Fundamentally it's hard to reconcile both worlds though. Building non-trivial multi-module projects on the JVM is inherently complex especially when you throw in multiple build targets, multiple toolchains, multiple platforms...
With simpler build tools (like in Go or Rust) you shift this complexity elsewhere, typically in a Makefile and/or a Docker/OCI based build pipeline, and these can get pretty complex too. Let alone distributed build tools like Bazel.
With scala-cli, there's not even a need to download a Java runtime or a language distribution. You can let the runner do its thing, or pass options to choose the JVM and the language version to use, or even write those options into special headers in the code files. You can also write tests, format code... it's all built-in. And in cases the code outgrows the tool and there's a need to migrate to a different build tool, there's even a feature to export the build to Sbt or Mill.
How do you figure out dependencies? Import statements in .java files give you the packages to import, but those package names could be provided by one or more .jar files and, regardless, the package names need not bear any relation to the jar name or its group/artifact IDs (if pulling from e.g. a maven-style repository, which basically everyone does).
For multi-module projects, how do you figure out the dependencies between the modules, even? Sure, you could probably figure that out by parsing all the .java files in all modules and figuring out what they provide and import, but that would be slower than maven, probably.
You could certainly do this for small, dependency-free programs, but it would be such a niche use case that I don't think it would be worth the time.
Dependencies? It's okay to use a dependency list file for these - I guess I don't consider this config compared to the stuff I usually find in a Gradle or Maven file. The thing I'm alergic to is all the stuff that isn't a dependency list.
In Go, these go in a go.module file that's automatically updated by the build tooling, which runs instantly & has a cache you never need to think about. Go has the advantage of import paths being URLs that specify the dependency too, but I think my Go-For-Java tool would use reverse package import search from an online service to map eg com.foo.bar.something.Potato to the appropriate package "foo-bar" from Maven Central or whatever. Building that index seems like a trivial program to write for the average Java engineer.
The more I think about this "go for java" idea, the more I want to build it "in anger" just to see how off-base I am. Maybe I really am just going to re-implement or wrap sbt, mill, gradle, https://www.jbang.dev/ idk. It just feels like the experience could be an order of magnitude simpler as an end-user with some conventions strictly enforced by the tooling.