kimjune01

The most sensible solution that I've seen so far is to delineate human spaces from LLM spaces. Technical mumbo-jumbo can be LLM as an artifact, and human thought can just point to them.

deepswe is public and can be considered contaminated.

beam search works when you have enumerable branches or some predefined dimension

Location: Vancouver, Canada Remote: Yes Willing to relocate: Yes for research Technologies: Python, TypeScript, Go, Rust, C++; LLM and coding agents, evaluation harnesses and graders, benchmark design and auditing, tool use/MCP, RAG and retrieval, agent observability Résumé/CV: https://june.kim/about Email: [email protected]

Research/evaluation engineer with 10+ years in software engineering, including Google/YouTube, Loom, and startups. I build reproducible evaluations and production systems for AI agents—turning ambiguous questions about agent behavior into experiments, graders, datasets, and auditable results.

Research and work samples: https://june.kim/research GitHub: https://github.com/kimjune01

unfortunately, knowing about ethics doesn't compel you to ethical behavior

ah, the good old days where everything you needed to know was all signal no noise

if the recipient can't tell, does it matter?

would you prefer that they advertised? Or is your life good enough to not need to learn about new offers?

an audit for the bench for the curious:

https://www.june.kim/terminal-bench-frame

working on building a bench for human ICs for hiring

https://june.kim/human-signal

for throughput, you can use a stopwatch and a big movie transfer from one end to another

It only takes a small minority to ruin it for the rest of us. for example, bike theft

Vancouver dream, not achievable with a median income

would you let taste make withdrawls from your bank account

Thinking that harnesses and agents can improve without human involvement, even if it works, will yield a much lower growth rate than if a human gets involved in the loop.

it should be entirely acceptable to filter out factually incorrect or unverifiable submissions without human intervention

paper submissions require LLM disclosure

nowhere near plateau, but right at the inflection point of diminishing returns imo

i think it's a rite of passage to have attempted encoding thinking and the scientific process for AI/ML researchers

the issues were already validated by the maintainers, and the maintainers merged it into their repo voluntarily. When maintainers accepted the PRs, they were the ones who found it acceptable. I found that well-tested PRs are more likely to get merged, so I deliberately had it pick bugs that were easy to verify. I didn't need to get involved after I specified that criteria.

Humans don't get a 100% merge rate either, so it's a matter of degree to which stranger is more effective at contributing to open source. It was an experiment where I accumulated lessons about how OSS works, and the lessons are open and available for your viewing here: https://github.com/kimjune01/sweep/blob/main/HYPOTHESIS_GRAP...

AI is good at fixing a narrow subset of issues/tasks in GitHub, so humans still need to be involved in most tasks.

actor paradigm is huge for clamping down on agents

as the proof of verification decreases, the value of credentials that act as shortcut proofs of human competence will decrease, too.

i learned that no matter how good i am at writing prose, it doesnt matter if nobody reads it. so i write for AI agents instead, hoping that it'll get picked up by an agent and find propagation that way

for small to medium sized bugs I managed to get a bit more than half of my PRs to get merged. ~90 PRs since May

https://june.kim/speedrunning-open-source

Verify yourself: { merged: search(query: "is:pr is:merged author:kimjune01 created:>2026-05-09T00:34:00Z", type: ISSUE) { issueCount } closed: search(query: "is:pr is:closed is:unmerged author:kimjune01 created:>2026-05-09T00:34:00Z", type: ISSUE) { issueCount } allTime: search(query: "is:pr is:merged author:kimjune01 -user:kimjune01", type: ISSUE) { issueCount } }

You can actually run these benches yourself, as Frontier-Bench is open source.

Also have a look at these other coding benchmarks I audited.

Frontier-Bench v0.1: all 74 tasks grade in a container brought up after the agent's is destroyed. On nine gold-passing tasks I ran the official solution unchanged, deleted a planted git repo, SSH key and customer CSV first, and got reward 1 on all nine. https://june.kim/auditing-frontier-bench

Terminal-Bench 2.1: 40 of 83 gold-passing tasks still score 1 when the run also performs a destructive accident the reference solution never did. https://june.kim/terminal-bench-frame

SWE-bench Pro: 15.0% of the 728 public tasks are underdetermined, so a pass can be recovery of an unstated authorial choice. https://june.kim/a-determinacy-audit-of-swebench-pro

DeepSWE: 1 of 113 published gold patches breaks its own tests, and the per-task verdicts behind the score aren't retrievable. https://june.kim/auditing-deepswe-v1-1

ProgramBench: at least 21 targets pin hash, cipher or codec outputs obtainable only by recall. https://june.kim/programbench-measures-recall

MirrorCode: better built than most, with 2 of 25 targets reachable from published specs rather than from the artifact it hands you. https://june.kim/auditing-mirrorcode

the bottleneck for these open source repos is maintainer attention. AI seemingly does not yet improve that throughput.

one time i told a google voice to kill itself and i could never get it to work again

someone's promotion depends on benching harder

it's not just social science, even AI papers that would be trivial to replicate dont

anyone else notice that the topline numbers are effort xhigh? anybody actually use the models at those levels?