mcyc

Ah, too bad we never crossed paths!

Thanks for your comments here. I agree that a lot of labs were pretty unserious. One other thing that I should have mentioned in the article is that labs in Japan are like little contracting businesses, and the professors have to find funding in a way that seems much harder than the US. This leads to a lot of variability in the financial situations of labs, so yeah my concerns about funding/etc were probably understated (and my lab was likely one of the best funded in the nation).

As for domestic conferences, typically my lab used them as staging for their papers that they would send to top international conferences, so the quality was still pretty good. I do get the sense that the quality outside of the T1 labs drops off pretty quickly.

> ... but I never collaborated with them (in fact, our lab has hardly ever published a paper with >2 authors).

This was something I had in the original draft of the article, but it was removed for being a bit niche. TiTech (and perhaps other schools, but not universal) had a restriction in place where only one student could claim primary authorship on a paper for the purposes of their graduation requirements (typically 1 conference + 1 journal paper). Even if two students contributed equally, they had to sign a form designating one of them the "true" primary author. This massively discourages collaboration. I absolutely hated this policy, but there was basically nothing I could do at an institutional level to change it. I ended up being able to collaborate pretty freely, since I reached my publication requirements quickly, but this policy really hurts students who do their PhD in Japan just because of the numbers game. If two candidates are up for a position, and one has N papers (all first author, but 2-3 authors max) and the other has N + 10 papers (with <N first author, but all with huge groups), the one with N papers is at a disadvantage at first glance. And it just removes one of the primary skills you need to cultivate during a PhD: collaborating with other researchers.

Hmm, the tuition is extremely low relative to the US. On the order or $4-5k USD/year. I am not sure how many additional fees (like facilities, etc.) there are, but I guess not many. Being self funded was common for Japanese students. AFAIK no international students that I knew were self funded, up to the tail of their graduate program when their MEXT ran out.

Now that I think about it, one thing I am not sure about is if self-funded students' travel/conference fees were covered. I would guess they are covered through a grant and it is just the tuition that is self funded, but I do not know for certain. Going to >1 international conference a year would be far more than the tuition.

So, I would guess it would cost ~10k/year to be fully self funded. You can recoup some of that cost working (on a student visa this is limited though), and your lab may also have an RA stipend (but this is both very low [<$20/hr] and counts towards your 28 hour/week limit).

My advice though is to never do a self-funded PhD. Being a PhD "student" is a misnomer. You are a research employee and should be compensated as such.

Yeah, it is definitely a harder limit for international students for the reason you stated (but this article is geared towards foreigners after all).

What field were you in? I am surprised to hear that you think Japan's lab culture is much better than the US. I do think it varies a lot by lab, but overall the internal collaboration and guidance (if not from the PI, from senior members) was much better in American labs than in Japanese labs in my experience.

Thanks!

My decision to start my PhD in Korea was under very different circumstances. I was studying a totally different field (automata theory, so closer to mathematics than ML) where longer PhDs were normal, I was used to the US system, I wasn't working or married, etc. I expected it to take ~6 years when I started.

I basically had to choose between living in the Canadian countryside, Helsinki, or Seoul in order to be in a lab studying what I wanted. I was already pretty into Korean culture (I minored in Korean in college, studied abroad there, and was a serious StarCraft player) so that choice was easy!

When I started looking again, I knew I wanted a shorter PhD (especially since I had a masters and industry + industry research experience), so Japan was pretty attractive.

I will admit that I think 3 years is a little short to make substantive progress outside of machine learning, so unless you come in with a lot of research experience and a good idea of what you want to work on, I think a longer PhD would be beneficial, but the opportunity cost is so high that it really doesn't seem like a great option on paper.

Thank you! And, re: 3 years, that's good to know, I should have been a bit more precise.

What field were you supervising?

3 years in CS is, imo, a little tight but not bad. But I always wondered how people in other fields (stem and otherwise) did it. CS simply doesn't have to deal with the extraneous factors that come with, e.g., biology or climatology.

I am the author of this. Happy to discuss!

One thing to note is that, after I submitted the final draft of this article, the Japanese government ended the MEXT university track or is at least planning to. There was a memo, but the status is not clear.

Anyway, I had a great time doing my PhD this way, and generally I recommend that people work before a PhD and try to work during it (if you are in CS). It gives you a lot of freedom and, at least for me, it is very helpful to have more than one pipeline of work at a time so that when one slows down (as it inevitably does in research), you can draw on the other.

Lichess has a checkmate captcha that I think is cute.

It requires you to solve a mate-in-one puzzle to, e.g., post on the forums.

(Sorry, don't have a better link, there wasn't any non-technical I could find about it).

https://www.reddit.com/r/chess/comments/q19wgq/til_lichess_d...

Yeah, it is interesting to me that it is coming from the _city_'s government. I've seen sovereign AI things at the country level, but this is the first municipal one I have seen.

You are right about most tokenizers being heavily biased towards English, but the situation is not so bad for Portuguese. Here are some results on the Goldfish corpus [1] with a few different tokenizers. This measures #characters in corpus / #subwords in tokenized corpus.

```

Llama3

english, 0.216

portuguese, 0.285

italian, 0.287

greek, 0.592

```

```

Gemma4

english, 0.219

portuguese, 0.246

italian, 0.249

greek, 0.537

```

```

Kimi2.6

english, 0.214

portuguese, 0.310

italian, 0.308

greek, 0.716

```

Portuguese is worse than English certainly, but it is on par with Italian (which I think has more overlap with English) and much better than Greek (since it doesn't use the Latin script and is definitely not prioritized in the tokenizer construction).

On your second point, tokenizer transfer allows for extending/modifying a tokenizer without retraining the model from scratch. The simplest version of this is tokenizer extension + continual pretraining, where you just add a bunch more tokens to the vocab for the language/domain that you want to improve and train a little more. It's been done for Japanese [2] and Indic languages, but afaik not Portuguese.

So I think that continual pretraining for a large base model would have probably been fine for this case with huge cost savings. But it is good to have the ability to train your own base models, so I don't think this is such a bad idea.

-----------------------

[1]: https://huggingface.co/datasets/goldfish-models/fish-food

[2]: https://arxiv.org/abs/2404.17790

This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested:

Some libraries:

- Outlines, a nice library for structured generation

  - https://github.com/dottxt-ai/outlines

- Guidance (already covered by FlyingLawnmower in this thread), another nice library

  - https://github.com/guidance-ai/guidance

- XGrammar, a less-featureful but really well optimized constrained generation library

  - https://github.com/mlc-ai/xgrammar

  - This one has a lot of cool technical aspects that make it an interesting project

Some papers:

- Efficient Guided Generation for Large Language Models

  - By the outlines authors, probably the first real LLM constrained generation paper

  - https://arxiv.org/abs/2307.09702

- Automata-based constraints for language model decoding

  - A much more technical paper about constrained generation and implementation

  - https://arxiv.org/abs/2407.08103

- Pitfalls, Subtleties, and Techniques in Automata-Based Subword-Level Constrained Generation

  - A bit of self-promotion. We show where constrained generation can go wrong and discuss some techniques for the practitioner

  - https://openreview.net/pdf?id=DFybOGeGDS

Some blog posts:

- Fast, High-Fidelity LLM Decoding with Regex Constraints

  - Discusses adhering to the canonical tokenization (i.e., not just the constraint, but also what would be produced by the tokenizer)

  - https://vivien000.github.io/blog/journal/llm-decoding-with-regex-constraints.html

- Coalescence: making LLM inference 5x faster

  - Also from the outlines team

  - This is about skipping inference during constrained generation if you know there is only one valid token (common in the canonical tokenization setting)

  - https://blog.dottxt.ai/coalescence.html

This is a nice attitude. I think HN is overall pretty nice for geeking out and also hearing other people geek out, but there is still a strain of elitism (not like StackExchange thankfully) and so I'm happy to see comments like this.

You can cross whitespace boundaries by setting flag `--split-on-whitespace` to false (it's true by default).

https://github.com/google/sentencepiece/blob/master/doc/opti...

Just a minor nit: SentencePiece is a library, not a tokenization algorithm. It implements two tokenization algorithms, Unigram and BPE.

BPE builds vocabularies from the base up so I assume you are talking about Unigram which starts with a big vocabulary and trims it.

The details of UnigramLM are here https://arxiv.org/pdf/1804.10959, and the part about vocabulary seeding is Section 3.2.

Basically, it just selects all substrings that appear in the corpus up to a certain length (and then maybe trims it a little by discarding rare substrings or something to reduce the initial size a bit and make things faster).

People may also be interested in Pynini [1], a python wrapper (+ a lot of additional ease-of-use functionality) of OpenFst [2] (a really great library for transducers).

There are some good tutorials in the form of homework assignments (from like Johns Hopkins and some others) that go through Pynini use cases.

[1] https://www.openfst.org/twiki/bin/view/GRM/Pynini

[2] https://www.openfst.org/