This is great! It's so hard to find bugs in software tools because the conventional wisdom is that "it's never a bug in the compiler."

Interesting that you had to create two patches for python 2.x and 3.x. (Frustrating, no?)

Seems like you were able to get this submitted and fixed fairly quickly; the Python guys seem like a welcoming group.

Has anyone else had experience contributing to this particular project?

FWIW, In my experience, the Python community has consistently been full of high-quality, very welcoming people. I don't understand why different parts of the open source world vary in such things, but they do seem to (in my limited experience) and the Python world is consistently awesome.

Your mileage may vary, of course...

(and, also fwiw, ned batchelder has a reputation for exemplifying that spirit – just trying to give credit where it's due.)

I had to make two patches because the testing style has changed slightly between the two, but also, it's guaranteed to apply better if I make them against the tip of each version.

I contributed a tiny fix for a bug in Tkinter, Python's built in GUI module. There's a contributor form to sign before your first contribution can be accepted, but everyone was friendly, and it was committed quickly once I'd returned the form.

"I discovered this working on code that grades student submissions at edX"

It sounds to me like the guy who submitted 40k spaces is the one who discovered the bug ;). Congrats on getting your patch accepted though!

This is slightly off-topic but I always wondered: why are commits in the CPython repo have their username changed from the author of the change to the person who committed it?

Completely guessing here, but it might be a tool thing. For example it's roughly what happens when I merge your code with git and then push it to subversion with git-svn.

What you describe is a probable scenario, but I think the real reason is in the patch process. Contributions in python don't appear to happen pull request style ("here's my repo with my changes, please merge that in"), but rather through submitting patches ("here's a .patch file with my changes, please apply that to tip").

So what happens isn't that someone with write access merges in changes (which, unless tools get in the way, should preserve authorship), but applies the patch - which attributes that code change to the person applying the patch.

In git the patch process is the same but the original author is retained. Git stores the author separately from the committer.

Does anyone know if hg has a similar author vs committer distinction? If so, perhaps the tool is displaying committers by default?

Looking into it today, it seems what a patch looks like depends on the tool. Git by default (using a command like git format-patch) adds a From: header value that contains the original author. Mercurial (using a command like hg export) includes the original author, but it shows up in commented lines.

However, looking at the devguide (http://docs.python.org/devguide/patch.html), they tell you to run hg diff, which creates a barebones patch file with no author metadata.

Contributions to hg are working the same but the author isn't changed, so this is presumably some policy specific to CPython.

I wonder if PyPy has a similar bug - since PyPy can do a much better job at sandboxing Python programs, it might be especially important how it handles tokenization of those programs.

Whoa, why the negativity? Dude found a bug in CPython, fixed it, and had his patch accepted. Regardless of whether he fixed the bug that you wanted fixed or not, it's always good to have more eyes and hands on a project, especially if people are getting involved for the first time.

OP did have a patch submitted and merged into CPython right? You are talking as though all the OP did was to add a strip in the program.

I know you from irc! good work sir!

link bait. actual title: i fixed an obscure bug in the cpython tokenizer.

I apologize for the "link bait." I didn't intend it as such, I was actually making fun of my "accomplishment," having contributed a tiny patch that fixed an obscure problem.

I can see how it reads very differently on Hacker News...

Screw it— you have nothing to apologize for. It is because of people like you— fixing bugs big an small— that Python is a viable for big projects.

And to the dude who put it on his resume— good move. This helps a ton when someone is screening resumes.

You have nothing to apologize for!

FWIW Ned, I had to update my (generally lame) résumé recently and was a bit embarrassed to put CPython on my list of projects that I've contributed to due to the same type of thing: it was a small patch that fixed an obscure issue.

Anyways, I did it anyhow (with the caveat to make me feel better :P) but I felt the same way about the "accomplishment" :). So congrats!

(This is tos9 by the way).

Sorry for the negative comment (I'm about to delete). I missed the tongue-in-cheek nature of your posting and as others pointed out. It was a real bug.

meh, forget the haters. congrats on your bugfix, I'd be pretty excited about contributing to CPython, too!

Don't be silly. I have 2 super-trivial patches in the Linux kernel tree (one to fix an array-indexing bug, and another to blacklist a feature on a piece of broken hardware). They're simple patches, but I'm nonetheless really happy to have contributed to such a giant project. You shouldn't apologize for feeling the same and wanting to post to HN about it!

Worry not, your title is fine.

Which is what you would expect, unless the OP was Guido, wouldn't you?

If my patch, how much so ever minuscule, was merged into CPython, I would be similarly elated as well.