Yep, the permit is tied to a specific job for the first two years[1]. From what I understand, it's pretty easy to get approval for a different job, as long as the job description is similar. Interestingly, my residence permit also says "selbständig Tätigkeit gestattet" (self-employment permitted), so it seems I could freelance here without needing a different visa.

[1] http://www.berlin.de/labo/auslaender/dienstleistungen/index....