The (old) C# compiler, as the VB compiler, was actually part of the .NET framework starting with .NET 2. Since .NET comes pre-installed on Windows, you also have a compiler. However, the new C# 6 compiler is no longer included in the framework, I think. When you install .NET 4.6 it even adds a notice to the old compiler's greeting message:

This compiler is provided as part of the Microsoft (R) .NET Framework, but only supports language versions up to C# 5, which is no longer the latest version. For compilers that support newer versions of the C# programming language, see http://go.microsoft.com/fwlink/?LinkID=533240.