News for hackers. Stuff that matters.
Re:Windows NT Kernel Contributor Explains Why Performance is... by kevingadd on Saturday May 11, 2013 @02:18AM (#5689875)
It's nice to see that internal developers feel the same way about XNA that external developers (who used to build XNA games, or still build XNA games) do.

From the outside I always assumed the constant flood of new, half-baked features instead of fixes and improvements to old ones was caused by interns and junior devs looking for glory - sad to hear that's actually partly true. I always considered frameworks like WPF (or Flex, for that matter) 'intern code' - not that interns necessarily wrote them, but they reek of not-experienced-enough engineers trying to solve problems by writing a bunch of new code, instead of fixing existing code.

It really is too bad, though. There are parts of the NT kernel (and even the Win32 API) that I consider a joy to use - I love IOCP, despite its warts, and APIs like MsgWaitForMultipleObjects are great tools for building higher-level primitives.

Plus, say what you want about GDI (there's a lot wrong with it at this point), but it's still a surprisingly efficient and flexible way to do 2D rendering, despite the fact that parts of it date back to before Windows 3.1. Some really smart people did some really good API design over time over at Microsoft...

Re:Windows NT Kernel Contributor Explains Why Performance is... by quotemstr on Saturday May 11, 2013 @02:30AM (#5689899)
Actually, I think one NT's largest advantages over POSIX systems is process management: yes, the venerable CreateProcess API.

See, in Windows, processes are first class kernel objects. You have handles (read: file descriptors) that refer to them. Processes have POSIX-style PIDs too, but you don't use a PID to manipulate a process the way you would with kill(2): you use a PID to open a handle to a process, then you manipulate the process using the handle.

This approach, at a stroke, solves all the wait, wait3, wait4, SIGCHLD, etc. problems that plague Unixish systems to this day. (Oh, and while you have a handle to a process open, its process ID won't be re-used.)

It's as if we live in a better, alternate universe where fork(2) returns a file descriptor.

You can wait on process handles (the handle becomes signaled and the wait completes when the process exits). You can perform this waiting using the same functions you use to wait on anything else, and you can use WaitForMultipleObjects as a kind of super-select to wait on anything.

If you want to wait on a socket, a process, and a global mutex and wake up when any of these things becomes available, you can do that. The Unix APIs for doing the same thing are a mess. Don't even get me started on SysV IPC.

Another thing I really like about NT is job objects (http://msdn.microsoft.com/en-us/library/windows/desktop/ms68...). They're a bit like cgroups, but a bit simpler (IMHO) to set up and use.

You can apply memory use, scheduling, UI, and other restrictions to processes in job objects. Most conveniently of all, you can arrange for the OS to kill everything in a job object if the last handle to that job dies --- the closest Linux has is PR_SET_PDEATHSIG, which needs to be set up individually for each child and which doesn't work for setuid children.

(Oh, and you can arrange for job objects to send notifications to IO completion ports.)

Yes, Windows gets a lot wrong, but it gets a lot right.

Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @04:42AM (#5690111)
> WaitForMultipleObjects as a kind of super-select to wait on anything

Except for the 64 handle limit, which makes it largely useless for anything that involves server applications where the number of handles grows with the number of clients. So then you'd spawn "worker" threads, each handling just 64 handles. And that's exactly where your code starts to go sour - you are forced to add fluff the sole purpose of which is to work around API limitations. What good is the super-select if I need cruft to actually use it in the app?

And don't get me started on the clusterfuck of the API that is IOCP.

> Yes, Windows gets a lot wrong, but it gets a lot right.

Oh, no, it doesn't.

Specifically, what Windows does not get is that it's other developers that are using its services and APIs, not just MSSQL team that can peek at the source code and cook up a magic combination of function arguments that does not return undocumented 0x8Fuck0ff. I have coded extensively for both Linux and Windows, including drivers and various kernel components, and while Windows may get things right at the conceptual level, using what they end up implemented as is an inferior and painful experience.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @06:32AM (#5690256)
> not just MSSQL team that can peek at the source code and cook up a magic combination of function arguments that does not return undocumented

Don't leave us hanging... what examples are you referring to?

Re:Windows NT Kernel Contributor Explains Why Performance is... by shubb on Saturday May 11, 2013 @07:14AM (#5690311)
One problem that the WINE project (windows emulator for unix)has is that windows is full of undocumented APIs, some of which might be really quite useful, which are used by windows internal developers.

For WINE, if you are going to write an emulator that runs Microsoft code (like the .net framework, or notepad.exe), you need to find the correct behavior of these undocumented APIs by trial and error.

For a developer on Windows, it is perhaps frustrating that MS identifies and solves problems with it's platform, but doesn't publicly release these solutions to you.

I work for some random company that makes an SDK, although I mostly make products using it. Many of our best features are hidden - we don't ship the header files with the SDK, but the symbols are there in the binaries. Someone doesn't think they are useful enough to justify the testing expense for a full release. I guess MS have the same thing.

Re:Windows NT Kernel Contributor Explains Why Performance is... by randomfool on Saturday May 11, 2013 @02:44PM (#5691620)
No. You can complain about insufficiently documented APIs, or internal APIs that you wish you could use but are not documented, but please do not complain about undocumented APIs that are used by Microsoft applications outside of Windows.

Ever since the monopoly rulings against Microsoft, they have had extremely strict internal controls preventing non-OS code from using undocumented APIs. If a DLL or EXE does not ship on the Windows client install disk, it cannot reference APIs which are not documented in MSDN. Full stop.

You're referencing an API that inadvertently didn't get documented, but everyone knows about it, has blogged about it and lots of articles mentioning to use it? Too bad. Either stop using the API or delay release by 3 months for the next MSDN update.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @02:07PM (#5691454)
> undocumented APIs

"undocumented APIs" != "a magic combination of function arguments"

Any function that's not documented can be called an undocumented API, any program has them -- Linux, Windows, whatever.

But you were talking about the argument combinations being undefined, not the functions themselves. What are some examples of that?

Re:Windows NT Kernel Contributor Explains Why Performance is... by shubb on Monday May 20, 2013 @06:53AM (#5736910)
Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Monday May 20, 2013 @10:55PM (#5741642)
What about them?
Re:Windows NT Kernel Contributor Explains Why Performance is... by mikevm on Saturday May 11, 2013 @07:52AM (#5690357)
>One problem that the WINE project (windows emulator for unix)

Wine Is Not an Emulator.

Re:Windows NT Kernel Contributor Explains Why Performance is... by fnordfnordfnord on Saturday May 11, 2013 @09:43AM (#5690560)
What shall we call WINE then? A "pretender"? I'm serious btw. I've never seen a good concise explanation of the way WINE works in comparison to say, an emulator. I have a fairly good idea myself, but I wouldn't want to have to give more than a very brief talk over it.
Re:Windows NT Kernel Contributor Explains Why Performance is... by gizmo686 on Saturday May 11, 2013 @10:19AM (#5690634)
From the WINE website: "Wine (originally an acronym for "Wine Is Not an Emulator") is a compatibility layer capable of running Windows applications on several POSIX-compliant operating systems, such as Linux, Mac OSX, & BSD. Instead of simulating internal Windows logic like a virtual machine or emulator, Wine translates Windows API calls into POSIX calls on-the-fly, eliminating the performance and memory penalties of other methods and allowing you to cleanly integrate Windows applications into your desktop." [1]

[1]http://www.winehq.org/about/

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @03:57PM (#5691896)
> What shall we call WINE then?

tl;dr: compatibility layer

Re:Windows NT Kernel Contributor Explains Why Performance is... by fnordfnordfnord on Saturday May 11, 2013 @12:17PM (#5691069)
I should have been more specific. How would you introduce WINE to freshmen, not CS students who might need to study WINE in detail, but well, here's a specific case, college freshmen who are expected to install WINE and then install and use software such as LTSpice, DipTrace, etc.? Or, more generally, how to explain it to neophytes who are expected to become moderately sophisticated computer users who might need to use WINE, but will probably never be software devs.
Re:Windows NT Kernel Contributor Explains Why Performance is... by HardlyYakka on Saturday May 11, 2013 @03:59PM (#5691899)
Something like:

WINE is a translator that allows Windows programs (those that have an .exe extension) to talk to other Operating Systems.

or perhaps:

Many people need wine if you want them to work. A windows program on Linux is no exception.

Re:Windows NT Kernel Contributor Explains Why Performance is... by justin66 on Saturday May 11, 2013 @05:05PM (#5692098)
> WINE is a translator that allows Windows programs (those that have an .exe extension) to talk to other Operating Systems.

I'm sure the aforementioned freshmen would like a no-bullshit explanation for why "emulator" hurts the WINE people's feelings but "translator" does not. It's always been a mystery to me.

Re:Windows NT Kernel Contributor Explains Why Performance is... by HardlyYakka on Saturday May 11, 2013 @06:12PM (#5692301)
> no-bullshit explanation

Hah! Not really possible, especially considering that the project used to be called windows emulator:

  http://www.faqs.org/faqs/windows-emulation/wine-faq/

I'd say they've changed the meaning so they can reinforce that it's not just another virtual machine ( and that it's faster than one).

Which is true, it doesn't emulate another hardware architecture's calls, just the kernel and OS magic needed.

In that regard, think of say a console game or arcade game emulator on a computer- that will have to emulate the whole architecture of said game system as well as the game.

Contrast that to a game program for the the x86 architecture, but not for windows- you could run it natively on windows providing you change everything the game asks for that's part of the Gamesystem380 system into something windows understands, and vice versa.

Re:Windows NT Kernel Contributor Explains Why Performance is... by plorkyeran on Saturday May 11, 2013 @10:20AM (#5690638)
A reverse-engineered implementation of the Win32 API?
Re:Windows NT Kernel Contributor Explains Why Performance is... by fnordfnordfnord on Saturday May 11, 2013 @12:20PM (#5691075)
Now we'll also have to explain what an API is, and then add some details about the Win32 API.
Re:Windows NT Kernel Contributor Explains Why Performance is... by illuminate on Saturday May 11, 2013 @12:02PM (#5691013)
Not pithy enough.
Re:Windows NT Kernel Contributor Explains Why Performance is... by asveikau on Saturday May 11, 2013 @01:06PM (#5691245)
It's a set of libraries, and a loader for EXEs and DLLs. That's the simplest, most literal way I can think of to explain it.
Re:Windows NT Kernel Contributor Explains Why Performance is... by shubb on Saturday May 11, 2013 @08:16AM (#5690396)
@Mike - Yes, that is true enough.
Re:Windows NT Kernel Contributor Explains Why Performance is... by acqq on Saturday May 11, 2013 @05:40AM (#5690200)
As far as I see you had a contact with Win API but you haven't understood enough, based on your complaints. Moreover, why do you mention "MySQL" as the team with the access to kernel sources?
Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @05:53AM (#5690210)
You see wrong. And I meant MSSQL, of course. Fixed.
Re:Windows NT Kernel Contributor Explains Why Performance is... by acqq on Saturday May 11, 2013 @07:33AM (#5690333)
Can you than please give a technical reason for you to insist on waiting on more than 64 threads in a single API call, having simply too many threads for good performance anyway when IO Completion Ports could be used instead?
Re:Windows NT Kernel Contributor Explains Why Performance is... by ambrop7 on Saturday May 11, 2013 @09:29AM (#5690531)
He probably doesn't have 64 threads, but he may have much more than 64 files or sockets. After all, the response was to the statement that "WaitForMultipleObjects as a kind of super-select to wait on anything". It clearly isn't, with this arbitrary limitation.

The problem with Windows is that there isn't any such super-select, unlike on Linux (see epoll, signalfd, timerfd...). You should take a look at my SO answer which lists six different ways to use sockets in Windows: http://stackoverflow.com/questions/11830839/when-using-iocp-...

Re:Windows NT Kernel Contributor Explains Why Performance is... by acqq on Saturday May 11, 2013 @12:09PM (#5691042)
Whatever he has more, waitfor is not the optimal way on Windows for that so it's still a complaint that round holes on Win32 are poor fit for his cubes. First choosing tools then complaining that they don't fit to some goal is typical for beginners.
Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @12:52PM (#5691194)
Do tell me, oh, enlightened one, what I should be using if I have 10K idle network connections and I can't really use IOCP, because I need keep things reasonably portable. That's not even considering that IOCP is a great example of premature optimization at cost of clarity and simplicity. It tries to solve a problem that I don't have and it doesn't offer a simpler solution that would just do. It is not a "hole" for my square peg, it's a freaking snowflake.

And expecting a modern OS to have an uncomplicated O(1) API for monitoring sockets is clearly too much to ask.

Re:Windows NT Kernel Contributor Explains Why Performance is... by ambrop7 on Saturday May 11, 2013 @02:06PM (#5691448)
It's possible to abstract IOCP and non-blocking sockets into a single interface with reasonable efficiency. For example:

- To send data, user calls Socket::send(const char *data, size_t length). This starts sending the given data; the buffer must remain available while data is being sent. IOCP implementation will initiate I/O using WSASend() or similar.

- When send operation is complete, the socket implementation calls a Done() callback of the user, and from that point on, the user is allowed to call Socket::send() again. With proper implementation, send() can directly be called from the callback.

This interface is very easy to implement on Linux. But on Windows there's a complication, because it's non-trivial to just stop an IOCP I/O operation that is in progress, in case you decide you don't need the socket any more and want to red rid of it NOW. If you just forget about it, there could be a crash when you release your buffer but Windows is still using it. A simple solution is to CancelIo() the socket and do blocking GetQueuedCompletionStatus()s until you get an event inidicating the completion of the pending I/O operation - taking care to queue up any unrelated results you may have gotten, for later processing. However, a more efficient solution is to use reference counting or similar with your buffers, so they only get released when Windows is done with them.

I didn't say it's easy, but it's possible.

Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @02:36PM (#5691579)
Yes, I know the abstraction is possible. In fact, I ended up implementing a nearly complete BSD socket API emulation on top of IOCP. Just don't lose the sight of the context, which is Windows does a lot of things right and my point being is that it doesn't.
Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @02:22PM (#5691515)
> The problem with Windows is that there isn't any such super-select, unlike on Linux (see epoll, signalfd, timerfd...)

> I can't really use IOCP, because I need keep things reasonably portable

LOL!

You outright dismiss the solution because it isn't "portable". What does "portable" mean here? Portable in a particular platform?!

Portable between what exactly? IOCP has been there since Windows NT 3.51 -- it's "portable" between the Windows-es.

Your epoll, signalfd, timerfd, etc. are Linux functions They're "portable" between the Linux-es.

i.e. The "portable" solutions you like are just as non-portable as the "non-portable" solutions you outright dismiss, making Microsoft's Windows designers look like idiots.

You're never going to find a solution while considering the Linux platform to be the "portable platform" and Windows to be the weird kid on the block.

Sadly, you're not the only person who thinks like this.

Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @03:18PM (#5691776)
A scalable variation of BSD socket API is supported by every major OS with a sole exception of Windows. It's an abstraction that is both simpler and more universal than IOCP model, not to mention that it also predates it by a decade.

So "LOL!" yourself, bud.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @03:44PM (#5691861)
> A scalable variation of BSD socket API is supported by every major OS with a sole exception of Windows

It's funny, you mention "epoll, signalfd, timerfd" as an example of a portable API, and then when I point out they're Linux APIs, you suddenly start talking about "a scalable variation" of the BSD sockets API. As for your "scalable variation of BSD sockets API", why don't you tell us the name so we know what you're referring to? Are you talking about Kqueue for example, or something else?

> an abstraction that is both simpler

I/O completion ports are just read-write queues... you ReadFile(), and when it's done, an item goes into the queue. (You can insert your own tasks into the queue, too, if you want.) Now just pop off the next item in the queue and off goes your thread. What part of that was so complex compared to your portable API (which for some reason you didn't feel was appropriate to mention the name of)?

> and more universal

Can't reply to it without knowing what "scalable variation" you're referring to, and why you didn't actually mention it as an example of portability when you were actually talking about the subject. Is portability actually a concern for you or no? (If it is, why did you bring up non-portable examples?!)

Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @05:43PM (#5692210)
> you mention "epoll, signalfd, timerfd"

It wasn't me who mentioned it.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @05:54PM (#5692242)
D'oh, sorry for the mixup. You were completely supporting that user so I didn't realize you were different people.

My comment remains essentially unchanged, though, with the same question as before:

What's this great portable API you're referring to?

Re:Windows NT Kernel Contributor Explains Why Performance is... by ams6110 on Saturday May 11, 2013 @01:39PM (#5691347)
Can you name any software that doesn't have an internal API that's not meant for use by 3rd parties?
Re:Windows NT Kernel Contributor Explains Why Performance is... by yuhong on Saturday May 11, 2013 @02:51AM (#5689946)
My personal annoyance is the introduction of win32k in NT4. With per-session CSRSS existing since WinFrame and Vista+ having Session 0 Isolation, not to mention much faster processors, and the removal of XPDM and the requirement of the DWM in Win8, it doesn't make as much sense anymore.

Update: And I forgot to mention the font parsing security issues (look up Duqu), and the issues with user mode callbacks too.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @06:44AM (#5690273)
> the issues with user mode callbacks too.

Genuine question (because I really don't know) -- how do Linux GUI frameworks work without kernel-mode callbacks? What do they use instead, when they need to send messages to other windows?

Re:Windows NT Kernel Contributor Explains Why Performance is... by qb45 on Saturday May 11, 2013 @07:27AM (#5690326)
X11 clients communicate with the server using Unix domain sockets or TCP sockets. Requests, responses and event notifications are exchanged as messages.

Asynchronous communication is achieved using syscalls like select and poll which take a set of file descriptors (files, special files, sockets) and block the calling thread until some descriptor from this set has new data available for reading, starts accepting writes or signals an error.

GUI toolkits repeatedly poll the X server descriptor, parse incoming messages and call application-defined callbacks.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @02:10PM (#5691474)
> poll and select

That sounds like it's basically the same thing as Windows (GetMessage or MsgWaitForMultipleObjectsEx anyone?)... only that now you don't have the ability to use SendMessage, and need to rely exclusively on PostMessage instead.

Is that correct?

Re:Windows NT Kernel Contributor Explains Why Performance is... by quotemstr on Monday May 13, 2013 @04:10AM (#5697661)
Between threads, you rely exclusively on PostMessage anyway. You can't directly call into a routine on a different thread, so SendMessage to a window owned by another thread just posts the message to a special queue that's serviced before the regular message queue, then blocks until the other thread's window procedure processes the message.

In all boils down to message-passing: in Windows, the win32k is the trusted third party that mediates interactions between different window-system clients. In the X11 case, the X server and the window manager work together to do the same job. The architecture are fundamentally similar.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Monday May 13, 2013 @05:33AM (#5697867)
Hmm... yeah, that does sound very similar.

Why does Windows need kernel-to-user callbacks in the first place then? Now I'm thinking it shouldn't even be necessary at all...

Re:Windows NT Kernel Contributor Explains Why Performance is... by quotemstr on Monday May 13, 2013 @10:04PM (#5702666)
Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Tuesday May 14, 2013 @12:25AM (#5703004)
Thanks, I took a look at that... but it doesn't seem to quite explain what I meant. I was thinking more like, there's no need for SendMessage() to go from user->kernel->user, even when it's on the same thread. It could just do a system call to get the wndproc for the given window, then call the wndproc from user-mode directly.

Wouldn't that completely avoid needing to call back into user-mode from kernel mode?

Re:Windows NT Kernel Contributor Explains Why Performance is... by yuhong on Thursday May 16, 2013 @08:52PM (#5722156)
It does this already when the window is on the same thread if no window hooks are installed. When the window is on a different thread, it is GetMessage or PeekMessage on the target thread that notices the sent message and does the user-mode callback to call the window procedure.
Re:Windows NT Kernel Contributor Explains Why Performance is... by quotemstr on Saturday May 11, 2013 @02:59AM (#5689962)
Nobody likes win32k, but it's there and it works.

Okay, you're in charge. Are you sure you can't find a better way to deploy shareholder capital than using it to remove win32k?

Re:Windows NT Kernel Contributor Explains Why Performance is... by Zigurd on Saturday May 11, 2013 @11:03AM (#5690788)
"Okay, you're in charge. Are you sure you can't find a better way to deploy shareholder capital than using it to remove win32k?"

Could that be why developing an OS inside of a shareholder-owned corporation is inherently hobbled compared with developing it as an open source project?

Re:Windows NT Kernel Contributor Explains Why Performance is... by kvb on Saturday May 11, 2013 @11:25AM (#5690877)
Open source projects have to prioritize resources, too. It's not like open source projects are known for their lack of cruft.
Re:Windows NT Kernel Contributor Explains Why Performance is... by Zigurd on Saturday May 11, 2013 @12:18PM (#5691070)
> It's not like open source projects are known for their lack of cruft.

True, but resource allocation is distributed between volunteer contributors, sponsored contributors who may have different sponsors, downstream consumers of a project, and people involved in project governance. That can make it easier to try new approaches.

Re:Windows NT Kernel Contributor Explains Why Performance is... by kvb on Saturday May 11, 2013 @03:46PM (#5691868)
I don't think commercial vs. open source really affects that much - regardless of open-source-ness, different projects are run in different ways. Just to give two examples, surely Office's switch to the Ribbon (and putting Office Apps online) was more speculative (or disruptive, or innovative, or whatever you want to call it) than anything being done by the open source office suites. Likewise, Visual Studio 2010 represented a big break from the past with a switch to many managed APIs driving their extensibility story (including the move to WPF for their UI), which was a huge change from their existing COM APIs.

Not all commercial projects are run like Windows, and it seems to me that most of their conservatism doesn't come from lazy or shoddy or shortsighted developers and managers, but rather their large number of hardware and software partners who take dependencies on undocumented implementation details (see Raymond Chen's blog for a virtually infinite supply of such anecdotes).

Re:Windows NT Kernel Contributor Explains Why Performance is... by enjolras on Saturday May 11, 2013 @08:07AM (#5690384)
Note that the problem you describe for process is not a fundamental design choice. For instance, FreeBSD has added such API.

http://www.freebsd.org/cgi/man.cgi?query=pdfork

Re:Windows NT Kernel Contributor Explains Why Performance is... by CountSessine on Saturday May 11, 2013 @06:01PM (#5692264)
I wonder if this is going to make it to OSX at some point, given the commonality between OSX and FreeBSD?
Re:Windows NT Kernel Contributor Explains Why Performance is... by astrange on Saturday May 11, 2013 @09:35PM (#5692784)
OS X has provided such APIs for a while - you can create processes using posix_spawn rather than fork, and monitor them with kqueues.

That said, if you're writing an app you might be able to go higher level and use XPC services, which can be much less pain.

Re:Windows NT Kernel Contributor Explains Why Performance is... by jcrites on Saturday May 11, 2013 @03:15AM (#5689992)
IO Completion Ports are a pretty nice feature too, in an of themselves.
Re:Windows NT Kernel Contributor Explains Why Performance is... by Maven911 on Saturday May 11, 2013 @03:12AM (#5689986)
Great explanation
Re:Windows NT Kernel Contributor Explains Why Performance is... by seanmcdirmid on Saturday May 11, 2013 @07:20AM (#5690319)
> I always considered frameworks like WPF (or Flex, for that matter) 'intern code' - not that interns necessarily wrote them, but they reek of not-experienced-enough engineers trying to solve problems by writing a bunch of new code, instead of fixing existing code.

This is unfair and unfounded accusation. If you look at the time frame, all the major platform were working on their first hardware accelerated UI toolkits and went through the similar teething problems (Cocoa anyone?). WinForms was a dead end, there was no fixing to do. WPF has turned out well enough, and WinRT has evolved into something very efficient (e.g. by using ref counting rather than GC).

> Plus, say what you want about GDI (there's a lot wrong with it at this point), but it's still a surprisingly efficient and flexible way to do 2D rendering

Not anymore. I get it that some love to use antique APIs and computer systems just for the sake of being retro, but when every computer ships these days with a GPU, using GDI is not even close to pragmatic.

Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Saturday May 11, 2013 @09:18AM (#5690509)
> ...WinRT has evolved into something very efficient (e.g. by using ref counting rather than GC).

Ah you mean by causing cache stalls when doing increments/decrements after each assignment and allowing for cyclic references?

Re:Windows NT Kernel Contributor Explains Why Performance is... by jjoonathan on Saturday May 11, 2013 @11:13AM (#5690826)
If the ref counting is manual or optimized, it doesn't happen after every assignment, only ones which change object ownership. If you're really nuts about avoiding cache hits, you can pack most ref counts into the low bits of the class pointer, since allocations tend to be 16-byte aligned, including allocations for class objects. Assuming you use the object at least once following a change of ownership, the overall cost in cache misses becomes 0. Or you can store the references in a global table with better cache properties (IIRC this is what ObjC does). Or you can rely on the fact that cache lines are typically large enough to grab a few words at a time, so fetching the class pointer will automatically fetch the refcount (again with the assumption that you use objects at least once per assignment). Honestly, I'm having difficulty imagining how cache behavior could become a problem unless you wanted it to.

As for not allowing cycles, I consider that a feature. The headache of memory management doesn't go away with GC. You still have to avoid inadvertently keeping references to objects through the undo stack & so on. Unintentional strong refs creep through GC code just as easily as memory leaks creep through code with manual memory management. Almost universally, I find that GC'd projects large enough to have memory management best practices implicitly do away with this freedom at the first opportunity by calling for acyclic or single-parent object ownership graphs. These restrictions primarily make it easier to think about object lifecycle -- the fact that they allow refcounting to suffice for memory management is icing on the cake.

Re:Windows NT Kernel Contributor Explains Why Performance is... by pcwalton on Saturday May 11, 2013 @02:58PM (#5691684)
> If the ref counting is manual or optimized, it doesn't happen after every assignment, only ones which change object ownership.

The Rust compiler does this. Even so, 19% of the binary size in rustc is adjusting reference counts.

I am not exaggerating this. One-fifth of the code in the binary is sitting there wasted adjusting reference counts. This is much of the reason we're moving to tracing garbage collection.

All of those optimizations that you're talking about (for example, packing reference counts into the class pointer) will make adjusting reference counts take even more code size.

> As for not allowing cycles, I consider that a feature. The headache of memory management doesn't go away with GC.

The fact that memory management is still an issue doesn't mean that we shouldn't make it work as well as we can.

In large software projects, cycles are a persistent issue. Gecko had to add a backup cycle collector over its reference counting framework, and observed memory usage decreased significantly once it was implemented.

Re:Windows NT Kernel Contributor Explains Why Performance is... by zooko_LeastAuth on Saturday May 11, 2013 @08:18PM (#5692636)
That's the evolution Python went through as well: reference counting, then (in CPython 2.0) an adjunct cycle-collector, then (in PyPy) a non-reference-counting garbage collector.

Hopefully in future versions of PyPy an even better garbage collector. :-)

Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Saturday May 11, 2013 @05:57PM (#5692254)
Very nice explanation, except you are forgetting WinRT is actually an extension of COM.

This means each increment/decrement is a virtual method call with the corresponding invocation costs.

This is why the .NET runtime caches the WinRT objects, and even re-uses them instead of making a 1:1 use like C++/CX does.

Re:Windows NT Kernel Contributor Explains Why Performance is... by seanmcdirmid on Saturday May 11, 2013 @09:39AM (#5690547)
The caching misbehavior of referencing counting has been greatly exaggerated, especially in the context of UI where responsiveness is much more important than raw CPU speed. Also, the ref counting tradeoff seems to work better for device (e.g. all the cool kids [1] are doing it).

[1] https://developer.apple.com/library/mac/documentation/Genera...

.NET still does GC, it is only the WinRT APIs (something like COM) that manage resources through ref counting. There is some cool interop magic that makes this somewhat transparent to the programmer.

Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Saturday May 11, 2013 @10:37AM (#5690696)
> The caching misbehavior of referencing counting has been greatly exaggerated, especially in the context of UI where responsiveness is much more important than raw CPU speed. Also, the ref counting tradeoff seems to work better for device (e.g. all the cool kids [1] are doing it).

Apple's ARC is a compiler hack. It only works if you happen to use set of Cocoa libraries that are recognized by the compiler and allow for some magic incantation.

You cannot make any Objective-C library ARC aware.

ARC is a good solution for Objective-C, because Apple never managed to have their GC working safely in all situations, given the underlying C type system.

> .NET still does GC, it is only the WinRT APIs (something like COM) that manage resources through ref counting. There is some cool interop magic that makes this somewhat transparent to the programmer.

True, but C++ does not and you still have the performance penalty of reference counting.

The only way to have reference counting with good performance is do what Parasol or Objective-C do, by removing unnecessary increment/decrement operations thanks to dataflow analysis.

Which is something that you cannot do just by using the WinRT runtime and need to rely on the cleverness of the compilers.

Actually, are you aware that reference counting tends to be presented on the first chapter of many CS books about garbage collection as poor man's GC?

Re:Windows NT Kernel Contributor Explains Why Performance is... by gurkendoktor on Saturday May 11, 2013 @11:19AM (#5690854)
> Apple's ARC is a compiler hack. It only works if you happen to use set of Cocoa libraries that are recognized by the compiler

I don't understand. How does ARC rely on more than what NSObject provides? I'm coming from the C++ shared_ptr world and ARC does not feel much different so far (less explicit, but easier to optimise).

Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Saturday May 11, 2013 @06:04PM (#5692273)
No, it is an hack.

Just read this document.

https://developer.apple.com/library/mac/#releasenotes/Object...

You need to write code with lots of special cases to be able to have ARC active and not fall into strange compiler errors.

Re:Windows NT Kernel Contributor Explains Why Performance is... by astrange on Saturday May 11, 2013 @09:30PM (#5692775)
The compiler errors are actually quite clear.

Most of the restrictions (bridge casts and naming methods) are interoperation with non-ARC code, and from working on Java/JNI I can assure you GC<>manual interaction was worse.

It is of course still quite possible to get abandoned memory through retain cycles.

Re:Windows NT Kernel Contributor Explains Why Performance is... by mikeash on Saturday May 11, 2013 @10:29PM (#5692879)
What part of that is supposed to explain your rather bizarre claim that ARC only works with special Apple libraries?
Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Sunday May 12, 2013 @01:30AM (#5693268)
> The compiler understands Objective-C methods that return Core Foundation types follow the historical Cocoa naming conventions
Re:Windows NT Kernel Contributor Explains Why Performance is... by mikeash on Sunday May 12, 2013 @09:40AM (#5694169)
That's very much an uncommon special case, though. You don't need that to use ARC, and rarely come across it.
Re:Windows NT Kernel Contributor Explains Why Performance is... by mikeash on Saturday May 11, 2013 @12:22PM (#5691082)
It doesn't even rely on NSObject. Any root class that provides a compliant implementation of retain, release, and autorelease will work with ARC.
Re:Windows NT Kernel Contributor Explains Why Performance is... by seanmcdirmid on Saturday May 11, 2013 @07:42PM (#5692564)
> True, but C++ does not and you still have the performance penalty of reference counting.

Not very relevant. For UI code, responsive resource allocation and de-allocation is much more important than saving a few % CPU cycles.

> Actually, are you aware that reference counting tends to be presented on the first chapter of many CS books about garbage collection as poor man's GC?

Yes! Sometimes the conventional wisdom doesn't hold in the end, or it doesn't hold for all use cases. But I should have been more clear that ref counting is GC, whereas what I refer to as GC involves sweeping reference roots periodically to identify garbage.

Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Sunday May 12, 2013 @01:45AM (#5693293)
> > True, but C++ does not and you still have the performance penalty of reference counting.

> Not very relevant. For UI code, responsive resource allocation and de-allocation is much more important than saving a few % CPU cycles.

I beg to differ. I consider missile control radar systems of military applications a bit more important than UIs.

There are military applications using real time GC systems, which I can list here if you want.

I also have a German magazine here (Making Games Magazin 1/13) that explains how Witcher2 for the Xbox 360 makes use of a GC.

Anyway one thing I do agree, automatic memory management, regardless of the form, should be the any of any programming language.

Pure manual memory management should be treated like Assembly. It will never go away and should only be used for the very few cases where there is no other way around the problem at hand.

Re:Windows NT Kernel Contributor Explains Why Performance is... by seanmcdirmid on Sunday May 12, 2013 @08:10PM (#5696482)
I'm not anti-GC, I just think that reference counting is appropriate for UI code. .NET still supports GC, and the collector is very good compared to our competitors; I'm not saying we should give that up!

I should have mentioned that the primary reason WinRT seems to use reference counting is for interop reasons: its much easier to great a UI toolkit that is programmable in both C# and C++. Note that WPF never had a C++ interface, while WinRT has one without the need for any bridge code.

Reference counting has some interesting applications where it is more appropriate trace-based GC. Heck, sometimes we want old-fashioned arenas (Pascal!), it would be nice if our languages could give us some choices.

Manual memory management is annoying but sometimes necessary. I can see why native programmers don't want to give it up as it constitutes the primary performance difference these days between native and managed languages.

Re:Windows NT Kernel Contributor Explains Why Performance is... by kevingadd on Saturday May 11, 2013 @07:33AM (#5690332)
GDI has been GPU accelerated literally forever. Vista may have dropped hardware acceleration for GDI, but it was promptly brought back in Windows 7.

Since the Win32 UI stack uses GDI, it was hardware accelerated before WPF even existed.

Re:Windows NT Kernel Contributor Explains Why Performance is... by seanmcdirmid on Saturday May 11, 2013 @08:02AM (#5690376)
Not the same thing! This is like comparing pre-DX8/programmable shader hardware acceleration to the fixed-function crud we had to deal with a long time ago. Ya, you can emulate fixed functions with programmable shaders, but its a poor way to make use of a modern GPU.

Here is a good discussion of the topic:

http://msdn.microsoft.com/en-us/library/windows/desktop/ff72...

Excerpts:

> When the GDI DDI was first defined, most display acceleration hardware targeted the GDI primitives. Over time, more and more emphasis was placed on 3D game acceleration and less on application acceleration. As a consequence the BitBlt API was hardware accelerated and most other GDI operations were not.

> In order to maintain compatibility, GDI performs a large part of its rendering to aperture memory using the CPU. In contrast, Direct2D translates its APIs calls into Direct3D primitives and drawing operations. The result is then rendered on the GPU. Some of GDI?s rendering is performed on the GPU when the aperture memory is copied to the video memory surface representing the GDI window.

> Existing GDI code will continue to work well under Windows 7. However, when writing new graphics rendering code, Direct2D should be considered, as it takes better advantage of modern GPUs.

Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Saturday May 11, 2013 @04:21AM (#5690083)
The "Inside Windows Kernel" book series are quite interesting to understand on how it all works, and even some of the initial VMS influence in the original kernel design.
Re:Windows NT Kernel Contributor Explains Why Performance is... by wsc981 on Saturday May 11, 2013 @04:55PM (#5692073)
> "It's nice to see that internal developers feel the same way about XNA that external developers (who used to build XNA games, or still build XNA games) do. From the outside I always assumed the constant flood of new, half-baked features instead of fixes and improvements to old ones was caused by interns and junior devs looking for glory ..."

Am I understanding you correct; are you implying that you think XNA was created by juniors? Would you say that because of this, it's a good thing XNA is being killed?

Cause personally I think it's been a terrible decision by Microsoft to kill XNA. A lot of indie game developers have relied on XNA and I really feel Microsoft can use the indie support. Sure, big name games might be a priority, but personally I feel most _interesting_ work is being done by indies. Indies tend to be less concerned with proven formulas and seem to see it more of a creative outlet for themselves[1]. I think it's a good thing frameworks like Monogame[2] exist, so developers can still put their existing XNA knowledge to good use - and not just limited to the Windows platform, but e.g. iOS and Android as well.

The Monogame website might not show very impressing examples, but a game like Bastion[3] was ported to other platforms using Monogame, showing very high-quality work can be created with XNA.

[1]: http://www.youtube.com/watch?v=GhaT78i1x2M

[2]: http://monogame.codeplex.com

[3]: http://supergiantgames.com/index.php/2012/08/bastions-open-s...

Re:Windows NT Kernel Contributor Explains Why Performance is... by jcrites on Saturday May 11, 2013 @03:23AM (#5690006)
What is the feeling about XNA? I haven't followed the area, so I found it unfortunate that it was treated in the original post without explanation. Was XNA a good thing or a bad thing? Why?
Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Saturday May 11, 2013 @04:27AM (#5690091)
It was loved by Indies for XBox 360 development, got so popular, Microsoft decided to make it the way to develop games on the Windows Phone 7.

When Windows Phone 8 came out, the native renaissance was full speed inside Microsoft, C++ with DirectX became the official way and XNA was canned while Microsoft left MonoGame and Unity pick up the C# developers still willing to invest into the platform.

The sad thing is that this is not a native vs managed issue, because even C# gets compiled to native code when targeting Windows Phone 8. Just plain product management decision to kill the product, which was being sold as the way to develop Windows Phone games.

Re:Windows NT Kernel Contributor Explains Why Performance is... by illuminate on Saturday May 11, 2013 @12:05PM (#5691025)
"XNA was canned while Microsoft left MonoGame and Unity pick up the C# developers still willing to invest into the platform"

I have no insider information to suggest otherwise, but don't you think that a replacement will be released with the new console?

Re:Windows NT Kernel Contributor Explains Why Performance is... by bentcorner on Saturday May 11, 2013 @01:06PM (#5691242)
My gut feeling is that even if they do, indie developers aren't going to fall for that one again.
Re:Windows NT Kernel Contributor Explains Why Performance is... by ams6110 on Saturday May 11, 2013 @01:48PM (#5691371)
What choice will they have?
Re:Windows NT Kernel Contributor Explains Why Performance is... by cpleppert on Saturday May 11, 2013 @01:47PM (#5691369)
No because microsoft isn't going to support a toolkit it has no intention of using on mobile or the desktop and will never be used by commercial game developers.
Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Saturday May 11, 2013 @06:06PM (#5692278)
Are you aware there was a Premium version of XNA for AAA studios?
Re:Windows NT Kernel Contributor Explains Why Performance is... by eropple on Sunday May 12, 2013 @12:35AM (#5693166)
As far as I know, the XDK Extensions were intended for XBLA developers, not "AAA studios".
Re:Windows NT Kernel Contributor Explains Why Performance is... by pjmlp on Sunday May 12, 2013 @01:50AM (#5693306)
It was announced in a different way at a Meltdown or PDC conference, around 2008 if I am not mistaken.

Sadly those videos seem to no longer be online after MSDN was redone.

Maybe someone can correct me if I am wrong.

Re:Windows NT Kernel Contributor Explains Why Performance is... by illuminate on Saturday May 11, 2013 @04:51PM (#5692063)
"a toolkit it has no intention of using on mobile or the desktop"

Right, I find it likely that XNA was killed to work on a better toolkit for all three.