News for hackers. Stuff that matters.
Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @04:42AM (#5690111)
> WaitForMultipleObjects as a kind of super-select to wait on anything

Except for the 64 handle limit, which makes it largely useless for anything that involves server applications where the number of handles grows with the number of clients. So then you'd spawn "worker" threads, each handling just 64 handles. And that's exactly where your code starts to go sour - you are forced to add fluff the sole purpose of which is to work around API limitations. What good is the super-select if I need cruft to actually use it in the app?

And don't get me started on the clusterfuck of the API that is IOCP.

> Yes, Windows gets a lot wrong, but it gets a lot right.

Oh, no, it doesn't.

Specifically, what Windows does not get is that it's other developers that are using its services and APIs, not just MSSQL team that can peek at the source code and cook up a magic combination of function arguments that does not return undocumented 0x8Fuck0ff. I have coded extensively for both Linux and Windows, including drivers and various kernel components, and while Windows may get things right at the conceptual level, using what they end up implemented as is an inferior and painful experience.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @06:32AM (#5690256)
> not just MSSQL team that can peek at the source code and cook up a magic combination of function arguments that does not return undocumented

Don't leave us hanging... what examples are you referring to?

Re:Windows NT Kernel Contributor Explains Why Performance is... by shubb on Saturday May 11, 2013 @07:14AM (#5690311)
One problem that the WINE project (windows emulator for unix)has is that windows is full of undocumented APIs, some of which might be really quite useful, which are used by windows internal developers.

For WINE, if you are going to write an emulator that runs Microsoft code (like the .net framework, or notepad.exe), you need to find the correct behavior of these undocumented APIs by trial and error.

For a developer on Windows, it is perhaps frustrating that MS identifies and solves problems with it's platform, but doesn't publicly release these solutions to you.

I work for some random company that makes an SDK, although I mostly make products using it. Many of our best features are hidden - we don't ship the header files with the SDK, but the symbols are there in the binaries. Someone doesn't think they are useful enough to justify the testing expense for a full release. I guess MS have the same thing.

Re:Windows NT Kernel Contributor Explains Why Performance is... by randomfool on Saturday May 11, 2013 @02:44PM (#5691620)
No. You can complain about insufficiently documented APIs, or internal APIs that you wish you could use but are not documented, but please do not complain about undocumented APIs that are used by Microsoft applications outside of Windows.

Ever since the monopoly rulings against Microsoft, they have had extremely strict internal controls preventing non-OS code from using undocumented APIs. If a DLL or EXE does not ship on the Windows client install disk, it cannot reference APIs which are not documented in MSDN. Full stop.

You're referencing an API that inadvertently didn't get documented, but everyone knows about it, has blogged about it and lots of articles mentioning to use it? Too bad. Either stop using the API or delay release by 3 months for the next MSDN update.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @02:07PM (#5691454)
> undocumented APIs

"undocumented APIs" != "a magic combination of function arguments"

Any function that's not documented can be called an undocumented API, any program has them -- Linux, Windows, whatever.

But you were talking about the argument combinations being undefined, not the functions themselves. What are some examples of that?

Re:Windows NT Kernel Contributor Explains Why Performance is... by shubb on Monday May 20, 2013 @06:53AM (#5736910)
Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Monday May 20, 2013 @10:55PM (#5741642)
What about them?
Re:Windows NT Kernel Contributor Explains Why Performance is... by mikevm on Saturday May 11, 2013 @07:52AM (#5690357)
>One problem that the WINE project (windows emulator for unix)

Wine Is Not an Emulator.

Re:Windows NT Kernel Contributor Explains Why Performance is... by fnordfnordfnord on Saturday May 11, 2013 @09:43AM (#5690560)
What shall we call WINE then? A "pretender"? I'm serious btw. I've never seen a good concise explanation of the way WINE works in comparison to say, an emulator. I have a fairly good idea myself, but I wouldn't want to have to give more than a very brief talk over it.
Re:Windows NT Kernel Contributor Explains Why Performance is... by gizmo686 on Saturday May 11, 2013 @10:19AM (#5690634)
From the WINE website: "Wine (originally an acronym for "Wine Is Not an Emulator") is a compatibility layer capable of running Windows applications on several POSIX-compliant operating systems, such as Linux, Mac OSX, & BSD. Instead of simulating internal Windows logic like a virtual machine or emulator, Wine translates Windows API calls into POSIX calls on-the-fly, eliminating the performance and memory penalties of other methods and allowing you to cleanly integrate Windows applications into your desktop." [1]

[1]http://www.winehq.org/about/

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @03:57PM (#5691896)
> What shall we call WINE then?

tl;dr: compatibility layer

Re:Windows NT Kernel Contributor Explains Why Performance is... by fnordfnordfnord on Saturday May 11, 2013 @12:17PM (#5691069)
I should have been more specific. How would you introduce WINE to freshmen, not CS students who might need to study WINE in detail, but well, here's a specific case, college freshmen who are expected to install WINE and then install and use software such as LTSpice, DipTrace, etc.? Or, more generally, how to explain it to neophytes who are expected to become moderately sophisticated computer users who might need to use WINE, but will probably never be software devs.
Re:Windows NT Kernel Contributor Explains Why Performance is... by HardlyYakka on Saturday May 11, 2013 @03:59PM (#5691899)
Something like:

WINE is a translator that allows Windows programs (those that have an .exe extension) to talk to other Operating Systems.

or perhaps:

Many people need wine if you want them to work. A windows program on Linux is no exception.

Re:Windows NT Kernel Contributor Explains Why Performance is... by justin66 on Saturday May 11, 2013 @05:05PM (#5692098)
> WINE is a translator that allows Windows programs (those that have an .exe extension) to talk to other Operating Systems.

I'm sure the aforementioned freshmen would like a no-bullshit explanation for why "emulator" hurts the WINE people's feelings but "translator" does not. It's always been a mystery to me.

Re:Windows NT Kernel Contributor Explains Why Performance is... by HardlyYakka on Saturday May 11, 2013 @06:12PM (#5692301)
> no-bullshit explanation

Hah! Not really possible, especially considering that the project used to be called windows emulator:

  http://www.faqs.org/faqs/windows-emulation/wine-faq/

I'd say they've changed the meaning so they can reinforce that it's not just another virtual machine ( and that it's faster than one).

Which is true, it doesn't emulate another hardware architecture's calls, just the kernel and OS magic needed.

In that regard, think of say a console game or arcade game emulator on a computer- that will have to emulate the whole architecture of said game system as well as the game.

Contrast that to a game program for the the x86 architecture, but not for windows- you could run it natively on windows providing you change everything the game asks for that's part of the Gamesystem380 system into something windows understands, and vice versa.

Re:Windows NT Kernel Contributor Explains Why Performance is... by plorkyeran on Saturday May 11, 2013 @10:20AM (#5690638)
A reverse-engineered implementation of the Win32 API?
Re:Windows NT Kernel Contributor Explains Why Performance is... by fnordfnordfnord on Saturday May 11, 2013 @12:20PM (#5691075)
Now we'll also have to explain what an API is, and then add some details about the Win32 API.
Re:Windows NT Kernel Contributor Explains Why Performance is... by illuminate on Saturday May 11, 2013 @12:02PM (#5691013)
Not pithy enough.
Re:Windows NT Kernel Contributor Explains Why Performance is... by asveikau on Saturday May 11, 2013 @01:06PM (#5691245)
It's a set of libraries, and a loader for EXEs and DLLs. That's the simplest, most literal way I can think of to explain it.
Re:Windows NT Kernel Contributor Explains Why Performance is... by shubb on Saturday May 11, 2013 @08:16AM (#5690396)
@Mike - Yes, that is true enough.
Re:Windows NT Kernel Contributor Explains Why Performance is... by acqq on Saturday May 11, 2013 @05:40AM (#5690200)
As far as I see you had a contact with Win API but you haven't understood enough, based on your complaints. Moreover, why do you mention "MySQL" as the team with the access to kernel sources?
Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @05:53AM (#5690210)
You see wrong. And I meant MSSQL, of course. Fixed.
Re:Windows NT Kernel Contributor Explains Why Performance is... by acqq on Saturday May 11, 2013 @07:33AM (#5690333)
Can you than please give a technical reason for you to insist on waiting on more than 64 threads in a single API call, having simply too many threads for good performance anyway when IO Completion Ports could be used instead?
Re:Windows NT Kernel Contributor Explains Why Performance is... by ambrop7 on Saturday May 11, 2013 @09:29AM (#5690531)
He probably doesn't have 64 threads, but he may have much more than 64 files or sockets. After all, the response was to the statement that "WaitForMultipleObjects as a kind of super-select to wait on anything". It clearly isn't, with this arbitrary limitation.

The problem with Windows is that there isn't any such super-select, unlike on Linux (see epoll, signalfd, timerfd...). You should take a look at my SO answer which lists six different ways to use sockets in Windows: http://stackoverflow.com/questions/11830839/when-using-iocp-...

Re:Windows NT Kernel Contributor Explains Why Performance is... by acqq on Saturday May 11, 2013 @12:09PM (#5691042)
Whatever he has more, waitfor is not the optimal way on Windows for that so it's still a complaint that round holes on Win32 are poor fit for his cubes. First choosing tools then complaining that they don't fit to some goal is typical for beginners.
Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @12:52PM (#5691194)
Do tell me, oh, enlightened one, what I should be using if I have 10K idle network connections and I can't really use IOCP, because I need keep things reasonably portable. That's not even considering that IOCP is a great example of premature optimization at cost of clarity and simplicity. It tries to solve a problem that I don't have and it doesn't offer a simpler solution that would just do. It is not a "hole" for my square peg, it's a freaking snowflake.

And expecting a modern OS to have an uncomplicated O(1) API for monitoring sockets is clearly too much to ask.

Re:Windows NT Kernel Contributor Explains Why Performance is... by ambrop7 on Saturday May 11, 2013 @02:06PM (#5691448)
It's possible to abstract IOCP and non-blocking sockets into a single interface with reasonable efficiency. For example:

- To send data, user calls Socket::send(const char *data, size_t length). This starts sending the given data; the buffer must remain available while data is being sent. IOCP implementation will initiate I/O using WSASend() or similar.

- When send operation is complete, the socket implementation calls a Done() callback of the user, and from that point on, the user is allowed to call Socket::send() again. With proper implementation, send() can directly be called from the callback.

This interface is very easy to implement on Linux. But on Windows there's a complication, because it's non-trivial to just stop an IOCP I/O operation that is in progress, in case you decide you don't need the socket any more and want to red rid of it NOW. If you just forget about it, there could be a crash when you release your buffer but Windows is still using it. A simple solution is to CancelIo() the socket and do blocking GetQueuedCompletionStatus()s until you get an event inidicating the completion of the pending I/O operation - taking care to queue up any unrelated results you may have gotten, for later processing. However, a more efficient solution is to use reference counting or similar with your buffers, so they only get released when Windows is done with them.

I didn't say it's easy, but it's possible.

Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @02:36PM (#5691579)
Yes, I know the abstraction is possible. In fact, I ended up implementing a nearly complete BSD socket API emulation on top of IOCP. Just don't lose the sight of the context, which is Windows does a lot of things right and my point being is that it doesn't.
Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @02:22PM (#5691515)
> The problem with Windows is that there isn't any such super-select, unlike on Linux (see epoll, signalfd, timerfd...)

> I can't really use IOCP, because I need keep things reasonably portable

LOL!

You outright dismiss the solution because it isn't "portable". What does "portable" mean here? Portable in a particular platform?!

Portable between what exactly? IOCP has been there since Windows NT 3.51 -- it's "portable" between the Windows-es.

Your epoll, signalfd, timerfd, etc. are Linux functions They're "portable" between the Linux-es.

i.e. The "portable" solutions you like are just as non-portable as the "non-portable" solutions you outright dismiss, making Microsoft's Windows designers look like idiots.

You're never going to find a solution while considering the Linux platform to be the "portable platform" and Windows to be the weird kid on the block.

Sadly, you're not the only person who thinks like this.

Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @03:18PM (#5691776)
A scalable variation of BSD socket API is supported by every major OS with a sole exception of Windows. It's an abstraction that is both simpler and more universal than IOCP model, not to mention that it also predates it by a decade.

So "LOL!" yourself, bud.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @03:44PM (#5691861)
> A scalable variation of BSD socket API is supported by every major OS with a sole exception of Windows

It's funny, you mention "epoll, signalfd, timerfd" as an example of a portable API, and then when I point out they're Linux APIs, you suddenly start talking about "a scalable variation" of the BSD sockets API. As for your "scalable variation of BSD sockets API", why don't you tell us the name so we know what you're referring to? Are you talking about Kqueue for example, or something else?

> an abstraction that is both simpler

I/O completion ports are just read-write queues... you ReadFile(), and when it's done, an item goes into the queue. (You can insert your own tasks into the queue, too, if you want.) Now just pop off the next item in the queue and off goes your thread. What part of that was so complex compared to your portable API (which for some reason you didn't feel was appropriate to mention the name of)?

> and more universal

Can't reply to it without knowing what "scalable variation" you're referring to, and why you didn't actually mention it as an example of portability when you were actually talking about the subject. Is portability actually a concern for you or no? (If it is, why did you bring up non-portable examples?!)

Re:Windows NT Kernel Contributor Explains Why Performance is... by huhtenberg on Saturday May 11, 2013 @05:43PM (#5692210)
> you mention "epoll, signalfd, timerfd"

It wasn't me who mentioned it.

Re:Windows NT Kernel Contributor Explains Why Performance is... by wfunction on Saturday May 11, 2013 @05:54PM (#5692242)
D'oh, sorry for the mixup. You were completely supporting that user so I didn't realize you were different people.

My comment remains essentially unchanged, though, with the same question as before:

What's this great portable API you're referring to?

Re:Windows NT Kernel Contributor Explains Why Performance is... by ams6110 on Saturday May 11, 2013 @01:39PM (#5691347)
Can you name any software that doesn't have an internal API that's not meant for use by 3rd parties?