News for hackers. Stuff that matters.
Re:Compiler bug? Linker bug? Windows Kernel bug by mehrdadn on Monday February 26, 2018 @03:50AM (#16464022)
Huh. There's a memory-mapped I/O bug I've known about for years, I wonder if this is related to it? It's something along the following lines, but I forget the exact details, so some bits (no pun intended) might be wrong: if you create a memory mapping and then close the underlying file handle, you can even delete the file at that point, but the memory mapping will still hang around and pretend to work, and you won't know it's not working. I'm not sure exactly what happens under the hood but I recall it doesn't behave sanely.
Re:Compiler bug? Linker bug? Windows Kernel bug by leeter on Monday February 26, 2018 @11:15AM (#16466334)
IIRC this isn't a bug but a compatibility hack. The pages really are unmapped in your process but if you try to access them the kernel will remap them for you and lie when you try to remap that file (as it already had the handle open and assigned to your process). This was done because a few well known pieces of software did this back before protected mode. So MS pretends the hack to keep them working. I'm pretty sure this goes away if you list the latest OS in your manifest and have DEP and NX turned on for your app.
Re:Compiler bug? Linker bug? Windows Kernel bug by mehrdadn on Monday February 26, 2018 @06:21PM (#16469893)
This doesn't seem to be the case. On the latest Windows 10 I tried out this test program

  HANDLE file = CreateFile(TEXT("\\\\.\\C:\\mmap.bin"), GENERIC_ALL | SYNCHRONIZE, FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, NULL, OPEN_ALWAYS, 0 /*can also try: FILE_FLAG_DELETE_ON_CLOSE*/, NULL);
  if (file != INVALID_HANDLE_VALUE)
  {
  	HANDLE mapping = CreateFileMapping(file, NULL, PAGE_EXECUTE_READWRITE, 0, 1, NULL);
  	void *ptr = MapViewOfFile(mapping, FILE_MAP_ALL_ACCESS, 0, 0, 0);
  	CloseHandle(file);
  	*static_cast<unsigned char *>(ptr) += 1;
  }
with /NXCOMPAT and even /DYNAMICBASE and with the embedded manifest

  <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  <assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1" xmlns:asmv3="urn:schemas-microsoft-com:asm.v3">
  	<assemblyIdentity type="win32" name="Microsoft.Windows.Foo" version="1.0.0.0"></assemblyIdentity>
  	<trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
  		<security>
  			<requestedPrivileges>
  				<requestedExecutionLevel level="asInvoker" uiAccess="false"></requestedExecutionLevel>
  			</requestedPrivileges>
  		</security>
  	</trustInfo>
  	<compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
  		<application>
  			<supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}"></supportedOS>
  		</application>
  	</compatibility>
  </assembly>
but it seems to still modify the file after closure even after the CloseHandle call.
Re:Compiler bug? Linker bug? Windows Kernel bug by leeter on Tuesday February 27, 2018 @11:16AM (#16474803)
When I get time I might actually debug this to see what's going on. I'm quite curious what happens when you access the pointer again. If the Kernel really is remapping the pages again as the anecdotes from Raymond would suggest it might be. Or if it just hasn't reclaimed them yet. VMMap and ProcessExplorer should give the story.
Re:Compiler bug? Linker bug? Windows Kernel bug by MaxBarraclough on Monday February 26, 2018 @05:06AM (#16464271)
That's undefined behaviour then, right? Even if it's not the sane behaviour.

I imagine that might be reason enough for them to treat it as a low-priority.

Re:Compiler bug? Linker bug? Windows Kernel bug by mehrdadn on Monday February 26, 2018 @05:15AM (#16464302)
I'm not sure... UB in general seems to be a language-level construct. OSes (and below) don't really have much UB to speak of, and I assume this is because they are expected to provide security and stability regardless of misbehaving user programs. And also, UB is usually said in response to a program expecting a certain behavior, but in this case I'm not worried about the program itself, but about the system and where exactly the black hole that the physical memory is mapped to is (given that it is no longer a file). In this case, I wouldn't be surprised if there are serious stability or even security issues lying behind this... I just haven't bothered to investigate.
Re:Compiler bug? Linker bug? Windows Kernel bug by barrkel on Monday February 26, 2018 @05:28AM (#16464353)
It would rather be classified as undocumented behaviour. Undocumented behaviour is frequently maintained in OSes because programs become reliant on it, to the point that sometimes bugs need to be emulated to avoid breakage.
Re:Compiler bug? Linker bug? Windows Kernel bug by DoofusOfDeath on Monday February 26, 2018 @07:22AM (#16464774)
It's been a long time since I've looked at Windows documentation, but I wonder if it's reasonable to treat their documentation as the actual contract offered by the OS?

I had the impression that poor documentation forced a lot of Windows programmers to understand Microsoft's API's via experimentation.

Re:Compiler bug? Linker bug? Windows Kernel bug by MaxBarraclough on Monday February 26, 2018 @09:36AM (#16465527)
> UB in general seems to be a language-level construct

Not always. Various standard library functions can cause UB, and not just things like C's div function. In C++, calling front() on an empty std::vector<T> causes UB, for instance. http://www.cplusplus.com/reference/vector/vector/front/

I agree that from a system perspective, it doesn't look like good behaviour.

Re:Compiler bug? Linker bug? Windows Kernel bug by mehrdadn on Monday February 26, 2018 @09:42AM (#16465570)
> Not always. Various standard library functions can cause UB, and not just things like C's div function. In C++, calling front() on an empty std::vector<T> causes UB, for instance.

std::vector et al. are parts of the C++ language. "Language" != pure syntax.

Re:Compiler bug? Linker bug? Windows Kernel bug by MaxBarraclough on Monday February 26, 2018 @11:12AM (#16466311)
I think I see your point - that despite the way C++ libraries can and do expose UB if not used as intended, it doesn't mean that Windows does the same.

For what it's worth, I read that POSIX does have UB in the spec (as in, distinct from how the compiler is allowed to break things in terms of language-level UB), at least in some ghastly signal handling situations. I wasn't able to find at a glance whether Windows does too, or if it categorically doesn't.

> According to POSIX, the behavior of a process is undefined after it ignores a SIGFPE, SIGILL, or SIGSEGV signal that was not generated by kill(2) or raise(3).

Re:Compiler bug? Linker bug? Windows Kernel bug by zuzun on Monday February 26, 2018 @05:45AM (#16464420)
A call to unlink removes the name from the filesystem, but keeps the file around as long as it's still opened by a process; and mmap keeps the file opened, even if you close the original file descriptor.
Re:Compiler bug? Linker bug? Windows Kernel bug by mehrdadn on Monday February 26, 2018 @05:50AM (#16464441)
> A call to unlink removes the name from the filesystem, but keeps the file around as long as it's still opened by a process; and mmap keeps the file opened, even if you close the original file descriptor.

Are you familiar with Windows or just assuming all OSes works like *nix systems? That's definitely not at all close to how Windows file systems behave.

Re:Compiler bug? Linker bug? Windows Kernel bug by pjmlp on Monday February 26, 2018 @06:28AM (#16464581)
Not even all UNIXes.

It is implementation defined on the UNIX OS variant and mounted file system.

The beauty of POSIX is how it appears to be portable, while leaving quite a few things being implementation defined.

Re:Compiler bug? Linker bug? Windows Kernel bug by gpderetta on Monday February 26, 2018 @10:34AM (#16465948)
Not sure what UNIX variant are you thinking, but from POSIX:

> The mmap() function adds an extra reference to the file associated with the file descriptor fildes which is not removed by a subsequent close() on that file descriptor. This reference is removed when there are no more mappings to the file.

I'm sure there have been unix variants that got it wrong and might not even have been in posix from the beginning (it is there since at least SUSv2); still I think this is traditional unix behavior.

Re:Compiler bug? Linker bug? Windows Kernel bug by mehrdadn on Monday February 26, 2018 @06:30AM (#16464588)
Today I learned!
Re:Compiler bug? Linker bug? Windows Kernel bug by zuzun on Monday February 26, 2018 @07:41AM (#16464857)
Your comment contains no details whatsoever, then you described perfectly sane mmap behavior, so what do I know?
Re:Compiler bug? Linker bug? Windows Kernel bug by throwaway2048 on Monday February 26, 2018 @11:37AM (#16466563)
Windows dosent have the concept of an inode, so there is nothing for a file handle (what windows calls a file descriptor) to bind to except a file name, this means that a filename cannot be removed while it is being accessed somehow.

This is the cause of a lot of woes on windows, such as random access denied errors when attempting to delete stuff, and what forces a reboot when you update nearly anything on windows.

One of my favourite things related to this is that the Windows XP explorer had a bug where clicking on a file would generate a dynamic preview of an audio/video file in the sidebar, which would lock the file, rendering it impossible to delete because you had to click on it first....

Re:Compiler bug? Linker bug? Windows Kernel bug by benmmurphy on Monday February 26, 2018 @07:28AM (#16464807)
this reminds me how updating libraries in linux can be kind of dangerous depending on how you perform it. for example if you unlink the library then create a new library with the same name then the system behaves probably how you want it to. existing processes use the original library until they are restarted. however, if you open the existing library with O_TRUNC and overwrite the library then existing processes will use the new code which is super dangerous. by default cp will do the O_TRUNC behaviour, and tar will use the unlink behaviour. its interesting to think that stuff might be accidentally working for some people because of this choice in behaviour for tar.

i haven't actually tested this with a real library. this is what i've assumed happens from playing around with mmap(PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE).

Re:Compiler bug? Linker bug? Windows Kernel bug by zrm on Monday February 26, 2018 @10:31AM (#16465937)
Both of those are the wrong way to do it. :)

O_TRUNC for the reasons you stated and unlink because it causes a race between the unlink and the copy when a new process that tries to use the library will either fail because it doesn't exist or get a copy of the library that isn't completely written yet.

The correct way to do it is to write the new file to the same filesystem at a different path and then, once it's completely written, move it to the intended path. Move (i.e. rename(2)) is atomic on sane filesystems so anything that opens the file will get one complete copy or the other.