Thursday, May 18, 2006

Delaying the xnu-x86 source release  

[Updated Aug 7 2006; see below.]

Tom Yager at InfoWorld points out that Apple has released the source code to everything in the x86 build of OSX Darwin except for the kernel, xnu.

He also makes what appears to be a completely and utterly unsubstantiated statement about why:

Thanks to pirates, or rather the fear of them, the Intel edition of Apple’s OS X is now a proprietary operating system.

First of all, what? That's a bold statement. Got any source for that claim? That seems like sheer FUD, deliberately sensationalized to create a stir and bring people to the site (and therefore sell ads). From everything I've seen, and I've worked with these people, Apple's security team and upper management know better than to rely on security through obscurity. And to date I don't know of any official or even unofficial statement from Apple about xnu-x86 -- just the fact that several people have noticed that the source still hasn't been released.

[Update 5/22 via John Gruber: Apple's Product Manager for Open Source reiterates that Apple has not made any announcement yet, and drops what sounds like a big hint that xnu-x86 will eventually be released.]

Let me offer two guesses at the real reason why we haven't seen source to xnu-x86 yet:

  1. The xnu-x86 source might leak information about a future product. The obvious candidate is the only missing link in the x86 line, the pro desktops. You know, the ones that will replace the "PowerMac". Let's call them "Mac Pro" for lack of a better name.

    For example, if the highest end Mac Pro desktop machines were planned to have, say, 4 CoreDuos packed into them for a whopping 8 CPU cores, then chances are you would see traces of that support show up in several parts of xnu. And when Apple releases a major rev of xnu, there are always some people who pore over it looking at the diffs.

    Apple does have ways to keep prerelease stuff out of the source release, of course, but it adds a layer of complication and risk to go back and hack that up after the fact. Maybe this time they decided it was just simpler to drag their feet for a while until the entire new line has been announced.

  2. The xnu-x86 source might currently contain a small amount of licensed proprietary code that does not belong to Apple. If that's the case, they simply might not be legally allowed to release it in its current form.

    Maybe it's virtualization code from Intel, or some sort of Trusted Computing gobbledygook which is currently dormant. If they can't negotiate terms to release it, then they might have to factor the sourcebase somehow to link that other code in separately. Factoring that out seems like it would be totally possible, but kind of a messy task since it's at such a low level in the kernel and they can't sacrifice any performance to do so.

Personally I think #1 fits Apple's modus operandi perfectly. But #2 is also the kind of real-world consideration that could delay a source release for an unknown period of time while the lawyers work things out. I will grant that "fear of pirates" is another technically possible reason why we haven't seen the source, but then again the same could be said for "fear of ninjas".

We'll see what happens. Either way my gut feeling is that this is just a delay in the source release, not a permanent switch to a completely closed-source kernel.

[Update: Aug 7 2006: xnu-x86 has now been released. According to Kevin van Vechten's post at kernel.macosforge.org:

Several changes were made in order to publish the kernel (xnu) sources. As a result, the kernel built from these sources differs from the one found in the 10.4.7 software update. In order to accommodate these changes, several kernel extensions were also modified and must be downloaded and installed in order to run a kernel built from these sources on Mac OS X 10.4.7 for Intel.

Based on that comment, it sounds like the answer is #2 and for the xnu-x86 release they moved some code from the kernel into a kext.]

9 comments:

  • Johannes Fortmann said...

    Just wanted to note that
    - you can build an x86 kernel from the ppc kernel sources
    - the resulting kernel is missing Rosetta
    - Rosetta is definitely licensed from a third party
    - there is no chance whatsoever that Apple has the rights to distribute Rosetta code
    I rest my case (hoping that in 10.5 we will have the sources back... mmmh, precious source code...)

  • Drew Thaler said...

    So you think it's Rosetta that's holding up xnu? The only way that seems plausible is if Rosetta requires some kernel support.

    But it's very likely that Rosetta is entirely a userspace construct. It's not like the 68K emulator that had to be able to run 68K code anywhere and at any time. With Rosetta, only whole applications are emulated. Not parts of an app (ie: single plugins), not kernel extensions. The easiest way to emulate a userlevel application is with another userlevel application.

    Probably the lowest level Rosetta needs to hook in is LaunchServices, which is above Darwin proper and is already closed-source. There's no obvious need for kernel support.

    If it's proprietary code that's the holdup, I'm still betting on VT-x or LaGrande.

  • Drew Thaler said...

    Okay, looks like Rosetta can be invoked via an exec() call, not just via LaunchServices. And that implies that there's at least a Rosetta-launching hook in xnu. So I take it back, there is at least some support for Rosetta built into xnu. But it still seems like it's got to be very minimal. I'd be surprised if that were actually holding up the kernel source release...

  • Wes Felter said...

    There's no evidence that OS X uses VT-x or LaGrande, either. Maybe the proprietary code is TehSnappy™ licensed from NinjaSoft. :-)

  • Drew Thaler said...

    Yeah, hmmm. There may not be built-in support in OSX for VT-x yet. Parallels Desktop uses it, but upon further inspection it looks like they install their own hypervisor to do it.

    Okay, well, maybe Rosetta is the answer after all. :-)

  • arwyn said...

    Nah, can't be Rosetta.

    The real question is if Parallels Desktop is still going to work unmodified on the next great OS revision. Multiple hypervisors probably wouldn't get along very well unless they knew intimately about each other... or cooperated to some extent, altho maybe thats the same thing. Either way, WWDC might be interesting... or not.

  • Drew Thaler said...

    I'd take it as a given that Parallels will have to rev for Leopard. I don't think you can really have multiple hypervisors, period, and it seems obvious that Apple will be supporting VT-x directly in the kernel by then. (Why wouldn't they?) So they'll need a new VMM API for x86.

    The real question I have about Leopard is whether Apple will kill off 90% of Parallels' market segment by introducing some sort of Windows Classic mode. But I guess we'll have to wait and see. :-)

    But all that's moot for the question about xnu source and Tiger...

  • Mike said...

    If Leopard has its own Hypervisor, it's possible that Parallels won't be needed. I suspect Boot Camp will become something that uses virtualization to run Windows without rebooting.

  • Drew Thaler said...

    Well, it's possible, but there's a huge step between just creating a hypervisor and the actual implementation of a virtual machine.

    Apple has a lot of incentive to create its own hypervisor. The hypervisor in its simplest form is just a way to do IPC between the client OS and the host OS. It runs at such a low level in the system that Apple will probably want to be in control of it, and just provide an API for third parties to hook into. Having a hypervisor with a standard SPI interface will prevent all sorts of weird compatibility problems. ("I installed Parallels, then VMware, and now nothing works!")

    What I'm thinking Leopard will have for xnu-x86 is something similar in spirit to the old VMM API in xnu-ppc. IIRC it was essentially written by Connectix engineers to speed up Virtual PC. It's relatively small, usable by multiple virtual machines at a time, and there are half a dozen kernel engineers at Apple who could create one. I'm not too far in-depth in my knowledge of the way VT-x works, but I think odds are good that they could do it in a single source file.

    Creating an entire virtual machine like Parallels, VPC, VMware, etc ... well, that's a whole 'nother story. :-) That's man-years of labor and probably a full calendar year of debugging. Even Parallels spent months working out their bugs, and they were just porting a pretty solid product that already worked on two other OSes.

    For now, I'm betting on baby steps.