Delaying the xnu-x86 source release

[Updated Aug 7 2006; see below.]

Tom Yager at InfoWorld points out that Apple has released the source code to everything in the x86 build of OSX Darwin except for the kernel, xnu.

He also makes what appears to be a completely and utterly unsubstantiated statement about why:

Thanks to pirates, or rather the fear of them, the Intel edition of Apple’s OS X is now a proprietary operating system.

First of all, what? That's a bold statement. Got any source for that claim? That seems like sheer FUD, deliberately sensationalized to create a stir and bring people to the site (and therefore sell ads). From everything I've seen, and I've worked with these people, Apple's security team and upper management know better than to rely on security through obscurity. And to date I don't know of any official or even unofficial statement from Apple about xnu-x86 -- just the fact that several people have noticed that the source still hasn't been released.

[Update 5/22 via John Gruber: Apple's Product Manager for Open Source reiterates that Apple has not made any announcement yet, and drops what sounds like a big hint that xnu-x86 will eventually be released.]

Let me offer two guesses at the real reason why we haven't seen source to xnu-x86 yet:

1. The xnu-x86 source might leak information about a future product. The obvious candidate is the only missing link in the x86 line, the pro desktops. You know, the ones that will replace the "PowerMac". Let's call them "Mac Pro" for lack of a better name.

   For example, if the highest end Mac Pro desktop machines were planned to have, say, 4 CoreDuos packed into them for a whopping 8 CPU cores, then chances are you would see traces of that support show up in several parts of xnu. And when Apple releases a major rev of xnu, there are always some people who pore over it looking at the diffs.

   Apple does have ways to keep prerelease stuff out of the source release, of course, but it adds a layer of complication and risk to go back and hack that up after the fact. Maybe this time they decided it was just simpler to drag their feet for a while until the entire new line has been announced.

2. The xnu-x86 source might currently contain a small amount of licensed proprietary code that does not belong to Apple. If that's the case, they simply might not be legally allowed to release it in its current form.

   Maybe it's virtualization code from Intel, or some sort of Trusted Computing gobbledygook which is currently dormant. If they can't negotiate terms to release it, then they might have to factor the sourcebase somehow to link that other code in separately. Factoring that out seems like it would be totally possible, but kind of a messy task since it's at such a low level in the kernel and they can't sacrifice any performance to do so.

Personally I think #1 fits Apple's modus operandi perfectly. But #2 is also the kind of real-world consideration that could delay a source release for an unknown period of time while the lawyers work things out. I will grant that "fear of pirates" is another technically possible reason why we haven't seen the source, but then again the same could be said for "fear of ninjas".

We'll see what happens. Either way my gut feeling is that this is just a delay in the source release, not a permanent switch to a completely closed-source kernel.

[Update: Aug 7 2006: xnu-x86 has now been released. According to Kevin van Vechten's post at kernel.macosforge.org:

Several changes were made in order to publish the kernel (xnu) sources. As a result, the kernel built from these sources differs from the one found in the 10.4.7 software update. In order to accommodate these changes, several kernel extensions were also modified and must be downloaded and installed in order to run a kernel built from these sources on Mac OS X 10.4.7 for Intel.

Based on that comment, it sounds like the answer is #2 and for the xnu-x86 release they moved some code from the kernel into a kext.]

What the hell is going on here?

I don't often write short posts, but this May 12, 2006 article from one of my favorite op-ed writers, Richard Reeves, really struck a chord:

The only way to restore constitutional checks and balances in Washington before 2008 is for the opposition to win one house of Congress and have the power to call witnesses at public hearings and ask, under oath:

"What the hell is going on here?"

What is going on in the White House? The Defense Department? The CIA and the NSA? With gasoline prices? Along the border between California and Mexico? In Baghdad? In New Orleans? With Jack Abramoff and the K Street Gang? In Congress itself?

Or, who is listening to your phone calls? Are your taxes being used to teach torture techniques to your sons and daughters? Are the glaciers melting?

We'll be the last to know.

The nation flies blind when we have determined one-party government. That can and has happened in both parties over the centuries, but this White House is a particularly tough bunch, talking freedom around the world and taking it away at home. President Bush essentially has veto power over the Republican automatons in the Congress....

The rest of the article is worth reading, but that's really it in a nutshell. And it summarizes the entire problem that we face today.

The entire Republican Congress, including our own Ohio Senators Mike DeWine and George Voinovich, have thrown away all pretense of holding the President and the executive branch accountable. They are nothing but a rubber-stamp for the President's bad policies.

God forbid that anyone suggest the President might possibly have made a mistake! They can't admit to even one.

We are in that most terrifying of states where virtually every single decision that's being made is wrong, but nobody is willing to stop. It's like the worst death spiral you've ever seen from a meth addict, or a horror movie where you are yelling at the screen: "No, DON'T split up! The killer is taking you out one by one!" And yet we are powerless to stop it.

Democrats speak out daily, but it's the Republican majority that controls all the committee chairmanships and sets the agendas for both houses. And it's the Republicans who make sure that no actions that force accountability ever make it to the floor. In the past year alone I've seen video of at least five instances of Republican chairmen overruling perfectly valid objections from Democrats, silencing debate, and completely ignoring accountability.

If you wonder why Congress doesn't do something to stop the President, you are really asking why the Republicans won't do anything to stop the President.

You know what? That's a damn good question.

High-Resolution Images Could Choke Internet

Perhaps you've seen this article, entitled High-Definition Video Could Choke Internet, already:

Every day, it seems, a new service pops up offering to send you video over the Internet. "Desperate Housewives," Stephen Colbert heckling the president, clips of bad dancers at wedding parties: It's all there.

You may be up for it, but is the Internet?

The answer from the major Internet service providers, the telephone and cable companies, is "no." Small clips are fine, but TV-quality and especially high-definition programming could make the Internet choke.

Man. That just sounds really familiar to me for some reason. I've been online for more than a decade now, and this has been bugging me all day. So I started searching through my old floppy disks for news articles that I'd stashed away back in my university days. It turns out this argument isn't new at all! Check it out (sorry, no link to the original -- all I know about the origin is that I saved it off Usenet as jpegbad.txt):

High-Resolution Images Could Choke Internet
By SVEN PETERSON, Microcomputer Technology Writer
Sunday Mar 24, 1996

NEW YORK - Every day, it seems, a new service pops up offering to send you images or software over a new service called the World Wide Web, or WWW. Astronomy pictures of the day, ESPN.SportsZone.com, the Internet Movie Database, DOOM wads, Info-Mac archives: It's all there.

You may be ready for it, but is the Internet?

The answer, from major Internet service providers such as Prodigy, Compuserve, and eWorld, is "no." Small pictures are fine, but GIF animations, full-color JPEGs, and especially high-resolution images could make the Internet choke.

Most home Internet use is in brief bursts: an e-mail here, a gopher session there, followed by some Telnet activity. If people start browsing the World Wide Web like they watch TV -- for hours at a time -- that puts a strain on the Internet that it wasn't designed for, ISPs say, and beefing up the Internet's capacity to prevent that will be expensive.

To offset that cost, ISPs want to start charging content providers to ensure delivery of large JPEG files, for example.

Internet activists and consumer groups are vehemently against those plans, saying they amount to deliberately fouling the Internet's level playing field, one of the things that has encouraged communication and collaboration and may, in the future, provide a much-needed boost to the US economy. They want legislation to guarantee a "neutral" Internet, but prospects appear slim.

At the heart of the debate is a key question: How much would it really cost the Internet carriers to provide a couple of megabytes of JPEG-laden WWW pages over their networks every day?

The carriers are not telling, but there are ways to get close to an answer.

One data point: As a rough estimate, an always-on, 1000 kilobit-per-second connection to the Internet backbone in downtown Cleveland, purchased wholesale, costs an ISP $100 to $200 a month, according to the research firm GeoTelegraphy Inc. An ISP's business is carrying data from that connection to the customer.

One thousand kilobits per second is obviously a lot of bandwidth, so ISPs have to spread that bandwidth out over their subscribers. That connection can serve about 30 users who are using every bit of their 28.8 kilobit-per-second dialup connection. But analysts estimate that ISPs sell around 30 times more bandwidth to their end users than they can connect simultaneously to the Internet, meaning that these thousand kilobits are often shared by roughly 900 users.

In a way, dial-up internet service is like an old-fashioned telephone service, where there are always more lines leading from homes to the local switching station than there are going from the station out of the neighborhood. If everyone picks up the phone at once, there won't be enough outgoing lines for every call to go through. But the system works because that rarely happens.

Oversubscription doesn't present a problem as long as people are only using the Internet for e-mail, telnet, and the occasional FTP download. But if everyone in a neighborhood is trying to "surf" the WWW at the same time, it's just not going to work.

"The simple truth is that today's networks simply don't have the capacity to deliver all that customers expect," says Timothy Taker, Prodigy's top lobbyist.

The solution, of course, is to make the pipes connecting to the Internet fatter. To illustrate what that would mean, Compuserve's chief architect, Franz Kafka, uses the assumption that the cost of providing a month's worth of data to the average user, about 2 megabytes, costs the company $1. That's a fairly small amount compared to the $25 to $47 a month Compuserve charges for dial-up, but then the company has to pay for sales, support, maintenance and a host of other costs.

If that same user were to start downloading twenty print-quality high-resolution JPEGs per month, Compuserve's data cost, not including the cost of maintaining the dial-up line, would go up to $4.50 a month. Higher, but perhaps not high enough to break their business model.

But if the customer starts browsing the WWW like the average household watches broadcast TV, 8 hours a day, Compuserve's cost would go up to $112 a month, enough to inspire nightmares in the corporate accounting department according to Kafka.

"We don't expect to get to the point where we're charging anyone those kinds of prices for Internet service, but it does reflect the kind of impact that high-resolution JPEGs and GIFs could have on the network and business models for the Internet," Kafka said.

To deal with that, Kafka says Compuserve might put caps on the amount of data that a residential user gets for free, and charge extra if the user goes over. Other options include charging content providers extra for guaranteed delivery of JPEG images and GIF animations, which has raised the hackles of Internet content providers and activists.

However, Kafka's estimates for these costs aren't really Compuserve's. Like other ISPs, they don't disclose their actual costs. Instead, Kafka's base figure of $1 for 2 megabytes of data per month is based on an estimate by Dave Tricklestein, sysop of the Dial-Up Prime BBS, and Tricklestein thinks Kafka has it wrong.

"Traffic just isn't growing that fast," Tricklestein said. "It will grow and it will even accelerate, but not fast enough to turn into dollar amounts that really matter." The new WWW service is still just a small fraction of the total amount of internet traffic out there, and that's unlikely to change overnight, in Tricklestein's opinion.

In fact, he said, it will probably be at least two more years before the WWW takes off in a big way. Prices of network equipment like switches and routers have been falling, and that trend is likely to continue.

Tricklestein believes the danger of letting the carriers charge extra for guaranteed delivery is that they'll put the spending for upgrades into creating that extra "toll lane," and won't reduce oversubscription in the rest of the network even though it would be cheap to do so.

Both Compuserve and eWorld have said they won't degrade or block anyone's Internet traffic. But it's impossible to tell what goes on inside their networks.

So what's the message? Stay tuned, and watch your modem connection speeds.

Funny. Whatever happened to Compuserve and that short-lived dial-up business model where you got charged extra for overages? Oh yeah, consumers ran away from it in droves in favor of unlimited-download services so that they could browse the Web without restriction.