Recording Artist

Optimization Goals

2011-04-20T16:48:00.001-04:00

In my day job, I've worked with many different teams of programmers. Game engine, game tools, OS, SDK, middleware, test engineers, you name it. Many of the people I work with are the best in the industry, true world-class programmers. And others, well, aren't... and that's okay too. :-)

Personally, I find that I've learned a ton from interacting with all of these different groups.

One of the most important lessons I've learned — and I'm frankly surprised this isn't more widely talked about — is that there are many different and equally valid possible optimization goals in programming.

When I talk to a new team, that's often the first thing I try to find out: what are your primary goals? What are you looking for?

The most common definition of optimization involves improving the runtime speed of the code. For example, game engine programmers are often very focused on this, because they have to cram as much work as they can into one frame — generally either 1/30th of a second (33.3ms) or 1/60th of a second (16.6ms). Naturally, this leads to a lot of counting cycles and downcoding inner loops into assembly.

However, game tool programmers tend to be a little more willing to sacrifice speed in favor of optimizing for other goals: development time, correctness, and ease-of-use. For example, it's often OK to use an unoptimized implementation of some algorithm in a tool, since a high-end PC will run it "fast enough". In other cases a tool might only run infrequently or overnight. Or it might be feasible to simply throw more hardware at the problem.

You also see it among different teams: a studio making a AAA game might be very focused on optimizing runtime speed, while a small indie team might be focused on optimizing for development cost — doing the work as cheaply as possible. These teams will have very different optimization goals, to the point that what the AAA game has downcoded to assembly might be C++ objects and STL in the other.

Then there are the sports games — which live in a fascinating alternate world all their own. These guys have to iterate and ship a new version once a year, every single year! Slipping is simply not an option, so they tend to optimize first for reliability, upgradeability, and ease-of-maintenance.

This got me thinking: what are all the possible optimization goals you might have?

Possible Optimization Goals

Here are some things that you may choose to optimize for in your development. It's impossible to optimize for all of these things simultaneously, but it's common to target more than one.

runtime speed - Run as fast as possible on your target system. Most of the optimization literature focuses on this. It involves choosing algorithms carefully, choosing your data layout carefully, looking for early exit opportunities, hand-optimizing inner loops, and so on.
development time - Complete the engineering work as quickly as possible. In other words, finish the job in a day or a week, rather than a month. Optimizing for development time may mean that you use the highest-level language available that will do the job, along with whatever third-party frameworks you can pull in to solve the problems you need solved. C#, Python, Ruby, etc.
development cost - Complete the engineering work as cheaply as possible. This is closely related to development time, but slightly different! The primary difference is the choice of who is doing the work: it might be cheaper (and a better investment) to have an intern use a task as a learning project and work on it for three months, than to have a senior programmer crank it out in three weeks.
correctness - Be absolutely sure there are no mistakes or bugs. Bug-free code is always a goal, of course, but sometimes it's even more important than usual. There are two common situations I can think of where correctness is your primary goal: (1) when creating a reference implementation of an algorithm, for possible later optimization. (2) when you're making important architectural decisions based on the results of some investigation or other; you'll generally double- and triple-check the investigation, if you're smart.
ease-of-use - Make sure users can grasp your interface quickly and achieve what they need to do. Funnily enough, this applies to both applications (user interfaces) and to middleware (developer interfaces). In both cases, you need to understand how people will expect to use the product. The fastest or easiest way to do things might not be the most understandable.
ease-of-maintenance - Make the code so straightforward a monkey could maintain it. Or you, in a few years. ;-) If you've ever had to go back and bug-fix some code that you (or god help you, someone else) wrote more than five years ago, you'll appreciate the value of this. Code that is well-written, easy to read, well-commented, and shaped well to fit the problem is code that is optimized for ease of maintenance. To some extent, this goal can be a cause of "not-invented-here" syndrome; the preference for in-house code is often born from a perfectly rational and reasonable desire to know how to maintain that code.
reliability - Never fail. Keep on going in the face of problems. At runtime, if the program gets unexpected input, don't crash: flag it, log it, and continue on with a sensible fallback behavior. At development time, always have a reference implementation to fall back to if the optimized version isn't working. In the extreme, reliability might require a reduction (hopefully small) in average-case performance just to ensure that the code can reliably handle uncommon edge cases. (LibC's memset and memcpy are examples of that.)
upgradeability - Make sure every subsystem can be upgraded. This is most commonly achieved with modularity. Good module separation should mean that individual pieces can be swapped out without too much trouble. If you remove modularity in order to run faster with today's situation, then you may be unable to upgrade to handle tomorrow's situation without a full rewrite.
iteration time - Minimize the time difference between when you make a change and when you are able to view and test the result. This covers a lot of ground and may include things like build time reduction, file loading, dynamic loading of updated assets, console boot time, and more. This is surprisingly important: the people who write the best software will almost always tell you that rapid iteration is king.

Did I miss any? What are you optimizing for?

Variants of CompareAndSwap

2010-03-16T16:19:00.001-04:00

I ran across an interesting issue the other day while trying to make a cross-platform atomics header work on Mac OS X.

Compare-and-Swap (hereafter, CAS) is one of the fundamental atomic primitives: given a working CAS you can implement any other atomic operation.

But there are in fact two common variants of CAS:

    // Returns the old value from *ptr
    int ValueCAS(volatile int *ptr, int compare, int swap);

    // Returns true if a swap occurred
    bool BoolCAS(volatile int *ptr, int compare, int swap);

That's OK, though, because it's theoretically possible to convert either one into the other.

It's trivial to implement BoolCAS in terms of ValueCAS. Here's how you do it:

    // GOOD: this works correctly.
    bool BoolCAS(volatile int *ptr, int compare, int swap) {
        return compare == ValueCAS(ptr,compare,swap);
    }

But for what I was doing, I needed to go the other way around. Mac OS X provides BoolCAS functions in <libkern/OSAtomic.h>, and I needed to convert those to ValueCAS functions.

It turns out to be subtly difficult to go the other way around, and I thought I'd share the reasons why.

First attempt

My first attempt was fairly simple, using just an extra load and branch:

    // BAD: do not use! Broken!
    int ValueCAS(volatile int *ptr, int compare, int swap) {
        if (BoolCAS(ptr,compare,swap))
            return compare;
        else
            return *ptr;
    }

The idea is that if the CAS worked, then the old value of *ptr must have been the same as compare, so we can just return that. That's absolutely correct.

However, if the CAS failed, what was the previous value? I was naïvely re-reading *ptr. But my unit tests (oh yes, this is the kind of thing you unit-test thoroughly) showed that very rarely — once per several million iterations — it would fail to do the right thing.

What was going wrong?

The problem comes down to how callers detect success and failure from ValueCAS. If ValueCAS returns compare, the CAS is assumed to have succeeded. If ValueCAS returns any other value, the CAS is assumed to have failed.

Once we recognize this, we can see that the naïve implementation above has not one, but two failure cases:

Spurious CAS failure. This can happen on nearly any system, for a variety of reasons. I'd rather not get too deep into explanations of CPU behavior here, but the typical example might be if an interrupt occurred at exactly the wrong moment. Even if *ptr == compare when the CAS runs, a spurious failure might exit the CAS without changing the value of *ptr. This would cause ValueCAS to incorrectly report success!
Race between CAS and reload. If more than one thread is atomically modifying *ptr, then there's a window between the CAS and the reload where *ptr could be changed to anything at all — including getting set back to compare. Again, this would cause ValueCAS to incorrectly report success!

Another approach

One tempting way to fix it is to simply lie about the old value of *ptr. I didn't actually try this, but I'll admit I considered it for a moment:

    // BAD: do not use! Semantically incorrect!
    int ValueCAS(volatile int *ptr, int compare, int swap) {
        if (BoolCAS(ptr,compare,swap))
            return compare;
        else
            return (compare-1); // or any value other than compare
    }

But this is wrong! It breaks the semantics of the CAS. CAS is supposed to return the old value. But the value this function returns is totally bogus, and may never have existed at *ptr. To help imagine why this is bad, consider what might happen if the caller is using the value as a bitmask. We might wind up returning an illegal mask with bits that had never been set. This could have unintended, ugly, subtle side-effects.

Third time's the charm

As far as I can tell, the best way to fix it is by adding both an extra load and two branches:

    // GOOD: works correctly
    int ValueCAS(volatile int *ptr, int compare, int swap) {
        int old;
        do {
            if (BoolCAS(ptr,compare,swap))
                return compare;
            old = *ptr;
        } while (old == compare); // never return compare if CAS failed
        return old;
    }

And sure enough, it's a little heavy-handed, but it works. Proper branch hinting will help, but I'm still a bit surprised that it's so expensive (relatively speaking) to do this conversion.

Can anyone come up with a better way?

A Christmas Present to Myself: My New Office

2009-12-19T15:12:00.001-05:00

I've moved my home office from one room of the basement to another.

I know, how boring, but this has given me a much-appreciated chance to completely rip out everything and get rid of the mess and abandoned stuff that I don't really use any more. (Heck, back when I moved into my former office, I was still working for Apple.)

As part of the move, I'm making an effort to make everything as wireless and high-speed as I can.

Upgrades include:

Tons of electrical outlets. Spent several days with an electrician to get this all done. 3 drops, each three feet apart, and each drop has 4 outlets above the desk and 4 outlets below the desk. Plus we put a bonus 4-outlet drop on the side. Grand total: 28. Yes, twenty-eight! I'm incredibly happy with the result... you have no idea how much I hate power strips. :-)
Clean wired network. I still had some older Cat-5 cables which kept me from getting maximum throughput. Now all hubs are 1Gb Ethernet and cables are Cat-6 or Cat-5e. This should scale up to 10Gb Ethernet when the time comes.
Wireless speakers. Specifically, Harman-Kardon SoundSticks II attached to an AirPort Express, driven with iTunes and Airfoil. Yes, I poked fun at AirTunes when it first came out, but it's become surprisingly useful. I do virtually all of my work on a laptop these days, and when I'm in my office I want good sound without having a cable plugged in.
AirPort Base Station with simultaneous dual-band. I'm getting a little tired of upgrading base stations, honestly. But since I have a ton of devices on my wireless network, and use it to access other machines on my local net, having 802.11n actually run at 802.11n speed is worth it.
WiEx zBoost YX510-PCS-CEL cellular booster. Properly installed with an antenna on the roof, this has given me 5 bars — in the basement, with AT&T. Infinity percent better than what I could get before! Love it!
Hawking HBB1 broadband booster. OK, this provides almost no feedback so I'm not actually sure how I'm supposed to measure its effectiveness. But it's cheap, dead simple to set up, and the network feels more responsive with it enabled. With so many wireless devices and no QoS in Apple's AirPort, I figured that having some traffic shaping was better than none.
Wall-mounted shelf to hold all the networking gear (DSL modem, base station, cell booster, broadband booster). I prefer to keep things off the desk since (ideally) there should be no need to touch this stuff once it's running. Individual wall mounts are out since the walls are concrete. A simple black shelf from Home Depot did the trick nicely — now everything is elevated and out of the way.

I'm also asking Santa for these extras:

Atomic wall clock, because I enjoy having an analog clock around. And once you've had a clock that automatically sets itself from the WWVB atomic time broadcast, you can't go back.
Indoor/outdoor thermometer. My new desk is right next to a window, so this is easy. I want one with a wired probe, not wireless, because I prefer it to be reliable and low-maintenance.

(FYI, some of the above are affiliate links. I only ever do that for products that I use myself and that I'm very happy with.)

Removed, and recycled or craigslisted:

Several 40' runs of cable that are no longer needed, since my desk is now much closer to where the services come into the house.
Three power strips and two extension cords. Yay!
Quite a few mystery DC power adapters. Often something like a FireWire hub or USB DVD burner that I used to use and now don't, or a charger for something. I'm not a big gadget freak, but after five years in one place these definitely accumulate. (Seriously, wireless power can't get here fast enough.)
Last year's AirPort 802.11n base station. Sigh.
An old 32-bit Gateway PC that doesn't meet my needs any more.
PowerMac G5 that I no longer use.
2 Cinema Displays with ADC connections. These were really a great investment and they lasted an incredibly long time. But sorry ADC, you're dead, and converting you to DVI is an expensive pain in the ass.
Apple DVI to ADC adapter with broken plastic. Huge by modern standards - about the size of a Mac mini.
Ancient software and unread books that had piled up on my bookshelf. Yeah, I'm pretty sure that copy of DiskWarrior from 2002 isn't going to do me any good any more.
My home server. I used to maintain a web and file server at home. (It was a PowerMac G4, nearly ten years old now.) I didn't derive a huge benefit from it, but most of the time the cost was minimal too — just a constant-but-small amount of maintenance. But over time, I found that periodically it created an enormous amount of trouble and expense when a hard drive failed or I needed to upgrade something. I've realized that there's just no value in my messing around with that stuff any more, so I've switched to offsite hosting and online backup.

I've also ordered some new iMacs for work - to use both as a local distributed build cluster, and for local testing. I'm going to put Windows 7 x64 on one, and the other will have OSX 10.6 running a couple of Linux VMs.

One of the nice side benefits is that moving has really given me a chance to clean up the cabling. Every single cable I've installed so far is neat and tidy, either cut to length or neatly bundled with a cable tie — so everything is tangle-free. I'm sure this will only last right up until I need to reconfigure everything in a hurry for some new task, but it's pretty sweet for now. :-)

Done anything nice for your office lately?

The Economics of School Supplies

2009-12-03T11:02:00.001-05:00

I'm interested in economics for one particular reason: incentives are fascinating.

There's the psychology aspect: how can you influence people's behavior? To really do this well you need to understand how people think.

There's the engineering aspect: human society is an enormously complex system, even more complex than the systems we work with in technology today. We can do things to nudge a complex system in one direction or another, but we can't always predict what will happen to it.

And then there are the results: you provide incentives in order to get people to do what you want. And that's a powerful thing! I don't mean this in a cold-blooded way; everybody does this every day of their lives.

Incentives in the Classroom

My wife is a high school teacher, and we had an interesting talk the other day. It was about how she handles providing a classroom supply of extra paper, extra pens, and all that for kids to use when they forget their own.

In years past, she had a system for this. She kept a supply closet, and early in September would give the kids extra credit for bringing in up to two items for the supply closet. Since she did this at the beginning of the year, kids quickly donated all kinds of stuff to get the easy points, and the closet filled up. Before the year had even started she had a year's worth of supplies.

This year, for the first time, she consciously tried not to do that, because she reasoned that it's not really an academic thing — so it wasn't a good idea to give out points for it. Makes sense, right? And besides, the school provides a certain amount of supplies for the teachers which she might be able to use.

But I'm sure you can guess what happened. With almost 200 kids in and out of her classroom each day, all it takes is a few percent each day to need paper and it'll add up fast. So she very quickly ran through her school-provided allotment, and by November, she was bringing in supplies from home to make up the difference.

And thus the dilemma: should she offer extra credit again? Or stick to her principles and not give out points?

Three approaches

As we talked, we came up with basically three approaches to the problem.

Do nothing. This is what she had been doing, and she wasn't really happy with it. The kids could borrow from each other — and some did. Or she could provide her own supplies — which she had started to do. Or kids would just fail the day's assignment when they didn't have the supplies for it... but that's very hard on a public school teacher who wants every kid to succeed.
Positive incentives. This is the extra-credit approach. She had mixed feelings about this at first, until I pointed out that the extra credit was just a tiny fraction of their grade — something like 1% of a single quarter. That isn't going to turn a failing student into a passing one; it might turn a B into a B+, and so on, but that's all. And then she raised the excellent point that roughly 20% (!!) of their grade is class participation anyway. Providing supplies really helps the community of the classroom, just like class participation does, so that's a fine place to apply the extra credit.
Negative incentives. I suggested that another approach would be to penalize the students for coming to class unprepared. Charge kids 1 point (or even just 1/10th of a point) each day that they borrow paper or a pen. The size of the incentive doesn't matter, it just matters that it's negative.

Now, different people will have different feelings about each of these. And you can do different variations on them — charging a penny instead of a point for paper, etc. But those are the basic choices.

Creating an Atmosphere

In practice, both the positive and negative incentives should work. But I pointed out that the main difference between these two is not the first-order results of the incentive ... the main difference is how she will be perceived after applying each one.

Why? Well, even as you incentivize the students with one of these actions, they in turn give you an incentive back with how they react.

If you're the teacher who is chintzy about paper and deducts fractions of points for every little thing, you'll be a Scrooge and the students won't like you. Negative reinforcement yields negative emotions.

But on the other hand, what if you're the teacher who provides extra-credit and then free paper? Why, then you're the wonderfully generous and nice teacher whom they love. Positive reinforcement yields positive emotions.

It's almost Machiavellian: do you want to be feared or loved?

Knowing my wife as I do, in her classroom she works best when she's loved. And so I told her to go back to what she had been doing — give out extra credit to fill up the supply closet. It's funny that after actually dissecting the problem and analyzing it, we came up with the same solution that she'd done more or less instinctively in the first place.

Remembering Apollo 11

2009-07-20T01:51:00.001-04:00

Today marks the 40th anniversary of the Apollo 11 moon landing.

I hope you're listening to the mission audio, or following the JFK Presidential Library's webcast. The landing itself will be replayed at 4:17pm EDT, and Neil Armstrong's first steps should be at 10:56pm EDT. NASA has been doing a number of things for the occasion, including releasing some very cool photos from orbiters which show the Apollo landing sites as they exist today.

The anniversary is getting a lot of attention this year. It's on magazine covers, it's on TV, and there's an undercurrent of buzz among my geek friends about it.

So I figured I'd take a break from my usual hectic schedule of "not blogging" to share some thoughts.

The first-time cost

Going to the moon was fantastically expensive.

Just think: landing on the moon required an absolutely unparalleled effort. It required the US to bring together technology, manufacturing, research, money, management, a boatload of sciences (psychology, physiology, physics, chemistry, materials, culinary, more), training, politics, leadership, public relations, the military, hard work, and a bottomless amount of determination.

The men who walked on the moon did so at the pinnacle of an enormous pyramid of marshaled resources — virtually the entire resources of a nation, bent to a single purpose. And they did it all with 1960's technology, in a spacecraft that had less computational power than your phone does today. (Heck, probably less than your refrigerator.) This is the stuff of which legends are made.

That pyramid of resources was amazing — and fragile.

We got to the moon and it was pretty freakin' cool. But it wasn't cool enough.

Snapping back

After getting there the first time, we just kind of stopped. The last mission that landed on the moon was Apollo 17 in 1972, just a few short years after the first one.

Why? Well, the moon just wasn't that interesting.

Hang on, I'd like to clarify that. In fact there's a ton of interesting stuff on the moon. But it's probably fair to say the cost of that vast, overreaching pyramid of resources was greater than the return we were getting back.

The truth is that the moon really didn't have anything that 1970's America really needed; no useful minerals, no water, no oil, and we can't even live there. So it makes sense that after briefly overreaching ourselves just to get there, that there would be a period of time where the pendulum swung back: where we'd stop going entirely.

But there is one key thing to remember. It's not 1970 any more.

What a difference 40 years makes

The cost of everything trends downward over time. Or you can put it another way: our civilization gets comparatively wealthier over time, so that a constant cost eventually becomes much more affordable. It doesn't really matter which way you want to look at it, since they're two sides of the same coin.

In the 1960's the cost of putting a man into space — not just the raw materials and rockets, but the entire infrastructure — was so large that it could only be attempted by the world's largest industrial powers.

In the 2000's we've seen the cost fall to the point where it just takes the resources of a single large corporation. In the next two years we should see commercial sub-orbital space travel offered by Virgin Galactic. And you can bet Virgin will have competitors before long.

Space travel isn't quite "cheap", but it's getting surprisingly affordable. It won't be long before we're back at the moon. In fact, I'll go so far as to make a few easy predictions:

A man will walk on the moon again by 2019. That's less than ten years, so on or before the 50th anniversary of Apollo 11. NASA has a plan to do it, and so do the Chinese. Barring bureaucratic screwups, another Great Depression, or the singularity, it seems very likely.

A woman will walk on the moon by 2019. The title of "first woman on the moon" is still wide open, and there's no shortage of talented female astronauts. It's an easy bet that the first return mission will include one.

What can the moon offer us now?

Why would we go back, anyway? Science-fiction fans will have no shortage of ideas about the long-term future, but I find it more interesting to consider the short term — because that's really where the rubber meets the road.

What are the realistic, short-term benefits that we might get from the moon when we go back?

A few thoughts:

Tourism. We already have space tourism, where rich people pay millions of dollars simply to go into space. Once commercial spaceflight takes off — and Virgin Galactic plans its inaugural flight within just two years — how long do you think it'll be until someone wants a tour of the moon?
An outpost. Rather like McMurdo Station in Antarctica, an outpost on the moon would be a center for scientific research and exploration, as well as a central base for other activities.
Mining, and ultimately manufacturing. Silicon, aluminum, and iron are probably the most useful and easily-extracted resources in the short term. (Many people talk about Helium-3, but it's not currently exploitable so I'm leaving it off.) But why would we mine the moon for this stuff instead of getting it from Earth? It's useful, but not exceptionally scarce.
The answer is really that it's about relative scarcity. These raw materials are not exceptionally scarce here on Earth, but they sure are hard to come by up in orbit!
It's a question of cost again. It's expensive to get things out of the earth's gravity well. If you want something even moderately large on the moon or in space — habitats, satellites, ships, mirrors, radiation shields — it may be cheaper to build it in place, with materials from the moon instead of from Earth.

I'd guess that these things are likely to happen in roughly that order: Tourism first, because there is a huge pent-up demand. This will be followed soon afterward by a small outpost — mostly for research, but also supporting (and partially funded by) tourism. And finally the mining and manufacturing, which can then can be used to build up the outpost even more.

Even though we've gotten used to space travel as a NASA thing, it's possible these may not be governmental projects at all, or they may be hybrid ventures which are part commercial and part subsidized. Either way, the feeling that the pendulum is finally swinging back is terrific. I can't wait!

Now it's your turn. What do you think the moon will have to offer us during the 2020's?

The Case Against Insensitivity

2007-12-02T02:36:00.001-05:00

One of the most controversial parts of my earlier post, Don't Be a ZFS Hater, was when I mentioned off-handedly in the comments that I don't like case-insensitivity in filesystems.

Boy, did that spur a storm of replies.

I resolved to not pollute the ZFS discussion with a discussion of case-insensitivity and promised to make a separate blog post about it. It took a while, but this is that post. I blame a busy work schedule and an even busier travel schedule. (Recently in the span of two weeks I was in California, Ohio, London, Liverpool, London, Bristol, London, Amsterdam, London, then back to Ohio. Phew!)

Here's Why Case-Insensitive Filesystems Are Bad

I've worked in and around filesystems for most of my career; if not in the filesystem itself then usually a layer just above or below it. I'm speaking from experience when I tell you:

Case-insensitivity is a bad idea in filesystems.

And here's why:

It's poorly defined.
Every filesystem does it differently.
Case-insensitivity is a layering violation.
Case-insensitivity forces layering violations upon other code.
Case-insensitivity is contagious.
Case-insensitivity adds complexity and provides no actual benefit.

I'll expand on each of these below.

It's poorly defined

When I say "case-insensitive", what does that mean to you?

If you only speak one language and that language is English, it probably seems perfectly reasonable: map the letter a to A, b to B, and so on through z to Z. There, you're done. What was so hard about that?

But that's ASCII thinking; the world left that behind a long time ago. Modern systems are expected to deal with case differences in all sorts of languages. Instead of a simple 26-letter transformation, "case insensitivity" really means handling all the other alphabets too.

The problem with doing that, however, is that it brings language and orthography into the picture. And human languages are inherently vague, large, messy, and constantly evolving.

Can you make a strict definition of "case insensitivity" without any hand-waving?

One way to do it is with an equivalence table: start listing all the characters that are equal to other characters. We can go through all the variants of Latin alphabets, including a huge list of accents: acute, grave, circumflex, umlaut, tilde, cedilla, macron, breve, dot, ring, ogonek, hacek, and bar. Don't forget to find all the special ligatures and other letters, too, such as Æ vs æ and Ø vs ø.

Okay, our table is pretty big so far. Now let's start adding in other alphabets with case: Greek, Armenian, and the Cyrillic alphabets. And don't forget the more obscure ones, like Coptic. Phew. It's getting pretty big.

Did we miss any? Well, for any given version of the Unicode standard it's always possible to enumerate all letters, so it's certainly possible to do all the legwork and prove that we've got all the case mappings for, say, Unicode 5.0.0 which is the latest at the time of this writing. But Unicode is an evolving standard and new characters are added frequently. Every time a new script with case is added we'll need to update our table.

There are also some other hard questions for case insensitivity:

Digraph characters may have three equivalent mappings, depending on how they are being written: all-lowercase, all-uppercase, or title-case. (For example: ǳ, Ǳ, or ǲ.) But this breaks some case-mapping tables which didn't anticipate the need for an N-way equivalence.
The German letter ß is considered equal to lowercase ss. Should "Straße" and "STRASSE" be considered equivalent? They are in German. But this breaks some case-mapping tables which didn't anticipate the need for an N-to-M character translation (1:2, in this case).
Capital letters can significantly alter the meaning of a word or phrase. In German, capital letters indicate nouns, so the word Essen means "food", while the word essen means "to eat". We make similar distinctions in English between proper nouns and regular nouns: God vs god, China vs china, Turkey vs turkey, and so on. Should "essen" and "Essen", or "china" and "China" really be considered equivalent?
Some Hebrew letters use different forms when at the end of a word, such as פ vs ף, or נ vs ן. Are these equivalent?
In Georgian, people recently experimented with using an obsolete alphabet called Asomtavruli to reintroduce capital letters to the written language. What if this had caught on?
What about any future characters which are not present in the current version of the Unicode standard?

Case is a concept that is built into written languages. And human language is inherently messy. This means that case-insensitivity is always going to be poorly defined, no matter how hard we try.

Every filesystem does it differently

Unfortunately, filesystems can't engage in hand-waving. Filesystem data must be persistent and forward-compatible. People expect that the data they wrote to a disk last year should still be readable this year, even if they've had an operating system upgrade.

That's a perfectly reasonable expectation. But it means that the on-disk filesystem specification needs to freeze and stop changing when it's released to the world.

Because our notion of what exactly "case-insensitive" means has changed over the past twenty years, however, we've seen a number of different methods of case-insensitivity emerge.

Here are a handful of the most popular case-insensitive filesystems and how they handle case-mapping:

FAT-32: ASCII upper- and lower-case letters, but a-z and A-Z are considered identical. Also variable IBM code pages in high ASCII.
HFS: ASCII upper- and lower-case letters, but a-z and A-Z are considered identical. Also variable Mac encodings in high ASCII.
NTFS: Case-insensitive in different ways depending on the version of Windows that created the volume.
HFS+: Case-insensitive with a mapping table which was frozen circa 1996, and thus lacks case mappings for any newer characters.

None of these — except for NTFS created by Vista — are actually up-to-date with the current Unicode specification. That's because they all predate it. Similarly, if a new filesystem were to introduce case-insensitivity today, it would be locked into, say, Unicode 5.0.0's case mappings. And that would be all well and good until Unicode 5.1.0 came along.

The history of filesystems is littered with broken historical case mappings like a trail of tears.

Case-insensitivity is a layering violation

When people argue for case-insensitivity in the filesystem, they almost always give user interface reasons for it. (The only other arguments I've seen are based on contagion, which I'll talk about in a moment.) Here is the canonical example:

My Aunt Tillie doesn't know the difference between letter.txt and Letter.txt. The filesystem should help her out.

But in fact this is a UI problem. The problem relates to the display and management of information, not the storage of this information.

Don't believe me?

When any application displays items in a window, who sorts them case-insensitively? The filesystem? No! The application does it.
When you type-select, typing b-a-b-y to select the folder "Baby Pictures" in an application, who does the case-insensitive mapping of the letters you type to the files you select? The filesystem? No! The application again.
When you save or copy files, who does the case-insensitive test to warn you if you're creating "file.txt" when "File.txt" already exists? The filesystem? Yes!

Why does the third question have a different answer than the rest?

And we've already talked about how filesystems are chronically out-of-date with their case mappings. If your aunt is a Turkish Mac user, for example, she's probably going to notice that the behavior of the third one is different for no good reason. Why are you confusing your Aunt Tülay?

One last point was summarized nicely by Mike Ash in the comments of Don't Be a ZFS Hater. I'll just quote him wholesale here:

Yes, Aunt Tillie will think that "Muffin Recipe.rtf" and "muffin recipe.rtf" ought to be the same file. But you know what? She'll also think that "Muffin Recipe .rtf" and "Recipe for Muffins.rtf" and "Mufin Recipe.txt" ought to be the same file too.

Users already don't generally understand how the OS decides whether two files are the same or not. Trying to alleviate this problem by mapping names with different case to the same file solves only 1% of the problem and just isn't worth the effort.

I agree completely.

Case-insensitivity forces layering violations upon other code

All too often, pieces of code around the system are required to hard-code knowledge about case-insensitive filesystem behavior. Here are a few examples off the top of my head:

Collision prediction. An application may need to know if two files would conflict before it actually writes either of them to disk. If you are writing an application where a user creates a group of documents — a web page editor, perhaps — you may need to know when banana.jpg and BANANA.JPG will conflict.

The most common way that programmers solve this is by hard-coding some knowledge about the case-insensitivity of the filesystem in their code. That's a classic layering violation.
Filename hashing. If you are writing code to hash strings that are filenames, you probably want equivalent paths to generate the same hash. But it's impossible to know which files are equivalent unless you know the filesystem's rules for case-mapping.

Again, the most common solution is a layering violation. You either hard-code some knowledge about the case-insensitivity tables, or you hard-code some knowledge about your input data. (For example, you may just require that you'll never, never, ever have multiple access paths for the same file in your input data. Like all layering violations, that might work wonderfully for a while ... right up until the day that it fails miserably.)

I'm sure there are more examples out there.

Case-insensitivity is contagious

This is the worst part. It's all too easy to accidentally introduce a dependence on case-insensitivity: just use an incorrect path with bad case.

The moment somebody creates an application or other system that inadvertently depends on case-insensitivity, it forces people to use a case-insensitive filesystem if they want to use that app or system. And that's one of the major reasons why case-insensitivity has stuck around — because it's historically been very difficult to get rid of.

I've seen this happen with:

Source code. Some bozo writes #include "utils.h" when the file is named Utils.h. Sounds innocent enough, until you find that it's repeated dozens of times across hundreds of files. Now that project can only ever be compiled on a case-insensitive filesystem.
Game assets. A game tries to load lipsync.dat instead of LIPSYNC.DAT. Without knowing it, the artist or developer has accidentally locked that game so that it can only run on a case-insensitive filesystem. (This causes real, constant problems in game pipelines; teams create and test their games on case-insensitive NTFS and don't notice such problems until it's burned to a case-sensitive UDF filesystem on DVD or Blu-Ray.)
Application libraries. DLLs and shared library references are sometimes generated by a build script which uses the wrong case. When that happens, the application may simply fail to launch from a case-sensitive filesystem.
Miscellaneous data files. Sometimes an application will appear to run on a case-sensitive filesystem but some feature will fail to work because it fails to load a critical data file: the spell-checking dictionary, a required font, a nib, you name it.

Happily, since Mac OS X shipped in 2001, Apple has been busy solving its own problems with case-insensitivity and encouraging its developers to test with case-sensitive filesystems. Two important initiatives in this direction have been NFS home directories and case-sensitive HFSX.

The upshot of it is that Mac OS X is actually very friendly to case-sensitive disks these days; very little that's bad happens when you use case-sensitive HFSX today.

Case-insensitivity adds complexity with no actual benefit

I'm going to make an assertion here:

ONE HUNDRED PERCENT of the path lookups happening on your Mac right now are made with correct case.

Think about that for a moment.

First off, you may think this contradicts the point I just made in the previous section. Nope; I'm simply rounding. The actual figure is something like 99.999%, and I'd probably get tired of typing 9's before I actually approached the real number. There are infinitesimally few path accesses made with incorrect case compared to the ones that are made with the proper case.

Modern computers make hundreds of filesystem accesses per second. As I type this single sentence in MarsEdit on Mac OS X 10.4.11, my computer has made 3692 filesystem accesses by path. (Yes, really. MarsEdit's "Preview" window is invoking Perl to run Markdown, which loads a handful of modules, and then WebKit re-renders the page. That's a lot of it, but meanwhile there's background activity from Mail, Activity Monitor, iChat, SystemUIServer, iCalAlarmScheduler, AirPort Base Station Agent, Radioshift, NetNewsWire, Twitterrific, and Safari.)

Under Mac OS X you can measure it yourself with this command in Terminal:

sudo fs_usage -f filesys | grep / > /tmp/accesses.txt

The vast majority of file accesses are made with paths that were returned from the filesystem itself: some bit of code read the contents of a directory, and passed the results on to another bit of code, which eventually decided to access one of those files. So most of the time the filesystem is getting back the paths that it has returned earlier. Very very few accesses are made with paths that come directly from an error-prone human, which is why essentially 100% of filesystem accesses are made with correct case.

But if essentially all filesystem accesses are made with the correct case to begin with, why do we even have case-insensitivity at all?

We've already discussed the problems of contagion, which is a circular justification: we have to do it because someone else did it first. We've also discussed UI decisions being incorrectly implemented in the bottommost layer of the operating system. Other than those two, what good is it?

I don't have an answer to that. For the life of me I can't come up with any reason to justify case-insensitive filesystems from a pure design standpoint. That leads me to my closing argument, which is...

A thought experiment

Suppose case-insensitive filesystems had never been invented. You're the leader of a team of engineers in charge of XYZZYFS, the next big thing in filesystems. One day you tell the other people who work on it:

"Hey! I've got this great idea! It's called case-insensitivity. We'll take every path that comes into the filesystem and compare it against a huge table to create a case-folded version of the path which we'll use for comparisons and sorting. This will add a bunch of complexity to the code, slow down all path lookups, increase our RAM footprint, make it more difficult for users of our filesystem to handle paths, and create a compatibility nightmare for future versions if we ever decide to change the table. But, you see, it'll all be worth it, because... _________________."

Can you fill in the blank?

ZFS Hater Redux

2007-10-10T06:19:00.001-04:00

MWJ has responded to my last post, Don't Be a ZFS Hater, with a post of their own: You don't have to hate ZFS to know it's wrong for you.

I don't like the point-by-point quote and response format — it's way too much like an old-school Usenet flamewar. So I will simply try to hit the high points of their arguments.

Where we agree

ZFS is not ready to deploy to the entire Mac OS X user base today. There's still some work to be done.
ZFS isn't necessary for most of today's Macintosh computers. If you have been using your Mac with no storage-related problems, then you can keep on using it that way. Perform regular backups and you'll be just fine.
It would be an absolutely terrible idea to take people's perfectly working HFS+ installations on existing computers and forcibly convert them to ZFS, chuckling evilly all the while. Not quite sure where that strawman came from.
ZFS fatzaps are expensive for small files. If it were true that 20% of the files in a Mac OS X installation required a fatzap (pdf link to ZFS-on-disk specification), that would indeed be unnecessarily wasteful.
A typical Mac OS X 10.4.x installation has on the order of about 600,000 files.

I think that's about it. But of course there are a number of places where we disagree too.

ZFS would be awfully nice for a small segment of the Mac OS X user base if it were ready today.

If you spend any amount of time managing storage — if drives have gone bad on you, if you have ever run out of space on a desktop system and needed to add a drive (or two), if you have a RAID array — then you are the sort of user that could see some immediate benefit.

But of course as we already agreed, it's not ready today. You haven't been "cheated" and I'm sure you don't feel that way. But feel free to look forward to it: I sure am.

ZFS — or something with all the features of ZFS — will be more than nice, it will be necessary for tomorrow's Macintosh computers.

Both storage sizes and consumer consumption of storage grow exponentially. I tried to make this point last time, but MWJ seems to have misunderstood and accused me of misquoting. Let's try again.

In 1997, 20GB of storage meant a server RAID array. Ten years later, in 2007, 20GB of storage is considered "not enough" by most people. Across my entire household I have drives larger than that in my computer, in my TiVo, in my PlayStation 3, and even in my iPod. Now let's extrapolate that into the future.

In 2007, 20TB of storage means a server RAID array. Ten years from now, in 2017, 20TB of storage will similarly be considered "not enough". MWJ scoffed at ZFS because it's really pretty good at the problems of large storage. But you know what? A solution to managing that much data will need to be in place in Mac OS X well before 20TB drives become the norm. Better hope someone's working on it today.

Meanwhile — and this is what scares the pants off me — the reliability numbers for hard drives have improved much more slowly than capacity.

Here's a fairly typical Seagate drive with a capacity of ~150GB = ~1.2 x 10¹² bits. The recoverable error rate is listed as 10 bits per 10¹² bits. Let's put those numbers together. That means that if you read the entire surface of the disk, you'll typically get twelve bits back that are wrong and which a retry could have fixed. (Updated Oct 11 2007: In the comments, Anton corrected me: I should've used the unrecoverable error rate here, not the recoverable error rate. The net result is that in ideal operating conditions bit errors occur over 100x less frequently than I originally suggested. However, it's still not zero. The net result is still a looming problem when you scale it across (installed base) x (storage consumption) x (time). See the comment thread.)

Yes, really. Did you catch the implications of that? Silent single-bit errors are happening today. They happen much more often at high-end capacities and utilizations, and we often get lucky because some types of data (video, audio, etc) are resistant to that kind of single-bit error. But today's high end is tomorrow's medium end, and the day after tomorrow's low end. This problem is only going to get worse.

Worse, bit errors are cumulative. If you read and get a bit error, you might wind up writing it back out to disk too. Oops! Now that bit error just went from transient to permanent.

Still think end-to-end data integrity isn't worth it?

Apple using ZFS rather than writing their own is a smart choice.

As I hope I made abundantly clear in the last post, extending HFS+ to the future that we can see looming is just not an option — its structure is simply too far removed from these problems. It's really just not worth it. It's pretty awesome that the original HFS design scaled as far as it did: how many people can come up with a 20-year filesystem? But you have to know when to throw in the towel.

So if you accept that the things I described above are real, looming problems, then Apple really does need a filesystem with at least several of the more important attributes of ZFS.

The choices at this point are essentially twofold: (1) start completely from scratch, or (2) use ZFS. There's really no point in starting over. ZFS has a usable license and has been under development for at least five years by now. By the time you started over and burned five years on catching up it would be too late.

And I really do want to reiterate that the shared community of engineers from Apple, Sun, and FreeBSD working on ZFS is a real and measurable benefit. I've heard as much from friends in CoreOS. I can't understand the hostility to this very clear and obvious fact. It's as if Apple suddenly doubled or tripled the number of filesystem engineers it has available, snagging some really brilliant guys at the top of their profession in the process, and then multiplied its testing force by a factor of 10.

(To respond to a query voiced by MWJ, HFS+ never gathered that community when it was open-sourced because the design was already quite old at that point. It frankly didn't have anything new and exciting to offer, and it was saddled with performance problems and historical compromises of various kinds, so very few people were interested in it.)

ZFS fatzaps are unlikely to be a significant problem.

This gets a bit technical. Please skip this section if you don't care about this level of detail.

MWJ really pounded on this one. That was a bit weird to me, since it seemed to be suggesting that Apple would not expend any engineering effort on solving any obvious glaring problems with ZFS before releasing it. That's not the Apple I know.

But okay, let's suppose that we're stuck with ZFS and Mac OS X both frozen as they stand today. Let's try to make an a priori prediction of the actual cost of ZFS fatzaps on a typical Mac OS X system.

Classic HFS attributes (FinderInfo, ExtendedFinderInfo, etc) are largely unnecessary and unused today because the Finder uses .DS_Store files instead. In the few cases where these attributes are set and used by legacy code, they should fit easily in a small number of microzaps.
Extended attributes may create fatzaps. Today it seems like extended attributes are typically used on large files: disk images, digital photos, etc. This may provoke squawking from the peanut gallery, but once a file is above a certain size — roughly a couple of megabytes — using an extra 128KiB is negligible. If you have a 4MiB file and you add 128KiB to track its attributes, big deal: you've added 3%. It's not nothing, but it's hardly a significant problem.
Another likely source of fatzaps in ZFS on Mac OS X is the resource fork. But with Classic gone, new Macs ship with virtually no resource forks on disk. There are none in the BSD subsystem. There are a handful in /System and /Library, mostly fonts. The biggest culprits are large old applications like Quicken and Microsoft Office. A quick measurement on my heavily-used one-year-old laptop shows that I have exactly 1877 resource forks out of 722210 files — that's 0.2%, not 20%.
(Fun fact: The space that would be consumed by fatzap headers for these resource files comes out to just 235 MiB, or roughly six and a half Keyboard Software Updates. Again: not nothing, but hardly a crisis to scream about.)

Want to measure it yourself? Amit Singh's excellent hfsdebug utility will show you a quick summary. Just run "sudo hfsdebug -s" and look at the numbers for "files" and "non-zero resource forks". Or try "sudo hfsdebug -b attributes -l any | less" to examine the files which have extended attributes on your disk.

ZFS snapshots don't have to be wasteful

The cheesesteak analogy was cute. But rather than imagining that snapshots just eat and eat and eat storage until you choke in a greasy pile of death, it would help if we all actually understand how hard drive storage is actually used in practice, and how ZFS can work with that.

There are three major classes of stored data.

Static data is data that you want to keep and almost never modify. This is your archive. Photographs, music, digital video, applications, email, etc. Archives are additive: unless you really run out of room, you rarely delete the old — you only add new stuff. You want the contents safe and immediately accessible, but they are essentially unchanging.

Snapshotting static data is close enough to free that you won't notice: the only cost is the basic cost of the snapshot. No extraneous data copies are ever created, because you never modify or delete this stuff anyway.

Dynamic data is data that you want to keep, but are modifying with some frequency. This is whatever you are working on at the moment. It might be writing a novel, working in Photoshop, or writing code: in all cases you keep saving new versions over the old.

Snapshotting dynamic data is more expensive, because if you do it too much without recycling your old snapshots then you can build up a large backlog.

Transient data is data that should not be persistent at all. These are your temporary files: local caches, scratch files, compiler object files, downloaded zip files or disk images, etc. These may be created, modified, or deleted at any moment.

Snapshotting transient data is generally a bad idea — by definition you don't care that much about it and you'd prefer it to be deleted immediately.

Got all that? Okay. Now I need to make a couple of points.

First, I assert that virtually all of the data on personal computer hard drives is static most of the time. Think about that. The operating system is static the whole time you are using it, until you install a system update. (And even then, usually just a few hundred megabytes change out of several gigabytes.) Your /Applications folder is static. Your music is static. And so on. Usually a few percent of your data is dynamic, and a few more percent is transient. But in most cases well over 95% is static. (Exceptions are easy to come up with: Sometimes you generate a large amount of transient data while building a disk image in iDVD or importing DV footage. That can shift the ratio below 95%. But once that task is complete you're back to the original ratio.)

Second, the biggest distinction that matters when snapshotting is separating persistent data from transient data. Taking snapshots of transient data is what will waste disk space in a hurry. Taking snapshots of dynamic data as a local backup is often valuable enough that it's okay to burn the small amount of disk space that it takes, because remember: that's the actual data that you're actively working on. And as we already mentioned, snapshots of static data are free.

Now here's where it gets interesting.

With ZFS, snapshots work on the filesystem level. Because it no longer uses the "big floppy" model of storage, new filesystems are very cheap to create. (They are almost as lightweight as directories, and often used to replace them.) So let's create one or more special filesystems just for transient data and exclude them from our regular snapshot process. In fact on Mac OS X that's easy: we have well-defined directories for transient data: ~/Library/Caches, /tmp, and so on. Link those all off to one or more transient filesystems and they will never wind up in a snapshot of the important stuff. I wouldn't expect users to do this for themselves, of course — but it could certainly be set up that way automatically by Apple.

Once the transient data is out of the picture, our snapshots will consist of 95% or more static data — which is not copied in any way — and a tiny percentage of dynamic data. And remember, the dynamic data is not even copied unless and until it changes. The net effect is very similar to doing an incremental backup of exactly and only the files you are working on. This is essentially a perfect local backup: no duplication except where it's actually needed.

Will you want to allow snapshots to live forever? Of course not. One reasonable model for taking backup snapshots might be to remember 12 hourly snapshots, 7 daily snapshots, and 4 weekly snapshots. If you are getting tight on storage the system could take new snapshots less frequently and expire them more aggressively. Remember: when nothing is changing the snapshots don't take up any space.

Wrap-up: Listen to the smart guys

Some very smart people at Sun started the ball rolling by putting an awful lot of thought into the future of storage, and they came up with ZFS.

After they announced it and started talking about it, other brilliant people at Apple (and FreeBSD, and NetBSD) paid attention to what they were doing. And they listened, and thought about it, and looked at the code, and wound up coming around to the side of ZFS as well.

If you think I'm smart, just know that I'm in awe of some of the guys who've been involved with this project.

If you think I'm stupid, why, I look forward to hearing from you in the comments.

Don't be a ZFS Hater

2007-10-06T03:03:00.002-04:00

John Gruber recently linked to — and thus gave credibility to — a MWJ post ripping on a fairly reasonable AppleInsider post about ZFS. Representative quote:

“We don't find HFS Plus administration to be complex, and we can't tell you what those other things mean, but they sound really cool, and therefore we want them. On the magic unlocked iPhone. For free.”

Har har har. Wait. Hold on a minute. Why is it suddenly fashionable to bash on ZFS?

Part of it is a backlash to the weird and obviously fake rumor about it becoming the default in Leopard, I guess. (No thanks to Sun's CEO Jonathan Schwartz here, who as far as I know has never publicly said anything about why he either misspoke or misunderstood what was going on back in June.)

But don't do that. Don't be a ZFS hater.

A word about my background

Let's get the credentials out of the way up front. Today I work on a file I/O subsystem for PlayStation 3 games. Before that, I worked in Apple's CoreOS filesystems group. Before that, I worked on DiscRecording.framework, and singlehandedly created the content subframework that streamed out HFS+, ISO-9660, and Joliet filesystems. Before that, I worked on the same thing for Mac OS 9. And before that, I worked on mass storage drivers for external USB/FireWire drives and internal ATA/ATAPI/SCSI drives.

You might say I know a thing or two about filesystems and storage.

What bugged me about the article

ZFS is a fine candidate to replace HFS+ eventually. It's not going to happen overnight, no. And it'll be available as an option for early adopters way before it becomes the default. But several years from now? Absolutely.
The bizarre rants about ZFS wasting processor time and disk space. I'm sorry, I wasn't aware that we were still using 30MHz machines with 1.44MB floppies. ZFS is great specifically because it takes two things that modern computers tend to have a surplus of — CPU time and hard disk space — and borrows a bit of it in the name of data integrity and ease of use. This tradeoff made very little sense in, say, 1992. But here in 2007 it's brilliant.
Sneeringly implying that HFS+ is sufficient. Sure, HFS+ administration is simple, but it's also inflexible. It locks you into what I call the "big floppy" model of storage. This only gets more and more painful as disks get bigger and bigger. Storage management has come a long way since the original HFS was created, and ZFS administration lets you do things that HFS+ can only dream of.
Claiming that RAID-Z is required for checksums to be useful. This is flat-out wrong. Sure, RAID-Z helps a lot by storing an error-correcting code. But even without RAID-Z, simply recognizing that the data is bad gets you well down the road to recovering from an error — depending on the exact nature of the problem, a simple retry loop can in fact get you the right data the second or third time. And as soon as you know there is a problem you can mark the block as bad and aggressively copy it elsewhere to preserve it. I suppose the author would prefer that the filesystem silently returned bad data?
Completely ignoring Moore's Law. How dumb do you need to be to willfully ignore the fact that the things that are bleeding-edge today will be commonplace tomorrow? Twenty gigabyte disks were massive server arrays ten years ago. Today I use a hard drive ten times bigger than that just to watch TV.

Reading this article made me feel like I was back in 1996 listening to people debate cooperative vs preemptive multitasking. In the Mac community at that time there were, I'm ashamed to say, a lot of heated discussions about how preemptive threading was unnecessary. There were some people (like me) who were clamoring for a preemptive scheduler, while others defended the status quo — claiming, among other things, that Mac OS 8's cooperative threads "weren't that bad" and were "fine if you used them correctly". Um, yeah.

Since then we've thoroughly settled that debate, of course. And if you know anything about technology you might be able to understand why there's a difference between "not that bad" and "a completely new paradigm".

ZFS is cool

Let's do a short rundown of reasons why I, a qualified filesystem and storage engineer, think that ZFS is cool. I'll leave out some of the more technical reasons and just try to keep it in plain English, with links for further reading.

Logical Volume Management. Hard disks are no longer big floppies. They are building blocks that you can just drop in to add storage to your system. Partitioning, formatting, migrating data from old small drive to new big drive -- these all go away.
Adaptive Replacement Caching. ZFS uses a smarter cache eviction algorithm than OSX's UBC, which lets it deal well with data that is streamed and only read once. (Sound familiar? It could obsolete the need for F_NOCACHE.)
Snapshots. Think about how drastically the trash can metaphor changed the way people worked with files. Snapshots are the same concept, extended system-wide. They can eliminate entire classes of problems.
I don't know about you, but in the past year I have done all of the following, sometimes more than once. Snapshots would've made these a non-issue:
- installed a software update and then found out it broke something
- held off on installing a software update because I was afraid something might break
- lost some work in between backups
- accidentally deleted an entire directory with a mistyped rm -rf or SCM delete command.
Copy-on-write in the filesystem makes snapshots super-cheap to implement. No, not "free", just "so cheap you wouldn't possibly notice". If you are grousing about wasted disk space, you don't understand how it works. Mac OS X uses copy-on-write extensively in its virtual memory system because it's both cheap and incredibly effective at reducing wasted memory. The same thing applies to the filesystem.
End-to-end data integrity. Journaling is the only thing HFS+ does to prevent data loss. This is hugely important, and big props are due to Dominic Giampaolo for hacking it in. But journaling only protects the write stage. Once the bits are on the disk, HFS+ simply assumes they're correct.
But as disks get larger and cheaper, we're finding that this isn't sufficient any more. The odds of any one bit being wrong are very small. And yet the 200GB hard disk in my laptop has a capacity of about 1.6 trillion bits. The cumulative probability that EVERY SINGLE ONE of those bits are correct is effectively zero. Zero!
Backups are one answer to this problem, but as your data set gets larger they get more and more expensive and slow. (How do you back up a terabyte's worth of data? How long does it take? Worse still, how do you really know that your backup actually worked instead of just appearing to work?) So far, disk capacity has consistently grown faster than disk speed, meaning that backups will only continue to get slower and slower. Boy, wouldn't it be great if the filesystem — which is the natural bottleneck for everything disk-related — helped you out a little more on this? ZFS does.
Combinations of the above. There are some pretty cool results that fall out of having all of these things together in one place. Even if HFS+ supported snapshots, you'd still be limited by the "big floppy" storage model. It really starts to get interesting when you combine snapshots with smart use of logical volume management. And we've already discussed how RAID-Z enhances ZFS's basic built-in end-to-end data integrity by adding stronger error correction. There are other cool combinations too. It all adds up to a whole which is greater than the sum of its parts.

Is any of this stuff new and unique to ZFS? Not really. Bits and pieces of everything I've mentioned above have showed up in many places.

What ZFS brings to the table is that it's the total package — everything all wrapped up in one place, already integrated, and in fact already shipping and working. If you happened to be looking for a next-generation filesystem, and Apple is, you wouldn't need to look much further than ZFS.

Still not convinced?

Okay, here are three further high-level benefits of ZFS over HFS+:

Designed to support Unix. There are a lot of subtleties to supporting a modern Unix system. HFS+ was not really designed for that purpose. Yes, it's been hacked up to support Unix permissions, node locking, lazy zero-fill, symbolic links, hard links, NFS readdir semantics, and more. Some of these were easy. Others were painful and exhibit subtle bugs or performance problems to this day.
Designed to support modern filesystem concepts. Transactions. Write cache safety. Sparse files. Extended metadata attributes. I/O sorting and priority. Multiple prefetch streams. Compression. Encryption. And that's just off the top of my head.
HFS+ is 10-year-old code built on a 20-year-old design. It's been extended to do some of this, and could in theory be extended to do some of the others... but not all of them. You'll just have to trust me that it's getting to the point where some of this stuff is just not worth the engineering cost of hacking it in. HFS+ is great, but it's getting old and creaky.
Actually used by someone besides Apple. Don't underestimate the value of a shared standard. If both Sun and Apple start using the same open-source filesystem, it creates a lot of momentum behind it. Having more OS clients means that you get a lot more eyes on the code, which improves the code via bugfixes and performance enhancements. This makes the code better, which makes ZFS more attractive to new clients, which means more eyes on the code, which means the code gets better.... it's a virtuous circle.

Is ZFS the perfect filesystem? I doubt it. I'm sure it's got its limitations just like any other filesystem. In particular, wrapping a GUI around its administration options and coming up with good default parameters will be an interesting trick, and I look forward to seeing how Apple does it.

But really, seriously, dude. The kid is cool. Don't be like that.

Don't be a ZFS hater.

Updates:

MWJ's response: You don't have to be a ZFS hater to know it's wrong for you.
My followup: ZFS Hater Redux

MarsEdit Markdown Scripts updated to 1.0.3

2007-10-04T16:18:00.001-04:00

No piece of software, however simple, is bug-free.

Daniel Jalkut was kind enough to point out to me that my Markdown scripts for MarsEdit didn't deal properly with Unicode text.

Silly me. I'd forgotten that AppleScript (with its very early-1990s roots) still needs to be explicitly told not to lose data when writing out files. He offered a fix — a few «class utf8» coercions in the right place and all was well again. That was 1.0.1.

However, just immediately after release, I discovered that the scripts needed to do some more aggressive transcoding of UTF8 into ASCII in order to get Python to read the file in html2text. So I've added support for that. This slows down the HTML-to-Markdown reverse conversion a little bit, but at least it's correct now. That was 1.0.2.

Finally, later in the evening I realized it was pretty stupid to write AppleScript code to transcode the UTF8 into ASCII, because AppleScript's support for Unicode is so horribly primitive and I was doing it character by character. So I rewrote that whole section as a one-line Perl script. Now it's blazing fast. And that's 1.0.3.

I've updated the scripts. Download them now!

Microbenchmarking

2007-09-04T02:26:00.002-04:00

I want to talk a little about a technique that can be very useful. Like all programming tools, however, it's a double-edged sword and can easily be abused.

I'm talking about microbenchmarks. A microbenchmark is a form of benchmark. Let's quickly define our terms:

Benchmark: noun: a relative measure of the performance of an operation (or set of operations) in computer software. Many factors may be used to create this measure, such as memory use or I/O, but when unspecified the measure is usually speed.; verb: to create or run a benchmark.

Microbenchmarks

Sometimes the term "benchmark" is too broad. I want to specifically talk about single benchmarks that are limited in scope. One commonly used term for this is "microbenchmark".

Microbenchmark: noun: a benchmark that measures the performance of a single operation which is a very small portion of a larger system.; verb: to create or run a microbenchmark.

I would consider each of the following to be an example of a microbenchmark:

Implementation comparisons. Suppose you are choosing between multiple implementations of the C standard library routine strcmp(). You might use a microbenchmark to compare different implementations against each other until you find the one that's right for you.
Equivalent function comparison. Sometimes there's more than one way to achieve your goal. If you want to know which is faster, you might use a microbenchmark to compare the different functions.
Inter-language primitive comparisons. Microbenchmarks are often used to compare programming languages. For example, Java, C++, and C# are often microbenchmarked to see which has the faster string comparison, stack allocation, heap allocation, and so on.
Individual aspects of a large system's performance. Examples might include adding a row to a database table, sorting a large list of input files, or even application launch time. These aspects may be measured in isolation (i.e., without a strict performance target) with the presumption that speeding them up is generally useful, since the end user's experience is comprised of lots of these small operations.

I think the last example above is the most generic description of a microbenchmark, because it encompasses the others. When you are benchmarking strcmp(), you are really measuring an individual aspect of the C library's performance. And when you are comparing language primitives, you are again measuring an individual aspect of the language's performance.

The prefix "micro-" is relative, of course. If your job description is "Chief Programmer: strcmp()", then measuring the performance of strcmp() is simply benchmarking, not microbenchmarking. But if you are working on a large modern GUI application for a consumer OS, measuring the performance of strcmp() becomes a microbenchmark because it's such a small part of your overall system.

Losing Track of the Macro

Before I start talking about how to do microbenchmarks, there's an important point to be made. Improperly applied microbenchmarks can actually be hazardous to real-world performance.

Benchmarking goes hand-in-hand with optimization when it's done correctly. You benchmark first, and then optimize. Similarly, microbenchmarking goes hand-in-hand with microoptimization. And microoptimization is rarely a good idea.

That's not to say that microbenchmarks are bad. They can be really useful when applied correctly. I remember the competitions at Apple to create the fastest BlockMove for PowerPC, which were often repeated with each new generation of processor. Since it was one of the most heavily-used system calls in the Mac OS, each incremental improvement made the OS faster. Many individual benchmarks, such as drawing and text manipulation, got faster whenever BlockMove was improved.

The danger comes not from the microbenchmarking itself. The real problem occurs when you focus so much on the micro that you forget about the big picture. There are three main negative outcomes possible:

Dangers of Microbenchmarking and Microoptimization

Wasting development time. Newbie programmers sometimes burn weeks optimizing something that doesn't really need optimizing. This doesn't matter much when you're doing it as a hobby, but it matters a lot when you're in a real job.
Making something else slower. Sometimes performance can be like a carpet. Push it down over here, and it pops up over there. If the operation that you're microbenchmarking does not crop up often in real-world scenarios, you may find that all your micro-optimizing actually decreases your real-world performance.
Introducing bugs. Sloppy optimization can be worse than no optimization. Your code might work today, but the trick you used might not be future-proof.

Performance is Chunky

There are two important lessons that I've learned about software performance over the years:

Lesson 1: Performance is "chunky". Not all parts of the code contribute equally to the software's speed. A small piece of code might turn out to be a very big chunk of the performance, and a big piece of code might be almost immaterial. You need to know what your chunks look like before you can optimize intelligently.

Lesson 2: Design trumps everything. Choosing the right design or algorithm for your code is far more important than almost any microoptimization you can make. If your code calls strcmp() 4 million times in a loop, you're better off changing your design to call it less often. There's no reason to waste time looking for a faster strcmp().

From these two lessons we can pretty easily derive the types of optimizations that work best. Here they are, from most effective to least effective:

Ordered Rules for Optimization

Want to make your code faster? Follow these rules in order.

Design for efficiency. Understand the real task. Eliminate redundancy. Understand how your algorithm is performing, and the ways it might not be optimal. Redesign the whole thing until it is optimal.
Optimize areas where your code spends a lot of time. Sometimes there is an inner loop which is repeated millions of times. The first answer is always rule #1: "don't do that" — try to redesign your code to reduce the number of times you perform that operation. But if it truly can't be avoided, then go ahead and expend some effort on speeding it up.
Microoptimization. Pick a single operation and make it faster in isolation. This is almost always the wrong thing to do.

Why are microbenchmarking and microoptimization almost always wrong? Because you should be taking care of #1 or #2 first. And the fundamental essence of programming is that the first two are never complete.

When should you microbenchmark?

If you are writing part of an operating system, a virtual machine, or the runtime for a programming language. Compiler writers may also microbenchmark the output of their compiler.

Very occasionally, it might be useful to do it as part of the first two optimization tasks. Perhaps you're choosing between different algorithms, one which involves calling functionA() a thousand times, and one which involves calling functionB() a thousand times. Rather than writing two complete implementations and comparing them, it might be instructive to do a quick microbenchmark to see which of the two is faster in isolation.

You can also do it for your own amusement and education, or for a blog entry. While not terribly useful, it can be informative and teach you a thing or two about your OS.

How to write a microbenchmark

With all that said, let's get down to the details. Suppose that you eventually find yourself in a position where you really do need to run a microbenchmark.

I use a ten-step process:

Consider all the factors that may impact performance. There may be non-obvious considerations like memory alignment, cache characteristics, or data set size. Decide exactly how you want to test your operation with those factors in mind.
Write a loop to perform the operation a large number of times. Any single measurement might be way off, because your computer is busy doing all sorts of things like processing interrupts in the background that are out of your control. So you need to average over a longish interval. Try starting with about a hundred thousand iterations, then add multiples of ten until you wind up with a total runtime of a second or two.
Use a high-resolution timer to record the start and stop time. Your data is only as good as your timer. Find the best one available. On Mac OS X that's mach_absolute_time(), and on Windows it's QueryPerformanceCounter(). Other platforms usually have an equivalent high-resolution timer. Call it only twice: once before the big loop, and once after.
Write code to compute the result. The approximate time it takes for a single operation will be elapsedTime / numberOfIterations. The elapsedTime is generally (stopTime - startTime) / timerFrequency. Watch out for integer overflow and floating-point precision loss: it's easiest to just use double-precision math.
Consider whether you should compile your code at the highest optimization level available. Compilers generate significantly slower code when optimization is off. You're almost certainly interested in the performance of the fully optimized build rather than the unoptimized debug build, but optimization can introduce its own difficulties. It depends on what exactly you're benchmarking.
Compile, then disassemble your test to make sure it's really doing what you think it's doing. This step is non-negotiable!! If you can't read assembly, you shouldn't be doing microbenchmarking. Compilers can be tricky and there's no guarantee that it's actually doing what you think it's doing until you look. If you are testing strcmp(), for example, you might find that the compiler has decided to optimize out all ten million calls in your inner loop because you're using a string constant instead of a variable. Or perhaps your ten million iterations of input = input + 1 are being optimized out to input = input + 10000000. If this is happening to you, you'll need to figure out a way to force the compiler to do what you want. With gcc, for example, you might need to link functions in via separate modules to prevent inlining, or specify additional flags on the command line.
Quiesce your system. You don't have to shut everything down, but do make sure that you're not running under abnormally heavy load. Most systems these days have a CPU meter which can tell you what's going on. If you're writing code on an embedded system, make sure that any asynchronous startup operations (like initializing the network stack) have finished before you begin.
Run the test and get the results. If it runs too quickly or too slowly, you might need to adjust your iteration count and try again. If you get what looks like a good result, there's one more step:
Run the test several more times and make sure your result is reproducible. This will also give you an idea of the typical error in your measurement.
If you support different platforms or processors, run the test on each one. Sometimes constructs which work well with one processor are the worst-case for another. Occasionally operating systems will change the implementation of their standard library functions enough to perturb performance. Don't assume that just because it's faster on one configuration that it will always be faster everywhere.

Code Sample

Here's a piece of sample code I wrote to microbenchmark the speed of a function call on Mac OS X. I've removed the function body in order to simply show the overall technique.

/* foo.cpp */
#include <stdio.h>
#include <mach/mach_time.h>

int testfunc(int arg1, int arg2);

int main(int, const char **)
{
    struct mach_timebase_info  tb;
    uint64_t   start, stop, elapsed;
    double     seconds;
    const int  iterations = 300000000;
    int result = 0;

    mach_timebase_info(&tb);

    // Run the test.
    start = mach_absolute_time();
    for (int i=0; i<iterations; ++i)
        result = testfunc(result,i);
    stop = mach_absolute_time();
    elapsed = stop - start;

    // Print results.
    seconds = ((double)elapsed * tb.denom / tb.numer) / (1000*1000*1000);
    printf("%d iterations = %0.9lf seconds\n", iterations, seconds);
    printf("1 iteration = %0.9lf seconds\n", seconds/iterations);
    return 0;
}

/* bar.cpp */
int testfunc(int, int)
{
    return 0;
}

Notice how the function that it's calling is in a completely different source file? That's because on my first attempt gcc's inlining optimizer actually completely eliminated the function call. This was only visible upon inspecting the disassembly (step 6). Putting the target function in another module was the easiest way to stop the function from being inlined. (Yes, I tried -fno-inline and __attribute__((noinline)). Neither has the desired effect.)

I've chosen to ignore the overhead of the loop itself in this measurement. It's essentially negligible for most types of microbenchmarks. Furthermore, as long as you wind up comparing the result against another test run with identical loop overhead, it will simply cancel from both sides of the equation.

So that's it. Anyone have any interesting situations where you've found that you needed to run a microbenchmark? Does your technique differ from mine? Found any unexpected results? Let me know in the comments.

What should TV shows cost?

2007-08-31T19:52:00.001-04:00

While reading Daring Fireball today, I saw that Apple has posted a press release scolding NBC for wanting to more than double the cost of their content.

Apple® today announced that it will not be selling NBC television shows for the upcoming television season on its online iTunes® Store (www.itunes.com). The move follows NBC's decision to not renew its agreement with iTunes after Apple declined to pay more than double the wholesale price for each NBC TV episode, which would have resulted in the retail price to consumers increasing to $4.99 per episode from the current $1.99. ABC, CBS, FOX and The CW, along with more than 50 cable networks, are signed up to sell TV shows from their upcoming season on iTunes at $1.99 per episode.

'We are disappointed to see NBC leave iTunes because we would not agree to their dramatic price increase,' said Eddy Cue, Apple's vice president of iTunes. 'We hope they will change their minds and offer their TV shows to the tens of millions of iTunes customers.'

First of all, I'm totally amused that Apple is dragging this dirty laundry out in public to shame NBC. It's a smart business move, and I have no objections to it: I just get a kick out of big corporations fighting.

Second, it brings up an excellent question about the value of content. Particularly content that is sometimes expensive to produce. What exactly do consumers perceive as the "right price" for something like a TV show?

What Content Producers Think

I think content producers operate on a couple of different mental models when it comes to pricing their content for digital distribution. Most of these models are terribly flawed, because they're not looking at the consumer's point of view.

Production Cost: "The more expensive it is to produce, the higher the price should be."

Length: "A one-hour TV show should cost around the same as a one-hour CD."

Cross-Media Comparisons: (To the most expensive alternative.) "We can sell a full season of DVDs for $60, which works out to $5 per show."

Threat of Withholding: "You're going to buy it at whatever price we ask, or else you won't get it at all."

These models probably only make sense to you if you're a content producer talking about your own content. As a consumer, they probably make you angry.

What Consumers Think

As a consumer, these are the things I think about when I am considering buying content.

Total Usage Time: "How much total time will I spend enjoying this content?"

Total usage time is very different from the length of the content. Total usage time is equal to the length, multiplied by the number of uses I expect to get from it.

For a CD or album, the total usage time is vastly different from that of a TV show. When I buy music, I expect to listen to it again and again. Most TV shows I only watch once. Sure, if it's a download I could watch it again and again, but odds are that the 45 minutes of content in your hour-long show is only going to get 45 minutes of my time. Or less. Movies and concert videos are somewhere in the middle: particularly good movies will sometimes get a second or third viewing, but still nothing approaching what a good CD gets.

Quality and Extras: "Is this particularly high-quality content? Does it have something extra that I'm interested in, like commentary or subtitles?"

Although this isn't my primary factor, I'm generally willing to pay a little more for content that goes the extra mile. Sometimes that means higher resolution (Blu-Ray vs DVD, or SACD or DVD-Audio vs CD vs MP3), sometimes that means particularly extraordinary special effects, sometimes it's commentary from a director or actor that I admire, sometimes it's a behind-the-scenes "Making Of" featurette.

For example, I almost never buy TV shows on DVD because they're just too expensive. But in the few rare cases where I do, it's because it's a show that I particularly enjoy and would like in a high-quality format with lots of extras.

Cross-Media Comparisons: (To the least expensive alternative.) "Can I get everything I want somewhere else, for less money?"

Sometimes I simply want to catch up on a show's plot developments. Reality shows are a good example: I sure don't watch them for the quality of the writing. If I'm traveling and I miss a showing of American Idol or Who Wants to Be A Superhero?, I'm not going to spend any significant amount of money at all on finding out who got kicked off this week. I'd rather just look it up online.

If it's a show like Battlestar Galactica that I actually do want to watch and appreciate, well, I own a DVR which lets me record the "free" showings from network or cable TV. Chances are unless I'm in a big hurry, I'll just watch the version that was automatically recorded for me once I get home.

This is also where piracy comes in. As Steve Jobs has said, content producers have to compete with piracy as a business model. Video-sharing sites and BitTorrent servers may offer the content at no cost. Piracy is a funny thing. It isn't attractive to most people ... until you start withholding or overcharging for content. Then it spikes. Personally I don't have any qualms about watching all or part of a TV show on YouTube if it isn't available anywhere else.

You can go into great depth parsing the legal arguments around both of those last two, but you'll be wasting your time. Consumers don't. The fact is that in our minds — yours and mine — TV show downloads are essentially competing with "free", twice over.

Convenience vs Importance: "Is the price I'm paying (in both money and inconvenience) proportionate to how much I care?"

Sometimes I'm only mildly interested in your content. I think most web browsing falls under this umbrella, which is why it's so difficult to make money by charging for content on the web. If I'm only mildly interested, I'm not going to pay you anything at all for it, but I might spend a few minutes watching it ... if it's free and easy to do so.

Of course, there are times when I'm extremely interested in your content. I'm generally more willing to pay money for it in that case. But the number of things that are quite that important to me necessarily has to be kept very small — because I don't have an infinite amount of money. (IMPORTANT NOTE: If you would like to give me an infinite amount of money, e-mail me and we'll talk.)

Resolving the difficulties

That's it. Those four criteria are what I use to evaluate prices as a consumer, and I would be willing to bet that I'm not alone. Realistically, I think the consumer models are what will ultimately drive the cost.

So what's the fair price for a TV show? If you go by total usage time, the value of a TV show is substantially less than the value of a single track of music, despite the fact that it costs more to produce. The convenience factor of getting it whenever you want it may bump up the price you're willing to pay, but I think that $2 is actually already too expensive in the minds of many consumers. They'd be better at $1 ... or even free.

What's the fair price of a TV show to you?

Markdown Scripts for MarsEdit

2007-08-26T12:57:00.001-04:00

I've created a few AppleScripts for MarsEdit to simplify the process of using Markdown to post to any weblog (like, say, Blogger) that doesn't natively support it.

Translate Markdown to HTML

This script takes text written in Markdown syntax and translates it into HTML. Extraneous line breaks are eliminated, because many blog hosts like to translate them into <br /> tags.

This works particularly well because MarsEdit has native support for previewing content written in Markdown. I normally use the Markdown preview mode while composing a post, then convert it to HTML just before posting.

But wait, you say: what if you need to edit a post?

No problem, says me. Then you use the other script.

Translate HTML to Markdown

This script uses Aaron Swartz's html2text to translate the post from HTML back into Markdown. It works on any HTML post, not just ones that you've created with the other script. Then you're free to edit the post, convert it back to HTML, and repost it.

Updated: Now with Unicode support

Update, Oct 4 2007: MarsEdit's author, Daniel Jalkut, alerted me to the fact that these scripts had problems with non-Roman text and offered a fix. Sure enough, I'd forgotten that AppleScript needs special care to do the right thing and handle text files as UTF-8. I've updated them appropriately.

Get them now!

Interested? Download the scripts now!

MarsEdit Markdown Scripts

BASIC for the iPhone

2007-08-19T14:54:00.001-04:00

This year C4[1] played host to Iron Coder Live. The API was "iPhone" and the theme was "conspiracy". The winner was well-deserved. But I had fun doing a hack of my own.

There was just one problem: I don't have an iPhone.

As I was driving from Cleveland to Chicago, I had a lot of time—about five hours—to think about what I might do for the contest. Without an actual iPhone it had to be something that could be largely developed offline and only run on the phone at the contest. That meant that I would need to stick to the, ahem, "official API" of the iPhone: HTML and JavaScript.

Although the phone is cool, I think it's even cooler that a whole community has built up to create the SDK that Apple failed to provide. After talking with some old friends at the conference I decided that it would be fun to join the conspiracy and create my own SDK for the iPhone, offering the ultimate retro-cool language: BASIC for the iPhone.

The version I showed at the conference garnered a very respectable 5th place, which I'm more than satisfied with considering its far more polished competition. I've since cleaned it up, GPLed it, and rechristened it. And here it is:

ippleSoft BASIC

The interface has been scaled for the iPhone, but it works just fine in Safari and Firefox.

ippleSoft BASIC for the iPhone - Run it now!
View the source
Command reference - also available by entering HELP at the prompt.

Notes

Naming: I tried to pick a name that wouldn't infringe on anyone's existing trademark. That's harder than you might think: "iPhone BASIC" would surely lead to a cease-and-desist letter from the hounds. iBASIC, IP-BASIC, etc were all taken. Then I ran into a stroke of luck: It turns out "AppleSoft" is no longer a registered trademark of Apple. They let it expire in 2001. Thus: ippleSoft.

Entirely JavaScript, entirely free. I've made it available under GPL v2. There's no public repository yet, but hey, it's GPL, so you can go and create one yourself. I've released code into the public domain before, but this is the first project I've ever released under GPL.

I'm not a JavaScript programmer. I learned about the language and tried to follow best practice, though. Well, as much as I could during a single beer-soaked weekend.

Autoscroll: The output display will autoscroll in Mac Safari, but not iPhone Mobile Safari. You can flick the output display to scroll it, though. If anyone with an iPhone has a suggestion on how to make it autoscroll properly please let me know!

Language: For now I've kept it pretty close to a strict subset of AppleSoft BASIC. There's a lot it doesn't do yet: graphics and text positioning are not supported, and neither are INPUT or GET. Those are pretty high on the list of priorities though.

One final thought

You know what? Text-based interfaces suck on the iPhone. There's just no getting around this. It's not the output so much as the input. The iPhone feels great when you're pushing buttons and flicking the display to scroll. But the keyboard is really lame.

I'm not sure if that's a feature or a bug for ippleSoft BASIC. Personally I find it kind of perversely amusing to use the iPhone to run a language originally designed for teletypes. But it would be interesting to explore ways to take the text out of BASIC.

Loading and running programs from elsewhere on the web would be a natural. A keyword-board like the one on the Timex-Sinclair TS1000 might be useful if you really must write code on the iPhone. INPUT and GET may need to be supplemented with some type of dialog interface. Anything else?

What do you think?

Crazy Del.icio.us

2007-03-23T19:26:00.001-04:00

Do you use del.icio.us? If not, you might want to. I've been using it for a while now, and the more I use it the more I like it.

What is del.icio.us?

It's a social bookmarking site. Here's a description in their own words.

Why don't you just tell me what social bookmarking is?

Okay, fine. Really, it's exactly what it sounds like.

Just like your web browser's "Bookmarks" menu, you post links that you want to remember to your account. Unlike your web browser's "Bookmarks" menu, the links are shared and visible to anyone who wants to find them. You can even see what your friends are bookmarking, and turn around and bookmark the same sites yourself if you find them interesting.

When you add a bookmark, you can provide a brief comment or excerpt to help remind you what it is. You can also assign tags to help categorize links. Since it's public, your list of bookmarks is accessible from any computer you might be at -- not just your home computer.

For example, here's my del.icio.us, containing a bunch of random links that I thought were worth remembering for one reason or another:

http://del.icio.us/drewthaler

Notice the nice, easy to remember URL. That's very typical of del.icio.us. Also, take a look at the links on the right side of that page -- I tag everything, and those links make it easy to find all links with a certain tag. Maybe you're only interested in my Apple links, or my PS3 links:

You can also combine tags. Here are my links that are tagged with both programming and humor:

http://del.icio.us/drewthaler/programming+humor

Easy, huh? And the best part (for me) is that if I want to remember a link I saved six months ago, I can do the same thing. I just hop over to my page and zoom down through the categories until I find it. Heck, I don't even have to be on my own computer!

Isn't there some other site that does this?

Yes and no. According to Wikipedia's entry on social bookmarking there have been several similar services over the years, going back to 1996 or so. But most of them died in the dot-com boom and bust. del.icio.us was founded in 2003, and is probably one of the earliest "Web 2.0" sites. It was recently acquired by Yahoo!, so it's not going anywhere.

You might be thinking of Digg or Reddit, which are social news sites. But that isn't quite the same thing. In fact, I find that social news starts out cool but quickly gets annoying. Sites are inevitably biased by the demographics of the top users, and the comment threads are worse than useless. del.icio.us has its own version of social news, in the hotlist and popular links accessible from the main page. But you can also ignore them entirely.

You might also be thinking of Twitter, but that's more of a lifeblogging site. You could abuse Twitter to create a sort of social bookmarking system, like someone I know did, but that's not really the site's intention.

It also sounds a little like Delicious Library, but there's absolutely no connection. :-)

What else makes it cool?

You can subscribe to the links of other people -- both via the web and via RSS. So if you find interesting people who have accounts, you can see what they're paying attention to. You can view my network if you want to.

People whose links I follow include:

Mark Pilgrim, known for his "Dive Into" series on Greasemonkey, accessibility, and more. Mark also aggregates his links on diveintomark, but I follow them on del.icio.us.
Jon Rentzsch, of Red Shed Software and C4.
Dave Shea, web designer and creator of CSS Zen Garden.
Daniel Jalkut, another former Apple employee and owner of Red Sweater Software.
Rosyna from Unsanity, an old friend from #macdev on IRC.
John Gruber of Daring Fireball used to use his account, but stopped using it once he realized he could make people pay him for the RSS feed instead. :-)

What do you like most about it?

Three things.

First, it provides a nice sort of link-blog functionality without having to spam everyone with a ton of small posts. This blog, as I've said before, is not a link blog. It's more of an essay blog. If you want random links from me these days, however, you can get them -- just go to my del.icio.us.

Second, it gives me a way to offload some of my memory to a computer. It's not as extreme as some of the current experiments in lifeblogging, but it's a good compromise for me. And since my bookmarks are on a public server run by Yahoo! I don't really need to worry too much about backing them up or continually copying them from one machine to another. Someone else handles that.

Third, I like checking out the bookmarks of other people. It's pretty nifty to fire up my RSS reader and see that Rosyna has posted some crazy Japanese Spider-Man video, or that Mark Pilgrim saved an interesting article about XML , or that Rentzsch just saved a nifty Mac programming link.

Are you using del.icio.us? If so, post a comment or drop me an email letting me know your username.

Oh, baby baby

2007-02-05T17:55:00.000-05:00

Please welcome my son Archer Douglas Thaler, born on Jan 30th 2007 at 5:10am. He was 7lbs 11oz, 20 inches long, and came out pink and screaming -- just the way they're supposed to.

There are a few more pictures available. Today he's exactly one week old. He's been a great kid so far, not fussy at all and often sleeping at night for a good 4-5 hours at a stretch. His name is a tribute to his great-grandfather Orion, as well as just a cool name that we liked.

I'll be a little busy for a while between work and family, but I hope to get back into posting soon. Lots of good technical stuff to talk about.

Better Food, Just as Fast

2007-01-06T13:46:00.000-05:00

Do you find yourself going out to dinner a lot, or bringing home takeout food more often than you should?

We sure do. Even though we've been making an effort to eat lighter meals at healthier places -- our faves from around here include Panera, Chipotle, Zoup!, and our local Heinen's grocery store -- it's just not the same as cooking and eating at home. There's a real difference in quality, quantity, and healthiness. And it's expensive, too!

Worse yet, it's a slippery slope. Once you get into the habit of getting takeout for dinner, it's easy to slip and start eating less healthy fast food too. (cough KFC's beautiful-but-deadly mashed potato bowls, I'm looking at you.)

My wife Nancy recently came up with a solution. She'd heard about this place nearby called Simply Done Dinners. Apparently -- and I had no idea this even existed -- places like this are a growing trend around the country. It's basically a place where you can come in, put together homemade dinners, bring them home, and toss them in the freezer to cook later.

Simply Done Dinners is local to the Cleveland area, and they have only four stores right now: Parma, North Olmsted, Twinsburg, and Medina. If you're not from around here, don't worry. There's probably a similar store near you. Take a look at this HUGE directory of easy meal-prep stores. See? I told you it was a growing trend. :-)

Our First Taste

The first time we tried it out, Nancy simply ordered six dinners online ($109) and had the store in Twinsburg assemble them ($29) so that all she had to do was pick it up. There was no preparation involved: she simply walked in and grabbed the prepared dinner packages. We got one of each of the following:

Apple Pork Chops with Sweet Potatoes
Cheeseburger Noodle Casserole
Cod Provencal
Lemon Garlic Chicken
Pasta Broccoli Bake
Southwestern Meatloaf

Each dinner package is designed for 4-6 people. For our family of three -- papa bear, mama bear, and baby bear, with proportional appetites -- we found that we got just under two meals per person out of each. Either two dinners from each, or dinner plus a tasty lunch of leftovers the next day. The meatloaf actually lasted much longer: I used it for lunch sandwiches for almost a week, but that was balanced out by the pork chops which got left out for too long and had to be thrown away. All in all, I figure that we got roughly 30 meals from the order.

If you do the math, that works out to $4.60 per meal per person: a little more expensive than buying all the ingredients yourself, but without any of the hassle. It's also about what you'd pay for a fast food "value meal", and maybe 50 cents to a dollar more than a frozen TV dinner at the grocery store. Really, a surprisingly good value for the kind of quality that you get.

Of course, these are just main courses and don't come with sides. (Some other chains do provide sides, but they also charge a little more.) So we'd add rolls, or a salad, or some veggies as needed. That adds a bit to the cost-per-meal, strictly speaking, but we already had all that stuff around the house anyway so we didn't factor it in.

The dinners come in a big disposable pan, the kind that you can just toss in the oven. There's a sticker with cooking directions on the side. Some needed baking, some needed broiling, and other stuff could be done in a crock-pot. (We avoided the crock-pot ones the first time, since we didn't have one.) In all cases it's pretty much drop-it-in-and-wait; no skills required. A bachelor could do it with one hand tied behind his back. :-) They're all designed to be stored in the freezer, but you have to remember to pull them out into the fridge to thaw 48 hours before you want them. We pulled out one or two a week, and eventually started a rotation where every time we used one we'd move another into the fridge so that it would be ready in a couple of days.

What about the taste? Well, we have a diverse set of tastebuds in our family: I'll eat anything, but Nancy can be a picky eater (to put it mildly). And even though Olivia's pretty darn adventurous for a 10-year-old kid, eating sushi and cow tongue among other things, she doesn't like everything. Sometimes she opts out of dinner and just gets a veggie burger instead.

But even in our divided household everyone unanimously agreed: the food was delicious! I guess that's the benefit of having a professional chef selecting the recipes.

Better still, it was all stuff that we either wouldn't have made, or something that was a unique flavor that we probably wouldn't have thought of. So it didn't actually conflict with any regular cooking -- we still made our simple family-friendly foods like nachos, pizza, salads, burgers, soup, sandwiches, pasta, etc. It was more like a supplement to our regular meals which pushed the fast food out of our diet for a while.

Cooking Too: Electric Boogaloo

Today we received our second order. Nancy is now nine months pregnant (!) and we're anticipating that eating will be a little bit hectic for a while once the baby comes, so we wanted to stock up on easy and nutritious food. This time instead of ordering 6, we ordered 12. And we figured we'd skip the "pick it up" option and try preparing it ourselves. Here's what we chose:

Balsamic Thyme Chicken
2 x Apricot Chicken (Heart Healthy)
Crabmeat and Swiss Cheese Bake
2 x Creamy Potato Soup
2 x Down Under Chicken
Rice Vegetable Cheese Casserole
2 x Tropical Pork Chops
Onion Bacon Crusted Cod

Preparation was really a lot of fun. Nancy and I went together, and we both agreed: it was all the fun parts of cooking without any of the hard stuff! No chopping, no searching for bowls, no scavenging ingredients or trying to work out what you can substitute because you forgot to buy something, and of course the best part -- NO DISHES!!

Working together we finished making our twelve dinners in just over an hour -- that's about five minutes each. Here's how it works:

Each store has a fixed menu which changes every month. There are different recipe stations around the room, one station per recipe. Each station has containers of the ingredients you need at waist level, with the appropriate measuring cup for the recipe already in it. Cheese, mayo, chopped onions, chili sauce, apricot jelly, cooked bacon, etc. (If you want a visual, think of the scene behind the counter at a Subway restaurant, but one that is customized for just a single sandwich.) Some ingredients are pre-packaged in containers that are just the right size for the recipe: for example, one casserole had a bag of exactly the right number of bread squares ready to go. Herbs and spices, sugar, oil, seasoning salts, etc for the recipe are all at shoulder height, and each one has the appropriate measuring spoon next to it. The meat is in a big refrigerator on one wall: thawed, pre-cut, and also ready to go. Along the side there's a sink for washing your hands, and a big rack full of all the utensils you could want and all the pans you need. Oh, and there's yet another station with ziploc bags and a big roll of plastic wrap.

To make your order, you simply find an open station (there was only one other couple there, so this wasn't hard) and start working. Since everything is already prepped for you, you basically wind up dumping stuff together. It's not hard at all. You're free to tweak the proportions if you want; we only did this a little bit since the recipes seem to have a good mix to begin with.

I'll admit that our very first one (the Down Under Chicken) was a little intimidating simpy because it was a new experience. We took it pretty slowly to make sure we were doing everything right. But once we figured out how it worked, we got into a groove and it couldn't have been easier -- we banged out the rest without a hitch.

When you're making it yourself you can also split the orders, so that instead of one meal that serves 4-6 you can make two meals of 2-3 each. We did that on many of the orders, in an attempt to reduce the amount of leftovers. We'll see how that works out.

At one point Nancy commented, "Geez, I feel like we're on a TV show!" You know how in cooking shows they just describe what you'd do, and then whip it out already done? And dirty dishes aren't even discussed? It was exactly like that. Almost a magical experience.

Net cost: $197 for twelve dinners, plus about two hours of our time on a Saturday morning. If the previous ratio holds and we get 60 meals out of it, that will come out to $3.28 per meal.

I'd love to tell you how the food tastes, but since we just made it this morning I can't say yet. But boy, did it ever did look good when we made it!

By the way, in case you were wondering: I'm not affiliated with these folks in any way, nor do I know the guy who runs the store. (Although we chatted afterward and he seemed pretty cool.) I just think it's a great service. And it's the kind of place that's just awesome for families, and which I even kind of wish I'd had access to as a bachelor -- because sometimes you just want a nice dinner without a lot of effort.

The verdict? Two thumbs way up. The dinners are cheap, fun to make, delicious, and easy ... and we're back to eating at a table, instead of from a paper bag.

Japanese PS3 commercials

2006-11-07T01:09:00.000-05:00

Everyone knows Sony's marketing can be, uh, interesting. Most people have seen the US commercials for the PlayStation 3, but not so many people have seen the Japanese commercials.

Since the Japanese launch of the PS3 is scheduled for November 11th (aka 11/11), most of them involve people seeing 11/11 in odd places. They're short and interesting and there aren't any creepy babies.

Here are some YouTube links for your viewing enjoyment.

Concent (The Socket)

Foku (The Fork)

Entotsu (The Smokestack)

Taikan (The Sensation)

Blogger: Condensed Label List

2006-11-03T16:09:00.000-05:00

This is a continuation of the series about customizations I made to my Blogger template after switching to the Blogger beta.

This time I'm going to talk about the condensed list of labels that I've put in the sidebar. Synonyms for "labels" used by other blogging tools include "categories" and "sections". For user navigation I like "sections", personally, so that's what the sidebar says.

Why customize it?

The default list of labels is a standard bulleted list that displays vertically.

The problem with vertical lists is that, well, they're an inefficient use of vertical space in the sidebar. If you have a lot of labels, you'll wind up with an extremely large portion of your sidebar devoted to this list. The vertical space that the list occupies will be far out of proportion to its importance.

Frankly, the list of labels isn't that important to me. It's a nice touch, and I would like them to be there, but I wouldn't say that it needs an entire screenful in my sidebar. So I set out to fix that.

Widget modifications

Here are the modifications I made to the default "Labels" widget to accomplish this.

<b:widget id='Label1' locked='false' title='Sections' type='Label'>
<b:includable id='main'>
<b:if cond='data:title'>
<h2><data:title/></h2>
</b:if>
<div class='widget-content'>
<ul style='padding-left:10px; margin:0; text-indent:0;'>
<b:loop values='data:labels' var='label'>
<li style='display: inline; list-style-type: none; padding-right: 5px; line-height: 140%;'>
<b:if cond='data:blog.url == data:label.url'>
<data:label.name/> (<data:label.count/>)
<b:else/>
<a expr:href='data:label.url'><data:label.name/></a> (<data:label.count/>)
</b:if>
</li>
</b:loop>
</ul>
<b:include name='quickedit'/>
</div>
</b:includable>
</b:widget>

Widget explanations

Since the styles here are very highly specific to this single list, and unlikely to be re-used, I decided to mix CSS with HTML here by assigning the styles via the style attribute on the tags. This isn't normally good practice, but Blogger templates do it a lot already, and it's actually kind of convenient here.

The display: inline; style on the LI tag makes the list horizontal. Adding list-style-type: none; removes the bullets.

That's almost good enough right there! But during testing I found that the original template used normal spaces in between the label and the numeric post count. Since I am expecting the list to wrap -- it wraps five times at my normal web browser size -- I need it to wrap cleanly. With a normal space, you often get the break in between the label and the post count, which doesn't look right. So I converted the regular space into a non-breaking space ( ).

Why ' ', rather than something like ' '? Because I'm still shooting for XHTML eventually, and XHTML does not recognize ' ' as an entity. The Unicode code point ' ' is the direct equivalent.

After all that was done, padding, margins, and line height were adjusted to my personal preference. Your needs may vary.

Dead ends

Although this looks simple, I tried a couple of other things before settling on this solution.

Putting the list in a paragraph - I considered this as an option, but was unable to get Blogger's template language to do it. You could do it as a space-separated list, but as soon as you add separators of any sort -- like commas -- there's no apparent way to avoid putting a separator in at the end. Blogger supports an "isLast" property on labels when used in the post widget, but it does not do so in the labels widget. On further consideration I decided that this was probably a bad approach, anyway, since it's not as accessible as using real lists.
Changing font size based on post count - This is a relatively new twist in Web design. Flickr's tags are a popular example. However, I couldn't figure out a way to do it without resorting to Javascript. Blogger has a very minimal expression language for templates, but I haven't been able to find any detailed documentation for it. The closest I came was doing something like this:
<font expr:size='data:label.count'> ... </font>
But that didn't cap the maximum font size -- potentially leading to problems as I write more posts in each category. For now I've tabled this idea. It would still be interesting, but I think the best way to do it in Blogger would be to leave the base template HTML as-is, then dynamically modify the HTML with Javascript on the client side.
Adding some interesting CSS decoration - I considered doing this too. But the only decent decoration I could think of was a rounded button look similar to what Safari uses in its Bookmarks toolbar. That would have been possible, but I'd need to create and edit some images and use this method to get the rounded corners. Worse, it wouldn't display correctly in Internet Explorer so I'd need to do a CSS hack to hide it. So I tabled this idea too. Even so, I might look back into this in the future.

More template tips

Other entries in this series:

Blogger: Image as Blog Header

Blogger: Image as Blog Header

2006-11-02T23:30:00.000-05:00

I'm going to do a brief series about how I wound up customizing my Blogger template (styles and widgets) after switching to the Blogger beta. First up, how I replaced the header with an image.

Why customize it?

Like a lot of people, I prefer to use an image instead of text for the header of my blog. I also want it to be clickable and always be a link back to the "home" page of the blog, which is a widely-adopted convention.

But I do not want to simply encode an inline image. For reasons of accessibility, I prefer a normal H1 tag that gets translated to an image by CSS.

Since I've been trying to keep the look-and-feel of my old template, I started with this image from my old template. It's 790 pixels wide by 105 pixels high.

Widget modifications

Here are the modifications I made to the default "Header" widget to accomplish this. Data that is specific to my blog, and which you'll want to change if you copy this code, is displayed in red.

<b:widget id='Header1' locked='true' title='Header' type='Header'>
<b:includable id='main'>
<div class='titlewrapper'>
<h1 class='title'>
<a href='http://drewthaler.blogspot.com/'><span/><data:title/></a>
</h1>
</div>
<div class='descriptionwrapper'>
<p class='description'><span><data:description/></span></p>
</div>
</b:includable>
</b:widget>

Widget explanations

The original Blogger template had a conditional so that the header was only a link if we weren't on the main page already. I removed the conditional because I think it makes more sense to always make it a link.

The original template also used 'data:blog.homepageUrl' for the link, but this resolved to 'http://drewthaler.blogspot.com/index.html', which is redundant. I wanted the shorter and more future-proof URL 'http://drewthaler.blogspot.com/' so I changed the link manually.

I left the description code as-is, because I don't use it. You may want to do something different.

You'll notice a single <span/> tag in the middle. That's required for CSS image replacement.

Stylesheet modifications

Here are the CSS additions that go with it. These are additions to the original stylesheet.

/* Replace the header with a graphic on computer displays */
@media screen, projection, tv {
h1.title {
width: 750px;
height: 89px;
left: 0;
top: 0;
padding: 15px 0 0 40px;
margin: 0;
border-bottom: 1px solid black;
z-index: 0;
}
h1.title a span {
background: url(http://homepage.mac.com/drewthaler/images/blog-header.gif) no-repeat;
position: absolute;
top: 0;
left: 0;
display: block;
width: 790px;
height: 105px;
z-index: 1;
}
#sidebar {
z-index: 2;
}
}

Stylesheet explanations

The @media directive means that these styles should only be applied for certain types of media. For screen, projection, and tv renderers I prefer the nicer look. But they will not be applied for screen readers and printing.

I'm using the Gilder/Levin Method for CSS image replacement. In this method, the span does all the work: it gets resized and repositioned, and the image is displayed as a CSS background image for it. The image is clickable because it's on the span, and the span is inside the link tag.

The span is given a z-index of 1 so that it sits above the original H1. The H1 text is still there, hiding underneath at z-index 0. Then the sidebar is given a z-index of 2 so that it will float above the header image if the two overlap.

The pixel sizes (790px by 105px) are from the size of my image. The padding (15px 0 0 40px) and bottom border are in there to make CSS-on/images-off browsing still look acceptable. It's a personal preference and based on the size of my image and text, so you might need to tweak it for your setup. The H1's width of 750px is the image's width (790px) minus the horizontal padding (40px), and the H1's height of 89px is the image's height (105px) minus the vertical padding (15px) minus the one-pixel bottom border (1px).

When Benchmarks Attack

2006-10-31T06:04:00.000-05:00

In my professional life I've found myself measured by an external benchmark many times. One time that comes to mind was when I was writing disk drivers for Mac OS 8.

The disk driver's job in Mac OS 8 was simply to pass requests from the filesystem layer to the underlying storage medium. Requests would come in for such-and-such amount of data at such-and-such an offset from the start of the volume. After some very minimal translation, we'd pass this request onward to the ATA Manager, or USB Manager, or FireWire libraries.

(Sounds easy, right? Well, yes, but the devil was in the details. USB and FireWire supported "hot-plugging", i.e. attaching or removing the drive while the computer was still running. Supporting this in an OS that was never designed for it was a big chore. CD and DVD drives were also much more difficult to handle than regular hard drives, for similar reasons.)

But we ran into a problem as we were preparing our drivers for release. The hardware manufacturers that were buying our drivers wanted to make sure our drivers were "fast". So they ran disk benchmarks against our drivers. Not an unreasonable thing to do, you might say, although as I've stated there really wasn't a lot of code in the critical path. The problem is that most of the disk benchmarks turned out to be inappropriate as measures of driver performance.

Measuring weight with a ruler

One common disk benchmark of the time was to read sequential 512-byte blocks from a large file. In Mac OS 8 these reads were passed directly down to the disk driver.

Remember, the disk driver's job was supposed to be to simply pass the incoming requests on to the next layer in the OS. The vast majority of the time would be spent accessing the hardware. So no matter what the requests were, in effect this test should have returned almost exactly the same results for all drivers, with code differences accounting for less than 1% of the results. Right? Wrong.

As we ran the tests, we found out that on this particular benchmark, our drivers were much slower than existing third-party drivers (our competition, more or less). Dramatically slower, in fact -- if we ran the test in 100 seconds, they ran it in 10 seconds.

Upon further examination, we found out the reason why they did so well on this benchmark. They had already been optimized for it!

We discovered that the other drivers contained caching logic which would cluster these small 512-byte sequential reads into larger 32KiB or so chunks. Doing so would decrease the number of round-trips to the hardware needed, which increased their performance on this benchmark.

Do what sells, not what's right

Now, it's important to understand that this tiny-sequential-read benchmark doesn't even reflect real-world use. The fact that larger I/O requests perform better than smaller I/O requests has been well known for decades, and almost every commercial application out there used large I/O requests for this very reason. Buffered I/O is built into the standard C libraries, for pete's sake. In the real world, requests that came into the disk driver tended to be either large or non-sequential.

Modifying our drivers to do the same thing would basically be pointless -- a lot of work for very little performance difference.

(It only gets worse when you realize that our disk driver was directly underneath the Mac OS disk cache. A driver-level cache was almost completely redundant. And the right place to do this sort of readahead caching would have been there, in the existing OS cache layer, not in each individual disk driver.)

But ... sigh. The real world doesn't always make sense. These small sequential reads were a well-known benchmark, even if it was a disk benchmark and not a driver benchmark. And that simple fact elevated this atypical scenario into the eyes of a lot of people who didn't really understand what it meant. To them, if our driver didn't do the same thing as the competition, we wouldn't measure up.

De-optimizing to improve the benchmark

So in order to make sales we were forced to burn another month adding and testing driver-level caching so that we'd perform better on this benchmark.

The irony? The work we did to improve this atypical benchmark actually increased the amount of time we spent in the driver code, and sometimes increased the amount of data we read, which in turn decreased real-world performance by as much as 1-2%.

But hey, at least that benchmark got faster.

And we sold our driver.

Switched to Blogger Beta

2006-10-30T02:00:00.000-05:00

Yesterday I moved this blog to the new Blogger Beta.

I went whole-hog and switched to a new template as well. I'm still tweaking it, and XHTML compliance is temporarily gone. But for the most part things should be back to normal. RSS readers may see that every post has been modified. This was a side effect of the switch, although in fact I added categories/labels to all posts once I saw that they were going to be touched anyway.

Why switch? The mainline Blogger service (1.0) has been having boatloads of problems over the past month or two. It hasn't been very well maintained and it seems like it's just barely holding together. All the Google blogs are using the beta now, and in fact there are some very nice features in the beta that I was tempted by.

Cool stuff that you get from the switch:

MUCH faster publishing. 1.0 generated a zillion static HTML files, the 2.0 beta uses a database.
Labels, aka categories.
Backlinks, aka trackbacks. Better than MT's, since these are automatic. (Presumably filled in by Google crawls?)
Comment feeds, both global and per-post.
Better template management. It's more powerful and exposes more, and it's smart enough to let you add and rearrange certain things on your page with no effort.
Widgets, aka server-side directives in your template.

Downsides of the switch:

RSS reset. Everything in my RSS feeds got reset when I switched. Not a huge problem.
Porting customizations. To get some of the nicer features you need to upgrade your template. If you had your blog moderately customized, and change your template like I did, then it can be kind of a chore to port your customizations over. For me it took about a day.
Limited Safari support. Posting from Safari works just fine. Template editing seems like it has to be done in Firefox.
It's a beta. And knowing Google, it will probably remain a beta for another three years or so. Dude. Can't you guys commit?

I haven't seen any other major downsides yet, but I'll let you know.

Video Game Tips for Parents

2006-10-29T03:34:00.001-05:00

I want to talk a little about a couple of lessons that I've learned from several years of experience with my daughter and video games.

If you haven't done so already, I'd also recommend reading these other tips from an earlier post, where I describe the things I went through trying to figure out which PlayStation 2 games would work for my 6 year old daughter.

Here's what I've learned:

Never, EVER buy a game without reading a review first. It's just a bad idea. That brings us to our second rule:
Never, EVER buy a game without reading a review first. Seriously. I mean it! If you're new to video games, you might assume that most games are all about the same and it doesn't matter what you buy. Not a chance. Would you say the same for books and movies? There are some steaming pieces of trash out there which have been slapped together with spackle and duct tape, and they still sell a few copies because they're put in the store with fancy box art.
This rule has several corollaries. Here's a list of very bad reasons to buy a game "cold" (i.e., without reading a review) that people can fall victim to:
Bad Reason #1: It uses the name of some great game that you remember. Many classic video games have been resurrected into horrible zombie caricatures of their former greatness. Frogger is one of the worst offenders.
Bad Reason #2: It's a sequel to a game you or your child liked. Often the sequel changes the gameplay. And sometimes the rights to a game are transferred to another studio, and when that happens the second studio usually does a ham-handed job with the sequel. (Movies are the same way. Consider Grease and Grease 2.)
Bad Reason #3: It's a TV or movie tie-in. TV and movie tie-ins are very often poorly made. Sometimes they just whip them together with no concern for gameplay or difficulty. There are some terrific games which are exceptions to this rule, of course -- many of Nickelodeon's games are pretty good -- but you won't know unless you read a review.
Bad Reason #4: The box art and description makes it sound interesting. Um, yeah. Every game is a "great adventure" and "exciting" and "fun" by the time the marketing people are done with it. The box has no relation whatsoever to the actual game.
Look for the Greatest Hits titles. This is one place where the box can actually tell you something about the game. Sony's "Greatest Hits" are titles that have sold particularly well -- they are the gaming equivalent of a record going platinum. The nice thing is that the games get a lower price when this happens. So even though these games cost around $20, they're often much more fun than newer games which cost $50. Of course, brisk sales don't necessarily mean the game is age-appropriate, so you should still check reviews.
Don't be afraid of older games. Quality is far more important. The age of the game really doesn't matter: a fun game will always be a fun game. It doesn't matter if the game is "new" or not; it will still be new to your child! Any parent who's watched their kid get hooked on The Cosby Show or Full House or some other unexpected sitcom will understand how the same logic applies directly to games. Younger kids aren't hooked on fancy graphics and don't care about what's latest and greatest; they just want a game that's fun. As an extra bonus, older games are cheaper -- often by a lot!
Buy games used, and trade in old games. Most dedicated game stores, such as EB Games, offer used games for sale and offer you store credit on trade-ins. This is a great way to save a lot of money. The used games will be literally as good as new. Trading in is also a useful lesson for your kids; I've made it a point to help my daughter pick which games she wants to trade in. We keep some old favorites around because she still plays them, or because they are good for sleepovers, but most eventually go back. Then we shop for new games, trying to see how much we can get without spending more than $10.
Consider renting games. It's up to you to decide whether this approach works for you. I don't do it with my daughter, because she simply doesn't play games enough to justify it, but some people do. It can be a fun way to try out a game without making a $40-$50 commitment. Just watch out for late fees! These days you can avoid them completely. Blockbuster now offers no-late-fee rentals (online and in-store) with their GamePass service, and there are a ton of NetFlix-like online rental services, including GameFly, GameLender, GPlay, and others.
Swap games with friends. Your child might not even consider this unless you suggest it. Loaning games to friends is a nice way to socialize and it saves money, too. And it gives the kids something else to talk about!
Play the games with your child. Especially with younger kids, but even as your child gets older this is important. It's a lot like reading a book together. It's a way to have fun and share an experience. When your child is particularly young, you might find that he gets stuck and gives up on a game quietly without really saying anything. If you help get past the tricky spots, he can go on having fun for longer. Or perhaps your kid will be like my daughter, and get (happily) scared when you reach the big, intimidating "boss" monster together, and she'll shove the controller over to you and want you to take over for a while. I usually play just long enough to show her how to do it, and then pass it back to her to try. Sometimes, if she asks, I'll get her past the hard part to the point where she wants to take over again. It's fun for both of you, and can be a great confidence builder.
For big games, consider a hint book / strategy guide. Don't do this all the time, of course. But there are some larger games (such as Kingdom Hearts) where I found it to be very useful. What's great about these guides for kids ages 5-7 is that it can encourage their reading skills. My daughter wasn't really motivated to read much when she was 6, but the game strategy guide really got her going! It sounds silly, but it worked.

Any tips that you want to share?

Great PS2 Games for Young Kids, Part Two

2006-10-28T13:00:00.001-04:00

Well, it's been (ahem) about a year and a half since I promised a "part two" to my earlier post on Great PS2 Games for Young Kids, Part One. And it's time for the next installment. Three more games this time:

Monster Rancher 3
Dance Dance Revolution
Kingdom Hearts

[This post used to contain a lengthy list of tips. To trim it to a more manageable size, I've moved it to a new entry: Video Game Tips for Parents.]

Okay. With that out of the way, let's get to the games! Three more good ones here. Although Olivia is now age 10, I'll talk about some games that she loved between the ages of about 7 and 9.

Monster Rancher 3

This is actually a series of games, ranging from Monster Rancher 1 to 4, plus the relatively new (and very different) Monster Rancher EVO. At the time we got it, the fourth game was out but I'd heard more favorable reviews of the third game. Following my own advice -- don't be afraid of old games! -- I went with number three. It might be hard to find the third game these days; I hear that the fourth one is also good, but I can't say for sure since we've only played this one.

You play as a monster trainer. The game starts by creating a monster, either from some pre-made guys or -- the better choice -- from a "saucer stone", which is a code name for "any old random CD or DVD you have lying around the house". (You'll have to swap discs manually, at which point you should pause and take a moment to teach your child about how to put CDs away properly when they're done with them.) Each disc will create a unique monster with different looks and abilities, and the same disc will create the same type of monster if used again. Your monster starts as a baby, and you feed it and train it to help it grow stronger. Over time it matures, grows old, and eventually dies, at which point you can start over again. There is a little bit of anime-style story, and there is a tournament where the monsters can optionally fight (without blood, and losers aren't really "killed", but there is cartoonish violence -- monsters chomp each other, whap each other with giant cartoon mallets, etc) to gain levels and prizes. And that's pretty much it.

What's fascinating about this game is the way that kids just love it. But it's the kind of game that can drive an adult crazy! Why? Because it's cute, and there's a lot of repetition. Kids that are 10 and under love repetition; just like they enjoy watching the same video over and over, or reading the same book over and over, they don't mind doing the same thing in a game over and over. There is a lot of repetition in the game as you train your monster. You'll see the same animations over and over (sigh, and over) again. And yet there's still some variety -- you can train in different ways, or use different monsters, or go to a completely new area to train in.

The controls are menu-based, but pretty simple to figure out. There is some text that your child should pay attention to. For example, the game will suggest that it's time to feed your monster, or to let it rest. Perhaps the trickiest part is saving the game and continuing after your monster dies (which is necessary to open up new areas). Olivia had a tendency to just start new games, since she got attached to her monster and didn't want to save over it after it had grown old and died. But doing that meant that she never got to train anywhere other than the first area.

The game can also help build the skills your child needs to keep a pet. Monster Rancher isn't quite a "virtual pet" simulator, like a Tamagotchi or Nintendogs, but it shares some of the same elements. If you treat your monster nicely, he'll be sweet and lovable and grow stronger. If you spoil him too much, he'll get fat and temperamental. If you punish him too much or don't feed him enough, he might get mean. And even though it can be very sad when a monster grows old and dies, that's part of life too. The game handles it well and you can quickly start over with a new baby monster -- which gets a boost from the "heart" left over by the previous one.

Kids also really love the "saucer stone" part. Since you don't know what any disc will give you, you'll have to try a bunch of them and see what you get. It's surprisingly entertaining, and kids love it because it's something they never thought might be possible. It's a really unique gameplay mechanism that I've never seen anywhere else. Just be careful: unattended, your kids might quickly make a mess of your CD collection! We gave Olivia permission try lots of CDs but always made sure that she was careful with them and put everything back as soon as she was done.

Some people might be concerned about the whole "raising animals to fight" part. That's your call. Personally, this wasn't a problem for us. First, we make sure our daughter is able to distinguish fantasy from reality (always a good thing) and that she knows how to behave in real life. Second, the monster fighting is very much treated as a sport in the game -- just like boxing, karate, or amateur wrestling might be. There are tournaments, prizes, and so on. So it's not as if you're teaching your kids to enjoy cockfights or the WWF. It's an extremely mild game.

Dance Dance Revolution

This is another game that appeals to people across all ages. You've probably seen the arcade version in a mall. It's the one that has people stepping on four arrows while dance music plays, often drawing a crowd when the better players are "on stage". There's no real gender bias to the game, but my impression is that with younger kids it's more popular with girls, while among teenagers it's more popular among boys. I don't know this for sure; that's just the impression I get. Somebody could probably write a PhD thesis dissecting the relationship of this game to gender roles. :-)

First of all, you absolutely need to get the special "mat" controller that lets you dance. Some versions come with it bundled, or you can buy it separately. They sell some very expensive ones, but the regular old cheap one works well. Once you have that the game is very simple, but challenging. You're given a wide variety of songs to choose from, each with its own dance moves. All you need to do is just step on the arrows at the right time.

Sounds easy, right?

You wish. The guys at the mall may make it look simple, but DDR is very physical and can be challenging both for you and your child. You'll want to start out with the difficulty dialed all the way down to the lowest setting. It's okay to screw up -- and you will -- as long as you keep trying. Stick with it! After a while you'll start getting the hang of it. When you're first starting out, it's not unusual to have to stop after a few songs because you just get too tired and sweaty. I've found that taking turns of either one or three songs works well.

What's great about this game is that it's a very physical thing, and it's something your child will want to practice. It's not very mentally challenging, but it can be tricky trying to figure out how to do all the moves you need to get through a song. It's sort of like puzzle-solving with your feet. It's not a game that you'll "beat" any time soon; you will be able to continue playing and improving for a long time. It's also an easy game to pick up and play; there's no story, and nothing gets in your way. You just start it up and go. And it's great at parties as long as you make sure the kids share.

There are a lot of versions of this game. Some come with the "mat" controller bundled, some don't. As mentioned before, you'll really need the controller. We chose to get the Dance Dance Revolution Extreme Bundle, which included it. (NOTE: The preceding link lists a price of $199.95, which is insane. It should be about $40, which is the price you'll find for the almost equivalent DDR Extreme 2 Bundle. I e-mailed the seller and he said, paraphrased, that the price is deliberately high because it's new and unopened and someone stupid might buy it at that price.)

The primary difference between the versions is really the song lists. They tend to be techno dance music, and more or less appropriate for everyone. You might hear any of these songs on the radio. Your standards may vary, of course, but we really didn't find anything that we were concerned about in the game.

Kingdom Hearts

Kingdom Hearts has a special kind of magic. It turns Disney movies into three-dimensional worlds that you can run through and play around in, and that's just fantastic. Since it contains a lot of characters and stories that your child will already know, it really is a lot of fun, and unlocking the game's worlds can be really rewarding. And if you look at the top of the box you'll see that it's now become a "Greatest Hits" title, which is why it's gotten so cheap.

This is a tricky one to put on the list, but it's just so good that I had to do it. Despite its Disney theme, this is definitely not an easy game for a young child. It's very long, complicated, can be very difficult, and has a few scary points. I would say that it's probably targeted a little more at the tween to teen market. Olivia and I started playing it when she was only 7, but we didn't finish for months and months! It definitely shouldn't be a child's first exposure to video games, and you should be prepared to help your child play it almost the whole way through. In fact, I would strongly suggest getting a hint book like the Kingdom Hearts Official Strategy Guide ... these can be a lot of fun, and can encourage your child's reading skills.

You play Sora, a young boy who lives on an island with his friends. After some lengthy training and a series of opening sequences which seem to take forever, eventually your island is ripped apart. You land in a central hub called Traverse Town, and eventually meet up with Donald and Goofy -- who have been sent by King Mickey to find you and then deal with the problem of "Heartless", shadowy monsters which are invading all the Disney worlds. Eventually you'll visit the worlds of (and fight the major villains from) a lot of different Disney movies: Little Mermaid, Hercules, Nightmare Before Christmas, Alice in Wonderland, Aladdin, Peter Pan, and more.

There is violence in this game: fighting monsters is most of what you do. Fighting involves swordfights with a "Keyblade" that looks like a giant key. You're rarely fighting people; most of the fighting is against Heartless, who look like shadowy insects. Otherwise the content should be fine for all ages. Not all of the characters are Disney characters; some are Square-Enix characters (from other games made by the same game studio) and some were created just for this game.

The controls are moderately complicated. The basic moves are just running around, jumping, and hitting the attack button -- easily mastered by most kids if they've played a few games before. But there are a lot of menus, and even items that you can optionally buy and use (healing potions, magic, and so on). Saving goes through a menu as well, and must be done at a save point. There's no autosave, so your child must remember to save or else lose her progress.

The worlds you run around in can be big and complicated, and it's not always clear what to do next. But I've found that because it's a Disney theme, kids are very highly motivated to explore and look around. They will probably get stuck a lot, and the game makes it worse by creating certain situations where you just can't make forward progress and you have to give up and go to another world. For example, the Colosseum in Hercules isn't open for a long time, and there's no obvious reason why. That's why I'd suggest a hint book, with maps and strategies. When your child gets stuck, help him figure out how and where to look up the answer. You'll be helping him with his game, and very sneakily teaching him valuable skills like "how to use a reference book"!

There is also a sequel, Kingdom Hearts II, which is very similar to the first game. It's also long and complicated, but almost as good as the first game. I found that Olivia's enthusiasm ran out after a while on the sequel, but that might be because it didn't come out until two years after we finished the first one.

Wrap-Up

I still believe that the most important thing you can do as a parent is to read reviews of games before you buy them. Games can be expensive, and you don't want to buy bad ones.

There are a few places I go to find reviews. Amazon frequently has reviews written by kids and parents. I like GameSpot's reviews, although as a parent you sometimes need to translate a little: the criticisms of "too short" and "too easy" often mean the game will be just right for kids. IGN also has reviews, but the reviews aren't as well-written; sometimes it feels like they are written for gaming insiders who follow every little bit of news.

I hope to write more in this series, but I encourage you to write your own experiences in the comments. Let me know what you've tried, and what worked and didn't!

New Parallels Desktop Beta

2006-08-08T13:30:00.001-04:00

A new beta update of Parallels Desktop has been released that updates Parallels Desktop for Mac to Build 1862 Beta.

This update fixes the disk caching policy problem that PD Tweaker was created for. If you were wondering, PD Tweaker is perfectly safe to run with the new update and doesn't conflict with it at all. But it will be unnecessary once you have configured your VM properly.

Curiously, instead of just fixing it they decided to make it an option under the virtual machine settings -> Options -> VM flags:

Choose virtual hard disk cache policy for better performance of:

[X] Virtual machine
[ ] Mac OS X

"Virtual machine" is the default setting. For the technically-minded among you, this is like having a radio button to select between two behaviors: "incorrect" and "correct", and defaulting to "incorrect".

In this case the correct behavior is to select the second option, "Mac OS X". There are rare scenarios where you might want the other behavior, for example if you were setting up a dedicated server box that does nothing except run virtual machines. But in almost all other cases the second option is correct. I'll blog more about this later.

There are several significant other improvements, including video acceleration -- which completely eliminates the display-update lag when you're typing in Windows, hooray! All in all it seems like a very worthwhile update so far. Remember to re-install the Parallels Tools on your client OS after updating so that you get the new drivers.

CLImax 1.5d2 re-released

2006-06-30T19:58:00.001-04:00

Any old-school scripters out there might be happy to hear that I've re-released CLImax 1.5d2, which I recently rediscovered on a long-lost backup.

CLImax is an AppleScript command-line interface for the Mac OS. It was originally written in 1996 for System 7.5, and development was stopped in 1998. Ten years after its first release, version 1.5d2 can still run under Classic today. It's not as convenient as it was under 9.2.2 natively, but it's not bad either.

I've received a bunch of positive comments about CLImax from people including Bill Cheeseman (of AppleScript Sourcebook), Ray Barber (of MacScripter), and Peter Hosey (of Adium X). Peter was the one who actually got me to re-release it.

CLImax 1.5d2 is now FREEWARE and available at no charge. Be sure to visit the CLImax homepage for more information, including screenshots.

I'm currently soliciting feedback on whether it'd be worth porting it to Mac OS X and creating a modern version. Thoughts?

PD Tweaker 1.0

2006-06-22T10:44:00.001-04:00

For all you fans of Parallels Desktop out there, I've released a quick little hack to fix some problems in their initial release (aka: build 1848). It's called PD Tweaker.

It's very simple and only does two things.

Optimizes caching for HDD and SAV files

Caching large files is actually harmful to your Mac's overall performance. HDD files do not need caching because the client OS will already have a cache layer. SAV files are streamed in and out and don't need to reside in the disk cache.
Always writes HDD and SAV files all the way to disk

Your data is precious. Especially data like a HDD file that took you hours to install and configure. Shouldn't you treat it that way?

I wrote it primarily because although Parallels Desktop is a great product, I got fed up with the way it wasted my entire 2GB of RAM with the HDD file and made my machine virtually unusable. With PD Tweaker installed, you will not only notice greatly improved performance from your Mac as a whole, but your HDD files will be safer from data corruption as well. And best of all, it's entirely free and the source code is available.

It uses Unsanity's Application Enhancer 2.0 to do its thing.

More information (including some rationale and a technical explanation) is available on the PD Tweaker website. What are you waiting for? Go there now!

Delaying the xnu-x86 source release

2006-05-18T04:34:00.001-04:00

[Updated Aug 7 2006; see below.]

Tom Yager at InfoWorld points out that Apple has released the source code to everything in the x86 build of OSX Darwin except for the kernel, xnu.

He also makes what appears to be a completely and utterly unsubstantiated statement about why:

Thanks to pirates, or rather the fear of them, the Intel edition of Apple’s OS X is now a proprietary operating system.

First of all, what? That's a bold statement. Got any source for that claim? That seems like sheer FUD, deliberately sensationalized to create a stir and bring people to the site (and therefore sell ads). From everything I've seen, and I've worked with these people, Apple's security team and upper management know better than to rely on security through obscurity. And to date I don't know of any official or even unofficial statement from Apple about xnu-x86 -- just the fact that several people have noticed that the source still hasn't been released.

[Update 5/22 via John Gruber: Apple's Product Manager for Open Source reiterates that Apple has not made any announcement yet, and drops what sounds like a big hint that xnu-x86 will eventually be released.]

Let me offer two guesses at the real reason why we haven't seen source to xnu-x86 yet:

The xnu-x86 source might leak information about a future product. The obvious candidate is the only missing link in the x86 line, the pro desktops. You know, the ones that will replace the "PowerMac". Let's call them "Mac Pro" for lack of a better name.
For example, if the highest end Mac Pro desktop machines were planned to have, say, 4 CoreDuos packed into them for a whopping 8 CPU cores, then chances are you would see traces of that support show up in several parts of xnu. And when Apple releases a major rev of xnu, there are always some people who pore over it looking at the diffs.
Apple does have ways to keep prerelease stuff out of the source release, of course, but it adds a layer of complication and risk to go back and hack that up after the fact. Maybe this time they decided it was just simpler to drag their feet for a while until the entire new line has been announced.
The xnu-x86 source might currently contain a small amount of licensed proprietary code that does not belong to Apple. If that's the case, they simply might not be legally allowed to release it in its current form.
Maybe it's virtualization code from Intel, or some sort of Trusted Computing gobbledygook which is currently dormant. If they can't negotiate terms to release it, then they might have to factor the sourcebase somehow to link that other code in separately. Factoring that out seems like it would be totally possible, but kind of a messy task since it's at such a low level in the kernel and they can't sacrifice any performance to do so.

Personally I think #1 fits Apple's modus operandi perfectly. But #2 is also the kind of real-world consideration that could delay a source release for an unknown period of time while the lawyers work things out. I will grant that "fear of pirates" is another technically possible reason why we haven't seen the source, but then again the same could be said for "fear of ninjas".

We'll see what happens. Either way my gut feeling is that this is just a delay in the source release, not a permanent switch to a completely closed-source kernel.

[Update: Aug 7 2006: xnu-x86 has now been released. According to Kevin van Vechten's post at kernel.macosforge.org:

Several changes were made in order to publish the kernel (xnu) sources. As a result, the kernel built from these sources differs from the one found in the 10.4.7 software update. In order to accommodate these changes, several kernel extensions were also modified and must be downloaded and installed in order to run a kernel built from these sources on Mac OS X 10.4.7 for Intel.

Based on that comment, it sounds like the answer is #2 and for the xnu-x86 release they moved some code from the kernel into a kext.]

What the hell is going on here?

2006-05-16T01:29:00.001-04:00

I don't often write short posts, but this May 12, 2006 article from one of my favorite op-ed writers, Richard Reeves, really struck a chord:

The only way to restore constitutional checks and balances in Washington before 2008 is for the opposition to win one house of Congress and have the power to call witnesses at public hearings and ask, under oath:

"What the hell is going on here?"
What is going on in the White House? The Defense Department? The CIA and the NSA? With gasoline prices? Along the border between California and Mexico? In Baghdad? In New Orleans? With Jack Abramoff and the K Street Gang? In Congress itself?
Or, who is listening to your phone calls? Are your taxes being used to teach torture techniques to your sons and daughters? Are the glaciers melting?
We'll be the last to know.
The nation flies blind when we have determined one-party government. That can and has happened in both parties over the centuries, but this White House is a particularly tough bunch, talking freedom around the world and taking it away at home. President Bush essentially has veto power over the Republican automatons in the Congress....

The rest of the article is worth reading, but that's really it in a nutshell. And it summarizes the entire problem that we face today.

The entire Republican Congress, including our own Ohio Senators Mike DeWine and George Voinovich, have thrown away all pretense of holding the President and the executive branch accountable. They are nothing but a rubber-stamp for the President's bad policies.

God forbid that anyone suggest the President might possibly have made a mistake! They can't admit to even one.

We are in that most terrifying of states where virtually every single decision that's being made is wrong, but nobody is willing to stop. It's like the worst death spiral you've ever seen from a meth addict, or a horror movie where you are yelling at the screen: "No, DON'T split up! The killer is taking you out one by one!" And yet we are powerless to stop it.

Democrats speak out daily, but it's the Republican majority that controls all the committee chairmanships and sets the agendas for both houses. And it's the Republicans who make sure that no actions that force accountability ever make it to the floor. In the past year alone I've seen video of at least five instances of Republican chairmen overruling perfectly valid objections from Democrats, silencing debate, and completely ignoring accountability.

If you wonder why Congress doesn't do something to stop the President, you are really asking why the Republicans won't do anything to stop the President.

You know what? That's a damn good question.

High-Resolution Images Could Choke Internet

2006-05-14T21:05:00.001-04:00

Perhaps you've seen this article, entitled High-Definition Video Could Choke Internet, already:

Every day, it seems, a new service pops up offering to send you video over the Internet. "Desperate Housewives," Stephen Colbert heckling the president, clips of bad dancers at wedding parties: It's all there.
You may be up for it, but is the Internet?
The answer from the major Internet service providers, the telephone and cable companies, is "no." Small clips are fine, but TV-quality and especially high-definition programming could make the Internet choke.

Man. That just sounds really familiar to me for some reason. I've been online for more than a decade now, and this has been bugging me all day. So I started searching through my old floppy disks for news articles that I'd stashed away back in my university days. It turns out this argument isn't new at all! Check it out (sorry, no link to the original -- all I know about the origin is that I saved it off Usenet as jpegbad.txt):

High-Resolution Images Could Choke Internet By SVEN PETERSON, Microcomputer Technology Writer Sunday Mar 24, 1996 NEW YORK - Every day, it seems, a new service pops up offering to send you images or software over a new service called the World Wide Web, or WWW. Astronomy pictures of the day, ESPN.SportsZone.com, the Internet Movie Database, DOOM wads, Info-Mac archives: It's all there. You may be ready for it, but is the Internet? The answer, from major Internet service providers such as Prodigy, Compuserve, and eWorld, is "no." Small pictures are fine, but GIF animations, full-color JPEGs, and especially high-resolution images could make the Internet choke. Most home Internet use is in brief bursts: an e-mail here, a gopher session there, followed by some Telnet activity. If people start browsing the World Wide Web like they watch TV -- for hours at a time -- that puts a strain on the Internet that it wasn't designed for, ISPs say, and beefing up the Internet's capacity to prevent that will be expensive. To offset that cost, ISPs want to start charging content providers to ensure delivery of large JPEG files, for example. Internet activists and consumer groups are vehemently against those plans, saying they amount to deliberately fouling the Internet's level playing field, one of the things that has encouraged communication and collaboration and may, in the future, provide a much-needed boost to the US economy. They want legislation to guarantee a "neutral" Internet, but prospects appear slim. At the heart of the debate is a key question: How much would it really cost the Internet carriers to provide a couple of megabytes of JPEG-laden WWW pages over their networks every day? The carriers are not telling, but there are ways to get close to an answer. One data point: As a rough estimate, an always-on, 1000 kilobit-per-second connection to the Internet backbone in downtown Cleveland, purchased wholesale, costs an ISP $100 to $200 a month, according to the research firm GeoTelegraphy Inc. An ISP's business is carrying data from that connection to the customer. One thousand kilobits per second is obviously a lot of bandwidth, so ISPs have to spread that bandwidth out over their subscribers. That connection can serve about 30 users who are using every bit of their 28.8 kilobit-per-second dialup connection. But analysts estimate that ISPs sell around 30 times more bandwidth to their end users than they can connect simultaneously to the Internet, meaning that these thousand kilobits are often shared by roughly 900 users. In a way, dial-up internet service is like an old-fashioned telephone service, where there are always more lines leading from homes to the local switching station than there are going from the station out of the neighborhood. If everyone picks up the phone at once, there won't be enough outgoing lines for every call to go through. But the system works because that rarely happens. Oversubscription doesn't present a problem as long as people are only using the Internet for e-mail, telnet, and the occasional FTP download. But if everyone in a neighborhood is trying to "surf" the WWW at the same time, it's just not going to work. "The simple truth is that today's networks simply don't have the capacity to deliver all that customers expect," says Timothy Taker, Prodigy's top lobbyist. The solution, of course, is to make the pipes connecting to the Internet fatter. To illustrate what that would mean, Compuserve's chief architect, Franz Kafka, uses the assumption that the cost of providing a month's worth of data to the average user, about 2 megabytes, costs the company $1. That's a fairly small amount compared to the $25 to $47 a month Compuserve charges for dial-up, but then the company has to pay for sales, support, maintenance and a host of other costs. If that same user were to start downloading twenty print-quality high-resolution JPEGs per month, Compuserve's data cost, not including the cost of maintaining the dial-up line, would go up to $4.50 a month. Higher, but perhaps not high enough to break their business model. But if the customer starts browsing the WWW like the average household watches broadcast TV, 8 hours a day, Compuserve's cost would go up to $112 a month, enough to inspire nightmares in the corporate accounting department according to Kafka. "We don't expect to get to the point where we're charging anyone those kinds of prices for Internet service, but it does reflect the kind of impact that high-resolution JPEGs and GIFs could have on the network and business models for the Internet," Kafka said. To deal with that, Kafka says Compuserve might put caps on the amount of data that a residential user gets for free, and charge extra if the user goes over. Other options include charging content providers extra for guaranteed delivery of JPEG images and GIF animations, which has raised the hackles of Internet content providers and activists. However, Kafka's estimates for these costs aren't really Compuserve's. Like other ISPs, they don't disclose their actual costs. Instead, Kafka's base figure of $1 for 2 megabytes of data per month is based on an estimate by Dave Tricklestein, sysop of the Dial-Up Prime BBS, and Tricklestein thinks Kafka has it wrong. "Traffic just isn't growing that fast," Tricklestein said. "It will grow and it will even accelerate, but not fast enough to turn into dollar amounts that really matter." The new WWW service is still just a small fraction of the total amount of internet traffic out there, and that's unlikely to change overnight, in Tricklestein's opinion. In fact, he said, it will probably be at least two more years before the WWW takes off in a big way. Prices of network equipment like switches and routers have been falling, and that trend is likely to continue. Tricklestein believes the danger of letting the carriers charge extra for guaranteed delivery is that they'll put the spending for upgrades into creating that extra "toll lane," and won't reduce oversubscription in the rest of the network even though it would be cheap to do so. Both Compuserve and eWorld have said they won't degrade or block anyone's Internet traffic. But it's impossible to tell what goes on inside their networks. So what's the message? Stay tuned, and watch your modem connection speeds.

Funny. Whatever happened to Compuserve and that short-lived dial-up business model where you got charged extra for overages? Oh yeah, consumers ran away from it in droves in favor of unlimited-download services so that they could browse the Web without restriction.

Apple's new laptop

2006-01-12T13:16:00.001-05:00

It's now been two days since Apple announced the first machines ever to ship with Intel processors, due to ship in February. (See my earlier post: Apple's future with Intel.) Steve said something about a new Intel-based iMac or something, I guess, but nobody cares because he also announced a new pro laptop that uses an Intel chip.

The usual flurries of oohs and ahhs, glowing reviews, playful cynicism, and plugs from unexpected directions have all played out. Now people are starting to look a little more closely at the offerings.

Missing Pieces

Finally, for the first time I have no connection whatsoever to Apple and I can talk about a new release without the fear of getting arbitrarily fired. So I'll dig in from my new perspective as an Apple outsider and take a look.

Rosyna examines what's not in the new laptop in his enigmatically titled Lost in Transition: Overcane of Antflower Milk. Don't worry about the title; he's just like that.

To sum up: no S-video, no FireWire 800, no modem, slightly lower resolution (1440x900, when the previous laptop was 1440x960), no dual-layer DVD burning. And from looking at the battery and the power brick, it seems likely that power consumption is actually higher on this machine than the previous iteration. (Not a big deal to me, since that's the natural progression anyway -- but worth noting.)

I'd add two things to that list:

No PC card slot. It was traded up for an ExpressCard/34 slot, which will ultimately be a good thing... but first, people will have to start making cards that fit it. To call the current offerings sparse would be generous. That'll be fixed in a year, but it may be a concern for a handful of early adopters who need PC cards for one reason or another.
No two-button trackpad. This doesn't matter when you're running Mac OS X, because OSX uses control-click. But tough luck if you were hoping to dual-boot into Windows; you simply can't run Windows usefully with a one-button mouse.

Many people have noticed that Apple is being suspiciously quiet about battery life. No statements on battery life are available, which is very odd for a laptop announcement. My sources suggest that they're simply not done with the final power management code, so they don't want to release actual metrics yet. If I may insert a side comment here, my own experience and tendency toward cynicism suggests that they will ship a half-assed version of power management with the machine, then patch it later with a couple of system updates to get to a version that actually works. Call it a hunch.

Performance

As the saying goes, "There are lies, damn lies, and benchmarks." Apple is claiming a whopping 4x speed boost over the previous PowerBook G4. But take a look at this breakdown of the benchmarks from Apple themselves:

Notice how only Modo -- an application that is heavily tuned for Intel chips -- is listed as about 4x faster, and everything else gets in the neighborhood of 2x. From Luxology's site:

The modo rendering engine deeply leverages various Intel Technologies to improve scalability and performance. Our bucket rendering provides near linear scales in performance with multiple processors and Intel® Pentium 4 processors with HT Technology.

Everything else that hasn't been hand-tuned quite as much for Intel chips only gets a boost in the neighborhood of 2x. Funny thing, really... 2x is just about the performance boost you'd get going from a single G4 to a dual G4.

From examining those benchmarks, my professional opinion is that it looks like a single core of the new Intel chip runs a little faster in these tests than a comparably-clocked G4, or roughly equal to a comparably-clocked G5.

The real performance boost comes from the fact that there are two cores. It's like the laptop went from a single-processor G4 to a dual-processor G5. That's a real boost, of course -- the machine really does seem to be twice as fast as the old machine, which is awesome and worthy of praise. But it's not because it's Intel vs PowerPC; most of the boost is due to the upgrade from single-processor to dual-processor.

As for people getting over-excited about Intel's SSE vs PowerPC's Altivec, my personal opinion -- as someone who's written code for both -- is that in the final analysis they're really quite similar. Sure, there are differences. Each is good at different things, and optimizing for one is not the same as optimizing for the other. Altivec has more registers, but then again you pay for those extra registers during context switches. Switching to SSE is more of a lateral move than a step back. It'll take a while for Apple and other vendors to convert everything that was Altivec-optimized to be fully SSE-optimized, but they'll get there.

Update: Ars Technica comes through with a hands-on look at the new iMac, and had a chance to run some benchmarks. Since they use the same CPU, the iMac probably has performance very similar to the new laptop. Check it out for more details.

Dreams of Dual-Booting

One of the reasons I was really interested in the new laptop was the possibility of dual-booting into Windows, and later, when the software became available, running Windows in a VMware-style shell.

My current work (contracting for Sony on PSP/PS3) requires me to use Windows XP on a daily basis. While I've mostly managed to customize my Windows system to the point where it satisfies my needs, I still really miss a lot of the nicer little features of the Mac: iChat, iCal, iPhoto, even AppleScript. And when I'm traveling I can't really work on Mac software without bringing two laptops along -- which is obnoxious both because of the extra weight and the extra hassle at the dog-and-pony show that passes for airport "security" in this country.

A single dual-booting laptop would have been a great solution for me -- I'd sign up to buy it right now if I thought the new laptop would deliver on that front, even without a two-button trackpad.

But it's been reported that Windows XP won't boot the laptop, because it uses EFI rather than a BIOS to boot. Longhorn, aka Windows Vista, should work... but who knows when that's coming out? Right now it's supposed to be the end of 2006, but it appears to be even money on whether Microsoft will actually make that date. The Longhorn schedule and magical ballooning feature list has looked a lot like Copland's so far, which isn't very heartening.

In summary, don't get this laptop for its capability to run two OSes; it will be a while before it can. Perhaps once there's a consumer VMware product for Mac OS X that boots XP I'll take a second look at it.

Update: In an interesting twist, Intel has firmly stated that you can definitely boot Windows XP with an EFI Core Duo system if the vendor provides a BIOS compatibility shim. Interesting! However, Ars Technica reports that they had no joy installing either XP or Vista on the new Intel iMac, so the shim is either not there right now or just not provided with the iMac.

But now that it seems at least technically possible, I bet we'll see a BIOS shim for the new laptop -- either factory installed, or as a download. Keep your eyes peeled; it may not be over yet!

The Good Stuff

So far this has seemed way too negative. That's not really how I feel. Overall I have to say I really do like the laptop as a Mac.

Things like lack of S-video and a modem don't bother me personally. Now that wi-fi has really taken off, it's not the crisis it used to be to be stuck in a hotel room without a modem -- you just have to find the nearest Starbucks. If you are one of the 2% of users who really need to use S-video or a modem regularly, you can get an adapter dongle. No, it's not as nice. But it works fine, and it helps bring the price of the laptop down for the rest of us.

I think the built-in iSight is nice; while not always useful, it's one of those things that's nice to have standard. Video chat and other applications of the camera (think Flickr) will increase dramatically if the iSight is now going to be standard on every new machine. I don't like bringing my iSight with me when I travel, but I'd definitely use a built-in one to video chat with my wife if I had it.

And the speed, ah, the speed. This machine really is fast. If you are looking for a fast new Mac laptop, this is what you've been waiting for.

What's up with the name?

I haven't even talked about the name yet. Apple used to sell the PowerBook. I guess they still do, at least for a while. The brand had good name recognition despite that awkward capital B in the middle that nobody actually bothered to type.

But this new laptop has been rebranded and is no longer a PowerBook. Instead, it's called a "MacBook Pro". Hardly anybody I've talked to likes the name, myself included. But I might be able to guess at the rationale behind it. Bear with me.

"PowerBook" is a brand name. A PowerBook actually has three unique brands associated with it: "PowerBook", "Mac", and the superbrand "Apple". Personally, I think it's a very strong name, and it has the benefit of 15 years of brand-building success behind it. People know what you're talking about when you say PowerBook.

MacBook Pro, on the other hand, is a brand extension. A double brand extension, really. "MacBook Pro" is an extension of "MacBook", which itself is an extension of "Mac". On the face of it, from everything the The 22 Immutable Laws of Branding tell us, this is a much much weaker name. Why on earth did Apple decide to make this change?

The only reason that makes sense to me is that Apple must have decided that they're going to start consolidating the Mac brand. Rather than having separate brands under the Mac umbrella, everything Mac is now going to include Mac in the name. If that's true, then the "iBook" name will going away to be replaced by just plain "MacBook". "PowerMac" will go away, to be replaced by something like "Macintosh Pro".

Why consolidate the Mac brand? Perhaps, and just perhaps -- this is wild speculation -- it's to get ready for a possible future move to OSX running on non-Apple PCs. Let me be clear here: I don't expect such a move of the OS to non-Apple PCs for at least five years or more. But Apple of course has to be thinking about the possibility of competing with Microsoft in the future.

If they were to start letting OSX run on non-Apple PCs, they might want to rebrand "Mac OS X" to "Apple OS X" to make a distinction between the high-end Mac computer brand and other non-Mac computers. But doing that would weaken the Mac brand, right? So that may be why we're seeing the brand consolidation now, years before any of these other changes take place. It's about strengthening the brand in anticipation of possible weakening later.

Just a thought.

I'm so totally Extreme

2005-11-30T02:10:00.001-05:00

This weekend I finally upgraded my wireless network at home. For the past three or four years I've been using a second generation AirPort base station, which was 11Mbps with 40-bit WEP. It's been thoroughly out-of-date for a while now, but it met my needs, was rock-solid, and worked like a charm so I didn't feel the need to upgrade. And after all, 5.5Mbps is still more bandwidth than my real-world connection to the internet, so I knew that switching to the higher-bandwidth AirPort Extreme wasn't going to make a difference in simple web browsing.

But this week I finally broke down and made the jump to the latest and greatest. A combination of two things finally pushed me over the edge:

I've been doing a lot more work lately that involves transferring things between machines in my office. Local transfers from machine-to-machine are where 54Mbps can really make a difference.
The signal strength wasn't so great everywhere in the house, and I wanted to use WDS with an AirPort Express to extend the range of my network.

Another small factor was that if I upgraded, I could have iTunes send its output directly to my stereo via AirTunes. Personally I think AirTunes is kind of a gratuitous and somewhat goofy feature. It's certainly not enough to make me spend a couple of hundred dollars on new equipment. But I have to admit that once the other (much better) reasons pushed me over the edge, I was looking forward to giving it a shot.

So I made the plunge and bought an AirPort Extreme Base Station and an AirPort Express, and hooked everything up.

Perhaps because I like pain, or maybe just because I wanted to see if it would work, I decided to set up the AirPort network from a Windows laptop. Okay, the real reason is that my old TiBook recently died from catastrophic hinge failure and I couldn't find where I'd left the charger for the iBook, which meant that I didn't have a wireless-capable Mac handy.

Setting up the main base station worked like a charm. I plugged it in, ran the Windows version of the admin utility -- which I have to say is really very nice -- and saved the configuration from the old base station into a file on my desktop, then imported it into the new base station. Happily, it copied out my DSL PPPoE account and password. In the process I upgraded the wireless security from WEP to WPA, set my server box as the default host (DMZ), and configured the base station so that it would syslog to one of my machines. Restarted the base station, connected with the new password, and everything worked flawlessly. Great!

Getting the Express set up to extend my network, however, was a little bit trickier. I started at the obvious place with the AirPort Express Assistant for Windows. But somewhere in the middle of the setup as it jiggled Windows XP and base station settings, it failed while reading from the base station with an error -4: "bad param". Tried again several times with the same result. Hmmm. Not so great, and virtually impossible to diagnose what went wrong.

Why, no problem, says I, I'll just configure it directly with the admin utility.

That's much easier said than done. I did get it done in the end, but it certainly didn't go as smoothly as I was hoping it would. There are a lot of non-obvious details that need to be just right before everything works. Most of the answers can be found in the admin utility's help if you know where and how to look, but it takes some digging.

Here are some tips from my experience with setting up WDS manually:

Terminology. You have the main, relay, and remote stations.
- The main base station is the one connected to the internet via Ethernet.
- Relay stations connect between base stations, and do not have an Ethernet connection.
- Remote stations provide services to clients, and do not have an Ethernet connection.
As far as I can tell there's not a lot of practical difference between relay and remote, since both main and relays can be configured to accept client connections too. It may be the case that remote stations are able to dedicate more bandwidth to clients than main or relay stations, but I'm not sure about that. In any event it seems more like something you'd only be concerned about for a large-scale installation with lots of client computers and extremely heavy traffic -- I doubt it matters for home networks.
MAC addresses. Before you start, you need to write down the AirPort MAC addresses of all the base stations involved. It's on the outside of the base station, or you can get it from the base station chooser. If you do that, though, remember that the stations will be broadcasting different wireless networks at first. So you need to join each one in turn, then select it in the chooser (no, not that chooser) and write the AirPort MAC address down somewhere.
"Distribute IP addresses" should only be set on the main base station, not on relays and remotes. Thankfully, the admin utility warns you about this.
All stations must use the same channel. Pick a channel (I like 3 and 10) and set both base stations to it. The admin utility tells you that you can't use 'Automatic', but neglects to mention that all base stations have to use the same channel -- which is kind of an important detail.
Set up the main base station first, then relays, then remotes. When you set up a WDS main base station, you'll need to enter the MAC addresses for the WDS remote and relay stations that will be allowed to connect. It doesn't work if you go the other way, because a remote base station won't be able to connect to the main base station until the main base station has been configured.
Use different SSIDs (network names) at first. It doesn't matter whether base stations connected by WDS have the same network name or different names. Toms Networking recommends that you use different SSIDs, while Apple's Designing AirPort Extreme Networks for Windows recommends on page 38 that you use the same SSID. But if you plan on giving them the same name eventually, don't start out that way! Give the networks different names so that you can be sure you're connecting to the right base station when testing to make sure that it works. Once everything is working you may then decide to set them to the same SSID; it's up to you.
Know where the reset button is. If you make a mistake and can't find one of the base stations on the wireless network anymore, hold down the reset button for about seven seconds (until it starts flashing quickly) to give it a hard reset. The button is recessed, but can be pushed with a paperclip, staple, ball-point pen, or stereo miniplug.
Double-check all of your settings if you are importing settings from an older base station to a newer base station. At one point in the process I noticed that my main base station's transmission strength was apparently set to the lowest setting -- 10%, instead of 100%. I can only speculate about why that happened. The older base station didn't have adjustable signal strength, so perhaps it pulled in a zero value rather than the default of 100% when it imported my old configuration to the new base station.

Phew. Anyway, after a little bit of futzing around, I've got it working and I'm happy. The speed on the local network is much better, and the WDS extension has made the signal strength much stronger throughout the house. It wasn't too much of an ordeal and I got it all sorted out in an hour or two, but it was certainly harder than it should have been. Apparently I'm not the only one who thinks so, either.

But as a bonus to reward all my hard work, I can now have iTunes play through my stereo via AirTunes. Dude, I'm like so extreme.

Ohio Needs Reform

2005-11-08T23:20:00.001-05:00

[Update 11:20pm: Wow. The full results aren't yet in, but with about 79% of the votes counted it looks like all four of the RON issues got squashed badly. What a shame -- so many Ohioans will be disenfranchised by this, and the state will continue to be mired in economic woes. The current batch of crooks who got us into this mess will continue to rig the system to keep themselves in office. A sad day for Ohio.]

If you live in Ohio, you've probably heard about the Reform Ohio Now proposals that are on the ballot as Issues 2, 3, 4, and 5.

These are at issue in the election on November 8th. That's TODAY! Make sure you get to the polls and vote YES on these issues!

What are these issues about?

I would recommend reading either the summary or the actual text yourself, but in a nutshell:

Issue 2 makes it easier for Ohioans to vote by mail. This is a growing trend in voting, since many precincts have lines of up to several hours to vote and many people can't take that kind of time off from work to vote. It's proven successful in other states, and it's long overdue here: Ohio has some of the most restrictive rules for absentee ballots in the nation.
Issue 3 puts limits on state campaign contributions. These limits are similar to the McCain-Feingold limits in place for national campaigns. These are designed as a step towards making sure that the super-rich can't have an undue influence on politics.
Issue 4 fights gerrymandering. This is what it's called when elected politicians redraw political boundaries to favor themselves. Gerrymandering leads to fractured districts and unfair elections, and is designed to artificially skew the balance so that the politicians elected do not actually represent the true mix of views in the state. In Ohio, many counties are sliced up into two or three different districts, and two congressional districts (OH-5 and OH-18) cover parts of sixteen different counties. Ohio as a whole votes almost exactly 50-50 Republican vs Democratic, but out-of-control gerrymandering has conveniently adjusted the boundaries to disenfranchise voters so that our Congressional representatives are two-thirds Republican.
Issue 5 creates an independent panel to administer elections. Right now we have strongly partisan career politicians like Ken Blackwell in charge of oversight for the elections of themselves and their cronies, and they do a deliberately bad job of it -- wilfully disenfranchising thousands of Ohioans in order to push the vote in the direction they want it to go.

Issues 2, 3, 4, and 5 are from the people of Ohio

These issues come from people just like you and me, people who are fed up with the amount of corruption in Ohio. There's a phrase that describes the situation here, where the political machine is making the rules for itself and overseeing itself: "the fox guarding the henhouse". Not surprisingly, that fox has been burying its face in all the eggs it can steal.

Corruption? Our state has plenty. You've heard of Tom Noe, the coin dealer and state Republican crony who got $50 million in unaudited sweetheart investment contracts from the Ohio Bureau of Worker's Compensation, and was disgraced for allegedly embezzling and just outright "losing" up to $10 million of that money. He was recently indicted for money-laundering, accused of circumventing the campaign finance donation limits by giving people money that they would then donate to various campaigns. The hell of it is that the money he used was probably from that $10 million he "lost" of the state's money.

Follow the trail: crony given contract by corrupt Republican officials, crony uses that contract to steal money from the state, crony uses stolen money to make sure corrupt Republican officials get re-elected. It's a vicious circle, and we need a change.

A group named Ohio First has been trying to portray these amendments as coming from "special interests". But they're full of shit -- fact is, they are the special interests. The average donation to Ohio First is $12,941. And remember, that's the average. How many Ohioans do you know that can afford to donate that much to a political campaign? Not even a candidate. A campaign against a couple of amendments. Meanwhile the average donation to Reform Ohio Now is $545. That reflects the large number of small donations (over three thousand and counting) from people like you and me across the state supporting the amendments.

Vote YES on Issues 2, 3, 4, and 5

On November 8th, vote for reform. Vote YES on Issues 2, 3, 4, and 5. While you're at it, I'd suggest voting YES on the following:

YES on State Issue 1 -- investment in high-tech and Ohio businesses. Ohio needs to invest and plan for the future, and this seems like a good start.
Cuyahoga residents should vote YES for County Issue 6 — funding for Tri-C.
Cuyahoga residents should also vote YES for County Issue 7. If this issue doesn't pass, funding for the Cuyahoga County Board of Mental Retardation and Developmental Disabilities will be cut by 60% and many programs that help retarded and developmentally disabled kids and adults will be cut.

Any other Brecksvillians out there? Here in Brecksville we've got Issue 14, which appears to be an adjustment of zoning height restrictions to make it easier to build McMansions in the new developments. Currently each McMansion has to get individual variances, which are almost always granted. I believe this would just make the variances permanent across the board. Personally, I am probably going to vote against it because I'm not in favor of McMansions in general -- they tend to be energy-wasteful and detract from the small-town atmosphere of Brecksville. But given their popularity around here and the deliberately vague wording of the issue, I'm afraid it's going to pass anyway.

Thick as Thieves

2005-10-31T01:34:00.001-05:00

Have you noticed there've been an awful lot of politicians getting busted lately? I was curious so I started to make a list.

Jack Abramoff, famously influential Republican lobbyist with close connections to Tom DeLay. Indicted on five counts of wire fraud and one count of conspiracy to commit wire fraud, August 11, 2005.
Adam Kidan, influential Republican lobbyist and Abramoff's partner. Indicted on five counts of wire fraud and one count of conspiracy to commit wire fraud, August 11, 2005.
David Safavian, Republican procurement chief at the White House Office of Management and Budget. Arrested and subsequently indicted for five counts of obstruction and related charges for lying to investigators in an attempt to cover up Abramoff's dealings. September 19, 2005.
Senator Bill Frist, Republican Senate Majority Leader. Under formal investigation by the SEC for insider trading, for dumping large quantities of HCA stock just before a bad quarterly report. Frist originally claimed the stock was held in a blind trust, but that was proven completely false. No indictment yet, but seems very likely. September 23, 2005 and ongoing.
Congressman Tom DeLay, Republican House Majority Leader. Indicted on charges of conspiracy to break campaign finance laws. Indicted again on charges of felony money laundering just a few days later. September 28, 2005 and October 3, 2005.
Tom Noe, Ohio rare-coin dealer and top Republican donor. Indicted on three counts for illegally circumventing campaign finance limits to funnel money to the 2004 election campaign of George W. Bush. Also continues to be under heavy investigation for losing up to $12 million belonging to the State of Ohio that Republican buddies got the Workers' Comp Bureau to invest in an unprecedented, unsecured, and unaudited "rare coin mutual fund" that he ran. It's not a big leap to suggest that he may have been embezzling from the state and turning that money directly into campaign contributions to Republicans, but investigations are still pending. October 27, 2005.
I. Lewis "Scooter" Libby, Vice-President Cheney's chief of staff. Indicted on charges of perjury and obstruction of justice for lying to investigators in the CIA leak case involving Valerie Plame Wilson. October 28, 2005.

Notice anything? Number one, they're all Republicans. Why do you think that is? If you are the type to foolishly insist that this is some sort of partisan thing, and all of the indictments are frame-ups or something, then why aren't Republicans retaliating by getting Democrats indicted the same way? Is it because the Dems are all evil and bad, and the saintly Republicans are "above all that" and prefer to practice squeaky-clean politics? Pfft. Yeah, that doesn't even pass the laugh test. Pull the other one.

Number two, they're all very big fish in their areas of Republican politics. Frist and DeLay are the Senate and House majority leaders, people. It doesn't get much bigger than that. The lobbyists are huge in their field, and wield vast amounts of influence with the money they command. Safavian was director of procurement at the White House. Libby is no mere functionary in the Vice-President's office, but rather his chief of staff. And Noe is so thoroughly wrapped up in Ohio state politics that five out of the seven Ohio Supreme Court justices had to recuse themselves from a case that came before them as part of Noe's Coingate because they had received campaign funds from him. Look it up; he's connected to nearly every single major Republican office-holder in the state.

Worst of all, that whole list described above does not represent all the criminal activity that's gone on. It's just the stuff where they have enough evidence to go after these guys.

Beyond the criminal activity, don't even get me started on incompetence. Where do you think FEMA is going to be when there's an emergency near you? Will the National Guard have enough troops and equipment on this side of the globe to respond?

These guys are out of control.

Hurricane Rita

2005-09-21T13:23:00.000-04:00

Would you believe there is another nasty hurricane threatening the Gulf coast? Hurricane Rita is a ~~CAT 4~~ CAT 5 storm which is moving westward across the Gulf of Mexico. It's currently projected to hit the coast of Texas.

(I've copied the satellite map pictured here to my own web space to reduce bandwidth usage; click to see a live update from Weather Underground.)

[Update 11:50pm: Rita is now a CAT 5. It also has a measured central pressure of 898 millibars. The CP of a hurricane is a rough measure of its strength; the lower the pressure, the stronger the hurricane. Rita's CP makes it the third most intense hurricane ever recorded, and it is likely to intensify over the next 12 hours. The measurement may not have recorded the actual lowest pressure, either:

DROPSONDE DATA FROM AN AIR FORCE RESERVE UNIT RECONNAISSANCE AIRCRAFT AT 623 PM CDT ... 2323Z ... INDICATED THE CENTRAL PRESSURE HAS FALLEN TO BELOW 899 MB...OR 26.55 INCHES.
THE DROPSONDE INSTRUMENT MEASURED 32 KT/35 MPH WINDS AT THE SURFACE...WHICH MEANS IT LIKELY DID NOT RECORD THE LOWEST PRESSURE IN THE EYE OF RITA. THE CENTRAL PRESSURE IS PROBABLY AT LEAST AS LOW AS 898 MB...AND PERHAPS EVEN LOWER.

Rita is now stronger than Katrina was at its strongest, and I repeat, it's likely to intensify tomorrow.]

[Update Thu 1:00pm: Rita is expected to weaken to a CAT 3 or 4 prior to landfall, but the meteorologists at Weather Underground are saying that the storm surge will be equivalent to a CAT 5, just as Katrina's was.]

The Texas coast has a hardier shoreline that is not as fragile as the Louisiana/Alabama/Mississippi coasts, but the real concern is that it will do further damage to our country's energy infrastructure. A lot of oil drilling and refining infrastructure was damaged by Katrina, and Rita has the potential to damage a lot more -- and with the damage from Katrina still being repaired, this could be like kicking us when we're down.

We're all familiar with the way that Katrina drove up gas prices around the country to over $3/gallon. Prices have recovered since then, coming back down to a still-painful $2.69/gallon or so. This was not because refinery capacity magically recovered -- Katrina's damage to refineries is still there. What happened is that the undamaged refineries redirected their efforts toward making more gasoline as opposed to other oil products like heating oil, because gasoline was in such high demand and was selling for such a high price. That's a relief at the pump, but doesn't bode well for winter heating costs.

Now Rita is aiming right at a series of drilling platforms and refineries off the short of Texas. The primary danger for the average person's wallet is the refineries. If the absolute worst-case scenario is true, Rita could shut down a third of the already-reduced refinery capacity of the United States. Even if that happened for only a few days, it would be absolutely brutal to the economy and to your wallet.

Rita is still well on the east side of the Gulf of Mexico, and won't make landfall for a couple of days. Currently the NWS is projecting a Saturday morning landfall at 7am.

I expect that this will be the best source for Rita information over the next few days:

The Oil Drum - a progressive blog about Peak Oil. They have extensive information on Rita already.

Other places to keep your eye on include:

Jeff Masters at Weather Underground
Steve Gregory at Weather Underground
National Hurricane Center from the National Weather Service

And I can't recommend this enough for detailed, easy-to-read background information:

DarkSyde's OIL STORM RITA: Forecast & Energy Primer

This also presents a difficult, perhaps no-win situation for President Bush. Should he act early to avoid potential damage from Rita? Obviously the answer is yes, because to do otherwise would be foolish. But unless he approaches it with extreme humility -- a sort of "we've now learned our lesson" approach -- he's going to be damned by most of the country for acting quickly to save white people and oil in Texas, while letting black people struggle and die for days in New Orleans.

Does the President have a single drop of humility in him? I sure haven't seen any yet. What do you think?

Changing the DNS query timeout in Windows XP

2005-09-09T10:27:00.000-04:00

I've been having some networking trouble lately. When my PC laptop is busy downloading a file, Windows XP starts failing to resolve DNS queries. So even simple lookups that I know must be cached at multiple levels, like www.google.com, start failing to resolve. Windows just times out after fifteen seconds and gives up.

Needless to say, this makes web browsing while downloading a file insanely frustrating.

My Mac laptops don't seem to have the same problem. I have no idea whether this is a problem with my ISP, my wireless router, Windows itself, or some combination of the three. And frankly, as an end user I don't care and shouldn't have to care. I just want it to stop sucking.

I set out to see if I could increase the client-side DNS timeout so that Windows would be a little more forgiving about slow DNS responses. It turns out there is a way to do that, though it's nearly impossible to find via a web search. (Even Windows experts, which I make no claim to be, seem to have trouble with this one because it's so obscure.)

Here's the registry setting to increase the DNS client-side timeout in Windows 2000 and XP:

HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ Tcpip \ Parameters \ DNSQueryTimeouts

[Update Nov 3, 2006: Fixed the above link. It used to point here, but that link now redirects you to the main page for the Windows 2000 resource kit. Remember, kids, cool URLs don't change.]

Read the above link for details. The registry entry does not exist by default; you have to create it. I don't suggest you do this lightly unless you're familiar with using regedit to tweak parameters.

The default value when the property isn't present is documented to be "1 2 2 4 8 0", which appears to represent that 15-second total timeout. (15 = 1 + 2 + 4 + 8. It's not clear to me exactly what the other 2 is for; it may be redundant.)

I wanted something a little longer, so I quadrupled all the numbers to "4 8 8 16 32 0".

Now I have a 60-second total timeout, with the final query given 32 seconds to get through. In practice this has proven to be a long enough timeout that Windows can continue to resolve DNS names even when my network connection is busy.

And that's good news. I'm much happier again, and I can continue to use my PC laptop without wanting to chuck it out the window every time I download a file.

Katrina's Aftermath

2005-09-09T02:00:00.000-04:00

Wow. Well, I've been meaning to post a wrapup to follow up my previous entry, but the unfolding events in New Orleans have left me a little shell-shocked. It's been difficult to gather my focus enough to write a coherent entry. This is going to be a long one, because I have dozens of links and a lot of pieces to put together.

As you probably know, the situation in New Orleans started out bad, and then went from bad to worse. But it didn't even slow down at that point -- it quickly passed through awful, lingered a while in horrific, sped up as it passed through OMG, and ended up somewhere in "WTF is going on!?"

Throughout the past week I've relied upon several sources for information. Many of those were the links I shared with you in my last entry. But I've also been very impressed by how thoroughly the reader-contributed diaries at Daily Kos (in the sidebar on the right side) have covered the situation. Daily Kos is a political blog with a large community from the left and center. As with any community blog, like Slashdot, you have to sometimes take things with a grain of salt. There are occasional wacky conspiracy theorists who read and post there, but there are also economists, scientists, authors, engineers, members of Congress, you name it.

What happened

Southern Louisiana and the city of New Orleans specifically were declared a national emergency on August 27th, just prior to the hurricane's arrival. Then the governor of Louisiana requested specific funds and relief resources from the federal government the day before the hurricane arrived. Here's the governor's request for aid, dated August 28th, and the President's response, dated August 29th.

I originally thought the local authorities had dropped the ball on early evacuation, but it appears that I might have been mistaken. The New Orleans area started evacuation and contraflow on Saturday, two days before the storm's Monday landfall. That's not bad. I think the problem was that people didn't take the hurricane seriously until everyone else was taking it seriously, so for many people evacuation waited until nearly the last minute. There was to my knowledge no coordinated attempt to use buses or trains to evacuate people without cars over the weekend.

Just before the hurricane arrived, many people who either chose not to or could not evacuate (many because they did not have cars, and there was no plan for using public transportation to evacuate) were moved out of the city of New Orleans into temporary shelters at the Superdome and convention center. Other scattered pockets took refuge elsewhere in the city. Some people are angry over this because it was unknown even at the time whether the Superdome could withstand the expected winds, which were expected to be up to 150mph. But I think this anger is misplaced. This was the right move. For the short duration of the hurricane, the Superdome was probably the safest place to be. There is no situation in which the Superdome would have been less safe than a house in New Orleans: if the wind had been strong enough to rip the roof off the Superdome, it would have been more than strong enough to flatten an ordinary home. Evacuating to large, well-built structures like the Superdome and convention center was absolutely the right thing to do for the duration of the hurricane.

I have to qualify that statement with "for the duration of the hurricane", of course, because of what happened next.

Astonishingly, the drowning city of New Orleans was closed off and nobody was allowed to enter or leave. For the people trapped in the flooded, burning city there was literally no place to go. The convention center and Superdome turned from safe havens into hellish deathtraps -- with nobody allowed in or out there was no food, no water, no sanitation, no medicine, no nothing. And this continued, unbearably, for days and days. As people were rescued from other areas of the city they were sent to one of these two locations, which only served to increase the crowding and filth because nobody was being let out. As Geraldo Rivera pointed out in this must-watch emotional TV horror show, they could have just walked right out if they'd been allowed to.

It went on like that not for one day, not for two days, not for three days... but continued and grew worse over FIVE DAYS until finally, mercifully, someone got their act together. The human conditions down there are still awful this week, but the evacuation seems to be finally proceeding forward.

Accountability

So who is responsible?

FEMA: To begin with, there appears to have been an utter breakdown in the effectiveness of FEMA, the Federal Emergency Management Agency. These are the guys that are supposed to move very quickly and mobilize state and local resources to handle emergencies and save lives. Instead of saving lives, we get lists like this from Constructive Interference:

1000 folks from the Lafayette area with 500 boats head to NO to aid the rescue get turned back by FEMA.
Wal-Mart trucks with food and water get turned back by FEMA. More here
The USS Bataan off the coast of LA ready to help, but underused by FEMA. See also here
Shipments of diesel fuel being turned back by FEMA

firemen from Houston turned away by FEMA
More fire fighters turned away.
Angel Flight South Central seaplanes getting a run around from FEMA
DMAT Teams available but still on call.
Red Cross kept out of New Orleans. Note: this is actually attibuted to the state Homeland Security department. More information on who is responsible for this decision would be helpful. See also here
Northcom ready to act, but not given needed orders.
Mobile medical lab stalled in Mississippi. Doctors and hospitals offering aid but not getting response. This article is a bit confusing as to whether the problem is at the state or federal level or both.
NYT article discusses the controversy and provides additional claims of FEMA interfering generally and some specific examples.
Florida airboaters stalled by FEMA.
Water tanker aircraft for fire fighting and Amtrack trains for evacuation. See here.
Morticians turned away.
Maryland fire fighters blocked by FEMA.
Generators turned away by FEMA in Slidel.

There's even more in the full story on CI. It seems pretty clear that rather than aiding and directing relief efforts to where they were needed most, FEMA acted as an awful hindrance to relief.

Some ugly stories have come out about several of the leaders of FEMA. It's getting tedious to link everything, but essentially a lot of the leadership seems to be Bush cronies with no prior emergency management experience. Here's more on FEMA director Michael Brown (college roommate of Bush's campaign manager), Chief of Staff Patrick Rhode (Bush's campaign event manager), and Deputy Chief of Staff Scott Morris (marketing director and media strategist for Bush's campaign). None have any previous emergency response experience to speak of; their positions appear to be political patronage and nothing more. [Update: TIME magazine reports that Brown's resume appears to be thoroughly padded, making him a great deal less qualified than he claims -- which wasn't very qualified to begin with.] Whether by malice or incompetence, it seems likely that people like these appear to have utterly crippled FEMA's ability to respond.

The Executive Branch of Government: Yes, not the legislative or judicial branches of government... this was a failure of the executive. In our country we have the guys that make the laws, the guys that interpret the laws, and the guys that enforce the laws. This was a complete and utter failure of enforcement and execution.

A lot of people are angry at George Bush. A very few people are still trying to defend him, even though Bush's actions seem more and more indefensible every day. Here's my take: Although Bush wasn't running FEMA, he appointed the people that were. As President he had a responsibility to pick capable directors, and it's becoming clear that he didn't do that. Bush also sets the budget for FEMA, which by all accounts has been slashed considerably. (Why? Take your pick: tax cuts for the rich, or the war in Iraq, or both. But that's another topic.)

The hell of it is that just last year FEMA and New Orleans ran a training scenario that is chillingly similar to what actually happened with Katrina. The scenario was entitled "Hurricane Pam". The first part of the training involved simply running through the scenario and seeing what happens. That was completed, but many loose ends were discovered -- including the fact that evacuees in the Superdome and other areas would be stranded with no place to go. The second part of the training, where plans would have been developed to fix all the things that went wrong during the exercise, was dropped due to a lack of funding.

Like him or not, Bush is the de facto leader of our country. And dammit, the leader of our country should be proactively handling things like a leader. Take responsibility, fire people that screw up, get in there and do it yourself if you have to. That's the American way. Don't cower and hide and blame everything on your subordinates. (Unfortunately that's sometimes the American way too, but we're not as proud of that one.)

But Bush and the people under him have shown absolutely no leadership -- in fact, negative leadership. Really what we're seeing is that they have bungled it just absolutely beyond all recognition, and are too busy patting each other on the back to notice what the entire rest of the country has noticed -- that it's a giant steaming clusterfuck. I'm half afraid that we're just going to see Medals of Freedom all around and Michael Brown will be rewarded with a Supreme Court nomination. That really might make me lose it.

Some of Bush's defenders have pointed out that hey, the mayor of New Orleans and the governor of Louisiana may share some of the blame. That's very possible. But all the documents and interviews I've seen suggest that they were screaming upward in the hierarchy for help even before the hurricane hit, and everything went to FEMA, who proceeded to drop the ball. FEMA could only be overridden by DHS (represented by Chertoff) or by Bush himself, neither of whom did anything. Under the mayor and governor prior to the hurricane, New Orleans achieved about 80% evacuation which isn't bad. And moving everyone who was left to the Superdome for the duration of the hurricane was the right thing to do. It really sounds like FEMA was the big screwup. Still, the mayor and governor are part of the executive branch too, so they will have to be open and accountable to the public for their actions as well.

(Some of the dumber or more desperate Bush apologists have been trying to claim that the mayor and governor should have ALL of the blame, leaving Bush and FEMA scot-free -- which is just a brown, drippy, steaming crock of bullshit.)

Update: Newsweek just published an article entitled "How Bush Blew It". Despite the unflattering title, it does not lay all the problems at Bush's feet. It's actually a pretty fair all-around discussion of the events and mistakes and problems of the executive branch's response to the storm. It reads like a strikingly honest assessment, which is something I'm not used to seeing from the press after several years of Iraq war cheerleading. Neither Bush, Chertoff, Brown, Blanco, nor Nagin come out looking all that good, frankly, though some look better than others.

The Legislative Branch: There were failures in the legislative branch as well. Repeated abominations like the pork-filled transportation bill have diverted federal funds from where they can do the most good. (In this year's version, Alaska was awarded over $400 million to build bridges to a Senator's vanity... for no viable economic reason whatsoever.) Forward-thinking programs that would have slowed or reversed the erosion of the buffer around New Orleans were cut. Those are not as immediate as the failures in the executive branch, but the legislature's problems are more subtle and more entrenched, and probably more difficult to solve.

Southern Racism: [Added Sep 9th at 3:45pm.] It's coming out that the police chief of Gretna, Louisiana was apparently responsible for pinning people in New Orleans. The people who were trapped were predominantly black.

In an interview with UPI, Gretna Police Chief Arthur Lawson confirmed that his department shut down the bridge to pedestrians: "If we had opened the bridge, our city would have looked like New Orleans does now: looted, burned and pillaged."

I don't think active 'drown as many darkies as we can' racism was at play here. (In the deep South you never know -- but I'm going to at least attempt to give him the benefit of the doubt. For now.) But if it's not that, then it's absolutely a matter of race-based fear, the fear that comes as a result of practical segregation. That fear is strengthened by what's called 'structural racism', a long-standing problem in the US where race and economic class are tied together, and racial divisions strengthen class divisions and vice versa.

Conclusion

Wow. I haven't I haven't even had a chance to get into the economic effects -- such as gas never seeing the underside of $3.00/gallon again because of damage to refineries and drilling, or heating costs quadrupling from 2003 prices because of damages to the natural gas supply from the Gulf. Or the environmental effects -- at least 20 drilling platforms were destroyed and are uncapped, and several large oil storage tanks have started leaking. Or the effect on our status as 'superpower', which appears to have been largely erased as a stunned world sees the country utterly unable to handle a thoroughly predictable disaster. (Although you know, a little dose of realism may be a good thing.) Or the very real possibility that all of the above economic effects will crash the housing bubble and start a real, full-fledged depression in the US and in the world.

But for now at least I can't go on. I need a break, and you probably do too.

I pray for everyone in the region, and I pray for our country. We badly need to regain our senses and do it quickly.

If you haven't donated yet, now is the time. I recommend giving to the Red Cross first. If you have anything left over, the ASPCA is doing what they can for the animals in the region. Both are fine organizations with a lot of able volunteers who will make your dollar go far.

Hurricane Katrina

2005-08-29T12:28:00.000-04:00

[Updates at the end of this post.]

As I write this at about 1:30 AM eastern time, the outer edges of Hurricane Katrina have just started to touch Louisiana and New Orleans. Nobody yet knows how it will turn out.

Katrina is a full-fledged Category Five hurricane with constant winds of 150-175 miles per hour, and gusting up to 200 mph. If that isn't bad enough, 20 to 25 feet of flooding is expected. That much water will basically destroy all buildings up to the third floor. The NOAA had an extremely sobering description of the anticipated damage up for a while (original seems to be gone, this via Atrios):

MOST OF THE AREA WILL BE UNINHABITABLE FOR WEEKS...PERHAPS LONGER. AT LEAST ONE HALF OF WELL CONSTRUCTED HOMES WILL HAVE ROOF AND WALL FAILURE. ALL GABLED ROOFS WILL FAIL...LEAVING THOSE HOMES SEVERELY DAMAGED OR DESTROYED.
THE MAJORITY OF INDUSTRIAL BUILDINGS WILL BECOME NON FUNCTIONAL. PARTIAL TO COMPLETE WALL AND ROOF FAILURE IS EXPECTED. ALL WOOD FRAMED LOW RISING APARTMENT BUILDINGS WILL BE DESTROYED. CONCRETE BLOCK LOW RISE APARTMENTS WILL SUSTAIN MAJOR DAMAGE...INCLUDING SOME WALL AND ROOF FAILURE.
HIGH RISE OFFICE AND APARTMENT BUILDINGS WILL SWAY DANGEROUSLY...A FEW TO THE POINT OF TOTAL COLLAPSE. ALL WINDOWS WILL BLOW OUT.
AIRBORNE DEBRIS WILL BE WIDESPREAD...AND MAY INCLUDE HEAVY ITEMS SUCH AS HOUSEHOLD APPLIANCES AND EVEN LIGHT VEHICLES. SPORT UTILITY VEHICLES AND LIGHT TRUCKS WILL BE MOVED. THE BLOWN DEBRIS WILL CREATE ADDITIONAL DESTRUCTION. PERSONS...PETS...AND LIVESTOCK EXPOSED TO THE WINDS WILL FACE CERTAIN DEATH IF STRUCK.
POWER OUTAGES WILL LAST FOR WEEKS...AS MOST POWER POLES WILL BE DOWN AND TRANSFORMERS DESTROYED. WATER SHORTAGES WILL MAKE HUMAN SUFFERING INCREDIBLE BY MODERN STANDARDS.
THE VAST MAJORITY OF NATIVE TREES WILL BE SNAPPED OR UPROOTED. ONLY THE HEARTIEST WILL REMAIN STANDING...BUT BE TOTALLY DEFOLIATED. FEW CROPS WILL REMAIN. LIVESTOCK LEFT EXPOSED TO THE WINDS WILL BE KILLED.

Most of the area has been evacuated, but about a hundred thousand people who don't own a car have been relocated to the Superdome and other shelters. It's unknown whether the Superdome can withstand that kind of wind and flooding if it gets hit full on -- there's a very frightening possibility that it might become a death trap.

The brand-new (literally, it's nine days old) the Weather Channel blog has some excellent commentary from the meteorologists themselves, and so far at least I find it a lot more informal and interesting than watching TV. This personal note from Lucas about his father and grandmother reminds us how many, many people are affected by this storm across the country. If they can keep the blog updated through the day tomorrow then it should be a very good source of information. ~~Highly recommended.~~ (Update: Not much going on there now that Katrina has made landfall -- probably too busy today. Weather Underground has some slightly better info at the link below.)

Please, pray for everyone in the area. If you want to take action, please, give money to the Red Cross. A cash donation is infinitely better than a donation of goods -- especially for an emergency like this. They buy in bulk direct from the appropriate distributor (often at cost) and deliver it right where it's needed.

Update: More blogging sources: Mark Kraft is collecting eyewitness reports of people who didn't evacuate. Sounds like some people are going to be blogging til the power goes out. Teece has more, and ends on the disturbing comment "I would have liked to have seen New Orleans." The eye is now visible on long-range radar, at least until the radar is destroyed by the storm. The Weather Underground blogs are also good sources.

Here's that Red Cross link again: Please donate.

Update (11am ET): More blogs describing it as it happens:

Times-Picayune reporters sheltering inside the TP building. Some of the reports are chilling.
Sun Herald reporters.
Metroblogging New Orleans
The Irish Trojan
And there's even a Flickr group with pictures.

Pieces of the Superdome's roof are peeling away~~, but so far it appears to be just the foamy membrane and not the structural steel~~. (Update 11:40am ET: AP is reporting that a few chunks of metal have come off leaving the interior exposed to the sky, but they are relatively small. (20' x 5', which is small compared to the size of the dome.) [12:30pm ET: NOLA now has pictures of the roof]. Originally I heard 100,000 souls there, but now the numbers being reported are more like 9,000 or 10,000 (or 26,000, or ... you name it). Why? Part of that may be because they were running security and bag checks at the door (oh my God, have we progressed to this?) and there was still a huge line outside as the winds started to get really, really bad.

Here's that Red Cross link again: Please donate.

United Federation of Chat

2005-08-24T04:55:00.000-04:00

Google has just come out with Google Talk, their new IM service.

The application part of it is very pretty; looking at the screenshots, it's essentially iChat for Windows. Simple, clean, and easy to use.

In fact, it looks so much like iChat that it probably borders on copyright-infringing, but hey -- that's for Google and Apple to work out between themselves. It's Jabber-based, supports SSL, and seems very nice overall.

But an IM service is an IM service, and we've got too many of them. The real news is buried in the middle of a page called "Additional Resources". That's where Google quietly announces that they are starting an initiative to merge all the separate IM networks into one. Emphasis mine:

What is "service choice" and how does Google Talk enable it?
Service choice is something you have with email and, for the most part, with your regular phone service today. This means that regardless of whom you choose as your email service provider (Gmail, Hotmail, Yahoo! Mail, your school or ISP, etc), you can email anyone who is using another service provider. The same applies to phone service. You can call someone even if they do not use the same phone company as you do. This allows you to choose your service provider based on other more important factors, such as features, quality of service, and price, while still being able to talk to anyone you want.
Unfortunately, the same is not true with most popular IM and VOIP networks today. If the people you want to talk to are all on different IM/VOIP services, you need to sign up for an account on each service and connect to each service to talk to them.
We plan to partner with other willing service providers to enable federation of our services. This means that a user on one service can communicate with users on another service without needing to sign up for, or sign in with, each service.

What the... federation? Whoa, you mean I won't need to have a separate ICQ number and AIM account and Yahoo Messenger account and MSN Chat account to keep in touch with all of my friends and family and work contacts? Why, that ... that would actually be good for the end user. Unpossible!

Crazy as it seems, it looks like Google has a shot at doing it. Rather than standardizing on one network, their description makes me think they are taking a network-to-network bridge approach. So you'd still use your existing account and chat application, and you'd still have all the network-specific features you were used to, but you'd be able to talk to more people. Perfect. I would also expect, given Google's track record, that it would be done 'right': ie, it would be decentralized and all the networks would be peers. Networks that join up would not be beholden to Google in any way.

If that's so, then it seems likely that we'll see consolidation in reverse order of marketshare: all the small players will join up with Google immediately because it's good business sense for them: Their networks suddenly become huge and they are free to differentiate themselves on software alone. Mid-size networks like ICQ will probably follow soon thereafter. Yahoo and MSN might hold out for longer because they would hate to give in to Google, but popular demand from their users will make it necessary.

And then there was one.

The big one, AOL, will probably be harder to win over. I don't have a solid source for current IM marketshare numbers, which is partly because it's a confusing mess since so many people are forced to use multiple IM services. But the data I've been able to find suggest that AOL and AIM have a massive dominance in the IM field, perhaps covering about 50-70% of all IM users. The problem, of course, is that AIM is proprietary and AOL has refused to open it up for a long time now.

I don't know what will happen there. Clearly Google thinks it's worth a shot. And AOL has opened up the AIM network before, for iChat, so it's possible they'll do it again in the name of an open standard. But the company's sluggish history with moving onto the Internet and supporting email and web standards suggests that they will only be dragged into compliance kicking and screaming; as long as there's a buck to be made by being incompatible they would rather be incompatible. It's possible that for quite a while we'll probably be faced with AIM on one side and everyone else on the other. While not perfect, that would at least be better than what we have now.

So what do you think? Are you concerned that Google has got its sticky fingers into too much stuff? Got any links to actual IM marketshare numbers? Let me know.

Apple's future with Intel

2005-07-28T18:00:00.000-04:00

Or, how I learned to stop worrying and love x86

It's been a while since I've posted. Work and family has kept me very busy lately. But since my last post the Mac world has received big news which would be impossible to let go without comment: Apple is switching the Macintosh computer to Intel processors. The keynote video stream where Steve made the announcement is still worth watching if you haven't seen it yet.

Enough time has passed that I'm going to fast-forward through the obvious comments:

Why switch? PowerPC clock speeds were stagnant, and temperatures and power consumption were staying too high. Fixing those problems was possible, but would have required a lot of expensive investment from Apple. And the problems wouldn't stay fixed: Apple would have to keep paying to push the development of the PowerPC, as they've been doing for years now. It finally reached the point where the price just wasn't worth it. Every other reason suggested is secondary.

Did I see it coming? Like a lot of people, I knew how much of the OS ran on x86 to begin with, and was well aware that it was an open possibility. Personally I figured it was inevitable that Mac OS X would run on x86 eventually. It's just happening a little sooner and in a more dramatic fashion than I had thought. I didn't anticipate the abandonment of PowerPC.

The leaks beforehand had all the hallmarks of being authentic and deliberate. As soon as I saw them coming out I felt that they were probably true. The leaks went to real news sources first, not the rumor sites. I believe the first mention came in the Wall Street Journal, and the WSJ's factual reporting is second to none. They would not have printed it if it hadn't been confirmed in some way. (Their opinion page is another matter.) And the Journal's Walt Mossberg seems to be a friend of Steve Jobs; he's been granted special early access to all sorts of Apple technologies and usually gives them glowing reviews. The WSJ is a friendly voice to Apple and they would be an excellent place to leak and start creating a buzz.

Now that that's all out of the way, let's move on. By the way, although I have friends at Apple and I'm probably covered by a ton of NDAs over the years, I want to emphasize that I'm not disclosing any confidential information here. This is all personal speculation based on publicly-available information.

Why Intel?

Why did Apple switch to Intel specifically? And what about the details? A lot of folks have been freaking out and wondering why Apple didn't say anything specifically about x86-64 or AMD.

First up, I'm pretty sure the cost savings to Apple will be more than just the CPU. The talk I've heard says that Intel offers bulk discounts to vendors who switch to Intel across the board, and Apple seems like a likely candidate for such a switch. Intel makes a a lot more than just CPUs, after all -- audio chips, SCCs, ethernet controllers, I/O processors, SATA controllers, and PCI chipsets, just to name a few. Apple already uses a lot of non-CPU Intel chips in Macs, Airport base stations, and iPods. It's certainly feasible that Apple might be going all-Intel-all-the-time.

Nor is it just chips, either. Remember, Intel designs motherboards too. Right now Apple does its board design in-house, which creates a significant lead time for product development as new boards are tested and reworked. Discounted Intel boards -- perhaps fully tested and qualified before they even reach Infinite Loop -- could be yet another potential cost savings. Apple has a penchant for pushing design boundaries with iMacs and laptops, true, but the company still does a brisk business selling desktop machines where space is not at a premium. Besides, Intel has been stuffing Pentiums into tiny spaces lately -- witness their recent clone of the Mac mini.

Could AMD compete with all that Intel has to offer? Frankly, it seems unlikely.

The door will still be open for AMD and other x86 chip vendors. I don't think Apple is going to start using Intel-specific features that AMD can't compete with -- that would be a foolish return to the single-chip-source problems that plagued Apple with the PowerPC. It's likely that the companies have made a deal where Apple commits to Intel for some number of years on negotiated terms, and thereafter the door is open to renegotiate or seek a better deal. So AMD might still get their foot in the door eventually. If Intel starts causing problems you bet Apple will switch to AMD or someone else; but as long as they offer a good deal they will be difficult to beat.

64-bitness

What about 64-bit support? Some folks were upset because Apple had done all this work toward 64-bit PowerPC and seems to now be ditching it. That work absolutely is not wasted. Anyone who thinks Apple is suddenly ditching 64-bit computing should put down the crack pipe -- it's not gonna happen. Sure, after Intel came up with the all-but-failed and incompatible IA-64/Itanium architecture, AMD turned around and created a much more popular backward-compatible architecture called x86-64. Point to AMD. But Intel at least recognized its mistake and cloned x86-64 for use in recent chips.

Apple may not make the jump to 64-bit Intel chips in its very first release, though I think it's at least possible that they might. But if they start with 32-bit chips, I will guarantee that they'll be on 64-bit chips within the year. And further, that they'll be going with the x86-64 architecture rather than the Itanium. It's the only decision that makes sense.

Should you care?

My personal opinion is that as a consumer you probably won't need to care about the transition.

Think about it: why do you buy a Macintosh? Do you really care about the chip that's inside it? No. What you want from a Mac is that you want the nice user experience, you want your apps to work, and you don't want the machine to stink up your desktop and crash and be virus-ridden like Windows.

Apple's switch to x86 will not change anything about the operating system or the applications: it will look and work exactly the same as your current Mac. It will still be just as stable and easy-to-use, all of your software will still run, all of your data will copy over and work just fine, and no viruses are going to magically jump over from Windows onto your Mac just because the hardware is the same.

Should you delay hardware purchases while you wait for the new machines? Well, you could. Apple seems to be expecting a certain number of consumers to make that choice -- they have more cash on hand than ever before ($7.5 billion USD), which it's pretty clear is a buffer against an anticipated temporary decline in Mac sales.

But the new machines won't be out for a couple of years. The first new ones might not be all that great, either; you may want to wait a few months for the second generation. Personally I'm planning on following my normal upgrade schedule: I'll keep my dual 2GHz G5 til it's on its last legs or until I get new free hardware, whichever comes first. The typical Mac lifetime is around four years (as opposed to two for PCs), and that's lengthened a bit by the way PowerPC clock speeds have been lagging, so I probably won't need to upgrade until the new machines are out anyway. As for my laptop, a late-model TiBook, it's getting pretty elderly. I wouldn't mind getting a new PowerPC laptop right now if I could afford it -- but so far furniture and family stuff has delayed me and I don't really need a new one badly enough yet.

So for now I think it's not a big deal: buy a new Mac if you want one and don't worry about it. As we get closer to the release of the first Intel Macs, though, the dropoff in Mac sales will be steeper. People will naturally wait for the new machines.

It'll be interesting to see what Jobs does with the release of the new machines: Will the release date be known or widely anticipated, for example Macworld SF 2007? Or will they get sprung on everyone unexpectedly? Several years of watching how Steve Jobs does things leads me to suspect the latter -- they will probably release the machines at least four months earlier than anyone expects. But it's an open question.

Compatibility and speed

Apple is including a way to run PowerPC apps which it is calling Rosetta. As reported by C|Net and Wired, this is Transitive's multipurpose DR emulator.

In an ironic twist, I've heard that Microsoft is also using Transitive's emulator ... but to translate x86 to PowerPC so that old games can run on the next-gen Xbox. Man. What's next -- cats and dogs living together?

I've tried out the emulator on a development system. Rosetta delivers great performance for simple apps. For a geek like me it's really cool: PowerPC apps just work transparently, and they run fast too!

However, anything that is heavily Altivec-enhanced will probably take a big hit when running under the emulator compared to running on a G5. The emulator does not emulate Altivec, so the app will first fall back to unvectorized floating-point, which could drop its speed to 25% of the vectorized floating-point. Then it would incur the cost of emulation on top of that. So some specialized operations running through the emulator might in theory be as much as 5x slower than on a comparable G5 system. As soon as the app goes native, however, it will gain all of its performance back and more.

But that only affects heavily vectorized applications. Your word processor will continue to work fine under emulation and will in fact probably be faster than before: the extra CPU speed will give it a boost. Games are mostly pumping video textures out using OpenGL, so they will probably not be affected.

Probably the most visible area where you'll see applications slow down is anything that uses QuickTime to compress or decompress video. That includes video playback; frame rates may drop significantly under some codecs. Why? Remember, Rosetta is not a mixed-mode architecture; it runs an entire app from top to bottom. And QuickTime loads and runs inside your application. So even though a native version of QuickTime will be available to native apps, an emulated app will run emulated QuickTime which will be much slower. If I've got all that right, then you can bet Apple is making an extra push to evangelize QuickTime developers to port as quickly as possible.

G4 emulation: Someone asked me the other day whether Apple might upgrade the emulator, which currently emulates a PowerPC G3, to emulate a G4 with Altivec. Frankly I doubt that will happen. It would take a big investment of time and money. And the benefit Apple would get from doing that is only in the short term, during the transition period where you want all the extra speed possible out of the emulator. After a while CPU speeds will increase so much that it just won't matter. The ROI (return on investment) just isn't there; too much work and too little benefit.

Classic

Apple has not announced plans for making the Classic environment run on the Intel machines. As a practical matter the current release of x86 Tiger does not support Classic. But have you noticed that they have not vocally announced that Classic is dead, either?

My guess is that Apple may be working on porting Classic, unannounced. There's a not-insignificant minority of Mac software that still needs it. And Apple has a long history of unbroken compatibility which it would be a shame to end.

In a lot of ways Classic is just a normal application: a lot of hardware-specific details were abstracted out of OS9 during its final years. The emulator doesn't need to be ported, after all: there's no reason why you couldn't run Classic's 68K-to-PPC emulator inside the new PPC-to-x86 emulator.

Of course, in other ways Classic is still very 'special'; there are many hacks all over xnu for it that give it special access to supervisor-level PowerPC stuff which Rosetta isn't going to emulate. Still, it seems like it'd be well worth Apple's time to sic a couple of guys on it for a year or two -- the potential gain is huge.

I suspect there are probably some doubts about whether it's technically possible. That's why there hasn't been a big announcement one way or another. But I'll go out on a limb and say there's a good chance someone will figure out a way to make it work.

Windows emulation

Here's an interesting one. Running x86 will make it a lot easier to run Windows applications on your Mac. Things like Virtual PC will run just about at full speed. Of course, Microsoft might not be very interested in porting Virtual PC to the new architecture, since Mac OS X just got a lot closer to becoming a competitor to Windows. if past history is any indication they'll probably do it eventually, but I bet they will drag their heels.

Other options exist, of course, such as VMware and WINE. It seems very likely that VMware is working on porting their software to x86 Tiger right now, and DarWINE is already underway.

Personally I think this would be an interesting thing for Apple to investigate. They might choose to be hands-off and just bundle VMware, Virtual PC, or DarWINE, but all of these are a little clunkier than what Apple prefers. It's possible that Apple might choose to enter the market themselves and deliver integrated Win32 emulation in some way.

Dual-booting into Windows: This sort of falls under this category. As others have noted, in addition to virtual PCs, it will be possible to dual-boot or triple-boot your Mac. Darwin can easily support the partition map styles used on Windows, so there's no reason you couldn't have a Linux partition, a Windows partition, and a Mac OS X partition. Mac OS X's built-in BSD and X11 are so good, however, that there's probably not a lot of reason for anyone who's not a Linux developer to dual-boot into Linux.

HP, Dell, and more

What about all the talk about whether Apple could or should release Mac OS X in a general release to other PC makers, like HP and Dell? Some people think that would've been a better idea. This attitude is exemplified by the somewhat goofy article Apple's Colossal Disappointment which was posted to Slashdot recently.

I call the article goofy because the author pays no attention whatsoever to business realities, nor does he seem to grasp the concept that policies may change over time. His basic complaint is that Apple is limiting its OS to run on its own hardware for the time being, and he thinks that's a mistake. (Or a "colossal disappointment", to use his exaggerated phrase.)

Here's my take. There were three possible ways Apple could have used its x86 code.

Switch the Mac hardware to x86
Allow a select few PC vendors to ship hardware that runs Mac OS X.
Release Mac OS X widely for anyone and everyone with a PC, the way Microsoft releases Windows.

Apple started with item one for the reasons mentioned above -- the PowerPC has been weighed in the balance and found wanting. But the second two courses of action are not immediately feasible, nor are they wise business decisions.

Sure, they are potentially desirable goals that are at least on Apple's radar. I guarantee you Steve Jobs is thinking about them; when interviewed by Fortune magazine he mentioned that three of the biggest PC makers have repeatedly asked him to license Mac OS X for them to bundle. (My guess: the big three are HP/Compaq, Dell, and Sony.)

But it makes absolutely no sense to do all of those things at once.

What do you think would happen if Apple switched to x86 and immediately let others release x86 machines with Mac OS X too? It would undercut Mac hardware sales in a big way. Apple has been burned by cloning before in the 1990's. The circumstances are different enough now that they may try it again, but they will definitely start out slowly and carefully. It will be several years before you see anyone but Apple shipping Mac OS X.

How about making a widespread general release? First of all, you have the same problem as above where Mac hardware sales are undercut. But okay, perhaps you might make that up with increased software sales. No big whoop. But the real cost comes in the support infrastructure. Mac hardware is fairly dependable and predictable, and Apple does a good job with the small amount of tech support that is needed. But making a widespread release would suddenly skyrocket the number of support calls being made -- people would buy it for their JalopyTech 3000 computer and something would break. Apple gets a call. Frankly, Apple doesn't have the support infrastructure to handle a twenty-fold increase in phone calls. It might be possible to get there in the long run, but it won't happen overnight.

The correct course of action from a business perspective is to do exactly what Apple is doing -- switch their own machines only at first. The other things will still remain possible future directions, and I think that both are likely in the long run. But Apple is an "old" company by tech industry standards, and doesn't plan on biting off more than it can chew at any one time.

The Lockdown

Will Apple use LaGrande as part of a scheme to make sure Mac OS X is running on Mac hardware? Maybe. But it seems very likely to me that Apple won't try TOO hard. They want to discourage the casual user from running an unsupported configuration, and they will. But it's not worth the effort (and would frankly be counterproductive) to lock out the hardcore geeks.

A small amount of 'geek piracy' will cost Apple practically nothing in hardware sales. Contrary to their huge presence on the internet, the actual real-world market share of geeks is fairly small. The vast majority of Apple's and Microsoft's markets are people who wouldn't know how to perform that sort of hack, and wouldn't want to, because they know they wouldn't get support for it. Overall it's far simpler to just buy a Mac in the first place.

Honestly, it's like free advertising... let the geeks hack OSX to run on their machines. Once most geeks start using OSX their prejudices evaporate and they quickly get hooked on it, talk to their friends about it, recommend it, and so on. And that will ultimately grow the Mac's marketshare.

A brief history of purity at Apple

2005-05-13T23:24:00.000-04:00

I was reading Ben Goodger's take on the recent Safari-KHTML kerfuffle. He's basically right. What I find particularly amusing is that ten years ago Apple would have been on the opposite side of the debate... with Apple fighting for purity and the open-source guys just putting in a quick hack to make it work.

It took Apple quite a while to get over its obsessions with purity in software design. Don't get me wrong; purity isn't a bad thing at all. In fact it's an important goal to strive for. But you have to make compromises to ship a useful product sometimes.

Ten years ago, more or less, Mac programmers like me were trying to deal with lovely gems that came out of Apple like these:

Apple Events -- in theory a great concept which used an incredibly flexible and extensible tagged-data interface. Problem was that its API and implementation were so annoyingly pure that they required allocation of a lot of temporary objects and a lot of memory got copied around needlessly. In the end it was much slower than any contemporary RPC/IPC call you could imagine.

(Today: Apple Events have been kept alive, although with the implementation optimized and the API expanded to be faster and less pure. Several even-less-elegant but faster-and-easier methods of RPC/IPC became available with OSX and were quickly adopted by many grateful programmers.)
AppleScript -- in theory another great concept, designed around an abstracted open scripting architecture which could support multiple scripting dialects, and which allowed those flexible and extensible bits of tagged data from the Apple Event manager to be passed around. But only one dialect was ever completed, and the architecture wound up making it painful to implement from the developer's point of view. Heck, even the one-and-only dialect was fairly clunky from the user's point of view. This might make you wonder whom exactly it was designed for.

(Today: AppleScript has stayed alive too, although the way developers and users actually use it in practice frequently disobeys the pure object-verb syntax of the original design. And what AppleScript support is out there generally comes for free from AppKit these days. Do you know anyone who's actually written code to properly parse a 'whose' clause lately?)
AOCE, or Apple Open Collaboration Environment -- in theory a great concept, trying to create a system that integrated mail, address books, digital signatures, networking, and more before all of these things were widespread. Its problem was that rather than attacking each problem individually, it tried to do them all at once. And it had a nicely object oriented design, but rather than allowing direct access to any of the dozens of classes or hundreds of members, it created accessors for every single operation. Ultimately it collapsed under its own weight. It was monolithic, huge, and incredibly difficult to understand. One of the Inside Macintosh: AOCE books was almost 1400 pages long -- big, 8.5" x 11" pages too. My phone book was smaller. And that book was only one of at least three...

(Today: Deceased, just like all those trees.)
Open Transport -- a networking stack and API that had a very theoretically pure design and was supposed to, in theory and if you used it right, deliver rockin' performance. It did deliver better performance than the older MacTCP, but it was generally much too obnoxious for most developers to use directly. The most popular way to use it was via a wrapper library such as GUSI that made it more socket-like.

(Today: Deceased. On OSX, Apple re-implemented the APIs with glue that calls through to Unix sockets.)
Newton -- another very pure concept. A tablet that you can just write on, and it will just recognize your handwriting! Cool! Electronic book! Electronic paper! The problem is that the technology and/or design clearly wasn't quite up to the task. It was big, clunky, expensive, and slow. The handwriting recognition technology was good in theory but notoriously imperfect in practice, but Apple stuck with it anyway.

(Today: Deceased. A few years later, when Palm did basically the same product, they put in a hack called Graffiti so that they didn't have to solve the problem of generalized handwriting recognition.)
OpenDoc -- a very nice idea that was (and probably still is) ahead of its time. Rather than documents being things created by applications, they were just collections of objects which were essentially peers and had defined relationships to each other. If anything, OpenDoc was less a victim of its design and probably a victim of its circumstances: it was C++ and required COM, which was burdensome in terms of both performance and licensing, and it had to run on a rather lackluster and nonstandard OS which limited its portability.

(Today: Deceased. But interestingly, Apple is just now starting to regain OpenDoc-like functionality with things like KVC, bindings, and CoreData. Thirteen years later.)

I think you can see what I'm getting at here. Apple being chastised for putting in quick hacks to make things work, rather than going for purity? By a bunch of Linux coders? Ah, sweet sweet irony.

Branching and Integration

I definitely feel the pain of the KHTML guys. But in the end what happened was inevitable.

See, one way or another Apple wound up on a branch. I don't know whether Apple chose to deliberately make incompatible changes, or if KHTML refused to accept some of their changes and forced them out on a branch. But regardless of your interpretation that's the way it ended up. And that was the first step towards KHTML's problems.

There should be (but probably isn't) a lesson in Software Engineering 101 about what happens when you have work proceeding on two different branches of a source tree. Integration -- the unpopular gruntwork everyone loves to hate -- starts to rear its ugly head. As long as the teams devote roughly the same amount of time to developing their branches, the branches grow in parallel and each spends a roughly equal amount of time integrating changes back and forth.

But what happens when one of those branches suddenly has a great deal of work invested in it, and the other doesn't? The team maintaining the less-vigorous branch starts spending more and more of their time on integration and less on development. Integration sucks; it's a necessary evil, but nobody likes doing it. Quickly it becomes less like fun and more like work. So the less-vigorous branch is in danger of withering even further.

It's made exponentially worse if the less-vigorous branch ever refuses some of the changes in the more-vigorous branch, because that causes the source bases to diverge even further. Now not only is there more integration work, but it's harder too. If the team is made up of part-time volunteers it can kill their enthusiasm for the project completely.

In a nutshell, that's pretty much what is happening (or has happened) with Apple's full-time engineering team and KHTML's part-time engineering team. Apple exacerbated the problem with what some are calling "code bombs", ie releases of an entire tree at once, but you're fooling yourself if you think the same problem would not exist regardless. The problem is really in the quantity of changes being made. Is it Apple's obligation to go back and do all the work to make their changes work on Konqueror? Not at all.

It kinda sucks for the KHTML guys, but the solution is clear even if they don't want to admit it: transition the upper levels over from the less-vigorous branch over to the more-vigorous branch. Rather than always integrating lots of changes from B back into A, start using B and port a few changes from A forward into B. It sounds like this is what Maciej suggested.

Happily, it sounds like they are pursuing something like this even as we speak.

Hey check me out -- I'm Dancin'! I'm Dancin'!

2005-04-10T15:35:00.000-04:00

Hitachi recently announced perpendicular recording technology, which is meant to improve the density at which bits can be put down on hard disks.

Storage density is exactly that -- the density at which information can be packed onto a storage medium like a hard disk. As density increases, that means that you can fit more data into the same size space, or fit the same amount of data into a smaller space, or both.

Over the years storage density has been increasing steadily. This has led to larger-capacity drives in physically smaller containers. Twenty years ago hard disks were clunkier and held a lot less data. 20MiB hard disks could be purchased but were expensive and rare. Today if you purchase a computer you'll probably get a 50GiB or larger hard disk, or over 2500 times the capacity, and it will take up a much smaller space. That's why the tiny little iPod mini can hold 6GiB of data.

But this rapidly increasing density has led to a problem. Magnetic storage technology is nearing its limits; today we are making bits so small and packing them so closely together that if we go any smaller, bits are in danger of flipping spontaneously from external factors. Since the bits are where your data is stored, you can't have them flipping arbitrarily -- because then you can't read what you just wrote. It'd be like trying to write on an etch-a-sketch that someone is shaking.

The arbitrary point at which this starts to happen and data is lost is called the superparamagnetic limit. Storage technology is getting very close to the superparamagnetic limit. Since bits need to be kept large enough that this doesn't happen, that in turn means that magnetic hard drives are close to their theoretical maximum density.

Hitachi has come up with a dodge that avoids the superparamagnetic limit for a little while longer. Rather than arranging data bits so that their magnetic poles aligned linearly along the surface of the disk pointing at each other (longitudinal recording), Hitachi is arranging them so that their magnetic poles are aligned parallel to each other and pointing away from the surface of the disk (perpendicular recording), with corresponding innovations in how the bits are read and written. This vertical alignment makes them less vulnerable to flipping and again increases the possible storage density. The result is that magnetic bits can be packed several times more densely than before, which means that drives will continue to be able to be made even smaller and higher-capacity than before. For example, rather than 6GiB iPod minis this will eventually make it possible to have 60GiB iPod minis.

It's all very serious and interesting and geeky.

So why exactly did they chose to promote this technology with a crazy Schoolhouse-Rock-style video including "Actuator Man" and little rectangular bits disco dancing?

Good question. Don't get me wrong, I think the video's great... I'm just a little baffled. Then again, it fits right in with Sanyo's HD-BURN promotion. Mr. CD-R, doubling of charming points!

Sometimes this is a bizarre industry I work in. But I'm glad people have a sense of humor about it.

Google Satellite Maps

2005-04-05T01:05:00.000-04:00

Google Maps just added an option for satellite photos. And I have to say ... wow. That is just wicked cool.

Not every location is covered by a high-resolution satellite photo, but a lot of places are. In particular, my town of Brecksville is available at the highest zoom level. I can see my house!

Play around with it for a while. Zoom in and out. It's incredibly fun. Here are a few neat locations to start with:

Hornblower's Barge and Grill in Cleveland. Zoom all the way in, and it's due north of the marker, in the water by the airport. The long, skinny ship is the U.S.S. Cod, a WWII submarine. Just southwest of that you can also see the Rock and Roll Hall of Fame (the odd white building with the triangular glass window and bullseye in front) and Browns Stadium. Looks like a building obscures the view of the Free Stamp, which is just south of there.
Apple's campus
Microsoft's campus
Golden Gate Bridge
The Hollywood Sign
Central Park in New York City
The White House. Notice how the roof is blanked out and many buildings nearby are blurred; this censoring has been standard practice for satellite images released in the US for years. Also, you can scroll almost due southwest across the river and you'll see the Pentagon.
Diamond Head State Monument in Honolulu
Mount St Helens
Michael Jackson's Neverland Valley Ranch
Las Vegas Boulevard near the Luxor. The Bellagio's fountains are clearly visible to the north.
The Boneyard at Davis-Monthan Air Force Base, full of decommissioned planes. Zoom in.

I'm incredibly stoked to see this feature in Google Maps. Yes, there are other sources for satellite and aerial imagery... but up until now it's either been slow and clunky, or a pay service. Integrating it with Google's terrific map interface makes all of that data extremely accessible for the first time.

Got any neat satellite image finds of your own? Let me know in the comments!

Nixon Approval Ratings

2005-04-04T13:34:00.000-04:00

From time to time as I look at current Presidential approval ratings, I find myself wanting to look at the week-by-week job approval ratings of former President Nixon. They are out there and available, of course, but they're surprisingly hard to find.

The best source I've found so far is the Roper Center at the University of Connecticut. They have a page with the data, but their graph quite frankly stinks. The value axis is compressed, there aren't any gridlines, and it just doesn't tell you much. So I compiled their data and made my own graphs, which I'll share with you now.

First, some background and the raw data:

The Washington Post's Watergate timeline
Detailed chronologies of Watergate events in 1972, 1973, and 1974.
Raw data on Nixon approval polls as a tab-delimited text file.
Raw data compiled into an Excel file with graphs.

And now the charts:

Stacked charts showing the poll results (click to zoom):
Approval only for Nixon's entire presidency (click to zoom):
Approval only for Nixon's second term (click to zoom):
Net approval rating (approve minus disapprove) for Nixon's entire presidency (click to zoom):
Net approval rating (approve minus disapprove) for Nixon's entire presidency, annotated (click to zoom):
Net approval rating (approve minus disapprove) for Nixon's second term (click to zoom):
Net approval rating (approve minus disapprove) for Nixon's second term, annotated (click to zoom):

I think there are a few very interesting points that are visible in the raw data.

First of all, Nixon started with a substantial number of "no opinion" ratings -- as high as 35%. But the ambivalence quickly disappeared. There was substantial movement from "no opinion" to "disapprove" during the first few months of his Presidency. I haven't done a thorough analysis, but from the few others I've checked it looks like other presidents show the same trend in their approval. This doesn't seem to be specific to Nixon.

Second, Nixon was much more popular than Bush throughout most of his Presidency. President Bush's approval shot up after 9/11, but has steadily eroded ever since. Bush spent most of 2004 hovering under 50% approval and with a net approval of less than zero. Nixon didn't get that low until his former counsel was testifying before the special Senate Watergate panel describing the political espionage that he'd personally taken part in.

Third, to me it looks like Nixon was seen as "above the fray" at the start of Watergate. Everything was somebody else's fault and he didn't know about any of it. His approval rating was at an all-time high as the Watergate burglars were convicted, despite the fact that it was a fairly high profile case and some of them were former Nixon aides. It wasn't until one of the convicted former aides, James McCord, started to make allegations of obstruction of justice and point fingers higher up that things started to take a turn for the worse.

Fourth, July 1973 was an incredibly bad month for Nixon. His net approval rating dropped by a whopping 25%. At the end of June, his former counsel testified that he was involved personally in the Watergate break-in and was involved in the obstruction of justice. Then in July he refused to testify before the Watergate committee, citing executive privilege; the existence of the White House tapes was revealed; and he refused to hand over the tapes. All this contributed to a serious drop in public approval from which he never recovered.

Fifth, even as the worst came out there was still a solid core of about 25% of the country who never abandoned Nixon and continued to approve of him.

All in all, a very interesting set of data indeed.

[Update: December 1st, 2005] Probably because of the plight of our current president, a whole lot of people have been coming here to look at the gory details of Nixon's approval ratings. I should point out that Professor Pollkatz has a pretty nice chart up comparing raw numbers from Bush, Nixon, and Clinton. And he updates his Bush numbers regularly so you can keep track of the progress. Go check it out!

And keep in mind that despite all the recent indictments, President Bush himself has not really had a decisive Watergate moment yet, and probably won't while the Republicans control Congress.

Songfight Mix

2005-03-09T19:15:00.000-05:00

I ran across Songfight the other day via Boing Boing. It's an online songwriting contest; several themes are posted and you have one week to write, perform, and upload a song for your chosen theme.

Some of the submissions are great. (Some aren't, or at least aren't to my taste.) I spent an hour or so sifting through the archives and picked out some favorites and made a mix. It's just a quick glance at what they have to offer, and there are probably more great tracks that I missed, but check it out. All MP3s are free downloads.

* Bite Size - Mellified Man
* Indier Than Thou - MC Frontalot
Shut Up And Sit Down - deshead
Sarah Vowell's Motto - Figure 8
Five Minutes - Josh Woodward
Shower Scene - Josh Woodward
Left/Right - Hobbes
I'm With Steve Now - Gyre (featuring Emma Lee)
We Are All Famous - Ants (Invisible)
The Truth About Aspartame - Consume
* Piece of My Heat - The Def Author
Massive Intelligence Failure - Southwest Statistic
The Truth About Aspartame - The BeWells
* Hollywood Fantasy - Loyalty Day
* Bonjour, Mon Amie - Josh Woodward
* Things To Do In New York - Fifteen Years
As Icons of My Youth Fall Silent - Forty Second Songs
Quentin Tarantino - Doctor Worm
I Thought I Loved You - Sven Mullet

The asterisks mark songs that are my particular favorites, but they're all pretty interesting. Track #2 is not actually from Songfight, but MC Frontalot has participated in the past and I thought it went well in the mix. Sue me.

Notice anything? Josh Woodward made the list three times. I was curious, so I checked him out. Turns out he's got a blog and he's local to me, relatively speaking. Findlay's about two hours away. I don't know if we'll make it over to see him at a live show, but perhaps we'll take a detour on one of our trips up to Michigan if the timing is right. He seems like an interesting guy... not many songwriters use grep to write songs.

Great PS2 Games for Young Kids, Part One

2005-03-03T21:58:00.000-05:00

More than 80 million homes have a Playstation 2, and ours is one of them. I originally bought it several years ago to play a few games for myself. Then my stepdaughter came into my life, and I started looking at the console in an entirely new way.

The PS2 is not really a gaming system for young children. Go to any store and look around; the vast majority of the games created for it are really targeted much more at teenagers and twenty-somethings. A lot of games either have very mature subject matter, or are very complicated and require a lot of skill and coordination and concentration. When Olivia started playing games on my PS2 at the age of 6, I had two thoughts. First of all, I firmly believe that video games are good for kids so I was very happy that she was excited about playing these games... but in the next moment I very quickly realized that most of the games I had were completely wrong for her.

Even after scratching the obvious ones like Grand Theft Auto, Devil May Cry, and Evil Dead, almost all of the games I had were unplayable. Final Fantasy X? Dark Cloud? No way; far too complicated. In fact, RPGs in general were just too involved. Ratchet & Clank? It's actually pretty hard for a 6-year-old. Jak and Daxter? Better, but it was really easy for her to get lost in the map and not have any idea what to do next. Even the simple-seeming cartoony games wound up requiring a lot of visualization, control skills, and just plain gaming experience that she didn't quite have.

To use a technical term, she was brand-new to the world of video games, and most PS2 games were either too challenging in general, or lacked sufficient scaffolding to teach her the skills she needed to play them. What's that, you ask?

Scaffolding is a term from education which describes the process of helping a student (or in this case, gamer) learn by providing supports which are later removed as experience is gained. Walkers for babies and training wheels on a bike are examples of simple scaffolding.

Video games will often try to provide scaffolding in the form of an in-game tutorial. The crucially important part, however, is that whatever scaffolding is present usually assumes a basic minimum skill level. In today's games that contain immersive 3D landscapes, for example, that means that the game designers often assume you can figure out where inside that world you are and where to go next. Easy enough for me, but it was a brand-new challenge for Olivia. Another skill that is often assumed is understanding the in-game physics well enough to be able to jump and land on a target platform. Again, not so easy for a kid just exploring video games for the first time. Even the "save game" interface that so many games have is tricky; kids forget to save or accidentally save over old games. Autosave is much easier. When too many of these difficulties add up, the kid spends more time frustrated than engaged, so as I was looking for a game for a six-year-old to play I wanted scaffolding that started at the ground up, moved forward slowly, and really kept the game from getting too hard.

So I started doing some research. That took some time, because there really aren't any good sites that look at games specifically from the perspective of kids and parents. The big game mega-sites like GameSpot and IGN tend to be focused on tweens, teens and twenty-somethings. A game might be great for young kids, but these sites may score it lower than it deserves because it's too simple for their target market. The reverse happens too: a game might score high because it's great fun for an adult, but it's just too darn hard for a younger child.

But finally, after a few false starts and a lot of trips to my local EB Games, we've found some games that Olivia loves. I can't possibly fit them all into one blog post, so this is just the first installment in a series.

Sly Cooper and the Thievius Raccoonus

One of Olivia's early favorites. Sly Cooper is a sly, charming, and honorable thief. Think about a dashing movie Robin Hood reincarnated as a raccoon, and you're in the right ballpark. He's trying to recover the book that is his family's legacy, the Thievius Raccoonus, which was stolen by another not-so-honorable gang of thieves called the Klaww Gang. As Sly proceeds through the game and recovers pages from the book, he learns cool new moves.

The characters, voices, and storytelling are just fantastic. The cel-shaded graphics are smooth and very cartoony, a familiar look to kids. The levels aren't overly complicated; in fact, despite being three-dimensional, most levels progress very linearly so it's hard to get lost. Sly's pal Bentley pops up several times in each level and tells you what you need to do -- audibly, not just in on-screen text which kids are inclined to skip over. There is violence, but it's fairly cartoonish; for example, Sly whacks bad guys with his cane and has to avoid barking dogs on leashes. Enemies only take one whack to defeat, and have simple patterns. A few guns appear but they are also big and cartoonish, and Sly himself only fights with his cane. The game's focus is really not on fighting. If anything, the best way to get through a level is to avoid getting into fights. The challenges in the game also work very smoothly and are very forgiving.

The controls are both simple and consistent, and the game is much better because of that. Sly has many special moves that look great on screen: he can grab ropes, land on tiny points, use his cane to swing from hooks, and more. And most of these feats are triggered by a single button. The game uses context clues to "just do the right thing" when you press it, and it's very forgiving about how close you are to the target. If you're anywhere near a rope, the button makes Sly grab it. If you're jumping near a spiky point, the button makes Sly land on it. And so on. This is fun for adult gamers, but it's particularly great for kids; it really makes you feel like you're doing these terrific-looking stunts when really all you've done is press a button.

Because the whole game is so consistent and provides a lot of direction, it was easy for her to understand. Of course, it still took a fair bit of help from dad to get her through some of the levels the first time through. A lot of the levels look hard but really aren't, and she had to get over being freaked out by those. Others were legitimately difficult and just needed practice; I got her past those the first several times, but she's since gotten good enough to handle them on her own.

In the end, one of the best signs of a great game is when a kid goes back and plays it again and again and again... and that's what Olivia has done with Sly Cooper. She started playing it at age 6, and will still go back to it almost two years later. Fantastic. Well worth the money.

There's a sequel out now, Sly 2: Band of Thieves. It's a very different game from the first. Olivia has played it and likes it well enough, but it's more difficult and is skewed a bit more toward the tween/teen age spectrum.

Katamari Damacy

Not only do Olivia and I like this game, but her mom does too! That's rare; my wife usually doesn't find video games interesting at all. But this game's weirdness and simplicity attracted even her.

The stars have fallen from the sky, and you play a tiny little guy, the Prince, whose father is the King of All Cosmos. He orders you in his own bizarre quirky way to roll up ordinary objects into an ever-growing ball (called a katamari) until the ball reaches the size which has been set as the goal for each level. You start out rolling a 10cm ball, rolling up thumbtacks and coins and caramels... and as you grow you are soon rolling up dogs, cats, and people... then cars and buildings... then trees, boats, and giant squid... and finally skyscrapers, airports, and entire islands. The thrill this provides is difficult to explain until you've tried it. There's literally nothing you can't roll up into your ball once it's big enough.

The music is beautifully composed; like the rest of the game, it's simultaneously weird and wonderful. The controls are simple and accessible to all ages. The two joysticks are used together to roll your ball, and that's really all you need to know. There is no violence to speak of. Okay, so some levels are filled with people screaming and running away as your giant katamari swallows up their town, but nobody is killed and it's very absurdly unrealistic. The net effect is much more comical and silly than scary. Some of the levels can be challenging, but this is made up for by the fact that every single level has replay value. If you can't beat level 4 within the time limit, you can go back and play levels 1 through 3 over and over again and still be just as entertained.

Highly recommended for all ages. The only danger is that the adults will want to spend more time playing it than the kids!

EyeToy: Play

The original EyeToy games are a lot of fun for everyone. Olivia and I like EyeToy, it's fun at sleepovers, and it's another game that has the very rare Mom Seal of Approval too!

EyeToy is a small camera that you attach to your PS2's USB port. The game that comes with the EyeToy is called EyeToy: Play. Control is handled entirely through the camera -- you don't use the normal controller at all. The camera displays an image as if the TV were a mirror, and it's overlaid by graphics. Waving your hand over a button will select it. There are a bunch of different mini-games to choose from that use the camera in different ways: Kung Fu, window washing, dancing, soccer, and more. One game just uses crazy camera effects to distort the image in various ways, and kids love it.

Using the camera as the controller is unusual but surprisingly simple and intuitive. It can be a little tricky to set the camera up initially -- mom and dad need to help with that. You have to draw the blinds, and and then get plenty of artificial light in the room. But once it's set up it's fun for the whole family. And it's also the sort of game you can just leave a bunch of kids alone with. Sharing is easy and sometimes not even necessary, like in the crazy-camera-effects section. There's a tiny bit of violence in some of the games, like Kung Fu, but it's so cartoonish -- your onscreen image is a giant, and little tiny guys jump at you and you get to smack them -- that it winds up being absurd and comical.

Highly recommended for all ages. Kids as young as a few years old should be able to understand this and interact with it. And it's still fun as a party game even after the novelty wears off. Well worth the money.

More to Come

That's all for now, but I'm not done. Keep your eyes out for the next installment!

[Update: Great PS2 Games for Young Kids, Part Two is now available.]

Weapons of Social Destruction

2005-02-05T16:55:00.000-05:00

A few days ago George W. Bush got up in front of Congress and gave his State of the Union address for 2005.

He started off by reminding us how wonderfully the economy has been performing. I think he's got a point there. It's certainly true that I've never felt quite the same way about my job security since he was elected.

Without a doubt, the keystone of his speech came when he warned us that Social Security has weapons of mass destruction. Although a shocking statement, this was not a surprise to anyone who's been paying attention. The administration has been laying the groundwork for months now.

Remember when Secretary of Defense Rumsfeld said that no federal program "poses a greater or more immediate threat to the security of our people and the stability of the world" than Social Security? Vice-President Cheney also stated forcefully that Social Security "is busy enhancing its capabilities in the field of chemical and biological agents," and continues to pursue "an aggressive nuclear weapons program."

In fact, President Bush himself said in a recent interview that "There is a real threat, in my judgment, a real and dangerous threat to America" in the form of Social Security.

Before he resigned, Secretary of State Colin Powell even appeared before the UN, held up a vial of anthrax, and showed some fuzzy photographs of the areas where he claimed Social Security was hiding its weapons systems. "How do I know that? How can I say that? Let me give you a closer look. Look at the image on the left. On the left is a close-up of one of the four chemical bunkers. The two arrows indicate the presence of sure signs that the bunkers are storing chemical munitions."

Wow, that sounds frightening! Shouldn't Americans be concerned about this? I know what many of you are thinking: Break out the duct tape and plastic sheeting! But not so fast.

Remember just a few years ago, when the very same administration told us that it was critical that we invade Iraq, because the country was about to go bankrupt? We all know how well that turned out.

During the lead-up to the war, Vice-President Cheney assured us that there was no recourse other than invasion. "With an aging population, and a steadily falling ratio of workers to retirees," Iraq "is on a course to eventual bankruptcy." Treasury Secretary John Snow claimed that the country was "in jeopardy" and "it's a pretty serious situation".

Bush himself said in a previous State of the Union address that "By the year 2042," Iraq would "be exhausted and bankrupt. If steps are not taken to avert that outcome, the only solutions would be drastically higher taxes."

Sound familiar? In the end it turned out that Iraq wasn't going to run out of money after all. Over a thousand American troops gave their lives, more than a hundred thousand Iraqis were killed, and countless more on both sides wounded severely -- all in the name of restoring solvency to Iraq. Three years and hundreds of billions of dollars in tax money later, we now know that Iraq was never in danger of going bankrupt.

Are the parallels clear yet? They should be. It's too late for all those who died fulfilling the President's folly in Iraq, but it's not too late for us here in America.

Does Social Security really have weapons of mass destruction? Will privatizing it really cause future generations to greet us as liberators?

The answer is no. The President has cried wolf once too often, and the American people are wising up.

Vote No on Gonzales

2005-01-26T13:40:00.000-05:00

Alberto Gonzales made it through the Senate Judiciary Committee today, with the vote split evenly along party lines. I'm sad to say that one of my representatives, Mike DeWine of Ohio, toed the Republican party line like a good little kid and voted to confirm him. Now it goes to a vote in the full Senate.

I've written my senators already. Or, I should say, I've written to their staff. It's pretty much a given that the senators don't bother to read any correspondence they receive from the ordinary people they represent, but I had to try anyway. Here is a copy of my letter to Senator Voinovich. It's a shame; Voinovich is well-regarded here in Cleveland for helping to turn the city around, and I've heard him speak: he seems basically like a good guy. If not for the fact that since he's been in office he's been like a rubber stamp for the bad policies of the Bush administration, I might have even voted for him.

Please listen to your conscience and vote NO on Alberto Gonzales for Attorney General.

I am very disappointed to hear that Sen. DeWine voted to confirm Alberto Gonzales as Attorney General in the SJC, and I hope that you won't make the same mistake when it comes to a vote in the full Senate.

Gonzales is the author of the US justification for torture. That's the issue, and that is how history will judge you.

As a citizen, a representative, a father, a husband, a grandfather, a member of the church, and a Cleveland resident... Please help our country put a stop to this madness. No more rubberstamping bad policies. You are a better man than this.

If you don't stand up, who will?

Senator Voinovich is one of a scant hundred Americans with the opportunity to stand up and really make his voice heard, to speak for the millions of Americans he represents and say no to torture. Will he do so? Probably not. But if you feel the same way, I urge you to write to your Senators and try to convince them to see reason. Gonzales doesn't deserve a place in the U.S. government, and confirming him would be a mistake with far-reaching implications.

If Wishes Were iPods

2005-01-25T07:44:00.000-05:00

How popular is the iPod? How about the iTunes Music Store?

Insanely popular.

You already knew that, of course. But I've got a few sources of both subjective and objective data to share. First of all, since subjective data is so much more fun, let's look at Amazon's new feature, the Most Wished For list in electronics. Here's what it looks like right now as I write this:

Today, iPods and iPod accessories fill 9 of the top 25 slots. I've counted the iTunes Music Store card, since it's a pretty safe bet that almost everyone using the iTunes Music Store regularly either has an iPod or will be getting one soon. Other closely related items didn't count because they are not exclusive to the iPod: the AirTunes base station, the USB 2.0 cable, the headphones.

Just a few days ago, the same list showed 13 of 25 items that were iPod-related. The only MP3 player on the list at that time was the Creative Zen Micro 5GB, sliding in at number 22.

So there are a lot of people out there wishing for iPods. By Amazon's wish list metric, more people want iPods than want digital cameras, DVD players, TiVos, wireless networks, or any other MP3 player. You should definitely keep in mind that this is an entirely subjective and arbitrary sample; practically speaking it's meaningless, but I found it interesting anyway.

iTunes Music Store Sales

Want some real data? I've got that too. Not about the iPod, but about the iTunes Music Store. As I pointed out above, the popularity of the iTMS should generally track the popularity of the iPod pretty well. They are marketed together, and combined they provide a lot more functionality than either one alone.

How well is the iTMS doing? Apple does not release specific numbers about how many tracks the iTunes Music Store sells at any given time, but they do make press releases when they hit certain milestones. Here they are:

April 28, 2003 - Launch
May 5, 2003 - 1 million
May 14, 2003 - 2 million
June 23, 2003 - 5 million
September 8, 2003 - 10 million
December 15, 2003 - 25 million
March 15, 2004 - 50 million
July 11, 2004 - 100 million
October 14, 2004 - 150 million
December 16, 2004 - 200 million
January 24, 2005 - 250 million

Here's what that looks like on a chart:

A strong exponential growth trend is clearly visible. This trend has actually been obvious since July 2004, and every data point since then has followed the same path. Graphing the data on a logarithmic scale makes that even more clear.

The growth is almost a flat line against the log scale. I'd say it's probably flattening out slightly, but it's hard to say for sure. One argument against that is that we're currently getting data points more often, so we could just be seeing seasonal fluctuation that we didn't have the resolution to see before.

But one thing is clear: even in the flattest period we've seen, iTMS total sales are still increasing exponentially, at an astonishing rate of about 14% per month. At times it's as high as 26% per month. That's anywhere from 500% to 1000% growth per year.

iTMS to hit a billion in 2005

If this keeps up -- and the January 24th press release follows the trend exactly, and gives no reason to suggest it will stop -- then the current growth rate suggests that Apple will sell its one BILLIONTH song through the iTunes Music Store by the end of 2005. And that's actually a pretty conservative statement.

It only took ten months for the sales to increase tenfold from 10M to 100M tracks. If it takes the same amount of time to go from 100M to 1B, the store would reach 1,000,000,000 tracks sold by the beginning of May. Unbelievably, that's less than four months away.

My personal impression of the data is that the growth rate is slowing a bit, and it'll reach a billion in September or October 2005. But we'll see what the future holds.

Other music services

So how are all those other music stores doing? Not so well.

The IFPI reported last week that over 200 million songs were legally downloaded in 2004, compared to 20 million in 2003. If both these numbers and Apple's are considered accurate, and if they are measuring roughly the same data set, then we can actually get some idea about the market share of the iTMS.

(For the sake of this discussion, I'm going to assume both sets of numbers are both accurate and consistent, even though they might not be perfectly so. I'm inclined to believe Apple's numbers. They should be capable of very accurate tracking and are subject to shareholder lawsuits if they misrepresent their sales. The IFPI has no reason to be inaccurate, but neither would it be held accountable if it got any numbers wrong. I don't know whether they are measuring the same data; Apple is probably counting free downloads that don't generate revenue, and the IFPI might not be. Apple is probably also counting every track, including audio books, while the IFPI release was only interested in digital music -- however, I doubt that audio books are more than a drop in the bucket of Apple's sales.)

The IFPI's numbers state that over 220M songs have been legally downloaded and purchased over 2003 and 2004, which just so happens to extend back through the lifetime of Apple's iTunes Music Store. Well, Apple just reported on December 16th that it had sold 200M songs over the lifetime of its store. If Apple has really sold about 200M of those 220M songs, that gives them a market share of more than 90%.

To put it in perspective, these two sets of numbers suggest that everybody else added together has sold about 20M songs over the entire lifetime of their stores. Apple sold more than twice that number in just the past month, selling 50M in the period from Dec 16th to Jan 24th.

When you read an article like the IFPI's release about the explosive growth of "legal downloading", they really mean "the iTunes Music Store". Because honestly, that's where all the action is right now.

Anonymity and the Web

2005-01-04T23:26:00.000-05:00

From time to time I encounter people who wonder why I don't use some creative online name. You know, some AOL-inspired unique handle like "drewsome" or "hyperthought" or "knockinboots16". Better still, when I intelligently argue an opinion against someone who'd rather not be rational, I've received dark comments about how it's surprising that I'd use my real name to "spout that kind of garbage".

(As an aside, I find it interesting that the latter kind of vague threat -- and it is definitely meant as a threat -- has only so far come from people who hold far-right-wing political views. Whenever that happens, though, it's a sure sign that you've won an argument; remember, violence is the last refuge of the incompetent.)

So why do I use my real name, anyway?

Part of it is that I can't be bothered to keep finding new names when the one I want is taken. For example, I could choose to be "drewsome" on half a dozen accounts, but then have to come up with something new when it's taken elsewhere. Well, that just stinks. My full name is not the most unique one in the world -- I'm no Deep Jawa -- but I can get by with using it most of the time.

I also find that using my real name helps a lot in business, since there's no need to explain why I've chosen it. It's simple, to the point, and easy to remember if you know my name, which you should if you're doing business with me. And if you are using it to email me, you will have to spell my last name correctly. Hint: there is no Y in it whatsoever.

But one of the biggest reasons is that I truly believe that anonymity on the web is an illusion. And since it is an illusion I don't see that it's worth the bother to go out of my way to maintain it. I try to act online the same way I do in real life: I'm polite, I respect the opinions of others, and even when I have a very strong opinion I still try to see both sides of the story.

That's quite a sweeping statement to make, so let me clarify it. I do not mean that anonymity is an illusion in the absolute sense. It's still very possible in this day and age to keep your identity hidden online from anyone but the government if you work at it, and if you're extra paranoid it's probably even possible to keep hidden from the government. (Although I guarantee they do not reveal everything they might be able to do to track you down.)

But for the average person, if you spend a reasonable amount of time online interacting with others and you don't go well out of your way to protect your identity, you are not anonymous. You may feel anonymous, but it's an illusion.

If you make comments on blogs and use the same identity over a period of time so that people can recognize you, you are not anonymous.

If you ever mention personal details of your life online or fill them out in forms, you are not anonymous.

If you have an internet connection in your home which keeps the same IP address for a long period of time, whether it's static or dynamic, you are not anonymous.

If you maintain an account on a webserver which keeps logs (and pretty much all of them do, with rare exceptions), then to anyone with access to the logs you are not anonymous.

The technologically savvy among you are saying, "Sure, but there are ways to get around all that." And you're right. There are. But in the end they are impractical for the average person. To be anonymous to everyone but the government, you need to be fairly antisocial. You can't maintain a consistent identity -- for any system where an account is required, you generally would have to create multiple names/accounts and never linger in any one for too long. Furthermore you'd need to watch out for quirks of spelling, typing, word choice, and grammar which might provide hints to which accounts are the same people. And you must trust the webmaster of the site, since most server software normally keeps logs of which accounts are accessed from which IP. Anonymizing proxy servers are possible, but due to the vanishingly small number of people that use them, you can actually make yourself less anonymous by doing so! And most obviously, but perhaps the thing that people forget most often, is that you can NEVER reveal to anyone your real name; the memory of a web server is much longer than the memory of a person, and in many cases things you say on the web NEVER go away.

A recent example of how anonymity failed on the web came to my attention this summer. Remember the book Unfit for Command? It was not much more than an ugly hack job written to smear John Kerry, but the book got some attention from the paid conservative onanists. Then a micro-scandal flared up when it was revealed that one of the co-authors, Jerry Corsi, had made some nasty, hateful, and often racist comments on the nasty, hateful, and often racist far-right-wing political website Free Republic. Oops.

It turns out that Corsi fell victim to the last and most obvious pitfall -- he revealed his name, and it later came back to bite him. But take a closer look. Joe Conason of Salon had just documented the links between the Swift Boat Veterans for Truth and Free Republic, two organizations whose names are very much Orwellian doublespeak. To the best of my recollection, Conason was first to pick up on the link and then someone else found Corsi's name. But even if Corsi hadn't made the obvious slip, it really might not have been very long before Corsi was identified from his posts on Free Republic anyway.

Don't believe me? Perhaps you remember the protestors who "infiltrated" the Republican National Convention and made a nuisance of themselves. They were quickly tackled and subdued, but a TV camera caught a young-looking guy kicking one of them, a girl, while she was pinned down on the floor by event security. A lot of people were outraged, and there were a flurry of blog posts like this one trying to identify him.

With nothing to go on but his obvious political leanings and a vidcap of his fairly nondescript face, a likely candidate named Scott Robinson from UPenn's Wharton was tracked down. He eventually denied it, and it's not quite proved beyond a reasonable doubt. But there are things that make you go hmmm. Someone with a guilty conscience cropped him out of a picture on the National Taxpayer's Union website while the brouhaha was going on. Hmmm. People came forward who knew him and said yeah, sounds exactly like something he would do. Hmmm. The chair of the UPenn College Republicans even said it looks like him. Hmmm. What do you think?

There are many more stories like these two. I'll spare you the details, but I've been working online for more than a decade (closer to two decades, if you count BBS systems), and the same pattern keeps showing up.

Are you really anonymous? No. Not really, not when somebody really wants to know.

The Echo Chamber

2004-11-17T19:36:00.000-05:00

After you've been bouncing around weblogs for a while, you start to notice that there's an awful lot of echo (echo... echo...) in the blogosphere. (sphere... sphere...) In fact, sometimes it's downright annoying to see a dozen different posts on the same subject. What ever happened to diversity? Or sticking to what you know, rather than floating wild-assed guesses and making inane comments about something you know very little about?

Sure, people do that a lot in their everyday conversation. And blogs by their nature are going to reflect that. Believe me, I have no illusions of anything changing. And it's not just blogs -- 24 hour news channels, news magazines, and tabloids all suffer from the same effect.

Personally, I find it a lot more interesting when someone speaks with authority about their field of expertise, or talks about things that are decidedly off the bandwagon. That's why I keep tabs on Iraq via Juan Cole's Informed Comment, look for interesting and solid investigative reporting at Salon, geek out with Daring Fireball, and occasionally check for updates on below-the-media world events through Soj's excellent Flogging the Simian.

Linking and bandwagoneering has its place, certainly. It can provide a lot of ready detail and perspective on whatever topics are being attacked by the bandwagon at the moment. However, it can also serve as a distraction from other topics. After a certain point, the information content of each additional link starts to decrease, until the additional links are more noise than signal.

The more interesting thing to me is increasing the diversity of topics, which you get from people who post original articles with meaningful content. And in the long run, I think that's going to be what the Internet is really all about. Original content will be the pearls that people seek out in the ocean of information -- or more colloquially, the bucket of shit -- that is the 'net.

And that's my take on blogging in a nutshell: Be the pearl, not the shit.

'Nuff said.

The Bottle Let Me Down

2004-09-22T19:29:00.000-04:00

I ran across this album in iTunes while searching for something else. It caught my attention first because of the title, and then the whole concept and the interesting glimpses available through the preview.

Check it out: The Bottle Let Me Down - Country Songs for Young Suckers. (At Amazon for those who aren't iTunes-enabled.) Nice play on words. Yes, it's a kid's album, and yes, it's got a country theme... but that's not a problem because you have an open mind, right? Besides, most of them are pretty mild and they're honest country songs; not that glitzy fake western stuff.

Figuring it was worth $10 just for kicks, I bought it. There are a few notable standouts in the album.

The musical:

Sad and Dreamy, by Alejandro Escovedo - "I hit the big 1-0, I feel so old, candy just doesn't taste as good any more." Oh, life is so hard when you turn ten whole years old! The mournful tone reminds me of Nil Lara.
Three Little Fishes, by Andy Hopkins & Jon Rauhouse - definitely a kid's song, but the refrain is catchy.
Little Red Riding Hood, by Freakwater - surprisingly dark and sexy. "You're everything a big bad wolf could want." Rowr.
Rubber Duckie, by Kelly Hogan - she's got a very sexy voice, on par with Stacy Kent, and this is a fun song which shows it off well.

And the silly:

Godfrey, by Robbie Fulks - "Godfrey, the sickly unemployed amateur children's magician... he's got tricks you haven't seen." Whoa.
I Am My Own Grandpa, by the Asylum Street Spankers - fast-paced familial silliness, served up in bluegrass.
Funky Butt, by Devil in a Woodpile - "She got the funky butt, stinky butt... open up the window and let the funk go out." The laid-back New Orleans blues sound is actually pretty nice on the ears too.

John Kerry, Hero

2004-08-11T16:44:00.000-04:00

Here are a few collected stories about the lives that John Kerry has saved with his quick, decisive actions.

There are more stories besides the life-saving ones, if you choose to seek them out, about Kerry's sympathy and caring. For the most part I'll leave that up to you. But really, these four stories show exactly why John Kerry is a man deserving of everyone's respect regardless of your political views.

Think about the man you're reading about here, and then consider the motives of those who have been attempting to smear him. And always remember to check non-partisan sources like Annenberg Political FactCheck for the real truth.

Jim Rassmann in 1969
Source: Los Angeles Times, March 13 2004

Rassmann was 21 at the time, a Special Forces lieutenant in charge of a company of American and Chinese fighters. On that day, they traveled on a convoy of five patrol boats led by the 25-year-old Kerry, a Navy lieutenant — and they were on the run, being chased down the Bay Hap River by enemy soldiers firing guns and rockets.

The group had already lost one soldier that day. As they sped down the river, one boat was blown out of the water, and then another. An explosion wounded Kerry in the arm and threw Rassmann into the river. Rassmann dove to the bottom to avoid being run over by the other boats. When he surfaced, he saw the convoy had gone ahead.

Viet Cong snipers fired at him, and Rassmann submerged over and over to avoid being hit. The bullets came from both banks, and Rassmann had nowhere to go. He began thinking his time had come, but the fifth time he came up, he saw the convoy had turned around. Kerry had ordered the boats back to pick up the man overboard.

Kerry's boat, under heavy fire, sidled up to the struggling soldier. Rassmann tried to scramble up a cargo net at the bow but was too exhausted to make it all the way. He clung to the net as bullets whizzed past.

"Next thing I knew, John came out in the middle of all this," Rassmann says. "I couldn't believe it. He was going to get killed. He ran to the edge, reached over with his good arm [Kerry had been wounded in his right arm] and pulled me over the lip."

Rassmann later recommended Kerry for the Silver Star, and was upset when the Army instead awarded Kerry a lesser Bronze Star with a "V" for valor. The medal citation described Kerry's actions on the river that day.

Former US Senator Chic Hecht in 1988
Source: Las Vegas Sun, February 6 2004.

Former U.S. Sen. Chic Hecht of Nevada is a staunch Republican, but he thanks his lucky stars for Democratic presidential hopeful Sen. John Kerry of Massachusetts.

On July 12, 1988, Hecht was attending a weekly Republican luncheon when a piece of apple lodged firmly in his throat.

Hecht stumbled out of the room, thinking he might vomit but not wanting to do it in front of his colleagues. Sen. Kit Bond, R-Mo., thumped his back, but Hecht quickly passed out in the hallway.

Just then, Kerry stepped off an elevator, rushed to Hecht's side and gave him the Heimlich maneuver -- four times.

The lifesaving incident made international news, and Dr. Henry Heimlich, who invented the maneuver in 1974, called Hecht to say that had Kerry intervened just 30 seconds later Hecht might have been in a vegetative state for life.

"This man gave me my life," the 75-year-old Hecht said Thursday.

Hecht said he was amazed that Kerry acted so quickly -- some people were assuming that he was having a heart attack.

"He knew exactly what to do," he said. "But a lot of people know what to do. They just don't size up the situation immediately."

Del Sandusky
Source: Washington Post, July 28 2004

When George Butler, a college friend, broke his hip and femur in 1994, Kerry tracked him down the next day at a hospital in New York. Butler had been lying alone, worrying about a blood clot going to his brain. Suddenly, there was Kerry ringing his phone. He was in Tokyo, where it was 4 a.m.

Another time, Sandusky, Kerry's former helmsman, called him from Illinois and said, "I'm ready to cash it in. I can't stop the bad dreams, and I can't stop the drinking." Over the next 24 hours, Kerry talked to Sandusky -- canceling meetings, instructing aides to yank him off the Senate floor -- until his isolated friend agreed to check into a treatment center. Over the next 12 weeks, Kerry called Sandusky's doctors to make sure they hadn't forgotten his crewmate.

Licorice the hamster - Alexandra Kerry at the Democratic National Convention, July 30 2004

Well, it is an incredible experience to be here tonight. And I have to admit that it hasn't been easy to sift through years of memories about my father and find those few that might best tell you who John Kerry really is. So let me just begin with one July day when Vanessa and I were kids. It's a silly story, but it's true, and it's one of my favorite memories about my father.

We were standing on a dock waiting for a boat to take us on a summer trip. Vanessa, the scientist, had packed all of her animals, including her favorite hamster. Our overzealous golden retriever got tangled in his leash and knocked the hamster cage off the dock. We watched as Licorice, the unlucky hamster, as he became termed, bubbled down into a watery doom.

Now, that might have been the end of the story, a mock funeral at sea and some tears for a hamster lost. But my dad jumped in, grabbed an oar, fished the cage from the water, hunched over the soggy hamster and began to administer CPR. Now, there are still to this day, there are some reports of mouth-to-mouth, but I admit that's probably a trick of memory. The hamster was never quite right after that, but he lived.

Now, like I said, it may sound silly and we still laugh about it today, but it was serious. And that's what mattered to my father.

First posts

2004-08-08T19:07:00.000-04:00

First posts are pretty much useless things. So let's just get this out of the way quick, Annie Hall style, and not have to worry about it. Right now my darling wife is heading home with sushi for the family so I don't have much time. We all like sushi here, even my 8-year-old stepdaughter, a fact for which I am incredibly grateful.