When Benchmarks Attack

In my professional life I've found myself measured by an external benchmark many times. One time that comes to mind was when I was writing disk drivers for Mac OS 8.

The disk driver's job in Mac OS 8 was simply to pass requests from the filesystem layer to the underlying storage medium. Requests would come in for such-and-such amount of data at such-and-such an offset from the start of the volume. After some very minimal translation, we'd pass this request onward to the ATA Manager, or USB Manager, or FireWire libraries.

(Sounds easy, right? Well, yes, but the devil was in the details. USB and FireWire supported "hot-plugging", i.e. attaching or removing the drive while the computer was still running. Supporting this in an OS that was never designed for it was a big chore. CD and DVD drives were also much more difficult to handle than regular hard drives, for similar reasons.)

But we ran into a problem as we were preparing our drivers for release. The hardware manufacturers that were buying our drivers wanted to make sure our drivers were "fast". So they ran disk benchmarks against our drivers. Not an unreasonable thing to do, you might say, although as I've stated there really wasn't a lot of code in the critical path. The problem is that most of the disk benchmarks turned out to be inappropriate as measures of driver performance.

Measuring weight with a ruler

One common disk benchmark of the time was to read sequential 512-byte blocks from a large file. In Mac OS 8 these reads were passed directly down to the disk driver.

Remember, the disk driver's job was supposed to be to simply pass the incoming requests on to the next layer in the OS. The vast majority of the time would be spent accessing the hardware. So no matter what the requests were, in effect this test should have returned almost exactly the same results for all drivers, with code differences accounting for less than 1% of the results. Right? Wrong.

As we ran the tests, we found out that on this particular benchmark, our drivers were much slower than existing third-party drivers (our competition, more or less). Dramatically slower, in fact -- if we ran the test in 100 seconds, they ran it in 10 seconds.

Upon further examination, we found out the reason why they did so well on this benchmark. They had already been optimized for it!

We discovered that the other drivers contained caching logic which would cluster these small 512-byte sequential reads into larger 32KiB or so chunks. Doing so would decrease the number of round-trips to the hardware needed, which increased their performance on this benchmark.

Do what sells, not what's right

Now, it's important to understand that this tiny-sequential-read benchmark doesn't even reflect real-world use. The fact that larger I/O requests perform better than smaller I/O requests has been well known for decades, and almost every commercial application out there used large I/O requests for this very reason. Buffered I/O is built into the standard C libraries, for pete's sake. In the real world, requests that came into the disk driver tended to be either large or non-sequential.

Modifying our drivers to do the same thing would basically be pointless -- a lot of work for very little performance difference.

(It only gets worse when you realize that our disk driver was directly underneath the Mac OS disk cache. A driver-level cache was almost completely redundant. And the right place to do this sort of readahead caching would have been there, in the existing OS cache layer, not in each individual disk driver.)

But ... sigh. The real world doesn't always make sense. These small sequential reads were a well-known benchmark, even if it was a disk benchmark and not a driver benchmark. And that simple fact elevated this atypical scenario into the eyes of a lot of people who didn't really understand what it meant. To them, if our driver didn't do the same thing as the competition, we wouldn't measure up.

De-optimizing to improve the benchmark

So in order to make sales we were forced to burn another month adding and testing driver-level caching so that we'd perform better on this benchmark.

The irony? The work we did to improve this atypical benchmark actually increased the amount of time we spent in the driver code, and sometimes increased the amount of data we read, which in turn decreased real-world performance by as much as 1-2%.

But hey, at least that benchmark got faster.

And we sold our driver.

Switched to Blogger Beta

Yesterday I moved this blog to the new Blogger Beta.

I went whole-hog and switched to a new template as well. I'm still tweaking it, and XHTML compliance is temporarily gone. But for the most part things should be back to normal. RSS readers may see that every post has been modified. This was a side effect of the switch, although in fact I added categories/labels to all posts once I saw that they were going to be touched anyway.

Why switch? The mainline Blogger service (1.0) has been having boatloads of problems over the past month or two. It hasn't been very well maintained and it seems like it's just barely holding together. All the Google blogs are using the beta now, and in fact there are some very nice features in the beta that I was tempted by.

Cool stuff that you get from the switch:

• MUCH faster publishing. 1.0 generated a zillion static HTML files, the 2.0 beta uses a database.
• Labels, aka categories.
• Backlinks, aka trackbacks. Better than MT's, since these are automatic. (Presumably filled in by Google crawls?)
• Comment feeds, both global and per-post.
• Better template management. It's more powerful and exposes more, and it's smart enough to let you add and rearrange certain things on your page with no effort.
• Widgets, aka server-side directives in your template.

Downsides of the switch:

• RSS reset. Everything in my RSS feeds got reset when I switched. Not a huge problem.
• Porting customizations. To get some of the nicer features you need to upgrade your template. If you had your blog moderately customized, and change your template like I did, then it can be kind of a chore to port your customizations over. For me it took about a day.
• Limited Safari support. Posting from Safari works just fine. Template editing seems like it has to be done in Firefox.
• It's a beta. And knowing Google, it will probably remain a beta for another three years or so. Dude. Can't you guys commit?

I haven't seen any other major downsides yet, but I'll let you know.

Video Game Tips for Parents

I want to talk a little about a couple of lessons that I've learned from several years of experience with my daughter and video games.

If you haven't done so already, I'd also recommend reading these other tips from an earlier post, where I describe the things I went through trying to figure out which PlayStation 2 games would work for my 6 year old daughter.

Here's what I've learned:

• Never, EVER buy a game without reading a review first. It's just a bad idea. That brings us to our second rule:

• Never, EVER buy a game without reading a review first. Seriously. I mean it! If you're new to video games, you might assume that most games are all about the same and it doesn't matter what you buy. Not a chance. Would you say the same for books and movies? There are some steaming pieces of trash out there which have been slapped together with spackle and duct tape, and they still sell a few copies because they're put in the store with fancy box art.

  This rule has several corollaries. Here's a list of very bad reasons to buy a game "cold" (i.e., without reading a review) that people can fall victim to:

  Bad Reason #1: It uses the name of some great game that you remember. Many classic video games have been resurrected into horrible zombie caricatures of their former greatness. Frogger is one of the worst offenders.

  Bad Reason #2: It's a sequel to a game you or your child liked. Often the sequel changes the gameplay. And sometimes the rights to a game are transferred to another studio, and when that happens the second studio usually does a ham-handed job with the sequel. (Movies are the same way. Consider Grease and Grease 2.)

  Bad Reason #3: It's a TV or movie tie-in. TV and movie tie-ins are very often poorly made. Sometimes they just whip them together with no concern for gameplay or difficulty. There are some terrific games which are exceptions to this rule, of course -- many of Nickelodeon's games are pretty good -- but you won't know unless you read a review.

  Bad Reason #4: The box art and description makes it sound interesting. Um, yeah. Every game is a "great adventure" and "exciting" and "fun" by the time the marketing people are done with it. The box has no relation whatsoever to the actual game.

• Look for the Greatest Hits titles. This is one place where the box can actually tell you something about the game. Sony's "Greatest Hits" are titles that have sold particularly well -- they are the gaming equivalent of a record going platinum. The nice thing is that the games get a lower price when this happens. So even though these games cost around $20, they're often much more fun than newer games which cost $50. Of course, brisk sales don't necessarily mean the game is age-appropriate, so you should still check reviews.

• Don't be afraid of older games. Quality is far more important. The age of the game really doesn't matter: a fun game will always be a fun game. It doesn't matter if the game is "new" or not; it will still be new to your child! Any parent who's watched their kid get hooked on The Cosby Show or Full House or some other unexpected sitcom will understand how the same logic applies directly to games. Younger kids aren't hooked on fancy graphics and don't care about what's latest and greatest; they just want a game that's fun. As an extra bonus, older games are cheaper -- often by a lot!

• Buy games used, and trade in old games. Most dedicated game stores, such as EB Games, offer used games for sale and offer you store credit on trade-ins. This is a great way to save a lot of money. The used games will be literally as good as new. Trading in is also a useful lesson for your kids; I've made it a point to help my daughter pick which games she wants to trade in. We keep some old favorites around because she still plays them, or because they are good for sleepovers, but most eventually go back. Then we shop for new games, trying to see how much we can get without spending more than $10.

• Consider renting games. It's up to you to decide whether this approach works for you. I don't do it with my daughter, because she simply doesn't play games enough to justify it, but some people do. It can be a fun way to try out a game without making a $40-$50 commitment. Just watch out for late fees! These days you can avoid them completely. Blockbuster now offers no-late-fee rentals (online and in-store) with their GamePass service, and there are a ton of NetFlix-like online rental services, including GameFly, GameLender, GPlay, and others.

• Swap games with friends. Your child might not even consider this unless you suggest it. Loaning games to friends is a nice way to socialize and it saves money, too. And it gives the kids something else to talk about!

• Play the games with your child. Especially with younger kids, but even as your child gets older this is important. It's a lot like reading a book together. It's a way to have fun and share an experience. When your child is particularly young, you might find that he gets stuck and gives up on a game quietly without really saying anything. If you help get past the tricky spots, he can go on having fun for longer. Or perhaps your kid will be like my daughter, and get (happily) scared when you reach the big, intimidating "boss" monster together, and she'll shove the controller over to you and want you to take over for a while. I usually play just long enough to show her how to do it, and then pass it back to her to try. Sometimes, if she asks, I'll get her past the hard part to the point where she wants to take over again. It's fun for both of you, and can be a great confidence builder.

• For big games, consider a hint book / strategy guide. Don't do this all the time, of course. But there are some larger games (such as Kingdom Hearts) where I found it to be very useful. What's great about these guides for kids ages 5-7 is that it can encourage their reading skills. My daughter wasn't really motivated to read much when she was 6, but the game strategy guide really got her going! It sounds silly, but it worked.

Any tips that you want to share?

Great PS2 Games for Young Kids, Part Two

Well, it's been (ahem) about a year and a half since I promised a "part two" to my earlier post on Great PS2 Games for Young Kids, Part One. And it's time for the next installment. Three more games this time:

• Monster Rancher 3
• Dance Dance Revolution
• Kingdom Hearts

[This post used to contain a lengthy list of tips. To trim it to a more manageable size, I've moved it to a new entry: Video Game Tips for Parents.]

Okay. With that out of the way, let's get to the games! Three more good ones here. Although Olivia is now age 10, I'll talk about some games that she loved between the ages of about 7 and 9.

Monster Rancher 3

This is actually a series of games, ranging from Monster Rancher 1 to 4, plus the relatively new (and very different) Monster Rancher EVO. At the time we got it, the fourth game was out but I'd heard more favorable reviews of the third game. Following my own advice -- don't be afraid of old games! -- I went with number three. It might be hard to find the third game these days; I hear that the fourth one is also good, but I can't say for sure since we've only played this one.

You play as a monster trainer. The game starts by creating a monster, either from some pre-made guys or -- the better choice -- from a "saucer stone", which is a code name for "any old random CD or DVD you have lying around the house". (You'll have to swap discs manually, at which point you should pause and take a moment to teach your child about how to put CDs away properly when they're done with them.) Each disc will create a unique monster with different looks and abilities, and the same disc will create the same type of monster if used again. Your monster starts as a baby, and you feed it and train it to help it grow stronger. Over time it matures, grows old, and eventually dies, at which point you can start over again. There is a little bit of anime-style story, and there is a tournament where the monsters can optionally fight (without blood, and losers aren't really "killed", but there is cartoonish violence -- monsters chomp each other, whap each other with giant cartoon mallets, etc) to gain levels and prizes. And that's pretty much it.

What's fascinating about this game is the way that kids just love it. But it's the kind of game that can drive an adult crazy! Why? Because it's cute, and there's a lot of repetition. Kids that are 10 and under love repetition; just like they enjoy watching the same video over and over, or reading the same book over and over, they don't mind doing the same thing in a game over and over. There is a lot of repetition in the game as you train your monster. You'll see the same animations over and over (sigh, and over) again. And yet there's still some variety -- you can train in different ways, or use different monsters, or go to a completely new area to train in.

The controls are menu-based, but pretty simple to figure out. There is some text that your child should pay attention to. For example, the game will suggest that it's time to feed your monster, or to let it rest. Perhaps the trickiest part is saving the game and continuing after your monster dies (which is necessary to open up new areas). Olivia had a tendency to just start new games, since she got attached to her monster and didn't want to save over it after it had grown old and died. But doing that meant that she never got to train anywhere other than the first area.

The game can also help build the skills your child needs to keep a pet. Monster Rancher isn't quite a "virtual pet" simulator, like a Tamagotchi or Nintendogs, but it shares some of the same elements. If you treat your monster nicely, he'll be sweet and lovable and grow stronger. If you spoil him too much, he'll get fat and temperamental. If you punish him too much or don't feed him enough, he might get mean. And even though it can be very sad when a monster grows old and dies, that's part of life too. The game handles it well and you can quickly start over with a new baby monster -- which gets a boost from the "heart" left over by the previous one.

Kids also really love the "saucer stone" part. Since you don't know what any disc will give you, you'll have to try a bunch of them and see what you get. It's surprisingly entertaining, and kids love it because it's something they never thought might be possible. It's a really unique gameplay mechanism that I've never seen anywhere else. Just be careful: unattended, your kids might quickly make a mess of your CD collection! We gave Olivia permission try lots of CDs but always made sure that she was careful with them and put everything back as soon as she was done.

Some people might be concerned about the whole "raising animals to fight" part. That's your call. Personally, this wasn't a problem for us. First, we make sure our daughter is able to distinguish fantasy from reality (always a good thing) and that she knows how to behave in real life. Second, the monster fighting is very much treated as a sport in the game -- just like boxing, karate, or amateur wrestling might be. There are tournaments, prizes, and so on. So it's not as if you're teaching your kids to enjoy cockfights or the WWF. It's an extremely mild game.

Dance Dance Revolution

This is another game that appeals to people across all ages. You've probably seen the arcade version in a mall. It's the one that has people stepping on four arrows while dance music plays, often drawing a crowd when the better players are "on stage". There's no real gender bias to the game, but my impression is that with younger kids it's more popular with girls, while among teenagers it's more popular among boys. I don't know this for sure; that's just the impression I get. Somebody could probably write a PhD thesis dissecting the relationship of this game to gender roles. :-)

First of all, you absolutely need to get the special "mat" controller that lets you dance. Some versions come with it bundled, or you can buy it separately. They sell some very expensive ones, but the regular old cheap one works well. Once you have that the game is very simple, but challenging. You're given a wide variety of songs to choose from, each with its own dance moves. All you need to do is just step on the arrows at the right time.

Sounds easy, right?

You wish. The guys at the mall may make it look simple, but DDR is very physical and can be challenging both for you and your child. You'll want to start out with the difficulty dialed all the way down to the lowest setting. It's okay to screw up -- and you will -- as long as you keep trying. Stick with it! After a while you'll start getting the hang of it. When you're first starting out, it's not unusual to have to stop after a few songs because you just get too tired and sweaty. I've found that taking turns of either one or three songs works well.

What's great about this game is that it's a very physical thing, and it's something your child will want to practice. It's not very mentally challenging, but it can be tricky trying to figure out how to do all the moves you need to get through a song. It's sort of like puzzle-solving with your feet. It's not a game that you'll "beat" any time soon; you will be able to continue playing and improving for a long time. It's also an easy game to pick up and play; there's no story, and nothing gets in your way. You just start it up and go. And it's great at parties as long as you make sure the kids share.

There are a lot of versions of this game. Some come with the "mat" controller bundled, some don't. As mentioned before, you'll really need the controller. We chose to get the Dance Dance Revolution Extreme Bundle, which included it. (NOTE: The preceding link lists a price of $199.95, which is insane. It should be about $40, which is the price you'll find for the almost equivalent DDR Extreme 2 Bundle. I e-mailed the seller and he said, paraphrased, that the price is deliberately high because it's new and unopened and someone stupid might buy it at that price.)

The primary difference between the versions is really the song lists. They tend to be techno dance music, and more or less appropriate for everyone. You might hear any of these songs on the radio. Your standards may vary, of course, but we really didn't find anything that we were concerned about in the game.

Kingdom Hearts

Kingdom Hearts has a special kind of magic. It turns Disney movies into three-dimensional worlds that you can run through and play around in, and that's just fantastic. Since it contains a lot of characters and stories that your child will already know, it really is a lot of fun, and unlocking the game's worlds can be really rewarding. And if you look at the top of the box you'll see that it's now become a "Greatest Hits" title, which is why it's gotten so cheap.

This is a tricky one to put on the list, but it's just so good that I had to do it. Despite its Disney theme, this is definitely not an easy game for a young child. It's very long, complicated, can be very difficult, and has a few scary points. I would say that it's probably targeted a little more at the tween to teen market. Olivia and I started playing it when she was only 7, but we didn't finish for months and months! It definitely shouldn't be a child's first exposure to video games, and you should be prepared to help your child play it almost the whole way through. In fact, I would strongly suggest getting a hint book like the Kingdom Hearts Official Strategy Guide ... these can be a lot of fun, and can encourage your child's reading skills.

You play Sora, a young boy who lives on an island with his friends. After some lengthy training and a series of opening sequences which seem to take forever, eventually your island is ripped apart. You land in a central hub called Traverse Town, and eventually meet up with Donald and Goofy -- who have been sent by King Mickey to find you and then deal with the problem of "Heartless", shadowy monsters which are invading all the Disney worlds. Eventually you'll visit the worlds of (and fight the major villains from) a lot of different Disney movies: Little Mermaid, Hercules, Nightmare Before Christmas, Alice in Wonderland, Aladdin, Peter Pan, and more.

There is violence in this game: fighting monsters is most of what you do. Fighting involves swordfights with a "Keyblade" that looks like a giant key. You're rarely fighting people; most of the fighting is against Heartless, who look like shadowy insects. Otherwise the content should be fine for all ages. Not all of the characters are Disney characters; some are Square-Enix characters (from other games made by the same game studio) and some were created just for this game.

The controls are moderately complicated. The basic moves are just running around, jumping, and hitting the attack button -- easily mastered by most kids if they've played a few games before. But there are a lot of menus, and even items that you can optionally buy and use (healing potions, magic, and so on). Saving goes through a menu as well, and must be done at a save point. There's no autosave, so your child must remember to save or else lose her progress.

The worlds you run around in can be big and complicated, and it's not always clear what to do next. But I've found that because it's a Disney theme, kids are very highly motivated to explore and look around. They will probably get stuck a lot, and the game makes it worse by creating certain situations where you just can't make forward progress and you have to give up and go to another world. For example, the Colosseum in Hercules isn't open for a long time, and there's no obvious reason why. That's why I'd suggest a hint book, with maps and strategies. When your child gets stuck, help him figure out how and where to look up the answer. You'll be helping him with his game, and very sneakily teaching him valuable skills like "how to use a reference book"!

There is also a sequel, Kingdom Hearts II, which is very similar to the first game. It's also long and complicated, but almost as good as the first game. I found that Olivia's enthusiasm ran out after a while on the sequel, but that might be because it didn't come out until two years after we finished the first one.

Wrap-Up

I still believe that the most important thing you can do as a parent is to read reviews of games before you buy them. Games can be expensive, and you don't want to buy bad ones.

There are a few places I go to find reviews. Amazon frequently has reviews written by kids and parents. I like GameSpot's reviews, although as a parent you sometimes need to translate a little: the criticisms of "too short" and "too easy" often mean the game will be just right for kids. IGN also has reviews, but the reviews aren't as well-written; sometimes it feels like they are written for gaming insiders who follow every little bit of news.

I hope to write more in this series, but I encourage you to write your own experiences in the comments. Let me know what you've tried, and what worked and didn't!