An Apple Genius Saved my Visit to Macworld Expo

In recent months, my MacBook has occasionally slowed down to a sleepy snail's crawl: applications would respond with a major lag, switching between applications would take a few seconds, and the time for launching another application -- including Activity Monitor for hunting down the culprit -- could be measured in minutes. I still have not found a general root cause for this behavior yet, but force-shutting down and rebooting would always get the machine back to its proper performance without any further problems.

The last time I had to apply this remedy, though, I could no longer log into my main user account after the machine had rebooted. And this was just minutes after I had left Phil Schiller's keynote, looking at a full week of Macworld Expo '09 still ahead of me.

Bad timing, really bad timing...

"Open Sesame!" no more

When I tried logging in after the reboot, the computer would apparently accept my username and password, because the login dialog box disappeared. However, after a few seconds of just showing an empty screen, it would simply display the login dialog again.

Usually, the simple ad-hoc fix would have been to hook up my portable hard drive to the Mac and reboot from that drive, which contains a complete clone of the MacBook's internal drive, courtesy of SuperDuper!. I would have been back in business within a minute.

This time, though, I had left the portable drive at my girlfriend's place for what I consider good reasons -- even when looking at it after the mess you are about to read about --, but I'll spare you the details. My plan was to buy an additional portable drive at the San Francisco Apple store, clone the 'Book's drive to it, and be done with it. As you may guess, I hadn't gotten around to doing that when the login troubles began. Bummer with a capital "B."

What saved me from a massive mood swing in this situation was the fact that I use several user accounts on my Mac, including one that is used exclusively for administering the machine. That account is, in fact, the only admin account on the machine, and it does not feature any third-party software like preferences panels that could cause problems, so there's a reasonable chance that this account will work fine even when others don't. And, luckily, I could log into the admin account without any problems. Same thing with the other two user accounts: login worked like a charm. Despite a few more attempts, though, the main user account still remained a no-can-go zone.

First Aid for accessing the machine

Because I could log into all other accounts, I assumed that during whatever the machine did while it had slowed down, one crucial file like a UNIX login script file or somesuch had been damaged, but that I had not lost any other data. So to make sure that that wouldn't happen after all, I went over to the Apple store near Market street and bought that portable disk that I should have bought much earlier.

Hooked it up, used SuperDuper! to make a clone, and tried booting from the freshly created clone. The Mac came up fine, but the problem persisted. At least I could easily retrieve some files now: by rebooting from the internal drive and checking the "Ignore ownership for this volume" option for the clone, I could easily regain access to my password database, my Macworld Expo registration PDFs, by copying the respective files to the internal volume.

Sometimes, finding the files that contain the actual data is a bit tricky for applications that maintain an "internal" database instead of individual files stored in the Documents folder. Bare Bones's Yojimbo, which I use for securely storing my passwords, is just one such example. Usually, you can find these files somewhere in the ~/Library/Application Support/ folder hierarchy.

After resurrecting just a few files this way, I realized how tedious a task it would be to recreate a comfortable work environment in another account on the machine (keep in mind that we're talking "Major Apple Event of the Year Week" here), so I packed up my Mac and headed over to the Apple store, hoping to get some Genius Bar help for a more efficient and encompassing fix. Unfortunately, the drop-in-patient approach did not work, as there were no more open slots at the Genius Bar left that day, but I at least got an appointment for early afternoon the next day.

As an aside, the important thing to mention here is that I was absolutely certain that I had not lost any seriously valuable files. Not so much because I could successfully log into the other accounts on the MacBook, but because I knew, that, even when worse had (or would have) come to worst, there were a total of three recent and complete backups of everything that was on the 'Books internal drive in other places on this Blue Sphere.

Admittedly, they were of no use to me for solving the login problem ordeal, but there is nothing that is as important for a computer user's peace of mind than knowing that you have a reliable backup strategy in place.

And now, back to our main story -- let's fast-forward to Wednesday.

An official Genius at work

When I arrived at the Apple Store for my appointment, I only had to wait for a few minutes before it was my turn to approach the Genius Bar, and I explained my misery to Apple Genius Zach Leach (who has kindly allowed me to mention his full name), upon which he immediately went to work. Here's what he did to (almost) fully restore my Mac to a fully functional state.

As a first step, he booted the machine in "Verbose Mode" (hold down Command-V while booting) to check if any error message would pop up during the boot process or after typing in my login credentials. Unfortunately, the machine just "hung there" after the login attempt, so he rebooted the machine via the power key and now held down Cmd-S to launch the Mac in "Single User Mode," which provides immediate and full root access to the computer.

"root" is the name of the most powerful user account on a UNIX machine, and full root access means that you have access to literally every file on your computer, including those pesky system files that are usually hidden from your view in the Finder. If you don't feel comfortable using the Terminal, messing around in root territory is not such a grand idea: inadvertently delete or move just one important system file, and your machine may not boot anymore at all. Just sayin'.

When booting into Singe User Mode, the file system is mounted read-only, so to make any changes, you need to mount the drive for write access via mount -uw /. After doing so, Zach successively "hid" the LaunchAgents, Caches, and Preferences folders by renaming them via the UNIX mv command and then rebooted the computer, but the problem persisted even when "hiding" all three.

Next up was cleanly recreating the user account that was acting up: to do so, Zach renamed the user folder -- in this case, /Users/jochenw -- to "hide" it from the OS. He then removed the account by deleting it via the Accounts preferences pane, renamed the user folder to its original name, and then re-created the user, using the exact same "short name" in the Accounts prefs.

The OS detects that a corresponding user folder already exists and offers to use that folder for the account to be created. That way, you may be able to fix a bug in the accounts database without losing any user data and without having to backup the user folder to another location first.

Alas, it did not solve the problem with my MacBook, which meant that something deep in OS X's guts had been damaged when I force-shut down the machine the day before. Time to take out the Virtual Sledge Hammer.

Which, in this case, was a clean re-install of OS X in Archive-and-Install mode. For these purposes, the Geniuses have a portable hard drive containing several versions of OS X at hand, which makes for a much faster installation process compared to using the standard DVD approach. Still, Zach estimated that it would take some 20..30 minutes for the installation to complete, which gave me ample time to roam the area around Union Square on this gorgeously sunny day, grab a Caramel Macchiato at a nearby Starbucks, and eventually head back to the Apple Store.

When choosing "Archive and Install," the Installer creates a new folder on the target disk, called Previous Systems, inside which reside time-stamped folders for each installation (attempt) of the OS, which, in turn, contain the previous /Applications, /Library, /System, and /User folders. Which means that, despite a spankingly clean new system installation, all user folders with all the files they contained are still stored on the internal drive of the machine. To tie them back into the system, you simply re-create the user accounts and move the user folders from the archive inside the /Previous Systems folders to the /Users folder. This includes, by the way, the Shared folder, for which, obviously, is not assigned to an individual user account.

Once Zach had moved all user folders over to the /Users account, I took over from him to create all four users accounts with their original short names, and for each, OS X offered to use the respective folders with the similar names. And that was it.

Done. Finally. Time to reboot, and this had to work now, because... Well, because!

So reboot the machine we did, and wait for the login screen we did, and type in the credentials we did, and stare at the screen we did, and rejoice we did when the spinning beach ball indicated some serious processing going on after the login. And finally cheer I did when the desktop for my main account appeared on the screen.

So there it was, my familiar work environment. Earlier I had written that Zach managed to "to (almost) fully restore my Mac to a fully functional state." As was to be expected from a fresh OS install, what was missing was any third-party software I had installed and, more specifically, any software that drops components into the /Library folder, e.g., QuickTime plugins like Perian or Flip4Mac, or iLife applications with their theme files.

Still, I couldn't have been happier with the service I experienced at the Apple store. Not only did Zach manage to fix the core problem. He also showed me exactly what he was doing all along the way so I could add new trouble-shooting techniques to my virtual Mac Tool Belt and also share the process with you, valued reader.

And I haven't even mentioned the courtesy displayed by all staff at the Genius Bar, yet. Oh, and let's not forget that this service did not cost me a single cent, either. Call it the anti-hotline service experience if you will: friendly, efficient, effective, and at an unbeatable price.

The aftermath

Back at the hotel, I could reinstall some applications by just dragging them from the Applications folder inside the /Previous Systems folder to /Applications. Still, right now, some applications are not working, e.g., Aperture, but re-installing them properly from their DVDs will be a no-brainer once I'm back home.

As an important side note, after the fresh OS installation, third-party firewalls may be disabled, and OS X's own built-in firewall is not activated by default. So before connecting your Mac to any network after this kind of deep-in-the-OS-internals trouble-fixing, do active the firewall to prevent any intrusions.

Lessons to be Learned

So, have I lost anything during this whole ordeal? Well, as mentioned before, at least I have not lost any valuable data; in fact, I have not lost any data. However, all in all, I have probably invested some 6..8 hours of very precious time that I would much rather have spent on the Macworld Expo show floor or in classes. But that's the kind of smack in your face that impatience gets you. Stupid me.

I have, however, also gained something, namely a few insights that will come in handy whenever I will be confronted with a similar situation in the future:

• Only force-shut down your Mac if it is absolutely necessary

  Although I have not verified this by analyzing log files, etc. as I have definitely lost too much time with this problem already, I am sure that shutting down the machine while it was doing something important in the background was what caused the problem in the first place.

  Hence, it is a very good idea to apply the power-button force-shut down only in cases where the machine has completely frozen up, i.e., it won't accept any keyboard or mouse input anymore, and nothing at all is happening on screen. Apply such brute-force digital bail-outs only as a final resort. Should your Mac "just" slow down, try to be patient and shut it down properly via the Shut Down command.

• Always, always bring an external hard drive with a functioning clone of the Mac's internal drive

  The one thing that could have saved me from this major hassle would have been an external hard drive with a reasonably recent clone image of the internal disk: plug it in, switch it on, boot from it -- voilà!

  Hence, do invest in such a hard drive, and if, for whatever reason, you cannot bring that drive you already have, by all means buy another one. I've paid around $160 for a 320GB bus powered drive with both Firewire and USB 2 connectors, and you'd probably get even betters deals when buying online versus brick-n-mortar retail. Considering how much time I've had to waste for having the machine fixed, thought, that's just peanuts.

• Create a dedicated administrator account on your Mac

  In the hopefully unlikely case that you end up in a similar situation where you cannot log into your main account anymore and don't have an emergency external drive at hand, the only way you can at least try to get to your data is via another user account. So, if you haven't done so yet, do create (at least) one more account besides your main work account.

Saved from despair

A lot of emails, few article stubs, and two or three SuperDuper! backups later, I think back to how I felt on Tuesday morning, and I realize that what could have easily ruined the whole Macworld Expo experience for me -- both practically as well as mood-wise --, has been reduced to losing "just" a few hours of time. Thanks to Zach, the San Francisco Apple Genius who saved my visit to Macworld Expo.