EqualLogic Madness

Over the last year or so, we’ve had the chance to play with Dell’s EqualLogic line of SAN arrays. We’ve sold a few to some customers, and as far as I’m concerned, the way those things operate are 10% logic and 90% magic. A couple of the guys from the office have gotten some formal EqualLogic training, but it’s still something that only kind of makes sense to me. So if I goof up the technical details or say something completely idiotic, forgive me. Sometimes I know not what I say.

Anyways.

We’ve got a customer who bought an array back in the spring that currently houses their VMDK files for the VMs that they run on. It’s an entry level SAN with two hot spares, and at some point last week one of the hot spare drives vanished. By vanished, I don’t mean failed…I mean the thing disappeared from the management software altogether. It was as if someone had gone in and physically removed the drive from the array. So we call Dell and we get a replacement drive dispatched.

Monday we notice that another drive has failed (leaving us with zero hot spares), so I call Dell and get a replacement drive dispatched for that slot as well. There’s some confusion about what’s going on, since a) I don’t really know what I’m doing with this thing, b) we did a firmware upgrade on the array over the weekend, and c) we’ve now got two separate problems with the same array happening at the same time.

So Tuesday evening I go onsite to the customer and replace the nonpresent drive, but the replacement drive doesn’t show up. The logs indicate that a drive was removed and re-inserted into the slot, but the software indicates that no drive is present. Since Dell suggested it (don’t try this at home or at the office!) I re-seated the drive. Still nothing. Since I may still have a known good drive, I put it in the slot for the other clearly failed drive. That one comes online right away, gets marked as a hot spare, and…

…the previously nonexistent hot spare comes online and is marked as failed.

Figure that one out.

I’ve got a few suspicions as to what the problem might be. My money’s on a bad controller, but the only way to test it is to pull the controller module and force a failover, which isn’t something I’m too keen on doing yet. Since I have Thursday and Friday off…I’ll update on Monday with how the next set of tests go. Wish me luck…

Update
I haven’t gotten around to doing any further testing on the array because the customer who owns the hardware has ordered another beefier array to join to the group. Once this gets in and I can get at the array and run tests during business hours, I’ll post an update.

Update X2
We finally had the chance to run some tests on the array. Read about the troubleshooting and the results here.