Bits on Bytes bits on solaris, programming, and all things tech

This morning while I’m still asleep my wife goes “Chris your computer is making a noise…” and after repeating herself ten times I finally comprehend what she’s saying and start listening. Sure enough there was a loud clicking noise coming from my server in the other room. I figured I had just lost a hard drive but I wasn’t worried because I have a spare that’s supposed to pop into action when this happens. So I hit the power button once and listen as it shuts itself down cleanly.

Looking back I probably should have been more cautious but hey, I was still 98% asleep. When I finally got a chance to look at it, I find out the spare wasn’t actually added to my storage pool. Oops, my bad. Well I still wasn’t worried because my storage pool (all my really important data) was still up and rockin’:

peemus@serenity ~ $ zpool status -v

pool: tank
state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:
      NAME            STATE        READ    WRITE    CKSUM
      tank            DEGRADED        0        0        0
        raidz1        DEGRADED        0        0        0
          c2d0        ONLINE          0        0        0
          c2d1        ONLINE          0        0        0
          c3d0        ONLINE          0        0        0
          c3d1        ONLINE          0        0        0
          c1t0d0      UNAVAIL         0        0        0  cannot open
          c1t1d0      ONLINE          0        0        0
          c1t2d0      ONLINE          0        0        0
          c1t3d0      ONLINE          0        0        0

errors: No known data errors

So what did I do next? Added my spare and replaced my failed drive of course!

peemus@serenity ~ $ sudo zpool add tank spare c1t4d0s4
peemus@serenity ~ $ sudo zpool replace tank c1t0d0 c1t4d0s4
peemus@serenity ~ $ zpool status -v

pool: tank
state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: resilver in progress, 0.05% done, 2h16m to go
config:
      NAME            STATE        READ    WRITE    CKSUM
      tank            DEGRADED        0        0        0
        raidz1        DEGRADED        0        0        0
          c2d0        ONLINE          0        0        0
          c2d1        ONLINE          0        0        0
          c3d0        ONLINE          0        0        0
          c3d1        ONLINE          0        0        0
          spare       DEGRADED        0        0        0
            c1t0d0    UNAVAIL         0        0        0  cannot open
            c1t4d0s4  ONLINE          0        0        0
          c1t1d0      ONLINE          0        0        0
          c1t2d0      ONLINE          0        0        0
          c1t3d0      ONLINE          0        0        0
      spares
        c1t4d0s4      INUSE     currently in use

errors: No known data errors

Check that out! So to recap, it was my fault I didn’t have a spare configured, but ZFS let me add my spare and replace my failed drive with it - all while my data stayed up. You can even see it scrubbing data over to the spare!

ZFS rocks.