Solaris 11 Boot Environment (BE) and ZFS rpool disk replacement

[cah]

Last week, hsiao.net was experiencing HD issue in the rpool.
In the beginning, c6d0s0 was showing write errors. I used 'zpool clear rpool' to clear the write errors. rpool was good for a couple of days.

Code: Select all

  pool: rpool
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
	repaired.
  scan: scrub repaired 0 in 1h55m with 0 errors on Wed Sep 10 02:04:21 2014
config:

	NAME        STATE     READ WRITE CKSUM
	rpool       DEGRADED     0     0     0
	  mirror-0  DEGRADED     0     0     0
	    c5d0s0  ONLINE       0     0     0
	    c6d0s0  FAULTED      0   242     0  too many errors

errors: No known data errors

Then, c5d0s0 was showing 488K write errors:

Code: Select all

  pool: rpool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
  scan: resilvered 207G in 1h39m with 0 errors on Wed Sep 10 15:05:52 2014
config:

	NAME        STATE     READ WRITE CKSUM
	rpool       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    c5d0s0  ONLINE       0  488K     0
	    c6d0s0  ONLINE       0     0     0

errors: No known data errors

Then, it was marked "faulted" "corrupted data" and it was no longer accessible.

Code: Select all

  pool: rpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
	invalid.  Sufficient replicas exist for the pool to continue
	functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-4J
  scan: resilvered 4.54G in 7h58m with 0 errors on Sun Sep 14 21:28:27 2014
config:

	NAME        STATE     READ WRITE CKSUM
	rpool       DEGRADED     0     0     0
	  mirror-0  DEGRADED     0     0     0
	    c5d0s0  FAULTED      0     0     0  corrupted data
	    c6d0s0  ONLINE       0     0     0

errors: No known data errors

hsiao.net was still working until I rebooted it.
I was no longer able to boot up from the latest BE (solaris-3)!!

Working BE:

Code: Select all

%beadm list
BE        Active Mountpoint Space   Policy Created          
--        ------ ---------- -----   ------ -------          
solaris   -      -          1.15G   static 2012-03-25 22:47 
solaris-1 -      -          34.71M  static 2012-06-22 18:32 
solaris-2 -      -          141.13M static 2012-11-21 18:47 
solaris-3 NR     /          28.40G  static 2012-11-21 19:05 

I could boot up solaris-2 but all updated data after 2012-11-21 was kept in solaris-3 and was unavailable in solaris-2 BE.

I took out the broken c5d0s0 and it sounded strangely. I then found one Seagate Baracuda 1 TB SATA drive from Amazon and ordered it.
It arrived on 09/18/2014 the day I returned from MSP.
I thought it would just work when I replace the drive but it still didn't work.

On 09/19/2014, I decided to detach c5d0s0 from the mirror and I was able to boot solaris-3 BE!!!
That means the faulted c5d0s0 was the culprit to hang the BE during boot up.

When I tried to attach c5d0s0 back to the rpool mirror, it complained c5d0s0 does not exist.
From 'format', I know for sure, the new Seagate Baracuda drive is at c5d0.

0. c5d0 <ST1000DM-Z1D4MWF-0001 cyl 60797 alt 2 hd 255 sec 126>
/pci@0,0/pci-ide@8,1/ide@0/cmdk@0,0
1. c6d0 <WDC WD50- WD-WCASU404856-0001 cyl 60797 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@8,1/ide@1/cmdk@0,0

However, it just doesn't want to take it. I then tried c5d0p0 and it took it into mirror and resilvered it.
However, installgrub didn't work. It needs to have a raw device in slice (ex. c5d0s0).

Code: Select all

cahtoh02:/root%/sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5d0p0
raw device must be a root slice (not s2)
Unable to gather device information for /dev/rdsk/c5d0p0

I then did a comparison between c5d0 and c6d0 in 'format' after I created Solaris2 partition in c5d0:
c5d0

Code: Select all

Part      Tag    Flag     Cylinders         Size            Blocks
  0 unassigned    wm       0                0         (0/0/0)              0
  1 unassigned    wm       0                0         (0/0/0)              0
  2     backup    wu       0 - 60796      931.46GB    (60797/0/0) 1953407610
  3 unassigned    wm       0                0         (0/0/0)              0
  4 unassigned    wm       0                0         (0/0/0)              0
  5 unassigned    wm       0                0         (0/0/0)              0
  6 unassigned    wm       0                0         (0/0/0)              0
  7 unassigned    wm       0                0         (0/0/0)              0
  8       boot    wu       0 -     0       15.69MB    (1/0/0)          32130
  9 alternates    wm       1 -     2       31.38MB    (2/0/0)          64260

c6d0

Code: Select all

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       1 - 60796      465.72GB    (60796/0/0) 976687740
  1 unassigned    wu       0                0         (0/0/0)             0
  2     backup    wu       0 - 60798      465.74GB    (60799/0/0) 976735935
  3 unassigned    wu       0                0         (0/0/0)             0
  4 unassigned    wu       0                0         (0/0/0)             0
  5 unassigned    wu       0                0         (0/0/0)             0
  6 unassigned    wu       0                0         (0/0/0)             0
  7 unassigned    wu       0                0         (0/0/0)             0
  8       boot    wu       0 -     0        7.84MB    (1/0/0)         16065
  9 unassigned    wu       0                0         (0/0/0)             0

I did not have slice 0 configured in c5d0. No wonder /dev/dsk/c5d0s0 doesn't exist!!
I then configured slice 0 to the following and labeled it.

Code: Select all

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       3 - 60752      930.74GB    (60750/0/0) 1951897500
  1 unassigned    wm       0                0         (0/0/0)              0
  2     backup    wu       0 - 60796      931.46GB    (60797/0/0) 1953407610
  3 unassigned    wm       0                0         (0/0/0)              0
  4 unassigned    wm       0                0         (0/0/0)              0
  5 unassigned    wm       0                0         (0/0/0)              0
  6 unassigned    wm       0                0         (0/0/0)              0
  7 unassigned    wm       0                0         (0/0/0)              0
  8       boot    wu       0 -     0       15.69MB    (1/0/0)          32130
  9 alternates    wm       1 -     2       31.38MB    (2/0/0)          64260

NOTICE: I purposely left some (44 to be exact) cylinders unused because the next 1 TB drive I will be buying might be a little smaller. If that's the case, I cannot use it in the mirror. It can only be equal or larger size than the current one. That's why I reduced 44 cylinders for this possibility. Ideally, I should get another identical HD but who knows if this HD's quality is up to my expectation.

After this, I was able to attach c5d0s0 to rpool mirror!

Code: Select all

% zpool attach rpool c6d0s0 c5d0s0

Then, resilvering started.

Code: Select all

%zpool status
  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Sep 20 05:53:26 2014
    24.1G scanned out of 207G at 35.1M/s, 1h28m to go
    24.1G resilvered, 11.64% done
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c6d0s0  ONLINE       0     0     0
            c5d0s0  ONLINE       0     0     0  (resilvering)

errors: No known data errors

After that, I just need to install the boot block on the new disk.

Code: Select all

cahtoh02:/root%/sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5d0s0
stage2 written to partition 0, 282 sectors starting at 50 (abs 32180)
stage1 written to partition 0 sector 0 (abs 32130)

Then, I can boot up from this new disk.
1. Shut down the system
2. Power up the machine
3. Press 'ESC'
4. Enter 'Boot' selection screen
5. Choose "ST1000DM-Z1D4MWF" (new Seagate 1 TB disk)
6. Select "solaris-3" BE if necessary

Everything is back to normal as of 09/20/2014.
Hopefully, it will be up for some time.

[cah]

c6d0s0 in rpool had been reporting unavailable for a few days.
I decided to get a new 4 TB SATA HD and use the existing 1 TB SATA backup HD to be the mirror disk in rpool.
After I backed up all data from the 1 TB backup HD to 4 TB new HD, I replaced the failed c6d0s0 SATA HD I got from Hsi a few years ago today (07/19/2015).

Physical replacement is straightforward but there were more work to do in OS.

1. Use "format" command to set up the replaced HD.
2. Configure c6d0s0 (replaced HD)

Code: Select all

format> disk


AVAILABLE DISK SELECTIONS:
       0. c5d0 <ST1000DM-Z1D4MWF-0001 cyl 60797 alt 2 hd 255 sec 126>
          /pci@0,0/pci-ide@8,1/ide@0/cmdk@0,0
       1. c6d0 <drive type unknown>
          /pci@0,0/pci-ide@8,1/ide@1/cmdk@0,0
Specify disk (enter its number)[0]: 1

Enter number of data cylinders: 16383
Enter number of alternate cylinders[2]: 3
Enter number of physical cylinders[16386]: 
Enter number of heads: 16
Enter number of data sectors/track: 63
Enter rpm of drive[3600]: 7200
Enter format time[default]: 
Enter cylinder skew[default]: 
Enter track skew[default]: 
Enter tracks per zone[default]: 
Enter alternate tracks[default]: 
Enter alternate sectors[default]: 
Enter cache control[default]: 
Enter prefetch threshold[default]: 
Enter minimum prefetch[default]: 
Enter maximum prefetch[default]: 
Enter disk type name (remember quotes): "HDS721010CLA332"
selecting c6d0
No current partition list
No defect list found
[disk formatted, no defect list found]
No Solaris fdisk partition found.

format> disk


AVAILABLE DISK SELECTIONS:
       0. c5d0 <ST1000DM-Z1D4MWF-0001 cyl 60797 alt 2 hd 255 sec 126>
          /pci@0,0/pci-ide@8,1/ide@0/cmdk@0,0
       1. c6d0 <DEFAULT cyl 60797 alt 2 hd 255 sec 126 cyl 60797 alt 2 hd 255 sec 126>
          /pci@0,0/pci-ide@8,1/ide@1/cmdk@0,0
Specify disk (enter its number)[1]: 1
selecting c6d0
Controller working list found
[disk formatted, defect list found]

3. Delete existing NTFS partition and create a Solaris02 partition and make it active.

Code: Select all

Delete partition 1
Then, create a partition

format> fdisk
             Total disk size is 60800 cylinders
             Total disk size is 60800 cylinders
             Cylinder size is 32130 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============  =====   ===   ======   ===
          1       Active    Solaris2          1  60799    60799    100


Total disk cylinders available: 60797 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0 unassigned    wm       0                0         (0/0/0)              0
  1 unassigned    wm       0                0         (0/0/0)              0
  2     backup    wu       0 - 60796      931.46GB    (60797/0/0) 1953407610
  3 unassigned    wm       0                0         (0/0/0)              0
  4 unassigned    wm       0                0         (0/0/0)              0
  5 unassigned    wm       0                0         (0/0/0)              0
  6 unassigned    wm       0                0         (0/0/0)              0
  7 unassigned    wm       0                0         (0/0/0)              0
  8       boot    wu       0 -     0       15.69MB    (1/0/0)          32130
  9 alternates    wm       1 -     2       31.38MB    (2/0/0)          64260

4. Set up partition 0 on the replaced disk

Code: Select all

partition> 0
Part      Tag    Flag     Cylinders         Size            Blocks
  0 unassigned    wm       0                0         (0/0/0)              0

Enter partition id tag[unassigned]: root
Enter partition permission flags[wm]: 
Enter new starting cyl[3]: 
Enter partition size[0b, 0c, 3e, 0.00mb, 0.00gb]: 60750c


partition> p
Current partition table (original):
Total disk cylinders available: 60797 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       3 - 60752      930.74GB    (60750/0/0) 1951897500
  1 unassigned    wm       0                0         (0/0/0)              0
  2     backup    wu       0 - 60796      931.46GB    (60797/0/0) 1953407610
  3 unassigned    wm       0                0         (0/0/0)              0
  4 unassigned    wm       0                0         (0/0/0)              0
  5 unassigned    wm       0                0         (0/0/0)              0
  6 unassigned    wm       0                0         (0/0/0)              0
  7 unassigned    wm       0                0         (0/0/0)              0
  8       boot    wu       0 -     0       15.69MB    (1/0/0)          32130
  9 alternates    wm       1 -     2       31.38MB    (2/0/0)          64260

5. Label it and quit

Code: Select all

partition> l 
Ready to label disk, continue? y

6. Replace HD information saved

Code: Select all

cahtoh02:/root%format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c5d0 <ST1000DM-Z1D4MWF-0001 cyl 60797 alt 2 hd 255 sec 126>
          /pci@0,0/pci-ide@8,1/ide@0/cmdk@0,0
       1. c6d0 <Hitachi-   JP2940HD0NEKA-0001 cyl 60797 alt 2 hd 255 sec 126>
          /pci@0,0/pci-ide@8,1/ide@1/cmdk@0,0
Specify disk (enter its number):

7. Attach it to existing rpool

Code: Select all

cahtoh02:/root%zpool attach -f rpool c5d0s0 c6d0s0
Make sure to wait until resilver is done before rebooting.

8. Check rpool status

Code: Select all

oh02:/root%zpool status rpool
  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function in a degraded state.
action: Wait for the resilver to complete.
        Run 'zpool status -v' to see device specific details.
  scan: resilver in progress since Sun Jul 19 16:29:48 2015
    88.4G scanned out of 333G at 2.60G/s, 0h1m to go
    0 resilvered
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            c5d0s0  ONLINE       0     0     0
            c6d0s0  DEGRADED     0     0     0  (resilvering)

errors: No known data errors

It scanned first then resilvered (synced)

Code: Select all

cahtoh02:/root%zpool status rpool
  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function in a degraded state.
action: Wait for the resilver to complete.
        Run 'zpool status -v' to see device specific details.
  scan: resilver in progress since Sun Jul 19 16:29:48 2015
    333G scanned
    22.7G resilvered at 50.9M/s, 6.77% done, 1h44m to go
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            c5d0s0  ONLINE       0     0     0
            c6d0s0  DEGRADED     0     0     0  (resilvering)

errors: No known data errors

.....

9. Use bootadm on x86 systems with Solaris 11.1 and above.
The install-bootloader subcommand installs the system bootloader. It supersedes the functionality of installgrub(1M) on x86 as well as supporting installation of GRUB2's bootloader on x86.

Code: Select all

cahtoh02:/root%bootadm install-bootloader -P rpool c6d0s0
cahtoh02:/root%

Need to run the following test to see if it boots from the replaced HD:

Code: Select all

1. Shut down the system
2. Power up the machine
3. Press 'ESC' at RAID controller screen or right after it (or it will miss the chance to select which HD to boot)
4. Enter 'Boot' selection screen
5. Choose "Hitachi-   JP2940HD0NEKA-0001"
6. Select "solaris-6" BE if necessary

10. c5d0 is a 1 TB HD but the old c6d0 HD was only 500GB so the rpool can not exceed its size. Now that both disks are 1 TB, the rpool should be able to expand. It does not automatically expand it because the "autoexpand" was set as "off".

Code: Select all

cahtoh02:/root%zpool get all rpool | more
NAME   PROPERTY       VALUE                 SOURCE
rpool  allocated      333G                  -
rpool  altroot        -                     default
rpool  autoexpand     off                   default
rpool  autoreplace    off                   default
rpool  bootfs         rpool/ROOT/solaris-6  local
rpool  cachefile      -                     default
rpool  capacity       71%                   -
rpool  dedupditto     0                     default
rpool  dedupratio     1.00x                 -
rpool  delegation     on                    default
rpool  failmode       wait                  default
rpool  free           131G                  -
rpool  guid           15585389463118454899  -
rpool  health         ONLINE                -
rpool  listshares     off                   default
rpool  listsnapshots  off                   default
rpool  readonly       off                   -
rpool  size           464G                  -
rpool  version        35                    default

cahtoh02:/root%df -h
Filesystem             Size   Used  Available Capacity  Mounted on
rpool/ROOT/solaris-6   457G    15G       124G    12%    /
/devices                 0K     0K         0K     0%    /devices
/dev                     0K     0K         0K     0%    /dev
ctfs                     0K     0K         0K     0%    /system/contract
proc                     0K     0K         0K     0%    /proc
mnttab                   0K     0K         0K     0%    /etc/mnttab
swap                   1.9G   1.6M       1.9G     1%    /system/volatile
objfs                    0K     0K         0K     0%    /system/object
sharefs                  0K     0K         0K     0%    /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap2.so.1   139G    15G       124G    12%    /lib/libc.so.1
fd                       0K     0K         0K     0%    /dev/fd
rpool/ROOT/solaris-6/var   457G   7.0G       124G     6%    /var
swap                   1.9G    56K       1.9G     1%    /tmp
rpool/VARSHARE         457G   121M       124G     1%    /var/share
rpool/export           457G    32K       124G     1%    /export
rpool/export/home      457G   136G       124G    53%    /export/home
rpool/export/home/cah   457G   103G       124G    46%    /export/home/cah
rpool/export/home/tcl   457G    58G       124G    32%    /export/home/tcl
rpool                  457G   5.0M       124G     1%    /rpool
rpool/VARSHARE/zones   457G    31K       124G     1%    /system/zones
rpool/VARSHARE/pkg     457G    32K       124G     1%    /var/share/pkg
rpool/VARSHARE/pkg/repositories   457G    31K       124G     1%    /var/share/pkg/repositories

Use the following command to set it on:

Code: Select all

cahtoh02:/root%zpool set autoexpand=on rpool

rpool reflected the expanded size after the command:

Code: Select all

cahtoh02:/root%zpool get all rpool | more   
NAME   PROPERTY       VALUE                 SOURCE
rpool  allocated      333G                  -
rpool  altroot        -                     default
rpool  autoexpand     on                    local
rpool  autoreplace    off                   default
rpool  bootfs         rpool/ROOT/solaris-6  local
rpool  cachefile      -                     default
rpool  capacity       35%                   -
rpool  dedupditto     0                     default
rpool  dedupratio     1.00x                 -
rpool  delegation     on                    default
rpool  failmode       wait                  default
rpool  free           595G                  -
rpool  guid           15585389463118454899  -
rpool  health         ONLINE                -
rpool  listshares     off                   default
rpool  listsnapshots  off                   default
rpool  readonly       off                   -
rpool  size           928G                  -
rpool  version        35                    default

cahtoh02:/root% df -h 
Filesystem             Size   Used  Available Capacity  Mounted on
rpool/ROOT/solaris-6   914G    15G       580G     3%    /
/devices                 0K     0K         0K     0%    /devices
/dev                     0K     0K         0K     0%    /dev
ctfs                     0K     0K         0K     0%    /system/contract
proc                     0K     0K         0K     0%    /proc
mnttab                   0K     0K         0K     0%    /etc/mnttab
swap                   1.9G   1.6M       1.9G     1%    /system/volatile
objfs                    0K     0K         0K     0%    /system/object
sharefs                  0K     0K         0K     0%    /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap2.so.1
                       596G    15G       580G     3%    /lib/libc.so.1
fd                       0K     0K         0K     0%    /dev/fd
rpool/ROOT/solaris-6/var
                       914G   7.0G       580G     2%    /var
swap                   1.9G    56K       1.9G     1%    /tmp
rpool/VARSHARE         914G   121M       580G     1%    /var/share
rpool/export           914G    32K       580G     1%    /export
rpool/export/home      914G   136G       580G    19%    /export/home
rpool/export/home/cah
                       914G   103G       580G    16%    /export/home/cah
rpool/export/home/tcl
                       914G    58G       580G    10%    /export/home/tcl
rpool                  914G   5.0M       580G     1%    /rpool
rpool/VARSHARE/zones   914G    31K       580G     1%    /system/zones
rpool/VARSHARE/pkg     914G    32K       580G     1%    /var/share/pkg
rpool/VARSHARE/pkg/repositories
                       914G    31K       580G     1%    /var/share/pkg/repositories