SUN SPARC T3-2 OS installation notes

[cah]

When I was trying to install OS on T3-2, I issued the following command from the OK prompt:

Code: Select all

boot cdrom

Instead of reading from DVD, it tried to reach network to install.

I then checked devaliases from OK prompt and found "dvd" is one of the devices and there's another "rcdrom" on the list.

I then tried:

Code: Select all

boot dvd

And, it read from the DVD.

This is new to me. It had always been "boot cdrom" even T3-1 but I had to use "boot dvd" on T3-2.

After the OS is installed, I tried to install the companion software. After inserting the disk into the DVD drive, I was able to cd to /cdrom and found cdrom1 is the new name. The old name is cdrom0.

I don't know why there are such changes but I figure it is worth noting these differences for future reference.

[cah]

When I was working on T3-2 to get the LUN from StorageTek 2540, T3-2 saw 2 disks (c1t0d1 and c4t2d1). Both show 2.7TB.

I knew it can not possibly use both because I have RAID 1 on the disk array and the capacity should be just 2.7TB.
I purposely tried to put both disks into zfs pool and it failed, of course.

I then opened a ticket with Oracle/SUN asking about this.
I was told by the tech support that I did not enable the multipathing on fiber channel controllers.
He introduced me to "stmsboot -e" command. I checked on the man page and tried it:

Code: Select all

%stmsboot -e 

WARNING: stmsboot operates on each supported multipath-capable controller
         detected in a host. In your system, these controllers are

/pci@400/pci@2/pci@0/pci@8/SUNW,emlxs@0/fp@0,0
/pci@500/pci@2/pci@0/pci@a/SUNW,emlxs@0/fp@0,0
/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@1
/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@2
/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@4
/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@8
/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@80

If you do NOT wish to operate on these controllers, please quit stmsboot
and re-invoke with -D { fp | mpt | mpt_sas} to specify which controllers you wish
to modify your multipathing configuration for.

Do you wish to continue? [y/n] (default: y) n

After I saw so many controllers could be impacted, I canceled the action.
After reading the man page further, I decided to enable multipathing on fiber channels only by issuing the following command:

Code: Select all

%stmsboot -D fp -e
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y) 
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y) 
updating /platform/sun4v/boot_archive

It rebooted the server.
After the reboot, I tried "format" command again and now it shows:

Code: Select all

%format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c3t3EDC40EB4E13BFC7d0 <LSI-Logical Volume-3000-278.46GB>
          /pci@400/pci@2/pci@0/pci@e/scsi@0/iport@v0/disk@w3edc40eb4e13bfc7,0
       1. c3t3718410C46D7BFE8d0 <LSI-LogicalVolume-3000 cyl 65533 alt 2 hd 64 sec 139>
          /pci@400/pci@2/pci@0/pci@e/scsi@0/iport@v0/disk@w3718410c46d7bfe8,0
       2. c9t600A0B800075F94F000002D94DBEC363d0 <SUN-LCSM100_F-0735-2.73TB>
          /scsi_vhci/ssd@g600a0b800075f94f000002d94dbec363
Specify disk (enter its number)[0]: 2
selecting c9t600A0B800075F94F000002D94DBEC363d0
[disk formatted]
format> p


PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        !<cmd> - execute <cmd>, then return
        quit
partition> p
Current partition table (original):
Total disk sectors available: 5855353822 + 16384 (reserved sectors)

Part      Tag    Flag     First Sector          Size          Last Sector
  0        usr    wm               256         2.73TB           5855353822    
  1 unassigned    wm                 0            0                0    
  2 unassigned    wm                 0            0                0    
  3 unassigned    wm                 0            0                0    
  4 unassigned    wm                 0            0                0    
  5 unassigned    wm                 0            0                0    
  6 unassigned    wm                 0            0                0    
  8   reserved    wm        5855353823         8.00MB           5855370206    

Notice: The device's target number "600A0B800075F94F000002D94DBEC363" is the volume's WWN from the StorageTek 2540 disk array:

Code: Select all

World Wide Name:
60:0A:0B:80:00:75:F9:4F:00:00:02:D9:4D:BE:C3:63

Then, I tried to create a new pool oracledatapool and use this newly found device:

Code: Select all

zpool create oracledatapool c9t600A0B800075F94F000002D94DBEC363d0

Check if it created the pool and mount point:

Code: Select all

%df -h /oracledatapool
Filesystem             size   used  avail capacity  Mounted on
oracledatapool         2.7T    21K   2.7T     1%    /oracledatapool

The multipathing is now enabled and the volume / LUN has been recognized and put into a newly created ZFS pool.

[cah]

After attaching the expansion tray (tray 0) with 12 disks in it to the disk array controller, we saw a disk failure and it delayed the virtual disk creation by 2 weeks (poor warranty service from Partners Data in my opinion).

I was able to replace drive #5 last Friday (08/24/2012). I created the virtual_disk_2 right after the successful drive replacement.

When I was working on it today (08/27/2012), I realized I need to create a new volume to get the WWN number that will be detected by OS later on. I then created a new volume (Oracle_Data_Volume_2) and change the existing volume to Oracle_Data_Volume_1. I then mapped the volume to the Default Storage Domain as LUN 2.

I then used 'devfsadm" to try to get the new volume/virtual disk detected but it didn't see the new volume from Solaris.

I then have to try 'devfsadm -C' and then format shows the new disk (#3 at the bottom of the list):

Code: Select all

       0. c3t3EDC40EB4E13BFC7d0 <LSI-Logical Volume-3000-278.46GB>
          /pci@400/pci@2/pci@0/pci@e/scsi@0/iport@v0/disk@w3edc40eb4e13bfc7,0
       1. c3t3718410C46D7BFE8d0 <LSI-LogicalVolume-3000 cyl 65533 alt 2 hd 64 sec 139>
          /pci@400/pci@2/pci@0/pci@e/scsi@0/iport@v0/disk@w3718410c46d7bfe8,0
       2. c9t600A0B800075F94F000002D94DBEC363d0 <SUN-LCSM100_F-0735-3.27TB>
          /scsi_vhci/ssd@g600a0b800075f94f000002d94dbec363
       3. c9t600A0B800075F93600000342503B498Cd0 <SUN-LCSM100_F-0735-3.27TB>
          /scsi_vhci/ssd@g600a0b800075f93600000342503b498c

From here, I should be able to run the following zpool command to add it to existing oracledatapool:

Code: Select all

zpool add oracledatapool c9t600A0B800075F93600000342503B498Cd0 

SUN Storage Common Array Manager is showing the new virtual disk being initializing for hours. I am not sure if I can go ahead and add the new volume to the pool or just want until the initialization is completed. Since there's no urgency on this, I would prefer to wait.

NOTICE: Be very careful with 'attach' command under zpool. 'attach' add disks into the mirror [RAID-1] while 'add' is to concatenate/strip disk(s) to existing pool [RAID-0].

[cah]

Came in this morning and saw another ticket was opened by ASR saying drive 05 in tray 0 may be failing.

Oracle is going to send us another disk shortly.
After replacing the failing disk, I will then add the virtual disk volume to oracledatapool.

[cah]

The replacement drive arrived around noon.
I had a PPM meeting at noon so I didn't get a chance to work on it until the meeting is over.

From the past experience, it will take about 3 hours to reconstruct the new drive.
The drive was replaced around 1 PM. The reconstruction should be done around 4 PM.
If so, I should be able to add the new virtual disk volume to oracledatapool and make it 6 TB!

In fact, it completed the reconstruction by 3 PM.
It took just about 2 hours. Nice.

[cah]

After the new drive reconstruction was done, I added it to existing zpool - oracledatapool:

Code: Select all

orazone01% zpool add oracledatapool c9t600A0B800075F93600000342503B498Cd0

Checking zpool status:

Code: Select all

orazone01% zpool status oracledatapool
  pool: oracledatapool
 state: ONLINE
 scrub: none requested
config:

        NAME                                     STATE     READ WRITE CKSUM
        oracledatapool                           ONLINE       0     0     0
          c9t600A0B800075F94F000002D94DBEC363d0  ONLINE       0     0     0
          c9t600A0B800075F93600000342503B498Cd0  ONLINE       0     0     0

errors: No known data errors

Checking zfs:

Code: Select all

orazone01% zfs list oracledatapool
NAME             USED  AVAIL  REFER  MOUNTPOINT
oracledatapool  1.07T  5.32T    21K  none

Checking zpool - oracledatapool:

Code: Select all

orazone01% zpool list oracledatapool
NAME             SIZE  ALLOC   FREE    CAP  HEALTH  ALTROOT
oracledatapool  6.50T   333G  6.17T     5%  ONLINE  -

Getting all zpool properties:

Code: Select all

orazone01% zpool get all oracledatapool
NAME            PROPERTY       VALUE       SOURCE
oracledatapool  size           6.50T       -
oracledatapool  capacity       5%          -
oracledatapool  altroot        -           default
oracledatapool  health         ONLINE      -
oracledatapool  guid           2777870903478614918  default
oracledatapool  version        22          default
oracledatapool  bootfs         -           default
oracledatapool  delegation     on          default
oracledatapool  autoreplace    off         default
oracledatapool  cachefile      -           default
oracledatapool  failmode       wait        default
oracledatapool  listsnapshots  on          default
oracledatapool  autoexpand     on          local
oracledatapool  free           6.17T       -
oracledatapool  allocated      333G        -

The disk array is now expanded to its max capacity with 2 trays.
We are ready for data migration!