Steve and I spent more time with IBRIX folks and we got the restore started.
If the machines do not crash, we are in really good shape.

Because, we can compare the current files with the pristine backup to see if they got corrupted.

I firewalled off the machines so that only AdminVPN, subnet 64.0 can get into the machine while restoring in progress.

Things to save for a long time that might be useful to refer to (the ib-fsck, dsmsched.log).
If it crashes again, we will likely need to format segment 4 and hopefully succeed.

Ran these from /state/partition1/:
dsmc restore -subdir=yes -replace=no /ibrixfs1/shared/ > ./ibrixfs1_shared.log 2>&1
dsmc restore -subdir=yes -replace=no /ibrixfs1/home/ > ./ibrixfs1_home.log 2>&1
dsmc restore -subdir=yes -replace=no /ibrixfs1/software/ > ./ibrixfs1_software.log 2>&1


Couple remaining things:
1) 2 drives were left unplugged (because they were suspects, I will call the person I worked with Dell to see if it's safe or should I plug back in).
    Or should I plug in the new drives. Since in PV220s, bad drives can cause huge issues.
2) I renamed /ibrixfs1/home/karroje to /ibrixfs1/home/karroje_donotouch. 
     karroje can still access some of those (with the exception of that one).

Attached are the logs for ib-fsck results and last tivoli backup logs. We should keep it somewhere -- if we were to refer to the pristine backup we saved.


Robin