NFS Migration (Celerra/VNX to ISILON) – the troubleshooting part

Recently I had a post on the NFS migrations from Celerra/VNX systems to Isilon array. This post is an extension to that, describing how to tackle the common errors/issues during the migration activities. I had to try different things in troubleshooting the issues, but sometimes, we may not be in a position to spend a lot of time in troubleshooting. Hence I thought of putting this post to listing some of the issues, I faced – you may come across.

Continue reading

First of all, the most common issue is with the checkpoint creation on the source (Celerra/VNX).

If you are getting a message similar to ‘msg SnapSure file system creation failed’ during any copy session, the command is failing to create a snapshot of the source filesystem. It could be due to many reasons including space shortage. You may try creating a manual snapshot on the source FS by logging into the source system. You will have to troubleshoot this issue if the manual checkpoint creation fails. free up some space if space is the issue. You may try the below command for a manual checkpoint creation on the source system.

#fs_ckpt Test_FS -name Test_FS-ckpt01 -Create

Next, Permission issues… I know I don’t have to mention it here. Mostly, from the error message itself it will be clear what you have to check. Ensure you are using the correct credentials for the NDMP user in the migration command. This user should have sufficient rights on the source system. You must have sufficient privileges on the target system you are logged on. If not the super user, the logged in user should be capable of creating and modifying the directories and their contents.

Another one could be related to the snapshot creations on the target – Isilon array. Snapshot creation may fail due to many issues as said above including space issue. In the below example, the snapshot creation failed as there is an existing snap with the same name.

ISILON568# isi_vol_copy_vnx Cel097-dm2:/Test_FS/NFS01 /ifs/data/Test_NFS01 -sa ndmp:NDMPpass -incr
Snapshot with conflicting name ‘isi_vol_copy.011.1.snap’ found. Remove/Rename the conflicting snapshot to continue with further migration runs.
snapshot already exists
ISILON568-1#

We can either delete or rename the existing snap to resolve this issue. Here it was safe to delete the existing snap, and so I did the same.

ISILON568-1# isi snapshot snapshots list| grep isi_vol_copy.011
134 isi_vol_copy.011.0.snap /ifs/.ifsvar/modules/isi_vol_copy/011/persistent_store
136 isi_vol_copy.011.1.snap /ifs/.ifsvar/modules/isi_vol_copy/011/persistent_store
ISILON568-1#

ISILON568-1# isi snapshot snapshots delete –snapshot=134
Are you sure? (yes/[no]): y
ISILON568-1#

Other issues could be anything related to network connectivity. The source and target systems should be able to communicate for a successful migration. Similarly, an ongoing migration session may fail due to network disruptions also.

I will update this post, if I find more similar issues during the migration sessions. Feel free to have your suggestions/questions in the comments section below.

Dell EMC Elect 2017 – Don’t be late, nominate now

Dell had their TechCenter Rockstar program running and EMC had the EMC Elect program to appreciate the social contributions. From this year, following the merger, Dell EMC have combined these programs to form the all new Dell EMC Elect program.

Wanna be a founding member of the Dell EMC Elect program? If you believe your (or your peer’s) contributions deserve to be recognised, nominate now.

Continue reading

Dell Rockstar and EMC Elect were two of the leading community recognition programs in the industry.  VIP treatment at major Dell EMC events (even though I’ve never been to EMC World before), blogger briefing sessions, goodies etc… are some of the attractions of the Elect program. With the new edition, the expectations are really high.

Here’s the link for you to nominate self or your peers. Your contributions will be validated by an expert panel formed by EMC Elect founders and members.

Time is running out. Nominations are open until 17th March only.

VMware vExpert 2017 announced

VMware, a few hours back announced the batch of 2017 vExperts. I am honoured to be a member of this group, fourth year in a row for me.

This year, there are 2017 members in the group. I would like to congratulate all my fellow members..!

I am sure that there’s no need for an introduction about the program for anyone. But, if you want to check few of the program benefits or FAQs you may find it here.

You may find the full list of 2017 members and more details here in the announcement.

NFS migration from Celerra/VNX to ISILON

I was working on a few NFS migrations from our Celerra/VNX systems to the ISILON systems. I thought of putting a post with the steps and gotchas in a Celerra-ISILON migration. It took so long, and now it is the time for that post.

Our first step should be to identify the hosts accessing the export and to understand the type of applications running on the export. We should work with server/application teams for this and for finding the suitable window for a possible migration window. The size of the exports, network parameters, and the nature of applications being used – we should take many things into consideration as we do the pre-work of the migration. ISILON by default, do not support 32-bit exports running on the NFS exports and this need a special bit setting to be enabled. So, this is one thing we should be ready for as we plan the migration. We will discuss how this is done as we discuss the actual migration steps.

Once we are done with all the pre-works, we will start the migration steps. To reduce the downtime window during the actual data migration, we will run a pre-copy of the data to the target ISILON array.

Continue reading

Create the NFS export directory and a quota on the target ISILON array. While setting the quota limits, we should consider the source FS size. Here I am assuming a 100 GB source FS and setting 90GB advisory threshold on the target directory.

mkdir Test_NFS01

isi quota quotas create /ifs/data/Test_NFS01 directory –advisory-threshold 90GB –hard-threshold 100GB –container yes

You can try listing the quota and it should be showing 0GB as the current utilisation. Now initiate a full copy of the data from the source (Celerra/VNX) export to the newly created directory.

isi_vol_copy_vnx Cel097-dm2:/Test_FS/NFS01 /ifs/data/Test_NFS01 -sa NDMPuser:NDMPpass -full

The isi_vol_copy_vnx utility creates a snapshot of the source Filesystem and initiates data transfer from it. The NDMP credentials on the array need to be used in the command in the place of NDMPuser and NDMPpass. Once done, we have one full copy of the data on the target array. You can now list the quota again and you should be seeing the change in utilisation now.

isi quota quotas list | grep NFS01

Now we are all set for the migration window. (Note: In case if the number of days between full-copy and the actual cutover is high – maybe due to a change in schedule etc… – you may run another incremental copy with incr switch a few hours prior to the cut-over. This will ensure the delta will be less and will help in completing the migration faster.)

The application team will start the migration proceedings by stopping the application instances running on the export. The server team will unmount the exports from their end after all the applications are stopped. We can un-export the source export and this will ensure there are no new writes happening to the export being migrated.

server_export server_x  -unexport -perm /Test_FS/NFS01

We will initiate a final incremental copy of the export to migrate the changes from the source export.

isi_vol_copy_vnx Cel097-dm2:/Test_FS/NFS01 /ifs/data/Test_NFS01 -sa NDMPuser:NDMPpass -incr

After the successful completion of the data transfer, we are good to create a new export on the target directory. We will add the export permissions as in the source export. We will create an NFS alias for the export for our convenience. NFSZone in the below commands needs to be replaced with NFS Zone on your ISILON array.

isi nfs exports create /ifs/data/Test_NFS01 –zone NFSZone –root-clients host1, host2 –read-write-clients host1, host2 –read-only-clients host5 –description Test_migration -f

isi nfs alias create /Test_NFS01 /ifs/data/Test_NFS01 –zone NFSZone

Now the 32 bit part. As discussed already, if your application is 32 bit, you have to enable 32-bit settings on the new NFS export.

isi nfs exports modify EXPID –zone=NFSZone –return-32bit-file-ids=yes

Replace the EXPID with the ID of the target export. You can now verify the bit settings by viewing the export.

isi nfs exports view EXPID –zone=NFSZone | grep -i return

We are now done with the migration. The server team can now mount the new export on the servers and application can be run on the new export.

Next… ?

You may read our post on some of the common issues in migration and their troubleshooting steps.

Time for a change – a new beginning


I am very pleased to announce that I have joined the Dell EMC’s Converged division today.

The converged division is one of the areas of toughest competition these days and it is very important for each vendor to deliver their best to remain in the competition. Dell EMC is the leader in converged systems sector with 43% share of the total market revenue as per the Q2 results from IDC. It will be very interesting to work with the leaders and will be equally challenging in the effort for continuous improvement.

In my previous roles, I was working closely with EMC as a partner and later as a customer.

The company has gone through the biggest transformation following the merger with Dell which was completed a few months ago. The merger announcement took the entire industry by surprise and remained a hot topic for many months. But all those concerns had no place as we have seen one of the smoothest mergers in IT. The company recently at the Dell EMC World, announced their new delivery model with Dell servers taking the compute part of their HCI systems. This will further accelerate the growth and will offer cost-efficiency for the customers.

Looking forward to the new challenges and opportunities with the new role. I would request all your prayers and support for me for my new assignment with Dell EMC Converged.