Parallelcluster 2.1.1 with raid 0 config on Cent OS 7 fails in create cluster

[ahmedelz]

Environment:

• AWS ParallelCluster 2.1.1
• OS: Cent OS 7
• Scheduler: SGE
• Master instance type: m5.large
• Compute instance type: m5.xlarge

Bug description and how to reproduce:
Deploying a ParallelCluster 2.1.1 with Raid 0 configuration fails with this error.

Beginning cluster creation for cluster: cluster1
Creating stack named: parallelcluster-cluster1
Status: parallelcluster-cluster1 - ROLLBACK_IN_PROGRESS
Cluster creation failed.  Failed events:
  - AWS::EC2::Instance MasterServer Received FAILURE signal with UniqueId i-0ecca142dxxxxx

I thought the failure could be because I'm using encrypted EBS volumes with custom KMS key but I commented out both encrypted and ebs_kms_key_id settings but still the same failure.

Additional context:
Any other context about the problem. E.g.:

• configuration file without any credentials or personal data.

[global]
update_check = true
sanity_check = true
cluster_template = default

[aws]
aws_region_name = us-west-2

[cluster default]
vpc_settings = vpc-0094xxxxx
key_name = cdns-cluster
base_os = centos7
compute_instance_type = m5.2xlarge
master_instance_type = m5.large
#compute_root_volume_size = 20
#master_root_volume_size = 20
initial_queue_size = 0
tags = {"BU" : "IT", "Sub_BU" : "IT"}
raid_settings = rs
#extra_json = { "cluster" : { "ganglia_enabled" : "yes" } }

[vpc vpc-0094xxxxx]
vpc_id = vpc-0094xxxxx
master_subnet_id = subnet-06cxxxxxx
use_public_ips = false
ssh_from = 172.16.0.0/12

[raid rs]
shared_dir = raid
raid_type = 0
num_of_raid_volumes = 2
volume_size = 100
encrypted = true
ebs_kms_key_id = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

When I created the cluster with --norollback option, I can see that the master has a 20GB disk mounted and exported under /shared and also noticed that the 2 disks for the raid0 configuration are not attached to the master.

Attachments:
cfn-init.log
cloud-init.log

[sean-smith]

Thanks for the detailed bug report, problem is the attachment point /dev/sdb already has a device attached:

botocore.exceptions.ClientError: An error occurred (InvalidParameterValue) when calling the AttachVolume operation: Invalid value '/dev/sdb' for unixDevice. Attachment point /dev/sdb is already in use

We get the device from: https://github.com/aws/aws-parallelcluster-cookbook/blob/develop/files/default/attachVolume.py#L43

It appears that line is failing to get the devices currently running on Centos7.

[ahmedelz]

Yes, /dev/sdb has that 20GB disk that wasn't expected to be there in the first place.

[sean-smith]

Hi,

This bug only effects NVME based instances, c5 and m5, so as a temporary workaround you can use a non-nvme based instance such as M4.

We've patched the issue in aws/aws-parallelcluster-cookbook#253 which will be part of parallelcluster in the next release. Thanks for the bug report!

[ahmedelz]

I changed master_instance_type to m4.large but got similar failure for MasterServer:

AWS::EC2::Instance MasterServer Received FAILURE signal with UniqueId i-0840c166xxxxx

[sean-smith]

Please re-create the cluster with the --norollback flag and upload the /var/log/cfn-init.log file.

I was able to launch an m4 based Raid 0 cluster using parallelcluster 2.1.1 with no issues.

[ahmedelz]

I attached cfn-init.log

I see two problems:

1. A 20 GB disk is still created and attached to the master on /dev/sdb. This is not expected because my configuration doesn't have any ebs_settings section.
2. Per cfn-init.log, it wasn't able to access the KMS key while attaching the 2 RAID encrypted disks to Master. The disks are encrypted with the correct KMS key. I was able to attach the disks to Master from the AWS console to the Master instance on /dev/sdc, /dev/sdd.

cfn-init.log

[sean-smith]

Hi,

1. Is expected, we mount a /shared ebs volume by default where we mount /opt and /home in order to share ssh config, scheduler setup ect.
2. Can you retry without the kms key and encrypted flag?