Skip to content

Conversation

@russellbrooks
Copy link
Contributor

Found this while some teammates were spinning up a Metaflow stack as part of a hackathon:

When deploying the AWS CloudFormation stack, if the SageMaker Notebook param is enabled there will be an error:

The following resource(s) failed to create: [SageMakerNotebookInstance]. Rollback requested by user.

Notebook Instance Lifecycle Config 'arn:aws:sagemaker:us-east-1:xxx:notebook-instance-lifecycle-config/basicnotebookinstancelifecycleconfig-s8ydcbss7bcb' for Notebook Instance 'arn:aws:sagemaker:us-east-1:xxx:notebook-instance/hackathon2023-notebookinstance-xru98eyd' took longer than 5 minutes. Please check your CloudWatch logs for more details if your Notebook Instance has Internet access.

Digging into the SageMaker Notebook instance logs, we can find it's from this:

/tmp/OnCreate_2023-01-24-18-3833l4zk30: line 8: initctl: command not found

This looks to be due to SageMaker swapping from Amazon Linux 1 to Amazon Linux 2, with further discussion here aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples#76

After updating the lifecycle config for the new command, the stack was created successfully and notebook instance had the expected metaflow params preconfigured.

@savingoyal savingoyal merged commit 49199f3 into outerbounds:master Jan 25, 2023
@russellbrooks russellbrooks deleted the Fix-SageMaker-Notebooks-for-AL2-AMI branch January 25, 2023 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants