CodeDeploy agent does not use force option of ruby symlink filetools

[hegyre]

Hello team,

We are facing an issue where CodeDeploy is unable to OVERWRITE symlinks during an in-place deployment.

This is hapening since we use --file-exists-behavior "OVERWRITE" option in the deploy script (aws deploy create-deployment).

I've digged, and:

This is the error I got from CodeDeploy:

2017-11-29 18:14:43 ERROR [codedeploy-agent(3901)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: Errno::EEXIST - File exists - (../squizlabs/php_codesniffer/bin/phpcs, /var/www/vhosts/staging/httpdocs/vendor/bin/phpcs) - /usr/share/ruby/fileutils.rb:349:in `symlink'
/usr/share/ruby/fileutils.rb:349:in `block in ln_s'
/usr/share/ruby/fileutils.rb:1574:in `fu_each_src_dest0'
/usr/share/ruby/fileutils.rb:347:in `ln_s'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/install_instruction.rb:235:in `execute'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:49:in `block (2 levels) in install'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:48:in `each'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:48:in `block in install'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:47:in `open'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:47:in `install'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:136:in `block in <class:CommandExecutor>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:68:in `execute_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:134:in `process_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:67:in `perform'
/opt/codedeploy-agent/lib/instance_agent/agent/base.rb:28:in `run'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:38:in `block in run'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:55:in `with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:37:in `run'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:70:in `block in run_with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:55:in `with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:69:in `run_with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:33:in `block in start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:22:in `loop'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:22:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:in `block in spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:196:in `block in spawn_children'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:195:in `times'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:195:in `spawn_children'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:134:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:37:in `block in start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:36:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:36:in `start'
/opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:43:in `block (2 levels) in <main>'
/opt/codedeploy-agent/vendor/gems/gli-2.11.0/lib/gli/command_support.rb:126:in `call'
/opt/codedeploy-agent/vendor/gems/gli-2.11.0/lib/gli/command_support.rb:126:in `execute'
/opt/codedeploy-agent/vendor/gems/gli-2.11.0/lib/gli/app_support.rb:284:in `block in call_command'
/opt/codedeploy-agent/vendor/gems/gli-2.11.0/lib/gli/app_support.rb:297:in `call'
/opt/codedeploy-agent/vendor/gems/gli-2.11.0/lib/gli/app_support.rb:297:in `call_command'
/opt/codedeploy-agent/vendor/gems/gli-2.11.0/lib/gli/app_support.rb:79:in `run'
/opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:90:in `<main>'
2017-11-29 18:14:44 DEBUG [codedeploy-agent(3901)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2017-11-29 18:14:44 INFO  [codedeploy-agent(3901)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1352_rpm.

In /usr/share/ruby/fileutils.rb:349, we have:

 def ln_s(src, dest, options = {})
    fu_check_options options, OPT_TABLE['ln_s']
    fu_output_message "ln -s#{options[:force] ? 'f' : ''} #{[src,dest].flatten.join ' '}" if options[:verbose]
    return if options[:noop]
    fu_each_src_dest0(src, dest) do |s,d|
      remove_file d, true if options[:force]
      File.symlink s, d
    end
  end

In /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/install_instruction.rb:235, we have:

def execute(cleanup_file)
          # NO need to check if file already exists in here, because if that's the case,
          # the CopyCommand entry should not even be created by Installer
          cleanup_file.puts(@destination)
          if File.symlink?(@source)
            FileUtils.symlink(File.readlink(@source), @destination)
          else
            FileUtils.copy(@source, @destination, :preserve => true)
          end
        end

The comments clearly shows that It's not expected to have a CopyCommand here.

I think that this def was written when the option --file-exists-behavior "OVERWRITE" was not yet implemented.

Now that we can use this option, you should add :force => true to the symlink parameter, like this:

def execute(cleanup_file)
          # NO need to check if file already exists in here, because if that's the case,
          # the CopyCommand entry should not even be created by Installer
          cleanup_file.puts(@destination)
          if File.symlink?(@source)
            FileUtils.symlink(File.readlink(@source), @destination, :force => true)
          else
            FileUtils.copy(@source, @destination, :preserve => true)
          end
        end

As a last word, I specify here that the Deployment is a new deploy and application and thus it does not have the "history" of a previous one. That is why I use option --file-exists-behavior "OVERWRITE"

[huckphin]

This needs to be in place. We are running into it. Do you want to me open up a PR for this?

[hegyre]

Hello Huckphin,

That would be great !

However I saw that in fact it should be fixed, but only if --file-exists-behavior is set to OVERRIDE

[huckphin]

@hegyre That is good information. We are consistently using the --file-exists-behavior OVERRIDE due to how I have setup our deployments, but I am only seeing this behavior on one of our services.

One thing that is very abnormal to all of our services is that we have consolidated all of our frontend code from six different services into one service. There was a lot of duplication across our services, and we are going to pare it all down, but our deployment artifact is very large: 600 MB. This is orders of magnitude larger than any other service that we are using. For the record, at the moment, I believe that this is a red herring, and not the real issue.

The other thing that could be related is that I have attached a 100 GB ebsmount to each member of the cluster that requires these large tarballs to be deployed. To accomodate this, I have setup a symlink:

/opt/codedeploy-agent -> /ebsmount/codedeploy-agent

I am thinking that this could be the cause, but, I have performed this same routine for a handful of our other services that required it, and they seem to be deploying without encountering this issue.

Also, on troubleshooting this, I have attempted to make it work by deleting the CodeDeploy application and the CodeDeploy deploymentgroup for this cluster of servers to no effect.

Maybe...I need to install a new version of the CodeDeploy agent? Is there an easy way to do that?

Hope this helps.

[hegyre]

@huckphin ,

About symlink /opt/codedeploy-agent to an EBS mount, I don't think there can be an issue there, because the issue is when codedeploy agent operate at the "install" stage to the destination directory which is not /opt/codedeploy-agent

Our artifact is about 600 MB too because we are deploying lots of different files for differents countries, but the size should not be an issue

About the new version of the agent, we are already using the verson 1.0.1.1352
https://docs.aws.amazon.com/codedeploy/latest/userguide/codedeploy-agent.html#codedeploy-agent-supported-versions

Here's the page to update the agent: https://docs.aws.amazon.com/codedeploy/latest/userguide/codedeploy-agent-operations-update.html

At least, we have made a work-around by using a beforeInstall script which removes the destination directory before the "install" event by the codedeploy agent. So there is no more orphaned symlinks. But it would be good to have it done natively by the agent itself

[huckphin]

@hegyre

Nevermind my last suggestions. @hegyre you are right it was not related to the Symlink issue at all. That was a red herring. The issue was related to how I was provisioning the mounts, completely unrelated to anything with the codedeploy-agent.

I stopped using snapshots to provision the volume, and the issue went away.

[annamataws]

I am adding this to our task list.

[jasonbigl]

now is 2022 and this is still a bug in the latest codedeploy agent

I have to use the BeforeInstall hook to delete all my symbolic link files to avoid this problem, please merge the pull...