Add Support for "Overwrite" instruction in appspec.yml "Files" section

[colinbjohnson]

Feature Request:

Would like the ability to have an "overwrite" instruction in the appspec.yml "Files" section.

Use:

In my initial testing, I wanted to clobber all files in /var/www/html with new files. This resulted in a failure because the files already existed. Specifically, the following failure: 2015-06-13 20:58:40 ERROR [codedeploy-agent(7589)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: RuntimeError - File already exists at location /var/www/html/index.html - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:113:ingenerate_normal_copy'`

Desired Behavior:

I'd like to be able to create an appspec.yml file with the following section:
files:
- source: web
destination: /var/www/html/
overwrite: true
Notice I've added the "overwrite" instruction, which is currently not offered.

Current Workaround:

I've been working around by utilizing the hook section to do file copies. The hooks/shell script workaround is sufficient, rendering this a lower priority feature request.

[surybala]

Hi,
Thanks for the feedback. I've added this feature request to our backlog.

[CNG]

I would also love this feature.

[madisonmay]

Add one more name to the list.

[kingdango]

Of course we need this feature!
But it should not be simply overwrite: true -- it should allow for a few options:
overwrite: none (default)
overwrite: purge
overwrite: merge

Purge removes all files in the destination before performing copying the new files.

Merge leaves the destination as-is, adds the files from source and will replace an files in the destination that also exist in the source.

As for now I guess I need to stage my source files in a temp location and then run a script to achieve this. 🐼 😢

[bslatner]

+1 for this feature as well.

Side note: the Windows agent does NOT exhibit this behavior. I've been writing several appspecs for Windows services and I've never run into this. Only when we went to deploy to an Ubuntu server for the first time did we see this behavior.

[JoshuaEstes]

+1

[etcsayeghr]

+1

[ixaxaar]

+1

[vitalyzeme]

+1

[praddc]

+1

[gabriellhrn]

+1

[JoshuaEstes]

So, I hope this helps other people find a solution. In my situation I work a lot with Magento so I deal with PHP, nginx. Composer is also tossed in the mix. I found a solution that works for me and for my case. If anyone has any comments or suggestions, please let me know.

---

I wanted to be have the production directory synced with the repository that I was using so any file modifications would be reflected. Not just updates to files, but delete the files that are no longer required. I think this is what everyone really wants. The solution I found was to deploy the code into a temporary directory (/var/www/release) and then rsync that directory over to the production directory (/var/www/magento).

appspec.yaml

version: 0.0
os: linux
files:
    - source: ./
      destination: /var/www/release/
permissions:
    - object: /var/www/magento/
      owner: www-data
      group: www-data
hooks:
    BeforeInstall:
        - location: scripts/beforeInstall.bash
    AfterInstall:
        - location: scripts/afterInstall.bash

scripts/beforeInstall.bash

#!/usr/bin/env bash

# I want to make sure that the directory is clean and has nothing left over from
# previous deployments. The servers auto scale so the directory may or may not
# exist.
if [ -d /var/www/release ]; then
    rm -rf /var/www/release
fi
mkdir -vp /var/www/release

scripts/afterInstall.bash

#!/usr/bin/env bash

# I have left a few things in here and will explain this further (see below)
rsync --delete-before --verbose --archive --exclude "htdocs/media/" --exclude ".*" --exclude "htdocs/var/" --exclude "htdocs/app/etc/local.xml" /var/www/release/ /var/www/magento/ > /var/log/deploy.log

The rsync command is what allows you to overwrite all the files you need and it also deletes files if they are deleted from the repository. In my case I am deploying a git repo and there are times when extensions/files are removed and those need to be removed on the server.

There are also directories that are excluded from the rsync command. You do not need to sync cache directories and in my case the htdocs/media directory is a mounted S3 bucket and there are lots of files we do not need to delete.

All of the output is put into a deployment log (/var/log/deploy.log) file that I can use in case something goes wrong and can see the exact files deleted and synced over. When I was first experimenting with this, I ran rsync in dry-run mode and checked the log file to make sure files were synced correctly. I HIGHLY recommend you do the same.

---

Some things I want to try when I make the time, is to rsync files directly from the code-deploy directory. For each deployment, AWS CodeDeploy puts the code into a specific directory which you can reach by some of the environment variables it sets. This should speed up the deploy a little since it isn't putting the files in two places.

There are some things that have been left out of the scripts. Certain directories are chowned with www-data user/group since the files are served used nginx. You may need something similar. The code that places Magento into maintenance mode and removes it from maintenance mode has also been removed since this needs to be generic. You may need similar logic.

---

You can read more about the various hooks and when they occur in the docs at http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref.html and also see the environment variables that can be used within the scripts.

[maorhayoun]

+1

[sverdier]

+1

[aronhaltamz]

+1

[TKLaracuente]

@JoshuaEstes You beautiful SOB, this is exactly what I needed.

[jody-greene]

+1

[JoshuaEstes]

Thanks, glad I could help!

[cesc1989]

+1

[lperezdude]

+1

[oswmilanez]

+1

[oswmilanez]

+1

[knick-burns]

This is a good workaround. The issue I have is with Amazon and the fact that it's been over a year and yet this feature is still not implemented.

[Saeven]

+1. It's a must.

[agutierrezl]

+100

[ashancock]

+1

[ratancs]

any update on this feature?

[GoodMirek]

+1

[jonpchin]

Any update on this?

[engineerakki]

@philstrong
Was this picked up in your sprint ?
Is there any official timeline for this feature ?

[quentinhayot]

Is this really happening ?!!!
2021 WILL be a good year after all ! :')

[dreaddesign]

if this is fixed, i would be very happy. :) +1

[philstrong]

We’re going to investigate and report back

…

On Wed, Mar 10, 2021 at 2:35 PM Eric Mckee ***@***.***> wrote: if this is added, i would be very happy. :) +1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUCBC55MKX2QDM4AABLQ3TC63XHANCNFSM4BIIWZCQ> .

[philstrong]

This will be coming soon in the next release of the agent. @chrisdibble did all of the work so he can explain the feature/fix a little better.

[waldoj]

Great news! Thanks so much for your attention to this.

[chrisdibble]

Not a huge change - we're just allowing the file_exists_behavior to be set as part of the appspec file, which will take precedence over the corresponding command line option for file_exists_behavior for the upcoming release. We're not allowing fine grained control of file_exists_behavior per file/directory listed in the appspec files section, though. We'll consider that if demand from customers still exists following the upcoming release.

[0cool-f]

Not a huge change - we're just allowing the file_exists_behavior to be set as part of the appspec file, which will take precedence over the corresponding command line option for file_exists_behavior for the upcoming release. We're not allowing fine grained control of file_exists_behavior per file/directory listed in the appspec files section, though. We'll consider that if demand from customers still exists following the upcoming release.

Thanks a lot for fixing it. You may not have noticed that, but this is exactly THE huge fix thousands of people have been waiting for.

[alinoudev]

On what date its planned to release?

[quentinhayot]

Not a huge change - we're just allowing the file_exists_behavior to be set as part of the appspec file, which will take precedence over the corresponding command line option for file_exists_behavior for the upcoming release. We're not allowing fine grained control of file_exists_behavior per file/directory listed in the appspec files section, though. We'll consider that if demand from customers still exists following the upcoming release.

Guys, this is it !

On what date its planned to release?

Hopefully soon ! I'll just put the file_exists_behavior in the appspec already...

[ghost]

Looking forward to this!

[s-larbi]

Any updates regarding the overwrite feature? It takes time to delete everything and setup apps from scratch. The feature would be a great time saver.

[philstrong]

Releasing it now to first regions as 1.3.2

…

On Fri, Apr 30, 2021 at 6:51 PM s-larbi ***@***.***> wrote: Any updates regarding the overwrite feature? It takes time to delete everything and setup apps from scratch. The feature would be a great time saver. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUCBBWIJSEC4VCYZXDPOTTLMX6FANCNFSM4BIIWZCQ> .

[abhijith7025]

I think there is already a feature

files:

• source:
  destination: c:\temp
  file_exists_behavior: OVERWRITE

read this out https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-files.html

[s-larbi]

I have tried this before and it didn't work: "The deployment failed because a specified file already exists at this location: /var/www/html/website-folder/sub-folder/file-name.php"...

[0cool-f]

I think there is already a feature

files:

* source: 
  destination: c:\temp
  file_exists_behavior: OVERWRITE

read this out https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-files.html

mate you should always read before write something: this bug exists because this feature, which it seems you haven't tried, does not work, in fact it's a bug...do you really think that an issue could remain open and debated for years, waiting for a doc page that we all haven't read? 😶

[s-larbi]

I hope this bug is fixed ASAP. It would be a great time saver.

[0cool-f]

I hope this bug is fixed ASAP. It would be a great time saver.

As Phil said, after the release cycle of 1.3.2 it will work.

Phil's team was fast, but "ASAP" is a relative concept, have you read the date of issue?

[philstrong]

1.3.2 is out in some regions e.g. us-east-1 but rolling out to the rest. We will update GitHub once all commercial regions are released which will be Thurs or Friday (5/13 or 5/14).
sa-east-1, ap-northeast-1, us-west-1, eu-west-1, ap-southeast-1 are also released

[quentinhayot]

Jeff, Give this guy a cigar !

[philstrong]

Issue is fixed in 1.3.2. Any and all feedback is welcome.

[waldoj]

Thank you, @philstrong, for getting this issue resolved. This will be a great little quality-of-life improvement for a lot of people. I'm grateful. 🙏🏻

[andreprawira]

hi @philstrong is file_exists_behavior: OVERWRITE also applicable to hooks section? because im also getting file already exist in codedeploy temp folder when i changed a folder name in the script. For example,

if my previous appspec.yml was

hooks:
  AfterInstall:
    - location: scripts\install_dependencies
      timeout: 300

and then i changed it to

hooks:
  AfterInstall:
    - location: scripts-linux\install_dependencies
      timeout: 300

i'd get that file already exist error, so if i add

hooks:
  AfterInstall:
    - location: scripts-linux\install_dependencies
      timeout: 300
      file_exists_behavior: OVERWRITE

will that solve the problem?

[andreprawira]

@philstrong sorry for the multiple comments but im also facing this The deployment failed because a specified file already exists at this location: C:\inetpub\www\ec2django/asgi.py error, this is on a windows appspec.yml, here is the script

`version: 0.0
os: windows
files:

• source: \ec2django
  destination: C:\inetpub\www\ec2django
  file_exists_behavior: OVERWRITE`

any idea where i got it wrong? that should have overwritten the previously existing files right?

[mwjones-aws]

@andreprawira to address your first comment, no. file_exists_behavior is not supported in the hooks section.

With respect to your second comment, the service team will try to reproduce the problem. Can we have a little more information?

1. Please post the full appspec.yml
2. What file-exists-behavior are you passing to CreateDeployment?