High memory consumption

[sharok]

Hello. When I restart codedeploy-agent it takes about 26MB. But, after one deploy it takes 300-350MB. Is it ok or not? Memory doesn't free when deploy is finished. Therefore, I get error about memory allocation on the next build.

[amartyag]

Hi,
Can you please tell us the size of the deployment bundle ? Also, can you try to do the deployment once without the install step and see if the memory still spikes? Thanks

[elijahchancey]

+1 I'm seeing high memory consumption too. For further info please see Case ID 1557344221

[elijahchancey]

Any updates on this?

[feverLu]

Sorry, but what do you mean by Case ID 1557344221?

[thegranddesign]

I'm seeing CodeDeploy taking 157MB of memory which is crazy high for something that simply waits and polls a server. I can see if maybe while it's doing other things like interacting with the deploy, but once it's done, it should release that memory.

[thegranddesign]

Also FWIW, it's the polling process that's taking up all the memory. The "master" process is sitting at 15MB which is very reasonable.

[talentedmrjones]

I'm seeing memory continue to grow with each deploy and not be freed. We are currently over 1GB!

total kB 1051392 765580 762328

[jangie]

+1. I am seeing the polling process taking 1.138g after two days of it being up. One possible compounding issue is that we have two applications that depend on codedeploy for deployment on this given instance, but it still seems excessive.

[jangie]

For people still experiencing this issue: In our case, this seems to have been resolved for our use case following an update of the ruby version. The ruby versions in question were 2.0.0 p353 (bad memory consumption) and 2.0.0 p598 (much more reasonable memory consumption). While there was also a simultaneous OS update, I believe that this would not have influenced this issue.

[thegranddesign]

I'm on Ruby 2.3 and this is still an issue.

[michaelsobota]

+1. We are on Ruby 2.3.1 and experiencing the same behavior.

[michaelsobota]

Our bundle size is ~80MB -- mem usage prior to a deployment is 28588kb RSS and jumps to 674184kb post deployment, after ~2-3 runs through our pipeline we get OOM failures on the deployment.

28588 766 codedeploy-agent: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller of master 762

674184 766 codedeploy-agent: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller of master 762

[mattlo]

I'm also encountering this issue. Been running code deploy for a few months and we're seeing memory creeping on every deploy. As mentioned on #6, I added a few commands when the code deploy succeeds

sudo /etc/init.d/codedeploy-agent stop
sudo /etc/init.d/codedeploy-agent start
echo 3 > sudo /proc/sys/vm/drop_caches

There is nothing other than the CodeDeploy agent running on the Ruby process.

• The constant memory was before I cleared the memory from CodeDeploy
• After dumping the memory (terminating CodeDeploy), I modified appspec.yml config to include the script above on ValidateService, added custom script to dump memory after CI has indicated a successful build.
• The memory spike is on code deployment, and the memory dips right back down to pre-deploy levels.
• Thankfully, CodeDeploy Console still seems to think the instance is a successful deploy, so it continues with the rollout config even though the agent stops for a several seconds. CodeDeploy Console will fail sometimes, so you can't use the leverage the ValidateService hook, even if you have a long timeout configuration.

Digging deeper, AWS CodeDeploy must be holding the bundle within memory and not releasing it after the hook. There probably needs to be a modification somewhere in the code base to deallocate it. Aggressively changing the ruby GC is another option, but may not be an easy option for those with Ruby applications.

[jcowley]

Seeing this as well. Memory usage continually creeps up. After a dozen or so deploys, the agent is holding onto ~500MB of memory. Eventually, deployments start failing because the deployment scripts can no longer allocate memory.

This seems like a pretty egregious problem to have open for so long. (See issue #6 as well.)

[jcowley]

So here's my hacky workaround to the aws-codedeploy-agent memory leak issue. I added a line to schedule a restart of the agent to the end of the last script (e.g. a script that runs from the ValidateService hook in the appspec.yml file):

at -M now + 2 minute <<< $'service codedeploy-agent restart'

Without something like this, builds start failing for me with 'Cannot Allocate Memory' errors in the logs after about 2-3 builds.

Just to be clear, this isn't a "fix" that resolves the issue, so please don't close it. It's just a workaround hack. The real fix might be to ensure that the codedeploy agent is not reading files into memory with every build.