Skip to content

Spot interruption event logs incorrect time #757

@aarongorka

Description

@aarongorka

Describe the bug
When NTH receives a spot interruption notice and logs the interruption time, it logs the current time rather than the actual interruption time.

See also #606

Steps to reproduce

  1. Set up NTH in Deployment/SQS mode
  2. Configure EventBridge to send spot termination notices
  3. Wait for a spot termination notice
  4. Observe log with "Spot Interruption event received"

Expected outcome
Spot interruption time should be roughly +2 minutes from when the event was received. Ideally we can use the actual interruption time as supplied by AWS rather than guestimating it though.

Application Logs

2023-01-23T00:18:31.086824792Z aws-node-termination-handler-6574867b74-z7l57 {"level":"info","event":{"EventID":"spot-itn-event-62363231313539382d666634352d343631612d646537642d346232353565353832386463","Kind":"SPOT_ITN","Monitor":"SQS_MONITOR","Description":"Spot Interruption event received. Instance i-099447d7bc6f9dde7 will be interrupted at 2023-01-23 00:18:29 +0000 UTC \n","State":"","AutoScalingGroupName":"eks-main-20220713005232834100000009","NodeName":"ip-10-9-139-214.ec2.internal","NodeLabels":null,"Pods":null,"InstanceID":"i-099447d7bc6f9dde7","ProviderID":"aws:///us-east-1b/i-099447d7bc6f9dde7","IsManaged":true,"StartTime":"2023-01-23T00:18:29Z","EndTime":"0001-01-01T00:00:00Z","NodeProcessed":false,"InProgress":false},"time":"2023-01-23T00:18:31Z","message":"Adding new event to the event store"}
2023-01-23T00:20:18.474299920Z aws-node-termination-handler-6574867b74-z7l57 {"level":"info","node_name":"ip-10-9-139-214.ec2.internal","reason":"Spot Interruption event received. Instance i-099447d7bc6f9dde7 will be interrupted at 2023-01-23 00:18:29 +0000 UTC \n","time":"2023-01-23T00:20:18Z","message":"Node successfully cordoned and drained"}

Environment

  • NTH App Version: v1.18.2 (Helm chart version 0.18.5)
  • NTH Mode (IMDS/Queue processor): Queue processor
  • OS/Arch: Amazon Linux 2, x86
  • Kubernetes version: 1.24
  • Installation method: Helm

Metadata

Metadata

Labels

staleIssues / PRs with no activity

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions