Automatically retry on certain HTTP status in response from ServiceNow #280
Description
Summary of the new feature/enhancement
In a busy environment, web exceptions are sometimes returned by the module. At times it is a useful HTTP error like 429 with a retry-after timespan in seconds. Other times it is a transport level exception due to network congestion, etc. In either case, simply retrying with some smarts (where possible) would greatly improve the reliability when the module is used in an unattended automation process.
Proposed technical implementation details
Implement an error handler in the Invoke-ServiceNowRestMethod, which retries request on HTTP status codes 429, 503, 502, 408, 504 and 409. When server returns retry-after then honour that value. Otherwise a default wait time of 5 seconds can be used between retries for certain errors. Default number of retries (RetryAttempt) maybe 2. I suggest adding RetryAttempt to ServiceNowSession object.
See reference https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/
Sample code
This function can be used to call Get-ServiceNowRecord with retries as described above, for e.g:
Invoke-SNowShellMethodWithRetry -MethodName Get-ServiceNowRecord -Parameters @{Table=$table; Filter=$filter; Fields=$fields; IncludeTotalCount=$true; First=500}
You can also test various HTTP status codes by passing Invoke-TestMode in the MethodName. TestMode uses https://httpstat.us/429 by default. To test another, pass a hashtable with Uri in Parameters, for e.g.:
Invoke-SNowShellMethodWithRetry -MethodName Invoke-TestMode -Parameters @{Uri='https://httpstat.us/504'}
Function Invoke-SNowShellMethodWithRetry {
param(
[Parameter(mandatory)]
[ValidateSet('Get-ServiceNowRecord','Invoke-TestMode')]
[string]$MethodName,
[Parameter(mandatory)]
[hashtable]$Parameters,
[Parameter()]
[ValidateRange(1,5)]
[UInt16]$RetryAttempts = 2,
[Parameter()]
[ValidateRange(1,10)]
[UInt16]$WaitTimeInSeconds = 5
)
$retryLoopActive = $true
$response = $null
do {
try {
switch ($MethodName) {
Get-ServiceNowRecord {
$response = Get-ServiceNowRecord @Parameters
break
}
Invoke-TestMode {
if ($Parameters.ContainsKey('Uri')) {
$response = Invoke-WebRequest @Parameters
} else {
Write-Host "TestMode is active. Getting HTTP 429 response from https://httpstat.us/429"
Write-Host "To check other HTTP status codes or your own Uri, pass the Uri in Parameters hashtable, e.g. @{Uri='https://httpstat.us/504'}"
$response = Invoke-WebRequest -Uri 'https://httpstat.us/429'
}
break
}
Default {
throw "Calling ServiceNow-PowerShell method $MethodName via this function is not currently supported. Call $MethodName directly instead."
}
}
$retryLoopActive = $false
} catch {
# This handler is designed to handle HTTP exceptions only
if ($_.Exception.GetType().Name -notin 'HttpResponseException','WebException') { throw $_ }
# Grab a pointer to the original exception
$origException = $_
try {
# Retry with or without wait based on source of exception
# Reference used by this handler https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/<status-code>
switch ($origException.Exception.Response.StatusCode.value__) {
{ $_ -in @(429,503) } {
Write-Warning "Server returned HTTP error $_ ($($origException.Exception.Response.StatusCode))"
if ($null -eq $origException.Exception.Response.Headers.RetryAfter) {
Write-Debug "Request will be retried $RetryAttempts time(s) after $WaitTimeInSeconds seconds"
Start-Sleep $WaitTimeInSeconds
} else {
Write-Debug "Request will be retried $RetryAttempts time(s) after $($origException.Exception.Response.Headers.RetryAfter.ToString()) seconds"
Start-Sleep $origException.Exception.Response.Headers.RetryAfter.ToString()
}
break
}
{ $_ -in @(502,408) } {
Write-Warning "Server returned HTTP error $_ ($($origException.Exception.Response.StatusCode))"
Write-Debug "Request will be retried $RetryAttempts time(s)"
break
}
{ $_ -in @(504,409) } {
Write-Warning "Server returned HTTP error $_ ($($origException.Exception.Response.StatusCode))"
Write-Debug "Request will be retried $RetryAttempts time(s) after $WaitTimeInSeconds seconds"
Start-Sleep $WaitTimeInSeconds
break
}
default {
Write-Warning "Server returned HTTP error $_ ($($origException.Exception.Response.StatusCode))"
throw $origException
}
}
$RetryAttempts --
if ($RetryAttempts -lt 1) {
$retryLoopActive = $false
throw $origException
}
} catch {
#Write-Error $_
throw $origException
}
}
} while ($retryLoopActive)
return $response
}