Handle transient error messages while editing config file during hot reload

[aaronburtle]

In order to hot reload we must monitor the config file for changes. This results in constant reads by the file watcher that we instantiate to do this task. However, when editing the config file, that editor will take a write-lock on the config file, resulting in the file watcher throwing errors. These errors are expected and will continue until the editor has relinquished its write lock on the config file.

Before we introduce the hot-reload feature, we must determine how we will handle these error messages.

For example, some editors will only take a write lock while saving changes made to the file. In this case, the error messaging will be limited, perhaps only a single message, and is likely to cause confusion. It might be best to ignore these limited transient errors by not recording the first small number of errors.

But it could also be the case that the editor has taken a write-lock the entire time the file is open, not just during saves. In this case a hot reload will not be possible until that editor is closed, and the error messaging will be valuable. We should maintain these error messages, perhaps by making sure we do in fact log errors beyond the first few.

Whatever the design decision, we must gracefully handle both cases so that the error messaging is consistent and clear.

Related to #2064

Related to #2108

[seantleonard]

TODO:

1. Update code which reads file contents of config file to be enclosed in Try/Catch:

   data-api-builder/src/Config/FileSystemRuntimeConfigLoader.cs

   Line 151 in dec9d0e

   string json = _fileSystem.File.ReadAllText(path);

   1. If IOException occurs on File.ReadAllText , we should introduce some sort of exponential backoff which optimistically assumes that file handle will become available within the backoff period.

Partially handled by #2424, with an added mitigation measure towards reading a file that is monitored by hot reload:

1. Exponential backoff added for when DAB gets the file hash and determines whether a hot reload operation should occur. While the hashing mechanism means that another read of the file occurs (another read handle acquired), the backoff mechanism protects against the IOExceptions that occurs.