Need concurrency control for Function key generation

[mathewc]

When generating host level secrets, we have concurrency control here. However when generating Function keys we do not here. This can lead to errors when attempting to persist the blob. The error that occurs is Azure/azure-storage-net#780.

We need to add locking here as well.

Following is the stack trace for the error when it occurs is below. This came up recently in CRI 224204858, but I see instances of this error happening in production across many applications.

Microsoft.Azure.Storage.StorageException : The specified block list is invalid.
   at async Microsoft.Azure.WebJobs.Script.WebHost.BlobStorageSecretsRepository.WriteAsync(ScriptSecretsType type,String functionName,ScriptSecrets secrets) at D:\a\1\s\src\WebJobs.Script.WebHost\Security\KeyManagement\BlobStorageSecretsRepository.cs : 115
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.PersistSecretsAsync[T](T secrets,String keyScope,Boolean isNonDecryptable) at D:\a\1\s\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs : 568
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.GetFunctionSecretsAsync(String functionName,Boolean merged) at D:\a\1\s\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs : 162
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.WebHost.Controllers.KeysController.GetFunctionKeys(String functionName) at D:\a\1\s\src\WebJobs.Script.WebHost\Controllers\KeysController.cs : 118
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.WebHost.Controllers.KeysController.Get(??) at D:\a\1\s\src\WebJobs.Script.WebHost\Controllers\KeysController.cs : 50
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper,ObjectMethodExecutor executor,Object controller,Object[] arguments)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at async Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next,Scope& scope,Object& state,Boolean& isCompleted)
   at async Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next,Scope& scope,Object& state,Boolean& isCompleted)
   at async Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.WebHost.Middleware.FunctionInvocationMiddleware.Invoke(HttpContext context) at D:\a\1\s\src\WebJobs.Script.WebHost\Middleware\FunctionInvocationMiddleware.cs : 40
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.WebHost.Middleware.ExceptionMiddleware.Invoke(HttpContext context) at D:\a\1\s\src\WebJobs.Script.WebHost\Middleware\ExceptionMiddleware.cs : 27