The Remove-AzureStorageDirectory command is missing a -Force parameter. If the directory is not empty, then an exception is thrown.
- Azure PowerShell module version 0.8.12
- WMF / PowerShell 5.0 November 2014 Preview
Remove-AzureStorageDirectory : The remote server returned an error: (409) Conflict. HTTP Status Code: 409 - HTTP Error Message: The specified directory is not empty.

Cheers,
Trevor Sullivan
Microsoft MVP: PowerShell