Skip to content

Add Initial support for CBOR protocol including extension package and Marshaller generators #3879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>net472</TargetFramework>
<AssemblyName>AWSSDK.Extensions.CborProtocol</AssemblyName>
<PackageId>AWSSDK.Extensions.CborProtocol</PackageId>
<GenerateAssemblyTitleAttribute>false</GenerateAssemblyTitleAttribute>
<GenerateAssemblyConfigurationAttribute>false</GenerateAssemblyConfigurationAttribute>
<GenerateAssemblyCompanyAttribute>false</GenerateAssemblyCompanyAttribute>
<GenerateAssemblyProductAttribute>false</GenerateAssemblyProductAttribute>
<GenerateAssemblyDescriptionAttribute>false</GenerateAssemblyDescriptionAttribute>
<GenerateAssemblyCopyrightAttribute>false</GenerateAssemblyCopyrightAttribute>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<NoWarn>$(NoWarn);CS1591</NoWarn>
<SignAssembly>True</SignAssembly>
</PropertyGroup>

<ItemGroup>
<Compile Remove="**/obj/**" />
</ItemGroup>

<Choose>
<When Condition=" '$(AWSKeyFile)' == '' ">
<PropertyGroup>
<AssemblyOriginatorKeyFile>..\..\..\sdk\awssdk.dll.snk</AssemblyOriginatorKeyFile>
</PropertyGroup>
</When>
<Otherwise>
<PropertyGroup>
<AssemblyOriginatorKeyFile>$(AWSKeyFile)</AssemblyOriginatorKeyFile>
</PropertyGroup>
</Otherwise>
</Choose>

<ItemGroup>
<PackageReference Include="System.Formats.Cbor" Version="9.0.5" />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a general question, but do you have plans to also include these new dependencies in the build system when copying over the dll's? I had to do that when adding System.Text.JSON as an example

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should part of the build system task that we added, will mention that on the task's decription so we don't miss that.

</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\..\..\sdk\src\Core\AWSSDK.Core.NetFramework.csproj" />
</ItemGroup>
</Project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFrameworks>netstandard2.0;netcoreapp3.1;net8.0</TargetFrameworks>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need netcoreapp3.1 in here if cbor is only supported in net8 and netstandard?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should work on netcoreapp3.1 too, it will just show a warning.

<AssemblyName>AWSSDK.Extensions.CborProtocol</AssemblyName>
<PackageId>AWSSDK.Extensions.CborProtocol</PackageId>
<GenerateAssemblyTitleAttribute>false</GenerateAssemblyTitleAttribute>
<GenerateAssemblyConfigurationAttribute>false</GenerateAssemblyConfigurationAttribute>
<GenerateAssemblyCompanyAttribute>false</GenerateAssemblyCompanyAttribute>
<GenerateAssemblyProductAttribute>false</GenerateAssemblyProductAttribute>
<GenerateAssemblyDescriptionAttribute>false</GenerateAssemblyDescriptionAttribute>
<GenerateAssemblyCopyrightAttribute>false</GenerateAssemblyCopyrightAttribute>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<SuppressTfmSupportBuildWarnings>true</SuppressTfmSupportBuildWarnings>
<NoWarn>$(NoWarn);CS1591</NoWarn>
<SignAssembly>True</SignAssembly>
</PropertyGroup>

<ItemGroup>
<Compile Remove="**/obj/**" />
</ItemGroup>

<PropertyGroup Condition="'$(TargetFramework)' == 'net8.0'">
<WarningsAsErrors>IL2026,IL2075</WarningsAsErrors>
<IsTrimmable>true</IsTrimmable>
</PropertyGroup>

<Choose>
<When Condition=" '$(AWSKeyFile)' == '' ">
<PropertyGroup>
<AssemblyOriginatorKeyFile>..\..\..\sdk\awssdk.dll.snk</AssemblyOriginatorKeyFile>
</PropertyGroup>
</When>
<Otherwise>
<PropertyGroup>
<AssemblyOriginatorKeyFile>$(AWSKeyFile)</AssemblyOriginatorKeyFile>
</PropertyGroup>
</Otherwise>
</Choose>

<ItemGroup>
<PackageReference Include="System.Formats.Cbor" Version="9.0.5" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\..\..\sdk\src\Core\AWSSDK.Core.NetStandard.csproj" />
</ItemGroup>
</Project>
122 changes: 122 additions & 0 deletions extensions/src/AWSSDK.Extensions.CborProtocol/CborWriterExtensions.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/

using System;
using System.Formats.Cbor;
using Amazon.Util;

namespace AWSSDK.Extensions.CborProtocol
{
public static class CborWriterExtensions
{
/// <summary>
/// Writes the DateTime as UnixEpochSeconds which is the only type we support for CBOR.
/// </summary>
/// <param name="writer">The CBOR writer to use.</param>
/// <param name="value">The DateTime value to write.</param>
public static void WriteDateTime(this CborWriter writer, DateTime value)
{
writer.WriteTag(CborTag.UnixTimeSeconds);
writer.WriteOptimizedNumber(AWSSDKUtils.ConvertToUnixEpochSecondsDouble(value));
}

/// <summary>
/// Writes a double using the smallest CBOR representation that preserves value and precision.
/// </summary>
/// <param name="writer">The CBOR writer to use.</param>
/// <param name="value">The double value to write.</param>
public static void WriteOptimizedNumber(this CborWriter writer, double value)
{
if (double.IsNaN(value) || double.IsInfinity(value))
{
writer.WriteDouble(value); // Write NaN or Infinity as a double.
return;
}

// If the value is an integer (without fractional part), write it as Int64 or UInt64.
if (value % 1 == 0)
{
if (value >= long.MinValue && value <= long.MaxValue)
{
// If the value fits within the signed 64-bit integer (long) range,
// WriteInt64 serializes it into the smallest CBOR type representation
// that can contain its value without loss of precision.
writer.WriteInt64((long)value);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Add a comment for future readers that CborWriter will take care of compacting the long to the least number of bytes. This threw me off since a double and long in .NET use the same number of bytes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some details that it serializes values into the smallest possible value.

return;
}

if (value >= 0 && value <= ulong.MaxValue)
{
// If the value is non-negative and fits within the unsigned 64-bit range,
// WriteUInt64 serializes it into the smallest possible CBOR type representation.
writer.WriteUInt64((ulong)value);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Add a comment for future readers that CborWriter will take care of compacting the long to the least number of bytes. This threw me off since a double and long in .NET use the same number of bytes.

return;
}
}

// Check if value can safely be represented as float32
float floatCandidate = (float)value;
if ((double)floatCandidate == value)
{
WriteOptimizedNumber(writer, floatCandidate);
return;
}

// If none of the above conditions are satisfied, write the value as a double.
writer.WriteDouble(value);
}

/// <summary>
/// Writes a float using the smallest CBOR representation that preserves value and precision.
/// This method uses manual encoding to avoid writing as a half-precision float.
/// </summary>
/// <param name="writer">The CBOR writer to use.</param>
/// <param name="value">The float value to write.</param>
public static void WriteOptimizedNumber(this CborWriter writer, float value)
{
// If the value is an integer (without fractional part), write it as Int64 or UInt64.
if (value % 1 == 0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this safe to do? I don't know much about CBOR but it seems that if the user passes in a float value they'd want to keep that extra precision instead of writing it as an int. Am I missing something here? Since it is binary does it not matter?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When if (value % 1 == 0) is true, then we don't have anything after the decimal point to keep. It was part of the SEP to send the numbers as the smallest version possible that doesn't lose any data.

{
if (value >= long.MinValue && value <= long.MaxValue)
{
// If the value fits within the signed 64-bit integer (long) range,
// WriteInt64 serializes it into the smallest CBOR type representation
// that can contain its value without loss of precision.
writer.WriteInt64((long)value);
return;
}

if (value >= 0 && value <= ulong.MaxValue)
{
// If the value is non-negative and fits within the unsigned 64-bit range,
// WriteUInt64 serializes it into the smallest possible CBOR type representation.
writer.WriteUInt64((ulong)value);
return;
}
}

// Manual encoding to avoid half-precision floats
var bytes = new byte[5];
bytes[0] = 0xFA; // CBOR float32 marker
BitConverter.GetBytes(value).CopyTo(bytes, 1);

// Ensure the bytes are in the correct endian order for CBOR.
if (BitConverter.IsLittleEndian)
Array.Reverse(bytes, 1, 4);

writer.WriteEncodedValue(bytes);
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@

<Project>
<PropertyGroup>
<BaseIntermediateOutputPath>$(MSBuildProjectDirectory)\obj\$(MSBuildProjectName)</BaseIntermediateOutputPath>
</PropertyGroup>
</Project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
using Amazon.Runtime;
using Amazon.Runtime.Internal;
using Amazon.Runtime.Internal.Transform;
using Amazon.Runtime.Internal.Util;
using Amazon.Util;
using System;
using System.Collections.Generic;
using System.Formats.Cbor;
using System.IO;

namespace AWSSDK.Extensions.CborProtocol.Internal
{
public class CborMarshallerContext : MarshallerContext
{
public CborWriter Writer { get; private set; }

public CborMarshallerContext(IRequest request, CborWriter writer)
: base(request)
{
Writer = writer;
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
using System;
using System.Collections.Concurrent;
using System.Formats.Cbor;
using System.Threading;

namespace AWSSDK.Extensions.CborProtocol.Internal
{
public static class CborWriterPool
{
// Internal pool storage using thread-safe collection
private static readonly ConcurrentBag<CborWriter> _pool = new ConcurrentBag<CborWriter>();

// Maximum number of CborWriter instances the pool can hold
private static int _maxPoolSize = 16;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current value of _maxPoolSize is temporary, I need evaluate and test it more before merging the feature branch.


/// <summary>
/// Gets or sets the maximum size of the writer pool.
/// Minimum value is 1.
/// </summary>
public static int MaxPoolSize
{
get => Volatile.Read(ref _maxPoolSize);
set => Volatile.Write(ref _maxPoolSize, Math.Max(1, value));
}

/// <summary>
/// Retrieves a CborWriter from the pool, or creates a new one if the pool is empty.
/// </summary>
public static CborWriter Rent()
{
if (_pool.TryTake(out var writer))
{
return writer;
}
// Create a new CborWriter if the pool is empty
return new CborWriter(CborConformanceMode.Canonical, true);
}

/// <summary>
/// Returns a CborWriter to the pool for reuse.
/// If the pool is already full then the writer will be discard.
/// </summary>
public static void Return(CborWriter writer)
{
if (_pool.Count >= Volatile.Read(ref _maxPoolSize))
return;

writer.Reset(); // Ensure the writer is in a clean state before pooling
_pool.Add(writer);
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: AssemblyTitle("AWSSDK.Extensions.CborProtocol")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("Amazon.com, Inc")]
[assembly: AssemblyProduct("AWS SDK for .NET extensions for Cbor protocol support")]
[assembly: AssemblyDescription("AWS SDK for .NET extensions for Cbor protocol support")]
[assembly: AssemblyCopyright("Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.")]
[assembly: AssemblyTrademark("")]

// Setting ComVisible to false makes the types in this assembly not visible
// to COM components. If you need to access a type in this assembly from
// COM, set the ComVisible attribute to true on that type.
[assembly: ComVisible(false)]

[assembly: AssemblyVersion("4.0")]
[assembly: AssemblyFileVersion("4.0.0.0")]
Loading