Multiple problems when using with Unity/MacOS

[staszain]

Hi, I'm evaluating this library for some time now in Unity. I managed to import it into the project with some issues (for some reason it doesn't load the first time you run the game), however it doesn't work out of the box for me. I've compiled the library from 1.1.0 and took the same version of the bindings. Here are the problems I've encountered so far:

1. client.DispatchCallback(status) is getting called with info structure containing garbage. It looks like there is a problem with marshalling. I was able to fix this with adding "ref" to the StatusCallback declaration:
   public delegate void StatusCallback(ref StatusInfo info, IntPtr context);
   I'm not an expert in c# marshalling and not sure this is a right fix and the system is now working as it supposed to, or is it going to break other platforms.

2. Once StatusCallback is changed, the structure still seems to be off. It appears that some of the field values are shifted to neighbour fields. I was able to fix this by adding StructLayoutAttribute.Pack = 4 to StatusInfo structure declaration:
   [StructLayout(LayoutKind.Sequential, Pack = 4)]
   public struct StatusInfo {
   ...
   as it looks like "connection" variable takes whole 8 bytes instead of 4. Not sure why Unity/Mono has a different default Pack setting, but I think this should fix the problem for Mono and not break other platforms.

3. Just a suggestion to mention deinitialization procedure, especially if you set a debug callback like described in the "Usage":

Library.Initialize();
NetworkingUtils utils = new NetworkingUtils();
utils.SetDebugCallback(DebugType.Everything, debug);

you absolutely MUST set it to None when you're done:

utils.SetDebugCallback(DebugType.None, null);
Library.Deinitilize();

otherwise Unity will freeze on the next launch (probably because debug callback is now invalid and crashes silently somewhere inside).

[nxrighthere]

client.DispatchCallback(status) is getting called with info structure containing garbage. It looks like there is a problem with marshalling. I was able to fix this with adding "ref" to the StatusCallback declaration:

You are right, the signature was incorrect, it should be passed by reference.

Going to investigate other problems too.

[nxrighthere]

Just a suggestion to mention deinitialization procedure, especially if you set a debug callback like described in the "Usage":

Library.Initialize();
NetworkingUtils utils = new NetworkingUtils();
utils.SetDebugCallback(DebugType.Everything, debug);

you absolutely MUST set it to None when you're done:

utils.SetDebugCallback(DebugType.None, null);
Library.Deinitilize();

otherwise Unity will freeze on the next launch (probably because debug callback is now invalid and crashes silently somewhere inside).

I think this should be resolved in the native library, we can simply zero out/reset all global data at deinitialization with GameNetworkingSockets_Kill() since we done with it, like: SteamNetworkingSocketsLib::s_pfnDebugOutput, SteamNetworkingSocketsLib::g_eSteamDatagramDebugOutputDetailLevel.

Other game engines might be affected by this as well, since usually libraries there are loaded once, and global data/variables will persist in memory while the application remains alive.

@fletcherdvalve What do you think about this?

[nxrighthere]

Once StatusCallback is changed, the structure still seems to be off. It appears that some of the field values are shifted to neighbour fields. I was able to fix this by adding StructLayoutAttribute.Pack = 4 to StatusInfo structure declaration:
[StructLayout(LayoutKind.Sequential, Pack = 4)]
public struct StatusInfo {
...
as it looks like "connection" variable takes whole 8 bytes instead of 4. Not sure why Unity/Mono has a different default Pack setting, but I think this should fix the problem for Mono and not break other platforms.

That one is strange. I'm unable to reproduce this on my end.

Native size of structures on Ubuntu:

Managed size of structures on macOS Catalina 10.15.4 (Unity 2020.1.0b2):

Managed size of structures on Windows 10 (Unity 2019.3.2f1):

Managed size of structures on Windows 10 (.NET Core 3.1.200):

Can you please run the following script and confirm that on your end?

using System;
using System.Runtime.InteropServices;
using Unity.Entities;
using Unity.Jobs;
using UnityEngine;

namespace Valve.Sockets {
	public class Test : JobComponentSystem {
		protected override void OnCreate() {
			Debug.Log("Address size: " + Marshal.SizeOf(typeof(Address)) + " bytes");
			Debug.Log("Configuration size: " + Marshal.SizeOf(typeof(Configuration)) + " bytes");
			Debug.Log("StatusInfo size: " + Marshal.SizeOf(typeof(StatusInfo)) + " bytes");
			Debug.Log("ConnectionInfo size: " + Marshal.SizeOf(typeof(ConnectionInfo)) + " bytes");
			Debug.Log("ConnectionStatus size: " + Marshal.SizeOf(typeof(ConnectionStatus)) + " bytes");
			Debug.Log("NetworkingIdentity size: " + Marshal.SizeOf(typeof(NetworkingIdentity)) + " bytes");
			Debug.Log("NetworkingMessage size: " + Marshal.SizeOf(typeof(NetworkingMessage)) + " bytes");
		}

		protected override JobHandle OnUpdate(JobHandle inputDependencies) => inputDependencies;
	}
}

[staszain]

Can you please run the following script and confirm that on your end?

It appears the problem is on C++ end. I have 704 bytes for the StatusInfo instead of 712, which probably means that some C++ defaults on Mac are to blame. I obviously didn't make any changes to the library.

[nxrighthere]

What compiler are you using?

[staszain]

AppleClang 11.0.3.11030032

[nxrighthere]

I have a feeling that this is because not all integer types are strictly set to exact-width, and some of those are compiler-specific, but I'm not sure.

I'll move this issue to the native repository if @fletcherdvalve doesn't appear here next Monday-Tuesday.

[zpostfacto]

Can you figure out where the difference is coming in?

[nxrighthere]

I might be wrong, but, probably what is affecting the layout is: int m_eEndReason; in SteamNetConnectionInfo_t and int m_cbSize; in SteamNetworkingIdentity which encapsulated into SteamNetConnectionInfo_t. It seems that only those guys are not strictly set to exact-width.

[zpostfacto]

But his size is 8 bytes smaller, right? Given that those int fields are 4 bytes on all platforms I know of, this would imply that they are 0 bytes on Apple clang. :)

I understand that it is legal for the C spec for int to be a size other than 4 bytes. I am just not aware of any ABI we care about that decided to make what would be an absolutely disastrous decision.

It's got to be some alignment thing or some enum or something.

[nxrighthere]

Yea, hard to diagnose this without the compiler and OS (I don't have it). So @staszain it would be great if you can help us to determine what exactly is affecting the layout.

[staszain]

I will be happy to provide any assistance you need from my side.

As I mentioned I believe the first shift comes from StatusInfo.connection field, which is UInt32 as declared in Valve.Sockets. It takes either 8 or 4 bytes depending on Pack attribute parameter and the entire content of the ConnectionInfo structure is shifted 4 bytes depending on that. I will double check.