Building MM and Debugging ModDBInterop

[Frosferes2]

I've spent some time familiarising myself with VS and F#. If I can get MM to build successfully, I can move on to debugging the issues I brought up earlier.

I'm using VS Community 2022. When I attempt to open and build the MMDotNet solution, I get notifications that I'm missing specific versions of the .NETFramework reference assemblies (4.0, 4.5 etc.). But when I try to install these, the installer says they're already part of my OS and won't proceed. The reference manager says the targetted assemblies were not found.

Is there a way to solve the missing references using Paket or the Github build options you mentioned before? Apologies for the elementary questions. Once this is solved there shouldn't be any more barriers to debugging.

On a side note, I've made a Blender script to generate a akeleton for imported MM meshes. Bone positions are approximated using the weighted mean position of all the vertices in a given vertex group, where mean weights come from the vertex weights. If you are interested, I can share it with you, though I'm not really sure how to do so via Github. I also want to make a partner script that can use this skeleton to symmetrize the vertex groups for more convenient weight painting.

[jmquigs]

I use vs community 2019, the github action uses this as well. Not surprised that 2022 has Reference errors but I always struggle to fix them.

If you follow the github action you should be able to get it to build https://github.com/jmquigs/ModelMod/blob/master/.github/workflows/dotnet-desktop.yml

You can ignore the TPLib for now, that lib isn't needed (its just binaries of d3dx and similar files that I didn't want to host in a public repo).

[Frosferes2]

Okay I figured out the problem. I had to install the required .NET targetting packs. But since the .NET 4.0, 4.5 etc. are out of support for VS 2022, I had to add VS2019 to the installer and add only the targetting pack components. The app is building and running correctly now.

I'm going to try to implement proper tangent and binormal calculation using the reference site in the script comment to see if this fixes the light deflection issues.

[Frosferes2]

I realised the same thing you did in #10, i.e. that we can export the tangent space directly from blender instead of calculating it ourselves. I have written my own extraction method which ensures every loop normal/tangent/binormal has a unique vertex index, and are then also sorted by those same indices. This should ensure each vertex is assigned the correct normal, tangent and binormal.

Now we have tn and bn lines in addition to vn lines, which should be able to be imported by MM. This is something I will work on another day.

Since I have edited the export script, this branch will now conflict with the other one I made a pull request for. Should I merge the two branches when I have finished the edits to the MM code?

[jmquigs]

That's cool! I'm interested in seeing if those blender-generated vectors work better. I was disappointed with the results I got from the DXMesh library, but I couldn't be sure if it was just because I had a "dirty mesh" with duplicate verts or faces and something like that.

You can go ahead and merge it into your existing PR if that is the easiest. It could just be a new feature of moving to the more recent version of blender.
FWIW your original PR will be hard to merge directly, because from what I can tell it doesn't seem backwards compatible, and it modifies the mmobj scripts in place (I would prefer to have separate script directories for both old and new versions of blender, so that I can continue to use 2.79b, since I have not yet mentally upgraded to the more recent versions.)

One thing I have strived for is to make sure the mmobj format is stable - for instance you don't need to reexport mmobj files when upgrading to a new version of MM. So any changes to the structure of the mmobj need corresponding changes in the F# loader so that it is able to load both old and new versions.

This would similarly apply for this change. if the bn and tn lines are available, then fine use that data, otherwise the F# code
should fall back to whatever it was doing before (even if that produces sub optimal results, its better than just straight failing to load a mod.)

[Frosferes2]

@jmquigs You're correct that the new scripts are not compatible with Blender 2.7x. I think it may be possible to have install.py recognise the Blender version you're using and install the appropriate scripts, but it's a question of if it is worth trying to implement this. The code changes I made to the normals writing protocol should be cross compatible, they would just need copying over. How you want to proceed here is up to you. I feel the majority of potential users will be on the latest Blender version, so maybe making the 2.7x scripts optional is best? Though, without modifying install.py, they would probably have to be installed manually. That, or they could be contained in their own individual nested zip files.

I am just making some changes to the F# code. I have added tangent and binormal resize arrays and log lines to readOBJ(). Is the mesh object from this function passed directly to modmBinormalTangent()? Also, if I'm not mistaken, it appears only the tangent is being passed to the renderer, and the binormal is not present in this function. Is the binormal computed internally by the renderer from the normal cross tangent? I am still getting familiar with F#, so apologies for the frequent questions. I will make the old tangent calculation a fallback for if the tangent/binormal data is missing from the mmobj.

Also, when I build the app and try to launch GW2 I get a popup with the error Start failed: Can't find source files for copy. What is the best way to fix this? Do I need to change the build location?

EDIT: After adding the bin with the x64 DX dlls to the repo, the game now launches, but MM does not load correctly. The log outputs this: ThreadId(1)/36329ms: Error creating CLR: CLRInitFailed("Error: Managed code version mismatch. Ensure that the d3d9.dll loaded by the game is the latest version. You may need to copy the new version in from your ModelMod directory. Click 'Start' in the ModelMod Launcher for more details.") The DX dlls in the GW2 directory are the same as the ones in the repo bin. If I launch MM from the exe in the release build, this error doesn't happen. Any idea what's causing this?

[Frosferes2]

I'm trying to figure out what components are missing from the build version that prevents it from working like the release. The setup instructions in the dev guide appear to be outdated as there is no longer a solution named ModelMod in the project. Unfortunately Im struggling to troubleshoot this myself due to my inexperience with VS.

[jmquigs]

@Frosferes2 You can get a "managed code version mismatch" if the F# code does not match the version the rust code expects.
The managed code version is here: https://github.com/jmquigs/ModelMod/blob/49f01fd5cc7139526769d789b1bb637ba6c0dd46/MMManaged/Interop.fs#L60C37-L60C37
While the rust code version is here:
https://github.com/jmquigs/ModelMod/blob/49f01fd5cc7139526769d789b1bb637ba6c0dd46/Native/dnclr/src/dnclr.rs#L85C10-L85C10

Both are 3 in the committed code. Since I presume you are not rebuilding the rust stuff, I would expect that to be "3". You can try printing the versions that are being compared in the F# code before it throws the exception to see what rust is passing in.

Its important for these versions to to be the same, because if they aren't the rust code won't have the proper layout of the F# interop interfaces, which can lead to mysterious crashes.

[Frosferes2]

@jmquigs Initialising logging early in Interop and printing the native versions for the Rust and F# code shows a discrepancy as you said: the Rust code is version 2 and the F# is 3. I am using the d3d11.dll from the MM Beta 1.2.0.66, which I assume the native version is read from when the CLR is initialised. The compile date of the rust code reported by the log file aligns with when this release was published: ModelMod initialized, built with rustc: 1.68.0 2023-03-06, git hash: be4246665daedbf6f07aab3582513ecf30230506, build date: 2023-03-24 14:18:05.456161900 +00:00. The fact that the packaged build works by itself leads me to believe the native version is 2 in the build but not the repo for some reason. Am I wrong here? It's a little odd.

Removing the fail condition allows the CLR to initialise regardless, and the MM UI loads in game. But I assume this is not best practice, since you said version differences can lead to unexpected errors. Do you know what the differences are between native Rust versions 2 and 3? If I were to build the version 3 Rust binaries myself, how would I go about doing it?

[Frosferes2]

@jmquigs I figured out how to build the Rust binaries and moved and renamed the newly generated hook_core.dll to the MM bin. The build is now fully working and I have been able to start debugging.

Exciting news! I've discovered the source of the lighting bug in the F# code and it is not related to the tangents after all. When WriteElement is called, both the vertex index and vertex normal index are extracted from the element index record v. For some reason I am not sure of, the vertex normal indices are out of sync with the actual vertex indices in this record. I wrote both these values to the log and found that the vertex normal index values would not exceed around 300 and would often repeat, despite the mesh containing several thousand vertices. This would explain why all the normals face a similar direction: they are all copies of only the first few vertex normals.

I changed both modmNormal and modmBinormalTangent to call the normal and tangent vectors respectively using the vertex index instead of the normal index and this seems to almost completely fix the lighting. There is still some weirdness, but this might be due to my texture or it could be related to the axis basis of the vectors as I mentioned in a past comment. I say this because it seems to me like the lighting is correct, but oriented in the wrong direction. I will have to test this further.

This method will cause an error if the length of the vertex list does not exactly match the normals list, and I still want to implement the pre-calculated tangent space from Blender. It will take some more F# learning to implement this intelligently though.

[Frosferes2]

@jmquigs Sorry for the incessant comments. I just wanted to let you know that I believe I've managed to fix the issues with blend weight mode. Changing line 466 from UNorm to UInt allows the mode to load. However, after doing this I got the same bug as I mentioned in this comment from nearly 2 years ago.

After some investigation, it turns out the problem was with writeMeshBW. Occasionally when adding a new vertex to the mesh, its weights will be rounded in the byte array such that the sum of the array is not 255. When this happens, the vertex does not have a valid binding and stretches off in world space.

To correct this, we can add the difference between 255 and the array sum to the highest (first) weight value. The sum of a byte array is also a byte, and so will overflow if over 255. To solve this we can initialize the weight array as ints, and then convert to byte when passing to the binary writer. This seems to have resolved the issue as none of my remodels have has this stretching problem since implementing this.

I would like to commit the changes I've made to a new PR soon. The changes to the managed code can exist independently of the Python stuff, as I'm not sure how you would like to handle the Blender scripts. I'm still not too familiar with Git, so I don't want to flood the page with tiny PRs. When you can, please let me know how you want to go about this.

[Frosferes2]

@jmquigs Hi again. I just discovered that the position transformations that MM uses to translate between the coordinate systems for OGL and DX is incorrect. At present, going from Blender to DX gives MM the transformations: rot_y_180 rot_x_90. This results in transformations that are oriented correctly, but mirrored in the X axis. This can be easilly missed, as if the mesh is symmetrical in the X axis, then there is no noticable difference in terms of the mesh geometry.

Blender's coordinate system is right handed, whereas DX is left handed. There is therefore no way to map one onto the other without one axis being mirrored. If this were applied to the current transformation list, it would look like: flip_x rot_y_180 rot_x_90. However, a more direct translation list would be flip_z rot_x_90. According to the MS website, to translate a right handed coordinate system into a left handed one, we need to flip the sign on one of the axes, and reverse the winding on the mesh triangles by swapping the second and third vertex index of each triangle. Implementing a position flip is easy enough, but I am still working on the reverse winding.

This also means that the snapshot transformations are incorrect. Which expedites the need to have working snapshotting for DX11. Problem is, I really don't know what to do to fix this. So I would need some leads from you if I were to attempt it.

[jmquigs]

Hey @Frosferes2 sorry I've been not responding, my life over the past few weeks has been much more complicated than I'd like.

I've actually known about that inverted X for quite a while now but never fixed it. I think what be involved is for the snapshot code to come up with a new transform convention name and export the verts using that fixed convention. For backwards compatibility we'd still want to be able to load the old transform though (but they could continue to use the unfixed axis). Additional on a re-export the python code could optionally fix the export to use the new transform. I'd least that is how it would look in a perfect world.

The DX11 snapshotting code is actually sort of working as far as I recall though its been months since I looked at it. The issue is that with DX11 the game needs to be started with a special flag so that snapshottable data is copied into system memory. This is enabled with a "precopy" flag as seen here: https://github.com/jmquigs/ModelMod/blob/master/Native/hook_core/src/hook_device_d3d11.rs#L697 . Its ok to run the game in snapshot mode all the time, as it garbage collects data after a period of time, its just a bit of a performance drain, especially when the game is loading a lot of assets.

I had started to modify the launcher so that it could be started in either precopy or "playback" mode but never finished that work. I got stuck on what kind of design would be optimal to pass the flag to the rust code, a silly thing in retrospect. Its still pretty unlikely that I'll be able to get to it soon. But if you wanted to play around with snapshotting yourself you'd need to start there. You may need to build the rust code as I'm not sure if the beta version I made had the latest changes to enable that.

[Frosferes2]

Hi @jmquigs don't worry about not responding, life takes priority after all. I hope things settle down for you soon.

I have added a new 'flip' keyword to the 3D transform parser, similar to what is used in the 2D UV transform. This instructs the transformer to take the negative value of whatever vector element is being flipped. After doing this, changing an existing Blender mesh's transform strings from rot_x_90 rot_y_180 scale_0.1' to 'flip_z rot_x_-90 scale_0.1 is able to import the mesh such that it's chirality is the same between the game and Blender. However, since the Blender mesh is now mirrored compared to what it should be, it needs to be corrected by scaling all vertices along the X axis by -1 in edit mode, making sure the transform is centered on the object origin. I think the new transform basis should work for snapshotting as well once it's going. Perhaps this new basis could be added to the snapshot profiles list. Bare in mind that this basis affects the normals/tangents as well as vertex positions, so position mirroring will likely mean the lighting is also inverted.

On that note, I have been able to get the lighting on the imported mesh accurate enough that I can't tell if any further discrepancies are because of the transform basis or something to do with the mesh export step. This is something I will look into when snapshotting is going. I have been able to build the Rust code with Cargo locally by fudging the Git hash. I'll have a go at setting this precopy flag manually and see what happens. I have aimed to ensure the changes I made to the F# code are backwards compatible with previous mods. I take it you're not going to be available to review them at the moment given IRL commitments, right? Perhaps I will push my changes onto my MM branch and submit a pull request once everything is done. We can discuss individual code changes is said PR later, hopefully when you have more time.

[Frosferes2]

@jmquigs indeed you were right. Forcing GLOBAL_STATE.run_conf.precopy_data to true enables snapshotting again. I had to build the release version of the Rust native, as the debug version causes GW2 to crash. Is this a known issue?

Some initial observations about the snapshotting:

1. A position axis flip in the snapshot profile works to extract a mesh with the correct mirroring, but this also causes the face normals to be flipped. This is to be expected, but for some reason doesn't happen when reading a mmobj back into MM. I may need to set up a face winding reversal function to only call when snapshotting and not loading mods.
2. The snapshotter outputs several texture files per object: tex_0, tex_1 etc. Texture 0 is the diffuse map combined with the transparency layer, and opens as intended. However, some other texture types appear to be corrupted or perhaps incorrectly formatted, as they will not open in the image editors I tried. These maps could include normal, specular, metalic (in the case of armor) etc. I would like to be able to access these maps as well. I wonder if you know what the problem with them may be?
3. Certain components of some objects do not appear to snapshot correctly. For example, I am trying to snapshot the Dark Tyrant outfit, and some of the plate parts including the gauntlets and the chestplate are not appearing in the snapshot. Do you know of why only some of the outfit would be captured like this? Other parts like the hanging cloth in this outfit and several back items are also not captured. I assume this has something to do with the cloth physics these pieces have.
4. Snapshots don't seem to be generating the VBDecl file that MM wants to create a new mod. What does this file contain that is necessary for mods?

[jmquigs]

Hey @Frosferes2 , glad you got it sort of working

I always build in release build. If it crashes in debug, well that may or may not be a problem. Possibly the rust code is panicking, which would crash the whole game. I never use debug because its too slow. For iterative dev builds I use this script under git bash to copy the dll and run the game:
https://github.com/jmquigs/ModelMod/blob/master/Native/hook_core/rb.sh

On other issues:

1. I am not sure what is going on here
2. Yes I noticed this as well. Could be that the format is a newer dx11 specific format and tools don't know how to load it. It could also be that the d3dx save texture function doesn't know how to to interpret the data and is filling the file with garbage. d3dx is very old and very obsolete. Probably MM should use DirectXTex or some newer library to copy out those images. Or it could be that the data is specific to the shader being used at render time, in which case no tool would be able to preview it. The code to save textures is here, as you can see it just calls out to d3dx: https://github.com/jmquigs/ModelMod/blob/master/Native/d3dx/src/d3dx.rs#L203
3. Yes this is a long standing issue. The GW engines are odd in that they will render certain complex meshes both with GPU-based animation (which MM can modify) and software animation (which MM can't use) And it will use both on a single object at the same time. Software rendered pieces will be fully transformed into their world position and animation frame. If you load them up in blender (you can directly import them from the snapshot dir, you don't need to create a mod), you can click on them in the object explorer and use the snap->object to origin feature to view them. These cannot be modified using modelmod, though I believe you can hide them with a deletion mod.
4. VBDecl used to store the d3d9 vertex layout. In d3d11, a file containing vertex elements should be written out instead (though I can't remember if I did this or not in the the snapshot code). I also don't think I modified the mod creation tool to support that new file. Blender doesn't need it, but it is required when the mod is loaded, so the mod yaml file should reference it.

Of course feel free to fork and branch so that you can commit your work. I won't be offended, in fact I'm happy someone is finally working on it other than me. The project certainly has some limitations but its still good for some use cases, I'd like to see it continue to work with gw2 at least.