Skip to content

Cross-compile Windows binaries on Linux #405

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 28 commits into from
Jun 29, 2020
Merged

Cross-compile Windows binaries on Linux #405

merged 28 commits into from
Jun 29, 2020

Conversation

@OJ
Copy link
Contributor

@OJ OJ commented Jun 4, 2020 •
edited by smcintyre-r7
Loading

This PR requires the following to be merged before landing:

Description

This PR contains changes to the CMake build that allow for most of the Meterpreter DLLs to be built from Linux. The only two things that I can't get working with the mingw-w64 compiler are ext_server_python and ext_server_powershell. I think that with these two we're hitting limitations of the toolsets, so I'm not sure how to proceed from here. COM related stuff is a pain, which is needed for Powershell, and there's a few assembly quirks that are killing it for Python.

So the full list of things that do build fine is everything apart from the above... even kiwi builds!

I made heavy use of @plowsec's work as a reference point for getting this going. While I didn't directly pull in any of his code from his PRs (mostly because the structure I already had in place from doing CMake builds on Windows), his efforts were invaluable in getting me going. So .. thank you!

The builds on Linux aren't clean. There are a lot of warnings and I don't think it's necessary at this point to go through each and try to remove them. This is something that can and should be done over time, rather than a large up-front activity.

This also contains changes to the Dockerfile that's used for building payloads. More information on that below.

Building

From my experience, this is not as simple as I was hoping and varies across Linux distros. I had a completely sh!tfight trying to get it working on Fedora. I learned (the hard way) that Fedora doesn't have the right compilers available through dnf. To make it work you'd have to build the compilers and install them manually.

The primary issue here is that threading-related stuff in C++ code ends up with a dependency on threading libraries. By default, if the standard compiler (such as the one on Fedora) is used, then you end up with a dependency on the winpthread library. Given that this library can't be statically linked, it results in binaries that have a hard-dependency on a DLL that almost certainly doesn't exist on the target machine. Hence, they crash when loaded.

The win32 versions of the compiler result in the C++ threading functionality being tied to Windows threads, which means the winpthread library dependency goes away. This is the default compiler that is available on Ubuntu. While doing the initial work on this PR, I had intended to build a docker container that could be used to build things easily so long as Docker is installed. After discovering this issue with linking against winpthread, I switched to the Docker option immediately.

I had to update the base Ubuntu version in the Dockerfile so that the appropriate tooling was available, this might have caused issues with the Java build based on what I saw while trying to build Java. I'd like to make sure I haven't done anything stupid here, so if I could please get @timwr and @jmartin-r7 to validate this I'd appreciate it.

Even if you run a distro with everything set up properly, I would recommend building directly out of the docker container.

Docker container changes

  • As already mentioned, the base Ubuntu version was changed.
  • The mingw-w64 toolset was added.

Makefile and CMakeLists

The CMakeLists.txt and Makefile have had a bunch of options now that allow for compilation if specific entities. I have not yet ported this to the Windows side.

These commands exist and result in building using the toolset installed on the host itself (not the container)

  • make meterpreter - builds all meterpreter binaries.
  • make meterpreter-x64 - builds all x64 meterpreter binaries.
  • make meterpreter-x86 - builds all x64 meterpreter binaries.
  • make meterpreter-metsrv - builds x86 and x64 metsrv binaries.
  • make meterpreter-metsrv-x64 - builds x64 metsrv binary.
  • make meterpreter-ext-stdapi-x64 - builds x64 stdapi binary.
  • make meterpreter-plugin-screenshot - builds x86 and x64 screenshot binaries.
  • ... and so and so forth for all the things.

All of the above commands have docker-based equivalents.

  • make docker-container - builds the docker container from the Dockerfile.
  • make docker* - this is the same as invoking make meterpreter* from inside the docker container and should work with all the examples shown above.

The commands that are prefixed with docker (other than docker-container) all mount the current folder into the container so that the container can build the current version of the source. The ID of the current user is passed to Docker and that's the context that is used when compiling. This means that the binaries that are generated on the file system are owned by the current user.

Sample run

Kicking off...
image

Finishing up...
image

Verification

From the c/meterpreter folder ...

  • Make sure that running the make-cmake.bat builds on Windows still works without any issues/warnings.
  • Make sure that running the make.bat builds on Windows still works without any issues/warnings.
  • Make sure the docker container builds by running make docker-container.
  • Make sure the binaries build without error by running make docker
  • Copy the binaries over from the output folder to the data/meterpreter folder ready for testing (make install does this if you have the projects checked out side by side).
  • Fire up MSF and generate some payloads. Go nuts with testing ALL THE THINGS.

Obviously it'd be ideal to get a full regression test done on this, but I'm not sure that's going to be possible.

Before landing

  • Build the docker container from the new Dockerfile and make sure that's pushed up to the Docker hub.
  • Modify the Makefile so that the DOCKER_CONTAINER value is updated to point back to rapid7:build:meterpreter
  • Validate the all the stuff still works once the Docker container has been changed.

Future work

  • Get the python and powershell extensions building.
  • Drink. A lot.

Thanks

Cheers again to @ccondon-r7, @bcook-r7, @adfoster-r7 and @jmartin-r7 for the help/support behind the scenes. Huge shout out t @plowsec for breaking the ground on this.

@OJ OJ requested review from bcook-r7, jmartin-tech and timwr June 4, 2020 00:40
@OJ OJ mentioned this pull request Jun 4, 2020
Merged
@OJ
Copy link
Contributor Author

OJ commented Jun 4, 2020

I'll sort the conflicts out in a sec. DOH!

@OJ
Copy link
Contributor Author

OJ commented Jun 4, 2020

Actually, I will resolve the conflicts when the other PRs have landed :)

@gentilkiwi
Copy link

I love your work @OJ <3, but aren't file bigger in size in this particular mode ?

@OJ
Copy link
Contributor Author

OJ commented Jun 7, 2020 via email

@OJ OJ added the delayed label Jun 10, 2020
@OJ
Copy link
Contributor Author

OJ commented Jun 10, 2020

Deliberately marking this as delayed while we merge all the other stuff first. When they're done, I'll make sure this gets updated/rebased/etc onto 6.x so that the merge is clean.

This was referenced Jun 19, 2020
Closed
Closed
Closed
Closed
@pussinboots1992
Copy link

Awesome stuff, thanks !!! AV vendors will hate this... :-)

@OJ
Copy link
Contributor Author

OJ commented Jun 19, 2020

I'll sort out the conflicts on this as soon as #401 has been sorted.

@pussinboots1992 pussinboots1992 mentioned this pull request Jun 20, 2020
Merged
7 tasks
@pussinboots1992
Copy link

I was able to run "make docker-container" without any issues, I did see some red when building the image though...

@OJ , this is what I get when running "make docker":

root@kali:~/dev/metasploit-payloads/c/meterpreter# make docker
-- Build Type not specified, defaulting to 'Release'.
-- Configuring done
-- Generating done
-- Build files have been written to: /meterpreter/workspace/build/mingw-x86
make[1]: Entering directory '/meterpreter/workspace/build/mingw-x86'
make[2]: Entering directory '/meterpreter/workspace/build/mingw-x86'
make[3]: Entering directory '/meterpreter/workspace/build/mingw-x86'
make[3]: Leaving directory '/meterpreter/workspace/build/mingw-x86'
[ 26%] Built target jpeg
make[3]: Entering directory '/meterpreter/workspace/build/mingw-x86'
make[3]: Leaving directory '/meterpreter/workspace/build/mingw-x86'
make[3]: Entering directory '/meterpreter/workspace/build/mingw-x86'
[ 27%] Building C object metsrv/CMakeFiles/metsrv.dir/meterpreter/source/metsrv/base.c.obj
In file included from /meterpreter/source/metsrv/base.c:5:
/meterpreter/source/metsrv/metsrv.h:21:10: fatal error: ../ReflectiveDLLInjection/inject/src/GetProcAddressR.h: No such file or directory
   21 | #include "../ReflectiveDLLInjection/inject/src/GetProcAddressR.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [metsrv/CMakeFiles/metsrv.dir/build.make:64: metsrv/CMakeFiles/metsrv.dir/meterpreter/source/metsrv/base.c.obj] Error 1
make[3]: Leaving directory '/meterpreter/workspace/build/mingw-x86'
make[2]: *** [CMakeFiles/Makefile2:394: metsrv/CMakeFiles/metsrv.dir/all] Error 2
make[2]: Leaving directory '/meterpreter/workspace/build/mingw-x86'
make[1]: *** [Makefile:84: all] Error 2
make[1]: Leaving directory '/meterpreter/workspace/build/mingw-x86'
make: *** [Makefile:27: meterpreter-x86-build] Error 2
make: *** [Makefile:312: docker] Error 2
root@kali:~/dev/metasploit-payloads/c/meterpreter#

@OJ
Copy link
Contributor Author

OJ commented Jun 21, 2020 via email

@pussinboots1992
Copy link

@OJ, sorry, I suck...it worked, had to create the output directory manually.

@OJ
Copy link
Contributor Author

OJ commented Jun 21, 2020 via email

OJ added 6 commits June 25, 2020 10:28
@OJ
Remove DLL exports from Meterpreter
e6441ad 
Add support for loading RDI-related stuff using ordinals instead of
function names. Remove exports from the extensions/etc. This is another
step in the direction to make the DLLs less obvious.

Extensions no longer have their own name in the library metadata.
They're all "extension.dll". Metsrv is now "server.dll" and the two
non-extensions are "plugin.dll". I was going for something a little less
obvious.

This required changes to the RDI functionality.
@OJ
Kinda almost sorta working for metsrv
060feec
@OJ
Get rid of the clang noise
aa3b586
@OJ
Get x86 working for metsrv
d820fb6
@OJ
Fix small compiler warning
dee84d6
@OJ
Lots of changes + stdapi support
effe6b9
@OJ
Copy link
Contributor Author

OJ commented Jun 25, 2020

OK did a rather gross rebase dance, but I think we're good now.

@jmartin-tech
Copy link
Contributor

jmartin-tech commented Jun 25, 2020 •
edited
Loading

Native build results from msbuild are showing an error with ext_server_kiwi:

22:28:44   ..\..\source\extensions\kiwi\mimikatz\mimikatz\modules\kuhl_m_standard.c(131): error C2308: concatenating mismatched strings [C:\Users\vagrant\metasploit-payloads\c\meterpreter\workspace\ext_server_kiwi\ext_server_kiwi.vcxproj]
22:28:44 
22:28:44     14 Warning(s)
22:28:44     1 Error(s)
22:28:44 
22:28:44 Time Elapsed 00:00:57.44
22:28:44 Finished 2020-06-25 03:28:43.088
22:28:44 Copying: ../c/meterpreter/output/elevator.x86.dll -> ./data/meterpreter/elevator.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_extapi.x86.dll -> ./data/meterpreter/ext_server_extapi.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_incognito.x86.dll -> ./data/meterpreter/ext_server_incognito.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_lanattacks.x86.dll -> ./data/meterpreter/ext_server_lanattacks.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_peinjector.x86.dll -> ./data/meterpreter/ext_server_peinjector.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_powershell.x86.dll -> ./data/meterpreter/ext_server_powershell.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_priv.x86.dll -> ./data/meterpreter/ext_server_priv.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_python.x86.dll -> ./data/meterpreter/ext_server_python.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_sniffer.x86.dll -> ./data/meterpreter/ext_server_sniffer.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_unhook.x86.dll -> ./data/meterpreter/ext_server_unhook.x86.dll
22:28:44 Copying: ../c/meterpreter/output/ext_server_winpmem.x86.dll -> ./data/meterpreter/ext_server_winpmem.x86.dll
22:28:44 Copying: ../c/meterpreter/output/metsrv.x86.dll -> ./data/meterpreter/metsrv.x86.dll
`

@OJ
Copy link
Contributor Author

OJ commented Jun 25, 2020 via email

@OJ
Copy link
Contributor Author

OJ commented Jun 25, 2020

Oh this might be VS 2013 specific!

@OJ
Copy link
Contributor Author

OJ commented Jun 25, 2020

Builds clean in VS2017 and VS2019, but breaks in VS2013. Could be the platform toolset. Will fix.

@OJ
Copy link
Contributor Author

OJ commented Jun 25, 2020

Yeah it's a platform toolset issue. v120_xp results in the break while v141_xp builds clean.

@jmartin-tech
Copy link
Contributor

Thanks for checking.

Test automation compile starts from a clean checkout with submodule init. Compile done on VS2013 edition env used when enabling compile on newer VS platforms.

@OJ
Copy link
Contributor Author

OJ commented Jun 25, 2020

To be honest, this shouldn't be something tied to the toolset version. VS2019 should have broken as well! Anyways, fix incoming in the mimikatz repo on the cross-compile-linux branch. Will update the ref in here as well.

@OJ
Copy link
Contributor Author

OJ commented Jun 25, 2020

OK should be good to go. Thanks @jmartin-r7 !

image

image

@jmartin-tech
Copy link
Contributor

Confirmed:

08:34:07     0 Error(s)
08:34:07 
08:34:07 Time Elapsed 00:02:09.48
08:34:07 Finished 2020-06-25 13:33:58.412

@smcintyre-r7 smcintyre-r7 self-assigned this Jun 25, 2020
@smcintyre-r7
Copy link
Contributor

While running docker-container I seem to be consistently getting this error message that doesn't stop the build operation. I just wanted to check if that's expected. The make docker command seems to be working, I'm still in the process of testing it. It appeared to have worked once but a git clean -xfd wasn't quite enough to have it build cleanly a second time. Should make clean be all that's required to run the operation again?

Docker Error Message 
Downloading from central: https://repo.maven.apache.org/maven2/log4j/log4j/1.2.12/log4j-1.2.12.jar
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-manager/2.0/plexus-compiler-manager-2.0.jar (4.6 kB at 10 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/commons-logging/commons-logging-api/1.1/commons-logging-api-1.1.jar
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.2.2/plexus-classworlds-2.2.2.jar (46 kB at 96 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/com/google/collections/google-collections/1.0/google-collections-1.0.jar
Downloaded from central: https://repo.maven.apache.org/maven2/commons-logging/commons-logging-api/1.1/commons-logging-api-1.1.jar (45 kB at 88 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/xbean/xbean-reflect/3.4/xbean-reflect-3.4.jar (134 kB at 257 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/log4j/log4j/1.2.12/log4j-1.2.12.jar (358 kB at 578 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-container-default/1.5.5/plexus-container-default-1.5.5.jar (217 kB at 343 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/com/google/collections/google-collections/1.0/google-collections-1.0.jar (640 kB at 1.0 MB/s)
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 18 source files to /metasploit-payloads/java/javapayload/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] Source option 5 is no longer supported. Use 6 or later.
[ERROR] Target option 1.5 is no longer supported. Use 1.6 or later.
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for JavaPayload for Metasploit (Parent project) 1-SNAPSHOT:
[INFO] 
[INFO] JavaPayload for Metasploit (Parent project) ........ SUCCESS [  0.005 s]
[INFO] JavaPayload for Metasploit ......................... FAILURE [ 10.614 s]
[INFO] Java Meterpreter (Parent project) .................. SKIPPED
[INFO] Java Meterpreter Shared ............................ SKIPPED
[INFO] Java Meterpreter ................................... SKIPPED
[INFO] Java Meterpreter StdApi Plugin ..................... SKIPPED
[INFO] Java Meterpreter Debug Loader ...................... SKIPPED
[INFO] Java Meterpreter extension archetype ............... SKIPPED
[INFO] JavaPayload Compatibility Checks (Parent project) .. SKIPPED
[INFO] JavaPayload Compatibility Checks (Java 1.6) ........ SKIPPED
[INFO] JavaPayload Compatibility Checks (Java 1.5) ........ SKIPPED
[INFO] JavaPayload Compatibility Checks (Android API) ..... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11.908 s
[INFO] Finished at: 2020-06-25T21:52:18Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project Metasploit-JavaPayload: Compilation failure: Compilation failure: 
[ERROR] Source option 5 is no longer supported. Use 6 or later.
[ERROR] Target option 1.5 is no longer supported. Use 1.6 or later.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :Metasploit-JavaPayload
make: *** [Makefile:8: android] Error 1
Removing intermediate container 985df15313ff
 ---> 42d64c728455
Step 14/22 : ENV ANDROID_HOME /usr/local/android-sdk
 ---> Running in dd90f07e84a5
Removing intermediate container dd90f07e84a5
 ---> b0fcc8a66ac5
Step 15/22 : ENV ANDROID_NDK_HOME /usr/local/android-ndk
 ---> Running in 291cb181c0da
Removing intermediate container 291cb181c0da
 ---> 2061826d5626
Step 16/22 : ENV PATH $PATH:$ANDROID_HOME/tools
 ---> Running in 26fd961de3f9
Removing intermediate container 26fd961de3f9
 ---> 833435ef11c4
Step 17/22 : ENV PATH $PATH:$ANDROID_HOME/platform-tools
 ---> Running in c5b3779a6084
Removing intermediate container c5b3779a6084
 ---> 837d4c16da9d
Step 18/22 : ENV PATH $PATH:$ANDROID_NDK_HOME
 ---> Running in 7b745db4bced
Removing intermediate container 7b745db4bced
 ---> 3ed6e69cc06d
Step 19/22 : ENV JENKINS_HOME /var/jenkins_home
 ---> Running in e87a6cb3cd1d
Removing intermediate container e87a6cb3cd1d
 ---> 111ad5e3e3be
Step 20/22 : RUN useradd -d "$JENKINS_HOME" -u 1001 -m -s /bin/sh jenkins
 ---> Running in fc0e8f69d883
Removing intermediate container fc0e8f69d883
 ---> f172d437df7b
Step 21/22 : VOLUME "$JENKINS_HOME"
 ---> Running in 7236f9684394
Removing intermediate container 7236f9684394
 ---> d54fd8c8a27b
Step 22/22 : RUN chown -R jenkins "$JENKINS_HOME"
 ---> Running in 0681e5fae63e
Removing intermediate container 0681e5fae63e
 ---> 5f91e9ca0529
Successfully built 5f91e9ca0529
Successfully tagged win-meterpreter-build:latest

@OJ
Copy link
Contributor Author

OJ commented Jun 25, 2020

@smcintyre-r7 That's odd, because @timwr's PR to include the updated alternatives should have fixed that. Tim, would you mind helping here please mate?

@timwr
Copy link
Contributor

timwr commented Jun 26, 2020

@smcintyre-r7 @OJ OJ#13 was the fix for that issue but I think the change got lost in the rebase.
I've sent another PR here: OJ#14 and confirmed it fixes the issue.
I will PR it to master/6.x too if needed

@OJ
Copy link
Contributor Author

OJ commented Jun 26, 2020

Thank you Tim!

@smcintyre-r7
Copy link
Contributor

Alright that commit fixed the Java error for me in docker, so thank you for that.

With that fix in place (and a manually created output) everything began to build correctly.

I validated both build bat scripts mentioned in the PR still build correctly without any errors. I also build and tested the binaries using docker. My tests involved building them, copying them as noted in the PR and before opening a Meterpreter session with PSexec and using the test script listed below. I tried both x86 and x64 on Windows 7 SP1 and x64 on Windows 10. Everything appeared to function correctly.

Meterpreter Test Script 
load espia      
load extapi     
load incognito   
load kiwi        
load lanattacks  
load peinjector  
load powershell  
load python      
load sniffer     
load unhook     
load winpmem

# Stdapi
run post/test/meterpreter
run post/test/file

# Espia
screengrab

# Extapi
window_enum
service_enum
clipboard_get_data

# Incognito
list_tokens -u

# Kiwi
creds_all
creds_wdigest
wifi_list

# Lanattacks
dhcp_start
dhcp_stop

# Sniffer
sniffer_interfaces

# Python
python_execute "print('Hello World')"
python_reset

# Powershell
powershell_execute "Write-Host 'Hello World'"

@OJ
Copy link
Contributor Author

OJ commented Jun 26, 2020 via email

@OJ
Copy link
Contributor Author

OJ commented Jun 26, 2020

Should be good now @smcintyre-r7 !

@smcintyre-r7
Copy link
Contributor

After talking to @jmartin-r7, for now I'm going to land this as is without pushing the new container publicly. The new container ID, once pushed however will be rapid7/msf-ubuntu-x64-meterpreter:latest.

I also verified that commit 9f859fe fixed the missing output directory.

With that sorted, I'm going to go ahead and get this landed now. Thanks @OJ!

@smcintyre-r7 smcintyre-r7 merged commit c04196c into rapid7:6.x Jun 29, 2020
@jmartin-tech
Copy link
Contributor

New container published to rapid7/msf-ubuntu-x64-meterpreter:latest see #417

@OJ
Copy link
Contributor Author

OJ commented Jun 29, 2020 via email

@jmartin-tech
Copy link
Contributor

oops, looks like I sent it to master already, will get that cherry-picked in

@OJ
Copy link
Contributor Author

OJ commented Jun 29, 2020

Ah cool! Thanks Jeffrey.

@OJ OJ deleted the cross-compile-linux branch June 29, 2020 22:17
@smcintyre-r7 smcintyre-r7 mentioned this pull request Sep 2, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@timwr timwr Awaiting requested review from timwr

@jmartin-tech jmartin-tech Awaiting requested review from jmartin-tech

@bcook-r7 bcook-r7 Awaiting requested review from bcook-r7

Assignees

@smcintyre-r7 smcintyre-r7

Labels

enhancement msf6 windows

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@OJ @gentilkiwi @pussinboots1992 @jmartin-tech @smcintyre-r7 @timwr @adfoster-r7