Set DYLD_LIBRARY_PATH when running tidy5 during the build

[ryandesign]

I'm the maintainer of tidy in MacPorts. I'm configuring tidy 4.9.24 to build with shared library support enabled. The build fails with this message:

./tidy5 -xml-help > /opt/local/var/macports/build/_Users_rschmidt_macports_dports_www_tidy/tidy/work/tidy-html5-ca06201c3abb3829d6be6465e433ac72c8b15449-x86_64/build/cmake/tidy-help.xml
dyld: Library not loaded: /opt/local/lib/libtidy5.4.dylib
  Referenced from: /opt/local/var/macports/build/_Users_rschmidt_macports_dports_www_tidy/tidy/work/tidy-html5-ca06201c3abb3829d6be6465e433ac72c8b15449-x86_64/build/cmake/./tidy5
  Reason: image not found
/bin/sh: line 1:  2307 Trace/BPT trap: 5       ./tidy5 -xml-help > /opt/local/var/macports/build/_Users_rschmidt_macports_dports_www_tidy/tidy/work/tidy-html5-ca06201c3abb3829d6be6465e433ac72c8b15449-x86_64/build/cmake/tidy-help.xml
make[2]: *** [man] Error 133

The problem is that the tidy5 program that was just compiled cannot find the libtidy5 library it needs because it has not yet been installed. The solution is to set the DYLD_LIBRARY_PATH environment variable to the absolute path to the build/cmake directory in which the library can be found.

[geoffmcl]

@ryandesign, I understand the problem. cmake is trying to run tidy5 before the shared libraries are installed!

I would certainly seek some advice on how to overcome this problem!

One idea might be to specifically set the environment variable you mentioned with -

  set(ENV{DYLD_LIBRARY_PATH} "${CMAKE_BINARY_DIR}")

Maybe someone will get a chance to experiment with this... would probably need a switch around if UNIX or APPLE concerning the variable name...

Another idea would be to add an option, like say -

option(BUILD_MAN_PAGE "Set ON to generate the man pages" OFF)

Then later, add to the manual build section -

if (UNIX AND BUILD_MAN_PAGE)

This would get through the initial build, which could then be installed. Then cmake is run a second time adding -DBUILD_MAN_PAGE:BOOL=ON. Now tidy would correctly find the installed shared libraries...

And of course this only happens if configuring to build the shared libraries... so another possible options is to ALWAYS build the static library, and always link this with the console app, tidy, and that could be separate to whether shared libraries are also built/installed...

Perhaps Issue #190 is related to this, since @skynet is also trying to build and use the shared library...

As indicated could really do with some unix HELP in getting this chicken and egg situation working the best possible way ;=()

[ryandesign]

You can conditionalize the use of DYLD_LIBRARY_PATH if you like, but only OS X uses it, so there would be no harm in setting it on other systems. Similarly there is no harm to setting LD_LIBRARY_PATH on OS X since it does not use it.

Attempting to build without building manpages, then installing the program and library, then building and installing the manpages isn't going to be workable in a package management system like MacPorts. MacPorts will build the entire software, then install the entire software. There is no opportunity to do any additional building after installation.

Shared libraries are preferable on OS X. Use of static libraries is discouraged on OS X.

I am already working around this problem in MacPorts by having MacPorts set DYLD_LIBRARY_PATH as an environment variable before calling make, but you should set it yourself in your build system so that this is not necessary.

[geoffmcl]

@ryandesign now building BOTH static and shared libraries... version 4.9.30 onward...

Due to the fact windows uses the same archive library name, have appended and 's' to the static library, and the tidy console app will link with this library...

Should now no longer be required to set the DYNA_LIBRARY_PATH... but will do no harm ;=))

This change may also assist with issue #190

[geoffmcl]

Now building both static library, for linking with tidy, AND the shared libraries for install...

No comments for many weeks, so closing this for now...

Feel feel to re-open, or post a new issue... thanks...

[ryandesign]

@ryandesign now building BOTH static and shared libraries... version 4.9.30 onward...

Ok. We were already enabling both static and shared libraries in the MacPorts build of tidy, and this seems like a reasonable default.

Due to the fact windows uses the same archive library name, have appended and 's' to the static library,

That's weird. I do see a two other MacPorts ports I have installed that use a "_s" suffix on static libraries, but it is definitely not the norm to suffix static libraries on UNIX. But I have no experience with Windows.

and the tidy console app will link with this library...

That's weird, in that it's an unnecessary waste of disk space. (Now there's two installed copies of the static library—one in libtidy5s.a and one in the tidy executable.)

Should now no longer be required to set the DYNA_LIBRARY_PATH... but will do no harm ;=))

Correct, 4.9.35 builds fine without DYLD_LIBRARY_PATH set.

[geoffmcl]

@ryandesign glad to hear it builds fine without the need for DYLD_LIBRARY_PATH set.

I guess there is no standard for the static library suffix, but I too have seen some that use s, so chose that. And some that really spell it out like _static, for freeglut, libpng, and zlib with static. Some go the other way and rename the DLL import library, adding _imp to the shared lib like libcurl. Assimp even adds the bitness with _static32...

And boost goes absolutely MAD with the name, like libboost_system-vc100-mt-1_53.lib. This is version 1.53 created using MSVC10. Here the mt denotes the use of the static runtimes, and would be md for the dynamic runtimes! Not quite the same, but similar idea.

The important issue is that they can NOT have the same name in Windows, since they both have the same .lib extension. This is not a problem in unix since one is .a (or .la), and the other is .so, with soft links to the SO version, like .so.4 and .so.4.9.35, etc...

But changing the static library name also seems good for unix, in that probably most prefer to use the shared. The developer has to really know about the static library name convention, and make a specific provision to use that if desired.

And NO, there are NOT two installed copies of the static library! One of the good things about a static library is that the linker only extracts the services needed when building tidy5.exe. I do NOT think is just embeds the whole library in the binary.

So while tidy5s.lib weighs in at (a meagre!) 1.5 MB, tidy5.exe is only 410 KB ;=)). This seems the same in unix, with the static library libtidy5s.a is 836 KB, and again tidy5 is 575 KB.

But anyway, in todays market of massive cheap HDD space, tidy is tiny, very light weight in total compared to many other packages ;=)).

What is still to be done is for me to dig up the courage and interest to build a release 5.0.0, or at least a 5.0.0RC1, and drop this 5 suffix... Have scheduled that for the end of this month, but have put it off several time before ;=() for various reasons...

[ryandesign]

I guess there is no standard for the static library suffix, but I too have seen some that use s, so chose that. And some that really spell it out like _static, for freeglut, libpng, and zlib with static. Some go the other way and rename the DLL import library, adding _imp to the shared lib like libcurl. Assimp even adds the bitness with _static32...

And boost goes absolutely MAD with the name, like libboost_system-vc100-mt-1_53.lib. This is version 1.53 created using MSVC10. Here the mt denotes the use of the static runtimes, and would be md for the dynamic runtimes! Not quite the same, but similar idea.

According to the documentation, "-mt" means multithreaded (it means that on UNIX too). Static libraries are indicated with the "lib" prefix while import libraries do not use a prefix.

The important issue is that they can NOT have the same name in Windows, since they both have the same .lib extension. This is not a problem in unix since one is .a (or .la), and the other is .so, with soft links to the SO version, like .so.4 and .so.4.9.35, etc...

That's the relevant point: you're describing the situation on Windows, and are applying it to UNIX, where it isn't applicable.

Note that the static library is .a, the shared library is .dylib on OS X and .so on other UNIX systems, and .la is the libtool control file which describes the shared library.

But changing the static library name also seems good for unix, in that probably most prefer to use the shared. The developer has to really know about the static library name convention, and make a specific provision to use that if desired.

My point is that it is extremely uncommon for static and shared library names to differ on UNIX. (Out of almost 1000 static libraries installed on my system by MacPorts, I only spotted 2 using a "_s" suffix.) Your examples libpng, zlib and boost all use the same library names on UNIX, with the only difference being the extension. (On my Mac, as installed by MacPorts, I have libpng.a and libpng.dylib; libz.a and libz.dylib; libboost_system-mt.a and libboost_system-mt.dylib.) If you want your software to be different, ok, but if you want to do what people expect, then the way you had it before was better.

And NO, there are NOT two installed copies of the static library! One of the good things about a static library is that the linker only extracts the services needed when building tidy5.exe. I do NOT think is just embeds the whole library in the binary.

So while tidy5s.lib weighs in at (a meagre!) 1.5 MB, tidy5.exe is only 410 KB ;=)). This seems the same in unix, with the static library libtidy5s.a is 836 KB, and again tidy5 is 575 KB.

I agree that the size of the tidy executable is smaller than the size of the static library, so presumably only the parts of the static library that are used are embedded into the executable. But that's still wasted space that would not be wasted if you used the shared library, like you did before.

But anyway, in todays market of massive cheap HDD space, tidy is tiny, very light weight in total compared to many other packages ;=)).

Which is still not a good reason to waste space.

If you don't want to change it again, it's fine; it does work the way it is now. I'm just saying I disagree with your reasons for having made the changes.

[geoffmcl]

@ryandesign, yes a small correction ;=(). In boost's case -mt does indeed mean just multithreaded. Should have check their documention!!!

On first glance I assumed it was related to the MSVC /MT compiler flag, which denotes multithread and static version of the run-time library, as opposed to /MD, which denotes multithread and DLL version of the run-time library. See documentation. Boost has other markers for this... lesson: never assume ;=))

And now checking further into a recent Windows 7 MSVC10 build of boost libraries, it seems they can build BOTH versions, with an -s and without... but not for all libaries... and likewise with and without -mt for some... as well as with and without -gd (debug) for all... Usually at least 4/5/6 versions for each library... some 272 32-bit, and 266 64-bit libraries. WOW! Impressive, and not really MAD!