mail/deplibs.html - libtool - Git at Google

 <html>
 <head>
 <title>Libtool Inter-library Dependencies</title>
 <body bgcolor="#ffffff">
 <center><h1><img src="/graphics/libtool.gif" width=477 height=192
  alt="Libtool"></h1></center>

 <h1>Inter-library Dependencies</h1>

 <p>About twice a week, for the last five weeks, I've been receiving
 bug reports which tell me that libtool's inter-library dependency
 handling is broken.

 <p>I know.  I broke it intentionally, until I have the time to fix it
 myself, or somebody else takes the time to help me with it.

 <p>These same people often give me a simple one-line patch which
 re-enables my old, simplistic inter-library dependencies, but nobody
 seems to want to test things thoroughly and come up with a real
 solution.

 <p>If you don't care about the history, and you just want to help me
 out, jump to <a href="#solution">the bottom of this document</a>.

 <h2>Background</h2>

 <p>Libtool's basic premise is to make static and shared libraries
 behave the same way from a programmer's point of view.  This allows
 users to build a libtoolized package with or without shared libraries,
 determined at configuration time.  It does this by using a
 <dfn>libtool object</dfn> (<samp>.lo</samp>) and <dfn>libtool
 archive</dfn> (<samp>.la</samp>) abstraction, so that the package
 maintainer can use libtool to operate on these files without making
 any assumptions about their underlying representation.

 <p>For the most part, this abstraction works well, and has made
 libtool as popular as it is today.  Without this abstraction, it would
 be significantly harder to port libtool to new platforms.

 <p>Unfortunately, what this abstraction has also done is reveal some
 fundamental inconsistencies with most shared library implementations.
 Every shared library implementation works well for `hello world'-type
 examples, but very few are robust and well-designed so that libtool
 doesn't need special tricks in order to build correct, featureful
 shared libraries.

 <p>Providing inter-library dependencies is one feature that has
 revealed these kinds of inconsistencies.

 <h2>The problem</h2>

 <p>My original inter-library dependency code received rigourous testing
 in beta releases of <a href="http://www.red-bean.com/guile/">GNU
 Guile</a>.  As soon as the Guile people started using my code, I
 received a flood of bug reports.  People were reporting that
 <samp>libguile</samp> (Guile's main shared library) was failing to
 link, or that programs linked against <samp>libguile</samp> were
 dumping core.

 <p>Not good.

 <p>The Guile people chased this bug down to the following scenario:

 <ol>
 <li>The user's system has a static regexp library installed,
 <samp>librx.a</samp>.

 <li>Guile's <samp>configure</samp> script detects that the
 <samp>-lrx</samp> linker option can be used to link in
 <samp>librx</samp>.

 <li>When <samp>libguile</samp> is built, the <samp>-lrx</samp> linker
 option is used.

 <li>Some linkers fail at this point, because they don't allow shared
 libraries to contain or depend on static libraries.

 <li>If the linker didn't fail, then a few programs are linked against
 <samp>libguile</samp>.

 <li>On some systems, these programs core dump because
 <samp>libguile</samp> is a shared library that contains non-PIC code
 (from <samp>librx</samp>).
 </ol>

 <h2>The interim solution</h2>

 <p>I needed some way to respond to these reports.  I saw my options (in
 order of my preference) as:

 <ol>
 <li>Write code in <samp>ltmain.sh</samp> to prevent static libraries
 from appearing in inter-library dependencies.  This would take some
 work, but obviously is the best solution.

 <li>Find the systems that fail, and turn off inter-library
 dependencies on only those systems.

 <li>Force the package maintainer to guarantee that static libraries
 never appear in inter-library dependencies.
 </ol>

 <p>I immediately vetoed the last solution, because that would violate
 the whole point of using libtool, and would cause a lot of people to
 waste time solving a problem that really should be fixed by libtool.

 <p>I preferred the first solution, but at the time of the reports, I
 didn't see an obviously simple mechanism for detecting the difference
 between shared and static libraries.

 <p>So, in the meantime, I tried turning off inter-library dependencies
 on the systems that failed.

 <p>I quickly discovered, to my chagrin, that many systems fail.  So,
 it was be simpler for me to turn off <em>all</em> inter-library
 dependencies, then find out which systems work, rather than vice
 versa.

 <h2>The current situation</h2>

 <p>I've been busy trying to avoid bankruptcy.  It's been over three
 months since I first turned off inter-library dependencies, and I
 still haven't completed the solution I want.

 <p>I've been gearing up for the 1.1 release of libtool, because there
 is a high demand for a stable public release.  So, I'm not going to
 introduce any destabilizing changes to the inter-library dependency
 code until after 1.1 is released.

 <h2><a name="solution">The solution</a></h2>

 <p>So, I want to tell you how you can help me solve the various
 dilemmas surrounding this issue:

 <ol>
 <li>I need to find out more about the nature of the problems I ran
 into with Guile.  Unfortunately, I cannot reproduce them in simple
 tests on my own platform (i586-pc-linux-gnulibc1), even though I think
 they were reported here.  I need to find out which platforms already
 have perfect inter-library dependency support, how to work around the
 problem on other platforms, and, more importantly, exactly
 <em>why</em> some systems give me problems and others don't.

 <p>On correct platforms, you can link <em>any</em> static library
 against a shared library via the <samp>-lNAME</samp> option without
 the linker complaining, then link a program against this library and
 run it without dumping core.  I know that this scenario will always
 work fine on the following systems:

  <ul>
  <li>None reported yet.
  </ul>

 <p>I also know that on some systems, you can create a shared library
 linked against a static one, but running programs linked against such
 a library will dump core:

  <ul>
  <li>None reported yet.
  </ul>

 <p>Finally, there are some systems which won't even allow you to link
 a shared library against a static one:

  <ul>
  <li>Solaris 2.x
  </ul>

 <li>Help me figure out a good, portable way to detect if a given
 <samp>-lNAME</samp> option refers to a shared library or not, since
 that is needed as a workaround to the problem.  Some suggestions so
 far have been:

  <ul>
  <li>Link the library against a tiny test program, and:

   <ul>
   <li>run <samp>ldd(1)</samp> on the test program and search for
   <samp>libNAME</samp> in the ldd output.
   <li>use the <samp>-verbose</samp> flag for GNU ld in order to see
   which library is actually linked.
   <li>run some sort of other program to determine if the library was
   dynamic.
   </ul>

  <li>Track the <samp>-LDIR</samp> flags, do a search for the library,
  and then check whether it is shared by:

   <ul>
   <li>using the <samp>file(1)</samp> program.
   <li>looking at its suffix.
   </ul>
  </ul>

 <li>Contact any people you know who might be interested, get them to
 read this page, so that they can help me solve the problem.

 <li>Send me money, so that I can devote more of my time to solving the
 problem.
 </ol>

 <p>Thank you for your help, and have fun.

 <hr>
 <a href="http://www.gimp.org/"><img src="/graphics/gfx_by_gimp.gif"
 align="right" border=0></a>

 <p><a href="libtool.html">Back to the libtool home page.</a>

 <address>
 Gordon Matzigkeit <a href="mailto:gord@profitpress.com">&lt;gord@profitpress.com&gt;</a>
 </address>
 </html>
	<html>
	<head>
	<title>Libtool Inter-library Dependencies</title>
	<body bgcolor="#ffffff">
	<center><h1><img src="/graphics/libtool.gif" width=477 height=192
	alt="Libtool"></h1></center>

	<h1>Inter-library Dependencies</h1>

	<p>About twice a week, for the last five weeks, I've been receiving
	bug reports which tell me that libtool's inter-library dependency
	handling is broken.

	<p>I know. I broke it intentionally, until I have the time to fix it
	myself, or somebody else takes the time to help me with it.

	<p>These same people often give me a simple one-line patch which
	re-enables my old, simplistic inter-library dependencies, but nobody
	seems to want to test things thoroughly and come up with a real
	solution.

	<p>If you don't care about the history, and you just want to help me
	out, jump to <a href="#solution">the bottom of this document</a>.

	<h2>Background</h2>

	<p>Libtool's basic premise is to make static and shared libraries
	behave the same way from a programmer's point of view. This allows
	users to build a libtoolized package with or without shared libraries,
	determined at configuration time. It does this by using a
	<dfn>libtool object</dfn> (<samp>.lo</samp>) and <dfn>libtool
	archive</dfn> (<samp>.la</samp>) abstraction, so that the package
	maintainer can use libtool to operate on these files without making
	any assumptions about their underlying representation.

	<p>For the most part, this abstraction works well, and has made
	libtool as popular as it is today. Without this abstraction, it would
	be significantly harder to port libtool to new platforms.

	<p>Unfortunately, what this abstraction has also done is reveal some
	fundamental inconsistencies with most shared library implementations.
	Every shared library implementation works well for `hello world'-type
	examples, but very few are robust and well-designed so that libtool
	doesn't need special tricks in order to build correct, featureful
	shared libraries.

	<p>Providing inter-library dependencies is one feature that has
	revealed these kinds of inconsistencies.

	<h2>The problem</h2>

	<p>My original inter-library dependency code received rigourous testing
	in beta releases of <a href="http://www.red-bean.com/guile/">GNU
	Guile</a>. As soon as the Guile people started using my code, I
	received a flood of bug reports. People were reporting that
	<samp>libguile</samp> (Guile's main shared library) was failing to
	link, or that programs linked against <samp>libguile</samp> were
	dumping core.

	<p>Not good.

	<p>The Guile people chased this bug down to the following scenario:

	<ol>
	<li>The user's system has a static regexp library installed,
	<samp>librx.a</samp>.

	<li>Guile's <samp>configure</samp> script detects that the
	<samp>-lrx</samp> linker option can be used to link in
	<samp>librx</samp>.

	<li>When <samp>libguile</samp> is built, the <samp>-lrx</samp> linker
	option is used.

	<li>Some linkers fail at this point, because they don't allow shared
	libraries to contain or depend on static libraries.

	<li>If the linker didn't fail, then a few programs are linked against
	<samp>libguile</samp>.

	<li>On some systems, these programs core dump because
	<samp>libguile</samp> is a shared library that contains non-PIC code
	(from <samp>librx</samp>).
	</ol>

	<h2>The interim solution</h2>

	<p>I needed some way to respond to these reports. I saw my options (in
	order of my preference) as:

	<ol>
	<li>Write code in <samp>ltmain.sh</samp> to prevent static libraries
	from appearing in inter-library dependencies. This would take some
	work, but obviously is the best solution.

	<li>Find the systems that fail, and turn off inter-library
	dependencies on only those systems.

	<li>Force the package maintainer to guarantee that static libraries
	never appear in inter-library dependencies.
	</ol>

	<p>I immediately vetoed the last solution, because that would violate
	the whole point of using libtool, and would cause a lot of people to
	waste time solving a problem that really should be fixed by libtool.

	<p>I preferred the first solution, but at the time of the reports, I
	didn't see an obviously simple mechanism for detecting the difference
	between shared and static libraries.

	<p>So, in the meantime, I tried turning off inter-library dependencies
	on the systems that failed.

	<p>I quickly discovered, to my chagrin, that many systems fail. So,
	it was be simpler for me to turn off <em>all</em> inter-library
	dependencies, then find out which systems work, rather than vice
	versa.

	<h2>The current situation</h2>

	<p>I've been busy trying to avoid bankruptcy. It's been over three
	months since I first turned off inter-library dependencies, and I
	still haven't completed the solution I want.

	<p>I've been gearing up for the 1.1 release of libtool, because there
	is a high demand for a stable public release. So, I'm not going to
	introduce any destabilizing changes to the inter-library dependency
	code until after 1.1 is released.

	<h2><a name="solution">The solution</a></h2>

	<p>So, I want to tell you how you can help me solve the various
	dilemmas surrounding this issue:

	<ol>
	<li>I need to find out more about the nature of the problems I ran
	into with Guile. Unfortunately, I cannot reproduce them in simple
	tests on my own platform (i586-pc-linux-gnulibc1), even though I think
	they were reported here. I need to find out which platforms already
	have perfect inter-library dependency support, how to work around the
	problem on other platforms, and, more importantly, exactly
	<em>why</em> some systems give me problems and others don't.

	<p>On correct platforms, you can link <em>any</em> static library
	against a shared library via the <samp>-lNAME</samp> option without
	the linker complaining, then link a program against this library and
	run it without dumping core. I know that this scenario will always
	work fine on the following systems:

	<ul>
	<li>None reported yet.
	</ul>

	<p>I also know that on some systems, you can create a shared library
	linked against a static one, but running programs linked against such
	a library will dump core:

	<ul>
	<li>None reported yet.
	</ul>

	<p>Finally, there are some systems which won't even allow you to link
	a shared library against a static one:

	<ul>
	<li>Solaris 2.x
	</ul>

	<li>Help me figure out a good, portable way to detect if a given
	<samp>-lNAME</samp> option refers to a shared library or not, since
	that is needed as a workaround to the problem. Some suggestions so
	far have been:

	<ul>
	<li>Link the library against a tiny test program, and:

	<ul>
	<li>run <samp>ldd(1)</samp> on the test program and search for
	<samp>libNAME</samp> in the ldd output.
	<li>use the <samp>-verbose</samp> flag for GNU ld in order to see
	which library is actually linked.
	<li>run some sort of other program to determine if the library was
	dynamic.
	</ul>

	<li>Track the <samp>-LDIR</samp> flags, do a search for the library,
	and then check whether it is shared by:

	<ul>
	<li>using the <samp>file(1)</samp> program.
	<li>looking at its suffix.
	</ul>
	</ul>

	<li>Contact any people you know who might be interested, get them to
	read this page, so that they can help me solve the problem.

	<li>Send me money, so that I can devote more of my time to solving the
	problem.
	</ol>

	<p>Thank you for your help, and have fun.

	<hr>
	<a href="http://www.gimp.org/"><img src="/graphics/gfx_by_gimp.gif"
	align="right" border=0></a>

	<p><a href="libtool.html">Back to the libtool home page.</a>

	<address>
	Gordon Matzigkeit <a href="mailto:gord@profitpress.com"><gord@profitpress.com></a>
	</address>
	</html>