tree 11af8fe5ed84d1d8083f9df62e24ef50e864de1d
parent 7cee18263c234073bfe88cbc962b1fc68509df82
author Nick Alcock <nick.alcock@oracle.com> 1560956212 +0100
committer Nick Alcock <nick.alcock@oracle.com> 1561118642 +0100

libctf: fix ctf_open endianness problems with raw CTF files

ctf_open (or, rather, ctf_fdopen, which underlies it) has several
endianness problems, even though it was written after the
endian-swapping code was implemented, so should have been endian-aware.

Even though the comment right above the relevant check says that it wil
check for CTF magic in any endianness, it only checks in the native
endianness, so opening raw LE CTF files on BE, or vice-versa, will fail.
It also checks the CTF version by hand, without ever endianness-swapping
the header, so that too will fail, and is entirely redundant because
ctf_simple_open does the job properly in any case.  We have a similar
problem in the next if block, which checks for raw CTF archives: we are
checking in the native endianness while we should be doing a le64toh()
on it to check in little-endian form only: so opening CTF archives
created on the local machine will fail if the local machine is
big-endian.

Adding insult to injury, if ctf_simple_open then fails, we go on and try
to turn it into a single-element CTF archive regardless, throwing the
error away.  Since this involves dereferencing null pointers it is not
likely to work very well.

libctf/
	* ctf-open-bfd.c: Add swap.h and ctf-endian.h.
	(ctf_fdopen): Check for endian-swapped raw CTF magic, and
	little-endian CTF archive magic.  Do not check the CTF version:
	ctf_simple_open does that in endian-safe ways.  Do not dereference
	null pointers on open failure.