x86-64: conditionalize tests using --32

Using this option doesn't really work when no support for any 32-bit
target was configured in (as is the case for at least cloudabi and
rdos).