rescoff: ensure file is PE

read_coff_rsrc makes one check on object file contents, the existence
of a .rsrc section.  It doesn't check that the file is PE but blindly
accesses bfd pe_data.  Fix that by adding the necessary checks.
Also, the "resources nest too deep" error isn't an overrun, ie. the
"address out of bounds" message isn't correct.  Fix that too.