gdb/python: stop using PyObject_IsInstance in py-disasm.c
The PyObject_IsInstance function can return -1 for errors, 0 to
indicate false, and 1 to indicate true.
I noticed in python/py-disasm.c that we treat the result of
PyObject_IsInstance as a bool. This means that if PyObject_IsInstance
returns -1, then this will be treated as true. The consequence of
this is that we will invoke undefined behaviour by treating the result
from the _print_insn call as if it was a DisassemblerResult object,
even though PyObject_IsInstance raised an error, and the result might
not be of the required type.
I could fix this by taking the -1 result into account, however,
gdb.DisassemblerResult cannot be sub-classed, the type doesn't have
the Py_TPFLAGS_BASETYPE flag. As such, we can switch to using
PyObject_TypeCheck instead, which only return 0 or 1, with no error
case.
I have also taken the opportunity to improve the error message emitted
if the result has the wrong type. Better error message make debugging
issues easier.
I've added a test which exposes the problem when using
PyObject_IsInstance, and I've updated the existing test for the
improved error message.
Approved-By: Tom Tromey <tom@tromey.com>
3 files changed
