[gdb/testsuite] Fix some incorrect uses of decimal

Dejagnu defines $decimal as:
...
set decimal "\[0-9\]+"
...

If we add an equivalent definition in lib/gdb.exp:
...
set decimal "(?:\[0-9\]+)"
...
we run into this type of ERRORs:
...
couldn't compile regular expression pattern: parentheses () not balanced
...

This is due to using "\[$decimal\]" instead of "\\\[$decimal\\]" or
"[subst_var {\[$decimal\]}]".

Add such definitions of decimal and hex, and apply the same technique to
positive and octal.

Fix the errors mostly using subst_var.

Tested on x86_64-linux.