Skip to content

Reduced precision in output

using DEFAULT_NUM_FORMAT = "%f" causes the output to have reduced precision. example using an f-string:

f"{0.0008534400105:f}"
Out[76]: '0.000853'

# vs.
f"{0.0008534400105}"
Out[77]: '0.0008534400105'

This applies similarly to numpy.savetxt(fname, X, fmt='%.18e' ...; unfortunately, the default fmt also doesn't seem well-suited:

f"{0.0008534400105:.18e}"
Out[78]: '8.534400104999999726e-04'

  • One solution is of course to provide a specific format for the output (or a sequence of formats), that matches that of the input.
    • That still leaves unnecessary decimal places for VMISS, see #14 (closed)
  • using %g %e might be better-suited:
f"{0.0008534400105:e}"
Out[87]: '8.534400e-04'

f"{0.0008534400105:g}"
Out[88]: '0.00085344'
Edited by Florian Obersteiner
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information