Incorrect number renderer precision with scientific notation

Incorrect number renderer precision with scientific notation

Seb33300Seb33300 Posts: 9Questions: 3Answers: 0
edited October 2 in Free community support

Description of problem:

If I define a column to be formatted with a precision of 2, I am expecting the precision to be the number of digits after the decimal (applied to the float number).

Examples:

Expected (precision 2):
123.456789 => 123.46
0.0000123456789 => 0.00

Current result (precision 2):
123.456789 => 123.46 (OK)
0.0000123456789 => 1.23 x 10-5 (WRONG = precision fixed after the 1 instead of decimal)

Expected (precision 7):
123.456789 => 123.4567890
0.0000123456789 => 0.0000123 => 1.23 x 10-5

Current result (precision 7):
123.456789 => 123.4567890 (OK)
0.0000123456789 => 1.2345678 x 10-5 (WRONG)

Replies

  • Seb33300Seb33300 Posts: 9Questions: 3Answers: 0

    Just did some research and precision may be "significant figures" or "decimals" depending on the context.

    So I believe here, the renderer is applying "significant figures"

  • allanallan Posts: 65,162Questions: 1Answers: 10,796 Site admin

    Hi,

    I've just created this little test case showing the issue: https://live.datatables.net/socirone/93/edit .

    The problem is this little piece of code.

    My intention had been for precision (significant figures), but I do see that the wording in the docs isn't quite right and I accept that this might not be the behaviour that is wanted.

    There should probably be an optional flag to say which display mode is to be used. I've noted that down to add.

    Allan

Sign In or Register to comment.