we all know floating point numbers can be treacherous, but what are specific examples of when they've betrayed you?

so far I have:

* money calculations where you need to get the exact correct answer
* twitter's tweet IDs are bigger than 2^53 and can't be represented as a Javascript number, you have to use a string
* implementing numerical algorithms (like matrix multiplication) (I've never done this)

Follow

@b0rk

I would phrase the money thing rather as "the exact correct answer _according to rules incompatible with ones for FP_".

I think there are very many problems that are instances of "implementing a numerical algorithm", even if people who don't think in these terms won't call them that. For example, computing a sum of a long sequence of numbers (where you can acquire errors by adding many numbers that are below the smallest representable difference in the sum accumulated so far).

Sign in to participate in the conversation
Qoto Mastodon

QOTO: Question Others to Teach Ourselves
An inclusive, Academic Freedom, instance
All cultures welcome.
Hate speech and harassment strictly forbidden.