> @jrose @fclc People who want NaN in their integers are welcome to do that in the privacy of their own bedrooms with new types they invent for this purpose.
That reminds me of OCaml's 63 bit integers. In the privacy of OCaml's own bedroom, they use one bit of their integers as a flag to distinguish pointers from integers.
The results are quite.. interesting. However, it does go to show that having slightly weird private integer types is actually totally workable, so the quote from the discussion might be meant as trolling, but it's less crazy than it sounds like.
The difference is that (a) that bit in OCaml has a purpose, while disregarding 0x80..00 hasn't; and (b) I assume that it was in OCaml from the beginning, so it wouldn't break existing software in probably very interesting ways (hence the suggestion that this should be done only "in the privacy of their own bedrooms with new types", i.e. not for already-used types)...
> > @jrose @fclc People who want NaN in their integers are welcome to do that in the privacy of their own bedrooms with new types they invent for this purpose.
And yet I bet they use IEEE754 floats without second thought.
I agree with the quote and use IEEE754 floats with second thought every time. In my perfect world floats don’t have inf, nan and base-2 exponent. And integers are 0-sized exponent floats.
Typically in the unsafe languages under discussion they use the IEEE754 types without even a first thought. For example you can expect them to sort such types (you can define rules for sorting them, but if you just didn't in C too bad your program has Undefined Behaviour, while in C++ today it's even worse you've silently not written a C++ program at all and it never had any defined behaviour even before it tried to execute your invalid sort)
At that point, why not introduce special i63 and u63 types and use the hidden bit 63 as 'soft carry flag' to detect over/undeflow. 63 bits of integer range ought to be enough for anyone ;)
(as soon as bit 63 is set, the number would be in 'overflow state', but at the same time the lower 63 bits would preserve the wrap-around value in case that's useful).
On the other hand, compilers are already free to check the CPU carry flag bit and trap on it. Those flag bits are just not exposed to programming languages (which IMHO is a serious 'dissonance' between assembly and high level languages), also CPU flags can't be carried around with their associated result value and go 'viral' like NaNs (but whether that's even useful instead of trapping immediately is another question).
...which is a bit of a weak reason when there are hardly any 32-bit CPUs around anymore though. Picking a narrower integer type makes sense in specific situations like tightly packed structs, but in that case it usually doesn't matter whether the integer range is 2^31 or 2^32, if you really need the extra bit of range it's usually ok to go to the next wider integer type.
DSPs already do something like this, offering the option of saturating arithmetic operations, in addition to the normal modulo arithmetic. It's the operation that's special, not the representation in memory.
The normal way to do it is to use the "ETSI Basic Operator" library, which provides saturating arithmetic operations, among other things [1]. If the hardware supports saturated arithmetic the compiler maps the operations to the relevant instructions, otherwise it provides a software implementation.
Example:
Modulo arithmetic: int16_t a,b,c; a = b + c;
Saturating 16-bit arithmetic using ETSI: int16_t a,b,c; a = add(b, c);
Processors with saturating capabilities usually clip at 0x7FFFFFF and 0x80000000, so the clipping thing has no relation to the original topic. But in fixpoint DSP, 0x8000 (add trailing zeros if you like) is often the only possibility to achieve a multiplicative identity by using a = - b * 0x8000, with the negation part of the assembly instruction.
Of course today in Rust we're not allowed to bless such types for ourselves, this is reserved to the standard library (core::num::NonZeroU32 is a 4-byte unsigned integer which can't be zero for example) because it's relying on a nasty permanently unstable compiler-only feature to say "Hey, not all possible representations the right size for this type are valid".
Rust has one very obvious set of types with such a niche, the references &T, but it also has not only the particular custom integers NonZero{U8,U16,U32,U64,I8,I16,I32,I64} and the Non-null pointers, but OS specific types (for Unix file descriptors, and Windows handles respectively) and of course char, it is 4 bytes but Unicode is explicitly limited to far fewer values, so that's a niche too.
But today making more is not allowed in stable Rust, so while my nook crate is an interesting curiosity (and some day I'll implement integer math traits etc.) it cannot be stabilised and I was persuaded that the path forward is a whole lot of work to deliver the Pattern Types feature which can do this stably.
Arbitrary niches aren't intended to be permanently unstable (at least not in the sense of explicit implementation details like lang items and intrinsics), people have expressed a desire to stabilize them for years (as discussed during the original nonzero int types RFC). If something has changed in that regard, I'd appreciate a link to a discussion.
The niche concept isn't permanently unstable, but the present mechanism is. Today the only way to make such a type is to use the magic attributes
#[rustc_nonnull_optimization_guaranteed]
#[rustc_layout_scalar_valid_range_start]
and
#[rustc_layout_scalar_valid_range_end]
As their names hint, these are permanently unstable compiler-only type hints.
I asked that some means be found to stabilize some equivalent hint (presumably with a name that doesn't look like a compiler-only feature) in one of the long series of tickets spawned after 3334 and I was persuaded that this is a terrible idea and won't happen.
Instead the proposed path forward is (at least was then) Pattern Types. Build at least an MVP of the full blown Pattern Types feature instead of a custom attribute.
In a new, green field programming language design: maybe.
You will have FFI interop issues: what if some foreign gives you such a value, which is correct in its domain. It will have to be specially handled and widened/promoted to a type that can hold it. It looks like friction, though.
In an existing language, like C? Making currently correct programs blow up with a trap exception is a nonstarter.
I think there is no hardware support for treating 0x80... as a trap, so if you actually want it to trap, rather than leaving it at undefined behavior, your compiler has to emit check code, at least in code paths where it is not obvious whether the value is or isn't a trap.
Also, it would be strangely inconsistent for the 0x80... to predictably trap wherever it occurs, but arithmetic overflow to have undefined consequences. The two want to be tied together.
Null is a valid value for pointers, but it is absolutely ordinary and normal for functions to refuse to accept null on the pain of crashing and burning.
No it isn't. In my experience, there are few higher quality signals a codebase is utter garbage than having `if (arg == NULL)` checks at the top of every function... Almost nothing in libc does that, it is entirely unnecessary.
I realize that wasn't your point, but this is an enormous pet peeve of mine...
This is a simplistic view. I implement nil/null values everywhere including integers and at the external border there is a test for that special value. It's not like you're using this sort of thing for just another number so it's natural that there is an adaption.
> You will have FFI interop issues: what if some foreign gives you such a value, which is correct in its domain. It will have to be specially handled and widened/promoted to a type that can hold it.
That's more or less what you have to do in Java if your FFI is passing you an unsigned type. You need to use a 64-bit signed long in Java to hold a uint32_t, and a BigInteger to hold a uint64_t.
I mean, you can certainly store that unsigned 32-bit int from the FFI call in a 32-bit signed Java int, but then you have to use special methods on the Integer class to do unsigned comparisons and arithmetic.
Agreed, this is a lot of friction, but there's long-standing precedent, at least.
Even if the current INT_MIN were redefined to be a NaN or a trap, that would do almost nothing to prevent signed integer overflow bugs. Sure, INT_MAX + 1 would be an error, but INT_MAX+2, INT_MAX+100 and so on would still be equal to INT_MIN + 1, INT_MIN + 99 etc.
Besides, -fwrapv was basically the default in most C compilers (and probably C++ as well) before the last 10 years or so. Many large systems thus use it today as well (e.g. the Linux kernel, I believe).
> Besides, -fwrapv was basically the default in most C compilers (and probably C++ as well) before the last 10 years or so.
It seems like people were complaining as early as 2007. Richard Biener (who's linked in the thread link below) said that GCC 2.95.3, which came in 2001, optimized away signed overflow checks as well.
> signed type overflow is undefined by the C standard, use unsigned int for the
> addition or use -fwrapv
You have GOT to be kidding?
All kinds of security issues are caused by integer wraps, and you are just telling me that with gcc 4.1 and up I cannot test for them for signed data types any more?!
You are missing the point here. There HAS to be a way to get around this. Existing software uses signed int and I can't just change it to unsigned int, but I still must be able to check for a wrap! There does not appear to be a work around I could do in the source code either! Do you expect me to cast it to unsigned, shift right by one, and then add or what?!
PLEASE REVERT THIS CHANGE. This will create MAJOR SECURITY ISSUES in ALL MANNER OF CODE. I don't care if your language lawyers tell you gcc is right. THIS WILL CAUSE PEOPLE TO GET HACKED.
I found this because one check to prevent people from getting hacked failed.
Thanks for looking into the history! Interesting to see that this wrong-headed behavior has now been around in GCC for longer than the status quo before.
Other compilers are different though - MSVC only seems to have started optimizing around this in ~2016 or so, icc seems to have done something around this in 2020, and I'm sure many embedded compilers ignore it even today. I couldn't find anything definitive on clang, but given its history I assume it probably always enforced this mistake.
I'd also note that several major C programs only work when signed integer overflow is defined: I mentioned Linux previously, but Python is another one. And Chrome and Firefox are considering enabling -fwrapv and equivalents.
R does this [1]. It is the sentinel that it uses internally to represent a missing value (i.e. NA) in integer arrays. Integer operations overflow to this value.
Wait so we're going to give up -fwrapv (a UB mitigation) for the explicit purpose of introducing more undefined behavior?
I don't see any way this change wouldn't lead to an enormous number of bugs, all for the (dubious) goal of having the range of signed integers be symmetrical.
-fwrapv is an UB mitigation that often just hides bugs and consequently creates more UB down the road, in places where it is less easy to diagnose. For example identifying an integer overflow is quite easy but placing bounds checks on all allocations in an ABI-preserving way is much more difficult.
No. What? No. A trap is by definition undefined behavior.
"Certain object representations need not represent a value of the object type. If the stored value of an object has such a representation and is read by an lvalue expression that does not have character type, the behavior is undefined. If such a representation is produced by a side effect that modifies all or any part of the object by an lvalue expression that does not have character type, the behavior is undefined."
-ftrapv is an implementation-provided flag that defines the behavior of some operations that are otherwise not defined by the language standard. The implementation could define the behavior (under the same or different flags) to this newly introduced trap representation as well (no need to actually trap).
Having said that this should be a new type, adding more UB to existing types is a bad idea.
Readit News
> @jrose @fclc People who want NaN in their integers are welcome to do that in the privacy of their own bedrooms with new types they invent for this purpose.
That reminds me of OCaml's 63 bit integers. In the privacy of OCaml's own bedroom, they use one bit of their integers as a flag to distinguish pointers from integers.
See https://blog.janestreet.com/what-is-gained-and-lost-with-63-...
The results are quite.. interesting. However, it does go to show that having slightly weird private integer types is actually totally workable, so the quote from the discussion might be meant as trolling, but it's less crazy than it sounds like.
And yet I bet they use IEEE754 floats without second thought.
(as soon as bit 63 is set, the number would be in 'overflow state', but at the same time the lower 63 bits would preserve the wrap-around value in case that's useful).
On the other hand, compilers are already free to check the CPU carry flag bit and trap on it. Those flag bits are just not exposed to programming languages (which IMHO is a serious 'dissonance' between assembly and high level languages), also CPU flags can't be carried around with their associated result value and go 'viral' like NaNs (but whether that's even useful instead of trapping immediately is another question).
Deleted Comment
The normal way to do it is to use the "ETSI Basic Operator" library, which provides saturating arithmetic operations, among other things [1]. If the hardware supports saturated arithmetic the compiler maps the operations to the relevant instructions, otherwise it provides a software implementation.
Example:
Modulo arithmetic: int16_t a,b,c; a = b + c;
Saturating 16-bit arithmetic using ETSI: int16_t a,b,c; a = add(b, c);
[1] https://www.itu.int/rec/T-REC-G.191/en
Of course today in Rust we're not allowed to bless such types for ourselves, this is reserved to the standard library (core::num::NonZeroU32 is a 4-byte unsigned integer which can't be zero for example) because it's relying on a nasty permanently unstable compiler-only feature to say "Hey, not all possible representations the right size for this type are valid".
Rust has one very obvious set of types with such a niche, the references &T, but it also has not only the particular custom integers NonZero{U8,U16,U32,U64,I8,I16,I32,I64} and the Non-null pointers, but OS specific types (for Unix file descriptors, and Windows handles respectively) and of course char, it is 4 bytes but Unicode is explicitly limited to far fewer values, so that's a niche too.
But today making more is not allowed in stable Rust, so while my nook crate is an interesting curiosity (and some day I'll implement integer math traits etc.) it cannot be stabilised and I was persuaded that the path forward is a whole lot of work to deliver the Pattern Types feature which can do this stably.
Arbitrary niches aren't intended to be permanently unstable (at least not in the sense of explicit implementation details like lang items and intrinsics), people have expressed a desire to stabilize them for years (as discussed during the original nonzero int types RFC). If something has changed in that regard, I'd appreciate a link to a discussion.
#[rustc_nonnull_optimization_guaranteed] #[rustc_layout_scalar_valid_range_start] and #[rustc_layout_scalar_valid_range_end]
As their names hint, these are permanently unstable compiler-only type hints.
I asked that some means be found to stabilize some equivalent hint (presumably with a name that doesn't look like a compiler-only feature) in one of the long series of tickets spawned after 3334 and I was persuaded that this is a terrible idea and won't happen.
Instead the proposed path forward is (at least was then) Pattern Types. Build at least an MVP of the full blown Pattern Types feature instead of a custom attribute.
In a new, green field programming language design: maybe.
You will have FFI interop issues: what if some foreign gives you such a value, which is correct in its domain. It will have to be specially handled and widened/promoted to a type that can hold it. It looks like friction, though.
In an existing language, like C? Making currently correct programs blow up with a trap exception is a nonstarter.
I think there is no hardware support for treating 0x80... as a trap, so if you actually want it to trap, rather than leaving it at undefined behavior, your compiler has to emit check code, at least in code paths where it is not obvious whether the value is or isn't a trap.
Also, it would be strangely inconsistent for the 0x80... to predictably trap wherever it occurs, but arithmetic overflow to have undefined consequences. The two want to be tied together.
Functions in C code bases checking for null is the rare exception rather than the rule.
When they do check, that doesn't turn null into a trap representation.
I realize that wasn't your point, but this is an enormous pet peeve of mine...
That's more or less what you have to do in Java if your FFI is passing you an unsigned type. You need to use a 64-bit signed long in Java to hold a uint32_t, and a BigInteger to hold a uint64_t.
I mean, you can certainly store that unsigned 32-bit int from the FFI call in a 32-bit signed Java int, but then you have to use special methods on the Integer class to do unsigned comparisons and arithmetic.
Agreed, this is a lot of friction, but there's long-standing precedent, at least.
Besides, -fwrapv was basically the default in most C compilers (and probably C++ as well) before the last 10 years or so. Many large systems thus use it today as well (e.g. the Linux kernel, I believe).
It seems like people were complaining as early as 2007. Richard Biener (who's linked in the thread link below) said that GCC 2.95.3, which came in 2001, optimized away signed overflow checks as well.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=30475
---
> signed type overflow is undefined by the C standard, use unsigned int for the
> addition or use -fwrapv
You have GOT to be kidding?
All kinds of security issues are caused by integer wraps, and you are just telling me that with gcc 4.1 and up I cannot test for them for signed data types any more?!
You are missing the point here. There HAS to be a way to get around this. Existing software uses signed int and I can't just change it to unsigned int, but I still must be able to check for a wrap! There does not appear to be a work around I could do in the source code either! Do you expect me to cast it to unsigned, shift right by one, and then add or what?!
PLEASE REVERT THIS CHANGE. This will create MAJOR SECURITY ISSUES in ALL MANNER OF CODE. I don't care if your language lawyers tell you gcc is right. THIS WILL CAUSE PEOPLE TO GET HACKED.
I found this because one check to prevent people from getting hacked failed.
THIS IS NOT A JOKE. FIX THIS! NOW!
Other compilers are different though - MSVC only seems to have started optimizing around this in ~2016 or so, icc seems to have done something around this in 2020, and I'm sure many embedded compilers ignore it even today. I couldn't find anything definitive on clang, but given its history I assume it probably always enforced this mistake.
I'd also note that several major C programs only work when signed integer overflow is defined: I mentioned Linux previously, but Python is another one. And Chrome and Firefox are considering enabling -fwrapv and equivalents.
[1] https://stackoverflow.com/a/56507840/1485877
I don't see any way this change wouldn't lead to an enormous number of bugs, all for the (dubious) goal of having the range of signed integers be symmetrical.
"Certain object representations need not represent a value of the object type. If the stored value of an object has such a representation and is read by an lvalue expression that does not have character type, the behavior is undefined. If such a representation is produced by a side effect that modifies all or any part of the object by an lvalue expression that does not have character type, the behavior is undefined."
Having said that this should be a new type, adding more UB to existing types is a bad idea.
I don't hate the idea, but doing it for existing int types might be a non-starter due to backward compatibility.