x86-retpoline flag (target modifier) to enable retpoline-related target features

[azhogin]

-Zx86-retpoline flag is target modifier (tracked to be equal in linked crates). And so, this PR is dependent from #133138.
Enables target features:
+retpoline-external-thunk, +retpoline-indirect-branches, +retpoline-indirect-calls.

Also this PR forbids to specify those target features manually (warning).

[rustbot]

r? @compiler-errors

rustbot has assigned @compiler-errors.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo, @GuillaumeGomez

[compiler-errors]

r? compiler

[bors]

☔ The latest upstream changes (presumably #136085) made this pull request unmergeable. Please resolve the merge conflicts.

[chenyukang]

r? compiler

[fee1-dead]

r? compiler

[lcnr]

r? @davidtwco maybe 😅

[Darksonn]

I don't think retpoline actually changes the ABI, so it might not need to be a target modifier. On the other hand, as an exploit mitigation, we do still want to avoid cases where you don't enable the mitigation in every compilation unit.

cc @andyhhp do I understand things correctly?

[azhogin]

I don't think retpoline actually changes the ABI, so it might not need to be a target modifier. On the other hand, as an exploit mitigation, we do still want to avoid cases where you don't enable the mitigation in every compilation unit.

I read your post in #116852:

I've investigated this issue in more details. It turns out that the retpoline mitigations do not affect ABI and can be mixed.

Mixing them gives up their advantage, though. Linking together two compilation units that disagree on whether to apply the mitigation may make both vulnerable to what they attempt to mitigate.

That's why I thought it should be a target modifier.

[andyhhp]

I don't think retpoline actually changes the ABI, so it might not need to be a target modifier. On the other hand, as an exploit mitigation, we do still want to avoid cases where you don't enable the mitigation in every compilation unit.

cc @andyhhp do I understand things correctly?

It's important to distinguish the general indirect-thunk transform from retpoline which is one specific transform (and is not effective on any Intel CPUs since Skylake).

retpoline is the specifically transform of an indirect call/jmp instruction into a call; int3; rewrite-ra; ret ROP gadget with the same net effect, but causes prediction to come from the RSB rather than the BTB.

The need to rewrite the return address on the stack necessitates the use of a GPR, and this is observable call/jmp-ee but any non-parameter register will do, so it doesn't impact the main function call ABI.

Even within retpoline itself, there are variations; inline retpolines are an option, but tend to be heavy on the code bloat side, which is why out-of-line retpolines in common sections are often preferred.

The alternatives to repoline are lfence;jmp (used on some Atom processors where retpoline isn't safe. Previously recommended by AMD but later rescinded), and re-inlining the original indirect branch (when hardware is believed safe, which is a contentious point because it can be made safe on AMD, but only "pretty safe" Intel.)

As to separate complication units, you can mix and match, but if any one piece of your whole process isn't taking safety precautions, then you've lost.