Sitelet https://github.com/RustCrypto/formats/pull/2451
Skip to content

der: order FlagSet by its BIT STRING encoding - #2451

Open
yuxi-liu-wired wants to merge 1 commit into
RustCrypto:masterfrom
yuxi-liu-wired:fix/der-flagset-value-ord
Open

yuxi-liu-wired wants to merge 1 commit into
RustCrypto:masterfrom
yuxi-liu-wired:fix/der-flagset-value-ord

Conversation

@yuxi-liu-wired

Copy link
Copy Markdown
Contributor

ValueOrd for FlagSet<T> compares bits() as an integer. The DER encoding, a NamedBitList BIT STRING, puts flag 0 in the most significant bit of the first octet and drops trailing zero bits, so the two orders disagree. For a 9-flag set:

{A}  bits 0x001  ->  03 02 07 80
{B}  bits 0x002  ->  03 02 06 40   <- first in DER

Over all 512×512 pairs of a 9-flag set, 84,822 pairs order differently.

X.690 11.6 orders SET OF components by their encodings. A SetOfVec of a type that contains a FlagSet can therefore encode out of DER order. An example is x509-cert's DistributionPoint, which derives ValueOrd and has reasons: Option<ReasonFlags>.

The fix compares the encoded values, using a 17-byte stack buffer: one unused-bits octet plus up to 16 data octets for u128 flags.

Test: bit_string::tests::flagset_valueord_is_encoding_order checks the {A}/{B} example and every pair of a 9-flag set against the order of their encodings. It fails on master.

This PR is independent of the other der ordering PRs (tag octet order, UtcTime, Option::der_cmp, ContextSpecific recursion) and merges cleanly with each.

This PR was produced by AI agents (Claude) while auditing every DerOrd/ValueOrd impl in der after differential fuzzing of der/x509-cert found the tag-order issue.

`ValueOrd for FlagSet<T>` compared `bits()` as an integer. The DER
encoding (a NamedBitList BIT STRING) puts flag 0 in the most significant
bit of the first octet and drops trailing zero bits, so the integer order
and the encoding order disagree, e.g. for a 9-flag set:

    {A}  bits 0x001  ->  03 02 07 80
    {B}  bits 0x002  ->  03 02 06 40   (sorts first in DER)

X.690 11.6 orders `SET OF` components by their encodings, so `SetOfVec`
of a type containing a `FlagSet` (e.g. x509-cert's `DistributionPoint`
with `reasons: Option<ReasonFlags>`) can encode out of DER order.

Compare the encoded values instead (on a stack buffer).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant