Sitelet https://web.archive.org/web/20260104130157/https://github.com/github/codeql/pull/9185
Skip to content

Conversation

@redsun82
Copy link
Contributor

@redsun82 redsun82 commented May 17, 2022 •

This allows to avoid bypassing label type correcness in the extractor,
and allows to independently resolve TBD extractions, as with this
approach TBD nodes do have the correctly typed trap label. The TBD
status is now a predicate on the QL side.

The code taking care of making unknwon entities be printed as TBD(Class)
is in the hand written Element.qll file.

This requires:

  • a default visit using the correct type, which is achieved via macro
    metaprogramming in VisitorBase.h, following the way
    swift::ASTVisitor is programmed
  • a mapping from labels to corresponding binding trap entries. The
    functor is defined in TrapTagTraits.h and instantiated in generated
    TrapEntries.h
  • Binding trap entries for TBD unknown entities must not have any other
    field than the id (after all, we are supposed to not extract them
    yet). This is why all unextracted fields in schema.yml have been
    commented out, and will be uncommentend when visitors are added

closes https://github.com/github/codeql-c-team/issues/1024

@redsun82 redsun82 requested review from MathiasVP, jketema and sashabu May 17, 2022 07:48
@redsun82 redsun82 requested review from a team as code owners May 17, 2022 07:48
@github-actions github-actions bot added the Swift label May 17, 2022
@redsun82 redsun82 added the no-change-note-required This PR does not need a change note label May 17, 2022
@redsun82 redsun82 force-pushed the redsun82/swift-tbd-rework branch from c12533d to 78bed81 Compare May 17, 2022 08:30
@jketema
Copy link
Contributor

jketema commented May 17, 2022

There are quite a lot of code scanning complaints. What is the deal with those?

@MathiasVP
Copy link
Contributor

There are quite a lot of code scanning complaints. What is the deal with those?

These are because ql-for-ql doesn't know about our autogeneration scheme for Swift. I've written about it here. Since that's a private repo I'll copy/paste the relevant part:

Since #9100 we added getAPrimaryQlClass for Swift the ql/primary-ql-class-consistency query currently (rightfully) reports a lot of inconsistencies.

This is because we generate two classes for each ... thing ... in Swift. For instance, we have ApplyExpr and ApplyExprBase for a function call. The ApplyExprBase class is an autogenerated and private class that contains (among other things) the implementation of getAPrimaryQlClass, whereas ApplyExpr contains additional manually written predicates.

This means that ApplyExprBase has a getAPrimaryQlClass predicate that returns "ApplyExpr" (instead of the expected "ApplyExprBase").

It should be straightforward to fix the ql-for-ql query to recognize this. The issue I wrote has two solutions, and Erik has suggested a third solution (and they should all be one-line fixes in ql-for-ql).

@redsun82 redsun82 force-pushed the redsun82/swift-tbd-rework branch from 78bed81 to 0452e2d Compare May 17, 2022 09:26
@redsun82
Copy link
Contributor Author

@sashabu or @jketema , can you have a look at the C++ part? If you start looking, can you immediately comment, so the other does not need to? And @MathiasVP, you might want to have a look at elements/Element.qll, which is the only non-generated QL code required for this to work.

@MathiasVP
Copy link
Contributor

And @MathiasVP, you might want to have a look at elements/Element.qll, which is the only non-generated QL code required for this to work.

Thanks for the heads up. This LGTM 👍.

@sashabu
Copy link
Contributor

sashabu commented May 17, 2022

@sashabu or @jketema , can you have a look at the C++ part? If you start looking, can you immediately comment, so the other does not need to?

I'll step back since Jeroen's already started commenting.

@sashabu sashabu removed their request for review May 17, 2022 12:35
@redsun82 redsun82 force-pushed the redsun82/swift-tbd-rework branch from 0452e2d to 128ba22 Compare May 17, 2022 12:40
@jketema
Copy link
Contributor

jketema commented May 17, 2022

I'll step back since Jeroen's already started commenting.

Just to be clear: I started commenting on the QL part, which I am much more interested in.

@redsun82 redsun82 force-pushed the redsun82/swift-tbd-rework branch 4 times, most recently from 0d91158 to 83e1071 Compare May 20, 2022 07:20
This allows to avoid bypassing label type correcness in the extractor,
and allows to independently resolve TBD extractions, as with this
approach TBD nodes do have the correctly typed trap label. The TBD
status is now a predicate on the QL side.

This requires:
* a default visit using the correct type, which is achieved via macro
  metaprogramming in `VisitorBase.h`, following the way
  `swift::ASTVisitor` is programmed
* a mapping from labels to corresponding binding trap entries. The
  functor is defined in `TrapTagTraits.h` and instantiated in generated
  `TrapEntries.h`
* Binding trap entries for TBD unknown entities must not have any other
  field than the `id` (after all, we are supposed to not extract them
  yet). This is why all unextracted fields in `schema.yml` have been
  commented out, and will be uncommentend when visitors are added
@redsun82 redsun82 force-pushed the redsun82/swift-tbd-rework branch from 83e1071 to da00bf9 Compare May 20, 2022 07:52
Copy link
Contributor

@jketema jketema left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks ok, but I think there are some details I'm missing. See questions below.

trap.emit(UnknownAstNodesTrap{label, name});
}
using Trap = BindingTrapOf<E>;
static_assert(sizeof(Trap) == sizeof(label),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ToBindingTrapFunctor seems pretty empty to me, so doesn't this always hold? I'm probably just missing details here.

Copy link
Contributor Author

@redsun82 redsun82 May 20, 2022 •

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ToBindingTrapFunctor is implemented by TrapEntries.h in the generated headers. This particular assertion fires whenever a final class in schema.yml has a single property (i.e. non-optional, non-repeated). For example uncommenting line 995 in schema.yml (field introducer_int in class ConcreteVarDecl) and regenerating the dbscheme file will then result in

./swift/extractor/SwiftDispatcher.h:34:5: error: static_assert failed due to requirement 'sizeof(codeql::ConcreteVarDeclsTrap) == sizeof (label)' "Binding traps of unknown entities must only have the `id` field (the class should be empty in schema.yml)"
    static_assert(sizeof(Trap) == sizeof(label),
    ^             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

which should point in the correct direction.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't get this, because:

template<>
struct ToBindingTrapFunctor<TrapLabel<FileTag>> {
  using type = FilesTrap;
};

pretty much looks like an empty class to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, sorry, I did not get that that was your doubt.

It's the type nested alias that does the trick. From TrapTagTraits:

template <typename T>
using BindingTrapOf = typename detail::ToBindingTrapFunctor<TrapLabelOf<T>>::type;

so for example BindingTrapOf<swift::VarDecl> will resolbe to typename detail::ToBindingTrapFunctor<TrapLabel<ConcreteVarDeclTag>>::type which in the generated file is defined as ConcreteVarDeclsTrap.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. But then why do this at all and not just:

static_assert(sizeof(E) == sizeof(label),

Copy link
Contributor Author

@redsun82 redsun82 May 20, 2022 •

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure but that's because there's actual work needed to implement underlying_type it's not just an alias like we have here.

I think that alias is there to hide away an implementation detail, which does the actual work in terms of template specialization.

Would you find it clearer if we wrote something like

  template <typename Label>
  using BindingTrapOf = typename detail::ToBindingTrap<Label>::type;

and then in the code using it did:

  using Trap = BindingTrapOf<decltype(label)>;

or more explicitly

  using Label = TrapLabelOf<E>;
  using Trap = BindingTrapOf<Label>;
  Label label = assignNewLabel(e);
  ...

?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand less and less of what this is code is doing. Why is all this indirect mess needed at all? This is going to cause a lot of trouble for anyone who is going to maintain this in the future.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want something that's just 2 lines of code instead of 12.

Copy link
Contributor Author

@redsun82 redsun82 May 20, 2022 •

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that should probably just be TrapLabel<E>{label}.?

Hmm, no, labels cannot be emitted in place of trap entries. TrapLabelOf<E> which resolves to TrapLabel<TrapTagOf<E>> (for example, TrapLabel<ConcreteVarDeclTag>) denotes a typed trap label, so something that is a # label in the trap file. We need to emit an actual trap (like concrete_var_decls(#1e)) which in C++ is done printing a ConcreteVarDeclsTrap instance.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for the record: @redsun82 and I discussed this face-to-face. For now I'm happy with this. I think it would be good to have another detailed look at how this is set up. My main issue is that a lot of the template programming being done here is needed to patch up the macro programming that the Swift compiler forces upon us, which feel rather brittle and difficult to maintain, due to this being split over quite a number of generated and manually implemented files. I wonder if this could be simplified significantly by moving towards more macro programming or moving towards more template programming.

@redsun82 redsun82 merged commit da7e700 into main May 20, 2022
@redsun82 redsun82 deleted the redsun82/swift-tbd-rework branch May 20, 2022 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-change-note-required This PR does not need a change note Swift

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants