Python: failing to parse t-strings and f-strings with an alignment modifier and `=` as fill character

[grahamcracker1234]

CodeQL does not support t-strings, a new 3.14 feature, failing with:

Could not process some files due to syntax errors

A parse error occurred while processing <file>, and as a result this file could not be analyzed. Check the syntax of the file using the python -m py_compile command and correct any invalid syntax.

Where <file> contains a t-string.

[redsun82]

👋 @grahamcracker1234 CodeQL support for t-strings landed 9 days ago with version 2.24.0 (change note). Please make sure you install the latest version, and you should be good to go 🙌

[grahamcracker1234]

I see no reference to t-strings in the 2.24.0 change note you provided. I could be missing something though.

Never mind, I now see:

• The Python extractor now supports template strings as defined in PEP-750, through the classes TemplateString and JoinedTemplateString.

Not quite sure what JoinedTemplateString refers to, but does this support the new syntax (t"value: {foo}") as well as the class Template()?

Nevertheless, the running version was already 2.24.0

In test_dedent.py, the only special syntax that I would have thought CodeQL would struggle to parse was t-strings.

I don't think this issue should be closed.

[grahamcracker1234]

Potentially, if the t-strings parsing is truly working, it may have to do with the custom format specifications used in dedent.

[redsun82]

Aha, let me reopen this with a different title and have a look.

[redsun82]

aha, found the problem: we fail to parse both f-strings and t-strings that use = as a fill character and specify an alignment modifier (in your case it was {header:=^17}). We probably parse = as an alignment modifier instead of a filler character (= can be used to enable padding after the sign on numbers) and then choke on multiple align modifiers being present. I'll relay this to the team working on it, and we will hopefully release a fix soon.

Thanks for reporting this!

[redsun82]

For the record: the relevant part of the format specification mini language is

If a valid align value is specified, it can be preceded by a fill character that can be any character and defaults to a space if omitted

So =^ (or =<, =>) gives = the role of fill character, overriding its padding specification behaviour. Curiously, CodeQL currently works on {x:<^20} or {x:>^20}, so the problem is specific to =.

[grahamcracker1234]

Thanks for reopening!

Due to objects being able to use arbitrary format specifications with __format__, any string should be valid, regardless of the convention of the format specification mini-language, no?

I think, the only caveat is that it can be interpolated with a value (t"{value:^{count}}"), so I think the parser only need to ensure there are balanced curly brackets { }, while recognizing {{ and }} as escaped characters, whenever the runtime actually interprets them.