Day 14 – The Magic Of Q (Part 2)

A few days after having done The Magic Of Q presentation, Lizzybel was walking through the corridors of North Pole Grand Central and ran into some of elves that had attended that presentation. When will you do the final part of your presentation? one of them asked. We could do it now if you want, Lizzybel answered, while walking to the presentation room and opening the door.

The room was empty. Most of the elves entered, one of them asked: Will there be a video? Lizzybel thought for a moment, and said: No, but there will be a blog post about it! That elf hesitated a bit, then said: Ok, I will read it when it gets online, and went the away. The others sat down and Lizzybel continued:

There are two other adverbs that haven’t been covered yet, and there’s a new one if you’re feeling brave. Let’s start with the two already existing ones:

short   long      what does it do
=====   ====      ===============
:x      :exec     Execute as command and return results
:to     :heredoc  Parse text until terminator

Shelling out

The :x (or :exec) adverb indicates that the resulting string should be executed as an external program using shell. For example:

say q:x/echo Hello World/; # Hello World␤

And since you can skip the : if there’s only one adverb, you could also have written this as the maybe more familiar:

say qx/echo Hello World/; # Hello World␤

Of course, you can also have variables interpolated by using qq:

my $who = 'World';
say qqx/echo Hello $who/; # Hello World␤

But one should note that this is a huge security issue if you are not 100% sure about the value of the variable(s) that are being interpolated. For example:

my $who = 'World; shutdown -h now';
say qqx/echo Hello $who/; # Hello World␤

would produce the same output as above, except it could possibly also shutdown your computer immediately (if you were running this with sufficient rights). Now imagine it’d be doing something more permanently destructive! Ouch!

So generally, you should probably be using run (which does not use a shell, so has fewer risks) or go all the way with full control with a Proc object, or possibly even better, with a Proc::Async object.

Until the end

The :to (or :heredoc) adverb does something very special!

Different from all other adverbs, it interprets the text between // as the end marker, and takes all text until that marker is found at the start of a line. So it basically stops normal parsing of your code until that marker is found. And this is usually referred to as a “heredoc“.

Of course, if so needed you can also interpolate variables (by using qq rather than q), but these variables would be interpolated inside the heredoc, not in the marker. For instance:

my $who = 'World';
say qq:to/GREETING/;
Hello $who
GREETING
# Hello World␤

It is customary, but not needed in any way, to use a term for the marker that sort of describes what the text is about.

As you may have noticed, the resulting string of a heredoc always ends with a newline (“␤“). There is no adverb to indicate you don’t want that. But you can call the .chomp method on it like so:

my $who = 'World';
say qq:to/GREETING/.chomp;
Hello $who
GREETING
# Hello World

You can indent the end marker for better readability, for instance if you’re using it inside an if structure. That won’t affect the resulting string:

my $who = 'World';
if 42 {
    say qq:to/GREETING/;
    Hello $who
    GREETING
}
# Hello World␤

The text inside will have the same amount of whitespace removed from the beginning of each line, as there is on the start of the line with the end marker.

What many people don’t know, is that you can have multiple heredocs starting on the same line. Any subsequent heredoc will start immediately after the previous one. You can for instance use this in a ternary like so:

my $who = 'Bob';
say $mood eq 'good' ?? qq:to/GOOD/ !! qq:to/BAD/;
Hi $who!
GOOD
Sorry $who, the VHS is still broken.
BAD

Depending on the $mood, this will either say “Hi Bob!␤” or “Sorry Bob, the VHS is still broken.␤“.

Formatting

Since the 2023.06 release of the Rakudo Compiler, the 6.e.PREVIEW language version contains the Format class. This RakuAST powered class takes a printf format specification, and creates an immutable object that provides a Callable that takes the expected number of arguments. For example:

printf "%04d - %s\n", 42, 'The answer'; # 0042 - The answer␤

You can now save the logic of the format in a Format object, and call that with arguments. Like so:

use v6.e.PREVIEW;
my $format = Format.new("%04d - %s\n");
print $format(42, 'Answer'); # 0042 - Answer␤

Now, why would this be important, you might ask? Well, it isn’t terribly important if you use a format only once. But in many situations, a specific format is called many times in a loop. For instance when processing a log file:

for lines {
    m/^ (\d+) ':' FooBarBaz (\w+) /;
    printf "%04d - %s\n", $0, $1;
}

Because of the way printf formats are implemented in Raku, this is very slow.

This is because each time printf is called with a format string, the whole format is interpreted again (and again) using a grammar. During this parsing of the format, the final result string is created from the given arguments. This is much slower than calling a block with arguments, as that can be optimized by the runtime. In trial runs speed improvements of 100x faster, have been observed.

The new Format class can do this once, at compile time even! And by storing it in a constant with a & sigil we can use that format as if it is a named subroutine!

use v6.e.PREVIEW;
my constant &logged = Format.new("%04d - %s\n");
for lines {
    m/^ (\d+) ':' FooBarBaz (\w+) /;
    print logged($0, $1);
}

So what does this have to do with quoting adverbs, you might ask? Well, when the 6.e language level is released, this will also introduce:

short   long      what does it do
=====   ====      ===============
:o      :format   Create Format object for the given string

If the given string is a constant string, then the above example can be written as (without needing to define a constant):

use v6.e.PREVIEW;
for lines {
    m/^ (\d+) ':' FooBarBaz (\w+) /;
    print qqo/%04d - %s\n/($0, $1);
}

And by this time the remaining elves were a bit flabbergasted.

Well, that’s about it. That’s it what I wanted to tell about the magic of Q! said Lizzybel. The elves had a lot of questions, but those questions did not make it to this blog post. Too bad.

Maybe the readers of the blog post will ask the same questions in the comments, thought Lizzybel after writing up all of these events.