Day 12 – Making a simple bot in Raku

Making IRC bots is incredibly simple in Raku, thanks to IRC::Client. It allows you to create a very simple bot in about 20 lines of code. There’s a plugin system that allows easy re-use of code between multiple bots, and adding customized features can be as easy as dropping in an anonymous class.

So, let’s get to it!

Get your dependencies installed

Raku uses zef as the standard module installer, and if you’re reading this, I’m assuming you have it available to you. Install IRC::Client with zef, and you should be good to get started.

zef install IRC::Client

Setting up the bot

To set up the bot, we’ll need to have a nickname to use, a server to connect to and a list of channels to join. To make it easier to run this is a program from your shell, I’ll be using a MAIN sub as well.

use IRC::Client;

sub MAIN () {
  IRC::Client.new(
    nick => 'raku-advent',
    host => 'irc.darenet.org',
    channels => < #advent >,
  ).run;
}

Let’s save this in a file called bot.pl6, and run it.

perl6 bot.pl6

This will run, and if you’re in the channel you specified in channels, you should see the bot joining in a short moment. However, the program itself doesn’t seem to provide any output. It would be highly convenient, especially during development, to show what it’s doing. This is possible by enabling the debug mode. Adding this to the new method call, making it look as follows.

IRC::Client.new(
  nick => 'raku-advent',
  host => 'irc.darenet.org',
  channels => < #advent >,
  debug => True,
).run;

If you restart the application now, you will see there’s a lot of output all of a sudden, showcasing the IRC commands the bot is receiving and sending in response. Now all we need to do is add some functionality.

Making the bot work

As described earlier, functionality of the bot is added in using plugins. These can be any class that implements the right method names. For now, we’ll stick to irc-to-me, which is a convenience method which is triggered whenever the bot is spoken to in a private message, or directly addressed in a channel.

The simplest example to get started with here is to simply have it respond with the message you sent to the bot. Let’s do this by adding an anonymous class as a plugin to the new method call.

IRC::Client.new(
  nick => 'raku-advent',
  host => 'irc.darenet.org',
  channels => < #advent >,
  debug => True,
  plugins => [
    class {
      multi method irc-to-me ($e) {
        $e.text
      }
    }
  ],
).run;

When you restart the bot and talk to it on IRC, you will see it responding to you with the same message you sent it.

 <@tyil> raku-advent: hi
 <raku-advent> tyil, hi
 <@tyil:> raku-advent: how are you doing
 <raku-advent> tyil, how are you doing

Adding some real features

So, you’ve seen how easy it is to get started with a simple IRC bot in just over a dozen lines. Let’s add two features that you may want your bot to support.

For convenience sake, I will only cover the class implementing the features, not the entire IRC::Client.new block.

Uptime

First off, let’s make the bot able to show the time its been running for. For this, I’ll make it respond to people asking it for “uptime”. We can use the irc-to-me convenience method for this again. After all, we probably don’t want it to respond every time someone discusses uptime, only when the bot is asked directly about it.

In Raku, there’s a special variable called $*INIT-INSTANT, which contains an Instant of the moment the program started. We can use this to easily get the Duration that the program has been running for.

class {
  multi method irc-to-me ($ where *.text eq 'uptime') {
    my $response = "I've been alive for";
    my ($seconds, $minutes, $hours, $days, $weeks) =
      (now - $*INIT-INSTANT).polymod(60, 60, 24, 7);

    $response ~= " $weeks weeks" if $weeks;
    $response ~= " $days days" if $days;
    $response ~= " $hours hours" if $hours;
    $response ~= " $minutes minutes" if $minutes;
    $response ~= " $seconds seconds" if $seconds;

    $response ~ '.';
  }
}

Now, whenever you ask the bot for uptime, it will respond with a human friendly uptime notification.

 <@tyil> uptime
 <@tyil> raku-advent: uptime
 <raku-advent> tyil, I've been alive for 5 minutes 8 seconds.

User points

Most channels have a bot that keeps track of user points, or karma as it’s sometimes referred to. There’s a module already that does this for us, called IRC::Client::Plugin::UserPoints. We don’t have to do much apart from installing it and adding it to the list of plugins.

zef install IRC::Client::Plugin::UserPoints

Once this finishes, the module can be used in your code. You will need to import it with a use statement, which you can put directly under the use IRC::Client line.

use IRC::Client;
use IRC::Client::Plugin::UserPoints;

Now, in the list of plugins, add it as a new entry.

plugins => [
  IRC::Client::Plugin::UserPoints.new,
  class {
    ...
  },
],

This plugin makes the bot respond to !scores, !sum and whenever a nick is
given points using a ++ suffix, for instance, t​yil++.

 <@tyil> raku++
 <@tyil> raku++
 <@tyil> !scores
 <raku-advent> tyil, « raku » points: main: 2

Finding plugins

All plugins for IRC::Client that are shared on the community have the prefix IRC::Client::Plugin::, so you can search for that on modules.perl6.org to find plugins to use. Of course, you can easily add your own plugins to the ecosystem as well!

Winding down

As you can see, with some very simple code you can add some fun or important
tools to your IRC community using the Raku programming language. Try it out and
have some fun, and share your ideas with others!

Day 11 – Packaging with Libarchive

Distributing physical gifts involves wrapping them up into packages, but suppose you want to distribute digital gifts. How can you use Raku to help you wrap them up? Enter Libarchive!

Simple wrapping files into a package

Let’s wrap up just two files, myfile1 and myfile2 into a single package.zip file. (Libarchive just as easily creates tar files, cpio, rar, even iso9660 images for cds or dvds.)

use Libarchive::Simple;

given archive-write('package.zip') {
    .add: 'myfile1', 'myfile2';
    .close;
}

This very simple syntax looks a little weird for those unfamiliar… here is a more ‘traditional’ way of writing the same thing:

use Libarchive::Write;

my $handle = Libarchive::Write.new('package.zip');
$handle.add('myfile1', 'myfile2');
$handle.close;

What is the difference? Libarchive::Simple provides a few shorthand routines for accessing the various Libarchive functionalities. One of these is archive-write() which is identical to Libarchive::Write.new().

The second example takes the return from new() and stores it in the variable $handle. Then we call two methods on that variable to add the files, and close the file.

The given statement makes this even simpler by topicalizing that variable, that is, storing it in the topic variable $_. Since $_ can be used as the default object for method calls, we don’t need to explicitly refer to it when calling methods.

.add('myfile1') is equivalent to $_.add('myfile1')

But what happened to the parentheses? Another little shorthand when calling methods — rather than surrounding your arguments to a method with parentheses, you can just precede them with a colon:

.add: 'myfile1';

Nice! I love programming with Raku!

Package a bunch of files by smartmatching

A handy routine to help in your packaging is dir(). It will return a lazy list of IO::Path objects for a directory. By happy coincidence, Libarchive add can take IO::Path just as easily as a filename.

given archive-write('package.zip') {
    .add: 'mydir', dir('mydir');
    .close;
}

Note we’ve added the directory itself first, then used dir() to get a list of the files inside mydir, which also get added. If you don’t include the directory itself, it won’t be part of the package. That works fine most of the time, depending on your format and your unpackaging program, but it is good practice to include the directory to make sure it gets created the way you want it to.

dir has an extra feature — it can filter the directory by smartmatching the string with a :test argument. Lets include only jpeg files, allowing them to end in either .jpg or .jpeg:

given archive-write('package.zip') {
    .add: 'mydir', dir('mydir', test => /:i '.' jpe?g $/);
    .close;
}

Ecosystem modules like File::Find or Concurrent::File::Find can easily generate even more complicated lists of files for including by recursively adding an entire hierarchy to the package.

Create your files on the fly while packaging

You aren’t limited to adding existing files. You can use the write() method to generate a file for the package on the fly. You can specify content as a Str, a Blob, or even an IO::Handle or IO::Path to get the content from.

given archive-write('package.zip') {
    .write: 'myfile', q:to/EOF/;
        Myfile
        ------
        A special file for a special friend!
        EOF
    .close;
}

Here we’re using a special Raku quoting construct called the heredoc.

The q:to/EOF/ says to use the lines following up until the EOF marker and make them into the content of a file named ‘myfile’ included in the package file. As a friendly benefit, the amount of indentation of the terminator is automatically removed from each line to the quoted lines. How convenient!

Stream your package instead of making files

Making files with your packages is great and all, but I’ve got a web site I want to return my custom CD images from — why bother with a temporary file? Just stream the output on the fly!

For this example, we’re going to stream the package as an iso9660 file (the image used for CD writers) to STDOUT, but you can stream to other programs too.

given archive-write($*OUT, format => 'iso9660') {
    .add: 'myfile1', 'myfile2', 'mydir', dir('mydir');
    .close;
}

Usually the format can be inferred from the suffix on a specified filename, but since we are streaming there is no filename, so the format must be specified. $*OUT is a special filehandle that is automatically opened for you for writing to STDOUT.

Burn that image to a CD and mount it and you’ll see the specified files. So easy!

Libarchive has so many cool features it would take days to go over them all, but I hope this little intro has whet your appetite for packaging things up. Raku has fantastic syntax, features and expressivity that make it so easy to interface with libraries like this.

Have fun packaging your own things, be they physical or digital!

Day 10 – A Teaser

Santa has a special treat: a teaser if you will. A part of a chapter from the upcoming book “Migrating Perl to Raku”, to be published January 2020.


Optimization Considerations

If you are an experienced Perl programmer, you have (perhaps inadvertently) learned a few tricks to make execution of your Perl program faster. Some of these idioms work counter-productively in Raku. This chapter deals with some of them and provides the alternative idioms in Raku.

Blessed Hashes Vs. Objects

Objects in Perl generally consist of blessed hashes. As we’ve seen before, this implies that accessors need to be made for them. Which means an extra level of indirection and overhead. So many Perl programmers “know” that the object is basically a hash, and access keys in the blessed hash directly.

Consequently, many Perl programmers decide to forget about creating objects altogether and just use hashes. Which is functionally ok if you’re just using the hash as a store for keys and associated values. But in Raku it is better to actually use objects for that from a performance point of view. Take this example where a hash is created with two keys / values:

    # Raku

    for ^1000000 {    # do this a million times

        my %h = a => 42, b => 666;

    }

    say now - INIT now;  # 1.4727555

Now, if we use an object with two attributes, this is more 4x as fast:

    # Raku

    class A {

        has $.a;

        has $.b;

    }


    for ^1000000 {

        my $obj = A.new(a => 42, b => 666);

    }

    say now - INIT now;  # 0.3511395

But, you might argue, accessing the keys in the hash will be faster than calling an accessor to fetch the values?

Nope. Using accessors in Raku is faster as well. Compare this code:

    # Raku

    my %h = a => 42, b => 666;


    for ^10000000 {    # do this ten million times

        my $a = %h<a>;

    }

    say now - INIT now;  # 0.4713363


To:

    # Raku

    class A {

        has $.a;

        has $.b;

    }

    my $obj = A.new(a => 42, b => 666);


    for ^10000000 {

        my $a = $obj.a;
    
}

    say now - INIT now;  # 0.36870995

Note that using accessors is also faster, albeit not much, but still significantly so.

So why is the accessor method faster in Raku? Well, really because Raku is able to optimise to attributes in an object to a list, easily indexed by a number internally. Whereas for a hash lookup, the string must be hashed before it can be looked up. And that takes a lot more work than just a lookup by index.

Of course, as with all benchmarks, this is just a snapshot in time. Optimisation work continues to be performed on Raku, which may change the outcome of these tests in the future. So, always test yourself if you want to be sure that some optimisation is a valid approach, but only if you’re really interested in squeezing performance out of your Raku code. Remember, premature optimisation is the root of all evil!


Santa hopes you liked it.

Day 9: a chain (or Russian doll) of containers

A lot of electronic in a couple of containers

If you’re in the business, you’ve probably by now heard about containers. They can be described as executables on steroids, or also, as its namesake, a great way of shipping applications anywhere, or have then stored and ready to use whenever you need them. These kinda-executables are called images, and you can find them in a number of places called registries, starting with Docker Hub, joined lately by the GitHub Container Registry and other places like Quay.io or RedHat container catalog. These last are up and coming, you and have to add them to your default configuration. Most containers are registered in Docker Hub anyway.

Also, since they are kinda-executables, they are architecture and operating system specific. Docker hub marks their architecture and operating system, with Linux being the most common ones. You can, however, run Linux images everywhere, as long as the Docker daemon is running in a Linux virtual machine; that is also the default configuration in Macs.

Of course, there’s a slew of containers you can use with Raku, even if we don’t really have one we can call official. I’m going to go with my own, since, well, I’m more familiar with them. But there’re these nightly images by tyil, for instance, or the kinda-official Rakudo Star images, which have not been updated since, well, Rakudo Star itself was updated last March.

The Alps as seen from a plane

Let’s start with the basic image, the tiny Russian Doll with Nicky the tsar. Since it’s going to be inside, we need to make it real tiny. Here it is, jjmerelo/alpine-perl6:

FROM alpine:latest
LABEL version="2.2" maintainer="JJMerelo@GMail.com" perl6version="2019.11"

# Environment
ENV PATH="/root/.rakudobrew/versions/moar-2019.11/install/bin:/root/.rakudobrew/versions/moar-2019.11/install/share/perl6/site/bin:/root/.rakudobrew/bin:${PATH}" \
    PKGS="curl git perl" \
    PKGS_TMP="curl-dev linux-headers make gcc musl-dev wget" \
    ENV="/root/.profile" \
    VER="2019.11"

# Basic setup, programs and init
RUN mkdir /home/raku \
    apk update && apk upgrade \
    && apk add --no-cache $PKGS $PKGS_TMP \
    && git clone https://github.com/tadzik/rakudobrew ~/.rakudobrew \
    && echo 'eval "$(~/.rakudobrew/bin/rakudobrew init Sh)"' >> ~/.profile \
    && eval "$(~/.rakudobrew/bin/rakudobrew init Sh)"\
    && rakudobrew build moar $VER \
    && rakudobrew global moar-$VER \
    && rakudobrew build-zef\
    && zef install Linenoise App::Prove6\
    && apk del $PKGS_TMP \
    && RAKUDO_VERSION=`sed "s/\n//" /root/.rakudobrew/CURRENT` \
       rm -rf /root/.rakudobrew/${RAKUDO_VERSION}/src /root/zef \
       /root/.rakudobrew/git_reference

# Runtime
WORKDIR /home/raku
ENTRYPOINT ["raku"]

This image was created just last week, after the release of Raku 2019.11, the first one to actually be called Raku and the one that calls its executable raku too.

First thing you see is that FROM which declares the teeny container that’s inside this one. We’re using Alpine Linux, a distribution little known outside the containerer community, that uses a couple of tricks to avoid bloating the number of files, and thus the size, of the container. This image will add up to less than 300 MBs, while an equivalent image with Debian or Ubuntu will be twice as much. That means that downloading it will take half as much, which is what we’re looking for.

Because there’s this thing, too: real-life containers, when empty, can be Russian-dolled and put inside one another so that they don’t occupy so much space. Something similar happens to containers. They are built putting layers on top of each other, the inner layer usually an operating system. Let’s check out the rest.

The next LABELs are simply tags or metadata that can be extracted from the image by inspection. Not really that important.

But the ENV block kinda is, over all the first one, which defines the PATH that is going to be used across the Russian doll buildup. The rest of the variables are mainly used while building the image. They will also help to make it somewhat generic, so that we can just change the value of a variable and get a new version; we put that into VER.

So far, no building has taken place, but in this humongous RUN statement is where we download rakudobrew, put it to work building the version contained in VER, set that version as the default one, install zef and a couple of modules we are going to need, and then delete what we will no longer be needing in the rest of the outer dolls to keep the whole thing small.

Finally, after setting up a working directory, we define an entry point, which is the real executable-within-the-executable. The container can be used in place of this command, so that anything that can be done with raku, can be done with this this executable. For instance, let’s run this program:

my @arr;
my ($a, $b) = (1,1);
for ^5 {
    ($a,$b) = ($b, $a+$b);
    @arr.push: ($a.item, $b.item);
    say @arr
};
say @arr;

We will give our containerized Raku an alias:

alias raku-do='docker run --rm -t -v `pwd`:/home/raku  jjmerelo/alpine-perl6'

We can run the program above with:

raku-do itemizer-with-container.p6

But you can take it a step further. Create this shell script and put it in the path:

#!/bin/bash

docker run --rm -t -v `pwd`:/home/raku  jjmerelo/alpine-perl6 $@

You can then use this in the shebang line: !/usr/bin/env raku-do.sh. This will create a throwaway image, that will be ephemerally created to run the script, and then thrown away (that’s the --rm in the line). The current directory (pwd) will be aliased to /home/raku, remember, our working directory, which means that the raku inside the container will see it right there. You see? With this you can have raku run wherever docker is installed. Pretty much everywhere, nowadays.

But let’s build up on this. Containers are extensively used for testing, since instead of building and installing, you can put everything in a single container, and download and use it for testing straight away. That’s is actually what made a containerer out of me, the long 20 minutes it took to brew rakudo for a few seconds of testing for every module. After that base container, I created this one, jjmerelo/test-perl6. Here it is:

FROM jjmerelo/alpine-perl6:latest
LABEL version="4.0.2" maintainer="JJ Merelo <jjmerelo@GMail.com>"

# Set up dirs
RUN mkdir /test
VOLUME /test
WORKDIR /test


# Will run this
ENTRYPOINT perl6 -v && zef install --deps-only . && zef test .

This is actually simplicity itself: the only thing that changes is the entrypoint and the working dir. Instead of running directly the raku compiler, it does a couple of things: install dependencies needed to run the tests, and then issue zef test . to run the tests.

That really speeds things up when testing. Put it in your .travis.yml file this way:

language: minimal

services:
  - docker

install: docker pull jjmerelo/test-perl6

script: docker run -t -v  $TRAVIS_BUILD_DIR:/test jjmerelo/test-perl6

And you’re good to go. Takes all of a minute and a half, as opposed to more than 20 minutes if you use the official Travis image, which is based in rakudobrew.

The Russian doll does not stop there: jjmerelo/perl6-test-openssl includes additional Alpine packages which are needed to install OpenSSL. And, based on that one, jjmerelo/perl6-doccer, which packs everything that’s needed to test the Raku documentation.

You should really try this yourself. If you’ve got even just a few additional modules to download when testing your module, just build up from the test-perl6 image and get your own! You’ll save time, and also save computing time, thus saving energy.

Actual Russian, or maybe Latvian, dolls, bought in Riga during PerlCon

 

Day 8 – Parsing Firefox’ user.js with Raku (Part 2)

Yesterday, we made a short Grammar that could parse a single line of the user.js that Firefox uses. Today, we’ll be adding a number of test cases to make sure everything we want to match will match properly. Additionally, the Grammar can be expanded to match multiple lines, so we can let the Grammar parse an entire user.js file in a single call.

Adding more tests

To get started with matching other argument types, we should extend the list of
test cases that are defined in MAIN. Let’s add a couple to match true,
false, null and integer values.

my @inputs = (
  'user_pref("browser.startup.homepage", "https://searx.tyil.nl");',
  'user_pref("extensions.screenshots.disabled", true);',
  'user_pref("browser.search.suggest.enabled", false);',
  'user_pref("i.have.no.nulls", null);',
  'user_pref("browser.startup.page", 3);',
);

I would suggest to update the for loop as well, to indicate which input it is currently trying to match. Things will fail to match, and it will be easier to see which output belongs to which input if we just print it out.

for @inputs {
  say "\nTesting $_\n";
  say UserJS.parse($_);
}

If you run the script now, you’ll see that only the first test case is actually
working, while the others all fail on the argument. Let’s fix each of these
tests, starting at the top.

Matching other types

To make it easy to match all sorts of types, let’s introduce a proto regex. This will help keep everything into small, managable blocks. Let’s also rename the argument rule to constant, which will more aptly describe the things we’re going to match with them. Before adding new functionalities, let’s see what the rewritten structure would be.

rule argument-list {
  '('
  <( <constant>+ % ',' )>
  ')'
}

proto rule constant { * }

rule constant:sym<string> {
  '"'
  <( <-["]>+? )>
  '"'
}

As you can see, I’ve given the constant the sym adverb named string. This makes it easy to see for us that it’s about constant strings. Now we can also easily add additional constant types, such as booleans.

rule constant:sym<boolean> {
  | 'true'
  | 'false'
}

This will match both the bare words true and false. Adding just this and running the script once more will show you that the next two test cases are now working. Adding the null type is just as easy.

rule constant:sym<null> {
  'null'
}

Now all we need to pass the 5th test case is parsing numbers. In JavaScript, everything is a float, so let’s stick to that for our Grammar as well. Let’s accept one or more numbers, optionally followed by both a dot and another set of numbers. Of course, we should also allow a - or a + in front of them.

rule constant:sym<float> {
  <[+-]>? \d+ [ "." \d+ ]?
}

Working out some edge cases

It looks like we can match all the important types now. However, there’s some edge cases that are allowed that aren’t going to work yet. A big one is of course a string containing a "`. If we add a test case for this, we can see it failing when we run the script.

my @inputs = (
  ...
  'user_pref("double.quotes", "\"my value\"");',
);

To fix this, we need to go back to constant:sym, and alter the rule to take escaped double quotes into account. Instead of looking for any character that is not a ", we can alter it to look for any character that is not directly following a \, because that would make it escaped.

rule constant:sym<string> {
  '"'
  <( .*? <!after '\\'> )>
  '"'
}

Parsing multiple lines

Now that it seems we are able to handle all the different user_pref values that Firefox may throw at us, it’s time to update the script to parse a whole file. Let’s move the inputs we have right now to user.js, and update the MAIN subroutine to read that file.

sub MAIN () {
  say UserJS.parse('user.js'.IO.slurp);
}

Running the script now will print a Nil value on STDOUT, but if you still have Grammar::Tracer enabled, you’ll also notice that it has no complaints. It’s all green!

The problem here is that the TOP rule is currently instructed to only parse a single user_pref line, but our file contains multiple of such lines. The parse method of the UserJS Grammar expects to match the entire string it is told to parse, and that’s causing the Grammar to ultimately fail.

So, we’ll need to alter the TOP rule to allow matching of multiple lines. The easieset way is to wrap the current contents into a group, and add a quantifier to that.

rule TOP {
  [
    <function-name>
    <argument-list>
    ';'
  ]*
}

Now it matches all lines, and correctly extracts the values of the user_pref statements again.

Any comments?

There is another edge case to cover: comments. These are allowed in the user.js file, and when looking up such files online for preset configurations, they’re often making extensive use of them. In JavaScript, comments start with // and continue until the end of the line.

We’ll be using a token instead of a rule for this, since that doesn’t handle whitespace for us. The newline is a whitespace character, and is significant for a comment to denote its end. Additionally, the TOP rule needs some small alteration again to accept comment lines as well. To keep things readable, we should move over the current contents of the matching group to it’s own rule.

rule TOP {
  [
  | <user-pref>
  | <comment>
  ]*
}

token comment {
  '//'
  <( <-[\n]>* )>
  "\n"
}

rule user-pref {
  <function-name>
  <argument-list>
  ';'
}

Now you should be able to parse comments as well. It shouldn’t matter wether they are on their own line, or after a user_pref statement.

## Make it into an object

What good is parsing data if you can’t easily play with it afterwards. So, let’s make use of Grammar Actions to transform the Match objects into a list of UserPref objects. First, let’s declare what the class should look like.

class UserPref {
  has $.key;
  has $.value;

  submethod Str () {
    my $value;

    given ($!value) {
      when Str  { $value = "\"$!value\"" }
      when Num  { $value = $!value }
      when Bool { $value = $!value ?? 'true' !! 'false' }
      when Any  { $value = 'null' }
    }

    sprintf('user_pref("%s", %s);', $!key, $value);
  }
}

A simple class containing a key and a value, and some logic to turn it back into a string usable in the user.js file. Next, creating an Action class to make these objects. An Action class is like any regular class. All you need to pay attention to is to name the methods the same as the rules used in the Grammar.

class UserJSActions {
  method TOP ($/) {
    make $/.map({
      UserPref.new(
        key => $_[0].made,
        value => $_[1].made,
      )
    })
  }

  method constant:sym<boolean> ($/) {
    make (~$/ eq 'true' ?? True !! False)
  }

  method constant:sym<float> ($/) {
    make +$/
  }

  method constant:sym<null> ($/) {
    make Any
  }

  method constant:sym<string> ($/) {
    make ~$/
  }
}

The value methods convert the values as seen in the user.js to Raku types. The TOP method maps over all the user_pref statements that have been parsed, and turns each of them into a UserPref object. Now all that is left is to add the UserJSActions class as the Action class for the parse call in MAIN, and use its made value.

sub MAIN () {
  my $match = UserJS.parse('user.js'.IO.slurp, :actions(UserJSActions));

  say $match.made;
}

Now we can also do things with it. For instance, we can sort all the user_pref statements alphabatically.

sub MAIN () {
  my $match = UserJS.parse('user.js'.IO.slurp, :actions(UserJSActions));
  my @prefs = $match.made;

  for @prefs.sort(*.key) {
    .Str.say
  }
}

Sorting alphabetically may be a bit boring, but you have all sorts of possibilities now, such as filtering out certain options or comments, or merging in multiple files from multiple sources.

I hope this has been an interesting journey into parsing a whole other programming language using Raku’s extremely powerful Grammars!

The complete code

parser.pl6

class UserPref {
  has $.key;
  has $.value;

  submethod Str () {
    my $value;

    given ($!value) {
      when Str  { $value = "\"$!value\"" }
      when Num  { $value = $!value }
      when Bool { $value = $!value ?? 'true' !! 'false' }
      when Any  { $value = 'null' }
    }

    sprintf('user_pref("%s", %s);', $!key, $value);
  }
}

class UserJSActions {
  method TOP ($/) {
    make $/.map({
      UserPref.new(
        key => $_[0].made,
        value => $_[1].made,
      )
    })
  }

  method constant:sym<boolean> ($/) {
    make (~$/ eq 'true' ?? True !! False)
  }

  method constant:sym<float> ($/) {
    make +$/
  }

  method constant:sym<null> ($/) {
    make Any
  }

  method constant:sym<string> ($/) {
    make ~$/
  }
}

grammar UserJS {
  rule TOP {
    [
    | <user-prefix>
    | <comment>
    ]*
  }

  token comment {
    '//' <( <-[\n]>* )> "\n"
  }

  rule user-pref {
    <function-name>
    <argument-list>
    ';'
  }

  rule function-name {
    'user_pref'
  }

  rule argument-list {
    '('
    <( <constant>+ % ',' )>
    ')'
  }

  proto rule constant { * }

  rule constant:sym<string> {
    '"'
    <( .*? <!after '\\'> )>
    '"'
  }

  rule constant:sym<boolean> {
    | 'true'
    | 'false'
  }

  rule constant:sym<null> {
    'null'
  }

  rule constant:sym<float> {
    <[+-]>? \d+ [ "." \d+ ]?
  }
}

sub MAIN () {
  my $match = UserJS.parse('user.js'.IO.slurp, :actions(UserJSActions));
  my @prefs = $match.made;

  for @prefs.sort(*.key) {
    .Str.say
  }
}

user.js

// Comments are welcome!

user_pref("browser.startup.homepage", "https://searx.tyil.nl");
user_pref("extensions.screenshots.disabled", true); //uwu
user_pref("browser.search.suggest.enabled", false);
user_pref("i.have.no.nulls", null);
user_pref("browser.startup.page", +3);
user_pref("double.quotes", "\"my value\"");

Day 7 – Parsing Firefox’ user.js with Raku

One of the simplest way to properly configure Firefox, and make the configurations syncable between devices without the need of 3rd party services, is through the user.js file in your Firefox profile. This is a simple JavaScript file that generally contains a list of user_pref function calls. Today, I’ll be showing you how to use the Raku programming language’s Grammars to parse the content of a user.js file. Tomorrow, I’ll be expanding on the basis created here, to allow people to programmatically interact with the user.js file.

The format

Let’s take a look at the format of the file first. As an example, let’s use the startup page configuration setting from my own user.js.

user_pref("browser.startup.homepage", "https://searx.tyil.nl");

Looking at it, we can deconstruct one line into the following elements:

  • Function name: in our case this will almost always be the string user_pref;
  • Opening bracket;
  • List of arguments, seperated by ,
  • Closing bracket;
  • A ; ending the statement.

We can also see that string arguments are enclosed in ". Integers, booleans and null values aren’t quoted in JavaScript, so that’s something we need to take into account as well. But let’s set those aside for now, and first get the example line parsed.

Setting up the testing grounds

I find one of the easiest ways to get started with writing a Grammar is to just write a small Raku script that I can execute to see if things are working, and then extend the Grammar step by step. The starting situation would look like this.

grammar UserJS {
  rule TOP { .* }
}

sub MAIN () {
  my @inputs = ('user_pref("browser.startup.homepage", "https://searx.tyil.nl");');

  for @inputs {
    say UserJS.parse($_);
  }
}

Running this script should yield a single Match object containing the full test string.

「user_pref("browser.startup.homepage", "https://searx.tyil.nl");」

The and markers indicate that we have a Match object, which in this case signifies that the Grammar parsed the input correctly. This is because the placeholder .* that we’re starting out with. Our next steps will be to add rules in front of the .* until that particular bit doesn’t match anything anymore, and we have defined explicit rules for all parts of the user.js file.

Adding the first rule

Since the example starts with the static string user_pref, let’s start on matching that with the Grammar. Since this is the name of the function, we’ll add a rule named function-name to the grammar, which just has to match a static string.

rule function-name {
  'user_pref'
}

Next, this rule needs to be incorporated with the TOP rule, so it will actually be used. Rules are whitespace insensitive, so you can re-order the TOP rule to put all elements we’re looking for one after another. This will make it more readable in the long run, as more things will be tacked on as we continue.

rule TOP {
    <function-name>
    .* 
}

Running the script now will yield a little more output than before.

「user_pref("browser.startup.homepage", "https://searx.tyil.nl");」
 function-name => 「user_pref」

The first line is still the same, which is the full match. It’s still matching everything, which is good. If it didn’t, the match would fail and it would return a Nil. This is why we keep the .* at the end.

There’s an extra line this time, though. This line shows the function-name rule having a match, and the match being user_pref. This is in line with our expectations, as we told it to match that literal, exact string.

Parsing the argument list

The next part to match is the argument list, which consists of an opening bracket, a closing bracket to match and a number of arguments in between them. Let’s make another rule to parse this part. It may be a bit naive for now, we will improve on this later.

rule argument-list {
  '('
  .+
  ')'
}

Of course, the TOP rule will need to be expanded to include this as well.

rule TOP {
    <function-name>
    <argument-list> 
    .* 
}

Running the script will yield another line, indicating that the argument-list rule matches the entire argument list.

「user_pref("browser.startup.homepage", "https://searx.tyil.nl");」
 function-name => 「user_pref」
 argument-list => 「("browser.startup.homepage", "https://searx.tyil.nl")」

Now that we know this basic rule works, we can try to improve it to be more accurate. It would be more convenient if we could get a list of arguments out of it, and not include the brackets. Removing the brackets is the easier part, so let’s do that first. You can use the <( and )> markers to indicate where the result of the match should start and end respectively.

rule argument-list {
  '('
  <( .+ )>
  ')' 
}

You can see that the output of the script now doesn’t show the brackets on the argument-list match. Now, to make a list of the arguments, it would be easiest to create an additional rule to match a single argument, and match the , as a seperator for the arguments. We can use the % operator for this.

rule argument-list {
  '('
  <( <argument>+ % ',' )>
  ')'
}

rule argument {
  .+
}

However, when you try to run this, all you’ll see is a Nil as output.

Debugging a grammar

Grammars are quite a hassle to debug without any tools, so I would not recommend trying that. Instead, let’s use a module that makes this much easier: Grammar::Tracer. This will show information on how the Grammar is matching all the stuff. If you use Rakudo Star, you already have this module installed. Otherwise, you may need to install it.

zef install Grammar::Tracer

Now you can use it in the script by adding use Grammar::Tracer at the top of the script, before the grammar declaration. Running the script now will yield some content before you see the Nil.

TOP
| function-name
| * MATCH "user_pref"
| argument-list
| | argument
| | * MATCH "\"browser.startup.homepage\", \"https://searx.tyil.nl\");"
| * FAIL
* FAIL

Looking at this, you can see that an argument is being matched, but it’s being too greedy. It matches all characters up until the end of the line, so the argument-list can’t match the closing bracket anymore. To fix this, we must update the argument rule to be less greedy. For now, we’re just matching strings that appear within double quotes, so let’s change the rule to more accurately match that.

rule argument {
  '"'
  <( <-["]>+? )>
  '"'
}

This rule matches a starting ", then any character that is *not* a ", then another ". There’s also <( and )> in use again to make the surrounding " not end up in the result. If you run the script again, you will see that the argument-list contains two argument matches.

「user_pref("browser.startup.homepage", "https://searx.tyil.nl");」
 function-name => 「user_pref」
 argument-list => 「"browser.startup.homepage", "https://searx.tyil.nl"」
  argument => 「browser.startup.homepage」
  argument => 「https://searx.tyil.nl」

I’m ignoring the output of Grammar::Tracer for now, since there’s no problems arising. I would generally suggest just leaving in there until you’re completely satisfied with your Grammars, so you can immediately see what’s going wrong where during development.

The statement’s end

Now all there’s left to explicitly match in the TOP rule, is the statement terminator, ;. This can replace the .*, since it’s the last character of the string.

rule TOP {    
    <function-name>
    <argument-list>
    ';' 
}

The final Grammar should look like this.

grammar UserJS {
  rule TOP {
    <function-name>
    <argument-list>
    ';' 
  }

  rule function-name { 
    'user_pref' 
  } 

  rule argument-list {
    '('
     <( <argument+ % ',' )>
    ')' 
  } 

  rule argument {    
    '"'
       <( <-["]> )>
    '"'
  }

Now, the problem here is that it’s still quite naïve. It won’t deal with double quotes inside strings, not with Boolean values or integers. The current Grammar is also not capable of matching multiple lines. All of these problems can be solved, some easier than others. Come back here tomorrow to learn how!

Day 6 – Put some (GitHub) Actions in your Raku (repositories)

After being in beta for quite some time, GitHub actions were finally introduced to the general public in November 2019. They have very soon become ubiquitous, over all combined with the other release that were recently made by GitHub, the package (and container) registry.

We can put them to good use with our Raku modules. Well see how.

We could use some action

An action is a script that is triggered by an event in your repository. In principle, anything you or a program does when interacting with a repository could trigger an action. Of course, this includes git actions, which include basically pushing to the repository, but also all kinds of things happening in the repository, from changes in the wiki to adding a review to a pull request.

And what kind of things can you do? GitHub creates a container with some basic toolchains, as well as language interpreters and compilers of your choice. At the very basic level, what you have is a container where you can run a script triggered by an event.

GitHub actions reside in a YAML file places within the .github/workflows directory in your repository. Let’s go for our first one:

name: "Merry Christmas"
on: [push]
jobs:
seasonal_greetings:
runs_on: ubuntu-latest
steps:
- name: Merry Xmas!
run: echo Merry Xmas!
view raw hola.yaml hosted with ❤ by GitHub

This script is as simple as it gets. It contains a single job, with a single step. Let’s go little by little:

  • We give it a name, “Merry Christmas”. That name will show up in your list of actions
  • on is the list of events that will trigger this action. We will just list a single event.
  • jobs is an array that will include the list of jobs that will be run sequentially.
  • Every job will have its own key in the, which will be used to refer to it (and also to store variables, more on this later), and can run in its own environment, which you have to select. We’ll take ubuntu-latest, which is a Bionic box, but there are other to choose from (more on this later).
  • A job has a series of steps, every one with a name and then a sequence of commands. run will run on whatever environment is defined in that specific step; in this case, a simple shell script that prints Merry Xmas!

Since we’ve instructed via the on command to run every time there’s a push to the repository, the tab Actions will show the result of running it, just like this. If nothing goes wrong, and how could it, since it’s simply a script, it will show green check marks and produce the result:

Merry Xmas from a GitHub Action

These steps form a kind of pipeline, and every step can produce an output or change the environment that is going to be used in the next step; that means that you can create pipe actions that just process input and produce something for an output, like this one

name: "One step up"
on: [push]
jobs:
seasonal_greetings:
runs-on: ubuntu-latest
steps:
- name: Pre-Merry Xmas!
env:
greeting: "Merry"
season: "Xmas"
run: |
sentence="$greeting $season!"
echo ::set-env name=SENTENCE::$sentence
- name: Greet
id: greet
run: |
output=$(python -c "import os; print(os.environ['SENTENCE'])")
echo ::set-output name=printd::$output
- name: Run Ruby
env:
OUTPUT: ${{ steps.greet.outputs.printd }}
run: /opt/hostedtoolcache/Ruby/2.6.3/x64/bin/ruby -e "puts ENV['OUTPUT']"
view raw using-steps.yaml hosted with ❤ by GitHub

The first step in this action, code-named “Pre-Merry Xmas!”, declares a couple off environment variables via env. We will collate them in a single sentence. But here comes the gist of it: GitHub Actions use meta-sentences, preceded with ::, that are printed to output and interpreted as commands for the next step. In this case, ::set-env sets an environment variable.

The next step showcases the use of Python, which is another default tool in this environment; as a matter of fact, it’s included in every environment out there, together with Node; you can use it in its default version or set the version as an action variable. This step also uses a similar mechanism to set, instead of an environment variable, an output that can be used by the next step.

Unlike Python, Ruby does not have a default version available in the path; however, it’s only a matter of finding the path to it and you can use it, like here. This step also uses the output of the previous step; GHAs have contexts, in this case a step context, which can be used to access the output of previous steps. steps.greet.outputs.printd access the context of the step whose id is greet (which we declared via the id key there), and since we declared the output to be called printd, outputs.printd will retrieve the output by that name. Contexts are not available from within the action environment, which is why we need to assign it first to an environment variable. Output will look like this, and it will use green check marks, as well as reveal the output in the raw log and if you click on the step name.

If you are a long-term Perl use like I am, you will miss that. Ruby, Python, Node, popular languages, fair enough. But Perl is in the base Ubuntu 16.04 install. Even if we can use that environment, it seems to have been eliminated from there. Where do we have to go to use Perl? To the Windows environments. Let’s use it to create a polite bot that greets you when you create or edit an issue:

name: "We 🎔 Perl"
on:
issues:
types: [opened, edited, milestoned]
jobs:
seasonal_greetings:
runs-on: windows-latest
steps:
- name: Checkout
uses: actions/checkout@v2-beta
- name: Maybe greet
id: maybe-greet
env:
HEY: "Hey you!"
GREETING: "Merry Xmas to you too!"
BODY: ${{ github.event.issue.body }}
TOKEN: ${{ secrets.GITHUB_TOKEN}}
ISSUE: ${{ github.event.issue.number }}
run: |
$output=(perl utils/printenv.pl)
$body= @{ 'body'= $output } | ConvertTo-Json
$header= @{
'Authorization'="token $ENV:TOKEN"
}
Invoke-RestMethod -Uri "https://api.github.com/repos/JJ/raku-advent-calendar-article-2019/issues/$ENV:ISSUE/comments&quot; -Method 'Post' -Body $body -Headers $header

Check out first the on command, that is set to be fired every time an issue is created, edited or assigned a milestone, an action that, for some reason, is called being milestoned.

This lawn has been milestoned

The main difference you see above is the presence of the windows-latest as the environment this action will be run on. But next we see another nice things of actions: they can be simply published in GitHub, and can be reused. This checkout action does what it says: checks out the repo code, which is not available by default. We are not really going to run any check on the code, but we need the little Perl script we’ve created. More on this later.

The next step is the one that actually will operate when an issue is created, changed or, wait for it, milestoned. We declare two different environment variables: one will be used to comment on issues that don’t mention “Merry”, the other if they do. But the nice thing comes next: we can work with the issue body, which is available as a context variable: github.event.issue.body. The next variable is the magic key that opens the door to the GitHub API. No need to upload it or anything, it will be there ready for you, and GitHub will keep track of it and hide it wherever it appears. We will also need the issue number to comment on it, and we store it in the $ISSUE variable.

Let’s next run the action. We will use the fantastic Perl regexes to check for the presence of the word Merry in the body, using this mini-script:

print( ( ($ENV{BODY} =~ /Merry/) == 1)? $ENV{GREETING} : $ENV{HEY});

The next few PowerShell commands are, by far, the most difficult part of this article.

We run the script so that we capture, and store, the result in a variable. And the next commands create PowerShell hashes, and $body is converted to JSON. By using Invoke-RestMethod we use GitHub API to create a comment with the greetings in the issue that was milestoned or any or the other stuff.

Issue commented and milestoned

As the image above shows, couple of comments: one when it was created and the other, well, check the image.

However, last time we checked this was a Raku Advent Calendar, right? We want our Raku!

Using Raku in GitHub actions

Last time I checked, Raku was not among the very limited number of languages that are available in any of the environments. However, that does not mean we cannot use it. Actions can be upgraded with anything that can be installed, in the case of Windows using Chocolatey (or downloading it via curl or any other command). We’ll also use it to run a real test. Dummy, but real. All actions actually either succeed or fail; you can use that for carrying out tests. Check out this action:

name: "We 🎔 Raku"
on: [push, pull_request]
jobs:
test:
runs-on: windows-latest
steps:
- name: Checkout
uses: actions/checkout@v2-beta
- name: Install and update
run: |
cinst rakudostar
$env:Path = "C:\rakudo\bin;C:\rakudo\share\perl6\site\bin;$env:Path"
refreshenv
zef test .

Which is testing using this script:

#!/usr/bin/env perl6
use v6;
use Test;
constant $greeting = "Merry Xmas!";
constant $non-greeting = "Hey!";
is( greet( "Hey", $greeting, $non-greeting), $non-greeting, "Non-seasonal salutation OK");
is( greet( "Merry Xmas!", $greeting, $non-greeting), $greeting, "Seasonal salutation OK");
done-testing;
sub greet( $body, $greeting, $non-greeting ) {
($body ~~ /M|m "erry"/)??$greeting!!$non-greeting;
}
view raw 00-regex.p6 hosted with ❤ by GitHub

The regex here uses the Raku syntax to perform more or less the same thing that the previous Perl script did, but let’s focus on the action above. It runs three PowerShell commands, one of them using Chocolatey to install Rakudo Star, and then set the command path and refresh it so that it can be used in the last command, the usual zef test . that actually runs the tests.

Rakudo Star has not been updated since March; a new update is coming very soon, but meanwhile, the combination Windows/GitHub Actions/Rakudo is not really the best way to go, since the bundled zef version is broken and can’t be updated from within a GitHub action.

This test takes quite a while; you have to download and install Raku every single time, plus it does not work if you need to install any additional module. Fortunately, there are many more ways to do it. Meet the Raku container.

Using dockerized actions

GitHub actions can be created in two different environments. One of them is called node12, and can actually run any operating system, the other is docker, which is Linux exclusive.

These containers will be built on the run and then executed, with commands executed directly inside the container. By default, the ENTRYPOINT of the container will be run, as usual. Previously, we have used actions/checkout for checking out the repository; these official actions can be complemented with our own; in this case, we will use the Raku container action which you can also check out in the Actions markecplace.

This action basically contains a Dockerfile, this one:

FROM jjmerelo/alpine-perl6:latest
LABEL version="4.0.2" maintainer="JJ Merelo <jjmerelo@GMail.com>"
# Set up dirs
ENV PATH="/root/.rakudobrew/versions/moar-2019.11/install/bin:/root/.rakudobrew/versions/moar-2019.11/install/share/perl6/site/bin:/root/.rakudobrew/bin:${PATH}"
RUN mkdir /test
VOLUME /test
WORKDIR /test
# Will run this
ENTRYPOINT raku -v && zef install --deps-only . && zef test .
view raw raku-test.Dockerfile hosted with ❤ by GitHub

This Dockerfile does little more than establish the system PATH and an entry point that can be used for testing. It does not have anything that is Action-specific.

It uses the very basic Alpine Raku container, which is the basis for a whole series of Raku testing containers.

But again, let’s go back to where the action is, that is, er, the action.

name: "We 🎔 Ubuntu, Docker and Raku"
on: [push, pull_request]
jobs:
adventest:
runs-on: ubuntu-latest
name: AdvenTest
steps:
- name: Checkout
uses: actions/checkout@v1
- name: Runs tests
uses: JJ/raku-container-action@master

Sweet and simple, right?

Yes, I couldn’t help but call the test for the Advent Calendar AdvenTest.

It checks out the repository using the official checkouting action, and then runs the test, which is the default command in the Dockerfile that is created in that action. It would also install ecosystem dependencies, if there were any.

How long does this one take? Just short of 30 seconds, or one quarter of what the other one took.

Tell me more!

GitHub actions are a world of possibilities (and occasionally, also a world of pain). Containerized tools mean that you will be able to work on the repository and the world at large using your favorite language, that is, Raku, starting actions from any kind of events, interactive or periodical; for instance, you could schedule tests every week, or start deployments when tests have been cleared.

If you liked CI tools such as Travis or CircleCI, you will love GitHub actions. Put them to good use in your Raku repositories.

Day 5 – Modular Raku Command Line Apps

Modular Raku Command Line Apps

It was a three weeks before Christmas and Santa’s workshop was a mess. Elves were running around trying to get everything ready and it didn’t look like anything would be.

As soon as Santa walked in he was surrounded by a horde of unhappy elves all complaining at once.

“The system is too slow!” one piped up “It takes a second just to print out a help file.”

“And the help file is wrong! It’s out of date.” said a second.

“I just found the command to add move a child from the Naughty to Nice would except something else and die silently!!!” a third and extremely harrased voice, belonging to an elf with a very long list broke through.

Santa was worried, he’d spent much of the year working with a lot of the Dev-elfs to update their older systems to use Raku and they thought everything was going swimingly. All the different modules worked really well in testing and the single command to manage the various system was the only section he’d not really been involved in.

He looked around for Snowneth the Elf in charge of that project for some answers but couldn’t see him. A quick set of questions and Santa found out Snowneth had come down with Elf flu a few months ago and was still of sick.

Santa kicked himself for not keeping up with things, but there was so much to do, he made a note to pop in and check on Snowneth later. Now to find out what had gone wrong and why no one had told him!

He opened up his laptop and tried out the main system service in test mode :

helper -?

A couple of seconds later the system gave him a command line prompt… nothing else.

helper -h

This time he got a documentation page, a quick scan of it and he could see a number of the commands were documented incorrectly and there were at least two more recent ones missing. Fearing what he would see he opened up the git log, all the commits had been done by a number of names he recognised. The junior Dev-elfs who had been assigned to Snowneth’s team.

He closed his laptop and went to find a coffee machine.


A few hours, and coffees, later, Santa and a number of young worried looking elves were huddled in a small office. Santa had his laptop open and the code for the service script was open in front of them, some of the 1000 lines of it.

By now Santa and found out what had happened, Snoweth had gone sick and everyone had thought someone else was going to appoint a new team lead. Meanwhile the juniors did their best to get the job done. Santa took a moment to get his breath and compose his voice.

“Firstly I have to apologise. I’m sure you’ve all be under a lot of stress and you’ve all done your best to get this vital work done.”

The juniors perked up, by now they’d all got the message that their work was causing the workshop to run slow and maybe even cancel Christmas! They had been expecting to be shouted at, maybe even sent to work mucking out the reindeer.
The traditional job for an elf who had messed up.

“We can get into what happened later, for now we need to fix this code as fast as possible and get things unblocked. I’m hoping you can all help me to get this done.”

The juniors by now where nodding and smiling, Santa was happy, he should have been on top of this mess and he hoped by getting them involved in the clean up they could regain some confidence.

“Right, so what are the problems we’ve got?”

Everyone spoke up and they quickly whiteboarded a list :

  • Slow to start for any command
  • Some commands have incorrect input validation
  • Out of Date documentation

There were a few other things but there three seemed to be the main ones.

“Ok. So slow startup. I think this is obvious right?”

All the juniors looked at him, then one raised his hand.

“Sniffy isn’t it? What do you think? And no need to raise your hand, just speak up… In turn try not to shout over each other.”

“Is it because the script is so long? I think I read that a Raku script is compiled when you call it? If we make the script shorter it will run faster?”

“That’s pretty much it yes, though less complex would be maybe a better goal. We’ve got a lot of if clauses and the like. The important thing to remember is Module code is precompiled but the main script isn’t. So wherever possible we should be using modules.”

The junior nodded their heads. Then one spoke up.

“But we are using the Modules, they are all there at the top of the script but then we have to work out what command people are calling then work out what arguments they are passing. Then…” Santa held up his hand.

“I can see that Wibbly. I think that’s where we should start.” He pointed to a section of code on the screen. “Who can tell me what’s wrong with this.”

use Workshop::ListController::Nice;

my $command = @*ARGV[0];

if ( $command ~~ 'list' ) {
    my $list-command = @*ARGV[1];
    if ( $list-command ~~ 'add' ) {
        if ( @*ARGV[2] ~~ 'nice' ) {
           Workshop::ListController::Nice.add-child( @*ARGV[3] ); 
        }

And so on. The juniors looked at it and talked among themselves. Sniffy spoke up again.

“It’s a bit complicated, it didn’t start that way but as we added commands it got bigger. Also we’re not checking the childs name is valid”

“That’s true but what I wondered was why are you reading @*ARGS?”

There was a look of confusion on their faces.

“Why don’t we have a MAIN sub routine?”

Still confusion.

“Ok. You all go look that up. I’lll whip up an example.”

He quickly typed while the elves went to the Raku Docs site and started searching. As he heard their exclamations raise then slowly quieten he’d turned away from the keyboard.

use Workshop::ListController::Nice;

multi sub MAIN( "list", "add", "nice", ValidChildName $child ) { Workshop::ListController::Nice.add-child( @*ARGV[3] ); }
multi sub MAIN( "list", "add", "nice", Str $invalid-child ) { die "Invalid childname {$invalid-child}"; }

There was a round of applause from the juniors.

“So we can use the Raku multi dispatch with MAIN to create all our commands down to a quite granular level and also have type checking in place. I think this is a good place to start. I’d like us to get to the point where all we have in this script is MAIN off we go.”


The next day and things were looking better, the script was faster to start and the input validation issues had been resolved. Everyone was feeling a lot better. Santa had even got the time to see Snoweth and make sure he was OK.

“Ok… Now lets look at this.”

He pointed at the largest subroutine left in the file, it started like this.

multi sub MAIN( :$h where so * ) {
    say "Usage:";
    say "  helper list add nice [child] : Adds a child to the Nice list."
    say "  helper stable rota list : Lists the current rota for cleaning the stables."

One of the elves jumped to her feet, eyes sparkling.

“Gimlina I believe you have some thoughts on this.”

“We should use declarator blocks!” Santa smiled, he’d had to hold back the young elf’s enthusiasm yesterday in order to keep the team focused on the task in hand. He smiled and nodded.

“Carry on.”

“If we add declarator blocks to our subroutines and arguments we get pregenerated documentaion for free. And it gets updated whenever the code changes.”

“Can you give a demonstration?”

She smiled and brought up some code.

#| Add a child to the nice list
multi sub MAIN( "list", "add", "nice", 
    ValidChildName $child #= Child name to add
) { 
    Workshop::ListController::Nice.add-child( @*ARGV[3] ); 
}

multi sub MAIN( "list", "add", "nice", Str $invalid-child ) is hidden-from-USAGE { die "Invalid childname {$invalid-child}"; }

“So the #| does what?”

“Attaches the block to whatever comes after it. #= attaches the block to the preceding item.”

“And is hidden-from-USAGE?”

“Well when you call the script and it doesn’t know what to do, or you pass -? it calls the USAGE function and that displays the $*USAGE string. Which is generated from the declarator blocks. But some subs you don’t want to display, so you can hide them.”

She quickly typed

helper -?
Usage:
  helper list add nice [child] -- Add a child to the nice list
  
        Child name to add 

Santa nodded and the rest of the juniors burst into enthusiastic cheers. Gimlina looked happy as they all turned to adding documentation to the helper script.


Things were looking better, the workshop was sounding a lot happier and the documentation was going well. Santa was sitting in with the juniors just checking final things over when he saw a merge request in one of the module repositories. He looked confused, they’d been in code freeze for a week now and only urgent bug fixes should be raised and he’d not heard of any issues beyond the helper script. He cursed himself again and thought that maybe he did need some product manager elves.

When he opened the merge request his eyes widened, he turned to one of the quieter juniors, they were very smart but tended to not speak up, just kept their head down and worked on things.

“Erry? What’s the MR in the Workshop::ListController::Nice module?”

As he asked, and waited on them to come over he took a look at the code, his eyes now so wide they looked like they would bulge out of his head.

#| Add a child to the nice list
multi sub MAIN is export(:MAIN) ( "list", "add", "nice", 
    ValidChildName $child #= Child name to add
) { 
    Workshop::ListController::Nice.add-child( @*ARGV[3] ); 
}

multi sub MAIN( "list", "add", "nice", Str $invalid-child ) is hidden-from-USAGE is export(:MAIN) { die "Invalid childname {$invalid-child}"; }

“Well I was thinking sir” the elf, in a suprisingly loud voice, said from next to him. “You said the module code is precompiled. So if we moved our MAIN subs into the modules they’d be precompiled.”

“And the module teams could manage their own command line interfaces and documentation!” Santa exclaimed happily, Erry nodded smiling.

The rest of the day was a riot of work as all the other dev-elfs (who had been looking forward to a nice month long code freeze before the post holiday backlog fighting) got pulled in and told the new plan.

By the end of the day the helper script was a long line of module use statements and one solitary function.

use Workshop::ListController::Nice :MAIN;

#| Display the help 
multi sub MAIN( :h($help) where so * ) {
    say $*USAGE;
}

Santa looked at it and at Snowneth who had finally made it back to the workshop that afternoon. He shrugged.

“I didn’t know about -? and I have got used to used to -h. Can’t hurt right?”

Santa nodded and they went off to help work in the workshop.

Day 4 – Not tripping over tripcodes

Greetings. Today we are going to look at an implementation of tripcodes, a kind of hashing used for signing posts anonymously on the Internet.

There are different algorithms to do so, but one that we are interested in is one generating non-secure, old-fashioned tripcodes.

So what is it?

Say there is a website allowing to leave comments while staying anonymous. No registration, no login, no usernames.

You respond to a post and then a person is responding to your response. You start a conversation. You know that your posts are yours. But what about all the other users? Are you still talking to the same person or some bunch of kids around playing their tricks on you? No idea! To resolve that sort of confusion in some situations, a tripcode can be used.

The idea is simple: along with your post you can pass your wanted nickname and a password. The website takes a password and hashes it into a tripcode. On displaying posts, the tripcodes are attached to messages, so you can make sure this is the same person who knows the password. Of course, nobody demands people to claim authorship of their posts, but we are leaving that aside, as we are interested in an implementation.

Examples at hands

Implementing the algorithm takes a single subroutine. Yet we need a way to test our tripcodes. Let’s define some testing rules for your tripcode subroutine:

use Test;

is tripcode('a'), '!ZnBI2EKkq.';
is tripcode('¥'), '!9xUxYS2dlM';
is tripcode('\\'), '!9xUxYS2dlM';
is tripcode('»'), '!cPUZU5OGFs';
is tripcode('?'), '!cPUZU5OGFs';
is tripcode('&'), '!MhCJJ7GVT.';
is tripcode('&amp;'), '!QfHq1EEpoQ';
is tripcode('!@#heheh'), '!eW4OEFBKDU';

Raku code

Now let’s take a look at the algorithm:

  • Escape HTML characters
  • Convert all characters to CP932 encoding. For characters where it is not possible, use ? symbol
  • Decode resulting bytes as UTF8
  • Generate salt for our hash. To do this, add H. string to our decoded string (as it might be empty!), take second and third characters. Next, substitute any “weird” characters (in ASCII terms, anything that has code below 46 (.) and above 122 (z)) with a dot.
  • Translate some non-word characters (:;<=>?@[\\]^_`) into ABCDEFGabcdef.
  • Use UNIX function crypt with the decoded string and a salt we got, and took last 10 characters of it.
  • That’s all!

There are quite a lot of steps, but let’s see how we can code such a task in Raku.
Let’s start with a sub declaration:

sub tripcode($pass is copy) {

}

We are going to modify the $str variable in-place, so is copy trait of the parameter will help us against the “passed Str value is immutable” error.

Next, escape HTML:

sub tripcode($pass is copy) {
    $pass .= trans(['&', '<', '>'] => ['&amp;', '&lt;', '&gt;']);
}

With the trans method, we can replace characters in a string using “left to right” correspondence, so & is replaced with &amp;, < is replaced with &lt; etc.

Next thing – dances with Windows 932.

$pass .= trans(['&', '<', '>'] => ['&amp;', '&lt;', '&gt;']);
$pass = ([~] $pass.comb.map({ (try .encode('windows-932')) // Buf.new(0x3F) })).decode;

Let’s imagine writing this line step by step:

# split $pass into single characters
$pass.comb
# for every character in the list resultring from `comb` method call
$pass.comb.map({  })
# try to encode it into the encoding we want
$pass.comb.map({ try .encode('windows-932') })
# when `try` returned `Nil`, use `//` operator which means `If the left side is not defined,`use the right side
$pass.comb.map({ ((try .encode('windows-932')) // Buf.new(0x3F) })
# Use [~] unary metaoperator, which is a shortcut for "join this array using this operator to join two single elements"
([~] $pass.comb.map({ (try .encode('windows-932')) // Buf.new(0x3F) }))
# At last, decode the resulting buffer and assign it to the variable
$pass = ([~] $pass.comb.map({ (try .encode('windows-932')) // Buf.new(0x3F) })).decode;

Now we need to generate some salt for our hash.

my $salt = "{$pass}H.".substr(1, 2).subst(/<-[. .. z]>/, '.').trans(':;<=>?@[\\]^_`' => 'ABCDEFGabcdef');

Firstly, we add H. part to the password, then taking second and first characters using substr call. Note the second call is subst, which replaces anything outside of regex range with a dot. Here, substr is a short for substring, while subst is a short for substitute. Then goes our trans method.

As the next thing, we need to call UNIX crypt function. Luckily, we don’t need to implement it! In Raku’s ecosystem there is already a module Crypt::Libcrypt written by Jonathan Stove++. Let’s install it:

zef install Crypt::Libcrypt

Now we can import this module and have crypt subroutine at our service. The last line is simple:

'!' ~ crypt($pass, $salt).substr(*-10, 10);

We don’t need to write an explicit return statement, as the last statement of a block is considered to be its return value. A call to crypt subroutine and our old friend substr with the first argument looking funny. The second argument is, as usual, a number of characters we want, while the first one is an expression with Whatever Star used. On call, the substr caller’s length is passed into this micro-block of code, so it is translated into 'foo'.substr('foo'.chars() - 10, 10) (but smarter inside).

Comprising everything, we get a full definition:

sub tripcode($pass is copy) {
    $pass .= trans(['&', '<', '>'] => ['&amp;', '&lt;', '&gt;']);
    $pass = ([~] $pass.comb.map({ (try .encode('windows-932')) // Buf.new(0x3F) })).decode;
    my $salt = "{$pass}H.".substr(1, 2).subst(/<-[. .. z]>/, '.').trans(':;<=>?@[\\]^_`' => 'ABCDEFGabcdef');
    '!' ~ crypt($pass, $salt).substr(*-10, 10);
}

Check it:

> perl6 tripcode.p6
ok 1 - 
ok 2 - 
ok 3 - 
ok 4 - 
ok 5 - 
ok 6 - 
ok 7 - 
ok 8 -

A success, all the checks we prepared pass! As we successfully implemented the algorithm using only four lines of code, it is time to refill some hot drink. Have a nice day!

Day 3 – Stack Frame Reduction

Stack Frame Reduction

What is a Stack Frame?

 width=

For those not familiar with the stack, it is a bit of memory for your program to use. It is fast but limited.
Whenever you call a procedure (function, method,… naming is a complicated thing) your program gets a bit of storage on the stack, which we call a frame.
The stack frame gets used for storing parameters, local variables, temporary storage, and some information about the calling context.
This means that if you have a recursive procedure call your program keeps asking for stack frames until you eventually return a value and the memory is freed up.

A quick and simple example:

Let us take the standard example of a basic recursive sorting algorithm:

sub factorial (Int $n --> Int) {
        $n == 0 ?? 1 !! $n * factorial($n - 1)
}

This is a very simple example of recursion, and usually we don’t have to worry about stack frame buildup in this code. That said, this is a good starting point for showing how to reduce the buildup.

GOTO reduction:

basic
Didn’t Larry start with Basic?

This way of reducing stack frame buildup should be familiar to most people, it’s the way procedural programming handles recursion.

The most basic implementation of this pattern looks like this:

sub factorial (Int $n is copy --> Int) {
        my Int $result = 1;
        MULT:
        $result *= $n;
        $n--;
        goto MULT if $n > 0;
        return $result;
};

GOTO is not yet implemented in Raku, but it should be fairly obvious we can easily replace this with an existing keyword:

sub factorial (Int $n is copy --> Int) {
        my Int $result = 1;
        while $n > 0 {
                $result *= $n;
                $n--;
        }
        return $result;
}

This does defeat the purpose of trying to use recursion, though. Therefore Raku offers the samewith keyword:

sub factorial (Int $n --> Int) {
        $n == 0 ?? 1 !! $n * samewith($n - 1);
}

There we go, recursion without incurring a thousand stack frames. I still think we’re missing something, though…

TRAMPOLINE!

trampoline-by-charles-cheng
Everything is better with trampolines, with penguins, in space, or on ice.

A trampoline is a design pattern in Functional Programming. It is a little complicated compared to normal GOTO-style reduction, but in the right hands it can be very powerful.
The basics behind the trampoline pattern are as follows:

  • We can expect to do something with the value we’re computing.
  • We can just pass our TODO into the function that computes the value.
  • We can have our function generate its own continuation.
sub trampoline (Code $cont is copy) {
        $cont = $cont() while $cont;
}

So we pass the trampoline a function. That function is called. The function optionally returns a follow-up. As long as we get a follow-up, we keep calling it and assigning the result until we’re done.

It requires a little reworking of the factorial function:

sub factorial (Int $n, Code $res --> Code) {
        $n == 0 ?? $res(1) !! sub { factorial($n - 1, sub (Int $x) { $res($n * $x) }) }
}

To unpack that heap of stacked functions:

  • If $n is 0, we can just move on to the continuation.
  • Otherwise we return an anonymous function that calls factorial again.
  • The previous step propagates until we arrive at 0, where we get the result called with 1.
  • That multiplies the previous $n with 1, and propagates the result backwards.
  • Eventually the result is propagated to the outermost block and is passed into the continuation.

The way we would use the trampoline then follows:

trampoline(sub { factorial($n, sub (Int $x) { say $x; Nil }) });

Again, a bunch of tangled up functions to unpack:

  • We send an anonymous function to the trampoline that calls factorial with a number $n, and an anonymous continuation.
  • The continuation for the factorial is to say the result of the factorial and stop (the Nil).

Bonus round

Why would you use a trampoline for something that could be done easier with a regular for loop?

sub factorial-bounce (Int $n --> Code) {
        sub { factorial($n, sub ($x) { say $x, factorial-bounce($x) }) }
}