Day 18: What is my concurrent or parallel Raku program doing?

Raku makes it easy to introduce a bit of parallelism into a program – at least, when the solution lends itself to that. Its asynchronous programming capabilities also shine when building concurrent applications. During the last year, I’ve enjoyed employing both of these in my work.

However, I also discovered that something was missing. It was relatively straightforward to build the parallel and concurrent things that I wanted in Raku. However, once they were built, I found it hard to reason about what they were doing. For example:

  • For data parallel operations, what degree of parallelism was being achieved?
  • What tasks executed in parallel, and were there missed opportunities for further task parallelism?
  • Where was time being spent in different stages of an asynchronous workflow? For example, if a HTTP request could trigger background work, which later led to a message being sent over a WebSocket, where did the time go between these steps?
  • Given a Cro web application is also an asynchronous pipeline, where is time going on request processing? Is some piece of middleware surprisingly eating up a lot of time?

At first, I started putting in bits of timing code that wrote results out to the console. Of course, I either had to remove it again afterwards, or guard it with an environment variable if I wanted to keep it in the code. And even then, the data it produced was quite hard to interpret. This all felt rather inefficient.

Thus, Log::Timeline was born. Using it, I learned a lot about my application’s behavior. Best of all, I didn’t just get answers to the questions I had, but also some that I hadn’t even considered asking. In this advent post, I’ll show how you can use this module also.

Trying it out

The first way we might make use of Log::Timeline doesn’t actually involve using the module at all, but rather using some library that already uses it to do logging. Since Cro::HTTP has done this (since 0.8.1), we can get our very first glimpse at Log::Timeline in action by building a Cro HTTP application.

Aside from something that produces log events, we also need some way to view the logs. At present, there are two modes of Log::Timeline output: writing a file in JSONLines format or sending them over a socket. The second of these is used by the log viewer in Comma (a Raku IDE), meaning that we can view the output live as our application runs.

Thus, assuming we already installed Cro, we can:

  1. Create a new Cro Web Application project in Comma (works much like using cro stub at the command line)
  2. Create a “Cro Service” Run Configuration (choose Edit Configurations on the Run menu)
  3. On the Run Menu, choose “Run ‘service.p6’ and show Timeline”

The service will start up, and display the URL it is running at in the console (probably http://localhost:20000). If we make a request to it, then flip to the Timeline tab, we’ll see the request logged (in fact, do it in a browser then probably two requests are logged, since a favicon.ico is automatically requested). The requests can be expanded to see how the processing time within them is spent.

cro-app-1

Cro can process requests to HTTP application in parallel. In fact, it’s both concurrent (thanks to use of asynchronous I/O) and parallel (all steps of request processing are run on the Raku thread pool). So, if we use the Apache Benchmark program to send 100 requests, 3 at a time, we’d hope to see an indication that up to 3 requests were being processed in parallel. Here’s the command:

ab -n 100 -c 3 http://localhost:20000/

And we do, indeed, see the parallelism:

cro-app-2

Similarly, if we jack it up to 10 concurrent requests:

ab -n 100 -c 10 http://localhost:20000/

Then we will see something like this:

cro-app-3

Looking a little more closely, we see that the “Process Request” task is logged as part of the HTTP Server category of the Cro module. However, that’s not all: tasks (things that happen over a period of time) can also have data logged with them. For example, the HTTP request method and target are logged too:

data

We might also notice that requests are displayed in alternating shades of color; this is to allow us to differentiate two tasks if they occur “back to back” with no time between them (or at least, none visible at our zoom level).

Adding Log::Timeline support to an application

What if we want to add Log::Timeline support to our own an application, so we can understand more about its behavior? To illustrate this, let’s add it to jsonHound. This is an application that looks through JSON files, and ensures that they are in compliance with a set of rules (its was built for checking the security of router configurations, but in principle could be used for much more).

When jsonHound is run, there are two steps:

  1. Loading the rules, which are written in Raku
  2. Checking each specified JSON file against the rules; if there is more than one file, they will be checked in parallel

We’ll create a Log::Timeline task for each of these. They go into a JsonHound::LogTimelineSchema module. The code in the module looks like this:

unit module JsonHound::Logging;

use Log::Timeline;

class LoadRules does Log::Timeline::Task["jsonHound", "Run", "Load Rules"] {
}

class CheckFile does Log::Timeline::Task["jsonHound", "Run", "Check File"] {
}

First, we use the Log::Timeline module. Then we create a class for each task that does the Log::Timeline::Task role (there is also an Event role, which can be used to log events that happen at a particular moment in time).

Next, we need to modify the program to use them. First, in the code that we want to add logging to, we need to use our task schema:

use JsonHound::LogTimelineSchema;

And now we can go to work. Loading the ruleset looks like this:

my $*JSON-HOUND-RULESET = JsonHound::RuleSet.new;
my $rule-file = $validations.IO;
CompUnit::RepositoryRegistry.use-repository:
        CompUnit::RepositoryRegistry.repository-for-spec(
                $rule-file.parent.absolute);
require "$rule-file.basename()";

We wrap it up as follows:

JsonHound::Logging::LoadRules.log: {
    my $rule-file = $validations.IO;
    CompUnit::RepositoryRegistry.use-repository:
            CompUnit::RepositoryRegistry.repository-for-spec(
                    $rule-file.parent.absolute);
    require "$rule-file.basename()";
}

When we’re not running with any logging output, the block is just executed as normal. If, however, we were to have the socket output, then it would send a message out over the socket when the block’s execution begins, and another when it ends.

The per-file analysis looks like this:

.($reporter-object) for @json-files.race(:1batch).map: -> $file {
    # Analysis code here
}

That is, we take the JSON files, and then map over them in parallel. Each produces a result, which we then invoke with the reporter. The exact details matter little; all we really need to do is wrap the analysis in our task:

.($reporter-object) for @json-files.race(:1batch).map: -> $file {
    JsonHound::Logging::CheckFile.log: {
        # Analysis code here
    }
}

Handily, the log method passes along the return value of the block.

What we’ve done so far will work, but we can go one step better. If we look at the log output, we might see a JSON input file that takes a long time to process, but we’d not know which one it is. We can annotate the log entry with whatever data we choose, simply by passing named arguments.

.($reporter-object) for @json-files.race(:1batch).map: -> $file {
    JsonHound::Logging::CheckFile.log: :$file, {
        # Analysis code here
    }
}

And with that, we’re ready! After adding a run configuration in Comma, and running it with the timeline viewer, we get a chart like this:

jsonhound

The future

While Log::Timeline can already offer some interesting insights, there’s still plenty more functionality to come. Current work in progress uses a new MoarVM API to subscribe to GC events and log them. This means it will be possible to visualize when GC runs and how much time is spent in it – and to correlate it with other events that are taking place.

I’m also interested in exposing various Rakudo-level events that could be interesting to see plotted on the timeline. For example, it would be possible to provide information on lock waiting times, or await times, or Supply contention times. Other ideas are plotting the times files or other resources are opened and closed, which may in turn help spot resource leaks.

Of course, just because there may be a wide range of things we could log does not mean they are all useful, and logging carries its own overhead. The use of the LogTimelineSchema naming convention looks forward to a further feature: being able to introspect the full set of events and tasks available. In Comma, we’ll then provide a UI to select them.

In closing…

No small amount of the time we spend solving problems with our programs goes on finding out what happens as they run. Good tooling can provide a window into the program’s behavior, and in some cases point out things that we might not even have considered. Log::Timeline is a pretty small module, and the visualizer didn’t take terribly long to implement either. However, the payback in terms of useful information makes it one of the most worthwhile thing I’ve built this year. I hope you might find it useful too.

Day 17: Maze Maker for Fun

Maze Maker

I remember when I was in school, I used to play the game where you help the mouse go through the maze and reach the target as quickly as possible. Even
when I finished my college, I would still wonder how one can create a maze. Those who create maze must be super ingenious.

It was only recently I decided to decode the mystery of maze creation. I asked my friend Google and guess what, I was presented with plenty of solutions in various languages. Some very difficult to follow and some not so difficult. Having got my head around the algorithm, I decided to do it in my first language i.e. Perl. It didn’t took me long to convert the algorithm into Perl script.

As you all know, I am still a new player in Raku. Thankfully I have plenty of support when it comes to Raku queries. For the first time, I decided to be part of Raku Advent Calendar. Thanks to JJ Merelo, I was given a slot. At that point, I didn’t have a clue what I am going to do. But I knew I have plenty of ideas that can be converted into Raku.

As you guessed it correctly, I picked my favourite maze maker script for the Raku Advent Calendar. With the help of many people on Twitter, I got my first draft ready in Raku. But it wasn’t creating the tunnel as expected. I went to my friend, Scimon Proctor, for guidance. He helped me with the Raku magic and all working smoothly now.

It was time to do some housekeeping before I hand it over to JJ Merelo. Having spent around 30 minutes, I got my final draft ready as below:

use v6;

my %opposite-of = ( 'north' => 'south', 'south' => 'north',  'west' => 'east', 'east'  => 'west' );
my @directions  = %opposite-of.keys;

sub MAIN(Int :$height = 10, Int :$width = 10) {
    my $maze;
    make-maze( 0, 0, $maze, $height, $width );
    say render-maze( $maze, $height, $width );
}

#
#
# METHODS

sub make-maze( $x, $y, $maze is rw, $height, $width ) {

    for ( @directions.pick(@directions) ) -> $direction {
        my ( $new_x, $new_y ) = ( $x, $y );

        if    ( 'east'  eq $direction ) { $new_x += 1; }
        elsif ( 'west'  eq $direction ) { $new_x -= 1; }
        elsif ( 'south' eq $direction ) { $new_y += 1; }
        else                            { $new_y -= 1; }

        if not-visited( $new_x, $new_y, $maze, $height, $width ) {
            $maze[$y][$x]{$direction} = 1;
            $maze[$new_y][$new_x]{ %opposite-of{$direction} } = 1;
            make-maze( $new_x, $new_y, $maze, $height, $width );
        }
    }
}

sub not-visited( $x, $y, $maze, $height, $width ) {

    # check the boundary
    return if $x < 0 or $y < 0;
    return if $x > $width - 1 or $y > $height - 1;

    # return false if already visited
    return if $maze[$y][$x];

    # return true
    return 1;
}

sub render-maze($maze, $height, $width) {

    my $as_string = " " ~ ( "_ " x $width );
    $as_string ~= "\n";

    for ( 0 .. $height - 1 ) -> $y {
        $as_string ~= "|";
        for ( 0 .. $width - 1 ) -> $x {
            my $cell = $maze[$y][$x];
            $as_string ~= $cell<south> ?? " " !! "_";
            $as_string ~= $cell<east>  ?? " " !! "|";
        }
        $as_string ~= "\n";
    }

    return $as_string;
}

What next?

I am thinking of converting this into a Raku Library, something like Games::Maze. I am working on it, hopefully I should have it ready before Christmas.

If you think it can be improved in any way then please do share with me. I would love to hear from you and learn from your experience. Till then have fun with it from a Perl Fan, who equally loves Raku.

Day 16: Raku powered jmp-ing to the coalface

Santa makes sure his elves can get to and from the workplace quickly. I wish it was the same for computer programming! Sometimes it takes a while to wade through a codebase to find where the real work needs to happen.

jmp is a Raku powered command-line utility I use for searching and jumping around large codebases and command output. It provides a terminal frontend to your favourite $code-searching-tool (e.g., ag, git grep, ack etc) so you can quickly jump into your favourite $EDITOR (e.g., vim, code, emacs etc).

It works like this:

jmp-demo

When jmp-ing into large codebases I often need to visit lots of different files and repositories. Sometimes this blows my short-term memory buffer. To try and stay in flow I leave a breadcrumb trail of comments throughout the code (i.e., “# COAL”).  Later when I need to get back to the coalface of file locations I do a jmp search for ‘COAL’.

shell> jmp to COAL

That works OK but it’s a manual process and requires cleaning up all the COAL markers afterwards. Can jmp help out a bit more here?

It would be cool if jmp automatically remembered where I’d visited in a codebase so I didn’t need to leave COAL markers around. Sounds like a bit more Raku is required!

To achieve this – jmp needs a memory:

# keep a file-based record of the searches and jmps
class JMP::Memory {

    has $.max-entries = 100; # keep a record of the last 100 jmps
    has $!file;
    has @!latest-jmps;

    #| get a list of the most recent JMP::File::Hits
    method get-recent-jmps ($last-n-jmps) { ... }

    #| write the memory file 
    method save ($jmp-command, $hit) { ... }

    submethod TWEAK { ... }

}

So let’s fill this class out.

TWEAK is a special submethod for helping to lazily finish building objects after an instance is created. Not all invocations of jmp need memory recall so it’s good not to process the history file if we can avoid it. The $!file and @!latest-jmps are encapsulated as private attributes so they are easy to change in future if needed.

submethod TWEAK {

    $!file = $*HOME.add('.jmp.hist').path;
    return unless $!file.IO.e;
    return without my $history = from-json($!file.IO.slurp);
    @!latest-jmps = $history.List;

}

The handy global $*HOME IO::Path object helps to create a cross-platform location for the jmp history file (e.g., ~/.jmp.hist). If the file exists we slurp the file in one go and parse the json payload into a .List of jmp records with the help of JSON::Tiny::from-json.

Whenever a search hit occurs we need to save the jmp command and the selected hit:

#| write the memory file 
method save ($jmp-command, $hit) {

    # for each incoming hit - we record two entries - the jmp command
    my %jmp-record = %(
        current-directory => $*CWD.path,
        jmp-command       => $jmp-command,
    );

    # and the selected destination
    my %hit-record = %(
        line-number   => $hit.line-number,
        file-path     => $hit.file-path,
        full-path     => $hit.full-path,
        matching-text => $hit.matching-text,
    );

    @!latest-jmps.unshift(%hit-record);
    @!latest-jmps.unshift(%jmp-record);

    my @hits = @!latest-jmps;

    @!latest-jmps = @hits.head($!max-entries);

    # dump to disk
    $!file.IO.spurt(to-json(@!latest-jmps));
}

Unshifting the record to the start of the list means more recent jmps appear at the top. Limiting the head() of the list ensures the list won’t grow too big and jmp stays fast.

When the user calls jmp with no arguments a list of recent jmps is displayed.

method get-recent-jmps ($last-n-jmps) {
    # return a list of JMP::File::Hits 
    my @recent-jmps;
    for @!latest-jmps.head($last-n-jmps) -> %hit {
        my $hit = %hit<jmp-command>:exists
                ?? JMP::Memory::Command.new(|%hit)
                !! JMP::Memory::Hit.new(|%hit);
        @recent-jmps.push($hit);
    }
    return @recent-jmps;
}

The for loop iterates through the previously recorded jmp commands and the selected search hits. Here is the full source code of JMP::Memory.

Now jmp can help you to return quickly to the coding coalface without leaving a mess behind.

To install the latest version of jmp, first install Raku, then use zef the Perl 6 module manager to install it:

shell> zef install jmp
shell> zef upgrade jmp   # if you've installed it before

Looking forward to using more Raku-powered utilities in 2020.

Happy Christmas!

Day 15 – Santa had too much eggnog

Santa had too much eggnog

We’re just over a week from Christmas and Santa is sending his elves the final present lists. Unfortunately, Santa had a bit too much eggnog and so the list that he sent to his elves was … not the greatest. Take a look at some of it:

Johnny
 - 4 bsaeball gluvs
 - 2 batts
 - 2 ballz
Mary
 - 3 fancee dols
 - 1 dressss
 - 1 bbaskebtall

Santa somehow managed to keep a nice format that we could mostly process with regexen, so the elves started hammering away at a nice grammar:

grammar Santa'sList {
  rule TOP        {       <kid's-list>+    }
  rule kid's-list {     <name>     <gift>+ }
  rule gift       { '-' <quantity> <item>  }
  token name      { <-[\n]>   }
  token quantity  { <.digit>+ }
  token item      { <.alpha>+ % \h+ }
}

While the elves figured that they could try to figure out what he meant in an action object, they decided it would be more interesting to create a token that they could reuse not just in the grammar, but in any random regex — these elves are crafty!

They wanted to make a new token that they’d call <fuzzy> that could somehow capture Santa’s drunken scribblings (can we call his typed list a scribbling?). But regex syntax doesn’t actually allow for doing any kind of fuzzy matching. But here Raku’s engine comes to the rescue. So first they created a code block inside of the token. Code blocks are normally defined with just { 🦋 } but because they needed to define the success of a match, they opted instead of the <?{ 🦋 }> conditional block, which will not only run the code, but will also fail if the block returns a false-y value.

  token fuzzy {
    (<.alpha>+ % \h+)
    <?{
      # «ö» code here
    }>
  }

Before they started writing their code, they did two other things. First they named the capture to be a bit easier to maintain down the road. And secondly, they realized they needed to actually get the list of possible toys into the token somehow. So they added a signature to the token to pass it in.

  token fuzzy(**@toys) {
    $<santa's-text>=(<.alpha>+ % \h+)
    <?{
      # «ö» code here
    }>
  }

Now they could begin the code itself. They would take Santa’s text, and compare it to each of the possible toys, and decide which one was the closest match:

  token fuzzy(**@toys) {
    $<santa's-text>=(<.alpha>+ % \h+)
    <?{
      my $best = @toys
                   .map({ $^toy, qgram($toy,$<santa's-text>.Str)})
                   .sort( *.tail )
                   .tail;
      say "Santa meant to write {$best[0]}";
    }>
  }

The Q-gram function they used creates N-grams for each word, and compares them to see how many they have in common. With testing they found that the best value for N (the length of each substring) was about half the average length. The way that Raku works, writing the Q-gram function was super easy:

  #| Generate space-padded N-grams of length n for string t.
  sub ngrams = -> \t, \n {
    my \s = (' ' x n - 1)  ~ t ~  (' ' x n - 1);
    do for ^(t.chars + n) { s.substr: $_, n }
  }

  #| Calculate Q-gram score using bag operations
  sub qgram (\a, \b) {
    my \q  = (a.chars + b.chars) div 4;
    my \aₙ = ngrams(a,q).BagHash;
    my \bₙ = ngrams(b,q).BagHash;

    (aₙ ∩ bₙ) / (aₙ ∪ bₙ)      # Coefficient de communauté de Jaccard
  }

Raku let the elves calculate N-grams in just two clean lines of code, and then use those to calculate the Jaccard-index between the two strings in just four more easy to read lines of code.

Putting this back into their grammar, they ended up with the following:

grammar Santa'sList {
  rule TOP        {       <kid's-list>+    }
  rule kid's-list {     <name>     <gift>+ }
  rule gift       { '-' <quantity> <item>  }
  token name      { <-[\n]>   }
  token quantity  { <.digit>+ }
  token item      { <fuzzy(@gifts)> }
  token fuzzy     { … }
  sub ngrams      { … }
  sub qgrams      { … }
}

That’s a pretty handy format, but an important problem remains. How do they get access to the best matched text? If they were to match and request, say, $<kid's-list>[0]<gift>[0]<item> they would only get Santa’s original illegible mess. They could do an action but that requires doing a parse with actions, which means the fuzzy token is tied to the vagaries of grammar parsing. Works fine here, but… less reusable.

But elves are good at packing and wrapping. They decide to make a package that wraps the fuzzy token so that both Santa’s original and the corrected version are easily accessible in a DWIM manner. This ‘package’ can’t be declared with package or module, though, because the wrapping process requires using the special sub EXPORT. Their basic process looks like the following:

sub EXPORT {
  # Make the fuzzy token in the elves' factory
  my token fuzzy (*@words) { … } 

  # Wrap it in wrapping paper (apply a role) so it's prettier (easier to use)
  &fuzzy.wrap( … )

  # Ship out (export) the wrapped version
  Map.new( '&fuzzy' => &fuzzy )
}

Any other special tools the elves need can be included in the EXPORT block, for example, the Q- and N-gram functions. So how will they actually do the wrapping? First, they design the paper, that is, a parameterized role that will override the .Str to give the clean/corrected value, but also provide access to a .fuzz function to allow access to older values:

  role Fuzzy[$clear,$fuzz] {
    method Str  { $clear }
    method fuzz { $fuzz  }
  }

Now, the wrapped function could look something like the following:

  &fuzzy.wrap(
    sub (|) {
      my $match = callsame;

      # Failed match evals to false, and is just passed along
      # Successful match gets Fuzzy role mixed in.
      $match
        ?? $match but Fuzzy[$match.??, $match.??]
        !! $match
    }
  );

There’s a small problem. The results of the calculations they ran inside of the token aren’t available. One solution they thought of involved adding new parameters to to the fuzzy token with the trait is raw so that the values could be passed back, but that felt like something the old C++ elves would do. No, Santa’s Raku elves had a better idea: dynamic variables. They made two of them, and refactored the original fuzzy method to assign to them:

  my token fuzzy(**@toys) {
    $<santa's-text>=(<.alpha>+ % \h+)
    <?{
      my $best = @toys
                  .map({ $^toy, qgram($toy,$<santa's-text>.Str)})
                  .sort( *.tail )
                  .tail;
      $*clear = $best[0];
      $*fuzz  = ~$<santa's-text>;
    }>
  }

  &fuzzy.wrap(
    sub (|) {
      my $*fuzz;
      my $*clear;

      my $match = callsame;   # sets $match to result of the original

      $match
        ?? $match but Fuzzy[$*clear, $*fuzz]
        !! $match
    }
  );

They did a test with some values and all went well, until an item wasn’t found:

"I like the Norht Pole" ~~ /I like the $<dir>=<fuzzy: <North South>> Pole/;
say $<dir>.clear;   # --> "North"
say $<dir>.fuzz;    # --> "Norht"

"I like the East Pole" ~~ /I like the $<dir>=<fuzzy: <North South>> Pole/;
say $<dir>.clear;   # --> "North"
say $<dir>.fuzz;    # --> "East"

What happened? The elves realized that their token was matching no matter what. This is because the <?{ 🦋 }> block will only fail if it returns a falsey value. The last statement, being an assignment of a string, will virtually always be truthy. To fix this, they added a simple conditional to the end of the block to fail if the Q-gram score wasn’t sufficiently high.

  my token fuzzy(**@toys) {
    $<santa's-text>=(<.alpha>+ % \h+)
    <?{
      my $best = @toys
                   .map({ $^toy, qgram($toy,$<santa's-text>.Str)})
                   .sort( *.tail )
                   .tail;

      $*clear = $best[0];
      $*fuzz  = ~$<santa's-text>;

      # Arbitrary but effective score cut off.
      $best[1] > 0.33
    }>
  }

With that, they were done, and able to process Santa’s horrible typing.

Of course, there were a lot of improvements that the elves could still make to make their fuzzy token more useful. After they had made use of it (and taken the eggnog away from Santa so they wouldn’t need it), they polished it up so that it could bring joy to everyone.


With that, I can also announce the release of Regex::FuzzyToken. To use it, just do like the elves and in a grammar or any other code, say use Regex::FuzzyToken and the token fuzzy will be imported into your current scope. It has a few extra features, so take a look at its readme for information on some of its options.

While not everyone will use or have need of a fuzzy token, I hope that this shows off some interesting possibilities when creating tokens that might be better defined programmatically, as well as other cool Raku features like Bag operators, dynamic variables, and parameterized roles.

Edit March 2020: A small change was made in the EXPORT sub. It now returns a Map instead of a Hash. Maps do not containerized their values.

Day 14 – Thinking Beyond Types: an Introduction to Rakudo’s MOP

It’s Christmas season! Christmas would not be Christmas without the caroling that’s part of the festivities, so let’s make it possible to sing some.

We could simply make a carol one giant string, but that’s not good enough. Being a song, carols often have a chorus that’s repeated in between verses. If we were to store it as one string, we’d be repeating ourselves. On top of that, people are not perfect; they may forget a verse of a carol, or even just a single line of a verse. We need a type to represent a carol. This could be a type of song, but since we only care about carols, it’s a bit early to abstract this out.

Now, to make this more interesting, let’s handle this without making instances of any type of any kind. All behaviour for all Christmas carols will be handled by type objects. This will be enforced using the Uninstantiable REPR.

At first, we might have a Christmas::Carol role:

role Christmas::Carol is repr<Uninstantiable> {
    proto method name(::?CLASS:U: --> Str:D)        {*}
    proto method verse(::?CLASS:U: Int:D --> Seq:D) {*}
    proto method chorus(::?CLASS:U: --> Seq:D)      {*}
    proto method lyrics(::?CLASS:U: --> Seq:D)      {*}

    method sing(::?CLASS:U: --> ::?CLASS:U) {
        .say for @.lyrics;
        self
    }
}

This would then be done by a class representing a specific carol:

class Christmas::Carol::JingleBells does Christmas::Carol {
    multi method name(::?CLASS:U: --> 'Jingle Bells') { }

    multi method verse(::?CLASS:U: 1 --> Seq:D) {
        lines q:to/VERSE/
        Dashing through the snow
        In a one-horse open sleigh
        O'er the fields we go
        Laughing all the way
        Bells on bobtails ring
        Making spirits bright
        What fun it is to ride and sing
        A sleighing song tonight
        VERSE
    }
    multi method verse(::?CLASS:U: 2 --> Seq:D) {
        lines q:to/VERSE/
        A day or two ago
        I thought I'd take a ride
        And soon, Miss Fanny Bright
        Was seated by my side
        The horse was lean and lank
        Misfortune seemed his lot
        He got into a drifted bank
        And then we got upset
        VERSE
    }
    multi method verse(::?CLASS:U: 3 --> Seq:D) {
        lines q:to/VERSE/
        A day or two ago
        The story I must tell
        I went out on the snow
        And on my back I fell
        A gent was riding by
        In a one-horse open sleigh
        He laughed as there I sprawling lie
        But quickly drove away
        VERSE
    }
    multi method verse(::?CLASS:U: 4 --> Seq:D) {
        lines q:to/VERSE/
        Now the ground is white
        Go it while you're young
        Take the girls tonight
        And sing this sleighing song
        Just get a bobtailed bay
        Two forty as his speed
        Hitch him to an open sleigh
        And crack, you'll take the lead
        VERSE
    }

    multi method chorus(::?CLASS:U: --> Seq:D) {
        lines q:to/CHORUS/
        Jingle bells, jingle bells
        Jingle all the way
        Oh, what fun it is to ride
        In a one-horse open sleigh, hey
        Jingle bells, jingle bells
        Jingle all the way
        Oh, what fun it is to ride
        In a one-horse open sleigh
        CHORUS
    }

    multi method lyrics(::?CLASS:U: --> Seq:D) {
        gather for 1..4 {
            take $_ for @.verse($_);
            take "";
            take $_ for @.chorus;
            take "" if $_ != 4;
        }
    }
}

There’s a problem with this approach, though. What happens if you want to hold onto a collection of Christmas carols to carol around the neighbourhood with?

use Christmas::Carol::JingleBells;
use Christmas;:Carol::JingleBellRock;
use Christmas::Carol::DeckTheHalls;
use Christmas::Carol::SilentNight;
# And so on...

That’s no good! You don’t need to know who wrote a Christmas carol in order to sing it. On top of that, no one thinks of Christmas carols in terms of symbols; they think of them in terms of their name. To represent them effectively, we need to make it so we can look up a Christmas carol using its name, while also making it possible to introduce new carols that can be looked up this way at the same time. How can we do this?

The way we’ll be using here requires a bit of explanation on how types in Raku work.

The Metaobject Protocol

Classes may contain three different types of method declarations. The two you most commonly see are public and private methods:

class Example {
    method !private-method(|) { ... }
    method public-method(|)   { ... }
}

There is a third type of method declaration you can make, which is exclusive to classes (this a lie, but this is the case when writing Raku with Rakudo alone), by prefixing the method’s name with ^. Calls to these are typically made using the .^ dispatch operator, which you often see when you need to introspect an object (.^name, .^methods, etc.). However, these don’t behave at all like you would otherwise expect a method to. Let’s take a look at what the invocant and parameters are when we invoke a method of this type using the .^ dispatch operator:

class Example {
    method ^strange-method(\invocant: |params --> List:D) {
        (invocant, params)
    }
}

say Example.^strange-method;
# OUTPUT:
# (Perl6::Metamodel::ClassHOW+{<anon>}.new \(Example))

Whoa whoa whoa, WTF? Why is the class this type of method is declared in its first parameter instead of its invocant? What even is that object that ended up as its invocant instead, and where is it coming from?

Before this can be explained, first we’ll need to understand a little bit about what the metaobject protocol (MOP) is. The MOP is a feature specific to Rakudo through which the behaviour for all objects that can exist in Raku are implemented. These are implemented based on kinds (types of types), such as classes, roles, and grammars. The behaviour for any type is driven by what is called a higher-order working (HOW), which is a metaobject. These are typically instances of a metaclass of some sort. For instance, HOWs for classes are created by the Metamodel::ClassHOW metaclass.

The HOW for any given object can be introspected. How is this done? How, you ask? How? HOW!? By calling HOW on it, of course!

role Foo { }
say Foo.HOW.^name; # OUTPUT: Metamodel::ParametricRoleGroupHOW

Methods of HOWs are called metamethods, and these are what are used to handle the various behaviours that types can support. Some examples of behaviours of kinds that metamethods handle include type names, attributes, methods, inheritance, parameterization, and typechecking. Since most of these are not features specific to any one kind, these are often mixed into metaclasses by metaroles. For instance, the Metamodel::Naming metarole is what handles naming for any type that can be named.

So that third type of method declaration from earlier? That doesn’t actually declare a method for a class; instead, it declares a metamethod that gets mixed into that class’ HOW, similarly to how metaroles are used. The .^ dispatch operator is just sugar for invoking a metamethod of an object using that object as its first argument, which metamethods accept in most cases. For instance, these two metamethod calls are equivalent:

say Int.^name;         # OUTPUT: Int
say Int.HOW.name(Int); # OUTPUT: Int

Metamethods are the tool we’ll be using to implement Christmas carols purely as types.

Spreading the Joy

To start, instead of having a Christmas::Carol role that gets done by Christmas carol classes, let’s make our carols roles mixed into a Christmas::Carol class instead. Through this class, we will stub the methods a Christmas carol should have, like it was doing as a role, but in addition will hold on to a dictionary of Christmas carols by their name.

We can store carols using an add_carol metamethod:

my constant %CAROLS = %();
method ^add_carol(Christmas::Carol:U, Str:D $name, Mu $carol is raw --> Mu) {
    %CAROLS{$name} := $carol;
}

Now we can mark roles as being carols like so:

role Christmas::Carol::JingleBells { ... }
Christmas::Carol.^add_carol: 'Jingle Bells', Christmas::Carol::JingleBells;

This isn’t a great API for people to use though. A trait could make it so this could be handled from a role’s declaration. Let’s make an is carol trait for this:

multi sub trait_mod:<is>(Mu \T, Str:D :carol($name)!) {
    Christmas::Carol.^add_carol: $name, T
}

Now we can define a role as being a carol like this instead:

role Christmas::Carol::JingleBells is carol('Jingle Bells') { ... }

To make it so we can fetch carols by name, we can simply make our Christmas::Carol class parametric. This can be done by giving it a parameterize metamethod which, given a name, will create a Christmas::Carol mixin using any carol we know about by that name:

method ^parameterize(Christmas::Carol:U $this is raw, Str:D $name --> Christmas::Carol:U) {
    self.mixin: $this, %CAROLS{$name}
}

Now we can retrieve our Christmas carols by parameterizing Christmas::Carol using a carol name. What will the name of the mixin type returned be, though?

say Christmas::Carol['Jingle Bells'].^name;
# OUTPUT: Christmas::Carol+{Christmas::Carol::JingleBells}

That’s a bit ugly. Let’s reset the mixin‘s name during parameterization:

method ^parameterize(Christmas::Carol:U $this is raw, Str:D $name --> Christmas::Carol:U) {
    my Christmas::Carol:U $carol := self.mixin: $this, %CAROLS{$name};
    $carol.^set_name: 'Christmas::Carol[' ~ $name.perl ~ ']';
    $carol
}

This gives our Jingle Bells carol a name of Christmas::Carol["Jingle Bells"] instead. Much better.

Let’s add one last metamethod: carols. This will return a list of names for the carols known by Christmas::Carol:

method ^carols(Christmas::Carol:U --> List:D) {
    %CAROLS.keys.list
}

With that, our Christmas::Carol class is complete:

class Christmas::Carol is repr<Uninstantiable> {
    proto method name(::?CLASS:U: --> Str:D)        {*}
    proto method chorus(::?CLASS:U: --> Seq:D)      {*}
    proto method verse(::?CLASS:U: Int:D --> Seq:D) {*}
    proto method lyrics(::?CLASS:U: --> Seq:D)      {*}

    method sing(::?CLASS:U: --> ::?CLASS:U) {
        .say for @.lyrics;
        self
    }

    my constant %CAROLS = %();
    method ^add_carol(Christmas::Carol:U, Str:D $name, Mu $carol is raw --> Mu) {
        %CAROLS{$name} := $carol;
    }
    method ^carols(Christmas::Carol:U --> List:D) {
        %CAROLS.keys.list
    }
    method ^parameterize(Christmas::Carol:U $this is raw, Str:D $name --> Christmas::Carol:U) {
        my Christmas::Carol:U $carol := self.mixin: $this, %CAROLS{$name};
        $carol.^set_name: 'Christmas::Carol[' ~ $name.perl ~ ']';
        $carol
    }
}

multi sub trait_mod:<is>(Mu \T, Str:D :carol($name)!) {
    Christmas::Carol.^add_carol: $name, T
}

Now, this is great and all, but how is this an improvement on our original code? By defining carols this way, we no longer need to know the symbol for a carol in order to sing it, and we no longer need to know which module even declared the carol in the first place. So long as we know that the Christmas::Carol class exists, we know all of the carols all of the modules we import happen to be aware of.

This means there can be a module defining a collection of carols:

use Christmas::Carol::JingleBells;
use Christmas::Carol::JingleBellRock;
use Christmas::Carol::DeckTheHalls;
use Christmas::Carol::SilentNight;
unit module Christmas::Carol::Collection;

From another module, we can make another collection using this, and define more carols:

use Christmas::Carol::Collection;
use Christmas::Carol::JingleBells::BatmanSmells;
unit module Christmas::Carol::Collection::Plus;

We can then import this and easily sing all of the original module’s carols, in addition to the ones this new module adds, by name:

use Christmas::Carol;
use Christmas::Carol::Collection::Plus;

Christmas::Carol[$_].sing for Christmas::Carol.^carols;

At this point, you may be wondering: “Couldn’t you just write code that does the same thing as this using instances?”. You’d be right! What this shows is that while there is a protocol for working with types when using Rakudo, the behaviour of any given type isn’t particularly unique; it’s driven mainly by metaobjects that you can have complete control over.

Just with metamethod declarations in classes alone, you can augment or override behaviours of any type that supports inheritance. This is far from the extent of what the MOP allows you to do with types! Alas, the more advanced features for working with the MOP would require more explanation, and would be best left for another time.

Day 13 – A Little R&R

A Little R&R

camelialoveferris

Introduction

Raku is a really nice language. Versatile, expressive, fast, dwimmy. The only problem I sometimes have with it is that it can be a little slow. Fortunately that can easily be solved by the NativeCall interface, which makes it easy to call C code in your Raku program. Now, as nice as C is, it is a rather old language with some limitations. A newer language that can fill its niche is known as Rust. I’ll show some examples of having Raku talk to Rust.

FFI

Rust code can be called from other languages using the FFI standard. FFI stands for “Foreign Function Interface” and allows you to export a Rust library as a standard shared library (a .so file on Linux or .dll on Windows). This is done by adding the following section in your Cargo.toml:

[lib]
crate-type = ["cdylib"]

After adding this section you will find the libraries in your target/debug or target/release folder when you build the library with Cargo. Also be sure to add the libc dependency to get access to standard C types.

Primitives

We can use the same primitive types as in C: numbers (and chars) and arrays.

Numbers and chars

Rust:

#[no_mangle]
pub extern fn addition(a: u32, b:32) -> u32 {
        a + b
}

Raku:

use NativeCall;
sub addition(uint32, uint32) returns uint32 is native('foo') { * }

Note the #[no_mangle], this keeps the name of the function unchanged in the final library file. While Rust has standardized name mangling (contrary to C++, where the name mangling is platform-dependent), it is still nice to call a function with it’s original name.

Arrays and strings

Rust:

use std::ffi::CStr;
use std::os::raw::c_char;

#[no_mangle]
pub unsafe extern fn count_chars(s: *const c_char) -> u32 {
        CStr::from_ptr(s).to_str().unwrap().chars().count() as u32
}

#[no_mangle]
pub extern fn lunch() -> *mut c_char {
        let c_string = CString::new("🌮🍚").expect("CString::new failed");
        c_string.into_raw()
}

#[no_mangle]
pub unsafe extern fn free_lunch(ptr: *mut c_char) {
        let _ = CString::from_raw(ptr);
}

Raku:

sub count_chars(Str is encoded('utf8')) returns uint32 is native ('foo') { * }
sub lunch() returns CArray[uint8] is native('foo') { * }
sub free_lunch(CArray[uint8]) is native('foo') { * }

Rust has first class support for UTF-8, making it a great fit for Raku. Using CString also guarantees to add a null byte at the end of the string, so you can get the significant bytes by looping until the AT-POS() value equals 0… if, that is, you choose to return an array rather than populate it.

Structs

Rust:

use std::mem::swap;

#[repr(C)]
pub struct Point {
    x: f32,
    y: f32,
}

impl Point {
        fn print(&self) {
                println!("x: {}, y: {}", self.x, self.y);
        }
}

#[no_mangle]
pub unsafe extern "C" fn flip(p: *mut Point) {
    swap(&mut (*p).x, &mut (*p).y);
        (*p).print();
}

Raku:

class Point is repr('CStruct') {
    has num32 $.x;
    has num32 $.y;
}

sub flip(Pointer[Point]) is native('./librnr.so') { * }

sub flipper {
    my Point $p .= new(x => 3.Num, y => 4.Num);
    say "x: ", $p.x, ", y: ", $p.y;
    flip(nativecast(Pointer[Point], $p));
}

Rust separates objects into structs (which we are all familiar with), and traits, which are kind of like roles in Raku.

Concurrency

Rust:

#[no_mangle]
pub extern "C" fn multithread(count: i32) {
    let threads: Vec<_> = (1..8)
        .map(|id| {
            thread::spawn(move || {
                println!("Starting thread {}", id);
                let mut x = 0;
                for y in 0..count {
                    x += y;
                    println!("Thread {}, {}/{}: {}", id, y, count, x);
                }
            })
        })
        .collect();

    for t in threads {
        t.join().expect("Could not join a thread!");
    }
}

Raku:

sub multithread(int32) is native('./librnr.so') { * }

sub multi-multithread {
    my @numbers = (3_000..50_000).pick(10);

    my @promises;

    for @numbers -> $n {
        push @promises, start {
            multithread($n);
            True;
        };
    }

    await Promise.allof(@promises);
}

Rust and Raku both have first-class concurrency support. This allows you to easily tweak your programs to get the highest possible performance.

Closing statement

These were some examples of interactions between Rust and Raku, two of the most promising languages when looking to the future. If you found this interesting, be sure to check out Andrew Shitov’s A Language a Day articles. Thanks for reading and happy holidays.

Day 12 – Making a simple bot in Raku

Making IRC bots is incredibly simple in Raku, thanks to IRC::Client. It allows you to create a very simple bot in about 20 lines of code. There’s a plugin system that allows easy re-use of code between multiple bots, and adding customized features can be as easy as dropping in an anonymous class.

So, let’s get to it!

Get your dependencies installed

Raku uses zef as the standard module installer, and if you’re reading this, I’m assuming you have it available to you. Install IRC::Client with zef, and you should be good to get started.

zef install IRC::Client

Setting up the bot

To set up the bot, we’ll need to have a nickname to use, a server to connect to and a list of channels to join. To make it easier to run this is a program from your shell, I’ll be using a MAIN sub as well.

use IRC::Client;

sub MAIN () {
  IRC::Client.new(
    nick => 'raku-advent',
    host => 'irc.darenet.org',
    channels => < #advent >,
  ).run;
}

Let’s save this in a file called bot.pl6, and run it.

perl6 bot.pl6

This will run, and if you’re in the channel you specified in channels, you should see the bot joining in a short moment. However, the program itself doesn’t seem to provide any output. It would be highly convenient, especially during development, to show what it’s doing. This is possible by enabling the debug mode. Adding this to the new method call, making it look as follows.

IRC::Client.new(
  nick => 'raku-advent',
  host => 'irc.darenet.org',
  channels => < #advent >,
  debug => True,
).run;

If you restart the application now, you will see there’s a lot of output all of a sudden, showcasing the IRC commands the bot is receiving and sending in response. Now all we need to do is add some functionality.

Making the bot work

As described earlier, functionality of the bot is added in using plugins. These can be any class that implements the right method names. For now, we’ll stick to irc-to-me, which is a convenience method which is triggered whenever the bot is spoken to in a private message, or directly addressed in a channel.

The simplest example to get started with here is to simply have it respond with the message you sent to the bot. Let’s do this by adding an anonymous class as a plugin to the new method call.

IRC::Client.new(
  nick => 'raku-advent',
  host => 'irc.darenet.org',
  channels => < #advent >,
  debug => True,
  plugins => [
    class {
      multi method irc-to-me ($e) {
        $e.text
      }
    }
  ],
).run;

When you restart the bot and talk to it on IRC, you will see it responding to you with the same message you sent it.

 <@tyil> raku-advent: hi
 <raku-advent> tyil, hi
 <@tyil:> raku-advent: how are you doing
 <raku-advent> tyil, how are you doing

Adding some real features

So, you’ve seen how easy it is to get started with a simple IRC bot in just over a dozen lines. Let’s add two features that you may want your bot to support.

For convenience sake, I will only cover the class implementing the features, not the entire IRC::Client.new block.

Uptime

First off, let’s make the bot able to show the time its been running for. For this, I’ll make it respond to people asking it for “uptime”. We can use the irc-to-me convenience method for this again. After all, we probably don’t want it to respond every time someone discusses uptime, only when the bot is asked directly about it.

In Raku, there’s a special variable called $*INIT-INSTANT, which contains an Instant of the moment the program started. We can use this to easily get the Duration that the program has been running for.

class {
  multi method irc-to-me ($ where *.text eq 'uptime') {
    my $response = "I've been alive for";
    my ($seconds, $minutes, $hours, $days, $weeks) =
      (now - $*INIT-INSTANT).polymod(60, 60, 24, 7);

    $response ~= " $weeks weeks" if $weeks;
    $response ~= " $days days" if $days;
    $response ~= " $hours hours" if $hours;
    $response ~= " $minutes minutes" if $minutes;
    $response ~= " $seconds seconds" if $seconds;

    $response ~ '.';
  }
}

Now, whenever you ask the bot for uptime, it will respond with a human friendly uptime notification.

 <@tyil> uptime
 <@tyil> raku-advent: uptime
 <raku-advent> tyil, I've been alive for 5 minutes 8 seconds.

User points

Most channels have a bot that keeps track of user points, or karma as it’s sometimes referred to. There’s a module already that does this for us, called IRC::Client::Plugin::UserPoints. We don’t have to do much apart from installing it and adding it to the list of plugins.

zef install IRC::Client::Plugin::UserPoints

Once this finishes, the module can be used in your code. You will need to import it with a use statement, which you can put directly under the use IRC::Client line.

use IRC::Client;
use IRC::Client::Plugin::UserPoints;

Now, in the list of plugins, add it as a new entry.

plugins => [
  IRC::Client::Plugin::UserPoints.new,
  class {
    ...
  },
],

This plugin makes the bot respond to !scores, !sum and whenever a nick is
given points using a ++ suffix, for instance, t​yil++.

 <@tyil> raku++
 <@tyil> raku++
 <@tyil> !scores
 <raku-advent> tyil, « raku » points: main: 2

Finding plugins

All plugins for IRC::Client that are shared on the community have the prefix IRC::Client::Plugin::, so you can search for that on modules.perl6.org to find plugins to use. Of course, you can easily add your own plugins to the ecosystem as well!

Winding down

As you can see, with some very simple code you can add some fun or important
tools to your IRC community using the Raku programming language. Try it out and
have some fun, and share your ideas with others!

Day 11 – Packaging with Libarchive

Distributing physical gifts involves wrapping them up into packages, but suppose you want to distribute digital gifts. How can you use Raku to help you wrap them up? Enter Libarchive!

Simple wrapping files into a package

Let’s wrap up just two files, myfile1 and myfile2 into a single package.zip file. (Libarchive just as easily creates tar files, cpio, rar, even iso9660 images for cds or dvds.)

use Libarchive::Simple;

given archive-write('package.zip') {
    .add: 'myfile1', 'myfile2';
    .close;
}

This very simple syntax looks a little weird for those unfamiliar… here is a more ‘traditional’ way of writing the same thing:

use Libarchive::Write;

my $handle = Libarchive::Write.new('package.zip');
$handle.add('myfile1', 'myfile2');
$handle.close;

What is the difference? Libarchive::Simple provides a few shorthand routines for accessing the various Libarchive functionalities. One of these is archive-write() which is identical to Libarchive::Write.new().

The second example takes the return from new() and stores it in the variable $handle. Then we call two methods on that variable to add the files, and close the file.

The given statement makes this even simpler by topicalizing that variable, that is, storing it in the topic variable $_. Since $_ can be used as the default object for method calls, we don’t need to explicitly refer to it when calling methods.

.add('myfile1') is equivalent to $_.add('myfile1')

But what happened to the parentheses? Another little shorthand when calling methods — rather than surrounding your arguments to a method with parentheses, you can just precede them with a colon:

.add: 'myfile1';

Nice! I love programming with Raku!

Package a bunch of files by smartmatching

A handy routine to help in your packaging is dir(). It will return a lazy list of IO::Path objects for a directory. By happy coincidence, Libarchive add can take IO::Path just as easily as a filename.

given archive-write('package.zip') {
    .add: 'mydir', dir('mydir');
    .close;
}

Note we’ve added the directory itself first, then used dir() to get a list of the files inside mydir, which also get added. If you don’t include the directory itself, it won’t be part of the package. That works fine most of the time, depending on your format and your unpackaging program, but it is good practice to include the directory to make sure it gets created the way you want it to.

dir has an extra feature — it can filter the directory by smartmatching the string with a :test argument. Lets include only jpeg files, allowing them to end in either .jpg or .jpeg:

given archive-write('package.zip') {
    .add: 'mydir', dir('mydir', test => /:i '.' jpe?g $/);
    .close;
}

Ecosystem modules like File::Find or Concurrent::File::Find can easily generate even more complicated lists of files for including by recursively adding an entire hierarchy to the package.

Create your files on the fly while packaging

You aren’t limited to adding existing files. You can use the write() method to generate a file for the package on the fly. You can specify content as a Str, a Blob, or even an IO::Handle or IO::Path to get the content from.

given archive-write('package.zip') {
    .write: 'myfile', q:to/EOF/;
        Myfile
        ------
        A special file for a special friend!
        EOF
    .close;
}

Here we’re using a special Raku quoting construct called the heredoc.

The q:to/EOF/ says to use the lines following up until the EOF marker and make them into the content of a file named ‘myfile’ included in the package file. As a friendly benefit, the amount of indentation of the terminator is automatically removed from each line to the quoted lines. How convenient!

Stream your package instead of making files

Making files with your packages is great and all, but I’ve got a web site I want to return my custom CD images from — why bother with a temporary file? Just stream the output on the fly!

For this example, we’re going to stream the package as an iso9660 file (the image used for CD writers) to STDOUT, but you can stream to other programs too.

given archive-write($*OUT, format => 'iso9660') {
    .add: 'myfile1', 'myfile2', 'mydir', dir('mydir');
    .close;
}

Usually the format can be inferred from the suffix on a specified filename, but since we are streaming there is no filename, so the format must be specified. $*OUT is a special filehandle that is automatically opened for you for writing to STDOUT.

Burn that image to a CD and mount it and you’ll see the specified files. So easy!

Libarchive has so many cool features it would take days to go over them all, but I hope this little intro has whet your appetite for packaging things up. Raku has fantastic syntax, features and expressivity that make it so easy to interface with libraries like this.

Have fun packaging your own things, be they physical or digital!

Day 10 – A Teaser

Santa has a special treat: a teaser if you will. A part of a chapter from the upcoming book “Migrating Perl to Raku”, to be published January 2020.


Optimization Considerations

If you are an experienced Perl programmer, you have (perhaps inadvertently) learned a few tricks to make execution of your Perl program faster. Some of these idioms work counter-productively in Raku. This chapter deals with some of them and provides the alternative idioms in Raku.

Blessed Hashes Vs. Objects

Objects in Perl generally consist of blessed hashes. As we’ve seen before, this implies that accessors need to be made for them. Which means an extra level of indirection and overhead. So many Perl programmers “know” that the object is basically a hash, and access keys in the blessed hash directly.

Consequently, many Perl programmers decide to forget about creating objects altogether and just use hashes. Which is functionally ok if you’re just using the hash as a store for keys and associated values. But in Raku it is better to actually use objects for that from a performance point of view. Take this example where a hash is created with two keys / values:

    # Raku

    for ^1000000 {    # do this a million times

        my %h = a => 42, b => 666;

    }

    say now - INIT now;  # 1.4727555

Now, if we use an object with two attributes, this is more 4x as fast:

    # Raku

    class A {

        has $.a;

        has $.b;

    }


    for ^1000000 {

        my $obj = A.new(a => 42, b => 666);

    }

    say now - INIT now;  # 0.3511395

But, you might argue, accessing the keys in the hash will be faster than calling an accessor to fetch the values?

Nope. Using accessors in Raku is faster as well. Compare this code:

    # Raku

    my %h = a => 42, b => 666;


    for ^10000000 {    # do this ten million times

        my $a = %h<a>;

    }

    say now - INIT now;  # 0.4713363


To:

    # Raku

    class A {

        has $.a;

        has $.b;

    }

    my $obj = A.new(a => 42, b => 666);


    for ^10000000 {

        my $a = $obj.a;
    
}

    say now - INIT now;  # 0.36870995

Note that using accessors is also faster, albeit not much, but still significantly so.

So why is the accessor method faster in Raku? Well, really because Raku is able to optimise to attributes in an object to a list, easily indexed by a number internally. Whereas for a hash lookup, the string must be hashed before it can be looked up. And that takes a lot more work than just a lookup by index.

Of course, as with all benchmarks, this is just a snapshot in time. Optimisation work continues to be performed on Raku, which may change the outcome of these tests in the future. So, always test yourself if you want to be sure that some optimisation is a valid approach, but only if you’re really interested in squeezing performance out of your Raku code. Remember, premature optimisation is the root of all evil!


Santa hopes you liked it.

Day 9: a chain (or Russian doll) of containers

A lot of electronic in a couple of containers

If you’re in the business, you’ve probably by now heard about containers. They can be described as executables on steroids, or also, as its namesake, a great way of shipping applications anywhere, or have then stored and ready to use whenever you need them. These kinda-executables are called images, and you can find them in a number of places called registries, starting with Docker Hub, joined lately by the GitHub Container Registry and other places like Quay.io or RedHat container catalog. These last are up and coming, you and have to add them to your default configuration. Most containers are registered in Docker Hub anyway.

Also, since they are kinda-executables, they are architecture and operating system specific. Docker hub marks their architecture and operating system, with Linux being the most common ones. You can, however, run Linux images everywhere, as long as the Docker daemon is running in a Linux virtual machine; that is also the default configuration in Macs.

Of course, there’s a slew of containers you can use with Raku, even if we don’t really have one we can call official. I’m going to go with my own, since, well, I’m more familiar with them. But there’re these nightly images by tyil, for instance, or the kinda-official Rakudo Star images, which have not been updated since, well, Rakudo Star itself was updated last March.

The Alps as seen from a plane

Let’s start with the basic image, the tiny Russian Doll with Nicky the tsar. Since it’s going to be inside, we need to make it real tiny. Here it is, jjmerelo/alpine-perl6:

FROM alpine:latest
LABEL version="2.2" maintainer="JJMerelo@GMail.com" perl6version="2019.11"

# Environment
ENV PATH="/root/.rakudobrew/versions/moar-2019.11/install/bin:/root/.rakudobrew/versions/moar-2019.11/install/share/perl6/site/bin:/root/.rakudobrew/bin:${PATH}" \
    PKGS="curl git perl" \
    PKGS_TMP="curl-dev linux-headers make gcc musl-dev wget" \
    ENV="/root/.profile" \
    VER="2019.11"

# Basic setup, programs and init
RUN mkdir /home/raku \
    apk update && apk upgrade \
    && apk add --no-cache $PKGS $PKGS_TMP \
    && git clone https://github.com/tadzik/rakudobrew ~/.rakudobrew \
    && echo 'eval "$(~/.rakudobrew/bin/rakudobrew init Sh)"' >> ~/.profile \
    && eval "$(~/.rakudobrew/bin/rakudobrew init Sh)"\
    && rakudobrew build moar $VER \
    && rakudobrew global moar-$VER \
    && rakudobrew build-zef\
    && zef install Linenoise App::Prove6\
    && apk del $PKGS_TMP \
    && RAKUDO_VERSION=`sed "s/\n//" /root/.rakudobrew/CURRENT` \
       rm -rf /root/.rakudobrew/${RAKUDO_VERSION}/src /root/zef \
       /root/.rakudobrew/git_reference

# Runtime
WORKDIR /home/raku
ENTRYPOINT ["raku"]

This image was created just last week, after the release of Raku 2019.11, the first one to actually be called Raku and the one that calls its executable raku too.

First thing you see is that FROM which declares the teeny container that’s inside this one. We’re using Alpine Linux, a distribution little known outside the containerer community, that uses a couple of tricks to avoid bloating the number of files, and thus the size, of the container. This image will add up to less than 300 MBs, while an equivalent image with Debian or Ubuntu will be twice as much. That means that downloading it will take half as much, which is what we’re looking for.

Because there’s this thing, too: real-life containers, when empty, can be Russian-dolled and put inside one another so that they don’t occupy so much space. Something similar happens to containers. They are built putting layers on top of each other, the inner layer usually an operating system. Let’s check out the rest.

The next LABELs are simply tags or metadata that can be extracted from the image by inspection. Not really that important.

But the ENV block kinda is, over all the first one, which defines the PATH that is going to be used across the Russian doll buildup. The rest of the variables are mainly used while building the image. They will also help to make it somewhat generic, so that we can just change the value of a variable and get a new version; we put that into VER.

So far, no building has taken place, but in this humongous RUN statement is where we download rakudobrew, put it to work building the version contained in VER, set that version as the default one, install zef and a couple of modules we are going to need, and then delete what we will no longer be needing in the rest of the outer dolls to keep the whole thing small.

Finally, after setting up a working directory, we define an entry point, which is the real executable-within-the-executable. The container can be used in place of this command, so that anything that can be done with raku, can be done with this this executable. For instance, let’s run this program:

my @arr;
my ($a, $b) = (1,1);
for ^5 {
    ($a,$b) = ($b, $a+$b);
    @arr.push: ($a.item, $b.item);
    say @arr
};
say @arr;

We will give our containerized Raku an alias:

alias raku-do='docker run --rm -t -v `pwd`:/home/raku  jjmerelo/alpine-perl6'

We can run the program above with:

raku-do itemizer-with-container.p6

But you can take it a step further. Create this shell script and put it in the path:

#!/bin/bash

docker run --rm -t -v `pwd`:/home/raku  jjmerelo/alpine-perl6 $@

You can then use this in the shebang line: !/usr/bin/env raku-do.sh. This will create a throwaway image, that will be ephemerally created to run the script, and then thrown away (that’s the --rm in the line). The current directory (pwd) will be aliased to /home/raku, remember, our working directory, which means that the raku inside the container will see it right there. You see? With this you can have raku run wherever docker is installed. Pretty much everywhere, nowadays.

But let’s build up on this. Containers are extensively used for testing, since instead of building and installing, you can put everything in a single container, and download and use it for testing straight away. That’s is actually what made a containerer out of me, the long 20 minutes it took to brew rakudo for a few seconds of testing for every module. After that base container, I created this one, jjmerelo/test-perl6. Here it is:

FROM jjmerelo/alpine-perl6:latest
LABEL version="4.0.2" maintainer="JJ Merelo <jjmerelo@GMail.com>"

# Set up dirs
RUN mkdir /test
VOLUME /test
WORKDIR /test


# Will run this
ENTRYPOINT perl6 -v && zef install --deps-only . && zef test .

This is actually simplicity itself: the only thing that changes is the entrypoint and the working dir. Instead of running directly the raku compiler, it does a couple of things: install dependencies needed to run the tests, and then issue zef test . to run the tests.

That really speeds things up when testing. Put it in your .travis.yml file this way:

language: minimal

services:
  - docker

install: docker pull jjmerelo/test-perl6

script: docker run -t -v  $TRAVIS_BUILD_DIR:/test jjmerelo/test-perl6

And you’re good to go. Takes all of a minute and a half, as opposed to more than 20 minutes if you use the official Travis image, which is based in rakudobrew.

The Russian doll does not stop there: jjmerelo/perl6-test-openssl includes additional Alpine packages which are needed to install OpenSSL. And, based on that one, jjmerelo/perl6-doccer, which packs everything that’s needed to test the Raku documentation.

You should really try this yourself. If you’ve got even just a few additional modules to download when testing your module, just build up from the test-perl6 image and get your own! You’ll save time, and also save computing time, thus saving energy.

Actual Russian, or maybe Latvian, dolls, bought in Riga during PerlCon