Day 8 – Make it Snow 2.0: The Snowfall Strikes Back

Introduction

Seven years ago I wrote a blog post on the previous incarnation of this advent calendar that demonstrated a new library I had written, called Terminal::Print, by making a (very primitive) snowfall simulator.

However, I was never entirely pleased with the outcome, especially after I saw this video about an implementation in APL that incorporated variable flake sizes, speeds, and turbulence.

Let’s address this by creating a brand-new snowfall simulator using a new Raku library, Dan Vu’s excellent Raylib::Bindings. These are generated bindings for the superb single-header game development library raylib.

Let’s snow!

Every raylib program works by first initializing it’s window:init-window(screen-width, screen-height, "Snowfall Simulator").

The init-window function has the side effect of setting up the OpenGL context. This is important, because without this context we aren’t able to load textures into GPU’s VRAM. Unlike many frameworks, init-window does not return a window object.

Font time

We want to draw the * character on tthe screen to represent our snowflake. There should be some way to use the UTF-8 characters from .TTF files, but I could not get it working when trying to use .

raylib does not draw fonts (or images) directly but rather supports loading fonts into memory and then creating images from them and then loading those images into textures.

my $image = image-text('*', 32, $white);
my $texture = load-texture-from-image($image);
unload-image($image);

Note the call to unload-image. We are going to need to be very careful about how most of our memory is managed when using raylib. We call unload-image($image) immediately after generating the texture because we have no further use for the image.

while forever (until not)

After init-window, the most important piece of a raylib program is its while loop. This is the loop that does all updating and rendering of the program’s state.

We also can’t forget to call set-target-fps, otherwise our FPS is entirely determined (rather than simply constrained) by our processing speed.

Let’s look at a fully working script that draws a single * in the center of the screen:

use v6.*;
use Raylib::Bindings;

constant screen-height = 800;
constant screen-width  = 800;

constant black = init-black;
constant white = init-raywhite;

init-window(screen-width,screen-height,"Snowfall Simulator");

my $image = image-text('*', 32, white);
my $texture = load-texture-from-image($image);
unload-image($image);

set-target-fps(60);

while !window-should-close {
    begin-drawing;
    clear-background(black);
    draw-texture-ex(
      $texture,
      Vector2.init(screen-width/2e0, screen-height/2e0),
      0e0,
      1e0,
      white
    );
    end-drawing;
}

unload-texture($texture);
close-window;

Try adjusting the fourth parameter of draw-texture-ex to higher values: this controls the scale of the texture. We will be using this to have different sized snowflakes.

Design Strategy

Before we dive into coding the actual snowfall, let’s first consider what we actually want from the program. Then we can design around those features.

  1. Snowflakes of various sizes falling down the screen.
    1. These snowflakes should appear natural as they enter the top of the screen — no easily discernable regularity or synchronization should be seen.
  2. A wind to blow the flakes.
  3. An accumulation of snowflakes on the bottom of the screen.

Looking at requirement one, snowflakes of various sizes, we could individualize the size of each snowflake. However, considering the sub-requirement of appearing naturally without apparent synchronization, it probably makes sense to do our work in layers. If we are going to be using layers anyway, we might as well attach the scale to the layer and save space on our eventual Snowflake object.

What should we base our layers on, in the end? In Raku it’s a natural fit to use interval supplies at various frequencies to trigger the creation of new snow as well as to trigger updates to the layer’s snowflakes. So, each layer will have both an interval supply and an update supply.

We want don’t want the snowflakes to fall uniformly, so one thing we can do is assign a random weight to each snowflake. We can then use this when we calculate position to make a heavier snowflake fall faster.

Wind is extremely simple to integrate, if a bit tricky to tune. We’ll have this as an attribute of the layer as well.

For accummulation we will create an array that is the length of the total number of pixels. The array will be one-dimensional for performance. We will encapsulate this behavior in a class. But! There is an important caveat: We want to update this accumulator array from multiple threads, so we will make it a monitor class using OO::Monitors.

Now our proposed classes look something like the following:

The final code

Rather than break this down step by step, for the sake of time I will instead present you with the full code of a snowfall simulator. Then I will explain my process for how I arrived there

use Raylib::Bindings;
use NativeCall;
use OO::Monitors;

## Initialization

constant screen-width = 1024;
constant screen-height = 768;

init-window(screen-width, screen-height, "Snowfall Simulator");

my int32 $zero = 0;
my int32 $size = 48;

my $white = init-white;
my $black = init-black;

## Implementation Classes
class Snowflake {
    my $TEXTURE; # Only one needed

    has Bool $.falling is rw = True;
    has Vector2 $.pos is required is rw;
    has Num() $.weight is required;

    method initialize-texture {
        my $image = image-text('*', 32, $white);
        $TEXTURE = load-texture-from-image($image);
        unload-image($image);
    }
    method unload-texture { unload-texture($TEXTURE) }
    method texture { $TEXTURE }
}

constant BOUNDS-SCALE = 4e0;

monitor SnowfallAccumulator {
    has @.pixels = 0 xx (screen-width * screen-height);

    method fill-pixel($x, $y, $scale) {
        my $range = (BOUNDS-SCALE * $scale).Int;
        for (-$range)..$range -> $x-offset {
            for (-$range)..$range -> $y-offset {
                next unless ($y + $y-offset) > 0  && ($x + $x-offset) > 0;
                @!pixels[(($y + $y-offset) * (screen-width)) + ($x + $x-offset)] ||= 1;
            }
        }
    }

    method check-pixel($x, $y) {
        @!pixels[($y * screen-width) + $x]
    }
}

my SnowfallAccumulator $accumulator .= new;

class SnowfallLayer {
    has Snowflake @.snowflakes;
    has Num $.scale is required;

    has Int $.layer-number is required;

    has Vector2 $.wind is rw = Vector2.init(1.001e0 + (2.5e0 * $!scale) * ((1..3).pick %% 2 ?? -1 !! 1), 1.02e0);

    has Bool $.started = False;
    has Supply $!update-supply = Supply.interval((($!scale + 0.5) / 30 + ($!layer-number / 1000)));
    has Supply $!more-supply = Supply.interval($!scale, $!scale + ($!layer-number / 50));

    submethod TWEAK() {
        self!fill-snowflakes;
        $!update-supply.act: {
            $!started && @!snowflakes.grep(*.falling).map: {
                $_.pos.y = $_.pos.y + (($_.weight + $!scale) * 10) + $!wind.y;
                $_.pos.x = $_.pos.x + (1 + ($!wind.x + (0.05 * (1..2).roll)));
                if $_.pos.x > screen-width {
                    $_.pos.x = $_.pos.x - screen-width;
                }
                if $_.pos.x < 0 {
                    $_.pos.x = $_.pos.x + screen-width;
                }
                if $_.pos.y >= screen-height || $accumulator.check-pixel($_.pos.x, $_.pos.y) {
                    my $size = BOUNDS-SCALE * $!scale;
                    $_.pos.y = screen-height - $size;
                    until not $accumulator.check-pixel($_.pos.x, $_.pos.y) {
                        $_.pos.y = $_.pos.y - $size;
                    }
                    $_.pos.x = $_.pos.x + (Bool.pick ?? -$size !! Bool.pick ?? 0 !! $size);
                    $accumulator.fill-pixel($_.pos.x, $_.pos.y, $!scale);
                    $_.falling = False;
                }
            };
        }
        $!more-supply.act: { $!started && self!fill-snowflakes };
    }

    method !fill-snowflakes() {
        for (^screen-width).pick(((1 .. 8).roll).abs) -> $x {
            @!snowflakes.push: Snowflake.new: :pos(Vector2.init($x.Num, (^16).roll.Num)), :weight((1..5).roll * $!scale);
        }
    }

    method render {
        for @!snowflakes.grep(Snowflake:D) -> $flake {
            draw-texture-ex($flake.texture, $flake.pos, 0e0, $!scale, $white);
        }
    }

    method start { $!started = True }
}

class SnowfallRenderer {
    has @.layers;

    submethod TWEAK {
        constant scales = (0.1e0, 0.2e0, 0.25e0, 0.3e0, 0.33e0, 0.36e0, 0.4e0, 0.5e0);
        @!layers = scales.kv.map: -> $layer-number, $scale { SnowfallLayer.new: :$scale, :$layer-number }
    }

    method start  { @!layers.map: *.start }
    method render { @!layers.map: *.render }
}

## Rendering
set-target-fps(60);
Snowflake.initialize-texture;
my $renderer = SnowfallRenderer.new;
$renderer.start;
while !window-should-close {
    begin-drawing;
    clear-background($black);
    $renderer.render;
    end-drawing;
}

Snowflake.unload-texture;
close-window;

Process notes

I implemented the Snowflake class and then immediately began working on SnowfallLayer . Once I had the $layer.render producing falling snow, I created the SnowfallRenderer class to own and render multiple layers.

Interval supplies are created to update the positions of the snowflakes. Note that this is done outside of the context of the main loop — while we can afford to visit every flake and draw it, we don’t want to pay the math and assignment costs while rendering our frames.

We create snowflake layers of different scales to fake a sense of depth in the field of view. Each layer has it’s own unique scale as well as its own wind. We make this wind dependent on scale so that the far away snow seems to be more affected by the wind than the closer up snow.

Once the snow hits an area that the accumulator has marked occupied, we rewind it by a fraction of the total size of the character and then mark it as no longer falling. Since we only do our movement processing on falling snowflakes, these fallen flakes will stay in their final positions.

Issues

There is occasional slow-down after enough snow has been accumulated. This is because iterating over the list of extant snowflakes for each layer is taking an increasing amount of time. I’ve tried several strategies to mitigate this but I have a feeling that I will have to revisit my entire design.

De-structuring the Snowflake objects into individual arrays of weights, positions, and falling states would be one possible avenue to explore. I might also try creating the objects in an entirely different language. (Sound strange? More on that to come in a future post…)

The snow

In the end we have this result:

Video of the Snowfall Simulator described in this article

I’m quite pleased with the way it looks. It’s leaps and bounds ahead of what we did with Terminal::Print. We are missing Unicode, which would be nicer, but we need to leave a few items to address for the eventual Make It Snow 3.0.

Until then, stay cozy!

Day 7 – The Magic Of $/

Santa was dabbling a bit with regexes in the Raku Programming Language, just to satisfy their curiosity. Trying to find out how many children have a first name that starts with a vowel. That seemed like a nice little project!

Since the Big Database Server at the North Pole was already heating up a lot, and the amount of ice on the North Pole was already dwindling at alarming rates, Santa decided to not use the database at all for this dabbling. Instead Santa decided to use the “words” file that you can find on Unix systems in /usr/share/dict/words, and copied it to “kiddos”.

Santa had already learned about Bags earlier in the year, and had also tried out several features of the regex syntax of Raku. And so came up with the following program:

my %b is Bag = "kiddos".IO.lines.map: {
    ~$0 if /^ (a | e | i | o | u) /;
}
for %b.sort(-*.value) {
    state $total;
    FIRST say "Names that start with vowels:";
    FIRST say "-----------------------------";
    printf "%1s: %5d\n", .key, .value;
    $total += .value;
    LAST printf "-------- +\n%8d\n", $total;
}

Which outputs:

Names that start with vowels:
-----------------------------
u: 16179
a: 14537
i:  8303
e:  7818
o:  7219
-------- +
   54056

Santa decided that the FIRST and LAST phasers were a really nice way to keep the summarizing code in a single scope. The fact that you could use a state variable for that as well, even made things even nicer.

But the little program took longer to execute than Santa liked. Even though Santa’s computer was not the most modern one, it still had 4 CPUs, and only one was used in this little program. Somewhere Santa had read about being able to execute a map in parallel. “Hmmmm” Santa mumbled.

Ah yes, race is what they where looking for! Just adding 5 letters should make all the difference!

#                               👇
my %b is Bag = "kiddos".IO.lines.race.map: {
    ~$0 if /^ (a | e | i | o | u) /;
}

But, but, that didn’t work at all? Every time the program would crash with strange, totally unexpected errors all starting with:

A worker in a parallel iteration (hyper or race) initiated here:
  in block ………

Santa decided to call in Lizzybel to tell them that they had found a bug in Raku. Santa actually felt quite proud about that!

Lizzybel was quite taken aback by the weirdness of the errors as well. Why would that not work? They started digging and after a few hours, realized that there was at least a very simple workaround:

my %b is Bag = "kiddos".IO.lines.race.map: {
    my $/;  # give this scope its own $/
    ~$0 if /^ (a | e | i | o | u) /;
}

Santa was really happy that such a little change would fix the issue and allow the program to run as fast as it could on Santa’s computer. But Santa also wanted to know why this was necessary! So Lizzybel explained:

As you know, matching will set the $/ variable. And $0 is just short for $/[0]. However, you almost never need to define a $/ yourself, because there’s one defined in every submethod or mainline automatically for you. So your code is really:

my $/;
my %b is Bag = "kiddos".IO.lines.race.map: {
    ~$0 if /^ (a | e | i | o | u) /;
}

But inside any other block { … }, Lizzybel rattled on, no $/ will be defined for you. This allows this common code structure to work:

"foo bar" ~~ / (\w+) /;
if $0 {
    say $0;  # 「foo」
}

Lizzybel continued: So the $/ implicitly used with $0 inside the if, refers to the $/ that is automatically defined in the current or outer lexical scope (in this case, the mainline of the program). If there would have been a separate $/inside the block of the if statement, it would mask the outer $/, and thus show an undefined value for $0, like this:

"foo bar" ~~ / (\w+) /;
if $0 {
    my $/;
    say $0;  # (Any)
}

Again, Lizzybel continued: In Raku it was decided that $/ would always refer to a (usually automatically) defined $/ in any of the lexically outer sub-like scopes if there is none in the current scope. This is not an issue with any single-threaded program. But becomes an issue when multiple threads are accessing the same $/ simultaneously. And that causes the strange errors that you see.

But, couldn’t this be handled automatically by Raku? Why would I have to add a my $/ there? This feels like a bit of a trap!, Santa interjected.

This can not be fixed generally, or very easily, said Lizzybel, and continued: That’s because the .race method is just like any other method in Raku, there is nothing special about it. It produces a RaceSeq object which happens to have a .map method that takes a block as a parameter. What that block actually contains, is completely opaque to the executor. And the result of the .map just happens to have an .iterator method that serializes the results that are calculated in parallel.

Hmmmm, said Santa, now almost grumbling.

Not deterred by Santa, Lizzybel continued again: And since the solution is pretty simple, it was decided that this should really be documented as a trap. At least until there’s a smarter way to handle this situation.

I guess I’m a bit of a sorcerer’s apprentice when it comes to multi-threading, Santa mumbled under their breath. Lizzybel heard it nonetheless and responded: In a way we all are, Santa, as I didn’t immediately grok the issue either.

After this reassurance Santa happily continued with dabbling in Raku, never too old to learn! And all was well.

Day 6 – The Future Of POD6

by Kay Rhodes

Some people believe that “code should be self documenting”. They believe that we don’t need inline docs because you can just look at the code and see what it’s doing. The reality is that everyone’s brain works differently. Everyone processes new information differently.

My brain’s extra divergent. I have a working memory that approaches that of your average goldfish. I’m rediscovering the little plastic castle, every time around the bowl. As a result, I’ve spent a lot of time thinking about how to convey knowledge to future me. Inline documentation of our code is one of those ways.

When we write internal docs there are three main aspects to it: what we choose to write, how we organize it, and how we encode it.

Tools Matter

Here in Rakuland we use Pod6 for our inline docs. I don’t think many people really think about it. Folks seem to use it because it’s there, and because it’s what the people before them have used. “That’s what we’ve always done” doesn’t mean the thing you’ve always done is the best it could be. It doesn’t mean that it’s what you should be doing. It might be, but that’s not a given.

To be explicit, I have a lot of respect for Pod6, its predecessor Pod, and the folks that worked on both of them. If it wasn’t for the amazing Pod docs written by the developers who came before me, I might not have the career I do. But, Pod(6) has never made me happy. In my travels beyond the shores of Perl and Raku I’ve learned about a number of other ways of documenting code, and what those things can do for us.

I’m hoping that by the end of this post, some of you will be inspired to dig into Pod6 and make it do more for us. I’m also hoping that some of you will be inspired to write an alternative.

What we choose to write

Over in Ruby land we have YARD. YARD isn’t the greatest code documentation tool, but there’s a lot it does well. It’s also the one I’m most familiar with.

YARD, like Pod6 never explicitly says “you should write this stuff here”. Unlike Pod6, it implicitly sets expectations about what should be written. It does this in two ways. First it provides a simple mechanism for encoding different types of information whose usage is self-evident. Second, it provides examples that use those mechanisms to actually document code.

If we provide examples of best practices for documenting code we set expectations. We also provide a helpful references for new users who haven’t memorized the syntax yet. If we do that when we introduce the documentation tool people have a pattern to follow from the start. Pod6’s docs don’t really do that. While there are some examples of Pod6 next to code none of them actually show how to document that code. Instead, they show how a given mechanism works.

If we look at YARD’s features page the first code example we see is this:

# Reverses the contents of a String or IO object.
#
# @param [String, #read] contents the contents to reverse
# @return [String] the contents reversed lexically
def reverse(contents)
  contents = contents.read if respond_to? :read
  contents.reverse
end

This doesn’t begin to cover the full capabilities of YARD. What it does do is set early expectations, both of what it’s like to use YARD, and how they expect you to use it. This example implicitly conveys what YARD’s creators expect its users to include:

  • a short description of what the function does
  • a list of its parameters, their types and some idea of what those params are expected to contain.
  • the same information about what the method returns.

I’ve written thousands of lines of Raku at this point, and I have no idea what is expected when documenting my methods. With no clear guidance, we each go off and do things our own way. Even if we manage to convince people that inline docs are worthwhile, there’s nothing guiding what they put in them. As a result, there’s no consistency, and we can’t have any expectation about what kind of support those docs will provide to the people who need them.

Flexibility is great, but guidance is great too. I honestly believe that one of the single best things we could do, to help people considering, and learning Raku is to be consistent about what we document about our code. We need a standard. It doesn’t have to be an “official” standard approved by the core team. But it needs to be something that every developer has a passing familiarity with. We need to provide clear guidance for people learning Pod6, but we also need to follow that guidance when we write our code.

A good standard for documenting code is something can be understood without ever reading the docs for the tool. If you’re a new Ruby dev, and you see a handful of examples like the one above, you’ll know what’s expected of you when you add functions. You’ll document what they take in, what they put out, and give them a short description.

So, what does Raku consider best practices for documenting our code, and what can we do to help make sure that every developer understands them?

How we organize our docs

How do we organize our docs? Do we document a class at the start of the file, or the end? When do choose to document the inputs and outputs of a function, do those come before or after the description? What about example usage? Where does that go? When I document a function, should I do that directly above the function, or should I put all the documentation at the end in one big section?

I don’t know.

Raku seems to be following in Perl’s footsteps; “There’s more than one way to do it.” I have some opinions about where things should go, and how we should order them. You do to. You write your code according to yours. I write my code according to mine.

This is easy for us, but it’s terrible for the people we want to use our code. First, they can’t have any expectations about what will be documented. Now, they can’t have any expectations about where they’ll find it.

A consistent organizational structure enables readers to rapidly find the information they need. I may not even realize you have docs, if I’m expecting them inline, but you’ve put them all at the bottom. When we don’t organize the things we do write consistently, readers have to spend time reading all of it to find what they need.

YARD’s examples make it clear that in Ruby we organize a method’s docs by first providing the description, then the inputs, then the outputs. But, what about usage examples? Where do they go? Honestly, I don’t know. I think shouldgo after documenting the return value. I’ve seen unofficial docs that show examples after the parameters and beforethe return. That seems weird to me. The end result is that without clear guidance from the creators, I put them in one place, and other people put them elsewhere.

When we provide guidance – be it implicit, or explicit – we need to be sure that it covers all the things that are expected. This includes the things that don’t come up very often. Providing usage examples is pretty uncommon. We should still know exactly where to put it.

How we encode things

When it comes to encoding things, Pod6 is a lot more like Markdown than a code documentation tool. It can help you format your text in useful ways, but it doesn’t really provide us any tools to encode information about our code.

Because we have no way to specifically encode parameters, we have no way to extract parameters. This means we can’t generate HTML docs that consistently include notes about parameters, and when they do we can’t guarantee where the notes about each one will appear. This is the organizational problem all over again. It’s harder to find things if you don’t know where they’ll be and they’re not consistently there.

In the prior section I said that I wasn’t sure where the “right” place to document example usage was in YARD docs. But, the fact that it’s a thing with a specific encoding mechanism provides a huge value. The HTML docs I generate can always have it organized according to the standard. Even if I’m completely ignorant of what that standard is. Even if I’m inconsistent in how I do it in my code. The generated HTML docs can consistently create produce a standard organizational structure. The tool knows what its working with, so it can put it in the right place.

Because Pod6 doesn’t really have a way to encode elements of our code, we can’t generate a consistent output. We can’t help people find the information we need.

More importantly, we can’t extract any specific type of information, so we can’t write tools that do useful things in it.

Imagine you’re working in the REPL, figuring out how something works. You start to call a method but realize you don’t remember what order the parameters go in, or what it returns. Because we have no way of specifically encoding a method’s parameters or return value we can’t extract them. You have to leave the REPL, go find the file that contains it, and then the method in that file. If it’s coming from a dependency you may have to track that library down on GitHub and start hunting around.

It doesn’t have to be that way. In some languages it isn’t. There are languages with documentation tools that leverage their structured encoding to expose that information in the REPL. In those languages you can ask the system for a functions docs within the REPL. You can also ask for the specific pieces that might be encoded in it. You can have it tell you just the return value of the foo method.

If we want the benefits that come from having specific ways of recording different types of information then one of three things has to happen.

Extend Pod6 itself

Obviously, we could extend Pod6 itself. This would be great for the community, but it might take years because of conflicting beliefs about what’s important, and how we should do it.

Build something that supports a superset of Pod6

This is what YARD did. YARD took the standard Ruby documentation tool (RDoc), and made a new tool that supported a superset of its functionality. It gives you all of the old stuff, plus the additional features they wanted.

This was a pretty great choice. It means that someone familiar with YARD finds codebases with RDoc totally readable, and vice versa. It also means that those old codebases can have their docs regenerated using YARD’s tooling.

One of the biggest benefits to this approach is that it can be easy to get started. If you’re happy with the Pod6 codebase, you could just fork it, add a single feature. Do that and you’ve improved something for yourself. One feature is not really worth releasing as a new codebase, but this approach makes it easy to iterate towards a goal. It gives you something functional that can be used, and improved as you have time.

Write something else

The last option is to make a completely separate alternative.

Twenty years ago I would probably think “Yeah, right. I’ll ‘just’ make my own Pod6 alternative.” However, a while back I was writing a fair bit of Chicken Scheme, and I wanted to be able to write structured inline docs using Markdown. I wanted to be able to provide docs with a description, parameters, what it returns, code examples, and arbitrary notes. I also wanted all of those things to be exposed when I was working in the REPL. So, I started writing.

It took me about 400 lines of new code to have a tool that did all of that. 1000 in you include a couple files i copied from another project of mine. That’s not because I’m some amazingly efficient Scheme developer. I’m really not. I love scheme, but I’m not that great at it.

It was because I did a lot of creative corner cutting. The following example is not something great. It’s just an example to show how much you can do when you combine a tiny bit of code with some creative thinking.

(doc-fun "greet"
	  "## Public: greet [name]
	Generates a greeting string.

	### Parameters:
	* name - (optional) the name of the person to greet

	### Returns:
	a string

	### Examples:
	    (greet "Bob")

	### Notes:
	This is *obviously* a contrived example.")

Because I delimited each “type” of information with a Markdown header, all I had to do to extract a section was to look for a line starting with hashtags. Then I’d read everything up to the next one or the end. Markdown’s intended to be human readable, so I don’t need to do anything fancy when I display it in the REPL. No markdown parser required. While I love the details that YARD gives me about parameters I don’t actually need code to specifically encode them to improve my quality of life. Instead, I just needed to set expectations for how we document things with this tool.

Parameters appear in a list. They start with the parameter name followed by a hypen a parenthesized note if its optional, and a description. Future versions could break each list item into its component parts.

I didn’t even write something to convert it to HTML. I didn’t need to. There are lots of Markdown to HTML converters out there. Inserting an HTML snippet into a pretty template is trivial. My tool just generates a folder structure of Markdown files that I can run through other people’s tools.

Is it cheating? Maybe. Does it work? Yup. Could it be better? Sure.

I love what my tool can do for me. I love how it increased my productivity when writing Scheme. I can iterate on it and give future versions more capabilities.

Writing a tool that matches your thinking, and supports your desires can be done in such a small amount of code. Knowing that, I’d like you to take a moment and consider what would your ideal documentation tool look like? Is it something that others might want to use?

Summary

Pod6 is a great starting point. It can do a lot, but I think we can do better. I think we can do better in how we introduce new developers to documenting their Raku. I think we can do a better job of showing them what they should document, and how they should organize it. I think we can improve the features it offers developers.

I don’t care how we solve these problems. I just care that they get solved. I’d love to see Pod6 get new capabilities. I’d love to have an alternative to Pod6.

Mostly, I’d love to see people start valuing inline documentation more. I’d love to open up your codebase and find the help I need, where I expect it. I don’t want to have to figure out your quirky way of organizing things.

We few Raku devs have the world’s greatest text manipulation tool just waiting for us to use it. Let’s do that. Let’s make our documentation tools as bad-ass as the code they document.

Day 5 – The Elves go back to Grammar School

It was Christmas Day in the workhouse

the snow was raining fast

a barefooted man with clogs on

came slowly running past

anon

‘Twas the month before Christmas when a barefooted Elf with clogs on realised that they needed to load around 1_000_000_000 addresses into Santa’s Google maps importer so that he could optimise his delivery route and get all the presents out in one night.

On the plus side, all the addresses came in via emails and pdf attachments as computer readable unicode text. But how to read all those addresses into a consistent format so that they could be fed to the maps app?

Then they remembered that Mrs. Claus had sent them all to (Raku) Grammar school and that now was the time to brush up on their parsing skills.

Peeking at the Answer

So, to work backwards, as is most comfortable for Elven folk, we start with some address text and feed it to be parsed by a raku Grammar.

#!/usr/bin/env raku
use v6.d;
use lib '../lib';
use Data::Dump::Tree;
use Grammar::Tracer;
my ($address,$match);
$address = q:to/END/;
123, Main St.,
Springfield,
IL 62704
USA
END
$address ~~ s:g/','$$//;
$address ~~ s:g/<['\-%]>//;
$address .= chomp;
$match = AddressUSA::Grammar.parse($address, :actions(AddressUSA::Actions));
say ~$match;
say $match;
ddt $match.made;

Note the use of Grammar::Tracer, this is an essential design and debug tool to work out what is happening in the parsing process with various text inputs.

There’s a little preparation of our text to prune out unwanted punctuation and line endings using the ~~ s:g/// substitution operator.

The result of the parsing operation is a new Address object, so the output with ddt is:

.AddressUSA @0
├ $.street = 123, Main St..Str
├ $.city = Springfield.Str
├ $.state = IL.Str
├ $.zipcode = 62704.Str
└ $.country = USA.Str

All neatly tagged and ready for import to a structured system.

Grammar, Rhyme and Meter

The Grammar itself looks like this..

grammar AddressUSA::Grammar does Address::Grammar::Base {
token TOP {
<street> \v
<city> \v
<state-zip> \v?
[ <country> \v? ]?
}
token state-zip {
^^ <state> <.ws>? <zipcode> $$ #<.ws> is [\h* | \v]
}
token state {
\w \w
}
token zipcode {
\d ** 5
}
}

Sooo – as with any well written elven code, it is easy for the casual reader to grasp what is going on (with a little prior know how of the Raku Grammar basics work, such as token TOP)

Casting a more expert eye, Rudolph (for it was he that crafted this tutorial) has a few expert remarks:

  • note the use of \v and \h to denote vertical and horizontal whitespace
  • since Raku does not dictate indentation, we can go with an aligned layout to better echo our input text
  • the query ? and square brackets [] help us with options (e.g. there may be no coontry line in the text)
  • token state looks for two word characters
  • token zipcode looks for 5 digits
  • there is a bit of magic in <state-zip> that picks out <state> and <zipcode> regardless of whether they are on the same line or on two lines
  • oh, and ^^ is the start of a line and $$ is the end (ie just before a \v)

So far, so ginchy.

Grammar Fundaments

Well, Rudolph was holding a bit up his sleeve (ed?) here, since raku doesn’t come with tokens like <street>, <city>, etc out of the box. An attentive Elf would spot the does Address::Grammar::Base … let’s look at that now:

my @street-types = <Street St Avenue Ave Av Road Rd Lane Ln Boulevard Blvd>;
role Address::Grammar::Base {
token street {
^^ [<number> ','? <.ws>]? <plain-words> <.ws> <street-type> '.'? $$
}
token number {
\d*
}
token plain-words {
<plain-word>+ % \h
}
token plain-word {
\w+ <?{ "$/" !~~ /@street-types/ }>
}
token street-type {
@street-types
}
token town { <whole-line> }
token city { <whole-line> }
token county { <whole-line> }
token country { <whole-line> }
token whole-line {
^^ \V* $$
}
}

So that explains where the Base tokens are built up and underlines the fact that a Raku grammar is just a fancy class and that tokens are just fancy methods. That way you can use the same role composition via the keyword does.

So key tools used here:

  • You can use any Raku Array like ‘@street-types‘ directly in a token, the elements are treated as if this was a set of alternate Str values [‘Street’ | ‘St’ | …]
  • The look-ahead assertion <?{ … }> is a cool way to call any code that returns True or False – in this case to check that the (stringified) match “$/” is not a street-type
  • And the “repeats” symbol % modifies the quantifier to add a separator pattern – in this case \h for a horizontal whitespace character

Actions of Compassion

“In the quaint streets of Victorian London, a benevolent soul anonymously gifted warm blankets to shivering orphans, embodying the true spirit of Christmas compassion in Dickensian fashion.”

ChatGPT 3.5

That just leaves the grammar Actions to be added in. In our example below, first we define our class AddressUSA to hold the results and then we make a new instance of it with the action method TOP

class AddressUSA {
has Str $.street;
has Str $.city;
has Str $.state;
has Str $.zipcode;
has Str $.country = 'USA';
}
class AddressUSA::Actions {
method TOP($/) {
my %a;
%a<street> = $_ with $<street>.made;
%a<city> = $_ with $<city>.made;
%a<state> = $_ with $<state-zip><state>.made;
%a<zipcode> = $_ with $<state-zip><zipcode>.made;
%a<country> = $_ with $<country>.made;
make AddressUSA.new: |%a
}
method street($/) { make ~$/ }
method city($/) { make ~$/ }
method state($/) { make ~$/ }
method zipcode($/) { make ~$/ }
method country($/) { make ~$/ }
}

The action methods use make and made commands to percolate those parts of the Grammar that we want to select up to the method TOP and then into the new object attributes with Address.new: |%a

  • This shows that the associative accessors we normally see with hashes like %a<street> are very similar to accessors on the match object itself like $<street>, because (err) they are the same just that $<> is syntatic sugar for $/<>
  • You can use them like %a{‘street’} if you prefer more typing and more line noise

And so, my fine friends, to another year of -0fun

There is nothing in the world so irresistibly contagious as laughter and good humor.

Charles Dickens, A Christmas Carol

I wish each and every one of you a Merry Christmas and a Happy New Year.

~librasteve

PS. All the code for this story is available as a single file in this… Github Gist

Day 4 – Embedding a stack-based programming language in Raku

When @lizmat asked me to write a post for the Raku advent calendar I was initially a bit at a loss. I have spent most of the year not writing Raku but working on my own language Funktal, a postfix functional language that compiles to Uxntal, the stack-based assembly language for the tiny Uxn virtual machine.

But as Raku is nothing if not flexible, could we do Uxntal-style stack-based programming in it? Of course I could embed the entire Uxntal language in Raku using a slang. But could we have a more shallow embedding? Let’s find out.

Stack-oriented programming

An example of simple arithmetic in Uxntal is

    6 4 3 ADD MUL

This should be self-explanatory: it is called postfix, stack-based or reverse Polish notation. In infix notation, that is 6*(4+3). In prefix notation, it’s MUL(6, ADD(4,3)). The integer literals are pushed on a stack, and the primitive operations ADD and MUL pop the arguments they need off the stack an push the result.

The mighty  operator, part I: definition

In Raku, we can’t redefine the whitespace to act as an operator. I could of course do something like

    my \stack-based-code = <6 4 3 ADD MUL>;

but I don’t want to write an interpreter starting from strings. So instead, I will define an infix operator . Something like this:

    6 ∘ 4 ∘ 3 ∘ ADD ∘ MUL

The operator either puts literals on the stack or calls the operation on the values on the stack.

By necessity,  is a binary operator, but it will put only one element on the stack. I chose to have it process its second argument, and ignore the first one,because in that way it is easy to terminate the calculation. However, because of that, the first element of each sequence needs to be handled separately.

Returning the result

To obtain a chain of calculations, the operator needs to put the result of every computation on the stack. This means that in the example, the result of MUL will be on the stack, and not returned to the program. To return the final result to the program, I slightly abuse Uxntal’s BRK opcode. On encountering this opcode, the value of the computation is returned and the stack is cleared (in native Uxntal, BRK simply terminates the program). So a working example of the above code is

    my \res = 6 ∘ 4 ∘ 3 ∘ ADD ∘ MUL ∘ BRK;

Some abstraction with subroutines

Uxntal allows to define subroutines. They are just blocks of code that you jump to. In my Raku implementation we can simply define custom subroutines and call them using the Uxntal instructions JSR (jump and return), JMP (jump and don’t return, used for tail calls) and JCN (conditional jump).

my \res =  3 ∘ 2 ∘ 1 ∘ INC ∘ ADD ∘ MUL ∘ 4 ∘ &f ∘ JMP;

sub f {
    SUB ∘ 5 ∘ MUL ∘ 2 ∘ ADD ∘ RET
}

(the instruction RET is called JMP2r in Uxntal)

Stack manipulation operations

One of the key features of a stack language is that it allows you to manipulate the stack. In Uxntal, there are several operations to duplicate, remove and reorder items on the stack. Here is a contrived example

my \res =
    4 ∘ 2 ∘ DUP ∘ INC ∘        # 4 2 3
    OVR ∘                      # 4 2 3 2
    ROT ∘                      # 4 3 2 2
    ADD ∘ 2 ∘ ADD ∘ MUL ∘ BRK; # 42

Keeping it simple

Uxntal has more ALU operations and device IO operations. It also has a second stack (return stack) and operations to move data between stacks. Furthermore, every instruction can take the suffix ‘2’, meaning it will work on two bytes, and ‘k’, meaning that it will not consume its arguments but leave them on the stack. I am omitting all these for simplicity.

The mighty  operator, part II: implementation

With the above, we have enough requirements to design and implement the operator. As usual, I will eschew the use of objects. It was my intention to use all kind of fancy Raku features such as introspection but it turns out I don’t need them.

We start by defining the Uxntal instructions as enums. I could use a single enum but grouping them makes their purpose clearer.

enum StackManipOps is export <POP NIP DUP SWP OVR ROT BRK>;
enum StackCalcOps is export <ADD SUB MUL INC DIV>;
enum JumpOps is export <JSR JMP JCN RET>;

We use a stateful custom operator with the stack @wst (working stack) as state. The operator returns the top of the stack and is left-associative. Anything that is not an Uxntal instruction is pushed onto the stack.

our sub infix:<∘>(\x, \y)  is export {
    state @wst = ();

    if y ~~ StackManipOps {
        given y {
            when POP { ... }
            ...
        }
    } elsif y ~~ StackCalcOps {
        given y {
            when INC { ... }
            ...
        }
    } elsif y ~~ JumpOps {
        given y {
            when JSR { ... }
            ...
        }
    } else {
        @wst.push(y);
    }

    return @wst[0]
}

This is not quite good enough: the operator is binary, but the above implementation ignores the first element. This is only relevant for the first element in a sequence. We handle this using a boolean state $isFirst. When True, we simply call the operator again with Nil as the first element. The $isFirst state is reset on every BRK.

    state Bool $isFirst = True;
    ...
    if $isFirst {
        @wst.push(x);
        $isFirst = False;
        Nil ∘ x
    }

The final complication lies in the need to support conditional jumps. The problem is that in e.g.

    &ft ∘ JCN ∘ &ff ∘ JMP

depending on the condition, ft or ff should be called. If ft is called, nothing after JCN should be executed. I solve this by introducing another boolean state variable, $skipInstrs, which is set to True when JCN is called with a true condition. 

    when JCN {
        my &f =  @wst.pop;
        my $cond = @wst.pop;
        if $cond>0 {
            $isFirst = True;
            f();
            $skipInstrs = True;
        }
    }

The boolean is cleared on encountering a JMP or RET:

    if $skipInstrs {
        if (y ~~ JMP) or (y ~~ RET) {
            $skipInstrs = False
        }
    } else {
        ...
    }

This completes the implementation of the operator . The final structure is:

our sub infix:<∘>(\x, \y)  is export {
    state @wst = ();
    state Bool $isFirst = True;
    state $skipInstrs = False;

    if $skipInstrs {
        if (y ~~ JMP) or (y ~~ RET) {
            $skipInstrs = False
        }
    } else {

        if $isFirst and not (x ~~ Nil) {
            @wst.push(x);
            $isFirst = False;
            Nil ∘ x
        }

        if y ~~ StackManipOps {
            given y {
                when POP { ... }
                ...
            }
        } elsif y ~~ StackCalcOps {
            given y {
                when INC { ... }
                ...
            }
        } elsif y ~~ JumpOps {
            given y {
                when JSR { ... }
                ...
            }
        } else {
            @wst.push(y);
        }
    }
    return @wst[0]
}

Memory and pointers

Like most stack languages, Uxntal also has load and store operations to work with memory. Uxntal does not have a separate instruction memory, so you can mix code and data and even write self-modifying code. There are load and store operations on absolute (LDA,STA) and relative (LDR,STR) addresses. In my Raku implementation, I don’t distinguish between those. I use arrays as named stretches of memory. So for example the following native Uxntal snippet

@array #11 #22 #33
;array #0001 ADD2 LDA ( puts 0x22 on the stack )

becomes

    my @array = 0x11, 0x22, 0x33;
    @array ∘ 0x0001 ∘ ADD ∘ LDA

and that would be close enough, were it not that in Uxntal memory is declared after subroutines. So what I actually need to do is

    (array) ∘ 0x0001 ∘ ADD ∘ LDA;
    sub array { [ 0x11, 0x22, 0x33 ] }

The way I handle the pointer arithmetic is by pattern matching on the type. Instructions ADDSUB and INC can take an integer or a label, which in Raku is an Array. The valid argument type for these operations (in pseudo-code type definitions) is

Pair  = (Fst,Int)
Fst = Array | (Fst,Int)
Addr = Int | Array | Pair

In words, it can be an integer, a label or a pair where the second element of the pair must be an integer and the first is either a label or a pair.

For example for the INC operation, we do

    given (arg) {
        when Int { push @wst,arg+1}
        when Array { push @wst,(arg,1)}
        when List { push @wst,(arg,1)}
    }

For ADD and SUB we do something similar but check if either arg is Int, Array or List. If both arguments are Int, we return the result of the operation; if only one of the arguments is an Int, we return the pair of arguments as a List; otherwise we throw an error as it is not a valid type.

The non-integer return values of this kind of arithmetic are used in LDA and STA. Here, the only valid type is the following:

Addr = Array | (Addr, Int)

In other words, an address must always be relative to a label. So we will check if the argument is not an integer.

Hello, Uxntal

With this machinery we can run the following Uxntal “Hello World” program

    |0100 ( start of program in Uxntal )

        ;hello JSR2

    BRK

    @hello
        ;hello-word ;print-text JSR2
    JMP2r

    @print-text ( str* -- )
        ;loop JSR2
    JMP2r

    @loop
        DUP2 LDA #18 DEO
        INC2 DUP2 LDA ;loop JCN2
    JMP2r

    @hello-word "Hello 20 "World! 00
    ( the #18 DEO instruction prints a byte on STDOUT )

in Raku as follows (assuming we have aliases with the suffix 2):

#|0100
        &hello ∘ JSR2 ∘ 
    BRK;

    sub hello {
        (hello-world) ∘ &print-text ∘ JSR2 ∘ 
        RET
    }

    sub print-text { # str* --
        &loop ∘ JSR2 ∘ 
        RET
    }

    sub loop {
        DUP2 ∘ LDA ∘ 0x18 ∘ DEO ∘ 
        INC2 ∘ DUP2 ∘ LDA ∘ &loop ∘ JCN2 ∘ 
        RET
    }

    sub hello-world { ["Hello,",0x20,"World!", 0x0a,0x00] }

As this program has a loop implemented as a tail recursion, it is complete in terms of illustrating the features of a stack-based program in Uxntal.

Conclusion

So in conclusion, we can embed a stack-based programming language such as Uxntal in Raku purely by defining a single binary operator and a few enums for the instructions. This is mainly courtesy of Raku’s state variables and powerful pattern matching.

The code for this little experiment is available in my raku-examples repo, with two sample programs stack-based-programming.raku and hello-uxntal.raku and the module implementing the operator Uxn.rakumod.

Day 3 – Helping the Elves Sort Their Mail

Lately, I’ve been a bit obsessed with Dave Thomas’s CodeKata series, and especially solving these problems in Raku. In this post, I want to talk about different ways of writing Raku and how to measure performance. We’ll focus on part 2 of Kata 11: Sorting It Out.

Approach #1: Let’s Use a Regex!

Raku, being a descendant of Perl, has an awesome regex engine. So, my first solution was straightforward enough:

sub sort-characters-regex($msg) {
    $msg
      .lc
      .subst(/:ignorecase <-[a..z]>/)
      .comb
      .sort
      .join
}

We convert the message to lowercase (.lc), case-insensitively remove characters that aren’t letters (.subst(/:ignorecase <-[ a..z]>/)), convert the message to a Seq of characters (.comb), sort them (.sort) and join them back to a string (.join). But just how fast (or slow!) is that? Let’s find out:

sub benchmark(&fn) {
    constant MESSAGE = 'When not studying nuclear physics, Bambi likes to play beach volleyball.';
    constant RUNS = 100_000;

    fn(MESSAGE) for ^RUNS;

    say "{RUNS} runs of &fn.name() took " ~ (now - ENTER now) ~ 's.';
}

benchmark &sort-characters-regex; # OUTPUT: «100000 runs of sort-characters-regex took 6.761854624s.␤»

6.76 seconds. Not bad, if performance isn’t a concern.

Approach #2: Imperative Programming

The original post says:

Are there any ways to perform this sort cheaply, and without using built-in libraries?

This got me thinking: why, that sounds like a Hash! Let’s try it.

sub sort-characters-imperatively($msg) {
    my %chars;
    constant LETTERS = ('a'..'z').join;
    for $msg.comb -> $ch {
        my $l = lc $ch;
        if LETTERS.contains($l) {
            %chars{$l}++;
        }
    }
    my $s = '';
    for LETTERS {
        $s ~= $_ x (%chars{$_} // 0);
    }
    return $s;
}

We iterate over each character of the message, lowercase it, check if it’s a letter and increment that letter’s count. Then, we initialize an empty string $s, and for each letter, we create a string containing said letter as many times as needed with the x operator, and append it to $s. We use // 0, which uses the definedness operator //, to provide a default value in case the letter has no associated count, i.e. is not present in the original message.

It’s quite a bit more code, but it’s also faster:

100000 runs of sort-characters-imperatively took 3.77489963s.

Approach #3: Functional Programming (or the Middle Road)

Is there another way to write this function? It’d be a strange question to ask if the answer were no, so… yes. We’ll use the Bag class. If you’ve ever used Python, it’s like a collections.Counter. If not, according to the docs, a Bag is

a collection of distinct elements in no particular order that each have an integer weight assigned to them signifying how many copies of that element are considered “in the bag.”

For example: "wheelbarrow".comb.Bag returns Bag(a b e(2) h l o r(2) w(2)). This means that “wheelbarrow” has one A, one B, two E’s, and so on. That sounds exactly like what we need:

sub sort-characters-functionally($msg) {
    my %frequency := $msg.lc.comb.Bag;
    return ('a'..'z').map({ $_ x %frequency{$_} }).join;
}

This approach is short and sweet, and somewhere in the middle of the other two in terms of performance:

100000 runs of sort-characters-functionally took 5.038593895s.

Note that Raku also enables us to effortlessly parallelize the code, just by adding .hyper:

sub sort-characters-functionally-hypered($msg) {
    my %frequency := $msg.lc.comb.Bag;
    return ('a'..'z').hyper.map({ $_ x %frequency{$_} }).join;
}

Sadly, that only hurts performance in this case:

100000 runs of sort-characters-functionally-hypered took 49.788108188s.

Ouch.

Still, hyper (and its cousin race) often make code run faster, and they are great tools to have in your toolbox.

Conclusion

Now, which approach should we use in production? Any of the three approaches seem fine. It just depends on what your goal is. Is it performance? Or readability? Or something in between? I love that Raku doesn’t force you to write code in any particular way. You can write it in a way that makes sense to you.

I hope you enjoyed this post, and I hope you learned something new. Merry Christmas!

Acknowledgments

This blog post could not have been written without the thoughtful feedback of Elizabeth Mattijsen. Thank you for all you’ve done for me and the Raku community at large.

Day 2 – An update on raku.land

A couple of years ago we gave a talk at the The Raku Conference 2021 which served both as an announcement and an introduction of raku.land. It was an exciting time for us, since it was the first time we were officially addressing the community, and we were keen to see how our work would be received and what impact it would have.

Fast-forward to today, and some of you might have noticed that the venerable modules.raku.org domain now redirects to raku.land.

We thought this was a good chance to take a look at what lead us here, where the new site is, where it is going, and how you—yes, you!—can help get it there.

What happened to the old site?

The original modules.raku.org site was written to solve a very specific problem: the Raku community needed a way to browse the modules that had been released to its fledgling ecosystem. It also served as an early test-bed for the integration of some community features that grew around that ecosystem, like the citation index, for example.

This was in the early days of Raku, before there was a satisfactory way to build and serve a website with the features that were needed, so this initial version was built using Mojolicious, one of Perl’s shining web frameworks.

There was a lot to like in this website, but as time went on, and its original maintainers moved on to different projects, it was largely left unmaintained. This not only meant that it started to lag behind some of the new features that had been added to the language (most importantly what at the time was the new zef ecosystem), but it was also becoming harder for the infrastructure team to keep the site running.

This came to a head a couple of weeks ago, when the site went down and could not be brought back up. That precipitated a decision that had been in discussion for the better part of a year, and the domain was redirected to raku.land.

This is the end of an era. Raku would not be where it is today were it not for what the old site allowed us to do, and we celebrate it for what it achieved. But this is also the start of a new era.

In with the new

What began as an experiment about using Raku in a real website, has now started to become an essential part of the day-to-day activities of the Raku community. It was the first web-based front-end for the zef ecosystem, and for a little over a year it has powered the canonical URLs used by Rakudo Weekly News.

We take this responsibility very seriously, so we’ve tried to respond to the user feedback we have received since then. Some of the concrete changes to the site since our initial announcement include the ability to display multiple versions of a distribution, and to link to specific ones; an endpoint to render recent releases in JSON; the display of reverse dependencies, and dependencies by stage; and the addition of a dark theme.

We know that there are still plenty of things that the site could do better, but we also know that the best chance we have to accomplish them is with the help of the community at large. This is true specially for some of the trickiest problems we need to grapple with, where the difficulty lies not in solving the problem, but in deciding how to solve the problem.

Tough problems to solve

The main problem that raku.land aims to solve—namely, that of cataloguing and organising information about Raku distributions—is a difficult one. And although it is a well known problem (after all, a lot of languages have distribution ecosystems), some aspects of it are particularly difficult because we are dealing with Raku distributions (after all, not many languages support multiple ecosystems!).

Take the following example: if you use Digest::MD5, which module are you loading? The answer will actually depend on the distributions that have been installed in your system. You could be loading the Digest::MD5 module provided by Digest:auth<zef:grondilu>, or you could be loading the one provided by Digest::MD5:auth<github:cosimo>. These are incompatible distributions, written by different authors, released on different ecosystems… but both provide a module with the same name. We would all agree these are different, but it’s up to you to disambiguate.

That particular example happens across ecosystems, but it doesn’t have to. Take JSON::Class: there’s JSON::Class:auth<zef:vrurg> and JSON::Class:auth<zef:jonathanstowe>. These are both on the same ecosystem (zef) but other than a shared name, these are also incompatible. Chances are we would also agree these are different.

On the other hand, you have several distributions that have historical releases on one ecosystem, but then moved to a newer one when it became available. Take Cro:auth<cpan:JNTHN> and Cro:auth<zef:cro>, for example. These distributions share the same codebase, and have been released by the same author, but since they’ve been released on separate ecosystems they have different authorities. So, are these the same?

A lot of the hardest problems raku.land has to deal with stem from this disambiguation, and from the fact that sometimes we would expect these “ambiguous” distributions to be treated as logically different, and sometimes as logically the same. For Raku the language these are all unambiguously different, but for Raku programmers the answer will depend.

The ramifications are wide-ranging: they affect everything from our ability to have short canonical distribution URLs, to how distributions are listed on the site, to our ability to detect when a distribution depends on another: if it lists “Digest::MD5” as a dependency, how can we know which one that will be? Whatever solution we come to in raku.land for these issues must come as a decision from the community, because the site needs to reflect the ecosystems it serves, and not the other way around. So we need your help.

A community project

The term “community project” is a slippery one. We all think we understand the term when we read it, but when it comes to deciding whether a specific project does or does not belong to “the community”, opinions differ.

Regardless of these differences though, raku.land is a project that was always developed with the interests of the community as a top priority. And as such, it’s a project that relies on the participation of the community for its success.

We’ve tried to make it easier for anyone to get involved and make contributions: we’ve kept the source-code lightweight so that it’s (hopefully!) easier to jump in and understand; and we’ve tried to document how the code and its deployment works. We have a roadmap of things that we would like to see in the future, and the team is reachable through a project email address and our own IRC channel on libera.chat.

Although the goals in that roadmap could possibly be more fleshed out, there are a lot of ambitious ideas we have for the future:

  • Source-code browsing
  • Properly linked rendered documentation
  • A unified API to expose some of the information we collect across ecosystems
  • The ability to render diffs between versions
  • Improvements to our search feature
  • … and more!

What we lack the most right now is the help of people like you. Some of these are hard problems to solve, but some of them are entirely achievable goals that we could tackle right now.

And there are multiple ways to help. We want technical writers who can help us flesh out the documentation wherever it’s needed. We need help raising bug reports and triaging tickets. We need more voices contributing to the conversation and pointing out where we’ve grown complacent. And of course, we can always use help implementing new features and integrating with new and exciting parts of the ecosystem.

From our point of view, raku.land is undoubtedly a community project, and the best way to move it forward is to get involved. Come join us!

Day 1 – Rocking Raku Meets Stodgy Debian

A unique method for installing Raku on Debian hosts.

Rumbling in the rack room

Santa’s IT department was now fully invested in using Raku, and it was paying off in increased programmer efficiency as well as toy output.

But old elf Eli, a system administrator, was grumbling to himself that it was a lot of work keeping the many hosts (mostly GNU/Linux, for great savings on hardware) with many different Linux flavors current given the rapid development of Raku.

He had been trying to convince his boss to gradually convert all systems to Debian, but his boss was reluctant to do so because of a conundrum: Debian has a long development cycle (good for IT stability) but Raku has a short development cycle (roughly monthly).

Systems other than Debian tended to keep Raku more current and typically stayed within 12 months of the latest Raku. But that effort often resulted in more OS maintenance than Debian required. (Note the proliferation of Linux flavors had come about due to flashy young elves, freshly out of grad school, who favored the OS-of-the-month they used in school. To Eli’s chagrin, Santa was a softy and too kind to interfere.)

Eli had long ago selected Debian as his choice of GNU/Linux systems for several reasons including its program packaging system as well as its roughly three-year upgrade cycle.

Unfortunately, that conflicted with Raku’s almost monthly upgrade cycle. Given the long cycle, Debian’s Raku version tended to be quite outdated (although usable for most purposes). Eli wanted a maintainable way to keep Raku current as well as make it a primary coding language on each new system.

The steadfast but unheralded folks on the Raku release team have provided many ways to install Raku on several flavors of GNU/Linux as well as MacOS and Windows. Eli started using the ‘rakudo-pkg’ method when it was first introduced years ago.

But, when it was announced it would not keep up with Debian versions after the end of the Long Term Support (LTS) period, he started considering other available methods to see which was best for complete automation both for regular upgrades as well as during initial Debian installations (including Debian’s preseeding installation).

Inspiration and perspiration

After considering methods published on https://rakudo.org, he decided the binary downloads looked to be the easiest.

An archive of code compiled for the desired system, accompanied by an identifying SHA-256 hash and signed by one of the holders of the public keys published on the site, and unpacked into a standard directory should be possible with Raku alone without other modules than the one he would create.

With the tantalizing smell of figgy pudding coming from the mess hall, he started to think deep thoughts while doing his regular duties. Finally, eureka! Why not use the Debian system Raku to bootstrap a current Raku—genius!

Eagerly he began to gobble pudding while coding…. After too much pudding and many code iterations, mistakes, and going down rabbit holes, and with help from fellow Debianites on the Debian users mailing list, he came up with his shiny new Raku module distribution: RakudoBin (not yet published, but available online).

He solved the bootstrap problem by using a special path setup so the new module’s installation script could use the system Raku while other existing or new Raku programs would have the latest Raku first in the default user PATH. Because of the lengths of the paths defined in the actual host system, the paths are represented here by an alias, [pathX], where ‘X’ is the path segment:

  • pathA /opt/rakudo/bin:/opt/rakudo/share/perl6/site/bin
  • pathB /usr/local/bin:/usr/bin:/bin
  • pathC /usr/local/games:/usr/games

The final system path for all users is established by creating the following system files:

$ cat /etc/environment
pathA:pathB:pathC
$ cat /etc/profile
pathA:pathB:pathC

Standard new user files in /etc/skel are okay as they are. But a missing file is added to correct the long-standing lack of a reliable graphical login solution for path setting, at least for the Mate deskstop:

# existing files NOT modified:
/etc/skel/.bash_logout
/etc/skel/.bashrc
/etc/skel/.profile
# added, not Debian standard, solves graphical
#   login path setting problem:
$ cp /etc/skel/.profile /etc/skel/.xsessionrc

That solution was achieved with much trial and error on a new host with freshly installed Debian 12 (Bookworm), plus lots of help from fellow Debian users, and a slightly outdated Debian online document.

The latest set of Rakudo binary files for a GNU/Linux system consist of:

FileSize
rakudo-moar-2023.11-01-linux-x86_64-gcc.tar.gz19.72 MB
rakudo-moar-2023.11-01-linux-x86_64-gcc.tar.gz.asc228.00 B
rakudo-moar-2023.11-01-linux-x86_64-gcc.tar.gz.checksums.txt1.02 KB

They are downloaded and checked for hash validity, then unpacked into directory /opt/rakudo. The paths required to use the installed binaries are /opt/rakudo/bin and /opt/rakudo/share/perl6/site/bin.

The installation script sets the standard path to put the new paths before the standard paths as shown below.

Debian standard path:

/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

RakudoBin modified path (with game paths removed):

/opt/rakudo/bin:/opt/rakudo/share/perl6/site/bin:pathB:pathC

Note the path finds the newly installed executables before the system’s since those are under directory /usr/bin. Eli solved the bootstrap problem by putting this as the shebang line in the installation script:

#!/usr/bin/raku

One other tricky problem Eli solved, and was quite proud of, was checking for the latest version without downloading large files. The easiest method, knowing the URL and format of the file set, is to just attempt to download it with this code:

shell "curl -1sLf https://path/file -o /localpath/file";

If the file doesn’t exist, the curl command throws an error. If we can trap the error, we can say the file doesn’t exist. Thanks to the help of Nick [no, not Santa, IRC #raku’s Nick Logan, aka @ugexe]), we can do that easily with this simple line in a check-https sub returning a Bool:

sub check-https("https://path/file" --> Bool {
    try {
        so quietly shell
          "curl -1sLf https://path/file -o /localpath/file"
    } // False
}

Here’s a link for some help for a more general method for future improvements to check a partial download with large files.

The installation script, by default, carefully asks for conrirmation before proceeding with various system-modifying parts. Using the quiet option assumes all is approved and no further permissions are requested.

Party time

In summary, module RakudoBin will give system administrators a practical way to (1) have a stable Debian system and (2) upgrade their Raku version to the latest. The module will also provide an easy way to quickly convert a standard Debian installation into a standard Raku host with common paths for all new users.

Santa’s Epilogue

Don’t forget the “reason for the season:” ✝

As I always end these jottings, in the words of Charles Dickens’ Tiny Tim, “may God bless Us, Every one!” [1]

Footnotes

  1. A Christmas Carol, a short story by Charles Dickens (1812-1870), a well-known and popular Victorian author whose many works include The Pickwick PapersOliver TwistDavid CopperfieldBleak HouseGreat Expectations, and A Tale of Two Cities.

The 2022 Raku Advent Posts

(in chronological order, with comment references)

Day 25: Rakudo 2022 Review

In a year as eventful as 2022 was in the real world, it is a good idea to look back to see what one might have missed while life was messing with your (Raku) plans.

Rakudo saw about 1500 commits this year, about the same as the year before that. Many of these were bug fixes and performance improvements, which you would normally not notice. But there were also commits that actually added features to the Raku Programming Language. So it feels like a good idea to actually mention those more in depth.

So here goes! Unless otherwise noted, all of these changes are in language level 6.d, and available thanks to several Rakudo compiler releases during 2022.

New REPL functionality

It is now possible to refer to values that were produced earlier, using the $*N syntax, where N is a number greater than or equal to 0.

$ raku
To exit type 'exit' or '^D'
[0] > 42
42
[1] > 666
666
[2] > $*0 + $*1
708

Note that the number before the prompt indicates the index with which the value that is going to be produced, can be obtained.

New MAIN options

You can now affect the interpretation of command line arguments to MAIN by setting these options in the %*SUB-MAIN-OPTS hash:

allow-no

Allow negation of a named argument to be specified as --no-foo instead of --/foo.

numeric-suffix-as-value

Allow specification of a numeric value together with the name of a single letter named argument. So -j2 being the equivalent of --j=2.

So for example, by putting:

my %*SUB-MAIN-OPTS = :allow-no, :numeric-suffix-as-value;

at the top of your script, you would enable these features in the command-line argument parsing.

New types

Native unsigned integers (both in scalar, as well as a (shaped) array) have finally become first class citizens. This means that a native unsigned integer can now hold the value 18446744073709551615 as the largest positive value, from 9223372036854775807 before. This also allowed for a number of internal optimisations as the check for negative values could be removed. As simple as this sounds, this was quite an undertaking to get support for this on all VM backends.

my uint  $foo = 42;
my uint8 $bar = 255;
my  int8 $baz = 255;

say $foo; # 42
say $bar; # 255
say $baz; # -1

say ++$foo; # 43
say ++$bar; # 0
say ++$baz; # 0

And yes, all of the other explicitly sized types, such as uint16uint32 and uint64, are now also supported!

New subroutines

A number of subroutines entered the global namespace this year. Please note that they will not interfere with any subroutines in your code with the same name, as these will always take precedence.

NYI()

The NYI subroutine takes a string to indicate a feature not yet implemented, and turns that into a Failure with the X::NYI exception at its core. You could consider this short for ... with feedback, rather than just the “Stub code executed”.

say NYI "Frobnication";
# Frobnication not yet implemented. Sorry.

chown()

The chown subroutine takes zero or more filenames, and changes the UID (with the :uid argument) and/or the GID (with the :gid argument) if possible. Returns the filenames that were successfully changed. There is also a IO::Path.chown method version.

my @files  = ...;
my $uid    = +$*USER;
my changed = chown @files, :$uid;
say "Converted UID of $changed / @files.elems() files";

Also available as a method on IO::Path, but then only applicable to a single path.

head(), skip(), tail()

The .head.skip and .tail methods got their subroutine counterparts.

say head 3, ^10; # (0 1 2)
say skip 3, ^10; # (3,4,5,6,7,8,9)
say tail 3, ^10; # (7 8 9)

Note that the number of elements is always the first positional argument.

New methods

Any.are

The .are method returns the type object that all of the values of the invocant have in common. This can be either a class or a role.

say (1, 42e0, .137).are;        # (Real)
say (1, 42e0, .137, "foo").are; # (Cool)
say (42, DateTime.now).are;     # (Any)

In some languages this functionality appears to be called infer, but this name was deemed to be too ComputerSciency for Raku.

IO::Path.inode|dev|devtype|created|chown

Some low level IO features were added to the IO::Path class, in the form of 5 new methods. Note that they may not actually work on your OS and/or filesystem. Looking at you there, Windows 🙂

  • .inode – the inode of the path (if available)
  • .dev – the device number of the filesystem (if available)
  • .devtype – the device identifier of the filesystem (if available)
  • .created – DateTime object when path got created (if available)
  • .chown – change uid and/or gid of path (if possible, method version of chown())

(Date|DateTime).days-in-year

The Date and DateTime classes already provide many powerfule date and time manipulation features. But a few features were considered missing this year, and so they were added.

A new .days-in-year class method was added to the Date and DateTime classes. It takes a year as positional argument:

say Date.days-in-year(2023);  # 365
say Date.days-in-year(2024);  # 366

This behaviour was also expanded to the .days-in-month method, when called as a class method:

say Date.days-in-month(2023, 2);  # 28
say Date.days-in-month(2024, 2);  # 29

They can also be called as instance methods, in which case the parameters default to the associated values in the object:

given Date.today {
    .say;                # 2022-12-25
    say .days-in-year;   # 365
    say .days-in-month;  # 31
}

New Dynamic Variables

Dynamic variables provide a very powerful way to keep “global” variables. A number of them are provided by the Raku Programming Language. And now there is one more of them!

$*RAT-OVERFLOW

Determine the behaviour of rational numbers (aka Rats) if they run out of precision. More specifically when the denominator no longer fits in a native 64-bit integer. By default, Rats will be downgraded to floating point values (aka Nums). By setting the $*RAT-OVERFLOW dynamic variable, you can influence this behaviour.

The $*RAT-OVERFLOW dynamic variable is expected to contain a class (or an object) on which an UPGRADE-RAT method will be called. This method is expected to take the numerator and denominator as positional arguments, and is expected to return whatever representation one wants for the given arguments.

The following type objects can be specified using core features:

Num

Default. Silently convert to floating point. Sacrifies precision for speed.

CX::Warn

Downgrade to floating point, but issue a warning. Sacrifies precision for speed.

FatRat

Silently upgrade to FatRat, aka rational numbers with arbitrary precision. Sacrifies speed by conserving precision.

Failure

Return an appropriate Failure object, rather than doing a conversion. This will most likely throw an exception unless specifically handled.

Exception

Throw an appropriate exception.

Note that you can introduce any custom behaviour by creating a class with an UPGRADE-RAT method in it, and setting that class in the $*RAT-OVERFLOW dynamic variable.

class Meh {
    method UPGRADE-RAT($num, $denom) is hidden-from-backtrace {
        die "$num / $denom is meh"
    }
}
my $*RAT-OVERFLOW = Meh;
my $a = 1 / 0xffffffffffffffff;
say $a;     # 0.000000000000000000054
say $a / 2; # 1 / 36893488147419103230 is meh

Note that the is hidden-from-backtrace is only added so that any backtrace will show the location of where the offending calculation was done, rather than inside the UPGRADE-RAT method itself.

New Environment Variables

Quite a few environment variables are already checked by Rakudo whenever it starts. Two more were added in the past year:

RAKUDO_MAX_THREADS

This environment variable can be set to indicate the maximum number of OS-threads that Rakudo may use for its thread pool. The default is 64, or the number of CPU-cores times 8, whichever is larger. Apart from a numerical value, you can also specify "Inf” or "unlimited" to indicate that Rakudo should use as many OS-threads as it can.

These same values can also be used in a call to ThreadPoolScheduler.new with the :max_threads named argument.

my $*SCHEDULER =
  ThreadPoolScheduler.new(:max_threads<unlimited>);

INSIDE_EMACS

This environment variable can be set to a true value if you do not want the REPL to check for installed modules to handle editing of lines. When set, it will fallback to the behaviour as if none of the supported line editing modules are installed. This appears to be handy for Emacs users, as the name implies 🙂

New experimental features

Some Raku features are not yet cast in stone yet, so there’s no guarantee that any code written by using these experimental features, will continue to work in the future. Two new experimental features have been added in the past year:

:will-complain

If you add a use experimental :will-complain to your code, you can customize typecheck errors by specifying a will complain trait. The trait expects a Callable that will be given the offending value in question, and is expected to return a string to be added to the error message. For example:

use experimental :will-complain;
my Int $a will complain { "You cannot use -$_-, dummy!" }
$a = "foo";
# Type check failed in assignment to $a; You cannot use -foo-, dummy!

The will complain trait can be used anywhere you can specify a type constraint in Raku, so that includes parameters and attributes.

:rakuast

The RakuAST classes allow you to dynamically build an AST (Abstract Syntax Tree programmatically, and have that converted to executable code. What was previously only possible by programmatically creating a piece of Raku source code (with all of its escaping issues), and then calling EVAL on it. But RakuAST not only allows you to build code programmatically (as seen in yesterday’s blog post), it also allows you to introspect the AST, which opens up all sorts of syntax / lintifying possibilities.

There is an associated effort to compile the Raku core itself using a grammar that uses RakuAST to build executable code. This effort is now capable of passing 585/1355 test-files in roast completely, and 83/131 of the Rakudo test-files completely. So still a lot of work to do, although it has now gotten to the point that implementation of a single Raku feature in the new grammar, often creates an avalanche of now passing test-files.

So, if you add a use experimental :rakuast to your code, you will be able to use all of the currently available RakuAST classes to build code programmatically. This is an entire new area of Raku development, which will be covered by many blog posts in the coming year. As of now, there is only some internal documentation.

A small example, showing how to build the expression "foo" ~ "bar":

use experimental :rakuast;

my $left  = RakuAST::StrLiteral.new("foo");
my $infix = RakuAST::Infix.new("~");
my $right = RakuAST::StrLiteral.new("bar");

my $ast = RakuAST::ApplyInfix.new(:$left, :$infix, :$right);
dd $ast;  # "foo" ~ "bar"

This is very verbose, agreed. Syntactic sugar for making this easier will certainly be developed, either in core or in module space.

Note how each element of the expression can be created separately, and then combined together. And that you can call dd to show the associated Raku source code (handy when debugging your ASTs).

For the very curious, you can check out a proof-of-concept of the use of RakuAST classes in the Rakudo core in the Formatter class, that builds executable code out of an sprintf format.

New arguments to existing functionality

roundrobin(…, :slip)

The roundrobin subroutine now also accepts a :slip named argument. When specified, it will produce all values as a single, flattened list.

say roundrobin (1,2,3), <a b c>;        # ((1 a) (2 b) (3 c))
say roundrobin (1,2,3), <a b c>, :slip; # (1 a 2 b 3 c)

This is functionally equivalent to:

say roundrobin((1,2,3), <a b c>).map: *.Slip;

but many times more efficient.

Cool.chomp($needle)

The .chomp method by default any logical newline from the end of a string. It is now possible to specify a specific needle as a positional argument: only when that is equal to the end of the string, will it be removed.

say "foobar".chomp("foo"); # foobar
say "foobar".chomp("bar"); # foo

It actually works on all Cool values, but the return value will always be a string:

say 427.chomp(7); # 42

DateTime.posix

DateTime value has better than millisecond precision. Yet, the .posix method always returned an integer value. Now it can also return a Num with the fractional part of the second by specifying the :real named argument.

given DateTime.now {
    say .posix;        # 1671733988
    say .posix(:real); # 1671733988.4723697
}

Additional meaning to existing arguments

Day from end of month

The day parameter to Date.new and DateTime.new (whether named or positional) can now be specified as either a Whatever to indicate the last day of the month, or as a Callable indicating number of days from the end of the month.

say Date.new(2022,12,*);   # 2022-12-31
say Date.new(2022,12,*-6); # 2022-12-25

Additions in v6.e.PREVIEW

You can already access new v6.e language features by specifying use v6.e.PREVIEW at the top of your compilation unit. Several additions were made the past year!

term nano

nano term is now available. It returns the number of nanoseconds since midnight UTC on 1 January 1970. It is similar to the time term but one billion times more accurate. It is intended for very accurate timekeeping / logging.

use v6.e.PREVIEW;
say time; # 1671801948
say nano; # 1671801948827918628

With current 64-bit native unsigned integer precision, this should roughly be enough for another 700 years 🙂

prefix //

You can now use // as a prefix as well as an infix. It will return whatever the .defined method returns on the given argument).

use v6.e PREVIEW;
my $foo;
say //$foo; # False
$foo = 42;
say //$foo; # True

Basically //$foo is syntactic sugar for $foo.defined.

snip() and Any.snip

The new snip subroutine and method allows one to cut up a list into sublists according the given specification. The specification consists of one or more smartmatch targets. Each value of the list will be smartmatched with the given target: as soon as it returns False, will all the values before that be produced as a List.

use v6.e.PREVIEW;
say (2,5,13,9,6,20).snip(* < 10);
# ((2 5) (13 9 6 20))

Multiple targets can also be specified.

say (2,5,13,9,6,20).snip(* < 10, * < 20);
# ((2 5) (13 9 6) (20))

The argument can also be an Iterable. To split a list consisting of integers and strings into sublists of just integers and just strings, you can do:

say (2,"a","b",5,8,"c").snip(|(Int,Str) xx *);
# ((2) (a b) (5 8) (c))

Inspired by Haskell’s span function.

Any.snitch

The new .snitch method is a debugging tool that will show its invocant with note by default, and return the invocant. So you can insert a .snitch in a sequence of method calls and see what’s happening “half-way” as it were.

$ raku -e 'use v6.e.PREVIEW;\
say (^10).snitch.map(* + 1).snitch.map(* * 2)'
^10
(1 2 3 4 5 6 7 8 9 10)
(2 4 6 8 10 12 14 16 18 20)

You can also insert your own “reporter” in there: the .snitch method takes a Callable. An easy example of this, is using dd for snitching:

$ raku -e 'use v6.e.PREVIEW;\
say (^10).snitch(&dd).map(*+1).snitch(&dd).map(* * 2)'
^10
(1, 2, 3, 4, 5, 6, 7, 8, 9, 10).Seq
(2 4 6 8 10 12 14 16 18 20)

Any.skip(produce,skip,…)

You can now specify more than one argument to the .skip method. Before, you could only specify a single (optional) argument.

my @a = <a b c d e f g h i j>;
say @a.skip;       # (b c d e f g h i j)
say @a.skip(3);    # (d e f g h i j)
say @a.skip(*-3);  # (h i j)

On v6.e.PREVIEW, you can now specify any number of arguments in the order: produce, skip, produce, etc. Some examples:

use v6.e.PREVIEW;
my @a = <a b c d e f g h i j>;
# produce 2, skip 5, produce rest
say @a.skip(2, 5);        # (a b h i j)
# produce 0, skip 3, then produce 2, skip rest
say @a.skip(0, 3, 2);     # (d e)
# same, but be explicit about skipping rest
say @a.skip(0, 3, 2, *);  # (d e)

In fact, any Iterable can now be specified as the argument to .skip.

my @b = 3,5;
# produce 3, skip 5, then produce rest
say @a.skip(@b);           # (a b c i j)
# produce 1, then skip 2, repeatedly until the end
say @a.skip(|(1,2) xx *);  # (a d g j)

Cool.comb(Pair)

On v6.e.PREVIEW, the .comb method will also accept a Pair as an argument to give it .rotor_-like capabilities. For instance, to produce trigrams of a string, one can now do:

use v6.e.PREVIEW;
say "foobar".comb(3 => -2);  # (foo oob oba bar)

This is the functional equivalent of "foobar".comb.rotor(3 => -2)>>.join, but about 10x as fast.

Changed semantics on Int.roll|pick

To pick a number from 0 till N-1, one no longer has to specify a range, but can use just the integer value as the invocant:

use v6.e.PREVIEW;
say (^10).roll;     # 5
say 10.roll;        # 7
say (^10).pick(*);  # (2 0 6 9 4 1 5 7 8 3)
say 10.pick(*);     # (4 6 1 0 2 9 8 3 5 7)

Of course, all of these values are examples, as each run will, most likely, produce different results.

More interesting stuff

There were some more new things and changes the past year. I’ll just mention them very succinctly here:

New methods on CompUnit::Repository::Staging

.deploy.remove-artifacts, and .self-destruct.

:!precompile flag on CompUnit::Repository::Installation.install

Install module but precompile on first loading rather than at installation.

New methods on Label

.file and .line where the Label was created.

.Failure coercer

Convert a Cool object or an Exception to a Failure. Mainly intended to reduce binary size of hot paths that do some error checking.

Cool.Order coercer

Coerce the given value to an Int, then convert to Less if less than 0, to Same if 0, and More if more than 0.

Allow semi-colon

Now allow for the semi-colon in my :($a,$b) = 42,666 because the left-hand side is really a Signature rather than a List.

Summary

I guess we’ve seen one big change in the past year, namely having experimental support for RakuAST become available. And many smaller goodies and tweaks and features.

Now that RakuAST has become “mainstream” as it were, we can think of having certain optimizations. Such as making sprintf with a fixed format string about 30x as fast! Exciting times ahead!

Hopefully you will all be able to enjoy the Holiday Season with sufficient R&R. The next Raku Advent Blog is only 340 days away!