Day 14 – Thinking Beyond Types: an Introduction to Rakudo’s MOP

It’s Christmas season! Christmas would not be Christmas without the caroling that’s part of the festivities, so let’s make it possible to sing some.

We could simply make a carol one giant string, but that’s not good enough. Being a song, carols often have a chorus that’s repeated in between verses. If we were to store it as one string, we’d be repeating ourselves. On top of that, people are not perfect; they may forget a verse of a carol, or even just a single line of a verse. We need a type to represent a carol. This could be a type of song, but since we only care about carols, it’s a bit early to abstract this out.

Now, to make this more interesting, let’s handle this without making instances of any type of any kind. All behaviour for all Christmas carols will be handled by type objects. This will be enforced using the Uninstantiable REPR.

At first, we might have a Christmas::Carol role:

role Christmas::Carol is repr<Uninstantiable> {
    proto method name(::?CLASS:U: --> Str:D)        {*}
    proto method verse(::?CLASS:U: Int:D --> Seq:D) {*}
    proto method chorus(::?CLASS:U: --> Seq:D)      {*}
    proto method lyrics(::?CLASS:U: --> Seq:D)      {*}

    method sing(::?CLASS:U: --> ::?CLASS:U) {
        .say for @.lyrics;
        self
    }
}

This would then be done by a class representing a specific carol:

class Christmas::Carol::JingleBells does Christmas::Carol {
    multi method name(::?CLASS:U: --> 'Jingle Bells') { }

    multi method verse(::?CLASS:U: 1 --> Seq:D) {
        lines q:to/VERSE/
        Dashing through the snow
        In a one-horse open sleigh
        O'er the fields we go
        Laughing all the way
        Bells on bobtails ring
        Making spirits bright
        What fun it is to ride and sing
        A sleighing song tonight
        VERSE
    }
    multi method verse(::?CLASS:U: 2 --> Seq:D) {
        lines q:to/VERSE/
        A day or two ago
        I thought I'd take a ride
        And soon, Miss Fanny Bright
        Was seated by my side
        The horse was lean and lank
        Misfortune seemed his lot
        He got into a drifted bank
        And then we got upset
        VERSE
    }
    multi method verse(::?CLASS:U: 3 --> Seq:D) {
        lines q:to/VERSE/
        A day or two ago
        The story I must tell
        I went out on the snow
        And on my back I fell
        A gent was riding by
        In a one-horse open sleigh
        He laughed as there I sprawling lie
        But quickly drove away
        VERSE
    }
    multi method verse(::?CLASS:U: 4 --> Seq:D) {
        lines q:to/VERSE/
        Now the ground is white
        Go it while you're young
        Take the girls tonight
        And sing this sleighing song
        Just get a bobtailed bay
        Two forty as his speed
        Hitch him to an open sleigh
        And crack, you'll take the lead
        VERSE
    }

    multi method chorus(::?CLASS:U: --> Seq:D) {
        lines q:to/CHORUS/
        Jingle bells, jingle bells
        Jingle all the way
        Oh, what fun it is to ride
        In a one-horse open sleigh, hey
        Jingle bells, jingle bells
        Jingle all the way
        Oh, what fun it is to ride
        In a one-horse open sleigh
        CHORUS
    }

    multi method lyrics(::?CLASS:U: --> Seq:D) {
        gather for 1..4 {
            take $_ for @.verse($_);
            take "";
            take $_ for @.chorus;
            take "" if $_ != 4;
        }
    }
}

There’s a problem with this approach, though. What happens if you want to hold onto a collection of Christmas carols to carol around the neighbourhood with?

use Christmas::Carol::JingleBells;
use Christmas;:Carol::JingleBellRock;
use Christmas::Carol::DeckTheHalls;
use Christmas::Carol::SilentNight;
# And so on...

That’s no good! You don’t need to know who wrote a Christmas carol in order to sing it. On top of that, no one thinks of Christmas carols in terms of symbols; they think of them in terms of their name. To represent them effectively, we need to make it so we can look up a Christmas carol using its name, while also making it possible to introduce new carols that can be looked up this way at the same time. How can we do this?

The way we’ll be using here requires a bit of explanation on how types in Raku work.

The Metaobject Protocol

Classes may contain three different types of method declarations. The two you most commonly see are public and private methods:

class Example {
    method !private-method(|) { ... }
    method public-method(|)   { ... }
}

There is a third type of method declaration you can make, which is exclusive to classes (this a lie, but this is the case when writing Raku with Rakudo alone), by prefixing the method’s name with ^. Calls to these are typically made using the .^ dispatch operator, which you often see when you need to introspect an object (.^name, .^methods, etc.). However, these don’t behave at all like you would otherwise expect a method to. Let’s take a look at what the invocant and parameters are when we invoke a method of this type using the .^ dispatch operator:

class Example {
    method ^strange-method(\invocant: |params --> List:D) {
        (invocant, params)
    }
}

say Example.^strange-method;
# OUTPUT:
# (Perl6::Metamodel::ClassHOW+{<anon>}.new \(Example))

Whoa whoa whoa, WTF? Why is the class this type of method is declared in its first parameter instead of its invocant? What even is that object that ended up as its invocant instead, and where is it coming from?

Before this can be explained, first we’ll need to understand a little bit about what the metaobject protocol (MOP) is. The MOP is a feature specific to Rakudo through which the behaviour for all objects that can exist in Raku are implemented. These are implemented based on kinds (types of types), such as classes, roles, and grammars. The behaviour for any type is driven by what is called a higher-order working (HOW), which is a metaobject. These are typically instances of a metaclass of some sort. For instance, HOWs for classes are created by the Metamodel::ClassHOW metaclass.

The HOW for any given object can be introspected. How is this done? How, you ask? How? HOW!? By calling HOW on it, of course!

role Foo { }
say Foo.HOW.^name; # OUTPUT: Metamodel::ParametricRoleGroupHOW

Methods of HOWs are called metamethods, and these are what are used to handle the various behaviours that types can support. Some examples of behaviours of kinds that metamethods handle include type names, attributes, methods, inheritance, parameterization, and typechecking. Since most of these are not features specific to any one kind, these are often mixed into metaclasses by metaroles. For instance, the Metamodel::Naming metarole is what handles naming for any type that can be named.

So that third type of method declaration from earlier? That doesn’t actually declare a method for a class; instead, it declares a metamethod that gets mixed into that class’ HOW, similarly to how metaroles are used. The .^ dispatch operator is just sugar for invoking a metamethod of an object using that object as its first argument, which metamethods accept in most cases. For instance, these two metamethod calls are equivalent:

say Int.^name;         # OUTPUT: Int
say Int.HOW.name(Int); # OUTPUT: Int

Metamethods are the tool we’ll be using to implement Christmas carols purely as types.

Spreading the Joy

To start, instead of having a Christmas::Carol role that gets done by Christmas carol classes, let’s make our carols roles mixed into a Christmas::Carol class instead. Through this class, we will stub the methods a Christmas carol should have, like it was doing as a role, but in addition will hold on to a dictionary of Christmas carols by their name.

We can store carols using an add_carol metamethod:

my constant %CAROLS = %();
method ^add_carol(Christmas::Carol:U, Str:D $name, Mu $carol is raw --> Mu) {
    %CAROLS{$name} := $carol;
}

Now we can mark roles as being carols like so:

role Christmas::Carol::JingleBells { ... }
Christmas::Carol.^add_carol: 'Jingle Bells', Christmas::Carol::JingleBells;

This isn’t a great API for people to use though. A trait could make it so this could be handled from a role’s declaration. Let’s make an is carol trait for this:

multi sub trait_mod:<is>(Mu \T, Str:D :carol($name)!) {
    Christmas::Carol.^add_carol: $name, T
}

Now we can define a role as being a carol like this instead:

role Christmas::Carol::JingleBells is carol('Jingle Bells') { ... }

To make it so we can fetch carols by name, we can simply make our Christmas::Carol class parametric. This can be done by giving it a parameterize metamethod which, given a name, will create a Christmas::Carol mixin using any carol we know about by that name:

method ^parameterize(Christmas::Carol:U $this is raw, Str:D $name --> Christmas::Carol:U) {
    self.mixin: $this, %CAROLS{$name}
}

Now we can retrieve our Christmas carols by parameterizing Christmas::Carol using a carol name. What will the name of the mixin type returned be, though?

say Christmas::Carol['Jingle Bells'].^name;
# OUTPUT: Christmas::Carol+{Christmas::Carol::JingleBells}

That’s a bit ugly. Let’s reset the mixin‘s name during parameterization:

method ^parameterize(Christmas::Carol:U $this is raw, Str:D $name --> Christmas::Carol:U) {
    my Christmas::Carol:U $carol := self.mixin: $this, %CAROLS{$name};
    $carol.^set_name: 'Christmas::Carol[' ~ $name.perl ~ ']';
    $carol
}

This gives our Jingle Bells carol a name of Christmas::Carol["Jingle Bells"] instead. Much better.

Let’s add one last metamethod: carols. This will return a list of names for the carols known by Christmas::Carol:

method ^carols(Christmas::Carol:U --> List:D) {
    %CAROLS.keys.list
}

With that, our Christmas::Carol class is complete:

class Christmas::Carol is repr<Uninstantiable> {
    proto method name(::?CLASS:U: --> Str:D)        {*}
    proto method chorus(::?CLASS:U: --> Seq:D)      {*}
    proto method verse(::?CLASS:U: Int:D --> Seq:D) {*}
    proto method lyrics(::?CLASS:U: --> Seq:D)      {*}

    method sing(::?CLASS:U: --> ::?CLASS:U) {
        .say for @.lyrics;
        self
    }

    my constant %CAROLS = %();
    method ^add_carol(Christmas::Carol:U, Str:D $name, Mu $carol is raw --> Mu) {
        %CAROLS{$name} := $carol;
    }
    method ^carols(Christmas::Carol:U --> List:D) {
        %CAROLS.keys.list
    }
    method ^parameterize(Christmas::Carol:U $this is raw, Str:D $name --> Christmas::Carol:U) {
        my Christmas::Carol:U $carol := self.mixin: $this, %CAROLS{$name};
        $carol.^set_name: 'Christmas::Carol[' ~ $name.perl ~ ']';
        $carol
    }
}

multi sub trait_mod:<is>(Mu \T, Str:D :carol($name)!) {
    Christmas::Carol.^add_carol: $name, T
}

Now, this is great and all, but how is this an improvement on our original code? By defining carols this way, we no longer need to know the symbol for a carol in order to sing it, and we no longer need to know which module even declared the carol in the first place. So long as we know that the Christmas::Carol class exists, we know all of the carols all of the modules we import happen to be aware of.

This means there can be a module defining a collection of carols:

use Christmas::Carol::JingleBells;
use Christmas::Carol::JingleBellRock;
use Christmas::Carol::DeckTheHalls;
use Christmas::Carol::SilentNight;
unit module Christmas::Carol::Collection;

From another module, we can make another collection using this, and define more carols:

use Christmas::Carol::Collection;
use Christmas::Carol::JingleBells::BatmanSmells;
unit module Christmas::Carol::Collection::Plus;

We can then import this and easily sing all of the original module’s carols, in addition to the ones this new module adds, by name:

use Christmas::Carol;
use Christmas::Carol::Collection::Plus;

Christmas::Carol[$_].sing for Christmas::Carol.^carols;

At this point, you may be wondering: “Couldn’t you just write code that does the same thing as this using instances?”. You’d be right! What this shows is that while there is a protocol for working with types when using Rakudo, the behaviour of any given type isn’t particularly unique; it’s driven mainly by metaobjects that you can have complete control over.

Just with metamethod declarations in classes alone, you can augment or override behaviours of any type that supports inheritance. This is far from the extent of what the MOP allows you to do with types! Alas, the more advanced features for working with the MOP would require more explanation, and would be best left for another time.