RFC 112 by Richard Proctor: Assignment within a regex

Richard wanted to

Provide a simple way of naming and picking out information from a regex without having to count the brackets.

I can say without hesitation that Raku (and before its rename, Perl 6) has achieved this goal — but all the details are different than proposed.

The reason is two-fold.

For one, Richard assumed a pretty straight-forward extension to Perl 5’s regex syntax, and his proposed syntax, (?$hours=..) for a named capture made sense. Instead, Raku regex syntax is pretty much a new thing, where all non-alphanumeric characters are potentially meta characters, and thus either used or reserved for one purpose or another. This made easier syntax than (?$name=regex) available for named captures.

The second is even more profound: The Raku designers realized that regexes could only be truly powerful if reuse was built in from the ground up. And the best way to make that happen was to make them first class.

I want to dwell on that point a bit: consider the power of functions (and closures) as first-class citizens in modern programming languages. Lisp has shown us what you can do with them, and now basically every programming language has got them. Dynamic languages like Perl, Ruby, Javascript and Python were pretty early adopters, modern statically typed languages like C# and F# also got them; even Java caught up eventually. Java didn’t even have functions, just methods, and now it’s got closures that you can pass around.

In my humble opinion, raising regexes to the level of first-class citizens and introducing a concise call syntax gave regexes a similar boost.

In the old days, it was common wisdom that you cannot parse XML (or other arbitrarily nested languages) with regexes, because they are not a regular language in the computer science sense. Perl 5 has some workarounds for that, but they are so clunky and verbose that I haven’t even seen them recommended much, and my general impression is that if you use them, it’s just for the lack of good alternatives.

Not so in Raku: <subrule> in a regex calls another regex called subrule, and so you have recursion (and, relevant to the discussion of RFC 112, named captures). This recursion moves regexes from regular into context-free language territory in the Chomsky Hierarchy. But more than recursion, the named regexes allow much easier reuse, testing in isolation and all that other wonderful stuff that first-classiness gave to functions. It also moved the sentiment towards parsing XML and other languages with regexes from “are you serious?” to “sure, it’s the best tool”.

The call syntax <subrule> implies a named capture, and it turns out that’s convenient enough that explicit named captures (not tied to a call) are actually pretty rare in real-world parsers. An explicit syntax for that exists though, it’s $<capturename>=[...].

Which brings us to the second interesting bit: RFC 112 doesn’t just talk about named captures, but implies that they are directly stored into variables of the same name.

This is problematic for a variety of reasons:

  • Scoping. A regex can be declared and used at two very different parts of the program. Forcing the variable to be in scope would make the capture syntax a source variables with an unnecessary large scope (dare I say global?), which is a clear anti pattern.
  • Quantifiers. In RFC 211 syntax, what would have happened with a regex like (?$char=.)+ matching the string abc? What’s in $char? The sigil implies a scalar, so… maybe the last capture, c? And throw away all the other matches? Doesn’t sound too good. Or maybe (?@char=.) would have an array with all captures, but then, when writing a regex, you’d have to know if anybody later wants to use that inside a quantifier. Not stellar either.
  • Composition. Binding matches to a variable assumes the regex is used as a top-level construct, and not part of larger thing.
  • Recursion. Do I even need to elaborate? Probably not.

The solution that’s implemented in Raku now is more suited to world of first-class regexes: for each regex match there’s a Match object. The top-level match object is stored in the variable $/, so accessing a named capture key is $/<key>, and there’s even a short-hand for that, $<key>. Just two character longer than the originally proposed $key.

This solution shines though when used in the context of composition, for example. Since the named capture corresponds to a regex match, it’s also a Match object, and so we arrive at a tree of matches (all alike). Or rephrased: a regex match already is a syntax tree.


I think this RFC is a good example how a real pain point and problem was identified, and a solution proposed. Aspects of this solution have survived the language design process, but most details haven’t, because the language changed much more than it barely being Perl 5 plus a few extensions through RFCs.

I have been participating in the Perl 6 project since around the year 2007, and have watched some of these transformations; for regexes, the majority of the consolidation and redesign work had already been done. I watched the implementations become more powerful, and even helped a little here and there.

Living in this process was a magical experience, just as magical as the result is now.

RFC1: Threads

RFC 1, by Bryan C. Warnock: Threads

It might or might not be the case that the need for a real multithreaded architecture in Perl was the real motive behind the creation of what was initially called simply Perl, then Perl 6, and eventually Raku.

It was probably late 90s or early 10s, when we had a contract with a big company that needed to download stuff from the web really fast. We needed those threads, and they finally arrived in Perl 5.8.8. However, our threads were very basic, didn’t need any kind of communication, just the bare parallel thing, and underneath them, operating system processes were used; there were no real threads at the Perl VM level. And they were sorely needed. Which is why RFC1 read:

Implementation of Threads in Perl

It was originally proposed on August 1st (hence the 20th aniversary thing), and finally frozen a couple of month later, by September 28th.

It basically proposes a way to implement low-level threads, including new namespaces (global, for sharing variables among threads) as well as the Threads class, with this example:

use Threads;
# the main thread has all four above in its arena

my $thread2 = Threads->new(\&start_thread2);
#...

sub start_thread2 { ... }

The main thread is implicit, and gets all other modules into its namespace, the second one inherits from the main thread. It makes sense, in general, except it’s a very low level mechanism to use threads, and in fact it looks more like a way to handle processes than what we call nowadays threads. There’s another RFC for those, which are called “lightweight threads”, which was started a few week later and frozen pretty much at the same time. It contains the graphic simile:

Perl → Swiss-army chain saw; Perl with threads → juggling chain saws

It’s difficult to see what’s the difference between them, except for the explicit sharing of variables and the fact that it uses Thread instead of Threads as the main class.

Eventually, that was the keyword chosen for threads in Raku: Thread. This uses new to create a thread, but you have then to issue a .run to actually run it. Alternatively, you can simply use .start to create and run a thread inmediately.

This is taken pretty much directly from the example in the Thread manual page, and shows the differences between Raku and what its early inceptions looked like. It uses a map to start 10 threads (using a Range); every thread will work on a range of numbers to check if there’s a prime in then. After cribbing out a few easy ones, it will simply check, using the is-prime function, if the number is prime, and it will print the number and the thread it’s in. The $*THREAD variable allows for easy introspection of the thread one is in, which will make this print something like this:

...
Prime 76579 found in Thread<4>(Checking primes from 0 to 100000)
Prime 994997 found in Thread<13>(Checking primes from 900000 to 1000000)
Prime 655043 found in Thread<10>(Checking primes from 600000 to 700000)
Prime 483991 found in Thread<8>(Checking primes from 400000 to 500000)
Prime 169283 found in Thread<5>(Checking primes from 100000 to 200000)
Prime 995009 found in Thread<13>(Checking primes from 900000 to 1000000)
Prime 761533 found in Thread<11>(Checking primes from 700000 to 800000)
...

Every thread has, by design, specialized in a specific range; thread number 13 gets from 900K to 1000K, for instance. Working with threads is much more efficient, but a process needs to be pinned to a specific thread to do this. This is why low-level thread access is not really the best way to create a concurrent program. Working with higher-level APIs makes a lot of more sense.

However, in 2000 it was enough to have the insight that a thread engine was needed for a modern, 100-year language like Raku. And Bryan C. Warnock, who became famous because of the Warnock’s Dilemma, had, if not the insight of the original idea, at least the laziness, impatience and hubris of putting it down in what eventually became the first RFC for Raku, 20 years ago today.

The origin of Warnock’s dilemma, according to Wikipedia, is pretty much in the same month, and actually originated in the bootstrap (for perl6) mailing list. And it is totally related to the fact that the response to that RFC was underwhelming, which indicates that either no one cared, or it was just perfect. I tend to think the latter, so thanks, Bryan, for this.

Threads on the world

20 years ago tomorrow

20 years ago, on the first of August, the inception of a language started to, well, incept.

Actually, it started a bit earlier than that. Perl was in need of change, so it was decided that the community itself should propose what the language needed to do to go forward one step, from Perl 5 to Perl 6. A call for requests for change was made; every request for change should include possible changes to Perl, as well as, if possible, an implementation proposal, laying out how to proceed. The procedure didn’t lack criticism, but it can’t be said that it was not received, in general, with such an enthusiasm that August 1st already saw the first RFC, pretty much at the same time as some instructions from Larry Wall on how to actually proceed.

The rest is history. It looks a bit like sacred history, since those RFCs were picked up (and apart) by Larry Wall’s apocalypses, explained later by Damian Conway’s exegeses, and roasted in the synopsis, which eventually became the roast repository, the actual specification of the language.

Which is now called Raku. But that’s another story.

To celebrate this part of the history and the people that brought us where we are now, starting tomorrow, we’ll publish 20 articles, one a day, that will focus on one or a few RFCs and show what they eventually became in today’s Raku. So come back every day for a piece of Raku, of history, and of Raku history!

Happy new year!

This year’s advent calendar is over, and it leaves us lots of articles on metaprogramming, applications, useful Raku modules, how to migrate from Perl, programming techniques and even how to work with Raku inside containers.

No more articles until next year, and the call is already open: if you want to tell something about Raku and its surroundings, add your name to the list with possible title, and you’re in!

Meanwhile, you’ve got raku.org and docs.raku.org for all your Raku needs.

Raku means enjoyment. So we wish you all a lot of Raku for next year!

Day 24: The Grinch of Raku, Part 2: Hold Your Horses

In 2017, the Grinch ruined Christmas by showing off some of the naughty things you can do with Raku’s features. Unfortunately, while his heart grew by three sizes that year, there’s more than one Grinch! This Grinch will be doing something extra naughty this year, taking some inspiration from the JavaScript community.

You may have heard of JSFuck, which is a tool that allows you to write any JavaScript code using only the characters [, ], (, ), +, and !. This is something you’d only expect to be possible in a language like JavaScript, right? That’s not entirely true! To prove this, let’s port it to Raku. Since this can’t be implemented using the exact same set of characters, our restrictions will be that only non-alphanumeric ASCII characters may be used in the translated code, and string literals must not be used.

Generating Primitives

The first thing we’ll need to do is find a way to generate some primitives. The ones from JavaScript that are of interest to us are booleans, numbers, and strings; any other type of primitive can be represented through other means. These are generated mainly through type coercion on empty arrays, which also happens to be possible to do in Raku.

True and False can be generated in Raku using the ! prefix operator, similarly to how you can in JavaScript:

say ![];  # OUTPUT: True
say !![]: # OUTPUT: False

Using this in combination with the + prefix operator, we can generate any whole number, which is also the case in JavaScript:

say +[];         # OUTPUT: 0
say +![];        # OUTPUT: 1
say +![] + +![]; # OUTPUT: 2

In JavaScript, + also happens to be used to concatenate strings. When used with two empty arrays, + will coerce both to strings and concatenate them, which results in an empty string. + doesn’t behave like this in Raku, so we’ll need to use the ~ operator instead:

say (~[]).perl; # OUTPUT: ""

What about strings that aren’t empty though? In JavaScript, strings are iterable, which allows for certain characters to be used when stringifying values other than empty arrays. This isn’t the case in Raku! It’s time to start getting creative.

String bitwise operators allow you to perform the same bitwise operations you can perform on numbers on codepoints in strings. Using the ~^ infix operator, we can generate a null character given 0 and 0:

say ord +[] ~^ +[]; # OUTPUT: 0

We can’t generate the characters we need very easily with the ~+, ~|, and ~^ operators alone though. There is a way to do this using that null character, but we need a lowercase letter of some sort first. We can grab the letter "e" from "True" if we use a regex:

say ~(![] ~~ /...(.)/)[+[]]; # OUTPUT: e

Using an infinite sequence with these two characters, we can generate most of the characters in ASCII:

my Str:D @chars = (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
say @chars[65..90];  # OUTPUT: (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z)
say @chars[97..122]; # OUTPUT: (a b c d e f g h i j k l m n o p q r s t u v w x y z)

Now that we can generate the characters in the string "&chr", we’ll be able to generate any Unicode string after the next step.

Evaluating Code

Most of the JavaScript code that can be generated depends on the Function constructor in order to work. Using it, you can arbitrarily generate a function at runtime. As far as I know, it’s not possible to generate code like this in Raku without using &EVAL. There’s a problem we need to solve if we are to use it, though.

We can use string literals with &EVAL just fine:

say EVAL "'Hello, world!'"; # OUTPUT: Hello, world!

But if we try to use a value that is unknown at compile-time with it, we’ll get an exception warning about the security implications of what we’re doing, telling us to use the MONKEY-SEE-NO-EVAL pragma:

say EVAL my $ = "'Hello, world!'"; # Throws X::SecurityPolicy::Eval

That’s not good in our case! We can’t set this pragma without alphanumeric characters. It’s time to get naughty. What happens if we try to use &EVAL using an indirect symbol lookup?

say ::('&EVAL')(my $ = "'Hello world!'"); # OUTPUT: Hello, world!

Perfect! Along with this, using indirect symbol lookup we can also call the &chr routine to generate a string for any Unicode codepoint. In combination, this allows us to translate any valid Raku code.

Hold Your Horses

We’re ready to start writing code for our port of JSFuck. This will simply be a script that takes some Raku code as input and outputs its translation. All of the subroutines used (apart from &MAIN) will be pure. Now, let’s give this port a bit of a nicer name than the obvious choice and call it Hold Your Horses instead.

Our first subroutine will be &from-uint, which will translate numbers. We could just add 1 to 0 repeatedly until we get the number we’re looking for, but this will generate huge amounts of code for larger codepoints. One way we can shorten the code this generates is if we represent numbers as being products of prime numbers. This can be further shortened by representing prime numbers greater than 5 as being a sum of products of prime numbers:

use Prime::Factor;

sub from-uint(UInt:D $x, Int:D $remainder = 0 --> Str:D) is pure {
    proto sub translate(UInt:D --> Str:D) is pure {*}
    multi sub translate(0 --> '+[]') { }
    multi sub translate(1 --> '+![]') { }
    multi sub translate(UInt:D $x --> Str:D) {
        join ' + ', '+![]' xx $x
    }

    if $x <= 5 {
        my Str:D $translation = $x.&translate;
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    } elsif $x.is-prime {
        from-uint $x - 1, $remainder + 1
    } else {
        my Str:D $translation = $x.&prime-factors».&from-uint.fmt: '(%s)', ' * ';
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    }
}

Now we can implement &from-str, which will parse code input by the user. This needs to map each codepoint in the given code to a Hold Your Horses number, which can be done by looking up a character in the sequence of characters from earlier if it is within its range, otherwise &chr can be called. Since we’re using this sequence every time we see a character that is included by it, this will be stored in $_ by our next subroutine. Since translating a single codepoint can be quite intensive, let’s use the experimental is cached trait with our helper subroutine that handles this to avoid having to do it more than once for any given codepoint:

use experimental :cached;

sub from-str(Str:D $code --> Str:D) is pure {
    my Int:D constant LIMIT = 'z'.ord.succ;

    proto sub translate(UInt:D --> Str:D) is pure is cached {*}
    multi sub translate(UInt:D $codepoint where 0..^LIMIT --> Str:D) {
        sprintf '.[%s]', $codepoint.&from-uint
    }
    multi sub translate(UInt:D $codepoint where LIMIT..* --> Str:D) {
        sprintf '::(%s)(%s)',
                '&chr'.ords».&translate.join(' ~ '),
                $codepoint.&from-uint
    }

    sprintf '::(%s)(%s)',
            '&EVAL'.ords».&translate.join(' ~ '),
            $code.ords».&translate.join(' ~ ')
}

Now we can implement &hold-your-horses, which will handle the full translation of code input by the user. All this needs to do is store the sequence from earlier in $_ before calling &from-str:

sub hold-your-horses(Str:D $code --> Str:D) is pure {
    Qc:to/TRANSLATION/.chomp
    $_ := (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
    {$code.&from-str};
    TRANSLATION
}

With &MAIN added, our script is now complete:

use v6.d;
use experimental :cached;
use Prime::Factor;
unit sub MAIN(Str:D $code) {
    say hold-your-horses $code
}

sub from-uint(UInt:D $x, Int:D $remainder = 0 --> Str:D) is pure {
    proto sub translate(UInt:D --> Str:D) is pure {*}
    multi sub translate(0 --> '+[]') { }
    multi sub translate(1 --> '+![]') { }
    multi sub translate(UInt:D $x --> Str:D) {
        join ' + ', '+![]' xx $x
    }

    if $x <= 5 {
        my Str:D $translation = $x.&translate;
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    } elsif $x.is-prime {
        from-uint $x - 1, $remainder + 1
    } else {
        my Str:D $translation = $x.&prime-factors».&from-uint.fmt: '(%s)', ' * ';
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    }
}

sub from-str(Str:D $code --> Str:D) is pure {
    my Int:D constant LIMIT = 'z'.ord.succ;

    proto sub translate(UInt:D --> Str:D) is pure is cached {*}
    multi sub translate(UInt:D $codepoint where 0..^LIMIT --> Str:D) {
        sprintf '.[%s]', $codepoint.&from-uint
    }
    multi sub translate(UInt:D $codepoint where LIMIT..* --> Str:D) {
        sprintf '::(%s)(%s)',
                '&chr'.ords».&translate.join(' ~ '),
                $codepoint.&from-uint
    }

    sprintf '::(%s)(%s)',
            '&EVAL'.ords».&translate.join(' ~ '),
            $code.ords».&translate.join(' ~ ')
}

sub hold-your-horses(Str:D $code --> Str:D) is pure {
    Qc:to/TRANSLATION/.chomp
    $_ := (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
    {$code.&from-str};
    TRANSLATION
}

Now, does this actually work? For brevity’s sake, let’s say it works as intended if say "Hello, world! 👋" can be translated and run:

bastille% raku hold-your-horses.raku 'say "Hello, world! 👋"' > hello-world.raku
bastille% raku hello-world.raku
Hello, world! 👋

Perfect! This is the script’s output:

$_ := (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
::(.[(+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) + +![])] ~ .[(+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) + +![]) + +![])] ~ .[(+![] + +![] + +![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])])(.[(+![] + +![] + +![] + +![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) + +![]] ~ .[((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![])] ~ .[(+![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![])] ~ .[((+![] + +![]) * (+![] + +![] + +![]) + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) * (+![] + +![] + +![] + +![] + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![])] ~ ::(.[(+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])])((+![] + +![] + +![] + +![] + +![]) * (+![] + +![] + +![] + +![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) + +![]) + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])) ~ .[(+![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) + +![])]);

Wrapping Up

Raku is quite a large language with an extensive set of features. These can be combined in some very interesting ways! Here, using a combination of type coercion, string bitwise operators, regexen, sequences, indirect symbol lookup, and a loophole with &EVAL, we were able to be naughty Grinches again this year and port JSFuck from JavaScript. If you’re tempted to say something is impossible to write in Raku, hold your horses; it may very well be possible to do with the right tools.

Day 23 – A Raku Advent Helper

Introduction

I have been writing Raku Advent articles annually since 2016, and it’s always been a struggle for me to get a reliable transformation of my source file into the Raku Advent WordPress (WP) website without something getting changed by WP. Then, the menus are terrible and editing can be troublesome. In this article I hope to show how the situation can be improved.

Background

The great name change to Raku this year unfortunately happened late in the year and there was not a lot of time to get a new Raku Advent website ready. Consequently, theme selection and tweaking, confusion over the actual Raku Advent website link, and unfortunate article cancellations were wrinkles in the normally smoother process. However, we plan to improve the website before the 2020 Advent season, and also get commitments earlier with concrete drafts available sooner. In the meantime, in this hastily prepared stand-in article, I will go into a bit of detail on some help we hope to offer.

Article creation

Since my first experience with WordPress, I have found these things that make it awkward for me to use WP:

  • Small editing window
  • Noticeable lag time while editing increases fumble-finger errors
  • Enter the desired schedule time in the website time zone (TZ) but see it displayed in your local TZ (with little or no hint as to what you are seeing) [See Note 1]
  • Confusing editing contexts and widget placements

I’m sure most of my problems with WP are self-induced, but I do prefer a more TeX-like document production work flow.

Prior years

In past years I’ve created the articles in Gihub-flavored markdown, manually (with the assistance of my Emacs editor) converted each paragraph to single, long lines, and then posted it in a Github gist. After that, I used the tool p6advent-md2html.p6, developed by @zoffix [Note 2] and modified by @SimonProctor, to extract the html from Github’s representation of the markdown which results in a nice highlighting of code blocks. Finally, that html is copied and pasted into WP and a publishing schedule set up. That original process is outlined here:

  1. Write the post in Github-favored Markdown text
  2. Collapse each paragraph to one long line
  3. Paste the source into a Github gist
  4. Use the existing Advent tool to extract the resulting html representation to one’s local computer
  5. Copy the html and paste it into the blank, html view of the selected WP editor
  6. View the finished product and check for errors

If errors are found:

  1. Correct the errors in the WP editor

OR

  1. Correct the errors in the source
  2. Repeat steps 2 through 6 again

That process is not so bad the first time through it, but when, inevitably, errors are found, one has the choice of either manually editing it on WP or modifying the source and going through the entire process again! Neither choice is very good.

2019’s Advent goal: reduce WP pain

I decided this year to help my article-creation situation so I created a Raku tool to eliminate some of the problems. It’s available to the public as of today:

$ zef install RakuAdvent::WordPress

That module provides the tool make-wp-input. So my new steps as of this year:

  1. Write the post in raw html
  2. Run my new Advent tool (make-wp-input) to format the source into WP-acceptable html
  3. Copy the html and paste it into the blank, html view of the selected WP editor
  4. View the finished product and check and correct for errors

If errors are found:

  1. Correct the errors in the WP editor

OR, preferably,

  1. Correct the errors in the source
  2. Repeat steps 2 through 4 again

Thus, in my new process, I’ve eliminated a couple of steps, but I still have to copy/paste my clean WP source into the WordPress editor—but that’s because I have not taken advantage of the available APIs from WordPress and Github to do the drudge work.

However, in spite of other limitations, the new tool has been a huge help in easing the use of live code examples in an article. In my sandbox where I write my article, I create the code samples in their own files and then add the

<!-- insert file-name lang -->

lines as needed in the location needed. That way, I can edit the live code and test it to make sure it works, but don’t have to change the source using that code.

Tips for Raku authors

Here are some ideas I’ve found helpful while developing articles for the Raku Advent:

  • See help for the WP scheduling calendar in this video
  • Take advantage of your default personal WP website to experiment
  • View the finished product as printed PDF, and check and correct for errors (this has been a very good way for me to look over my article at leisure while sipping an egg nog with my BF “by the fire.” 😊; see Ref. 1 for an excellent html-to-pdf converter)

Wish lists

Here are some things I hope to do with make-wp-input in the New Year:

  1. Convert html source to Github-flavored markdown
  2. Handle html tables
  3. Allow paragraphs in the source html to be recognized by either blank lines above and below the text or a line with a closing tag on the line before the text or an opening tag on the line following the text
  4. Use Github’s APIs [Ref. 2] to manipulate markdown source to a Github gist and get html results back from it
  5. Use WordPress’s APIs [Ref. 3] to manipulate one’s article on WordPress (including setting or updating the publication schedule)

And here are some things I hope the community can do (or at least agree upon) for the Raku Advent website:

  1. Improve the theme and code styling.
  2. Use the old Perl 6 Advent theme?
  3. Sign up for article slots earlier in the year, and start the article (at least in skeleton form) as a scheduled one on the Raku Advent website.

Summary

This year has seen a lot of changes in the Raku community, especially with the name change, and not all are done yet. One area that still needs work is improving the new Raku Advent website. We also hope to make it easier to create and post Raku Advent articles as well as get more participation. Note the 2020 schedule is open now, so you can get your slot early and avoid last minute shopping, er, Raku Adventing!

I ❤️ Raku! 😊

Merry Christmas and a Happy, Blessed New Year to all!


APPENDIX


Notes

  1. I have filed an issue with WordPress to help with time zone identification in the scheduling calendar.
  2. Names preceded by @ are IRC or Github aliases.

References

  1. wkhtmltopdf (available as a Debian package)
  2. Github API
  3. WordPress API

Raku modules used

  • RakuAdvent::WordPress (v.0.0.2)

Day 22: Off Course

You may not have heard about my Perl 6 Courses, and I don’t blame you.

It has been quite a journey.

It started in September 2018 with my Perl6 In 45+45 Minutes introduction to Perl 6 at the Nordic Perl Workshop in Oslo. The very first time a held a presentation at a conference…

I got positive feedback, and wondered if I could build on it. The idea of a full blown course matured, and I started working on the accompanying textbook first.

The book and course is meant as an introduction to Raku, for people already familiar with programming.

I pitched the course to PerlCon 2019 in Riga, and they accepted it. The organiser asked me to promote it, and the result was my Perl 6 blog Perl 6 Musings (at the absolutely fantastic address «perl6.eu»).

Unfortunately that didn’t work out, and the course was cancelled due to too few participants.

  Beginning Raku, 1. Edition (December 2019)  

Pages: 370

File size: ~ 11 Mbyte (pdf)


arnesom.github.io/Beginning-v1.00.pdf

I am giving away this first version of the book for free. I do reserve the right to print the book and sell it. You are free to distribute the pdf file or print it. You are also free to distribute printed copies, but you may not get paid for it.

Feel free to use the code samples, either as they are or as inspiration for your own work. Atribution would be nice, but isn’t required.

I will be grateful for feedback, and do so at the Github page for the book – or by email to the address shown in the book. I intend to publish a revised version of the book if I receive feedback that warrants an update.

Not Complete

The next course, «Advanced Raku» continues where this one ends. As the book is meant as a reference, I have chosen to make a combined book for both courses, called «Raku Explained». The second half (the «Advanced Raku» part) is unfinished, but I have published a preliminary Table of Contents and Index so that you can see what the whole book intends to cover.


  Raku Explained, v0.01 (December 2019)  

Pages: 30 (Table of Contents & Index only)

File size: ~ 5 Mbyte (pdf)


arnesom.github.io/Explained-v0.01.pdf

I am also interested in feedback on the topics in the second part (chapter 18 – 32).

Day 21: Searching for a Red gift

Alabaster Snowball, the elf, was searching for a gift for the person he had drawn on the North Pole’s Secret Santa. He had the great honour to draw Santa! What to give for the one who gives everyone’s presents? So he was searching on the internet for some keywords he knew Santa would like:

  • automatic letter reader
  • resistant boots
  • red sleigh accessories
  • red cap that does not fly off in wind
  • red jacket
  • red

Wait a minute! Is Red going to :api<2>?!! Alabaster Snowball has already read about that ORM for Raku. But it seems this new :api<2> version is taking it to the next level.

That’s it! I’ll give Santa a Red:api<2> PoC as gift! I know he has been playing with Raku, and I think it would be great to change all that collection of SQL strings on the NiceList model to a well made set of ORM classes.

Reading the documentation, Snowball learned that it would be very easy to create it’s first model:

use Red:api<2>;

unit model Child;

has UInt $!id              is id;
has Str  $.name            is column;
has Str  $.country         is column;

He started using Red:api<2> and creating a new model that represents a table child with 3 columns (id, name and country). As easy as that.

Alabaster could now just connect into a database, create the table, and start inserting children:

use Red:api<2>;
red-defaults default => database "SQLite";

Child.^create-table: :unless-exists;

Child.^create: :name<Fernanda>, :country<England> ;
Child.^create: :name<Sophia>,   :country<England> ;
Child.^create: :name<Dudu>,     :country<Scotland>;
Child.^create: :name<Rafinha>,  :country<Scotland>;
Child.^create: :name<Maricota>, :country<Brazil>  ;
Child.^create: :name<Lulu>,     :country<Brazil>  ;

And to list all children created:

.say for Child.^all.sort: *.name;

And that would run this query:

SELECT
   child.id, child.name, child.country 
FROM
   child
ORDER BY
   child.name

And prints:

Child.new(name => "Dudu", country => "Scotland")
Child.new(name => "Fernanda", country => "England")
Child.new(name => "Lulu", country => "Brazil")
Child.new(name => "Maricota", country => "Brazil")
Child.new(name => "Rafinha", country => "Scotland")
Child.new(name => "Sophia", country => "England")

If it’s needed, Santa can classify children by country:

my %by-country := Child.^all.classify: *.country;

And to discover what countries have children registered:

say %by-country.keys;

That would run:

SELECT
   DISTINCT(child.country) as "data_1"
FROM
   child

And that would return:

(England Scotland Brazil)

If he needs to get all children from England:

.say for %by-country<England>;

That would run:

SELECT
   child.id, child.name, child.country 
FROM
   child
WHERE
   child.country = ?

-- BIND: ["England"]

That would return:

Child.new(name => "Fernanda", country => "England")
Child.new(name => "Sophia", country => "England")

It’s working great! How about storing the gifts? Is there a way to store what a child asked by year?

# Gift.pm6
use Red:api<2>;

unit model Gift;

has UInt $!id            is serial;
has Str  $.name          is column{ :unique };

has      @.asked-by-year is relationship( *.gift-id, :model<ChildAskedOnYear> );

method child-asked-on-year(UInt $year = Date.today.year) {
    @!asked-by-year.grep(*.year == $year)
}

method asked-by(UInt $year) {
    self.child-asked-on-year(|($_ with $year)).map: *.child
} 
# Child.pm6
use Red:api<2>;

unit model Child;

has UInt $!id              is id;
has Str  $.name            is column;
has Str  $.country         is column;

has      @.asked-by-year   is relationship( *.child-id, :model<ChildAskedOnYear> );

method asked(UInt $year = Date.today.year) {
    @!asked-by-year.grep: *.year == $year
}
# ChildAskedOnYear.pm6
use Red:api<2>;

unit model ChildAskedOnYear;

has UInt $!id       is serial;
has UInt $.year     is column = Date.today.year;
has UInt $!child-id is referencing(*.id, :model<Child>);
has UInt $!gift-id  is referencing(*.id, :model<Gift>);

has      $.child    is relationship( *.child-id, :model<Child> );
has      $.gift     is relationship( *.gift-id,  :model<Gift>  );

Alabaster Snowball thought that way he could get all information he would need. Creating new gifts is easy!

for <doll ball car pokemon> -> $name {
    Gift.^create: :$name;
}

How about searching? Alabaster Snowball writes a new line:

.say for Gift.^all

And it returns all the gifts. But what if we want only the gifts that end with “ll”?

.say for Gift.^all.grep: *.name.ends-with: "ll"

That will run a query like:

SELECT
   gift.id, gift.name 
FROM
   gift
WHERE
   gift.name like '%ll'

Snowball wondered if it is possible to find what a child has asked:

.say for Child.^find(:name<Fernanda>).asked.map: *.gift

That runs:

SELECT
   child_asked_on_year_gift.id, child_asked_on_year_gift.name 
FROM
   child_asked_on_year
    LEFT JOIN gift as child_asked_on_year_gift ON child_asked_on_year.gift_id = child_asked_on_year_gift.id
WHERE
   child_asked_on_year.child_id = ? AND child_asked_on_year.year = 2019

And what if we want to know the last year’s gift?

.say for Child.^find(:name<Fernanda>).asked(2018).map: *.gift
SELECT
   child_asked_on_year_gift.id, child_asked_on_year_gift.name 
FROM
   child_asked_on_year
    LEFT JOIN gift as child_asked_on_year_gift ON child_asked_on_year.gift_id = child_asked_on_year_gift.id
WHERE
   child_asked_on_year.child_id = ? AND child_asked_on_year.year = '2018'

How do we know how many of each gift should be built?

say ChildAskedOnYear.^all.map(*.gift.name).Bag
SELECT
   child_asked_on_year_gift.name as "data_1", COUNT('*') as "data_2"
FROM
   child_asked_on_year
    LEFT JOIN gift as child_asked_on_year_gift ON child_asked_on_year.gift_id = child_asked_on_year_gift.id
GROUP BY
   child_asked_on_year_gift.name

The documentation for Red is on https://fco.github.io/Red/ and some examples used here can be found on https://github.com/FCO/Red/blob/join/examples/xmas/index.p6