Day 25 – Raku 2023 Review

How time flies. Yet another year has flown by. A year of many ups and a few downs.

Rakudo saw about 2000 commits this year, up about 30% from the year before. There were some bug fixes and performance improvements, which you would normally not notice. Most of the work has been done on RakuAST (about 1500 commits).

But there were also commits that actually added features to the Raku Programming Language that are immediately available, either in the 6.d language level, or in the 6.e.PREVIEW language level. So it feels like a good idea to actually mention the more noticeable ones in depth.

So here goes! Unless otherwise noted, all of these changes are in language level 6.d. They are available thanks to the Rakudo compiler releases during 2023, and all the people who worked on them.

New classes

Two publicly visible classes were added this year, one in 6.d and one in 6.e.PREVIEW:

Unicode

The Unicode class is intended to supply information about the version of Unicode that is supported by Raku. It can be instantiated, but usually it will only be used with its class methods: version and NFG. When running with MoarVM as the backend:

say Unicode.version; # v15.0
say Unicode.NFG;     # True

Format

The Format class (added in 6.e. PREVIEW) provides a way to convert the logic of sprintf format processing into a Callable, thereby allowing string formatting to be several orders of magnitude faster than using sprintf.

use v6.e.PREVIEW;
my constant &formatted = Format.new("%04d - %s\n");
print formatted 42, "Answer";  # 0042 - Answer

New methods

.is-monotonically-increasing

The Iterator role now also provides a method called .is-monotonically-increasing. Returns False by default. If calling that method on an iterator produces True, then any consumer of that iterator can use that knowledge to optimize behaviour. For instance:

say [before] "a".."z"; # True

In that case the reduction operator could decide that it wouldn’t have to actually look at the values produced by "a".."z" because the iterator indicates it’s monotonically increasing:

say ("a".."z").iterator.is-monotonically-increasing;  # True

New Dynamic Variables

Dynamic variables provide a very powerful way to keep “global” variables. A number of them are provided by the Raku Programming Language. And now there are two more of them!

$*EXIT / $*EXCEPTION

Both of these dynamic variables are only set in any END block:

• $*EXIT contains the value that was (implicitly) specified with exit()
• $*EXCEPTION contains the exception object if an exception occurred, otherwise the Exception type object.

File extensions

The .rakutest file extension can now be used to indicate test files with Raku code. The .pm extension is now officially deprecated for Raku modules, instead the .rakumod extension should be used. The .pm6 extension is still supported, but will also be deprecated at some point.

Support for asynchronous Unix sockets

With the addition of a .connect-path and .listen-path method to the IO::Socket::Async class, it is now possible to use Unix sockets asynchronously, at least on the MoarVM backend.

Improvements on type captures

In role specifications, it is possible to define so-called type captures:

role Foo[::T] {
    has T $.bar;
}

This allows consuming classes (or roles for that matter) to specify the type that should be used:

class TwiddleDee does Foo[Int] { }  # has Int $.bar
class TwiddleDum does Foo[Str] { }  # has Str $.bar

The use of these type captures was actually pretty limited. Fortunately, the possibilities have recently been extended significantly!

New arguments to existing functionality

Some methods have functionality added:

Stringification of integers

The .Str method of the Int class now optional takes either a :superscript or a :subscript named argument, to stringify the value in either superscripts or subscripts:

dd 456.Str(:superscript); # "⁴⁵⁶"
dd 456.Str;               # "456"
dd 456.Str(:subscript);   # "₄₅₆"

Min/max/sort options

The .min / .max methods now accept the :k, :v, :kv and :p named arguments. This is specifically handy if there can be more than one minimum / maximum value:

my @letters = <a b d c d>;
dd @letters.max;       # "d"
dd @letters.max(:k);   # (2, 4)
dd @letters.max(:v);   # ("d", "d")
dd @letters.max(:kv);  # (2, "d", 4, "d")
dd @letters.max(:p);   # (2 => "d", 4 => "d")

One could see this as a generalization of the .minpairs / .maxpairs methods, which now also accept a comparator as the first (optional) argument.

The .sort method now also accepts a :k named argument, returning the sorted indices in the original list, instead of the sorted list:

my @letters = <a c b d>;
say @letters.sort;               # (a b c d)
say @letters.sort(:k);           # (0 2 1 3)
say @letters[@letters.sort(:k)]; # (a b c d)

This can provide quite significant memory savings for large lists.

Better handling of out-of-bounds values

In 6.e.PREVIEW, the handling of negative values with the .log and .sqrt methods will now produce a Complex value, rather than a NaN. Relatedly, the .sign method can now also be called on Complex values.

use v6.e.PREVIEW;
say sqrt -1; # 0+1i
say log -1;  # 0+3.141592653589793i
say i.sign;  # 0+1i

RakuAST

About 75% of this year’s work was done on the RakuAST (Raku Abstract Syntax Tree) project. It basically consists of 3 sub-projects, that are heavily intertwined:

1. Development of RakuAST classes that can be used to represent all aspects of Raku code in an object-oriented fashion.
2. Development of a grammar and an actions class to parse Raku source code and turn that into a tree of instantiated RakuAST objects.
3. Development of new features / bug fixing in the Raku Programming Language and everything else that has become a lot easier with RakuAST.

RakuAST classes

There is little more to say about the development of RakuAST classes other than that there were 356 of them at the start of the year, and 440 of them at the end of the year. As the development of these classes is still very much in flux, they are not documented yet (other than in the test-files in the /t/12rakuast directory).

On the other hand, the RakuAST::Doc classes are documented because they have a more or less stable API to allow for the development of RakuDoc Version 2.

Raku Grammar / Actions

The work on the Raku Grammar and Actions has been mostly about implementing already existing features. This is measured by the number of Rakudo (make test) and roast (make spectest) test-files that completely pass with the new Raku Grammar and Actions. And these are the changes:

• make test: 95/137 (69.3%) → 110/151 (72.8%)
• make spectest: 585/1355 (43.2%) → 980/1356 (72.3%)

So there are quite a few features left to implement. Although it must be said that many tests are hinging on the implementation of a single feature, and often cause an “avalanche” of additional test-files passing when it gets implemented.

If you’d like to try out the new Raku Grammar and Actions, you should set the RAKUDO_RAKUAST environment variable to 1. The .legacy method on the Raku class will tell you whether the legacy grammar is being used or not:

$ raku -e 'say Raku.legacy'
True
$ RAKUDO_RAKUAST=1 raku -e 'say Raku.legacy'
False

Bug fixing

Several long standing bugs in Rakudo have been fixed in the new Raku Grammar / Actions. You can find these with the “Fixed in RakuAST” tag in the Rakudo issues. Fixes were usually a side-effect of re-implementation, or an easy fix after re-implementation.

New language features

The FIRST phaser can be reliably used in any block scope, thereby providing an alternative to once. And it returns the value, so you can use it to e.g. initialize a variable.

Unicode synonyms for -> and <-> are now accepted: → (2192 RIGHTWARDS ARROW) and ↔ (2194 LEFT RIGHT ARROW).

Vulgar fractions are now completely supported: ²/₃, 4¹/₃ and 4⅔ are now valid ways to express <2/3>, <13/3> and <14/3> (which was actually a long-standing request).

The rules for attaching Declarator Blocks to Raku source items have been simplified and made more consistent. One could consider this a bug fix, rather than a new feature :-). In short: declarator blocks are attached to the last attachable object on a line, rather than the first.

Creating/Inspecting/Debugging ASTs

Since most of the RakuAST classes have not been documented yet, it is often hard to figure out how to implement certain semantics from scratch. However, if you can express these semantics in Raku source code, there is a method that can help you with this: Str.AST. It takes the string, and parses this using the Raku grammar, and returns a RakuAST::StatementList object with the result.

For instance, how to do “my $a = 42; say $a” in RakuAST:

$ raku -e 'say Q|my $a = 42; say $a|.AST'
RakuAST::StatementList.new(
  RakuAST::Statement::Expression.new(
    expression => RakuAST::VarDeclaration::Simple.new(
      sigil       => "\$",
      desigilname => RakuAST::Name.from-identifier("a"),
      initializer => RakuAST::Initializer::Assign.new(
        RakuAST::IntLiteral.new(42)
      )
    )
  ),
  RakuAST::Statement::Expression.new(
    expression => RakuAST::Call::Name::WithoutParentheses.new(
      name => RakuAST::Name.from-identifier("say"),
      args => RakuAST::ArgList.new(
        RakuAST::Var::Lexical.new("\$a")
      )
    )
  )
)

Note the use of Q|| here: it’s nothing special, just an easy way to make sure nothing is inadvertently being interpolated in the string.

What you see here is the .raku output of the RakuAST tree. Note that it is carefully indented for easier adaptation / integration into other code.

To run the code in this AST, you can call the .EVAL method:

$ raku -e 'Q|my $a = 42; say $a|.AST.EVAL'
42

It is also possible to convert a RakuAST tree back to Raku source code with the .DEPARSE method:

$raku -e 'say Q|my $a = 42; say $a|.AST.DEPARSE'
my $a = 42;
say $a

Methods giving Raku core and other developers a lot of tools to work with!

Methods on RakuAST objects

These methods are more intended to be used by people wanting to build / modify an existing RakuAST tree. In short:

• .map, .grep, .first: select objects matching given condition, provide @*LINEAGE inside the code blocks.
• .rakudoc: specialized version of .map selecting RakuAST::Doc objects.
• .literalize: attempt to create a RakuAST::Literal object out of the invocant (basically: constant folding).

RakuDoc

The legacy Pod parser was replaced by a RakuDoc parser, implemented from scratch. Which made parsing of Pod about 3x as fast. Through this re-implementation, it became much easier to add new features in RakuDoc, which resulted in the RakuDoc Version 2 project that Richard Hainsworth reported about.

The --rakudoc command-line argument has been added, similar to --doc. But instead of loading the Pod::To::Text, it will load the new RakuDoc::To::Text module to produce the documentation.

Localization

At the first Raku Core Summit, Richard Hainsworth not only made compelling points about the Raku documentation, they also introduced the idea of localization of the Raku Programming Language: being able to program Raku in your native language!

Learning a programming language can be difficult enough. And especially so if English is not your native language.

So far 6 languages are supported (to various degrees): DE (German), FR (French), HU (Hungarian), IT (Italian), NL (Dutch) and PT (Portuguese). The .AST and .DEPARSE methods have been adapted to allow a localization language to be specified. So to convert a piece of Raku code to Dutch, one can do:

$ raku -e 'say Q|my $a = 42; say $a|.AST.DEPARSE("NL")'
mijn $a = 42;
zeg $a

Or convert a piece of code in Dutch, into Hungarian:

$ raku -e 'say Q|mijn $a = 42; zeg $a|.AST("NL").DEPARSE("HU")'
enyém $a = 42;
mond $a

Of course, we would like to see as many localizations as possible. To create a localization in your native language, you will need to translate about 600 items in a text file (more information).

The localization effort will have its effects on documentation, IDEs and Public Relations. These will still need to further developed / investigated. But the end goal, being able to teach programming to all children in the world, is a worthy cause!

Documentation

The documentation update process was renewed, and the documentation site was re-styled, thanks to the many members of the Raku Documentation Team. And put live thanks to the Raku Infra Team. Who all deserve many kudos for their work behind the scenes.

Thank You

JJ Merelo decided to step down from the Raku Steering Council. Again a big thank you for all that he’s done for Raku.

Rainbow Butterfly Award 2023

The 2023 Rainbow Butterfly Award was awarded to Oleksander Kiryuhin (aka sena_kun aka Altai-man) for their tireless efforts as release manager of the Raku Programming Language for two years (2020-2021), and their work on getting a more functional Raku documentation in general, and a better documentation web site in particular.

The Raku Conference

Andrey Shitov tried very hard to get an in-person Raku Conference together, but alas had to cancel for various hard to control reasons. Instead, the Third Raku Conference was once again held online. We’ll always have the videos!

In Memoriam

The Rakudo Weekly News brought the sad news that Ben Davies (aka kaiepi, aka @mrofnet) passed away in the early hours of 14 January. Ben has supplied many very useful Pull Requests to the MoarVM, NQP and Rakudo repositories, and thus was almost a Rakudo core member. He is and will be missed.

Raku Core Summit

In June, Wendy van Dijk and Elizabeth Mattijsen organized the very first Raku Core Summit: Three+ days of in person discussions, hacking, making plans and finally having some quality time to work on something that has been bugging for a long time.

Looking forward to the second Raku Core Summit, so this can become a very nice tradition!

Summary

Looking back, an amazing amount of work has been done in 2023!

The Raku core developers gained another member: John Haltiwanger. Which will help the RakuAST work going forward, and the next language release of the Raku Programming Language getting closer!

Hopefully you will all be able to enjoy the Holiday Season with sufficient R&R. The next Raku Advent Blog is only 341 days away!