Featured

The 2025 Raku Advent Posts

(in chronological order, with comment references)

Featured

All the blog posts of 2021

Featured

It’s that time of the year

When we start all over again with advent calendars, publishing one article a day until Christmas. This is going to be the first full year with Raku being called Raku, and the second year we have moved to this new site. However, it’s going to be the 12th year (after this first article) in a row with a Perl 6 or Raku calendar, previously published in the Perl 6 Advent Calendar blog. And also the 5th year since the Christmas release, which was announced in the advent calendar of that year.

Anyway. Here we go again! We have lined a up a full (or eventually full by the time the Advent Calendar is finished) set of articles on many different topics, but all of them about our beloved Raku.

So, enjoy, stay healthy, and have -Ofun reading this nice list of articles this year will be bringing.

Day 19 – Welcome to the Megadungeon

Welcome, weary traveler. It is known that you come bearing earnest questions and we congratulate you on managing to arrive here, on this precipice of doom. Not a brutal doom, nor even a fearful doom, but a doom as in a fate open to you.

See, your arrival here stands foretold. In fact, we built this very precipice in anticipation of you. Since we expected you to come from many different paths, we laid out long and sturdy carpets in untallied directions.

There has been a long-standing contrast observed, that Perl was the Hobbit to Raku’s Lord of the Rings. While somewhat amusing, I have always felt that this analogy failed in two key ways.

First, it fails to give Perl’s legacy its due. Perl’s presence is felt in the deepest firmaments of our operating systems. If it must play the role of a ‘smaller sibling,’ using the Hobbit as reference just feels too under-stated. It’s an amazing book, but especially with today’s often almost anti-rose-tinted perspective on Perl’s contributions on everything from build systems, bug report tools, testing frameworks, centralized distribution of modules, even to our very understanding of DNA… I think we can do better by Perl.

Second, comparing Raku to the Lord of the Rings… while it is an indisputably impressive work of singular imagination, this comparison nevertheless fails to describe the sheer, nigh unbelievable audacity that the project — the dream — that is now known as Raku actually represents.

I’d like to take this opportunity to explore a new spin on all of this. If Perl 1-4 represents the Hobbit, then Perl 5 (now, at-last-and-once-again known as simply Perl) is the Lord of the Rings. Perl has certainly gone to Mount Doom and back. And when it finally arrived back home, it found that its position in the world had been irrevocably changed.

That is also to say that Perl represents an achievement of similar scope and magnitude as to Tolkien’s genre-defining series. Nothing that came after it ever looked or felt quite the same. But none would ever again dare to arrive without (hash)maps in place!

But where does that leave Raku? Raku is Dungeons and Dragons. D&D’s very existence hinges on works like Lord of the Rings (see Appendix N for the larger context of influences). Dungeons and Dragons is also certainly audacious: it’s essentially an audacity amplifier! Whatever scope you can envision, you can make possible. That fits pretty well into our famous position: “make the hard things easy and the impossible things feasible.” (How’s that for audacity? Also, this is my personal paraphrase. Use responsibly!)

All you need is a group of like-minded people who agree to both take it seriously and have a whole lot of fun while doing so.

It is understandable if you feel a bit trepidatious. It’s a known crossover effect from the fact that you may have already made the choice, though it is only now presented before you… beyond the threshold, into the darkness of the deep… some greater force urges you on, compelling you to strive forth and make a place for your name in the Great Hall of CONTRIBUTORS.

A megadungeon is a specific type of role-playing “adventure path” designed to be a permanent fixture of a larger campaign. Player characters can dip in and out of the megadungeon as they please, exploring it to whatever depth feels appropriate. But a key feature of a megadungeon is that it (almost) always has another level, another staircase leading into seemingly-unknown-yet-also-clearly-already-ventured-upon corridors and chasms. (Maybe I just took the first step in creating my own heartbreaker RPG system called Chasms & Corridors?)

The “almost” qualifier is that every megadungeon does have a last level, at least as designed. You can theoretically get there. But the chances of Real Life allowing it are minimal. And so the comparison to D&D also fits the historical record of Raku, especially if you consider core development to be a megadungeon.

It is not uncommon for our core developers to reach a point in their journey where they find they must turn back in order to find their way again. Their paths may or may not ever venture back into Mount Camelia, but the marks they left remain and endure and shape the experience of all who come after. We are all so very grateful to have had the time with you that you chose to share, and wish you nothing but the best in all of your new adventures.

At the risk of belaboring this metaphor, I’ll leave on a note of highlighting the stratification of core development in this context. Work can be done by those lovely creatures blessed with darksight (read: C skills and compiler mind) at the MoarVM level. The fact that they can get into the lowest areas of Mount Camelia and emerge with treasure is a testament to either their fortitude, their foolhardiness, or both.

Meanwhile, there has been a years long project to create an entirely new compiler front-end, essentially building a brand new way of existing and traveling throughout the megadungeon that is Raku core development. And all the while, progress and improvements have been made to the implementation of the core setting.

With regards to the new front end: We are close, maybe very close. But even so… for those of us still delving the nooks and crannies of our respective levels, we know that once we get there, it will be as it ever was: just as we put the last of our current efforts to polish, we will inevitably see some glimmer in the dark to once again drive us on.

You will then, as always, be welcomed to come along, for a spell or for a song, as we carve a dream of it.

This post is dedicated to all of us who have put heart, mind, and/or life into this audacious dream of ours. We love you, we miss you, we are happy for you, wherever you are.

Happy holidays!

Day 18 – Hallo, Wêreld!

by habere-et-dispertire

The first elf language is the sharing of emotions. Next comes their mother tongue. When elves start speaking their first programming language, then they are already onto their third language!

Santa imagines raku draws inspiration from many principles of Unix philosophy In particular, localization of raku plays into the robustness principle.

When writing to Santa, it doesn’t matter which language you write your wishlist in — for Santa is learning all the languages he can:

say 'Hello, World!' # English, or
sê 'Hallo, Wêreld!' # Afrikaans

It is kinder thought Santa — especially to those writing their first list — to get their feet wet by writing in a language closer to their heart.

TLDR

use L10N::AF;

# Trim whitespace (leading and trailing)
sit .sny vir '/pad/na/lêer'.IO.lyne

# List lines containing specified text
.sit as .bevat: 'soekteks' vir '/pad/na/lêer'.IO.lyne

Install

To speak with the alf elf, the summoning charm is via zef:

$ zef install L10N::AF

To write your wishlist in Afrikaans, don’t forget to include use L10N::AF near the top of your list. Or use the prepaid postage envelopes labelled afrku or lekker:

$ lekker -e 'sê "Hallo, Wêreld!"'
Hallo, Wêreld!

Other languages

The talkative elves have discovered that between them they already speak eleven languages. To find an elf that speaks your language, replace XX in use L10N::XX with the elf’s nickname:

LanguageISO 639 Set 1
AfrikaansAF
DutchNL
EsperantoEO
FrenchFR
GermanDE
HungarianHU
ItalianIT
JapaneseJA
PortuguesePT
ChineseZH
WelshCY

Preservation Of Huffmanization

Santa doesn’t much like chimneys as they are too long. Why huff and puff when he can send his smaller elves down, like , so, vra and met:

use L10N::AF;

#| Pangram test / Toets vir 'n pangram
sub is-pangram ($sin = vra 'Sin: ') {
    sê so .kl.bevat: alle 'a'..'z' met $sin
}

is-pangram q :tot /GEDIG/;
    Bask with grammars,
    Coax via allomorphs,
    Sequences of huffmanized joy.
GEDIG
# True

Decorations

Festive elves sprinkled decorations on some of the presents, like (say) and reël (rule). This did not slow Santa as the first four things he swallowed from the explosion at his ASCII factory were U, T, F and 8. Diacritics are also fun to input when you configure your keyboard to use a compose key. The keystroke sequences often form a visual composition then, for example:

CharacterKeystrokes
êCompose + e + ^`
ëCompose + e + "

Dual Translations

Some elves couldn’t make up their minds what they wanted for Xmas. Santa has allowed them to wrap their own gifts twice which has made them happy again for now:

AfrikaansEnglish
basisnaambasename
aanpas-lêermoduschmod
aanpas-lêerskapchown
gidsdir
gidsnaamdirname
sifgrep
kophead
koppellink
maak-gidsmkdir
drukfprintf
skrap-gidsrmdir
slaapsleep
sorteersort
sterttail
ontkoppelunlink

( Words in common shell usage have been kept as alternate translations — either can be used. )

Arrays

Some elves need help remembering which end of the sleigh they are servicing and whether they are loading or unloading:

Santa decided to label both ends of the sleigh and split the elves into fore and aft teams:

EnglishAfrikaans
pop, pushtrek-einde, druk-einde
shift, unshifttrek-begin, druk-begin

Composable

Raku and Afrikaans share a composable aspect.
Santa’s elves giggle when making associations between concepts:

AfrikaansEnglish
Waar, OnwaarTrue, False
skrap, skrap-nl, skrap-gidsdelete, chomp, rmdir
maak-oop, maak-toe, maak, maak-gidsopen, close, make, mkdir
gee, teruggee, opgee, gegewe, vergewe*return, returns, fail, given, forgiven*
afrond, afrondaf, afrondopround, floor, ceiling
stop, stopseindie, fatal

Sensitivity

Abuse by coercive control is not yet addressed by the Inclusive Naming Initiative. It is recognized in law in England and Wales (2015) and Ireland (2019). Santa is peaceable and so with his awareness of escalatory metaphors and imprecise terminology of the non-elves, he has tried to find translations that are more descriptive, less disproportionate and better suited to a programming language.

In programming context, the word “coerce” is used to change a value between types.
Here Santa chose herskep (recreate) rather than “dwang” (coerce). This ties in with the root term skep (create) and with omskep (map/convert). Santa wondered if his de-escalations mattered:

EnglishAfrikaansEnglish re-translation
CONTROLBESTUURmanage
diestophalt
fatalstopseinstop signal
snitchaanskouto witness

But was rather happy to express himself…

Day 17 – An issue with evaluation

Lizzybel was walking the corridors of North Pole Central when Steve, the Physics Elf, came up to her. “Have you seen my issue on this very nice module of yours?”, he asked.

“Oof, I guess I must have missed that, sorry!”, said Lizzybel, while thinking to herself “I really should look more at my modules every now and then”. “What is the issue about?” “Well, you know my nice App::Crag module? Well, in some cases it just generates some warnings that do not make a lot of sense in that context, so I’d like to be able to get rid of them. But I can’t”, said Steve, “It’s the CodeUnit module”.

Lizzybel tried to hide her blush, but to no avail. Steve saw that, and said: “No worries, it’s but a nuisance” and hurried away smiling, as all of the working at the North Pole don’t just work out by themselves.

CodeUnit

“Ah, yes, CodeUnit, a module that provides a unit for execution of code”, thought Lizzybel, “that cute one that I abstracted from the REPL module I worked on earlier this year. Ah, and there’s Steve‘s issue!”.

Time to look at the code! So why did setting $*ERR to /dev/null not work? Aaah… found it:

CONTROL {
    when CX::Warn {
        .gist.say;
        .resume;
    }
}

“Stupid Lizzybel“, she thought. Control messages (of which warnings are of type CX::Warn) where caught in the CONTROL block, but then output on STDOUT with .say. That should have been .note! Ok, that’s easy enough to fix.

More Options

Thinking about this some more, Lizzybel decided to make it slightly more configurable. “Why not add a :keep-warnings flag and attribute to indicate that you don’t want warnings to be shown immediately, but when you want it?”, she thought to herself. Why not indeed!

A little bit of hacking later, that piece of the code now read:

CONTROL {
    when CX::Warn {
        $!keep-warnings ?? @!warnings.push($_) !! .gist.note;
        .resume;
    }
}

So, now if we don’t want to keep warnings, they are shown on STDERR (because of .note). And if we want to keep warnings, the warning in $_ will be added to the warnings. Now for a way to get the warnings from the CodeUnit object.

method warnings() {
    @!warnings ?? @!warnings.splice.List(:view) !! BEGIN ()
} 

And with that it’s also possible to obtain any warnings, and reset them at the same time!

Now the only thing to do, was to update the documentation, add some more tests and release the new version of the CodeUnit module to the Raku ecosystem with mi6 release! And so it was done.

But, but, but, how?

A few hours later, Steve came running to Lizzybel, alerted by the closing of his issue, and a little short of breath, said: “That’s all very nice, and thank you. But what magic is .splice.List(:view) and BEGIN ()?”.

“Ah, yes. Maybe a little bit too core-ish for a module. But then again, that module is already bound tightly to the Rakudo implementation of the Raku Programming Language“, she answered, “so it felt ok to do that here”.

“So the .splice method without any arguments is an optimized way to both return the whole invocant array and reset that array to not having any elements. The :view named argument to the .List method is an implementation detail that provides an optimized way to change an Array into an immutable List. Provided, of course, that the underlying Array doesn’t change, and no attempts are made to change any element in the List.”, she continued. “Hmm…”, said Steve, not entirely convinced.

Lizzybel continued further: “If there are no warnings, then we could also convert the Array to a List, but that felt a bit wasteful. So why not return an empty List with () in that case?”.

“But why the BEGIN then?”, asked Steve. “Well, an empty List currently in Rakudo is equivalent to List.new, so creates a new object each time. But conceptually speaking, all empty Lists are the same. So why not always return the same one, created at compile time with the BEGIN phaser”, Lizzybel lectured. “That feels like that should be done automatically by the compiler”, said Steve, while wandering away again, busy as ever. “You’re probably right”, thought Lizzybel, “I should make that happen for RakuAST“.

Inspiration

And while Steve was walking away, Lizzybel realized she finally had a subject to write a Raku Advent post about!

Day 16 – Melian and the Helpers of Evergreen

by Sawyer X

Operation Evergreen began quietly that winter.

Santa had announced it as a workshop-wide initiative to improve the speed and efficiency of analytics across every team (logs, metrics, scheduling, routing, and storage). The elves who maintained the Workshop’s growing systems found themselves gathered around long benches, examining slow queries, performance charts, and the output of tools that strained under seasonal load.

One of the key engines behind Evergreen was Melian, a high-performance caching service designed to answer structured queries faster than any relational database or caching server could hope to. Melian’s protocol was compact and binary; queries were made of tables, indexes, and tight payloads. Round-trip times were measured in single microseconds.

Melian already had clients in Perl, Python, Node.js, PHP, and C. But Evergreen wasn’t only about speed, it was also about making engineering work pleasant. In the workshop, comfort matters, even when you’re racing the calendar.

So one team of elves wondered: Could Raku be the next great way to talk to Melian?

They suspected it could. And they were right.

Enter Raku

These elves were familiar with Raku‘s reputation: expressive, concurrent, type-driven, joyful with binary data, and backed by a runtime that had been quietly optimized for real workloads. But this was the first time they’d attempt to pair Raku with a high-throughput caching server.

To their surprise, Raku behaved as if it had been waiting for precisely this job.

Why Raku Was a Natural Fit for Melian

Melian’s goals were simple: serve rows faster than anything else and handle extremely high request rates while avoiding unnecessary overhead and keeping the client interface small and predictable.

That meant the client language needed to handle structured bytes, concurrency, efficient encoding/decoding, and predictable performance without becoming tedious for the elves writing the code.

Raku checked every box.

Bufs, Blobs, and Native Binary Handling

The elves quickly discovered that Raku treats binary data as a first-class citizen.

For a protocol like Melian’s, you don’t want ceremony. You want to construct frames, inspect bytes, slice, encode, decode, and rebuild messages without tripping over your language.

Raku’s built-in buf8, Blob, and typed native arrays made that simple. These weren’t add-ons or foreign modules; they were part of the core. Conversions between Blobs, Bufs, native arrays, and flat integer lists were natural and efficient.

This meant the elves could remain close to the metal, constructing packets in the exact format Melian required, while still enjoying Raku’s expressive and readable style. Nothing felt clumsy or forced.

It was the first moment the elves realized: this language might actually be built for this kind of work.

Types That Make the Interface Behave

The next advantage became clear when designing the interface.

Raku lets you describe the shape of your data directly in method signatures:

method fetch-by-int(
  UInt:D $table-id, UInt:D $index-id, Int:D $key --> Promise
) { ...

If a caller passes the wrong type, Raku catches it immediately. No mysterious errors. No silent failures. The elves loved this because Evergreen required reliability as much as raw speed.

Of course, Raku could have made this even fancier. Its multi-method dispatch system could distinguish between fetch(Int), fetch(Str), fetch(Blob), or even custom types. Many languages struggle with this; Raku does it effortlessly and beautifully.

But for clarity (and in keeping with other Melian client libraries), the elves chose to expose explicit methods such as fetch-by-int-from() and fetch-by-string-from().

The key point remained: Raku let them decide the interface, and it supported both elegant magic and predictable simplicity.

Promises, Auto-Scheduling, and Combinators

Melian encourages parallelism. Most workloads involve fetching many keys at once. That meant concurrency mattered.

The elves knew Raku had Promises, but they didn’t realize how good they were.

Raku’s Promises:

  • Auto-schedule onto the thread pool.
  • Requires no executor setup.
  • Integrate cleanly with typed signatures.
  • Compose using Promise.allof, Promise.anyof, and then chains.
  • Behave predictably even with deep asynchronous pipelines.

The elves had spent years managing thread pools manually in other languages. Here, the threading happened automatically. Not only did this simplify Evergreen’s code, but it made Melian’s parallelism feel lightweight.

Even better, Raku offered concurrency tools (like react, whenever, supplies, and channels) that served as powerful constructs, letting elves orchestrate constant streams of cache traffic if the need arose.

They didn’t use these for the basic client, but knowing the tools existed gave the elves confidence: Raku scales with you, not against you.

Parsing Strength: Grammars and Beyond

Melian’s wire protocol is intentionally small, but some elves wondered if future versions might include richer metadata or structured commands.

Raku’s grammars reassured them.

Raku is one of the strongest parsing languages ever built. Grammars are not a library. They are a language feature that provides tokenization, backtracking, parse trees, and consistent error reporting. All are built directly into the syntax.

Even though Melian doesn’t require grammars today, the elves knew that if Evergreen ever needed a more expressive protocol, Raku would handle it without friction.

Rakudo and MoarVM

Finally, the elves examined how this all actually ran.

The client was implemented in Rakudo, powered by MoarVM. MoarVM shines when dealing with:

  • native buffers
  • concurrent workloads
  • object specialization
  • byte-level operations
  • predictable execution of Promise chains

The elves noted something unusual: Raku’s abstractions didn’t cost them performance. MoarVM carried its weight quietly and efficiently.

Melian was fast, and Raku let the elves keep it fast.

A Walk Through the Raku Client

With these features combined, the Raku Melian client ended up being compact and comfortable:

  • binary frames built from buf8
  • typed method signatures
  • Promise-based fetch operations
  • lazy schema bootstrapping
  • readable byte decoding

The elves could issue dozens (or hundreds) of concurrent requests, await them, and process results transparently. Where other languages required scaffolding, Raku needed only a few lines.

Evergreen’s goal was speed without sacrificing happiness. Here, Raku delivered both.

A Greener Christmas

By the time Operation Evergreen wrapped up, the Workshop’s analytics systems were faster than ever. Melian handled impossible workloads with ease. And the Raku client provided an expressive, type-safe, concurrent interface that let elves build tools quickly and confidently.

The elves didn’t replace any existing languages. They simply added another one: one that complemented Melian beautifully.

This was the true spirit of Evergreen: enhancing what worked, exploring what could, and giving the workshop more ways to succeed.

Melian stayed fast. Raku made using it pleasant. And somewhere between them, Christmas became just a little greener.

Do It Yourself: Try Melian with Raku

Step 1: Create a SQLite Database

Create a file called workshop.db and load this schema:

CREATE TABLE production (
    id INTEGER PRIMARY KEY,
    elf_name TEXT NOT NULL,
    toys_made INTEGER NOT NULL
);

INSERT INTO production (elf_name, toys_made) VALUES
    ('Elanor', 42),
    ('Tarn',   44),
    ('Pip',    39);

You can inspect it with:

sqlite3 workshop.db 'SELECT * FROM production'

Step 2: Point Your Melian Daemon at the Table

Run Melian in the command line with:

MELIAN_DB_DRIVER=sqlite MELIAN_SQLITE_FILENAME=workshop.db MELIAN_TABLE_TABLES='production#0|60|id#0:int;elf_name#1:string' ./melian-server

This gives us a SQLite DB driver, using the production table, defining two queryable columns (id and elf_name), assuming each will return only one row.

Step 3: Query the Data from Raku

Here is a small example that fetches entries for one elf across multiple hours:

use Melian;

my $melian   = Melian.new;
my @names    = <Elanor Pip>;
my @promises = @names.map({
    $melian.fetch-by-string-from( 'production', 'elf_name', $_);
});

await Promise.allof(@promises);
@promises>>.result.say;

From here, you can extend the example to compute averages, compare elves, or schedule periodic lookups using concurrency constructs.

Letters from the North Pole

Operation Evergreen pushed the Workshop to explore new territory: faster caching, better concurrency, and cleaner interfaces. Melian provided the performance. Raku provided the language that made the client both fast and pleasant to use.

With strong binary types, a real type system, Promises and combinators, a parsing engine ready for any protocol – all built on Rakudo + MoarVM’s efficient execution model.

Day 15 – An expression language for Vixen

#raku-beginners: korvo: Hi! I’m trying out Raku in stead of META II for a toy compiler.

(5 minutes later) korvo: I’m merely trying to compile a little expression language because my angle of repose has slightly increased, and multiple folks have recommended Raku for parsing and lightweight compilers.

(20 minutes later) korvo: [T]hanks for a worthy successor to META II. This is the first enjoyable parser toolkit I’ve used in a while; I spent almost no time fussing over the Regex tools, found it easy to refactor productions, and am spending most of my time trying to handle strings and lists and I/O.

Happy Holiday! As we enter the winter-solstice season, it’s worth reflecting on the way that hunkering down for the cold can bring about new avenues of exploration. I have a jar of pickled homegrown banana peppers in the corner of my refrigerator slowly evolving into something delicious; I do have to shake it every few days, but it will be worth it to add that fruity sour spice to any dish. Similarly, this can be a season for letting old concepts ferment into new concepts.

I have written so many parsers and parser toolkits. I’m tired of writing parsers yet again. So, after hearing about Raku’s builtin support for grammars, I decided that I would commit to trying Raku for my next parsing task. How hard could it be? I’ve already used and forgotten Perl 5 and Ruby.

I don’t need to reinvent Raku. ~ me, very tired

Problem

I’m working on a language that I call Vixen. I should back up.

At the beginning of the year, Joel Jakubovic blogged that The Unix Binary wants to be a Smalltalk Method, Not an Object. They argue that, while we have traditionally thought that Unix files correspond to objects, we should instead correspond Unix directories with objects and Unix files with methods. By “object” I mean a bundle of state and behavior which communicates with other objects by passing messages to them. This is a big deal for folks who study what “object” means, but not really interesting for the wider programming world. However, they followed it up with a prototype and a paper, The Unix Executable as a Smalltalk Method: And its implications for Unix-Smalltalk unification. Jakubovic provides a calling convention, which we call Smalltix, that allows us to treat a Unix system as if it were a Smalltalk-like message-passing object-based system. Crucially, there isn’t a single language for programming Smalltix, because of fragmentation: a Unix system already has many different languages for writing executable programs, and adding another language would only fragment the system further.

Okay! So, I’m working on Vixen, a fork of Smalltix. Jakubovic used Bash and Smalltalk-style classes; I’m simplifying by using execline and Self-style prototypes. Eventually, I’ve got a few dozen little scripts written with execline. Can I simplify further?

Now, I fully admit that execline is something of an alien language, and I should explain at least some of it before continuing. Execline is based on the idea of Bernstein chain loading; the interpreter takes all arguments in argv as a program and calls into multiple helpers which incrementally rewrite argv into a final command. Here’s an example method that I call “debug:” which takes a single argument and prints it to stderr. First it uses the fdmove helper to copy file descriptor 2 to file descriptor 1, shadowing stdout with stderr; finally, it echoes a string that interpolates the first and second items of argv. The calling convention in Smalltix and Vixen is that argv’s zeroth item is the method, the first item is the receiving object passed as a path to a directory, and the rest of the items are positional arguments. By tradition, there is one colon “:” in the name of a method per argument, so “debug:” takes one argument; also by tradition, the names of methods are called verbs. Since this method takes one positional argument, we pass the -s2 flag to the execline interpreter execlineb to collect argv up to index 2.

#!/usr/bin/env -S execlineb -s2
fdmove -c 1 2
echo "debug: ${1}: ${2}"

For something more complicated, here is a method named “allocateNamed:” which augments some other “allocate” method with the ability to control the name of a directory. This lets us attach names to otherwise-anonymous objects. Here, we import the name “V” from the environment envp to turn it into a usable variable. In Vixen, I’ve reserved the name “V” to refer to a utility object that can perform calls and other essential tasks. The backtick helper wraps a subprogram in a curly-brace-delimited block and captures its output. The foreground helper runs two subprograms in sequence; there’s also an if helper which exits early if the first subprogram fails.

#!/usr/bin/env -S execlineb -s2
importas -iS V
backtick -E path { ${V}/call: $1 allocate }
foreground { mkdir ${path}/${2} }
echo ${path}/${2}

Now, as the reader may know, object-based languages are all about messages, object references, and passing messages to object references. In some methods, like this one called “hasParent”, we are solely passing messages to objects; the method is merely a structure which composes some other objects. This is starting to be a lot of code; surely there’s a better way to express this composite?

#!/usr/bin/env -S execlineb -s1
importas -iS V
backtick -E parent { ${V}/call: $1 at: "parent*" }
${V}/call: $parent exists

Syntax

Let’s fragment the system a little bit by introducing an expression language just for this sort of composition. Our justification is that we aren’t going to actually replace execline; we’re just going to make it easier to write. We’ll scavenge some grammar from a few different flavors of Smalltalk. The idea is that our program above could be represented by something like:

[|^(self at: "parent*") exists]

For non-Smalltalkers, this is a block, a fundamental unit of code. The square brackets delimit the entire block. The portion to the right of the pipe “|” is a list of expressions; here there is only one. When the final expression starts with a caret “^”, it will become the answer or return value; there’s a designated Nil object that is answered by default. Expressions are merely messages passed to objects, with the object on the left and the message on the right. If a message verb ends with a colon “:” then it is called a keyword and labels an argument; for each verb with a colon there is a corresponding argument. The builtin name self refers to the current object.

The parentheses might seem odd at first! In Smalltalk, applications of verbs without arguments, so-called unary applications, bind tighter than keyword applications. If we did not parenthesize the example then we would end up with the inner call "parent*" exists, which is a unary application onto a string literal. We also must parenthesize to distinguish nested keyword applications, as in the following example:

[:source|
obj := (self at: "NixStore") intern: source.
^self at: obj name put: obj]

Here we can see the assignment token “:=” for creating local names. The full stop “.” occurs between expressions; it creates statements, which can either assign to a name or not. We can also see a parameter to this block, “:source”, which occurs on the left side of the pipe “|” and indicates that one argument can be passed along with any message.

Grammar

Okay, that’s enough of an introduction to Vixen’s expression language. How do we parse it? That’s where Raku comes in! (As Arlo Guthrie might point out, this is a blog post about Raku.) Our grammar features everything I’ve shown so far, as well as a few extra features like method cascading with the semicolon “;” for which I don’t have good example usage.

grammar Vixen {
    token id       { <[A..Za..z*]>+ <![:]> }
    token selector { <[A..Za..z*]>+ \: }
    token string   { \" <-["]>* \" }
    token param    { \: <id> }
    token ass      { \:\= }

    rule params { <param>* % <ws> }

    proto rule lit             {*}
          rule lit:sym<block>  { '[' <params> '|' <exprs> ']' }
          rule lit:sym<paren>  { '(' <call> ')' }
          rule lit:sym<string> { <string> }
          rule lit:sym<name>   { <id> }

    rule chain { <id>* % <ws> }

    proto rule unary {*}
          rule unary:sym<chain> { <lit> <chain> }

    rule keyword { <selector> <unary> }

    proto rule message {*}
          rule message:sym<key>  { <keyword>+ }

    rule messages { <message>* % ';' }

    proto rule call {*}
          rule call:sym<cascade> { <unary> <messages> }

    proto rule assign {*}
          rule assign:sym<name> { <id> <ass> <call> }
          rule assign:sym<call> { <call> }

    rule statement { <assign> '.' }

    proto rule exprs {*}
          rule exprs:sym<rv>  { <statement>* '^' <call> }
          rule exprs:sym<nil> { <statement>* <call> }

    rule TOP { '[' <params> '|' <exprs> ']' }
}

Writing the grammar is mostly a matter of repeatedly giving it example strings. The one tool that I find indespensible is some sort of debugging tracer which indicates where a parse rule has failed. I used Grammar::Tracer, available via zef. I’m on NixOS, so language-specific package managers don’t always play nice, but zef works and is recommended. First I ran:

$ zef install Grammar::Tracer

And then I could start my file with a single import in order to get tracing:

import Grammar::Tracer;

Actions

The semantic actions transform the concrete syntax tree to abstract syntax. This sort of step is not present in classic META II but is essential for maintaining sanity. I’m going to use this grammar for more than a few weeks, so I wrote a few classes for representing abstract syntax and a class of actions. Some actions are purely about extraction; for example, the method for the params production merely extracts a list of matches and extracts the Str for each match.

    method params($/) { make $.values.map: *.Str; }

Some actions contain optimizations that avoid building abstract syntax. The following method for unary handles chained messages, where we have multiple unary applications in a row; we want a special case for zero applications so that the VUnary class can assume that it always has at least one application.

    method unary:sym<chain>($/) {
        my $receiver = $.made;
        my @verb = $.made;
        make @verb ?? VUnary.new(:$receiver, :@verb!! $receiver;
    }

Some actions build fresh abstract syntax not in the original program. The following method for exprs handles the case when there is no return caret; the final expression is upgraded to a statement which ignores its return value and the name Nil is constructed as the actual return value.

    method exprs:sym<nil>($/) {
        my @statements = $.values.map: *.made;
        my $call = $.made;
        @statements.push: VIgnore.new(:$call);
        my $rv = VName.new(:n("Nil"));
        make VExprs.new(:@statements, :$rv);
    }

Getting the actions right was difficult. I ended up asking for hints on IRC about how to work with matches. The .values method is very useful.

Abstract syntax

I had a couple false starts with the abstract syntax. I think that the right mentality is to have one node per production, but to have one role per compiler action. If necessary, change the grammar to make the abstract syntax easier to generate; Raku is flexible enough to allow grammars to be refactored. Rules like params, chain, and keyword were broken out to make life easier.

By the way, starting at this point, I am only showing excerpts from the complete compiler. The compiler is available in a separate gist. Classes may be incomplete; only relevant methods and attributes are shown.

For example, there is a role for emitting literals. A parenthesized call just unwraps the parentheses; a string is represented by itself.

role EmitLiteral {
    method prepareLiteral($compiler) { ... }
}
class VParen does EmitLiteral {
    has Call $.call;
    method prepareLiteral($compiler) { $.call.prepareLiteral: $compiler; }
}
class VStr does EmitLiteral {
    has Str $.s;
    method prepareLiteral($compiler) { $.s; }
}

We can also have a role for performing a call. We need two flavors of call: call and bind to a name, and also call without binding. It’s much easier to compile chains and cascades with the option to bind or not bind. We can put both roles onto a single class, so that a cascading application both performs a call and also evaluates to a literal expression.

role Call {
    method prepareBind($name, $compiler) { ... }
    method prepareOnly($compiler) { ... }
}
class VCall does Call does EmitLiteral {
    has EmitLiteral $.unary;
    has VMessage @.cascades;
    # NB: @cascades is inhabited!
    method prepareBind($name, $compiler) {
        my $receiver = $.unary.prepareLiteral: $compiler;
        my $last = @.cascades[*-1];
        for @.cascades[0 ...^ @.cascades.elems - 1] {
            my ($verb, @row= $_.prepareMessage: $compiler;
            $compiler.callOnly: $receiver, $verb, @row;
        };
        my ($verb, @row= $last.prepareMessage: $compiler;
        $compiler.callBind: $name, $receiver, $verb, @row;
    }
    method prepareOnly($compiler) {
        my $receiver = $.unary.prepareLiteral: $compiler;
        for @.cascades {
            my ($verb, @row= $_.prepareMessage: $compiler;
            $compiler.callOnly: $receiver, $verb, @row;
        };
    }
    method prepareLiteral($compiler) {
        my $name = $compiler.gensym;
        self.prepareBind: $name, $compiler;
        "\$" ~ $name;
    }
}

A first compiler

We’ll start by compiling just one block. Our compiler will include a gensym: a method which can generate a symbol that hasn’t been used before. I’m not trying very hard here and it would be easy for a malicious user to access generated symbols; we can fix that later. The compiler is mostly going to store calls; each call can either be a backtick or an if (or foreground) depending on whether it binds a name.

class Compiler {
    has Int $!syms;
    method gensym { $!syms += 1; "gs" ~ $!syms; }

    has Str %.lines;
    method push($line) { %.lines
 ~= $line ~ "\n"; }

    method callBind($name, $receiver, $verb, @args) {
        self.push: "backtick -E $name \{ " ~ formatCall($receiver, $verb, @args~ " \}";
    }
    method callOnly($receiver, $verb, @args) {
        self.push: "if \{ " ~ formatCall($receiver, $verb, @args~ " \}";
    }

    method assignName($from, $to) { self.push: "define $to \$$from"; }
}

The method .assignName is needed to handle assignments without intermediate calls, as in this := that.

class VName does Call does EmitLiteral {
    has Str $.n;
    method prepareBind($name, $compiler) { $compiler.assignName: $.n, $name; }
    method prepareOnly($compiler) {;}
    method prepareLiteral($compiler) { "\$" ~ $.n; }
}

Calling into Vixen

To compile multiple blocks, we will need to emit multiple blocks. A reasonable approach might be to emit a JSON Object where each key is a block name and each value is a String containing the compiled block. I’m feeling more adventurous than that, though. Here’s a complete Smalltix/Vixen FFI:

sub callVixen($receiver, $verb, *@args) {
    my $proc = run %*ENV ~ "/call:", $receiver, $verb, |@args, :out;
    my $answer = $proc.out.slurp: :close;
    $proc.sink;
    $answer.trim;
}

Vixen is merely a calling convention for processes; we can send a message to an object by doing some string formatting and running a subprocess. The response to a message, called an answer, is given by stdout. Non-zero return codes indicate failure and stderr will contain useful information for the user. The rest of the calling convention is handled by passing envp and calling the V/call: entrypoint.

In addition to passing V in the environment, we will assume that there are Allocator and NixStore objects. Allocator allocates new objects and NixStore interacts with the Nix package manager; we will allocate a new object and store it in the Nix store. The relevant methods are V/clone: anAllocator, which allocates a shallow copy of V and serves as a blank object template, and NixStore/intern: anObject, which recursively copies an object from a temporary directory into the Nix store.

The reader doesn’t need to know much about Nix. The only relevant part is that the Nix store is a system-wide immutable directory that might not be enumerable; it’s a place to store packages, but it’s hard to alter packages or find a package that the user hasn’t been told about.

Name analysis

We will need to know whether a name is used by a nested block. When we create an object representing that block, we will provide that object with each name that it uses. This is called name-use analysis or just use analysis and it is a type of name analysis. The two effects worth noting are when an expression uses a name and when a statement assigns to a name. We track the used names with a Set[Str]. For example, a keyword message uses a name if any of its arguments use a name:

class VMessage {
    has VKeyword @.keywords;
    method uses(--> Set[Str]) { [(|)@.keywords.map({ $_.uses }) }
}

A sequence of expressions has its usage computed backwards; every time an expression is assigned to a name, we let that assignment shadow any further uses by removing it from the set of used names. This can be written with reduce but it’s important to preserve readability since this sort of logic can be subtly buggy and often must be revisted during debugging.

class VExprs {
    has EmitStatement @.statements;
    has EmitLiteral $.rv;
    method uses(--> Set[Str]) {
        my $names = $.rv.uses;
        for @.statements.reverse {
            $names = ($names (-) $_.writes) (|) $_.call.uses;
        };
        $names;
    }
}

The .writes method merely produces the set of assigned names:

class VAssignName does EmitStatement {
    has Str $.target;
    method writes(--> Set[Str]) { Set[Str].new($.target) }
}

A second compiler

We now are ready to compile nested blocks. The overall workflow is to compute a closure for the inner block whose names are all used names in the block, except for parameters and global names. We rename everything in the closure with fresh symbols to avoid clashes and allow names like “self” to be closed over. We produce two scripts. One script accepts the closure’s values and attaches them to a new object; one script loads the closure and performs the action in the nested block upon the new object. We call into Vixen to allocate the prototype for the block, populate it, and intern it into the Nix store. Everything else is support code.

        my $closureNames = $uses (-) ($params (|) %globals);
        my %closure = $closureNames.keys.map: { $_ => $compiler.gensym ~ "_" ~ $_ };
        my $valueVerb = @.params ?? "value:" x @.params.elems !! "value";
        my $closureVerb = %closure ?? %closure.keys.map(* ~ ":").join !! "make";
        my $valueBlock = produceValueBlock($compiler, %closure, @.params, $.exprs);
        my $closureBlock = cloneForClosure($compiler, %closure);
        my $obj = callVixen(%*ENV, "clone:", $allocator);
        $compiler.writeBlock: $obj, $valueVerb, $valueBlock;
        $compiler.writeBlock: $obj, $closureVerb, $closureBlock;
        my $interned = callVixen(%*ENV, "intern:", $obj);

One hunk of support code is in the generation of the scripts with produceValueBlock and cloneForClosure. These are open-coded actions against the $compiler object:

sub cloneForClosure($compiler, %closure) {
    my $name = $compiler.genblock;
    $compiler.pushBlock: $name, %closure.keys;
    my $obj = $compiler.gensym;
    my $selfName = $compiler.useName: "self";
    $compiler.callBind: $obj, $selfName, "clone:", ($allocator,);
    my $rv = $compiler.useName: $obj;
    for %closure.kv -> $k, $v {
        my $arg = $compiler.useName: $k;
        $compiler.foreground: "redirfd -w 1 $rv/$v echo " ~ $arg;
    }
    $compiler.finishBlock: $rv;
    $name;
}
sub produceValueBlock($compiler, %closure, @params, $exprs) {
    my $name = $compiler.genblock;
    $compiler.pushBlock: $name, @params;
    my $selfName = $compiler.useName: "self";
    for %closure.kv -> $k, $v { $compiler.callBind: $k, $selfName, $v, [] };
    my $rv = $exprs.compileExprs: $compiler;
    $compiler.finishBlock: $rv;
    $name;
}

The compiler was augmented with methods for managing scopes of names and reassigning names, so that the define helper is no longer used at all. There’s also a method .writeBlock which encapsulates the process of writing out a script to disk.

class Compiler {
    has Hash[Str@.scopes;
    method assignName($from, $to) { @.scopes[*-1]{ $to } = $from }
    method useName($name) {
        for @.scopes.reverse {
            return "\$\{" ~ $_$name } ~ "\}" if $_$name }:exists;
        };
        die "Name $name not in scope!";
    }
    method writeBlock($obj, $verb, $blockName) {
        spurt "$obj/$verb", %.lines$blockName }.trim-leading;
        chmod 0o755, "$obj/$verb";
    }
}

Closing thoughts

This compiler is less jank than the typical compiler. There’s a few hunks of duplicated code, but otherwise the logic is fairly clean and direct. Raku supports a clean compiler mostly by requiring a grammar and an action class; I had started out by writing imperative spaghetti actions, and it was up to me to decide to organize further. To optimize, it might be worth virtualizing assignments so that there is only one convention for calls; this requires further bookkeeping to not only track renames but also name usage. Indeed, at that point, the reader is invited to consider what SSA might look like. Another possible optimization is to skip allocating empty closures for blocks which don’t close over anything.

It was remarkably easy to call into Vixen from Raku. I could imagine using the FFI as scaffolding to incrementally migrate this compiler to a self-hosting expression-language representation of itself. I could also imagine extending the compiler with FFI plugins that decorate or cross-cut compiler actions.

This blogpost is currently over 400 lines. The full compiler is under 400 lines. We put Raku into a jar with Unix, Smalltalk, and Nix; shook it up, and let it sit for a few days. The result is a humble compiler for a simple expression language with a respectable amount of spice. Thanks to the Raku community for letting me share my thoughts. To all readers, whether you’re Pastafarian or not, whether you prefer red sauce or white sauce or even pesto, I hope that you have a lovely and peaceful Holiday.

Day 14 – Taming Concurrency

Hello everyone and a merry advent time!

Today I’d like to show case a neat little mechanism that allows building concurrent applications without having to worry about the concurrency much. I’ve built this tool as part of the (yet to be released) Rakudo CI Bot. That’s a middle man that watches GitHub for things to test (both via API hooks that GitHub calls, as well as a regularly called puller), pushes test tasks to one or more CI backends (e.g. Azure), monitors those and pulls results once available and reports the results back to GitHub. It has to deal with a lot of externally triggered, thus naturally concurrent events.

So there is a fundamental need to bring synchronity into the processing of these events so things don’t step on each others toes. This is how I did it.

I’ve created components (simple singleton classes) that each is responsible for one of the incoming event sources. Those are:

  • The GitHubCITestRequester that receives CITests requests seen on GitHub both, via a poller and a web hook that GitHub itself calls.
  • The OBS backend that pushes test tasks to the Open Build Service and monitors it for test results.
  • The CITestSetManager that connects the components. It reads test tasks from the DB and sends them off to the test backends. It also monitors the results and acts whenever all tests for a task completes.

Data is persisted in a DB. Each table and field lies in the responsibility of one of these components. So naturally, because the responsibility of each data unit is clear the components are nicely separated. But the tools often need to hand work over to some other component. They do so by creating the respective DB entries (either new rows or filling out fields) and then notifying that other component that data is waiting to be processed.

This notification needs to check a few boxes:

  • The processing should start as soon as possible once work is waiting.
  • There should always only be a single worker processing the work of a component. This simplifies the design a lot as I don’t have to synchronize workers.
  • The notifiers shouldn’t be blocked.

I modelled the above via a method on the each of the above mentioned component classes called process-worklist. That method looks at the DB, retrieves all rows that want to be processed and does what needs to be done.

But this method can be called from multiple places and there is no guarantee that no worker is active when a call is done.

So I need a mechanism that ensures that

  • a call never blocks,
  • the method is run when it’s called and no run is in progress,
  • when a run is already in progress, queues another run right after the running one finishes,
  • but doesn’t pile up runs, because a single run will see and process all the work that was added in the mean time.

I achieved all of the above with a method trait called is serial-dedup. Add that trait to a method and it behaves exactly as described above. This is the code:

unit module SerialDedup;

my class SerialDedupData {
    has Semaphore $.sem is rw .= new(1);
    has $.run-queued    is rw = False;
}

my role SerialDedupStore {
    has SerialDedupData %.serial-dedup-store-variable;
}

multi sub trait_mod:<is>(Method $r, :$serial-dedupis export {
    my Lock $setup-lock .= new;
    $r.wrap(my method ($obj:) {
        my $d;
        $setup-lock.protect: {
            if !$obj.does(SerialDedupStore) {
                $obj does SerialDedupStore;
            }
            $d := $obj.serial-dedup-store-variable{$r.name//= SerialDedupData.new;
        }

        if $d.sem.try_acquire() {
            my &next = nextcallee;
            $d.run-queued = False;
            start {
                &next($obj);
                $d.sem.release();
                $obj.&$r() if $d.run-queued;
            }
        }
        else {
            $d.run-queued = True;
        }
    });
}

It works by wrapping the method the trait was added to. State is persisted in a field injected into the object by doesing a Role holding that field. The rest of the code is a pretty straight forward implementation of the required semantics. It tracks whether a call is running via the $.sem semaphore, and keeps a note whether another run was requested (by a concurrent call to the method) in the $.run-queued boolean. And
always runs the wrapped method in a separate thread.

All in all, I love how straight forward, easy and short the code doing this is. Especially when considering that this mechanism is the heart of the concurrency handling of a non-trivial application.

Day 13 – Christmas Crunching Part II

Christmas is nearly on us, 10 shopping days to go. Things at the pole were gathering pace, with so much left to do.

Rudolph (him again) was pacing up and down, mashing his cheroot. He cast his mind back to the App::Crag calcs he had done last time – sure all the distances, times and speeds had worked out. But something was still missing. Could he be sure that all the prerequisites were finalised, that the crucial flight would be a success again this year?

Then click, his nose lit up like a lightbulb – what about the fuel. Would they have enough juice in the powerplant to maintain optimium power output throughout the entire long night?

Once again, he cracked open his laptop and his hooves whirred away on the keys.

Present Power

He realised that vertical work against the pull of Earth’s gravity would be the key factor, Santa would need his winch to lower and raise him back up every chimney on the planet…

Green Christmas

New this year was the installation of a large, Lithium ion battery pack in the sleigh. Rudi wondered how many wind turbines would be needed in Lapland to charge up the battery and the capacity of the cabling required…

Rudolph nodded sagely and lit his pipe, it would be alright on the night after all.

Rudolph’s calm and cheery,
ready from the flight—
he’s puffing on his cherrywood pipe,
glowing in the night.

~librasteve


Credits

Some of the App::Crag features in play tonight were:

  • ?<some random LLM query>
  • ^<25 mph> – a standard crag unit
  • ?^<speed of a diving swallow in mph> – put them together to get units
  • 25km – a shortcut if you have simple SI prefixes and units
  • $answer = 42s – crag is just vanilla Raku with no strict applied

Checkout the crag-of-the-day for more – but beware, this is kinda strangely addictive.

Day 12 – Mathematician’s Yahtzee

Santa was playing and losing yet another game of Yahtzee with Mrs. Claus and the elves when a thought occurred to him: why don’t you get points for rolling the first 5 digits of the Fibonacci sequence (1, 1, 2, 3, 5)? For that matter, why isn’t there a mathematician’s version of Yahtzee where you get points for rolling all even numbers, all odd, etc. After a bit of brainstorming, Santa came up with the following rules:

  • Rolling all odd numbers replaces 3 of a kind (because ‘odd’ has 3 letters).
  • Rolling all even numbers replaces 4 of a kind (because ‘even’ has 4 letters).
  • Rolling pi (a 3, 1, and 4) replaces a full house.
  • Rolling all prime numbers replaces a small straight.
  • Rolling the Fibonacci sequence (1, 1, 2, 3, 5) replaces a large straight.
  • A Yahtzee (all five of the same number) is still a Yahtzee.

Santa and the others tried an experimental game with these rules, but it was more difficult than they expected. Everyone was so used to identifying a small/large straight, full house, etc, that it was hard to break those habits and identify the new patterns. So, Santa decided to write a Raku program to help him practice.

To start with, Santa created a function that uses the roll method to simulate rolling an arbitrary number of dice, then called it with 5 dice and printed the output:

sub roll-dice($n-dice) {
    return (1..6).roll($n-dice);
}

my @dice = roll-dice(5);
print-summary(@dice);

The print-summary function will be defined later.

Next up is allowing the user to re-roll an arbitrary number of dice up to two times. The user is prompted to enter which numbers they want to re-roll, using spaces to separate each number. To stop, the user can simply press enter. Getting input from the user is done via the prompt function.

for 1..2 -> $reroll-count {
    my $answer = prompt("> Which numbers would you like to re-roll? ");
    last if !$answer;
    my @indices;
    for $answer.split(/\s+/-> $number {
        my $index = @dice.first: $number, :k;
        if !$index.defined {
            note "Could not find number $number";
            exit 1;
        }
        @indices.push: $index;
        @dice[$index]:delete;
    }
    for @indices -> $index {
        @dice[$index= roll-dice(1).first;
    }
    print-summary(@diceif $reroll-count == 1;
}

print-summary(@dice);

A few notes on the above code:

  • Once a user’s input is received, it’s split on whitespace, looping over each number.
  • The first method finds the first die with that number, while the :k adverb makes first return the index of that number (rather than the number itself).
  • If the number isn’t found, note is used to print to standard error and then the program exits.
  • The index is tracked by pushing it onto an array, and then the die value is deleted.
  • Using the delete adverb on an array creates a hole at that index. That’s okay, though, because the next loop fills it in with a new roll. Note that the roll-dice function always returns a Seq, even when rolling a single die, so the first method is used to get the first (and only) value from the sequence.

At this point Santa is halfway done. He doesn’t just want to print out the dice that were rolled, though, he also wants to identify all the new rules the dice match. To start with, identifying a Yahtzee in an array of dice is simple: just count the number of unique values. If it’s one, it’s a Yahtzee:

@dice.unique.elems == 1

For a Fibonacci sequence, Santa uses a Bag, which is a collection that keeps track of duplicate values. Two bags can be compared with the eqv (equivalence) operator. (Santa learned the hard way you can’t use == here because it turns a Bag into a Numeric, which would be the number of elements in the Bag).

@dice.Bag eqv (1, 1, 2, 3, 5).Bag

For pi, Santa brushes up on his rusty set theory. After some research, he makes use of the ⊂ (subset of)  operator, which returns true if all the left-side elements are in the right-side elements (and the right side must have more elements, which it will since there are 5 dice).

(3, 1, 4)  @dice

All prime numbers are easily identified by combining the all method with is-prime. This creates an all Junction that runs is-prime on each value and collapses into a single boolean.

@dice.all.is-prime

Something similar can be done for all even numbers, this time using the %% (divisibility) operator, which returns true if the left side is evenly divisible by the right side.

@dice.all %% 2

For all odd numbers, none is used instead of all.

@dice.none %% 2

Finally, putting it all together into a function results in this:

sub print-summary(@dice where *.elems == 5) {
    with my @summary {
        .push: 'yahtzee'   if @dice.unique.elems == 1;
        .push: 'fibonacci' if @dice.Bag eqv (1, 1, 2, 3, 5).Bag;
        .push: 'pi'        if (3, 1, 4)  @dice;
        .push: 'all-prime' if @dice.all.is-prime;
        .push: 'all-even'  if @dice.all %% 2;
        .push: 'all-odd'   if @dice.none %% 2;
    };
    my $output = @dice.join(' ');
    if ( @summary ) {
        $output ~= " (@summary.join(', '))";
    }
    say $output;
}

Note the signature to the function uses a type constraint (where *.elems == 5) to prove there are always 5 dice; an error is thrown if there are not 5 elements.

Here’s a sample run of the program:

$ raku math-yahtzee.raku
3 3 1 6 2
> Which numbers would you like to re-roll? 3 6
4 3 1 5 2 (pi)
> Which numbers would you like to re-roll? 4
1 3 1 5 2 (fibonacci)

This is exactly what Santa wanted to help him practice Mathematician’s Yahtzee! He had a lot of fun writing the program, especially the print-summary function, and is especially impressed that the program did not need to use any modules. There’s plenty more that could be done to improve this program, but Santa will stop here.

Day 11 – Raku To The Stars

Datastar is a hypermedia systems library in the same tradition as htmx, Unpoly, Alpine AJAX, and Hotwire‘s Turbo. These libraries are generally Javascript/Typescript bundles that utilize the HTML standard to allow you to declaratively write AJAX calls or JS-powered CSS transitions as HTML tag attributes instead of hand-writing Javascript. @librasteve has been working on Air and the HARC stack which seeks to deeply integrate htmx into the Raku ecosystem, and so I highly recommend reading his posts to get a better understanding of why Hypermedia always been a compelling sell.

htmx in particular makes no prescription on handling browser-side, component-local state. Carson Gross, the creator of htmx, lists Alpine, his own project hyperscript, Web Component frameworks such as lit, and several others as options depending on the use case.

A Datastar Primer

Datastar does not take this approach; it aims to handle both browser-side state using signals and the server-side connectivity which htmx and Phoenix LiveView do. The main differentiating factor for Datastar is that it automatically handles Server-Sent Events (SSE) and text/event-stream responses, making it really good for real-time applications. Datastar also allows you to return a regular text/html response just like htmx; it morphs the HTML fragment into the DOM using a forked version of the same DOM-morphing library htmx uses. Datastar also accepts a application/json response from the server which it uses to patch signals (which are JSON objects), and it also accepts a text/javascript response from the server, which it uses to run custom Javascript on the browser.

Raku ❤ Datastar

Finally realizing I don’t have to write any React code in my side projects, I wrote a Datastar library in Raku, combining the two things I’ve recently taking a liking to.

Raku’s best on display

Multimethods and Multisubs

Let’s take a walk through the actual code and see how I’ve utilized Raku’s expressivity, starting with the use of multi subs and multi methods:

multi patch-signals(
    Str $signals, 
    Str :$event-id, 
    Bool :$only-if-missing, 
    Int :$retry-duration
is export {
    ...
}

multi patch-signals(%signals, *%options) {
    samewith to-json(%signals, :!pretty), |%options;
}

The reason we use multi subs here is to allow for the possibility that the signals may be serialized into a Str before patch-signals is called. However, assuming an Associative is passed as the first argument, we just turn it into a Str and call the same function.

Metaoperators

Let’s also take a look at the body of patch-signals, starting with this:

my @signal-lines = 
    "{SIGNALS-DATALINE-LITERAL} " X$signals.split("\n");

I love this line. Let’s start with what this code does; it prepends the word signal to every line of actual stringified JSON. So for example if we have this as our sample stringified JSON:

{
    "selected-element-index": 0,
    "selected-element-value": "Nominative"
}

The actual output of the first line will the following split per-line:

signal {
signal     "selected-element-index": 0,
signal     "selected-element-value": "Nominative"
signal } 

which is what Datastar expects when parsing signals being sent downstream from the server. We’re using X~ the same way Bruce Gray outlines it here, as a scaling/map operator to prepend "signal" to each line of the stringified JSON, an incredibly clever use of the cross-product meta-operator.

Another meta-operator we use is the reduction metaoperator [], mainly to join the lines of the resulting SSE response into one string:

class SseGen {
    method Str { [~@!data-lines }
}

DSL Building using dynamic variables and blocks

Let’s examine patch-signals‘s first statement:

fail 'You can only call this method inside a datastar { } block' 
    unless $*INSIDE-DATASTAR-RESPONSE-GENERATOR;

We use dynamic variables to help enforce the usage of these functions within the datastar block as shown here:

datastar {
    patch-signals { current-index => 0 };
}

sub datastar is defined as:

sub datastar(&fis export {
    my $*INSIDE-DATASTAR-RESPONSE-GENERATOR = True;
    my $*render-as-supply = False;
    my SseGen $*response-generator .= new;

    f();

    $*render-as-supply  
      ?? $*response-generator.Supply 
      !! $*response-generator.Str
}

We make use of blocks and dynamic variables here to have an implicit variable be passed down to functions that make use of it, so that we get very easy to read code. This is largely inspired from what I saw in Matthew Stuckwisch’s presentation here.

A Few More Ideas

Another improvement I had in mind was modifying the multi functions so that you could theoretically write code in a functional manner using the feed operators like this if you wanted:

SseResponse.new
    ==> patch-elements("<div></div>")
    ==> patch-signals({ a => 2, b => 3 })
    ==> as-supply();

You would theoretically import this functionality by adding the following to the top of your file: use DataStar :functional;

Another idea that I had was the following:

given SseGen.new {
    .patch-elements: "<div>Hello there</div>";
    .patch-signals: { a => 2, b => 3 };

    .Supply
}

This makes use of the given control structure which assigns an expression to the topic variable $_, and Raku’s syntax sugar for calling methods on $_ which is to just invoke method without a receiver/object.

I hope to make it easier for developers using this library to follow the TMTOWTDI philosophy by providing multiple strategies for interfacing with the library.

Next Steps

Here are the next steps regarding this library:

  • Run the full Datastar test suite against this library to make sure I’m on the same page.
  • Rename DataStar to Datastar. Datastar is the preferred name for the package and I incorrectly named it when I was in a rush to release this package.
  • Integrate this with Cro through a new package Cro::Datastar and integrate it with Air via a new package: Air::Jetstream.

Happy holidays everyone!

Day 10 – Santa’s Finance Department

Cron Meets Raku

The Finance Department’s computers had been converted to Debian Linux with Rakuized software along with all the other departments at the North Pole headquaters, and its employees enjoyed the Windows-free environment. However, an inspection by a firm hired to evaluate efficiency practices found some room for improvement.

Much of the work day was spent perusing email (as well as some postal mail) and entering purchase and payment information into the accounting software. The review team suggested that more automation was possible by investing in programs to (1) extract the information from the emails and (2) use optical character recognition (OCR) on digital scans of paper invoices.

The review team briefed Santa Claus and his department heads after their work was finished. After the team departed, Santa asked the IT department to assist the finance department in making the improvements.

Note the IT department is now using ChatGPT as a programming aid, so some of the new projects rely on it heavily for assistance in areas of little expertise as well as handling boiler plate (e.g., boring) coding. But any code used is tested thouroughly.

Extracting data from email

Gmail is the email system used currently with an address of “finance.santa@gmail.com” for the department. All bills and correspondence with external vendors use that address.

Normally Raku would be the programming language of choice, but Python is used for the interaction with Gmail because Google has well-defined Python APIs supported by Google.

In order to access Gmail programmatically, we need a secret token for our user. Following is the one-time interactive process using Python:

cd /path/to/gmail-finance-ingest # a directory to contain most Python code
python3 -m venv .venv
. .venv/bin/activate
pip install .
gmail-finance-ingest init-auth \
  --credentials=/home/finance/secret/google/credentials.json \
  --token=/home/finance/secret/google/token.json

That launches the browser, the user approves access, that token is saved. After that, no more interaction is needed; cron (the Linux scheduler) can use the same token.

In order to handle the mail approriately, we use a yaml file to identify expected mail and its associated data as shown here in example file config.yml:

data_root: /home/finance/gmail-bills

sources:
  - name: city-utilities
    gmail_query: 'from:(billing@mycity.gov) has:attachment filename:pdf'
    expect: pdf
    subdir: city-utilities

  - name: electric-utility
    gmail_query: 'from:(noreply@powerco.com) subject:(Your bill) has:attachment filename:pdf'
    expect: pdf
    subdir: electric-utility

  - name: amazon
    gmail_query: 'from:(order-update@amazon.com OR auto-confirm@amazon.com)'
    expect: email
    subdir: amazon

Following is the bash script to handle the finance department’s config file:

. /home/finance/path/to/gmail-finance-ingest/.venv/bin/activate
gmail-finance-ingest sync \
  --config=/home/finance/gmail-finance-config.yml \
  --credentials=/home/finance/secret/google/credentials.json \
  --token=/home/finance/secret/google/token.json

Automating the process

Linux cron is used to automate various email queries, thus saving a lot of manual, boring work by staff.

cron is a time-based job scheduler in Linux and other Unix-like operating systems. It enables users to schedule commands or scripts (known as cron jobs) to run automatically at specific times, dates, or intervals.

The driver program is a Python package named gmail_finance_ingest.

Here is the bash script used to operate on emails:

#!/bin/bash
set -e

LOGDIR="$HOME/log"
mkdir -p "$LOGDIR"

# Activate venv
. "$HOME/path/to/gmail-finance-ingest/.venv/bin/activate"

gmail-finance-ingest sync \
  --config="$HOME/gmail-finance-config.yml" \
  --credentials="$HOME/secret/google/credentials.json" \
  --token="$HOME/secret/google/token.json" \
  >> "$LOGDIR/gmail-sync.log" 2>&1

Following is the cron code used to update email scans daily:

15 3 * * * /home/finance/bin/run-gmail-sync.sh

For processing the data we handle several types which are identified in the expected emails and identified in the config file by the keywords shown below:

  1. text embedded in the mail – expect=email
  2. PDF attachments – expect=pdf
  3. attachments or enclosed chunks of scanned documents – expect=ocr

Type 3 is not yet handled.

The collected data is parsed by type and the pertinent output is placed in CSV tables for bookkeeping purposes. Such tables can be used as source for Linux accounting programs like GnuCash. The department has been using that program sincs the big Linux/Debian transition.

Emails which cannot be evaluated by machine are automatically forwarded to designated department staff to handle manually.

Other work

The IT folks have other projects not formally published yet, but some are in final testing stage and are usable now. See the summary below for related Raku projects such as an Access-like database program and a check-writing program.

Summary

The products mentioned above are still works-in-progress, but their development can be followed on GitHub now at:

An automated email interrogator for known sendees.

A program to print a business-size check on a standard single-check form available from ?

An Access-like relational database management system capable of using CSV tables as a backing store.

Epilogue

Don’t forget the “reason for the season:” ✝

As I always end these jottings, in the words of Charles Dickens’ Tiny Tim, “may God bless Us, Every one!” [2]

Footnotes

A Christmas Carol, a short story by Charles Dickens (1812-1870), a well-known and popular Victorian author whose many works include The Pickwick Papers, Oliver Twist, David Copperfield, Bleak House, Great Expectations, and A Tale of Two Cities.