After reading Santa’s horrible code, Lizzybel thought it might be time to teach the elves (and maybe Santa) a little bit about hypering and racing in the Raku Programming Language.
So they checked with all the elves that were in their latest presentation, to see if they would be interested in that. ”Sure“, one of them said, “anything that will give us some more free time, instead of having to wait for the computer!“ So Lizzybel checked the presentation room (still empty) and rounded up the elves that could make it. And started:
A simple case
Since we’re going to mostly talk about (wallclock) performance, I will add timing information as well (for what it’s worth: on an M1 (ARM) processor that does not support JITting).
Let’s start with a very simple piece of code, to understand some of the basics: please give me the 1 millionth prime number!
$ time raku -e 'say (^Inf).grep(*.is-prime)[999999]' 15485863 real 6.41s user 6.40s sys 0.04s
Looks like 15485863 is the 1 millionth prime number! What we’re doing here is taking the infinite list of 0 and all positive integers (^Inf), select only the ones that are prime numbers (.grep(*.is-prime)), put the selected ones in a hidden array and show the 1 millionth element ([999999]).
As you can see, that takes 6.41 seconds (wallclock). Now one thing that is happening here, is that the first 999999 prime numbers are also saved in an array. So that’s a pretty big array. Fortunately, you don’t have to do that. You can also skip the first 999999 prime numbers, and then just show the next value, which would be the 1 millionth prime number:
$ time raku -e 'say (^Inf).grep(*.is-prime).skip(999999).head' 15485863 real 5.38s user 5.39s sys 0.02s
This was already noticeable faster: 5.38s! That’s already about 20% faster! If you’re looking for performance, one should always look for things that are done needlessly!
This was still only using a single CPU. Most modern computers nowadays have more than on CPU. Why not use them? This is where the magic of .hyper comes in. This allows you to run a piece of code on multiple CPUs in parallel:
$ time raku -e 'say (^Inf).hyper.grep(*.is-prime).skip(999999).head' 15485863 real 5.01s user 11.70s sys 1.42s
Well, that’s… disappointing? Only marginally faster (5.38 -> 5.01)? And use more than 2 times as much CPU time (5.41 -> 11.70)?
The reason for this: overhead! The .hyper adds quite a bit overhead to the execution of the condition (.is-prime). You could think of .hyper.grep(*.is-prime) as .batch(64).map(*.grep(*.is-prime).Slip).
In other words: create batches of 64 values, filter out the prime numbers in that batch and slip them into the final sequence. That’s quite a bit of overhead compared to just checking each value for prime-ness. And it shows:
$ time raku -e 'say (^Inf).batch(64).map(*.grep(*.is-prime).Slip).skip(999999).head' 15485863 real 9.55s user 9.55s sys 0.03s
That’s roughly 2x as slow as before.
Now you might ask: why the 64? Wouldn’t it be better if that were a larger value? Like 4096 or so? Indeed, that makes things a lot better:
$ time raku -e 'say (^Inf).batch(4096).map(*.grep(*.is-prime).Slip).skip(999999).head' 15485863 real 7.75s user 7.75s sys 0.03s
The 64 is the default value of the :batch parameter of the .hyper method. If we apply the same change in size to the .hyper case, the result is a lot better indeed:
$ time raku -e 'say (^Inf).hyper(batch => 4096).grep(*.is-prime).skip(999999).head' 15485863 real 1.22s user 2.79s sys 0.06s
That’s more than 5x as fast as the original time we had. Now, should you always use a bigger batch size? No, it all depends on the amount of work that needs to be done. Let’s take this extreme example, where sleep is used to simulate work:
$ time raku -e 'say (^5).map: { sleep $_; $_ }' (0 1 2 3 4) real 10.18s user 0.14s sys 0.03s
Because all of the sleeps are executed consecutively, this obviously takes 10+ seconds, because 0 + 1 + 2 + 3 + 4 = 10. Now let’s add a .hyper to it:
$ time raku -e 'say (^5).hyper.map: { sleep $_; $_ }' (0 1 2 3 4) real 10.19s user 0.21s sys 0.05s
That didn’t make a lot of difference, because all 5 values of the Range are slurped into a single batch because the default batch size is 64. These 5 values are then processed, causing the sleeps to still be executed consecutively. So this is just adding overhead. Now, if we change the batch size to 1:
$ time raku -e 'say (^5).hyper(batch => 1).map: { sleep $_; $_ }' (0 1 2 3 4) real 4.19s user 0.19s sys 0.03s
Because this way all of the sleeps are done in parallel, we only need to wait just over 4 seconds for this to complete, because that’s the longest sleep that was done.
Question: what would the wallclock time at least be in the above example if the size of the batch would be 2?
In an ideal world the batch-size would adjust itself automatically to provide the best overhead / throughput ratio. But alas, that’s not the case yet. Maybe in a future version of the Raku Programming Language!
Racing
“But what about the race method?“, asked one of the elves, “what’s the difference between .hyper and .race?“ Lizzybel answered:
The .hyper method guarantees that the result of the batches are produced in the same order as the order in which the batches were received. The .race method does not guarantee that, and therefore has slightly less overhead than .hyper.
You typically use .race if you put the result into a hash-like structure, like Santa did in the end, because hash-like structures (such as a Mix) don’t have a particular order anyway.
Degree
“But what if I don’t want to use all of the CPUs” another elf asked. “Ah yes, almost forgot to mention that“, Lizzybel mumbled and continued:
By default, .hyper and .race will use all but one of the available CPUs. The reason for this is that you need one CPU to manage the batching and reconstitution. But you can specify any other value with the :degree named argument, and the results may vary:
$ time raku -e 'say (^Inf).hyper(batch => 4096, degree => 64).grep(*.is-prime).skip(999999).head' 15485863 real 1.52s user 3.08s sys 0.06s
That’s clearly slower (1.22 -> 1.52) and takes more CPU (2.79 -> 3.08).
Because the :degree argument really indicates the maximum number of worker threads that will be started. If that number exceeds the number of physical CPUs, then you will just have given the operating system more to do, shuffling the workload of many threads around on a limited number of CPUs.
Work
At that point Santa, slightly red in the face, opened the door of the presentation room and bellowed: “Could we do all of this after Christmas, please? We’re running behind schedule!“. All of the elves quickly left and went back to work.
The answer is: 5. Because the first batch (0 1) will sleep 0 + 1 = 1 second, the second batch (2 3) will sleep 2 + 3 = 5 seconds, and the final batch only has 4, so will sleep for 4 seconds.
In this document we proclaim recent creation and updates of several Raku packages that facilitate the utilization of the OpenAI’s DALL-E 3 model. See [AAp1, AAp3, AAp4, AAp5].
The exposition of this document was designed and made using the Jupyter framework — we use a Raku Jupyter chat-enabled notebook, [AAp2].
Chat-enabled notebooks are called chatbooks.
We discuss workflows and related User eXperience (UX) challenges, then we demonstrate image-generation workflows.
The demonstrations are within a Raku chatbook, see “Jupyter::Chatbook”, [AAp4]. That allows interactive utilization of Large Language Models (LLMs) and related Artificial Intelligence (AI) services.
In the past, we made presentations and movies using DALL-E 2; see [AAv1, AAv2]. Therefore, in this document we use DALL-E 3 based examples. Nevertheless, we provide a breakdown table that summarizes and compares the parameters of DALL-E 2 and DALL-E 3.
Since this document is written at the end of the year 2023, we use the generation of Christmas and winter themed images as examples. Like the following one:
#% dalle, model=dall-e-3, size=landscape Generate snowfall with a plain black background. Make some of the snowflakes look like butterflies.
Document structure
Here is the structure of the document:
Image generation related workflows
Implementation and UX challenges
Image generation examples
Showing how the challenges are addressed.
Programmatic example
Via AI vision.
Breakdown table of DALL-E 2 vs DALL-E 3 parameters
Workflows
Just generating images
The simplest workflow is:
Pick a DALL-E model.
Pick image generation parameters.
Choose the result format:
URL — the image is hosted remotely
Base64 string — the image is displayed in the notebook, [AAp5], and stored locally
Generate the image.
If not satisfied or tired goto 1.
Review and export images
It would be nice, of course, to be able to review the images generated within a given Jupyter session and export a selection of them. (Or all of them.)
Here is a “single image export” workflow:
Review generated (and stored) images in the current session
Select an image
Export the image by specifying:
Index (integer or Whatever)
File name (string or Whatever))
Here is an “all images export” workflow:
Review generated (and stored) images in the current session
Export all images in the current session
The “automatic” file names in both workflows are constructed with a file name prefix specification (using the magic cell option “prefix”.)
In order to publish this article to GitHub and WordPress we used the “all image export” workflow.
Programmatic utilization
Programmatic workflows tie up image generation via “DALL-E direct access” with further programmatic manipulation. One example, is given in “AI vision via Raku”, [AA3]:
LLM-generate a few images that characterize parts of a story.
Do several generations until the images correspond to “story’s envisioned look.”
Narrate the images using the LLM “vision” functionality.
Use an LLM to generate a story over the narrations.
Challenges and implementations that address them
We can say that a Jupyter notebook provides a global “management session” that orchestrates (a few) LLM “direct access” sessions, and a Raku REPL session.
Here is a corresponding Mermaid-JS diagram:
#% mermaid
graph LR
JuSession{{Jupyter session}} <--> Raku{{"Raku<br>REPL"}}
JuSession --> DALLE{{"DALL-E<br>proxy"}}
DALLE <--> Files[(File system)]
DALLE <--> IMGs[(Images)]
Raku <--> Files
DALLE --> Cb([Clipboard])
Raku <--> Cb
DALLE -.- |How to exchange data?|Raku
subgraph Chatbook
JuSession
Raku
DALLE
IMGs
end
subgraph OS
Files
Cb
end
Remark: Raku chatbooks have Mermaid-JS magic cell that use “WWW::MermaidInk”, [AAp6].
Challenges
Here is a list of challenges to be addressed in order to facilitate the workflows outlined above:
How the Raku REPL session communicates with the DALL-E access “session”?
How an image is displayed in:
Jupyter notebook
Markdown document
What representation of images to use?
Base64 strings, or URLs, or both?
How images are manipulated in Raku?
What kind of manipulations are:
Desireable
Possible (i.e. relatively easy to implement)
How an image generated in a “direct access” Jupyter cell is made available in the running along Raku session?
More generally, how the different magic cells “communicate” their results to the computational cells?
If a Jupyter notebook session stores the generated images:
How to review images?
How to delete images?
How to select an image and export it?
How to export all images?
How to find the file names of the exported images?
Solution outline
Addressing the implementation and UX challenges is done with three principle components:
A dedicated Raku package for handling images in markup environments; see “Image::Markup::Utilities”, [AAp5]. Allows:
Streamlined display of images in Markdown and Jupyter documents
Image import from file path or URL
Image export to file system
Storing of DALL-E generated images within a Jupyter session.
In order to provide the Raku session access to DALL-E session artifacts.
Chatbook magic cell type for meta operations over stored DALL-E images.
In order to review, delete, or export images.
Additional solution elements are:
Export paths are placed in the clipboard.
DALL-E generation cells have the parameter “response-format” that allows getting both URLs and Base64 strings.
If the result is a URL then it is placed in the OS clipboard.
Instead of introducing an image Raku class we consider an image to be “just” a Base64 string or an URL. See [AAp5].
Here we ask the default LLM about Chinese horoscope signs of this and next year:
#% chat, temperature=0.3 Which Chinese horoscope signs are the years 2023 and 2024?
The Chinese zodiac follows a 12-year cycle, with each year associated with a specific animal sign. In 2023, the Chinese zodiac sign will be the Rabbit, and in 2024, it will be the Dragon.
Image generation followed by “local” display
Here we ask OpenAI’s DALL-E system (and service) to generate an image of a dragon chasing a rabbit:
#% dalle, model=dall-e-3, size=landscape
A Chinese ink wash painting of a zodiac dragon chasing a zodiac rabbit. Fractal snow and clouds. Use correct animal anatomy!
The DALL-E magic cell argument “model” can be Whatever of one of “dall-e-2” or “dall-e-3”.
Using remote URL of the generated image
Here we use a different prompt to generate a certain image that is both Raku- and winter related — we request the image of DALL-E’s response to be given as a URL and the result of the chatbook cell to be given in JSON format:
#% dalle, model=dall-e-3, size=landscape, style=vivid, response-format=url, format=json
A digital painting of raccoons having a snowball fight around a Christmas tree.
{ "data": [ { "url": "https://oaidalleapiprodscus.blob.core.windows.net/private/org-KbuLSsqssXAPQFZORGWZzuN0/user-Ss9QQAmz9L5UJDcmKnhxnRoT/img-ResmF1iVDSxEGb6ORTkjy200.png?st=2023-12-18T00%3A46%3A22Z&se=2023-12-18T02%3A46%3A22Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-17T19%3A56%3A04Z&ske=2023-12-18T19%3A56%3A04Z&sks=b&skv=2021-08-06&sig=YWFpfGlDry1L5C8aU1hMUhum02uTvyqwAyOgi3A9ZaU%3D", "revised_prompt": "Create a digital painting showing a merry scene of raccoons engaging in a playful snowball fight. The backdrop should consist of a snow-filled winter landscape illuminated by the soft glow of a magnificently decorated Christmas tree. The raccoons should be exuding joy, raising their tiny paws to throw snowballs, their eyes twinkling with mischief. The Christmas tree should be grand and towering, adorned with vibrant ornaments, strands of sparkly tinsel, and a star at the top, reflecting both tradition and cheer." } ], "created": 1702863982 }
Remark: DALL-E 3 model revises the given prompts. We can paste the result above into a Raku computational cell using the clipboard shortcut, (Cmd-V).
Here we display the image directly as a Markdown image link (using the command paste of “Clipboard“):
#% markdown
my $url = from-json(paste)<data>.head<url>;
""
Alternatively, we can import the image from the given URL and display it using the magic %% markdown (intentionally not re-displayed again):
# %% markdown
my $img = image-import($url);
$img.substr(0,40)
Here we export into the local file system the imported image:
Generally speaking, importing and displaying image Base64 strings is a few times slower than using image URLs.
Meta cells for image review and export
Here we use a DALL-E meta cell to see how many images were generated in this session:
#% dalle meta
elems
2
Here we export the second image (dragon and rabbit) into a file named “chinese-ink-wash.png”:
#% dalle export, index=1
chinese-ink-wash.png
chinese-ink-wash.png
Here we show all generated images:
#% dalle meta
show
Here we export all images (into file names with the prefix “advent2023”):
#% dalle export, index=all, prefix=advent2023
Programmatic workflow with AI vision
Let us demonstrate the OpenAI’s Vision using the “raccoons and snowballs” image generated above:
llm-vision-synthesize("Write a limerick based on the image.", $img)
In a forest so snowy and bright, Raccoons played in the soft moonlight. By a tree grand and tall, They frolicked with a ball, And their laughter filled the night.
Exercise: Verify that limerick fits the image.
The functions llm-vision-synthesize and llm-vision-function were added to “LLM::Functions” after writing (and posting) “AI vision via Raku”, [AA3]. We plan to make a more dedicated demonstration of those functions in the near future.
Breakdown of model parameters
As it was mentioned above, the DALL-E magic cell argument “model” can be Whatever of one of “dall-e-2” or “dall-e-3”.
Not all parameters that are valid for one of the models are valid or respected by the other — see the subsection “Create image” of OpenAI’s documentation.
Here is a table that shows a breakdown of the model-parameter relationships:
Parameter
Type
Required/Optional
Default
dall-e-2
dall-e-3
Valid Values
prompt
string
Required
N/A
✔️
✔️
Maximum length is 1000 characters for dall-e-2 and 4000 characters for dall-e-3
model
string
Optional
dall-e-2
✔️
✔️
N/A
n
integer or null
Optional
1
✔️
✔️ (only n=1)
Must be between 1 and 10. For dall-e-3, only n=1 is supported
quality
string
Optional
standard
❌
✔️
N/A
response_format
string or null
Optional
url
✔️
✔️
Must be one of url or b64_json
size
string or null
Optional
1024×1024
✔️ (256×256, 512×512, 1024×1024)
✔️ (1024×1024, 1792×1024, 1024×1792)
Must be one of 256×256, 512×512, or 1024×1024 for dall-e-2. Must be one of 1024×1024, 1792×1024, or 1024×1792 for dall-e-3 models
Shortly before take off, Rudolph (who wanted to get his facts 100% right) asked his friend ChattyGPT “what are the crucial factors for a successful Sleigh ride?”
Certainly, said Chatty! Here are three calculations for Santa:
Sleigh Speed and Distance Calculation:
Santa needs to calculate the optimal speed of his sleigh to cover the vast distances between homes around the world in a single night. This calculation involves factoring in variables such as the Earth’s rotation, to ensure he can reach every home within the limited time frame.
Gift Weight Distribution:
Santa has to ensure that his sleigh is balanced with the right distribution of gifts. This calculation involves considering the weight of each gift, its size, and the overall weight capacity of the sleigh.
Sleigh Energy Management:
In addition to distributing the gifts, Santa needs to optimize the energy efficiency of his sleigh. This calculation ensures that he can maintain the magical power required for the entire journey, taking into account the various stops and starts during gift deliveries.
How could Rudi check his numbers in time?
Calculator using RAku Grammars (CRAG)
Then he remembered that Raku, his favourite programming language could be used as a powerful Command Line calculator via the App::Crag module.
$ zef install App::Crag
1. Sleigh Speed and Distance Calculation
So, pondered Rudolph, let’s say I start at the North pole and head to the equator, how far is that?
With the French Revolution (1789) came a desire to replace many features of the Ancien Régime, including the traditional units of measure. […] A new unit of length, the metre was introduced – defined as one ten-millionth of the shortest distance from the North Pole to the equator passing through Paris…
$ crag 'say (:<510064471.91 sq km> 19110349.077).in: <sq km>' 26.69sq km per kid
so, if I fly a path to cover all the kids then I need to cover a strip about 26.69km wide, but my total path length is about double that due to zig zagging to visit each chimney…
guess we can ignore the masses of Santa, the Sleigh and the Reindeer then
3. Sleigh Energy Management
Now, we just need enough energy to accelerate (and brake) the Sleigh so that it can stop at each chimbley. And to keep up the average speed Santa needs, the maximum speed will need to be double the average:
errr … seems like we may need to use magic after all!
Under the Hood
I wrote App::Crag in response to a comment from a raku newb – they were looking for “the ultimate command line calculator”. That inspired me to pull together a bunch of raku modules into a unified command and to provide some sugar to avoid too much typing.
As with any raku CLI command, there is built in support for help text, just type ‘crag‘ on your terminal:
For this advent post I will tell you about how I got into the Raku Programming Language and my struggles of making raylib-raku bindings. I already have some knowledge about C, which helped me tremendously when making the bindings. I won’t explain much about Cpassing by value or passing by reference. So I suggest learning a bit of C if you are more interested.
Encountering Raku
I discovered Raku by coincidence in a Youtube video and I got intrigued by how expressive it is. While reading through the docs, the feature that caught my eye was the first class support for grammars!
Trying out the grammar was very intuitive if you have worked with some parser generator where EBNF is used, then it should be quite similar. I will definitely make a toy compiler/interpreter using Raku at some point. Before doing that I wanted to make a chip-8 emulator in Raku and want Raylib for rendering since I have used it before. Sadly there was no bindings for it in raku, but then maybe I could make the bindings?
Making native bindings
So I got a bit sidetracked from making the emulator and began reading through the docs for creating the bindings using Raku NativeCall. I began translating some simple functions in raylib.h to Raku Nativecall. My first attempt was something like this:
use NativeCall;
constant LIBRAYLIB = 'libraylib.so';
class Color is export is repr('CStruct') is rw { has uint8 $.r; has uint8 $.g; has uint8 $.b; has uint8 $.a; }
sub InitWindow(int32 $width, int32 $Height, Str $name) is native(LIBRAYLIB) {*} sub WindowShouldClose(--> bool) is native(LIBRAYLIB) {*} sub BeginDrawing() is native(LIBRAYLIB) {*} sub EndDrawing() is native(LIBRAYLIB) {*} sub CloseWindow() is native(LIBRAYLIB) {*} sub ClearBackground(Color $color) is native(LIBRAYLIB) {*}
my $white = Color.new(r => 255, g => 255, b => 255, a => 255); InitWindow(800, 450, "Raylib window in Raku!"); while !WindowShouldClose() { BeginDrawing(); ClearBackground($white); EndDrawing(); } CloseWindow();
Yay we got a window!
But something is clearly off, the background wasn’t white as I defined it to be.
It turns out that ClearBackground expects Color as value type as shown below:
RLAPI void ClearBackground(Color color);
The problem is Raku NativeCall only supports passing as a reference, not by value!
After asking the community for guidance, I got the solution to pointerize the function. Meaning we need to make a new wrapper function example: ClearBackground_pointerized(Color* color) which takes Color as a pointer and then call the original function with the dereferenced value:
Since Color must be pointer we need to allocate it on the heap using C‘s malloc function. We need to expose a malloc_Color function in C-intermediate code to be able to call this from Raku.
If you malloc you also need to free it, or else we will memory leak. We need to expose another function for calling free on Color.
void free_Color(Color* ptr){
free(ptr);
}
To make this more Object-Oriented we supply the Color class with an init method for mallocing it on the heap. We handle freeing Color by using the submethod DESTROY, whenever the GC decides to collect the resource it will also free Color from the heap.
class Color is export is repr('CStruct') is rw { has uint8 $.r; has uint8 $.g; has uint8 $.b; has uint8 $.a; method init(uint8 $r,uint8 $g,uint8 $b,uint8 $a --> Color) { malloc-Color($r,$g,$b,$a); } submethod DESTROY { free-Color(self); } }
The intermediate C-code of course needs to also be compiled into the raylib library.
... # using the modified libraylib.so constant LIBRAYLIB = 'modified/libraylib.so'; ... # Not using init to malloc my $white = Color.init(r => 255, g => 255, b => 255, a => 255); InitWindow(800, 450, "Raylib window in Raku!"); while (!WindowShouldClose()) { BeginDrawing(); ClearBackground($white); EndDrawing(); } CloseWindow();
Yes now it works as expected!
Phew! All this work has to be done for every function that is using values for arguments. Looking at raylib.h. That’s many functions!
Maybe that’s a reason for why nobody has made bindings for raylib, because it’s way too tedious!!!
At that point I wished that NativeCall would handle this us and almost didn’t bother working on making the bindings.
Then I had a great idea! Raku has Grammar support! What if I just parse the header file and automatically generate the bindings using the actions!
Generating bindings
So I began defining the grammar for raylib and not for C, since that’s a bigger task.
Now we will define the Actions to handle the cases for converting C to Raku bindings.
Below is simplified pointerization logic which is extracted from the raylib-raku module that I made. The Actions just contain arrays of strings holding the generated Raku bindings and the C-pointerized code.
First we need to pointerize a function only if it’s a value type. So to deduce this type must be an identifier and pointer must be nil.
Using Raku’s multiple dispatch and where clauses is a very slick way to handle different conditions.
multi method function($/ where $<type><identifier> && !$<pointer>) { my $return-type = ~$<type>; my $function-name = ~$<identifier>; my @current-identifiers;
# pointerizing the parameters which also extracts the identifiers # inside the parameters my $pointerized-params = self.pointerize-parameters($<parameters>, @current-identifiers);
# then we use the current-identifiers for creating the call to the # original c-function my $original-c-function = self.create-call-func(@current-identifiers, $<identifier>);
# creating the c-wrapper function; my $wrapper = q:s:to/WRAPPER/; $return-type $function-name_pointerized ($pointerized_parameters) { return $original-c-function; } WRAPPER
# when calling .made we convert it to raku types my $raku-func = q:s:to/SUB/; our sub $function-name $<parameters>.made() $<type>.made() is export is native(LIBRAYLIB) is symbol('$function-name_pointerized') {*} SUB
Rest of the code basically handles strings creation according to the type and or if the parameter needs to get pointerized.
# map for c to raku types has @.type-map = "int" => "int32", "float" => "num32", "double" => "num64", "short" => "int16", "char" => "Str", "bool" => "bool", "void" => "void", "va_list" => "Str";
method type($/) { if ($<identifier>) { make ~$<identifier>; } else { # translating c type to raku type make %.type-map{~$/}; } }
# Generating call to original function method create-call-func(@current-identifiers, $identifier) { my $call-func = $identifier ~ '('; for @current-identifiers.kv -> $idx, $ident { my $add-comma = $idx gt 0 ?? ', ' !! ''; # If it's a pointer then we must deref if ($ident[2]) { $call-func ~= ($add-comma ~ "*$ident[1]"); } # No deref else { $call-func ~= ($add-comma ~ "$ident[1]"); } } $call-func ~= ");"; return $call-func; }
# Generating pointerized parameters method pointerize-parameters($parameters, @current-identifiers) { my $tail = ""; # recursively calling pointerize-parameters on the rest my $rest = $parameters<parameters> ?? $parameters<parameters>.map(-> $p { self.pointerize-parameters($p, @current-identifiers) }) !! ""; if $rest { $tail = ',' ~ ' ' ~ $rest; }
# if is value type, do pointerization. if ($parameters<type><identifier> && !$parameters<pointer>) { return "$($parameters<type>)* $parameters<identifier>" ~ $tail; } else { return "$($parameters<type>) $parameters<identifier>" ~ $tail; } }
Again using multiple dispatch and where makes it easy to handle different C-types.
# Handling void multi method parameters($/ where $<pointer> && $<type> eq 'void') { make "Pointer[void] \$$<identifier>, {$<parameters>.map: *.made.join(',')}"; }
# Handling int multi method parameters($/ where $<pointer> && $<type> eq 'int') { make "int32 \$$<identifier> is rw, {$<parameters>.map: *.made.join(',')}"; }
# Handling char* multi method parameters($/ where $<pointer> && $<type> eq 'char' && !$<const>) { make "CArray[uint8] \$$<identifier>, {$<parameters>.map: *.made.join(',')}"; }
etc...
Generated code
The code above shows how we use the grammar and action to deduce the generating pointerized C-functions which was the most problematic case.
Of course we also need to handle malloc and free, callbacks, const, unsigned integers, and more. I left those out since I think the the code above demonstrates that we can use grammar and action to handle tricky cases for creating Raku bindings.
I took me about a week to finish the code generation logic. The generated Raku bindings are here and the generated C-pointerized-functions are here .
Conclusion
Overall it was a success!
By using grammar and actions we overcame the painful task of manually making the bindings.
Generally the Ethereum (Web3) keystore file is a kind of container for the private key, it has the specific structure mostly related to encryption details. Actually you will not find the private key there as a plain text, but the keystore file has everything to decrypt the private key… with some tricks surely.
When you work with Geth as a backend to access the blockchain, you have to work with accounts and therefore the «address/password» pairs or relevant private keys. Honestly, use of credential pairs is good enough for the most of the tasks, but since you want to boost the performance of your application and use it with authentication-less endpoints — you have to get some Geth-specific features directly in your app. Transaction signing, for example.
Transactions must be signed by a private key obviously, so you can somehow get private key and store it somewhere you want, but more flexible approach — manage existed keystore files.
Also you might be interested in Geth accounts management via keystore file direct access — fortunately Geth reloads keystores on the fly. Of course, I have to warn you about these hacking practices: you can corrupt or delete your account and eventually lose the access to your data on blockchain, but as a research or experiment — it’s ok 😜.
What is the Ethereum (Web3) keystore file
Overview
Roughly speaking, a keystore file is an encrypted representation of the private key. By structure — it is a JSON file with the following content:
The main member is the version. It defines the version of the keystore file and hence the encryption approach. I focus on the latest (3rd) generation of Ethereum keystores.
The other JSON members in keystore file are:
id is the random universally unique identifier generated on keystore file creation. Importing an account with Geth from a keystore file (and exporting it if needed) will by default lead to keystore with different id. If this is not desired, you have to use 3rd-party tools allow to reuse or explicitly define id;
address relies on private key and might be fetched with Bitcoin::Core::Secp256k1module;
crypto has all enctyption details to get private key from the keystore file, encrypted private key is stored in ciphertext.
Actual scope (real world tasks)
I mentioned a few real world tasks we can solve via direct keystore management, but let me show a hot one at least. I maintain a few full nodes for Sepolia test network. On node deployment there are two options for address management: use automatically generated address or import an existing one.
Both cases have their own specifics. The common thing: both need a positive account balance (some available funds) to start interacting with test network.
Tools (aka hacks) to get private key from keystore file
The first case (automatically generated address) is the trivial one if you do not want to manage (track) your balance with the 3rd party wallet: you just need to get some funds from the faucet (I use the faucet by Alchemy) and track funds transfer with Etherscan.
The magic happens when you want to add your automatically generated address to 3rd-party wallets. You have to fetch somehow private key from the keystore file initially generated in Geth and import it, for example, into MetaMask. I wrote a simple Node.js script as a hacker tool for that task. It’s a bit excessive and obviously not flexible approach — you should have Node.js installed, also script needs Keythereum package as a primary dependency. So eventually I had to install all that stuff to my Sepolia node server, added a few symlinks (script looks for keystore file in keystore folder explicitly) and run script from privileged user:
Another hacking approach — use Python. It’s a bit more scalable than Node.js: Python is pre-installed in the most of the popular Linux distributions and the script has just a single external dependency:
Since you got the raw private key, you can easily import it into Metamask and manage/track your account (balance at least) via a friendly UI.
Importing new accounts (private keys) into Geth client
The second case (import existing private key) into Geth client has two options under the hood as well. We can use Node.js again for raw private key to keystore file conversion (dependence to ethereumjs-wallet package). Another approach: built-in Clef account manager — new accounts might be added via the command line:
Input argument --keystore points to folder, where Geth stores its keystore files; importraw option triggers raw private key import; /pk is just the path to the sample text file with raw private key. Import via clef is used in Pheix CI/CD at GitLab.
Implementation in Raku
I started with the quick research — how it’s implemented in Geth client written in Go.
Initial state (it looks promising)
Not a lot of work and it looks like Raku has everything in its ecosystem for straight-forward implementation:
get keccak hash from derived key and keystore’s ciphertext with Node::Ethereum::Keccak256::Nativemodule;
decrypt ciphertext by derived key and keystore’s input vector with AES128: with OpenSSL or Gcrypt modules [ref1, ref2];
verify decrypted V3 key (raw private key) and get Ethereum address with Bitcoin::Core::Secp256k1module;
generate UUID v4 with Crypt::Randommodule — in case we want to create keystore file from raw private key.
Problems (are on the most important steps)
The main issues are at decryption, unfortunately the most important step. First issue is at Crypt::LibScrypt: custom KDF parameters could not be used because of module limitations. By default only scrypt-hash is implemented, it uses KDF constants hardcoded in Crypt::LibScrypt. Obviously those constants impact hash generation, so to get it working with Geth keystores we have to add binding to libscrypt_scrypt function from Native libscrypt library.
Next issue is at OpenSSL and Gcrypt, both do not include AES128-CTR implementation. I will consider details below. Finally I found the problem at Bitcoin::Core::Secp256k1 — compression was not configurable there, but we need to supportSECP256K1_EC_UNCOMPRESSED alongside SECP256K1_EC_COMPRESSED.
Let’s raise a few Pull Requests
Crypt::LibScrypt
On initial step I added a few principal updates to Crypt::LibScrypt module, as it was mentioned above we need a binding to internal hashing function libscrypt_scrypt. Finally the next bindings (and exported wrappers) were added:
libscrypt_salt_gen — method for salt generation, «must-have» feature for keystore file generation;
libscrypt_scrypt — main libscrypt‘s hashing method, totally configurable via user-defined KDF (Key Derivation Function) parameters;
libscrypt_mcf — method to transpose raw hash buffer to modular crypt format (MCF), stringified hash in MCF might be verified by scrypt-verify;
Updates to OpenSSL are quite straight-forward: I added AES128-CTR binding EVP_aes_128_ctr and implemented new encrypt and decrypt multi methods with mandatory :$aes128ctr argument (to call EVP_aes_128_ctr from).
Initially I tried Gcrypt as the module with AES128-CTR in place. Gcrypt is the great set of bindings for GNU libgcrypt. My expectation was: Gcrypt has a bindings for AES family with all available modes. Quote from module README:
It provides functions for all cryptograhic building blocks: symmetric cipher algorithms (AES, Arcfour, Blowfish, Camellia, CAST5, ChaCha20 DES, GOST28147, Salsa20, SEED, Serpent, Twofish) and modes (ECB, CFB, CBC, OFB, CTR, CCM, GCM, OCB, POLY1305, AESWRAP).
The first finding was: the mode switch is broken in Gcrypt since it was released. Just a typo in Gcrypt/Constants.rakumod: enum with modes is defined as Gcrypt::CipherMode, but in generic Gcrypt::Cipher class, mode is set up from Gcrypt::CipherModes 🤷. So looks like none tested it before, a bit of a dangerous beginning.
I did a quick fix and started tests, but decryption bellow gives wrong $secret buffer:
my buf8 $secret = Crypt::LibGcrypt::Cipher.new(:algorithm(GCRY_CIPHER_AES), :key($derivedkey.subbuf(0, 16)), :mode('CTR'), :iv($iv)).decrypt($ciphertext, :bin(True));
To debug that, I wrote simple C-application, where I compared libgcrypt and libssl decryption — it worked 100% similar to Raku implementation, libssl gave correct secret and libgcrypt gave incorrect one (but the same as I got from Raku script). That finding showed that I missed something in libgcrypt implementation, so I started investigation (googling actually). Next finding was: in libgcrypt embedded tests for AES in CTR mode counter vector ctr was used instead of initial vector iv. So I modified my C-application and 🎉 I finally got correct secret via libgcrypt.
So what updates should be added to Raku Gcrypt module eventually? Actually a few ones:
constructor upgrade — set up counter vector if needed.
While working on Gcrypt, my pull requests for Crypt::LibScrypt and OpenSSL were still pending (now are pending as well), so I decided to forkGcrypt to Crypt::LibGcrypt, because another one stuck PR will frustrate me finally.
Upgrade secp256k1 Raku binding
The quickest step! I’m the maintainer of Bitcoin::Core::Secp256k1, so I just pushed the updates to the source base 😍. The idea behind: add an option to get uncompressed key with serialize_public_key_to_compressed method. Looks a bit controversial (from naming perspective), but it’s just the feature for key serialization without compression.
Manage keystore files with 🤫
So looks like we got all «puzzles» working and finally after an evening spent on Node::Ethereum::KeyStore::V3 module coding, we are ready for 👇
Quick start
This module has everything under the hood to manage your keystore files and raw public keys. You can easily decrypt existed keystore:
useNode::Ethereum::KeyStore::V3;
my $password = 'node1';
my $decrypted = Node::Ethereum::KeyStore::V3.new(:keystorepath('./data/92745E7e310d18e23384511b50FAd184cB7CF826.keystore')).decrypt_key(:$password);
$decrypted<buf8>.say;
$decrypted<str>.say;
# Buf[uint8]:0x<63 27 35 B6 6A D8 75 10 8D EE F0 39 BE 85 5A AE 7F 70 26 53 FC C2 B2 EF B1 E5 66 6C 13 06 F2 FD>
# 632735b66ad875108deef039be855aae7f702653fcc2b2efb1e5666c1306f2fd
New keystore file could be generated from raw public key via a few calls:
useNode::Ethereum::KeyStore::V3;
my $secret = '632735b66ad875108deef039be855aae7f702653fcc2b2efb1e5666c1306f2fd';
my $password = 'node1';
my $ksobject = Node::Ethereum::KeyStore::V3.new(:keystorepath('./sample-keystore.json'));
my $keycrypto = $ksobject.keystore((:$password, :$secret);
$ksobject.save(:$keystore);
Command line utility
Node::Ethereum::KeyStore::V3 module distribution has ethkeystorev3-cli utility onboard. After the module installation it’s available via command line:
ethkeystorev3-cli --keystorepath=$HOME/sample-keystore-gen.json --password=111 --privatekey=632735b66ad875108deef039be855aae7f702653fcc2b2efb1e5666c1306f2fd
# keystore /home/user/sample-keystore-gen.json is successfully saved
Dependencies
Since my pull request to Crypt::LibScrypt is still pending, I decided to maintain my fork of Crypt::LibScrypt with personal auth. For now Node::Ethereum::KeyStore::V3 module depends on Crypt::LibScrypt:ver<0.0.7+>:auth<zef:knarkhov> (available in fez ecosystem).
Afterword
Ethelia service
Ethelia is a secure, authoritative and reliable Ethereum blockchain storage provider for different kinds of lightweight data.
Blockchain technology gives decentralization, security and resistance against data corruption or falsifying out of the box, and Ethelia delivers to end user the ability to store tamper-proof and sensitive data there.
Сonsider a network with functionally different nodes: IoT, programmable logic controllers, smart home systems, micro-services, standalone multi-threading or mobile applications. Every node runs own duty cycle with state changes and events emission.
In general non-critical state changes or usual events should not be logged, but once the node detects some anomaly or exceptional behavior, importance of logging increases exponentially. Such events might be stored on blockchain by Ethelia and the fact of storing will guarantee the data integrity and consistency.
I like sign up/in approach from Midjourney — do it via Discord. For Ethelia it’s done the same way, but Telegram is used instead. Registration model is described here.
Basic idea is — customer has to pass the registration interview with the bot and if interview is passed, bot registers new customer on Ethelia’s node. Telegram bot is written in Raku, the module for interviewing is at early beta and still under developing, currently I’m trying to combine static questions with the dynamic GPT4 ones (generated on the fly according the interview flow).
Node::Ethereum::KeyStore::V3 module is used there at the final step for actual account creation (keystore file generation to Geth keystore folder).
Since you are registered, you can post your lightweight data to a different Ethereum networks (our private PoA, Sepolia and mainnet) via Ethelia’s endpoint. The only limitation for this moment — API access token is available only during the auth session in web control panel.
Demo pitch
You are welcome to try it out! Thank you for reading. Happy Xmas 🎄🎅☃️🎁⛄🦌🎄
Santa only had a few days left to make sure everything was ready to go, but with all the stress of the season, he needed a break to recharge. He grabbed some cocoa and hid away in a nook to relax his body and distract his mind, tuning into one of his favorite Youtubers, Matt Parker.
Parker finds interesting mathematical problems that he attempts to untangle and present to the audience in a tractable way, and as he analyzes the problems, he often has to write “some horrible python code.” Santa, of course, will use his favorite language instead: the Raku Programming Language!
Maybe if Santa’s brain was working on one of these puzzles, it’d help him stop thinking about all the other work he was supposed to be doing.
The Problem
So, what to work on in this precious downtime? Santa wants to work on something a little practical, so he doesn’t feel too guilty about taking some time off – let’s figure out how much we’re going to have to expand the shop in the next few years!
A quick google search gets us to some UN data – surely that’s a good start. Santa creates a sandbox folder, and manually downloads and unzips it. For small projects like this, Santa likes to attack the problem in chunks rather than map the whole project at once. First, he makes sure he can read the data at all:
my $data-file = "data.csv".IO;
my $data = $data-file.lines;
my $headers = $data[0];
dd $headers.split(',');
Alright, the CSV starts with a row of headers, so we read it in, grab the first row, and do a data dump of that row. We ignore all the possible complexity of CSV, we’ll deal with that if we need to.
Filtering
We are only interested in getting estimates on kids, so let’s filter through the data. Santa can ignore anything where the starting age is 15 or higher, at least for this project.
We peeked at the headers, we know which columns the data we need is in, so we’ll hardcode it for now. Santa gets the age first since that’s our filter, and only prints out the data if the row is good!
my $data-file = "data.csv".IO;
my $data = $data-file.lines;
my $headers = $data[0];
my $count = 0;
for @($data) -> $line {
$count++;
next if $count == 1; # skip the headers
my @row = $line.split(',');
my $age = @row[15];
next if $age >= 15;
my $year = @row[12];
my $pop = @row[19];
dd $year, $age, $pop;
}
Cannot convert string to number: imaginary part of complex number must be followed by 'i' or '\i' in '0-4⏏' (indicated by ⏏)
What? There’s imaginary numbers in here? Santa adds some debug output to print the line before processing it, and sees:
15,934,g,,,,5,Development group,902,"Less developed regions, excluding least developed countries",2,Medium,1950,1950,0-4,0,5,113433.383,107834.33,221267.713
Not so simple
Ah, biscuits. Looks like our horribly simple start caught up with us, we do have to care about more complicated CSV data after all.
Rather than spending any more time on improving our CSV “parser” (currently only split), let’s get out the big hammer:
$ zef install Text::CSV
Santa quickly checks out the docs and updates his code:
use Text::CSV;
my $csv = Text::CSV.new;
my $io = open "data.csv", :r;
my @headers = $csv.header($io).column-names;
while (my @row = $csv.getline($io)) {
my $age = @row[15];
next if $age >= 15;
my $year = @row[12];
my $pop = @row[19];
}
He’s still using column numbers, but now that he’s switched over to Text::CSV, at least we can process the whole file.
Speed?
Problem with this version is it’s a little slow. To be fair, it is over 900,000 lines with 20 colums of CSV data. Santa is willing to cheat a little here: he’s just looking for some estimates, after all.
Maybe the Text::CSV has to do enough extra processing per line that it adds up, or maybe Raku’s default line iteration is more efficient than manually calling getline a bunch of times.
We’re impatient, so we’ll try addressing both at once: .lines to walk through the file, and then only using the CSV parser if it we know we got the wrong column count back. We may miss a line or two but this is good enough for our rough estimate. Santa adds up all the data for each year and prints out some samples.
use Text::CSV;
my $csv = Text::CSV.new;
my @lines = "data.csv".IO.lines;
my $headers = @lines.shift.split(',');
my $cols = $headers.elems;
my %estimate;
for @lines {
my @row = $_.split(','); # simple CSV
if @row.elems != $cols {
@row = $csv.getline($_); # real CSV
}
my $year = @row[12];
next if $year <= 2023;
my $age = @row[15];
next if $age >= 15;
my $pop = @row[19];
%estimate{$year}+=$pop;
}
say %estimate{2024};
say %estimate{2050};
19110349.077
19204147.428
Ah, much better. Now we can see we can expect a few more deliveries in 2050! Let’s improve the formatting a little and filter to output each decade and see how much we need to expand!
Pretty print
use Text::CSV;
my $csv = Text::CSV.new;
my @lines = "data.csv".IO.lines;
my $headers = @lines.shift.split(',');
my $cols = $headers.elems;
my %estimate;
for @lines {
my @row = $_.split(','); # simple CSV
if @row.elems != $cols {
@row = $csv.getline($_); # real CSV
}
my $year = @row[12];
next if $year <= 2023;
next unless $year %% 10;
my $age = @row[15];
next if $age >= 15;
my $pop = @row[19];
%estimate{$year} += $pop;
}
for %estimate.keys.sort -> $year {
say "$year: %estimate{$year}.fmt('%i')";
}
Oh! It’s a good thing we checked, looks like 2050 will be the peak, and then the projections go back down! Maybe we can avoid expanding the shop for a while!
Speed?
Even though we have our answer now, this still takes a few seconds to get through all the data, so one last round of changes! We can:
add some concurrency to race through the processing, we don’t care what order we process the data
use some Seq methods to deal with the first line of headers more cleanly
specify a type for the data we’re extracting
use a Mix instead of a Hash to handle the addition
change the logic a bit to grab all the data and only print what we want – makes it easier if we want to change our reporting later
use Text::CSV;
my $io = "data.csv".IO;
my $headers = $io.lines.head.split(',');
my $cols = $headers.elems;
my %estimate is Mix = $io.lines.skip.race(batch => 1024).map: {
my @row = .split(','); # simple CSV
if @row.elems != $cols {
@row = Text::CSV.new.getline($_); # real CSV
}
my int $year = @row[12].Int;
my int $age = @row[15].Int;
my int $pop = @row[19].Int;
$year => $pop if $year > 2023 && $age < 15;
}
for %estimate.keys.grep(* %% 10).sort -> $year {
say "$year: %estimate{$year}.fmt('%i')";
}
This does a little more work in about 40% of the time of the previous version since Santa made the work happen on multiple cores!
Other improvements?
Having gotten the quick answer he was looking for, Santa throws together a TODO file for next year’s estimator script:
Pull the file from the UN and unzip it in code if we haven’t already – and see if there’s an updated file name each year
Switch to a full Text::CSV version and figure out the best API to use for parallel processing. If we ever get embedded newlines in this CSV file, our cheat won’t work!
Use column headers instead of numbers to future proof against changes in the data file!
Wrap this into a MAIN sub so we can pass in the config we have hardcoded in the script
Wrapup
Now that Santa’s exercised his brain on this code, he’s ready to get back to the real work for the season!
Santa’s recommendation to you is to write some “horrible” Raku code, just like Matt Parker would. Of course, it’s not actually horrible, more “quick and dirty”. Remember, it’s OK to write something that just gets the job done, and not start with something polished.
It’s OK if you don’t necessarily understand all the nuances of the language (it’s big!), you just need enough to get the job done. You can always go back later and polish or iteratively improve it.
Raku even has this attitude baked in with gradual typing – you can add type strictures as you need. Much like writing a blog post, it’s easier to start with something and revise it than it is to face that blank file.
Remember, when optimizing your project, sometimes it’s OK to optimize for developer time!
As the Christmas approaches, the time for unwrapping presents is near. Let’s review a present Rakudo release 2023.12 is going to have for you.
Long ago I once got a question related to type capturing. Don’t remember the question itself, but remember the answer Jonathan Worthington gave to it (rephrasing up to what my memory managed to keep): “You can use it, but it’s incomplete.” This is what they call “to be optimistic”. Anyway, a lot has changed in this area since then.
Why
WWW::GCloud family of modules is inevitably heavily based on two whales: REST and JSON. The former is in strong hands of the Cro framework. For JSON I ended up with creating my own JSON::Class variant, with lazy deserializations support and many other features.
Laziness in JSON::Class is also represented by special collection data types: sequences and dictionaries. Both are not de-serializing their items until they’re read from. This is a kind of virtue that may play big role when dealing with, say, OCR-produced data structures where every single symbol is accompanied with at least its size and location information.
In a particular case of Google Vision it’d be the Symbol with this representation in my Raku implementation. The problem is that the symbols are barely needed all at once. Where JSON::Fast manages very well with these, producing a full tree of objects is costly. But while converting WWW::GCloud for using JSON::Class:auth<zef:vrurg> I stumbled upon a rather unpleasant issue.
Google APIs use a common pattern when it comes to transferring long lists of items, which involves paginating. A structure, that supports it, may look like ListOrgPoliciesResponse, or like an operations list, or many other of their kind. Since the nextPageToken field is handled consistently by all similar APIs, it makes sense to provide standardized support for them. Starting with a role that would unify representation of these responses. Something like:
role Paginatable[::ITEM-TYPE] {
has Str $.nextPageToken;
has ITEM-TYPE:D @.items;
}
See the real thing for more details, they are not relevant here. What is relevant is that for better support of JSON::Class laziness I’d like it to look more like this:
role Paginatable[::LAZY-POSITIONAL] {
has Str $.nextPageToken;
has @.items is LAZY-POZITIONAL;
}
class OpList is json(:sequence(Operation:D)) {}
class Operations is json does Paginatable[OpList] {}
Or, perhaps, like this:
role Paginatable[::RECORD-TYPE] {
my class PageItems is json(:sequence(RECORD-TYPE)) {}
has Str $.nextPageToken;
has @.items is PageItems;
}
class Operations is json does Paginatable[Operation:D] {}
Neither was possible, though due to different causes. The first one was failing because the LAZY-POSITIONAL type capture was never getting instantiated, resulting in an exception during attempt to create an @.items object.
The second case is even worse in some respect because under the hood JSON::Class creates descriptors for serializable entities like attributes or collection items. Part of descriptor structure is the type of the entity it represents. There was simply no way for the PageItems sequence to know how to instantiate the RECORD-TYPE generic.
Moreover, even if we know how to instantiate the generic, @.items would has to be rebound to a different type object, which has descriptors pointing at nominal (non-generic) types. As you can see, even to explain the situation takes time and words. And that is not to mention that the explanation is somewhat incomplete yet as there are some hidden rocks in these waters.
How
Rewinding forward all the doubts, like not wanting to invest into the legacy compiler, and all the development fun (and not so fun too), let’s skip to the final state of affairs. Before anything else is being said, please, keep in mind that all this is still experimental. Not like something, to be covered with use experimental pragma, but like something that might eventually be torn up from Rakudo codebase for good. OK, let’s get down to what’s been introduced or changed.
Instantiation Protocol
The way generics get instantiated is starting to receive a shape as a standard. Parts of the protocol will be explained below, where they relate to.
Generic Classes
This change is conceptual: a class can now be generic; even if a class is not generic, an instance of it can be.
If the latter doesn’t make sense at first, consider, for example, a Scalar, where an instance of it can be .is_generic().
What does it mean for a class to be generic? From the class developer point of view – just about anything. From the point of view of Raku metaobject protocol it means that GenericClass.^archetypes.generic is true. How would class’ HOW know it? By querying the is-generic method of GenericClass:
role R[::T] {
my class GenericClass {
method is-generic { True }
}
say "Class is generic? ", ?GenericClass.^archetypes.generic;
}
constant RI = R[Int];
So far, so good, but what does role do up there? A generic is having very little sense outside of a generic context, I.e. in a lexical scope that doesn’t have access to any type capture. The body of R[::T] role does create such context. Without it we’d get a compile time warning:
Potential difficulties:
Generic class 'GenericClass' declared outside of generic scope
An attempt to use the class would, for now, very likely end up with a cryptic error ‘Died with X::Method::NotFound’. It’s an LTA that is to be fixed, but for currently it’s a ‘Doctor, it hurts when I do this’ kind of situation. Apparently, having distinct candidates of is-generic for definite and undefined cases lets one to report different states for classes and their instances:
multi method is-generic(::?CLASS:U:) { False }
multi method is-generic(::?CLASS:D:) { self.it-depends }
Note 1 Be careful with class composition times. An un-composed class doesn’t have access to its methods. At this stage MOP considers all classes as non-generics. Normally it has no side effects, but trait creators may be confused at times.
Note 2 Querying archetypes of a class instance is likely to be much slower than querying the class itself. This is because instances are run-time things, whereas the class itself is mostly about compile-time. An instance can change its status as a result of successful instantiation of generics, for example.
Instantiation Method
OK, a class can now be a generic. How do we turn it into a non-generic? This would be a sole responsibility of INSTANTIATE-GENERIC method of the class. What the method does to achieve the goal and how it does it – we don’t care. Most typically one would need two candidates of the method: one for the class, one for an instance:
multi method INSTANTIATE-GENERIC(::?CLASS:U: $typeenv) {...}
multi method INSTANTIATE-GENERIC(::?CLASS:D: $typeenv) {...}
For example, instantiation of a collection object might look like:
multi method INSTANTIATE-GENERIC(::?CLASS:D: $typeenv) {
::?CLASS.INSTANTIATE-GENERIC($typeenv).STORE(self)
}
Instantiation of a class… Let me put it this way: we don’t have a public API for this yet.
All this is currently a newly plowed field, a tabula rasa. It took me a few hours of trial-and-error before there was working code for JSON::Class that is capable of creating a copy of an existing generic class, which is actually subclassing the original generic but deals with instantiated descriptors.
Eventually, I extracted the results of the experiment into a Rakudo test, which is free of JSON::Class-specifics and serves as a reference code for this task. It’d be reasonable to have that test in the Roast, but it relies on Rakudo implementation of Raku MOP, where many things and protocols are not standardized yet.
What’s next? I’d look at where it all goes, what uses people may find for this new area. Everything may take an unexpected turn with RakuAST and macros in place. Either way, some discussion must take place first.
Type Environment
It used to be an internal thing of the MOP. But with introduction of the public instantiation protocol we need something public to pass around.
What is “type environment” in terms of Raku MOP? Simply, it’s a mapping of names of generic types into something (likely) nominal. For example, for any of R[::T] role declaration when it gets consumed by a class as, say, R[Str:D] the type environment created by role specializer would have a key "T" mapping into Str:D type object.
There is a problem with the internal type environment objects: they are meaningless for Raku code without use nqp pragma and without using corresponding NQP ops. Most common case is when the environment is a lexical context of role’s body closure.
A new TypeEnv class serves as an Associative container for the low-level type environments. As an Associative, it provides the standard (though rather basic) interface, identical to that of the Map class:
say $typeenv<T>.^name; # outputs 'Str:D' in the context of the R[Str:D] example above
The class is now supported by the MOP making it possible to even create own type environments:
my %typeenv is TypeEnv = :T1(Foo), :T2(Bar);
my \instantiation = GenericClass.^instantiate_generic(%typeenv);
Instantiation In Expressions
Let’s consider a simplified example:
role R[::T] {
my class C {...} # Generic
method give-it-to-me {
C.new
}
}
If no special care taken by the compiler, the method will try to create an instance of a generic class C. Instead, Rakudo is now pre-instantiating the class as soon as possible. In today’s situation, this happens when the role gets specialized as this is the earliest moment when type environment takes its final shape. The pre-instantiation is then referenced at run-time.
Are We There Yet?
Apparently, far from it. I’m not even sure if there is an end to this road.
When first changes to Rakudo code started showing promising results I created a draft PR for reviews and optimistically named it with something about “complete instantiation”. Very soon the name was changed to “moving close to complete instantiation”.
The most prominent missing part in this area for now is instantiation of generic parameters in signatures, and signatures themselves. Having all we already have, this part should be much easier to implement. Or may be not, considering the current signature binding implementation.
And then there is another case, which I spotted earlier, but forgot to leave a note to myself and can’t now remember what it was about.
Speaking of the future plans, I wouldn’t say better than one person formulated it once in the past:
I Don’t Know The Future. I Didn’t Come Here To Tell You How This Is Going To End. I Came Here To Tell You How It’s Going To Begin.
So, let’s go straight to the…
Conclusion
Quoting another famous person:
Ho-Ho-Ho!
Hope you like this present! I guess it might not be up to what you expected from it. But we can work together to move things forward. In the meantime I would have to unwind my stack of tasks back to the project, where all that started a while ago… Quite a while…
This post is a continuation of Day 5 – The Elves go back to Grammar School, you may recall that our elfin friends had worked out how to parse all the addresses from the many, many children who had emailed in their wish lists.
edited by L. H. Jones, Public domain, via Wikimedia Commons
Now, as they sing along to “Christmas is Coming”, they realise that their AddressUSA::Grammar parser only covers mainland addresses in the USA. But what about Europe? What about the rest of the world? Oh my…
Christmas is saved, now Santa has the structured address info to load into his SatNav … we leave the other geos as an exercise for the elves.
Contact
Here’s the top level Contact.rakumod code:
use Contact::Address;
roleContact {
hasStr$.textisrequired;
hasStr$.countryisrequired
where*eq <USA UK>.any;
hasStr$.name;
has Address $.address;
submethodTWEAK {
my@lines=$!text.lines;
$!name=@lines.shift;
$!address= AddressFactory[$!country].new.parse:
@lines.join("\n");
}
methodStr { ... }
}
Key takeaways here are:
we use the built in TWEAK method to adjust our attributes immediately after the object is constructed … in this case parcelling out name and address construction
we choose to use the relaxed style of raku OO with public attributes so that (eg.) you can go say $contact.address.street if that’s what you want
You might recognise these classes from the previous post … now we have refactored the classes into a single common Address module and this gives the coding flexibility to keep all classes & methods separate, or to evolve them to move common code into composed roles.
Highlights are:
Girl, that’s really clear! It shows how raku objects can be used to contain real-world data, keeping the “labels” (house, street, city, etc) as has attributes.
It shows the application of an API definition role that stubs required methods with the { … } syntax (these methods are then mandatory for any class that does the role
It shows the application of a parameterized role – in this case the $country parameter can be specified via a Factory class pattern (with suitable default)
This allows for USA and UK variants (and, in future, others) to be checked with a where clause and then to be instanced as consumer of the Contact::Address role
Each branch of the factory will create a country-specific instance such as class Contact::Address::UK, class Contact::Address::USA, and more can be added
Each of these child objects has a .parse method as required by the API and that, in turn, loads the implementation with (eg.) use Contact::Address::UK::Parse, which loads the classContact::Address::UK::Parse child to perform the Grammar and Actions specific to that country
This code is intended to make it’s way into a new raku Contact module … that’s work in progress for now, but you are welcome to view / raise issues / make PRs if you would like to contribute…
This gives us a clear class & role hierarchy with extensibility such as more attributes of Contact (email, phone anyone?) and international coverage (FR, GE and beyond).
It keeps the Grammar and Action classes of the country-specific parsers together since they have an intimate context. And, since they are good citizens in the Raku OO model, they sit naturally in the tree.
Fröhliche Weihnachten!
…said Père Noël. Dammit, said the naughty elf, we’ve hardly started on the anglophone addresses and now we need to cope with all these accents and hieroglyphs (not to mention pictographic addresses).
Stay cool said Rudi, for he knew a thing or two about Raku’s super power regexes and grammars with Unicode support built right in.
A few days after having done The Magic Of Q presentation, Lizzybel was walking through the corridors of North Pole Grand Central and ran into some of elves that had attended that presentation. When will you do the final part of your presentation? one of them asked. We could do it now if you want, Lizzybel answered, while walking to the presentation room and opening the door.
The room was empty. Most of the elves entered, one of them asked: Will there be a video? Lizzybel thought for a moment, and said: No, but there will be a blog post about it! That elf hesitated a bit, then said: Ok, I will read it when it gets online, and went the away. The others sat down and Lizzybel continued:
There are two other adverbs that haven’t been covered yet, and there’s a new one if you’re feeling brave. Let’s start with the two already existing ones:
short long what does it do ===== ==== =============== :x :exec Execute as command and return results :to :heredoc Parse text until terminator
Shelling out
The :x (or :exec) adverb indicates that the resulting string should be executed as an external program using shell. For example:
say q:x/echo Hello World/; # Hello World
And since you can skip the : if there’s only one adverb, you could also have written this as the maybe more familiar:
say qx/echo Hello World/; # Hello World
Of course, you can also have variables interpolated by using qq:
my $who = 'World';
say qqx/echo Hello $who/; # Hello World
But one should note that this is a huge security issue if you are not 100% sure about the value of the variable(s) that are being interpolated. For example:
my $who = 'World; shutdown -h now';
say qqx/echo Hello $who/; # Hello World
would produce the same output as above, except it could possibly also shutdown your computer immediately (if you were running this with sufficient rights). Now imagine it’d be doing something more permanently destructive! Ouch!
So generally, you should probably be using run (which does not use a shell, so has fewer risks) or go all the way with full control with a Proc object, or possibly even better, with a Proc::Async object.
Until the end
The :to (or :heredoc) adverb does something very special!
Different from all other adverbs, it interprets the text between // as the end marker, and takes all text until that marker is found at the start of a line. So it basically stops normal parsing of your code until that marker is found. And this is usually referred to as a “heredoc“.
Of course, if so needed you can also interpolate variables (by using qq rather than q), but these variables would be interpolated inside the heredoc, not in the marker. For instance:
my $who = 'World';
say qq:to/GREETING/;
Hello $who
GREETING
# Hello World
It is customary, but not needed in any way, to use a term for the marker that sort of describes what the text is about.
As you may have noticed, the resulting string of a heredoc always ends with a newline (““). There is no adverb to indicate you don’t want that. But you can call the .chomp method on it like so:
my $who = 'World';
say qq:to/GREETING/.chomp;
Hello $who
GREETING
# Hello World
You can indent the end marker for better readability, for instance if you’re using it inside an if structure. That won’t affect the resulting string:
my $who = 'World';
if 42 {
say qq:to/GREETING/;
Hello $who
GREETING
}
# Hello World
The text inside will have the same amount of whitespace removed from the beginning of each line, as there is on the start of the line with the end marker.
What many people don’t know, is that you can have multiple heredocs starting on the same line. Any subsequent heredoc will start immediately after the previous one. You can for instance use this in a ternary like so:
my $who = 'Bob';
say $mood eq 'good' ?? qq:to/GOOD/ !! qq:to/BAD/;
Hi $who!
GOOD
Sorry $who, the VHS is still broken.
BAD
Depending on the $mood, this will either say “Hi Bob!” or “Sorry Bob, the VHS is still broken.“.
Formatting
Since the 2023.06 release of the Rakudo Compiler, the 6.e.PREVIEW language version contains the Format class. This RakuAST powered class takes a printf format specification, and creates an immutable object that provides a Callable that takes the expected number of arguments. For example:
printf "%04d - %s\n", 42, 'The answer'; # 0042 - The answer
You can now save the logic of the format in a Format object, and call that with arguments. Like so:
use v6.e.PREVIEW;
my $format = Format.new("%04d - %s\n");
print $format(42, 'Answer'); # 0042 - Answer
Now, why would this be important, you might ask? Well, it isn’t terribly important if you use a format only once. But in many situations, a specific format is called many times in a loop. For instance when processing a log file:
Because of the way printf formats are implemented in Raku, this is very slow.
This is because each time printf is called with a format string, the whole format is interpreted again (and again) using a grammar. During this parsing of the format, the final result string is created from the given arguments. This is much slower than calling a block with arguments, as that can be optimized by the runtime. In trial runs speed improvements of 100x faster, have been observed.
The new Format class can do this once, at compile time even! And by storing it in a constant with a & sigil we can use that format as if it is a named subroutine!
use v6.e.PREVIEW;
my constant &logged = Format.new("%04d - %s\n");
for lines {
m/^ (\d+) ':' FooBarBaz (\w+) /;
print logged($0, $1);
}
So what does this have to do with quoting adverbs, you might ask? Well, when the 6.e language level is released, this will also introduce:
short long what does it do ===== ==== =============== :o :format Create Format object for the given string
If the given string is a constant string, then the above example can be written as (without needing to define a constant):
use v6.e.PREVIEW;
for lines {
m/^ (\d+) ':' FooBarBaz (\w+) /;
print qqo/%04d - %s\n/($0, $1);
}
And by this time the remaining elves were a bit flabbergasted.
Well, that’s about it. That’s it what I wanted to tell about the magic of Q! said Lizzybel. The elves had a lot of questions, but those questions did not make it to this blog post. Too bad.
Maybe the readers of the blog post will ask the same questions in the comments, thought Lizzybel after writing up all of these events.
In parts 1 and 2 of these blog posts, I roughed out a simple ping chart program and then began to refactor and add features to improve the overall experience.
It’s functional, but there’s a lot to improve upon — it doesn’t use the screen real estate particularly well, there are some common network problems it can’t visualize, and frankly it just doesn’t look all that cool.
So let’s fix all that!
Another Dimension
A simple way to improve the chart’s overall information density is to encode more information into each rendered grid cell. Instead of always using the same glyph for every data point — providing no information other than its location — the shape, color, attributes, or pattern can be adjusted to show more useful information in each rendered cell of the screen.
The version of ping-chart in parts 1 and 2 only shows a relatively short history, since each grid column only represents at most one measurement (and possibly zero, if the “pong” reply packet was never received). Simply placing several measurements before moving on to the next column would improve that, but then overlaps become ambiguous. If ten measurements rendered as just three circles on the screen, how often did the measured ping times land on each of those circles? Were the measurements spread roughly evenly? Did most of them land on the highest or lowest circle?
The chart already looks a bit like a trail of bubbles or pebbles, so why not change the size of each pebble to indicate how often the measurement landed on a particular grid cell? There are many mappings usable for this, depending on which glyphs are available in the terminal font; here are a few obvious options:
ASCII: . : o O 0
Latin-1: · ° º o ö O Ö 0
WGL4: · ◦ ∙ ●
Braille: ⠄ ⠆ ⠦ ⠶ ⠷ ⠿ ⡿ ⣿
I’ll use ASCII for now, since every terminal font supports it. Making this work requires only a few changes to the update-chart sub. Instead of the original X coordinate calculation, I instead use:
state @counts;
constant @glyphs = « ' ' . : o O 0 »;
my $saturation = @glyphs.end;
my $x = ($id div $saturation) % $width + $x-offset;
This creates the @counts state variable to track how many measurements have landed on a particular Y coordinate and defines the glyphs to be used (including a leading space in the zero slot). The saturation point — the most measurements that can be recorded in a single column before moving forward — is calculated as the last glyph index (@glyphs.end), and finally the calculated X coordinate is (integer) divided by that saturation level to slow the horizontal movement appropriately.
Then I simply need to clear the @counts every time the chart moves to a new column:
# Clear counts
@counts = ();
And update the Y coordinate handling and glyph printing:
# Calculate Y coord (from scaled ping time)
# and cell content
my $y =
0 max $bottom - floor($time / $chart.ms-per-line);
my $c = @glyphs[++@counts[$y]];
# Draw glyph at (X, Y)
$grid.print-cell($x, $y, $c);
The tiny change to the calculation of $y ensures that it can never be negative, and thus is always a valid array index. It’s then used to update the @counts, select the appropriate glyph based on the latest count, and print the chosen glyph in the right spot.
It looks like this now; instead of identical pebbles, the trail looks much more like various-sized pebbles on a scattering of sand:
It’s much easier to see where the most common measurements lie, and where the outliers are. As a bonus the change to force the Y coordinate onto the chart grid makes it now possible to see how often large outliers appear; they are no longer simply ignored when printing, but rather appear as a smattering of dots at the very top.
As there were five non-blank glyphs chosen, this version now shows five times as much history at once — a bit more than six minutes of history in a default width-80 terminal window.
A wider selection of @glyphs could further improve on that, but there are rapidly diminishing returns — too many different glyphs and the glanceability is lost because it becomes hard to tell the difference between them just by visual size or “density”. This is why I didn’t just choose the digits 1..9; there is very little density distinction between them all, and the overall effect is more confusing than enlightening.
Heating Up
Instead of changing the particular glyph drawn, we could also change its color and brightness; a bright line through a dark-background chart (or a dark line through a light-background chart) would then show where most ping times were clustered.
The original 16 color ANSI palette supported virtually everywhere is completely awful for this purpose, especially since every operating system and terminal program uses a different mapping for these colors. Thankfully there’s a better replacement: most modern terminal emulators support the xterm-256color extended colors and map them all equivalently.
These added colors are mapped in two blocks: a 6x6x6 RGB color cube and a 24-level gray scale. By choosing appropriate points on the color cube, I can create a decent heat map gradient:
# Calculate and convert the colormap once
constant @heatmap-colors =
# Black to brick red
(0,0,0), (1,0,0), (2,0,0), (3,0,0), (4,0,0),
# Red to yellow-orange
(5,0,0), (5,1,0), (5,2,0), (5,3,0), (5,4,0),
# Bright to pale yellow
(5,5,0), (5,5,1), (5,5,2), (5,5,3), (5,5,4),
# White
(5,5,5);
constant @heatmap-dark =
@heatmap-colors.map: { ~(16 + 36 * .[0] + 6 * .[1] + .[2]) }
The formula used in the map converts a color cube RGB triple into a single color index in the range 16 to 231, which is what the terminal expects to see as a color specifier.
Another consideration is that subtly colored circles will probably be hard to distinguish; it would be clearer to just color the entire contents of each grid cell. The easiest way to do this is to set the cell background on a blank cell by using the on_prefix for the color and a blank space for the “glyph”.
Let’s look at the calculations of $saturation and $c again:
my $saturation = @heatmap-dark.end;
# ...
my $c = 'on_' ~ @heatmap-dark[++@counts[$y]];
Modifying the call to print-cell allows setting the color:
Here’s what that looks like (as an image screenshot rather than a text capture now, in order to show the colors):
It’s beginning to look better, with a vague fiery look and a clear bright band where the ping times are concentrated. Furthermore with fifteen non-black colors in the map, this version of the program now has another three-fold history expansion over the five-glyph version in the previous section — almost 20 minutes of history across a default terminal window.
Precision Flames
While the heat map version has considerably improved information density horizontally, it’s done nothing to change the vertical density; the ping time resolution is just as bad now as it was in the very first version. And because terminal fonts usually make monospace character cells twice as tall as they are wide, the whole chart looks like it’s been smeared vertically. Time to fix that.
Around 2004 Microsoft and various type foundries standardized a list of standard glyphs that modern fonts should supply, called Windows Glyph List 4 or WGL4 for short. This standard was very well supported as a minimum subset for fonts (both free and proprietary) and its full character repertoire was later included in the first stable version of Unicode, cementing it as a solid compatibility baseline.
Among the many very useful glyphs in WGL4 (and thus Unicode 1.1) are the “half blocks”, which split each character cell in half either horizontally or vertically, displaying the foreground color on one half and the background color on the other half. Using the horizontal half blocks can effectively double the chart’s ping time resolution and simultaneously get rid of the vertical smearing effect.
This time all the changes occur in the last few lines of the update-chart sub, starting with a new Y calculation:
# Calculate half-block resolution Y coord from
# scaled ping time
my $block-y = floor(2 * $time / $chart.ms-per-line);
my $even-by = $block-y - $block-y % 2;
my $y = 0 max $bottom - $even-by div 2;
@counts[$block-y]++;
# Determine top and bottom counts for
# half-block "pixels"
my $c1 = @counts[$even-by + 1] // 0;
my $c2 = @counts[$even-by] // 0;
# Create an appropriate colored cell, using half
$ blocks if needed
my $c = $c1 == $c2
?? $grid.cell(' ', 'on_' ~ @heatmap-dark[$c1])
!! $grid.cell('▀', @heatmap-dark[$c1] ~
' on_' ~ @heatmap-dark[$c2]);
# Draw colored cell at (X, Y)
$grid.change-cell($x, $y, $c);
$grid.print-cell($x, $y);
Since each grid cell now represents two half-block “pixels”, it’s necessary to keep track of both the per-half-block counts and the actual cell Y coordinate that a given block falls into. In addition since each cell could be generated as either one flat color or as two different colors, the code takes care to make an optimal custom grid cell, assign it with change-cell, and print it.
Here’s the result:
Much better — lots more detail and no distracting smearing effect.
Errors and Outages
While the half-block chart does a pretty good job showing network latency when the connection is relatively stable, it doesn’t do a good job of showing various errors: outages, individual lost packets, reordered packets, and so on. These can be detected by watching the sequence IDs carefully, and can be displayed on the top line of the chart to give a glanceable view of such problems.
First the error count for the current column must be kept as a new state variable and reset when the @counts are cleared for each new column:
state ($errors, @counts);
# ...
# Clear counts and errors
@counts = ();
$errors = 0;
Then the previous sequence ID must be kept as well, and used to detect sequencing problems and gaps:
# Determine if we've had any dropped packets or
# sequence errors, while heuristically accounting
# for sequence number wraparound
state $prev-id = 0;
$prev-id = -1 if $prev-id > $id + 0x7FFF;
$errors += $id > $prev-id + 1
?? $id - ($prev-id + 1)
!! $id <= $prev-id
?? 1
!! 0;
$prev-id = $id max $prev-id;
It’s not perfect — it can certainly get confused by particularly horrid conditions — but the above algorithm for sequence tracking is similar to the one used by ping itself and should be resilient to many common problems.
If there are any errors, they should be marked after the normal ping time colors are drawn:
# Mark errors if any
if $errors {
my $color = @heatmap-dark[$errors min $saturation];
$grid.print-cell(
$x, 0, {char => '╳', color => "black on_$color"}
);
}
This will indicate individual errors within a single column, but won’t show errors on skipped columns during an extended outage. To handle that, the first part of the code for moving to a new column needs some adjustment to update the error count and then mark the errors if any. Because the update-chart is now going to do the exact same error marking in two different places, it can be wrapped in a private helper sub called where needed:
# Helper routine for marking errors for a
# particular column
my sub mark-errors($x, $errors) {
if $errors {
my $color =
@heatmap-dark[$errors min $saturation];
$grid.print-cell($x, 0, {
char => '╳', color => "black on_$color"
});
}
}
# ...
# If there was a *valid* previous column,
# finish it off
if $prev-x >= $x-offset {
# Missing packets are counted as errors
$errors += 0 max $saturation - @counts.sum;
mark-errors($prev-x, $errors);
# Remove the old "current column" marker
# if it hasn't been overdrawn
$grid.print-cell($prev-x, $bottom, ' ')
if $grid.grid[$bottom][$prev-x] eq '^';
}
# ...
# Mark errors if any
mark-errors($x, $errors);
Here’s what an outage of a little less than a minute looks like now, showing a bright error bar on the top of the chart during the outage:
One Last Bug
There is a remaining subtle bug in the handling of long ping times. Counts are adjusted individually for each (quantized) ping time seen, but long times could map to a quantization bucket arbitrarily far off the top of the chart. Given only fifteen chances in each column, it’s unlikely that any two overlong times will map to the same (off screen) @counts bucket. So even though $y is forced onto the chart before printing, it will likely only ever show the darkest red color even if several very long pings were measured in that column.
To fix this, all of the overlong pings should be counted in a single bucket and be displayed appropriately in the top chart row. As with $errors, let’s track the number of overlong pings in a given column with a new state variable, and reset it when moving to a new column:
state ($errors, $over, @counts);
# ...
# Clear counts and errors
@counts = ();
$errors = 0;
$over = 0;
Then it’s simply a matter of special-casing the top row when drawing the ping time results:
Since this happens before the call to mark-errors, an actual error in a given column will replace any overlong mark that was already there. This is intentional: The top row of the chart is used for “problems”, lost packets have a worse effect on user experience than slow replies, and there’s not enough value in using the top two rows of the screen to separate the two problem types visually.
Here’s the final result, my network roasting on an open fire when the ping time variance has got rather bad for a while:
A Final Present
If you’ve made it this far, I’ve got one last little trick for you. You can change the window title by printing a short escape sequence, so that it’s easier to identify in the giant mess of windows on the typical desktop. (What, you’re going to try to claim your desktop doesn’t have dozens of windows open? Mine certainly does!)
Just add this right after initializing Terminal::Print in MAIN:
# Set window title
my $title = "$target - ping-chart";
print "\e]2;$title\e\\";
And that’s it! Happy holidays to all!
Appendix: (Possibly) Frequently Asked Question
But, but … color theory! The color map isn’t perceptually even!
Well yes, I did gloss over that a bit didn’t I?
In terms of perceptual distance between colors, it would seem much better to jump directly from bright yellow to white without the fine details of light yellows. The perceptual differences in light yellows are much less obvious than in the reds and oranges, so a color palette made from the heat map above appears to have 10 clearly different reds and oranges, and then a subtly-varying “smear” of light yellow. Jumping directly from bright yellow to white sets the two apart decently (though still not as much as the reds and oranges), so the color swatches would look more evenly spaced.
However in practice such a “corrected” map looks worse for the actual ping chart! Ping time jitter makes it unlikely that the top couple colors in the map will actually be shown, as that would require (nearly) every ping time to map to the very same pixel, and thus absolutely rock steady network performance. Aside from perhaps the loopback interface (the computer talking to itself), this is rather unlikely in actual practice. Thus to be able to produce the characteristic bright band of a mostly steady connection, the lightest colors need to be over-emphasized in the color map, which it happens the smooth walk through the light yellows achieves nicely.
Raku Advent Calendar 