Day 20 – Raku from Perl: Transforming Old Perl Code (Part 2)

Spoiler Alert!

When I started this two-part post, I was blissfully unaware of a similar topic in the Raku community; however, I was awakened to that fact when I first read Elizabeth Mattijsen’s Raku Weekly Blog on Monday, 9 Dec, and saw that the famous Perl and Raku expert, Jeff Goff, had written a multi-part series on porting a very complex Perl module, to Raku. The only saving grace for me is that the posts are highly technical, and its audience is for serious Raku hackers who want to produce native Raku code for the most complicated Perl modules in existence: a highly recommended read for the accomplished programmer who relishes reverse-engineering and all its tribulations! So, with a tip of my hat to Jeff, I will continue to slog along here in my old Perl garden which is so overgrown with newby, self-taught Perl.

One more side note: the author of the module Jeff is porting, John McNamara, is my favorite Perl module author, whose wonderful modules enabling Perl users to slice and dice Microsoft Excel were life-savers for me in my last job. I have had several e-mail discussions with John over the years, and he is a kind and very talented man who has contributed so much for Perl user. Many thanks, John!

Introduction

In this post we will continue to port old Perl code to Raku. In order to follow this Part 2 you should have read the Part 1 post. Make sure you clone the exercise code from Github so you can follow along since much of the actual porting problems and solutions are shown as git commits in each stage-N branch.

The next step in our porting adventure is to start transforming Perl modules to Raku. During the transition from Perl to Raku we will strive to duplicate (port) every Perl module to Raku.

In our previous article we continued to ensure the transformed program ran (without arguments) while we moved all its Perl subroutines to a Perl module. Now, while we port Perl modules, we will also check actual operation of the driver program. We expect to find many more challenges as we continue working through my old code, so let’s dig in!

Stage-2: where we left off

At the end of Part 1 I said there was one more thing I would do to the the driver program (manage-web-site.raku): replace the if/else blocks with when blocks, so I’ll do that now. Please go the exercise repository directory for 2019 and ensure you are in the branch stage-2 with no changes uncommitted.

In order to use the when blocks we must use the implicit topic variable ($_) instead of the $arg variable; however, at the moment we still need it so we remove it from the loop variable and temporarily declare it at the top of the block (see Note 1): my $arg = $_. Now let’s execute the program again, this time with the -h (help) option:

$ ./manage-web-site.raku -h
Use of uninitialized value of type Any in numeric context
  in block  at ./manage-web-site.raku line 184
Use of uninitialized value of type Any in numeric context
  in block  at ./manage-web-site.raku line 185
Use of uninitialized value of type Any in numeric context
  in block  at ./manage-web-site.raku line 186
No such method 'Int' for invocant of type 'Any'
  in block  at ./manage-web-site.raku line 186

Here are the offending lines:

# lines 181-187:
my $arg = $_;                    # <= new decl of $arg, set to topic variable's value
my $val;
my $idx = index $arg, '=';       # = 0) {                 # <= this is where the problem first surfaces
    $val = substr $arg, $idx+1;  # <= and then here
    $arg = substr $arg, 0, $idx; # <= and here
}

What has happened is another syntax change between Perl and Raku has surfaced: the index routine in Perl returns a ‘-1’ if the needle is not found, but in Raku it returns undefined so our existing Perl test for an integer throws an error. The solution is to test the return value for definedness. As in Perl, We can’t just test for zero with an if because that is a valid value but it isn’t truthy. So, we change our code to this:

# lines 184-187:
if $idx.defined {                # <= this is the only change needed
    $val = substr $arg, $idx+1;
    $arg = substr $arg, 0, $idx;
}

After a few more cleanups we’re ready to start porting a module. Ensure your git repo is on branch stage-2> and clean with no uncommitted changes. Then execute:

$ git checkout -b stage-3

Porting a Perl module

First, let’s look at a few of the common issues we will meet:

1. Converting Perl’s export methods to Raku’s much simpler syntax
2. Converting Perl-style calls to Raku’s function signatures
3. Converting Perl’s foreach loops to Raku’s for @arr -> { loops
4. Converting Perl’s for (my $i...) {...} loops to Raku’s loop (...) {...}

There will definitely be more issues, and I’m sure the changes I make may not be the ones a person more knowledgeable of modern Perl may make but remember, a lot of my old Perl predates modern Perl, and I’m not always taking the time to improve the code but just getting a first cut at a working solution.

Before we look at the old Perl we will look at working examples of handling export and signatures. The following Perl module (P5.pm) is using Damian Conway’s Perl6::Export::Attrs module for simpler and Raku-like export handling. Note the three types of argument passing.

package P5;

use feature 'say';
use strict;
use warnings;

# This module does NOT affect exporting to Raku, it only affects
# exporting to Perl programs. See program `usep5.pl` for examples.
use Perl6::Export::Attrs; # [from CPAN] by Damian Conway

our $VERSION = '1.00';

# Always exported:
# sub passing all args via @_ array:
sub sayAB :Export(:MANDATORY) {
    my $a = shift;
    my $b = shift;
    $b = 0 if !defined $b;
    say "\$a = '$a'; \$b = '$b'";
}

# Export sayABC when explicitly requested or when the ':ALL' export is set
# sub passing args via a hash:
sub sayABC :Export(:sayABC) {
    my $href = shift;
    my $a = $href->{a};
    my $b = $href->{b};
    my $c = $href->{c};
    $a = 0 if !defined $a;
    $b = 0 if !defined $b;
    $c = 0 if !defined $c;
    say "\$a = '$a'; \$b = '$b'; \$c = '$c'";
}

# Always exported:
# sub pass rw args via a ref
sub changeC :Export(:MANDATORY) {
    my $cref = shift;
    my $newc = shift;
    $${cref} = $newc;
}

1; # mandatory true value 

The following Perl program (usep5.pl) shows how to use the three types of argument passing. It also shows how the export-restricted sub (sayABC) is used.

#!/usr/bin/env perl
use feature 'say';
use strict;
use warnings;

use lib qw(.);
use P5 qw(:sayABC);

sayAB(1);             # using the @_ for passing arguments
sayABC({a=>1, b=>2}); # using a hash for passing arguments

my $c = 1;
say "\$c = $c";
changeC(\$c, 2); # using a reference to pass a read/write variable
say "\$c = $c";

We exercise the Perl script:

$ ./usep5.pl
$a = '1'; $b = '0'
$a = '1'; $b = '2'; $c = '0'
$c = 1
$c = 2

We then see the Raku version of the Perl script and module demonstrating the changes necessary to get the same output result (but see Note 2).

unit module P6;

#| Always exported:
#| ported from a Perl sub passing all args via @_ array:
sub sayAB($a, $b = 0) is export {
    say "\$a = '$a'; \$b = '$b'";
}

#| Export &sayABC when explicitly requested or when the ':ALL' export is set
#| ported from a Perl sub passing args via a hash:
sub sayABC(:$a, :$b, :$c = 0) is export(:sayABC) {
    say "\$a = '$a'; \$b = '$b'; \$c = '$c'";
}

#| Always exported:
#| ported from a Perl sub passing a read/write arg:
sub changeC($c is rw, $newc) is export {
    $c = $newc;
}

#!/usr/bin/env perl6

use lib ;
use P6 :ALL; #= <= ensures all exportable object are exported

sayAB 1;
sayABC :b, :a;

my $c = 1;
say "\$c = $c";
changeC $c, 2;
say "\$c = $c";

Executing the Raku script should show the same results:

$ ./usep6.raku
$a = '1'; $b = '0'
$a = '1'; $b = '2'; $c = '0'
$c = 1
$c = 2

Voila! I hope you can see the Raku module’s version is cleaner and simpler looking as compared to that of Perl.

To lead us to the first module we want to port we will execute the main option in the driver program:

$ ./manage-web-site.raku -gen
Collecting names by CS...
Unable to open file 'usafa-template1-letter.ps': No such file or directory
  in method call-args at /usr/local/rakudo.d/share/perl6/site/sources/ACCE801FB16076DAD1F96BE316DBFEDD148902C8 (Inline::Perl5) line 430
  in sub  at /usr/local/rakudo.d/share/perl6/site/sources/ACCE801FB16076DAD1F96BE316DBFEDD148902C8 (Inline::Perl5) line 935
  in block  at ./manage-web-site.raku line 449

If we look at the line (449) that apparently caused the failure we see the following:

build_montage(%CL::mates, $genS); #=  <= in module ./PicFuncs.pm5

and we see that module PicFuncs.pm5 is the entry point. So we will select that module to start with and copy it to PicFuncs.pm6. We open the new file in our favorite editor and see some changes we can make immdediately (starting at the first line):

• change package to unit module
• remove the next file lines of code
• change all sub lines reading :Export(:DEFAULT) to is export
• move sub build_montage to the top of the module to ease porting one sub at a time
• remove Perl code for true return at the bottom of the file

Now let’s take a look at some other things we notice:

• the module is using the G.pm which is in Perl format, so we will have to either convert that next or change the way we use it inside the PicFuncs.pm6 module. I choose to convert the G.pm module to G.pm6
• to ease the port we want to end the PicFuncs.pm6 module after the first sub by use of the =finish pod feature
• we have to change the way we use the PicFuncs module inside the manage-web-site.raku file

You can probably guess that soon we will be in a situation where we can’t get much farther without converting the rest of the Perl modules to Raku. But on to tending to the list above…

To get a check on our progress we execute the driver program again:

$ ./manage-web-site.raku -gen
===SORRY!=== Error while compiling /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm (PicFuncs)
This appears to be Perl 5 code. If you intended it to be Perl 6 code, please use a Perl 6 style declaration like "unit package Foo;" or "unit module Foo;", or use the block form instead of the semicolon form.
at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm (PicFuncs):1
------> package PicFuncs;⏏

Whoa, looks like we need to try to separate the two types of modules since the names confuse Raku. I’ll first try rearranging some code in the driver….

The ‘use’ statement and module searches

Note there are some differences in the way Perl and Raku handle the usestatement. In Perl, modules are searched for modules in paths in the following order: paths defined in the use lib statement, paths defined in the environment variables PERL5LIB and PERLLIB (in that order), and, finally, in paths defined in the @INC array. In any list of paths, individual paths are separated by the colon character (‘:’) which is consistent with the OS path separator.

Raku is very different. First, we cannot use the use statement in a module because it will be precompiled before use and the use statement cannot be precompiled. Second, Raku uses paths searched for in the following order: paths defined in the use lib statement, paths defined in the environment variable PERL6LIB, and paths defined during Raku installation. In any list of paths, individual paths are separated by the comma character (‘,’) which is consistent with lists in Raku. Here are the search paths on my computer for this test environment (PERL6LIB=foo,bar):

$ perl6 -e 'use lib ; use MyModule'
===SORRY!===
Could not find MyModule at line 1 in:
    file#/home/tbrowde/raku-advent/raku-advent-2019
    file#/home/tbrowde/raku-advent/raku-advent-2019/foo
    file#/home/tbrowde/raku-advent/raku-advent-2019/bar
    inst#/home/tbrowde/.perl6
    inst#/usr/local/rakudo.d/share/perl6/site
    inst#/usr/local/rakudo.d/share/perl6/vendor
    inst#/usr/local/rakudo.d/share/perl6
    ap#
    nqp#
    perl5#

Resume porting…

Now try again:

$ ./manage-web-site.raku -gen
===SORRY!===
Unsupported use of 'foreach'; in Perl 6 please use 'for'
at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):48
------>     foreach⏏ my $cs (@cs) {
Other potential difficulties:
    To pass an array, hash or sub to a function in Perl 6, just pass it as is.
    For other uses of Perl 5's ref operator consider binding with ::= instead.
    Parenthesize as \(...) if you intended a capture of a single variable.
    at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):23
    ------>     U65::get_keys_by_sqdn(\⏏%sqdn, $mref);

We see now the first glimpse of some of the loop differences between Perl and Raku. I’ll start with changing the loops. One warning about Perl’s for (my $i = 0; $i < $max; ++$i) {...}. The direct translation into Raku is this: loop (my $i = 0; $i < $max; ++$i) {...}, but therein lies a surprise. The loop index variable, which in the scope of the loop in Perl, is in the outer scope of the loop’s parentheses in Raku! So I got in the habit of rewriting a Perl loop to emphasize the proper scope of the index variable and a better pointer to possible problems of duplicate declarations:

my $i;
loop (my $i = 0; $i < $max; ++$i) {...}

Now we start again with execution testing…oops, another syntax problem:

$ ./manage-web-site.raku -gen
===SORRY!===
Unsupported use of @{$sqdn{$cs}; in Perl 6 please use @($sqdn{$cs) for hard ref or @::($sqdn{$cs) for symbolic ref
at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):55
------>         my @n = @{$sqdn{$cs}⏏};
Other potential difficulties:
    To pass an array, hash or sub to a function in Perl 6, just pass it as is.
    For other uses of Perl 5's ref operator consider binding with ::= instead.
    Parenthesize as \(...) if you intended a capture of a single variable.
    at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):23
    ------>     U65::get_keys_by_sqdn(\⏏%sqdn, $mref);

A huge dfference between Perl arrays and hashes and those of Raku. I’ll tend to those, too…another problem:

$ ./manage-web-site.raku -gen
===SORRY!=== Error while compiling /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs)
Unsupported use of ->(), ->{} or ->[] as postfix dereferencer; in Perl 6 please use .(), .[] or .{} to deref, or whitespace to delimit a pointy block
at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):137
------> 	        my $fname = $mref->{⏏$c}{file};

This problem is from the differences in signature and passing arguments to subs from the caller as discussed earlier in the simple examples. I will work through the signature to see if we can help the situation. I’m first going to look at the calling line in the driver we looked at before:

build_montage(%CL::mates, $genS);

We see two arguments: a hash and an apparent scalar. We changed those in Part 1 to Raku from their original Perl syntax using a reference. So we are fine on the calling side at the moment. Back to the called sub where the first few lines look like this (with comments stripped out):

sub build_montage is export {
    my $mref  = shift @_; # \%CL::mates
    my $cs    = shift @_;

We’re going to change that to:

sub build_montage(%mates, $cs) is export {

and proceed with other required changes…After quite a few changes we start running into missing routines:

$ ./manage-web-site.raku -gen
===SORRY!=== Error while compiling /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs)
Undeclared routines:
    convert_single_pic_to_eps used at line 159
    insert_logo used at lines 287, 297
    insert_pictures used at line 279

This looks like a good stopping place. As I said earlier, we are probably at the point where we will have to convert almost the whole project, or at least those modules which are used by the driving script as well as all the other modules they use. I checked out a stage-4 branch and did some more housekeeping and tidying, but I leave the continuation of porting for another day.

Summary

In these two articles you have seen one way to ease porting Perl code to Raku, and I hope they may help those who are considering moving to Raku see that it can be accomplished iteratively in smaller steps instead of taking great chunks of time. The results, for me, include code that is:

• more visually appealing (cleaner and less cluttered)
• easier to maintain
• easier to see where to improve it

If you are interested in seeing the rest of the conversion, execute .ask tbrowder on IRC channel ‘#raku’ and I’ll keep on plugging on the repo we’ve been using (and it will help me, too!). Happy Rakuing

I ❤️ ❤️ Raku! 😊

🎅 Merry Christmas 🎅 and 🥂 Happy New Year 🎉 to all and may ✝ “God bless Us, Every One!” ✝ [Ref. 1]

---

APPENDIX

---

Notes

1. I wasn’t sure if changing the value of the topic variable in the loop would allow the when blocks to continue to work as expected. Fortunately, it does.
2. There are some subtle (to me) issues when using various module export options in Raku. The current discussion in the docs here (see the ordered list of five Notes) do not describe exhaustively all possible combinations of export options, and as a result I (who was the person who wrote those notes) saw an error message when I used a new combination of conditions that, in my opinion, was misleading. Consequently I first suspected a Rakudo bug but now suspect it is an LTA (Less Than Awesome) error message. Keep an eye on Rakudo issue #3341 for a resolution.

References

1. A Christmas Carol, a short story by Charles Dickens (1812-1870), a well-known and popular Victorian author whose many works include The Pickwick Papers, Oliver Twist, David Copperfield, Bleak House, Great Expectations, and A Tale of Two Cities.