Day 1 – Raku from Perl: Transforming Old Perl Code

Posted byTom Browder (@tbrowder)Posted in2019, Perl

Introduction

I have been using Raku (Perl 6’s new name) since mid-2015 and really appreciate its nice features for programmers who may be lazy, non-touch typists, amateurs, old Perl lovers, non-parallel users, or wannabe hackers including:

  • kebab-case names
  • brace-less control statements
  • easy class construction
  • lexical block variables
  • copious built-in routines
  • use unicode natively
  • powerful, easy-to-use function signatures

but one of its featured non-core modules has really come into play for me recently, so I am highlighting it as a surprise Raku gift for me this year: the Raku module Inline::Perl5, written by Stefan Seifert (IRC #raku: ‘nine’; Github: ‘niner’).

Before proceeding, please note the new Raku links for the Raku home page, Raku docs, and Raku modules.

NOTE: At the moment I only use Raku on Debian Linux hosts, so I can’t help if you have any problems on Windows or Mac OSX with any of the following.

Background

I describe myself as a pragmatic programmer (i.e., getting the job done ASAP with no frills), with little formal programming training except during college in the main-frame, batch job era, and later, off-duty, while in my last job in the US Air Force. (See this document for more context.)

Soon after being hired in mid-1993 at my last civilian employer (another US DoD contractor, from whom I retired on 2016-01-011), I discovered Perl 4 and found it was the ideal language to create the tools I needed for our small local office to move from an intensive manual process to a more automated one (I was then using C heavily, and Perl was the first interpreted language I had used since Basic). Over the years, I eventually moved to Perl 5 (much later than I should have) and continued to grow and improve my company’s software tool box (much written at home on my own time), now on Redhat Linux computers, all in Perl, which included automatic image generation and document production (using Perl to write PostScript converted to PDF). The documents produced enabled my team of analysts to see standard results plots, tables, and other generated metrics and thus had more time and could easily write their detailed analyses which were then incorporated into the final products.

In addition, I built other products for my personal use including a personalized calendar with integrated database, Christmas Card address database with label maker, and several websites. In sum, I have a lot of old as well as more recent Perl code at my house!.

2015 and Raku

I had always hoped to see Raku (Perl 6) coming soon, because the existing language seemed to be a little clunkier than it could be, but CPAN and its wonderful module authors, especially Damian Conway, helped mitigate the bumps.

So I was very happy to join in the -Ofun when I checked on the progress of Raku in mid-2015 and saw the impending initial stable release. I immediately started trying to convert some of my Perl products to pure Raku starting with some of the CPAN modules important to me in my personal projects. The first was Perl’s Geo::Ellipsoid which was a real learning experience and took much longer than I thought. Eventually I published eight pure Raku modules to CPAN.

Fast forward to 2019

When I started porting my own tools to Raku this year the real fun began. When I built the original tools, much of it was done in a rush with little time for thought and design, and very little testing, and certainly not a test suite. Consequently, I had lots of ugly code sitting around ready to be ported to Raku. To paraphrase Dr. Strangelove [Ref. 1], I stopped worrying about the mess and started working on a Raku port.

Part 1: Preliminary testing

I first started with porting modules and then the programs that used them but found that to be far too labor intensive in many cases. I encountered problems with lack of signatures, much use of GOTOs, global variables in long main programs (a.k.a. scripts) with lots of subroutines, etc.

So I finally, just this year, decided to try using Inline::Perl5 to simplify my chore. I changed my porting process to:

  1. Move existing subroutines in Perl programs to Perl modules.
  2. Ensure the Perl programs continue to work as expected after step 1.
  3. Port the now-much-shorter Perl programs to Raku, a much easier task than before.

Before I seriously started I created a Raku script to find all my Perl files (using File::Find with regex /['.pm'|'.pl']$/`), read them line-by-line, and write them out again to see if there were any issues handling them with Raku, and I certainly did: in some of my very old code (mid 1990s) I got errors about malformed utf8 like this:

ERROR: something failed in file 'make_color_book.pl': Malformed UTF-8

I tried several methods to isolate the bad lines, including using the *nix tool od but that was painfully slow and visual inspection with vim didn’t always work. (I didn’t get around to using either Emacs or comma since I was doing the work remotely, so I don’t know if that would help.) Luckily, I stumbled on a trick while I was using a limited set of files for testing when I used this fragment in my program

try { my $string = slurp $infile }
if $! {
    note "Error attempting to slurp file '$infile'";
    note "$!";
}

and a UTF-8 error was detected I would get an error message like

Error attempting to slurp file 'PP.pm'
Malformed UTF-8 at line 179 col 66

which enabled me to easily see the problem character in the original file and change it to valid UTF-8.

Importing Perl modules into both Perl and Raku

When modifying the existing Perl modules to be used by Perl as well as Raku I found two final problems that overlap:

  1. In the Perl programs and their Perl modules, how does one handle sets of
    global variables found missing when the programs’ subroutines are
    moved into an existing or new Perl module?
  2. How does one export subs and vars from the Perl module into both Perl
    and Raku programs?

Problem 1: Global variables

Inline::Perl5 doesn’t currently describe accessing variables, and, of course, such practices are not recommended at all, but, with help from the author (Stefan Seifert), I found a way. We start with an example Perl module to be used with both Perl and Raku programs, say P5.pm, which looks like this (file P5.pm):

package P5;

use feature 'say';
use strict;
use warnings;

#| The following module does NOT affect exporting to Raku, it only
#| affects exporting to Perl programs. See program `usep5.pl` for
#| examples.
use Perl6::Export::Attrs; #= [from CPAN] by Damian Conway

our $VERSION = '1.00';

#| Always exported (no matter what else is explicitly or implicitly
#| requested):
our %h :Export(:MANDATORY);
our $pa :Export(:MANDATORY);

#| Export $pb when explicitly requested or when the ':ALL' export set
#| is requested.
our $pb :Export(:DEFAULT :pb);

#| Always exported:
sub set_vars :Export(:MANDATORY) {
    %h = ();
    $h{a} = 2;
    $pa = 3;
    $pb = 5;
}

#| Always exported (no matter what else is explicitly or implicitly
#| requested):
sub sayPA :Export(:MANDATORY) {
    say "  \$pa = $pa";
}

#| Always exported:
sub sayPB :Export(:DEFAULT :sayPB) {
    say "  \$pb = $pb";
}

#| Always exported:
sub sayH :Export(:MANDATORY) {
    foreach my $k (sort keys %h) {
        my $v = $h{$k};
        say "  key '$k', value '$v'";
    }
}
1; #= mandatory true return

Problem 2: Exporting global variables

As noted in module `P5.pm`, the export information provided by `Perl6::Export::Attrs` is only for the use of Perl code using `P5.pm` (it will not affect Raku programs using `P5.pm`). However, inserting `use Perl6::Export::Attrs;` in any Perl module greatly eases the task of exporting as desired without a lot of boiler plate Perl code. One doesn’t have to use it, but I highly recommend it. A bonus is that eventually porting the Perl module to Raku will be easier.

Perl programs using module P5

One can access the objects in the Perl module in a Perl program like this (file use5.pl):

#!/usr/bin/env perl
use feature 'say';
use strict;
use warnings;

use lib qw(.);
use P5 qw($pb sayPB); # <== notice the explicit requests

set_vars;

my %h = %P5::h;
say "Current globals in P5:";
foreach my $k (sort keys %h) {
    my $v = $h{$k};
    say "  key '$k', value '$v'";
}
sayPA();
sayPB();

say << "HERE";

Modify current globals in P5:
  \$P5::h{a} = 3
  \$P5::h{c} = 5 # a new key/value pair
  \$P5::pa = 4
  \$P5::pb = 6
HERE

$P5::h{a} = 3;
$P5::h{c} = 5;
$P5::pa = 4;
$P5::pb = 6;

say "Revised globals in P5:";
sayH();
sayPA();
sayPB();

Raku programs using module P5

And one can access the Perl module’s objects in a Raku program like this (file use5.raku)

#!/usr/bin/env perl6

#| IMPORTANT
#| Notice no explicit use of Inline::Perl5, but it
#| must be installed.
use lib:from '.'; #= Must define the Perl lib location with this syntax.
use P5:from;      #= Using the Perl module.

#| IMPORTANT
#| =========
#| Bind the hash variable so we can modify the hash.
#| For access only, use of the '=' alone is okay.
set_vars;

my %h := %*PERL5;

say "Current globals in P5:";
for %h.keys.sort -> $k {
    my $v = %h{$k};
    say "  key '$k', value '$v'";
}
sayPA();
sayPB();

say qq:to/HERE/;

Modify current globals in P5:
  \%h = 3
  \%h = 5 # a new key/value pair
  \$P5::pa = 4
  \$P5::pb = 6
HERE

%h = 3;
%h = 5;

#| IMPORTANT
#| Need this syntax to access or modify a scalar:
$P5::pa = 4;
$P5::pb = 6;

say "Revised globals in P5:";
sayH();
sayPA();
sayPB();

The three three test files all work together and provide a blueprint for working with my real code.

Part 2: Using real code

In this section I will be using files from one of my projects: my college class website (see it here). I started it in 2009 and have been adding to it and maintaining it often, so it has a lot of crufty Perl code. I have created a Github repository which contains the code I’ll be using in the following discussion. You can follow along by cloning it like this:

$ git clone https://github.com/tbrowder/raku-advent-extras.git

The code I will be using is in the raku-advent-extras/2019/ directory. The code should be sanitized so no non-public information is shown, and it will not be totally functional, but the main script, manage-web-site.pl, should always run if executed without any arguments. Let the games begin!

Finding global variables

Using the syntax examples above in my real Perl modules, I first moved the obviously marked global variables in a Perl program to a new Perl module named with a single letter for easy use such as G.pm (for Global). For example, finding a variable $start_time in the main program I would rename it to $G::start_time and put it into the G.pm module as our $start_time.

Then I exercised the program repetitively, finding more global variables at each run, adding them to the module, and so on until all globals were found.

The first real files to work with after defining Perl global variables are the program file manage-web-site.pl, and two Perl modules PicFuncs.pm and G.pm and they will be used in the rest of this article. To get to a common starting point, in the git repo:

$ git checkout stage-0

and ensure the main script runs with no arguments:

$ ./manage-web-site.pl
Usage: ./manage-web-site.pl -gen | -cvt [-final][-useborder][-usepics][-debug][-res=X]
                      [-force][-typ=X][-person][-stats][-warn]
Options:
[...snip...]

Now start a new branch: $ git checkout -b stage-1.

Stage-1: Move all subs in the main program to a new Perl module

At this point I’m going to finish moving all the Perl subs in manage-web-site.pl to a new module OtherSubs.pm. I’ll do it one at a time, execute the program to see if we have any problems, and so on until all (or most) subs are stashed in the new Perl module. The steps

  • Create OtherSubs.pm
  • Add use OtherSubs to the program
  • Remove sub dequote (not needed)
  • Move sub Build_web_pages to OtherSubs.pm
  • Execute manage-web-site.pl: PROBLEMS WITH GLOBAL VARS!!

I got the following symbols missing:

Global symbol "$CL_HAS_CHANGED"...
Global symbol "$CL_WAS_CHECKED"...
Global symbol "$GREP_pledge_form"...
Global symbol "$USAFA1965"...
Global symbol "$USAFA1965_tweetfile"...
Global symbol "$debug"...
Global symbol "$dechref"...
Global symbol "$force_xls"...
Global symbol "$real_xls"...

After I resolved that issue, I continued to move subs, and resolve new global variables, until all were moved. You should see a commit message after each sub was successfully moved. I stopped with one sub left the program file, sub zero_modes, since it is part of the option handling and shouldn’t normally be in a module.

Stage-2: Port the Perl program to Raku

For this part I started a new branch from the stage-1 branch: $ git checkout -b stage-2.

I’m sure every Raku programmer will proceed to port a Perl program to Raku in a different way, but following is my general recipe.

  1. Copy the existing program, in this case manage-web-site.pl, to an equivalent Raku name, manage-web-site.raku (see Notes 1 and 2 below).
  2. Change the shebang line to use perl6.
  3. Execute manage-web-site.pl: PROBLEMS!!

I got the following errors:

===SORRY!===
Could not find feature at line 3 in:
    inst#/home/tbrowde/.perl6
    inst#/usr/local/rakudo.d/share/perl6/site
    inst#/usr/local/rakudo.d/share/perl6/vendor
    inst#/usr/local/rakudo.d/share/perl6
    ap#
    nqp#
    perl5#

I then found one problem that I haven’t addressed in the general process: conflicting global symbols. That happened when I tried to use the Raku version of module Geo::Ellipsoid and some of the Perl versions were also using it. I solved the immediate problem by commenting out the Raku version and using the Perl version in the program file.

After I resolved that issue, I continued to remove or replace used modules, handle more global variables, find or ignore missing subroutines, replace =pod/=cut with =begin comment/=end comment, remove unneeded parens, use Raku idioms (e.g., Raku ‘for’ versus Perl ‘foreach’), and fix issues until all were resolved. You should see a commit message after each issue was successfully resolved. I also tried to clean up the code while I worked.

Stage-3: Tidying the Raku program

Finally, the program manage-web-site.raku runs (with no input arguments) with no errors. At this point I checked out a stage-3 branch for cleaning the program a bit: git checkout -b stage-3. I removed a lot of comments and removed parens. I also removed from use modules that aren’t now actually used in the program file after the subs were moved. Additionally, I made the help system a bit cleaner. I leave one obvious Raku feature to be added as an exercise for the user: in the ugly if/else blocks for option selection, change to use Raku’s when blocks.

We started with a manage-web-site.pl file with about 6600 lines of ugly Perl code and finished with a Raku version, manage-web-site.raku. with less than 800 lines and a much cleaner look. We’re not finished with the port yet: we still have to test each option for proper functioning (and I’m sure there be dragons 🐉!). Ideally, we’ll also add tests in the process. But we don’t have all the necessary content for that, so we’ll stop at this point (but, follow me on my next steps in Part 2 of this post on Day 9).

Summary

You have seen one way to ease porting Perl code to Raku, and I hope it may help those who are considering moving to Raku see that it can be accomplished iteratively in smaller steps instead of taking great chunks of time. Part 2 of this post on Day 9 will try to take the next baby step and convert a Perl module to Raku and have it be used by both Perl and Raku callers.

I ❤️ ❤️ Raku! 😊

🎅 Merry Christmas 🎅 and 🥂 Happy New Year 🎉 to all and, in the immortal words of Charles Dickens’s Tiny Tim, may ✝ “God bless Us, Every One!” ✝ [Ref. 2]

APPENDIX

Notes

  1. I actually started the file rename while accidentally in the stage-1 branch, sorry.
  2. The file extension of ‘.raku’ is the community-accepted convention for Raku executable programs. However, for the foreseeable future, its use (on *nix systems) depends on having the perl6 Rakudo compiler installed and one of two other conditions: (1) the user’s program file marked as executable with chmod x and having the proper shebang line as the first line of the file or (2) executing the program as perl6 myprog.raku. Sometime hopefully soon, when the Rakudo compiler’s executable is available as raku and it is installed on your system, replace perl6 in the instruction above with raku. (Windows and Mac users will have to get their instructions from other sources.)

References

  1. Movie (1964): Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (see Imdb.com)
  2. A Christmas Carol, a short story by Charles Dickens (1812-1870), a well-known and popular Victorian author whose many works include The Pickwick Papers, Oliver Twist, David Copperfield, Bleak House, Great Expectations, and A Tale of Two Cities.

Raku modules used (install with zef)

  • Inline::Perl5

Perl modules used from CPAN (install with cpanm)

  • Perl6::Export::Attrs
%d bloggers like this: