Day 1 – Batteries Included: Generating Thumbnails

It was a cold wintry night in the North Pole and Santa was in a mood.

“Naughty. Naughty. Naughty. Ni..aughty” he grumbled, checking his list. Then checking it again.

“Everything ok?” chipped cheerful Sparkleface the elf, bouncing into the room. “Isn’t it nice to have some cold weather for a change?”

Santa scowled at Sparkleface with an icy stare that froze all the water molecules in the room. He said nothing, gazing through Sparkleface into some distant place in another dimension.

Undeterred, Sparkleface continued: “did you see all those wonderful images we’ve received from the children of the world who are looking forward to the holiday, and have been sending us pictures of what they want for Christmas? Isn’t it great that everyone has cell phones these days and can so easily send us high resolution images instead of writing out lists by hand like in the olden days?”

This finally provoked a reply. “No. It is not great.”

Sparkleface started to say something when Santa lowered his voice
and continued —

“Let me tell you precisely why it is not great”

and began to Santasplain the situation:

“There are 2.2 billion children in the world, and so far we have received images from 90% of them. So that’s 1,980,000,000 images. Yes, most of them are high resolution — too high — many are over 20 megabytes each! and we need to email them out to our distributed team of elves! That’s too much data! We need to downscale them all into lower resolution versions before emailing them out. In other words, we need to convert them all into thumbnails.”

“So why don’t you write a script?” Sparkleface let the words come out before realizing the effect they would have.

“First of all” — Santa’s voice began to get louder — “in case you didn’t know, I am 1,750 years old. In fact, this year was my 1,750th birthday. Do you know how many times I have written a script in my lifetime to convert a directory of images into thumbnails?”

“Well…computers have only been around since…”

“Sixty three times. Here at the North Pole we have been using sophisticated technology since way before it was popular. We have always been early adopters — always needing the latest tech in order to help meet increasing demand — and have always had images of toys around, to keep track of things. And every year for some reason I need a new script for generating thumbnails — whether it’s for new image formats, compression schemes, or new use cases. This is the first year that it’s been because of high resolution images coming from cell phones, that we need to downscale.”

Santa continued: “Thumbnail generation scripts are tedious to write.” and listed the features that he wanted for this year’s script:

• the ability to use all of the CPUs available — scaling images in parallel to maximize our resource usage, and minimize the time it takes
• an accurate count of the number of images successfully scaled
• a way to do a dry-run
• some sort of verbose option to see what the program is doing
• a way to re-run the program and force overwriting of previous images
• some kind of documentation

Sparkleface smiled wryly, and sat down at Santa’s keyboard. “Give me a minute.”

Sparkleface then typed out a program faster than you can sing “Rudolph the Red Nosed Reindeer.”

“Okay try it, this script is called sparkle-sizer. `sparkle-sizer -h” will show the options”.

Santa tried it:

$ sparkle-sizer -h
Usage:
  ./sparkle-sizer [-n] [-v] [--force] [--degree[=Int]] [<dir>]

    -n                dry run
    -v                verbose
    --force           force regeneration of existing thumbnails
    --degree[=Int]    degree of parallelism

Santa ran it with --degree=12 and watched with glee as the output of htop showed the CPU usage:

“Not bad” he said. It’s using all of the CPUs! And he proceeded to look at the source code:

#!/usr/bin/env raku

sub mk-thumb(IO::Path $src, Bool :$force) {
  my $dst = "{$src.dirname}/thumb-{$src.basename}";
  return False if $dst.IO.e && !$force;
  (shell "gm convert -auto-orient $src -thumbnail '400x400>' $dst") == 0;
}

multi files(IO::Path $f where *.f) { $f }
multi files(IO::Path $f where *.d) {
  my @ls = $f.IO.dir(
      test => { not .starts-with('thumb' | '.' ) }
     );
  @ls.map: { |files($_) }
}

multi MAIN($dir = "photos",
  Bool :$n,          #= dry run
  Bool :$v,          #= verbose
  Bool :$force,      #= force regenerate existing thumbnails
  Int  :$degree = 4, #= degree of parallelism
) {

  say "dry run!" if $n;
  &shell.wrap: -> |c { note c.raku if $n || $v; callsame unless $n }

  my atomicint $converted = 0;
  my atomicint $considered = 0;

  files($dir.IO).race(:$degree).map: -> $f {
    with ++⚛$considered {
      note "$_ files considered" if $_ %% 100;
    }
    ++⚛$converted if mk-thumb($f, :$force);
  }

  say "converted $converted out of $considered";
}

“No dependencies?” asked Santa. “How does it work? Tell me more.”

Now it was Sparkleface’s turn to explain:

The beginning, mk-thumb, converts a single image into a thumbnail using “shell“. Raku has gradual typing, so we can enforce type constraints. Also if $force is true, we overwrite any old files.

Then we have “files“, to recursively traverse a directory. Note that Raku has multi dispatch — and the extra constraints on the arguments allow us to write make a sort of Prolog-style directory traversing set of routines.

We then have MAIN which declares the main program. Named arguments become command-line arguments, and comments are printed out as a help message.

We use “wrap” to get dry-run capability — wrapping the “shell” built-in — printing the arguments when we are verbose or doing a dry-run, and only calling the real “shell” for the non-dry-run case.

We have two counters for keeping track of files we’ve considered and converted. We use an atomic int so we can increment it from various threads and not worry about race conditions.

We then use “race” to run the conversion in batches across multiple threads. Note the atomic operators which allow us to increment our native variables and not worry about race conditions.

“Atomic operators” Santa said. “I thought they were snowflakes.” Santa’s mood had lightened.

While running the program, Sparkleface thought he could see a little twinkle in Santa’s eyes and had hope that Santa now had one less thing to worry about before the big night.