Day 16 – It’s Too Generic; Please Instantiate!

As the Christmas approaches, the time for unwrapping presents is near. Let’s review a present Rakudo release 2023.12 is going to have for you.

Long ago I once got a question related to type capturing. Don’t remember the question itself, but remember the answer Jonathan Worthington gave to it (rephrasing up to what my memory managed to keep): “You can use it, but it’s incomplete.” This is what they call “to be optimistic”. Anyway, a lot has changed in this area since then.

Why

WWW::GCloud family of modules is inevitably heavily based on two whales: REST and JSON. The former is in strong hands of the Cro framework. For JSON I ended up with creating my own JSON::Class variant, with lazy deserializations support and many other features.

Laziness in JSON::Class is also represented by special collection data types: sequences and dictionaries. Both are not de-serializing their items until they’re read from. This is a kind of virtue that may play big role when dealing with, say, OCR-produced data structures where every single symbol is accompanied with at least its size and location information.

In a particular case of Google Vision it’d be the Symbol with this representation in my Raku implementation. The problem is that the symbols are barely needed all at once. Where JSON::Fast manages very well with these, producing a full tree of objects is costly. But while converting WWW::GCloud for using JSON::Class:auth<zef:vrurg> I stumbled upon a rather unpleasant issue.

Google APIs use a common pattern when it comes to transferring long lists of items, which involves paginating. A structure, that supports it, may look like ListOrgPoliciesResponse, or like an operations list, or many other of their kind. Since the nextPageToken field is handled consistently by all similar APIs, it makes sense to provide standardized support for them. Starting with a role that would unify representation of these responses. Something like:

role Paginatable[::ITEM-TYPE] {
    has Str $.nextPageToken;
	has ITEM-TYPE:D @.items;
}

See the real thing for more details, they are not relevant here. What is relevant is that for better support of JSON::Class laziness I’d like it to look more like this:

role Paginatable[::LAZY-POSITIONAL] {
	has Str $.nextPageToken;
	has @.items is LAZY-POZITIONAL;
}
class OpList is json(:sequence(Operation:D)) {}
class Operations is json does Paginatable[OpList] {}

Or, perhaps, like this:

role Paginatable[::RECORD-TYPE] {
	my class PageItems is json(:sequence(RECORD-TYPE)) {}
	has Str $.nextPageToken;
	has @.items is PageItems;
}
class Operations is json does Paginatable[Operation:D] {}

Neither was possible, though due to different causes. The first one was failing because the LAZY-POSITIONAL type capture was never getting instantiated, resulting in an exception during attempt to create an @.items object.

The second case is even worse in some respect because under the hood JSON::Class creates descriptors for serializable entities like attributes or collection items. Part of descriptor structure is the type of the entity it represents. There was simply no way for the PageItems sequence to know how to instantiate the RECORD-TYPE generic.

Moreover, even if we know how to instantiate the generic, @.items would has to be rebound to a different type object, which has descriptors pointing at nominal (non-generic) types. As you can see, even to explain the situation takes time and words. And that is not to mention that the explanation is somewhat incomplete yet as there are some hidden rocks in these waters.

How

Rewinding forward all the doubts, like not wanting to invest into the legacy compiler, and all the development fun (and not so fun too), let’s skip to the final state of affairs. Before anything else is being said, please, keep in mind that all this is still experimental. Not like something, to be covered with use experimental pragma, but like something that might eventually be torn up from Rakudo codebase for good. OK, let’s get down to what’s been introduced or changed.

Instantiation Protocol

The way generics get instantiated is starting to receive a shape as a standard. Parts of the protocol will be explained below, where they relate to.

Generic Classes

This change is conceptual: a class can now be generic; even if a class is not generic, an instance of it can be.

If the latter doesn’t make sense at first, consider, for example, a Scalar, where an instance of it can be .is_generic().

What does it mean for a class to be generic? From the class developer point of view – just about anything. From the point of view of Raku metaobject protocol it means that GenericClass.^archetypes.generic is true. How would class’ HOW know it? By querying the is-generic method of GenericClass:

role R[::T] {
	my class GenericClass {
		method is-generic { True }
	}
	say "Class is generic? ", ?GenericClass.^archetypes.generic;
}
constant RI = R[Int];

So far, so good, but what does role do up there? A generic is having very little sense outside of a generic context, I.e. in a lexical scope that doesn’t have access to any type capture. The body of R[::T] role does create such context. Without it we’d get a compile time warning:

Potential difficulties:
    Generic class 'GenericClass' declared outside of generic scope

An attempt to use the class would, for now, very likely end up with a cryptic error ‘Died with X::Method::NotFound’. It’s an LTA that is to be fixed, but for currently it’s a ‘Doctor, it hurts when I do this’ kind of situation. Apparently, having distinct candidates of is-generic for definite and undefined cases lets one to report different states for classes and their instances:

multi method is-generic(::?CLASS:U:) { False }
multi method is-generic(::?CLASS:D:) { self.it-depends }

• Note 1 Be careful with class composition times. An un-composed class doesn’t have access to its methods. At this stage MOP considers all classes as non-generics. Normally it has no side effects, but trait creators may be confused at times.
• Note 2 Querying archetypes of a class instance is likely to be much slower than querying the class itself. This is because instances are run-time things, whereas the class itself is mostly about compile-time. An instance can change its status as a result of successful instantiation of generics, for example.

Instantiation Method

OK, a class can now be a generic. How do we turn it into a non-generic? This would be a sole responsibility of INSTANTIATE-GENERIC method of the class. What the method does to achieve the goal and how it does it – we don’t care. Most typically one would need two candidates of the method: one for the class, one for an instance:

multi method INSTANTIATE-GENERIC(::?CLASS:U: $typeenv) {...}
multi method INSTANTIATE-GENERIC(::?CLASS:D: $typeenv) {...}

For example, instantiation of a collection object might look like:

multi method INSTANTIATE-GENERIC(::?CLASS:D: $typeenv) {
	::?CLASS.INSTANTIATE-GENERIC($typeenv).STORE(self)
}

Instantiation of a class… Let me put it this way: we don’t have a public API for this yet.

All this is currently a newly plowed field, a tabula rasa. It took me a few hours of trial-and-error before there was working code for JSON::Class that is capable of creating a copy of an existing generic class, which is actually subclassing the original generic but deals with instantiated descriptors.

Eventually, I extracted the results of the experiment into a Rakudo test, which is free of JSON::Class-specifics and serves as a reference code for this task. It’d be reasonable to have that test in the Roast, but it relies on Rakudo implementation of Raku MOP, where many things and protocols are not standardized yet.

What’s next? I’d look at where it all goes, what uses people may find for this new area. Everything may take an unexpected turn with RakuAST and macros in place. Either way, some discussion must take place first.

Type Environment

It used to be an internal thing of the MOP. But with introduction of the public instantiation protocol we need something public to pass around.

What is “type environment” in terms of Raku MOP? Simply, it’s a mapping of names of generic types into something (likely) nominal. For example, for any of R[::T] role declaration when it gets consumed by a class as, say, R[Str:D] the type environment created by role specializer would have a key "T" mapping into Str:D type object.

There is a problem with the internal type environment objects: they are meaningless for Raku code without use nqp pragma and without using corresponding NQP ops. Most common case is when the environment is a lexical context of role’s body closure.

A new TypeEnv class serves as an Associative container for the low-level type environments. As an Associative, it provides the standard (though rather basic) interface, identical to that of the Map class:

say $typeenv<T>.^name; # outputs 'Str:D' in the context of the R[Str:D] example above

The class is now supported by the MOP making it possible to even create own type environments:

my %typeenv is TypeEnv = :T1(Foo), :T2(Bar);
my \instantiation = GenericClass.^instantiate_generic(%typeenv);

Instantiation In Expressions

Let’s consider a simplified example:

role R[::T] {
    my class C {...} # Generic
	method give-it-to-me {
		C.new
	}
}

If no special care taken by the compiler, the method will try to create an instance of a generic class C. Instead, Rakudo is now pre-instantiating the class as soon as possible. In today’s situation, this happens when the role gets specialized as this is the earliest moment when type environment takes its final shape. The pre-instantiation is then referenced at run-time.

Are We There Yet?

Apparently, far from it. I’m not even sure if there is an end to this road.

When first changes to Rakudo code started showing promising results I created a draft PR for reviews and optimistically named it with something about “complete instantiation”. Very soon the name was changed to “moving close to complete instantiation”.

The most prominent missing part in this area for now is instantiation of generic parameters in signatures, and signatures themselves. Having all we already have, this part should be much easier to implement. Or may be not, considering the current signature binding implementation.

And then there is another case, which I spotted earlier, but forgot to leave a note to myself and can’t now remember what it was about.

Speaking of the future plans, I wouldn’t say better than one person formulated it once in the past:

So, let’s go straight to the…

Conclusion

Quoting another famous person:

Hope you like this present! I guess it might not be up to what you expected from it. But we can work together to move things forward. In the meantime I would have to unwind my stack of tasks back to the project, where all that started a while ago… Quite a while…

Have a merry happy Christmas everybody!