Just in time for Christmas
This project started with the modest aim of documenting parts of Rakudoc V1 (what used to be called POD6 that had been specified, but not included in the original documentation.
Except … some parts of the specification had not been implemented in the Pod::To::HTML renderer. And some parts were outdated. So a little bit of trimming was needed.
Also … a couple of extra bits could be added, and tables have always been a problem.
Since the next Raku language level is going to be based on RakuAST, and the parser was being refactored, why not look through the whole specification? The redesign was more extensive than originally planned.
We (Damian Conway, Elizabeth Mattijsen, Aliaksandr Zahatski, and I) started this project in August, opened the consultation to the community and hoped to finish in November. But in fact we have only just completed the specification- we hope – in time for Christmas!
A rendered version of the complete specification can be found on the new-raku deployment site. The rendering is not perfect because some of the components have yet to be implemented.
Documentation seems a theme this year, with Kay Rhodes discussing some views and mentioning the new design. I thought I’d take the conversation further.
I remember attending the London launch of a magazine called Personal Computer World. The magazine no longer exists, and the new products it discussed can now be found in museums. An engineer involved in the development of the first general purpose chip (Intel’s 8008) gave a talk about how the Altair computer came about. A comment of his that I remember to this day was that in time the cost of the hardware would fall to near zero, and that it would be the software which would generate more revenue. At the time, his prediction seemed ridiculous; now it seems obvious.
His logic was that because the chips were still being developed, they were all expensive. As the pace of development slowed, the low material cost of the hardware would dominate. But software requires innovation, human input, and would constantly evolve.
My thought is that a similar trend can now be seen in software development. As new applications were developed, code was expensive. New versions of Windows, for example, brought radical new innovations, but more recently, there is little radical or world changing in operating systems.
Compared to when I first started coding, it was fairly standard to write your own sort procedures and algorithms were fundamental to good programming. Now it is simply a waste of effort to re-invent the wheel if a standard library has been implemented. Coding costs will fall because most coding will be about stitching libraries together (I’m simplifying of course for rhetorical effect).
What will matter – I would argue – is how to use the software, and that means documentation. So my prediction is that the success (and ultimately the retail price) of hardware/software combinations will depend on the quality of the documentation.
Another consideration is accessibility. Coding has to the present been dominated by English-language software engineers. But a substantial population of the planet does not speak or read English as a first language.
My view is that documentation systems, such as product support websites, should be designed to incorporate other languages from the ground up. The current paradigm is create the website in English (sometimes Japanese or Chinese), then to translate pages and have an ad hoc language switcher.
This paradigm means that the state of the page (where the browser cursor is located) is not mapped between pages because translation is not a simple word mapping. But for this to be possible, the base language and the translated supporting texts need to be synchronised at a section level (eg. Japanese documents order material differently, so even a sentence level or paragraph level mapping is not idiomatic!).
In order to create documentation like this, the underlying tools have to be robust and customisable, and the text structures need to contain meta data. RakuDoc V1 already contained many of these qualities, but all good designs can be improved. I’ll discuss some of these enhancements below.
Documentation culture
It remains a major hassle for me as an active user of Raku that when I need to refer to a module that I want to use, even if it is installed on my local system, I have to go online. Not only that, but the main source of readily accessible information is the README.
Despite having a great documenting tool in RakuDoc V1, it is underused and documentation is difficult to access. The two are unsurprisingly related: why spend so much time writing documentation if its almost impossible to access the documentation? For my Raku::Pod::Render module, I have four major documentation files, and they are interlinked, but there is no way for them to be made accessible unless they are rendered into Markdown and in the root directory of the distribution.
Even though there is directory doc specified for the META6.json file, it (or rather its content files) are not accessible programmatically when zef installs the distribution. The upshot is that automatically finding the documentation files, even if they have been written, is not easy.
A documentation goal of mine in the new year is a
Collectionplugin so that a local website can be generated using the documentation files of all installed modules. The website will be similar to the Raku documentation suite, with the same search functionality.
Inline documentation already exists
When I began to use the Comma IDE, I noticed that hovering over variable names (and other names) pulled up information attached at declaration. This has led me to document all my variables.
Once there is a use for the documentation technique, in this case RakuDoc declarator blocks, I began to use them extensively. It is interesting how my documentation habits have changed since using Comma, which leverages RakuDoc’s instructions.
Automatic documentation tools
Even though I document most declared things, I miss separately documenting subs, methods, roles and so on. It would be useful to have a tool to do this. For example, given a distribution with a META6.json file, the documentation tool would go through each of the provides items, extract all sub/class/method/rule etc declaration and create a separate ‘DISTRIBUTION.rakudoc’ file under docs with a =head1 block and the declaration, and then a stub text.
By creating headings with the declaration keywords (eg. sub, method, role), the Collection plugin that generates the data for the search for the Raku documentation suite will add these headings to the search engine for the locally installed modules website.
These suggestions are not a part of the RakuDoc specification, but should form a part of our coding culture.
What has changed between RakuDoc V1 and V2?
In one respect, almost nothing has changed. Anything that has been written in RakuDoc V1 will be rendered in RakuDoc V2 (as opposed to what could theoretically have been written in V1). However, the specification and the documentation of RakuDoc are now merged into the same document.
The experience of rendering the whole of the Raku documentation suite, together with the experience of implementing RakuDoc without using Raku (the Podlite project), raised questions that led to re-evaluations and clarification.
There are some new features, such as a new table syntax, but the most important change is a clearer distinction between the various components of RakuDoc, and the different use-cases (more below).
Kay Rhodes commented in their article about the documenting of variables and subroutines. To be honest I really liked this feature when I was working with Octave. It was included from the start in RakuDoc V1 and, as I have indicated above, they are called declarator blocks. However, I am totally unsurprised that Kay did not refer to them as part of POD6 because the description of them and how they were to be used escaped me too. I even implemented them in Raku::Pod::Render without knowing what they were used for. It was only when I saw them in action with Comma that I finally grokked them.
In addition, RakuDoc V1 has the =finish and =data constructs which replace Perl‘s __DATA__ constructs. Rakudo even implemented these constructs, but they were never documented, so no one really knew.
By re-writing the documentation and the specification, we were able to clarify what had always been in RakuDoc V1. In a sense, what we were able to do was to subject the first specification to a good edit, and to go back to the author for clarification and a better understanding of the initial design.
It quickly became clear that RakuDoc has at least two perspectives. One to an IDE, with constructs such as declarator blocks, =finish and =data, and the other to a text oriented documentation renderer, with constructs such as =head, =item, =table, and so on.
The re-writing also made other things much clearer, and so more extensible. RakuDoc always had directives, blocks (sub-divided into built-in, semantic, and custom), format codes, and meta data, but the distinction between directives and blocks was not obvious. Format codes included text that was modified, but also included data that was transformed, such as links and aliases. So, we changed the name to markup codes and these became in-line containers to be filled with text, whilst blocks indicated text that was broken into paragraphs and could contain other blocks.
Meta data options are probably the most powerful aspect of RakuDoc. In V1 they appear only to change some aspects of the way a block is rendered. However, meta data act in the same way as parameters do for methods or functions: they provide data to the block handler.
Standard meta data options include :toc, which indicates whether a block is included in a Table of Contents, and :caption – the text to be included with the block, say a Table.
For the purposes of synchronisation, every block may have its own :anchor, and blocks can be linked into sections. This means that sections within a text can be linked to similar sections in other documents.
Seeing the forest from the trees
There is a difference between the documents that form the text form part of a suite [individual trees], and the collection of all the documents that constitute the whole suite [the forest].
Take for example the Raku documentation suite. There are several hundred individual documents, one for every Type, core module, fundamental concept, and others for tutorials and orientation guides.
The whole collection needs to be tied together, a search function added, index pages constructed, and so on. Originally the index pages for the website were crafted from HTML, and links to each of the individual pages were generated from meta data associated with each source document. At one point in time, the information for the ‘language’ page was even held in a YAML configuration file.
However, it became clear that by creating custom RakuDoc blocks, which were rendered using data collected from all the source files, all of the pages in the website could be created using RakuDoc.
Documentation as the killer application
All websites contain text dispersed with images and active components, such as buttons. Modern websites use a variety of Markdown for the text components. But Markdown is minimalist by design, and has had to be extended to meet the need for Tables of Content, web services and so on. Each extension is bespoke, so there is a Github Markdown, and a WordPress Markdown.
Markdown fails when it comes to associating meta data to blocks of text, or adding inline components with side effects (this tends to be where bespoke extensions take over).
RakuDoc, by contrast, has always had the possibility of meta data, and new inline components.
Complex documents tend to be created with custom formats, such as rtf, docx, or otf. The problem with these formats is that there was never an intention that documents would be created manually, the formatting was always to be done with word-processors.
RakuDoc comes in between complex machine readable formats, and overly simplistic markup languages.
The very close combination of Raku as the processing language and RakuDoc as the markup language could create a niche application for documentation, yet because documentation has so many uses … .
Conclusion
The revision of RakuDoc was not motivated by a desire to improve, so much as a desire to clarify. The driving motivation has been to create the base for a documentation system that can be multilingual and handle complex text.
As one of the authors of the new revision, I strongly believe we have a specification that can be used to create documentation systems, not just for Raku software, but also for a much wider problem domain.
Raku Advent Calendar 