#Performance

The reference book here is the Jennifer language documentation: 155 chapters, 1.7 MiB of Markdown, built into 155 pages, a 1335-section search index, and a 727-page PDF. Everything below was taken with scripts/bench.sh on bare metal - an 8-core, 16-thread Ryzen 7 5800X running Arch Linux - against Jennifer 0.24.0-dev+15. Each figure is the fastest of three runs.

#The site scales to about 4.5x, not to the core count

Total CPU barely moves as the jobs rise to the core count - 35 s at -j 1, 39 s at -j 4 - and climbs only past it, to 54 s at -j 16, where half the jobs are landing on SMT siblings rather than on cores. So the limit is not CPU. It is that chapter sizes span two orders of magnitude - 135 KiB against a 7 KiB median - and a chapter cannot be split: it is one parse and one render.

The same run on a 6-core, 12-thread Ryzen 5 7600X3D gives 1.82x at -j 2 and 3.01x at -j 4 - within a few percent of the curve above, on different silicon, with a different core count and a different clock. The shape belongs to the book, not to the machine.

Work is handed out heaviest-first, each chapter going to whichever worker is least loaded at that moment. That is the greedy longest-processing-time rule, and on this book it does noticeably better than the obvious alternatives:

At -j 8 that is an exactly even split - 213 KiB is the total divided by eight, so no arrangement does better. At -j 16 it is the size of the single largest chapter, which is the floor by definition. Past that, more jobs cannot help; the one 135 KiB chapter decides when the build ends.

Sorting first was tried and is worse at high job counts, which is the counter-intuitive part: it packs the small chapters neatly but leaves nothing left to balance the big ones against.

Memory is the other axis: the site peaks at 169 MB at -j 1 and 464 MB at -j 16, because every worker holds its own chapter.

#The PDF dominates, and it is serial

Where a grimoire pdf of this book goes:

Six sevenths of it is inside the modules, and single-threaded. The parse and the layout are the same size to within a rounding error; writing the file does not register beside them.

Grimoire starts the PDF at the same time as the chapter render, so --pdf costs roughly max(site, pdf) rather than their sum - 37.9 s against a 35.7 s PDF and a 21.5 s site. But the PDF is the floor, and it is why more jobs do not make build --pdf faster: at -j 16 it takes 39.2 s, over a second longer than at -j 1, the extra workers competing with the one task that decides when the build ends.

The parse is one call over the whole assembled book. It was one call per chapter until recently, with the block trees concatenated onto a single root - the same tree, and for a long time a much cheaper one, because markdown.parse handed the whole line list to a collector once per fenced block, quote, and list, so a book-sized document paid for that copy on every block. That was worth 2.25x at its worst and 1.14x as late as +13.

The read-only borrow removed the copy, and the workaround came out with it. One call is what the code always wanted to be, and it lowers the peak as well: the spliced version held every chapter's blocks while a fresh chapter tree was being built beside them.

The PDF holds the whole book as one document, and is where a build reaches its memory high-water mark - 1.6 GB on a book this size, growing with it.

Re-derive the split at any time with the profiler:

jennifer profile bin/grimoire pdf --src <your-book> --out /tmp/pdfout

#What these numbers do not include

A benchmark file is a measurement of one commit against one interpreter, and both move. Two things about the code as it stands make a build faster or smaller than the tables say, and neither is measured here:

The build-time highlighter writes its prompt test and its identifier scan out inside render, so a book with [highlight] on pays no helper call per character of a code block. How much that is worth depends on the interpreter's parameter borrow, which collects most of the same cost.

The PDF is parsed as one document rather than a chapter at a time. The split between the first two rows of the phase table and the total disagree slightly for that reason, and peak memory is lower than the 1.6 GB shown.

Re-run scripts/bench.sh when the figures matter to a decision: it is the only way to have numbers for the machine, the book and the interpreter in front of you.

#Taking these numbers on your own machine

scripts/bench.sh runs the whole matrix - startup, the site at each --jobs setting, the PDF alone, both together, a profiler pass, and the markdown and pdf modules on their own - and writes one tab-separated file:

scripts/bench.sh ~/src/your-book          # the full matrix, tens of minutes
scripts/bench.sh --quick ~/src/your-book  # one repetition, two job settings

Nothing is written inside the book; every build goes to a temporary directory. The file records the machine as well as the timings, because a wall-clock figure without the CPU, the filesystem, and the interpreter version beside it cannot be compared with anything - and a figure from a virtual machine cannot be compared with one from the metal under it at all.

#Practical notes

• --jobs 0 (the default) is one task per CPU and is almost always the right answer. Raising it past the point above buys nothing; lowering it to 1 is useful for reading --verbose output in outline order.
• --no-search saves nothing measurable. Indexing rides along with the render that has already parsed the chapter.
• grimoire pdf skips the site entirely when the PDF is all you want, but since the two overlap, build --pdf costs about 6% more and gives you both.
• Output is deterministic at any job count, so a build is reproducible and diffable regardless of how the work was split.