Aquileo | unified - learn

Aquileo | Publish a plugin

Mon, 19 Aug 2024 00:00:00 GMT

Publish a plugin

This guide shows how to share a plugin with the world.

Stuck? Have an idea for another guide? See support.md.

Intro

You may have a plugin locally, that you use in a project, and think others might find useful too.

Below you’ll find some rules on how to share plugins with the world.

Package name

The package name must be prefixed with rehype-, remark-, retext-, and so on. The name of the ecosystem. Use this prefix only for plugins. For presets, use rehype-preset-, remark-preset-, retext-preset-, and so on.

npm

Plugins should be published to npm. You are free to publish them elsewhere too.

You should use a keyword rehype-plugin, remark-plugin, retext-plugin, and so on, to tag your package.

GitHub

You may use GitHub. Or an alternative.

Only GitHub is crawled for the explore page. You should use a topic rehype-plugin, remark-plugin, retext-plugin, and so on, to tag your repository.

Export map

You must expose a plugin that can be imported from that package name. So, there must be a plugin at rehype-some-plugin.

There may be several plugins in a package. One example is rehype-mathjax, which has plugins exposed as rehype-mathjax/browser, rehype-mathjax/chtml, rehype-mathjax/svg, and rehype-mathjax (as an alias for rehype-mathjax/svg). The reason for different plugins in the same project is that they each use different code. Which impact performance and bundle size.

`export default`

Your package must expose a plugin at the default export. It must be a function that must work when passed as x to unified().use(x).

There may be different exports too. One example is remark-rehype, which also exposes useful things at defaultFootnoteBackContent, defaultFootnoteBackLabel, and defaultHandlers, but the plugin itself is exported as default.

Docs

Write a good readme. Explain what the plugin does, when it should be used or not, how to configure it.

Tests

Add thorough tests. Use a CI. You should test your project in all maintained versions of Node.js.

License

Add a license. It should be MIT or ISC, as that aligns with most of the JavaScript ecosystem. It must be an open source license.

Aquileo | Intro to syntax trees

Wed, 14 Aug 2024 00:00:00 GMT

Introduction to syntax trees

unified uses abstract syntax trees (abbreviated as ASTs), that plugins can work on. This guide introduces what ASTs are and how to work with them.

What is an AST?

An abstract syntax tree (AST) is a tree representation of the syntax of programming languages. For us that’s typically markup languages.

As a JavaScript developer you may already know things that are like ASTs: The DOM and React’s virtual DOM. Or you may have heard of Babel, ESLint, PostCSS, Prettier, or TypeScript. They all use ASTs to inspect and transform code.

In unified, we support several ASTs. The reason for different ASTs is that each markup language has several aspects that do not translate 1-to-1 to other markup languages. Taking markdown and HTML as an example, in some cases markdown has more info than HTML: markdown has several ways to add a link (“autolinks”: <https://url>, resource links: [label](url), and reference links with definitions: [label][id] and [id]: url). In other cases, HTML has more info than markdown. It has many tags, which add new meaning (semantics), that aren’t available in markdown. If there was one AST, it would be quite hard to do the tasks that several remark and rehype plugins now do.

See “How to build a syntax tree” for more info on how to make a tree. See “Syntax trees in TypeScript” on how to work with ASTs in TypeScript.

What is unist?

But all our ASTs have things in common. The bit in common is called unist. By having a shared interface, we can also share tools that work on all ASTs. In practice, that means you can use for example unist-util-visit to visit nodes in any supported AST.

See “Tree traversal” for more info on unist-util-visit.

unist is different from the ASTs used in other tools. Quite noticeable because it uses a particular set of names for things: type, children, position. But perhaps harder to see is that it’s compatible with JSON. It’s all objects and arrays. Strings, numbers. Where other tools use instances with methods, we use plain data. Years ago in retext we started out like that too. But we found that we preferred to be able to read and write a tree from/to a JSON file, to treat ASTs as data, and use more functional utilities.

When to use an AST?

You can use an AST when you want to inspect or transform content.

Say you wanted to count the number of headings in a markdown file. You could also do that with a regex:

const value = `# Pluto

Pluto is a dwarf planet in the Kuiper belt.

## History

### Discovery

In the 1840s, Urbain Le Verrier used Newtonian mechanics to predict the
position of…`

const expression = /^#+[^\r\n]+/gm
const headings = [...value.matchAll(expression)].length

console.log(headings) //=> 3

But what if the headings were in a code block? Or if Setext headings were used instead of ATX headings? The grammar of markdown is more complex than a regex can handle. That’s where an AST can help.

import {fromMarkdown} from 'mdast-util-from-markdown'
import {visit} from 'unist-util-visit'

const value = `# Pluto

Pluto is a dwarf planet in the Kuiper belt.

## History

### Discovery

In the 1840s, Urbain Le Verrier used Newtonian mechanics to predict the
position of…`

const tree = fromMarkdown(value)

let headings = 0

visit(tree, 'heading', function () {
  headings++
})

console.log(headings) //=> 3

See “Tree traversal” for more info on unist-util-visit.

Aquileo | Create a retext plugin

Tue, 13 Aug 2024 00:00:00 GMT

Create a retext plugin

This guide shows how to create a plugin for retext that checks the amount of spaces between sentences.

Stuck? Have an idea for another guide? See support.md.

Case

Before we start, let’s first outline what we want to make. Say we have the following text file:

One sentence. Two sentences.

One sentence.  Two sentences.

We want to get a warning for the second paragraph, saying that one space instead of two spaces should be used.

In the next step we’ll write the code to use our plugin.

Setting up

Let’s set up a project. Create a folder, example, enter it, and initialize a new project:

mkdir example
cd example
npm init -y

Then make sure the project is a module, so that import and export work, by changing package.json:

--- a/package.json
+++ b/package.json
@@ -1,6 +1,7 @@
 {
   "name": "example",
   "version": "1.0.0",
+  "type": "module",
   "main": "noindex.js",
   "scripts": {
     "test": "echo \"Error: no test specified\" && exit 1"

Make sure example.md exists with:

One sentence. Two sentences.

One sentence.  Two sentences.

Now, let’s create an example.js file that will process our text file and report any found problems.

import fs from 'node:fs/promises'
import {retext} from 'retext'
import {reporter} from 'vfile-reporter'
import retextSentenceSpacing from './plugin.js'

const document = await fs.readFile('example.md', 'utf8')

const file = await retext()
  .use(retextSentenceSpacing)
  .process(document)

console.error(reporter(file))

Don’t forget to npm install retext vfile-reporter!

If you read the guide on using unified, you’ll see some familiar statements. First, we load dependencies, then we read the file in. We process that file with the plugin we’ll create next and finally we report either a fatal error or any found linting messages.

Note that we directly depend on retext. This is a package that exposes a unified processor, and comes with the parser and compiler attached.

When running our example (it doesn’t work yet though) we want to see a message for the second paragraph, saying that one space instead of two spaces should be used.

Now we’ve got everything set up except for the plugin itself. We’ll do that in the next section.

Plugin

We’ll need a plugin, and for our case also a transform which will inspect. Let’s create them in our plugin file plugin.js:

/**
 * @import {Root} from 'nlcst'
 * @import {VFile} from 'vfile'
 */

export default function retextSentenceSpacing() {
  /**
   * @param {Root} tree
   * @param {VFile} file
   * @return {undefined}
   */
  return function (tree, file) {
  }
}

First things first, we need to check tree for a pattern. We can use a utility to help us to recursively walk our tree, namely unist-util-visit. Let’s add that.

--- a/plugin.js
+++ b/plugin.js
@@ -3,6 +3,8 @@
  * @import {VFile} from 'vfile'
  */

+import {visit} from 'unist-util-visit'
+
 export default function retextSentenceSpacing() {
   /**
    * @param {Root} tree
@@ -10,5 +12,8 @@ export default function retextSentenceSpacing() {
    * @return {undefined}
    */
   return function (tree, file) {
+    visit(tree, 'ParagraphNode', function (node) {
+      console.log(node)
+    })
   }
 }

Don’t forget to npm install unist-util-visit.

If we now run our example with Node.js, as follows, we’ll see that visitor is called with both paragraphs in our example:

node example.js

{
  type: 'ParagraphNode',
  children: [
    { type: 'SentenceNode', children: [Array], position: [Object] },
    { type: 'WhiteSpaceNode', value: ' ', position: [Object] },
    { type: 'SentenceNode', children: [Array], position: [Object] }
  ],
  position: {
    start: { line: 1, column: 1, offset: 0 },
    end: { line: 1, column: 29, offset: 28 }
  }
}
{
  type: 'ParagraphNode',
  children: [
    { type: 'SentenceNode', children: [Array], position: [Object] },
    { type: 'WhiteSpaceNode', value: '  ', position: [Object] },
    { type: 'SentenceNode', children: [Array], position: [Object] }
  ],
  position: {
    start: { line: 3, column: 1, offset: 30 },
    end: { line: 3, column: 30, offset: 59 }
  }
}
no issues found

This output already shows that paragraphs contain two types of nodes: SentenceNode and WhiteSpaceNode. The latter is what we want to check, but the former is important because we only warn about whitespace between sentences in this plugin (that could be another plugin though).

Let’s now loop through the children of each paragraph. Only checking whitespace between sentences. We use a small utility for checking node types: unist-util-is.

--- a/plugin.js
+++ b/plugin.js
@@ -13,7 +13,23 @@ export default function retextSentenceSpacing() {
    */
   return function (tree, file) {
     visit(tree, 'ParagraphNode', function (node) {
-      console.log(node)
+      let index = -1
+
+      while (++index < node.children.length) {
+        const previous = node.children[index - 1]
+        const child = node.children[index]
+        const next = node.children[index + 1]
+
+        if (
+          previous &&
+          next &&
+          previous.type === 'SentenceNode' &&
+          child.type === 'WhiteSpaceNode' &&
+          next.type === 'SentenceNode'
+        ) {
+          console.log(child)
+        }
+      }
     })
   }
 }

If we now run our example with Node, as follows, we’ll see that only whitespace between sentences is logged.

node example.js

{
  type: 'WhiteSpaceNode',
  value: ' ',
  position: {
    start: { line: 1, column: 14, offset: 13 },
    end: { line: 1, column: 15, offset: 14 }
  }
}
{
  type: 'WhiteSpaceNode',
  value: '  ',
  position: {
    start: { line: 3, column: 14, offset: 43 },
    end: { line: 3, column: 16, offset: 45 }
  }
}
no issues found

Finally, let’s add a warning for the second whitespace, as it has more characters than needed. We can use file.message() to associate a message with the file.

--- a/plugin.js
+++ b/plugin.js
@@ -25,9 +25,15 @@ export default function retextSentenceSpacing() {
           next &&
           previous.type === 'SentenceNode' &&
           child.type === 'WhiteSpaceNode' &&
-          next.type === 'SentenceNode'
+          next.type === 'SentenceNode' &&
+          child.value.length !== 1
         ) {
-          console.log(child)
+          file.message(
+            'Unexpected `' +
+              child.value.length +
+              '` spaces between sentences, expected `1` space',
+            child
+          )
         }
       }
     })

If we now run our example one final time, we’ll see a message for our problem!

$ node example.js
3:14-3:16 warning Unexpected `2` spaces between sentences, expected `1` space

⚠ 1 warning

Further exercises

One space between sentences isn’t for everyone. This plugin could receive the preferred amount of spaces instead of a hard-coded 1.

If you want to warn for tabs or newlines between sentences, maybe create a plugin for that too?

If you haven’t already, check out the other articles in the learn section!

Aquileo | Create a remark plugin

Tue, 13 Aug 2024 00:00:00 GMT

Create a remark plugin

This guide shows how to create a plugin for remark that turns emoji shortcodes (gemoji, such as :+1:) into Unicode emoji (👍). It looks for a regex in the text and replaces it.

Stuck? Have an idea for another guide? See support.md.

Case

Before we start, let’s first outline what we want to make. Say we have the following file:

Look, the moon :new_moon_with_face:

And we’d like to turn that into:

Look, the moon 🌚

In the next step we’ll write the code to use our plugin.

Setting up

Let’s set up a project. Create a folder, example, enter it, and initialize a new project:

mkdir example
cd example
npm init -y

Then make sure the project is a module, so that import and export work, by changing package.json:

--- a/package.json
+++ b/package.json
@@ -1,6 +1,7 @@
 {
   "name": "example",
   "version": "1.0.0",
+  "type": "module",
   "main": "noindex.js",
   "scripts": {
     "test": "echo \"Error: no test specified\" && exit 1"

Make sure input.md exists with:

Look, the moon :new_moon_with_face:

Now, let’s create an example.js file that will process our file and report any found problems.

import fs from 'node:fs/promises'
import {remark} from 'remark'
import remarkGemoji from './plugin.js'

const document = await fs.readFile('input.md', 'utf8')

const file = await remark().use(remarkGemoji).process(document)

await fs.writeFile('output.md', String(file))

Don’t forget to npm install remark!

Note that we directly depend on remark. This is a package that exposes a unified processor, and comes with the markdown parser and markdown compiler attached.

Now we’ve got everything set up except for the plugin itself. We’ll do that in the next section.

Plugin

We’ll need a plugin, and for our case also a transform. Let’s create them in our plugin file plugin.js:

/**
 * @import {Root} from 'mdast'
 */

/**
 * Turn gemoji shortcodes (`:+1:`) into emoji (`👍`).
 *
 * @returns
 *   Transform.
 */
export default function remarkGemoji() {
  /**
   * @param {Root} tree
   * @return {undefined}
   */
  return function (tree) {
  }
}

That’s how most plugins start. A function that returns another function.

For this use case, we could walk the tree and replace nodes with unist-util-visit, which is how many plugins work. But a different utility is even simpler: mdast-util-find-and-replace. It looks for a regex and lets you then replace that match.

Let’s add that.

--- a/plugin.js
+++ b/plugin.js
@@ -2,6 +2,8 @@
  * @import {Root} from 'mdast'
  */

+import {findAndReplace} from 'mdast-util-find-and-replace'
+
 /**
  * Turn gemoji shortcodes (`:+1:`) into emoji (`👍`).
  *
@@ -14,5 +16,16 @@ export default function remarkGemoji() {
    * @return {undefined}
    */
   return function (tree) {
+    findAndReplace(tree, [
+      /:(\+1|[-\w]+):/g,
+      /**
+       * @param {string} _
+       * @param {string} $1
+       * @return {undefined}
+       */
+      function (_, $1) {
+        console.log(arguments)
+      }
+    ])
   }
 }

Don’t forget to npm install mdast-util-find-and-replace!

If we now run our example with Node.js, we’ll see that console.log is called:

node example.js

[Arguments] {
  '0': ':new_moon_with_face:',
  '1': 'new_moon_with_face',
  '2': {
    index: 15,
    input: 'Look, the moon :new_moon_with_face:',
    stack: [ [Object], [Object], [Object] ]
  }
}

This output shows that the regular expression matches the emoji shortcode. The second argument is the name of the emoji. That’s what we want.

We can look that name up to find the corresponding Unicode emoji. We can use the gemoji package for that. It exposes a nameToEmoji record.

--- a/plugin.js
+++ b/plugin.js
@@ -2,6 +2,7 @@
  * @import {Root} from 'mdast'
  */

+import {nameToEmoji} from 'gemoji'
 import {findAndReplace} from 'mdast-util-find-and-replace'

 /**
@@ -21,10 +22,10 @@ export default function remarkGemoji() {
       /**
        * @param {string} _
        * @param {string} $1
-       * @return {undefined}
+       * @return {string | false}
        */
       function (_, $1) {
-        console.log(arguments)
+        return Object.hasOwn(nameToEmoji, $1) ? nameToEmoji[$1] : false
       }
     ])
   }

Don’t forget to npm install gemoji!

If we now run our example again with Node…

node example.js

…and open output.md, we’ll see that the shortcode is replaced with the emoji!

Look, the moon 🌚

That’s it! For a complete version of this plugin, see remark-gemoji.

If you haven’t already, check out the other articles in the learn section!

Aquileo | Create a rehype plugin

Tue, 13 Aug 2024 00:00:00 GMT

Create a rehype plugin

This guide shows how to create a plugin for rehype that adds id attributes to headings.

Stuck? Have an idea for another guide? See support.md.

Case

Before we start, let’s first outline what we want to make. Say we have the following file:

<h1>Solar System</h1>
<h2>Formation and evolution</h2>
<h2>Structure and composition</h2>
<h3>Orbits</h3>
<h3>Composition</h3>
<h3>Distances and scales</h3>
<h3>Interplanetary environment</h3>
<p>…</p>

And we’d like to turn that into:

<h1 id="solar-system">Solar System</h1>
<h2 id="formation-and-evolution">Formation and evolution</h2>
<h2 id="structure-and-composition">Structure and composition</h2>
<h3 id="orbits">Orbits</h3>
<h3 id="composition">Composition</h3>
<h3 id="distances-and-scales">Distances and scales</h3>
<h3 id="interplanetary-environment">Interplanetary environment</h3>
<p>…</p>

In the next step we’ll write the code to use our plugin.

Setting up

Let’s set up a project. Create a folder, example, enter it, and initialize a new project:

mkdir example
cd example
npm init -y

Then make sure the project is a module, so that import and export work, by changing package.json:

--- a/package.json
+++ b/package.json
@@ -1,6 +1,7 @@
 {
   "name": "example",
   "version": "1.0.0",
+  "type": "module",
   "main": "noindex.js",
   "scripts": {
     "test": "echo \"Error: no test specified\" && exit 1"

Make sure input.html exists with:

<h1>Solar System</h1>
<h2>Formation and evolution</h2>
<h2>Structure and composition</h2>
<h3>Orbits</h3>
<h3>Composition</h3>
<h3>Distances and scales</h3>
<h3>Interplanetary environment</h3>
<p>…</p>

Now, let’s create an example.js file that will process our file and report any found problems.

import fs from 'node:fs/promises'
import {rehype} from 'rehype'
import rehypeSlug from './plugin.js'

const document = await fs.readFile('input.html', 'utf8')

const file = await rehype()
  .data('settings', {fragment: true})
  .use(rehypeSlug)
  .process(document)

await fs.writeFile('output.html', String(file))

Don’t forget to npm install rehype!

Note that we directly depend on rehype. This is a package that exposes a unified processor, and comes with the HTML parser and HTML compiler attached.

Now we’ve got everything set up except for the plugin itself. We’ll do that in the next section.

Plugin

We’ll need a plugin and for our case also a transform. Let’s create them in our plugin file plugin.js:

/**
 * @import {Root} from 'hast'
 */

/**
 * Add `id`s to headings.
 *
 * @returns
 *   Transform.
 */
export default function rehypeSlug() {
  /**
   * @param {Root} tree
   * @return {undefined}
   */
  return function (tree) {
  }
}

That’s how most plugins start. A function that returns another function.

Next, for this use case, we can walk the tree and change nodes with unist-util-visit. That’s how many plugins work.

Let’s start there, to use unist-util-visit to look for headings:

--- a/plugin.js
+++ b/plugin.js
@@ -2,6 +2,8 @@
  * @import {Root} from 'hast'
  */

+import {visit} from 'unist-util-visit'
+
 /**
  * Add `id`s to headings.
  *
@@ -14,5 +16,17 @@ export default function rehypeSlug() {
    * @return {undefined}
    */
   return function (tree) {
+    visit(tree, 'element', function (node) {
+      if (
+        node.tagName === 'h1' ||
+        node.tagName === 'h2' ||
+        node.tagName === 'h3' ||
+        node.tagName === 'h4' ||
+        node.tagName === 'h5' ||
+        node.tagName === 'h6'
+      ) {
+        console.log(node)
+      }
+    })
   }
 }

Don’t forget to npm install unist-util-visit!

If we now run our example with Node.js, we’ll see that console.log is called:

node example.js

{
  type: 'element',
  tagName: 'h1',
  properties: {},
  children: [ { type: 'text', value: 'Solar System', position: [Object] } ],
  position: …
}
{
  type: 'element',
  tagName: 'h2',
  properties: {},
  children: [
    {
      type: 'text',
      value: 'Formation and evolution',
      position: [Object]
    }
  ],
  position: …
}
…

This output shows that we find our heading element. That’s what we want.

Next we want to get a string representation of what is inside the headings. There’s another utility for that: hast-util-to-string.

--- a/plugin.js
+++ b/plugin.js
@@ -2,6 +2,7 @@
  * @import {Root} from 'hast'
  */

+import {toString} from 'hast-util-to-string'
 import {visit} from 'unist-util-visit'

 /**
@@ -25,7 +26,8 @@ export default function rehypeSlug() {
         node.tagName === 'h5' ||
         node.tagName === 'h6'
       ) {
-        console.log(node)
+        const value = toString(node)
+        console.log(value)
       }
     })
   }

Don’t forget to npm install hast-util-to-string!

If we now run our example with Node.js, we’ll see the text printed:

node example.js

Solar System
Formation and evolution
Structure and composition
Orbits
Composition
Distances and scales
Interplanetary environment

Then we want to turn that text into slugs. You have many options here. For this case, we’ll use github-slugger.

--- a/plugin.js
+++ b/plugin.js
@@ -3,6 +3,7 @@
  */

 import {toString} from 'hast-util-to-string'
+import Slugger from 'github-slugger'
 import {visit} from 'unist-util-visit'

 /**
@@ -17,6 +18,8 @@ export default function rehypeSlug() {
    * @return {undefined}
    */
   return function (tree) {
+    const slugger = new Slugger()
+
     visit(tree, 'element', function (node) {
       if (
         node.tagName === 'h1' ||
@@ -27,7 +30,8 @@ export default function rehypeSlug() {
         node.tagName === 'h6'
       ) {
         const value = toString(node)
-        console.log(value)
+        const id = slugger.slug(value)
+        console.log(id)
       }
     })
   }

Don’t forget to npm install github-slugger!

The reason const slugger = new Slugger() is there, is because we want to create a new slugger for each document. If we’d create it outside of the function, we’d reuse the same slugger for each document, which would lead to slugs from different documents being mixed. That becomes a problem for documents with the same headings.

If we now run our example with Node.js, we’ll see the slugs printed:

node example.js

solar-system
formation-and-evolution
structure-and-composition
orbits
composition
distances-and-scales
interplanetary-environment

Finally, we want to add the id to the heading elements. This is also a good time to make sure we don’t overwrite existing ids.

--- a/plugin.js
+++ b/plugin.js
@@ -22,16 +22,17 @@ export default function rehypeSlug() {

     visit(tree, 'element', function (node) {
       if (
-        node.tagName === 'h1' ||
-        node.tagName === 'h2' ||
-        node.tagName === 'h3' ||
-        node.tagName === 'h4' ||
-        node.tagName === 'h5' ||
-        node.tagName === 'h6'
+        !node.properties.id &&
+        (node.tagName === 'h1' ||
+          node.tagName === 'h2' ||
+          node.tagName === 'h3' ||
+          node.tagName === 'h4' ||
+          node.tagName === 'h5' ||
+          node.tagName === 'h6')
       ) {
         const value = toString(node)
         const id = slugger.slug(value)
-        console.log(id)
+        node.properties.id = id
       }
     })
   }

If we now run our example again with Node…

node example.js

…and open output.html, we’ll see that the IDs are there!

<h1 id="solar-system">Solar System</h1>
<h2 id="formation-and-evolution">Formation and evolution</h2>
<h2 id="structure-and-composition">Structure and composition</h2>
<h3 id="orbits">Orbits</h3>
<h3 id="composition">Composition</h3>
<h3 id="distances-and-scales">Distances and scales</h3>
<h3 id="interplanetary-environment">Interplanetary environment</h3>
<p>…</p>

That’s it! For a complete version of this plugin, see rehype-slug.

If you haven’t already, check out the other articles in the learn section!

Aquileo | unified in the browser

Fri, 09 Aug 2024 00:00:00 GMT

unified in the browser

unified is many different projects that are maintained on GitHub and distributed mainly through npm. Almost all the projects can be used anywhere: in Bun, Deno, Node.js, on the edge, or in browsers. To use our projects in a browser, you need to do one or two things. And there’s different ways to go about it.

Bundle

A common way to use unified in the browser is to bundle it with a bundler. You perhaps know bundlers already: webpack, Rollup, or esbuild. You might be using one. Or otherwise have a favorite. If not, esbuild is a good choice.

Bundling is almost always a good idea. It gives end users a single file to download. Often minified.

Say we have some code using unified in example.js:

import rehypeStringify from 'rehype-stringify'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import {unified} from 'unified'

const file = await unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeStringify)
  .process('Hello, *world*!')

console.log(String(file))

And you want to use that in some HTML called index.html:

<!doctype html>
<meta charset=utf8>
<title>Example</title>
<script src=example.min.js type=module></script>

To make example.js work in the browser, you can bundle it with esbuild. First, set up a package. Go to the folder in your terminal and run:

npm init --yes
npm install esbuild --save-dev rehype-stringify remark-parse remark-rehype unified

Then, bundle example.js:

npx esbuild --bundle --format=esm --minify --outfile=example.min.js example.js

Now, open index.html in a browser. When you open the console of your developer tools, you should see Hello, world!

You probably also want to configure the target environment for the browsers that you support. That way, JavaScript syntax which is too new for some browsers, will be transformed into older JavaScript syntax that works. Pass the --target flag to do this.

CDN

If you don’t want to bundle unified yourself, you can use a CDN.

A CDN hosts files for you. And they can process them for you as well. The nice thing is that you do not have to install and bundle things yourself. The downside is that you’re dependent on a particular server that you do not control.

One such CDN is esm.sh. Like the code above, you can use it in a browser like this:

<!doctype html>
<meta charset=utf8>
<title>Example</title>
<script type=module>
  import rehypeStringify from 'https://esm.sh/rehype-stringify@10?bundle'
  import remarkParse from 'https://esm.sh/remark-parse@11?bundle'
  import remarkRehype from 'https://esm.sh/remark-rehype@11?bundle'
  import {unified} from 'https://esm.sh/unified@11?bundle'

  const file = await unified()
    .use(remarkParse)
    .use(remarkRehype)
    .use(rehypeStringify)
    .process('Hello, *world*!')

  console.log(String(file))
</script>

Aquileo | Using plugins

Thu, 08 Aug 2024 00:00:00 GMT

Using plugins

You can use plugins and presets to extend unified by calling use on a processor.

A small example is:

import rehypeDocument from 'rehype-document'
import rehypeFormat from 'rehype-format'
import rehypeStringify from 'rehype-stringify'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import {read} from 'to-vfile'
import {unified} from 'unified'

const file = await read('example.md')

await unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeDocument, {title: '👋🌍'})
  .use(rehypeFormat)
  .use(rehypeStringify)
  .process(file)

console.log(String(file))

This example shows how several plugins are used. It shows that the order in which plugins are used is important. And it shows that plugins can be configured by passing options to them. In this case, rehypeDocument receives a title field.

Using presets is similar:

import rehypeParse from 'rehype-parse'
import rehypePresetMinify from 'rehype-preset-minify'
import rehypeStringify from 'rehype-stringify'
import {read} from 'to-vfile'
import {unified} from 'unified'

const file = await read('example.html')

await unified()
  .use(rehypeParse)
  .use(rehypePresetMinify)
  .use(rehypeStringify)
  .process(file)

console.log(String(file))

Presets themselves cannot receive options. Sometimes, you still want to pass options to a particular plugin in a preset. To configure a plugin in a preset, use it after the preset, with the correct options:

import rehypeMinifyWhitespace from 'rehype-minify-whitespace'
import rehypeParse from 'rehype-parse'
import rehypePresetMinify from 'rehype-preset-minify'
import rehypeStringify from 'rehype-stringify'
import {read} from 'to-vfile'
import {unified} from 'unified'

const file = await read('example.html')

await unified()
  .use(rehypeParse)
  .use(rehypePresetMinify)
  .use(rehypeMinifyWhitespace, {newlines: true})
  .use(rehypeStringify)
  .process(file)

console.log(String(file))

Aquileo | HTML and remark

Tue, 09 Mar 2021 00:00:00 GMT

HTML and remark

remark is a markdown compiler. It’s focus is markdown. It’s concerned with HTML in two ways:

markdown is often turned into HTML
markdown sometimes has embedded HTML

When dealing with HTML and markdown, both remark and rehype are used. This article shows some examples of how to do that.

How to turn markdown into HTML

remark handles markdown: it can parse and serialize it. But it’s not for HTML. That’s what rehype does, which exists to parse and serialize HTML.

To turn markdown into HTML, we need remark-parse, remark-rehype, and rehype-stringify:

import rehypeStringify from 'rehype-stringify'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import {unified} from 'unified'

const file = await unified()
  .use(remarkParse) // Parse markdown content to a syntax tree
  .use(remarkRehype) // Turn markdown syntax tree to HTML syntax tree, ignoring embedded HTML
  .use(rehypeStringify) // Serialize HTML syntax tree
  .process('*emphasis* and **strong**')

console.log(String(file))

This turns *emphasis* and **strong** into emphasis and strong, but it does not support HTML embedded inside markdown (such as *emphasis* and strong).

This solution is safe: content you don’t trust cannot cause an XSS vulnerability.

How to turn HTML into markdown

We can also do the inverse. To turn HTML into markdown, we need rehype-parse, rehype-remark, and remark-stringify:

import rehypeParse from 'rehype-parse'
import rehypeRemark from 'rehype-remark'
import remarkStringify from 'remark-stringify'
import {unified} from 'unified'

const file = await unified()
  .use(rehypeParse) // Parse HTML to a syntax tree
  .use(rehypeRemark) // Turn HTML syntax tree to markdown syntax tree
  .use(remarkStringify) // Serialize HTML syntax tree
  .process('<em>emphasis</em> and <strong>strong</strong>')

console.log(String(file))

This turns emphasis and strong into *emphasis* and **strong**.

How to allow HTML embedded in markdown

Markdown is a content format that’s great for the more basic things: it’s nicer to write *emphasis* than emphasis. But, it’s limited: only a couple things are supported with its terse syntax. Luckily, for more complex things, markdown allows HTML inside it. A common example of this is to include a <details> element.

HTML embedded in markdown can be allowed when going from markdown to HTML by configuring remark-rehype and rehype-stringify:

import rehypeStringify from 'rehype-stringify'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import {unified} from 'unified'

const file = await unified()
  .use(remarkParse)
  .use(remarkRehype, {allowDangerousHtml: true}) // Pass raw HTML strings through.
  .use(rehypeStringify, {allowDangerousHtml: true}) // Serialize the raw HTML strings
  .process('*emphasis* and <strong>strong</strong>')

console.log(String(file))

This solution is not safe: content you don’t trust can cause XSS vulnerabilities.

How to properly support HTML inside markdown

To properly support HTML embedded inside markdown, we need another plugin: rehype-raw. This plugin will take the strings of HTML embedded in markdown and parse them with an actual HTML parser.

import rehypeRaw from 'rehype-raw'
import rehypeStringify from 'rehype-stringify'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import {unified} from 'unified'

const file = await unified()
  .use(remarkParse)
  .use(remarkRehype, {allowDangerousHtml: true})
  .use(rehypeRaw) // *Parse* the raw HTML strings embedded in the tree
  .use(rehypeStringify)
  .process('*emphasis* and <strong>strong</strong>')

console.log(String(file))

This solution is not safe: content you don’t trust can cause XSS vulnerabilities.

But because we now have a complete HTML syntax tree, we can sanitize that tree. For a safe solution, add rehype-sanitize right before rehype-stringify.

Aquileo | Support tables in remark

Wed, 24 Feb 2021 00:00:00 GMT

How to support tables in remark

Tables are a non-standard feature in markdown: they are not defined in CommonMark and will not work everywhere.

Tables are an extension that GitHub supports in their GFM. They work on github.com in most places: a readme, issue, PR, discussion, comment, etc.

remark and unified can support them through a plugin: remark-gfm.

What are tables?

Tables in markdown are used for tabular data and look like this:

| Beep |   No.  |   Boop |
| :--- | :----: | -----: |
| beep |  1024  |    xyz |
| boop | 338845 |    tuv |
| foo  |  10106 | qrstuv |
| bar  |   45   |   lmno |

The result on a website would look something like this:

Beep	No.	Boop
beep	1024	xyz
boop	338845	tuv
foo	10106	qrstuv
bar	45	lmno

How to write tables

Use pipe characters (|) between cells in a row. A new line starts a new row. You don’t have to align the pipes (|) to form a nice grid. But it does make the source more readable.

The first row is the table header and its cells are the labels for their respective column.

The second row is the alignment row and there must be as many cells in it as in the header row. Each “cell” must include a dash (-). A cell can be aligned left with a colon at the start (:-), aligned right with a colon at the end (-:), or aligned center with colons at the start and end (:-:). This alignment cell is used to align all corresponding cells in its column.

Further rows are the table body and are optional. Their cells are the table data.

How to support tables

As tables are non-standard, remark does not support them by default. But it can support them with a plugin: remark-gfm. Let’s say we have some markdown with a GFM table, in an example.md file:

# Table

| Branch  | Commit           |
| ------- | ---------------- |
| main    | 0123456789abcdef |
| staging | fedcba9876543210 |

And a module set up to transform markdown with tables to HTML, example.js:

import fs from 'node:fs/promises'
import rehypeStringify from 'rehype-stringify'
import remarkGfm from 'remark-gfm'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import {unified} from 'unified'

const document = await fs.readFile('example.md', 'utf8')

const file = await unified()
  .use(remarkParse) // Parse markdown.
  .use(remarkGfm) // Support GFM (tables, autolinks, tasklists, strikethrough).
  .use(remarkRehype) // Turn it into HTML.
  .use(rehypeStringify) // Serialize HTML.
  .process(document)

console.log(String(file))

Now, running node example.js yields:

<h1>Table</h1>
<table>
<thead>
<tr>
<th>Branch</th>
<th>Commit</th>
</tr>
</thead>
<tbody>
<tr>
<td>main</td>
<td>0123456789abcdef</td>
</tr>
<tr>
<td>staging</td>
<td>fedcba9876543210</td>
</tr>
</tbody>
</table>

How to support tables in `react-markdown`

As tables are non-standard, react-markdown does not support them by default. But it can support them with a plugin: remark-gfm.

Let’s say we have some markdown with a GFM table, in an example.md file:

/// <reference lib="dom" />
// ---cut---
import React from 'react'
import {createRoot} from 'react-dom/client'
import Markdown from 'react-markdown'
import remarkGfm from 'remark-gfm'

const markdown = `| Branch | Commit |
| - | - |
| main | 0123456789abcdef |`

createRoot(document.body).render(
  <Markdown remarkPlugins={[remarkGfm]}>{markdown}</Markdown>
)

Yields in JSX:

console.log(
  <>
    <h1>Table</h1>
    <table>
      <thead>
        <tr>
          <th>Branch</th>
          <th>Commit</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>main</td>
          <td>0123456789abcdef</td>
        </tr>
      </tbody>
    </table>
  </>
)

Aquileo | Remove a node

Mon, 15 Jun 2020 00:00:00 GMT

How to remove a node

Once you have found the node(s) you want to remove (see tree traversal), you can remove them.

Prerequisites

Tree traversal — intro on how to walk trees and find specific nodes with unist-util-visit

Removing a node

For the most part, removing nodes has to do with finding them first (see tree traversal), so let’s say we already have some code to find all emphasis nodes.

First, our example.md file:

Some text with *emphasis*.

Another paragraph with **importance** (and *more emphasis*).

And a module, example.js:

import fs from 'node:fs/promises'
import remarkParse from 'remark-parse'
import {unified} from 'unified'
import {visit} from 'unist-util-visit'

const document = await fs.readFile('example.md', 'utf8')

const tree = unified().use(remarkParse).parse(document)

visit(tree, 'emphasis', function (node) {
  console.log(node)
})

Now, running node example.js yields (ignoring positions for brevity):

{
  type: 'emphasis',
  children: [ { type: 'text', value: 'emphasis', position: [Object] } ]
}
{
  type: 'emphasis',
  children: [ { type: 'text', value: 'more emphasis', position: [Object] } ]
}

As the above log shows, nodes are objects. Each node is inside an array at the children property of another node. In other words, to remove a node, it must be removed from its parents children.

The problem then is to remove a value from an array. Standard JavaScript Array functions can be used: namely, splice.

We have the emphasis nodes, but we don’t have their parent, or the position in the parent’s children field they are in. Luckily, the function given to visit gets not only node, but also that index and parent:

--- a/example.js
+++ b/example.js
@@ -7,6 +7,6 @@ const document = await fs.readFile('example.md', 'utf8')

 const tree = unified().use(remarkParse).parse(document)

-visit(tree, 'emphasis', function (node) {
-  console.log(node)
+visit(tree, 'emphasis', function (node, index, parent) {
+  console.log(node.type, index, parent?.type)
 })

Yields:

emphasis 1 paragraph
emphasis 3 paragraph

parent is a reference to the parent of node, index is the position at which node is in parent’s children. With this information, and splice, we can now remove emphasis nodes:

--- a/example.js
+++ b/example.js
@@ -8,5 +8,9 @@ const document = await fs.readFile('example.md', 'utf8')
 const tree = unified().use(remarkParse).parse(document)

 visit(tree, 'emphasis', function (node, index, parent) {
-  console.log(node.type, index, parent?.type)
+  if (typeof index !== 'number' || !parent) return
+  // Note: this is buggy, see next section.
+  parent.children.splice(index, 1)
 })
+
+console.log(tree)

Yields:

{
  type: 'root',
  children: [
    {
      type: 'paragraph',
      children: [
        {type: 'text', value: 'Some text with '},
        {type: 'text', value: '.'}
      ]
    },
    {
      type: 'paragraph',
      children: [
        {type: 'text', value: 'Another paragraph with '},
        {type: 'strong', children: [Array]},
        {type: 'text', value: ' (and '},
        {type: 'text', value: ').'}
      ]
    }
  ]
}

This looks great, but beware of bugs. We are now changing the tree, while traversing it. That can cause bugs and performance problems.

When changing the tree, in most cases you should signal to visit how it should continue. More information on how to signal what to do next, is documented in unist-util-visit-parents.

In this case, we don’t want the removed node to be traversed (we want to skip it). And we want to continue with the node that is now at the position where our removed node was. To do that: return that information from visitor:

--- a/example.js
+++ b/example.js
@@ -1,7 +1,7 @@
 import fs from 'node:fs/promises'
 import remarkParse from 'remark-parse'
 import {unified} from 'unified'
-import {visit} from 'unist-util-visit'
+import {SKIP, visit} from 'unist-util-visit'

 const document = await fs.readFile('example.md', 'utf8')

@@ -9,8 +9,9 @@ const tree = unified().use(remarkParse).parse(document)

 visit(tree, 'emphasis', function (node, index, parent) {
   if (typeof index !== 'number' || !parent) return
-  // Note: this is buggy, see next section.
   parent.children.splice(index, 1)
+  // Do not traverse `node`, continue at the node *now* at `index`.
+  return [SKIP, index]
 })

 console.log(tree)

This yields the same output as before, but there’s no bug anymore. Nice, we can now remove nodes!

Replacing a node with its children

One more thing to make this example more useful: instead of dropping emphasis and its children, it might make more sense to replace the emphasis with its children.

To do that, we can do the following:

--- a/example.js
+++ b/example.js
@@ -9,7 +9,7 @@ const tree = unified().use(remarkParse).parse(document)

 visit(tree, 'emphasis', function (node, index, parent) {
   if (typeof index !== 'number' || !parent) return
-  parent.children.splice(index, 1)
+  parent.children.splice(index, 1, ...node.children)
   // Do not traverse `node`, continue at the node *now* at `index`.
   return [SKIP, index]
 })

Yields:

{
  type: 'root',
  children: [
    {
      type: 'paragraph',
      children: [
        {type: 'text', value: 'Some text with '},
        {type: 'text', value: 'emphasis'},
        {type: 'text', value: '.'}
      ]
    },
    {
      type: 'paragraph',
      children: [
        {type: 'text', value: 'Another paragraph with '},
        {type: 'strong', children: [Array]},
        {type: 'text', value: ' (and '},
        {type: 'text', value: 'more emphasis'},
        {type: 'text', value: ').'}
      ]
    }
  ]
}

Aquileo | unified - learn

Aquileo | Publish a plugin

Publish a plugin

Contents

Intro

Package name

npm

GitHub

Export map

export default

Docs

Tests

License

Aquileo | Intro to syntax trees

Introduction to syntax trees

Contents

What is an AST?

What is unist?

When to use an AST?

Aquileo | Create a retext plugin

Create a retext plugin

Contents

Case

Setting up

Plugin

Further exercises

Aquileo | Create a remark plugin

Create a remark plugin

Contents

Case

Setting up

Plugin

Aquileo | Create a rehype plugin

Create a rehype plugin

Contents

Case

Setting up

Plugin

Aquileo | unified in the browser

unified in the browser

Contents

Bundle

CDN

Aquileo | Using plugins

Using plugins

Aquileo | HTML and remark

HTML and remark

Contents

How to turn markdown into HTML

How to turn HTML into markdown

How to allow HTML embedded in markdown

How to properly support HTML inside markdown

Aquileo | Support tables in remark

How to support tables in remark

Contents

What are tables?

How to write tables

How to support tables

How to support tables in react-markdown

Aquileo | Remove a node

How to remove a node

Contents

Prerequisites

Removing a node

Replacing a node with its children

`export default`

How to support tables in `react-markdown`