Syncing

1. Watching files

We can use a .dir-locals.el file to add a on-save hook to all the files here. I just need to beware of subdirectories, as they don’t inherit ancestor dirlocals.

We want the save hook to trigger a script which generates .html exports (as needed), and then syncs with the server.

Emacs Lisp

;;; Directory Local Variables
;;; For more information see (info "(emacs) Directory Variables")

((org-mode . ((eval
               . (add-hook
                  'after-save-hook
                  (lambda ()
                    (when (and (bound-and-true-p +public-documents-sync)
                               (process-live-p +public-documents-sync))
                      (message "Killing old in-progress sync")
                      (kill-process +public-documents-sync))
                    (let ((default-directory "~/Documents/Public")
                          (out-buffer (get-buffer-create "*public-sync*")))
                      (with-current-buffer out-buffer
                        (with-silent-modifications
                          (erase-buffer))
                        (read-only-mode 1)
                        (ansi-color-for-comint-mode-on)
                        (comint-mode))
                      (setq +public-documents-sync
                            (start-process "public-sync" out-buffer "~/Documents/Public/sync"))
                      (set-process-filter +public-documents-sync #'comint-output-filter)
                      (set-process-sentinel
                       +public-documents-sync
                       (lambda (process signal)
                         (when (and (eq 'exit (process-status process))
                                    (= 0 (process-exit-status process)))
                           (message "Published public documents")
                           (setq +public-documents-sync nil))))))
                  nil t)))))

2. Clean

Sometimes clutter happens accidentally. In that case just delete .build/, re-trangle this file, then re-run sync.

3. Updating exports

We use a little shell hack to immediately call Emacs on the file. Because it’s called with start-process it should be non-blocking.

Emacs Lisp

#!/usr/bin/env sh
":"; exec emacs --quick --script "$0" -- "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-

3.1. Prevent recursion

We don’t want this called on files in .build, as that would make .build/.build, etc.

Emacs Lisp

(when (or (string-match-p "\\.build" default-directory)
          (not (string-match-p "/Documents/Public/" default-directory)))
  (kill-emacs 1))

3.2. The head cache file

For the sake of the pagelist, we record the hash of the heads of the org files to detect either new files, or a changed title/subtitle.

Emacs Lisp

(setq org-head-hash-file ".orgheadhash")

3.3. Initialisation

Before we do anything else, I will provide format-all because all it gives me is pain 😢.

Emacs Lisp

(defalias #'format-all-buffer #'ignore)
(defvar format-all--format-table (make-hash-table))
(provide 'format-all)

The we load my config to get all the nice stuff.

Emacs Lisp

(load "~/.config/emacs/early-init.el")
(require 'doom-start)
(require 'flycheck) ; To avoid issues that crop up with org-flycheck.
(defmacro flycheck-prepare-emacs-lisp-form (&rest _))

Let’s not bother with recentf

Emacs Lisp

(require 'recentf)
(recentf-mode -1)
(advice-add 'recentf-mode :override #'ignore)
(advice-add 'recentf-cleanup :override #'ignore)
(advice-add 'recentf-save-list :override #'ignore)

Before we get to anything, we want to complain if something goes wrong and avoid cluttering the directory.

Emacs Lisp

(setq make-backup-files nil
      debug-on-error t
      debugger-batch-max-lines 200)

To actually perform this exporting, we need some libraries.

Emacs Lisp

(require 'ox)
(require 'ox-html)
(require 'ox-latex)
(require 'engrave-faces-html)

When opening the files, if there are any file-local variables trust them. I trust myself after all 🙂.

Emacs Lisp

(setq enable-local-eval t
      enable-local-variables :all)

Now to stop annoying functionality.

Emacs Lisp

(defalias '+format-buffer-h #'ignore)

3.4. Setup build dir

We’re about to produce a bunch of files, but we don’t want to pollute the main directory. So, we can copy over the .org files to a hidden subdirectory and symlink everything else.

First up, we want to make sure the build dir exists

Emacs Lisp

(setq build-dir (expand-file-name ".build/" default-directory))

(unless (file-exists-p build-dir)
  (make-directory build-dir))

(message "\e[0;36m• Syncing .build directory\e[0m")

Then we see if any source files have been deleted while we weren’t looking, and if so ensure nether they or any other files with the same stem (to account for exports of Org files) are allowed to continue their existence in the .build dir.

Emacs Lisp

(setq deleted-files (cl-remove-if-not
                     (lambda (f)
                       (not (or (member (file-name-extension f) '("txt" "tex" "pdf" "html" "org.html" "version"))
                                (string= f "svg-inkscape")
                                (string= f org-head-hash-file)
                                (string-match-p "^_minted.*" f)
                                (string-match-p "^pagelist\\.org" f)
                                (string-match-p "^.orgheadhash" f)
                                (string-match-p "^\\." f))))
                     (cl-set-difference (directory-files build-dir)
                                        (directory-files default-directory)
                                        :test 'string=))
      keep-files '("org-style.css" "org-style.js" "org-autoupdate.js")
      deleted-file-bases (cl-remove-duplicates (mapcar #'file-name-base deleted-files)))

(dolist (file-or-dir (directory-files build-dir))
  (when (and (member (replace-regexp-in-string "\\..*" "" file-or-dir) deleted-file-bases)
             (not (member file-or-dir keep-files)))
    (message "  Removing %s" file-or-dir)
    (if (file-directory-p (expand-file-name file-or-dir build-dir))
        (delete-directory (expand-file-name file-or-dir build-dir) t)
      (delete-file (expand-file-name file-or-dir build-dir)))))

Copying over new .org files is actually a bit more complicated, as we want to resolve symlinks and carry over file modification dates of existing files. This can all be accomplished in a shell one-liner, so let’s do that.

We’ll still save a list of new files so we can determine if there’s been any change in file existence later.

Emacs Lisp

(setq new-files (cl-set-difference (directory-files default-directory nil "\\.org$")
                                   (directory-files build-dir nil "\\.org$")
                                   :test 'string=))

(shell-command-to-string (format "cp -Lpf *.org %s"
                                 (shell-quote-argument build-dir)))

Then, we make sure we’ve got symlinks to every other file, so they can be carried over by rsync.

Since we’re using syncthing, and it doesn’t support symlinks 🥲, we’ll copy files into the build directory and expand symlinks along the way.

Emacs Lisp

(dolist (file-or-dir (directory-files default-directory))
  (unless (or (string= (file-name-extension file-or-dir) "org")
              (member file-or-dir '(".build" ".dir-locals.el" "sync"))
              (string-match-p "^\\.*$" file-or-dir)
              (file-exists-p (expand-file-name file-or-dir build-dir)))
    (call-process "cp" nil nil nil "-Lpfr" file-or-dir (expand-file-name file-or-dir build-dir))))

Finally, we set the active directory to .build.

Emacs Lisp

(setq pub-dir default-directory
      default-directory build-dir)

3.5. Generating the page-list

Emacs Lisp

(setq old-hash
      (if (file-exists-p org-head-hash-file)
          (with-temp-buffer
            (insert-file-contents org-head-hash-file)
            (buffer-substring-no-properties (point-min) (point-max)))
        "n/a")
      current-hash
      (with-temp-buffer
        (dolist (file (directory-files default-directory nil "\\.org$"))
          (when (file-exists-p (file-truename file))
            (insert "==> " file "\n")
            (insert-file-contents (file-truename file))
            (while (re-search-forward "^\\=#\\+" nil t)
              (forward-line 1))
            (unless (eobp)
              (delete-region (point) (point-max)))))
        (sha1 (buffer-string))))

(write-region current-hash nil org-head-hash-file nil :silent)

(message "Comparing org head hashes\n Old hash: %s\n New hash: %s" old-hash current-hash)

(unless (and (file-exists-p "pagelist.org") (string= old-hash current-hash))
  (message "\e[0;36m• Generating page list\e[90m\e[0m")
  (write-region "" nil "pagelist.org")
  (with-temp-buffer
    (insert
     "#+title: Pagelist
#+subtitle: An enumeration of the contents of the folder
#+author: TEC
#+date: "
     (format-time-string "%F")
     "\n\n"
     (let ((org-files (nreverse (delete "pagelist.org" (directory-files default-directory nil "\\.org$"))))
           entries)
       (dolist (org-file org-files)
         (when (and (not (string-match-p "^_" org-file)) (file-exists-p (file-truename org-file)))
           (with-temp-buffer
             (insert-file-contents org-file)
             (let ((org-inhibit-startup t)
                   org-mode-hook)
               (org-mode))
             (let ((org-keywords (org-collect-keywords '("title" "subtitle" "date"))))
               (push (list :file org-file
                           :title (cadr (assoc "TITLE" org-keywords))
                           :subtitle (cadr (assoc "SUBTITLE" org-keywords))
                           :date (cadr (assoc "DATE" org-keywords)))
                     entries)))))
       (mapconcat (lambda (entry)
                    (format "+ [[file:%s][%s]] %s :: %s"
                            (plist-get entry :file)
                            (plist-get entry :title)
                            (if (plist-get entry :date)
                                (concat " (/" (plist-get entry :date) "/)") "")
                            (if (plist-get entry :subtitle)
                                (plist-get entry :subtitle) "")))
                  entries
                  "\n"))
     "\n")
    (let ((after-save-hook nil))
      (write-file "pagelist.org"))))

3.6. Export the Org files

Broken links sometimes happen, but we don’t want that to be a show-stopper.

Emacs Lisp

(setq org-export-with-broken-links t)

We also want to make sure we actually export all the linked resources — I trust myself.

Emacs Lisp

(setq org-resource-download-policy t)

Usually when there’s inline code, I want it to be shown.

Emacs Lisp

(setq org-babel-default-inline-header-args '((:session . "none") (:results . "replace") (:exports . "both") (:hlines . "yes")))

Instead of having the whole style inline, it would be nicer to copy the style files once and link to them using a CSS <link>.

Emacs Lisp

(setq org-html-style-default
      (with-temp-buffer
        (insert-file-contents
         (expand-file-name "misc/org-export-header.html" doom-private-dir))
        (insert
         "\n<link rel=\"stylesheet\" type=\"text/css\" href=\"org-style.css\">
<script src=\"org-style.js\" async></script>\n")
        (buffer-string))
      org-html-scripts "")

If the CSS or JS files in my config are newer, let’s copy the new versions over.

Emacs Lisp

(let ((css-src (expand-file-name "misc/org-css/main.css" doom-private-dir))
      (css-dest (expand-file-name "org-style.css" build-dir))
      (js-src (expand-file-name "misc/org-css/main.js" doom-private-dir))
      (js-dest (expand-file-name "org-style.js" build-dir)))
  (when (file-newer-than-file-p css-src css-dest)
    (copy-file css-src css-dest t))
  (when (file-newer-than-file-p js-src js-dest)
    (copy-file js-src js-dest t)))

To avoid unnecessary work, we can look for every .org file without a newer .html companion.

Emacs Lisp

(setq modified-org-files nil)

(let ((org-files (directory-files default-directory t "\\.org$")))
  (dolist (org-file org-files)
    (if (and (file-exists-p (concat org-file ".html"))
             (file-newer-than-file-p (concat org-file ".html") org-file))
        nil ;; (message "\e[0;33m• %s appears unchanged, skipping\e[90m" (file-name-nondirectory org-file))
      (push org-file modified-org-files))))

Now we just need to export each modified file.

Emacs Lisp

(dolist (org-file modified-org-files)
  (message "\e[0;34m• %s\e[90m" (file-relative-name org-file default-directory))
  (with-current-buffer (find-file org-file)
    (message "Opened %s" org-file)
    (let ((inhibit-message nil)
          (base-name (file-name-base org-file))
          (version-file (file-name-with-extension org-file "version"))
          version
          before-save-hook)
      (when (file-exists-p version-file)
        (with-temp-buffer
          (insert-file-contents version-file)
          (let ((version-string (string-trim (buffer-string))))
            (setq version (if (string-match-p "\\`[0-9]+\\'" version-string)
                              (1+ (string-to-number version-string))
                            0)))))
      (when (file-symlink-p (expand-file-name org-file pub-dir))
        (goto-char (point-min))
        (while (search-forward ":file \"figures/" nil t)
          (message "\e[0;36m:file figures -> figures/%s\e[90m" base-name)
          (replace-match (format ":file \"figures/%s/" base-name)))
        (goto-char (point-min))
        (while (search-forward "[file:figures/" nil t)
          (message "\e[0;36m[file:figures -> figures/%s\e[90m" base-name)
          (replace-match (format "[file:figures/%s/" base-name)))
        (goto-char (point-min))
        (while (search-forward "src=\"figures/" nil t)
          (message "\e[0;36msrc=figures -> figures/%s\e[90m" base-name)
          (replace-match (format "src=\"figures/%s/" base-name))))
      (condition-case err
          (progn
            (let ((org-inhibit-startup t))
              (org-mode))
            (unless (string= "syncing" (file-name-base org-file))
              (org-babel-tangle-file org-file))
            (org-ascii-export-to-ascii)
            (if (save-excursion
                  (goto-char (point-min))
                  (not (re-search-forward "^#\\+options:.* live-update:t" nil t)))
                (org-html-export-to-html)
              (let ((org-html-head-include-scripts t)
                    (org-html-scripts (format "<meta name=\"version\" content=\"%s\" />\n<script src=\"org-autoupdate.js\" async></script>\n" version)))
                (org-html-export-to-html)))
            (write-region (format "%s\n" version) nil version-file)
            (read-event nil nil 0.01)
            (let ((html-buf (engrave-faces-html-buffer-standalone)))
              (with-current-buffer html-buf
                (write-region nil nil (concat org-file ".html"))))
            (org-latex-export-to-pdf))
        (error (message "  \e[0;31m%s\e[90m"  (error-message-string err))))
      (kill-buffer (current-buffer)))))

For each modified file, we want to make sure a .versions file exists, and increment it. The writing is handled with (write-region (format "%s\n" version) nil version-file) , we just need to get the current version.

export/get-versionEmacs Lisp

(when (file-exists-p version-file)
  (with-temp-buffer
    (insert-file-contents version-file)
    (let ((version-string (string-trim (buffer-string))))
      (setq version (if (string-match-p "\\`[0-9]+\\'" version-string)
                        (1+ (string-to-number version-string))
                      0)))))

Before we export the files though, we need to update the figure paths based on the modified directory structure.

export/figure-renameEmacs Lisp

(when (file-symlink-p (expand-file-name org-file pub-dir))
  (goto-char (point-min))
  (while (search-forward ":file \"figures/" nil t)
    (message "\e[0;36m:file figures -> figures/%s\e[90m" base-name)
    (replace-match (format ":file \"figures/%s/" base-name)))
  (goto-char (point-min))
  (while (search-forward "[file:figures/" nil t)
    (message "\e[0;36m[file:figures -> figures/%s\e[90m" base-name)
    (replace-match (format "[file:figures/%s/" base-name)))
  (goto-char (point-min))
  (while (search-forward "src=\"figures/" nil t)
    (message "\e[0;36msrc=figures -> figures/%s\e[90m" base-name)
    (replace-match (format "src=\"figures/%s/" base-name))))

With the .version files, it’s possible to check them and automatically reload the page. This complicates the HTML export slightly. Should we want to include any extra content, we can shove it into the (initially empty) org-html-scripts variable.

export/htmlEmacs Lisp

(if (save-excursion
      (goto-char (point-min))
      (not (re-search-forward "^#\\+options:.* live-update:t" nil t)))
    (org-html-export-to-html)
  (let ((org-html-head-include-scripts t)
        (org-html-scripts (format "<meta name=\"version\" content=\"%s\" />\n<script src=\"org-autoupdate.js\" async></script>\n" version)))
    (org-html-export-to-html)))

3.7. Exit

I used to use rsync for this, but now it’s managed by syncthing 🙂, so all we need to do is announce we’re finished and exit.

Emacs Lisp

(message "\e[0;32m⯀ Done\e[0m")
(kill-emacs 0)

4. The auto-update script

When the live-update:t setting is given in #+options:, we want to intermittently update the loaded HTML with the latest version.

This can be done with a javascript program that:

Checks the .version file to look for a new version, every few seconds
Upon noticing a new version, downloads the new .html
Recursively modifies the current DOM to match the new file

We could use something like HTMX but … too late now.

Javascript

// [1/3] Version information

<<js/fetch-version>>

// [2/3] Updating the DOM

<<js/update-dom>>

// [3/3] Checking for updates

<<js/check-updates>>

4.1. Fetching the version

The document comes with a <meta name="version" content="N" /> tag, which makes it easy to get the current version.

Javascript

let currentVersion = document.querySelector('meta[name="version"]')?.content;

Thanks to the .version files, it’s really easy to check if a newer version of the document is available.

Javascript

function getVersionFileUrl() {
  const currentPath = window.location.href;
  return currentPath.substring(0, currentPath.lastIndexOf('.')) + '.version';
}

const versionFileUrl = getVersionFileUrl();

function fetchVersion() {
  return fetch(versionFileUrl, { cache: 'no-cache' })
    .then(response => response.text())
    .then(version => version.trim())
    .catch(error => console.error('Error fetching version:', error));
}

4.2. Updating the DOM

In order to minimise the number of DOM updates we make, we’ll need a way of comparing elements. A recursive match should work well here.

Javascript

function isMatchingElement(el1, el2) {
  if (el1.nodeName !== el2.nodeName) return false;
  if (el1.nodeType === Node.TEXT_NODE && el2.nodeType === Node.TEXT_NODE) {
    return el1.textContent.trim() === el2.textContent.trim();
  }
  if (el1.nodeType !== Node.ELEMENT_NODE || el2.nodeType !== Node.ELEMENT_NODE) return false;
  if (el1.attributes.length !== el2.attributes.length) return false;
  for (const attr of el1.attributes) {
    if (el1.getAttribute(attr.name) !== el2.getAttribute(attr.name)) {
      return false;
    }
  }
  if (el1.childNodes.length !== el2.childNodes.length) return false;
  return Array.from(el1.childNodes).every((child, index) => isMatchingElement(child, el2.childNodes[index]));
}

Using this element match function to identify common children will allow us to make more surgical DOM updates. A little helper function to identify a matching child would be helpful though.

Javascript

function findMatchingElement(parent, reference, startIndex) {
  for (let i = startIndex; i < parent.childNodes.length; i++) {
    const child = parent.childNodes[i];
    if (isMatchingElement(child, reference)) {
      return child;
    }
  }
  return null;
}

Now we can try to implement our main DOM update function. It should recursively, minimally, update elements. This is the most annoying bit to implement, it basically needs to:

Leave unchanged children alone
Insert new children, and remove deleted children
Also update element attributes

Javascript

function updateElement(current, updated) {
  if (current.nodeName !== updated.nodeName) {
    current.replaceWith(updated.cloneNode(true));
    return;
  }

  if (current.nodeType === Node.TEXT_NODE) {
    if (current.textContent !== updated.textContent) {
      current.textContent = updated.textContent;
    }
    return;
  }

  updateAttributes(current, updated);

  // Map to store the position of nodes in the updated children
  const updatedPositionMap = new Map(Array.from(updated.childNodes).map((node, index) => [node, index]));

  const currentChildren = Array.from(current.childNodes);
  let lastPlacedNodeIndex = -1;
  let nodesToRemove = new Set(currentChildren);

  let numAdded = 0, numRemoved = 0, numMoved = 0;

  currentChildren.forEach((child, index) => {
    const updatedChild = findMatchingElement(updated, child);

    if (updatedChild) {
      // Update the node if it matches one in the updated children
      const [childAdded, childRemoved, childMoved] = updateElement(child, updatedChild);
      [numAdded, numRemoved, numMoved] = [numAdded + childAdded, numRemoved + childRemoved, numMoved + childMoved];
      nodesToRemove.delete(child);
      // Determine the position where the node should be in the updated list
      const updatedPosition = updatedPositionMap.get(updatedChild);
      if (updatedPosition > lastPlacedNodeIndex) {
        // Node is in the correct sequence, no need to move
        lastPlacedNodeIndex = updatedPosition;
      } else {
        // Node needs to be moved to the correct position
        const referenceNode = updated.childNodes[updatedPosition + 1] || null;
        current.insertBefore(child, referenceNode);
        lastPlacedNodeIndex = updatedPosition;
        numMoved += 1;
      }
    }
  });

  // Remove nodes that are not found in the updated children
  nodesToRemove.forEach(node => current.removeChild(node));
  numRemoved = nodesToRemove.size

  // Insert any new nodes that did not exist in the current children
  updatedPositionMap.forEach((position, node) => {
    if (!current.contains(node)) {
      const referenceNode = current.childNodes[position] || null;
      current.insertBefore(node.cloneNode(true), referenceNode);
      numAdded += 1;
    }
  });
  return [numAdded, numRemoved, numMoved];
}

We referenced an updateAttribute function, but we need to actually implement it now.

Javascript

function updateAttributes(currentElement, newElement) {
  const currentAttrs = currentElement.attributes;
  const newAttrs = newElement.attributes;
  for (const attr of currentAttrs) {
    if (!newElement.hasAttribute(attr.name)) {
      currentElement.removeAttribute(attr.name);
    }
  }
  for (const attr of newAttrs) {
    if (currentElement.getAttribute(attr.name) !== attr.value) {
      currentElement.setAttribute(attr.name, attr.value);
    }
  }
}

4.3. Checking for updates

With the version-fetching and DOM-updating function we’ve implemented so far, actually checking for an update is as easy as:

Checking to see if .version is changed
Fetching the new HTML document if so
Updating the DOM to match the new HTML document

Javascript

function fetchAndUpdateContent(initialVersion) {
    fetch(window.location.href, { cache: 'no-cache' })
        .then(response => response.text())
        .then(newHtml => {
            const parser = new DOMParser();
            const newDoc = parser.parseFromString(newHtml, 'text/html');
            newVersion = newDoc.querySelector('meta[name="version"]')?.content;
            currentVersion = newVersion;
            document.title = newDoc.title;
            const [nodesAdded, nodesRemoved, nodesMoved] = updateElement(document.body, newDoc.body);
            if (typeof MathJax == "object") {
                MathJax.startup.defaultReady();
            }
            console.info(`Updated document from version ${initialVersion} → ${newVersion}: ${nodesAdded + nodesRemoved + nodesMoved} nodes changed.`);
        })
        .catch(error => console.error('Error fetching new HTML:', error));
}

function checkForUpdates() {
    fetchVersion().then(newVersion => {
        if (currentVersion && newVersion !== currentVersion) {
            fetchAndUpdateContent(currentVersion);
            currentVersion = newVersion; // Assume we'll update to the new version
        }
    });
}

Now we could just use a run-every-interval function to update the document, but I think it would be good to stop doing so when the page isn’t visible, so let’s get a little fancy and stop the fetching when that happens, and restart it when the page is visited again.

Javascript

let checkInterval;
function startChecking() {
  checkInterval = setInterval(checkForUpdates, 15000); // Check every 15 seconds
}

function stopChecking() {
  clearInterval(checkInterval);
}

function handleVisibilityChange() {
  if (document.visibilityState === "visible") {
    checkForUpdates(); // Perform an immediate check
    startChecking();   // Restart the checking interval
  } else {
    stopChecking();    // Stop checking while the page is not visible
  }
}

document.addEventListener("visibilitychange", handleVisibilityChange);

// Initial setup
if (document.visibilityState === "visible") {
  startChecking();   // Start checking if the page is initially visible
} else {
  stopChecking();    // Otherwise, don't start the interval
}

1. Watching files#

2. Clean#

3. Updating exports#

3.1. Prevent recursion#

3.2. The head cache file#

3.3. Initialisation#

3.4. Setup build dir#

3.5. Generating the page-list#

3.6. Export the Org files#

3.7. Exit#

4. The auto-update script#

4.1. Fetching the version#

4.2. Updating the DOM#

4.3. Checking for updates#

1. Watching files

2. Clean

3. Updating exports

3.1. Prevent recursion

3.2. The head cache file

3.3. Initialisation

3.4. Setup build dir

3.5. Generating the page-list

3.6. Export the Org files

3.7. Exit

4. The auto-update script

4.1. Fetching the version

4.2. Updating the DOM

4.3. Checking for updates