Syncing
How I manage my frequently-synced public documents
1. Watching files
We can use a .dir-locals.el file to add a on-save hook to all the files here. I just need to beware of subdirectories, as they don't inherit ancestor dirlocals.
We want the save hook to trigger a script which generates .html exports (as needed), and then syncs with the server.
Emacs Lisp
;;; Directory Local Variables
;;; For more information see (info "(emacs) Directory Variables")
((org-mode . ((eval
. (add-hook
'after-save-hook
(lambda ()
(when (and (bound-and-true-p +public-documents-sync)
(process-live-p +public-documents-sync))
(message "Killing old in-progress sync")
(kill-process +public-documents-sync))
(let ((default-directory "~/Documents/Public")
(out-buffer (get-buffer-create "*public-sync*")))
(with-current-buffer out-buffer
(with-silent-modifications
(erase-buffer))
(read-only-mode 1)
(ansi-color-for-comint-mode-on)
(comint-mode))
(setq +public-documents-sync
(start-process "public-sync" out-buffer "~/Documents/Public/sync"))
(set-process-filter +public-documents-sync #'comint-output-filter)
(set-process-sentinel
+public-documents-sync
(lambda (process signal)
(when (and (eq 'exit (process-status process))
(= 0 (process-exit-status process)))
(message "Published public documents")
(setq +public-documents-sync nil))))))
nil t)))))
2. Clean
Sometimes clutter happens accidentally. In that case just delete .build/,
re-trangle this file, then re-run sync.
3. Updating exports
We use a little shell hack to immediately call Emacs on the file. Because it's
called with start-process it should be non-blocking.
Emacs Lisp
#!/usr/bin/env sh ":"; exec emacs --quick --script "$0" -- "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-
3.1. Prevent recursion
We don't want this called on files in .build, as that would make .build/.build, etc.
Emacs Lisp
(when (or (string-match-p "\\.build" default-directory)
(not (string-match-p "/Documents/Public/" default-directory)))
(kill-emacs 1))
3.2. The head cache file
For the sake of the pagelist, we record the hash of the heads of the org files to detect either new files, or a changed title/subtitle.
3.3. New initialisation
Emacs Lisp
(setq gc-cons-threshold 16777216
gcmh-high-cons-threshold 16777216)
(load "~/.config/emacs/lisp/doom.el")
(require 'doom-cli)
(doom-initialize-packages)
(setq subconf-root (expand-file-name "subconf" doom-user-dir))
(add-to-list 'load-path subconf-root)
Emacs Lisp
;; For some reason, these seem to behave a bit strangely.
(add-load-path! "~/.config/emacs/.local/straight/repos/parent-mode/"
"~/.config/emacs/.local/straight/repos/highlight-quoted/")
Emacs Lisp
(require 'engrave-faces)
(load "~/.config/doom/misc/config-publishing/doom-one-light-engraved-theme.el")
(require 'config-ox-html)
(require 'config-ox-latex)
(require 'vc) ; For the {{vc-time}} macro
;; For some face engraving
(require 'highlight-numbers)
(require 'highlight-quoted)
(require 'rainbow-delimiters)
Before we get to anything, we want to complain if something goes wrong and avoid cluttering the directory.
When opening the files, if there are any file-local variables trust them. I trust myself after all 🙂.
3.4. Initialisation
3.5. Setup build dir
We're about to produce a bunch of files, but we don't want to pollute the main directory. So, we can copy over the .org files to a hidden subdirectory and symlink everything else.
First up, we want to make sure the build dir exists
Emacs Lisp
(setq build-dir (expand-file-name ".build/" default-directory)) (unless (file-exists-p build-dir) (make-directory build-dir)) (message "\e[0;36m• Syncing .build directory\e[0m")
Then we see if any source files have been deleted while we weren't looking, and if so ensure nether they or any other files with the same stem (to account for exports of Org files) are allowed to continue their existence in the .build dir.
Emacs Lisp
(setq deleted-files (cl-remove-if-not
(lambda (f)
(not (or (member (file-name-extension f) '("txt" "tex" "pdf" "html" "org.html" "version"))
(string= f "svg-inkscape")
(string= f org-head-hash-file)
(string-match-p "^_minted.*" f)
(string-match-p "^pagelist\\.org" f)
(string-match-p "^.orgheadhash" f)
(string-match-p "^\\." f))))
(cl-set-difference (directory-files build-dir)
(directory-files default-directory)
:test 'string=))
keep-files '("org-style.css" "org-style.js" "org-autoupdate.js")
deleted-file-bases (cl-remove-duplicates (mapcar #'file-name-base deleted-files)))
(dolist (file-or-dir (directory-files build-dir))
(when (and (member (replace-regexp-in-string "\\..*" "" file-or-dir) deleted-file-bases)
(not (member file-or-dir keep-files)))
(message " Removing %s" file-or-dir)
(if (file-directory-p (expand-file-name file-or-dir build-dir))
(delete-directory (expand-file-name file-or-dir build-dir) t)
(delete-file (expand-file-name file-or-dir build-dir)))))
Copying over new .org files is actually a bit more complicated, as we want to resolve symlinks and carry over file modification dates of existing files. This can all be accomplished in a shell one-liner, so let's do that.
We'll still save a list of new files so we can determine if there's been any change in file existence later.
Emacs Lisp
(setq new-files (cl-set-difference (directory-files default-directory nil "\\.org$")
(directory-files build-dir nil "\\.org$")
:test 'string=))
(shell-command-to-string (format "cp -Lpf *.org %s"
(shell-quote-argument build-dir)))
Then, we make sure we've got symlinks to every other file, so they can be
carried over by rsync.
Since we're using syncthing, and it doesn't support symlinks 🥲, we'll copy files into the build directory and expand symlinks along the way.
Emacs Lisp
(dolist (file-or-dir (directory-files default-directory))
(unless (or (string= (file-name-extension file-or-dir) "org")
(member file-or-dir '(".build" ".dir-locals.el" "sync"))
(string-match-p "^\\.*$" file-or-dir)
(file-exists-p (expand-file-name file-or-dir build-dir)))
(call-process "cp" nil nil nil "-Lpfr" file-or-dir (expand-file-name file-or-dir build-dir))))
Finally, we set the active directory to .build.
3.6. Generating the page-list
Emacs Lisp
(setq old-hash
(if (file-exists-p org-head-hash-file)
(with-temp-buffer
(insert-file-contents org-head-hash-file)
(buffer-substring-no-properties (point-min) (point-max)))
"n/a")
current-hash
(with-temp-buffer
(dolist (file (directory-files default-directory nil "\\.org$"))
(when (file-exists-p (file-truename file))
(insert "==> " file "\n")
(insert-file-contents (file-truename file))
(while (re-search-forward "^\\=#\\+" nil t)
(forward-line 1))
(unless (eobp)
(delete-region (point) (point-max)))))
(sha1 (buffer-string))))
(write-region current-hash nil org-head-hash-file nil :silent)
(message "Comparing org head hashes\n Old hash: %s\n New hash: %s" old-hash current-hash)
(unless (and (file-exists-p "pagelist.org") (string= old-hash current-hash))
(message "\e[0;36m• Generating page list\e[90m\e[0m")
(write-region "" nil "pagelist.org")
(with-temp-buffer
(insert
"#+title: Pagelist
#+subtitle: An enumeration of the contents of the folder
#+author: TEC
#+date: "
(format-time-string "%F")
"\n\n"
(let ((org-files (nreverse (delete "pagelist.org" (directory-files default-directory nil "\\.org$"))))
entries)
(dolist (org-file org-files)
(when (and (not (string-match-p "^_" org-file)) (file-exists-p (file-truename org-file)))
(with-temp-buffer
(insert-file-contents org-file)
(let ((org-inhibit-startup t)
org-mode-hook)
(org-mode))
(let ((org-keywords (org-collect-keywords '("title" "subtitle" "date"))))
(push (list :file org-file
:title (cadr (assoc "TITLE" org-keywords))
:subtitle (cadr (assoc "SUBTITLE" org-keywords))
:date (cadr (assoc "DATE" org-keywords)))
entries)))))
(mapconcat (lambda (entry)
(format "+ [[file:%s][%s]] %s :: %s"
(plist-get entry :file)
(plist-get entry :title)
(if (plist-get entry :date)
(concat " (/" (plist-get entry :date) "/)") "")
(if (plist-get entry :subtitle)
(plist-get entry :subtitle) "")))
entries
"\n"))
"\n")
(let ((after-save-hook nil))
(write-file "pagelist.org"))))
3.7. Export the Org files
Broken links sometimes happen, but we don't want that to be a show-stopper.
We also want to make sure we actually export all the linked resources — I trust myself.
Usually when there's inline code, I want it to be shown.
Emacs Lisp
(setq org-babel-default-inline-header-args '((:session . "none") (:results . "replace") (:exports . "both") (:hlines . "yes")))
Instead of having the whole style inline, it would be nicer to copy the style files once and link to them using a CSS <link>.
Emacs Lisp
(setq org-html-style-default
(with-temp-buffer
(insert-file-contents
(expand-file-name "misc/org-export-header.html" doom-private-dir))
(insert
"\n<link rel=\"stylesheet\" type=\"text/css\" href=\"org-style.css\">
<script src=\"org-style.js\" async></script>\n")
(buffer-string))
org-html-scripts "")
If the CSS or JS files in my config are newer, let's copy the new versions over.
Emacs Lisp
(let ((css-src (expand-file-name "misc/org-css/main.css" doom-private-dir))
(css-dest (expand-file-name "org-style.css" build-dir))
(js-src (expand-file-name "misc/org-css/main.js" doom-private-dir))
(js-dest (expand-file-name "org-style.js" build-dir)))
(when (file-newer-than-file-p css-src css-dest)
(copy-file css-src css-dest t))
(when (file-newer-than-file-p js-src js-dest)
(copy-file js-src js-dest t)))
To avoid unnecessary work, we can look for every .org file without a newer .html companion.
Emacs Lisp
(setq modified-org-files nil)
(let ((org-files (directory-files default-directory t "\\.org$")))
(dolist (org-file org-files)
(if (and (file-exists-p (concat org-file ".html"))
(file-newer-than-file-p (concat org-file ".html") org-file))
nil ;; (message "\e[0;33m• %s appears unchanged, skipping\e[90m" (file-name-nondirectory org-file))
(push org-file modified-org-files))))
Now we just need to export each modified file.
Emacs Lisp
(dolist (org-file modified-org-files)
(message "\e[0;34m• %s\e[90m" (file-relative-name org-file default-directory))
(with-current-buffer (find-file org-file)
(message "Opened %s" org-file)
(let ((inhibit-message nil)
(base-name (file-name-base org-file))
(version-file (file-name-with-extension org-file "version"))
version
before-save-hook)
(when (file-exists-p version-file)
(with-temp-buffer
(insert-file-contents version-file)
(let ((version-string (string-trim (buffer-string))))
(setq version (if (string-match-p "\\`[0-9]+\\'" version-string)
(1+ (string-to-number version-string))
0)))))
(when (file-symlink-p (expand-file-name org-file pub-dir))
(goto-char (point-min))
(while (search-forward ":file \"figures/" nil t)
(message "\e[0;36m:file figures -> figures/%s\e[90m" base-name)
(replace-match (format ":file \"figures/%s/" base-name)))
(goto-char (point-min))
(while (search-forward "[file:figures/" nil t)
(message "\e[0;36m[file:figures -> figures/%s\e[90m" base-name)
(replace-match (format "[file:figures/%s/" base-name)))
(goto-char (point-min))
(while (search-forward "src=\"figures/" nil t)
(message "\e[0;36msrc=figures -> figures/%s\e[90m" base-name)
(replace-match (format "src=\"figures/%s/" base-name))))
(condition-case err
(progn
(let ((org-inhibit-startup t))
(org-mode))
(unless (string= "syncing" (file-name-base org-file))
(org-babel-tangle-file org-file))
(org-ascii-export-to-ascii)
(if (save-excursion
(goto-char (point-min))
(not (re-search-forward "^#\\+options:.* live-update:t" nil t)))
(org-html-export-to-html)
(let ((org-html-head-include-scripts t)
(org-html-scripts (format "<meta name=\"version\" content=\"%s\" />\n<script src=\"org-autoupdate.js\" async></script>\n" version)))
(org-html-export-to-html)))
(write-region (format "%s\n" version) nil version-file)
(read-event nil nil 0.01)
(let ((html-buf (engrave-faces-html-buffer-standalone)))
(with-current-buffer html-buf
(write-region nil nil (concat org-file ".html"))))
(org-latex-export-to-pdf))
(error (message " \e[0;31m%s\e[90m" (error-message-string err))))
(kill-buffer (current-buffer)))))
For each modified file, we want to make sure a .versions file exists, and
increment it. The writing is handled with (write-region (format "%s\n" version) nil version-file), we just need to get the current version.
export/get-versionEmacs Lisp
(when (file-exists-p version-file)
(with-temp-buffer
(insert-file-contents version-file)
(let ((version-string (string-trim (buffer-string))))
(setq version (if (string-match-p "\\`[0-9]+\\'" version-string)
(1+ (string-to-number version-string))
0)))))
Before we export the files though, we need to update the figure paths based on the modified directory structure.
export/figure-renameEmacs Lisp
(when (file-symlink-p (expand-file-name org-file pub-dir))
(goto-char (point-min))
(while (search-forward ":file \"figures/" nil t)
(message "\e[0;36m:file figures -> figures/%s\e[90m" base-name)
(replace-match (format ":file \"figures/%s/" base-name)))
(goto-char (point-min))
(while (search-forward "[file:figures/" nil t)
(message "\e[0;36m[file:figures -> figures/%s\e[90m" base-name)
(replace-match (format "[file:figures/%s/" base-name)))
(goto-char (point-min))
(while (search-forward "src=\"figures/" nil t)
(message "\e[0;36msrc=figures -> figures/%s\e[90m" base-name)
(replace-match (format "src=\"figures/%s/" base-name))))
With the .version files, it's possible to check them and automatically reload
the page. This complicates the HTML export slightly. Should we want to include
any extra content, we can shove it into the (initially empty) org-html-scripts variable.
export/htmlEmacs Lisp
(if (save-excursion
(goto-char (point-min))
(not (re-search-forward "^#\\+options:.* live-update:t" nil t)))
(org-html-export-to-html)
(let ((org-html-head-include-scripts t)
(org-html-scripts (format "<meta name=\"version\" content=\"%s\" />\n<script src=\"org-autoupdate.js\" async></script>\n" version)))
(org-html-export-to-html)))
3.8. Exit
I used to use rsync for this, but now it's managed by syncthing 🙂, so all we need to do is announce we're finished and exit.
4. The auto-update script
When the live-update:t setting is given in #+options:, we want to intermittently update the loaded HTML with the latest version.
This can be done with a javascript program that:
- Checks the .version file to look for a new version, every few seconds
- Upon noticing a new version, downloads the new .html
- Recursively modifies the current DOM to match the new file
We could use something like HTMX but … too late now.
Javascript
// [1/3] Version information <<js/fetch-version>> // [2/3] Updating the DOM <<js/update-dom>> // [3/3] Checking for updates <<js/check-updates>>
4.1. Fetching the version
The document comes with a <meta name="version" content="N" /> tag,
which makes it easy to get the current version.
Thanks to the .version files, it's really easy to check if a newer version of the document is available.
Javascript
function getVersionFileUrl() {
const currentPath = window.location.href;
return currentPath.substring(0, currentPath.lastIndexOf('.')) + '.version';
}
const versionFileUrl = getVersionFileUrl();
function fetchVersion() {
return fetch(versionFileUrl, { cache: 'no-cache' })
.then(response => response.text())
.then(version => version.trim())
.catch(error => console.error('Error fetching version:', error));
}
4.2. Updating the DOM
In order to minimise the number of DOM updates we make, we'll need a way of comparing elements. A recursive match should work well here.
Javascript
function isMatchingElement(el1, el2) {
if (el1.nodeName !== el2.nodeName) return false;
if (el1.nodeType === Node.TEXT_NODE && el2.nodeType === Node.TEXT_NODE) {
return el1.textContent.trim() === el2.textContent.trim();
}
if (el1.nodeType !== Node.ELEMENT_NODE || el2.nodeType !== Node.ELEMENT_NODE) return false;
if (el1.attributes.length !== el2.attributes.length) return false;
for (const attr of el1.attributes) {
if (el1.getAttribute(attr.name) !== el2.getAttribute(attr.name)) {
return false;
}
}
if (el1.childNodes.length !== el2.childNodes.length) return false;
return Array.from(el1.childNodes).every((child, index) => isMatchingElement(child, el2.childNodes[index]));
}
Using this element match function to identify common children will allow us to make more surgical DOM updates. A little helper function to identify a matching child would be helpful though.
Javascript
function findMatchingElement(parent, reference, startIndex) {
for (let i = startIndex; i < parent.childNodes.length; i++) {
const child = parent.childNodes[i];
if (isMatchingElement(child, reference)) {
return child;
}
}
return null;
}
Now we can try to implement our main DOM update function. It should recursively, minimally, update elements. This is the most annoying bit to implement, it basically needs to:
- Leave unchanged children alone
- Insert new children, and remove deleted children
- Also update element attributes
Javascript
function updateElement(current, updated) {
if (current.nodeName !== updated.nodeName) {
current.replaceWith(updated.cloneNode(true));
return;
}
if (current.nodeType === Node.TEXT_NODE) {
if (current.textContent !== updated.textContent) {
current.textContent = updated.textContent;
}
return;
}
updateAttributes(current, updated);
// Map to store the position of nodes in the updated children
const updatedPositionMap = new Map(Array.from(updated.childNodes).map((node, index) => [node, index]));
const currentChildren = Array.from(current.childNodes);
let lastPlacedNodeIndex = -1;
let nodesToRemove = new Set(currentChildren);
let numAdded = 0, numRemoved = 0, numMoved = 0;
currentChildren.forEach((child, index) => {
const updatedChild = findMatchingElement(updated, child);
if (updatedChild) {
// Update the node if it matches one in the updated children
const [childAdded, childRemoved, childMoved] = updateElement(child, updatedChild);
[numAdded, numRemoved, numMoved] = [numAdded + childAdded, numRemoved + childRemoved, numMoved + childMoved];
nodesToRemove.delete(child);
// Determine the position where the node should be in the updated list
const updatedPosition = updatedPositionMap.get(updatedChild);
if (updatedPosition > lastPlacedNodeIndex) {
// Node is in the correct sequence, no need to move
lastPlacedNodeIndex = updatedPosition;
} else {
// Node needs to be moved to the correct position
const referenceNode = updated.childNodes[updatedPosition + 1] || null;
current.insertBefore(child, referenceNode);
lastPlacedNodeIndex = updatedPosition;
numMoved += 1;
}
}
});
// Remove nodes that are not found in the updated children
nodesToRemove.forEach(node => current.removeChild(node));
numRemoved = nodesToRemove.size
// Insert any new nodes that did not exist in the current children
updatedPositionMap.forEach((position, node) => {
if (!current.contains(node)) {
const referenceNode = current.childNodes[position] || null;
current.insertBefore(node.cloneNode(true), referenceNode);
numAdded += 1;
}
});
return [numAdded, numRemoved, numMoved];
}
We referenced an updateAttribute function, but we need to actually implement it
now.
Javascript
function updateAttributes(currentElement, newElement) {
const currentAttrs = currentElement.attributes;
const newAttrs = newElement.attributes;
for (const attr of currentAttrs) {
if (!newElement.hasAttribute(attr.name)) {
currentElement.removeAttribute(attr.name);
}
}
for (const attr of newAttrs) {
if (currentElement.getAttribute(attr.name) !== attr.value) {
currentElement.setAttribute(attr.name, attr.value);
}
}
}
4.3. Checking for updates
With the version-fetching and DOM-updating function we've implemented so far, actually checking for an update is as easy as:
- Checking to see if .version is changed
- Fetching the new HTML document if so
- Updating the DOM to match the new HTML document
Javascript
function fetchAndUpdateContent(initialVersion) {
fetch(window.location.href, { cache: 'no-cache' })
.then(response => response.text())
.then(newHtml => {
const parser = new DOMParser();
const newDoc = parser.parseFromString(newHtml, 'text/html');
newVersion = newDoc.querySelector('meta[name="version"]')?.content;
currentVersion = newVersion;
document.title = newDoc.title;
const [nodesAdded, nodesRemoved, nodesMoved] = updateElement(document.body, newDoc.body);
if (typeof MathJax == "object") {
MathJax.startup.defaultReady();
}
console.info(`Updated document from version ${initialVersion} → ${newVersion}: ${nodesAdded + nodesRemoved + nodesMoved} nodes changed.`);
})
.catch(error => console.error('Error fetching new HTML:', error));
}
function checkForUpdates() {
fetchVersion().then(newVersion => {
if (currentVersion && newVersion !== currentVersion) {
fetchAndUpdateContent(currentVersion);
currentVersion = newVersion; // Assume we'll update to the new version
}
});
}
Now we could just use a run-every-interval function to update the document, but I think it would be good to stop doing so when the page isn't visible, so let's get a little fancy and stop the fetching when that happens, and restart it when the page is visited again.
Javascript
let checkInterval;
function startChecking() {
checkInterval = setInterval(checkForUpdates, 15000); // Check every 15 seconds
}
function stopChecking() {
clearInterval(checkInterval);
}
function handleVisibilityChange() {
if (document.visibilityState === "visible") {
checkForUpdates(); // Perform an immediate check
startChecking(); // Restart the checking interval
} else {
stopChecking(); // Stop checking while the page is not visible
}
}
document.addEventListener("visibilitychange", handleVisibilityChange);
// Initial setup
if (document.visibilityState === "visible") {
startChecking(); // Start checking if the page is initially visible
} else {
stopChecking(); // Otherwise, don't start the interval
}