Compare commits

...
Sign in to create a new pull request.
Author SHA1 Message Date
20c9504ea9 Delete empty and whitespace-only notes without sending a mail
This also fixes the -v output for such notes, which referred to a mail
that was never sent and was logged as a send failure.
2026-10-27 12:59:13 +01:00
6a0d4365ca Remove unreferenced uploads and attach them to expiry mails
HedgeDoc doesn't track uploaded files, so they stay on disk after their
note is deleted. If UPLOADS_PATH is set, expire now removes uploads that
no remaining note or revision references and that are older than -u DAYS
(default 1). check and emailcheck list the files that would be removed.

Expiry mails now include the note's uploads as attachments, up to a total
of MAX_ATTACHMENT_SIZE bytes (default 20 MB); files that don't fit are
listed in the mail text instead.
2026-10-27 12:55:20 +01:00
3 changed files with 185 additions and 24 deletions

View file

@ -26,6 +26,21 @@ by the mail server, the note will not be removed. Note however that this utility
successfully delivered that mail to the intended recipient; if the mail gets lost somewhere on the way, the note cannot successfully delivered that mail to the intended recipient; if the mail gets lost somewhere on the way, the note cannot
be recovered. be recovered.
If `UPLOADS_PATH` is set, files uploaded to the note (images and other attachments) are attached to the mail as well,
up to a total of `MAX_ATTACHMENT_SIZE` bytes. Files that do not fit are listed in the mail text instead.
### Expiring unreferenced uploads
Hedgedoc does not keep track of uploaded files, so they remain on disk after their note has been deleted, either by the
user or by `hedgedoc-expire`. If `UPLOADS_PATH` points to Hedgedoc's uploads directory (`/hedgedoc/public/uploads` in
the Hedgedoc container image), `hedgedoc-expire` will remove all files in that directory that are not referenced by any
remaining note or revision. To avoid deleting files that have just been uploaded, but whose note has not been saved yet,
only files older than the age given with `-u` are removed.
A file is considered referenced if its name appears in a link containing `/uploads/<name>` in any note or revision,
regardless of the host name in the link. Links to uploads from outside of Hedgedoc are not taken into account. Only the
`filesystem` upload type is supported.
## Running `hedgedoc-expire.py` ## Running `hedgedoc-expire.py`
Locally from the command line: Locally from the command line:
@ -42,6 +57,9 @@ From Docker Compose:
command: "-c -r 14 -n 95" command: "-c -r 14 -n 95"
environment: environment:
- "POSTGRES_HOSTNAME=database" - "POSTGRES_HOSTNAME=database"
- "UPLOADS_PATH=/uploads"
volumes:
- uploads:/uploads
depends_on: depends_on:
- database - database
``` ```
@ -67,16 +85,17 @@ Without it, the expiry action will be taken.
|--------|---------|-------------------------------------------------------------------| |--------|---------|-------------------------------------------------------------------|
| -n | 90 | remove all notes not changed in these many days | | -n | 90 | remove all notes not changed in these many days |
| -r | 7 | remove all revisions created more than these many days ago | | -r | 7 | remove all revisions created more than these many days ago |
| -u | 1 | remove unreferenced uploads older than these many days |
| -v | false | Print info on current action during `cron` and `expire` commandds | | -v | false | Print info on current action during `cron` and `expire` commandds |
Command is one of: Command is one of:
| Command | Description | | Command | Description |
|------------|------------------------------------------------------------------------------------------------------------| |------------|------------------------------------------------------------------------------------------------------------|
| check | Print a list of revisions and notes that would be expired, based on the given parameters for `-n` and `-r` | | check | Print a list of revisions, notes and uploads that would be expired, based on `-n`, `-r` and `-u` |
| cron | Run `expire` at 2 am local time each day. Will run until killed. | | cron | Run `expire` at 2 am local time each day. Will run until killed. |
| emailcheck | Send an email from the configured sender to themselves with the the check report. | | emailcheck | Send an email from the configured sender to themselves with the the check report. |
| expire | Expire old revisions and notes. | | expire | Expire old revisions, notes and unreferenced uploads. |
### Environment variables ### Environment variables
@ -85,19 +104,21 @@ To configure the Postgres database connection and the SMTP parameters to send ma
For the SMTP connection, the code assumes a standard submission protocol setup with enable StartTLS and authentication, For the SMTP connection, the code assumes a standard submission protocol setup with enable StartTLS and authentication,
so you will need to configure a username and password. so you will need to configure a username and password.
| Variable | Default | Description | | Variable | Default | Description |
|-------------------|-----------------------|-------------------------------------| |---------------------|-----------------------|---------------------------------------------------------------|
| POSTGRES_DATABASE | hedgedoc | database to connect to | | MAX_ATTACHMENT_SIZE | 20971520 | maximum total size of uploads attached to an expiry mail |
| POSTGRES_HOSTNAME | localhost | host of the database server | | POSTGRES_DATABASE | hedgedoc | database to connect to |
| POSTGRES_PASSWORD | geheim | password for the database | | POSTGRES_HOSTNAME | localhost | host of the database server |
| POSTGRES_PORT | 5432 | port number of the database server | | POSTGRES_PASSWORD | geheim | password for the database |
| POSTGRES_USERNAME | hedgedoc | username for the database | | POSTGRES_PORT | 5432 | port number of the database server |
| SMTP_FROM | | sender address for the expiry mails | | POSTGRES_USERNAME | hedgedoc | username for the database |
| SMTP_HOSTNAME | localhost | mail server hostname | | SMTP_FROM | | sender address for the expiry mails |
| SMTP_PASSWORD | | SMTP password | | SMTP_HOSTNAME | localhost | mail server hostname |
| SMTP_PORT | 587 | port to connect to | | SMTP_PASSWORD | | SMTP password |
| SMTP_USERNAME | | SMTP username | | SMTP_PORT | 587 | port to connect to |
| URL | http://localhost:3000 | base URL for linking to notes | | SMTP_USERNAME | | SMTP username |
| UPLOADS_PATH | | Hedgedoc uploads directory; uploads are ignored if not set |
| URL | http://localhost:3000 | base URL for linking to notes |
## Local Development Setup ## Local Development Setup

View file

@ -49,6 +49,9 @@ services:
command: "-v -r .001 -n .001 check" command: "-v -r .001 -n .001 check"
environment: environment:
- "POSTGRES_HOSTNAME=database" - "POSTGRES_HOSTNAME=database"
- "UPLOADS_PATH=/uploads"
volumes:
- uploads:/uploads
depends_on: depends_on:
- database - database

View file

@ -2,14 +2,18 @@
import argparse import argparse
import email import email
import json import json
import mimetypes
import re
import smtplib import smtplib
import ssl import ssl
import sys import sys
from datetime import datetime, timezone, timedelta from datetime import datetime, timezone, timedelta
from email import encoders
from email.mime.base import MIMEBase from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText from email.mime.text import MIMEText
from os import getenv from os import getenv
from pathlib import Path
from textwrap import dedent from textwrap import dedent
from time import sleep from time import sleep
@ -26,6 +30,7 @@ class Config:
self.verbose = False self.verbose = False
self.revision_age = timedelta(days=14) self.revision_age = timedelta(days=14)
self.note_age = timedelta(days=95) self.note_age = timedelta(days=95)
self.upload_age = timedelta(days=1)
self.postgres_hostname = getenv('POSTGRES_HOSTNAME', 'localhost') self.postgres_hostname = getenv('POSTGRES_HOSTNAME', 'localhost')
self.postgres_username = getenv('POSTGRES_USERNAME', 'hedgedoc') self.postgres_username = getenv('POSTGRES_USERNAME', 'hedgedoc')
@ -39,6 +44,8 @@ class Config:
self.smtp_password = getenv('SMTP_PASSWORD', '') self.smtp_password = getenv('SMTP_PASSWORD', '')
self.smtp_from = getenv('SMTP_FROM', '') self.smtp_from = getenv('SMTP_FROM', '')
self.url = getenv('URL', 'http://localhost:3000') self.url = getenv('URL', 'http://localhost:3000')
self.uploads_path = getenv('UPLOADS_PATH', '')
self.max_attachment_size = int(getenv('MAX_ATTACHMENT_SIZE', str(20 * 1024 * 1024)))
class EmailSender: class EmailSender:
@ -76,6 +83,9 @@ class EmailSender:
class HedgedocExpire: class HedgedocExpire:
# matches the file name in links to uploads, e.g. https://example.com/uploads/abc.png or /uploads/abc.png
UPLOAD_REFERENCE = re.compile(r'/uploads/([^/\s()\[\]<>"\'?#]+)')
def __init__(self, config: Config, email_sender: EmailSender): def __init__(self, config: Config, email_sender: EmailSender):
self.config = config self.config = config
self.email_sender = email_sender self.email_sender = email_sender
@ -168,6 +178,66 @@ class HedgedocExpire:
}) })
return revisions return revisions
def referenced_uploads(self, db, note_cutoff: datetime, revision_cutoff: datetime) -> set[str]:
"""
Collect the file names of all uploads referenced by notes updated since note_cutoff, and by revisions created
since revision_cutoff.
:param db: the database connection
:param note_cutoff: ignore notes last updated before this time
:param revision_cutoff: ignore revisions created before this time
:return: set of file names in the uploads directory
"""
referenced = set()
with db.prepare('''SELECT COALESCE("content", '') AS "text"
FROM "Notes"
WHERE "updatedAt" >= $1
UNION ALL
SELECT COALESCE("content", '') || COALESCE("lastContent", '') || COALESCE("patch", '') AS "text"
FROM "Revisions"
WHERE "createdAt" >= $2
''') as texts:
for row in texts(note_cutoff, revision_cutoff):
referenced.update(self.UPLOAD_REFERENCE.findall(row.text))
return referenced
def uploads_to_be_expired(self, db, note_cutoff: datetime, revision_cutoff: datetime) -> list[Path]:
"""
Get a list of all uploaded files that are not referenced by any note or revision, and are older than the
configured upload age. The age check avoids removing files that have just been uploaded, but whose note has not
been saved to the database yet.
:param db: the database connection
:param note_cutoff: notes last updated before this time do not count as references
:param revision_cutoff: revisions created before this time do not count as references
:return: list of paths to unreferenced files
"""
uploads = Path(self.config.uploads_path)
referenced = self.referenced_uploads(db, note_cutoff, revision_cutoff)
cutoff = (datetime.now(timezone.utc) - self.config.upload_age).timestamp()
files = []
for file in sorted(uploads.iterdir()):
if file.name.startswith('.') or not file.is_file() or file.is_symlink():
continue
if file.name not in referenced and file.stat().st_mtime < cutoff:
files.append(file)
return files
def check_uploads_to_be_expired(self, db) -> str:
"""
Return a list of uploads that will be expired, assuming the notes and revisions listed by the other checks have
been expired.
:param db: the database connection
:return: a multi-line text suitable for humans to read
"""
if not self.config.uploads_path:
return ''
r = f'Uploads to be deleted not referenced by any note and older than {humanize.naturaldelta(self.config.upload_age)}:\n'
now = datetime.now(timezone.utc)
for file in self.uploads_to_be_expired(db, now - self.config.note_age, now - self.config.revision_age):
stat = file.stat()
age = now - datetime.fromtimestamp(stat.st_mtime, timezone.utc)
r += f' {file.name} ({humanize.naturalsize(stat.st_size)}, {humanize.naturaldelta(age)})\n'
return r
def check_notes_to_be_expired(self, db) -> str: def check_notes_to_be_expired(self, db) -> str:
""" """
Return a list of notes that will be expired. Return a list of notes that will be expired.
@ -206,6 +276,35 @@ class HedgedocExpire:
r += f' {humanize.naturaldelta(rev["age"])}: {rev["revisionId"]}\n' r += f' {humanize.naturaldelta(rev["age"])}: {rev["revisionId"]}\n'
return r return r
def upload_attachments(self, content: str) -> tuple[list[MIMEBase], list[str]]:
"""
Create mail attachments for the uploads referenced in a note, up to the configured total size.
:param content: the note content
:return: the attachments, and the names of referenced uploads that could not be attached
"""
attachments = []
skipped = []
if not self.config.uploads_path:
return attachments, skipped
total = 0
for name in sorted(set(self.UPLOAD_REFERENCE.findall(content))):
file = Path(self.config.uploads_path) / name
if not file.is_file() or file.is_symlink():
continue
size = file.stat().st_size
if total + size > self.config.max_attachment_size:
skipped.append(name)
continue
total += size
content_type, _ = mimetypes.guess_type(name)
maintype, subtype = (content_type or 'application/octet-stream').split('/', 1)
attachment = MIMEBase(maintype, subtype)
attachment.set_payload(file.read_bytes())
encoders.encode_base64(attachment)
attachment.add_header('Content-Disposition', 'attachment', filename=name)
attachments.append(attachment)
return attachments, skipped
def expire_old_notes(self, db) -> None: def expire_old_notes(self, db) -> None:
""" """
Email old notes to their owners, then delete them. Email old notes to their owners, then delete them.
@ -215,9 +314,11 @@ class HedgedocExpire:
with db.prepare('DELETE FROM "Notes" WHERE "id" = $1') as delete_statement: with db.prepare('DELETE FROM "Notes" WHERE "id" = $1') as delete_statement:
for note in self.notes_to_be_expired(db): for note in self.notes_to_be_expired(db):
try: try:
# notes that are empty or only contain whitespace are deleted without sending a backup
if len(note["content"]) > 0: has_content = len((note["content"] or '').strip()) > 0
if has_content:
note_age = datetime.now(timezone.utc) - datetime.fromisoformat(note['updatedAt']) note_age = datetime.now(timezone.utc) - datetime.fromisoformat(note['updatedAt'])
attachments, skipped = self.upload_attachments(note["content"])
msg = MIMEMultipart() msg = MIMEMultipart()
msg['From'] = self.email_sender.mail_from msg['From'] = self.email_sender.mail_from
msg['To'] = self.email_from_email_or_profile(note) msg['To'] = self.email_from_email_or_profile(note)
@ -227,8 +328,12 @@ class HedgedocExpire:
It was lasted updated {note['updatedAt']}, {humanize.naturaldelta(note_age)} ago. We expire all notes It was lasted updated {note['updatedAt']}, {humanize.naturaldelta(note_age)} ago. We expire all notes
that have not been updated within {humanize.naturaldelta(self.config.note_age)}. that have not been updated within {humanize.naturaldelta(self.config.note_age)}.
Please find attached the contents of the latest revision of your note. Please find attached the contents of the latest revision of your note, and the files
uploaded to it.
''') + (dedent(f'''\
These uploaded files were too large to attach:
''') + ''.join(f' {name}\n' for name in skipped) + '\n' if skipped else '') + dedent(f'''\
The admin team for {self.config.url} The admin team for {self.config.url}
'''), 'plain', 'utf-8')) '''), 'plain', 'utf-8'))
@ -238,6 +343,8 @@ class HedgedocExpire:
filename = 'note' filename = 'note'
md.add_header('Content-Disposition', f'attachment; filename={filename}.md') md.add_header('Content-Disposition', f'attachment; filename={filename}.md')
msg.attach(md) msg.attach(md)
for attachment in attachments:
msg.attach(attachment)
self.email_sender.send(msg) self.email_sender.send(msg)
# email backup of the note sent, now we can delete it # email backup of the note sent, now we can delete it
@ -245,7 +352,10 @@ class HedgedocExpire:
if self.config.verbose: if self.config.verbose:
url = self.config.url + '/' + (note["alias"] if note["alias"] is not None else note["shortid"]) url = self.config.url + '/' + (note["alias"] if note["alias"] is not None else note["shortid"])
print(f'Note "{note["title"]}" ({url}) emailed to {msg["To"]}') if has_content:
print(f'Note "{note["title"]}" ({url}) emailed to {self.email_from_email_or_profile(note)}')
else:
print(f'Empty note "{note["title"]}" ({url}) deleted')
except Exception as e: except Exception as e:
print(f'Unable to send email to {self.email_from_email_or_profile(note)}: {e}', file=sys.stderr) print(f'Unable to send email to {self.email_from_email_or_profile(note)}: {e}', file=sys.stderr)
@ -261,22 +371,44 @@ class HedgedocExpire:
if self.config.verbose: if self.config.verbose:
print(f'Deleted {len(rows)} old revisions') print(f'Deleted {len(rows)} old revisions')
def expire_unreferenced_uploads(self, db) -> None:
"""
Remove all uploaded files that are not referenced by any remaining note or revision.
:param db: the database connection
:return:
"""
if not self.config.uploads_path:
return
# count every note and revision that is still in the database, including notes that failed to expire
epoch = datetime.fromtimestamp(0, timezone.utc)
deleted = 0
for file in self.uploads_to_be_expired(db, epoch, epoch):
try:
file.unlink()
deleted += 1
except Exception as e:
print(f'Unable to delete upload {file}: {e}', file=sys.stderr)
if self.config.verbose:
print(f'Deleted {deleted} unreferenced uploads')
def cmd_check(self) -> None: def cmd_check(self) -> None:
with pgsql.Connection((self.config.postgres_hostname, self.config.postgres_port), with pgsql.Connection((self.config.postgres_hostname, self.config.postgres_port),
self.config.postgres_username, self.config.postgres_password) as db: self.config.postgres_username, self.config.postgres_password) as db:
print(self.check_revisions_to_be_expired(db) + print(self.check_revisions_to_be_expired(db) +
self.check_notes_to_be_expired(db)) self.check_notes_to_be_expired(db) +
self.check_uploads_to_be_expired(db))
def cmd_emailcheck(self) -> None: def cmd_emailcheck(self) -> None:
with pgsql.Connection((self.config.postgres_hostname, self.config.postgres_port), with pgsql.Connection((self.config.postgres_hostname, self.config.postgres_port),
self.config.postgres_username, self.config.postgres_password) as db: self.config.postgres_username, self.config.postgres_password) as db:
report = self.check_revisions_to_be_expired(db) + self.check_notes_to_be_expired(db) report = (self.check_revisions_to_be_expired(db) + self.check_notes_to_be_expired(db) +
self.check_uploads_to_be_expired(db))
msg = MIMEMultipart() msg = MIMEMultipart()
msg['From'] = self.email_sender.mail_from msg['From'] = self.email_sender.mail_from
msg['To'] = self.email_sender.mail_from msg['To'] = self.email_sender.mail_from
msg['Subject'] = f'Hedgedoc Expire: Report' msg['Subject'] = f'Hedgedoc Expire: Report'
msg.attach(MIMEText(dedent(f'''\ msg.attach(MIMEText(dedent(f'''\
This report shows which notes and revisions would be deleted if expire would be run now. This report shows which notes, revisions and uploads would be deleted if expire would be run now.
''') + report + dedent(f'''\ ''') + report + dedent(f'''\
@ -289,6 +421,7 @@ class HedgedocExpire:
self.config.postgres_username, self.config.postgres_password) as db: self.config.postgres_username, self.config.postgres_password) as db:
self.expire_old_revisions(db) self.expire_old_revisions(db)
self.expire_old_notes(db) self.expire_old_notes(db)
self.expire_unreferenced_uploads(db)
def main(): def main():
@ -300,13 +433,14 @@ def main():
Notes that have not been updated in the specified time will be emailed to the creator and then deleted. Notes that have not been updated in the specified time will be emailed to the creator and then deleted.
Revisions of notes that have been created before the specified time will be deleted. Revisions of notes that have been created before the specified time will be deleted.
If UPLOADS_PATH is set, uploaded files no longer referenced by any note or revision will be deleted.
'''), '''),
epilog=dedent('''\ epilog=dedent('''\
command is one of: command is one of:
- check: print a list of revisions and notes to be expired - check: print a list of revisions, notes and uploads to be expired
- cron: run expire every 24 hours - cron: run expire every 24 hours
- emailcheck: send am email from the configured sender to themselves with the the check report - emailcheck: send am email from the configured sender to themselves with the the check report
- expire: expire old revisions and untouched notes - expire: expire old revisions, untouched notes and unreferenced uploads
See https://git.hamburg.ccc.de/CCCHH/hedgedoc-expire See https://git.hamburg.ccc.de/CCCHH/hedgedoc-expire
''') ''')
@ -315,6 +449,8 @@ def main():
help='remove all notes not changed in these many days') help='remove all notes not changed in these many days')
parser.add_argument('-r', '--revisions', metavar='DAYS', type=float, default=14, parser.add_argument('-r', '--revisions', metavar='DAYS', type=float, default=14,
help='remove all revisions created more than these many days ago') help='remove all revisions created more than these many days ago')
parser.add_argument('-u', '--uploads', metavar='DAYS', type=float, default=1,
help='remove unreferenced uploads older than these many days')
parser.add_argument('command', choices=['check', 'cron', 'emailcheck', 'expire'], default='check', nargs='?', parser.add_argument('command', choices=['check', 'cron', 'emailcheck', 'expire'], default='check', nargs='?',
help='action to perform') help='action to perform')
parser.add_argument('-v', '--verbose', action='store_true', default=False, parser.add_argument('-v', '--verbose', action='store_true', default=False,
@ -324,6 +460,7 @@ def main():
config = Config() config = Config()
config.note_age = timedelta(days=args.notes) config.note_age = timedelta(days=args.notes)
config.revision_age = timedelta(days=args.revisions) config.revision_age = timedelta(days=args.revisions)
config.upload_age = timedelta(days=args.uploads)
config.verbose = args.verbose config.verbose = args.verbose
mail = EmailSender(config.smtp_hostname, config.smtp_port, config.smtp_username, config.smtp_password, mail = EmailSender(config.smtp_hostname, config.smtp_port, config.smtp_username, config.smtp_password,
config.smtp_from) config.smtp_from)