Compare commits
3 changed files with 24 additions and 185 deletions
51
README.md
51
README.md
|
|
@ -26,21 +26,6 @@ by the mail server, the note will not be removed. Note however that this utility
|
|||
successfully delivered that mail to the intended recipient; if the mail gets lost somewhere on the way, the note cannot
|
||||
be recovered.
|
||||
|
||||
If `UPLOADS_PATH` is set, files uploaded to the note (images and other attachments) are attached to the mail as well,
|
||||
up to a total of `MAX_ATTACHMENT_SIZE` bytes. Files that do not fit are listed in the mail text instead.
|
||||
|
||||
### Expiring unreferenced uploads
|
||||
|
||||
Hedgedoc does not keep track of uploaded files, so they remain on disk after their note has been deleted, either by the
|
||||
user or by `hedgedoc-expire`. If `UPLOADS_PATH` points to Hedgedoc's uploads directory (`/hedgedoc/public/uploads` in
|
||||
the Hedgedoc container image), `hedgedoc-expire` will remove all files in that directory that are not referenced by any
|
||||
remaining note or revision. To avoid deleting files that have just been uploaded, but whose note has not been saved yet,
|
||||
only files older than the age given with `-u` are removed.
|
||||
|
||||
A file is considered referenced if its name appears in a link containing `/uploads/<name>` in any note or revision,
|
||||
regardless of the host name in the link. Links to uploads from outside of Hedgedoc are not taken into account. Only the
|
||||
`filesystem` upload type is supported.
|
||||
|
||||
## Running `hedgedoc-expire.py`
|
||||
|
||||
Locally from the command line:
|
||||
|
|
@ -57,9 +42,6 @@ From Docker Compose:
|
|||
command: "-c -r 14 -n 95"
|
||||
environment:
|
||||
- "POSTGRES_HOSTNAME=database"
|
||||
- "UPLOADS_PATH=/uploads"
|
||||
volumes:
|
||||
- uploads:/uploads
|
||||
depends_on:
|
||||
- database
|
||||
```
|
||||
|
|
@ -85,17 +67,16 @@ Without it, the expiry action will be taken.
|
|||
|--------|---------|-------------------------------------------------------------------|
|
||||
| -n | 90 | remove all notes not changed in these many days |
|
||||
| -r | 7 | remove all revisions created more than these many days ago |
|
||||
| -u | 1 | remove unreferenced uploads older than these many days |
|
||||
| -v | false | Print info on current action during `cron` and `expire` commandds |
|
||||
|
||||
Command is one of:
|
||||
|
||||
| Command | Description |
|
||||
|------------|------------------------------------------------------------------------------------------------------------|
|
||||
| check | Print a list of revisions, notes and uploads that would be expired, based on `-n`, `-r` and `-u` |
|
||||
| check | Print a list of revisions and notes that would be expired, based on the given parameters for `-n` and `-r` |
|
||||
| cron | Run `expire` at 2 am local time each day. Will run until killed. |
|
||||
| emailcheck | Send an email from the configured sender to themselves with the the check report. |
|
||||
| expire | Expire old revisions, notes and unreferenced uploads. |
|
||||
| expire | Expire old revisions and notes. |
|
||||
|
||||
### Environment variables
|
||||
|
||||
|
|
@ -104,21 +85,19 @@ To configure the Postgres database connection and the SMTP parameters to send ma
|
|||
For the SMTP connection, the code assumes a standard submission protocol setup with enable StartTLS and authentication,
|
||||
so you will need to configure a username and password.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---------------------|-----------------------|---------------------------------------------------------------|
|
||||
| MAX_ATTACHMENT_SIZE | 20971520 | maximum total size of uploads attached to an expiry mail |
|
||||
| POSTGRES_DATABASE | hedgedoc | database to connect to |
|
||||
| POSTGRES_HOSTNAME | localhost | host of the database server |
|
||||
| POSTGRES_PASSWORD | geheim | password for the database |
|
||||
| POSTGRES_PORT | 5432 | port number of the database server |
|
||||
| POSTGRES_USERNAME | hedgedoc | username for the database |
|
||||
| SMTP_FROM | | sender address for the expiry mails |
|
||||
| SMTP_HOSTNAME | localhost | mail server hostname |
|
||||
| SMTP_PASSWORD | | SMTP password |
|
||||
| SMTP_PORT | 587 | port to connect to |
|
||||
| SMTP_USERNAME | | SMTP username |
|
||||
| UPLOADS_PATH | | Hedgedoc uploads directory; uploads are ignored if not set |
|
||||
| URL | http://localhost:3000 | base URL for linking to notes |
|
||||
| Variable | Default | Description |
|
||||
|-------------------|-----------------------|-------------------------------------|
|
||||
| POSTGRES_DATABASE | hedgedoc | database to connect to |
|
||||
| POSTGRES_HOSTNAME | localhost | host of the database server |
|
||||
| POSTGRES_PASSWORD | geheim | password for the database |
|
||||
| POSTGRES_PORT | 5432 | port number of the database server |
|
||||
| POSTGRES_USERNAME | hedgedoc | username for the database |
|
||||
| SMTP_FROM | | sender address for the expiry mails |
|
||||
| SMTP_HOSTNAME | localhost | mail server hostname |
|
||||
| SMTP_PASSWORD | | SMTP password |
|
||||
| SMTP_PORT | 587 | port to connect to |
|
||||
| SMTP_USERNAME | | SMTP username |
|
||||
| URL | http://localhost:3000 | base URL for linking to notes |
|
||||
|
||||
## Local Development Setup
|
||||
|
||||
|
|
|
|||
|
|
@ -49,9 +49,6 @@ services:
|
|||
command: "-v -r .001 -n .001 check"
|
||||
environment:
|
||||
- "POSTGRES_HOSTNAME=database"
|
||||
- "UPLOADS_PATH=/uploads"
|
||||
volumes:
|
||||
- uploads:/uploads
|
||||
depends_on:
|
||||
- database
|
||||
|
||||
|
|
|
|||
|
|
@ -2,18 +2,14 @@
|
|||
import argparse
|
||||
import email
|
||||
import json
|
||||
import mimetypes
|
||||
import re
|
||||
import smtplib
|
||||
import ssl
|
||||
import sys
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from email import encoders
|
||||
from email.mime.base import MIMEBase
|
||||
from email.mime.multipart import MIMEMultipart
|
||||
from email.mime.text import MIMEText
|
||||
from os import getenv
|
||||
from pathlib import Path
|
||||
from textwrap import dedent
|
||||
from time import sleep
|
||||
|
||||
|
|
@ -30,7 +26,6 @@ class Config:
|
|||
self.verbose = False
|
||||
self.revision_age = timedelta(days=14)
|
||||
self.note_age = timedelta(days=95)
|
||||
self.upload_age = timedelta(days=1)
|
||||
|
||||
self.postgres_hostname = getenv('POSTGRES_HOSTNAME', 'localhost')
|
||||
self.postgres_username = getenv('POSTGRES_USERNAME', 'hedgedoc')
|
||||
|
|
@ -44,8 +39,6 @@ class Config:
|
|||
self.smtp_password = getenv('SMTP_PASSWORD', '')
|
||||
self.smtp_from = getenv('SMTP_FROM', '')
|
||||
self.url = getenv('URL', 'http://localhost:3000')
|
||||
self.uploads_path = getenv('UPLOADS_PATH', '')
|
||||
self.max_attachment_size = int(getenv('MAX_ATTACHMENT_SIZE', str(20 * 1024 * 1024)))
|
||||
|
||||
|
||||
class EmailSender:
|
||||
|
|
@ -83,9 +76,6 @@ class EmailSender:
|
|||
|
||||
|
||||
class HedgedocExpire:
|
||||
# matches the file name in links to uploads, e.g. https://example.com/uploads/abc.png or /uploads/abc.png
|
||||
UPLOAD_REFERENCE = re.compile(r'/uploads/([^/\s()\[\]<>"\'?#]+)')
|
||||
|
||||
def __init__(self, config: Config, email_sender: EmailSender):
|
||||
self.config = config
|
||||
self.email_sender = email_sender
|
||||
|
|
@ -178,66 +168,6 @@ class HedgedocExpire:
|
|||
})
|
||||
return revisions
|
||||
|
||||
def referenced_uploads(self, db, note_cutoff: datetime, revision_cutoff: datetime) -> set[str]:
|
||||
"""
|
||||
Collect the file names of all uploads referenced by notes updated since note_cutoff, and by revisions created
|
||||
since revision_cutoff.
|
||||
:param db: the database connection
|
||||
:param note_cutoff: ignore notes last updated before this time
|
||||
:param revision_cutoff: ignore revisions created before this time
|
||||
:return: set of file names in the uploads directory
|
||||
"""
|
||||
referenced = set()
|
||||
with db.prepare('''SELECT COALESCE("content", '') AS "text"
|
||||
FROM "Notes"
|
||||
WHERE "updatedAt" >= $1
|
||||
UNION ALL
|
||||
SELECT COALESCE("content", '') || COALESCE("lastContent", '') || COALESCE("patch", '') AS "text"
|
||||
FROM "Revisions"
|
||||
WHERE "createdAt" >= $2
|
||||
''') as texts:
|
||||
for row in texts(note_cutoff, revision_cutoff):
|
||||
referenced.update(self.UPLOAD_REFERENCE.findall(row.text))
|
||||
return referenced
|
||||
|
||||
def uploads_to_be_expired(self, db, note_cutoff: datetime, revision_cutoff: datetime) -> list[Path]:
|
||||
"""
|
||||
Get a list of all uploaded files that are not referenced by any note or revision, and are older than the
|
||||
configured upload age. The age check avoids removing files that have just been uploaded, but whose note has not
|
||||
been saved to the database yet.
|
||||
:param db: the database connection
|
||||
:param note_cutoff: notes last updated before this time do not count as references
|
||||
:param revision_cutoff: revisions created before this time do not count as references
|
||||
:return: list of paths to unreferenced files
|
||||
"""
|
||||
uploads = Path(self.config.uploads_path)
|
||||
referenced = self.referenced_uploads(db, note_cutoff, revision_cutoff)
|
||||
cutoff = (datetime.now(timezone.utc) - self.config.upload_age).timestamp()
|
||||
files = []
|
||||
for file in sorted(uploads.iterdir()):
|
||||
if file.name.startswith('.') or not file.is_file() or file.is_symlink():
|
||||
continue
|
||||
if file.name not in referenced and file.stat().st_mtime < cutoff:
|
||||
files.append(file)
|
||||
return files
|
||||
|
||||
def check_uploads_to_be_expired(self, db) -> str:
|
||||
"""
|
||||
Return a list of uploads that will be expired, assuming the notes and revisions listed by the other checks have
|
||||
been expired.
|
||||
:param db: the database connection
|
||||
:return: a multi-line text suitable for humans to read
|
||||
"""
|
||||
if not self.config.uploads_path:
|
||||
return ''
|
||||
r = f'Uploads to be deleted not referenced by any note and older than {humanize.naturaldelta(self.config.upload_age)}:\n'
|
||||
now = datetime.now(timezone.utc)
|
||||
for file in self.uploads_to_be_expired(db, now - self.config.note_age, now - self.config.revision_age):
|
||||
stat = file.stat()
|
||||
age = now - datetime.fromtimestamp(stat.st_mtime, timezone.utc)
|
||||
r += f' {file.name} ({humanize.naturalsize(stat.st_size)}, {humanize.naturaldelta(age)})\n'
|
||||
return r
|
||||
|
||||
def check_notes_to_be_expired(self, db) -> str:
|
||||
"""
|
||||
Return a list of notes that will be expired.
|
||||
|
|
@ -276,35 +206,6 @@ class HedgedocExpire:
|
|||
r += f' {humanize.naturaldelta(rev["age"])}: {rev["revisionId"]}\n'
|
||||
return r
|
||||
|
||||
def upload_attachments(self, content: str) -> tuple[list[MIMEBase], list[str]]:
|
||||
"""
|
||||
Create mail attachments for the uploads referenced in a note, up to the configured total size.
|
||||
:param content: the note content
|
||||
:return: the attachments, and the names of referenced uploads that could not be attached
|
||||
"""
|
||||
attachments = []
|
||||
skipped = []
|
||||
if not self.config.uploads_path:
|
||||
return attachments, skipped
|
||||
total = 0
|
||||
for name in sorted(set(self.UPLOAD_REFERENCE.findall(content))):
|
||||
file = Path(self.config.uploads_path) / name
|
||||
if not file.is_file() or file.is_symlink():
|
||||
continue
|
||||
size = file.stat().st_size
|
||||
if total + size > self.config.max_attachment_size:
|
||||
skipped.append(name)
|
||||
continue
|
||||
total += size
|
||||
content_type, _ = mimetypes.guess_type(name)
|
||||
maintype, subtype = (content_type or 'application/octet-stream').split('/', 1)
|
||||
attachment = MIMEBase(maintype, subtype)
|
||||
attachment.set_payload(file.read_bytes())
|
||||
encoders.encode_base64(attachment)
|
||||
attachment.add_header('Content-Disposition', 'attachment', filename=name)
|
||||
attachments.append(attachment)
|
||||
return attachments, skipped
|
||||
|
||||
def expire_old_notes(self, db) -> None:
|
||||
"""
|
||||
Email old notes to their owners, then delete them.
|
||||
|
|
@ -314,11 +215,9 @@ class HedgedocExpire:
|
|||
with db.prepare('DELETE FROM "Notes" WHERE "id" = $1') as delete_statement:
|
||||
for note in self.notes_to_be_expired(db):
|
||||
try:
|
||||
# notes that are empty or only contain whitespace are deleted without sending a backup
|
||||
has_content = len((note["content"] or '').strip()) > 0
|
||||
if has_content:
|
||||
|
||||
if len(note["content"]) > 0:
|
||||
note_age = datetime.now(timezone.utc) - datetime.fromisoformat(note['updatedAt'])
|
||||
attachments, skipped = self.upload_attachments(note["content"])
|
||||
msg = MIMEMultipart()
|
||||
msg['From'] = self.email_sender.mail_from
|
||||
msg['To'] = self.email_from_email_or_profile(note)
|
||||
|
|
@ -328,12 +227,8 @@ class HedgedocExpire:
|
|||
It was lasted updated {note['updatedAt']}, {humanize.naturaldelta(note_age)} ago. We expire all notes
|
||||
that have not been updated within {humanize.naturaldelta(self.config.note_age)}.
|
||||
|
||||
Please find attached the contents of the latest revision of your note, and the files
|
||||
uploaded to it.
|
||||
Please find attached the contents of the latest revision of your note.
|
||||
|
||||
''') + (dedent(f'''\
|
||||
These uploaded files were too large to attach:
|
||||
''') + ''.join(f' {name}\n' for name in skipped) + '\n' if skipped else '') + dedent(f'''\
|
||||
The admin team for {self.config.url}
|
||||
|
||||
'''), 'plain', 'utf-8'))
|
||||
|
|
@ -343,8 +238,6 @@ class HedgedocExpire:
|
|||
filename = 'note'
|
||||
md.add_header('Content-Disposition', f'attachment; filename={filename}.md')
|
||||
msg.attach(md)
|
||||
for attachment in attachments:
|
||||
msg.attach(attachment)
|
||||
self.email_sender.send(msg)
|
||||
|
||||
# email backup of the note sent, now we can delete it
|
||||
|
|
@ -352,10 +245,7 @@ class HedgedocExpire:
|
|||
|
||||
if self.config.verbose:
|
||||
url = self.config.url + '/' + (note["alias"] if note["alias"] is not None else note["shortid"])
|
||||
if has_content:
|
||||
print(f'Note "{note["title"]}" ({url}) emailed to {self.email_from_email_or_profile(note)}')
|
||||
else:
|
||||
print(f'Empty note "{note["title"]}" ({url}) deleted')
|
||||
print(f'Note "{note["title"]}" ({url}) emailed to {msg["To"]}')
|
||||
except Exception as e:
|
||||
print(f'Unable to send email to {self.email_from_email_or_profile(note)}: {e}', file=sys.stderr)
|
||||
|
||||
|
|
@ -371,44 +261,22 @@ class HedgedocExpire:
|
|||
if self.config.verbose:
|
||||
print(f'Deleted {len(rows)} old revisions')
|
||||
|
||||
def expire_unreferenced_uploads(self, db) -> None:
|
||||
"""
|
||||
Remove all uploaded files that are not referenced by any remaining note or revision.
|
||||
:param db: the database connection
|
||||
:return:
|
||||
"""
|
||||
if not self.config.uploads_path:
|
||||
return
|
||||
# count every note and revision that is still in the database, including notes that failed to expire
|
||||
epoch = datetime.fromtimestamp(0, timezone.utc)
|
||||
deleted = 0
|
||||
for file in self.uploads_to_be_expired(db, epoch, epoch):
|
||||
try:
|
||||
file.unlink()
|
||||
deleted += 1
|
||||
except Exception as e:
|
||||
print(f'Unable to delete upload {file}: {e}', file=sys.stderr)
|
||||
if self.config.verbose:
|
||||
print(f'Deleted {deleted} unreferenced uploads')
|
||||
|
||||
def cmd_check(self) -> None:
|
||||
with pgsql.Connection((self.config.postgres_hostname, self.config.postgres_port),
|
||||
self.config.postgres_username, self.config.postgres_password) as db:
|
||||
print(self.check_revisions_to_be_expired(db) +
|
||||
self.check_notes_to_be_expired(db) +
|
||||
self.check_uploads_to_be_expired(db))
|
||||
self.check_notes_to_be_expired(db))
|
||||
|
||||
def cmd_emailcheck(self) -> None:
|
||||
with pgsql.Connection((self.config.postgres_hostname, self.config.postgres_port),
|
||||
self.config.postgres_username, self.config.postgres_password) as db:
|
||||
report = (self.check_revisions_to_be_expired(db) + self.check_notes_to_be_expired(db) +
|
||||
self.check_uploads_to_be_expired(db))
|
||||
report = self.check_revisions_to_be_expired(db) + self.check_notes_to_be_expired(db)
|
||||
msg = MIMEMultipart()
|
||||
msg['From'] = self.email_sender.mail_from
|
||||
msg['To'] = self.email_sender.mail_from
|
||||
msg['Subject'] = f'Hedgedoc Expire: Report'
|
||||
msg.attach(MIMEText(dedent(f'''\
|
||||
This report shows which notes, revisions and uploads would be deleted if expire would be run now.
|
||||
This report shows which notes and revisions would be deleted if expire would be run now.
|
||||
|
||||
''') + report + dedent(f'''\
|
||||
|
||||
|
|
@ -421,7 +289,6 @@ class HedgedocExpire:
|
|||
self.config.postgres_username, self.config.postgres_password) as db:
|
||||
self.expire_old_revisions(db)
|
||||
self.expire_old_notes(db)
|
||||
self.expire_unreferenced_uploads(db)
|
||||
|
||||
|
||||
def main():
|
||||
|
|
@ -433,14 +300,13 @@ def main():
|
|||
|
||||
Notes that have not been updated in the specified time will be emailed to the creator and then deleted.
|
||||
Revisions of notes that have been created before the specified time will be deleted.
|
||||
If UPLOADS_PATH is set, uploaded files no longer referenced by any note or revision will be deleted.
|
||||
'''),
|
||||
epilog=dedent('''\
|
||||
command is one of:
|
||||
- check: print a list of revisions, notes and uploads to be expired
|
||||
- check: print a list of revisions and notes to be expired
|
||||
- cron: run expire every 24 hours
|
||||
- emailcheck: send am email from the configured sender to themselves with the the check report
|
||||
- expire: expire old revisions, untouched notes and unreferenced uploads
|
||||
- expire: expire old revisions and untouched notes
|
||||
|
||||
See https://git.hamburg.ccc.de/CCCHH/hedgedoc-expire
|
||||
''')
|
||||
|
|
@ -449,8 +315,6 @@ def main():
|
|||
help='remove all notes not changed in these many days')
|
||||
parser.add_argument('-r', '--revisions', metavar='DAYS', type=float, default=14,
|
||||
help='remove all revisions created more than these many days ago')
|
||||
parser.add_argument('-u', '--uploads', metavar='DAYS', type=float, default=1,
|
||||
help='remove unreferenced uploads older than these many days')
|
||||
parser.add_argument('command', choices=['check', 'cron', 'emailcheck', 'expire'], default='check', nargs='?',
|
||||
help='action to perform')
|
||||
parser.add_argument('-v', '--verbose', action='store_true', default=False,
|
||||
|
|
@ -460,7 +324,6 @@ def main():
|
|||
config = Config()
|
||||
config.note_age = timedelta(days=args.notes)
|
||||
config.revision_age = timedelta(days=args.revisions)
|
||||
config.upload_age = timedelta(days=args.uploads)
|
||||
config.verbose = args.verbose
|
||||
mail = EmailSender(config.smtp_hostname, config.smtp_port, config.smtp_username, config.smtp_password,
|
||||
config.smtp_from)
|
||||
|
|
|
|||
Loading…
Reference in a new issue