Email Notifications and Automated Delivery in RDataCore

Data without a way to tell humans what happened is only half a system. RDataCore now ships a full email stack so every pipeline, import, and auth event can reach the people who care: customers getting a welcome message after signing up, operators getting a nightly import summary, admins recovering a lost password, compliance teams reading an audit trail of every delivery.

The design has four moving parts. Two independent SMTP lanes keep operational mail and workflow mail separate. Database-managed Handlebars templates let admins edit subject lines and bodies without redeploying. An async Redis job queue smooths out delivery so workflows never block on a slow mail server. And a full System Log records every send (recipient, template, rendered content, status) for auditing and debugging.

This post walks through the architecture, shows the two workflow email primitives (per item and post run), and ends with a setup guide you can run against your own instance.

Most platforms send every email through the same SMTP relay. RDataCore splits the wiring on purpose.

Why two lanes? A high-volume workflow shouldn't be able to exhaust the reputation or throughput quota of the system relay that has to deliver a password reset. Operators can route them through different providers (e.g., a transactional provider for system, a marketing provider for workflow), apply different From addresses and display names, and monitor them independently.

Both DSNs are optional. When either is not configured, the corresponding features are automatically gated off: the UI checks a capabilities endpoint on startup and hides everything email-related for that lane. Below, the login page shows the feature-gated state when SYSTEM_SMTP_DSN is unset. The Forgot your password? flow is not offered, and any attempt to use it returns a clear message.

The two lanes cover very different jobs.

Workflow lane, per item (one email per record processed, while the pipeline keeps running):

• A welcome or confirmation email for each newly ingested customer from a nightly import.
• A threshold alert per sensor reading that crosses a safety limit.
• A receipt per line in an order import.

Workflow lane, post run (one email after the full batch completes):

• A nightly import summary showing processed / failed / total counts and duration.
• A failure digest to the ops channel when at least one item in a batch errored.
• A weekly stakeholder rollup that only fires on successful completion.

Which lane to pick is a question of semantics: if the mail is about the platform, it belongs on the system lane; if it's about data moving through the platform, it belongs on the workflow lane.

Workflow email comes in two distinct primitives, and understanding when to reach for each one is the key insight of the whole feature.

Send Email transform (per item). A Send Email step can be inserted anywhere inside a workflow's transform pipeline. It fires once per item, and the workflow continues immediately after queuing the email; delivery happens asynchronously on the worker. The UI spells this out: Sends one email per item processed. Use as a mid-pipeline notification; the workflow continues after queuing the email.

Recipients can be resolved from a field on the current record (e.g., the email field on a user row) or set to a constant. Template variables are filled from the normalized context: every field on the item plus the output of prior transforms. A Target Status Field captures the per-item outcome (queued, mail_not_configured) so later steps in the pipeline can branch on it.

Post-Run Action: Send Email (post run). Post-Run Actions live in a separate section of the workflow definition. They run once after all items are processed, and they have access to a rich run context: {{run.workflow_name}}, {{run.processed_items}}, {{run.failed_items}}, {{run.total_items}}, {{run.status}}, and {{run.duration_seconds}}. Combined with the step's normalized fields, this is enough to produce a useful summary message.

A Condition selector decides whether to fire: Always, On Success (all items passed), or On Failure (at least one item failed). This is the right place for summary notifications, reports, and alerts. The UI banner reads These actions execute once after all items are processed.

The distinction is simple: reach for per item when each record deserves its own message; reach for post run when the batch as a whole deserves one.

Every email RDataCore sends is rendered from a template stored in the email_templates table. Templates are Handlebars and support {{variable}}, {{nested.field}}, {{#if condition}}…{{/if}}, and {{#each array}}…{{/each}}. Missing variables render as empty strings, a deliberately forgiving default so a badly named field can't hard-fail a delivery.

There are two kinds of templates:

The template editor supports three bodies: subject, HTML body (for mail clients that render HTML), and plain-text body (the fallback). Variables the template references are auto-detected from the template source; admins can add a description next to each so future authors know what to pass in.

Workflow emails don't block the pipeline that produces them. When a Send Email step fires, it enqueues a SendEmailJob on a Redis queue (queue:email by default) and moves on. A worker process consumes the queue, resolves the right mail service based on the job's lane, renders the template, and sends via lettre.

Two things happen after every attempt, successful or not: a System Log entry is written with the full details (timestamp, recipient, template slug, rendered HTML, source lane, status), and an exponential backoff kicks in if the send failed transiently.

The System Logs tab is searchable and filterable. Operators can filter by type (email_sent, entity_created, auth_event, …), by resource, by status, and by date range. Clicking a row opens a detail overlay that renders the email HTML in a sanitized container, so it's possible to see exactly what a recipient received without leaving the admin UI.

Logs have a configurable retention (SYSTEM_LOGS_RETENTION_DAYS, default 90) and a purger cron (SYSTEM_LOGS_PURGER_CRON) that deletes expired entries. The same log stream captures admin entity CRUD and auth events, so an operator troubleshooting a delivery problem also has the context of the workflow runs and template edits that led up to it.

Here's the end-to-end setup for a self-hosted RDataCore instance.

1. Pick a mail relay. For local development, run Mailpit as a docker-compose sidecar; it catches outgoing mail in a local web UI without sending anything externally. In production, point the DSNs at your transactional provider (e.g., Postmark, SES, SendGrid) or your own relay.

2. Configure the DSNs. Both are URL-shaped and optional:

• SYSTEM_SMTP_DSN=smtp://user:pass@host:port?tls=true&from=system@example.com&from_name=RData%20System
• WORKFLOW_SMTP_DSN=smtp://user:pass@host:port?tls=true&from=workflow@example.com&from_name=RData%20Workflows

The from query parameter is required; user:pass@ and tls are optional (tls defaults to false). Leave either variable unset if you don't need that lane, and the UI will gate it off automatically.

3. Set FRONTEND_BASE_URL. This is required for the password reset flow; the seeded template uses it to build the {{reset_url}} link, e.g., FRONTEND_BASE_URL=http://admin.rdatacore.docker.

4. Optional tuning. You can adjust PASSWORD_RESET_THROTTLE_SECONDS (default 60), QUEUE_EMAIL_KEY (default queue:email), SYSTEM_LOGS_RETENTION_DAYS (default 90), and SYSTEM_LOGS_PURGER_CRON as your scale requires.

5. Edit the seeded system template. In the admin UI, open System Settings → Email Templates and edit password_reset. Review the subject and bodies, make sure your branding and tone are right, and save.

6. Create the workflow templates you need. Still under Email Templates, click Create Template to define a workflow template. Give it a slug, write the subject and bodies using Handlebars variables, and fill in the description for each variable so future admins understand what to pass in.

8. Verify. Run the workflow, open the System Logs tab, and filter by email_sent. You should see one entry per delivery with the rendered content available in the detail overlay.