This article walks through the thought process behind designing a minimalistic backup script in Bash. While there are robust and sophisticated backup solutions (such as zerobyte or plakar), the goal here is to build something simple and minimal. It also serves as a practical exercise for improving both design thinking and Bash scripting skills.
\nWe won’t start entirely from scratch. Instead, we’ll use the existing todiadiyatmo/bash-backup-rotation-script as a starting point and adapt and iterate it to fit our use case.
\nAs a sample application, we’ll use the latest MyBB forum running on PHP and MySQL, based on nemanjam/mybb-docker, deployed with Docker.
\nLet’s start by clearly defining the requirements the script should fulfill so we can address them properly:
\nThese core requirements are enough to get us started.
\nAt the beginning, we need to make some core decisions about how to structure the code that creates and manages backups, as well as how to organize the folder structure for the application code, backup scripts, and backup files.
\nNote: The terms “local” and “remote” are used relative to the server where the application being backed up is running. “Local” refers to the server’s filesystem, while “remote” refers to machines where backup copies are stored permanently. These are often devices (such as a laptop, Raspberry Pi, or home server) within a local network, so don’t be confused by the terminology.
\nWe prefer having local backups that can be quickly restored without needing to fetch and transfer data from a remote machine. At the same time, we also want true remote backups - synced copies stored on one or more external machines, since we treat our server instance as disposable.
\nThere are two main approaches to this, depending on whether the primary backup script is stored and scheduled locally or on a remote machine:
\nThe first approach is simpler and easier to understand, so we’ll go with that. It also provides a solid foundation if we decide to extend the system later and implement the second approach as well.
\n# Main, local backup script\nbackup-files-and-mysql.sh\n\n# Remote sync script\nbackup-rsync-local.shHowever, both approaches rely on a clear, predefined folder structure, which should be explicitly validated when the script runs.
\nBash scripts rely completely on relative paths, which means the script code and the surrounding folder structure are tightly coupled. A simple and understandable structure is a prerequisite for clean and reliable code.
\nBelow is the expected folder structure for both the server and local backups. In this context, “local” means local to the running application itself, on the same machine and file system. This backup repository serves as the source of truth for all other synchronized, remote backup copies.
\nHere, mybb/ is the application root directory, containing both the application files and the backup. Accordingly, mybb/backup/ is the backup directory, which includes the backup Bash scripts (mybb/backup/scripts/) and the backup data (mybb/backup/data/). The generated .zip archive contains a mysql_database/ folder for the MySQL dump, while adjacent asset folders retain their original names.
This folder structure is mandatory and fixed, as all file paths in the backup-files-and-mysql.sh script are defined relative to the script location (mybb/backup/scripts/).
Another useful detail is that the application files in mybb/ are versioned, including the backup-files-and-mysql.sh script. Since we don’t use a .env file for configuration, the production server contains an unversioned copy named backup-files-and-mysql-run.sh, which includes both the actual variables and executable code. This approach simplifies git pull operations and repository updates. Naturally, all backup data in mybb/backup/data/ is excluded from version control via .gitignore.
# Local (server) backup folder structure:\n#\n# mybb/\n# ├─ backup/\n# │ ├─ scripts/ - backup scripts\n# │ │ ├─ backup-files-and-mysql.sh - versioned\n# │ │ └─ backup-files-and-mysql-run.sh - current script\n# │ └─ data/ - backups data\n# │ ├─ mybb_files_and_mysql-daily-2026-01-20.zip\n# │ │ ├─ inc/\n# │ │ ├─ images/custom/\n# │ │ └─ mysql_database/\n# │ │ └─ mybb.sql\n# │ ├─ mybb_files_and_mysql-daily-2026-01-19.zip\n# │ ├─ mybb_files_and_mysql-weekly-2026-01-14.zip\n# │ └─ mybb_files_and_mysql-monthly-2026-01-01.zip\n# ├── data - Docker volumes\n# │ ├── mybb-data - PHP forum files\n# │ └── mysql-data - database data\n# ├── docker-compose.yml - containers definitions\n# |\n# ...\n# |\n# ├── .gitignore\n# └── README.mdThis is the synchronized backup repository, which is considered remote from the server’s perspective. In most cases, it resides on one of our local machines where backups are stored.
\nAs a synchronized mirror, its folder structure is identical, with one important distinction: it only contains the mybb/backup/ directory and does not include any application files, only the backups themselves. Additionally, this repository is not versioned.
# Remote (synced) backup folder structure:\n#\n# mybb/\n# └─ backup/\n# ├─ scripts/\n# │ └─ backup-rsync-local.sh - current script\n# └─ data/\n# ├─ .gitkeep\n# ├─ mybb_files_and_mysql-daily-2026-01-20.zip\n# │ ├─ inc/\n# │ ├─ images/custom/\n# │ └─ mysql_database/\n# │ └─ mybb.sql\n# ├─ mybb_files_and_mysql-daily-2026-01-19.zip\n# ├─ mybb_files_and_mysql-weekly-2026-01-14.zip\n# └─ mybb_files_and_mysql-monthly-2026-01-01.zipThis is the main script that creates a backup on the server’s local filesystem. It dumps the MySQL database running in Docker into a plain UTF-8 .sql file and also backs up predefined application files and folders. A temporary staging folder is used to create a well-structured .zip archive.
Let’s walk through the main script responsible for creating backups of MySQL database and arbitrary application assets, section by section.
\nEntire script: https://github.com/nemanjam/bash-backup/blob/main/backup-files-and-mysql.sh
\nThese are the real variables that can be freely adjusted to fit a specific application and use case. Normally, they would be defined in a .env file, but for simplicity, they are hardcoded directly in the Bash script. Modifying these values does not require changing the script’s code.
The variables are fairly self-explanatory:
\nDB_* - variables used for connecting to the MySQL database instance we want to back up.LOCAL_BACKUP_DIR - the directory where backup files are stored.SRC_CODE_DIRS - an associative array listing the files and directories to include in the backup.BACKUP_RETENTION_* - the number of daily, weekly, and monthly backups to retain.MAX_RETENTION - the upper limit for any BACKUP_RETENTION_* value.# ---------- Configuration ----------\n\n# MySQL credentials\nDB_CONTAINER_NAME=\"mybb-database\"\nDB_NAME=\"mybb\"\nDB_USER=\"mybbuser\"\nDB_PASS=\"password\"\n\n# Note: all commands run from script dir, NEVER call cd, for relative paths to work\n\n# Dirs paths\n# Local folder is root, all other paths are relative to it\n# script located at ~/traefik-proxy/apps/mybb/backup/scripts\nLOCAL_BACKUP_DIR=\"../data\"\n\n# File or directory\n# Relative to script dir, ../../ returns to: apps/mybb/\ndeclare -A SRC_CODE_DIRS=(\n [\"inc\"]=\"../../data/mybb-data/inc/config.php\"\n [\"images/custom\"]=\"../../data/mybb-data/images/custom\"\n)\n\n# Retention\nMAX_RETENTION=6 # 6 months for monthly backups\nBACKUP_RETENTION_DAILY=3\nBACKUP_RETENTION_WEEKLY=2\nBACKUP_RETENTION_MONTHLY=6These are additional configurable variables used specifically to control logging behavior. Since the backup script runs periodically via cron, proper logging is essential for monitoring, validation, and debugging. The last thing we want is a incorrect or misconfigured script silently producing invalid and unusable backups for months.
\nThe variables are as follows:
\nLOG_TO_FILE - a boolean that determines whether logs are written to a file or output to the terminal.LOG_FILE - the path to the log file.LOG_MAX_SIZE_MB and LOG_KEEP_SIZE_MB - used to prevent unlimited log file growth. LOG_MAX_SIZE_MB a float defining the maximum file size (in MB), after which the log is truncated, while LOG_KEEP_SIZE_MB defines the size to retain after truncation.LOG_TIMEZONE - the time zone used for log timestamps.# ---------- Logging vars ----------\n\n# Enable only when running from cron\n# Cron has no TTY, interactive shell does\nLOG_TO_FILE=false\n[ -z \"$PS1\" ] && LOG_TO_FILE=true\n\n# Log file\nLOG_FILE=\"./log-backup-files-and-mysql.txt\"\n\n# Log size limits (MB, float allowed)\nLOG_MAX_SIZE_MB=1.0 # truncate when log exceeds this\nLOG_KEEP_SIZE_MB=0.5 # keep last N MB after truncation\n\n# Timezone for log timestamps\nLOG_TIMEZONE=\"Europe/Belgrade\"These are constant global variables, defined in a single place and reused throughout the script. Unlike configuration variables, they are not meant to be modified, as they are tightly coupled with the script’s logic. Changing them requires corresponding updates to the code.
\nMYSQL_ZIP_DIR_NAME and FILES_ZIP_DIR_NAME - directory names used inside the archive.ZIP_PREFIX - prefix for backup archive filenames.FREQ_PLACEHOLDER - placeholder string to be replaced with the actual retention frequency in archive names.DATE - date string included in the archive filename.DAY_OF_* - numeric values (e.g., day of week/month) included in archive names.BACKUP_* - boolean flags derived from BACKUP_RETENTION_* variables.SCRIPT_DIR - absolute path of the current script, intended for resolving relative paths (currently unused).# ---------- Constants ----------\n\n# Zip vars\n# Both inside zip\nMYSQL_ZIP_DIR_NAME=\"mysql_database\" \nFILES_ZIP_DIR_NAME=\"source_code\"\n\n# Must match backup-rsync-local.sh\nZIP_PREFIX=\"mybb_files_and_mysql\"\nFREQ_PLACEHOLDER='frequency'\n\nDATE=$(date +\"%Y-%m-%d\")\nZIP_PATH=\"$LOCAL_BACKUP_DIR/$ZIP_PREFIX-$FREQ_PLACEHOLDER-$DATE.zip\"\n\n# Current day and weekday\nDAY_OF_MONTH=$((10#$(date +%d))) # Force decimal, avoid bash octal bug on 08/09\nDAY_OF_WEEK=$((10#$(date +%u))) # 1=Monday … 7=Sunday\n\n# Must do it like this for booleans\nBACKUP_DAILY=$([[ $BACKUP_RETENTION_DAILY -gt 0 ]] && echo true || echo false)\nBACKUP_WEEKLY=$([[ $BACKUP_RETENTION_WEEKLY -gt 0 ]] && echo true || echo false)\nBACKUP_MONTHLY=$([[ $BACKUP_RETENTION_MONTHLY -gt 0 ]] && echo true || echo false)\n\n# Script dir absolute path, unused\n# mybb/backup/scripts\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"At the top of the script, right below the variable and constant definitions, we conditionally call the setup_logging() function based on the LOG_TO_FILE variable. This function redirects both standard output and error output (e.g., from echo commands) to the LOG_FILE.
Additionally, it avoids a common pitfall of infinitely growing log files by truncating the file to LOG_KEEP_SIZE_MB whenever it reaches LOG_MAX_SIZE_MB. This ensures that only the most recent log entries are preserved.
# ---------- Enable logging ----------\n\nsetup_logging() {\n local max_size keep_size\n\n # Convert MB -> bytes (rounded down)\n max_size=$(echo \"$LOG_MAX_SIZE_MB * 1024 * 1024 / 1\" | bc)\n keep_size=$(echo \"$LOG_KEEP_SIZE_MB * 1024 * 1024 / 1\" | bc)\n\n # Ensure log file exists (do not truncate)\n if [ ! -f \"$LOG_FILE\" ]; then\n touch \"$LOG_FILE\"\n fi\n\n # Truncate log if too big\n local size size_mb\n size=$(stat -c%s \"$LOG_FILE\")\n size_mb=$(awk \"BEGIN {printf \\\"%.2f\\\", $size/1024/1024}\") # convert bytes -> MB\n\n if (( size > max_size )); then\n tail -c \"$keep_size\" \"$LOG_FILE\" > \"$LOG_FILE.tmp\" && mv \"$LOG_FILE.tmp\" \"$LOG_FILE\"\n\n # Log truncation message happens before the exec redirection, so it needs its own timestamp\n echo \"$(TZ=\"$LOG_TIMEZONE\" date '+%Y-%m-%d %H:%M:%S') [INFO] Log truncated: original_size=${size_mb}MB, max_size=${LOG_MAX_SIZE_MB}MB, keep_size=${LOG_KEEP_SIZE_MB}MB\" >> \"$LOG_FILE\"\n fi\n\n # Redirect stdout + stderr to log file with timestamps\n exec > >(while IFS= read -r line; do\n echo \"$(TZ=\"$LOG_TIMEZONE\" date '+%Y-%m-%d %H:%M:%S') $line\"\n done >> \"$LOG_FILE\") 2>&1\n\n # Per-run separator — just echo, timestamps added automatically\n echo\n echo \"========================================\"\n echo \"[INFO] Logging started\"\n echo \"[INFO] Log file: $LOG_FILE\"\n echo \"[INFO] Max size: ${LOG_MAX_SIZE_MB}MB, keep: ${LOG_KEEP_SIZE_MB}MB\"\n echo \"========================================\"\n echo\n}\n\nif [ \"$LOG_TO_FILE\" = true ]; then\n setup_logging\nfiBefore executing any other logic, we validate the existence and correctness of the configuration variables and ensure that the environment meets the basic requirements needed to produce a usable and meaningful backup. The following checks are performed:
\n# ---------- Validate config ------------\n\nis_valid_config() {\n local non_zero_found=0\n\n echo \"[INFO] Validating configuration...\"\n\n # Check that MySQL container is running\n if ! docker inspect -f '{{.State.Running}}' \"$DB_CONTAINER_NAME\" 2>/dev/null | grep -q true; then\n echo \"[ERROR] MySQL container not running or not found: DB_CONTAINER_NAME=$DB_CONTAINER_NAME\" >&2\n return 1\n fi\n\n # Check MySQL connectivity inside container\n if ! docker exec \"$DB_CONTAINER_NAME\" \\\n mysql -u\"$DB_USER\" -p\"$DB_PASS\" \"$DB_NAME\" -e \"SELECT 1;\" >/dev/null 2>&1; then\n echo \"[ERROR] MySQL connection failed: container=$DB_CONTAINER_NAME user=$DB_USER db=$DB_NAME\" >&2\n return 1\n fi\n\n # Check local backup directory variable is set, dir exists, and is not root\n if [ -z \"$LOCAL_BACKUP_DIR\" ] || [ ! -d \"$LOCAL_BACKUP_DIR\" ] || [ \"$LOCAL_BACKUP_DIR\" = \"/\" ]; then\n echo \"[ERROR] Local backup directory invalid: path=$LOCAL_BACKUP_DIR\" >&2\n return 1\n fi\n\n # Check source code paths exist (file or directory)\n for path in \"${SRC_CODE_DIRS[@]}\"; do\n if [ ! -e \"$path\" ]; then\n echo \"[ERROR] Source path missing: path=$SCRIPT_DIR/$path\" >&2\n return 1\n fi\n done\n\n # Validate retention values\n for var in BACKUP_RETENTION_DAILY BACKUP_RETENTION_WEEKLY BACKUP_RETENTION_MONTHLY; do\n value=\"${!var}\"\n\n if [[ ! \"$value\" =~ ^[0-9]+$ ]]; then\n echo \"[ERROR] Retention value is not a number: $var=$value\" >&2\n return 1\n fi\n\n if (( value > MAX_RETENTION )); then\n echo \"[ERROR] Retention value too large: $var=$value max=$MAX_RETENTION\" >&2\n return 1\n fi\n\n (( value > 0 )) && non_zero_found=1\n done\n\n if (( non_zero_found == 0 )); then\n echo \"[ERROR] All retention values are zero: daily=$BACKUP_RETENTION_DAILY weekly=$BACKUP_RETENTION_WEEKLY monthly=$BACKUP_RETENTION_MONTHLY\" >&2\n return 1\n fi\n\n # Delete existing temp backup file for this day (idempotent, can run on same day)\n if [[ -f \"$ZIP_PATH\" ]]; then\n rm -f \"$ZIP_PATH\"\n echo \"[WARN] Existing temporary backup file deleted: $ZIP_PATH\"\n fi\n\n echo \"[INFO] Configuration is valid. Creating backup...\"\n\n return 0\n}The create_backup() function is the core of the entire script. It assembles a MySQL database dump and file assets into a structured archive with well-organized relative paths, while also cleaning up any temporary files created during the process.
To build the archive with proper relative paths, we use a temporary STAGING_DIR directory. Inside this directory, the database dump is stored under a folder named by the MYSQL_ZIP_DIR_NAME constant, while file assets are grouped under the FILES_ZIP_DIR_NAME directory.
Using a combination of docker exec and mysqldump, we export the database contents as a plain UTF-8 .sql file and place it into the staging directory. The file is named after the database itself, using the DB_NAME configuration variable.
We then copy all asset files and directories defined in SRC_CODE_DIRS into the staging directory under the FILES_DIR parent folder. If SRC_CODE_DIRS is empty, the entire FILES_DIR directory is removed from the staging area to avoid unnecessary clutter.
Next, we create a .zip archive from STAGING_DIR using a subshell to safely change directories without affecting the main script, which would otherwise break relative path handling. Note that the script is intentionally designed to rely exclusively on relative paths. The resulting archive is saved to the ZIP_PATH location.
Finally, we remove the STAGING_DIR directory regardless of whether archive creation succeeds or fails. This ensures idempotency and prevents leftover temporary files from accumulating.
create_backup() {\n # Note: use staging dir with relative paths to have nice overview in GUI archive utility\n\n # Local scope\n # staging dir: mybb/backup/data/staging_dir\n # temp db dir: mybb/backup/data/staging_dir/mysql_database\n # working dir: mybb/backup/scripts\n local STAGING_DIR=\"$LOCAL_BACKUP_DIR/staging_dir\"\n local TEMP_DB_DIR=\"$STAGING_DIR/$MYSQL_ZIP_DIR_NAME\"\n local FILES_DIR=\"$STAGING_DIR/$FILES_ZIP_DIR_NAME\"\n\n # Reset staging dir from previous broken state \n rm -rf \"$STAGING_DIR\"\n mkdir -p \"$TEMP_DB_DIR\" # Will recreate staging dir\n mkdir -p \"$FILES_DIR\" # Folder to group all source code\n\n echo \"[INFO] Created staging directory: $STAGING_DIR\"\n echo \"[INFO] Created temporary DB directory: $TEMP_DB_DIR\"\n echo \"[INFO] Created files directory: $FILES_DIR\"\n\n # Dump MySQL as plain UTF-8 .sql\n docker exec \"$DB_CONTAINER_NAME\" sh -c \\\n 'mysqldump --no-tablespaces -u\"$DB_USER\" -p\"$DB_PASS\" \"$DB_NAME\"' \\\n > \"$TEMP_DB_DIR/$DB_NAME.sql\"\n\n echo \"[INFO] MySQL database dumped: db_name=$DB_NAME -> path=$TEMP_DB_DIR/$DB_NAME.sql\"\n\n # Copy source code folders grouped into FILES_DIR dir\n for SRC_CODE_DIR in \"${!SRC_CODE_DIRS[@]}\"; do\n SRC_CODE_DIR_PATH=\"${SRC_CODE_DIRS[$SRC_CODE_DIR]}\"\n cp -a \"$SRC_CODE_DIR_PATH\" \"$FILES_DIR/\"\n echo \"[INFO] Added to staging: $SRC_CODE_DIR_PATH -> $FILES_ZIP_DIR_NAME/\"\n done\n\n # Remove FILES_DIR if empty\n if [ -d \"$FILES_DIR\" ] && [ -z \"$(ls -A \"$FILES_DIR\")\" ]; then\n rm -rf \"$FILES_DIR\"\n echo \"[INFO] Removed empty files directory: $FILES_DIR\"\n fi\n\n # Create zip with clean relative paths\n # ( ... ) - subshell, cd wont affect working dir of the main script\n (\n cd \"$STAGING_DIR\" || {\n echo \"[ERROR] Failed to cd into staging directory: $STAGING_DIR\" >&2\n exit 1\n }\n\n # There was cd in subshell\n # Adjust zip path relative to staging_dir\n zip -r \"../$ZIP_PATH\" .\n ) || {\n echo \"[ERROR] Zip creation failed: $ZIP_PATH\" >&2\n rm -rf \"$STAGING_DIR\"\n exit 1\n }\n\n echo \"[INFO] Created zip archive: $ZIP_PATH\"\n\n # Cleanup\n rm -rf \"$STAGING_DIR\"\n echo \"[INFO] Removed staging directory: $STAGING_DIR\"\n echo \"[INFO] Backup file created successfully: $ZIP_PATH\"\n}The create_retention_copies() function manages retention by creating time-based copies (daily, weekly, monthly) of a freshly generated backup file. The original backup file path (ZIP_PATH) includes a placeholder string (frequency), defined by the FREQ_PLACEHOLDER constant.
For each retention option, the function evaluates the current date (e.g., Sunday for weekly, the first day of the month for monthly) and checks whether the corresponding BACKUP_* variable is enabled. If the conditions are satisfied, the original backup is copied and renamed by replacing the placeholder with the appropriate frequency.
To ensure idempotency, the function checks whether a retention copy for the current day already exists. If it does, it is removed before creating a new one, allowing the script to be safely re-run on the same day.
\nFinally, the temporary backup file with the placeholder name is deleted, as it is no longer needed after the retention copies are created.
\ncreate_retention_copies() {\n local IS_WEEKLY=$(( DAY_OF_WEEK == 7 )) # Sunday\n local IS_MONTHLY=$(( DAY_OF_MONTH == 1 )) # First day of month\n\n if [[ ! -f \"$ZIP_PATH\" ]]; then\n echo \"[ERROR] Backup file does not exist: $ZIP_PATH\"\n return 1\n fi\n\n for FREQ in daily weekly monthly; do\n case \"$FREQ\" in\n daily)\n [[ \"$BACKUP_DAILY\" == true ]] || continue\n ;;\n weekly)\n [[ \"$IS_WEEKLY\" -eq 1 && \"$BACKUP_WEEKLY\" == true ]] || continue\n ;;\n monthly)\n [[ \"$IS_MONTHLY\" -eq 1 && \"$BACKUP_MONTHLY\" == true ]] || continue\n ;;\n esac\n\n # Placeholder 'frequency' string replacement\n TARGET_FILE=\"${ZIP_PATH/$FREQ_PLACEHOLDER/$FREQ}\"\n\n # Delete existing backup for this frequency (idempotent, can run on same day)\n if [[ -f \"$TARGET_FILE\" ]]; then\n rm -f \"$TARGET_FILE\"\n echo \"[WARN] Existing $FREQ backup removed: $TARGET_FILE\"\n fi\n\n cp \"$ZIP_PATH\" \"$TARGET_FILE\"\n echo \"[INFO] $FREQ backup copied successfully: $TARGET_FILE\"\n done\n\n rm -f \"$ZIP_PATH\"\n echo \"[INFO] Removed temporary backup file: $ZIP_PATH\"\n}We don’t want to accumulate an unlimited number of backup copies; instead, we delete outdated ones according to the retention limits defined by the BACKUP_RETENTION_* variables.
The prune_old_backups() function enforces these retention limits by removing older backups for each frequency (daily, weekly, monthly).
Based on the BACKUP_RETENTION_* variables, we dynamically calculate the RETENTION integer value for each frequency. If it is zero or unset, we exit early from the loop.
We then list the contents of the LOCAL_BACKUP_DIR and delete outdated copies by processing the output of the ls command through a pipeline:
ZIP_PREFIX and FREQ variables.RETENTION lines, which effectively keeps the RETENTION most recent backups.xargs and rm -R, we delete all remaining filenames line by line, ignoring any error messages to keep logs clean.prune_old_backups() {\n for FREQ in daily weekly monthly; do\n # Determine retention variable dynamically\n RETENTION_VAR=\"BACKUP_RETENTION_${FREQ^^}\" # uppercase: daily -> DAILY\n RETENTION=\"${!RETENTION_VAR}\"\n\n # Skip if retention is zero or unset\n [[ -z \"$RETENTION\" || \"$RETENTION\" -le 0 ]] && continue\n\n # Find old backups and delete them\n ls -t \"$LOCAL_BACKUP_DIR\" \\\n | grep \"$ZIP_PREFIX\" \\\n | grep \"$FREQ\" \\\n | sed -e 1,\"$RETENTION\"d \\\n | xargs -d '\\n' -I{} rm -R \"$LOCAL_BACKUP_DIR/{}\" > /dev/null 2>&1\n\n echo \"[INFO] Pruned $FREQ backups, keeping last $RETENTION\"\n done\n}Finally, we invoke the functions defined above.
\nFirst, we ensure that all required configuration is correct. If validation fails, the script prints an error message to stderr and immediately exits with a non-zero status, preventing any further execution.
\nIf the configuration is valid, the script continues by creating a backup archive, then generating additional retention copies, and finally removing old backups.
\n# ---------- Main script ----------\n\nif ! is_valid_config; then\n echo \"[ERROR] Configuration validation failed. Aborting backup.\" >&2\n exit 1\nfi\n\ncreate_backup\n\ncreate_retention_copies\n\nprune_old_backups\n\necho \"[INFO] Backup completed successfully.\"Below is an example log entry from a successful run when creating a backup:
\n2026-04-07 00:30:01 ========================================\n2026-04-07 00:30:01 [INFO] Logging started\n2026-04-07 00:30:01 [INFO] Log file: ./log-backup-files-and-mysql.txt\n2026-04-07 00:30:01 [INFO] Max size: 1.0MB, keep: 0.5MB\n2026-04-07 00:30:01 ========================================\n2026-04-07 00:30:01 \n2026-04-07 00:30:01 [INFO] Validating configuration...\n2026-04-07 00:30:01 [INFO] Configuration is valid. Creating backup...\n2026-04-07 00:30:01 [INFO] Created staging directory: ../data/staging_dir\n2026-04-07 00:30:01 [INFO] Created temporary DB directory: ../data/staging_dir/mysql_database\n2026-04-07 00:30:01 [INFO] Created files directory: ../data/staging_dir/source_code\n2026-04-07 00:30:01 mysqldump: [Warning] Using a password on the command line interface can be insecure.\n2026-04-07 00:30:02 [INFO] MySQL database dumped: db_name=mybb -> path=../data/staging_dir/mysql_database/mybb.sql\n2026-04-07 00:30:02 [INFO] Added to staging: ../../data/mybb-data/images/custom -> source_code/\n2026-04-07 00:30:02 [INFO] Added to staging: ../../data/mybb-data/inc/config.php -> source_code/\n2026-04-07 00:30:02 adding: mysql_database/ (stored 0%)\n2026-04-07 00:30:02 adding: mysql_database/mybb.sql (deflated 83%)\n2026-04-07 00:30:02 adding: source_code/ (stored 0%)\n2026-04-07 00:30:02 adding: source_code/config.php (deflated 62%)\n2026-04-07 00:30:02 adding: source_code/custom/ (stored 0%)\n2026-04-07 00:30:02 adding: source_code/custom/logo-blue-153x75.png (stored 0%)\n2026-04-07 00:30:02 adding: source_code/custom/logo-blue-588x288.png (deflated 0%)\n2026-04-07 00:30:02 [INFO] Created zip archive: ../data/mybb_files_and_mysql-frequency-2026-04-07.zip\n2026-04-07 00:30:02 [INFO] Removed staging directory: ../data/staging_dir\n2026-04-07 00:30:02 [INFO] Backup file created successfully: ../data/mybb_files_and_mysql-frequency-2026-04-07.zip\n2026-04-07 00:30:02 [INFO] daily backup copied successfully: ../data/mybb_files_and_mysql-daily-2026-04-07.zip\n2026-04-07 00:30:02 [INFO] Removed temporary backup file: ../data/mybb_files_and_mysql-frequency-2026-04-07.zip\n2026-04-07 00:30:02 [INFO] Pruned daily backups, keeping last 3\n2026-04-07 00:30:02 [INFO] Pruned weekly backups, keeping last 2\n2026-04-07 00:30:02 [INFO] Pruned monthly backups, keeping last 6\n2026-04-07 00:30:02 [INFO] Backup completed successfully.Besides the main backup script running directly on the server, we use an additional script that replicates the original backup on remote machines.
\nThis script synchronizes the backups created on the server (which serves as the source of truth) to remote machines used for storing backup copies. It connects to the server via SSH, validates both the backup and local folder structure, and finally synchronizes the data using the rsync command.
Entire script: https://github.com/nemanjam/bash-backup/blob/main/backup-rsync-local.sh
\nSimilarly to the local scripts, these are configurable variables that would typically be defined in a .env file. They are used to configure the synchronization script.
REMOTE_HOST - the server host used for the SSH and rsync connections.REMOTE_BACKUP_DIR - the absolute path (avoid shell expansion) to the backup directory on the server (source of truth).LOCAL_BACKUP_DIR - the relative path to the local directory where backups are synchronized.MIN_BACKUP_SIZE_MB - a human-readable float value (in MB) representing the minimum valid backup size.MIN_BACKUP_SIZE_BYTES - the equivalent integer value in bytes, used for actual validation and computations.# ---------- Configuration ----------\n\nREMOTE_HOST=\"arm2\"\n# Full, absolute path - can't use ~/, used both locally and remote with ssh/rsync\nREMOTE_BACKUP_DIR=\"/home/ubuntu/traefik-proxy/apps/mybb/backup/data\"\n\n# Note: all commands run from script dir, NEVER call cd, for relative LOCAL paths to work\nLOCAL_BACKUP_DIR=\"../data\"\n\n# Minimum valid backup size, ZIP size, compressed\n# Only db for blank forum, zip=158.2 KiB\n# Float\nMIN_BACKUP_SIZE_MB=0.1\n# Integer\nMIN_BACKUP_SIZE_BYTES=$(echo \"$MIN_BACKUP_SIZE_MB * 1024 * 1024 / 1\" | bc) # rounded to integerThe only constant used is ZIP_PREFIX and it must match the one used in backup creation script backup-files-and-mysql.sh.
# ---------- Constants ----------\n\n# Must match backup-files-and-mysql.sh\nZIP_PREFIX=\"mybb_files_and_mysql\"\n\n# Script dir absolute path, unused\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"These are the same as in the Local backup script. The logging format is identical in both the local and remote scripts.
\n# ---------- Logging vars ----------\n\n# Enable only when running from cron\n# Cron has no TTY, interactive shell does\nLOG_TO_FILE=false\n[ -z \"$PS1\" ] && LOG_TO_FILE=true\n\n# Log file\nLOG_FILE=\"./log-backup-rsync-local.txt\"\n\n# Log size limits (MB, float allowed)\nLOG_MAX_SIZE_MB=1.0 # truncate when log exceeds this\nLOG_KEEP_SIZE_MB=0.5 # keep last N MB after truncation\n\n# Timezone for log timestamps\nLOG_TIMEZONE=\"Europe/Belgrade\"Identical as in the local script.
\nSame as in the local script, we validate the existence and correctness of the configuration variables before performing any other logic. In this case, we ensure that:
\n# ---------- Validate config ------------\n\nis_valid_config() {\n echo \"----------------------------------------\"\n echo \"[INFO] Validating configuration\"\n\n # Check SSH connectivity to remote host\n if ! ssh -o BatchMode=yes -o ConnectTimeout=5 \"$REMOTE_HOST\" \"true\" >/dev/null 2>&1; then\n echo \"[ERROR] Cannot connect to remote host via SSH: REMOTE_HOST=$REMOTE_HOST\" >&2\n return 1\n fi\n echo \"[INFO] SSH connection established: REMOTE_HOST=$REMOTE_HOST\"\n\n # Check remote backup directory exists\n if ! ssh \"$REMOTE_HOST\" \"[ -d \\\"$REMOTE_BACKUP_DIR\\\" ]\" >/dev/null 2>&1; then\n echo \"[ERROR] Remote backup directory does not exist: REMOTE_HOST=$REMOTE_HOST REMOTE_BACKUP_DIR=$REMOTE_BACKUP_DIR\" >&2\n return 1\n fi\n echo \"[INFO] Remote backup directory exists: $REMOTE_BACKUP_DIR\"\n\n # Check local backup directory exists\n if [ ! -d \"$LOCAL_BACKUP_DIR\" ]; then\n echo \"[ERROR] Local backup directory does not exist: path=$SCRIPT_DIR/$LOCAL_BACKUP_DIR\" >&2\n return 1\n fi\n echo \"[INFO] Local backup directory exists: $SCRIPT_DIR/$LOCAL_BACKUP_DIR\"\n\n echo \"[INFO] Configuration validation successful\"\n echo \"----------------------------------------\"\n\n return 0\n}Besides validating configuration variables, we also need to validate the remote backup on the server before syncing locally. This is a more complex task, so we break it down into a few smaller, reusable utility functions for clarity and readability.
\nget_latest_date() expects a list of filenames via stdin, extracts the date part in YYYY-MM-DD format, sorts them in descending order, and returns the top item, which is the latest date in the list.split_backup_types() accepts a list of filenames as a single string (e.g. output from ls) as the first argument. The second argument is a mutable associative array passed by reference, used to store the result. The function parses the raw input string and groups backup filenames into categories (daily, weekly, monthly), storing each group in the corresponding key of the associative array.check_count() compares remote and local backup counts (for a given type). If the remote count is lower than the local count, it prints an error and returns failure.check_date() does the same for dates. If the latest remote backup date is older than the latest local backup date, it prints an error and returns failure.bytes_to_human() converts a size in bytes into a human-readable format (KB, MB, GB). It is used to improve log readability.check_file_size() validates that all remote backup files meet a minimum size requirement. It fetches file names and sizes from the remote server via SSH and iterates through each file. For each one, it logs the size and checks whether it is smaller than the minimum allowed size MIN_BACKUP_SIZE_BYTES. If any file is too small, it logs the filename and exits early with an error.# ------------ Utils ------------\n\n# Extract latest YYYY-MM-DD date from backup filenames\nget_latest_date() {\n sed -E 's/.*-([0-9]{4}-[0-9]{2}-[0-9]{2})\\.zip/\\1/' \\\n | sort | tail -n 1\n}\n\n# Split a list of filenames into daily/weekly/monthly assoc array\nsplit_backup_types() {\n local files=\"$1\"\n declare -n arr=$2 # pass assoc array by name\n\n while IFS= read -r file; do\n case \"$file\" in\n *-daily-*.zip) arr[daily]+=\"$file\"$'\\n' ;;\n *-weekly-*.zip) arr[weekly]+=\"$file\"$'\\n' ;;\n *-monthly-*.zip) arr[monthly]+=\"$file\"$'\\n' ;;\n esac\n done <<< \"$files\"\n}\n\n# Ensure remote has at least as many backups as local\ncheck_count() {\n local remote_count=\"$1\"\n local local_count=\"$2\"\n local backup_type=\"$3\"\n\n if (( remote_count < local_count )); then\n echo \"ERROR: remote has fewer type=$backup_type backups than local, remote_count=$remote_count, local_count=$local_count\"\n return 1\n fi\n}\n\n# Ensure remote backups are not older than local\ncheck_date() {\n local remote_latest=\"$1\"\n local local_latest=\"$2\"\n local backup_type=\"$3\"\n\n if [[ -n \"$local_latest\" && \"$remote_latest\" < \"$local_latest\" ]]; then\n echo \"ERROR: remote type=$backup_type backup is older than local, remote_latest=$remote_latest, local_latest=$local_latest\"\n return 1\n fi\n}\n\n# Convert bytes to human-readable format\nbytes_to_human() {\n local size=$1\n if (( size < 1024 )); then\n echo \"${size}B\"\n elif (( size < 1024*1024 )); then\n echo \"$((size/1024))KB\"\n elif (( size < 1024*1024*1024 )); then\n echo \"$((size/1024/1024))MB\"\n else\n echo \"$((size/1024/1024/1024))GB\"\n fi\n}\n\n# Ensure all remote backups are larger than minimum size\ncheck_file_size() {\n local bad_file bad_file_size\n local remote_file size\n local remote_files_info\n\n # Store SSH output in a variable\n remote_files_info=$(ssh \"$REMOTE_HOST\" \"\n for f in $REMOTE_BACKUP_DIR/${ZIP_PREFIX}-*.zip; do\n [ -f \\\"\\$f\\\" ] || continue\n stat -c '%n %s' \\\"\\$f\\\"\n done\n \")\n\n # Iterate over each line in the variable\n while read -r remote_file size; do\n echo \"[INFO] Remote file: $remote_file, size=$(bytes_to_human $size)\"\n\n if (( size < MIN_BACKUP_SIZE_BYTES )); then\n bad_file=\"$remote_file\"\n bad_file_size=\"$size\"\n break\n fi\n done <<< \"$remote_files_info\"\n\n if [[ -n \"$bad_file\" ]]; then\n echo \"ERROR: remote backup file too small: $bad_file, size=$(bytes_to_human $bad_file_size), min=$(bytes_to_human $MIN_BACKUP_SIZE_BYTES)\"\n return 1\n fi\n\n echo \"[INFO] All remote backup files meet minimum size, min=$(bytes_to_human $MIN_BACKUP_SIZE_BYTES)\"\n return 0\n}The is_valid_backup() function is the final check before synchronization. It validates both remote (source of truth) and local backups, compares them for consistency, and ensures that we do not accidentally overwrite the local backup with a corrupted or inconsistent remote backup from the server. It composes the utility functions defined above and adds its own logic:
If any validation fails, the function prints an error and exits early with a non-zero status. If all checks pass, it confirms successful validation and returns success.
\n# ---------- Validation ----------\n\nis_valid_backup() {\n echo \"----------------------------------------\"\n echo \"[INFO] Validating backups\"\n\n # Local variables\n local -A remote_lists local_lists\n local remote_all_files local_all_files\n\n # Loop variables\n local backup_type\n local remote_list local_list\n local remote_count local_count\n local remote_latest local_latest\n\n # Global size validation (run once)\n if ! check_file_size; then\n echo \"ERROR: remote backup contains file(s) smaller than minimum size, min=$(bytes_to_human $MIN_BACKUP_SIZE_BYTES)\"\n return 1\n fi\n echo \"[INFO] Remote backup file sizes validated, min=$(bytes_to_human $MIN_BACKUP_SIZE_BYTES)\"\n\n # Store remote backup filenames in a variable and split, ignores .gitkeep\n remote_all_files=$(ssh \"$REMOTE_HOST\" \"ls -1 $REMOTE_BACKUP_DIR/${ZIP_PREFIX}-*.zip 2>/dev/null\")\n split_backup_types \"$remote_all_files\" remote_lists\n\techo \"[INFO] Remote backup file list loaded for type(s):\"\n\techo \"$remote_all_files\"\n\n # Store local backup filenames in a variable and split\n local_all_files=$(ls -1 \"$LOCAL_BACKUP_DIR/${ZIP_PREFIX}-*.zip\" 2>/dev/null)\n split_backup_types \"$local_all_files\" local_lists\n\techo \"[INFO] Local backup file list loaded:\"\n\techo \"$local_all_files\"\n\n for backup_type in daily weekly monthly; do\n echo \"[INFO] Checking backup type: $backup_type\"\n\n # Set filename lists\n remote_list=\"${remote_lists[$backup_type]}\"\n local_list=\"${local_lists[$backup_type]}\"\n\n # Check counts\n remote_count=$(echo \"$remote_list\" | grep -c . || true)\n local_count=$(echo \"$local_list\" | grep -c . || true)\n if ! check_count \"$remote_count\" \"$local_count\" \"$backup_type\"; then\n echo \"ERROR: backup count mismatch for type=$backup_type: remote=$remote_count is less than local=$local_count\"\n return 1\n fi\n echo \"[INFO] Backup count valid: type=$backup_type remote=$remote_count local=$local_count\"\n\n # Check latest dates\n remote_latest=$(echo \"$remote_list\" | get_latest_date)\n local_latest=$(echo \"$local_list\" | get_latest_date)\n if ! check_date \"$remote_latest\" \"$local_latest\" \"$backup_type\"; then\n echo \"ERROR: latest backup date mismatch for type=$backup_type: remote=$remote_latest is older than local=$local_latest\"\n return 1\n fi\n echo \"[INFO] Latest backup date valid: type=$backup_type date=$remote_latest\"\n done\n\n echo \"[INFO] Backup validation successful\"\n echo \"----------------------------------------\"\n\n return 0\n}Finally, we can invoke the validation functions from above and synchronize remote backup to the local machine.
\nIt first verifies that the configuration is valid, and then that the remote backups are valid. If any validation fails, the script aborts and logs error message.
\nIf both checks pass, it proceeds to mirror the remote backup directory locally using rsync, preserving structure and deleting any local files that no longer exist on the remote.
# ---------- Sync ----------\n\nif ! is_valid_config; then\n echo \"[ERROR] Configuration validation failed. Aborting script.\" >&2\n exit 1\nfi\n\n# Exit early if remote backup is not valid\nif ! is_valid_backup; then\n echo \"ERROR: Backup validation failed - aborting\"\n exit 1\nfi\n\n# Note: no fallback logic for now\n\necho \"[INFO] Remote backup valid - syncing data\"\n\n# Mirror remote data directory locally\nrsync -ah --progress --delete \"$REMOTE_HOST:$REMOTE_BACKUP_DIR/\" \"$LOCAL_BACKUP_DIR/\"\n\necho \"[INFO] Backup sync completed successfully.\"Below is an example log entry from a successful run when synchronizing a backup:
\n2026-04-07 00:45:02 ========================================\n2026-04-07 00:45:02 [INFO] Logging started\n2026-04-07 00:45:02 [INFO] Log file: ./log-backup-rsync-local.txt\n2026-04-07 00:45:02 [INFO] Max size: 1.0MB, keep: 0.5MB\n2026-04-07 00:45:02 ========================================\n2026-04-07 00:45:02 \n2026-04-07 00:45:02 ----------------------------------------\n2026-04-07 00:45:02 [INFO] Validating configuration\n2026-04-07 00:45:03 [INFO] SSH connection established: REMOTE_HOST=arm2\n2026-04-07 00:45:04 [INFO] Remote backup directory exists: / ... /traefik-proxy/apps/mybb/backup/data\n2026-04-07 00:45:04 [INFO] Local backup directory exists: / ... /mybb-backup/scripts/../data\n2026-04-07 00:45:04 [INFO] Configuration validation successful\n2026-04-07 00:45:04 ----------------------------------------\n2026-04-07 00:45:04 ----------------------------------------\n2026-04-07 00:45:04 [INFO] Validating backups\n2026-04-07 00:45:05 [INFO] Remote file: / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-daily-2026-04-05.zip, size=279KB\n2026-04-07 00:45:05 [INFO] Remote file: / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-daily-2026-04-06.zip, size=293KB\n2026-04-07 00:45:05 [INFO] Remote file: / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-daily-2026-04-07.zip, size=285KB\n2026-04-07 00:45:05 [INFO] Remote file: / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-monthly-2026-02-01.zip, size=292KB\n2026-04-07 00:45:05 [INFO] Remote file: / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-monthly-2026-03-01.zip, size=334KB\n2026-04-07 00:45:05 [INFO] Remote file: / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-monthly-2026-04-01.zip, size=332KB\n2026-04-07 00:45:05 [INFO] Remote file: / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-weekly-2026-03-22.zip, size=326KB\n2026-04-07 00:45:05 [INFO] Remote file: / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-weekly-2026-04-05.zip, size=279KB\n2026-04-07 00:45:05 [INFO] All remote backup files meet minimum size, min=102KB\n2026-04-07 00:45:05 [INFO] Remote backup file sizes validated, min=102KB\n2026-04-07 00:45:06 [INFO] Remote backup file list loaded for type(s):\n2026-04-07 00:45:06 / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-daily-2026-04-05.zip\n2026-04-07 00:45:06 / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-daily-2026-04-06.zip\n2026-04-07 00:45:06 / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-daily-2026-04-07.zip\n2026-04-07 00:45:06 / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-monthly-2026-02-01.zip\n2026-04-07 00:45:06 / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-monthly-2026-03-01.zip\n2026-04-07 00:45:06 / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-monthly-2026-04-01.zip\n2026-04-07 00:45:06 / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-weekly-2026-03-22.zip\n2026-04-07 00:45:06 / ... /traefik-proxy/apps/mybb/backup/data/mybb_files_and_mysql-weekly-2026-04-05.zip\n2026-04-07 00:45:06 [INFO] Local backup file list loaded:\n2026-04-07 00:45:06 \n2026-04-07 00:45:06 [INFO] Checking backup type: daily\n2026-04-07 00:45:06 [INFO] Backup count valid: type=daily remote=3 local=0\n2026-04-07 00:45:06 [INFO] Latest backup date valid: type=daily date=2026-04-07\n2026-04-07 00:45:06 [INFO] Checking backup type: weekly\n2026-04-07 00:45:06 [INFO] Backup count valid: type=weekly remote=2 local=0\n2026-04-07 00:45:06 [INFO] Latest backup date valid: type=weekly date=2026-04-05\n2026-04-07 00:45:06 [INFO] Checking backup type: monthly\n2026-04-07 00:45:06 [INFO] Backup count valid: type=monthly remote=3 local=0\n2026-04-07 00:45:06 [INFO] Latest backup date valid: type=monthly date=2026-04-01\n2026-04-07 00:45:06 [INFO] Backup validation successful\n2026-04-07 00:45:06 ----------------------------------------\n2026-04-07 00:45:06 [INFO] Remote backup valid - syncing data\n2026-04-07 00:45:07 receiving incremental file list\n2026-04-07 00:45:07 deleting mybb_files_and_mysql-daily-2026-04-04.zip\n2026-04-07 00:45:07 ./\n2026-04-07 00:45:07 mybb_files_and_mysql-daily-2026-04-07.zip\n2026-04-07 00:45:07 \n 0 0% 0.00kB/s 0:00:00 \n 292.31K 100% 1.91MB/s 0:00:00 (xfr#1, to-chk=5/10)\n2026-04-07 00:45:07 [INFO] Backup sync completed successfully.Now that we have implemented the scripts, we just need to schedule them to run daily by defining cron jobs on the server and on each machine that stores synced copies.
\nAs a reminder, we can list and edit cron jobs using the following commands:
\n# List crons\ncrontab -l\n\n# Edit crons\ncrontab -eWe need to carefully choose times that capture application data near the end of the day in our target time zone. The sync script must run after the backup creation script, so we need to estimate the execution time of the backup process and schedule the sync within a safe margin. With this in mind, we can schedule the backup at 23:30 and the sync at 23:45.
Another important detail is that both scripts rely on relative paths, so cron must execute them from the correct working directory. This can be achieved with cd /.../backup/scripts && bash ./my-script.sh. Additionally, we should use absolute paths in the cron configuration (avoiding shortcuts like ~ for the home directory), as such expansions may fail in a cron environment.
On server:
\n# Create backup every day at 23:30 Belgrade (UTC+2) time (21:30 UTC)\n30 21 * * * cd /home/username/traefik-proxy/apps/mybb/backup/scripts && /usr/bin/bash ./run-backup-files-and-mysql.shOn syncing machines:
\n# Sync backup every day at 23:45 Belgrade (UTC+2) time (21:45 UTC)\n45 21 * * * cd /home/username/mybb-backup/scripts && /usr/bin/bash ./run-backup-rsync-local.shInterestingly, I couldn’t find a reliable way to set a custom time zone for cron jobs. I tried setting the TZ and CRON_TZ variables, but they were ignored, and cron always fell back to UTC.
# None of these actually sets the time zone successfully\n# Always falls back to UTC\n\n# Global\nTZ=Europe/Belgrade\nCRON_TZ=Europe/Belgrade\n\n# Per job\n30 21 * * * TZ=Europe/Belgrade cd /home/username/traefik-proxy/apps/mybb/backup/scripts && /usr/bin/bash ./run-backup-files-and-mysql.sh\n30 21 * * * CRON_TZ=Europe/Belgrade cd /home/username/traefik-proxy/apps/mybb/backup/scripts && /usr/bin/bash ./run-backup-files-and-mysql.shWe described an example implementation along with the thought process behind designing a usable backup script. It is certainly not perfect or final, and it can be enhanced and improved in a number of ways. Here are some possible improvements:
\n.env file.ENABLE_ASSETS boolean flag to enable or disable including application file assets in the backup. Currently, this requires commenting out all keys in the SRC_CODE_DIRS associative array.backup-rsync-local.sh would send and execute the backup-files-and-mysql.sh script remotely and clean up the temporary backup after downloading.The assumption was that we need to track the state of an application, including the database, configuration (source files), and assets (images), and that we need a simple, minimalistic, yet functional solution. Although this is a quite common use case, after searching I couldn’t find a convincing, up-to-date Bash script for this purpose. So, I decided to build upon and adapt the closest existing script I could find. That process is described in this article.
\nThis is a pragmatic, custom script focused on simplicity, with no ambition to become a comprehensive backup solution covering many use cases and features. Such a solution would require a much larger scope of work, and many robust backup tools already exist.
\nHow do you approach creating and managing backups for your applications? Let me know in the comments.
\nMany of you can probably guess the point of this article just by reading the title, but it’s still useful to have a clear reminder backed by some real-world measurements. This will be a practical, straight-to-the-point article.
\nWhen copying the dist folder to a deployment server, the first instinct is usually to clear the existing folder and upload the new one using scp. While this works, you can achieve significant long-term improvements by replacing a few lines and using rsync instead.
The important facts to keep in mind are:
\ndist folder.Clearing the dist folder and scp the entire content each time simply ignores the facts above. scp it is not optimized for repeated copying where most files remain unchanged.
Bash deployment scripts and Github Actions deployment workflows run frequently, so any unnecessary time or performance overhead accumulates and wastes energy and resources. It’s important to optimize as much as possible, especially when it requires very little effort.
\nrsync is designed specifically for efficient file synchronization. Instead of copying everything every time, it compares the source and destination and transfers only the files that have changed.
This dramatically reduces the amount of data that needs to be sent during deployments. In most cases, only a small subset of files changes between builds, which means rsync can complete the transfer much faster than scp.
Another advantage is that rsync can resume partially transferred files and optionally compress data during transfer. These features make it especially well suited for automated deployment workflows where speed and reliability are important.
A typical rsync command used in deployments looks like this:
rsync -az --delete ./dist/ user@server:/var/www/siteSome commonly used flags include:
\n-a (archive) preserves permissions, timestamps, and recursively copies files.-z enables compression during transfer, which reduces network usage.--delete removes files on the destination that no longer exist in the source, keeping the deployment directory in sync.--partial allows interrupted transfers to resume instead of restarting from scratch.Together, these options make rsync a powerful and efficient tool for copying build artifacts during automated deployments.
\nA complete list of options is available in the command’s manual: https://download.samba.org/pub/rsync/rsync.1#OPTION_SUMMARY.
\nFor both scp and rsync, we will consider two examples: a Bash script used to deploy from a local development environment, and a Github Actions workflow. These represent two common approaches to deployments.
For the sake of context and completeness, the full scripts are included so you can reuse them or run your own tests and performance comparisons.
\nNaturally, the only truly important part is the scp line. However, let’s briefly review the rest of the script, since it demonstrates what we would typically use in a real-world scenario.
The first assumption is that we have a local dist folder containing the compiled application built with a local .env file, and an Nginx web server with a webroot directory on a remote server. Our Bash script accepts three input arguments: LOCAL_PATH, REMOTE_PATH, and REMOTE_HOST, which we validate before performing the copy.
Next, we establish an initial ssh connection to delete the existing application artifacts from the previous deployment. During this step, we also log some information by printing the file list and the total number of files in the Nginx webroot before and after removing the old files.
Note 1: When removing old artifacts, we delete the contents of the Nginx webroot directory, not the webroot directory itself. Removing the directory could disrupt the current Nginx session and would require restarting the Nginx process or container.
\nNote 2: Below the scp line, I also include a tar ... | ssh command example that compresses the artifacts before piping them through the SSH connection. In theory, this should provide performance similar to rsync in scenarios where we always completely clear the previous deployment. I will include it in the measurements so we can see how it performs.
# Navigate to ~/traefik-proxy/apps/nmc-nginx-with-volume/website\ncd $REMOTE_PATH\n\n# Clear the contents, not the `/website` path segment\nrm -rf * https://github.com/nemanjam/nemanjam.github.io/blob/main/scripts/deploy-nginx.sh
\n#!/bin/bash\n\nLOCAL_PATH=\"./dist\"\n# REMOTE_PATH=\"~/traefik-proxy/apps/nmc-nginx-with-volume/website\"\n# REMOTE_HOST=\"arm1\"\n\nREMOTE_PATH=$1\nREMOTE_HOST=$2\n\n# Check if all arguments are provided\nif [[ -z \"$REMOTE_PATH\" || -z \"$REMOTE_HOST\" ]]; then\n echo \"Incorrect args, usage: $0 <remote_path> <remote_host>\"\n exit 1\nfi\n\n# Navigate to the website folder on the remote server and clear contents of the website folder\nssh $REMOTE_HOST \"cd $REMOTE_PATH && \\\n echo 'Listing files before clearing:' && \\\n echo 'List before clearing:' && \\\n ls && \\\n echo 'Count before clearing:' && \\\n ls -l | grep -v ^l | wc -l && \\\n\n # Only possible to skip with rsync --delete\n\n echo 'Clearing contents of the folder...' && \\\n rm -rf * && \\\n echo 'List after clearing:' && \\\n ls && \\\n echo 'Count after clearing:' && \\\n find . -type f | wc -l && \\\n\n echo 'Copying new contents...'\"\n\n# Copy new contents, 320 MB\n# Using scp -rq, slowest, not resumable\nscp -rq $LOCAL_PATH/* $REMOTE_HOST:$REMOTE_PATH\n\n# Using tar, fast for cleaned dir\n# tar cf - -C \"$LOCAL_PATH\" . | ssh \"$REMOTE_HOST\" \"tar xvf - -C $REMOTE_PATH\" >/dev/null 2>&1Then we can call the Bash script like this by passing REMOTE_PATH and REMOTE_HOST arguments:
{\n \"scripts\": {\n // ...\n\n \"deploy:nginx:rpi\": \"bash scripts/deploy-nginx.sh '~/traefik-proxy/apps/nmc-nginx-with-volume/website' rpi\",\n \n // ...\n }\n}The Github Actions workflow provides even more context. It includes environment variables required by the application, sets up Node.js and pnpm, and builds the app. The rest is identical to the Bash script above: we use the appleboy/ssh-action action to establish an SSH connection and clear the previous deployment, and the appleboy/scp-action action to copy the built dist/ folder to the remote server using scp.
In the scp step, most arguments are self-explanatory, but one worth emphasizing is strip_components: 1. This prevents creating an additional dist/ path segment inside the Nginx webroot. In other words, we want the files copied to nmc-nginx-with-volume/website/*, not to nmc-nginx-with-volume/website/dist/*.
name: Deploy Nginx scp\n\non:\n push:\n branches:\n - 'main'\n tags:\n - 'v[0-9]+.[0-9]+.[0-9]+'\n pull_request:\n branches:\n - 'disabled-main'\n workflow_dispatch:\n\nenv:\n SITE_URL: 'https://nemanjamitic.com'\n PLAUSIBLE_SCRIPT_URL: 'https://plausible.arm1.nemanjamitic.com/js/script.js'\n PLAUSIBLE_DOMAIN: 'nemanjamitic.com'\n\njobs:\n deploy:\n runs-on: ubuntu-latest\n\n steps:\n - name: Checkout code\n uses: actions/checkout@v4\n with:\n fetch-depth: 1\n\n - name: Print commit id, message and tag\n run: |\n git show -s --format='%h %s'\n echo \"github.ref -> {{ github.ref }}\"\n\n - name: Set up Node.js and pnpm\n uses: actions/setup-node@v4\n with:\n node-version: 24.13.0\n registry-url: 'https://registry.npmjs.org'\n\n - name: Install pnpm\n uses: pnpm/action-setup@v4\n with:\n version: 10.30.1\n\n - name: Install dependencies\n run: pnpm install --frozen-lockfile\n\n - name: Build nemanjamiticcom\n run: pnpm build\n\n - name: Clean up website dir\n uses: appleboy/ssh-action@master\n with:\n host: ${{ secrets.REMOTE_HOST }}\n username: ${{ secrets.REMOTE_USERNAME }}\n key: ${{ secrets.REMOTE_KEY_ED25519 }}\n port: ${{ secrets.REMOTE_PORT }}\n script_stop: true\n script: |\n cd /home/ubuntu/traefik-proxy/apps/nmc-nginx-with-volume/website\n echo \"Content before deletion: $(pwd)\"\n ls -la\n rm -rf ./*\n echo \"Content after deletion: $(pwd)\"\n ls -la\n\n - name: Copy dist folder to remote host\n uses: appleboy/scp-action@v0.1.7\n with:\n host: ${{ secrets.REMOTE_HOST }}\n username: ${{ secrets.REMOTE_USERNAME }}\n key: ${{ secrets.REMOTE_KEY_ED25519 }}\n port: ${{ secrets.REMOTE_PORT }}\n source: 'dist/'\n target: '/home/ubuntu/traefik-proxy/apps/nmc-nginx-with-volume/website'\n # remove /dist path segment\n strip_components: 1Now we modify the existing Bash script and Github Actions workflow by replacing scp with rsync, while keeping the rest of the code identical.
Most of the script remains the same. However, since we use rsync --delete, we can omit the step that deletes the previous deployment. In fact, the initial SSH call is no longer necessary, but we will keep it for debugging and transparency.
Another option worth mentioning is --info=progress2, which is very convenient in a live terminal session because it displays the current transfer progress in a concise way. This provides reassurance that the network connection is active and the transfer is progressing.
https://github.com/nemanjam/nemanjam.github.io/blob/main/scripts/deploy-nginx.sh
\n#!/bin/bash\n\nLOCAL_PATH=\"./dist\"\n# REMOTE_PATH=\"~/traefik-proxy/apps/nmc-nginx-with-volume/website\"\n# REMOTE_HOST=\"arm1\"\n\nREMOTE_PATH=$1\nREMOTE_HOST=$2\n\n# Check if all arguments are provided\nif [[ -z \"$REMOTE_PATH\" || -z \"$REMOTE_HOST\" ]]; then\n echo \"Incorrect args, usage: $0 <remote_path> <remote_host>\"\n exit 1\nfi\n\n# Navigate to the website folder on the remote server and clear contents of the website folder\nssh $REMOTE_HOST \"cd $REMOTE_PATH && \\\n echo 'Listing files before clearing:' && \\\n echo 'List before clearing:' && \\\n ls && \\\n echo 'Count before clearing:' && \\\n ls -l | grep -v ^l | wc -l && \\\n\n # Only possible to skip with rsync --delete\n\n # echo 'Clearing contents of the folder...' && \\\n # rm -rf * && \\\n # echo 'List after clearing:' && \\\n # ls && \\\n # echo 'Count after clearing:' && \\\n # find . -type f | wc -l && \\\n\n echo 'Copying new contents...'\"\n\n# Using rsync, fastest, resumable, deletes without clearing, lot faster with reusing unchanged files (--delete)\nrsync -az --delete --info=stats2,progress2 $LOCAL_PATH/ $REMOTE_HOST:$REMOTE_PATH\n\n# List all files after copying\nssh $REMOTE_HOST \"cd $REMOTE_PATH && \\\n echo 'List after copying:' && \\\n ls && \\\n echo 'Count after copying:' && \\\n find . -type f | wc -l\"The workflow implements the same logic using the Burnett01/rsync-deployments action. Since this is not a live terminal session, --info=stats2 is sufficient for logging.
Unless you are actively debugging, avoid using the rsync -v flag, as overly verbose logs reduce readability.
name: Deploy Nginx rsync\n\non:\n push:\n branches:\n - 'main'\n tags:\n - 'v[0-9]+.[0-9]+.[0-9]+'\n pull_request:\n branches:\n - 'disabled-main'\n workflow_dispatch:\n\nenv:\n SITE_URL: 'https://nemanjamitic.com'\n PLAUSIBLE_SCRIPT_URL: 'https://plausible.arm1.nemanjamitic.com/js/script.js'\n PLAUSIBLE_DOMAIN: 'nemanjamitic.com'\n\njobs:\n deploy:\n runs-on: ubuntu-latest\n\n steps:\n - name: Checkout code\n uses: actions/checkout@v4\n with:\n fetch-depth: 1\n\n - name: Print commit id, message and tag\n run: |\n git show -s --format='%h %s'\n echo \"github.ref -> ${{ github.ref }}\"\n\n - name: Set up Node.js and pnpm\n uses: actions/setup-node@v4\n with:\n node-version: 24.13.0\n registry-url: 'https://registry.npmjs.org'\n\n - name: Install pnpm\n uses: pnpm/action-setup@v4\n with:\n version: 10.30.1\n\n - name: Install dependencies\n run: pnpm install --frozen-lockfile\n\n - name: Build nemanjamiticcom\n run: pnpm build\n\n - name: Deploy dist via rsync\n uses: burnett01/rsync-deployments@v8\n with:\n switches: -az --delete --info=stats2\n path: dist/\n remote_path: /home/ubuntu/traefik-proxy/apps/nmc-nginx-with-volume/website/\n remote_host: ${{ secrets.REMOTE_HOST }}\n remote_user: ${{ secrets.REMOTE_USERNAME }}\n remote_port: ${{ secrets.REMOTE_PORT }}\n remote_key: ${{ secrets.REMOTE_KEY_ED25519 }}For Bash script measurements, I used my local network to deploy to a Raspberry Pi server. I used 1 Gbps Ethernet and 5 GHz, 433 Mbps WiFi 5. For Github Actions workflows, I used the standard Github runners available on the free plan. For each case, I took a few measurements to eliminate random anomalies. I didn’t aim for statistical accuracy.
\nFor deployment, I used this very static Astro website you are currently reading. Its build artifacts consist of 1320 files totaling 347 MB (it contains a number of images).
\n\nLet’s comment on the results, starting from the worst option:
\nscp has the worst performance in every case (Ethernet (9.6 s), WiFi (188 s), Github Actions (43 s)). The WiFi result is especially bad (188 seconds). I don’t have an exact explanation, but the WiFi connection probably doesn’t handle a large number of files well.tar + SSH has decent performance (Ethernet (4.1 s), WiFi (16.6 s), Github Actions (32 s)), considering that it clears the destination and transfers all files every time. Interestingly, on Ethernet it even performs 2× better (4.1 s) than rsync (with synchronization enabled) (8.8 s). I explain this by the fact that hashing and comparing files in rsync can cost more than the file transfer itself on a stable, wired Ethernet connection.rsync (cleared) (delete the destination and transfer everything each time) is on par with tar + SSH (Ethernet (4.6 s), WiFi (15.1 s), Github Actions (29 s)). This makes sense because those two methods are basically doing the same thing.rsync (synchronization enabled) overall has the best performance (Ethernet (8.8 s), WiFi (14.6 s), Github Actions (10 s)), with the exception of Ethernet, which I already explained (hashing and file comparison can cost more than network transfer). The Github Actions result (10 s) is especially important, since CI is the most common way to deploy apps in practice. It also creates around 14× less network traffic (24 MB compared to 347 MB).Meaning, they rank in the following order (from best to worst):
\nrsyncrsync (cleared) and tar + SSH (equally fast)scp (worst in every scenario)Key takeaway: In Github Actions, rsync saves 43 - 10 = 33 seconds on each run compared to scp, which is a significant improvement.
Transferring files is just one of the steps within the deployment process. It is not even the most dominant one. If we look at the times for each step in the Github Actions default__deploy-nginx-rsync.yml workflow, we can see the following:
Set up job 2s\nBuild burnett01/rsync-deployments@v8 9s\nCheckout code 18s\nPrint commit id, message and tag 0s\nSet up Node.js and pnpm 5s\nInstall pnpm 1s\nInstall dependencies 6s\nBuild nemanjamiticcom 2m 25s\nDeploy dist via rsync 10s\nPost Install pnpm 0s\nPost Set up Node.js and pnpm 0s\nPost Checkout code 0s\nComplete job 0sDeploy dist via rsync is third on the list with 10 seconds. Checkout code is second with 18 seconds. That step already has the fetch-depth: 1 optimization; the repository simply has a large file size. The app build step Build nemanjamiticcom obviously takes the most time and has the greatest potential for optimizing performance and saving time. Although obvious, this fact is also formally articulated by Amdahl’s law, which states:
\n\nThe overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used.
\n
However, the app’s build step is also the most complex to optimize. It spans the app code implementation, build configuration, and caching on both Vite and Github Actions levels. Naturally, it is largely app-dependent and more challenging to generalize.
\nIf I look at the Astro build log, I can see this:
\n/_astro/snow1.DTiId6LS_Z2cIRXL.webp (reused cache entry) (+2ms) (1044/1101)This image, snow1.DTiId6LS_Z2cIRXL.webp, has the exact same name in each build and is cached and reused, which drastically improves performance.
On the other hand, in the build log I can also see:
\nλ src/pages/api/open-graph/[...route].png.ts\n ├─ /api/open-graph/blog/2026-01-03-nextjs-server-actions-fastapi-openapi.png (+1.42s) This is an Open Graph image generated using a Satori HTML template, and it is regenerated from scratch on each build. I can see two problems with this:
\nsrc/utils/gradients.ts uses Math.random(), which makes the generation non-deterministic. Instead, the gradient should use a pseudo-random, deterministic approach, for example by hashing the page title string.snow1.DTiId6LS_Z2cIRXL.webp is originally placed inside the src directory, which registers it as an Astro asset. As a result, Astro handles compression, naming, and caching during the build process. This is not the case with the src/pages/api/open-graph/[...route].png.ts static route and the Satori template; additional configuration would be required to enable caching.Anyway, that is a separate topic for a completely different article. In this one, we focus on the file transfer step, which can be optimized with minimal effort - simply by replacing a single command.
\nThe relevant files:
\ngit clone git@github.com:nemanjam/nemanjam.github.io.git\n\n# Bash\nscripts/deploy-nginx.sh\n\n# Github Actions\n.github/workflows/default__deploy-nginx-scp.yml\n.github/workflows/default__deploy-nginx-rsync.ymlYou might think, “This is a pretty long and verbose article for something that could be explained in two sentences” and you would probably be right. However, besides the main rsync vs scp point, I wanted to provide a drop-in script and workflow that you can reuse with minimal changes, just adjust the environment variables, build command, and deployment paths.
Additionally, real-world measurements help provide a realistic sense of how significant the performance improvements can be.
\nWhat methods do you use to optimize the deployment process in your projects? Let me know in the comments.
\nThis article focuses specifically on deploying static websites to Vercel. In a previous article https://nemanjamitic.com/blog/2026-02-22-vercel-deploy-fastapi-nextjs, we covered in detail how to deploy a full-stack application using the Vercel CLI from a local development environment. This time, we will use the same CLI inside a Github Actions runner to automate redeploying a static website on every push, for example, after adding a new blog article in markdown.
\nAs an example, we will deploy the same blog website you are currently reading. The site itself is a statically built Astro application.
\nVercel supports deployments through a Github integration (documented here: https://vercel.com/docs/git/vercel-for-github). You provide Vercel with your Github repository URL and read access, and Vercel automatically redeploys your application on every push. If you prefer not to grant Vercel access to your source code or Github repository, or if you want more control over the deployment process, you can instead use Github Actions, the approach described in this article.
\nAs with any Vercel deployment, you need to provide Vercel with additional information about the project’s build process, such as the framework, build command, and output directory, as well as which files should be included or ignored during deployment.
\nBefore adding any configuration files, go to your Vercel dashboard, create a new project, give it a name, and set all required environment variables.
\nThe contents of the vercel.json file are mostly self-explanatory. We specify the astro framework, and the build command and output directory match those used in the local development environment. With this configuration, Vercel knows exactly how to build the application.
https://github.com/nemanjam/nemanjam.github.io/blob/main/vercel.json
\n{\n \"framework\": \"astro\",\n \"buildCommand\": \"pnpm build\",\n \"outputDirectory\": \"dist\",\n \"cleanUrls\": true,\n \"trailingSlash\": false\n}For performance reasons, it is important to avoid uploading files that are not used during the build and deployment process, such as dependencies, .env* files, documentation, or Docker-related configuration. The .vercelignore file is used to exclude these unnecessary files. Additionally, on the free tier, your deployment must stay below the 250 MB size limit.
https://github.com/nemanjam/nemanjam.github.io/blob/main/.vercelignore
\n# Node / package managers\nnode_modules\n.pnpm-store\n.npm\n.yarn\n\n# ! Needed for commit info\n# Vercel omits it by default, now way to upload it\n# .git\n# .gitignore\n\n# Local env files\n.env\n.env.*\n!.env.*example\n\n# Logs\nnpm-debug.log*\nyarn-debug.log*\nyarn-error.log*\n\n# Docker & tooling\ndocker/\nscripts/\n\n# Documentation & notes\ndocs/\n\n# OS / editor junk\n.DS_Store\n.idea\n.vscode\n\n# Astro build cache\n.astro/*\n# Keep types if build needs them\n!.astro/types.d.ts\n\n# Github\n.github/The exact contents of this file depend on your specific project. To ensure you have excluded all unnecessary paths, go to your Vercel dashboard and navigate to My Project -> My Deployment -> Source, where you can clearly see exactly which files are uploaded.
\nOnce again, go to your Vercel dashboard and create an access token in your account settings. Add this token as the VERCEL_TOKEN Github repository secret. Then, in your Vercel project settings, copy your user (organization) ID and project ID and add them as the VERCEL_ORG_ID and VERCEL_PROJECT_ID Github repository secrets.
With this setup, Github is aware of your Vercel project, and NOT the other way around. Vercel only receives the compiled application artifacts and has no access to your Github repository or source code.
\nhttps://github.com/nemanjam/nemanjam.github.io/blob/main/.github/workflows/vercel__deploy-manual.yml
\nname: Deploy to Vercel manually\n\n# Docs example: https://vercel.com/kb/guide/how-can-i-use-github-actions-with-vercel\n\non:\n push:\n branches:\n - 'main'\n tags:\n - 'v[0-9]+.[0-9]+.[0-9]+'\n\n pull_request:\n branches:\n - 'disabled-main'\n\n workflow_dispatch:\n\npermissions:\n contents: read\n\nenv:\n # Project vars\n # Redundant, vercel pull will define them\n # SITE_URL: 'https://nemanjam.vercel.app'\n # PLAUSIBLE_DOMAIN: 'nemanjamitic.com'\n # PLAUSIBLE_SCRIPT_URL: 'https://plausible.arm1.nemanjamitic.com/js/script.js'\n\n # Vercel vars\n VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }} # user id\n VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID }}\n\njobs:\n deploy-vercel:\n runs-on: ubuntu-latest\n\n steps:\n - name: Checkout code\n uses: actions/checkout@v4\n with:\n fetch-depth: 1\n\n - name: Print commit id and message\n run: |\n git show -s --format='%h %s'\n echo \"github.ref -> ${{ github.ref }}\"\n\n - name: Set up Node.js\n uses: actions/setup-node@v4\n with:\n node-version: 24.13.0\n registry-url: 'https://registry.npmjs.org'\n\n - name: Install pnpm\n uses: pnpm/action-setup@v4\n with:\n version: 10.30.1\n\n - name: Install Vercel CLI\n run: pnpm add -g vercel\n\n - name: Pull Vercel production environment variables\n run: vercel pull --yes --environment=production --token=${{ secrets.VERCEL_TOKEN }}\n\n - name: Build project using Vercel\n run: vercel build --prod --token=${{ secrets.VERCEL_TOKEN }}\n\n - name: Deploy prebuilt project to Vercel\n run: vercel deploy --prebuilt --prod --token=${{ secrets.VERCEL_TOKEN }}Vercel’s official tutorial already does a good job of explaining the basics and provides a solid starting workflow file: https://vercel.com/kb/guide/how-can-i-use-github-actions-with-vercel. In this article, we will focus on a specific use case: deploying a static website.
\nLet’s start by explaining the Github repository secrets used in this workflow:
\nVERCEL_TOKEN - an access token that the Github Actions runner uses to authenticate with Vercel and create a deploymentVERCEL_ORG_ID - a Github user or organization ID that identifies who owns the deploymentVERCEL_PROJECT_ID - identifies the Vercel project being deployedThe VERCEL_ORG_ID and VERCEL_PROJECT_ID values are passed as environment variables and are defined at the workflow level, making them available to all jobs. The VERCEL_TOKEN is passed to individual commands as a command-line argument.
The first part of the workflow is standard and straightforward. We simply check out the repository (fetch-depth: 1 to fetch only the latest commit for speed), then install Node.js, pnpm, and the Vercel CLI. These steps set up the prerequisites needed to build and deploy the project in the following steps.
Here we are referring to your project’s environment variables. Since we are deploying a fully static website, all environment variables are strictly build-time variables, as explained here: https://nemanjamitic.com/blog/2025-12-21-static-website-runtime-environment-variables. The Vercel target environment does not need to define any variables because they are inlined during the build, immutable, and ignored afterward. This also means the build artifacts are specific to the environment they were built for.
\nAlthough variables in the target environment are ignored at runtime, it is still a good practice to define them in the Vercel dashboard and use Vercel as the single source of truth for your deployment. This allows you to easily pull them into the Github Actions runner using: vercel pull --yes --environment=production --token=${{ secrets.VERCEL_TOKEN }}
The --environment=production flag selects the production environment. To deploy to preview environments, you can create a separate workflow .yml file triggered by feature branches (any branch other than main) and use vercel pull with the --environment=preview option to fetch the corresponding variables.
on:\n push:\n branches-ignore:\n - mainNote: You can define your project’s environment variables using the env: key at the workflow or job level, but this is generally not recommended. Doing so will lead to conflicts with variables pulled via vercel pull and issues with overriding priority, unless you are confident in managing them. Relying exclusively on variables from vercel pull ensures clarity and simplicity.
At this point, we are ready to build the project using: vercel build --prod --token=${{ secrets.VERCEL_TOKEN }}. This command generates the application artifacts in the output folder specified in vercel.json, all within the Github Actions runner. After that, the Vercel CLI copies the framework’s output folder (defined in vercel.json) inside the .vercel/output folder, creating a deployment-ready package that can be uploaded directly to Vercel.
The final step is to upload the deployment-ready package inside the .vercel/output folder to Vercel using: vercel deploy --prebuilt --prod --token=${{ secrets.VERCEL_TOKEN }}. The --prebuilt option tells Vercel to skip the build step since the application has already been built in the Github Actions runner.
That’s it. Add the shown vercel.json, .vercelignore, and .github/workflows/vercel__deploy-manual.yml files to your repository, then run git push to trigger the workflow. Once it completes, you can view your website at <your-project-name>.vercel.app.
The relevant files:
\ngit clone git@github.com:nemanjam/nemanjam.github.io.git\n\n# Files\n.github/workflows/vercel__deploy-manual.yml\nvercel.json\n.vercelignore\n\n.dockerignore\n.gitignore\n\n# Vercel configuration and workflow in a clear diff\nhttps://github.com/nemanjam/nemanjam.github.io/commit/c0d6c6739b3215a6841a463115ec5242ea76e492CI/CD workflows are the standard way to handle deployments, and deploying to Vercel is no exception. By combining Github Actions with the Vercel CLI, you can implement a fully automated deployment pipeline with just a few lines of configuration.
\nThis approach gives you complete control over the build and deployment process while keeping your source code private and your security model explicit. Once in place, deployments become predictable, repeatable, and hands-off.
\nHow do you automate deployments to Vercel in your projects? Let me know in the comments.
\nvercel deploy --prebuilt option, Vercel docs https://vercel.com/docs/cli/deploy#prebuiltimport { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;\nimport DeploymentDiagramImage from ’../../../../content/post/2026/02-22-vercel-deploy-fastapi-nextjs/_images/vercel-fastapi-nextjs.png’;\nimport FrontendScreenshotImage from ’../../../../content/post/2026/02-22-vercel-deploy-fastapi-nextjs/_images/frontend-screenshot-1200x630.png’;\nimport BackendScreenshotImage from ’../../../../content/post/2026/02-22-vercel-deploy-fastapi-nextjs/_images/backend-screenshot-1200x630.png’;\nimport VercelButtonWizardScreenshotImage from ’../../../../content/post/2026/02-22-vercel-deploy-fastapi-nextjs/_images/vercel-button-wizard-screenshot.png’;
\nThis article is a practical guide that presents a single, proven approach for deploying a FastAPI and Next.js application on Vercel. It is not intended to be a comprehensive, in depth overview of all Vercel features or deployment options. The main idea is to show how to host demo apps on Vercel for free.
\nPython is one of the runtimes with first-class support on Vercel, as stated in the documentation: https://vercel.com/docs/functions/runtimes. Additionally, FastAPI is an officially supported backend framework and is documented here: https://vercel.com/docs/frameworks/backend/fastapi.
\nNaturally, Next.js is developed by Vercel and has full, native support on the platform. Given all of this, deploying a full-stack FastAPI and Next.js application on Vercel is entirely viable, and we can confidently proceed with the configuration and deployment.
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={DeploymentDiagramImage} alt=“Deployment diagram” />
\nThere are certain constraints on how a FastAPI and Next.js project must be structured and configured in order to be deployable on Vercel. When deploying with Docker, we typically use separate containers for the backend and frontend, since a container is designed to run a single process. Similarly, on Vercel we will use two separate deployments, one for the backend and one for the frontend, and connect them using the SITE_URL and API_URL environment variables.
In general, a full-stack application should be designed without making assumptions about the values of SITE_URL and API_URL, allowing the backend and frontend to be hosted independently on any arbitrary domains.
This is especially important for authentication and cookies, since the frontend and backend will use different *.vercel.app domains. Because these domains are included in the public suffix list (https://publicsuffix.org/list), cookie behavior cannot be fully controlled across them.
One effective solution is to enforce a clean separation of concerns: use FastAPI for authentication logic, and rely on Next.js API routes or server actions for setting and unsetting cookies. I covered this approach in detail in a previous article: https://nemanjamitic.com/blog/2026-02-07-github-login-fastapi-nextjs#architecture-overview.
\nNote: If your project originally hosts the backend and frontend on the same domain and relies on a reverse proxy such as Nginx or Traefik for routing, you will need to reconfigure this setup, as Vercel does not provide that level of routing control.
\nVercel offers two basic ways to deploy a project: 1. using the CLI to deploy from a local machine, and 2. deploying from a repository URL, such as Github or Gitlab. There are also several variations of these approaches, including the “Vercel button” which launches a setup wizard from a repository URL, and Github Actions, where the CLI is used inside a workflow runner.
\nIn this tutorial, we will use the CLI approach and deploy both the backend and frontend directly from our local development machine.
\nWe will begin by installing the Vercel CLI globally on our local machine and then logging in with our Vercel account.
\n# Install Vercel CLI\npnpm install -g vercel\n\n# Log in to Vercel\nvercel loginFirst, we need to expose a FastAPI entry point in a way that can be consumed by a Vercel serverless function.
\n\n# Required by vercel.json to locate the FastAPI `app` instance.\n# `# noqa: F401` prevents formatters from removing this import.\n\nfrom app.main import app # noqa: F401Then we can use it in vercel.json to define the build path for the serverless function and the root request handler for HTTP requests.
{\n \"builds\": [\n {\n \"src\": \"app/api/index.py\",\n \"use\": \"@vercel/python\"\n }\n ],\n \"routes\": [\n {\n \"src\": \"/(.*)\",\n \"dest\": \"app/api/index.py\"\n }\n ]\n}Next, we need to define the backend’s root project directory and specify which files should be included or excluded from the serverless function. This step is important to avoid unnecessary bloat, as it affects both performance and the 250 MB size limit for Vercel functions on the free plan. By default, the root project directory is the folder where you run the vercel CLI command, though you can override it during the deployment prompt.
The .vercelignore file is used to exclude files from being uploaded during deployment. It should be placed inside the root project directory. If you already have a properly configured .dockerignore, these two files are basically the same. Below is my .vercelignore configuration:
# Python\n.venv/\n__pycache__/\n.ruff_cache/\n.mypy_cache/\n.pytest_cache/\napp.egg-info\n*.pyc\n\n# Tests\n.coverage\nhtmlcov\n\nDockerfile*\n\n# ignore database\n!data/database\ndata/database/*\n!data/database/.gitkeepYou can also verify and debug any redundant files in the Vercel dashboard under My Project -> My Deployment -> Source. I highly recommend performing this check to ensure that no unnecessary bloat is included in the serverless function.
Additionally, add the backend/.vercel/ directory to your .gitignore file. This directory contains local Vercel configuration and should not be committed to Git.
# Local Vercel configuration\n.vercel/Your application already relies on certain environment variables that are configured in a specific way. However, their names and formats may not fully align with the predefined variables available in the Vercel environment, so some adaptation is usually required. In my case, this involved the following changes:
\nENVIRONMENT variable that needed to be derived from the predefined VERCEL_ENV variable and then reassigned using a Pydantic model validator.POSTGRES_SERVER, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB. These were replaced with a single DATABASE_URL variable, which is exposed by Neon by default. This value can then be piped into a SQLALCHEMY_DATABASE_URI computed property.\n# Vercel default vars\nVERCEL_ENV: str | None = None\n\n# ...\n\n# for Vercel and Neon\nDATABASE_URL: PostgresDsn | None = None\n\n# Local / Docker fallback\nPOSTGRES_SERVER: str | None = None\nPOSTGRES_PORT: int = 5432\nPOSTGRES_USER: str | None = None\nPOSTGRES_PASSWORD: str | None = None\nPOSTGRES_DB: str | None = None\n\n# ...\n\n@computed_field # type: ignore[prop-decorator]\n@property\ndef SQLALCHEMY_DATABASE_URI(self) -> PostgresDsn:\n # Vercel + Neon\n if self.DATABASE_URL:\n database_url = str(self.DATABASE_URL)\n # Force SQLAlchemy to use psycopg v3 on Vercel (Neon provides postgresql:// by default)\n if database_url.startswith(\"postgresql://\"):\n database_url = database_url.replace(\n \"postgresql://\", \"postgresql+psycopg://\"\n )\n return database_url\n\n # Local / Docker\n if not all(\n [\n self.POSTGRES_SERVER,\n self.POSTGRES_USER,\n self.POSTGRES_PASSWORD,\n self.POSTGRES_DB,\n ]\n ):\n raise ValueError(\n \"Either DATABASE_URL (Vercel/Neon) or POSTGRES_* variables must be set\"\n )\n\n return MultiHostUrl.build(\n scheme=\"postgresql+psycopg\",\n username=self.POSTGRES_USER,\n password=self.POSTGRES_PASSWORD,\n host=self.POSTGRES_SERVER,\n port=self.POSTGRES_PORT,\n path=self.POSTGRES_DB,\n )\n\n# ...\n\n@model_validator(mode=\"after\")\ndef resolve_environment(self) -> Self:\n if self.VERCEL_ENV == \"production\":\n self.ENVIRONMENT = \"production\"\n elif self.VERCEL_ENV == \"preview\":\n self.ENVIRONMENT = \"staging\"\n # else: keep whatever ENVIRONMENT was set from OS/.env\n return self\nAs already noted, SITE_URL is a particularly important variable, as it is the primary way to inform the backend about the domain on which the frontend is hosted. It is used throughout the backend for CORS whitelisting, redirects, constructing absolute URLs, etc.
You may not have this value available during the initial backend deployment. In that case, you can temporarily set a placeholder value to satisfy Pydantic validation. Once the actual frontend URL is known, update the variable via the Vercel dashboard or CLI and redeploy the application for the change to take effect.
\nBelow is the full list of required and optional environment variables used by this backend application:
\n\n# ------------ Required vars --------------\n\n# Frontend url\n# Used by the backend to generate links in emails to the frontend\nSITE_URL=my-frontend-url.vercel.app\n\n# Auth\nJWT_SECRET_KEY=my-secret\nSESSION_SECRET_KEY=my-secret\n\n# Superuser email and password\nFIRST_SUPERUSER=admin@example.com\nFIRST_SUPERUSER_PASSWORD=password\n\n# Postgres database, e.g. Neon\n# Format: postgresql://<username>:<password>@<host>/<database>?<query>\n# Neon example: postgresql://neondb_owner:npg_someHash@ep-solitary-moon-some-hash-pooler.c-3.us-east-1.aws.neon.tech/neondb?sslmode=require\nDATABASE_URL=\n\n# ------------ Optional vars --------------\n\n# Used in email templates and OpenAPI docs\nPROJECT_NAME=\"Full stack FastAPI template Next.js\"\n\n# Whitelisted frontend urls\n# SITE_URL is included by default\nBACKEND_CORS_ORIGINS=\"http://localhost,https://localhost,http://localhost:3000,https://localhost:3000,http://localhost:3001,https://localhost:3001,https://my-frontend-url.vercel.app\"\n\n# Environment: local, staging, production\n# If omitted defaults to VERCEL_ENV\nENVIRONMENT=production\n\n# If omitted defaults to 7 days = 24 * 7 = 168 hours\nACCESS_TOKEN_EXPIRE_HOURS=168\n\n# Github OAuth id and secret\n# Only a single deployment (callback url) per Github app is possible\nGITHUB_CLIENT_ID=\nGITHUB_CLIENT_SECRET=\n\n# Postgres database, e.g. Neon\n# Supply either all POSTGRES_* variables or a single DATABASE_URL\n# If both are defined DATABASE_URL has precedence\nPOSTGRES_SERVER=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_DB=app\nPOSTGRES_USER=postgres\nPOSTGRES_PASSWORD=passwordAt this point, the only remaining step is to obtain a valid DATABASE_URL that points to a migrated, seeded, and running database instance. You can use any cloud or self-hosted PostgreSQL instance, as long as it is accessible over the internet. In this case, we will use Neon, since it is an officially supported PostgreSQL integration on Vercel.
Create an account at https://neon.com, then create a new project and a new database. Next, connect the database to your Vercel project using the Neon integration and use the default production database branch. When the integration is added, Neon will automatically expose certain environment variables to your Vercel deployment, which you can customize in the Neon integration settings. The DATABASE_URL variable is exposed by default, and since we configured the backend to use it in the previous step, no further action is required at this stage.
At this point, the backend can connect to the database, but the database is still empty. We now need to run migrations to create the tables and seed the initial data.
\nNote: You might be tempted to automate database migrations and seeding (especially for demo apps) by running them on application startup. However, since Vercel is a serverless environment, there is no single, well-defined application start event (unlike a VPS or Docker-based setup). Instead, multiple instances can start independently, which would cause migrations and seeds to run multiple times in an unpredictable manner. This can lead to serious issues. For this reason, database migrations and initial seeding should be performed as a single, manual step.
\nWe will use our local development environment, where the application is already fully installed and configured, to run the migrations and seed the data. In the local .env file, simply replace the local development database connection with the remote production database connection that is already used by the Vercel deployment.
Also, make sure to check out the vercel-deploy branch locally, as only that branch is configured to handle the DATABASE_URL variable correctly.
# Checkout vercel-deploy branch that has Vercel configuration (backend/vercel.json, backend/.vercelignore, backend/app/api/index.py, modified backend/app/core/config.py)\ngit checkout vercel-deploy\n# Comment out local Postgres database\n# POSTGRES_SERVER=localhost\n# POSTGRES_PORT=5433\n# POSTGRES_DB=app\n# POSTGRES_USER=postgres\n# POSTGRES_PASSWORD=password\n\n# Neon database url example used on Vercel:\nDATABASE_URL=postgresql://neondb_owner:npg_some-slug@ep-rough-cherry-some-slug-pooler.c-3.us-east-1.aws.neon.tech/neondb?sslmode=requireNow we can run the migrations and seed the Neon remote database. Make sure that all backend dependencies are installed and that your Python virtual environment is activated. Be patient and allow the command to complete, as this process can take a few minutes.
\n# From /backend\ncd ./backend\n\n# Create virtual environment\nuv venv\n\n# Activate the environment\nsource .venv/bin/activate\n\n# Install dependencies\nuv sync\n\n# Migration and seed scripts need activated venv and Python dependencies\n\n# Await db, run migrations and seed (must have .env)\n# With a remote Neon database, this command can take a few minutes to complete\n# Be patient and do not interrupt it\nbash scripts/prestart.shOpen the Neon dashboard and verify that the user and item tables have been created and populated with the seed data. At this point, the backend is fully configured and ready to be deployed using the Vercel CLI.
First, install the vercel CLI and log in with your Vercel account. Then navigate to the backend directory and start the deployment wizard by running the vercel --prod command. The wizard will prompt you to:
./)# Install Vercel CLI\npnpm install -g vercel\n\n# Log in to Vercel\nvercel login\n\n# Navigate to the backend folder\ncd backend\n\n# Deploy for the first time (production)\n# Fill prompts for name, root directory `./` (vercel.json dir)\nvercel --prod\n\n# Add required environment variables (production) (after the wizard completes)\necho \"Full stack FastAPI template Next.js\" | vercel env add PROJECT_NAME production\necho \"https://my-frontend-url.vercel.app\" | vercel env add SITE_URL production\necho \"my-secret\" | vercel env add JWT_SECRET_KEY production\necho \"my-secret\" | vercel env add SESSION_SECRET_KEY production\necho \"admin@example.com\" | vercel env add FIRST_SUPERUSER production\necho \"password\" | vercel env add FIRST_SUPERUSER_PASSWORD production\necho \"postgresql://user:pass@host/db\" | vercel env add DATABASE_URL production\n# Set more optional variables...\n\n# List existing environment variables\nvercel env ls\n\n# Redeploy after changes\nvercel --prod # production\n\n# After deploy, the CLI outputs the URL\n# Example: https://api-full-stack-fastapi-template-nextjs-my-slug.vercel.app\n\n# Debug deployment\nvercel inspect https://api-full-stack-fastapi-template-nextjs-my-slug.vercel.app --jsonTreat the initial deployment as disposable. At this stage, the frontend is likely not deployed yet, which means the SITE_URL value is not available. You can temporarily set a placeholder URL that satisfies Pydantic validation. Once the frontend is deployed, update SITE_URL with the actual value using the CLI (as shown earlier) or via the Vercel dashboard (Project -> Settings -> Environment Variables).
The same approach applies to other environment variables, such as GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET. After updating any environment variable, you must redeploy the project for the changes to take effect.
# Update (remove and add) an existing env var SITE_URL\nvercel env rm SITE_URL production --yes\necho \"https://my-new-frontend-url.vercel.app\" | vercel env add SITE_URL productionOnce the deployment wizard completes successfully, you can verify the result by accessing the FastAPI backend at the URL printed in the terminal, for example: https://api-full-stack-fastapi-template-nextjs-my-slug.vercel.app/docs.
This will display the OpenAPI UI. The exact URL depends on how you named the project. You can view all associated URLs in the Vercel dashboard under the deployment settings.
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={BackendScreenshotImage} alt=“OpenAPI screenshot” />
\nDeploying the Next.js application is much simpler, as it requires fewer environment variables and no database. However, there are still a few details to keep in mind, which we will cover here.
\nWe use a vercel.json file to specify the framework, install and build commands, and the output directory. These settings mirror the local development environment and are pretty self-explanatory.
{\n \"framework\": \"nextjs\",\n \"installCommand\": \"pnpm install\",\n \"buildCommand\": \"pnpm turbo run build --filter=web\",\n \"outputDirectory\": \".next\"\n}Note that our Next.js application uses a monorepo setup with Turbo, which has a few implications:
\nvercel.json file is NOT placed at the monorepo root (frontend/). Instead, it lives in the Next.js application directory at frontend/apps/web/vercel.json. Despite this, the Vercel project root directory IS the monorepo root (frontend/), since all packages and source files must be uploaded in order to build the application successfully. This is the correct approach for deploying monorepo projects to Vercel..vercelignore file should also be placed in the monorepo root directory (frontend/).As with the backend, we need to ignore all unused local files during deployment to prevent unnecessary uploads, performance degradation, and excess bloat, and to stay below the 250 MB limit for a serverless function. Once again, it is a good idea to verify that nothing was missed by checking My Project -> My Deployment -> Source in the Vercel dashboard.
# Node.js\n**/node_modules/\n**/.next/\n**/.turbo/\n**/dist/\n\n# Ignore all env files, env vars are passed into container explicitly\n**/.env*\n\n# Tests\n**/test-results/\n**/playwright-report/\n**/blob-report/\n**/playwright/.cache/Similarly, add the frontend/.vercel/ directory to your frontend .gitignore file. This directory contains local Vercel configuration and should not be committed to Git.
# Local Vercel configuration\n.vercel/The frontend uses far fewer environment variables, only two in fact: API_URL, which points to the backend URL, and SITE_URL, which represents the frontend’s own URL. The value of SITE_URL can be derived from the predefined VERCEL_PROJECT_PRODUCTION_URL variable exposed by Vercel.
# ------------ Required vars --------------\n\n# Backend url\nAPI_URL=https://my-backend-url.vercel.app\n\n# ------------ Optional vars --------------\n\n# Frontend url\n# If omitted defaults to https:// + VERCEL_PROJECT_PRODUCTION_URL\nSITE_URL=https://my-frontend-url.vercel.appWith this in mind, we need to include VERCEL_PROJECT_PRODUCTION_URL in the Next.js environment variable handling logic.
Set it as a fallback value in the next-public-env schema.
export const { getPublicEnv, PublicEnv } = createPublicEnv(\n {\n NODE_ENV: process.env.NODE_ENV,\n SITE_URL: process.env.SITE_URL || `https://${process.env.VERCEL_PROJECT_PRODUCTION_URL}`,\n API_URL: process.env.API_URL,\n },\n { schema: (z) => getProcessEnvSchemaProps(z) }Include it in the process.env type.
declare namespace NodeJS {\n interface ProcessEnv {\n readonly NODE_ENV: 'development' | 'production' | 'test';\n readonly SITE_URL: string;\n readonly API_URL: string;\n readonly VERCEL_PROJECT_PRODUCTION_URL: string;\n }\n}Notify Turborepo.
\n\n{\n \"$schema\": \"https://turbo.build/schema.json\",\n \"ui\": \"tui\",\n \"globalEnv\": [\"NODE_ENV\", \"SITE_URL\", \"API_URL\", \"NEXT_RUNTIME\", \"VERCEL_PROJECT_PRODUCTION_URL\"],\n \n // ...\n\n}That’s it. Our Next.js frontend is ready for deployment.
\nDeploying the frontend is very similar to deploying the backend, with the main difference being the selection of the project root directory, since we are deploying a monorepo. Navigate to the frontend directory and start the deployment wizard by running the vercel --prod command. The wizard will prompt you to:
./apps/web/)# Install Vercel CLI\npnpm install -g vercel\n\n# Log in to Vercel\nvercel login\n\n# Navigate to the frontend folder\ncd frontend\n\n# Deploy for the first time (production)\n# Fill prompts for name, root directory `./apps/web/` (vercel.json dir)\nvercel --prod\n\n# Set required environment variables (production)\necho \"https://api-full-stack-fastapi-template-nextjs-my-slug.vercel.app\" | vercel env add API_URL production\n# Set more optional variables...\n\n# List existing environment variables\nvercel env ls\n\n# Redeploy after changes\nvercel --prod # production\n\n# After deploy, the CLI outputs the URL\n# Example: https://full-stack-fastapi-template-nextjs-my-slug.vercel.app\n\n# Debug deployment\nvercel inspect https://full-stack-fastapi-template-nextjs-my-slug.vercel.app --jsonMake sure to set the environment variables API_URL and SITE_URL (optional), either through the Vercel dashboard (Project -> Settings -> Environment Variables) or via the CLI. If SITE_URL is not set, it will fall back to the predefined VERCEL_PROJECT_PRODUCTION_URL variable, as explained earlier.
Remember to redeploy the project whenever you update environment variables to apply the changes.
\n# Update (remove and add) an existing env vars API_URL and SITE_URL\n\nvercel env rm API_URL production --yes\necho \"https://api-full-stack-fastapi-template-nextjs-my-slug.vercel.app\" | vercel env add API_URL production\n\nvercel env rm SITE_URL production --yes\necho \"https://full-stack-fastapi-template-nextjs-my-slug.vercel.app\" | vercel env add ITE_UR productionIf everything is configured correctly, your full-stack FastAPI and Next.js application should now be fully deployed and functional at the URL displayed in the terminal: https://full-stack-fastapi-template-nextjs-my-slug.vercel.app
The exact URL will depend on the name you chose for the project. Congratulations!
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={FrontendScreenshotImage} alt=“Frontend screenshot” />
\nIn addition to the CLI, Vercel also allows deploying a project by simply specifying the repository URL. Naturally, the repository must be properly prepared and configured beforehand. One variation of this method is the “Vercel Deploy” button, a URL-encoded href link that points to Vercel and includes query parameters for the repository URL, environment variables, integrations, and other necessary configuration details. When clicked, it launches a setup wizard to complete any remaining deployment configuration.
Vercel also provides a utility form to generate these buttons conveniently: https://vercel.com/docs/deploy-button. The “Vercel Deploy” button can be embedded in markdown or HTML pages, allowing visitors to deploy a live demo of your project quickly and effortlessly.
\nWith this in mind, we can define a single “Vercel Deploy” button to deploy both the backend and frontend of our FastAPI and Next.js project. It will clone a single Github repository and create two separate Vercel projects. We can then include the button’s Markdown in the project’s README.md file.
The button below specifies URLs (repository, demo, images, etc.) for this particular example, you can adjust them to match your own URLs.
\nThis is fairly self-explanatory, but let’s clarify the query parameters for completeness:
\nrepository-url - points to the Github repository and the vercel-deploy branch.repository-name - the name of the cloned repository.root-directories - specifies the backend (backend) and frontend root directories (frontend/apps/web).monorepo - indicates that a single Git repository contains multiple Vercel projects.totalProjects - number of Vercel projects to create.Additional parameters include:
\nproject-names - the names the backend and frontend Vercel projects.demo-* - information and URLs displayed in the wizard for demo purposes.The products parameter is particularly important, as it activates the Neon integration in the wizard to provision a new PostgreSQL database.
# URL for the \"Vercel Deploy\" button's href attribute\n\nhttps://vercel.com/new/clone\n?repository-url=https://github.com/nemanjam/full-stack-fastapi-template-nextjs/tree/vercel-deploy\n&root-directories=frontend/apps/web,backend\n&repository-name=full-stack-fastapi-template-with-next-js\n&monorepo=1\n&totalProjects=2\n&project-names=full-stack-fastapi-frontend,full-stack-fastapi-backend\n&demo-description=Build full-stack apps with Next.js and FastAPI.\n&demo-image=https://github.com/nemanjam/full-stack-fastapi-template-nextjs/raw/main/docs/screenshots/frontend-screenshot-1200x630.png\n&demo-title=Full stack FastAPI template with Next.js\n&demo-url=https://full-stack-fastapi-template-nextjs.vercel.app\n&skippable-integrations=1\n&products=[\n {\n \"type\": \"integration\",\n \"integrationSlug\": \"neon\",\n \"productSlug\": \"neon\",\n \"protocol\": \"storage\"\n }\n]Once the URL is constructed, it should be URL-encoded and then embedded in the markdown (or HTML) as shown below.
\n<!-- Markdown for the button -->\n\n[](urlencoded-url-from-above)The button and wizard will clone a single GitHub repository and create and deploy two separate backend and frontend Vercel projects. The deployments are not functional at this point. The wizard will not set any required environment variables for either the backend or frontend projects. Users will need to set the appropriate environment variables themselves in the Vercel dashboard.
\nThe wizard will also create an integration that provisions an unassigned, blank Neon database. Users will then need to assign the integration to the backend project and run migrations and seed the database manually, as described in the previous section: Database migrations and seed.
\nAfter clicking the “Vercel Deploy” button, the user will be taken to a form wizard, as shown below.
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={VercelButtonWizardScreenshotImage} alt=“Vercel button wizard” />
\nNote: You can also use two separate “Vercel Deploy” buttons to clone the backend and frontend as two Github repositories (and deploy them as two Vercel projects). This approach allows you to include the list of environment variables and their default values (env, envDefaults, and envDescription parameters) in the button’s URL.
You can see examples of such buttons at the following links: vercel-button-backend.md, vercel-button-frontend.md
\nThe relevant branch vercel-deploy and files:
git clone git@github.com:nemanjam/full-stack-fastapi-template-nextjs.git\n\n# Checkout the vercel-deploy branch\ngit checkout vercel-deploy\n\n# Backend\nbackend/vercel.json\nbackend/.vercelignore\n\nbackend/app/api/index.py\nbackend/app/core/config.py\n\n.env.vercel.example\ndocs/notes/vercel-deployment-backend.md\n\n# Frontend\nfrontend/apps/web/vercel.json\nfrontend/.vercelignore\n\nfrontend/apps/web/src/config/process-env.ts\nfrontend/apps/web/src/env.d.ts\nfrontend/turbo.json\n\nfrontend/apps/web/.env.vercel.example\ndocs/notes/vercel-deployment-frontend.md\n\n# Compare branches in a clear diff\nhttps://github.com/nemanjam/full-stack-fastapi-template-nextjs/compare/vercel-deploy?expand=1\n\n# Compare specific commits in a clear diff\nhttps://github.com/nemanjam/full-stack-fastapi-template-nextjs/compare/45c840d48cba2aeab07e0a66f8245110b852571e...e5fa4b4af3c19c8c2c584fe437b7298f9e342083Vercel is a quite viable option for hosting full-stack FastAPI and Next.js demo projects for free. It works well when the project is designed around serverless constraints. Keeping the backend and frontend as separate deployments, using environment variables consistently, and handling database migrations manually results in a setup that is simple, predictable, and reliable on the free tier.
\nThis guide focused on one practical approach that is easy to reproduce and debug. With the provided repository and Vercel Deploy buttons, you can use this setup as a solid baseline for demos, prototypes, or small production projects.
\nHave you done something similar yourself and used a different approach? Leave a comment below, I’m happy to hear your opinions.
\nimport { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;\nimport GithubLoginArchitectureImage from ’../../../../content/post/2026/02-07-github-login-fastapi-nextjs/_images/github-login-architecture-16-9.png’;\nimport OAuth2FlowDiagramImage from ’../../../../content/post/2026/02-07-github-login-fastapi-nextjs/_images/oauth2-diagram.png’;
\nIn this article, we will show how to implement Github login in a FastAPI and Next.js application. We use Github in this particular case, but the same approach applies to any OAuth provider, you only need to adjust the FastAPI redirect and callback endpoints. Since this is a Next.js app using server components, we will store the session in an HttpOnly cookie. We will dig into implementation details such as domains, cookies, redirects, and overall structuring to achieve a clean, maintainable, and robust solution.
\nOAuth2 flow sequential diagram: (Source gist)
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={OAuth2FlowDiagramImage} alt=“OAuth2 flow sequential diagram” />
\nLet’s begin with a quick reminder of how OAuth works in very simplified terms. OAuth is built around the principle of a trusted middleman: both we (our app) and the user know who Github is (the authorization server) and trust it. This means we can use Github to identify the user and obtain their information. For the user, this means Github vouches for our app’s identity and legitimacy, clearly showing what information the app will access and at what level, so the user can give informed consent. Of course, there are many more implementation details, but this is enough for a high-level overview.
\nFor our app, this practically means we need to register it with Github, obtain the app’s client ID and client secret, and then, in the backend, use an OAuth client library to implement two endpoints:
\nAdditionally, within the callback endpoint we store the user’s information (email, OAuth ID, name, avatar, etc.) in the database and use the autogenerated database user ID to generate a JWT access token, in the same way we do for a regular email/password authenticated user.
\nIn this way, we achieve a unified interface for authenticating users, regardless of whether they log in with Github or via email/password.
\nArchitecture diagram:
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={GithubLoginArchitectureImage} alt=“Github login architecture diagram” />
\nThe one obvious and constant assumption is that we will use a FastAPI backend and a Next.js frontend. When it comes to authorization, this leaves additional room for deciding how we structure the logic and separate concerns. There is more than one way to do this, and some approaches can be fragile and hard to maintain, which is exactly what we want to avoid.
\nLet’s go straight to the point and explain the optimal approach that we will use, and then briefly touch on some suboptimal alternatives and the kinds of problems they introduce.
\nSince we use Next.js and server components, we will store the session in an HttpOnly cookie so we can have private, server-side rendered pages. This means localStorage is not used for storing authentication state.
The frontend and backend are separate applications, which means they run as separate Node.js and Python processes and are deployed as separate containers (reminder: Docker containers are meant to run a single process per container). This also applies to the domains on which the frontend and backend run. Ideally, we want complete freedom to use any unrelated domains for both, without making assumptions about subdomains, prefixes, or shared domain structure.
\nCookies are tightly coupled to domains: a browser will accept and store a cookie only if it is set by a server running on the correct domain. The fact that Next.js provides API functionality is very fortunate, because those endpoints run on the same domain as the rest of the Next.js app (pages) and can set cookies for that exact domain. This removes the need to deal with cross-domain cookies, which are often complex and fragile.
\nCookies are tied to a domain and are impractical for passing arguments via HTTP responses. Cookies are meant for storing data, not for transmitting it. Consequently, for passing the access_token and expires values, we will use the response body (for server actions) and URL query parameters (for the OAuth redirect). Response bodies and URL query parameters are domain-independent and are designed for passing data between HTTP requests.
Separation of concerns on the backend: FastAPI will contain all backend logic, including authorization. This means it will implement both OAuth endpoints (the redirect and callback endpoints). Next.js will handle cookie setting and unsetting logic: via server actions for email/password login, and via Next.js API routes for Github login. As a reminder, a server action is essentially a POST endpoint under the hood and can set or unset cookies.
\nThe OAuth callback endpoint in FastAPI needs to initiate an uninterrupted redirect chain composed of two steps (FastAPI and Next.js API): Github -> FastAPI callback redirect -> Next.js API redirect -> Next.js home page. During this process, the access_token and expires values need to be passed as query parameters appended to the URL. Redirects are mandatory because the entire flow is driven by the browser, and we do not want the user’s browser to just land on a raw API response, but rather on the website’s home page as a successfully logged-in user.
There is some ambiguity caused by the redundancy of options, which can lead to suboptimal solutions if we do not think clearly enough. Let’s discuss some of them:
\nSince we have two backend-capable frameworks, we might be tempted to move a significant part of the authorization logic into Next.js APIs, for example, implementing the OAuth redirect endpoint, the callback endpoint, or even the email/password authentication endpoints there. This would introduce several serious problems, including unnecessary coupling of two backends to the same database and schema, backend deployment being split across two containers that must stay in strict sync, fragmented configuration and secrets management, violation of the “single source of truth” principle, potential read/write race conditions, and increased debugging and logging complexity.
\nTo prevent all of this, we enforce a clear separation of concerns: FastAPI acts as a complete, standalone backend, while Next.js APIs (and server actions) are responsible only for setting and unsetting cookies. Since Next.js runs on the frontend domain, this approach drastically simplifies and hardens cookie handling.
\nAnother pitfall is relying on cross-domain cookies by making assumptions about the domains used by the frontend and backend. For example, we might assume SITE_URL=https://my-website.com for the frontend and API_URL=https://api.my-website.com for the backend. In that case, the backend could tweak cookie properties such as SameSite=None and Domain=.my-website.com to get the browser to accept and store the cookie.
This introduces additional complexity and fragility into the authentication flow and deployment reliability, along with a number of problems and limitations. Some of them include a major mismatch between email/password login (where the cookie is set directly via a server action) and OAuth login, the inability to host the frontend and backend on completely different, unrelated domains (which is a legitimate requirement), and the inability to host the backend on a PaaS that uses domains included in the public suffix list (https://publicsuffix.org/list/), such as vercel.app.
Once again, this is solved by letting the Next.js API (and server actions) handle setting and unsetting cookies.
\nThat was a lot of text but still no code. On the other hand when we have clear mental model and worked out plan implementation is straight forward.
\nLike with any OAuth provider we need to register our app on Github and obtain client id and client secret. One Github specific is that you can have set only one redirect URL per app, so if you want multiple deployments you will need to create a separate app for each of them.
\nIt’s a straight forward process, go to your Github profile and open the following menus: Github (top-right avatar) -> Settings -> Developer settings (bottom of the left sidebar) -> OAuth Apps -> New OAuth App. Fill in your app info, including redirect URL where you should set the URL of your FastAPI callback endpoint, e.g. https://api.my-website.com/api/v1/auth/github/callback.
Then copy Client ID and Client secret and set inside the backend .env file.
\n# ...\n\nGITHUB_CLIENT_ID=Ov23liasdxhfaOJasdf12\nGITHUB_CLIENT_SECRET=c9ad7bc12977515fed61409492abe169212345\n\n# ...\nWe need to install OAuth client library, we will use authlib/authlib.
\n# Activate venv\nsource .venv/bin/activate\n\n# Install authlib\npoetry add authlibThen we can instantiate OAuth client:
\nGITHUB_OAUTH_CONFIG = {\n \"name\": \"github\",\n \"client_id\": settings.GITHUB_CLIENT_ID,\n \"client_secret\": settings.GITHUB_CLIENT_SECRET,\n \"access_token_url\": \"https://github.com/login/oauth/access_token\",\n \"authorize_url\": \"https://github.com/login/oauth/authorize\",\n \"api_base_url\": \"https://api.github.com/\",\n \"client_kwargs\": {\"scope\": \"user:email\"},\n}\n\n\ndef create_oauth() -> OAuth:\n oauth = OAuth()\n oauth.register(**GITHUB_OAUTH_CONFIG)\n return oauth\n\n\noauth = create_oauth()We can then use the instantiated OAuth client to implement the OAuth redirect and callback endpoints.
\nThe redirect endpoint is quite simple, almost trivial. When the user hits this endpoint, they are redirected to the Github login page, where they can give consent. The redirect_uri variable contains the absolute URL of our callback endpoint, which we define next.
@router.get(\"/login/github\")\nasync def login_github(request: Request):\n \"\"\"\n Redirect to Github login page\n Must initiate OAuth flow from backend\n \"\"\"\n redirect_uri = request.url_for(\"auth_github_callback\") # matches function name\n\n # rewrite to https in production\n if is_prod:\n redirect_uri = redirect_uri.replace(scheme=\"https\")\n\n return await security.oauth.github.authorize_redirect(request, redirect_uri)Now we can define the callback endpoint, which is where Github sends the user after they have logged in on Github. This part is a bit more complex.
\nGithub includes an authorization code as a URL parameter, which we use to obtain an OAuth access token. We then use this token to call two separate Github APIs: one to retrieve the user’s profile information (full name, username, and OAuth ID), and another to retrieve the user’s primary email address. Next, we find or create the user in our database. Finally, we use the user’s database ID to create a JWT token, in exactly the same way as we do for a regular email/password user.
\nNext, we calculate the expires value for the session cookie so that it matches the JWT access_token expiration. We then attach the access_token and expires values as query parameters to the redirect URL. The redirect URL is constructed as f\"{settings.SITE_URL}/api/auth/set-cookie\", pointing to a Next.js API endpoint (which we define next) that is responsible for actually setting the cookie. Finally, we redirect the user.
Once again, it is important to emphasize that the redirect is essential so the browser can follow the entire chain. We do not want the user to land on a raw API response, the home page is the final destination after a successful login.
\n@router.get(\"/auth/github/callback\")\nasync def auth_github_callback(\n request: Request, session: SessionDep\n) -> RedirectResponse:\n \"\"\"\n Github OAuth callback, Github will call this endpoint\n \"\"\"\n # Exchange code for access token\n token = await security.oauth.github.authorize_access_token(request)\n\n # Get user profile Github API\n user_info = await security.oauth.github.get(\"user\", token=token)\n profile = user_info.json()\n\n # Get primary email Github API\n emails = await security.oauth.github.get(\"user/emails\", token=token)\n primary_email = next((e[\"email\"] for e in emails.json() if e[\"primary\"]), None)\n\n logger.info(f\"Primary Github email: {primary_email}\")\n\n # Authenticate or create user\n user = crud.authenticate_github(\n session=session,\n primary_email=primary_email,\n profile=profile,\n )\n\n expires_delta = timedelta(hours=settings.ACCESS_TOKEN_EXPIRE_HOURS)\n\n access_token = security.create_access_token(user.id, expires_delta)\n\n # Absolute expiration timestamp (UTC)\n expires_at = datetime.now(timezone.utc) + expires_delta\n expires_timestamp = int(expires_at.timestamp())\n\n # Build redirect URL to Next.js cookie-setter\n base_url = f\"{settings.SITE_URL}/api/auth/set-cookie\"\n query = urlencode(\n {\n \"access_token\": access_token,\n \"expires\": expires_timestamp,\n }\n )\n redirect_url = f\"{base_url}?{query}\"\n\n response = RedirectResponse(url=redirect_url, status_code=302)\n\n return responseNote the fixed dict type Token used for passing cookie properties. It is important that this type is identical and shared between both OAuth and email/password flows, ensuring that they conform to the same interface.
class Token(SQLModel):\n access_token: str\n # Absolute Date, timestamp, sufficient\n expires: intNow that we have identified the user on Github and created the JWT access_token, the only remaining step is to set the cookie. As mentioned earlier, in the OAuth flow this is done in a Next.js API endpoint.
Below is the complete endpoint implementation. As you can see, it is not too complicated. We simply parse the access_token and expires values from the URL query parameters, use them to construct the cookie, and attach the cookie to a redirect response that sends the user to the home page. This final step sets the cookie, and that’s it.
It’s also worth mentioning that if the query parameters are invalid, we redirect the user back to the login page.
\nNote that we construct the cookie as host-only (domain: undefined), meaning it is valid only for the frontend domain. This is perfectly fine and exactly what we want, since in a Next.js app both pages and APIs run on the same domain.
export const GET = async (request: Request): Promise<Response> => {\n const { SITE_URL, NODE_ENV } = getPublicEnv();\n const isProd = NODE_ENV === 'production';\n\n const url = new URL(request.url);\n\n const accessToken = url.searchParams.get('access_token');\n const expiresParam = url.searchParams.get('expires');\n\n const hasAllData = accessToken && expiresParam;\n if (!hasAllData) {\n const loginUrl = new URL(`${LOGIN}?error=missing_auth_token`, SITE_URL);\n return NextResponse.redirect(loginUrl, { status: 302 });\n }\n\n // Convert Unix timestamp (seconds) to a JS Date object\n const expiresDate = new Date(Number(expiresParam) * 1000);\n\n const redirectUrl = new URL(DASHBOARD, SITE_URL);\n const response = NextResponse.redirect(redirectUrl, { status: 302 });\n\n response.cookies.set({\n name: AUTH_COOKIE,\n\n // passed from backend\n value: accessToken,\n expires: expiresDate,\n\n // frontend-specific\n httpOnly: true,\n secure: isProd,\n\n // host-only\n path: '/',\n sameSite: 'lax',\n domain: undefined,\n });\n\n return response;\n};Although a server action is used to set the session cookie only for the email/password login, it is important to explain the complete authentication picture, show how both login flows conform to the same interface, and highlight some differences.
\nIn contrast to the OAuth flow, which relies on redirects, the email/password login can simply call the FastAPI endpoint LoginService.loginAccessToken({ body }) and obtain the access_token and expires values from the response body to construct the cookie.
Once again, the cookie is host-only (domain: undefined) and included in the server action response. Under the hood, a server action is just a POST request, which effectively sets the cookie.
export const loginAction = async (\n _prevState: ApiResult,\n formData: FormData\n): Promise<ApiResult> => {\n const { NODE_ENV } = getPublicEnv();\n\n const body = Object.fromEntries(formData) as BodyLoginLoginAccessToken;\n const apiResponse = await LoginService.loginAccessToken({ body });\n\n const { response: _, ...result } = apiResponse;\n\n const isSuccess = isSuccessApiResult(result);\n // UI will display backend error\n if (!isSuccess) return result;\n\n const { access_token, expires } = result.data;\n const isProd = NODE_ENV === 'production';\n\n // Convert Unix timestamp (seconds) to a JS Date object\n const expiresDate = new Date(Number(expires) * 1000);\n\n const cookieStore = await cookies();\n\n cookieStore.set({\n name: AUTH_COOKIE,\n // args\n value: access_token,\n expires: expiresDate,\n // local\n httpOnly: true,\n secure: isProd,\n // host-only for exact frontend domain\n path: '/',\n sameSite: 'lax',\n domain: undefined,\n });\n\n // success result is ignored, just for type\n return result;\n};Another server action is used for logout. It simply unsets the cookie and applies to both email/password and OAuth logins, since both rely on the same session cookie.
\nexport const logoutAction = async (): Promise<void> => {\n const cookiesList = await cookies();\n cookiesList.delete(AUTH_COOKIE);\n\n redirect(LOGIN);\n};So what is different between the email/password and OAuth flows?
\nThe OAuth flow is based on two consecutive redirects: FastAPI callback endpoint -> Next.js API set-cookie endpoint -> Home page. This is mandatory because the callback endpoint is the only thing Github provides us, and the access_token and expires values must be passed as query parameters attached to the redirect responses.
In contrast, the email/password flow is based on an HTML form and a server action, which follows a standard request/response pattern. This allows the access_token and expires values to be sent directly in the response body.
The relevant files:
\ngit clone git@github.com:nemanjam/full-stack-fastapi-template-nextjs.git\ngit checkout 45c840d48cba2aeab07e0a66f8245110b852571e\n\n# Backend\nbackend/app/api/routes/login.py\nbackend/app/core/security.py\nbackend/app/crud.py\nbackend/app/models.py\n\n# Frontend container (Next.js)\n# API route\nfrontend/apps/web/src/app/api/auth/set-cookie/route.ts\n# Server action\nfrontend/apps/web/src/actions/auth.ts\n\n# Pull request with most of the code shown in a clear diff\nhttps://github.com/nemanjam/full-stack-fastapi-template-nextjs/pull/4Let’s conclude this article by summarizing the upsides and downsides of choosing to move the cookie-setting logic to Next.js server actions and APIs.
\nUpsides:
\nSITE_URL and API_URL environment variables.vercel.app without needing any additional modifications.Downsides:
\nHave you implemented something similar yourself? What approach did you choose? Let me know in the comments.
\nimport { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;\nimport DeploymentDiagramImage from ’../../../../content/post/2026/01-03-nextjs-server-actions-fastapi-openapi/_images/deployment-diagram-16-8.png’\nimport SequentialDiagramImage from ’../../../../content/post/2026/01-03-nextjs-server-actions-fastapi-openapi/_images/sequential-diagram.png’
\nIn 2026, Next.js and server components are the industry standard for building server-side rendered React websites. Next.js comes with API routes by default for building backend endpoints, but for various reasons you may want to use a completely different programming language and a non-TypeScript backend. For example, FastAPI is known for its excellent integration with ML and AI libraries, which are typically implemented in Python.
\nToday we will show how to implement one critical part of every full-stack app: data fetching and data mutations using Next.js and FastAPI. There is more than one way to do this, but we will aim to choose and implement the best one.
\nWe will not start completely from scratch, but instead reuse https://github.com/fastapi/full-stack-fastapi-template as a starting point. It already provides a solid foundation, especially on the backend, where we will only change session storage from localStorage to an HttpOnly cookie. In contrast, we will replace most of the frontend code by switching from TanStack Router, React Query, and Chakra to Next.js 16, ShadcnUI, and Tailwind CSS v4.
What is the real challenge here? Modern React 19 and Next.js 16 provide new, advanced features and a standardized workflow not only for fetching data into components but also for managing state. These span both the server and the client, and we should aim to fully leverage them.
\nSo the real goal is this: we want to use a non-TypeScript backend while at the same time preserving the well-established server components model for data fetching and the server actions model for data mutations, with the same level of type safety and state management we would have when using Next.js API endpoints.
\nVisual representation of the architecture we will build in this tutorial.
\nDeployment diagram:
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={DeploymentDiagramImage} alt=“Deployment diagram, client JavaScript, Next.js server and FastAPI server” />
\nSequential diagram:
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={SequentialDiagramImage} alt=“Sequential diagram, client JavaScript, Next.js server and FastAPI server” />
\nServer-side rendering and server components are central features of modern Next.js and React. The original starter stores the session in the browser’s localStorage, which prevents us from having server-side rendered private pages and components. We will modify this to store the session in an HttpOnly cookie instead, which we can access and read in server components.
In the original repository, there is already a pull request, Replace plaintext auth tokens with HttpOnly cookies #1606, that implements this. We will reuse it and adapt it to our needs. Let’s highlight the most important parts of this code and discuss them.
\nFirst, let’s create utilities to set and unset the auth cookie in API responses. Note the signature of the set_auth_cookie method: set_auth_cookie(subject, expires_delta, response). We pass in the subject (typically a user.id) and the token’s expiration, as well as a response argument of the base class type Response so that this utility can be applied to any specific response subclass. The create_access_token() method itself remains unchanged; the token is created the same way for both localStorage and cookies.
def set_auth_cookie(\n subject: str | Any, expires_delta: timedelta, response: Response\n) -> Response:\n # Cookie expiration and JWT expiration match\n # Note: cookie expiration must be in seconds\n expires_in_seconds = int(expires_delta.total_seconds())\n access_token = create_access_token(subject, expires_delta)\n\n # Dev defaults\n samesite = \"lax\"\n domain = None\n\n # Prod overrides\n if is_prod:\n samesite = \"none\"\n # Note: important for cross-site cookies in prod to succeed\n # api-site.rpi.example.com and site.rpi.example.com\n parsed = urlparse(settings.SITE_URL)\n domain = parsed.hostname # full domain\n\n # if it has subdomains whitelist cookie for \"1 level less\" subdomain, rpi.example.com\n host_segments = domain.split(\".\")\n if len(host_segments) > 2:\n domain = \".\".join(host_segments[1:]) # remove the first segment (head)\n\n logger.info(f\"domain: {domain}\")\n\n response.set_cookie(\n key=settings.AUTH_COOKIE,\n value=access_token,\n httponly=True,\n max_age=expires_in_seconds,\n expires=expires_in_seconds,\n samesite=samesite,\n secure=is_prod,\n domain=domain,\n )\n return responseThe core idea is straightforward: create a token, assign it to the cookie value, and attach the cookie to the response object.
There is some added cross-site cookie complexity specific to my use case, which you may not necessarily need to replicate, but let’s explain it for the sake of clarity.
\nIn general, in practice, the frontend and backend often don’t share the same domain, and you need to account for this because cookies are tied to a domain. For example, if the frontend is on my-website.com and the backend is on api.my-website.com, the backend must set domain = \"my-website.com\" and samesite = \"none\" for the browser to accept and store the cookie.
On my server, I use an additional Traefik TCP router that treats a dot . as a special delimiter character, which prevents it from correctly routing infinite-depth subdomains. As a result, I had to use a dash - instead for my backend domain.
# frontend url\nhttps://full-stack-fastapi-template-nextjs.arm1.nemanjamitic.com\n\n# backend url\nhttps://api-full-stack-fastapi-template-nextjs.arm1.nemanjamitic.comThe additional cookie domain logic essentially does the following: if the frontend is on a subdomain, it sets the cookie for the parent domain (one level up). For example, in this specific case, arm1.nemanjamitic.com.
That’s enough for this digression, I just wanted to emphasize that you must carefully adjust the domain property of a cross-site cookie depending on the URLs where you host your frontend and backend. Otherwise, the browser will reject the cookie, and authentication will fail.
Similarly, we use this code to unset the auth cookie. We simply return a JSONResponse containing an expired cookie with the same key. It can be improved, but it will suffice for now.
def delete_auth_cookie() -> JSONResponse:\n response = JSONResponse(content={\"message\": \"Logout successful\"})\n response.delete_cookie(\n key=settings.AUTH_COOKIE,\n path=\"/\",\n domain=None,\n httponly=True,\n samesite=\"lax\",\n secure=is_prod,\n )\n return responseNow it’s time to make use of these utilities to log in and log out a user.
\nFor login, we first verify that the user has provided a valid email and password. If the credentials are correct, we use the user.id to generate an access token, set it as the cookie value, and include the cookie in the response using the previously mentioned security.set_auth_cookie() utility.
backend/app/api/routes/login.py
\n@router.post(\"/login/access-token\")\ndef login_access_token(\n session: SessionDep, form_data: Annotated[OAuth2PasswordRequestForm, Depends()]\n) -> JSONResponse:\n \"\"\"\n OAuth2-compatible token login: get an access token for future requests (sent in an HTTP-only cookie)\n \"\"\"\n user = crud.authenticate(\n session=session, email=form_data.username, password=form_data.password\n )\n if not user:\n raise HTTPException(status_code=400, detail=\"Incorrect email or password\")\n elif not user.is_active:\n raise HTTPException(status_code=400, detail=\"Inactive user\")\n\n access_token_expires = timedelta(hours=settings.ACCESS_TOKEN_EXPIRE_HOURS)\n response = JSONResponse(content={\"message\": \"Login successful\"})\n\n return security.set_auth_cookie(user.id, access_token_expires, response)For logout, we simply use the security.delete_auth_cookie() utility to unset the cookie from the user’s browser.
Note: we are implementing this endpoint for the sake of completeness in the FastAPI backend. In our particular setup, however, we will use the Next.js server to clear the auth cookie.
\nbackend/app/api/routes/login.py
\n@router.post(\"/logout\", dependencies=[Depends(get_current_user)])\ndef logout() -> JSONResponse:\n \"\"\"\n Delete the HTTP-only cookie during logout\n \"\"\"\n return security.delete_auth_cookie()After implementing login and logout, we can use them to protect specific API endpoints by identifying the user from the request and checking whether they have sufficient privileges to access a resource or perform an action.
\nWe will use FastAPI’s dependency injection to centralize the logic for obtaining the auth cookie with CookieDep and for identifying the user who sent the cookie with CurrentUser. The get_current_user() method checks for the existence of the cookie, decodes and verifies the validity of the access token from the cookie, and finally uses the user.id from the token’s subject to query the user from the database.
cookie_scheme = APIKeyCookie(name=settings.AUTH_COOKIE)\n\nCookieDep = Annotated[str, Depends(cookie_scheme)]\n\n\ndef get_current_user(session: SessionDep, cookie: CookieDep) -> User:\n if not cookie:\n raise HTTPException(\n status_code=status.HTTP_401_UNAUTHORIZED,\n detail=\"Not authenticated\",\n )\n\n try:\n payload = jwt.decode(\n cookie, settings.JWT_SECRET_KEY, algorithms=[security.ALGORITHM]\n )\n token_data = TokenPayload(**payload)\n except (InvalidTokenError, ValidationError):\n raise HTTPException(\n status_code=status.HTTP_403_FORBIDDEN,\n detail=\"Could not validate credentials\",\n )\n\n user = session.get(User, token_data.sub)\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n if not user.is_active:\n raise HTTPException(status_code=400, detail=\"Inactive user\")\n\n return user\n\nCurrentUser = Annotated[User, Depends(get_current_user)]\nNow, any function or route in FastAPI can make use of the CurrentUser dependency to identify the user simply by including it as a typed argument. Below is a simple /me endpoint for illustration.
backend/app/api/routes/users.py
\n@router.get(\"/me\", response_model=UserPublic)\ndef read_user_me(current_user: CurrentUser) -> Any:\n \"\"\"\n Get current user.\n \"\"\"\n return current_userWith this HttpOnly cookie setup, authentication on the backend is complete. Essentially, we didn’t change much, we only moved the access token from the JSON body of the response to the cookie header.
\nNow that we have moved the session from localStorage to an HttpOnly cookie, we can identify the user in server components on the Next.js server. An important and obvious note worth repeating: HttpOnly cookies are accessible only on the Next.js server, not in the client-side JavaScript running in the browser.
Since our setup involves Next.js client -> Next.js server -> FastAPI server, we need to handle cookie transmission accordingly. We will continue using the @hey-api/openapi-ts package from the original template, but for our Next.js app, we will use the @hey-api/client-next client and configure it to handle the auth cookie properly.
Here is the configuration for generating the OpenAPI client:
\nfrontend/apps/web/openapi-ts.config.ts
\nconst config: HeyApiConfig = defineConfig({\n input: './openapi.json',\n output: {\n format: 'prettier',\n lint: 'eslint',\n path: './src/client',\n importFileExtension: null,\n },\n exportSchemas: true, // backend models types\n plugins: [\n // Note: order matters\n {\n name: '@hey-api/typescript',\n enums: 'javascript', // const objects instead of enums\n },\n '@hey-api/schemas', // default json, req.body, '{\"username\":\"abc\",\"password\":\"123\"}'\n {\n name: '@hey-api/sdk',\n asClass: true, // UsersService.readUserMe(), 'true' doesn't allow tree-shaking\n classNameBuilder: '{{name}}Service', // class Users -> UsersService\n // @ts-expect-error @hey-api/openapi-ts doesn't export types\n methodNameBuilder, // usersReadUserMe -> readUserMe\n },\n {\n name: '@hey-api/client-next',\n // relative from src/client/ folder\n runtimeConfigPath: '../lib/hey-api', // sets API_URL, auth...\n },\n ],\n});The important parts are: we choose the Next.js client @hey-api/client-next, control the SDK method names using the methodNameBuilder callback, and specify the path for the generated client runtime configuration with runtimeConfigPath.
There are several ways to set runtime configuration for the generated client. We will use the runtimeConfigPath field to specify the path to the configuration file, as this is the recommended approach according to the docs: https://heyapi.dev/openapi-ts/clients/next-js#runtime-api.
The auth cookie is originally stored in the browser and is included in client requests by default. An important note is that the cookie in these requests is valid only for the frontend domain (Next.js server) and not for the backend domain. If you try calling the FastAPI backend using this cookie, you will get a 403 Forbidden HTTP error. That’s why requests from client code need to be proxied through a Next.js API endpoint to the FastAPI backend.
In contrast, HTTP requests made from the Next.js server do not include the cookie automatically at all, so we need to forward it explicitly.
\nWe handle both of these requirements in a base runtime configuration at the SDK instance level in a single place, so we don’t have to repeat this logic for every specific call.
\nHere is the code:
\nfrontend/apps/web/src/lib/hey-api.ts
\nconst { CLIENT_PROXY } = ROUTES.API;\n\n/** Runtime config. Runs and imported both on server and in browser. */\nexport const createClientConfig: CreateClientConfig = (config) => {\n const { API_URL } = getPublicEnv();\n\n return {\n ...config,\n baseUrl: API_URL,\n credentials: 'include',\n fetch: isServer() ? serverFetch : clientFetch,\n };\n};\n\nconst serverFetch: typeof fetch = async (input, init = {}) => {\n // Note: Dynamic import to avoid bundling 'next/headers' on client\n const { cookies } = await import('next/headers');\n\n const cookieStore = await cookies();\n const cookieHeader = cookieStore\n .getAll()\n .map((c) => `${c.name}=${c.value}`)\n .join('; ');\n\n // Note: must append auth_cookie like this or content-type header will break in server actions\n const headers = new Headers(init.headers);\n headers.append('Cookie', cookieHeader);\n\n // test skeletons styling\n // await waitMs(3000);\n\n const response = fetch(input, { ...init, headers });\n\n return response;\n};\n\n/** Client-side fetch: forwards requests to api/client-proxy/[...path]/route.ts */\nconst clientFetch: typeof fetch = async (input, init = {}) => {\n const { API_URL } = getPublicEnv() as { API_URL: string };\n\n // hey-api sends absolute URL\n const url: string = typeof input === 'string' ? input : input.toString();\n\n // Normalize to relative URL\n // API_URL.length + 1 - removes leading slash, API_URL guaranteed not to have trailing slash\n const relativeUrl = url.startsWith(API_URL) ? url.slice(API_URL.length + 1) : url;\n\n // Build the proxy URL relative to Next.js API\n const proxyUrl = `${CLIENT_PROXY}${relativeUrl}`;\n\n const headers = new Headers(init.headers);\n\n return fetch(proxyUrl, { ...init, headers, credentials: 'include' });\n};\nThe important part here is the serverFetch override for the fetch function used by the generated client on the Next.js server. The serverFetch code typically runs either on a server-side rendered page or in a server action. Both of these have access to the request object (including headers and cookies) that originally comes from the browser.
Based on this, we read the cookies (including the auth cookie) and forward them to the fetch client, which makes requests from the Next.js server to the FastAPI endpoints. This creates the following chain: browser -> Next.js server -> FastAPI server.
There are a few additional tricks required to make this work correctly.
\nconst { cookies } = await import('next/headers'), to prevent bundling server-only code into the client bundle, which would break the build.new Headers() instance and use the append() method to pass cookies to fetch.API_URL environment variable using const { API_URL } = getPublicEnv() from the next-public-env package. For a reusable build, API_URL must be a runtime variable. Since this code runs both on the server and in the client, it must get the correct value in both environments. I wrote about reusable Next.js builds and runtime environment variables in detail in this article: Runtime environment variables in Next.js - build reusable Docker images.With this OpenAPI client, we can make HTTP authenticated calls from Next.js pages, server components, and server actions to protected FastAPI endpoints.
\nExample 1. Query in a page:
\nfrontend/apps/web/src/app/dashboard/items/[[…page]]/page.tsx
\n\nconst ItemsPage: FC<Props> = async ({ params }) => {\n const { page } = await params;\n const { currentPage, isValidPage } = parsePage(page);\n\n if (!isValidPage) notFound();\n\n const result = await ItemsService.readItems({\n query: {\n skip: (currentPage - 1) * PAGE_SIZE_TABLE,\n limit: PAGE_SIZE_TABLE,\n },\n });\n\n const items = result.data;\n\n // ...\n\n}Example 2. Query in a server component:
\nfrontend/apps/web/src/components/dashboard/home/list-recent-items.tsx
\nconst ListRecentItems: FC = async () => {\n const result = await ItemsService.readItems();\n\n throwIfApiError(result);\n\n const items = result.data?.data ?? [];\n\n // ...\n\n}Example 3. Mutation in a server action:
\nfrontend/apps/web/src/actions/item.ts
\nexport const itemCreateAction = async (\n _prevState: ApiResult,\n formData: FormData\n): Promise<ApiResult> => {\n const body = Object.fromEntries(formData) as ItemCreate;\n\n const apiResponse = await ItemsService.createItem({ body });\n\n const { response: _, ...result } = apiResponse;\n\n revalidatePath(ITEMS);\n\n return result;\n};The clientFetch implementation rewrites the original URL pointing to FastAPI to a Next.js API endpoint that serves as a proxy. With this, every call from the client code is routed to the Next.js API proxy endpoint.
The Next.js API endpoint then parses the authorization cookie and attaches it to the actual request to the corresponding FastAPI endpoint. It does the same for the response by forwarding the cookie from the FastAPI response back to the Next.js client code. In practice, it acts as a proxy middleman that attaches the cookie in both directions.
\nAs a result, HTTP requests from the client code can be authenticated correctly with FastAPI instead of being rejected with a 403 Forbidden error.
Note: This call is not displayed in the architecture diagram, it needs to be updated.
\nfrontend/apps/web/src/app/api/client-proxy/[…path]/route.ts
\nconst proxyHandler = async (request: NextRequest, { params }: ClientProxyRouteParam) => {\n const { path } = await params;\n\n const { API_URL } = getPublicEnv();\n\n // params.path just splits into array, this just joins back to string\n // path = ['api', 'v1', 'users', 'me'] -> http://api.localhost:8000/api/v1/users/me\n const backendUrl = `${API_URL}/${path.join('/')}`;\n\n // clone headers from client request\n const headers: Record<string, string> = {};\n request.headers.forEach((value, key) => (headers[key] = value));\n\n const { method } = request;\n const body = ['GET', 'HEAD'].includes(method) ? undefined : await request.arrayBuffer();\n\n const apiResponse = await fetch(backendUrl, { method, headers, body });\n\n const data = await apiResponse.arrayBuffer();\n const response = new NextResponse(Buffer.from(data), { status: apiResponse.status });\n\n // copy headers from backend response, excluding ones automatically controlled by Node/Next.js internally\n const excludeHeaders = ['content-encoding', 'transfer-encoding', 'connection'];\n apiResponse.headers.forEach(\n (value, key) => !excludeHeaders.includes(key) && response.headers.set(key, value)\n );\n\n // optional: return error if response not ok\n if (!apiResponse.ok) {\n console.warn('Client proxy returned error:', apiResponse.status, backendUrl);\n }\n\n return response;\n};\n\n// add PUT, DELETE, PATCH if needed\n\nexport const GET = proxyHandler;\n\nexport const POST = proxyHandler;Example 1. Query in a page’s client component:
\nfrontend/apps/web/src/app/page.tsx
\n'use client';\n\n// ...\n\nconst { LOGIN, DASHBOARD } = ROUTES;\nconst { HOME_PAGE_REDIRECT } = DELAY;\n\n// Must be client component to show loader before redirect\n\nconst HomePage: FC = () => {\n const router = useRouter();\n\n useEffect(() => {\n const redirect = async () => {\n // client side call, cookie valid only for Next.js API domain, and not for FastAPI domain\n const result = await UsersService.readUserMe();\n const currentUser = result.data;\n\n await waitMs(HOME_PAGE_REDIRECT);\n\n const redirectUrl = currentUser ? DASHBOARD : LOGIN;\n router.push(redirectUrl);\n };\n\n redirect();\n }, [router]);\n\n return (\n <div className=\"flex items-center justify-center min-h-screen\">\n <div className=\"text-center\">\n <div className=\"animate-spin rounded-full h-8 w-8 border-b-2 border-gray-900 mx-auto\"></div>\n <p className=\"mt-2 text-gray-600\">Redirecting...</p>\n </div>\n </div>\n );\n};The original idea of this article is to preserve the default React workflow, even when using a non-TypeScript backend. To achieve this, the client code must not be aware of the FastAPI endpoints and should never call them directly, but only through the Next.js server, which will forward the requests.
\nWe have already implemented passing the cookie from the browser to FastAPI via the Next.js server, but we have not yet implemented the opposite direction. The FastAPI login endpoint sets the cookie in the response; now we need to forward it through the Next.js server to the user’s browser. Here is the code that accomplishes this:
\nfrontend/apps/web/src/utils/actions.ts
\nexport const forwardCookiesFromResponse = async (response: Response): Promise<void> => {\n const rawCookies = response.headers.get('set-cookie');\n if (!rawCookies) return;\n\n const parsed = setCookieParser.parse(rawCookies);\n const cookieStore = await cookies();\n\n for (const c of parsed) {\n cookieStore.set({\n name: c.name,\n value: c.value,\n httpOnly: c.httpOnly,\n secure: c.secure,\n path: c.path ?? '/',\n sameSite: c.sameSite as any,\n expires: c.expires,\n });\n }\n};The utility method above retrieves the cookies from the FastAPI response object, parses them using the set-cookie-parser package, and finally attaches the parsed cookies to the Next.js response that will be returned as part of the server action.
We then call this utility method within the login server action, which sets the forwarded cookies in the user’s browser. Under the hood, a server action is just a POST request, and it can set cookies just like any other HTTP call.
\nfrontend/apps/web/src/actions/auth.ts
\nexport const loginAction = async (\n _prevState: ApiResult,\n formData: FormData\n): Promise<ApiResult> => {\n const body = Object.fromEntries(formData) as BodyLoginLoginAccessToken;\n const apiResponse = await LoginService.loginAccessToken({ body });\n\n const { response, ...result } = apiResponse;\n await forwardCookiesFromResponse(response);\n\n return result;\n};Now that we have configured the backend and HTTP client, it is time to handle form submission within our client code. For some time, react-hook-form has been the dominant forms package in the React ecosystem. There are a few tricks to integrate it properly with the useActionState React API and server actions.
Here is example code for creating an item:
\nfrontend/apps/web/src/components/dashboard/items/form-item-create.tsx
\n\nconst defaultValues: ItemCreateFormValues = {\n title: '',\n description: '',\n} as const;\n\nconst resolver = zodResolver(itemCreateSchema);\n\nconst FormItemCreate: FC<Props> = ({ onSuccess, onCancel }) => {\n const initialState = { data: undefined };\n const [state, formAction, isPending] = useActionState(itemCreateAction, initialState);\n\n const form = useForm<ItemCreateFormValues>({ resolver, defaultValues });\n\n const isSuccess = isSuccessApiResult(state);\n\n useEffect(() => {\n if (isSuccess) onSuccess?.();\n }, [isSuccess, onSuccess]);\n\n const isError = isErrorApiResult(state);\n\n const validateAndSubmit = (event: FormEvent<HTMLFormElement>) => {\n event.preventDefault();\n\n form.handleSubmit(() => {\n const formElement = event.target as HTMLFormElement;\n const formData = new FormData(formElement);\n\n startTransition(() => {\n formAction(formData);\n form.reset();\n });\n })(event);\n };\n\n return (\n <Form {...form}>\n <form action={formAction} onSubmit={validateAndSubmit} className=\"space-y-6\">\n <FormField\n control={form.control}\n name=\"title\"\n render={({ field }) => (\n <FormItem>\n <FormLabel>Title *</FormLabel>\n <FormControl>\n <Input {...field} placeholder=\"Enter item title...\" disabled={isPending} />\n </FormControl>\n <FormMessage />\n </FormItem>\n )}\n />\n\n <FormField\n control={form.control}\n name=\"description\"\n render={({ field }) => (\n <FormItem>\n <FormLabel>Description</FormLabel>\n <FormControl>\n <Input {...field} placeholder=\"Enter item description...\" disabled={isPending} />\n </FormControl>\n <FormMessage />\n </FormItem>\n )}\n />\n\n {isError && (\n <Alert variant=\"destructive\">\n <AlertDescription>{getApiErrorMessage(state.error)}</AlertDescription>\n </Alert>\n )}\n\n <div className=\"flex justify-end space-x-2\">\n <Button type=\"button\" variant=\"outline\" onClick={onCancel} disabled={isPending}>\n Cancel\n </Button>\n\n <Button type=\"submit\" disabled={isPending}>\n {isPending ? (\n <>\n <Loader2 className=\"mr-2 h-4 w-4 animate-spin\" />\n Creating...\n </>\n ) : (\n 'Create Item'\n )}\n </Button>\n </div>\n </form>\n </Form>\n );\n};The code above addresses two important requirements:
\nServer actions are designed to support form submission even with JavaScript disabled. In our code, we support this by assigning the action attribute on the <form /> tag to the formAction result returned by the useFormAction hook: <form action={formAction} ... />.
We use client-side JavaScript to validate the fields with Zod and display user-friendly error messages. For this, as well as for form submission when JavaScript is enabled, we use the validateAndSubmit event handler attached to the onSubmit event: <form onSubmit={validateAndSubmit} ... />.
I learned this trick from the tutorial linked in this GitHub discussion comment: react-hook-form/discussions/11832#discussioncomment-11832211.
\nReact provides the useActionState hook not only to handle form submission and call server actions, but also to manage state, all at the same time. It accepts the server action itemCreateAction as an argument and returns the formAction function, which can be used to bind to the form’s action attribute, invoked manually with an event handler, or both at the same time, as we do in this case.
const [state, formAction, isPending] = useActionState(itemCreateAction, initialState);State management is modeled in a way that is somewhat reminiscent of reducers in Redux. It uses the concept of previous and next state, where the next state is the result of applying a transformation to the previous state, in this case, the transformation occurs on the server. Naturally, it all starts with an initial state.
\nIn our example, useActionState accepts initialState as the second argument, and the server action response is contained within the state item in the returned tuple.
There are certain rules that apply to React server actions in general:
\nIn server actions, you shouldn’t throw exceptions but instead return errors within the result. This affects which particular generic we will use to type the OpenAPI client response, because in our case a server action is just a proxy to the respective FastAPI endpoint.
\nHere is the ItemsService.createItem() return type as an example. This type is reused for both FastAPI and the server action. This is intentional because the server action is just a proxy, and we want to avoid any unnecessary transformation of the results.
As mentioned earlier, errors should be included as return values. This type clearly shows that: the result is a union of success and error branches and additionally contains the raw HTTP Response object. You may recall that we already used it to extract the cookie in the login endpoint.
Promise<({\n data: ItemPublic;\n error: undefined;\n} | {\n data: undefined;\n error: HttpValidationError;\n}) & {\n response: Response;\n}>Server actions have limited serialization capabilities. You cannot return class instances, error instances, database model instances, etc. You should mostly rely on object literals for return values from server actions.
\nLet’s see the code example:
\nfrontend/apps/web/src/actions/item.ts
\nexport const itemCreateAction = async (\n _prevState: ApiResult,\n formData: FormData\n): Promise<ApiResult> => {\n const body = Object.fromEntries(formData) as ItemCreate;\n\n const apiResponse = await ItemsService.createItem({ body });\n\n const { response: _, ...result } = apiResponse;\n\n revalidatePath(ITEMS);\n\n return result;\n};Note that the response field is a class instance and is not serializable, which is why we omit it from the server action return value.
You may recall that we bound the action attribute to formAction in the form: <form action={formAction} ... >. Because of this, form values are received as FormData in the server action, and we use Object.fromEntries() to convert it to a plain object that can be forwarded as the HTTP request body to the FastAPI endpoint.
Just a quick reminder: useActionState is not the only way to call a server action. It is typically used with a form element, while also providing state and an isPending flag. We have already seen a code example for this.
When we have a simple void server action, we can skip the form and simply invoke the action within an event handler. Such calls are typically marked as lower priority by wrapping them with startTransition. Below is a code example:
frontend/apps/web/src/components/dashboard/items/dropdown-item.tsx
\nconst DropdownItem: FC<Props> = ({ item }) => {\n const [isPending, startTransition] = useTransition();\n\n const handleDeleteItem = (userId: string) => {\n startTransition(() => {\n itemDeleteAction(userId);\n });\n };\n\n // ...\n\n};The relevant files:
\ngit clone git@github.com:nemanjam/full-stack-fastapi-template-nextjs.git\ngit checkout be2b94b72b343563d21aeac29743099af8512f62\n\n# Backend\nbackend/app/core/security.py\nbackend/app/api/deps.py\nbackend/app/api/routes/login.py\nbackend/app/api/routes/users.py\n\n# Frontend\n\n# OpenAPI configuration\nfrontend/apps/web/openapi-ts.config.ts\nfrontend/apps/web/src/lib/hey-api.ts\n\n# Queries - Next.js server\nfrontend/apps/web/src/app/dashboard/items/[[...page]]/page.tsx\nfrontend/apps/web/src/components/dashboard/home/list-recent-items.tsx\n\n# Queries - Next.js client\nfrontend/apps/web/src/app/page.tsx\n\n# Next.js API proxy endpoint\nfrontend/apps/web/src/app/api/client-proxy/[...path]/route.ts\n\n# Mutations\n\n# Server actions\nfrontend/apps/web/src/utils/actions.ts\nfrontend/apps/web/src/actions/item.ts\nfrontend/apps/web/src/actions/auth.ts\n\n# Forms\nfrontend/apps/web/src/components/dashboard/items/form-item-create.tsx\nfrontend/apps/web/src/components/dashboard/items/dropdown-item.tsxCongratulations on reading this far. It was long to read, but also long to write and figure out.
\nNext.js is a comprehensive full-stack meta-framework, but there are situations where you might want to use it with a backend written in a completely different programming language. In this tutorial, we have demonstrated that it is entirely possible to bridge the modern Next.js ecosystem with a powerful Python-based FastAPI backend while preserving the full benefits of React Server Components and Server Actions. This approach shows that you don’t have to compromise on developer experience or type safety when using a non-TypeScript backend.
\nThe key idea is that Next.js remains the only thing the browser ever talks to. FastAPI is treated as an internal service behind the Next.js server. This keeps a clean React mental model: components call server actions, server actions call strongly typed OpenAPI clients, and only the Next.js server handles cookies, authentication, and cross-service communication. From the client’s perspective, nothing changes.
\nThe result is a stack where FastAPI does what it does best (Python, validation, data, ML, background jobs) and Next.js does what it does best (React, rendering, routing, and UX), without either one leaking into the other.
\nFeel free to explore the complete implementation in the accompanying repository. It serves as a solid starting point for anyone building AI-powered or data-intensive applications that require the rich AI/ML ecosystem and library availability of Python on the backend, along with the developer experience of modern React on the front.
\nHave you faced similar challenges and used a different approach? Feel free to share in the comments. I would love to hear about your experience.
\n@hey-api/openapi-ts configuration docs https://heyapi.dev/openapi-ts/configuration@hey-api/client-next client docs https://heyapi.dev/openapi-ts/clients/next-jsimport { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;\nimport OpenGraphBakedUrlImage from ’../../../../content/post/2025/12-21-static-website-runtime-environment-variables/_images/open-graph-baked-url.png’;
\nThis article is a live, “try and see” practical experiment. I will use this exact blog project, a static Astro website, and try to package it as a reusable Nginx Docker image that requires just a single .env file to run in any environment.
I will use this tutorial as a starting point: https://phase.dev/blog/nextjs-public-runtime-variables/. It describes the idea and the process, uses Next.js, and includes a shell replacement script that we can work with.
\nLet’s define our goal and requirements at the beginning. We will use a pure static website that consists only of assets, without any server-side runtime code. This is important because hosting a static website is simple, free, and widely available. We want reusable builds where no environment-specific data is bundled into the application code, but instead read from a single .env file.
Now it’s time to identify which data is environment-specific. In this particular website, these are the four environment variables:
\n\n# some example values\n\nSITE_URL=http://localhost:8080\nPLAUSIBLE_SCRIPT_URL=https://plausible.arm1.nemanjamitic.com/js/script.js\nPLAUSIBLE_DOMAIN=nemanjamitic.com\nPREVIEW_MODE=trueNotice the format of these variables: (almost) three URLs and one boolean value.
\nNow let’s clearly understand what makes this challenging. Once compiled, a static website becomes just a collection of static assets (.html, .js, .css, .jpg, etc.) deployed to an Nginx web folder. This practically means there is no server runtime and we cannot run any code, which greatly limits our power and control. The only runtime we have is the browser runtime, which is only useful to a limited extent when it comes to loading environment-specific data at runtime.
Before we start following the original tutorial and move on to the practical implementation, let’s reconsider the possible alternatives at our disposal. I already touched on this in my previous article about runtime environment variables: https://nemanjamitic.com/blog/2025-12-13-nextjs-runtime-environment-variables#alternative-approaches, but let’s go through them once again, since this article is entirely dedicated to runtime variables in purely static websites. Here are the alternatives:
\nThe first option is the original idea from the tutorial: using a shell script with the sed command to string-replace all placeholder values that were inlined at build time directly in the bundle. We will include this script in the Docker entrypoint so it can run when the container starts and insert environment-specific values. This way, we achieve an effect similar to start-time variables.
Nginx has certain features for injecting environment variables into responses. For example, sub_filter can perform string replacement in each text response. The upside of this approach is that the variables are truly runtime and will reflect any change immediately, but the major disadvantage is the performance overhead, especially under heavy traffic.
Another method is to rely on the JavaScript runtime in the browser and dynamically host and load <script src=\"./env.js\"></script> in the HTML of your root layout. It can have a few variations. For example, you can create an env.template file that holds placeholders for the replacement script called in the entrypoint, or you could even inline env.js directly in the nginx.conf itself:
location = /env.js {\n default_type application/javascript;\n return 200 \"window.__CONFIG__ = { API_URL: '$API_URL' }\";\n}The common aspect of all these approaches is that you need to run client-side JavaScript on a given page to access runtime variables through the `window` object, for example `window.__RUNTIME_ENV__.MY_VAR`. This is a disadvantage on its own and comes with a performance cost. For example, Astro enforces a __zero client-side JavaScript by default__ strategy precisely for performance reasons.\n\nAnother deal-breaker is that files which do not run client-side JavaScript cannot access these variables. Examples include `sitemap.xml` and `robots.txt`, which are very important SEO-related files, especially for static websites.And finally, the most important and interesting part: the practical implementation of the most promising alternative. We have spent quite a lot of time on the introduction and the overview of alternatives.
\nYou can review the exact implementation in the pull request at this link: https://github.com/nemanjam/nemanjam.github.io/pull/28
\nLet’s start with the shell replacement script. We will use the replace-variables.sh script from the tutorial as a starting point. After some trial and error and a few iterations, this is what the final script looks like:
#!/bin/sh\n\n# Note: sh shell syntax, NO bash in Alpine Nginx\n\n# Summary:\n# 1. Required variables are checked to be defined.\n# 2. Optional variables are initialized to empty string if undefined.\n# 3. All files in DIST_PATH with specified extensions are processed.\n# 4. Placeholders of the form PREFIX_VAR are replaced with actual environment variable values.\n\n# Define required and optional environment variables (space-separated strings for /bin/sh)\nREQUIRED_VARS=\"SITE_URL PLAUSIBLE_SCRIPT_URL PLAUSIBLE_DOMAIN\"\nOPTIONAL_VARS=\"PREVIEW_MODE\"\n\n# Variables that are baked as URL-shaped placeholders (https://BAKED_VAR)\nBAKED_URL_VARS=\"SITE_URL PLAUSIBLE_SCRIPT_URL\"\n\nPREFIX=\"BAKED_\"\n# Baked has always https://BAKED_VAR\n# Will be replaced with whatever VAR value http:// or https:// or any string\nURL_PREFIX=\"https://${PREFIX}\"\nFILE_EXTENSIONS=\"html js xml json\"\n\n# Read DIST_PATH from environment variable\n# Do not provide a default; it must be set\nif [ -z \"${DIST_PATH}\" ]; then\n echo \"ERROR: DIST_PATH environment variable is not set.\"\n exit 1\nfi\n\n# Check if the directory exists\nif [ ! -d \"${DIST_PATH}\" ]; then\n echo \"ERROR: DIST_PATH directory '${DIST_PATH}' does not exist.\"\n exit 1\nfi\n\n# Check required environment variables are defined\nfor VAR in $REQUIRED_VARS; do\n # POSIX sh-compatible indirect expansion\n eval \"VAL=\\$$VAR\"\n if [ -z \"$VAL\" ]; then\n echo \"$VAR required environment variable is not set. Please set it and rerun the script.\"\n exit 1\n fi\ndone\n\n# Default optional variables to empty string\nfor VAR in $OPTIONAL_VARS; do\n eval \"VAL=\\$$VAR\"\n if [ -z \"$VAL\" ]; then\n eval \"$VAR=''\"\n fi\ndone\n\n# Combine required and optional variables into a single string\nALL_VARS=\"$REQUIRED_VARS $OPTIONAL_VARS\"\n\n# Find and replace placeholders in files\nfor ext in $FILE_EXTENSIONS; do\n\n # Use 'find' to recursively search for all files with the current extension\n # -type f ensures only regular files are returned\n # -name \"*.$ext\" matches files ending with the current extension\n find \"$DIST_PATH\" -type f -name \"*.$ext\" |\n\n # Pipe the list of found files into a while loop for processing\n while read -r file; do\n # Read file once into a variable for faster checks\n FILE_CONTENT=$(cat \"$file\")\n FILE_REPLACED=0\n\n # Loop over each variable that needs to be replaced\n for VAR in $ALL_VARS; do\n\n PLACEHOLDER=\"${PREFIX}${VAR}\"\n URL_PLACEHOLDER=\"${URL_PREFIX}${VAR}\"\n\n # Get variable value (POSIX sh compatible)\n # Optional variables are guaranteed to have a value (possibly empty)\n eval \"VALUE=\\$$VAR\"\n\n # Escape VALUE for sed replacement:\n # - & → \\& (ampersand is special in replacement, expands to the whole match)\n # - | → \\| (pipe is used as sed delimiter)\n ESCAPED_VALUE=$(printf '%s' \"$VALUE\" | sed 's/[&|]/\\\\&/g')\n\n # Handle baked URL variables (e.g. https://BAKED_SITE_URL)\n # These must be replaced as full URLs to avoid invalid or double protocols\n for URL_VAR in $BAKED_URL_VARS; do\n # Check if current variable is a baked URL var\n if [ \"$VAR\" = \"$URL_VAR\" ]; then\n # Skip if URL placeholder is not present in this file, 2 - parent loop, i - case insensitive\n echo \"$FILE_CONTENT\" | grep -qi \"$URL_PLACEHOLDER\" || continue 2\n\n # Print file name once on first replacement\n if [ \"$FILE_REPLACED\" -eq 0 ]; then\n echo \"Processing file: $file\"\n FILE_REPLACED=1\n fi\n\n # Log replacement\n # Log $VALUE, because $ESCAPED_VALUE is just for sed\n echo \"replaced: $URL_PLACEHOLDER -> $VALUE\"\n\n # Replace full URL placeholder in-place, I - case insensitive\n sed -i \"s|$URL_PLACEHOLDER|$ESCAPED_VALUE|gI\" \"$file\"\n\n # Continue with next variable, 2 - parent loop\n continue 2\n fi\n done\n\n # Note: exits loop early if placeholder is not present in the file, i - case insensitive\n echo \"$FILE_CONTENT\" | grep -qi \"$PLACEHOLDER\" || continue\n\n # Print file name only on first replacement\n if [ \"$FILE_REPLACED\" -eq 0 ]; then\n echo \"Processing file: $file\"\n FILE_REPLACED=1\n fi\n\n # Log what is replaced\n if [ -z \"$VALUE\" ]; then\n echo \"replaced: $PLACEHOLDER -> (empty)\"\n else\n echo \"replaced: $PLACEHOLDER -> $VALUE\"\n fi\n\n # Perform in-place replacement using sed\n # \"s|pattern|replacement|g\" replaces all occurrences in the file\n # The | delimiter is used instead of / to avoid conflicts with paths\n # I - case insensitive\n # Example: BAKED_SITE_URL → https://example.com\n sed -i \"s|$PLACEHOLDER|$ESCAPED_VALUE|gI\" \"$file\"\n\n done\n done\ndoneI left the verbose comments in the script for clarity, but here are the most important points to keep in mind:
\n#!/bin/sh shell syntax because bash is not available by default in the Nginx Alpine image, and we want to keep the image size minimal.DIST_PATH environment variable as an argument to pass the path to the bundle into the script. At the top of the script, we validate and initialize all hardcoded and passed arguments, and exit early if invalid data is provided.REQUIRED_VARS and OPTIONAL_VARS.PREFIX=\"BAKED_\" and URL_PREFIX=\"https://${PREFIX}\"), and use the corresponding placeholders (PLACEHOLDER=\"${PREFIX}${VAR}\" and URL_PLACEHOLDER=\"${URL_PREFIX}${VAR}\"). This is necessary because a baked build for Astro will fail if we pass an invalid URL to the site option in astro.config.ts.sed instead of envsubst because it gives us more control over string replacement. Additionally, we escape special characters such as & and | in the sed input.Once we have a working and tested replacement script, it’s time to include it in the Docker entrypoint so it can run when the container starts and replace baked placeholders in the bundle with the actual environment variables from the current environment.
\nFortunately, the Nginx Alpine image already provides a dedicated pre-start folder, /docker-entrypoint.d, intended for entrypoint scripts. We define the ENV DIST_PATH=/usr/share/nginx/html environment variable because replace-variables.sh expects it as an input argument. Additionally, we include a 10- prefix in the script file name to define the execution order of scripts in the entrypoint. We want our script to run before any others.
Below is the complete runner stage of the docker/Dockerfile:
# -------------- runner --------------\n\nFROM nginx:1.29.1-alpine3.22-slim AS runtime\nCOPY ./docker/nginx.conf /etc/nginx/nginx.conf\n\n# set dist folder path for both web folder and script arg\nENV DIST_PATH=/usr/share/nginx/html\n\n# sufficient for SSG\nCOPY --from=build /app/dist ${DIST_PATH}\n\n# copy to pre-start scripts folder\n# 10-xxx controls the order\nCOPY ./scripts/replace-variables.sh /docker-entrypoint.d/10-replace-variables.sh\nRUN chmod +x /docker-entrypoint.d/10-replace-variables.sh\n\nEXPOSE 8080Finally, we define the actual values for the environment variables in the docker-compose.yml for testing:
\nservices:\n nmc-docker:\n container_name: nmc-docker\n build:\n context: .\n dockerfile: ./docker/Dockerfile\n\n platform: linux/amd64\n restart: unless-stopped\n environment:\n SITE_URL: 'http://localhost:8080'\n PLAUSIBLE_SCRIPT_URL: 'https://plausible.arm1.nemanjamitic.com/js/script.js'\n PLAUSIBLE_DOMAIN: 'nemanjamitic.com'\n PREVIEW_MODE: 'true'\n ports:\n - '8080:8080'\n networks:\n - defaultWith this, we are all set to run the container and apply the start-time environment variables. The original tutorial mentions certain trade-offs of this method, such as slower container startup and the risk of unintentional string replacements, but these are tolerable for our use case. But let’s actually see if there is more to it, in detail, in the next section.
\nThis is by far the most important section of the article. If you wanted a “TLDR” of the article, this would be it. Let’s review the issues one by one:
\nYou can have only string variables (or unions of string literals - enums). Let’s illustrate this with a code example:
\nMy website uses one boolean variable, PLAUSIBLE_DOMAIN, that enables preview of draft articles. Initially, it’s typed and validated as a boolean in both Zod and Astro schemas:
// Zod schema\n\nexport const booleanValues = ['true', 'false', ''] as const;\n\nexport const processEnvSchema = z.object({\n // ...\n PREVIEW_MODE: z\n .enum(booleanValues)\n .transform((value) => value === 'true')\n .default(false),\n // ...// Astro schema\n\nexport const envSchema = {\n schema: {\n // ...\n PREVIEW_MODE: envField.boolean({\n context: 'server',\n access: 'public',\n default: false,\n }),\n // ...But obviously, since our replacement method requires unique, baked placeholder values, we cannot use true and false directly. We must convert it to a union of string literals so that the baked placeholder value is valid at build time and the build can succeed. The type becomes: PREVIEW_MODE: 'true' | 'false' | '' | 'BAKED_PREVIEW_MODE'. Here is the updated code:
// Zod schema\n\n// src/utils/baked.ts\nexport const baked = <T extends string>(name: T): `BAKED_${T}` => `BAKED_${name}` as `BAKED_${T}`;\n\nexport const booleanValues = ['true', 'false', ''] as const;\n\nexport const processEnvSchema = z.object({\n // ...\n // Note: string union, not boolean, for baked\n PREVIEW_MODE: z\n .enum(booleanValues)\n .or(z.literal(baked('PREVIEW_MODE')))\n .default('false'),\n // ...// Astro schema\n\n// src/utils/baked.ts\nexport const baked = <T extends string>(name: T): `BAKED_${T}` => `BAKED_${name}` as `BAKED_${T}`;\n\nexport const envSchema = {\n schema: {\n // ...\n PREVIEW_MODE: envField.enum({\n context: 'server',\n access: 'public',\n values: [...booleanValues, baked('PREVIEW_MODE')],\n default: 'false',\n }),\n // ...And here is an example of how to use the new quasi-boolean variable:
\nexport const isPreviewMode = (): boolean => CONFIG_SERVER.PREVIEW_MODE === 'true';Issue no. 1 conclusion: It’s a bit of a workaround, but acceptable.
\nYou must bake and replace URL-shaped variables separately for the build to pass. The most typical and obvious variable is SITE_URL, which is assigned to the site: option inside astro.config.ts. This option is deeply integrated into the framework, used for routing, and passed within default Astro.props. If left undefined, Astro defaults to http://localhost:port. On the other hand, if you set it to a non-URL baked placeholder, e.g., BAKED_SITE_URL, the build will fail, as Astro internally passes it into the native new URL() constructor.
We solve this by treating URL-shaped variables separately, giving them a different prefix and replacement rule. This way, a baked placeholder can be a valid URL, e.g., https://BAKED_SITE_URL, allowing the build to succeed.
PREFIX=\"BAKED_\"\nURL_PREFIX=\"https://${PREFIX}\"\n\n# ...\n\nPLACEHOLDER=\"${PREFIX}${VAR}\"\nURL_PLACEHOLDER=\"${URL_PREFIX}${VAR}\"\n\n# ...\n\nsed -i \"s|$PLACEHOLDER|$ESCAPED_VALUE|gI\" \"$file\"\nsed -i \"s|$URL_PLACEHOLDER|$ESCAPED_VALUE|gI\" \"$file\"Fortunately, Astro doesn’t transform the site: option internally, so the placeholder maintains its integrity and this works as expected.
Issue no. 2 conclusion: It was a close call, but it works and is acceptable.
\nThis issue was somewhat obvious, but I still failed to predict it. Open Graph images are typically very important for SEO and the reach of static websites, especially for blogs or content-focused sites whose success largely depends on sharing on social networks.
\nOne obvious piece of information that an Open Graph image should include is the website URL. Since we made SITE_URL a start-time variable, only its placeholder is available at build time.
On my website, I use an Astro static endpoint in src/pages/api/open-graph/[…route].png.ts to dynamically render .png images from a Satori HTML template. This endpoint is called at build time, and the rendered .png images are included in the bundle. There is nothing we can do at runtime.
<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={OpenGraphBakedUrlImage} alt=“Open Graph image with incorrect SITE_URL placeholder” />
\nThe obvious consequence is that we can either:
\n![src/pages/api/open-graph/[…route].png.ts](\"https://gitlab.com/nemanjam/nemanjam.github.io/-/blob/feature/runtime-environment-variables/src/pages/api/open-graph/%5B...route%5D.png.ts\"){kind=link}
SITE_URL) from the Open Graph images, or use a unique and consistent SITE_URL_CANONICAL for all environments.Obviously, both options 1 and 2 are bad trade-offs and beyond what’s acceptable. At this point, it’s better to keep the existing setup and rebuild the website for each environment.
\nIssue no. 3 conclusion: It is unacceptable.
\nOften, you need to transform a URL variable, for example to extract the domain from the URL. Obviously, you can’t do this in Astro TypeScript code that runs at build time, because it would use values from the baked placeholders and inline the incorrect placeholder domain into the bundle. You must keep the baked variable intact during the build.
\nThe solution is to move that transformation to client-side JavaScript by including a <script /> with the transformation code that runs on page load. As mentioned earlier, this degrades page performance and SEO, because the client needs to parse and run the JavaScript to get the final content of the page.
<!-- title attribute needs just the domain from the SITE_URL -->\n\n<link\n id=\"rss-link\"\n rel=\"alternate\"\n type=\"application/rss+xml\"\n data-SITE_URL={SITE_URL}\n title=\"RSS feed\"\n href=\"/api/feed/rss\"\n/>\n\n<script>\n // or read it form DOM and data-SITE_URL attribute\n const siteUrl = window.__RUNTIME_ENV__.SITE_URL;\n const hostname = new URL(siteUrl).hostname;\n\n const link = document.getElementById('rss-link');\n\n if (link && hostname) {\n link.title = 'RSS feed for ' + hostname;\n }\n</script>As mentioned, this makes the code messy, overly verbose, and error-prone. It degrades page performance and SEO, and defeats Astro’s zero client-side JavaScript by default strategy. All of this is, once again, beyond acceptable.
\nIssue no. 4 conclusion: It is unacceptable.
\nThe experiment partially worked, but the results clearly show why a reusable build for a pure static website is a bad idea in practice.
\nIt is possible to inject start-time environment variables into a static bundle using shell scripts, Nginx entrypoints, and carefully crafted placeholders. With enough discipline, you can even make builds pass by separating string variables from URL-shaped variables, bending schemas, and moving certain logic into client-side JavaScript. However, every step in that direction erodes the very benefits that make static websites attractive in the first place.
\nA pure static website has no server runtime, no request context, and no dynamic execution environment. As soon as you try to retrofit runtime configuration into that model, you run into hard limitations. Non-string values must be faked, URLs must be handled as special cases, Open Graph images become impossible to render correctly, and any transformation of environment data leaks into client-side JavaScript. At that point, you are no longer building a clean static site, but a fragile system of workarounds that hurts performance, SEO, and maintainability.
\nIf you already have a dynamic, server-side rendered website that uses a server runtime and only need a few static pages, then a few workarounds to benefit from runtime environment variables and reusable builds can represent a reasonable trade-off. I already described that use case in the previous article: https://nemanjamitic.com/blog/2025-12-13-nextjs-runtime-environment-variables.
\nOn the other hand, if you want the simplicity, performance, and reliability of a pure static website, then accept rebuilds as part of the workflow.
\nAt first glance, you might think of environment variables as just a few values needed when the app starts, but as you dig deeper, you realize it’s far more complex than that. If you don’t clearly understand the nature of the value you’re dealing with, you’ll have a hard time running the app and managing its configuration across multiple environments.
\nLet’s identify a few dimensions that any environment variable can have:
\nThere are probably more, but this is enough to understand why it can be challenging to manage. We could go very wide, write a long article and elaborate each of these and their combinations, but since the goal of this article is very specific and practical - handling Next.js environment variables in Docker, we’ll focus just on the top three items from the list. Still, it was worth mentioning the others for context.
\nIf you search the Next.js docs, you will find a guide on environment variables, such as .env* filenames that are loaded by default, their load order and priority, variable expansion, and exposing and inlining variables with the NEXT_PUBLIC_ prefix into the client. In the self-hosting guide, you will also find a paragraph about opting into dynamic rendering so that variable values are read on each server component render, not just once at build time, and how this is useful for reusable Docker images.
A common scenario after reading the docs is to be aware of NEXT_PUBLIC_ and server variables and then scatter them around the codebase. If you use Docker and GitHub Actions, you will typically end up with something like this:
# Next.js app installer stage\nFROM base AS installer\nRUN apk update\nRUN apk add --no-cache libc6-compat\n\n# Enable pnpm\nENV PNPM_HOME=\"/pnpm\"\nENV PATH=\"$PNPM_HOME:$PATH\"\nRUN corepack enable\nRUN corepack prepare pnpm@10.12.4 --activate\n\nWORKDIR /app\n\n# Copy monorepo package.json and lock files\nCOPY --from=builder /app/out/json/ .\n# Install the dependencies\nRUN pnpm install --frozen-lockfile\n\n# Copy pruned source\nCOPY --from=builder /app/out/full/ .\n\n# THIS: set build time env vars\nARG ARG_NEXT_PUBLIC_SITE_URL\nENV NEXT_PUBLIC_SITE_URL=$ARG_NEXT_PUBLIC_SITE_URL\nRUN echo \"NEXT_PUBLIC_SITE_URL=$NEXT_PUBLIC_SITE_URL\"\n\nARG ARG_NEXT_PUBLIC_API_URL\nENV NEXT_PUBLIC_API_URL=$ARG_NEXT_PUBLIC_API_URL\nRUN echo \"NEXT_PUBLIC_API_URL=$NEXT_PUBLIC_API_URL\"\n\n# Build the project\nRUN pnpm turbo build\n\n# ...11c8a512…/.github/workflows/build-push-frontend.yml
\nname: Build and push Docker frontend\n\non:\n push:\n branches:\n - 'main'\n workflow_dispatch:\n\nenv:\n IMAGE_NAME: ${{ github.event.repository.name }}-frontend\n # THIS: set build time env vars\n NEXT_PUBLIC_SITE_URL: 'https://full-stack-fastapi-template-nextjs.arm1.nemanjamitic.com'\n NEXT_PUBLIC_API_URL: 'https://api.full-stack-fastapi-template-nextjs.arm1.nemanjamitic.com'\n\njobs:\n build:\n name: Build and push docker image\n runs-on: ubuntu-latest\n steps:\n\n # ...\n\n - name: Build and push Docker image\n uses: docker/build-push-action@v6\n with:\n context: ./frontend\n file: ./frontend/Dockerfile\n platforms: linux/amd64,linux/arm64\n progress: plain\n # THIS: set build time args\n build-args: |\n \"ARG_NEXT_PUBLIC_SITE_URL=${{ env.NEXT_PUBLIC_SITE_URL }}\"\n \"ARG_NEXT_PUBLIC_API_URL=${{ env.NEXT_PUBLIC_API_URL }}\"\n push: true\n tags: ${{ secrets.DOCKER_USERNAME }}/${{ env.IMAGE_NAME }}:latest\n 11c8a512…/frontend/package.json
\n{\n \"name\": \"full-stack-fastapi-template-nextjs\",\n \"version\": \"0.0.1\",\n \"private\": true,\n \"scripts\": {\n \"build\": \"turbo build\",\n \"dev\": \"turbo dev\",\n \"standalone\": \"turbo run standalone --filter web\",\n // THIS: set build time args\n \"docker:build:x86\": \"docker buildx build -f ./Dockerfile -t nemanjamitic/full-stack-fastapi-template-nextjs-frontend --build-arg ARG_NEXT_PUBLIC_SITE_URL='full-stack-fastapi-template-nextjs.local.nemanjamitic.com' --build-arg ARG_NEXT_PUBLIC_API_URL='api.full-stack-fastapi-template-nextjs.local.nemanjamitic.com' --platform linux/amd64 .\"\n\n // ...\n\n },\n\n // ...\n\n}In the code above, we can see that our Next.js app requires the NEXT_PUBLIC_SITE_URL and NEXT_PUBLIC_API_URL environment variables at build time. These values will be inlined into the bundle during the build and cannot be changed later. This means the Dockerfile must pass them as the corresponding ARG_NEXT_PUBLIC_SITE_URL and ARG_NEXT_PUBLIC_API_URL build arguments when building the image.
Leaving them undefined would break the build because they are validated with Zod inside the Next.js app, and validation runs at both build time and run time. Stripping the NEXT_PUBLIC_ prefix would also break the build, even without Zod, if they are used in client code.
Consequently, we need to pass these build arguments whenever we build the Docker image, for example in GitHub Actions and in the local build script defined in package.json.
Using this method, we would get a functional Docker image, but with one major drawback: it can be used only in a single environment because the NEXT_PUBLIC_SITE_URL and NEXT_PUBLIC_API_URL values are baked into the image at build time and are immutable.
To make this crystal clear, whatever we set for the NEXT_PUBLIC_SITE_URL and NEXT_PUBLIC_API_URL environment variables at runtime will be ignored because they no longer exist in the Next.js app. After the build they are replaced with string literals in the JavaScript bundle.
If, besides production, you also have staging, preview, testing environments, or other production mirrors, you would need to maintain a separate image with its own configuration code, build process, and registry storage for each of them. This means a lot of overhead.
\nMany people find this impractical, which you can see from the popularity of such issues in the Next.js repository:
\nBetter support for runtime environment variables #44628
\nDocker image with NEXT_PUBLIC_ env variables #17641
\nNot possible to use different configurations in staging + production #22243
\nThe solution is obvious: we should prevent any use of build-time (stale, immutable) variables and read everything from the target environment at runtime. This also means avoiding any NEXT_PUBLIC_* client variables.
To implement this, we must be well aware of where and when a given component runs:
\nThese components (or entire pages) are dynamically rendered on each request. They have access to any server data, including both public and private environment variables. No additional action is needed. In Next.js, we identify such components by their use of request resources such as cookies, headers, and connection:
\nimport { cookies, headers } from 'next/headers';\nimport { connection } from 'next/server';\n\nexport default async function Page() {\n const headersList = await headers();\n const cookiesList = await cookies();\n\n await connection(); // void\n}Such a page is pre-rendered once at build time in the build environment. It has access to server data, but it is converted to a static asset at build time and is immutable at runtime. We have two options:
\nimport { connection } from 'next/server';\n\nexport default async function Page() {\n\n // opt into dynamic rendering\n await connection();\n\n // ...\n}Set placeholder values for variables at build time and perform string replacement directly on the generated static HTML using sed or envsubst and a shell script included in ENTRYPOINT [\"scripts/entrypoint.sh\"] in the Dockerfile.
Note that these will be start-time variables, not true run-time variables, but most of the time that is sufficient because they are unique to each environment. However, they cannot change during the app’s run time once initialized.
\nWe won’t go into much detail about this method, it could be a good topic for a future article since it is quite useful for static, presentational websites. If you want to read more, here is an interesting and practical tutorial: https://phase.dev/blog/nextjs-public-runtime-variables/.
\nNext.js prevents exposing any variables to the client without the NEXT_PUBLIC_ prefix, but since those are inlined at build time, we simply won’t use them. For exposing environment variables to client components, we have a few options:
Pass variables as props from the parent server component like any other value. This is simple and convenient.
\nInside the dynamically generated root layout, render a <script /> tag that injects a window.__RUNTIME_ENV__ property into the global window object using the dangerouslySetInnerHTML attribute. We will actually use this method. Then, on the client, we can access the variables on the window object, for example window.__RUNTIME_ENV__.API_URL.
Also this is a good moment to validate runtime vars with Zod.
\nHere is the illustration code bellow:
\nimport { connection } from 'next/server';\n\nexport const runtimeEnvSchema = z.object({\n SITE_URL: z.url().regex(/[^/]$/, 'SITE_URL should not end with a slash \"/\"'),\n API_URL: z.url().regex(/[^/]$/, 'API_URL should not end with a slash \"/\"'),\n});\n\nconst RootLayout: FC<Props> = async ({ children }) => {\n await connection();\n \n const runtimeEnvData = {\n SITE_URL: process.env.SITE_URL,\n API_URL: process.env.API_URL,\n };\n\n // validate vars with Zod before injecting\n const parsedRuntimeEnv = runtimeEnvSchema.safeParse(runtimeEnvData);\n\n // if invalid vars abort\n if (!parsedRuntimeEnv.success) throw new Error('Invalid runtime environment variable found...');\n\n const runtimeEnv = parsedRuntimeEnv.data;\n\n return (\n <html lang=\"en\">\n <body>\n {/* Inline JSON injection */}\n <script\n dangerouslySetInnerHTML={{\n __html: `window.__RUNTIME_ENV__ = ${JSON.stringify(runtimeEnv)};`,\n }}\n />\n\n {children}\n </body>\n </html>\n );\n}sed to replace them with a shell script inside the JavaScript bundle when the container starts.We can see from this that the first two methods are the simplest and most convenient, so we will use them.
\nNote: Whenever an environment variable is available on the client, it is public by default. Make sure not to expose any secrets to the client.
\nalizeait/next-public-env packageWe could do this manually as shown in the snippet above, but there is already the alizeait/next-public-env package that handles all of this and also provides some more advanced handling.
\nCheck these 2 files for example:
\n\nUsage is obvious and straightforward: just define a Zod schema, mount <PublicEnv /> in the root layout, and use getPublicEnv() to access the variables wherever you need them.
You can see bellow how I did it:
\n# install package\n\npnpm add next-public-envfrontend/apps/web/src/config/process-env.ts
\n/** Exports RUNTIME env. Must NOT call getPublicEnv() in global scope. */\nexport const { getPublicEnv, PublicEnv } = createPublicEnv(\n {\n NODE_ENV: process.env.NODE_ENV,\n SITE_URL: process.env.SITE_URL,\n API_URL: process.env.API_URL,\n },\n { schema: (z) => getProcessEnvSchemaProps(z) }\n);frontend/apps/web/src/schemas/config.ts
\nimport { z } from 'zod';\n\nexport const nodeEnvValues = ['development', 'test', 'production'] as const;\n\ntype ZodType = typeof z;\n\n/** For runtime env. */\nexport const getProcessEnvSchemaProps = (z: ZodType) => ({\n NODE_ENV: z.enum(nodeEnvValues),\n SITE_URL: z.url().regex(/[^/]$/, 'SITE_URL should not end with a slash \"/\"'),\n API_URL: z.url().regex(/[^/]$/, 'API_URL should not end with a slash \"/\"'),\n});\n\n/** For schema type. */\nexport const processEnvSchema = z.object(getProcessEnvSchemaProps(z));frontend/apps/web/src/app/layout.tsx
\nimport { PublicEnv } from '@/config/process-env';\n\ninterface Props {\n children: ReactNode;\n}\n\nconst RootLayout: FC<Props> = ({ children }) => (\n <html lang=\"en\" suppressHydrationWarning>\n <body className={fontInter.className}>\n <PublicEnv />\n <ThemeProvider attribute=\"class\" defaultTheme=\"light\" enableSystem disableTransitionOnChange>\n {/* Slot with server components */}\n {children}\n <Toaster />\n </ThemeProvider>\n </body>\n </html>\n);\n\nexport default RootLayout;An example usage, for instance in instrumentation.ts, to log the runtime values of all environment variables for debugging purposes:
frontend/apps/web/src/instrumentation.ts
\n/** Runs only once on server start. */\n\n/** Log loaded env vars. */\nexport const register = async () => {\n if (process.env.NEXT_RUNTIME === 'nodejs') {\n const { prettyPrintObject } = await import('@/utils/log');\n const { getPublicEnv } = await import('@/config/process-env');\n\n prettyPrintObject(getPublicEnv(), 'Runtime process.env');\n }\n};baseUrl for OpenAPI clientThis is another typical and very important spot for using the API_URL environment variable. What makes it tricky is that it is included and runs on both the server and in the browser, but it is defined in a single place.
However, alizeait/next-public-env resolves this complexity very well on its own, and you can simply use getPublicEnv() to get the API_URL value while letting the package handle the rest.
frontend/apps/web/src/lib/hey-api.ts
\nimport { getPublicEnv } from '@/config/process-env';\n\n/** Runtime config. Runs and imported both on server and in browser. */\nexport const createClientConfig: CreateClientConfig = (config) => {\n const { API_URL } = getPublicEnv();\n\n return {\n ...config,\n baseUrl: API_URL,\n credentials: 'include',\n ...(isServer() ? { fetch: serverFetch } : {}),\n };\n};Variables that are the same for every environment can be left as NEXT_PUBLIC_ and inlined into the bundle. They should also be versioned in Git (their .env.* files). Since this is the case, the best approach is to store them as TypeScript constants directly in the source, because that is what they truly are - shared constants.
Build once - deploy everywhere. Use a single image and .env file with no redundancy.
Now that we have eliminated all build-time variables by converting them to run-time environment variables, we can simply remove all build arguments and environment variables from the Dockerfile, Github Actions build workflow, package.json build scripts, etc.
Note: During the build phase of a Next.js app, the global scope is also executed. Therefore, if you read any environment variables, such as process.env.MY_VAR_XXX, your code must be able to handle a default undefined value without throwing exceptions, as this would break the build.
Tip: To access environment variables, always use getPublicEnv() inside components and functions. Never call getPublicEnv() or read process.env in the global scope, this way, you won’t need to handle undefined environment variables explicitly for the build to pass.
Simply remove all build arguments and build-time environment variables from the Dockerfile:
# Not needed anymore, remove all build args\nARG ARG_NEXT_PUBLIC_SITE_URL\nENV NEXT_PUBLIC_SITE_URL=$ARG_NEXT_PUBLIC_SITE_URL\nRUN echo \"NEXT_PUBLIC_SITE_URL=$NEXT_PUBLIC_SITE_URL\"\n\nARG ARG_NEXT_PUBLIC_API_URL\nENV NEXT_PUBLIC_API_URL=$ARG_NEXT_PUBLIC_API_URL\nRUN echo \"NEXT_PUBLIC_API_URL=$NEXT_PUBLIC_API_URL\"This is the cleaned up Dockerfile that I am using to build Next.js app inside the monorepo: frontend/Dockerfile.
Also, don’t forget to clean up unused build arguments from the Github Actions workflow and package.json scripts for building the Docker image. You can see mine here: .github/workflows/build-push-frontend.yml, frontend/package.json.
\"scripts\": {\n \"docker:build:x86\": \"docker buildx build -f ./Dockerfile -t nemanjamitic/full-stack-fastapi-template-nextjs-frontend --platform linux/amd64 .\"\n},Once built, you can use that image to deploy to any environment. Naturally, you need to define and pass all runtime environment variables into the Docker container. In your docker-compose.yml, use the env_file: or environment: keys.
services:\n frontend:\n image: nemanjamitic/full-stack-fastapi-template-nextjs-frontend:latest\n container_name: full-stack-fastapi-template-nextjs-frontend\n restart: unless-stopped\n env_file:\n - .env\n environment:\n - PORT=3000\n \n # ...SITE_URL=https://full-stack-fastapi-template-nextjs.arm1.nemanjamitic.com\nAPI_URL=https://api-full-stack-fastapi-template-nextjs.arm1.nemanjamitic.com\nNODE_ENV=production\n\n# ...You can see docker-compose.yml and .env I am using here: apps/full-stack-fastapi-template-nextjs/docker-compose.yml, apps/full-stack-fastapi-template-nextjs/.env.example
In the Static page section, I already mentioned a few notes about runtime variables and static websites. Indeed, you have two options for runtime variables:
\nConvert the website from static to dynamically rendered SSR (rendered at request time). Note that this is a significant change: from this point, your website will require a Node.js runtime, which will greatly impact your deployment options, as you can no longer use static hosting.
\nThis is overkill just for the purpose of having runtime environment variables. Use it only if your website has additional reasons to use SSR.
\nPerform string replacement directly on bundle assets using sed, envsubst, etc. This is the right approach. There are other options, such as the Nginx subs_filter config option, but be careful with it, as it runs on each request and can waste CPU.
Another option to consider is using an ./env.js file instead of the usual .env. You can then host it with Nginx and load it into the app using <script src=\"./env.js\" />. After that, you can reference the variables with window.__RUNTIME_ENV__.MY_VAR.
Note that this won’t work well for usage in pure HTML pages. For example, Astro omits any client-side JavaScript by default, so you would need to use an additional inline <script /> tag to update the HTML, e.g., getElementById(\"my-id\")?.textContent = window.__RUNTIME_ENV__.MY_VAR, which is less optimal than the string replacement method.
Here is a quick, approximate code for illustration:
\n// define variables\n\nwindow.__RUNTIME_ENV__ = {\n SITE_URL: \"https://my-static-website.com\",\n PLAUSIBLE_DOMAIN: 'my-static-website.com',\n PLAUSIBLE_SCRIPT_URL: 'https://plausible.my-server.com/js/script.js',\n};# mount and host env.js file\n\nmy-static-website:\n image: nginx:1.29.1-alpine3.22-slim\n container_name: my-static-website\n restart: unless-stopped\n volumes:\n - ./website:/usr/share/nginx/html\n - ./env.js:/usr/share/nginx/html/env.js # this\n - ./nginx/nginx.conf:/etc/nginx/nginx.conf\n\n # ...<!-- Load env.js file -->\n\n<head>\n <meta charset=\"UTF-8\" />\n <title>My static website</title>\n\n <script src=\"./env.js\"></script>\n\n <!-- ... -->\n</head><!-- example usage -->\n\n<!-- example 1: assign var to text content -->\n<span id=\"my-element\"></span>\n\n<script>\n const mySpan = document.getElementById('my-element');\n mySpan.textContent = window.__RUNTIME_ENV__.MY_VAR;\n</script>\n\n<!-- example 2: assign var to script attribute -->\n<script>\n const script = document.createElement(\"script\");\n script.defer = true;\n script.type = \"text/partytown\";\n\n // dynamically set attributes from runtime env\n script.dataset.domain = window.__RUNTIME_ENV__.PLAUSIBLE_DOMAIN;\n script.src = window.__RUNTIME_ENV__.PLAUSIBLE_SCRIPT_URL;\n\n document.head.appendChild(script);\n</script>So, to conclude, the best approach is to use a shell script with sed or envsubst and add it to the Nginx Dockerfile ENTRYPOINT or the docker-compose.yml command:. Here is the link to the already mentioned practical tutorial again: https://phase.dev/blog/nextjs-public-runtime-variables/.
The relevant files:
\n# 1. Next.js app repo\n\n# https://github.com/nemanjam/full-stack-fastapi-template-nextjs/tree/e990a3e29b7af60831851ff6f909c34df6a7f800\n\ngit checkout e990a3e29b7af60831851ff6f909c34df6a7f800\n\n# run-time vars configuration\nfrontend/apps/web/src/config/process-env.ts\nfrontend/apps/web/src/schemas/config.ts\nfrontend/apps/web/src/app/layout.tsx\n\n# usages\nfrontend/apps/web/src/instrumentation.ts\nfrontend/apps/web/src/lib/hey-api.ts\n\n# 2. Deployment repo\n\n# https://github.com/nemanjam/traefik-prox/tree/f3c087184e851db20e65409a6dd145767dd9bc2b\n\ngit checkout f3c087184e851db20e65409a6dd145767dd9bc2b\n\napps/full-stack-fastapi-template-nextjs/docker-compose.yml\napps/full-stack-fastapi-template-nextjs/.env.exampleIf you go by inertia and mix and scatter run-time and build-time variables around the source code, build, and deployment configuration, you will end up with development and production environments that are difficult to manage, hard to debug and replicate bugs, have an unreliable deployment process, constantly require troubleshooting for missing or invalid environment variables, and result in redundant Docker images, among other issues.
\nSo, take a proactive approach: understand properly and identify the variables you are dealing with. One way to do this is to leverage the power and convenience of run-time environment variables.
\nWhat approach do you use to manage environment variables in Next.js apps? Feel free to share your experiences and opinions in the comments.
\nimport { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;
\nimport MazeSolverClassDiagramImage from ’../../../../content/post/2025/07-31-maze-solver/_images/maze-solver-class-diagram.png’;\nimport YarnDevImage from ’../../../../content/post/2025/07-31-maze-solver/_images/yarn-dev.png’;\nimport YarnTestVerboseImage from ’../../../../content/post/2025/07-31-maze-solver/_images/yarn-test-verbose.png’;\nimport YarnCoverageImage from ’../../../../content/post/2025/07-31-maze-solver/_images/yarn-coverage.png’;
\nPathfinding is a fundamental topic in computer science, with applications in fields like navigation, AI/ML, network routing, and many others. In this article, we compare four core pathfinding algorithms: breadth-first search (BFS), depth-first search (DFS), Dijkstra’s algorithm, and A* (A star) through a practical maze-solving example. We don’t just explain them in theory, we built a demo app where you can tweak maze inputs or edit the algorithm code and instantly see how it affects the output and efficiency.
\nOne of the key takeaways is how a tiny change, just a single line in the code can drastically alter an algorithm’s behavior. This highlights how critical implementation details are, even when the overall structure looks the same.
\nPaths in a maze form a tree structure or a graph if the maze contains cycles. That’s why tree and graph traversal algorithms can be used for finding paths and the shortest path in a maze.
\nAll 4 algorithms differ in just a few lines of code, but their behavior differs dramatically.
\nWe create a pragmatic, simplified OOP model of the maze and its behavior as a tradeoff, favoring clarity and concise instantiation of maze objects in tests.
\nA maze is represented as a binary matrix where 0 stands for a free space, 1 for a boundary, and * for a path. It also has start and end points. In the sense of a weighted graph 0 cell has zero weight and cell 1 has infinite weight.
export class Maze implements IMaze {\n private board: number[][];\n private start: Coordinate;\n private end: Coordinate;\n\n // ...\n}\n\nexport interface Coordinate {\n readonly x: number;\n readonly y: number;\n}\n\n// example maze\nconst testMaze: number[][] = [\n [0, 1, 0, 0, 0],\n [0, 1, 0, 1, 0],\n [0, 0, 0, 1, 0],\n [0, 1, 1, 1, 0],\n [0, 0, 0, 0, 0],\n];\n\nconst start: Coordinate = { x: 0, y: 0 };\nconst end: Coordinate = { x: 4, y: 4 };We use polymorphism and a simplified Factory pattern. MazeSolver is an abstract class that declares the findPath() method, which is implemented in each derived concrete solver class.
/**\n * Abstract base class for maze solving algorithms.\n * Implements common functionality for maze solvers.\n */\n\nexport abstract class MazeSolver implements IMazeSolver {\n protected maze: IMaze;\n\n protected abstract findPath(): Coordinate[] | null;\n\n // ...\n}We use encapsulation and coding towards interface, separating interfaces from implementations by exposing only the public class methods through the interfaces.
\nMaze interface:
\nexport interface IMaze {\n getBoard(): number[][];\n\n getStart: () => Coordinate;\n\n formatPath: (path: ReadonlyArray<Coordinate>) => string;\n\n // ...\n}Maze implementation:
\nexport class Maze implements IMaze {\n public getBoard(): number[][] { ... }\n\n public getStart(): Coordinate { ... }\n\n public formatPath(path: ReadonlyArray<Coordinate>): string { ... }\n\n // ...\n}Maze usage:
\nconst _maze2: IMaze = Maze.create(testMaze, start, end);\n\n// ...Class diagram:
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={MazeSolverClassDiagramImage} alt=“Maze solver class diagram” />
\nWe install dependencies, run the app, and run the tests as usual, like any other TypeScript app.
\n# install dependencies\nyarn install\n\n# enable or disable logging in src/config.ts\n\n# run the app in dev mode\nyarn dev\n\n# logging is disabled for tests by default\n\n# run tests\nyarn test\n\n# run tests in verbose mode\nyarn test-verbose\n\n# generate coverage report\nyarn coverageWe can see that different algorithms require a different number of steps for the same input maze. Example output for a given maze input:
\n<Image {…IMAGE_SIZES.FIXED.MDX_XS} src={YarnDevImage} alt=“App dev mode terminal output” />
\nWe can run tests that ensure for each algorithm:
\n<Image {…IMAGE_SIZES.FIXED.MDX_XS} src={YarnTestVerboseImage} alt=“Run tests in verbose mode” />
\nAnd calculate the code coverage:
\n<Image {…IMAGE_SIZES.FIXED.MDX_SM} src={YarnCoverageImage} alt=“Tests coverage table” />
\nNow for the most important and interesting part: let’s analyze the algorithm’s code and explain how it affects their behavior and efficiency.
\nBFS and DFS are basic traversal algorithms that ignore the weights of the edges, so they are applicable only to unweighted graphs.
\nThe actual code for BFS and DFS differs by only a single line, but they exhibit completely opposite behavior. BFS uses a queue (FIFO), while DFS uses a stack (LIFO), and this has a fundamental impact on how the next node candidate for the path is selected.
\n// BFS\nconst { coord, path } = queue.shift()!;\n\n// DFS\nconst { coord, path } = stack.pop()!;This is the array of coordinates that represents possible directions for movement. This array is iterated over in the algorithm’s inner loop.
\nexport const directions: Direction[] = [\n { x: 0, y: 1 }, // up\n { x: 1, y: 0 }, // right\n { x: 0, y: -1 }, // down\n { x: -1, y: 0 }, // left\n];Here is the complete BFS implementation (since all 4 algorithms share most of the same base) so we can have a better idea of what we are working with:
\nexport class MazeSolverBFS extends MazeSolver {\n /**\n * Implements the Breadth-First Search (BFS) algorithm to find a path from the start to the end of the maze.\n */\n protected findPath(): Coordinate[] | null {\n const start = this.maze.getStart();\n\n // Initialize the BFS queue with the start position.\n const queue: BFSQueueElement[] = [{ coord: start, path: [start] }];\n\n // Keep track of visited coordinates (as strings).\n const visited = new Set<string>();\n visited.add(`${start.x},${start.y}`);\n\n while (queue.length > 0) {\n // Count iterations.\n this.incrementStep();\n\n // The most important line. \n // FIFO - Takes the oldest element in the queue.\n const { coord, path } = queue.shift()!;\n\n // Check if end and exit.\n if (this.maze.isEnd(coord)) {\n return path;\n }\n\n // Print the current state of the maze.\n this.printBoard(coord, visited, path);\n\n // Always loops 4 times.\n for (const direction of directions) {\n // Calculate the next coordinate by applying the direction.\n const nextCoord: Coordinate = {\n x: coord.x + direction.x,\n y: coord.y + direction.y,\n };\n // Create a key for nextCoord (to check for uniqueness in the visited set).\n const coordKey = `${nextCoord.x},${nextCoord.y}`;\n\n // If nextCoord is not visited, is within bounds, and is walkable, add it to the potential path.\n if (\n !visited.has(coordKey) &&\n this.maze.isWithinBounds(nextCoord) &&\n this.maze.isWalkable(nextCoord)\n ) {\n visited.add(coordKey);\n queue.push({ coord: nextCoord, path: [...path, nextCoord] });\n }\n }\n }\n\n // Return null if no path to the end is found.\n return null;\n }\n}Since BFS uses a queue, it respects this structure and attempts to change direction in every iteration of the outer loop. Without obstacles and boundaries, this causes the algorithm to thoroughly inspect nodes closer to the starting node before moving further away. That’s why BFS can be inefficient for large trees and graphs where the end node is very distant from the starting node.
\nIn contrast, DFS also respects the initial order in the directions array but prioritizes the earlier elements. So, in the example above, it will always attempt to apply the up direction first before exploring other directions. Without obstacles and boundaries, this causes the algorithm to inspect distant nodes in a straight line. DFS can be efficient for finding a distant end node but can also be very inefficient for finding a nearby node if it happens to be in a different direction.
Not all graphs have edges with uniform weights. In such cases, we must use algorithms that are aware of weights (the cost between two nodes), such as Dijkstra and A*.
\nDijkstra’s algorithm is aware of the cost between two nodes (edge weight) and takes it into account when selecting the next node. It uses a priority queue to keep track of the cost history.
\n// Take the first element from the priority queue.\n// Choose the node that ads minimal cost.\nqueue.sort((a, b) => a.cost - b.cost);\nconst { coord, path, cost } = queue.shift()!;\n\n// ...\n\n// Test how much cost every new node ads to the path before adding it to the queue.\nconst nextCost = cost + this.maze.getCost(nextCoord);\n// ...\nif( ... && !costMap.has(coordKey) || nextCost < costMap.get(coordKey)!)Dijkstra keeps a history of the cost of the current path and when selecting the next node chooses the node that adds the minimal cost. If there are cycles it may access the same node from multiple paths and will choose the one with the minimal weight (shortest path). In graphs with constant edge weights it reduces to BFS. This can be observed in the screenshot above, where both BFS and Dijkstra take an equal number of steps because the maze has uniform weights of 1 and Infinity.
\nA* is the same as Dijkstra but besides keeping the history, it uses a heuristic function to predict the future - the direction in which the end node could be.
\nprotected heuristic(a: Coordinate, b: Coordinate): number {\n // Manhattan distance as the heuristic.\n // Can only move horizontally or vertically, not diagonally. In \"rectangles\".\n return Math.abs(a.x - b.x) + Math.abs(a.y - b.y);\n}\n\n// ...\n\nopenSet.sort(\n (a, b) =>\n // The most important line. The only difference from Dijkstra.\n // Cost = history + Manhattan distance from the end node.\n // prettier-ignore\n (a.cost + this.heuristic(a.coord, end)) -\n (b.cost + this.heuristic(b.coord, end))\n);If the heuristic function is well chosen it will make A* more efficient than the before mentioned algorithms. Consequently, if the heuristic function is poorly chosen it will degrade the algorithm efficiency.
\nIn this example, we can see how algorithm analysis and design is a very sensitive and subtle discipline that leaves no room for low focus or a lack of understanding of the domain. Although BFS, DFS, Dijkstra, and A* share most of their implementation, even a subtle change in the code can lead to a dramatic change in behavior.
\nIn the demo app, you can tweak the predefined mazes in the tests/fixtures/*.txt files and make your own observations. You can also check the resources and interactive playground listed in the References section.
Have you experimented with maze-solving and pathfinding algorithms before? Let me know in the comments.
\nimport { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;
\nimport TraefikLoadBalancerArchitectureImage from ’../../../../content/post/2025/05-29-traefik-load-balancer/_images/traefik-load-balancer-architecture.png’;
\nThis article is a continuation of Expose home server with Rathole tunnel and Traefik article, which explains how to permanently host websites from home by bypassing CGNAT. That setup works well for exposing a single home server (like a Raspberry Pi, server PC, or virtual machine), but it has a limitation: it requires one VPS (or at least one public network interface) per home server. This is because the Rathole server exclusively uses ports 80 and 443.
But it doesn’t have to be like this. We can reuse a single Rathole server for many tunnels and home servers, we just need a tool to load balance their traffic, as long as our VPS’s network interface provides enough bandwidth for our websites and services.
\nThis article explains how to achieve that using Traefik HTTP and TCP routers.
\nThe main problem here is that we can’t bind more than one port to ports 80 and 443, respectively. Only one service can listen on a given port at the same time. So something like this doesn’t exist:
services:\n\n rathole:\n image: rapiz1/rathole:v0.5.0\n container_name: rathole\n command: --server /config/rathole.server.toml\n restart: unless-stopped\n ports:\n # host:container\n - 2333:2333\n - 80:5080,5081 # non existent syntax, can't bind two ports to a single port\n - 443:5443,5444 # same\n volumes:\n - ./rathole.server.toml:/config/rathole.server.toml:roNeither the operating system nor Docker provides load balancing functionality out of the box, we need to handle it ourselves.
\nWe need to introduce a tool for load balancing traffic between tunnels. We will use Traefik, since we already use it with the Rathole client.
\nFor each home server, we need 2 tunnels: one for HTTP and another for HTTPS traffic:
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={TraefikLoadBalancerArchitectureImage} alt=“Traefik load balancer architecture diagram” />
\nReminder:
\nI already wrote about the advantage of resolving SSL certificates locally on the home server in the Architecture overview section of the previous article, but here is a quick recap:
\nSince we passthrough encrypted HTTPS traffic, Traefik can’t read the subdomain from an HTTP request as usual. Instead, we will run the Traefik router in TCP mode, using the HostSNIRegexp matcher. This will run the router on layer 4 (TCP) instead of the usual layer 7 (HTTP).
\nFor more in-depth info on how this works, you can read here: Server Name Indication (SNI).
\nNow that we understand the principle, we can get to the practical implementation.
\nBelow is the complete docker-compose.yml that defines the Traefik TCP router and the Rathole server with 2 HTTP/HTTPS tunnel pairs for 2 home servers: pi (OrangePi) and local (MiniPC), in my case.
version: '3.8'\n\nservices:\n traefik:\n image: traefik:v2.9.8\n container_name: traefik\n restart: unless-stopped\n command:\n - --providers.docker=true\n - --entrypoints.web.address=:80\n - --entrypoints.websecure.address=:443\n - --entrypoints.traefik.address=:8080\n - --api.dashboard=true\n - --api.insecure=false\n - --log.level=DEBUG\n - --accesslog=true\n ports:\n - 80:80\n - 443:443\n - 8080:8080\n volumes:\n - /var/run/docker.sock:/var/run/docker.sock:ro\n networks:\n - proxy\n labels:\n # Enable the dashboard at http://traefik.amd2.nemanjamitic.com\n # http for simplicity, no acme.json file\n - traefik.enable=true\n - 'traefik.http.routers.traefik.rule=Host(`traefik.amd2.${SITE_HOSTNAME}`)'\n - traefik.http.routers.traefik.entrypoints=web\n - traefik.http.routers.traefik.service=api@internal\n - traefik.http.routers.traefik.middlewares=auth\n - 'traefik.http.middlewares.auth.basicauth.users=${TRAEFIK_AUTH}'\n\n rathole:\n image: rapiz1/rathole:v0.5.0\n container_name: rathole\n command: --server /config/rathole.server.toml\n restart: unless-stopped\n ports:\n - 2333:2333\n volumes:\n - ./rathole.server.toml:/config/rathole.server.toml:ro\n networks:\n - proxy\n\n labels:\n ### HTTP port 80 - HTTP routers ###\n\n # pi.nemanjamitic.com, www.pi.nemanjamitic.com, *.pi.nemanjamitic.com, www.*.pi.nemanjamitic.com\n \n # Route *.pi.nemanjamitic.com -> 5080\n - 'traefik.http.routers.rathole-pi.rule=HostRegexp(`pi.${SITE_HOSTNAME}`, `www.pi.${SITE_HOSTNAME}`, `{subdomain:[a-z0-9-]+}.pi.${SITE_HOSTNAME}`, `www.{subdomain:[a-z0-9-]+}.pi.${SITE_HOSTNAME}`)'\n - traefik.http.routers.rathole-pi.entrypoints=web\n - traefik.http.routers.rathole-pi.service=rathole-pi\n - traefik.http.services.rathole-pi.loadbalancer.server.port=5080\n\n # Route *.local.nemanjamitic.com -> 5081\n - 'traefik.http.routers.rathole-local.rule=HostRegexp(`local.${SITE_HOSTNAME}`, `www.local.${SITE_HOSTNAME}`, `{subdomain:[a-z0-9-]+}.local.${SITE_HOSTNAME}`, `www.{subdomain:[a-z0-9-]+}.local.${SITE_HOSTNAME}`)'\n - traefik.http.routers.rathole-local.entrypoints=web\n - traefik.http.routers.rathole-local.service=rathole-local\n - traefik.http.services.rathole-local.loadbalancer.server.port=5081\n\n ### HTTPS port 443 with TLS passthrough - TCP routers ###\n\n # Route *.pi.nemanjamitic.com -> 5443\n - 'traefik.tcp.routers.rathole-pi-secure.rule=HostSNIRegexp(`pi.${SITE_HOSTNAME}`, `www.pi.${SITE_HOSTNAME}`, `{subdomain:[a-z0-9-]+}.pi.${SITE_HOSTNAME}`, `www.{subdomain:[a-z0-9-]+}.pi.${SITE_HOSTNAME}`)'\n - traefik.tcp.routers.rathole-pi-secure.entrypoints=websecure\n - traefik.tcp.routers.rathole-pi-secure.tls.passthrough=true\n - traefik.tcp.routers.rathole-pi-secure.service=rathole-pi-secure\n - traefik.tcp.services.rathole-pi-secure.loadbalancer.server.port=5443\n\n # Route *.local.nemanjamitic.com -> 5444\n - 'traefik.tcp.routers.rathole-local-secure.rule=HostSNIRegexp(`local.${SITE_HOSTNAME}`, `www.local.${SITE_HOSTNAME}`, `{subdomain:[a-z0-9-]+}.local.${SITE_HOSTNAME}`, `www.{subdomain:[a-z0-9-]+}.local.${SITE_HOSTNAME}`)'\n - traefik.tcp.routers.rathole-local-secure.entrypoints=websecure\n - traefik.tcp.routers.rathole-local-secure.tls.passthrough=true\n - traefik.tcp.routers.rathole-local-secure.service=rathole-local-secure\n - traefik.tcp.services.rathole-local-secure.loadbalancer.server.port=5444\n\nnetworks:\n proxy:\n external: trueLet’s start with the most important part: the labels on the rathole container that define load balancing on the two tunnels.
First, we define two HTTP routers using the HostRegexp() matcher. It takes HTTP traffic from the entrypoint on port 80 and load balances it between two tunnels on ports 5080 and 5081.
The second pair of labels defines a TCP router that takes traffic from the HTTPS entrypoint on port 443, passes it through without decrypting, and load balances it between tunnels on ports 5443 and 5444. Note that with the HostSNIRegexp() matcher, you can’t include escaped dots (.) in the regex, you must repeat the entire domain sequence to handle the www variant of the domain.
Also note that we use separate regex variants to match the root subdomain explicitly, e.g. pi.nemanjamitic.com and www.pi.nemanjamitic.com for both HTTP and TCP routers.
That’s it, this is the main load balancing logic definition.
\nNote: Because we use HostRegexp() and HostSNIRegexp() on the server, you will need to use Host() and HostSNI() matchers for the Traefik running on the client side of the tunnel, or you will get 404 errors without additional configuration. Regex matchers on both the server and client sides seem to be too loose.
Now it’s just left to write the config for the Rathole server that defines 2×2 tunnels. Just make sure to use a different token and port for each tunnel.
\n[server]\nbind_addr = \"0.0.0.0:2333\"\n\n[server.transport]\ntype = \"noise\"\n\n[server.transport.noise]\nlocal_private_key = \"private_key\"\n\n# separated based on token, also can NOT use same ports\n\n# pi\n[server.services.pi-traefik-http]\ntoken = \"secret_token_1\"\nbind_addr = \"0.0.0.0:5080\"\n\n[server.services.pi-traefik-https]\ntoken = \"secret_token_1\"\nbind_addr = \"0.0.0.0:5443\" \n\n# local\n[server.services.local-traefik-http]\ntoken = \"secret_token_2\"\nbind_addr = \"0.0.0.0:5081\"\n\n[server.services.local-traefik-https]\ntoken = \"secret_token_2\"\nbind_addr = \"0.0.0.0:5444\"Reminder: You just need to open port 2333 in the VPS firewall for the Rathole control channel and not for the ports 5080, 5081, 5443, or 5444, because they are used by Rathole internally.
Additionally, for the sake of debugging, we expose the Traefik dashboard using labels on the traefik container. To simplify the configuration and avoid handling the acme.json file, we expose it using HTTP.
Warning: When setting the dashboard hashed password via the TRAEFIK_AUTH environment variable, make sure to escape the $ characters properly or authentication will break. To do that, you need to use both double quotes \"...\" and the escape slash ‘\\’, as shown in the example below:
# install apache2-utils\nsudo apt install apache2-utils\n\n# hash the password\nhtpasswd -nb admin yourpassword# use BOTH \"...\" and \\$ to escape $ properly\n\n# this will work correctly\nTRAEFIK_AUTH=\"admin:\\$asd1\\$E3lsdAo\\$3Mertp57JJ4LVU.HRR0\"\n\n# this will break\nTRAEFIK_AUTH=\"admin:$asd1$E3lsdAo$3Mertp57JJ4LVU.HRR0\"\n\n# this will also break\nTRAEFIK_AUTH=admin:\\$asd1\\$E3lsdAo\\$3Mertp57JJ4LVU.HRR0The client part of the tunnel is almost the same as for a single home server. The only thing to keep in mind is to bind the specific client only to the tunnels that are meant for it, and not to all tunnels. Kind of obvious and self-explanatory, but just in case, let’s be very clear and explicit.
\nHere, we define the rathole.client.toml Rathole client config to bind the pi home server to its HTTP pi-traefik-http and HTTPS pi-traefik-https tunnels.
[client]\nremote_addr = \"123.123.123.123:2333\"\n\n[client.transport]\ntype = \"noise\"\n\n[client.transport.noise]\nremote_public_key = \"public_key\"\n\n# single client per tunnels pair\n\n# pi\n[client.services.pi-traefik-http]\ntoken = \"secret_token_1\"\nlocal_addr = \"traefik:80\"\n\n[client.services.pi-traefik-https]\ntoken = \"secret_token_1\"\nlocal_addr = \"traefik:443\" Similarly, here we define the rathole.client.toml config to bind the local home server to it’s HTTP local-traefik-http and HTTPS local-traefik-https tunnels.
[client]\nremote_addr = \"123.123.123.123:2333\"\n\n[client.transport]\ntype = \"noise\"\n\n[client.transport.noise]\nremote_public_key = \"public_key\"\n\n# single client per tunnels pair\n\n# local\n[client.services.local-traefik-http]\ntoken = \"secret_token_2\"\nlocal_addr = \"traefik:80\"\n\n[client.services.local-traefik-https]\ntoken = \"secret_token_2\"\nlocal_addr = \"traefik:443\"docker-compose.yml for the Rathole client and Traefik is exactly the same as it was for a single home server. I am repeating it here just for the sake of completeness.
version: '3.8'\n\nservices:\n\n rathole:\n image: rapiz1/rathole:v0.5.0\n container_name: rathole\n command: --client /config/rathole.client.toml\n restart: unless-stopped\n volumes:\n - ./rathole.client.toml:/config/rathole.client.toml:ro\n networks:\n - proxy\n\n traefik:\n image: 'traefik:v2.9.8'\n container_name: traefik\n restart: unless-stopped\n depends_on:\n - rathole\n command:\n # moved from static conf to pass email as env var\n - '--certificatesresolvers.letsencrypt.acme.email=${TRAEFIK_LETSENCRYPT_EMAIL}'\n security_opt:\n - no-new-privileges:true\n networks:\n - proxy\n # rathole will pass traffic through proxy network directly on 80 and 443\n # defined in rathole.client.toml\n environment:\n - TRAEFIK_AUTH=${TRAEFIK_AUTH}\n volumes:\n - /etc/localtime:/etc/localtime:ro\n - /var/run/docker.sock:/var/run/docker.sock:ro\n - ./traefik-data/traefik.yml:/traefik.yml:ro\n - ./traefik-data/acme.json:/acme.json\n - ./traefik-data/configurations:/configurations\n labels:\n - 'traefik.enable=true'\n - 'traefik.docker.network=proxy'\n - 'traefik.http.routers.traefik-secure.entrypoints=websecure'\n - 'traefik.http.routers.traefik-secure.rule=Host(`traefik.${SITE_HOSTNAME}`)'\n - 'traefik.http.routers.traefik-secure.middlewares=user-auth@file'\n - 'traefik.http.routers.traefik-secure.service=api@internal'\n\nnetworks:\n proxy:\n external: trueYou can use this setup to expose as many home servers as you want, in a cost-effective and practical way, as long as your VPS has enough network bandwidth to support their traffic. It can bring your homelab to another level.
\nWhat tool and method did you use to expose your home servers to the internet? Do you like this approach, are you willing to give it a try? Let me know in the comments.
\nHappy self-hosting.
\nv2.9 HostRegexp reference: https://doc.traefik.io/traefik/v2.9/routing/routers/#rulev2.9 HostSNIRegexp reference: https://doc.traefik.io/traefik/v2.9/routing/routers/#rule_1import { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;
\nimport RatholeTraefikArchitectureImage from ’../../../../content/post/2025/04-29-rathole-traefik-home-server/_images/rathole-traefik-architecture-16-9.png’;\nimport FirewallImage from ’../../../../content/post/2025/04-29-rathole-traefik-home-server/_images/firewall.png’;\nimport HomeServerContainersImage from ’../../../../content/post/2025/04-29-rathole-traefik-home-server/_images/home-server-containers.png’;\nimport OrangePiGifImage from ’../../../../content/post/2025/04-29-rathole-traefik-home-server/_images/orange-pi.gif’;
\nIn the previous article, I wrote about a temporary SSH tunneling technique to bypass CGNAT. This method is not suitable for exposing permanent services, at least not without autossh manager. Proper tools for this are rapiz1/rathole or fatedier/frp. I chose Rathole since it’s written in Rust and offers better performance and benchmarks.
80 and 443 for any other services aside from RatholeWe will use Rathole for an encrypted tunnel between the VPS and the local network. We will also use Traefik since we want to host multiple websites on our home server, just like you would on any server.
\nThe main question is where to run Traefik:
\nI highly prefer option 2 because, that way, the entire configuration is stored on our home server. The home server is almost tunnel-agnostic, and you can reuse it on any tunneled or non-tunneled server. Otherwise, we would need to maintain state between the VPS and the home server, debug both together, etc.
\nAnother point is that, with option 2, we avoid the gap of unencrypted traffic on the VPS between Traefik (TLS) and Rathole (Noise Protocol). You can read more about the comparison of these two options in this article: https://blog.mni.li/posts/caddy-rathole-zero-knowledge/.
\nThe downside is that Rathole will exclusively occupy ports 80 and 443 on the VPS, preventing any other process from using them. We won’t be able to run other web servers on that VPS, so it’s best to use a small one dedicated to this purpose.
Unless we use a load balancer Load balancing multiple Rathole tunnels with Traefik HTTP and TCP routers.
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={RatholeTraefikArchitectureImage} alt=“Rathole Traefik architecture diagram” />
\nWe will run the Rathole server inside a Docker container on our VPS. Rathole uses the same binary for both the server and client, you just pass the right option (--server or --client) and the .toml configuration file.
Here is the Rathole server configuration rathole.server.toml:
\n[server]\nbind_addr = \"0.0.0.0:2333\"\n\n[server.transport]\ntype = \"noise\"\n\n[server.transport.noise]\nlocal_private_key = \"private_key\"\n\n[server.services.traefik-http]\ntoken = \"secret_token_1\"\nbind_addr = \"0.0.0.0:5080\"\n\n[server.services.traefik-https]\ntoken = \"secret_token_1\"\nbind_addr = \"0.0.0.0:5443\"Let’s explain it: we choose port 2333 for the control channel and bind it to all interfaces inside the Docker container with the 0.0.0.0 IP. We choose the noise encryption protocol and specify a private key. The public key will be used on the Rathole client. The public and private key pair is generated with:
docker run -it --rm rapiz1/rathole --genkeyThen we define two tunnels: one for HTTP and another for HTTPS. For the HTTP tunnel, we define the name server.services.traefik-http, set the value for token, and choose port 5080, and again we bind it to all container interfaces with 0.0.0.0. Similarly, for HTTPS, we set the name to server.services.traefik-https, provide a token value, and choose port 5443.
Every tunnel has to have a unique name, token value, and port. With that fulfilled, a single Rathole server instance can have as many Rathole clients as needed, which is pretty convenient. For example, besides the existing home server on ports 5080 and 5443, we can expose another one using ports 5081 and 5444.
Token is just a random base64 string, we generate it by running this:
\nopenssl rand -base64 32After configuration file we define a Rathole server container with docker-compose.yml:
\nservices:\n\n rathole:\n image: rapiz1/rathole:v0.5.0\n container_name: rathole\n command: --server /config/rathole.server.toml\n restart: unless-stopped\n ports:\n # host:container\n - 2333:2333\n - 80:5080\n - 443:5443\n volumes:\n - ./rathole.server.toml:/config/rathole.server.toml:roIn the command, we set the --server option, pass the .toml configuration file, and mount it as a read-only bind-mount volume.
The important part is the port mappings. Here, you can see that the Rathole server container will occupy ports 2333, 80, and 443 exclusively on the host VPS. This practically means we won’t be able to run any other web servers on ports 80 and 443. We will also need to open ports 80, 443, and 2333 in the VPS firewall. You don’t need to open ports 5080 and 5443, those are used only by Rathole internally.
We run the Rathole client and Traefik inside Docker containers on the home server. Configuring the Rathole client and connecting it to Traefik is a bit more complex and tricky.
\nHere is the Rathole client configuration core/rathole.client.toml.example:
\n[client]\nremote_addr = \"123.123.123.123:2333\"\n\n[client.transport]\ntype = \"noise\"\n\n[client.transport.noise]\nremote_public_key = \"public_key\"\n\n# this is the important part\n# Rathole knows traffic comes from 5080 and 5443, control channel told him\n# DON'T do ANY mapping in docker-compose.yml\n# just pass traffic from Rathole on ports which Traefik expects (80 and 443)\n\n[client.services.traefik-http]\ntoken = \"secret_token_1\"\nlocal_addr = \"traefik:80\"\n\n[client.services.traefik-https]\ntoken = \"secret_token_1\"\nlocal_addr = \"traefik:443\"Let’s go through it. First, we define the VPS server IP remote_addr, the control channel port 2333, set the noise encryption protocol, and this time specify a public key remote_public_key.
Now comes the important and tricky part: defining tunnels and services. We repeat the service name and token that we used in the Rathole server config.
\nAnd now the most important part: local_addr, for this we target the Traefik hostname - service name from core/docker-compose.local.yml and the Traefik listening ports 80 and 443. That’s it. It might look simple and obvious, this is the correct setup. I must emphasize: don’t fall into temptation of setting any additional port mappings in core/docker-compose.local.yml, functionality will break, all should be done in core/rathole.client.toml.
Another note: You might wonder why ports 5080 and 5443 aren’t repeated anywhere in the client config core/rathole.client.toml. The answer is “no need for it”, we already specified port 2333 for the control channel, which will communicate all additional required information between the Rathole server and client.
Now that we have configured the Rathole client, we need to define Rathole client and Traefik containers.
\nHere is the Rathole client container and the important part of the Traefik container core/docker-compose.local.yml:
\nservices:\n\n rathole:\n # 1. default official x86 image\n image: rapiz1/rathole:v0.5.0\n\n # 2. custom built ARM image (for Raspberry pi)\n # image: nemanjamitic/my-rathole-arm64:v1.0\n\n # 3. build for arm - AVOID, use prebuilt ARM image above\n # build: https://github.com/rapiz1/rathole.git#main\n # platform: linux/arm64\n\n container_name: rathole\n command: --client /config/rathole.client.toml\n restart: unless-stopped\n volumes:\n - ./rathole.client.toml:/config/rathole.client.toml:ro\n networks:\n - proxy\n\n traefik:\n image: 'traefik:v2.9.8'\n container_name: traefik\n restart: unless-stopped\n\n # for this to work both services must be defined in the same docker-compose.yml file\n depends_on: \n - rathole\n\n # other config...\n\n networks:\n - proxy\n\n # leave this commented out, just for explanation\n # Rathole will pass Traffic through proxy network directly on 80 and 443\n # defined in rathole.client.toml\n # ports:\n # - '80:80'\n # - '443:443'\n\n # other config...Let’s start with the Rathole service. Similarly to the server command, we run the Rathole binary, this time in client mode with --client and we pass the client config file /config/rathole.client.toml which we also bind mount as volume. An important part is that we set both the Rathole and Traefik containers on the same external network proxy so they can communicate with each other and with the host.
Additional notes about the Rathole image:
\nx86 - By default, Rathole provides only the x86 image. If your home server uses that architecture, you are good to go.ARM - If you have an ARM home server (e.g., Raspberry Pi), you will have to build the image yourself or use a prebuilt, unofficial one. Avoid building images on low-power ARM single-board computers, as it will take a long time and require a lot of RAM and CPU power. Instead, either pre-build one yourself and push it to Docker Hub, or you can reuse my nemanjamitic/my-rathole-arm64:v1.0 image (which uses Rathole v0.5.0).Now, the Traefik container. It must be on the same proxy external network as Rathole. Another important part: It must wait for the Rathole container to boot up depends_on: rathole, because the traffic will come from the Rathole tunnel. Do not expose ports 80 and 443, Rathole has already bound those Traefik container ports, as we defined in the Rathole client config core/rathole.client.toml.
The rest of the Traefik container definition is left out here because it’s the usual configuration, unrelated to the Rathole tunnel. Below is a quick reminder about the general Traefik configuration.
\nTraefik reminder
\n.env file with variables needed for Traefik:cp .env.example .envSITE_HOSTNAME=homeserver.my-domain.com\n\n# important: must put value in quotes \"...\" and escape $ with \\$\nTRAEFIK_AUTH=\n\n# will receive expiration notifications\nTRAEFIK_LETSENCRYPT_EMAIL=myname@example.comdocker network create proxyacme.json file with permission 600:touch ~/homelab/traefik-proxy/core/traefik-data/acme.json\n\nsudo chmod 600 ~/homelab/traefik-proxy/core/traefik-data/acme.json# core/traefik-data/traefik.yml\n\ncertificatesResolvers:\n letsencrypt:\n acme:\n # always start with staging certificate\n caServer: \"https://acme-staging-v02.api.letsencrypt.org/directory\"\n # caServer: 'https://acme-v02.api.letsencrypt.org/directory'acme.jsontruncate -s 0 acme.jsonThat’s it. Once done, we can run Rathole client and Traefik containers on our home server with:
\ndocker compose -f docker-compose.local.yml up -d<Image {…IMAGE_SIZES.FIXED.MDX_XL} src={HomeServerContainersImage} alt=“Running containers on the home server” />
\nFortunately, Rathole makes it trivial to run multiple tunnels using a single Rathole server. We don’t need to open any additional ports in the firewall or run multiple container instances. What we do need are different tunnel names, token values, and ports. Those must be unique for each tunnel/service. Also, you will need a load balancer to bind ports 80 and 443 to more than one destination port, respectively.
I wrote a detailed tutorial on how to expose multiple home servers using a single Rathole server. You can read it here: Load balancing multiple Rathole tunnels with Traefik HTTP and TCP routers.
\nLike for any webserver, on the VPS you will need to open ports 80 and 443 to listen for HTTP/HTTPS traffic. Additionally you will need to open the port 2333 for the Rathole control channel - tunnel.
<Image {…IMAGE_SIZES.FIXED.MDX_XL} src={FirewallImage} alt=“Opened port for Rathole tunnel in the firewall” />
\nMost consumer-grade internet connections are behind a CGNAT. This setup allows you to bypass CGNAT and host an unlimited number of websites on your home server almost for free. You can use it for web servers in virtual machines, LXC containers, SBC computers, etc. - anywhere you can run Docker.
\nIt is simple, cheap, and you can set it up in 30 minutes. Like anything, it also has some downsides, one of them is the overhead latency caused by an additional network hop between the VPS and your home network, but it’s a reasonable tradeoff.
\nDid you make something similar yourself? Can you see room for improvement? Did you use a different method? You tried to run the code and need help with troubleshooting? Let me know in the comments.
\n.toml environment variables discussion https://github.com/rapiz1/rathole/issues/218import { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;
\nimport SshTunnelArchitectureImage from ’../../../../content/post/2025/04-20-ssh-tunnel-docker/_images/ssh-tunnel-architecture-16-9.png’;\nimport FirewallImage from ’../../../../content/post/2025/04-20-ssh-tunnel-docker/_images/firewall.png’;
\nimport TunnelDemoVideo from ’../../../../content/post/2025/04-20-ssh-tunnel-docker/_images/tunnel-demo.webm’;
\nMost consumer-grade internet connections are hidden behind CGNAT and are not reachable from the internet. This is done to save IP addresses, as IPv4 has a limited range. If you have a static public IPv4 or any IPv6 address, you won’t need the setup from this tutorial.
\nThere are already services like localtunnel or ngrok for this purpose, but when you actually start using them, you will often find out that they have limitations on their free plans. So, we will configure our own custom setup once and have it always available for convenient and practical usage which will save a lot of time and nerves in the long run.
\nThis is useful whenever you need to share your local project with others or provide a publicly accessible URL for your service so that external systems can reach it. This is often the case if you work remotely.
\nYes, you can use test deployments, but having a tunnel setup configured and being able to run it with a single terminal command saves a lot of time and energy.
\nPossible use cases:
\n<video {…IMAGE_SIZES.FIXED.MDX_LG} controls>\n
Without going too deep into computer networking theory, let’s explain port forwarding in simplified terms. Port forwarding is a mapping (binding) between two points (services) on a private network (or even on the same machine) that would otherwise be unreachable. You can think of it as a VPN for a single service (port).
\nSo it’s exactly what we need: we want to bind (redirect traffic from) a public port (1081 in our case) on the VPS, which acts as a gateway, to port 3000 on our local dev server that is not directly reachable from the internet. That’s it for the tunneling part, this setup is sufficient for serving HTTP traffic.
\nAdditionally, to support HTTPS and provide a user-friendly URL, we will add Traefik, which will handle HTTPS certificates and route traffic from port 443 to port 1081 of the tunnel.
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={SshTunnelArchitectureImage} alt=“SSH tunnel architecture diagram” />
\nWe already use SSH to access our VPS, but we prefer to keep that configuration untouched. So, we will run a separate SSH server inside a Docker container specifically for tunneling.
\nFor this, we will use linuxserver/openssh-server image. Linuxserver is an organization that maintains very stable Dokcer images for all kinds of purposes.
\nBy default, the SSH server doesn’t allow tunneling, so we need to modify the config in /etc/ssh/sshd_config and enable it.
AllowTcpForwarding yes\nGatewayPorts yessudo nano /etc/ssh/sshd_config\n\n# edit config...\n\nsudo systemctl restart sshdBut since we are using a Docker container, we will do it differently.
\nWe will use openssh-server-ssh-tunnel mod, which enables tunnelling in the linuxserver/openssh-server image. You can think of mods as presets (additional layers and configurations) for these images.
Here is docker-compose.yml for the SSH tunnel container:
version: '3.8'\n\nservices:\n openssh-server:\n image: linuxserver/openssh-server\n container_name: openssh-server\n restart: unless-stopped\n hostname: openssh-server #optional\n expose:\n - 1081 # tunneled service port, for Traefik\n ports:\n - 1080:2222 # 1080 is the main SSH connection port\n environment:\n # https://github.com/linuxserver/docker-mods/tree/openssh-server-ssh-tunnel\n - DOCKER_MODS=linuxserver/mods:openssh-server-ssh-tunnel\n - SHELL_NOLOGIN=false\n # set correct for current host user\n - PUID=1001\n - PGID=1001\n - TZ=Etc/UTC\n # important\n - PUBLIC_KEY \n # optional env vars bellow\n - SUDO_ACCESS=true \n - USER_NAME=username \n - PASSWORD_ACCESS=false \n volumes:\n - ./config:/config\n # Traefik configuration bellow\n labels:\n - 'traefik.enable=true'\n - 'traefik.docker.network=proxy'\n - 'traefik.http.routers.ssh-tunnel.rule=Host(`preview.${SITE_HOSTNAME}`)'\n - 'traefik.http.routers.ssh-tunnel.entrypoints=websecure'\n - 'traefik.http.routers.ssh-tunnel.service=ssh-tunnel'\n - 'traefik.http.services.ssh-tunnel.loadbalancer.server.port=1081' # matches exposed port\n networks:\n - proxy\n\nnetworks:\n proxy:\n external: trueLets explain the code above:
\nservices:\n openssh-server:\n image: linuxserver/openssh-server\n # ...\n ports:\n - 1080:2222 # 1080 is the main SSH connection portBy default, linuxserver/docker-openssh-server runs the SSH service on port 2222, to avoid conflicting with the usual port 22 that is used for host’s SSH service and it’s hardcoded in the Dockerfile. We will choose port 1080 for the main SSH connection, so we need to map it to port 2222 with SSH in the container. Port 1080 is used for the actual connection over the internet, and it is required to allow that port in VPS firewall.
So, let’s establish clear and precise naming from the beginning:
\n1080 - the main SSH connection port1081, 1082, 1083, ... - tunneled services remote portsAdditionally, you need to configure the SSH client on your dev machine to use port 1080 for SSH when connecting to this container. In this example I have named VPS host amd1 and SSH container host amd1c, you can use your own naming logic.
\n# ssh amd1 ssh container\nHost amd1c 123.123.123.123 # VPS IP\n HostName 123.123.123.123\n IdentityFile ~/.ssh/my-keys/amd1_ssh_container__id_ed25519 # private key file name\n User username\n Port 1080In the client SSH config above, you will notice the private key file amd1_ssh_container__id_ed25519. The public key is passed to the SSH container as an environment variable:
services:\n openssh-server:\n image: linuxserver/openssh-server\n # ...\n environment:\n - PUBLIC_KEY # importantYou generate SSH key pairs as usual, e.g.:
\nssh-keygen -t ed25519 -C \"myemail@gmail.com\" -f ~/.ssh/my-keys/amd1_ssh_container__id_ed25519Now, we choose which remote port we will use to expose our local dev server. If you’re using Traefik and don’t access this port directly via the browser, you don’t need to allow it in the VPS’s firewall.
\nservices:\n openssh-server:\n image: linuxserver/openssh-server\n # ...\n expose:\n - 1081 # tunneled service remote portThe other environment variables worth mentioning are the following:
\nservices:\n openssh-server:\n image: linuxserver/openssh-server\n # ...\n environment:\n # https://github.com/linuxserver/docker-mods/tree/openssh-server-ssh-tunnel\n - DOCKER_MODS=linuxserver/mods:openssh-server-ssh-tunnel\n - PUID=1001\n - PGID=1001We use DOCKER_MODS variable to specify openssh-server-ssh-tunnel mod. PUID and PGID are user and group IDs used to handle permissions between the host and the container. You get their values by running id -u && id -g on the VPS host. It is also a good idea to export them as global environment variables in ~/.bashrc file to make them available for all containers:
export MY_UID=$(id -u)\nexport MY_GID=$(id -g)Then you can pass them like this:
\n # ...\n environment:\n - PUID=$MY_UID\n - PGID=$MY_GIDThe SSH tunnel is now configured. Now you can access your local dev server by HTTP via the your VPS IP e.g. http://123.123.123.123:1081 or domain http://my-domain.com:1081.
Some browsers disallow insecure HTTP traffic by default, and you need to tweak the browser settings to allow it explicitly. This can be inconvenient when sending a demo link to a non-technical person. Additionally, some OAuth providers require HTTPS even for testing (e.g. Facebook). So let’s make an extra effort to do things properly and configure a HTTPS with a subdomain using Traefik.
\nIf you are running a VPS, chances are you already use a reverse proxy for handling certificates and subdomain routing. This example shows how to do it with Traefik.
\n\nservices:\n openssh-server:\n image: linuxserver/openssh-server\n container_name: openssh-server\n # ...\n expose:\n - 1081 # tunneled service remote port\n ports:\n - 1080:2222 # 1080 is the main SSH connection port\n\n # Traefik configuration bellow\n labels:\n - 'traefik.enable=true'\n - 'traefik.docker.network=proxy'\n - 'traefik.http.routers.ssh-tunnel.rule=Host(`preview.${SITE_HOSTNAME}`)'\n - 'traefik.http.routers.ssh-tunnel.entrypoints=websecure'\n - 'traefik.http.routers.ssh-tunnel.service=ssh-tunnel'\n - 'traefik.http.services.ssh-tunnel.loadbalancer.server.port=1081' # matches exposed port\n networks:\n - proxy\n\nnetworks:\n proxy:\n external: trueThe truth is, there is not much work to do here. All you need to do is to map the remote port of the tunnel 1081 to Traefik and define the URL on which you want to expose your local dev server via the environment variable e.g. SITE_HOSTNAME=preview.my-domain.com.
Everything else is just generic Traefik configuration. Also, don’t forget to add the wildcard A record for your subdomains (e.g., you might add a *.tunnels “namespace”) in your DNS provider’s dashboard and point it to your VPS IP. Additionally, create an external Docker network, e.g. named proxy as shown in the example above.
services:\n openssh-server:\n image: linuxserver/openssh-server\n container_name: openssh-server\n # ...\n expose:\n - 1081 # tunneled service remote port, passed to Traefik\n\n # ...\n # Traefik configuration bellow\n labels:\n # ...\n - 'traefik.http.routers.ssh-tunnel.rule=Host(`preview.${SITE_HOSTNAME}`)' # in .env file: SITE_HOSTNAME=my-domain.com\n - 'traefik.http.services.ssh-tunnel.loadbalancer.server.port=1081' # matches the exposed portIn the end, you just need to define 2 environment variables for your docker-compose.yml inside the .env file:
# full with subdomain, without 'https://'\nSITE_HOSTNAME=my-domain.com # or e.g. preview.my-domain.com\n\n# public ssh key\nPUBLIC_KEY=my-public-ssh-keyAbove is shown only the relevant Traefik configuration for the SSH tunnel container. A complete Traefik reverse proxy configuration requires additional static and dynamic configurations for the Traefik container, but that is outside the scope of this tutorial. You can search for examples of Traefik configurations or reuse mine, which is available in this repository: nemanjam/traefik-proxy.
\nSometimes your app runs more than a single service, e.g. frontend and backend. If you expose just the frontend from port 3000, note that localhost from, e.g. localhost:5000 won’t be resolved. Therefore, you need to tunnel all services and set the tunneled URLs in your .env files.
How to have more than one tunnel? Your first thought might be to run multiple SSH server containers, but fortunately, that is not necessary. You can tunnel as many services as you want through a single SSH connection. You just need to expose multiple ports on the SSH container and map them to multiple Traefik hosts with labels, as shown below:
\nversion: '3.8'\n\nservices:\n openssh-server:\n image: linuxserver/openssh-server\n container_name: openssh-server\n restart: unless-stopped\n hostname: openssh-server #optional\n # tunneled services, remote ports\n expose:\n - 1081 # tunnel1\n - 1082 # tunnel2\n - 1083 # tunnel3\n ports:\n - 1080:2222 # 1080 is the main SSH connection port\n environment:\n # https://github.com/linuxserver/docker-mods/tree/openssh-server-ssh-tunnel\n - DOCKER_MODS=linuxserver/mods:openssh-server-ssh-tunnel\n - SHELL_NOLOGIN=false\n # set correct for current host user\n - PUID=1001\n - PGID=1001\n - TZ=Etc/UTC\n # important\n - PUBLIC_KEY \n # optional env vars bellow\n - SUDO_ACCESS=true \n - USER_NAME=username \n - PASSWORD_ACCESS=false \n volumes:\n - ./config:/config\n # Traefik configuration bellow\n labels:\n # common config\n - 'traefik.enable=true'\n - 'traefik.docker.network=proxy'\n\n # tunnel1 (port 3000 -> 1081)\n - 'traefik.http.routers.ssh-tunnel1.rule=Host(`preview1.${SITE_HOSTNAME}`)'\n - 'traefik.http.routers.ssh-tunnel1.entrypoints=websecure'\n - 'traefik.http.routers.ssh-tunnel1.service=ssh-tunnel1'\n - 'traefik.http.services.ssh-tunnel1.loadbalancer.server.port=1081'\n\n # tunnel2 (port 5000 -> 1082)\n - 'traefik.http.routers.ssh-tunnel2.rule=Host(`preview2.${SITE_HOSTNAME}`)'\n - 'traefik.http.routers.ssh-tunnel2.entrypoints=websecure'\n - 'traefik.http.routers.ssh-tunnel2.service=ssh-tunnel2'\n - 'traefik.http.services.ssh-tunnel2.loadbalancer.server.port=1082'\n\n # tunnel3 (port 5001 -> 1083)\n - 'traefik.http.routers.ssh-tunnel3.rule=Host(`preview3.${SITE_HOSTNAME}`)'\n - 'traefik.http.routers.ssh-tunnel3.entrypoints=websecure'\n - 'traefik.http.routers.ssh-tunnel3.service=ssh-tunnel3'\n - 'traefik.http.services.ssh-tunnel3.loadbalancer.server.port=1083'\n\n networks:\n - proxy\n\nnetworks:\n proxy:\n external: trueIf you have a large number of services to tunnel, you might want to use a VPN to access all ports by default, but that’s rarely the case.
\nAnother point to make is that the SSH tunnel technique is most suitable for temporarily exposing services for demo purposes. For permanent tunnels, you would need to add autossh to keep the connection alive, but there are better tools for permanent tunnels, such as rapiz1/rathole or fatedier/frp.
For the main SSH connection, you will need to open a port in your VPS firewall, port 1080 in this example. Additionally, if you want to access tunnels directly via a port in the browser without Traefik, you will need to open those ports as well. Be mindful not to open too many unnecessary ports, as every newly opened port increases the attack surface.
<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={FirewallImage} alt=“Example opened ports in the firewall” />
\nYou start the tunnel with a single command like below. The -R option means remote port forwarding, followed by two IP:port pairs. The first pair is remote, and the second is local. At the end, you have the VPS host.
# command format\nssh -R [remote_addr:]remote_port:local_addr:local_port [user@]gateway_addr\n\n# example:\n# amd1c host is defined in ~/.ssh/config\nssh -R *:1081:localhost:3000 amd1c\n\n# access the url, e.g.\nhttps://preview1.my-domain.com\n\n# terminate tunnel, like any ssh connection\nexitYou can open multiple tunnels with a single command. Just specify the tunnels one after another before the host. Note that you must have these tunnels defined in your docker-compose.yml for the SSH server (exposed ports and Traefik host labels).
# tunnel frontend at port 3000 and backend at port 5000\nssh \\\n -R *:1081:localhost:3000 \\\n -R *:1082:localhost:5000 \\\n amd1c\n\n# access the urls, e.g.\nhttps://preview1.my-domain.com\nhttps://preview2.my-domain.com/apiPort forwarding is a basic networking technique that is very familiar to network engineers, but perhaps not often utilized by developers. It can be very useful and practical, especially in a remote work setting. As described in this tutorial, you just need to run a single container, configure the client and firewall, and once you have it set up, it can save you a lot of time and energy in the long run.
\nSSH remote port forwarding is just one of the many useful and cool SSH networking tricks. There are many others like dynamic port forwarding, SSH agent forwarding, X11 forwarding, SSH file system, etc. Do you use some of them? Please share in the comments bellow.
\nlinuxserver/docker-openssh-server image repository https://github.com/linuxserver/docker-openssh-serveropenssh-server-ssh-tunnel mod repository https://github.com/linuxserver/docker-mods/tree/openssh-server-ssh-tunnelimport { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;
\nimport LayoutShiftImage from ’../../../../content/post/2025/04-06-random-image-component/_images/layout-shift.png’;
\nimport OverviewVideo from ’../../../../content/post/2025/04-06-random-image-component/_images/overview.webm’;\nimport ResponsiveImagesVideo from ’../../../../content/post/2025/04-06-random-image-component/_images/responsive-images.webm’;
\nFor the sake of practice and fun let’s build a component that displays a random image on mouse click. It looks more fun and interactive than a static hero image. You can see it in action on the Home page of the my website.
\nThis functionality shares some common parts with the image gallery described in the previous article, such as the component hierarchy and including urls in the client. However, it also introduces some new elements, like a proper blur preloader.
\n<video {…IMAGE_SIZES.FIXED.MDX_LG} controls>\n
Again, we will use the similar structure MDX (index.mdx) -> Astro component (ImageRandom.astro) -> React components (ImageRandomReact.jsx and ImageBlurPreloader.jsx), and again, the client React components contain the most complexity.
Code (paraphrased):
\n// src/pages/index.mdx\n\n<ImageRandom />\n\n// src/components/ImageRandom.astro\n\n<ImageRandomReact {galleryImages} client:load />\n\n// src/components/react/ImageRandomReact.tsx\n\n<ImageBlurPreloader {...props} />This is the functionality shared with the image gallery. This time, we’ll use a fixed low-resolution image for the blur effect and a responsive, high-resolution hero image as the main one. The blur and main images will have different resolutions but share the same 16:9 aspect ratio.
The code is as follows:
\n// src/constants/image.ts\n\nexport const IMAGE_SIZES = {\n FIXED: {\n // blur image\n BLUR_16_9: {\n width: 64,\n height: 36,\n },\n // ... \n },\n RESPONSIVE: {\n // main image\n POST_HERO: {\n widths: [TW_SCREENS.XS, TW_SCREENS.SM, TW_SCREENS.MD, TW_SCREENS.LG],\n sizes: `(max-width: ${TW_SCREENS.XS}px) ${TW_SCREENS.XS}px, (max-width: ${TW_SCREENS.SM}px) ${TW_SCREENS.SM}px, (max-width: ${TW_SCREENS.MD}px) ${TW_SCREENS.MD}px, ${TW_SCREENS.LG}px`,\n },\n // ... \n },\n};\n\n// actual <img /> tag attributes that are generated with the POST_HERO\n\n<img\n sizes=\"(max-width: 475px) 475px, (max-width: 640px) 640px, (max-width: 768px) 768px, 1024px\"\n width=\"3264\"\n height=\"1836\"\n srcset=\"\n /_astro/amfi1.Cv2xkJ5B_1Lofkq.webp 475w,\n /_astro/amfi1.Cv2xkJ5B_Oxmi8.webp 640w,\n /_astro/amfi1.Cv2xkJ5B_X0wXS.webp 768w,\n /_astro/amfi1.Cv2xkJ5B_Z1u01H4.webp 1024w\n \"\n src=\"/_astro/amfi1.Cv2xkJ5B_26HGs8.webp\"\n/>\n\n// src/libs/gallery/transform.ts\n\nexport const heroImageOptions = {\n ...IMAGE_SIZES.RESPONSIVE.POST_HERO,\n};\n\n// src/libs/gallery/images.ts\n\nexport const getHeroImages = async (): Promise<HeroImage[]> => {\n const blur = await getCustomImages(blurImageOptions);\n const hero = await getCustomImages(heroImageOptions);\n\n const heroImages = mergeArrays(blur, hero).map(([blur, hero]) => ({\n blur: imageResultToImageAttributes(blur),\n hero: imageResultToImageAttributes(hero),\n }));\n\n return heroImages;\n};\n\n// src/components/ImageRandom.astro\n\nconst galleryImages = await getHeroImages();Responsive main image in action:
\n<video {…IMAGE_SIZES.FIXED.MDX_LG} controls>\n
Again, we have the same situation as in the image gallery. The key point is to include all image urls in the client and execute getRandomElementFromArray() in the client React component to display a random image at runtime. If we called the random function on the server, in the Astro component, we would end up with a single image that was randomly picked at build time - which is not what we want.
This is the code:
\n---\n// src/components/ImageRandom.astro\n\nconst galleryImages = await getHeroImages();\n---\n\n{/* include all the images in the client and let the client pick the random image */}\n\n<div {...props}>\n <ImageRandomReact {galleryImages} client:load />\n</div>\n\n// src/components/react/ImageRandom.tsx\n\nconst ImageRandomReact: FC<Props> = ({ galleryImages, className, divClassName, ...props }) => {\n // cache randomized images\n const randomImage = useMemo(() => getRandomElementFromArray(galleryImages), [galleryImages]);\n\n const [image, setImage] = useState(initialImage);\n\n // pick initial random image on mount\n useEffect(() => {\n setImage(randomImage);\n }, [setImage, randomImage]);\n\n // pick random image onClick\n const handleClick = async () => {\n const randomImage = getRandomElementFromArray(galleryImages);\n setImage(randomImage);\n };\n\n return (\n <ImageBlurPreloader\n {...props}\n blurAttributes={{ ...image.blur, alt: 'Blur image' }}\n mainAttributes={{ ...image.hero, onClick: handleClick, alt: 'Hero image' }}\n className={cn('cursor-pointer my-0', className)}\n divClassName={divClassName}\n />\n );\n};\nThis is the most interesting part of the feature. The first instinct when swapping the blur and main images might be to use a ternary operator to mount or unmount the appropriate image. But we actually can’t do that here. Why? Because both images need to remain mounted in the DOM to ensure the onLoad event works correctly for both the blur and main images. So instead of unmounting, we will use absolute positioning to place the main image above the blur image and toggle its opacity to show or hide it.
But there is more. Note that with the onLoad event, we have three possible values for the image’s src attribute (although the main image actually uses the srcset and sizes attributes). These are:
'' when both blur and main images are still loading. In this case we will show an empty <div /> of the same size as the main image.src attribute of the blur image, when the blur image is loaded but the main image is still loading.srcset and sizes attributes of the main image, once the main image has fully loaded.This is the code src/components/react/ImageBlurPreloader.tsx:
\nconst initialAttributes: ImgTagAttributes = { src: '' } as const;\n\nconst ImageBlurPreloader: FC<Props> = ({\n blurAttributes = initialAttributes,\n mainAttributes = initialAttributes,\n onMainLoaded,\n className,\n divClassName,\n}) => {\n const [isLoadingMain, setIsLoadingMain] = useState(true);\n const [isLoadingBlur, setIsLoadingBlur] = useState(true);\n\n const prevMainAttributes = usePrevious(mainAttributes);\n\n const isNewImage = !(\n prevMainAttributes?.src === mainAttributes.src &&\n prevMainAttributes.srcSet === mainAttributes.srcSet\n );\n\n // reset isLoading on main image change\n useEffect(() => {\n if (isNewImage) {\n setIsLoadingBlur(true);\n setIsLoadingMain(true);\n }\n }, [isNewImage, setIsLoadingMain, setIsLoadingBlur]);\n\n // important: main image must be in DOM for onLoad to work\n // unmount and display: none; will fail\n const handleLoadMain = () => {\n setIsLoadingMain(false);\n onMainLoaded?.();\n };\n\n const commonAttributes = {\n // blur image must use size from main image\n width: mainAttributes.width,\n height: mainAttributes.height,\n };\n\n const blurAlt = !isLoadingBlur ? blurAttributes.alt : '';\n const mainAlt = !isLoadingMain ? mainAttributes.alt : '';\n\n const hasImage = Boolean(\n isLoadingMain\n ? mainAttributes.src || mainAttributes.srcSet\n : blurAttributes.src || blurAttributes.srcSet\n );\n\n return (\n <div className={cn('relative size-full', divClassName)}>\n {hasImage && (\n <>\n {/* blur image */}\n <img\n {...blurAttributes}\n {...commonAttributes}\n alt={blurAlt}\n onLoad={() => setIsLoadingBlur(false)}\n className={cn('object-cover absolute top-0 left-0 size-full', className)}\n />\n\n {/* main image */}\n <img\n {...mainAttributes}\n {...commonAttributes}\n alt={mainAlt}\n onLoad={handleLoadMain}\n className={cn(\n 'object-cover absolute top-0 left-0 size-full',\n // important: don't hide main image until next blur image is loaded\n isLoadingMain && !isLoadingBlur ? 'opacity-0' : 'opacity-100',\n className\n )}\n />\n </>\n )}\n </div>\n );\n};That is a lot of code, so let’s break it down. First, note the use of relative and absolute classes to position the images on top of each other.
We set the initial src to an empty string in the initialAttributes variable. This sets the hasImage flag to true, unmounts the images, and displays an empty <div> that fills the parent container thanks to the size-full class (which is needed to prevent layout shift).
Next, note that we track the separate states isLoadingMain and isLoadingBlur for the main and blur images. Both are necessary so we can correctly show/hide the main image by changing its opacity from opacity-0 to opacity-100. The general idea is this: “Always keep the blur image below, just show or hide the main image above.”
Additionally, we track the previous main image, prevMainAttributes, to detect when a new image is selected via the onClick event passed from the parent component.
Finally, while an image is loading, we set its alt attribute (using the blurAlt and mainAlt variables) to an empty string to avoid rendering text in place of an empty image, as it doesn’t look nice.
Bonus tip: You can also experiment with the <img style={{imageRendering: 'pixelated'}} /> scaling style on the blur image if you find it more aesthetically pleasing.
This is also an interesting part. In general, the server always sends images with their sizes (at least it should), which makes handling layout shifts easier, so we should be able to solve it properly.
\nThe key point is this: Set the component’s actual size in the server component ImageRandom.astro and use w-full h-full (size-full) in the client ImageRandom.tsx React component to stretch it to fill the parent. This way, the size is resolved on the server, and there is no shift when hydrating the client component.
Lets see it in practice src/components/ImageRandom.astro#L21
\n---\n// add 'px' suffix or styles will fail\nconst { width, height } = Object.fromEntries(\n Object.entries(IMAGE_SIZES.FIXED.MDX_XL_16_9).map(([key, value]) => [key, `${value}px`])\n);\n---\n\n{/* height and width MUST be defined ON SERVER component to prevent layout shift */}\n{/* set height and width to image size but set real size with max-height and max-width */}\n\n<div\n class={cn('max-w-full max-h-64 md:max-h-96 my-8', className)}\n style={{ width, height }}\n {...props}\n>\n <ImageRandomReact {galleryImages} client:load />\n</div>\nWe use the max-w-... and max-h-... classes to set the actual (responsive) size for the server component, which the client component will fill.
The my-8 margin is there to override the vertical margin styles for the image component in the markdown (prose class). Remember, we have two actual, absolutely positioned <img /> tags in the DOM, so prose will add double margins, and we need to correct that.
Client component src/components/react/ImageBlurPreloader.tsx:
\nconst ImageBlurPreloader: FC<Props> = ({\n // ...\n className,\n divClassName,\n}) => {\n\n // ...\n\n return (\n <div className={cn('relative size-full', divClassName)}>\n {hasImage && (\n <>\n {/* blur image */}\n <img\n className={cn('object-cover absolute top-0 left-0 size-full', className)}\n />\n\n {/* main image */}\n <img\n className={cn(\n 'object-cover absolute top-0 left-0 size-full',\n )}\n />\n </>\n )}\n </div>\n );\n};In the client component, we simply stretch all elements with size-full to fill the parent server component.
With this in place, we achieve the following score for the cumulative layout shift:
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={LayoutShiftImage} alt=“Lighthouse score layout shift” />
\nThe relevant files:
\n# https://github.com/nemanjam/nemanjam.github.io/tree/c1e105847d8e7b4ab4aaffad3078726c37f67528\ngit checkout c1e105847d8e7b4ab4aaffad3078726c37f67528\n\n# random image code\nsrc/pages/index.mdx\nsrc/components/ImageRandom.astro\nsrc/components/react/ImageRandom.tsx\nsrc/components/react/ImageBlurPreloader.tsx\n\n# common code with gallery\nsrc/libs/gallery/images.ts\nsrc/libs/gallery/transform.ts\nsrc/constants/image.tsOnce again, we played around with images, Astro, and React. Have you implemented any similar components yourself, maybe a carousel? What was your approach? Do you have suggestions for improvements or have you spotted anything incorrect? Don’t hesitate to leave a comment below.
\ngetImage() function https://docs.astro.build/en/recipes/build-custom-img-component/import { Image } from ‘astro:assets’;
\nimport { IMAGE_SIZES } from ’../../../../constants/image’;
\nimport LayoutShiftBeforeImage from ’../../../../content/post/2025/04-02-astro-react-gallery/_images/layout-shift-before.png’;\nimport LayoutShiftAfterImage from ’../../../../content/post/2025/04-02-astro-react-gallery/_images/layout-shift-after.png’;
\nimport OverviewVideo from ’../../../../content/post/2025/04-02-astro-react-gallery/_images/overview.webm’;\nimport ResponsiveImagesVideo from ’../../../../content/post/2025/04-02-astro-react-gallery/_images/responsive-images-1.5x.webm’;\nimport InfiniteScrollLoaderVideo from ’../../../../content/post/2025/04-02-astro-react-gallery/_images/infinite-scroll-loader.webm’;
\nI wanted to have a simple, Instagram-like, scroll paginated gallery page on the website where I could share my everyday photos. Initially I implemented it using benhowell/react-grid-gallery package for gallery, and frontend-collective/react-image-lightbox for lightbox component. It worked ok, but since those are a bit legacy packages I was unable to upgrade to React 19, it loaded all images at once without scroll pagination and Lighthouse score wasn’t so great.
\nYou can see that implementation if you navigate back in Git history e0165b:
\n# in git history navigate back to the old gallery commit\ngit checkout e0165b295db2ccc72bbbb7be4bdd7eb48f7dedae\n\n# preview\nyarn clean && yarn install && yarn devI decided to reimplement it, did a quick research and decided to make my own gallery component and use dimsemenov/photoswipe package for lightbox. And that’s how this article got created, while implementing I took notes about the most important and interesting parts from the process. Look at it as not necessarily the absolute best way to make image gallery with Astro and React but as one of the ways that is proven in practice and works well.
\n{/* overview.webm - https://github.com/user-attachments/assets/59744bb9-3c87-4e2c-9b10-1c830f9af554 */}\n<video {…IMAGE_SIZES.FIXED.MDX_LG} controls>\n
This is the first dilemma and initial decision that affects all the future code that we write. Since this is a static website example we are naturally inclined to pre-render everything we can at build time, but can this work for images too?
\nAstro provides <Image /> component and it’s a server component like any other Astro component. It is clear that we will need onLoad, onClick events on a image and events aren’t possible on a server component. Yes, but maybe we can use client component wrapper and pass Astro <Image /> component as a slot so we can have best from both - Astro component for image optimization and a <div /> for events, could this work?
Not really, for any preload effects onLoad event needs to be on the <img /> tag, but more important is that we can’t pass any client props to the slot <Image /> component, we can generate only a single instance at build time. For any props values we would need to pregenerate separate image HTML which in this case is highly impractical.
Conclusion: We will use a React client component that supports interactivity and Astro getImage() function to optimize the images.
import.meta.glob()We want to stick to a static website, for performance reasons and convenient deployments. What way should we use to pass the image urls to the client? We could make a static API endpoint that serves JSON array. We could even make an parametrized API endpoint that serves optimized images.
\nRight away, why having an extra HTTP call for JSON on client when we can pregenerate image urls at build time, it’s not what we want.
\nFor a static API endpoint, since it’s static we would need to pre-render all params at build time, so we could do http://localhost/api/gallery/xl/image1.webp but not http://localhost/api/gallery/300x200/image1.webp and http://localhost/api/gallery/301x200/image1.webp, for that we would need to enable Astro server side rending and have Node.js runtime in production.
If we log a src attribute of an imported image in dev and prod mode we will see something like this:
// in dev\nhttp://localhost:3000/_image?href=/@fs/home/username/Desktop/nemanjam.github.io/src/assets/images/all-images/morning1.jpg?origWidth=4608&origHeight=2592&origFormat=jpg&w=1280&h=720&f=webp\n\n// in prod\nhttp://localhost:3000/_astro/morning1.CEdGhKb3_nVk9T.webpSo Astro is already serving images for us, with a dedicated API endpoint we would just accomplish human friendly url rewriting, that could be useful only if some external service fetches those images, which we don’t have here.
\nConclusion: We will use import.meta.glob('/src/assets/images/all-images/*.jpg') from Vite to import images as modules to obtain images at build time and pass them as props into the Gallery component.
The code is as follows src/libs/gallery/images.ts#L16:
\n\nexport const getGalleryImagesMetadata = (): ImageMetadata[] => {\n const imageModules = import.meta.glob<{ default: ImageMetadata }>(\n // can't be a variable\n '/src/assets/images/all-images/*.jpg',\n { eager: true }\n );\n\n // convert map to array\n const imagesMetadata = Object.keys(imageModules)\n // filter excluded filenames\n .filter((path) => !EXCLUDE_IMAGES.some((excludedFileName) => path.endsWith(excludedFileName)))\n // return metadata array\n .map((path) => imageModules[path].default);\n\n return imagesMetadata;\n};We will structure code like this: MDX (gallery.mdx) -> Astro component (Gallery.astro) -> React component (Gallery.jsx). The call stack is top-down, MDX is a declarative presentation layer, Astro component will resolve data - images, React component will handle events and define logic, it’s the most complex layer.
Code (paraphrased):
\n// src/pages/gallery.mdx\n\n<Gallery class=\"not-prose grow\" />\n\n// src/components/Gallery.astro\n\n<ReactGallery client:only=\"react\" images={randomizedGalleryImages} />\n\n// src/components/react/Gallery.tsx\n\n<div className=\"grid grid-cols-1 gap-1 sm:grid-cols-2 lg:grid-cols-3\">\n {loadedImages.map((image) => (\n <img {...imageProps} />\n )}\n</div>map() on the clientAgain, interesting and important point that is easy to forget is that images.map() needs to be in React component in order to have infinite scroll pagination. For that all image urls (and other props) need to be bundled and available on client, that is passed as props from Astro to the React component.
If we placed images.map() in the Astro component we would we would have a single image list as is without any interactivity (pagination on scroll).
Reminder: Static “backend” runs only once - at build time. We have a Node.js runtime only in development, and not in production - in there we have just a webserver static folder for serving assets. Kind of obvious, but it can sometimes be overlooked when we decide whether to put certain code in a server or client component.
\ngetImage() and <img srcset sizes />Astro provides getImage() function that we will use to optimize images and generate <img /> tag attributes for the client. It accepts the same arguments as the <Image /> component. Note, <img /> tag supports srcset and sizes attributes for responsive images which is sufficient for our use case. This time we don’t need <picture /> support for different images (art direction) and different formats.
We will prepare different image presets (sizes) in src/libs/gallery/transform.ts#L7:
\nNote that only thumbnail uses responsive image, and lightbox uses a fixed size image since Photoswipe lightbox doesn’t support responsive image (at least without a custom component).
\n\n// common props\nconst defaultAstroImageOptions = {\n format: 'webp',\n};\n\n// thumbnail preset\nexport const thumbnailImageOptions = {\n ...IMAGE_SIZES.RESPONSIVE.GALLERY_THUMBNAIL,\n};\n\n// lightbox preset\nexport const lightboxImageOptions = {\n ...IMAGE_SIZES.FIXED.MDX_2XL_16_9,\n};\n\n// getImage() wrapper\nexport const getCustomImage = async (options: UnresolvedImageTransform): Promise<GetImageResult> =>\n getImage({\n ...defaultAstroImageOptions,\n ...options,\n });After that we use getCustomImage() to optimize gallery images that we previously loaded with import.meta.glob() in src/libs/gallery/images.ts#L50:
\nexport const getGalleryImages = async (): Promise<GalleryImage[]> => {\n const thumbnails = await getCustomImages(thumbnailImageOptions);\n const lightBoxes = await getCustomImages(lightboxImageOptions);\n\n const galleryImages = mergeArrays(thumbnails, lightBoxes).map(([thumbnail, lightbox]) => ({\n thumbnail: imageResultToImageAttributes(thumbnail),\n lightbox: imageResultToImageAttributes(lightbox),\n }));\n\n return galleryImages;\n};\n\n// select only needed attributes for the <img /> tag\nexport const imageResultToImageAttributes = (imageResult: GetImageResult): ImgTagAttributes => ({\n src: imageResult.src,\n srcSet: imageResult.srcSet?.attribute,\n ...imageResult.attributes,\n});Now we have the ready <img /> attributes (props) available to pass into the React gallery client component.
Interesting part is configuring <img /> sizes (sizes and widths args in getImage()) attribute for responsive images in src/constants/image.ts#L86:
\nGALLERY_THUMBNAIL: {\n widths: [TW_SCREENS.XS, TW_SCREENS.SM],\n sizes: `(max-width: ${TW_SCREENS.SM}px) ${TW_SCREENS.SM}px, ${TW_SCREENS.XS}px`,\n},\n\n// actual <img /> tag attributes that are generated with the GALLERY_THUMBNAIL\n<img \n sizes=\"(max-width: 640px) 640px, 475px\" \n srcset=\"\n /_astro/river16.CcFOUvED_Z2d5kbP.webp 475w, \n /_astro/river16.CcFOUvED_Z16pb6L.webp 640w\n \" \n src=\"/_astro/river16.CcFOUvED_Z1Dswo2.webp\"\n width=\"4000\" \n height=\"2252\" \n/>\n\n// src/components/react/Gallery.tsx\n\n<div\n id={GALLERY_ID}\n className=\"pswp-gallery grid grid-cols-1 gap-1 sm:grid-cols-2 lg:grid-cols-3\"\n>\n...\n</div>If you are not familiar with defining responsive images, it’s not that complicated as it seems. The code above basically says, bellow SM screen breakpoint (640px) use SM size (width) (640px) image, and if screen is wider than SM use smaller XS (475px) image. Maybe unexpected to use smaller image for larger screen, but it makes sense when you look at responsive grid that is used for the gallery layout.
You can see in grid classes that bellow sm: breakpoint image uses full width of the layout and above sm: there are 2 images per row, above lg: 3 images per row, so it makes sense to use the larger image on smaller screens.
While configuring responsive images it’s advisable to preview what is generated in the browser and ensure that result meets the expectation, we have sharp images at all resolutions and not too large image files.
\n{/* responsive-images-1.5x.webm - https://github.com/user-attachments/assets/48d00b22-feac-40d4-a98b-2e840824fc7f */}\n<video {…IMAGE_SIZES.FIXED.MDX_LG} controls>\n
Large lightbox image will handle Photoswipe on its own, we won’t interfere with it for now. But we can have some nice effect on thumbnail images on infinite scroll. They are already small enough to load fast so no need to use smaller resolution image for blur preloader, we can achieve the same effect with a simple CSS transition.
\nThe following code does that src/components/react/Gallery.tsx#L132
\n\nconst [loadedImages, setLoadedImages] = useState<GalleryImage[]>([]);\n\nconst isLoadingPageImages = useMemo(\n () => !Object.values(loadedStates).every(Boolean),\n [loadedStates, loadedImages.length]\n);\n\nuseEffect(() => {\n const callback: IntersectionObserverCallback = (entries) => {\n // must wait here for images to load\n if (!isEnd && !isLoadingPageImages && entries[0].isIntersecting) {\n setPage((prevPage) => prevPage + 1);\n }\n };\n \n // ...\n \n // page dependency is important for initial load to work for all resolutions\n}, [observerTarget, page, isEnd, isLoadingPageImages]);\n\n\nconst handleLoad = (src: string) => {\n setLoadedStates((prev) => ({ ...prev, [src]: true }));\n};\n\n{loadedImages.map((image) => (\n// ...\n <img\n {...image.thumbnail}\n onLoad={() => handleLoad(image.thumbnail.src)}\n alt={loadedStates[image.thumbnail.src] ? 'Gallery image' : ''}\n className={cn(\n 'w-full transition-all duration-[2s] ease-in-out',\n loadedStates[image.thumbnail.src]\n ? 'opacity-100 blur-0 grayscale-0'\n : 'opacity-75 blur-sm grayscale'\n )}\n />\n))}Note that we have a map() call here and we are storing loading states for an array of images. This is because we want to have a smooth transition for the entire new page of images, not for each image separately because they will load randomly and that’s less esthetic. Important part is isLoadingPageImages variable, it is used to block loading a new page until all images from the previous page are loaded. This happens in the observer callback condition if (!isEnd && !isLoadingPageImages && entries[0].isIntersecting).
Another part is CSS transition, duration-[...] should be picked so it takes more than actual thumbnail image loading time. For the transition effect, you can play around with opacity and Tailwind’s filter classes and see what looks nicest to you.
We want to implement pagination through infinite scroll like e.g. Instagram. Obviously, for this, Gallery needs to be a client component and we will use IntersectionObserver to detect the bottom of the gallery and trigger loading a new page of images. For the observer we could use ready-made hooks from utility libraries like uidotdev/usehooks or streamich/react-use but lets go with our own custom implementation this time.
\nThe code for this is in src/components/react/Gallery.tsx#L76:
\n\n// sets only page\nuseEffect(() => {\n const callback: IntersectionObserverCallback = (entries) => {\n // must wait here for images to load\n if (!isEnd && !isLoadingPageImages && entries[0].isIntersecting) {\n setPage((prevPage) => prevPage + 1);\n }\n };\n const debouncedCallback = debounce(callback, OBSERVER_DEBOUNCE);\n const options: IntersectionObserverInit = { threshold: 1 };\n\n const observer = new IntersectionObserver(debouncedCallback, options);\n\n const observerRef = observerTarget.current;\n if (observerRef) observer.observe(observerRef);\n\n return () => {\n if (observerRef) observer.unobserve(observerRef);\n };\n // page dependency is important for initial load to work for all resolutions\n}, [observerTarget, page, isEnd, isLoadingPageImages]);There are 3 important parts in this code:
\npage state variable in the useEffect dependencies array because we want to trigger effect execution every time new page of images loads and height of gallery increases. Also note that we read page state value from the state setter callback argument setPage((prevPage) => prevPage + 1), that’s why we must also list page in useEffect dependencies array.if (!isEnd && !isLoadingPageImages && entries[0].isIntersecting), it practically means “load new page of images whenever 1. we haven’t loaded all images AND 2. previous page of images is fully loaded - for esthetics AND 3. the gallery is scrolled to the bottom - main prerequisite.callback() triggers quite often, so we need to limit the frequency by debouncing. Note OBSERVER_DEBOUNCE constant value needs to be fine tuned and validated through practical trial and error.Another important and interesting part is detecting bottom of the page and displaying loader UI:
\n\n{/* control threshold with margin-top */}\n{/* must be on top so loader doesn't affect it */}\n<div ref={observerTarget} className=\"mt-0\" />\n\n<div\n className={cn(\n // duration-500 is related to OBSERVER_DEBOUNCE: 300\n 'flex items-center justify-center transition-all duration-500 ease-in-out',\n shouldShowLoader ? 'min-h-48' : 'min-h-0'\n )}\n>\n {shouldShowLoader && <PiSpinnerGapBold className=\"size-10 sm:size-12 animate-spin mt-4\" />}\n</div>This can be tricky because they are circularly dependent - detection triggers showing loader and displaying loader affects position of detection <div ref={observerTarget}/>. Another thing ot note is that detection div has zero height and is placed either above or bellow the loader. It is important to be the above loader because we are interested in the bottom of the images, not the loader that will disappear from the UI in a few milliseconds anyway.
Another important part is controlling and fine-tuning the threshold of the observed element <div ref={observerTarget}/>. We do this by adjusting the positioning with className=\"mt-0\", controlling the observers callback execution frequency with OBSERVER_DEBOUNCE, setting the transition timing for the loader element duration-500, specifying how many images we load (number of rows in the gallery) using the pageSize constant, and how many pages of images we load initially on the first screen initialPage constant.
All of these parameters are connected together and you need to fine tune them for smooth infinite scroll experience. Also note that pageSize and initialPage constants are responsive and need to be defined for each breakpoint independently for full and ergonomic control.
You can see that in the constants file in src/constants/gallery.ts#L7
\n\nexport const GALLERY = {\n GALLERY_ID: 'my-gallery',\n // Todo: make it responsive\n /** step. */\n PAGE_SIZE: {\n XS: 1,\n SM: 2,\n LG: 3,\n },\n /** page dependency in useEffect is more important. To load first screen quickly, set to 3 pages */\n INITIAL_PAGE: {\n XS: 3,\n SM: 3,\n LG: 3,\n },\n /** fine tuned for scroll */\n OBSERVER_DEBOUNCE: 300,\n} as const;And the mapping to translate constants into usable pageSize and initialPage values are defined in utility functions in src/utils/gallery.ts#L8:
\nconst { PAGE_SIZE, INITIAL_PAGE } = GALLERY;\n\n// related to gallery grid css\nconst breakpointToPageKey = {\n XXS: 'XS',\n XS: 'XS',\n SM: 'SM',\n MD: 'SM',\n LG: 'LG',\n XL: 'LG',\n _2XL: 'LG',\n} as const;\n\nconst defaultPageKey = 'LG' as const;\n\nexport const getPageSize = (breakpoint: Breakpoint): number => {\n const key = breakpointToPageKey[breakpoint] ?? defaultPageKey;\n const pageSize = PAGE_SIZE[key];\n\n return pageSize;\n};\n\nexport const getInitialPage = (breakpoint: Breakpoint): number => {\n const key = breakpointToPageKey[breakpoint] ?? defaultPageKey;\n const initialPage = INITIAL_PAGE[key];\n\n return initialPage;\n};With this, we have a smooth scrolling experience on all screen sizes:
\n{/* infinite-scroll-loader-1.5x.webm - https://github.com/user-attachments/assets/c4deb895-5b48-43d6-ae4e-aa34518fc317 */}\n<video {…IMAGE_SIZES.FIXED.MDX_LG} controls>\n
Also pay attention how we “fetch” a new page of images to update:
\n\nconst fetchImagesUpToPage = (\n images: GalleryImage[],\n pageSize: number,\n nextPage: number\n): GalleryImage[] => {\n const endIndex = nextPage * pageSize;\n const isLastPage = endIndex >= images.length;\n\n // for fetchPageImages pagination startIndex must use loadedImages and not all images and page\n const selectedImages = images.slice(0, endIndex);\n\n // load all images for last page\n return !isLastPage ? sliceToModN(selectedImages, pageSize) : selectedImages;\n};\n\n// converts page to loaded images\nuseEffect(() => {\n const upToPageImages = fetchImagesUpToPage(images, pageSize, page);\n setLoadedImages(upToPageImages);\n}, [page, images, pageSize]);There are 2 important moments here:
\nimages.slice(0, endIndex);. Usually pagination implies a network and database calls that require both startIndex and endIndex, and if we went that path we would need to calculate startIndex by finding the last element of the loadedImages state array in the images array and pass those as arguments.pageSize constant is responsive it can change when e.g. user resizes the browser window, so we call sliceToModN(selectedImages, pageSize) for evenly loaded new row. Note that we don’t call this for the last page because, eventually, we want to load all images, and the correct loadedImages array length is important for calculating the isEnd variable.Layout shift is important web vitals parameter and it’s more challenging to optimize here since we are dealing with a dynamic client components. In the Gallery component we handle this by setting initialPage constant to load enough images to fill the initial gallery screen.
\nexport const GALLERY = {\n // ...\n PAGE_SIZE: {\n XS: 1,\n SM: 2,\n LG: 3,\n },\n INITIAL_PAGE: {\n XS: 3,\n SM: 3,\n LG: 3,\n },\n // ...\n} as const;Another optimization we can do is to stretch the empty gallery container element with flex grow. For that we need to modify the Page layout and pass the required Tailwind classes via the MDX frontmatter and articleClass prop.
You can see that in src/layouts/Page.astro#L38:
\n---\n\nimport Centered from '@/layouts/Centered.astro';\nimport { getOpenGraphImagePath } from '@/libs/api/open-graph/image-path';\nimport { cn } from '@/utils/styles';\n\nexport interface Content {\n // ...\n class?: string;\n /** for flex flex-grow min-height to prevent layout shift for client components */\n articleClass?: string;\n}\n\n// ...\n\nconst { title, description, class: className, articleClass } = content;\n\n// ...\n\n---\n\n<Centered {metadata} class={cn(className)}>\n {/* in general must not have flex, it will disable margin collapsing in MDX */}\n <article class={cn('my-prose', articleClass)}>\n <slot />\n </article>\n</Centered>Flex class is passed from MDX frontmatter in src/pages/gallery.mdx#L7:
\n\n---\nlayout: '../layouts/Page.astro'\n...\nclass: 'max-w-5xl'\narticleClass: 'grow flex flex-col'\n---\n\nimport Gallery from '../components/Gallery.astro';\n\n# Gallery\n\n<Gallery class=\"not-prose grow\" />This will reduce the shift of DOM elements size, it won’t make it perfect like in fully static page but for our use case it’s good enough.
\nAnother point to make is that flex container will disable margin collapsing which is important for proper vertical spacings in MDX generated HTML. So if you do that you will need to add an additional <div> wrapper element without flex to re-enable proper margin collapsing.
Lighthouse score, old gallery:
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={LayoutShiftBeforeImage} alt=“Old gallery Lighthouse score” />
\nLighthouse score, new gallery:
\n<Image {…IMAGE_SIZES.FIXED.MDX_MD} src={LayoutShiftAfterImage} alt=“New gallery Lighthouse score” />
\nPlease ignore the “Accessibility” score above, since the accessibility attributes aren’t yet tackled on the entire website.
\nFor previewing images in full screen lightbox we will use ready made library Photoswipe that looks solid, reliable and flexible. We will use a basic React example from the documentation.
\nThis is the code src/components/react/Gallery.tsx#L98
\n\n// lightbox\nuseEffect(() => {\n let lightbox: PhotoSwipeLightbox | null = new PhotoSwipeLightbox({\n gallery: '#' + GALLERY_ID,\n children: 'a',\n pswpModule: () => import('photoswipe'),\n });\n lightbox.init();\n\n return () => {\n lightbox?.destroy();\n lightbox = null;\n };\n}, []);\n\nreturn (\n <>\n <div\n id={GALLERY_ID}\n className=\"pswp-gallery grid grid-cols-1 gap-1 sm:grid-cols-2 lg:grid-cols-3\"\n >\n {loadedImages.map((image) => (\n <a\n key={`${GALLERY_ID}--${image.lightbox.src}`}\n // lightbox doesn't support responsive image\n href={image.lightbox.src}\n data-pswp-width={image.lightbox.width}\n data-pswp-height={image.lightbox.height}\n target=\"_blank\"\n rel=\"noreferrer\"\n >\n <img\n {...image.thumbnail}\n // ...\n />\n </a>\n ))}\n </div>\n {/* ... */}\n <>Note that for a simplicity sake we are using a simple fixed image and Photoswipe implements scale transition on its own. By default it uses a simple link <a href={image.lightbox.src}> to load the <img src /> in the full page lightbox.
This is a tradeoff for simplicity. Loading a responsive image with srcset would require integrating a custom component which could be a topic for another article. Another possible improvement is to enable closing lightbox on backdrop click on mobile which is not the case with the default config.
Lightbox image size is defined in src/libs/gallery/transform.ts#L24
\n\nexport const lightboxImageOptions = {\n ...IMAGE_SIZES.FIXED.MDX_2XL_16_9,\n};\n\n// src/constants/image.ts\n\nexport const IMAGE_SIZES = {\n FIXED: {\n // ...\n MDX_2XL_16_9: { width: TW_SCREENS._2XL, height: TW_SCREENS.HEIGHTS._2XL },\n },\n // ...\n};The relevant files:
\n# new gallery https://github.com/nemanjam/nemanjam.github.io/tree/c1e105847d8e7b4ab4aaffad3078726c37f67528\ngit checkout c1e105847d8e7b4ab4aaffad3078726c37f67528\n\nsrc/pages/gallery.mdx\nsrc/components/Gallery.astro\nsrc/components/react/Gallery.tsx\nsrc/libs/gallery/images.ts\nsrc/libs/gallery/transform.ts\nsrc/utils/gallery.ts\nsrc/constants/gallery.ts\nsrc/constants/image.ts\nsrc/components/react/hooks/useScrollDown.tsx\nsrc/components/react/hooks/useWidth.tsx\n\n# old gallery https://github.com/nemanjam/nemanjam.github.io/tree/e0165b295db2ccc72bbbb7be4bdd7eb48f7dedae\ngit checkout e0165b295db2ccc72bbbb7be4bdd7eb48f7dedaeThat was a pretty long read, thank you for your attention and dedication. Have you implemented an Astro image gallery yourself and used a different approach? Do you have suggestions for improvements or spotted anything incorrect? Don’t hesitate to leave a comment below.
\ngetImage() function https://docs.astro.build/en/recipes/build-custom-img-component/