Skip to main content

Incremental Sync for Changed Files

Rsync's greatest strength is its ability to detect and transfer only what changed. This page covers the techniques and flags that make incremental sync work in practice.

Why This Matters

A full copy of a 10 GB application directory takes minutes. An incremental rsync of the same directory (where only 50 MB changed) takes seconds. Understanding incremental sync is essential for efficient server management.

How Rsync Detects Changes​

By default, rsync decides whether a file needs transfer by comparing:

  1. File size — Has the byte count changed?
  2. Modification time (mtime) — Has the timestamp changed?

If either differs between source and destination, rsync transfers the file. Unchanged files are skipped entirely.

Detection MethodFlagSpeedAccuracyBest For
Size + timestamp (default)None neededFastGoodDaily backups, code deployment
Checksum comparison--checksumSlowerExactCritical data, database files
Skip newer files at destination--updateFastSafeMerge without overwriting newer edits

Sync Only Changed Files (Non-Destructive)​

The safest incremental pattern — copies new and modified files without deleting anything at the destination:

rsync -av SOURCE_DIR/ DEST_DIR/
  • -a — Archive mode (recursive, preserves permissions, timestamps, ownership)
  • -v — Verbose output (shows which files are transferred)

Result: Changed and new files are copied. Files that exist only at the destination remain untouched because --delete is not used.

Example: Backup a Web Application​

# Sync only changes to local backup
rsync -av /var/www/html/ /backup/www/

# Sync only changes to remote backup server
rsync -avz /var/www/html/ user@backup:/backups/www/

Example: Sync Database Dumps​

# Only transfer new or updated dump files
rsync -av /var/backups/mysql/ /mnt/external/mysql-backups/

Skip Files Newer at Destination (--update)​

When both source and destination may have been modified independently, --update prevents overwriting files that are newer at the destination:

rsync -av --update SOURCE_DIR/ DEST_DIR/

This is useful when:

  • Multiple people edit files on different servers
  • You want to merge changes without losing recent destination edits
  • You're syncing between staging and development environments

Example: Merge Configuration Changes​

# Sync configs without overwriting newer versions on the remote
rsync -av --update /etc/nginx/ user@server2:/etc/nginx/

Checksum-Based Change Detection (--checksum)​

When timestamps can't be trusted (clock drift, restored from backup, copied with cp), use checksum mode to compare file contents directly:

rsync -av --checksum SOURCE_DIR/ DEST_DIR/
Performance Impact

Checksum mode reads every file on both sides to compute hashes. This is significantly slower for large directories. Use it only when data integrity is critical — such as database dumps, financial records, or compliance data.

Example: Verify Python Application Integrity​

# Ensure deployed code exactly matches the repository build
rsync -av --checksum /build/output/ user@production:/var/www/app/

Keep Previous Versions on Overwrite (--backup)​

Combine incremental sync with automatic versioning to retain old copies of overwritten files:

rsync -av --update \
 --backup \
 --backup-dir="/backup/versions/$(date +%F_%H%M%S)" \
 SOURCE_DIR/ DEST_DIR/
  • --backup — Instead of overwriting, move the old destination file to a backup location
  • --backup-dir — Where to store the old versions (timestamped directory)

Result: The latest files are synced while previous destination versions are preserved for rollback.

Example: Version-Controlled Configuration Backups​

# Sync Nginx config with automatic versioning of overwritten files
rsync -av --backup --backup-dir="/backup/nginx/$(date +%F)" \
 /etc/nginx/ /backup/nginx/current/

Example: Application Deployment with Rollback Safety​

# Deploy to production with automatic backup of replaced files
rsync -avz --backup --backup-dir="/var/rollback/$(date +%F_%H%M)" \
 /staging/app/ user@production:/var/www/app/
tip

This pattern gives you automatic rollback points. If a deployment breaks something, you can restore from the versioned backup directory.

Exclude Unnecessary Files​

Speed up incremental syncs by skipping files that don't need to be backed up:

# Skip common build artifacts, caches, and version control
rsync -av --update \
 --exclude '.git' \
 --exclude 'node_modules' \
 --exclude '__pycache__' \
 --exclude '*.log' \
 --exclude 'cache/' \
 SOURCE_DIR/ DEST_DIR/

Common Excludes by Application Type​

ApplicationExclude Patterns
PHP Applicationsvendor/, cache/, *.log, .env, storage/framework/cache/
Python (Django, Flask)venv/, __pycache__/, *.pyc, .env
Node.jsnode_modules/, dist/, .env
General.git/, *.tmp, *.swp, logs/

Combining Incremental Flags​

Here's a production-ready incremental sync command that combines multiple techniques:

rsync -avz \
 --update \
 --backup --backup-dir="/backup/versions/$(date +%F)" \
 --exclude '.git' \
 --exclude 'node_modules' \
 --exclude '*.log' \
 --compress \
 --progress \
 /var/www/app/ user@backup-server:/backups/app/

This command:

  1. Transfers only changed files (-a with delta detection)
  2. Skips files newer at destination (--update)
  3. Preserves old versions of overwritten files (--backup)
  4. Excludes unnecessary files (.git, node_modules, logs)
  5. Compresses data for network transfer (--compress)
  6. Shows progress for each file (--progress)

Common Pitfalls​

PitfallImpactPrevention
Missing trailing slash on sourceCreates nested directories instead of syncing contentsAlways use SOURCE/ with trailing slash
Using --delete unintentionallyRemoves files at destination that don't exist at sourceOnly add --delete when you explicitly want mirror behavior
Trusting timestamps on restored filesRsync skips files that appear unchanged (same size + mtime)Use --checksum after restoring from backup
No --dry-run before productionUnexpected changes applied to live serverAlways preview with --dry-run or -n first

What's Next​