Uploading local folder changes to SFTP in real time

This recipe implements a real time folder backup with Syncplify AFT! scripting: a local folder is watched, and every file created or modified in it is uploaded to a remote SFTP server as soon as it appears. Two approaches are presented, one for folders that change rarely and one for folders that change often, and a third variant uses a virtual file system instead of a client object. All three use FsWatcher to watch the folder; the first two upload with SftpClient.

INFO

Everything on this page is for Syncplify AFT! v4.0.0 and later. If you are porting a v3 polling script, the breaking changes page shows what to change.

How it works

AFT! scripts can run as long lived jobs. Rather than executing once and exiting, this one blocks in WaitForHaltSignal(), which returns only when an operator stops the job from the web UI or through the API. That makes it ideal for reactive, always on automation such as real time backup.

The FsWatcher object watches a local directory, and optionally its subdirectories, for file system events. When one occurs it invokes a callback with an event object of two fields that matter:

• evt.Event: the type of event, one of EVT_CREATE, EVT_WRITE, EVT_REMOVE, EVT_RENAME and EVT_CHMOD
• evt.Object: the full path of the affected file or directory

The script filters the events it cares about and acts on them.

Choosing the right approach

The two strategies differ in how they manage the SFTP connection:

	Connect on demand	Persistent connection
SFTP connection	Opened and closed for each event	Opened once at startup
Best for	Infrequent changes	Frequent changes
Risk	Higher latency per upload	Idle timeout disconnections

If files in the watched folder change many times per minute, opening and closing a TCP connection for each event adds needless overhead, and a persistent connection is the better choice. If files change rarely, a few times per hour, a connection kept open indefinitely will hit the server's idle timeout (many servers, deliberately, do not let a client stay connected forever), and connecting on demand avoids that entirely.

Script 1: connect on demand

The SFTP client is created inside the event callback. Each detected change opens a new connection, uploads the file and closes the connection.

// Real time backup: CONNECT ON DEMAND
// For folders where files change infrequently. A fresh SFTP connection is opened
// and closed for each detected change, which avoids idle connection timeouts.

var sourceFolder = "C:\\YourSourceFolder";
var remoteFolder = "/remote/backup/path";

// The remote directory that mirrors the local directory of a file.
function remoteDirFor(localFile) {
    var relative = localFile.substring(sourceFolder.length).replace(/\\/g, "/");
    return remoteFolder + relative.substring(0, relative.lastIndexOf("/"));
}

// Which file system events trigger an upload.
var watcher = new FsWatcher();
watcher.NotifyCreate = true;
watcher.NotifyWrite  = true;
watcher.NotifyRemove = false;
watcher.NotifyRename = false;
watcher.NotifyChmod  = false;

try {
    // Watch the source folder recursively (true = include all subdirectories).
    watcher.WatchDir(sourceFolder, true);
    watcher.Start(function(evt) {
        // Ignore the events we do not care about.
        if (evt.Event !== EVT_CREATE && evt.Event !== EVT_WRITE) {
            return;
        }

        Log("change detected: " + evt.Object + ", connecting to SFTP");

        // Open a fresh connection for this event, upload the file, then close.
        var scli = new SftpClient();
        scli.Host = "your-sftp-server.example.com:22";
        scli.User = "your-username";
        scli.Pass = GetSecret("your-password-secret-name");
        scli.HostKeySHA256 = "your-servers-public-key-sha256-fingerprint";
        // A modified file must replace the copy already on the server.
        scli.Options.UploadPolicy = AlwaysOverwrite;

        if (scli.Connect()) {
            // Upload creates the remote directory if it does not exist yet.
            if (!scli.Upload(evt.Object, remoteDirFor(evt.Object))) {
                Log.Error("upload failed: " + evt.Object);
            }
            scli.Close();
        } else {
            Log.Error("could not connect to the SFTP server to upload: " + evt.Object);
        }
        scli = null;
    });

    // Block here until an operator sends a halt signal from the AFT! web UI or the API.
    WaitForHaltSignal();
    Log("halt signal received, shutting down");
} finally {
    // Always stop the watcher on exit, whether normal or due to an error.
    watcher.Stop();
}

A few things worth noting.

watcher.WatchDir(sourceFolder, true): the second argument enables recursive watching. Any file created or modified inside a subdirectory, at any depth, triggers the callback. FsWatcher also tracks subdirectories created while the job runs, so folders that did not exist when the watcher started are covered without any extra logic.

scli.Upload(evt.Object, remoteDirFor(evt.Object)): Upload puts a file into the remote directory you name and creates that directory if needed. The helper computes the directory from the local path, so the remote tree mirrors the local one: C:\YourSourceFolder\invoices\may.pdf lands in /remote/backup/path/invoices/may.pdf. (UploadWithPathR mirrors a whole tree in one call and serves batch jobs; for one file at a time, Upload with a computed directory is the direct route.)

scli.Options.UploadPolicy = AlwaysOverwrite: the default policy is NeverOverwrite, which leaves an existing remote file alone. A backup must replace it when the local file changes, so the policy is set explicitly.

scli.HostKeySHA256: the SHA-256 fingerprint of the server's host key. Setting it enables strict host key verification, which prevents man in the middle attacks. Get the fingerprint from your SFTP server administrator or by inspecting the server's known host key. Omitting it is not recommended in production.

The try and finally blocks: the watcher is started inside try and always stopped in finally. This guarantees that system resources (file handles and operating system notification handles) are released even if the script exits because of an error.

Script 2: persistent connection

The SFTP client is created and connected before the watcher starts. The same connection is reused for every upload throughout the lifetime of the job.

// Real time backup: PERSISTENT SFTP connection
// For folders where files change frequently. The SFTP connection is opened once
// at startup and kept alive by the steady upload activity.
// Connect first, then start watching. On halt, stop the watcher and close the connection.

var sourceFolder = "C:\\YourSourceFolder";
var remoteFolder = "/remote/backup/path";

// The remote directory that mirrors the local directory of a file.
function remoteDirFor(localFile) {
    var relative = localFile.substring(sourceFolder.length).replace(/\\/g, "/");
    return remoteFolder + relative.substring(0, relative.lastIndexOf("/"));
}

// Set up and connect the SFTP client before starting the watcher.
var scli = new SftpClient();
scli.Host = "your-sftp-server.example.com:22";
scli.User = "your-username";
scli.Pass = GetSecret("your-password-secret-name");
scli.HostKeySHA256 = "your-servers-public-key-sha256-fingerprint";
scli.Options.UploadPolicy = AlwaysOverwrite;

// Which file system events trigger an upload.
var watcher = new FsWatcher();
watcher.NotifyCreate = true;
watcher.NotifyWrite  = true;
watcher.NotifyRemove = false;
watcher.NotifyRename = false;
watcher.NotifyChmod  = false;

// Fail fast: abort if the server is unreachable at startup.
if (!scli.Connect()) {
    Log.Error("could not connect to the SFTP server, aborting");
    Exit(1);
}

try {
    // Watch the source folder recursively (true = include all subdirectories).
    watcher.WatchDir(sourceFolder, true);
    watcher.Start(function(evt) {
        if (evt.Event === EVT_CREATE || evt.Event === EVT_WRITE) {
            Log("uploading: " + evt.Object);
            if (!scli.Upload(evt.Object, remoteDirFor(evt.Object))) {
                Log.Error("upload failed: " + evt.Object);
            }
        }
    });

    // Block here until an operator sends a halt signal from the AFT! web UI or the API.
    WaitForHaltSignal();
    Log("halt signal received, shutting down");
} finally {
    // Always clean up both the watcher and the connection on exit.
    watcher.Stop();
    scli.Close();
}

The structure differs from script 1 in a few important ways.

Connect before watching: the scli.Connect() call happens at the top level, before WatchDir and Start. If the connection fails at startup, the script calls Exit(1) immediately, which marks the job as failed in the job history rather than as completed. There is no point starting the watcher if there is nowhere to send the files.

No per event connection setup: the callback is intentionally minimal. It logs the file name and calls Upload. The absence of connect and close calls inside the callback is not an oversight; it is the entire point of this pattern.

finally closes both: here the finally block must stop the watcher and close the SFTP connection. The watcher is stopped first, so that no upload is attempted after the connection is gone.

When to use each pattern

Use connect on demand when:

• the watched folder receives new or modified files at most a few times per hour
• the remote SFTP server enforces a short idle connection timeout
• you prefer simplicity and do not want to think about connection state

Use the persistent connection when:

• files change frequently, several times per minute or more
• upload latency matters and you want to avoid a TCP and SSH handshake per event
• the remote server's idle timeout is long enough, or keep alives are configured

Both patterns are valid. The choice is an operational one, based on your environment.

Watcher options worth knowing

• watcher.DelayBySeconds: waits that many seconds after an event before invoking the callback. Use it when the producer needs time to finish writing, otherwise a file that is still being written gets uploaded half done. Events are handled one after another, so the delay also spaces out the uploads.
• watcher.InclusionFilter and watcher.ExclusionFilter: arrays of glob patterns matched against the file name (not the full path). An event whose name does not pass them never reaches the callback, for example with watcher.InclusionFilter = ["*.pdf", "*.xlsx"] or watcher.ExclusionFilter = ["*.tmp", "*.part"].
• The Notify switches: only NotifyCreate is on by default. The scripts above enable NotifyWrite too, to catch modified files.

A VFS based alternative

For backup purposes like the ones on this page, a client object as used above is often the best option, because of its flexibility and advanced options: atomic uploads through temporary names, modification times adjusted after upload, versioning, rule based skipping. Client objects are purpose built to carry out these tasks.

If you want a quick way to achieve the same goal and do not need those features, you may prefer the simplicity of a virtual file system. A VFS manages the state of its remote connection automatically and internally, hiding all that complexity, and its credentials live in the VFS library rather than in the script.

Here is a version of the scripts above rewritten to use a VFS instead of a client object. The named VFS must exist in AFT! before the script runs.

// Real time backup using a virtual file system
// The VFS manages its connection internally (reconnecting as needed), so one script
// covers both the "infrequent changes" and the "frequent changes" scenarios with no
// connect or close logic anywhere. The named VFS must exist in the VFS library; its
// credentials, host key and options live in that profile, not in the script.

var sourceFolder = "C:\\YourSourceFolder";
var remoteFolder = "/remote/backup/path";

// Which file system events trigger an upload.
var watcher = new FsWatcher();
watcher.NotifyCreate = true;
watcher.NotifyWrite  = true;
watcher.NotifyRemove = false;
watcher.NotifyRename = false;
watcher.NotifyChmod  = false;

// Resolve the SFTP connection from the VFS library.
var sftp = new VirtualFSByName("your-sftp-vfs-name");

try {
    watcher.WatchDir(sourceFolder, true);
    watcher.Start(function(evt) {
        if (evt.Event !== EVT_CREATE && evt.Event !== EVT_WRITE) {
            return;
        }

        // The remote path: strip the local root and turn backslashes into slashes.
        var relativePath = evt.Object.substring(sourceFolder.length).replace(/\\/g, "/");
        var targetPath   = remoteFolder + relativePath;
        Log("uploading: " + evt.Object + " to " + targetPath);

        // ImportFile creates the target directory tree if it does not exist yet.
        var r = sftp.ImportFile(evt.Object, targetPath);
        if (!r.Ok()) {
            Log.Error("upload failed for " + evt.Object + ": " + r.ErrorMsg());
        }
    });

    // Block here until an operator sends a halt signal from the AFT! web UI or the API.
    WaitForHaltSignal();
    Log("halt signal received, shutting down");
} finally {
    // Always stop the watcher on exit, whether normal or due to an error.
    watcher.Stop();
}

Now you know the options at your disposal. The choice is yours.