Downloading new remote files as they arrive

This recipe watches a remote directory and downloads each new file the moment it appears, using the RemoteWatcher object. Unlike a cron scheduled script that lists the directory once and exits, this one runs as a persistent job and reacts in near real time: as soon as a poll cycle detects a new file, the callback fires and the download begins. It is the inbound mirror of Uploading local folder changes to SFTP in real time.

How it works

RemoteWatcher takes the name of a profile from the VFS library and polls the remote directory for you. You choose which events you care about, tell it which directory to watch, and hand it a callback. AFT! runs the polling loop, handles the connection, and tracks subdirectories created while the job runs, so a file that lands in a folder that did not exist at startup is not missed.

The callback receives an event object with two fields:

• evt.Event: the type of change, one of EVT_CREATE, EVT_REMOVE and EVT_MODIFY
• evt.Object: the full remote path of the file that changed

For an inbound pickup only EVT_CREATE matters. Files modified in place or deleted are irrelevant.

The script also opens a separate VirtualFSByName handle on the same profile. The watcher polls through its own connection; this second handle is for the ExportFile call that copies each remote file to the local inbox.

// Remote inbound file pickup
// Watches a named VFS for newly arrived files and downloads each one to a local
// inbound folder as it appears. Runs persistently until halted.

var remoteVfsName       = "your-sftp-vfs-name";   // named VFS profile from the VFS library
var remoteInboxDir      = "/remote/inbox";         // remote directory to poll for new arrivals
var localInboxDir       = "/local/inbound";        // local destination for downloaded files
var pollIntervalSeconds = 30;                      // how often to poll the remote directory

// Which remote events to act on.
// For most inbound pickup scenarios only newly created files matter.
var watcher = new RemoteWatcher(remoteVfsName);
watcher.NotifyCreate = true;
watcher.NotifyRemove = false;
watcher.NotifyModify = false;

// Optionally restrict pickups to specific file patterns.
// watcher.InclusionFilter = ["*.csv", "*.xml"];

// A VFS handle for downloading each newly detected file. The RemoteWatcher
// polls the directory on its own; this separate handle is for ExportFile.
var vfs = new VirtualFSByName(remoteVfsName);

try {
    watcher.WatchDir(remoteInboxDir, true);   // true = recurse into subdirectories

    watcher.Start(pollIntervalSeconds, function(evt) {
        if (evt.Event !== EVT_CREATE) {
            return;
        }

        // evt.Object is the full remote path of the newly detected file.
        var remotePath = evt.Object;
        var fileName   = ExtractName(remotePath) + ExtractExt(remotePath);
        var localPath  = localInboxDir + "/" + fileName;

        Log.Info("new remote file detected: " + remotePath);

        var r = vfs.ExportFile(remotePath, localPath);
        if (!r.Ok()) {
            Log.Error("download failed for " + remotePath + ": " + r.ErrorMsg());
        } else {
            Log.Info("downloaded to: " + localPath);
        }
    });

    // Block here until an operator sends a halt signal from the AFT! web UI or the API.
    WaitForHaltSignal();
    Log.Info("halt signal received, shutting down");
} finally {
    // Always stop the watcher on exit, whether normal or due to an error.
    watcher.Stop();
}

A few things worth noting.

watcher.WatchDir(remoteInboxDir, true): the second argument enables recursive watching. If the partner drops files into subdirectories, they are picked up too. RemoteWatcher also tracks subdirectories that appear at run time, so folders that did not exist at startup are covered without any extra logic on your part.

watcher.Start(pollIntervalSeconds, callback): the first argument is the polling interval in seconds. Unlike FsWatcher, which receives push notifications from the operating system, a remote watcher can only detect changes by comparing successive directory listings, so it must poll. Tune the interval to balance freshness against the load on the remote server; thirty seconds is a reasonable default for most trading partner scenarios.

The first poll reports everything that is already there. The watcher starts with an empty memory of the directory, so every file present at startup counts as newly created and gets downloaded. Start the job on an empty inbox, or let that first pass drain the backlog, which is usually what you want anyway.

InclusionFilter: if you only want specific file types, uncomment the filter line and list the patterns. Files whose names do not match are ignored before the callback is ever invoked.

The try and finally blocks ensure that watcher.Stop() is always called, even if an exception is thrown, which releases the polling loop.

Running this script

Because the script calls WaitForHaltSignal(), it runs as a persistent job rather than a one shot execution. Start it once with the Run button on the Scripts page of the web UI, or trigger it from the command line or through the REST API, and leave it running. To stop it, use the Halt button in the web UI, or call DELETE /v1/adm/jobs/{id} in the administrative API.