Relaying files from S3 to SFTP without local staging

A common integration pattern is moving files from a cloud object storage bucket (S3 or any S3 compatible service) to an SFTP server that a partner or a downstream system reads from. The naive approach downloads each file to the local disk first and then uploads it, which wastes disk space and roughly doubles the transfer time. The CopyToVFS method of Syncplify AFT! eliminates the intermediate step by streaming the data directly between the two virtual file systems in a single call. This recipe builds that relay as a scheduled cron job.

How it works

You open two VFS handles: one for the S3 source and one for the SFTP destination. Both are resolved by name from the VFS library, so no credential appears in the script. Then you list the source directory, loop over the files, and call src.CopyToVFS(srcPath, dest, dstPath) for each one. AFT! handles the streaming; your script is just the orchestration glue.

// S3 to SFTP relay
// Reads all files from a named S3 VFS and copies each one directly to a named
// SFTP VFS. No intermediate local file is created: CopyToVFS streams the data
// between the two virtual file systems. Designed to run as a scheduled cron job.

var s3VfsName   = "your-s3-vfs-name";    // source: named S3 VFS in the VFS library
var sftpVfsName = "your-sftp-vfs-name";  // destination: named SFTP VFS in the VFS library
var sourceDir   = "/outbox";             // source directory on S3
var destDir     = "/inbound/from-s3";    // destination directory on SFTP

var src  = new VirtualFSByName(s3VfsName);
var dest = new VirtualFSByName(sftpVfsName);

// List the source directory sorted oldest first, so that files arrive at the
// destination in the same order they were written.
var listing = src.ReadDir(sourceDir, SortByTime, SortAsc);
if (!listing.Ok()) {
    Log.Error("failed to list source directory: " + listing.ErrorMsg());
    Exit(1);
}

var entries = listing.Infos();
var relayed = 0;
var failed  = 0;

Log.Info("relaying " + entries.length + " entries from S3 to SFTP");

for (var i = 0; i < entries.length; i++) {
    var f = entries[i];

    // ReadDir returns both files and directories; skip directories.
    if (f.Type !== "FILE") {
        continue;
    }

    var srcPath = sourceDir + "/" + f.Name;
    var dstPath = destDir   + "/" + f.Name;

    var r = src.CopyToVFS(srcPath, dest, dstPath);
    if (!r.Ok()) {
        Log.Error("relay failed for " + f.Name + ": " + r.ErrorMsg());
        failed++;
    } else {
        Log.Info("relayed: " + f.Name + " (" + f.Size + " bytes)");
        relayed++;
    }
}

Log.Info("relay complete: " + relayed + " succeeded, " + failed + " failed");

if (failed > 0) {
    Exit(1);
}

src.ReadDir(sourceDir, SortByTime, SortAsc): listing oldest first means the SFTP server receives the files in the order they were written to S3. For many downstream systems that order matters, for example when files are numbered sequences or must be processed chronologically.

f.Type !== "FILE": ReadDir returns both files and directory entries. Skipping everything that is not a file keeps the relay from trying to copy directory prefixes as if they were objects.

Exit(1) at the end: a non zero exit code tells the AFT! job engine that the run was not entirely clean. It shows in the job history and can trigger alerts in external monitoring tools. A run that relayed everything successfully exits with the default code 0.

Turning the relay into a move

If the files should be moved rather than copied, use MoveToVFS instead of CopyToVFS:

var r = src.MoveToVFS(srcPath, dest, dstPath);
if (r.Ok()) {
    relayed++;
} else {
    failed++;
}

Using this with other back ends

The same script works for any combination of VFS types. Swap the named profiles in the VFS library to relay from Azure to Google Cloud Storage, from SFTP to S3, from the local disk to Azure, and so on. The script logic does not change at all.