Removing old files from a remote archive

Without a cleanup policy, a remote archive grows indefinitely. Storage costs accumulate quietly, and eventually someone has to do a manual purge under time pressure. This recipe is a scheduled Syncplify AFT! script that connects to an S3 bucket, inspects the modification time of every file in an archive directory, and deletes anything older than a configurable retention threshold. The same pattern works for any remote service AFT! supports: swap S3Client for SftpClient, AzureClient or GCSClient.

How it works

The script connects to S3 with credentials from the secrets store, lists the archive directory with LIST_FILES (which excludes directory prefixes), and iterates over the results. For each file it computes the age in milliseconds by comparing f.TimeStamp with the current time. Files within the retention window are skipped; files past it are deleted.

// Remote archive cleanup
// Connects to an S3 bucket and deletes the files in the archive directory that are
// older than the configured retention period. Intended to run as a scheduled cron
// job, for example once a day after the nightly backup completes.

var retentionDays = 90;         // delete files older than this many days
var archiveDir    = "/archive"; // remote directory to clean up

var cli = new S3Client();
cli.Region    = GetSecret("s3-region");
cli.AccessKey = GetSecret("s3-access-key");
cli.SecretKey = GetSecret("s3-secret-key");
cli.Bucket    = GetSecret("s3-bucket-name");

if (!cli.Connect()) {
    Log.Error("failed to connect to S3");
    Exit(1);
}

try {
    // LIST_FILES requests file entries only; directory prefixes are excluded.
    var files = cli.ListDir(archiveDir, LIST_FILES);
    Log.Info("scanning " + files.length + " files in " + archiveDir);

    var nowMs       = new Date().getTime();
    var retentionMs = retentionDays * 24 * 60 * 60 * 1000;
    var deleted     = 0;
    var retained    = 0;

    for (var i = 0; i < files.length; i++) {
        var f     = files[i];
        var ageMs = nowMs - new Date(f.TimeStamp).getTime();

        if (ageMs < retentionMs) {
            retained++;
            continue;
        }

        // ListDir returns bare file names, so build the full remote path.
        var fullPath = archiveDir + "/" + f.Name;
        var ageDays  = Math.floor(ageMs / (24 * 60 * 60 * 1000));

        if (cli.Delete(fullPath)) {
            Log.Info("deleted: " + f.Name + " (age: " + ageDays + " days)");
            deleted++;
        } else {
            Log.Error("failed to delete: " + fullPath);
        }
    }

    Log.Info("cleanup complete: " + deleted + " deleted, " + retained + " retained");
} finally {
    cli.Close();
}

new Date(f.TimeStamp).getTime(): f.TimeStamp is the modification time of the S3 object as reported by the API. Passing it to the Date constructor gives a JavaScript Date you can compare arithmetically. The difference in milliseconds, divided by the appropriate factor, is the age in seconds, minutes, hours or days.

LIST_FILES: passing this constant to ListDir tells AFT! to return file objects only. S3 "directories" are just key prefixes, not real objects, and they have no meaningful timestamp. Excluding them avoids a spurious deletion attempt.

The try and finally blocks ensure that cli.Close() is always called, even if the body throws an exception.

Testing before you delete

Add a dry run flag when you first deploy this to a production environment:

var dryRun = true;   // set to false to enable the actual deletion

if (dryRun) {
    Log.Info("dry run: would delete " + fullPath + " (age: " + ageDays + " days)");
} else {
    if (cli.Delete(fullPath)) { deleted++; }
}

Run it once with dryRun = true, review the job log, and flip the flag only when you are satisfied that the retention threshold is right.

Cleaning subdirectories too

ListDir looks at one directory. ListDirR lists a whole tree, but it reports each file by its bare name, without the subdirectory it sits in, so its output cannot feed Delete. To clean an archive that has subdirectories, put the loop above in a function and call it for every directory in the tree, which ListDir with LIST_DIRS enumerates one level at a time:

function cleanDir(dir) {
    // the listing and deletion loop from the script above, with archiveDir replaced by dir
}

function cleanTree(dir) {
    cleanDir(dir);
    var subdirs = cli.ListDir(dir, LIST_DIRS);
    for (var i = 0; i < subdirs.length; i++) {
        cleanTree(dir + "/" + subdirs[i].Name);
    }
}

cleanTree(archiveDir);

Using MinIO or another S3 compatible service

Add Endpoint and UsePathStyle before calling Connect:

cli.Endpoint     = "https://minio.internal.example.com";
cli.UsePathStyle = true;

Everything else stays the same.