Integrating Syncplify AFT! with an external workflow API

Many serious file transfer operations are steps of larger business workflows. A data pipeline might require a "claim" step, where the downstream system registers its intent to process a batch before any file moves, and a "completion" step, where the status is reported back once the transfer has finished, successfully or not. Syncplify AFT! can play the execution engine in such orchestration patterns by making HTTP calls before and after the actual file movement. This recipe builds a script that claims a job from an external API, performs the transfer, and always reports the outcome back to that same API, even if the transfer fails.

How it works

The script receives a job identifier as a run time parameter, so that the same script serves many jobs without modification. It calls a claim endpoint on the orchestration API to retrieve the VFS names and paths to use for this particular run, performs the transfer, and calls a report endpoint with the outcome.

// REST API integration: claim, transfer, report
// Receives a job ID as a run time parameter, claims the job from an external
// orchestration API to discover the source and destination details, performs the
// file transfer, and always reports the outcome back to the API.

var jobId    = Param("job_id");      // passed in when the job is triggered
var apiBase  = GetSecret("orch-api-url");
var apiToken = GetSecret("orch-api-token");

if (!jobId) {
    Log.Error("the job_id parameter is required");
    Exit(1);
}

// Step 1: claim the job from the orchestration API.
// This tells the API that AFT! is about to start work, and retrieves the
// source and destination details for this specific run.
var claimResp = new HttpCli()
    .Url(apiBase + "/jobs/" + jobId + "/claim")
    .Bearer(apiToken)
    .Post();

if (!claimResp.Ok() || claimResp.StatusCode() >= 300) {
    Log.Error("claim failed: " + (claimResp.Ok() ? "HTTP " + claimResp.StatusCode() : claimResp.ErrorMsg()));
    Exit(1);
}

var job = JSON.parse(claimResp.BodyAsString());
var transferOk = false;

// Step 2: perform the transfer.
try {
    var src = new VirtualFSByName(job.sourceVfs);
    var dst = new VirtualFSByName(job.destVfs);

    var cpResult = src.CopyToVFS(job.sourcePath, dst, job.destDir);
    if (!cpResult.Ok()) {
        Log.Error("transfer failed: " + cpResult.ErrorMsg());
    } else {
        Log.Info("transfer complete: " + job.sourcePath + " to " + job.destVfs + job.destDir);
        transferOk = true;
    }
} catch (err) {
    Log.Error("transfer exception: " + err.toString());
}

// Step 3: always report the outcome, success or failure.
// The orchestration system needs to know what happened, however the transfer
// ended, so that it can route the workflow correctly.
var reportPayload = JSON.stringify({
    jobId:  jobId,
    status: transferOk ? "completed" : "failed",
    agent:  "aft"
});

var reportResp = new HttpCli()
    .Url(apiBase + "/jobs/" + jobId + "/report")
    .Bearer(apiToken)
    .Header("Content-Type", "application/json")
    .ReqBody(reportPayload)
    .Post();

if (!reportResp.Ok() || reportResp.StatusCode() >= 300) {
    Log.Error("report failed: " + (reportResp.Ok() ? "HTTP " + reportResp.StatusCode() : reportResp.ErrorMsg()));
    // No Exit here: fall through to the final exit below with the right code.
}

if (!transferOk) {
    Exit(1);
}

Param("job_id"): Param retrieves a value passed to the job when it was triggered. Whether you trigger this script through the AFT! API or from the command line, you supply the parameter as a key and value pair. The script logic stays generic; the job specific data comes in at run time.

GetSecret("orch-api-token"): storing the bearer token as a named secret keeps the credential out of the script source entirely. Rotate the secret in the AFT! secrets store, and every script that calls GetSecret picks up the new value without any change.

new HttpCli(): the HTTP client is configured with fluent calls (Url, Bearer, Header, ReqBody) and then a verb is called. The request body is what ReqBody was given, sent verbatim; a body that is valid JSON gets Content-Type: application/json automatically, and an explicit Header call wins when you set one. Ok() tells whether the call completed, so a connection or TLS failure makes it false; it says nothing about the status the server answered with, hence the explicit check on StatusCode().

claimResp.BodyAsString(): returns the raw response body as a string. Wrap it in JSON.parse() to work with the object. HttpCli does not parse JSON by itself; which body fields to use is up to the script.

The report call sits outside the try and catch blocks and is sent regardless of the outcome. This is deliberately unconditional. If you report only on success, or only on error, the orchestration system can end up waiting forever for a status that never comes, which is worse than almost any other failure mode.

Triggering the script with a parameter

From the command line, on the machine that runs AFT! or another one that its API key allows:

aft start -n "Orchestrated transfer" -a "your-api-key" --params '{"job_id":"abc-123"}' --insecure

Through the REST API, from any HTTP client or orchestration system:

curl -sk -X POST https://127.0.0.1:44399/v1/jobs \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"scriptName":"Orchestrated transfer","params":{"job_id":"abc-123"}}'

Both forms take the parameters as a flat JSON object of string values. The job starts immediately, and the response carries a job ID. The details of both ways are in Triggering jobs from the command line and Triggering a job through the REST API.