Test the delivery as a whole—not just whichever file happens to be visible

Automatically Test SFTP Deliveries: Correlate, Claim, and Archive with Confidence

The broader approach is covered by the main article on automated interface quality. The published series covers JSON Schema, CSV and fixed-width files, HTTP contracts, and the stable one-file SFTP flow. This article shows how to turn the SFTP boundary into an operationally understandable delivery lifecycle.

The v3 profile as an explicit contract

Version 3 builds on the bounded directory selector from version 2. filePattern identifies the primary file. companionPattern identifies its companion. Both patterns are matched against basenames only. deliveryIdPattern contains a named deliveryId group that must produce the same value for both file names.

{
  "format": "sftp",
  "version": 3,
  "host": "sftp.example.test",
  "port": 22,
  "username": "quality-reader",
  "remoteDirectory": "/incoming",
  "filePattern": "^delivery-[A-Z0-9-]+[.]json$",
  "companionPattern": "^delivery-[A-Z0-9-]+[.]meta[.]json$",
  "deliveryIdPattern": "^delivery-(?[A-Z0-9-]+)(?:[.]meta)?[.]json$",
  "claimRemoteDirectory": "/claim",
  "archiveRemoteDirectory": "/archive",
  "knownHosts": "known_hosts",
  "privateKey": "id_ed25519",
  "contentFormat": "json",
  "contentProfile": "delivery.schema.json",
  "quarantineRemoteDirectory": "/quarantine",
  "stabilityChecks": 2,
  "stabilityDelaySeconds": 1,
  "maxAttempts": 2,
  "retryDelaySeconds": 1,
  "connectTimeoutSeconds": 10
}

The host-key and private-key files remain local files below the profile directory. The profile contains no passphrase; a protected key receives its passphrase only from the configured environment variable. The remote locations are intentionally separate: the producer writes to /incoming, while the validator owns /claim and /archive.

Multiple files, one delivery ID

A delivery is not complete just because one matching file exists. The v3 selector groups candidates by delivery ID. Exactly one primary and exactly one companion must form one complete delivery. This remains deterministic without hiding a business decision inside “take the newest file.”

Claiming and safe recovery

The claim directory is the validator’s ownership boundary. Once one complete delivery has been found, every selected file is renamed remotely from /incoming to /claim. The validator downloads only after the claim succeeds. A parallel run no longer sees those files in the producer-facing directory and cannot select them again.

1. Check the claim directory first for a delivery that was already taken.
2. If no claim exists, inspect the input directory for one complete delivery.
3. Rename every file in that delivery into the claim directory.
4. If the process stops, resume the claimed delivery on the next run.

That turns “try again” into a defined recovery path. A failed claim is an operational error with exit code 2. If only part of the claim succeeds, the adapter attempts to move the files it already claimed back to the input directory; the original transfer failure remains the actionable error.

Stability, integrity, and content

Claiming does not replace validation. It protects the handoff from the producer area into the active check. The established SFTP pipeline then continues:

1. Verify the host key and private-key access strictly.
2. Observe type, size, and modification time as a stable snapshot for every file.
3. Download all bytes into a temporary local directory.
4. Compare the optional SHA-256 value with the downloaded primary file.
5. Validate both the primary and companion file with the configured JSON, CSV, or fixed-width contract.

The shared content path is deliberately a first, bounded form of correlation: both files use the same content profile. Mixed formats or business rules such as “the total in file A must equal the total in file B” need a separate contract. Keeping that boundary visible makes the current behavior easier to trust and troubleshoot.

Archive or quarantine

A valid result is no longer just a terminal status. When claim and archive directories are configured, every checked file moves to the archive only after the complete delivery passes. When the hash or content contract fails, the entire claimed delivery moves to quarantine if a quarantine directory is enabled.

Final names include a UTC timestamp and a short collision-avoidance suffix. A later run cannot silently overwrite an earlier outcome. A failure during a final remote move is an operational error with exit code 2, not a data-contract finding.

The operational report

The shared report keeps status, valid, and findings. A v3 lifecycle may add a metadata object. It contains no payload or secrets—only the information needed to explain the state transition:

{
  "status": "VALID",
  "valid": true,
  "findings": [],
  "metadata": {
    "deliveryId": "D-200",
    "filesValidated": 2,
    "lifecycle": "ARCHIVED",
    "archivedFiles": [
      "/archive/delivery-D-200.json.20260928180000-ab12cd34.validated",
      "/archive/delivery-D-200.meta.json.20260928180000-ef56ab78.validated"
    ]
  }
}

Older adapters and v1/v2 SFTP runs keep the compact report because empty metadata is omitted. Automation can still decide from status and findings; the lifecycle metadata adds useful context for operations, audit trails, and manual recovery.

The local proof

Readers do not need a public SFTP account to reproduce this slice. Unit tests cover correlation, incomplete deliveries, claim recovery, and remote moves through a controlled transport fake. The integration test starts an embedded Apache MINA SSHD server and runs the production JSch transport against it.

mvnw.cmd -B -Dtest=SftpValidatorTest,SftpValidatorIntegrationTest test

mvnw.cmd -B clean package

Deliberate boundaries

• no “newest file wins” rule and no selection based on server ordering;
• exactly one primary and, when configured, exactly one companion per correlated delivery;
• both files currently use the same content profile;
• no password authentication, SSH agent, or automatic OpenSSH configuration;
• no semantic cross-file rules and no permanent directory watcher.

These boundaries are part of the contract. They keep a practical interface check from silently becoming a general-purpose SFTP orchestrator. A future step can now make an explicit decision about mixed formats and true cross-file business rules.