Out-of-Transfer File Validation

Out-of-transfer file validation is run as soon as the client uploads a to HST Server. The transfer is reported as complete and then the validation is run. The validation script uses the Aspera Reliable Query API to retrieve the list of files to validate and update the file status during validation. The transfer user who is transferring files to the server must be associated with Node API user credentials in order to use the API. These instructions describe how to associate a transfer user with Node API user credentials, create a validation script, and configure the server to use out-of-transfer file validation on files that it receives from specific transfer users, groups, or globally.

This approach has several benefits over inline file validation:

More efficient use of system resources because the ascp sessions can close before validation is completed.
Out-of-transfer file validation is applied to transfers that use HTTP(S) fallback transport.
Files are explicitly reported as "validating" to IBM Aspera Faspex through asperacentral. Files that are validated inline are reported as "transferring".

Associate the transfer user with a Node API username and password, if not already configured.
```
# /opt/aspera/bin/asnodeadmin -a -u node_username -p node_password -x transfer_user
```
To view existing Node API users and the transfer users associated with them, run the following command:
```
# /opt/aspera/bin/asnodeadmin -l
```
Create your validation script.
Note: The validation service must be executed on a system that has access to the storage.
The validation script should follow these steps:
1. Identify the files that need to be validated by using the Reliable Query REST API:
```
curl -X POST -u node_user:password -d '{ "file_transfer_filter": { "max_result": 20}, "validation": { "validator_id": "validator_id" } }' https://server_name:9092/services/rest/transfers/v1/files
```
  Where the validator_id is a unique ID to prevent simultaneous validation of the same file by different validators. The value for max_result sets a "batch size" for how many files are collected for validation by each POST request, and cannot exceed 1000.
  
  The POST request retrieves the files that are "to_be_validated", updates their state to "validating" and the owner to the validator ID, and returns the file list, with information similar to the following:
```
{
   "file_transfer_info_result": {
        "file_transfer_info" : [
            { "session_uuid":"9a2678c3-64db-4bc1-abd4-605ad7702230",
              "path" :"/tmp/src/dir","local_id":1,
              "file_id":"47203042-bb57-487f-95df-ad614d0a3720",
               "status":"validating", 
               "new_file":true, "error_code":0,
               "size":10000000,
               "start_offset":0,
               "bytes_written":10000000,
               "bytes_contiguous":0, "bytes_lost":0, "elapsed":0,"bytes_processed":0,
               "start_date":"2017-11-29T16:21:24Z",
               "checksum_type":"None"
 } ],
"iteration_token":"0000000000000003",
"remaining_result_count":1,
"result_count":1
} }
```
2. Validate the files and update the "bytes_processed".
  By updating the "bytes_processed", the GUI can display a progress bar:
```
curl -X PUT -u node_user:password -d '{ "validator_id": "validator_id", "files": [{ "session_uuid": "session_uuid", "file_id": "file_id", "status": "validating", "bytes_processed": bytes } ] }' https://server_name:9092/services/rest/transfers/v1/files
```
  Note: If a validator does not update the file status within the validation timeout, the file status is reset to "to_be_validated" and the file is released from the validator so that the file can be validated by a different validator. The default timeout is 5 minutes. To edit the validation timeout, go to Configuration > Transfer Server in the GUI and override the value for Post-Transfer Validation Timeout, or run the following command:
  # asconfigurator -x "set_central_server_data;validation_timeout,seconds"
3. Update the status of each file as validation completes or fails:
  If a file passes validation, update its status to "completed":
```
curl -X PUT -u node_user:password -d '{ "validator_id": "validator_id", "files": [{ "session_uuid": "session_uuid", "file_id": "file_id", "status": "completed" } ] }' https://server_name:9092/services/rest/transfers/v1/files
```
  If the file fails validation, update its status to "error" and provide an error code (as a number) and error description (as a string):
```
curl -X PUT -u node_user:password -d '{ "validator_id": "validator_id", "files": [{ "session_uuid": "session_uuid", "file_id": "file_id", "status": "error", "error_code": error_number, "error_description": "error_string" } ] }' https://server_name:9092/services/rest/transfers/v1/files
```
  For example, the body of a PUT request could contain the following information for three files:
```
{
    "validator_id": "my identifier",
    "files": [
       {
	"session_uuid": "1425c741-32bb-492d-b5e1-724c8bdb1fbf",
	"file_id": "11111111-11422dfb-5b8ed464-239783b8-09c78597",
	"status": "validating",
	"bytes_processed": 3
       },
       {
	"session_uuid": "1425c741-32bb-492d-b5e1-724c8bdb1fbf",
	"file_id": "22222222-11422dfb-5b8ed464-239783b8-09c78597",
	"status": "completed"
        },
       {
	"session_uuid": "1425c741-32bb-492d-b5e1-724c8bdb1fbf",
	"file_id": "33333333-11422dfb-5b8ed464-239783b8-09c78597",
	"status": "error",
          "error_code": 2,
          "error_description": "File not found"
        }
    ]
}
```
If all files validate and update successfully, HTTP 204 is returned. If one or more files have failed validation, HTTP 200 is returned. For each failed file, an entry is added to the result. If another HTTP code is returned, then a more general error, such as invalid JSON, has occurred.
Confirm that persistent storage is enabled (the default setting).
In the GUI, go to Configuration > Transfer Server and confirm that Persistent Storage is set to enable.
From the command line, run the following command:
```
# /opt/aspera/bin/asuserdata -c
```
In the output, locate the value for "persistent_store". If it is not set to "enable", run the following command:
```
# asconfigurator -x "set_central_server_data;persistent_store,enable"
```
Ensure that empty files and files that exist at the destination (and are skipped by the transfer session) are not ignored.
In the GUI, go to Configuration > Transfer Server and confirm that Ignore No-transfer Files is set to false.
From the command line, run the following command:
```
# asconfigurator -x "set_central_server_data;ignore_no_transfer_files,false"
```
If ignore_no_transfer_files is set to true, the workflow might fail when the transfer attempts to create empty files on the destination and they are not validated.
Schedule the validation.
The validation can be scheduled for one or more users (files that are transferred to the server by those users are validated), for one or more groups (files that are transferred to the server by users in the groups are validated), or globally (all files that are transferred to the server for all users are validated).

In the GUI, go to Configuration > File Handling for a user, group, or global, and set Run File Validation at File Stop to post_transfer.
From the command line, run the command corresponding to the scope of your configuration:
```
# asconfigurator -x "set_user_data;user_name,username;validation_file_stop,post_transfer"
# asconfigurator -x "set_group_data;group_name,groupname;validation_file_stop,post_transfer"
# asconfigurator -x "set_node_data;validation_file_stop,post_transfer"
```