Starting an Upload

Start an upload by POSTing the filename and size to /samples/init_upload. To actually perform the upload it is then necessary to transmit the file to the provided upload_url, with any additional_fields as needed. This second step varies depending on the upload_type. Currently only a standard upload type is available.

Performing the actual upload – "standard" upload type

The standard upload involves POSTing a sample up to 20GB in size to a provided HTTPS URL. The Content-Type should be multipart/form-data and the additional_fields should be included in the POST body.

The workflow using httpie or Python requests is:

# Start the upload
http --auth $ONE_CODEX_API_KEY: POST \ 
https://app.onecodex.com/api/v1/samples/init_upload \
filename=HiSeq_accuracy.fa size:=1189333

# Returns a 200 response with the following JSON body:
# {
#     "additional_fields": {
#         "AWSAccessKeyId": "XXXXXXXXXXXXXXXXXXXX",
#         "acl": "private",
#         "key": "user_xxxxxxxxxxxxxxxx/file_yyyyyyyyyyyyyyyy/${filename}",
#         "policy": "CiAgICAgICAgICAgAgICAgICB7ImJ1Y2tldCI6ICJyZWZnZW5vbWljcy11c2VyZGF0YS1kZXYtZW5jcnlwdGVkIiB9LAogICAS1lbmNyeXB0aW9uIjogIkFFUzI1NiJ9LAogICAgICAgICAgICAgICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgInVzZXJfNGFkYTU2MTAzZDlhNDhiOC9maWxlXzI4NDU5NTA4NTkyYTQ4MWMvIl0sCiAgICAgICAgICAgICAgICB7InN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyI6ICIyMDEifSwKICAgICAgICAgICAgICBdCiAgICAgICAgICAgIH0KICAgICAgICAgICAg",
#         "signature": "ELADfQLgxXXXXXXXXx/5D99Q9AY=",
#         "success_action_status": 201,
#         "x-amz-server-side-encryption": "AES256"
#     },
#     "sample_id": "28459508592a481c",
#     "upload_url": "https://refgenomics-userdata-dev-encrypted.s3.amazonaws.com"
# }

http -f POST \ 
    https://sample-upload-bucket.s3.amazonaws.com \
    AWSAccessKeyId="XXXXXXXXXXXXXXXXXXXX" acl="private" \ 
    key="user_xxxxxxxxxxxxxxxx/file_yyyyyyyyyyyyyyyy/${filename}" \ 
    policy="CiAgICAgICAgICAgAgICAgICB7ImJ1Y2tldCI6ICJyZWZnZW5vbWljcy11c2VyZGF0YS1kZXYtZW5jcnlwdGVkIiB9LAogICAS1lbmNyeXB0aW9uIjogIkFFUzI1NiJ9LAogICAgICAgICAgICAgICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgInVzZXJfNGFkYTU2MTAzZDlhNDhiOC9maWxlXzI4NDU5NTA4NTkyYTQ4MWMvIl0sCiAgICAgICAgICAgICAgICB7InN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyI6ICIyMDEifSwKICAgICAgICAgICAgICBdCiAgICAgICAgICAgIH0KICAgICAgICAgICAg" \
    signature="ELADfQLgxXXXXXXXXx/5D99Q9AY=" success_action_status=201 \
    x-amz-server-side-encryption="AES256" \ 
    file@HiSeq_accuracy.fa
    
# Returns a 201 response

# Confirm the upload (see below)
http --auth $ONE_CODEX_API_KEY: POST \ 
    https://app.onecodex.com/api/v1/samples/confirm_upload \
    sample_id="28459508592a481c"

# Start the upload
resp = requests.post('https://app.onecodex.com/api/v1/init_upload', 
                     json={'filename': 'HiSeq_accuracy.fa', 
                           'size': os.path.getsize('HiSeq_accuracy.fa')},
                     auth=(os.getenv('ONE_CODEX_API_KEY'), '')
                    ).json()

# Perform the upload (should return a 201)
requests.post(resp['upload_url'], data=resp['additional_fields'], 
             files={'file': open('HiSeq_accuracy.fa')})

# Finally, confirm the upload (should return a 200)
requests.post('https://app.onecodex.com/api/v1/samples/confirm_upload', 
              json={'sample_id': resp['sample_id']},
              auth=(os.getenv('ONE_CODEX_API_KEY'), ''))