Python Upload to S3 Bucket With Public Permission

• Language Back up
• Python
• Direct to S3 File Uploads in Python

This commodity was contributed by Volition Webberley

Will is a computer scientist and is enthused past almost all aspects of the technology domain. He is specifically interested in mobile and social computing and is currently a researcher in this surface area at Cardiff University.

Direct to S3 File Uploads in Python

English — 日本語に切り替える

Last updated November 24, 2020

Tabular array of Contents

• Uploading direct to S3
• Overview
• Prerequisites
• Initial setup
• Straight uploading
• Running the app
• Summary

Web applications often require the ability to permit users to upload files such as images, movies and archives. Amazon S3 is a pop and reliable storage choice for these files.

This commodity demonstrates how to create a Python application that uploads files directly to S3 instead of via a spider web application, utilising S3'southward Cross-Origin Resources Sharing (CORS) support. The commodity and companion repository consider Python 2.7, but should be by and large also compatible with Python 3.3 and above except where noted beneath.

Uploading straight to S3

A consummate example of the code discussed in this article is available for direct use in this GitHub repository.

The chief advantage of directly uploading is that the load on your application's dynos would exist considerably reduced. Using server-side processes for receiving files and transferring to S3 tin needlessly tie upwards your dynos and will mean that they volition non exist able to respond to simultaneous web requests equally efficiently.

If your application relies on some form of file processing betwixt the client's computer and S3 (such as parsing Exif data or applying watermarks to images), then y'all may demand to employ the use of extra dynos and pass the upload through your webserver.

The application uses client-side JavaScript and Python for signing the requests. It will therefore be a suitable guide for developing applications for the Flask, Bottle and Django spider web frameworks. The upload is carried out asynchronously so that you tin decide how to handle your application's catamenia afterwards the upload has completed (for example, a page redirect upon successful upload rather than a total folio refresh).

An example simple account-editing scenario is used as a guide for completing the various steps required to accomplish the direct upload and to chronicle the application of this to a wider range of use-cases. More information on this scenario is provided later.

Overview

S3 is comprised of a set of buckets, each with a globally unique proper name, in which individual files (known as objects) and directories, tin can be stored.

For uploading files to S3, y'all will need an Admission Key ID and a Secret Access Key, which act as a username and password. The admission key account volition need to have sufficient access privileges to the target bucket in order for the upload to be successful.

Please run into the S3 Article for more data on this, creating buckets and finding your Access Key ID and Secret Admission Cardinal.

The method described in this commodity involves the utilise of client-side JavaScript and server-side Python. In general, the completed image-upload procedure follows these steps:

• A file is selected for upload by the user in their web browser;
• JavaScript is so responsible for making a request to your web application on Heroku, which produces a temporary signature with which to sign the upload asking;
• The temporary signed request is returned to the browser in JSON format;
• JavaScript then uploads the file directly to Amazon S3 using the signed request supplied by your Python awarding.

This guide includes information on how to implement the client-side and server-side code to form the complete system. Later on following the guide, you should have a working barebones system, allowing your users to upload files to S3. Nonetheless, it is usually worth calculation extra functionality to help improve the security of the arrangement and to tailor it for your ain particular uses. Pointers for this are mentioned in the advisable parts of the guide.

Prerequisites

• The Heroku CLI has been installed;
• A Heroku application has been created for the current project;
• An AWS S3 bucket has been created. For sit-in purposes we assume a bucket has been created that permits the cosmos of public objects. In a production environment yous may want to utilise individual objects that can be accessed via signed URLs.

During the first few hours after a saucepan has been created S3 may render redirects in response to upload requests. If yous notice this behaviour, and so waiting a short while for your new bucket to completely settle should fix the problem.

Initial setup

Heroku setup

In club for your application to access the AWS credentials for signing upload requests, they will need to be added equally configuration variables in Heroku:

If you are testing locally before deployment, remember to add together the credentials to your local machine'south environment, too.

            $ heroku config:fix AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy Adding config vars and restarting app... done, v21     AWS_ACCESS_KEY_ID     => 30     AWS_SECRET_ACCESS_KEY => yyy                      

In improver to the AWS access credentials, fix your target S3 bucket'due south name (not the saucepan's ARN):

            $ heroku config:prepare S3_BUCKET=zzz Adding config vars and restarting app... washed, v21     S3_BUCKET     => zzz                      

Using config vars is preferable over configuration files for security reasons. Effort to avert placing passwords and access keys directly in your awarding's lawmaking or in configuration files.

S3 setup

You volition now need to edit some of the permissions properties of the target S3 bucket so that the last request has sufficient privileges to write to the bucket. In a web-browser, sign in to the AWS panel and select the S3 section. Select the appropriate saucepan and click the Permissions tab. A few options are at present provided on this page (including Block public access, Access Command List, Bucket Policy, and CORS configuration).

Firstly, ensure that "Block all public access" is turned off, and in particular turn off "Block public access to buckets and objects granted through new access control lists" and "Block public admission to buckets and objects granted through any access control lists" for the purposes of this project. Setting upwards the bucket in this mode allows us to read its contents without signed URLs, merely this may not be suitable for services running in production.

Next, you will need to configure the bucket's CORS (Cross-Origin Resource Sharing) settings, which will allow your application to access content in the S3 bucket. Each dominion should specify a set of domains from which access to the bucket is granted and as well the methods and headers permitted from those domains.

For this to piece of work in your application, click Edit and enter the post-obit JSON:

            [     {         "AllowedHeaders": [             "*"         ],         "AllowedMethods": [             "Get",             "Head",             "Postal service",             "PUT"         ],         "AllowedOrigins": [             "*"         ],         "ExposeHeaders": []     } ]                      

Click Save changes and close the editor.

This tells S3 to allow whatever domain access to the bucket and that requests can contain any headers. For security, you can change the 'AllowedOrigins' array to only accept requests from your domain.

If y'all wish to use S3 credentials specifically for this application, then more than keys can be generated in the AWS business relationship pages. This provides farther security, since you can designate a very specific prepare of requests that this gear up of keys are able to perform. If this is preferable to you, then you volition need to likewise fix up an IAM user in the Edit bucket policy option in your S3 bucket. In that location are various guides on AWS'southward web pages detailing how this can be achieved.

Direct uploading

The processes and steps required to accomplish a direct upload to S3 will be demonstrated through the use of a simple profile-editing scenario for the purposes of this article. This case will involve the user beingness permitted to select an avatar image to upload and enter some basic information to be stored as part of their business relationship.

In this scenario, the following procedure volition take identify:

• The user is presented with a web page, containing elements encouraging the user to choose an image to upload as their avatar and to enter a username and their own proper noun.
• An chemical element is responsible for maintaining a preview of the chosen image by the user. By default, and if no image is chosen for upload, a default avatar image is used instead (making the image-upload effectively optional to the user in this scenario).
• When a user selects an image to be uploaded, the upload to S3 is handled automatically and asynchronously with the process described before in this commodity. The image preview is then updated with the selected image one time the upload is complete and successful.
• The user is then free to move on to filling in the rest of the information.
• The user and then clicks the Submit push button, which posts the username, name and the URL of the uploaded image to the Python application to be checked and/or stored. If no image was uploaded past the user earlier the default avatar image URL is posted instead.

Setting up the customer-side lawmaking

No 3rd-political party code is required to complete the implementation on the customer-side.

The HTML and JavaScript can now be created to handle the file pick, obtain the request and signature from your Python application, and then finally brand the upload request.

Firstly, create a file chosen account.html in your application's templates directory and populate the head and other necessary HTML tags accordingly for your application. In the torso of this HTML file, include a file input and an element that will contain condition updates on the upload progress. In addition to this, create a class to allow the user to enter their username and full name and a subconscious input element to concord the URL of the chosen avatar prototype:

To see the completed HTML file, please see the appropriate code in the companion repository.

            <input type="file" id="file_input"/> <p id="status">Please select a file</p> <img id="preview" src="/static/default.png" />  <form method="POST" action="/submit_form/">   <input type="hidden" id="avatar-url" proper name="avatar-url" value="/static/default.png">   <input type="text" name="username" placeholder="Username">   <input type="text" name="full-name" placeholder="Full name">   <input type="submit" value="Update profile"> </form>                      

The #preview element initially holds a default avatar epitome (which would become the user's avatar if a new paradigm is not chosen), and the #avatar-url input maintains the current URL of the user's chosen avatar image. Both of these are updated by the JavaScript, discussed beneath, when the user selects a new avatar.

Thus when the user finally clicks the Submit button, the URL of the avatar is submitted, along with the username and full name of the user, to your desired endpoint for server-side handling.

The client-side code is responsible for achieving two things:

1. Retrieve a signed request from the app with which the epitome tin be POSTed to S3
2. Actually Mail the epitome to S3 using the signed asking

JavaScript's XMLHttpRequest objects can be created and used for making asynchronous HTTP requests.

To reach this, first create a <script> block and write some code that listens for changes in the file input, in one case the document has loaded, and starts the upload process.

            (function() {   document.getElementById("file_input").onchange = function(){     var files = document.getElementById("file_input").files;     var file = files[0];     if(!file){       return alert("No file selected.");     }     getSignedRequest(file);   }; })();                      

The code also determines the file object itself to exist uploaded. If i has been selected properly, it proceeds to call a function to obtain a signed POST request for the file. Next, therefore, write a function that accepts the file object and retrieves an advisable signed asking for information technology from the app.

            office getSignedRequest(file){   var xhr = new XMLHttpRequest();   xhr.open("Get", "/sign_s3?file_name="+file.name+"&file_type="+file.type);   xhr.onreadystatechange = function(){     if(xhr.readyState === 4){       if(xhr.status === 200){         var response = JSON.parse(xhr.responseText);         uploadFile(file, response.data, response.url);       }       else{         alarm("Could not become signed URL.");       }     }   };   xhr.send(); }                      

The in a higher place function passes the file'due south proper noun and mime type as parameters to the GET request since these are needed in the construction of the signed request, as volition be covered subsequently in this article. If the retrieval of the signed request was successful, the role continues past calling a function to upload the actual file:

            function uploadFile(file, s3Data, url){   var xhr = new XMLHttpRequest();   xhr.open up("Postal service", s3Data.url);    var postData = new FormData();   for(key in s3Data.fields){     postData.suspend(key, s3Data.fields[central]);   }   postData.suspend('file', file);    xhr.onreadystatechange = office() {     if(xhr.readyState === iv){       if(xhr.status === 200 || xhr.status === 204){         certificate.getElementById("preview").src = url;         document.getElementById("avatar-url").value = url;       }       else{         alert("Could not upload file.");       }    }   };   xhr.ship(postData); }                      

This office accepts the file to be uploaded, the S3 request data, and the URL representing the eventual location of the avatar image. The latter two arguments will be returned as function of the response from the app. The office, if the asking is successful, updates the preview element to the new avatar prototype and stores the URL in the hidden input so that it can exist submitted for storage in the app.

Now, in one case the user has completed the residue of the grade and clicked Submit , the name, username, and avatar image tin all exist posted to the same endpoint.

If you find that the page isn't working every bit you intend after implementing the system, then consider using console.log() to record whatsoever errors that are revealed by the onreadystatechange function and use your browser's error panel to help diagnose the trouble.

Information technology is good exercise to inform the user of any prolonged activeness in whatsoever form of awarding (web- or device-based) and to brandish updates on changes. Therefore a loading indicator could exist displayed betwixt selecting a file and the upload existence completed. Without this sort of data, users may suspect that the page has crashed, and could endeavour to refresh the page or otherwise disrupt the upload process.

Setting upwards the server-side Python code

This section discusses the use of Python for generating a temporary signature with which the upload request tin can exist signed. This temporary signature uses the AWS access key and surreptitious access key as a basis for the signature, merely users will not accept direct admission to this information. After the signature has expired, and so upload requests with the same signature volition not exist successful.

As mentioned previously, this article covers the product of an application for the Flask framework, although the steps for other Python frameworks will exist similar. Readers using Python 3 should consider the relevant data on Flask's website before continuing.

To see the completed Python file, please come across the advisable code in the companion repository.

Showtime by creating your master application file, application.py, and set up your skeleton application appropriately:

            from flask import Flask, render_template, request, redirect, url_for import os, json, boto3  app = Flask(__name__)  if __name__ == '__main__':   port = int(os.environ.become('PORT', 5000))   app.run(host='0.0.0.0', port = port)                      

The currently-unused import statements volition be necessary afterward on. boto3 is a Python library that will generate the pre-signed POST request. This, along with Flask, tin can be installed but using pip.

Next, in the same file, you lot will need to create the views responsible for returning the correct information back to the user's browser when requests are made to various URLs. Showtime define view for requests to /business relationship to render the page account.html, which contains the form for the user to complete:

            @app.route("/account/") def business relationship():     return render_template('account.html')                      

Please note that the views for the application will need to exist placed between the app = Flask(__name__) and if __name__ == '__main__': lines in application.py.

Now create the view, in the same Python file, that is responsible for generating and returning the signature with which the client-side JavaScript can upload the image. This is the kickoff request fabricated by the client before attempting an upload to S3. This view responds with requests to /sign_s3/:

            @app.route('/sign_s3/') def sign_s3():   S3_BUCKET = os.environ.go('S3_BUCKET')    file_name = asking.args.get('file_name')   file_type = request.args.get('file_type')    s3 = boto3.client('s3')    presigned_post = s3.generate_presigned_post(     Bucket = S3_BUCKET,     Key = file_name,     Fields = {"acl": "public-read", "Content-Type": file_type},     Conditions = [       {"acl": "public-read"},       {"Content-Type": file_type}     ],     ExpiresIn = 3600   )    return json.dumps({     'data': presigned_post,     'url': 'https://%due south.s3.amazonaws.com/%s' % (S3_BUCKET, file_name)   })                      

If your bucket is in a region that requires a v4 signature, and then you can modify your boto3 client configuration to declare this:

              s3 = boto3.client('s3', config = Config(signature_version = 's3v4'))                          

This lawmaking performs the post-obit steps:

• The request is received to /sign_s3/ and the S3 saucepan name is loaded from the environment.
• The proper name and mime type of the object to be uploaded are extracted from the Become parameters of the request (this stage may differ in other frameworks). The parameters are provided by the JavaScript discussed in the previous section.
• An S3 client is constructed using the boto3 library. At this phase, the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY prepare before are automatically read from the surroundings.
• The pre-signed POST request data is then generated using the generate_presigned_post function. To this is passed the saucepan proper name, the name of the file, some parameters to let the uploaded file to be publicly readable, and an expiry time of the signed asking (in seconds).
• Finally, the pre-signed asking data and the location of the eventual file on S3 are returned to the client as JSON.

You may wish to assign some other, customised name to the object instead of using the one that the file is already named with, which is useful for preventing accidental overwrites in the S3 bucket. This name could be related to the ID of the user'south business relationship, for example. If not, you should provide some method for properly quoting the name in example there are spaces or other awkward characters present. In addition, this is the stage at which you could provide checks on the uploaded file in order to restrict access to certain file types. For example, a simple cheque could exist implemented to allow just .png files to continue beyond this point.

Finally, in application.py, create the view responsible for receiving the account information after the user has uploaded an avatar, filled in the form, and clicked Submit . Since this will be a POST request, this volition too need to exist defined as an 'allowed access method'. This method will respond to requests to the URL /submit_form/:

            @app.road("/submit_form/", methods = ["POST"]) def submit_form():   username = request.form["username"]   full_name = request.form["total-name"]   avatar_url = request.form["avatar-url"]    update_account(username, full_name, avatar_url)    return redirect(url_for('profile'))                      

In this example, an update_account() function has been called, just creation of this method is not covered in this commodity. In your application, y'all should provide some functionality, at this stage, to allow the app to store these business relationship details in some form of database and correctly associate the information with the rest of the user's account details.

In addition, the URL for the profile page has non been defined in this commodity (or companion code). Ideally, for example, after updating the account, the user would be redirected dorsum to their own profile and then that they can run across the updated data.

Running the app

Everything should at present be in place to perform the direct uploads to S3. To test the upload, save any changes and use heroku local to start the application:

You will need a Procfile for this to exist successful. See Getting Started with Python on Heroku for information on the Heroku CLI and running your app locally. Too remember to correctly set your surroundings variables on your ain machine before running the application locally.

            $ heroku local 15:44:36 web.one  | started with pid 12417                      

Press Ctrl+C to return to the prompt. If your awarding is returning 500 errors (or other server-based issues), then start your server in debug mode and view the output in the Terminal emulator to assistance fix your problem. For example, in Flask:

            ... app.debug = True port = int(os.environ.get('PORT', 5000)) app.run(host='0.0.0.0', port=port)                      

Summary

This article covers uploading to Amazon S3 directly from the browser using Python to temporarily sign the upload request. Although the guide and companion code focuses on the Flask framework, the idea should hands carry over to other Python applications.

smithdowen1979.blogspot.com

Source: https://devcenter.heroku.com/articles/s3-upload-python

Share this post