Flask with Amazon S3 Part V
Now that we’ve managed to view, add, delete, and download files, we have an empty home (index) page that’s completely blank (not including our friendly “Hello, Bootstrap”). In this part, we will make use of our home page.
First, there are a couple of mistakes and things we can clean up a bit. You’ll notice there is a lot of repeated code when it comes to getting our S3 Resource and our Bucket in our app.py file. There is also another issue in which we set a global variable for an S3 Resource. However, if we’re relying on the AWS CLI for this, it’s not needed. Basically, passing in the credentials is only necessary if we’re using environment variables and not relying on the CLI. Let’s start by fixing this.
First, we can remove the top, global boto3.resource snippet. Then, we’ll add a new function, _get_s3_resource, right under the section were we initialize our app variable and our filters. We will check if the environment variables for our key and secret are set. If they are, we will use them to get our resource. If not, and we’re relying on the settings from when we setup the AWS CLI, we won’t pass them in.
At this point, our app.py, file should read:
app.py
...
app.jinja_env.filters['file_type'] = file_type
def _get_s3_resource():
if S3_KEY and S3_SECRET:
return boto3.resource(
's3',
aws_access_key_id=S3_KEY,
aws_secret_access_key=S3_SECRET
)
else:
return boto3.resource('s3')
@app.route("/")
def index():
return render_template("index.html")
...
This satisfies the part of getting the S3 Resource, however, we still need to get the Bucket object. We’ll add a second function (get_bucket) for that:
app.py
...
app.jinja_env.filters['file_type'] = file_type
def _get_s3_resource():
if S3_KEY and S3_SECRET:
return boto3.resource(
's3',
aws_access_key_id=S3_KEY,
aws_secret_access_key=S3_SECRET
)
else:
return boto3.resource('s3')
def get_bucket():
s3_resource = _get_s3_resource()
return s3_resource.Bucket(S3_BUCKET)
@app.route("/")
def index():
return render_template("index.html")
...
From our get_bucket, we will call _get_s3_resource, the previous function that we made. This will be helpful because all we will need to do in our route functions is call get_bucket.
app.py
@app.route('/files')
def files():
my_bucket = get_bucket()
summaries = my_bucket.objects.all()
return render_template('files.html', my_bucket=my_bucket, files=summaries)
@app.route('/upload', methods=['POST'])
def upload():
file = request.files['file']
my_bucket = get_bucket()
my_bucket.Object(file.filename).put(Body=file)
flash('File uploaded successfully')
return redirect(url_for('files'))
@app.route('/delete', methods=['POST'])
def delete():
key = request.form['key']
my_bucket = get_bucket()
my_bucket.Object(key).delete()
flash('File deleted successfully')
return redirect(url_for('files'))
@app.route('/download', methods=['POST'])
def download():
key = request.form['key']
my_bucket = get_bucket()
file_obj = my_bucket.Object(key).get()
The _get_s3_resource function is both prepended by an underscore. The reason for this is because it should be private (meaning they shouldn’t be getting called from another file/module). The only common exception is for unit testing purposes. In Python, this by convention only and isn’t possible to enforce.
That being said, we can clean this up a bit more. We can move these new functions into a separate file altogether. Let’s create a new file in our project’s root directory, resources.py.
resources.py
import boto3
from config import S3_BUCKET, S3_KEY, S3_SECRET
def _get_s3_resource():
if S3_KEY and S3_SECRET:
return boto3.resource(
's3',
aws_access_key_id=S3_KEY,
aws_secret_access_key=S3_SECRET
)
else:
return boto3.resource('s3')
def get_bucket():
s3_resource = _get_s3_resource()
return s3_resource.Bucket(S3_BUCKET)
We need to import the boto3 library and the constants from our config file. Now these are no longer being used in app.py, we can delete the unused imports. We will need to add the get_bucket function to our imports, however. Our imports in app.py should now be:
app.py
from flask import Flask, render_template, request, redirect, url_for, flash, \
Response
from flask_bootstrap import Bootstrap
from filters import datetimeformat, file_type
from resources import get_bucket
It’s always a good idea to break up app.py whenever possible since it’s so easy to clutter this file with too much stuff.
After a little cleanup, we can move on to adding new features. We’re not using our home page for anything, so we can use this for listing all of our S3 buckets. In our new resources.py file, we can add a new function to get our bucket list. In order to debug and check to see if we’re pulling the correct data (or any data at all for that matter), we can use the handy pretty-print library. We’ll need to import pprint in addition to creating our new bucket.
resources.py
...
from pprint import pprint
...
def get_buckets_list():
client = boto3.client('s3')
buckets = client.list_buckets()
pprint(buckets)
In order to run this function, we will need to call it form somewhere. In app.py, we can import our get_buckets_list function and call it from our index route.
app.py
...
from resources import get_bucket, get_buckets_list
...
@app.route("/")
def index():
buckets = get_buckets_list()
return render_template("index.html")
When running this, we should see all the bucket info printed to our terminal window in a readable format. We need to make sure our Flask application is running, then navigate to or refresh the index page. The data that should display in your terminal window should look something like this:
{'Buckets': [{'CreationDate': datetime.datetime(2017, 7, 4, 16, 30, 22, tzinfo=tzutc()),
'Name': 'First Bucket'},
{'CreationDate': datetime.datetime(2017, 2, 28, 21, 54, 13, tzinfo=tzutc()),
'Name': 'Second Bucket'},
{'CreationDate': datetime.datetime(2017, 9, 17, 15, 9, 44, tzinfo=tzutc()),
'Name': 'Third Bucket'},
'Owner': {'DisplayName': 'username',
...
So, what we’re looking at is a dictionary. However, we’re mainly interested in the list of buckets. Our next step will then be to modify our get_buckets_list function.
resources.py
...
def get_buckets_list():
client = boto3.client('s3')
return client.list_buckets().get('Buckets')
Since it’s obvious we only need to call the dictionary key of the client’s list_buckets method, we can remove our import and use of pretty-print. Now, our function will only return us the bucket list.
Returning back to app.py, we can pass our bucket list to our index template.
app.py
...
@app.route("/")
def index():
buckets = get_buckets_list()
return render_template("index.html", buckets=buckets)
...
In our index template, we can remove the “Hello Bootstrap” message and replace it with a new table.
...
{% block content %}
<div class="container">
<div class="col-12-xs">
<h3>Bucket List</h3>
<table class="table table-striped">
<tr>
<th>Bucket Name</th>
<th>Created</th>
</tr>
{% for bucket in buckets %}
<tr>
<td>{{ bucket['Name'] }}</td>
<td>{{ bucket['CreationDate'] }}</td>
</tr>
{% endfor %}
</table>
</div>
</div>
{% endblock %}
Recall also we have our datetimeformat filter which we’re already using on the files page. We can simply reuse this here also.
...
{% for bucket in buckets %}
<tr>
<td>{{ bucket['Name'] }}</td>
<td>{{ bucket['CreationDate'] | datetimeformat }}</td>
</tr>
{% endfor %}
...
Our new page should look something like the following:
Currently we have 2 pages, an index page displaying our buckets, and a files page displaying the objects for the bucket that was setup in our environment variable. With 2 pages, we are using templates that have a lot of duplicated code. This would be a good time to clean this up a bit. We will start with a new layout template. For this, lets create the new file:
touch templates/layout.html
The best place to start is with the items being loaded on every page. We’ll also make things like Font Awesome available on every page. This will make things easier as we add new features in the future.
templates/layout.html
{% extends "bootstrap/base.html" %}
{% block html_attribs %} lang="en"{% endblock %}
{% block head %}
{{super()}}
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.0.8/css/solid.css" integrity="sha384-v2Tw72dyUXeU3y4aM2Y0tBJQkGfplr39mxZqlTBDUZAb9BGoC40+rdFCG0m10lXk" crossorigin="anonymous">
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.0.8/css/fontawesome.css" integrity="sha384-q3jl8XQu1OpdLgGFvNRnPdj5VIlCvgsDQTQB6owSOHWlAurxul7f+JpUOVdAiJ5P" crossorigin="anonymous">
<link rel="stylesheet" href="/static/styles.css">
{% endblock %}
{% block title %}Flask S3 Browser{% endblock %}
{% block navbar %}
<div class="navbar navbar-fixed-top">
<!-- ... -->
</div>
{% endblock %}
<div class="container">{% block content %}{% endblock %}</div>
Extending Bootstrap giving to us via the Flask-Bootstrap package, along with our head block can go here. In our actual templates, we’ll be overriding the title as well as the content. For the content, we can just set this to an empty block for now.
Returning back to index template, we can now remove all this boilerplate stuff, and replace it with a different extends statement. Instead of each template file extending “bootstrap/base.html”, each one can extend our layout.html template.
templates/index.html
{% extends "layout.html" %}
{% block title %}S3 Bucket List{% endblock %}
{% block content %}
<div class="container">
<div class="col-12-xs">
<h3>Bucket List</h3>
<table class="table table-striped">
<tr>
<th>Bucket Name</th>
<th>Created</th>
</tr>
{% for bucket in buckets %}
<tr>
<td>{{ bucket['Name'] }}</td>
<td>{{ bucket['CreationDate'] | datetimeformat }}</td>
</tr>
{% endfor %}
</table>
</div>
</div>
{% endblock %}
Now our index template has very little code in it. Which is nice since we can just focus on the page content. Notice that we overrode the title block with our new title. If we didn’t include the title block, our page would simply render “Flask S3 Browser” which was setup in the layout. In addition, we overrode the content block with our page’s content.
Now that we’ve cleaned up our index template, we can do the same with our files template. Everything we need to change will be at the top of the file:
templates/files.html
{% extends "layout.html" %}
{% block title %}S3 Object List{% endblock %}
{% block content %}
<div class="container">
<div class="col-12-xs">
<h3>Bucket Info</h3>
<p>Created: {{ my_bucket.creation_date | datetimeformat }}</p>
Before we wrap up part 5, it would be nice to have some navigation since we have 2 pages with actual content. Since we’ve created the layout template, we will only need to put the nav bar in one place.
templates/layout.html
...
{% block navbar %}
<div class="container">
<div class="navbar">
<ul class="nav navbar-nav">
<li><a href="{{ url_for('index') }}">Buckets</a></li>
<li><a href="{{ url_for('files') }}">Files</a></li>
</ul>
</div>
</div>
{% endblock %}
...
Flask Bootstrap gave us a starting placeholder to put our nav bar. We’re going to wrap it into a container element so it lines up with the rest of our content. We can also use Flask’s url_for method to attach the routes to the href attributes for each link.