Amazon S3 upload and download using Python/Django

This article describes how you can upload files to Amazon S3 using Python/Django and how you can download files from S3 to your local machine using Python.

We assume that we have a file in /var/www/data/ which we received from the user (POST from a form for example).

You need to create a bucket on Amazon S3 to contain your files. This can be done using the Amazon console.

Now, we are going to use the python library boto to facilitate our work.

We define the following variables in settings.py of our Django project:

BUCKET_NAME = 'bucket_name'
AWS_ACCESS_KEY_ID = ...
AWS_SECRET_ACCESS_KEY = ...

Upload to S3

Here is the code we use to upload the picture files:

def push_picture_to_s3(id):
  try:
    import boto
    from boto.s3.key import Key
    # set boto lib debug to critical
    logging.getLogger('boto').setLevel(logging.CRITICAL)
    bucket_name = settings.BUCKET_NAME
    # connect to the bucket
    conn = boto.connect_s3(settings.AWS_ACCESS_KEY_ID,
                    settings.AWS_SECRET_ACCESS_KEY)
    bucket = conn.get_bucket(bucket_name)
    # go through each version of the file
    key = '%s.png' % id
    fn = '/var/www/data/%s.png' % id
    # create a key to keep track of our file in the storage 
    k = Key(bucket)
    k.key = key
    k.set_contents_from_filename(fn)
    # we need to make it public so it can be accessed publicly
    # using a URL like http://s3.amazonaws.com/bucket_name/key
    k.make_public()
    # remove the file from the web server
    os.remove(fn)
  except:
    ...

Download from S3

As you saw, you can access the file using the URL: http://s3.amazonaws.com/bucket_name/key but you can also use the boto library to download the files. I do that to create a daily backup of the bucket’s files on my local machine.

Here is the script to do that:

import boto
import sys, os
from boto.s3.key import Key

LOCAL_PATH = '/backup/s3/'
AWS_ACCESS_KEY_ID = '...'
AWS_SECRET_ACCESS_KEY = '...'

bucket_name = 'bucket_name'
# connect to the bucket
conn = boto.connect_s3(AWS_ACCESS_KEY_ID,
                AWS_SECRET_ACCESS_KEY)
bucket = conn.get_bucket(bucket_name)
# go through the list of files
bucket_list = bucket.list()
for l in bucket_list:
  keyString = str(l.key)
  # check if file exists locally, if not: download it
  if not os.path.exists(LOCAL_PATH+keyString):
    l.get_contents_to_filename(LOCAL_PATH+keyString)