This article describes how you can upload files to Amazon S3 using Python/Django and how you can download files from S3 to your local machine using Python.
We assume that we have a file in /var/www/data/ which we received from the user (POST from a form for example).
You need to create a bucket on Amazon S3 to contain your files. This can be done using the Amazon console.
Now, we are going to use the python library boto to facilitate our work.
We define the following variables in settings.py of our Django project:
BUCKET_NAME = 'bucket_name' AWS_ACCESS_KEY_ID = ... AWS_SECRET_ACCESS_KEY = ...
Upload to S3
Here is the code we use to upload the picture files:
def push_picture_to_s3(id): try: import boto from boto.s3.key import Key # set boto lib debug to critical logging.getLogger('boto').setLevel(logging.CRITICAL) bucket_name = settings.BUCKET_NAME # connect to the bucket conn = boto.connect_s3(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY) bucket = conn.get_bucket(bucket_name) # go through each version of the file key = '%s.png' % id fn = '/var/www/data/%s.png' % id # create a key to keep track of our file in the storage k = Key(bucket) k.key = key k.set_contents_from_filename(fn) # we need to make it public so it can be accessed publicly # using a URL like http://s3.amazonaws.com/bucket_name/key k.make_public() # remove the file from the web server os.remove(fn) except: ...
Download from S3
As you saw, you can access the file using the URL: http://s3.amazonaws.com/bucket_name/key but you can also use the boto library to download the files. I do that to create a daily backup of the bucket’s files on my local machine.
Here is the script to do that:
import boto import sys, os from boto.s3.key import Key LOCAL_PATH = '/backup/s3/' AWS_ACCESS_KEY_ID = '...' AWS_SECRET_ACCESS_KEY = '...' bucket_name = 'bucket_name' # connect to the bucket conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) bucket = conn.get_bucket(bucket_name) # go through the list of files bucket_list = bucket.list() for l in bucket_list: keyString = str(l.key) # check if file exists locally, if not: download it if not os.path.exists(LOCAL_PATH+keyString): l.get_contents_to_filename(LOCAL_PATH+keyString)