Upload files from Python to Django

In this post, we are going to upload files using Python to a Django server in an efficient way (streaming style on both sides). We will upload all the files found in 1 folder to our Django server. We will also use authentication to post our data to the server.

Client side

A very nice module to use is the poster module. It facilitates making POST requests using the standard multipart/form-data encoding, as well as enabling streaming POST requests.

We have a cookie in hand which we created using:

cj = cookielib.CookieJar()

We need to import few functions from the poster module:

from poster.encode import multipart_encode
from poster.streaminghttp import StreamingHTTPHandler, StreamingHTTPRedirectHandler, StreamingHTTPSHandler

We build and install our opener using the streaming handler functions of the poster module:

handlers = [StreamingHTTPHandler, StreamingHTTPRedirectHandler, StreamingHTTPSHandler]
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj), *handlers)
urllib2.install_opener(opener)

You could use register_openers() but it doesn’t support authentication so we call the internal functions ourself.

We need to read the list of files in the folder ‘/data’ and post them to the server:

params = {}
dir = u'/data/'
for f in os.listdir(dir):
  params[f] = open(dir+f)
data, headers = multipart_encode(params)
url = 'https://test/upload.py'
req = urllib2.Request(url, data, headers)
result = urllib2.urlopen(req)

Server side

We also don’t want to overwhelm the memory on the server side so we are going to read the data in chunks.

The files received are in request.FILES as a dictionary of UploadedFile objects.

If the file size is above 2.5MB (this can be modified in your settings.py), it will be split in multiple parts so reading doesn’t consume too much RAM.

We add the following in the view function handling upload.py requests:

for key, file in request.FILES.items():
  path = '/data/'+ file.name
  dest = open(path.encode('utf-8'), 'wb+')
  if file.multiple_chunks:
    for c in file.chunks():
      dest.write(c)
    else:
      dest.write(file.read())
    dest.close()

Here it is, we have all our files uploaded saved in the data folder on the server side.

If you enjoyed this article, check out my web app Gourmious and discover and share your favorite restaurant dishes.