Upload to Django with progress bar using Ajax and jQuery

September 30, 2010

In this article, I am going to describe how I implemented upload to Django + progress bar with Ajax and jQuery. I needed this feature so users could post their dish pictures on Gourmious and follow the upload’s progress.

Client Side

We need a form so the user can select a file to upload.

<form id="form_upload" action="/upload" method="POST">
  <input type="file" name="picture" id="picture" />
  <input type="hidden" id="X-Progress-ID" name="X-Progress-ID" value=""/>
  <input type="hidden" id="id" name="id" value=""/>
  <input id="form_submit_button" class="tp-button" type="submit" value="Submit" />
  </form>

We added 2 hidden inputs, first one is ‘X-Progress-ID’ which is the upload ID so we can support simultaneous uploads on the server side. We will see later how that value is handled by the server.

We also have the hidden input ‘id’ representing the dish ID in our case.

We want to use Ajax to send the POST request so it can be integrated nicely in a modern web interface along with a progress bar. To support that, we are going to use the jQuery Form plugin.

The function ajaxSubmit() is going to take care of everything for us.

We generate a random string for this upload ID and set the input value to that string.
We need to specify a URL to be called for the upload and 2 callback functions: one to be called before the request and 1 after.

$('#X-Progress-ID').val('random string');
var options = {
  dataType: 'xml',
  url: '/upload?X-Progress-ID='+$('#X-Progress-ID').val(),
  beforeSubmit: showRequest,
  success: showResponse
}
$('#form_upload').ajaxSubmit(options);

The showRequest callback can be as simple as:

function showRequest(formData, jqForm, options) {
    // do something with formData
    return True;
}

In the showResponse function, you need to process the response and act on it. In my case, I process some xml returned by the server with a status value.

function showResponse(response) {
    // do something with response
}

When the user presses submit, we want to display a progress bar so we use the following JS code to add a progress bar to the form. The progressBar() method is part of the jQuery progress bar plugin.

$('#form_upload').find('#form_submit_input').append('&lt;span id="uploadprogressbar"&gt;&lt;/span&lt;');
$('#form_upload').find('#uploadprogressbar').progressBar();

Now, we need to add a function running every few seconds to get the upload progress from the server and update the progress bar accordingly.

To do that, we use setInterval() to issue a GET request to the server to get the progress value using the JSON format. We pass the upload ID to the server. When the value null is returned, we know that the upload is finished.

function startProgressBarUpdate(upload_id) {
  $("#uploadprogressbar").fadeIn();
  if(g_progress_intv != 0)
    clearInterval(g_progress_intv);
  g_progress_intv = setInterval(function() {
    $.getJSON("/get_upload_progress?X-Progress-ID="
+ upload_id, function(data) {
      if (data == null) {
        $("#uploadprogressbar").progressBar(100);
        clearInterval(g_progress_intv);
        g_progress_intv = 0;
        return;
      }
      var percentage = Math.floor(100 * parseInt(data.uploaded) / parseInt(data.length));
      $("#uploadprogressbar").progressBar(percentage);
    });
  }, 5000);
}

Server side

First, we need a function in views.py to handle the upload. This function handles the request: “/upload?X-Progress-ID=xxxx”. We are reading the file chunk by chunk to not use too much RAM. In my case, I return some xml containing the status value: upload OK or not.

def upload(request):
  id = request.POST['id']
  path = '/var/www/pictures/%s' % id
  f = request.FILES['picture']
  destination = open(path, 'wb+')
  for chunk in f.chunks():
    destination.write(chunk)
  destination.close()
  # return status to client
  ...

How do we keep track of the upload progress? We need to use a different file upload handler. We are going to use UploadProgressCachedHandler. We just need the class from this snippet, not the view function which we are going to write ourself. You can add the class to a file named uploadprogresscachedhandler in your project.

This handler saves the upload progress to the cache so it can be retrieved easily when we receive the requests from the client.

To enable this handler, we need to add the following to settings.py:

from django.conf import global_settings
FILE_UPLOAD_HANDLERS = ('uploadprogresscachedhandler.UploadProgressCachedHandler', ) \
+ global_settings.FILE_UPLOAD_HANDLERS

We also need to enable the cache system. We are going to use memcached. This goes inside settings.py too.

CACHE_BACKEND = 'memcached://127.0.0.1:11211/'

You need to make sure memcached and the python bindings are installed on your server.

We need to add a function in views.py to return the upload progress asked by the client every few seconds during the upload. This function handles the request “/get_upload_progress?X-Progress-ID=xxxx”. The progress value is stored using the key “remoteaddress_uploadid”.

from django.utils import simplejson
from django.core.cache import cache
def get_upload_progress(request):
  cache_key = "%s_%s" % (request.META['REMOTE_ADDR'], request.GET['X-Progress-ID'])
  data = cache.get(cache_key)
  return HttpResponse(simplejson.dumps(data))

That’s it for now. Don’t hesitate to add comments to discuss more.

If you like this article, it would be cool if you can share your favorite restaurant dishes on Gourmious.

Facebook oauth and Graph API with Django

August 23, 2010

In this post, I am going to describe how I integrated Facebook into Gourmious, so users could post their favorite dishes on Gourmious and on Facebook at the same time.

I wanted to keep the Gourmious login so the users could decide to login using their Gourmious credentials or using the Facebook login feature. I also wanted all users to have a Gourmious account so I didn’t allow users to login using their Facebook credentials if they didn’t have a Gourmious account. I decided to support the following 3 scenarios :

– user has a Django account and he logs in with Facebook. I associate his Facebook account to his Django account. This needs to be done only once.
– user does not have a Django account and tries to login using Facebook. I ask him first to create a Django account and I associate both accounts.
– user logs in using his Django credentials.

Facebook oauth is easier than the old Facebook connect. I was so happy to migrate to this new scheme.

I assume that you already have a server running your Django application and a Facebook app.

We are going to use the Django app django_facebook_oauth to make our life easier. Make sure you have python simplejson package installed.

Facebook Platform uses the OAuth 2.0 protocol for authentication and authorization. Read this first before continuing.
Facebook authentication

Clone the Django Facebook oauth app source code:

git clone http://github.com/dickeytk/django_facebook_oauth.git

Copy the folder django_facebook_oauth to your Django project apps folder and rename it to facebook.

We will assume that your apps folder is called apps.

Django settings.py

We need to make the following changes:

  • Add ‘apps.facebook.backend.FacebookBackend’ to AUTHENTICATION_BACKENDS.
  • Add ‘apps.facebook’ to INSTALLED_APPS
  • Add your Facebook API app ID: API_ID = xxxx.
  • Add your Facebook secret key: APP_SECRET = xxxx.

Django urls.py

Add the following to urlpatterns:

(r'', include('apps.facebook.urls')),

Database changes

You need to add the following table so you can associate Django user IDs with Facebook user IDs and store the Facebook session token. You can use the syncdb feature or create the table manually.

CREATE TABLE `facebook_facebookuser` (
  `id` int(11) NOT NULL auto_increment,
  `user_id` int(11) NOT NULL,
  `facebook_id` varchar(150) NOT NULL,
  `access_token` varchar(150) default NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `user_id` (`user_id`),
  UNIQUE KEY `facebook_id` (`facebook_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8

Facebook authentication flow

1- user clicks on the Facebook login button
2- Django Facebook app authenticate view is called
3- facebook open graph authorize url is called with application ID, redirect uri = /authenticate, scope = ‘publish_stream’ to ask for permission to post to user’s wall
4- Dajgno Facebook app authenticate view is called with a parameter called code.
5- authentication using the Django Facebook app backend by passing the code back to Facebook along with the app secret.
6- Facebook returns a session token to be used for actions like posting messages to the user’s wall.
7- if the user is already in facebook_facebookuser table, login and redirect to home page. If user is not in the table, ask him to login using your app credentials so you can associate his Django app user id with his Facebook user id.

Django Facebook app views.py

I modified views.py to use my login form to associate Django user id with Facebook ID instead of using the register form included in the Django Facebook app.

When the token is returned and the user does not exist in the facebook table, I add a parameter to the request session to indicate that the user needs to login first so I can join the accounts.

if user != None:
  login(request, user)
  return HttpResponseRedirect('/')
else:
  # lines added
  request.session['ask_to_login_facebook'] = '1'
  return HttpResponseRedirect('/')
  #return HttpResponseRedirect('/register/')

I check for this session parameter in my template to popup a login form.

When the user enters his Django app credentials, I add an entry to the Facebook table before logging in the user. This operation just needs to be done once. After that, I know how to match a user Facebook id to the Django app user id so the user can directly login using the Facebook connect button.

This the code in my Django app views.py

# check credentials
user = authenticate(username=username, password=password)
if user is not None:
  if user.is_active:
      if 'fb_id' in request.session:
        fb_user = FacebookUser(user = user,
          facebook_id = request.session['fb_id'],
          access_token = request.session['fb_token'])
        fb_user.save()
        del request.session['fb_id']
        del request.session['fb_token']
      login(request, user)
      status = 0
    else:
      status = 1
  else:
    status = 1
    msg = _('Invalid password')

Django Facebook app backend.py

I modified this file to also add the token to the request session so I can use it when I add an entry to the facebook user table in views.py.

try:
  fb_user = FacebookUser.objects.get(facebook_id = str(profile["id"]))
except FacebookUser.DoesNotExist:
  request.session['fb_id'] = profile['id']
  # line added
  request.session['fb_token'] = access_token
  return None
fb_user.access_token = access_token
fb_user.save()

Django app template

This is what I added to my Django app login form to support the Facebook login button.

<form id="form_authenticate" action="/authenticate" method="POST">
  <p>blablabla</a></p>
  <input id="form_authenticate_button" type="image" src="link_to_facebook_button"                    onClick="javascript:$('#form_authenticate').submit();">
</form>

I also added a ‘Post to Facebook’ check box to my ‘Post dish review’ form so the user can decide what gets posted to his Facebook wall.

Post a message on the user’s wall

I used to post messages using the Javascript SDK but I now do it on the server side using the Facebook python SDK.

Add the file facebook.py (http://github.com/facebook/python-sdk/blob/master/src/facebook.py) to the Facebook app folder and rename it facebook_sdk.py .

We call put_wall_post() to post a message to the user’s wall.

Here is the code I am using in my Django app views.py to format the parameters before calling put_wall_post() .

def post_to_facebook(request):
  try:
    fb_user = FacebookUser.objects.get(user = request.user)
    # GraphAPI is the main class from facebook_sdp.py
    graph = GraphAPI(fb_user.access_token)
    attachment = {}
    message = 'test message'
    caption = 'test caption'
    attachment['caption'] = caption
    attachment['name'] = 'test name'
    attachment['link'] = 'link_to_picture'
    attachment['description'] = 'test description'
    graph.put_wall_post(message, attachment)
    return 0
  except:
    logging.debug('Facebook post failed')

That’s it for now.

If you enjoyed this article, check out my web app Gourmious to discover and share your favorite restaurant dishes. It would be cool if you could add some of your favorite restaurant dishes.

Upload files from Python to Django

January 24, 2010

In this post, we are going to upload files using Python to a Django server in an efficient way (streaming style on both sides). We will upload all the files found in 1 folder to our Django server. We will also use authentication to post our data to the server.

Client side

A very nice module to use is the poster module. It facilitates making POST requests using the standard multipart/form-data encoding, as well as enabling streaming POST requests.

We have a cookie in hand which we created using:

cj = cookielib.CookieJar()

We need to import few functions from the poster module:

from poster.encode import multipart_encode
from poster.streaminghttp import StreamingHTTPHandler, StreamingHTTPRedirectHandler, StreamingHTTPSHandler

We build and install our opener using the streaming handler functions of the poster module:

handlers = [StreamingHTTPHandler, StreamingHTTPRedirectHandler, StreamingHTTPSHandler]
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj), *handlers)
urllib2.install_opener(opener)

You could use register_openers() but it doesn’t support authentication so we call the internal functions ourself.

We need to read the list of files in the folder ‘/data’ and post them to the server:

params = {}
dir = u'/data/'
for f in os.listdir(dir):
  params[f] = open(dir+f)
data, headers = multipart_encode(params)
url = 'https://test/upload.py'
req = urllib2.Request(url, data, headers)
result = urllib2.urlopen(req)

Server side

We also don’t want to overwhelm the memory on the server side so we are going to read the data in chunks.

The files received are in request.FILES as a dictionary of UploadedFile objects.

If the file size is above 2.5MB (this can be modified in your settings.py), it will be split in multiple parts so reading doesn’t consume too much RAM.

We add the following in the view function handling upload.py requests:

for key, file in request.FILES.items():
  path = '/data/'+ file.name
  dest = open(path.encode('utf-8'), 'wb+')
  if file.multiple_chunks:
    for c in file.chunks():
      dest.write(c)
    else:
      dest.write(file.read())
    dest.close()

Here it is, we have all our files uploaded saved in the data folder on the server side.

If you enjoyed this article, check out my web app Gourmious and discover and share your favorite restaurant dishes.

Create a Debian package for your Django application

January 2, 2010

This post is about building a Debian package to deploy a Django application on a Debian server.

The package will take care of the following:

  • Install application
  • Web server configuration (in our case: Lighttpd with fastcgi)
  • Init.d script to start/stop Django server

We will also use a debconf template to ask 1 question to the user to make our package more dynamic.

Let’s assume you have a Django application located in /usr/share/djangoapp . This is where you have settings.py, manage.py, views.py etc…

First thing, we need to create a folder debian and copy the Django app folder into it:

mkdir ./debian
mkdir -p ./debian/usr/share/
cp -r /usr/share/djangoapp ./debian/usr/share/

We also need to create a folder names DEBIAN in debian/

mkdir ./debian/DEBIAN

DEBIAN files

This folder needs to contain the following files:

  • control
  • config
  • conffiles
  • templates
  • postinst
  • postrm
  • preinst
  • prerm

control

This is the most important file in a Debian package. It gives information on the package itself along with some dependencies. In our case, it looks like this:

Package: djangoapp
Version: 0.1-1
Section: devel
Priority: optional
Architecture: all
Depends: python, python-django, python-flup, debconf, lighttpd
Maintainer: Laurent Luce <laurent@tomnica.com>
Description: short description
 long description

Depends consists of a list of packages required for our package to run properly.

config

config is a script responsible for asking questions to the user before the package gets installed. In our case, we will ask one simple question: what is your full name?

#!/bin/sh -e

# Source debconf library.
. /usr/share/debconf/confmodule

# server type?
db_input critical djangoapp/username || true
db_go

As you can see, you need to include debconf confmodule to add configuration support. Our question is critical.

Wait a minute, where is the question definition? It is defined in the templates file

templates

This is where we define our question: string type.

Template: djangoapp/username
Type: string
Description: Username:
This package requires your full name.

conffiles
This file contains the configuration files generally located under /etc . We add our Django start script and the lighttpd configuration file.

/etc/init.d/django
/etc/lighttpd/lighttpd.conf

preinst
This script is run before the package gets installed.

#!/bin/bash
set -e

# stop django server
if [ -f /etc/init.d/django ]
then
  invoke-rc.d django stop
fi

postinst
This script is run after the package is unpacked.

#!/bin/sh
set -e

# Source debconf library.
. /usr/share/debconf/confmodule

db_get djangoapp/username
username="$RET"
# do what you want with this username

# register django
update-rc.d django defaults 90 >/dev/null
# start django
invoke-rc.d django start

db_stop

postrm
This script is run after the package is removed.

#!/bin/bash
set -e

if [ "$1" = "purge" -a -e /usr/share/debconf/confmodule ]; then
    # Source debconf library.
    . /usr/share/debconf/confmodule
    # Remove my changes to the db.
    db_purge
fi

if [ "$1" = "remove" ]; then

# Source debconf library.
. /usr/share/debconf/confmodule

# remove Django start script
update-rc.d -f django remove

# Remove my changes to the db.
db_purge
fi

prerm
This script is run before the package is removed.

#!/bin/bash
set -e

# stop django server
invoke-rc.d django stop

Web server configuration

Here is what needs to be modified in the web server configuration (in our case, lighttpd.conf):

mod_fastcgi needs to be added to the server modules.

The Django server will be running on port 3033 so we need to specify the connection here:

fastcgi.server = (
  "/djangoapp.fcgi" => (
    "main" => (
      # Use host / port instead of socket for TCP fastcgi
      "host" => "127.0.0.1",
      "port" => 3033,
      "check-local" => "disable",
  ))
)

Django init.d script

Now is time to add our Django start/stop script:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          FastCGI servers for Django
# Required-Start:    networking
# Required-Stop:     networking
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Django FastCGI
#

DJANGO_SITE="djangoapp"
SITE_PATH=/var/www
RUNFILES_PATH=$SITES_PATH/tmp
HOST=127.0.0.1
PORT_START=3033
RUN_AS=www-data

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="Django FastCGI"
NAME=$0
SCRIPTNAME=/etc/init.d/$NAME

#
#       Function that starts the daemon/service.
#
d_start()
{
    # Starting all Django FastCGI processes
    echo -n ", $DJANGO_SITE"
    if [ -f $RUNFILES_PATH/$DJANGO_SITE.pid ]; then
        echo -n " already running"
    else
       start-stop-daemon --start --quiet \
                       --pidfile $RUNFILES_PATH/$DJANGO_SITE.pid \
                       --chuid $RUN_AS --exec /usr/bin/env -- python \
                       $SITE_PATH/$DJANGO_SITE/manage.py runfcgi \
                       host=$HOST port=$PORT \
                       pidfile=$RUNFILES_PATH/$DJANGO_SITE.pid
    fi
}

#
#       Function that stops the daemon/service.
#
d_stop() {
    # Killing all Django FastCGI processes running
    echo -n ", $DJANGO_SITE"
    start-stop-daemon --stop --quiet --pidfile $RUNFILES_PATH/$SITE.pid \
                          || echo -n " not running"
    if [ -f $RUNFILES_PATH/$DJANGO_SITE.pid ]; then
        rm $RUNFILES_PATH/$DJANGO_SITE.pid
    fi
}

ACTION="$1"
case "$ACTION" in
    start)
        echo -n "Starting $DESC: $NAME"
        d_start
        echo "."
        ;;

    stop)
        echo -n "Stopping $DESC: $NAME"
        d_stop
        echo "."
        ;;

    restart|force-reload)
        echo -n "Restarting $DESC: $NAME"
        d_stop
        sleep 1
        d_start
        echo "."
        ;;

    *)
        echo "Usage: $NAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac

exit 0

This script needs to be placed in ./debian/etc/init.d/

You will also need to create a link from /var/www/djangoapp to /usr/share/djangoapp

changelog, changelog.Debian and copyright

Those are required for Debian packages:

changelog

djangoapp (0.1-1) unstable; urgency=low

  * Initial release.
    + Initial package

 -- Laurent Luce <laurent@tomnica.com>  Fri, 01 Jan 2010 00:00:00 +0000

In our case, changelog.Debian is identical to changelog.

copyright

DjangoApp

Copyright: bla bla

2010-01-01

The home page of xxx is at: 
http://www.xxx.com

Copyright xxx 2010

We need to place those files properly:

mkdir -p ./debian/usr/share/doc/djangoapp
cp changelog changelog.Debian copyright ./debian/usr/share/doc/djangoapp/
gzip --best ./debian/usr/share/doc/djangoapp/changelog
gzip --best ./debian/usr/share/doc/djangoapp/changelog.Debian

Build package

We need to fix the permission of our package files:

find ./debian -type d | xargs chmod 755

We build the package and verify it:

fakeroot dpkg-deb --build debian
mv debian.deb DjangoApp_0.1-1_all.deb
lintian DjangoApp_0.1-1_all.deb

Our package is now ready to be deployed.

If you enjoyed this article, check out my web app Gourmious and discover and share your favorite restaurant dishes.

 
Powered by Wordpress and MySQL. Theme by Shlomi Noach, openark.org