Princess Polymath

At LinkedIn we enjoy a monthly Hackday, and I usually use the opportunity to figure out new ways to combine our platform with other systems – I love combining different APIs to create new functionality.

This month I decided to use my hackday time to create an SMS notification system for our LinkedIn today system. Since I’m not always on the website, I frequently miss the viral articles being discussed until they’ve passed into my history, and I’d like to be able to jump in on the conversation while it’s hot.  Using the LinkedIn Today API (currently only available internally), the system periodically checks each members feed ‘s see if there are any articles which are currently generating a huge amount of activity .  Once these “hot” articles are found, they are sent via SMS (using Twilio) to the user’s cell phone and recorded in the DB.  The member can reply to the SMS with “save” to add the article to their saved articles at LinkedIn, “share” the article with their LinkedIn network, change the notification level or cancel notifications entirely.

In order to build this system, I realized that I had to get several different systems to work together, and working through this process I realized that I had to solve several common problems.  This series of posts will be a tutorial on how to do the following:

• Integrate Log in with LinkedIn into Django and deploy to Heroku
• Use the Heroku scheduler and python to send SMS messages using Twilio
• Handle SMS responses from Twilio in Django (on Heroku)

The final piece in the application I’m working on requires that I process SMS replies from Twilio.  To do this, I need a public-facing server that can handle a POST request from another server so it can do the right thing on my end.  This tutorial will walk through how to do that with a Django server on Heroku.  You can grab the twilio_sms application from within my GitHub project for this application, or you can build it based on the walkthrough here.

URL Setup

Add a mapping for /reply/ to the urls.py in your top level Django project.

urls.py

url(r'^reply', 'twilio_sms.views.sms_reply'),

You’ll need to tell Twilio where to send SMS replies on the dashboard.  For instance, my Heroku instance runs at http://falling-summer-4605.herokuapp.com so my SMS Response URL is set to http://falling-summer-4605.herokuapp.com/reply.

Creating twilio_sms

In order for that URL mapping to do the right thing it needs to point to the right application.  If you aren’t using the code samples from GitHub, you can create a new application with:

django-admin.py startapp twilio_sms

Now we need to build the views.py file to handle the responses.

The setup is similar to the linkedin application we made previously.

# Python
import oauth2 as oauth
import simplejson as json
import re
from xml.dom.minidom import getDOMImplementation,parse,parseString

# Django
from django.http import HttpResponse
from django.shortcuts import render_to_response
from django.http import HttpResponseRedirect
from django.conf import settings
from django.views.decorators.csrf import csrf_exempt
from django.contrib.auth.models import User

# Project
from linkedin.models import UserProfile,SentArticle

# from settings.py
consumer = oauth.Consumer(settings.LINKEDIN_TOKEN, settings.LINKEDIN_SECRET)
client = oauth.Client(consumer)

We’ll create a convenience method to build SMS responses in the format expected by the Twilio server (so that the user gets a reasonable SMS response)

def createSmsResponse(responsestring):
	impl = getDOMImplementation()
	responsedoc = impl.createDocument(None,"Response",None)
	top_element = responsedoc.documentElement
	sms_element = responsedoc.createElement("Sms")
	top_element.appendChild(sms_element)
	text_node = responsedoc.createTextNode(responsestring)
	sms_element.appendChild(text_node)
	html = responsedoc.toxml(encoding="utf-8")
	return html

Because the POST will be coming from an external server without a session cookie, we need to use the @csrf_exempt decorator to tell Django to allow these POSTs without authentication.  For security, you might check the incoming IP to make sure it’s coming from Twilio, or make sure that the other information matches what you expect.  For this demo, we’ll allow it to proceed assuming it’s the right thing.

Grab the parameters, get the user’s phone number, and determine which of our users matches that phone number, then grab their credentials and create the LinkedIn client to make requests.

@csrf_exempt
def sms_reply(request):
    if request.method == 'POST':
        params = request.POST
        phone = re.sub('\+1','',params['From'])
        smsuser = User.objects.get(userprofile__phone_number=phone)
        responsetext = "This is my reply text"
        token = oauth.Token(smsuser.get_profile().oauth_token, smsuser.get_profile().oauth_secret)
        client = oauth.Client(consumer,token)

Figure out what the user wants us to do (save, search, cancel, help, level)

commandmatch = re.compile(r'(\w+)\b',re.I)
        matches = commandmatch.match(params['Body'])
        command = matches.group(0).lower()

“Cancel” tells the system the user doesn’t want notifications anymore.  For now, we’re going to keep their user and profile around, so that we don’t send them all the same articles again in the future. But sendArticles.py won’t send them anything if the level is set to zero.

        # Cancel notifications by setting score to zero
        # Command is 'cancel'
        if command == 'cancel':
        	profile = smsuser.get_profile()
        	profile.min_score = 0
        	profile.save()
        	return HttpResponse(createSmsResponse("Today SMS Service Cancelled"))

“Level <number>” tells the system the user wants to change their notification level.  Higher means fewer messages, as this is used as the “score” check against articles based on relevance.  See the previous post on Twilio notifications to see how this is implemented.  Notice that in this and the following methods we’re doing generic error catching – there’s a few reasons why it might fail, but the important thing is to tell the user their action didn’t succeed and give them a hint as to why that might be.

        # Change level for notifications by setting score to requested level
        # Command is 'level \d'
        if command == 'level':
        	levelmatch = re.compile(r'level (\d)(.*)',re.I)
        	matches = levelmatch.search(params['Body'])

        	try:
	        	level = int(matches.group(1))
	        except:
	        	e = sys.exc_info()[1]
	        	print "ERROR: %s" % (str(e))
	        	return HttpResponse(createSmsResponse("Please use a valid level (1-9)."))

        	profile = smsuser.get_profile()
        	profile.min_score = level
        	profile.save()
        	return HttpResponse(createSmsResponse("Today SMS minimum score changed to %d" % int(level)))

“Save <article number>” saves an article to the user’s LinkedIn saved articles.  Remember that in the setup we grabbed the credentials for the user who sent the SMS based on their phone number, so this (and share) are done against the LinkedIn API on their behalf.  In this new (preview only) API JSON doesn’t seem to be working well, so I’m building and using XML.

        # Save an article
        # Command is 'save <articlenum>'
        if command == 'save':
        	savematch = re.compile(r'save (\d+)(.*)',re.I)
        	matches = savematch.search(params['Body'])
        	try:
	        	article = matches.group(1)
        		sentarticle = SentArticle.objects.get(user=smsuser, id=article)
	        except:
	        	e = sys.exc_info()[1]
	        	print "ERROR: %s" % (str(e))
	        	return HttpResponse(createSmsResponse("Please use a valid article number with save."))

        	responsetext = "Saved article: %s" % (sentarticle.article_title)
        	saveurl = "http://api.linkedin.com/v1/people/~/articles"

        	# Oddly JSON doesn't seem to work with the article save API
        	# Using XML instead
        	impl = getDOMImplementation()
        	xmlsavedoc = impl.createDocument(None,"article",None)
        	top_element = xmlsavedoc.documentElement
        	article_content_element = xmlsavedoc.createElement("article-content")
        	top_element.appendChild(article_content_element)
        	id_element = xmlsavedoc.createElement("id")
        	article_content_element.appendChild(id_element)
        	text_node = xmlsavedoc.createTextNode(sentarticle.article_number)
        	id_element.appendChild(text_node)
        	body = xmlsavedoc.toxml(encoding="utf-8")

        	resp, content = client.request(saveurl, "POST",body=body,headers={"Content-Type":"text/xml"})
        	if (resp.status == 200):
        		return HttpResponse(createSmsResponse(responsetext))
        	else:
        		return HttpResponse(createSmsResponse("Unable to save post: %s" % content))

“Share <article number> <comment>” shares an article to the user’s network with a comment.  The comment shouldn’t really be optional, but typing on T-9 keyboards is a pain, so I wanted to give a default share message.  I’m not sure I love it as an answer though…

        # Share an article
        # Command is 'share <articlenum> <comment>'
        # If no comment is included, a generic one is sent
        if command == 'share':
        	sharematch = re.compile(r'Share (\d+) (.*)')
        	matches = sharematch.search(params['Body'])
        	try:
        		article = matches.group(1)
	        	sentarticle = SentArticle.objects.get(user=smsuser, id=article)
        		comment = matches.group(2)
	        except:
	        	if sentarticle and not comment:
	        		comment = "Sharing an article from the LinkedIn SMS System"
	        	else:
	        		e = sys.exc_info()[1]
	        		print "ERROR: %s" % (str(e))
	        		return HttpResponse(createSmsResponse("Please use a valid article number with share and include a comment."))

        	responsetext = "Shared article: %s" % (sentarticle.article_title)
        	shareurl = "http://api.linkedin.com/v1/people/~/shares"
        	body = {"comment":comment,
        		"content":{
        			"article-id":sentarticle.article_number
       	 		},
        	"visibility":{"code":"anyone"}
        	}

        	resp, content = client.request(shareurl, "POST",body=json.dumps(body),headers={"Content-Type":"application/json"})
        	if (resp.status == 201):
        		return HttpResponse(createSmsResponse(responsetext))
        	else:
        		return HttpResponse(createSmsResponse("Unable to share post: %s" % content))

If we’ve fallen through to here, the user may have asked for ‘help’ – but whatever they did we didn’t understand it so we should give them the help text in any case.

       # If command is help, or anything we didn't recognize, send help back
        helpstring = "Commands: 'cancel' to cancel Today SMS; 'level #number#' to change minimum score;"
        helpstring += "'save #article#' to save; 'share #article# #comment#' to share"
        return HttpResponse(createSmsResponse(help string))

… and, if the request wasn’t a POST, send a 405 response back to the system (it won’t be Twilio, it might have been someone else).  This URL is only for processing these SMS messages.

    # If it's not a post, return an error
    return HttpResponseNotAllowed('POST')

For my Twilio SMS Demo App this month, I wanted to send notifications to LinkedIn members based on hot items for them in LinkedIn Today.  This is a tutorial on creating a python script to access user information in the Django database, pull information from a Web Service (in this case the LinkedIn Today API) and send an SMS to the user.  A couple of caveats are in order here to make sure that people don’t accidentally run afoul of the LinkedIn API Terms of Use when implementing something of this sort.

• The LinkedIn Today API is only available internally at this point, and should only be used as an example of what’s possible
• I have stored the user’s phone number in the database – when storing LinkedIn information you always need to notify the user that you’re doing that, explicitly.

Install the Twilio Library

% pip install twilio
% pip freeze > requirements.txt

Adding the Phone Number

Remember, if you’re going to store any information at all about a LinkedIn member, you need to explicitly tell them so.  That having been said, you can add the following to your LinkedIn application to get the phone number and store it.  I’m not going to provide the completed code to do this since you need to implement some user notification if you’re using this in a production system.

linkedin/models.py

class UserProfile(models.Model):
    user = models.ForeignKey(User)
    oauth_token = models.CharField(max_length=200)
    oauth_secret = models.CharField(max_length=200)
    phone_number = models.CharField(max_length=10)

linkedin/views.py

Modify the LinkedIn profile URL:

url = "http://api.linkedin.com/v1/people/~:(id,first-name,last-name,industry,phone-numbers)"

Add this logic before saving the user profile:

        if "phoneNumbers" in profile:
        	for phone_number in profile['phoneNumbers']['values']:
        		userprofile.phone_number = re.sub(r"\D","",phone_number['phoneNumber'])
        		if phone_number['phoneType'] == 'mobile':
        			continue

If you’ve already created your DB you’ll need to wipe it out and recreate it with these new models in order for this to work.  You can most easily do this by just starting a new project for this version – while you can drop the database locally and recreate it with syncdb, for heroku you’ll need to create a new cedar project so a new database will be created.

Creating the Job

Now we’ll walk through the code.  This file wants to live at your LinkedIn project level (so in the example, in the hellodjango directory).  It’s also available in the GitHub repository for django-linkedin-simple.

First, do some basic setup:

from django.core.management import setup_environ
import settings
setup_environ(settings)

from twilio.rest import TwilioRestClient
import oauth2 as oauth
import time
import simplejson
import datetime
import httplib2
import psycopg2

account = "TWILIO_ACCT_NUMBER"
token = "TWILIO_TOKEN"
twilioclient = TwilioRestClient(account, token)

consumer = oauth.Consumer(
        key="LINKEDIN_CONSUMER_KEY",
        secret="LINKEDIN_CONSUMER_SECRET")
url = "http://api.linkedin.com/v1/people/~/topics:(description,id,topic-stories:(topic-articles:(relevance-data,article-content:(id,title,resolved-url))))"

Grab the user profiles (this includes the user tokens and secrets)

users = User.objects.all()

Iterate over the users, grabbing their token, secret, django userid and phone number, then grab the LinkedIn Today feed for that user.

for djangouser in users:
	if djangouser.username == "admin":
		continue
	profile = UserProfile.objects.get(user=djangouser)
	token = oauth.Token(
        	key=profile.oauth_token,
        	secret=profile.oauth_secret)
	phone = profile.phone_number
	userid = profile.user_id
	user_articles = SentArticle.objects.filter(user=djangouser)

	# Now make the LinkedIn today call and get the articles in question
	client = oauth.Client(consumer, token)

	resp, content = client.request(url, headers={"x-li-format":'json'})
	results = simplejson.loads(content)

We only want to send alerts for articles which have a high “score”, indicating that they are currently hot topics for this user.  Look through the articles to find those “hot topics”:

	for topic in results['values']:
           for story in topic['topicStories']['values']:
                for article in story['topicArticles']['values']:
                        score = article['relevanceData']['score']
                        if score > 6:

Now, SMS messages are expensive for us (the application owner) and potentially for the user, so we only want to tell the user if they haven’t already heard about this article.

checkarticle = user_articles.filter(article_number__exact=article['articleContent']['id'])

If there aren’t any matching articles for this user, go ahead and send them a message, then log it in the database so we don’t tell them again later:

if len(checkarticle) == 0:
    # This is where we get the shortened URL from google because LinkedIn doesn't provide one
    http = httplib2.Http()
    body = {"longUrl": article['articleContent']['resolvedUrl']}
    resp,content = http.request("https://www.googleapis.com/urlshortener/v1/url?key=YOUR_GOOGLE_API_KEY","POST",body=simplejson.dumps(body),headers={"Content-Type":"application/json"})
    googleresponse = simplejson.loads(content)
    sentarticle = SentArticle(article_number=article['articleContent']['id'],user=djangouser,timestamp=datetime.datetime.today())
    sentarticle.save()
    bodytext = article['articleContent']['title'] + " " + googleresponse['id']
    bodytext += " ('save %s')" % sentarticle.id
    message = twilioclient.sms.messages.create(to="+1" + phone, from_="+YOUR_TWILIO_NUMBER", body=bodytext)

Great, we’ve got an application that’ll send messages for the users in the Django system when something hot is there to talk about.

Pushing to Heroku

Now we need the job to work on Heroku, and get scheduled.  If you went through my previous post using the existing GitHub example, the file is already up at Heroku, otherwise you’ll need to push the file to your Heroku instance:

% git add hellodjango/sendArticles.py
% git push heroku master

Whether it was already there or you just put it there, you should check to make sure that it runs correctly.  Make sure you’ve logged into the system using your LinkedIn ID and stored your phone number there.

% heroku run python hellodjango/sendArticles.py

Did it work?  Great!

Schedule it in Heroku

The Heroku Scheduler is a free add-on, but remember that since it uses machine time it can start racking up the charges if you don’t keep an eye on it.  You can add it using the command line, or you can add it under “Resources” for your application on the Heroku site.  Once you’ve added it you can manage it using the scheduler dashboard.

Pick whatever frequency you like, and put “python hellodjango/sendArticles.py” in the field, and you’re done.

Don’t forget to stop the scheduler when you’re no longer testing so you don’t get a surprise bill from Heroku later.

The next post will cover handling POST requests from Twilio in response to SMS replies from your users.

To build the LinkedIn Today SMS Notification System, I needed to find a framework where I could do the following:

• Integrate LinkedIn OAuth with a more general user management system
• Post the system to a public site
• Handle POST requests from external clients

For this set of criteria, I chose to use Django and publish it using Heroku.  This post is similar to the Django Getting Started Guide for Heroku, but it’s designed for this specific use case – Using LinkedIn OAuth as the login system and integrating it with the Django user management system.  The code for this example is available here.

Here are the prerequisites for this tutorial:

• Python, virtualenv, pip, and the Heroku toolbelt as described in Getting Started with Python on Heroku/Cedar.
• An installed version of Postgres to test locally
• A Postgres user and a database to use for Django (in this case I’m using ‘linkedin’)
• A LinkedIn API Key.  If you haven’t worked with the LinkedIn API before, you’re encouraged to go through the Quick Start Guide.

Create Your Django Environment

% mkdir hellodjango && cd hellodjango
% virtualenv --no-site-packages vent
% source venv/bin/activate
% pip install Django psycopg2 oauth2 simplejson
% django-admin.py startproject hellodjango
% cd hellodjango
% python manage.py startapp linkedin

Now that you’ve done all these pieces, you have a working django system.

File Setup

In order to add the LinkedIn integration, you’ll need to edit the settings and URL files.  I’ve created a GitHub repository with the code needed for this example, and you can use that as your model – but I’ll walk through the changes here so you understand how they work.

hellodjango/settings.py

Edit the DATABASES section to configure it to use postgres, and your database:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'linkedin',
        'USER': 'postgres',
        'PASSWORD': 'xxx',
        'HOST': '',
        'PORT': '5432',
    }
}

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'linkedin'
    # Uncomment the next line to enable the admin:
    # 'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',
)

AUTH_PROFILE_MODULE= 'linkedin.UserProfile'
LINKEDIN_TOKEN='xxx'
LINKEDIN_SECRET='xxx'
LOGIN_URL='/login/'

url(r'^login/?$', 'linkedin.views.oauth_login'),
url(r'^logout/?$', 'linkedin.views.oauth_logout'),
url(r'^login/authenticated/?$', 'linkedin.views.oauth_authenticated'),
url(r'^$','linkedin.views.home'),

from django.db import models
from django.contrib.auth.models import User
class UserProfile(models.Model):
    user = models.ForeignKey(User)
    oauth_token = models.CharField(max_length=200)
    oauth_secret = models.CharField(max_length=200)

# Python
import oauth2 as oauth
import cgi
import simplejson as json
import datetime
import re

# Django
from django.http import HttpResponse
from django.shortcuts import render_to_response
from django.http import HttpResponseRedirect
from django.conf import settings
from django.contrib.auth import authenticate, login, logout
from django.contrib.auth.models import User
from django.contrib.auth.decorators import login_required

# Project
from linkedin.models import UserProfile

# from settings.py
consumer = oauth.Consumer(settings.LINKEDIN_TOKEN, settings.LINKEDIN_SECRET)
client = oauth.Client(consumer)
request_token_url = 'https://api.linkedin.com/uas/oauth/requestToken'
access_token_url = 'https://api.linkedin.com/uas/oauth/accessToken'
authenticate_url = 'https://www.linkedin.com/uas/oauth/authenticate'

# /login
def oauth_login(request):

    # Step 0. Get the current hostname and port for the callback
    if request.META['SERVER_PORT'] == 443:
     current_server = "https://" + request.META['HTTP_HOST']
    else:
     current_server = "http://" + request.META['HTTP_HOST']
     oauth_callback = current_server + "/login/authenticated"

    # Step 1. Get a request token from Provider.
    resp, content = client.request("%s?oauth_callback=%s" % (request_token_url,oauth_callback), "GET")
    if resp['status'] != '200':
        raise Exception("Invalid response from Provider.")
    # Step 2. Store the request token in a session for later use.
    request.session['request_token'] = dict(cgi.parse_qsl(content))
    # Step 3. Redirect the user to the authentication URL.
    url = "%s?oauth_token=%s" % (authenticate_url,
        request.session['request_token']['oauth_token'])
    print url
    return HttpResponseRedirect(url)

# / (requires login)
@login_required
def home(request):

    html = "<html><body>"
    token = oauth.Token(request.user.get_profile().oauth_token,request.user.get_profile().oauth_secret)
    client = oauth.Client(consumer,token)
    headers = {'x-li-format':'json'}
    url = "http://api.linkedin.com/v1/people/~:(id,first-name,last-name,headline)"
    resp, content = client.request(url, "GET", headers=headers)
    profile = json.loads(content)
    html += profile['firstName'] + " " + profile['lastName'] + "<br/>" + profile['headline']
    return HttpResponse(html)

# /logout (requires login)
@login_required
def oauth_logout(request):

    # Log a user out using Django's logout function and redirect them
    # back to the homepage.
    logout(request)
    return HttpResponseRedirect('/')

#/login/authenticated/
def oauth_authenticated(request):
    # Step 1. Use the request token in the session to build a new client.
    token = oauth.Token(request.session['request_token']['oauth_token'],
        request.session['request_token']['oauth_token_secret'])
    if 'oauth_verifier' in request.GET:
        token.set_verifier(request.GET['oauth_verifier'])
    client = oauth.Client(consumer, token)
    # Step 2. Request the authorized access token from Provider.
    resp, content = client.request(access_token_url, "GET")
    if resp['status'] != '200':
        print content
        raise Exception("Invalid response from Provider.")
    access_token = dict(cgi.parse_qsl(content))
    headers = {'x-li-format':'json'}
    url = "http://api.linkedin.com/v1/people/~:(id,first-name,last-name,industry)"
    token = oauth.Token(access_token['oauth_token'],
        access_token['oauth_token_secret'])
    client = oauth.Client(consumer,token)
    resp, content = client.request(url, "GET", headers=headers)
    profile = json.loads(content)    

    # Step 3. Lookup the user or create them if they don't exist.
    firstname = profile['firstName']
    lastname = profile['lastName']
    identifier = profile['id']
    try:
        user = User.objects.get(username=identifier)
    except User.DoesNotExist:
        user = User.objects.create_user(identifier,
            '%s@linkedin.com' % identifier,
            access_token['oauth_token_secret'])                 user.first_name = firstname
        user.last_name = lastname
        user.save()
        # Save our permanent token and secret for later.
        userprofile = UserProfile()
        userprofile.user = user
        userprofile.oauth_token = access_token['oauth_token']
        userprofile.oauth_secret = access_token['oauth_token_secret']
        userprofile.save()
    # Authenticate the user and log them in using Django's pre-built
    # functions for these things.
    user = authenticate(username=identifier,
        password=access_token['oauth_token_secret'])

    login(request, user)
    return HttpResponseRedirect('/')

web: python hellodjango/manage.py runserver "0.0.0.0:$PORT"

venv
*.pyc

% python manage.py syncdb
% python manage.py runserver

% git init
% git add .
% git commit -m "My LinkedIn Django App"
% heroku create --stack cedar
% git push heroku master
% heroku run python hellodjango/manage.py syncdb 
% heroku open

I’m working on a little hacky toy for our upcoming hack day at work, and one of the pieces requires that I get some info from the Amazon “REST” API. It’s not REST in the least, but that’s not what I’m here to talk about. I’m here to talk about creating a signed request which works with the “Amazon Signature 2″ format needed for the API. There’s a Perl library to do everything with the API, which is great if that’s what you’re looking for, but it wasn’t. I just wanted to sign a simple static request to get information about books for which I had the ISBN. No Python library exists, but fortunately if you’re using Python 2.5+ you’ve got everything you need with very little code. I’m including it here to save you the effort of pulling together the pieces yourself.

# Create a signed request to use Amazon's API
import base64
import hmac
import urllib, urlparse
import time

from hashlib import sha256 as sha256

AWS_ACCESS_KEY_ID = 'XXX'
AWS_SECRET_ACCESS_KEY = 'YYY'
hmac = hmac.new(AWS_SECRET_ACCESS_KEY, digestmod=sha256)

def getSignedUrl(params):
    action = 'GET'
    server = "webservices.amazon.com"
    path = "/onca/xml"

    params['Version'] = '2009-11-02'
    params['AWSAccessKeyId'] = AWS_ACCESS_KEY_ID
    params['Service'] = 'AWSECommerceService'
    params['Timestamp'] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())

    # Now sort by keys and make the param string
    key_values = [(urllib.quote(k), urllib.quote(v)) for k,v in params.items()]
    key_values.sort()

    # Combine key value pairs into a string.
    paramstring = '&'.join(['%s=%s' % (k, v) for k, v in key_values])
    urlstring = "http://" + server + path + "?" + \
        ('&'.join(['%s=%s' % (k, v) for k, v in key_values]))

    # Add the method and path (always the same, how RESTy!) and get it ready to sign
    hmac.update(action + "\n" + server + "\n" + path + "\n" + paramstring)

    # Sign it up and make the url string
    urlstring = urlstring + "&Signature="+\
        urllib.quote(base64.encodestring(hmac.digest()).strip())

    return urlstring

if __name__ == "__main__":
    params = {'ResponseGroup':'Small,BrowseNodes,Reviews,EditorialReview,AlternateVersions',
                     'AssociateTag':'xxx-20',
                     'Operation':'ItemLookup',
                     'SearchIndex':'Books', 
                     'IdType':'ISBN',
                     'ItemId':'9780385086950'}
    url = getSignedUrl(params)
    print url

I need to get OpenCV working on my system for a couple of image processing tasks I have for work, and as with many complex software systems, it was somewhat difficult to get it working on my system, a macbook pro running Leopard (OS/X 10.5).

% mkdir build
% cd build
% cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -DBUILD_PYTHON_SUPPORT=ON ../

I was having some fun poking around at the Netflix API, and realized that there wasn’t a Netflix module for python, so I put one together.  First version is up at http://code.google.com/p/pyflix. Has installable code, an example file, and a py.test compatible test file.

For an upcoming project at work, I needed to get a better idea of how the AWS services work together, and wanted to also see how the EC2 instances could be used for parallel processing.  Sadly, I do not love Java, and although I would use it if pressed, I wanted to see if I could find a pythony way to process some data using a hadoop setup.

hadoop fs -put <file> s3://ID:SECRET@BUCKET/name_of_dir

There are, of course, other ways to move things to S3 buckets.  Pick one you like.

bin/hadoop-ec2 run

That’s a lie.  Try this instead:

bin/hadoop-ec2 launch-cluster <group_name> <number_of_slaves>

This will create a master hadoop node, and your slaves. For number_of_slaves you want to pick something <= 19 so that your total doesn’t exceed 20 (unless you have special privileges).

Now we have to move our snazzy mapper and reducer to the master:

bin/hadoop-ec2-env.sh
scp $SSH_OPTS /path/to/mapper.py root@$MASTER_HOST:/home
scp $SSH_OPTS /path/to/reducer.py root@$MASTER_HOST:/home

‘run’ apparently used to then log you into your master, but since we’re using launch-cluster, you’ll need to do it yourself:

ssh $SSH_OPTS root@<your_new_master>

And there you are! On your new master. Awesome. Now let’s move the data to our cluster (ID and SECRET are your AWS credentials, BUCKET is the bucket you created):

cd /usr/local/hadoop-<version>
bin/hadoop fs -mkdir files
bin/hadoop distcp s3://<ID>:<SECRET>@<BUCKET>/path/to/files files

Ok, great. Almost there. Now we need to run the thing:

hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop jar contrib/streaming/hadoop-0.18.0-streaming.jar -mapper mapper.py -file /home/mapper.py -reducer reducer.py -file /home/reducer.py -input files/* -output map-reduce.output

While it’s running, you can check out the neat web report hadoop creates at http://<server_name>:50030.  Go ahead, check it out.  It’s totally cool.

Don’t get me wrong. I do like django. I’ve been working back and forth in Pylons and Django, trying to learn each of them well enough so that I can figure out which one will give the right answer when I know better what the powers that be want.
So, if you, like me, have a debian etch box upon which you want to install django, have it work with the tutorials in the book and on the site (thus needing python2.5), using mod_python so that you can work on a remote server, with postgres, do the following:

• Use the Django from subversion. It lives in http://code.djangoproject.com/svn/django/trunk (don’t forget to uninstall python-django if you’ve already installed that.
• It requires python2.5. The packaged version is fine. apt-get install python2.5
• Get mod_perl as a package, because it will make all the connections correctly (but it will be linked to python2.4), and then
• Install apache2-prefork-dev so that you have the right apxs2 to build mod_python against python2.5
• Download mod_python from http://ftp.wayne.edu/apache/httpd/modpython/mod_python-3.3.1.tgz
• Configure it (with –apxs=/usr/bin/apxs2)
• Install it
• Get psycopg from http://www.initd.org/pub/software/psycopg/PSYCOPG-2-0/psycopg2-2.0.5.1.tar.gz. Don’t get fancy and try the new one. It doesn’t work.
• python setup.py install that sucker
• And then restart everything and all should be lovely in the world

So that’s…

svn co http://code.djangoproject.com/svn/django/trunk django
ln -s `pwd`/django /usr/lib/python2.5/site-packages/
ln -s `pwd`/django/django/bin/django-admin.py /usr/local/bin
apt-get install python2.5 libapache2-mod-python apache2-prefork-dev
wget http://ftp.wayne.edu/apache/httpd/modpython/mod_python-3.3.1.tgz
tar xzf mod_python-3.3.1.tgz
cd mod_python-3.3.1
./configure --apxs=/usr/bin/apxs2
sudo make install
cd ..
wget http://www.initd.org/pub/software/psycopg/PSYCOPG-2-0/psycopg2-2.0.5.1.tar.gz
tar xzf psycopg2-2.0.5.1.tar.gz
cd psycopg2.2.0.5.1
sudo python setup.py install
sudo /etc/init.d/apache2 restart

I spent a good deal of today trying to find these answers. So, you’re welcome