Thursday 21 March 2013

Sorry Google; you can Keep it to yourself — Tech News and Analysis

What he said.

Sorry Google; you can Keep it to yourself — Tech News and Analysis:

'via Blog this'

Friday 1 March 2013

Implementing OpenID authentication with Cherrypy

Originally published: 2010-05-09 on my old blog

Seeing as I've just spent most of the weekend trying to get OpenID authentication working on, I thought I'd share how I did it, and some pitfalls I encountered along the way.
Installation was easy, and the documentation on both OpenID and python-openid is reasonably good, however there seems to be a lack of sample code.  API references are all well and good, but they don't show you *exactly* how it should be done.
So, for anyone attempting the same thing, I'm going to reproduce the relevant sections of my login script for so that others don't have to waste their weekends. 
Firstly, you'll need to install python-openid itself, which seems to be quite well distributed, so on my debian-based system, this was as easy as "apt-get install python-openid".  So, with that installed, and your website already up and running with Python and some sort of web framework (Cherrypy in this case), you'll need to do something like the following:

#! /usr/bin/python
import cherrypy, logging
from openid.consumer.consumer import Consumer, AuthRequest, SuccessResponse
from import MemoryStore
class Login(object):
def __init__(self): = MemoryStore()
self.returnurl = ''
def default(self, *args, **kwargs):
if len(args) == 1:
self.returnurl = args[0]
return self.PrintLoginForm()
def index(self):
return self.PrintLoginForm()
def do(self,openid_url='',returnurl='',loginsubmit=''):
self.returnurl = returnurl
consumer = Consumer(cherrypy.session,
username = self.FormatUserName(openid_url)
cherrypy.log('initialised consumer', context='', severity=logging.DEBUG, traceback=False)
auth = consumer.begin(username)
cherrypy.log('begin called', context='', severity=logging.DEBUG, traceback=False)
newurl = auth.redirectURL('http://*','')
cherrypy.log('redirect called to '+ newurl, context='', severity=logging.DEBUG, traceback=False)
raise cherrypy.HTTPRedirect(newurl)
def verify(self, *args, **kwargs):
cherrypy.log('verifying ' + str(args) + str(kwargs), context='', severity=logging.DEBUG, traceback=False)
consumer = Consumer(cherrypy.session,
completedict = {'openid.mode':'check_authentication'}
for k,v in kwargs.iteritems():
completedict[k] = v
result = consumer.complete(completedict,'' + completedict['janrain_nonce'])
if type(result) is SuccessResponse:
cherrypy.log('complete success' + str(result.signed_fields), context='', severity=logging.DEBUG, traceback=False)
username = self.FormatUserName(result.identity_url)
# Do your site-specific login here
return self.PrintSuccess()
cherrypy.log('complete failure ' + str(result), context='', severity=logging.DEBUG, traceback=False)
return self.PrintFailure(result.message)
except Exception, inst:
cherrypy.log(str(inst), context='', severity=logging.DEBUG, traceback=False)
return self.PrintFailure()
def FormatUserName(self, usrname):
return usrname.strip().replace('http://','').rstrip('/')
def PrintSuccess(self):
return "<a href="/%s">Proceed</a>" % (self.returnurl)
def PrintFailure(self, extrainfo = ''):
return "Failed to log in - auth failed %s <a href="/login/%s">Try Again</a>" % (extrainfo, self.returnurl)
def PrintLoginForm(self):
return """
<p>Please log in using OpenID</p>­
<form action="/login/do" method="post">
<label for="openid_url">Your Open ID: </label><input name="openid_url" id="openid_url" style="background: transparent url(/img/openid-inputicon.gif) no-repeat" type="text">
<a href="">Get an Open ID</a>
<input name="loginsubmit" value="Log In" type="submit">
<input name="returnurl" value="%s" type="hidden">
""" % (self.returnurl)
One thing that tripped me up for a while was the "realm" parameter to auth.RedirectURL, I was simply putting "" whereas the real answer was "http://*", again not something you'll find out just by reading the API docs.  Obviously replace with your own domain name whereever you see it.

I've still got a little way to go (like retrieving nicknames and email addresses from the OpenID provider), but the above will certainly work for authentication.  Good luck (and let me know if you can help with the email retrieval!  Cheers.


Why use Vim?

Originally published: 2010-03-21 on my old blog

Why use Vim?

Some of you may have stumbled upon posts saying something like "real programmers use Vi or Emacs", so let's have a look at why anyone today would want to use a 30 year old editor.

For the purposes of this article, and personal preference I'll concentrate on Vi rather than Emacs - the war between those two factions is another book on its own.


Vi (pronounced vee-eye) was born in 1976, by Bill Joy, who was looking to create a useable multi-line editor for the Unix operating system and its main successor, Vim (Vi Improved) was born in 1992, written by Bram Moolenaar. Vim improves on Vi by adding many new features, especially those to support programmers (such as syntax highlighting), as well as being almost completely backwards-compatible with Vi. Vim is free, and open source software, although users are encouraged to donate money to Uganda by registering/sponsoring Vim - a model known as "CharityWare".

Why does it matter?

If you spend any significant time at a computer dealing with text, and you are a good typist, you will almost certainly benefit from using a more advanced text editor. Many of the people reading this article are likely to be using a computer and editing text around 8 hours a day. In other words even if you only get 10% better at manipulating text, you'll gain nearly 300 hours more spare time every year just by improving your editing skills. Vim is also completely keyboard driven, so you will save time and risk of injury by not constantly swapping between keyboard and mouse for selecting text and navigating the cursor. It's this improvement that I think is worth the steep learning curve of an editor like Vi.

What's wrong with my normal editor?

I'm not going to stand up and say that whatever you're using now is useless and terrible, because it probably isn't (unless you're still using Notepad of course! - in which case at least try NotePad++). Most modern editors also support syntax highlighting and autocompletion for many programming languages, however Vi has matured over many years, and is almost infinitely extendable and customisable. You can also almost guarantee that whatever system you use (or are forced to use), it'll either have Vi or Vim installed, or it's a short download away. One of my main reasons for learning it was wanting a common set of skills I could guarantee i'd be able to use anywhere, on Windows at work, on Linux at home, and over SSH to my hosting server. Vi has also taught me more about regular expressions, which have helped me many times in my professional life.

What does Vim do?

Among many other things:

  • Supported on almost any operating system
  • Syntax highlighting for hundreds of different programming languages
  • Repeat any action using the '.' key
  • Powerful regular expression support
  • Excellent integrated help system
  • Ability to interact with the command line without leaving the editor
  • Multiple cut and paste registers
  • Infinitely extendable and customisable

If you want to know more, take a look at Vim in 6 Kilobytes, Why oh why and the Vi lovers page

So how does it work?

Vim is different to most other editors because it uses a concept called 'Modal editing'. This basically means that keys on the keyboard carry out different functions depending on what mode you are in. The two main modes are 'Insert Mode' (allowing you to enter text like any other editor), and 'Normal Mode'. Normal mode is the default mode which Vim will start in, and it allows the keys on the keyboard to carry out various cursor movements and functions without having to resort to the mouse and to toolbars and dropdown menus. From normal mode you get get into insert mode by pressing the 'i' key, typing text and then pressing Esc to get back to normal mode. This was done for two main reasons, firstly a lot of early keyboards didn't have separate cursor keys (and even if they did you have to take your hands off the keyboard to press them), and secondly the recognition that when editing text, especially source code, you will tend to spend over 80% of the time reading and navigating through the code, and only a short amount of time actually entering new text. I'm not going to type thousands of words teaching you every command there is, as many people (not least of all the integrated help - type :help to get to it) have already done it.

You won't be able to pick up these commands and be productive with Vi after just a few hours, or even master it in a few weeks, but the pay-off is huge once you have mastered the basics. For some examples, see the videos here, here and here, the quickstart / cheat sheets here and a whole book here.


Once you get used to using Vi/Vim you will find it quite difficult and frustrating if you need to edit text without it. Several products have been made to address this, including ViEmu for Visual Studio, and even a Vi Mode (Vimperator) for Firefox. These allow you to interact with other applications using most (but not all) of the niceties you're used to in Vim. The person who wrote ViEmu also offers products to integrate with Microsoft Office and Outlook. You can also use the "It's all text" plugin for firefox to allow you to edit in Vim straight from the web.

Emacs vs Vi

I won't go into it too much, but Vi is typically smaller and faster than Emacs, and is installed by default on more systems. Emacs has more of a "kitchen sink" feel to it, and I feel some of its keystrokes are particularly difficult (for example the long stretch to press Control-Y to paste text). However having said that, Emacs is also very powerful, and if you can't get on with Vim it's definitely worth a try.


These are the reasons why I have chosen to invest time in learning this editor, and hopefully I'll have convinced a few of you to do the same, or at least give it a try. You will only really appreciate the power of Vim when you watch an expert editing with it, see the videos linked above for a good example. Oh, and if you can't yet touch type, you'll be far better off learning that first! Enjoy.

How best to manage passwords

Originally published: 2010-03-17 on my old blog.  Superseded by

How do you keep track of them and remain secure online?

This is an issue that everyone with internet access will have come across, although some may have sat down and thought about it more than others.  I'm betting that you have at least half a dozen passwords to remember.

Single password?

Many people will use a single password for all the sites and accounts that they access.  This makes it easy to remember, but is insecure because if just one of the sites is broken into, your username and password details will potentially be available not just for that site, but for everything that you access online.  Using a separate password for each site is the ideal solution, but few people will be able to remember more than 6 or 7 passwords.  Company requirements for changing passwords every 90 days complicate this even further.

Password strength and length

More and more websites are enforcing stricter rules on how long your password must be and whether it needs to contain any numbers or special characters.  Nearly all sites will mandate passwords that are 6 characters long.  The longer a password is, the more difficult it is to guess or crack.  One of the most common brute-force attacks is called a dictionary attack, literally trying each word in the dictionary in turn hoping to find a match.  Even substituting numbers or symbols for letters will easily be cracked, such as p455w0rd.  Of course the longer and more complex you make your passwords, the more difficult to remember they become.  Another trap that many people will fall into is using personal information as your password, such as a pet's name or favourite singer.  This information can be extracted from you using social engineering, and in many cases, with the likes of facebook around, you've probably already divulged this information online anyway!

'Static' or generated passwords?

If you choose to "manage" and store your passwords properly, there are two options, the first one is having a password protected (or encrypted) list or database of passwords, that way you only have to remember one password (to access the database), which then gives you access to all your passwords.  There are several examples of such software freely available on the Internet, two of the best are Keepass and Password Safe.  The other advantage to these pieces of software is they allow you to store a notes field (such as the memorable information some sites require if you ever forget your password).
The second option is to have a piece of software which generates your password for you, such as PasswordMaker.  This software allows you to use a relatively simple and easy-to-remember master password, it will then use this master password, together with the domain name of the site, and half a dozen other pieces of information (such as which characters are used in the password), as these are kept secret and all the pieces of information are combined (hashed) together, it becomes near impossible to get back to your master password and break into your other accounts.

A big disadvantage of this is the all-your-eggs-in-one-basket problem - if you lose the "master" password, you've lost everything.  To get around this, several of the tools mentioned allow you to export your passwords to a text file, which you can then print out.

Multiple computers 

Now all of these solutions are great if you only ever use a single computer for accessing your accounts, but most people will have two or more, for example, your work computer, your home computer and maybe a laptop too.  You then have the hassle of maintaining seperate databases, or trying to keep one database 'in sync' with one another.

How to get to your password safe in the first place?

Now many of you may have spotted a problem with all of this...  how do you log onto your computer if you don't remember the password (because it's in your password safe and you're not logged in yet!)?

This is a question I don't really have a satisfactory answer to.  One of the solutions would be to write your passwords down on paper, but we've all at one-time-or-another been told this is a bad idea.

Writing your passwords down

Why is it perceived to be such a bad idea to write down your passwords?  Actually, having your passwords written down isn't such a terrible idea, if they are locked away in your house.  A common or garden burglar is likely not interested in your passwords, even if he knew where to find them, and the number of people with access to your house is vastly less than the number of people who can get to an internet-connected computer.  As long as they are kept safe, this method is an ideal back-up to the electronic solutions.


Some companies and websites are starting to introduce measures to increase the security of their customers, such as two-factor authentication, single sign on, and one-time passwords.  One-time passwords are self explanatory, two-factor authentication increases security by (typically) using something you have (such as a keyfob which generates passwords) as well as something you know (your password).  Single sign-on is perhaps the best idea, but it requires every company or website to sign up to the same idea/technology.  One of the most promising is  Open ID.  

To conclude, there's no right answer to this problem, but there are several pitfalls and wrong answers.  Staying safe online requires a little bit of effort, but with passwords giving access to such important information as your online banking, it's worth the effort.  If you've got a solution you use, please share it by adding a comment.

Save money on your Google Adwords campaigns

Originally published: 2010-04-17 on my old blog

It's just occurred to me that if you've got a sufficiently well-known website (think Amazon, Tesco, etc) that there's no point in you bidding for your own name as a keyword in your Adwords campaign. If your website already occupies the top few listings in google's organic (unpaid) search results, then big companies could save a huge amount by not having a sponsored link as well.

I've noticed that a large proportion of 'ordinary' web users will click on a sponsored (paid) link without  realising, and an increasing number of people use Google / Yahoo / [insert favourite engine here] as their address bar (i.e. rather than typing Facebook into your browser's address bar (and letting the browser
sort out the http://www and .com bits automatically, they'll bring up a search engine instead (sometimes with comical results)

So if you run an Adwords campaign (and you rank well in organic search) consider not bidding for your
own brandname as a keyword on it's own, as it'll be cheaper for you if people just click the first organic result instead!

Beware of 0870

Originally published: 2010-05-01 on my old blog

I was shocked last month to get a mobile phone bill that was over £8. Yes, that may not sound much, but bear in mind that I have a calling plan with unlimited calls, texts and internet. So, after looking through the itemisation, it turns out that I spent £8 on a couple of calls to 0870 numbers.

Now, for a bit of background, there seem to be three types of "non-geographic" numbers:

  1. 0800 numbers, which are free from landlines but not from mobiles (more on that later),
  2. 0845 numbers, which are supposed to be charged the same as a national-rate call,
  3. and 0870 numbers which are just plain expensive, whether you use a landline or not.
None of these numbers are usually (there are a few exceptions) included in any free minutes or calling plans, whether you're using a mobile or a landline. That statement is my main bugbear and the reason for this post - why aren't they included in your free minutes?
Let's start with 0800 numbers, I have a vested interest in these numbers, since my wife runs one for her company (The Pass Lane). These numbers work by the recipient of the call paying for the cost of the call rather than the person making the call. These numbers should make it free for your customers to contact you, however with more and more people using mobiles, this isn't the case. The reason 0800 calls aren't free from mobiles is because the company doesn't necessarily want to pay the higher cost for you to call from a mobile. 0845 numbers are just supposed to be an easy to remember niceity, and 0870 numbers are expensive presumably because the company doesn't want you to phone them.
The mobile phone companies will allow you hundreds of free minutes, yet these 08xx numbers aren't included. You're likely to be paying around 40 pence for every minute you're on the phone to these numbers, and with most of them being call centres, a lot of that time will be spent on hold. There seems to be a slow movement towards 0300 numbers instead, which are included in your free minutes, but many companies have already invested in 08xx numbers and don't want to change all their literature, etc.
For 0800 numbers, if the company you're calling is prepared to pay for the cost of your call from a landline, and your mobile company is prepared to offer you free minutes to a landline, then surely that reduces the cost down to zero, and you (the person calling from a mobile) shouldn't have to pay anything??
One solution (to a problem that shouldn't exist), is Say No To, which allows you to search for a company or number, and tries to return a geographic number for them, which is then much cheaper, or free to call. This is well worth doing, as a typical 10 minute phone call could cost you £4.
So, my questions to no-one in particular (ok, maybe Ofcom) are:
  1. Why bother with 0300 numbers at all? We could just change 0800 or 0845 numbers to do the same
  2. Why aren't 08xx numbers included in free minutes?
  3. Why are 0870 numbers so massively expensive? Who's pocketing the cash?

Energy Saving Tip #1

Originally published: 2010-11-10 on my old blog

At our house I'm always keen to keep the energy bills (and by extension our carbon footprint)
as low as possible, so my first tip would be not to heat your water any more than you need to.

How many places have you been to where the hot tap is scolding hot, and you can't wash your
hands under it without burning yourself? Or when you need gallons of cold water in your bath
just to make it a comfortable temperature. You can alter this quite easily in most homes by
changing the value on the thermostat wrapped around the hot water cylinder. Mine's pictured below
and I've got it set to about 48 degrees C. This means that I can have a nice warm bath without needing
to add any cold water whatsoever, and only heats the water as much as anyone in the house needs.

I you haven't got a hot water cylinder in your house you may be able to achieve the same effect by
tweaking the "power" knob on your boiler.

Why you shouldn't use 3 AntiVirus programs at once

Originally published: 2010-04-19 on my old blog

I feel the need to follow up on a piece of playground gossip I overheard this afternoon. I overheard two people discussing how slowly their Microsoft Windows PC was running, despite having 3 different AntiVirus programs running.

In the vain hope that someone will read this advice, I'll explain why having 3 different AntiVirus programs running will only make things worse, not better.

Without going into too much detail, an AntiVirus program will scan files on your hard disk when they are accessed. Therefore one AntiVirus program will make your computer very slightly slower, but this downside
is greatly outweighed by the benefits that an (up-to-date) AntiVirus application will bring, protecting you from all sorts of nasty viruses and worms.

The problem comes when people install multiple AntiVirus programs to try to fix a slow or infected machine, in the belief that "more is better". So, as we know, AntiVirus programs scan our disks when files change. However two or more A/V programs begin to step on each others toes, the programs start to scan the files not only that you've accessed, but those that the other A/V programs have accessed, thus setting off a chain of events which will massively slow down your machine.

So, stick with a single AntiVirus program you trust, and most importantly, keep it up to date by downloading
new virus definitions on a regular basis. You might want a second program to scan for spyware or malware, but you'll only need one anti virus program.

Good, Free, AntiVirus programs for Windows include Avast and AVG. Don't click on, or install anything
that pops up on your screen unsolicited and promises to fix your machine for you!

Camera Cheat sheet - part 1 - The basics

Originally published: 2010-05-14 on my old blog

I've put this brief intro together in the hope that it will explain the basics
of some of the jargon you'll encounter around digital cameras, and how to make the
most of your photos. It's part one in a series of posts, the size of which I haven't decided yet.

Aperture or f/number

Relates to the size of the circular hole through which light enters the camera. A wider aperture means more light will enter the camera.
A higher f/number gives a smaller aperture (less light), a lower f/number gives a wider aperture (more light).
Aperture controls depth of field, which is the amount of an image which is in focus. A wider aperture (smaller number) gives a smaller depth of field, meaning
less of the area in front of and behind the object you have focussed on will be in sharp focus. Use wider apertures for portraits (such as f/11), and narrower ones (such as f/2.8) for landscapes.

Larger aperture lenses tend to be more expensive as they use higher-quality and more glass elements in their construction.

Shutter Speed

Controls how many milliseconds the shutter remains open, capturing light for your image. The longer the shutter is open, the more chance there is that you will either shake the camera,
or the subject you are photographing moves, blurring the image.
You will usually need a tripod for any exposure longer than 1/60 of a second. For fast-moving objects you will need faster
shutter speeds to capture them without blurring the image. (1/200 or faster). Cameras with vibration reduction or steady-shot technology will allow longer exposures without blurring the image through
camera shake, but will not affect motion-blur from fast-moving objects.

White balance and colour temperature

Often left on auto. A higher "K" (Kelvin) value, the higher the colour temperature, and the warmer (redder)
an image will look, a lower temperature results in a cooler (bluer) image.

For example you may have to change this to reduce the orange tinge you often get under tungsten lighting, by setting a lower colour temperature in post-processing, or selecting the correct temperature preset in the camera.

Focal length

Our eyes' natural focus length is around 50mm, anything lower than this is called wide-angle (17 - 50mm), anything higher than this (50mm+) is called telephoto. Wide angle lenses allow you to see a larger
field of view (more objects), but those objects will look smaller. Telephoto lenses will show you a smaller field of view, but the objects in that field will look larger (more "zoomed in").
Typically telephoto
lenses will have a higher f/number (thus let in less light) than wide-angle lenses.

Exposure Compensation

In certain circumstances, you may find that the exposure that your camera has picked for an image is too dark or too bright, in this case you can re-take the image and adjust the exposure compensation to correct it.
For example by
default a camera will underexpose snow, making it look grey, rather than white.


This number relates to the sensitivity of the camera to light. The default is normally 100 ISO. The higher the ISO the more sensitive to light, but the more noise (speckles) you will get on the image.

In digital cameras it will simply amplify the signal. Useful in low light, or at night, to increase the shutter speed you can use and reduce blur, but at the cost of slightly increased noise.


An image saved as RAW saves the un-processed information straight from the camera's sensor. It means that you will have to process the image before you can see it on screen or share it with friends. However it also means that the camera
hasn't decided for you what the image should look like, and things like the colour temperature can be set correctly.
It also allows you to make bigger changes to the brightness and colour of the image than you would by re-processing a JPEG
image that's already been saved by the camera.


Full Auto (AUTO on most camera dials) - Camera controls everything for you, including ISO and Flash

Program (P) - Camera controls Aperture and Shutter speed to give correct exposure, may or may not control ISO

Aperture Priority (A or Av) - Camera determines shutter speed to use, based on the aperture you have chosen to give a correct exposure

Shutter Priority (S or Tv) - Camera determines aperture to use, based on the shutter speed requested to give correct exposure

Full Manual (M) - All settings can be changed simultaneously, up to you to work out correct exposure (maybe using a light meter).

Try to use your camera in Aperture priority, Shutter Priority, and Program modes, rather than leaving it on Auto, to learn more about how these affect your image.