Adding google authorship to a web site allows google searchers to see a headshot of the author. This is a great way to make your content stand out from the rest and studies have shown these 'Rich Snippets' can improve your click through ratio. This article gives a step-by-step guide for setting it up on your site regardless of if your email address is on the same domain as your blog / site content.
If you're not on it already, you'll need a Google+ account to use the authorship tag, you don't have to be an 'active' user but you do require an account. Once you've created this make sure you verify the email address.
Next, create a link on your website to your Google+ profile as below, replacing the [profile_url] with your Google+ url (you can obtain this by just going to your profile and looking at the url, it should look like this: https://plus.google.com/123456789 , if there's extra string variables just re-create that url with your number). Ideally you want to be placing this around your name in the 'by Joe Bloggs' part of your posts.
<a href="[profile_url]?rel=author">Joe Bloggs</a>
Make sure you leave the ?rel=author as this is key for Google to assoicate the content with your account.
Next you need to create a reciprocal link from your profile back to your site. To do this visit the Contributor To Section and add a custom link.
Once you've completed the above check out your url on Google's structured data testing tool. And now, we wait...
Good question. It seems people have had mixed results on this from days to months (I've just set this up, so I'll let you know how this pans out for me! UPDATE: Photo started showing on new posts after 4 days) but there are a few things you can do to give you the best chance of getting shown as soon as possible:
Run your site through a validator to check the markup is correct, Google can be picky about this!
On your blog posts make sure the 'by Joe Bloggs' text matches the name on your Google+ account exactly.
Interestingly, clever Google does a check against your mugshot to make sure it's suitable. The following are a no-go:
...you get the idea. Choose a simple face-on picture of yourself that uses up most of the image and you're not wearing or doing something stupid (if you're looking on facebook it may take a while).