For some time I’ve been wanting to put together some easy to use git auto deployment scripts that I can use with GitHub and BitBucket. Despite hearing anecdotes to the contrary, I kept thinking to myself “It can’t be that hard to auto deploy a git repo from these services”. Now I’ve finally spent some time digging into it and have come to the conclusion that while it is not super simple, it is not impossible either. And now with code in hand, it should be much easier to get going!
Git Post Deploy Hooks
This entire approach makes use of the POST deploy hooks that are offered by GitHub and BitBucket. Both services allow you to POST a payload of information about incoming commits to a specific URL whenever new code is pushed to the repo. The Deploy script makes use of this POST data to see if it’s a recognized repo and branch, and if so, automatically pulls in the new changes for you. Setting up a service hook is slightly different between GitHub and BitBucket, but both services offer decent help articles on the subject.
GitHub: Post-Receive Hooks
BitBucket: POST Service Management
Getting from POST data to Git Auto Deploy
If you are in a hurry to set up your Auto Deploy, you’ll probably get there faster going straight to the GitHub repo: https://github.com/lkwdwrd/git-deploy
OK, great, so we know we can set up GitHub and BitBucket to ping a URL with some commit payload information, but how do we leverage this into a git auto deploy script? I decided my preferred approach was to create an endpoint to ping for each supported service. Also, because we basically want to auto git pull changes, most of the code to create an endpoint is exactly the same. In other words, most of it can be shared code. After some thought I decided I wanted a class structure where the main script is in an abstract class that is extended for each service so that new endpoints can easily be created for future services that offer similar POST data service hooks.
After getting the abstract parent class and two child classes set up for both GitHub and BitBucket, I abstracted out the registration array into a config file meant to be user editable. I feel this makes it slightly more user friendly because at the end of the day you can simply put your repo information in the configuration file, set up your POST hooks to ping the correct end point and BAM: Git Auto Deployment!
Well, almost… Let’s get everything set up!
Setup and Configuration
The first step to set up your auto deployment is to clone the deploy repo into wherever you would like the endpoints to be located. On WP Copilot I cloned the repo into deploy/. This means my deployment endpoints are https://lkwdwrd.com/deploy/github.php and https://lkwdwrd.com/deploy/bitbucket.php. These can be located wherever you wish on your site.
Once you’ve got the deploy script cloned into your desired location, you need to set up a repo to auto deploy. As an example I’ve written a WordPress plugin stored on GitHub that makes use of the excellent Prism JS code highlighter. I SSHed into my server and navigated to my wp-content/plugins folder. Once there I simply ran git clone https://github.com/lkwdwrd/prism-code-highlighting. I then entered the now present prism-code-highlighting/ directory and ran pwd, which gave me the full path to the repo. You’ll want to copy the file path down.
Going back to the deploy/ directory I updated my deploy-config.php file to register the Prism repository for auto deployment. I needed the name, the branch I wanted to deploy, and the path (the one I got by running pwd earlier). Note you can also specify a remote name if it is something other than ‘origin’, and you can specify a ‘post_deploy’ callback function if your code needs to perform additional functions after the code is pulled.
php
You can register as many repos as you want with the deploy script by adding a new element to the $repos array.
php
Once you have the repo information saved in the deploy-config.php file, set up a POST hook (see links above) to ping the appropriate endpoint. When you commit to the deploy branch you specified, GitHub or BitBucket will ping your endpoint, the script will recognize new code is available, and perform a git pull automatically for you.
Congratulations! You’re auto deploying your code!
Private Repos
One of the reason I like to have BitBucket available as an endpoint is the fact that they have unlimited private repositories for free. Something GitHub doesn’t have. You can work with private repos just like you do public ones, but you will need to set up an SSH deploy key for each private repo you want to deploy. It’s easy, but it took me a while to wrap my head around how this multiple SSH key thing works.
On your sever you can create an SSH key by running ssh-keygen -t rsa. With private repos you can only use one key with one repo. BitBucket wont let you use the same key to deploy multiple repos for security reasons. At first I was unhappy about it, but have come to terms with the fact that it is a decent security measure. Setting up multiple keys isn’t too hard, it just takes some understanding.
SSH into your server and then navigate to ~/.ssh/. Once there, run ls to see if you have a config file present. If you do not, create one. In that file you will want to use this pattern to give each repo it’s own SSH deploy key:
Host <alias>
HostName bitbucket.org
IdentityFile ~/.ssh/<filename>
You can put as many of these entries in the config file as you want. The <filename> is whatever filename you choose when running ssh-keygen -t rsa. I usually make it the same as the alias I choose. The <alias> can be whatever you want. This is an alias for the server you put in the HostName declaration. When you use the alias in url paths, your server will see it and replace the alias with the specified HostName and use the SSH key you specified in the IdentityFile declaration.
Yeah, confusing. So for example, if I’ve got the following in my ~/.ssh/config file:
Host wp-copilot-theme
HostName bitbucket.org
IdentityFile ~/.ssh/wp-copilot-theme
I will generate an ssh key ssh-keygen -t rsa -f ~/.ssh/wp-copilot-theme. Then I’ll run cat ~/.ssh/wp-copilot-theme.pub and copy the public key. I’ll enter this public key as the deploy key in my BitBucket repo (instructions). When that is set up I’ll use the <alias> I defined when I actually run git clone. GIT will then go to the .ssh config file and see “Oh, Replace this <alias> with this HostName, and use the SSH key in IdentityFile.” Or at least that is how it works in my head..
In our example I would run git clone git@wp-copilot-theme:wpcopilot/wcp-theme.git when I clone my private repo and it will resolve to git clone git@bitbucket.org:wpcopilot/wcp-theme.git and use the wp-copilot-theme ssh key. This will then be accepted by bitbucket because we’ve given them the .pub version of this key as our deploy key. The rest of the setup works just like it would if it was a public repo.
Finishing Up
And you now officially have auto deployment from GitHub or BitBucket, or heck, why not both? You can set up as many deploys as you want. Enjoy, and pull requests welcome!
April 19, 2013 at 5:15 am
It might be a week or two down the road but I will definitely give it a shot, I am a noob with 6 months of hard work behind me now and I am pulling the myriad piles of software @mylocalhost down again in preparation for a structured workflow that will include as much automation as I can employ. I have been curling-n-cloning-n-git’ing from repo for a couple of months but I still have not found a deployment configuration that I am satisfied with and being that I am as yet unable to burp myself after feeding I rely heavily upon the labor of those who go before me, my gut has no more experience that do I but it is telling me how good this method looks. As I progress my ability increases and someday I expect to awaken and discover that distant horizon I have been attempting to run down has begun to recede over my shoulder. I will gladly clone and begin my crude manipulations, and hopefully be able to make a meaningful contribution at some time, thanks.
May 15, 2013 at 11:01 am
So my brain is still frying! I want to set up lots of repos on bitbucket, but I don’t want to go through all this every time. Can’t I just write a PHP script to pull via HTTPS, call it deploy.php , then just modify:
$repo = “https://repos_location”;
$repo_pw = “bitbucket_repo_password”;
In other words, this is a great example of how to set up a SSH auto deploy. Thanks for putting it up! Anybody know of a script to autodeploy via HTTPS, where I could just plop it the root of the repo, clone in one time, then just ping deploy.php to pull down? I can’t believe this hassle of setting up SSH for each repo is the easiest way to go.
May 29, 2013 at 3:24 pm
Having a deploy key for each repo is a security thing. It takes an extra couple minutes to set up each private repo because of it, but generally you don’t have a ton of repos on any given site. This would be easier if you could use the same SSH key as the deploy key for multiple repos like you can with an account SSH key, but again it’s a security thing.
You could generate an SSH key on your server and stick it in your BitBucket SSH keys. This would give your server write access to your repos and it would be an non-authenticated key, which is why I would suggest avoiding it. However, doing it this way is would simplify the SSH key configuration somewhat and certainly be more convenient.
Another possibility is having one master repo that pulls in a bunch of your other repos as git sub-modules and just commit and incrementing change any time you make a change to one of the submodules. I’ve not really thought this through, but it would be an interesting thing to try. Hope you find what you’re looking for.
May 29, 2013 at 5:28 pm
Hey Luke,
This looks really cool! But could you please explain what the workflow of this script is? As in:
Git Push -> Git POST -> Webserver POST REQUEST -> Call script
And, what would you suggest when you don’t want your whole script to be able to access all scripts on your server?
Thanks,
Jeroen
May 29, 2014 at 9:05 am
Hey Jeroen,
Sorry for the late reply! This post has gotten neglected a bit as it was on a site that wasn’t well maintained. I’m hoping to keep up with it a little better here at its new home.
You are correct, the general workflow is:
Git push to remote -> Remote fires a web hook (a POST request) to the script -> the script runs a git pull on the server to get new changes.
Not quite sure what you are asking about accessing scripts on your sever. All git commands are hardcoded in an attempt to prevent command injection. To make things a little more secure you could implement checks for the referer before running the script. Depends on your needs/requirements.
November 28, 2013 at 6:49 am
What about if I have more than one webserver where I need to deploy automatically, and what about if my webservers are in autoscaling, so I don’t know how many and who are?
Thanks
May 29, 2014 at 9:10 am
Hey Fabio! If you are at the point that multiple servers and load balancing is in the equation, you may want to look at a more robust deployment system than a simple script like this one. Depending on your setup, you could make a callback function fire off an additional POST request to the provisioned clones and have them pull in the new changes. If you come up with anything for that situation using this script, I’d love an overview of how you ended up solving the problem.
May 28, 2014 at 6:43 pm
Do I require Git to be installed on the server? Also, it seems that I don’t have SSH access to the server, but I can generate SSH keys. Does that mean I can’t deploy with this script? Also… How on Earth it knows what repo to pull, or if it’s BitBucket or GitHub? Do I paste the HTTPS link of the repo to ‘remote’? I don’t get it.
May 29, 2014 at 8:54 am
Hey Daniel, thanks for taking the time to comment.
This script does require Git to be installed on the server and SSH access of some kind. You might be able to get it going through server panels of some kind, but that is going to be pretty tricky.
The general work flow is to clone the repo to the correct location on the server using Git over SSH. Using git over https requires a password, so you can’t have it pull in new changes automatically. You can set up both Bitbucket and Github with what they call deploy keys, which means you can read the repo over SSH, but can’t write to it.
Once the repository is cloned in the correct place, you can then set up this deploy script to pull in new changes as they are committed to the git remote. Because git holds the repository configuration information, the script doesn’t need to be aware of the git remote at all, so that is not part of the config file. It changes its working directory to the correct location (where your local repo is) and runs a pull to get the updates.
So 10,000 foot overview:
• Git knows where the remote repo is at and what to do when git pull is run.
• Webhooks from Github or Bitbucket tell us when there are new changes.
• The deploy script is the bridge between the Webhook and Git, telling Git to pull when the Webhook fires.
Hopefully that clarifies, feel free to reach out here or on Github if you have further questions! 🙂
May 29, 2014 at 1:55 pm
Thanks for the answer! I found out that my server does not allow such scripts to be run, neither Git or SSH. That infuriates me a bit, but I guess I should go for VPS hosting instead of shared. Once again, thanks for clarification!
August 6, 2014 at 10:52 pm
I’ve set up my auto deployment repo with bitbucket and it’s working when the repo is not private, but when I change it to private It’s not pulling as expected.
I followed every step of your tutorial but I don’t know how to make it work properly. I spent a lot of time on it.
I’ll appreciate your time to answer me or helping me somehow.
November 14, 2014 at 3:29 pm
Hey José,
Sorry for the delayed reply. For support, it’s easiest to keep track f things at https://github.com/lkwdwrd/git-deploy/issues.
There may be an issue with SSH key you use. You need to add it to the private repo as a deploy key. Private repos are sometimes hard to get working and the log doesn’t always show why. The PHP error log can be helpful in diagnosing problems. Make sure the SSH key you use is available to the user PHP runs as (usually www-data, nginx, etc).
November 13, 2014 at 3:50 am
Whenever i push a commit it goes to bitbucket but does not make changes on the website. But whenever i delete the repo in sourcetree and reclone it, it makes the changes just one time when i do next commit. I cant delete and re-clone it every time in sourcetree so please help.
November 14, 2014 at 3:16 pm
Hey Ashish,
For support It’s easiest to keep track of things on github (https://github.com/lkwdwrd/git-deploy) but even then I don’t have a ton of time to run support. Take a look at the deployment log, usually located in your deploy folder. Sometimes it will still give you a “deployment successful” message. If it is, then the script is generally working, but something else in the setup may be off. In these instances I’ve sometimes found clues in the PHP error log.
More often than not it’s an issue with the SSH keys.
Thanks for giving it a try!
March 27, 2015 at 4:45 pm
Hi, Im in progress trying this out on the impossible godaddy shared hosting. But regarding private repos, have you seen this bitbucket guidance about multiple ssh identities?
If I’m reading it correctly, this will allow you to reference one key for the server, rather than a separate key for each repro. What a blessing that would be!
https://confluence.atlassian.com/pages/viewpage.action?pageId=271943168
April 3, 2015 at 12:43 pm
Sadly, I was mistaken – but I have figured it out how to pull a mix of private and public repos. Full writeup here: http://stackoverflow.com/a/29433348/362445
On your remote, you can set up a server key and alias this for parent repo (pull and push), but this key will *not* work for the submodules. For submodule, Bitbucket at least will allow you to use one deploy (pull only) key across multiple repos. You will need to set up at least one deployment key, and assign it to each of them on bit bucket, and then create an alias ssh alias on your remote, using git as the host. I documented the steps I took in that SO post.
October 13, 2015 at 3:13 pm
It’s not possible to test this somehow without creating the hook on github (to avoid testing creating push). For example if I call /deploy/github.php it shows this error:
No payload present
A GitHub POST payload is required to deploy from this script.
October 15, 2015 at 3:16 pm
I’m receiving this output:
Could not create directory ‘/var/www/.ssh’. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. and the repository exists.
When running this:
echo system( ‘git pull origin development 2>&1’);
I’ve to extract that from your code because it was not showing error, on github it appears as if command worked like a charm but was not updating. It’s a private repo, how I can specify which public key to use?