============ A quick tour ============ Goal ---- This tutorial will expand on the :ref:`first-run-label` tutorial by taking a quick tour around some of the features of buildbot that are hinted at in the comments in the sample configuration. We will simply change parts of the default configuration and explain the activated features. As a part of this tutorial, we will make buildbot do a few actual builds. This section will teach you how to: - make simple configuration changes and activate them - deal with configuration errors - enable forcing builds - change the update step that checks out source code - enable and control the IRC bot - enable the GUI status client - enable the debug client - customize the web output icon and CSS Setting project name and URL ---------------------------- Let's start simple by customizing the buildbot's project name and URL. We continue where we left off in the :ref:`first-run-label` tutorial. Open a new terminal, and first enter the same sandbox you created before:: cd cd tmp/buildbot source sandbox/bin/activate $EDITOR master/master.cfg Now, look for the section marked *PROJECT IDENTITY* which reads:: ####### PROJECT IDENTITY # the 'projectName' string will be used to describe the project that this # buildbot is working on. For example, it is used as the title of the # waterfall HTML page. The 'projectURL' string will be used to provide a link # from buildbot HTML pages to your project's home page. c['projectName'] = "Buildbot" c['projectURL'] = "http://buildbot.sourceforge.net/" Change the last two lines to read:: c['projectName'] = "Buildbot Tutorial" c['projectURL'] = "http://www.google.com/" Now, from the terminal, type:: buildbot reconfig master You will see a handful of lines of output from the master log, much like this:: 2009-08-01 13:38:21+0200 [-] loading configuration from /home/thomas/dev/ext/buildbot/sandbox/master/master.cfg 2009-08-01 13:38:21+0200 [-] builder buildbot-full is unchanged 2009-08-01 13:38:21+0200 [-] removing IStatusReceiver 2009-08-01 13:38:21+0200 [-] configuration update started 2009-08-01 13:38:21+0200 [-] (Port 8010 Closed) 2009-08-01 13:38:21+0200 [-] Stopping factory 2009-08-01 13:38:21+0200 [-] adding IStatusReceiver 2009-08-01 13:38:21+0200 [-] twisted.web.server.Site starting on 8010 2009-08-01 13:38:21+0200 [-] Starting factory 2009-08-01 13:38:21+0200 [-] WebStatus using (/home/thomas/dev/ext/buildbot/sandbox/master/public_html) 2009-08-01 13:38:21+0200 [-] adding 0 new schedulers, removed 0 2009-08-01 13:38:21+0200 [-] adding 0 new changesources, removing 0 2009-08-01 13:38:21+0200 [-] configuration update complete Reconfiguration appears to have completed successfully. The important lines are the ones telling you that it is loading the new configuration at the top, and the one at the bottom saying that the update is complete. Now, if you go back to `the waterfall page `_, you will see that the project's name has changed, and now shows *Buildbot Tutorial* instead: .. image:: waterfall-tutorial.png :alt: waterfall with tutorial. If you click on either the top or bottom link, it will take you to Google, the URL you configured. Configuration errors -------------------- It is very common to make a mistake when configuring buildbot, so we might as well show you now what happens in that case and what you can do to fix the error. Open up the config again and introduce a syntax error by removing the first single quote in the two lines you changed, so they read:: c[projectName'] = "Buildbot Tutorial" c['projectURL'] = "http://www.google.com/" This creates a Python SyntaxError. Now go ahead and reconfig the buildmaster:: buildbot reconfig master This time, the output looks like:: 2009-08-01 13:52:23+0200 [-] loading configuration from /home/thomas/dev/ext/buildbot/sandbox/master/master.cfg 2009-08-01 13:52:23+0200 [-] error while parsing config file 2009-08-01 13:52:23+0200 [-] error during loadConfig 2009-08-01 13:52:23+0200 [-] Unhandled Error Traceback (most recent call last): File "/home/thomas/dev/ext/buildbot/sandbox/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/application/app.py", line 348, in runReactorWithLogging reactor.run() File "/home/thomas/dev/ext/buildbot/sandbox/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 1128, in run self.mainLoop() File "/home/thomas/dev/ext/buildbot/sandbox/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 1137, in mainLoop self.runUntilCurrent() File "/home/thomas/dev/ext/buildbot/sandbox/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 757, in runUntilCurrent call.func(*call.args, **call.kw) --- --- File "/home/thomas/dev/ext/buildbot/src/buildbot/master.py", line 511, in loadTheConfigFile self.loadConfig(f) File "/home/thomas/dev/ext/buildbot/src/buildbot/master.py", line 529, in loadConfig exec f in localDict exceptions.SyntaxError: EOL while scanning string literal (master.cfg, line 191) 2009-08-01 13:52:23+0200 [-] The new config file is unusable, so I'll ignore it. 2009-08-01 13:52:23+0200 [-] I will keep using the previous config file instead. Reconfiguration failed. Please inspect the master.cfg file for errors, correct them, then try 'buildbot reconfig' again. This time, it's clear that there was a mistake. in the configuration. Luckily, the buildbot master will ignore the wrong configuration and keep running with the previous configuration. The message is clear enough, so open the configuration again, fix the error, and reconfig the master. The first build --------------- By now you're probably thinking: "All this time spent and still not done a single build ? What was the name of this project again ?" You're right, so let's fix that. Edit the configuration again:: $EDITOR master/master.cfg Find the section that starts with *STATUS TARGETS*:: ####### STATUS TARGETS # 'status' is a list of Status Targets. The results of each build will be # pushed to these targets. buildbot/status/*.py has a variety to choose from, # including web pages, email senders, and IRC bots. c['status'] = [] # Use allowForce=True (boolean, not a string. ie: not 'True') to allow # Forcing Builds in the Web User Interface. The default is False. # from buildbot.status import html # c['status'].append(html.WebStatus(http_port=8010,allowForce=True)) from buildbot.status import html c['status'].append(html.WebStatus(http_port=8010)) Comment out the last two lines, and uncomment the two lines before that:: # Use allowForce=True (boolean, not a string. ie: not 'True') to allow # Forcing Builds in the Web User Interface. The default is False. from buildbot.status import html c['status'].append(html.WebStatus(http_port=8010,allowForce=True)) # from buildbot.status import html # c['status'].append(html.WebStatus(http_port=8010)) Just as before, reload the configuration. Now, go back to `the waterfall page`_. This time, click on the buildbot-full link, and scroll down. You will see some new options that allow you to force a build: .. image:: builder-force.png :alt: force a build. Type in your name and a reason, then click *Force Build*. After that, click on `view in waterfall `_. You will now see: .. image:: builder-exception.png :alt: an exception happened. Well, that kind of purple can't be a good thing. If you click on `stdio ` you can scroll down and see why it failed:: Logging in to :pserver:anonymous@cvs.sourceforge.net:2401/cvsroot/buildbot CVS password: cvs [login aborted]: connect to [cvs.sourceforge.net]:2401 failed: Connection refused elapsedTime=0.829763 The build failed because buildbot no longer uses CVS on Sourceforge. So that's easy to fix. We'll make the configuration use the new git URL instead. Edit the configuration file again, and find the section marked *BUILDERS* which has the following code:: cvsroot = ":pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot" cvsmodule = "buildbot" from buildbot.process import factory from buildbot.steps.source import CVS from buildbot.steps.shell import Compile from buildbot.steps.python_twisted import Trial f1 = factory.BuildFactory() f1.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="copy")) f1.addStep(Compile(command=["python", "./setup.py", "build"])) f1.addStep(Trial(testChanges=True, testpath=".")) Replace the two lines starting with cvs with a line that reads:: gitrepourl = "git://github.com/djmitche/buildbot.git" The line that imports CVS should be changed to import Git instead. The line that uses it (the first addStep) should be changed to only use the repourl keyword and the mode keyword. After changing the same section should look like this:: gitrepourl = "git://github.com/djmitche/buildbot.git" from buildbot.process import factory from buildbot.steps.source import Git from buildbot.steps.shell import Compile from buildbot.steps.python_twisted import Trial f1 = factory.BuildFactory() f1.addStep(Git(repourl=gitrepourl, mode="copy")) f1.addStep(Compile(command=["python", "./setup.py", "build"])) f1.addStep(Trial(testChanges=True, testpath=".")) Reconfigure the master. Take special care to verify that the master has been reconfigured successfully. Then go `back to force another build ` Force a new build, then go back to the waterfall view. The view now looks slightly better: .. image:: builder-git-failed.png :alt: git build failed. The update and compile steps are green, so that means they worked fine. The tests step has failed. Feel free to click on the various links to see the output you get from each step, successful or failed. In this particular case, the tests step has failed because the log of the test cannot be parsed correctly. (This is most likely a bug in buildbot: a forced build is not triggered by a particular change, while this particular test step only tests files affected by the change. No change, no tests.) Enabling the IRC bot -------------------- Buildbot includes and IRC bot that you can tell to join a channel and control to report on the status of buildbot. First, start an IRC client of your choice, connect to irc.freenode.org and join an empty channel. In this example we will use #buildbot-test, so go join that channel. (*Note: please do not join the main buildbot channel!*) Edit the config and look for the *STATUS TARGETS* section that you changed before to be able to force the build. Find the following lines:: # from buildbot.status import words # c['status'].append(words.IRC(host="irc.example.com", nick="bb", # channels=["#example"])) Uncomment those lines and modify them to read:: from buildbot.status import words c['status'].append(words.IRC(host="irc.freenode.org", nick="bbtest", channels=["#buildbot-test"])) Reconfigure the build master, The log output should contain a line like this:: 2009-08-01 15:35:20+0200 [-] adding IStatusReceiver You should see the bot now joining in your IRC client. Type:: bbtest: help to get a list of the commands the bot supports. Let's tell the bot to notify certain events:: bbtest: notify start bbtest: notify finished bbtest: notify failure The bot should have responded to each of the commands:: bbtest: notify on started The following events are being notified: ['started'] bbtest: notify on finished The following events are being notified: ['started', 'finished'] bbtest: notify on failure The following events are being notified: ['started', 'failure', 'finished'] Now, go back to the web interface and force another build. Notice how the bot tells you about the start and failure of this build:: build #3 of buildbot-full started including [] build #3 of buildbot-full is complete: Failure [failed tests] Build details are at http://localhost:8010/builders/buildbot-full/builds/3 You can also use the bot to force a build:: bbtest: force build buildbot-full test build This time, the bot is giving you more output, as it's specifically responding to your direct request to force a build, and explicitly tells you when the build finishes:: bbtest: force build buildbot-full test build build #5 of buildbot-full started including [] build #5 forced I'll give a shout when the build finishes build #5 of buildbot-full is complete: Failure [failed tests] Build details are at http://localhost:8010/builders/buildbot-full/builds/5 Hey! build buildbot-full #5 is complete: Failure [failed tests] Build details are at http://localhost:8010/builders/buildbot-full/builds/5 You can also see the new build in the web interface. Enable the GUI status client ---------------------------- Buildbot comes with a simple graphical application that can talk to a buildbot master. To enable it, change the master configuration. Underneath the section from before there are the following two lines:: # from buildbot.status import client # c['status'].append(client.PBListener(9988)) Uncomment them, then reconfig the master. The log should have the following line in it:: adding IStatusReceiver Note that the debug client requires `pygtk2 `_. If you are still using virtualenv and easy_install for buildbot, it will not be available in your sandbox (although it's most likely available on your system). In that case, we need to fiddle with $PYTHONPATH. Try starting the GUI client by typing:: buildbot statusgui localhost:9988 If that fails with:: ImportError: No module named gobject then try the following:: PYTHONPATH=$PYTHONPATH:/usr/lib64/python2.6/site-packages buildbot statusgui localhost:9988 Note that in that line, you should change lib64 to lib if you are on a 32-bit machine, and change python2.6 to match your Python version. Possibly you need to make further changes depending on your distribution. If that works, you will see a simple rectangular application that shows the status of the last build: .. image:: statusgui-idle.png :alt: idle status gui Force a build either through the IRC bot or the web interface, and you will notice this application will react and show the following: .. image:: statusgui-building.png :alt: building status gui The status gui application is not the most polished, but it does the job. *Note: there is also a status applet, but currently it is not that easy to install.* Enable the debug client ----------------------- There is a simple graphical debug client that can be useful for debugging buildbot. To enable it, find the *DEBUGGING OPTIONS* section and uncomment the debugPassword line:: c['debugPassword'] = "debugpassword" Reconfigure the master. This time you won't see anything in the logs, but that's fine. Start the client:: buildbot debugclient -m localhost:9989 -p debugpassword (Again, if you have an ImportError for gobject, adapt the $PYTHONPATH) You should see: .. image:: debugclient.png :alt: debug client *Note: make sure you use the right port! For the status GUI, we enabled port 9988. For the debug client, the same port should be used as for the slaves, which in our default configuration is 9989.* Click the 'connect' button. If all is well, the GUI should respond by going from 'Disconnected' to 'Connected'. Click 'poke IRC', and see the message from the bot in your IRC client. Click the 'commit' button. Now see the waterfall, and notice how there is a 2 minute count down: .. image:: waterfall-waiting.png :alt: waiting for a build The 'commit' button faked a commit by bob, and now buildbot is waiting for a while in case any other commits might land in the tree, since often developers commit patches in series. After that build has completed, try typing 'buildbot-full' in the 'Builder' field, then click 'Request Build'. Notice in the web status that the builder starts building. Customizing the web status -------------------------- Let's finish up this tutorial with some simple customizations to the web output. The master directory contains a public_html/ subdirectory which is what buildbot's web status serves up. First, let's add a favourite icon. It makes life easier when browsing through your browser bookmarks looking for that buildbot page. I like the little buildbot nut, so let's use that for now:: wget http://buildbot.net/favicon.ico mv favicon.ico master/public_html $EDITOR master/public_html/index.html Now, add the following line in the section:: Note that you do not need to restart the master in this case; the file gets served from disk on every page view. If you go back to the index page, you should now have a shortcut icon. *Shortcut or favorite icons are fickle beasts, so you might have to do anything from a ctrl-refresh to a browser restart.* Note: there is currently no way to have all pages serve the favicon. FIXME. Note: if you are going to set up or follow more than one buildbot, it helps to create different favourite icons to keep them apart. Don't go nuts. Second, let's customize the .css. A long long time ago at `Fluendo `_ we wanted a slightly more modern look for buildbot. Download `the css file `_ and place it in public_html/, then refresh the web page. Note: this updated .css might be out of date. If you want to customize the CSS yourself, take the original .css file as a starting point.