Whiskers - a tool for keeping track of your buildouts

Few years ago I released a first version of Whiskers together with buildout.sendpickedversions. It's about the time to push new version out.

What is Whiskers?

Whiskers is a Pyramid web application which stores information about your buildouts so that you don't have to manually check what they contain. All the data to Whiskers is transferred by buildout.sendpickedversions - an buildout extension which keeps track of packages your buildout uses and sends the data in json-format to Whiskers-server.

Reason for Whiskers?

Main reason for me to develop Whiskers was based on the need of knowing what packages I have in our servers. I'm working at the University of Jyväskylä and part of my job is to maintain our Plone instances. We have about 100 Plone instances with slightly varying setup. If we needed a new version of some specific package it meant lots of work to have a list of Plone-sites which needed the update. With Whiskers I know what to update after few clicks.

New tricks

Since last Plone conference in Arnhem I've tried to give Whiskers (and buildout.sendpickedversions) some well deserved attention and simply make it better.  Outcome of this slow process was finally published earlier this month and it has several new features:
  • Whiskers keeps track of hosts where buildouts are ran.
  • Whiskers keeps much better track of package requirements - every requirement and version is tracked.
  • Whiskers knows packages which are not used by any buildouts anymore.
  • Whiskers knows your buildouts history and has settings view where you can configure how many buildouts is being stored to DB.
  • New Twitter Bootstrap based UI.
Most above features needed a lots of changes to buildout.sendpickedversions - not to mention that things have changed a bit after zc.buildout 2.1.x release. New Whiskers requires you to use new release of buildout.sendpickedversions.

If you are using old version of Whiskers, please note that new version is not compatible with old one.

Help me to make Whiskers better

If you are using Whiskers and have ideas how to improve it, please share those ideas with me by creating a new issue in Github or send in a pull request.


cd under development packages nested folders without repeating yourself

To me Plone development environment means Fedora 12, terminator and vim. As I do lots of development in the shell there has been one annoying little thing which I'm tired of - cd:ing under development packages inner structure to where the actual code lives.

Doesn't seem much but if you repeat that many times a day you'll get tired of it. Of course you can always launch vim for the package root and open the file you wanted from there, but it's still extra effort.

Few weeks ago me and my colleague Jussi Talaskivi started to think that there has to be a script which takes you to your destination simply by parsing dotted name and using information from there to cd you to the correct place. After spending some time googling around we found none. Annoyed by this we decided to do this missing script by ourself.

Soon it turned out that this wasn't that easy task as it sounds - simple shell or python script where you just parse correct path from arguments wouldn't do the trick as they're launched in a separate process which doesn't have any effect on the working dir of your shell. Little research and try - error approach later we found working solution.

1. Create shell script (eg. pcd.sh) to somewhere on your $PATH which contains following lines:

cd `echo -n $1/;echo $1 | sed -e 's/\./\//g'`

2. Chmod your shell script so that you can execute it.

3. Edit your .bashrc and add following alias:

alias pcd=". pcd.sh"

Now pcd command should work as expected:

$ pwd
$ pcd very.long.packagename
$ pwd


The key element is the first dot in the alias. This makes your command to execute in the current shell session. As alias doesn't cope with arguments you'll need a script (in this case pcd.sh) for parsing to work. I know this isn't the most sophisticated piece of code and there probably are better ways to achieve the same result, but I couldn't find one.

I hope this example is useful for someone frustrated for the long nested structures we're all so familiar with.


Plone RPM deployment

It's been ages since I wrote my last post. I guess now it's time to make that up with post about Plone deployment. In a last few months I've had chance to invest part of my time at work to develop our Plone deployment model and since there hasn't been that many posts about the subject I thought I'd share my experiences for the community.

Before I get to the details here's some background information about the environment I've been dealing with. I'm working at the University of Jyväskylä, Finland. Our university is a heavy weight Plone adopter here in Finland. Plone is in use by every faculty and department. We have about 500 - 700 content managers and about 50 - 60 separate Plone instances. Our front page gets about 2 - 3 million hits page views / month. The important information in previous data is the amount of instances. 50 - 60 is just my estimation about our current status and that amount is increasing every year. I'm sure you got the idea so I'll continue to our previous deployment model which I'm sure is familiar to anyone who has had the pleasure to deploy Plone sites.

Current setup

We've got quite normal setup of RHEL servers, Apache + Varnish combination and loads of Plone sites. Our sites use mainly Plone 3.3.5, but we have increasing amount of Plone 4 sites and also few Plone 2.1 and 2.5 sites. We've used buildout to deploy new sites and update the old ones and as I've mentioned in my previous posts (Managing multiple plone buildouts, problems with plone version pinnging) it hasn't always been enjoyable experience. Don't get me wrong - buildout is awesome tool if you need to put up few sites every now and then and don't need to look after them, but running about 30 buildout scripts within few hours scheduled update window isn't fun. I know many of you think now that why didn't they script that? We did. We also developed tools to update packages automatically to avoid all the manual steps - yet we hit problems in updates, some of them were plain human errors (eg. wrong version pinnings) and some of them were because of some unpredictable behavior of software.

Reinventing the wheel?

Every now and then we've been thinking better deployment story for Plone-sites until we realized that the things we'd want to achieve sounded just like package management software. As we're using Red Hat os in our servers I hold my breath and jumped head first to the RPM world. There isn't too much information about deploying Plone sites as RPM-package on WWW. I knew folks at Weblion have developed environment for Debian based distros, but that didn't help much. Luckily Google revealed that Nikolay Kim from Enfoldsystems had been in talks with Fedora packagers about packaging Plone as RPM. As far as I understood from the mailing list messages this didn't work out well and Enfoldsystems packages never ended up to Fedoras repository due to packaging policy disagreements. Nevertheless Enfoldsystems have kindly published their Plone buildouts and RPM specs in their Subversion repository.

RPM setup

I used Nikolay Kims work as a base structure to fit things to our environment. Nikolay had divided Plone to two separate RPMs. First one (plone-base) contains all the common packages Plone needs. Another package contains the Plone-instance and its eggs. Nikolays packages should work out of the box as they are, but we needed to modify them a bit to suit our needs. I created own RPM-package for the Python virtualenvs - one for Python 2.4.6 and one for Python 2.6.5. I also created RPM-packages for few python libraries (python-imaging, python-lxml, python-ldap) so that they install to virtualenvs instead of system python. With these ready I modified Nikolays spec-files to use my virtualenv packages and modified buildout.cfg file so that it extends to our shared configuration and the initial setup was done. There was lots of manual work to do to add and test RPM-specs for our buildouts but now when it's done life seems to smile at me. To make my workload even lighter and to avoid dull copy-paste work I created custom paster template which creates buildout folder with RPM-spec ready for rpmbuild to do it's magic. I attached the rpmbuild process to Hudson by creating parametrized builds - when our developers includes certain commit message and pushes his changes to Mercurial this launches hudson build, tests and if everything went ok, finally rpmbuild. This is done with the help of hghudson package which I customized a bit to allow parametrized Hudson builds.

What we achieved?

You may wonder what makes RPM deployment better than buildout?

Here are some pros listed in no particular order:
  • It fits well with the regular operating system maintenance.
  • Our system administrators can now update servers/packages automatically without any knowledge about Plone.
  • If something goes wrong we can easily rollback to previous package version.
  • Deploying new site happens just like installing any other package with package management software.
  • We can be sure there isn't going to be any trouble with pypi or some other critical site being down when we're updating/creating new instance.
  • We still have buildout in the background to make developers life easy.
  • It saves time, nerves and makes it possible for me to once again enjoy Wednesdays (guess which day our scheduled maintenance window is?) :)

I know some of these can be achieved by using buildout as well - eg. by mirroring pypi, using eggproxy etc., but nevertheless my experience is that running buildouts on a production server has always a bigger chance to fail than updating tested RPM-package. I still have to admit that at the moment this deployment model is new to us and I probably won't know all the cons. Here are few which are more related to the setup process of RPM deployment model:

  • You'll need a way to create RPM-spec automatically.
  • RPM deployment isn't as straightforward than using buildout.
  • You'll get most out of the pros only when you're serving several instances.
  • You'll need to know at least basics about how to create RPM-packages to set up this kind of environment.

I hope this post will be useful to someone struggling with similar problems. I will be posting updates about the subject when our new deployment model has seen more use. I'd also like to thank Nikolay Kim and Enfoldsystems for the excellent work they have done to make Plone RPM-packaging being as easy as it is now. Thank you!


Plone 4, ZEO and supervisor

This post belongs also to the "lessons learned" category.

With Plone 3, ZEO and supervisor combination you've probably configured your supervisor to start plone instances by running $BUILDOUT/parts/client1/bin/runzope.

Problem is that with Plone 4 your $BUILDOUT/parts/client folder doesn't contain anything else than etc folder. You know starting instances by targeting supervisor to use $BUILDOUT/bin/client1 fg doesn't work like you'd expect (supervisor would control the client1 script - not the actual plone process).

My colleague Jussi Talaskivi figured that using 'console' argument instead of 'fg' for bin/client1 script should do the trick. With 'console' argument stopping, starting and restarting Plone 4 instances with supervisor works like a charm.

Below is full example of working supervisor configuration.


parts = supervisor

recipe = collective.recipe.supervisor
port = 8200
user = xxxx
password = xxxx
programs =
    10 zeo       /buildout/xxx/parts/zeoserver/bin/runzeo  true zope
    20 instance1 /buildout/xxx/bin/instance1 [console]     true zope
    20 instance2 /buildout/xxx/bin/instance2 [console]     true zope


Plone 4, ZEO and tempstorage

This is reminder for myself and hopefully for other people too so that they don't have to waste several hours of valuable time looking for this simple piece of information.

If you have experience setting up Plone 3 with ZEO and tempstorage (so that your sessions won't disappear in the middle of hardcore content managing) you might have trouble setting up same kind of setup while using Plone 4.

Thanks to David Glick who enlightened  me that plone.recipe.zeoserver doesn't include whole Zope 2 in it's Python environment anymore. In this specific scenario this means that even though you have tempstorage in your eggs folder it isn't included in pythonpath when zeoserver part is being processed and this leads to below error message:

Error: could not load package tempstorage: No module named tempstorage
Package name: 'tempstorage'
File name: 'component.xml'
Package path: None
For help, use bin/zeo-server -h

Solution to this is to add this simple oneliner to your buildout.cfg zeoserver part:

eggs = tempstorage


Usability and Plone

I've seen in here and there someone mentioning that usability of Plone is very good. Lot's of people - me included - like the fact that in Plone you don't have separate content management interface compared to some of Plones rivals. That counts for something when we're talking about good usability. Still that is only one quite small part of the whole picture. So what else is there? What do people like in Plone and where are the rough edges for end users? If general consensus is that Plone does have good usability, where is the actual proof of that?

On plone.org I found one page in developer documentation mentioning following: "Plone differentiates itself on usability. The intuitiveness of the user interface is what attracts people to Plone the most."

I interpret this sentence meaning about the "one view for all" approach. What bugs me in this is that this whole sentence about good usability is about how the UI works compared to Plones competitors. Saying that our approach has more intuitive UI than our competitors is one thing - proving that our claim is true is another thing. Yet I think the most important thing is missing from here. Proving that the UI we have has good usability in itself. Is the user experience better than our competitors and good enough for our end users or is it actually, really and without any doubt so good that it wouldn't be better without rethinking whole thing and designing even more intuitive UI?

In launchpad.net/plone it reads that "Plone is easy to use. The Plone Team includes usability experts who have made Plone easy and attractive for content managers to add, update, and mantain content."

Good usability is not just that someone says or writes it. You'll need to prove it too - and not only by comparing how our product does against competitors products - getting the end users opinion weights so much more.

Why I'm bringing this topic up? I'm doing my masters thesis about Plone usability and in the past week I've been searching usability studies about Plone to get some background material. When I started this I thought it'd be easy to find at least few through reports - knowing that people have mentioned Plones good usability quite often in separate posts. To my surprise I found only few short blogposts (Plone Usability Makes It As Easy As ABCUsability of CMS Home Sites) and few evaluation videos, but none of those are scientifically accurate enough to be material in masters thesis. Basically I didn't find any actual usability study made about Plone.

I hope you don't get me wrong in this post. I'm being over critical about this matter as the subject is maybe too close my job and interests. I've been working with Plone about 4 years now and this includes developing new products, giving end user support and administrating sites which gets millions hits/month. I'm trying to forget all this in my masters thesis which should to be scientifically approvable study. If I won't find any proof about the good usability I have to think (scientifically) that it is all just words and nothings been proven - yet.

If you happen to know some study what I've (obviously) missed I'd be more than thankful to know about it.


Speed comparsions between Plone and Wordpress

Jon Stahl wrote recently a blogpost about Plone being three times faster than Dropal, Joomla and Wordpress. We had a small discussion about this in my workplace and as my colleague pointed out this wasn't a really that comprehensive test that you could state Plone being 3 times faster than it's competitors. This seemed a bit unfair test considering how fast this has been spread in tweet/blogosphere, so I decided to repeat the test with a bit more critical viewpoint.

What's wrong in the original test? No one would consider opinion poll with 10 answers nowhere near trustworthy - it's all the same with requests. I didn't want to put up all the cms so I just set up second best (Wordpress 2.9.1) and compared that to my Plone development site (4.0a3). As a comparsion I did the first test with same ab command Jon used and my iMac gave following results:

Wordpress: 7.13 requests / second
Plone: 17.10 requests / second

So far we have clear winner and Jons data holds up... almost. Anyways.. to do a bit more realistic test I used 100 requests - first with no concurrency and then concurrency value set to 3 (ab -n 100 -c 3 http://...) Results are a bit different:

No concurrency, 100 requests
Wordpress: 7.18
Plone: 16.46

Concurrency 3, 100 requests
Wordpress: 13.04 requests/second
Plone: 16.50 request/second

To add some more requests I got just small differences anymore. With ab -n 200 -c 3:

Concurrency 3, 200 requests
Wordpress 13.09 req/s
Plone 16.37 req/s

Concurrency 4, 200 requests
Wordpress: 13.04 req/s
Plone 16.39 req/s

So when I just added some more requests situation was quite the same, but when we have (in the real world) that much requests in shortest time possible there are bound to be some concurrency. Adding concurrency to my test situation doesn't look that bad for Wordpress anymore.

Even though Wordpress did better job with more concurrency it still got beaten by Plone in my rough testing. Yet it's not that spectacular to be 1.2 times faster than being three times faster. And don't get me wrong - as a Plone developer and administrator I'm really thankful for all the people who have contributed to Plone tuning in one way or another. It's still good to be a bit critical before we claim that good results :)

To be clear, I'm not really familiar with Wordpress or anything which has something to do with PHP. As Jon I also put up MAMP environment and installed Wordpress there using PHP 5. There were no other customizations so I don't know how tuned it was for production use.  My Plone on the other hand is my development instance and it was running in debug mode on the foreground so it was in worst possible situation for Plone to run... or well.. I could have enabled very specific logging, but that would have been overkill for this test.