Short description

Currently Ganeti Web Manager lacks proper packaging. If a sysadmin decides to install GWM, he/she should be able to simply install via package manager and be done.

This project intends to build tools (or leverage existing ones) to automate creation of .deb and .rpm Ganeti Web Manager packages for Linux distributions. The actual work will also include further modularization of GWM, workflow automation and packages for opt-in GWM features, like VMs metric system (GSoC 2012).

Detailed description

Installation of GWM currently requires these steps:

1. Installing Python dependencies (pip, *-devel, virtualenv, fabric) -- all of them system-wide.
2. Cloning GIT repository / downloading tarball.
3. Running fabric with production or development setup.
4. Sourcing into virtual environment.
5. Proper setting up Django.
6. Dealing with South migrations.
7. Finally running a development server.

I intend to shorten this path for the end-user to just these steps:

1. Installing Ganeti Web Manager and its dependencies: via package manager
2. Configuring Django (/etc/ganeti-webmgr/settings.conf will provide some default settings)
3. Updating database / initializing (/usr/bin/gwm-update or /usr/bin/gwm-init)
4. Running server (/usr/bin/gwm-server - might not even be a development server, but rather small, lightweight server shipped with GWM, like Cherokee)

Actual work includes:

• modularization of GWM into separate Django apps: three weeks MILESTONE 1
• separating GWM dependencies into stand-alone projects: two weeks
  • for these dependencies: Django Object Permissions, Django Object Log, Django Muddle Users, Muddle, Twisted VNC Auth Proxy
  • proper git repositories
  • proper Python packages
  • including documentation
  • including tests
  • getting rid of Muddle (if desired)
  • most of above already done
• incorporating mock and git-buildpackage workflow into GWM repository
• setting up RPM repository somewhere at OSUOSL
• automation of building and testing GWM and its dependencies .deb and .rpm packages -- this can be done via GIT hooks (see below)
• providing DEB repository as well
• getting rid of Fabric: five weeks (last 5 steps) MILESTONE 2
• writing system /usr/bin/gwm-* applications: two weeks MILESTONE 3
• ADDITIONAL: building GWM opt-in metrics system .deb and .rpm packages
• ADDITIONAL: improving automation within GWM workflow (e.g. CI server)
• ADDITIONAL: work on current GWM issues

Additional steps will be taken into account if I work ahead of schedule.

Technical details

GWM modularization: separate Django applications for Nodes, Clusters, Virtual Machines, and maybe more. This involves separating appropriate views, models and templates into Django apps and actually might require changing GWM project structure into what is generated by Django since version 1.4.

GWM dependencies: documentation hosted on Read The Docs, unittests, packages uploaded to PyPI.

Building .deb packages: with help from git-buildpackage.

Building .rpm packages: not with Python's very own distutils, but rather with Fedora's mock.

At first I suggest to automate building distribution packages via GIT repository hooks, for example: a developer creates a new tag in repository, then a building script is invoked. Of course this can be done manually, too.

Structuring GWM projects: like they did at Synnefo. Everything mostly comes down to including configuration from /etc/ -- here are adequate lines.

What instead of Fabric for development process? The usual way Python projects are being developed is: set up virtual environment, install dependencies via pip (even from requirements_development.txt file). Getting rid of Fabric actually lengthens the list of steps to start developing GWM, but the advantages are:

• no invoking system commands from Python script
• 300 lines less to maintain
• more freedom for developers when it comes to virtual environments (for example I personally like using virtualenvwrapper).

Python virtual environments are known for not being portable (even though they claim to be). But we should not rely on end-user OS to provide all our dependencies. Thus DEB/RPM packages containing almost whole GWM project (with external dependencies) is what we should be trying to reach.

Hynek Schlawack wrote great article (Python Application Deployment with Native Packages) on how to achieve this goal.

Some keypoints and conclusions from his article:

• Virtual environments should be re-created upon installation on end-user system -- but only executables, (sym)links and directory structure. Python libraries will be copied from within DEB/RPM package.
• We have to watch out for packages from PyPI that needs to be compiled. In the perfect world we don't have any compiled dependencies. We could then simply use fpm to create RPMs/DEBs.
• If we have to provide some dependencies which require compilation, we should make source RPM (sRPM) and source DEB (DSC) packages first and then prepare them for various Linux distributions via mock and pbuilder or git-buildpackage, because fpm probably can't build packages for different target distributions at once.
• Some GWM dependencies that can be provided by end-user OS and we could rely on:
  • pyopenssl
  • simplejson (we could get rid of it if we bumped required Python version to 2.6)

Lightweight web server that might end up being included with GWM package: Cherokee.

RPM packages have higher priority than DEBs. When building RPMs, I have to take into account different distributions. mock helps with that issue.

Timeline

June 17 - June 23

GWM modularization

June 24 - June 30

end term exams, will not really be able to work too much

July 1 - July 15

GWM modularization continued

July 16 - July 29

work on GWM dependencies (documenting, unittesting, PyPI, etc)

July 30 - August 26

setting up RPM and DEB repositories, building GWM with its dependencies and packaging it to these repositories. Automating this process. Getting rid of Fabric.

August 27 - September 2

Work on /usr/bin/gwm-* tools.

September 3 - September 9

Exams again (in case I failed at previous attempt).

September 10 - September 16

Work on /usr/bin/gwm-* tools continued.

September 17 - September 27

Finishing touches, documentation, etc.

I do not plan to go on any longer than one day vacation. Only time I won't spend writing software during GSoC is when I have end term exams.