Re-inventing Python Packaging & Testing (0.9)
Holger Krekel, http://twitter.com/hpk42
Pycon Russia, Ekaterinburg, 25th Feb 2013
(XKCD of course)
Very happy to be here!
My background
- started games programming 1986
- studied computer science in the 1990ties
- lots of Assembler, C, C++, distributed programming
- came to Python around 2001
- co-founded PyPy project in 2003, core dev for many years
- author of pytest, tox, execnet libraries
And now for something completely different
Remember
Perl:
There is more than one way to do it.
Python:
There should be one -- and preferably only one -- obvious way to do it.
Second version syndrome
Python3 may have beaten Perl6
(from theregister.co.uk)
but Perl is coming back to haunt us ...
Comprehensive Perl Archive network (CPAN)
cpan.pm comes with base Perl installs:
- downloads/builds/installs and tests packages and deps
- cpan index has hundreds of public mirrors
- uses declarative metadata since 2003
- can verify cryptographic signing of packages
CPAN server / uploading
PAUSE is the upload server for perl packages:
- supports/can enforce versioning semantics
- development releases are mirrored but not indexed
- policy for taking over maintenance of abandoned packages
- provides a mirrored un-indexed "home" area for authors/uploaders for patches, little docs
Python Installer situation Feb 2013
- no download+install tool coming with core Python
- two ways to download/install packages (pip, easy_install):
- pip installs only source distributions
- easy_install also binary distributions
- slow: dependency discovery requires client-side crawling for packages in all download and homepage sites
- no reliable way to test before install
- no cryptographic signature verification
upload/PYPI server interaction today
- pypi.python.org server code not easily deployable on your laptop
- no enforced versioning semantics
- no policy for taking over maintenance of abandoned packages
- brittle protocol: network and server failures of any download/homepage server lead to aborted installs or installation of wrong versions
- metadata through code (setup.py) has tons of problems, has been hard so far to move away from.
and as to mirrors ...
We can stop the comparison already ...
(from www.cavstheblog.com)
Three Lessons learned
Comparing with others can quickly make you unhappy
Perl and Python both not living up to their mantras
There is a lot to improve on
Python Tools and PEPs to improve
- setuptools/distribute projects
- pip/distlib/pkglib/... projects
- PEP386 new version comparison for distutils
- PEP345 Packaging Metadata 1.2
- PEP376 infrastructure for managing distributions locally
- PEP405 virtual environments
- PEP426 Packaging Metadata Version 1.3 (finishing)
- PEP427 Wheel binary package format (APPROVED)
beware: the standardization trap
(one of the many great XKCD comics)
To avoid the standardization trap:
Don't demand that the world changes first
before your tool/idea can be used
(was also my motto when starting PyPy)
Rather: write meta tools
- configure and invoke existing tools
- make it work for most use cases
- enable new facilities/standards from there
example from the testing arena
(you do know the goat is Python's testing mascott, right?)
Python Testing tools
- nose: popular unittest-runner
- py.test: popular generic test runner
- unittest/2: standard library TestCases
- zope.testing: layered-setup unittest-runner
- twisted.trial: running twisted/async test cases
- many other methods, including "make test"
- setuptools has "setup.py test"
Would inventing a new test runner help to standardize testing?
tox: a "meta" test running tool
mission: standardize testing in Python
- configure multiple dependency configs
- install packages in virtualenv environments
- run tests with your tools of choice
- act as a frontend to CI servers
See http://tox.testrun.org for details.
But that's about testing ...
what about packaging/pypi?
I did a little prototyping ...
shown first here at Pycon Russia :)
devpi: a meta tool for packaging activities
devpi serves two purposes:
- it's a new compatible index and upload server
- subcommands for managing release and QA workflows
why a new approach to index serving?
there is no good existing pypi server to build these features:
- multiple indexes and staging (dev->QA->PROD)
- merging of upstream (pypi.python.org) indexes
- server-side crawling of download/homepages
- recording of automated test results with each package
- seemlessly allow offline usage (e. g. on a laptop)
Open Source principle:
talk is cheap, show working code!
$ devpi server
- multiple indexes, organized as ~USERNAME/INDEXNAME
- each index has an upstream index whose packages are seemlessly merged
- each index is "crawl-clean" for clients
- uploading a package to an index will not push it to upstream
- compatible for pip, easy_install and setup.py upload
enough theory - on to a a little demo :)
summary of devpi server demo insights
- upstream merging makes a private overlay index simple
- crawling at server-side makes it easy and fast for installers
- devpi maintains a nginx-servable static view
but:
typing in index server urls, maintaining .pypirc config is burdensome with multiple indexes ... therefore ...
devpi also provides "workflow" subcommands
use set current pypi index install download and install one or more packages upload build and upload packages from a checkout test download and test a package push push a package to another index
$ devpi test
example:
$ devpi test -e py25,py33 mypkg
- download and unpack a package from current index
- call tox for provisioning and performing tests
- stream test results from a test run (currently pytest only, nose/unittest to follow)
$ devpi push
example:
$ devpi push mypkg https://pypi.python.org
- pushes uploaded packages to another index
- currently specified by "PKGNAME-VER"
- in the future this could be done to constraints like "all tests on windows need to pass"
devpi: typical Open Source work flow
- start a localhost index server:
$ devpi server
- use a user-specific index:
$ devpi use http://localhost:3141/~hpk42/dev/
- upload a package from a checkout:
$ devpi upload
- perform tests on multiple machines from multiple users:
$ devpi test PKGNAME
- test results at http://localhost:3141/~hpk42/dev/tests
- when release is ready:
$ devpi push PKG https://pypi.python.org
devpi implementation plan
- to be MIT licensed
- test-driven development
- get early adopters and streamline processes according to real-life usage
- don't aim for standard lib or PEPs (for now)
my priorities
use cases from contracting companies
use cases from my and others open source work
Open areas / future work
- use VCS for index persistence?
- plugin system for upload/serve/test/install
- implement mirroring of index as static files
- support declarative metadata / Wheel format /...
- support server-side PGP signing and verification
- a public, redundant,global web service
- CI: per-commit automatic integration, testing and packaging
devpi going public mid 2013 ...
aiming for a <1.0 public release, to involve early adopters and contributors for finalizing processes and good defaults.
Approach me here at Pycon Russia 2013
and thanks for listening!
No public channels yet, but am going to announce them via: