-->

Tuesday, November 20, 2012

Back from PyCon Canada 2012

I just got back a few days ago from the 2012 edition of PyCon Canada, which was a great success. I wanted to thank the team who invited me for a fantastic experience: Diana Clarke who as conference chair did an incredible job, Greg Wilson from Software Carpentry with whom I had a chance to interact a lot (he already has a long list of ideas for the IPython notebook in teaching contexts we're discussing), Mike DiBernardo and the rest of the PyConCa team. They ran a conference with a great vibe and tons of opportunity for engaging discussion.

Thanks to Greg I also had a chance to give a couple of more academically-oriented talks at U. Toronto facilities, both at the Sunnybrook hospital and their SciNet HPC center, where we had some great discussions. I look forward to future collaborations with some of the folks there.

The PyConCa kindly invited me to deliver the closing keynote for the conference, and I tried to provide a presentation on the part of the Python world that I've been involved with, namely scientific computing, but that would be of interest to the broader Python development community in attendance here. I tried to illustrate where Python has been a great success for modern scientific research, and in doing so I took a deliberately biased view where I spent a good amount of time discussing IPython, which is how I entered that world in the first place.

This is the video of the talk:


and here are the accompanying slides.

I'm too far behind to do a proper recap of the conference itself, but I want to mention one of the highlights for me: a fantastic talk by Elizabeth Leddy, a prominent figure in the Plone world, on how to build sustainable communities. She had a ton of useful insight from in-the-trenches experience with the Plone foundation, and I fortunately got to pick her brain for a while after the talk on these topics. As we gradually build up somewhat similar efforts in the scientific Python world with NumFOCUS, I think she'll be a great person for us to bug every now and then for wisdom.

IPython at the sprints

I managed to stay for the two days of sprints after the end of the main conference, and we had a great time: a number of people made contributions to IPython for the first time, so I'd like to quickly recap here what happened.

Nose extension

Taavi Burns and Greg Ward of distutils fame fought hard on a fairly tricky but extremely useful idea on a suggestion from Greg Wilson: easy in-place use of nose to run tests inside a notebook. This was done by taking inspiration (and I think code) from Catherine Devlin's recent work on integrating doctesting inside the notebook.

The new nose extension hasn't been merged yet, but you can already get the code from github, as usual. Briefly (from Taavi's instructions), this little IPython extension gives you the ability to discover and run tests using Nose in an IPython Notebook.

You starty with a cell containing:

%load_ext ipython_nose

Then write tests that conform to Nose conventions, e.g.

  def test_arithmetic():
      assert 1+1 == 2

And where you want to run your tests, you add a cell consisting of

  %nose

and run it: that will discover your test_* functions, run them, and report how many passed and how many failed, with stack traces for each failure.

WebGL-based 3d protein visualization

RishiRamraj, Christopher Ing and Jonathan Villemaire-Krajden implemented an extremely cool visualization widget that can, using the IPython display protocol, render a protein structure directly in a 3d interactive window. They used Konrad Hinsen's MMTK toolkit, and the resulting code is as simple as:

from MMTK.Proteins import Protein
Protein('insulin')

You can see what the output looks like in this short video shot by Taavi Burns just as they got it working and we were all very excited looking at the result; the code is already available on github.

I very much look forward to much more of this kind of tools being developed, and in fact Cyrille Rossant wasted no time at all building off this to provide fast 2-d visualizations rendered via WebGL with his Galry library:


Software Carpentry

In addition to the Nose extension above, Greg Wilson had a ton of ideas on things that could be added to the notebook that he thinks would help in the context of teaching workshops such as those that Software Carpentry presents. Their audience is typically composed of beginning programmers, scientists who may be experts in their discipline but who have little to no formal computational training and are now tasked with managing often quite complex computational workflows. Since SWC recently announced they would be switching to the notebook as their main teaching platform, they obviously are thinking deeply about how to make the best use of it and where the notebook can improve for this kind of use case.

These are still conversations that I hope will turn soon into concrete issues/code repositories to begin testing them, but that kind of validated testing is very useful for us. Since at this point we have too many feature requests from multiple fronts to be able to satisfy them all, we are trying to focus on ensuring that IPython can support indivdual projects building their own custom tools and extensions. We can't possibly merge every last idea from every front into IPython, but we can work to ensure it's a flexible and coherent enough foundation that others can build their own highly customized experiences on top. Once these get widely tested and validated, it may be that pieces are clearly of generic enough value to percolate into the core, but in the meantime this approach means that other projects (SWC being just one example among many) don't need to wait for us to add every feature they need.

What we will focus on will be on addressing any limitations that our architecture may have for such extensibility to work well, so the life of third party projects isn't a fight against our interfaces.

A first-time contributor to open source

Last, but not least, I had the great experience of working with David Kua, a CS student from U. Toronto who had never made a contribution to open source and wanted to work on IPython. Right during the sprints we were able to merge his first pull request into nbconvert, and he immediately started working on a new one for IPython that by now has also been merged.

That last one required that he learn how to rebase his git repo (he had some extraneous commits originally) and go through a fair amount of feedback before merging: this is precisely the real world cycle of open source contributions. It's always great to see a brand new contributor in the making, and I very much look forward to many more work from David, whether he decides to do it in IPython or in any other open source project that catches his interest.

Note

Since I am now writing all my posts as IPython notebooks (even when there's no code, it's a really nice way to get instant feedback on markdown), you can get the notebook for this post from my repo.