Fernando Perez

The IPython notebook: a historical retrospective

noreply@blogger.com (Fernando Perez) — Mon, 09 Jan 2012 00:37:00 +0000

On December 21 2011, we released IPython 0.12 after an intense 4 1/2 months of development. Along with a number of new features and bug fixes, the main highlight of this release is our new browser-based interactive notebook: an environment that retains all the features of the familiar console-based IPython but provides a cell-based execution workflow and can contain not only code but any element a modern browser can display. This means you can create interactive computational documents that contain explanatory text (including LaTeX equations rendered in-browser via MathJax), results of computations, figures, video and more. These documents are stored in a version-control-friendly JSON format that is easy to export as a pure Python script, reStructuredText, LaTeX or HTML.

For the IPython project this was a major milestone, as we had wanted for years to have such a system, and it has generated a fair amount of interest online. In particular, on our mailing list a user asked us about the relationship between this effort and the well-known and highly capable Sage Notebook. In responding to the question, I ended up writing up a fairly detailed retrospective of our path to get to the IPython notebook, and it seemed like a good idea to put this up as a blog post to encourage discussion beyond the space of a mailing list, so here it goes (the original email that formed the base of this post, in case anyone is curious about the context).

The question that was originally posed by Oleg Mikulchenklo was: What is the relation and comparison between the IPython notebook and the Sage notebook? Can someone provide motivation and roadmap for the IPython notebook as an alternative to the Sage notebook? I'll try to answer that now...

Early efforts: 2001-2005

Let me provide some perspective on this, since it's a valid question that is probably in the minds of others as well. This is a long post, but I'm trying to do justice to over 10 years of development, multiple interactions between the two projects and the contributions of many people. I apologize in advance to anyone I've forgotten, and please do correct me in the comments, as I want to have a full record that's reasonably trustworthy.

Let's go back to the beginning: when I started IPython in late 2001, I was a graduate student in physics at CU Boulder, and had used extensively first Maple, then Mathematica, both of which have notebook environments. I also used Pascal (earlier) then C/C++, but those two (plus IDL for numerics) were the interactive environments that I knew well, and my experience with them shaped my views on what a good system for everyday scientific computing should look like. In particular, I was a heavy user of the Mathematica notebooks and liked them a lot.

I started using Python in 2001 and liked the language, but its interactive prompt felt like a crippled toy compared to the systems mentioned above or to a Unix shell. When I found out about sys.displayhook, I realized that by putting in a callable object, I would be able to hold state and capture previous results for reuse. I then wrote a python startup file to provide these features and some other niceties such as loading Numeric and Gnuplot, giving me a 'mini-mathematica' in Python (femto- might be a better description, in fairness). Thus was my 'ipython-0.0.1' born, a mere 259 lines to be loaded as $PYTYHONSTARTUP.

I also read an article that mentioned two good interactive systems for Python, LazyPython and IPP, not surprisingly also created by scientists. I say this because the natural flow of scientific computing pretty much mandates a solid interactive environment, so while other Python users and developers may like having occasional access to interactive facilities, scientists more or less demand them. I contacted their authors, Nathan Gray and Janko Hauser, seeking to join forces to create IPython; they were both very gracious and let me use their code, but didn't have the time to participate in the effort. As any self-respecting graduate student with a dissertation deadline looming would do, I threw myself full-time into building the first 'real' IPython by merging my code with both of theirs (eventually I did graduate, by the way).

The point of this little trip down memory lane is to show how from the very beginning, Mathematica and its notebooks (and the Maple worksheets before) were in my mind as the ideal environment for daily scientific work. In 2005 we had two Google SoC students and we took a stab at building, using Wx, a notebook system. Robert Kern then put some more work into the problem, but unfortunately that prototype never really became fully usable.

Sage bursts into the scene

In early 2006, William Stein organized the first Sage Days at UCSD and invited me; William and I had been in touch since 2005 as he was using IPython for the Sage terminal interface. I suggested Robert Kern come as well, and he demoed the notebook prototype he had at that point. It was very clear that the system wasn't production ready, and William was already starting to think about a notebook-like system for Sage as well. Eventually he started working on a browser-based system, and by Sage Days 2 in October 2006, as shown by the coding sprint topics, the Sage notebook was already usable.

For Sage, going at it separately was completely reasonable and justified: we were moving slowly and by that point we weren't even convinced the Wx approach would go anywhere. William is a force of nature and was trying to get Sage to be very usable very fast, so building something integrated for his needs was certainly the right choice.

We continued slowly working on IPython, and actually had another attempt at a notebook-type system in 2006-2007. By that point Brian Granger and Min Ragan-Kelley had come on board and we had built the Twisted-based parallel tools. Using this, Min got a notebook prototype working using an SQL/SQLAlchemy backend. We had the opportunity to work on many of these ideas during a workshop on Interactive Parallel Computation that William and I co-organized (along with others). Like Sage, this prototype used a browser for the client but it tried to retain the 'IPython experience', something the Sage notebook didn't provide.

Keeping the IPython experience in the notebook

This is a key difference of our approach and the Sage notebook, so it' worth clarifying what I mean, the key point being the execution model and its relation to the filesystem. The Sage notebook took the route of using the filesystem for notebook operations, so you can't meaningfully use 'ls' in it or move around the filesystem yourself with 'cd', because Sage will always execute your code in hidden directories with each cell actually being a separate subdirectory. This is a perfectly valid approach and has a number of very good consequences for the Sage notebook, but it is also very different from the IPython model where we always keep the user very close to the filesystem and OS. For us, it's really important that you can access local scripts, use %run, see arbitrary files conveniently, etc., as these are routine needs in data analysis and numerical simulation.

Furthermore, we wanted a notebook that would provide the entire IPython experience, meaning that magics, aliases, syntax extensions and all other special IPython features worked the same in the notebook and terminal. The Sage notebook reimplemented some of these things in its own way: they reused the % syntax but it has a different meaning, they took some of the IPython introspection code and built their own x?/?? object introspection system, etc. In some cases it's almost like IPython but in others the behavior is fairly different; this is fine for Sage but doesn't work for us.

So we continued with our own efforts, even though by then the Sage notebook was fairly mature. For a number of reasons (I honestly don't recall all the details), Min's browser-based notebook prototype also never reached production quality.

Breaking through our bottleneck and ZeroMQ

Eventually, in the summer of 2009 we were able to fund Brian to work full-time on IPython, thanks to Matthew Brett and Jarrod Millman, with resources from the NiPy project. Brian could then dig into the heart of the beast, and attack the fundamental problem that made IPython development so slow and hard: the fact that the main codebase was an outgrowth of that original merge from 2001 of my hack, IPP and LazyPython, by now having become an incomprehensible and terribly interconnected mess with barely any test suite. Brian was able to devote a summer full-time to dismantling these pieces and reassembling them so that they would continue to work as before (with only minimal regressions), but now in a vastly more approachable and cleanly modularized codebase.

This is where early 2010 found us, and then zerendipity struck: while on a month-long teaching trip to Colombia I read an article about ZeroMQ and talked to Brian about it, as it seemed to provide the right abstractions for us with a simpler model than Twisted. Brian then blew me away, coming back in two days with a new set of clean Cython-based bindings: we now had pyzmq! It became clear that we had the right tools to build a two-process implementation of IPython that could give us the 'real IPython' but communicating with a different frontend, and this is precisely what we wanted for cleaner parallel computing, multiprocess clients and a notebook.

When I returned from Colombia I had a free weekend and drove down from Berkeley to San Luis Obispo. Upon arriving at Brian's place I didn't even have zeromq installed nor had I read any docs about it. I installed it, and Brian simply told me what to type in IPython to import the library and open a socket, while he had another one open on his laptop. We then started exchanging messages from our IPython sessions. The fact that we could be up and running this fast was a good sign that the library was exactly what we wanted. We coded frantically in parallel: one of us wrote the kernel and the other the client, and we'd debug one of them while leaving the other running in the meantime. It was the perfect blend of pair programming and simultaneous development, and in just two days we had a prototype of a python shell over zmq working, proving that we could indeed build everything we needed. Incidentally, that code may still be useful to someone wanting to understand our basic ideas or how to build an interactive client over ZeroMQ, so I've posted it for reference as a standalone github repository.

Shortly thereafter, we had discussions with Eric Jones and Travis Oliphant at Enthought, who offered to support Brian and I to work in collaboration with Evan Patterson, and build a Qt console for IPython using this new design. Our little weekend prototype had been just a proof of concept, but their support allowed us to spend the time necessary to apply the same ideas to the real IPython. Brian and I would build a zeromq kernel with all the IPython functionality, while Evan built a Qt console that would drive it using our communications protocol. This worked extremely well, and by late 2010 we had a more or less complete Qt console working:

Over the summer of 2010, Omar Zapata and Gerardo Gutierrez worked as part of the Google Summer of Code project and started building both terminal- and Qt-based clients for IPython on top of ZeroMQ. Their task was made much harder because we hadn't yet refactored all of IPython to use zmq, but the work they did provided critical understanding of the problem at this point, and eventually by 0.12 much of it has been finally merged.

The value and correctness of this architecture became clear when Brian, Min and I met with the Enthought folks and Shahrokh Mortazavi and Dino Viehland from Microsoft. After a single session explaining to Dino and Shahrokh our design and pointing them to our github repository, they were able to build support for IPython into the new Python Tools for Visual Studio, without ever asking us a single question:

In October 2010 James Gao (a Berkeley neuroscience graduate student) wrote up a quick prototype of a web notebook, demonstrating again that this design really worked well and could be easily used by a completely different client:

And finally, in the summer of 2011 Brian took James' prototype and built up a fully working system, this time using websockets, the Tornado web server, JQuery for Javascript, CodeMirror for code editing, and MathJax for LaTeX rendering. Ironically, we had looked at Tornado in early 2010 along with ZeroMQ as a candidate for our communications, but dismissed it as it wasn't really the tool for that job; it now turned out to be the perfect fit for an asynchronous http server with Websockets support.

We merged Brian's work in late August while working on IRC from a boarding room at the San Francisco airport, just in time for me to present it at the EuroSciPy 2011 conference. We then polished it over the next few months to finally release it as part of IPython 0.12:

Other differences with the Sage notebook

We deliberately wrote the IPython notebook to be a lightweight, single-user program that feels like any other local application. The Sage notebook draws many parallels with the google docs model, by default requiring a login and showing all of your notebooks together, kept in a location separate from the rest of your files. In contrast, we want the notebook to just start like any other program and for the ipynb files to be part of your normal workflow, ready to be version-controlled just like any other, stored in your normal folders and easy to manage on their own. Update: as noted by Jason Grout, the Sage notebook was designed from the start to scale to big centralized multi-user servers (sagenb.org, with about 76,000 accounts, is a good example). The notebook that runs in the local user's computer is the same as the one in these large public servers.

There are other deliberate differences of interface and workflow:

We keep our In/Out prompts explicit because we have an entire system of caching variables that uses those numbers, and because those numbers give the user a visual clue of the execution order of cells, which may differ from the document's order.
We deliberately chose a structured JSON format for our documents. It's clear enough for human reading while allowing easy and powerful machine manipulation without having to write our own parsing. So writing utilities like a reStructuredText or LaTeX converter is very easy, as we recently showed.
Our move to zmq allowed us (thanks to Thomas Kluyver's tireless work) to ship the notebook working both on Python2 and Python3 out of the box. The current version of the Sage notebook only works on Python2, in part due to its use of Twisted. Update: William pointed out to me that the upcoming 5.0 version of the notebook will have a vastly reduced dependency on Twisted, so this will soon be less of an issue for Sage.
Because our notebook works in the normal filesystem, and lets you create .py files right next to the .ipynb just by passing --script at startup, you can reuse your notebooks like normal scripts, import one notebook from another or a normal python script, etc. I'm not sure how to import a Sage notebook from a normal python file, or if it's even possible.
We have a long list of plans for the document format: multi-sheet capabilities, LaTeX-style preamble, per-cell metadata, structural cells to allow outline-level navigation and manipulation such as in LyX, improved literate programming and validation/reproducibility support, ... For that, we need to control the document format ourselves so we can evolve it according to our needs and ideas.

As you see, there are indeed a number of key differences between our notebook and the sage one, but there are very good technical reasons for this. The notebook integrates with our architecture and leverages it; you can for example use the interactive debugger via a console or qtconsole against a notebook kernel, something not possible with the sage notebook.

In addition, Sage is GPL licensed while IPython is BSD licensed. This means we can not directly reuse their code, though when we have asked them to relicense specific pieces of code to us, they have always agreed to do so. But large-scale reuse of Sage code in IPython is not really viable.

The value of being the slowest in the race

As this long story shows, it has taken us a very long time to get here. But what we have now makes a lot of sense for us, even considering the existence of the Sage notebook and how good it is for many use cases. Our notebook is just one particular aspect of a large and rich architecture built around the concept of a Python interpreter abstracted over a JSON-based, explicitly defined communications protocol. Even considering purely http clients, the notebook is still just one of many possible: you can easily build an interface that only evaluates a single cell with a tiny bit of javascript like the Sage single cell server, for example.

Furthermore, since Min also reimplemented our parallel machinery completely with pyzmq, now we have one truly common codebase for all of IPython. We still need to finish up a bit of integration between the interactive kernels and the parallel ones, but we plan to finish that soon.

In many ways, our slow pace of development paid off:

We had multiple false starts that helped us much to better understand the hard parts of the problem and where the dead ends would lie.
We were still thinking about this all the time: even when we couldn't spare the time to actively work on it, we had no end of discussions on these things over the years (esp. Brian, Min and I, but also with others at meetings and conferences).
The Sage notebook was a great trailblazer showing both what could be done, and also how there were certain decisions that we wanted to make differently.
The technology of some critical third-party tools caught up in an amazing way: ZeroMQ, Tornado, WebSockets, MathJax, and the fast and capable Javascript engines in modern browsers along with good JS libraries. Without these tools we couldn't possibly have implemented what we have now.

As much as we would have loved to have a solid notebook years ago in IPython, I'm actually happy at how things turned out. We have now a very nice mix of our own implementation for the things that are really within our scope, and leveraging third party tools for critical parts that we wouldn't want to implement ourselves.

What next?

We have a lot of ideas for the notebook, as we want it to be the best possible environment for modern computational work (scientific work is our focus, but not its only use), including research, education and publication, with consistent support for clean and reproducible practices throughout. We are fairly confident that the core design and architecture are extremely solid, and we already have a long list of ideas and improvements we want to make. We are limited only by manpower and time, so please join us on github and pitch in!

Since this post was motivated by questions about Sage, I'd like to emphasize that we have had multiple, productive collaborations with William and other Sage developers in the past, and I expect that to continue to be the case. On certain points that collaboration has already led to convergence; e.g. the new Sage single cell server uses the IPython messaging protocol, after we worked closely with Jason Grout during Sage Days 29 in March 2011 thanks to William's invitation. Furthermore, William's invitations to several Sage Days events, as well as the workshops we have organized together over the years, offered multiple opportunities for collaboration and discussion that proved critical on the way to today's results.

In the future we may find other areas where we can reuse tools or approaches common to Sage and IPython. It is clear to us that the Sage notebook is a fantastic system, it just wasn't the right fit for IPython. I hope this very long post illustrates why, as well as providing some insights into our vision for scientific computing.

Last, but not least

From this post it should be obvious that what today's IPython is the result of the work of many talented people over the years, and I would like to thank all the developers and users who contribute to the project. But it's especially important to recognize the stunning quality and quantity of work that Brian Granger and Min Ragan-Kelley have done for this to be possible. Brian and I did our PhDs together at CU and we have been close friends since then. Min was an undergraduate student of Brian's while he was a professor at U. Santa Clara and the first IPython parallel implementation using Twisted was his senior thesis project; he is now a PhD student at Berkeley (where I work) so we continue to be able to easily collaborate. Building a project like IPython with partners of such talent, dedication, tenacity and generous spirit is a wonderful experience. Thanks, guys!

Please notify me in the comments of any inaccuracies in the above, especially if I failed to credit someone.

Austin trip: IPython at TACC and DataArray summit at Enthought

noreply@blogger.com (Fernando Perez) — Thu, 19 May 2011 07:00:00 +0000

TACC Software Days

I recently had the chance to speak at the UT Austin Texas Advanced Computing Center, during their Fifth Annual Scientific Software Day thanks to a kind invitation by Sergey Fomel and Victor Eijkhout. Since the audience wasn't specifically composed of Python users, I gave a general introduction to Python's role in scientific computing, but then spent most of my time presenting some of the recent work we've been doing on IPython, extending the model of basic interactive computing in what I think are interesting directions with multiple client models and new parallel computing interfaces. Sergey had asked me to provide a somewhat personal account, so the presentation is fairly biased towards my own path (and therefore IPython) through the scipy effort. Since the foucs of TACC is high-performance computing, I hope some of our new functionality on the IPython front will be useful to such users.

There were some very interesting presentations about the FLAME linear algebra library. I did my best to convince Robert van de Geijn of the interest there would be in the scientific Python community for exposing FLAME to Python, possibly as an alternative backend for the linear algebra machinery in scipy. Since FLAME uses a fair amount of code generation, I think the dynamic properties of Python would make it a great fit for the FLAME paradigm. We had some interesting discussions on this, we'll see where this develops...

Datarray summit at Enthought

In conjunction with this visit, we had been trying to organize a meeting at Enthought to make some progress on the datarray effort that was started after last year's scipy conference. I'd like to thank Sergey for kindly allowing my UT-funded visit to extend into the datarray summit. Enthought brought a large contingent of their in-house team (developers and interns like Mark Wiebe of numpy fame), and invited Wes Mc Kinney (pandas) and Matthew Brett (nipy) to participate (as well as others who couldn't make it). In all we had roughly 15 people, with Travis Oliphant, Eric Jones, Robert Kern, Peter Wang and Corran Webster --all very experienced numpy users/developers-- participating for most of the meeting, which was both a lot of fun and very productive. The datarray effort has continued to progress but fairly slowly, mostly due to my being timesliced into oblivion. But I remain convinced it's a really important piece of the puzzle for scientific python to really push hard into high-level data analysis, and I'm thrilled that Enthought is putting serious resources into this.

We spent a lot of time on the first day going over use cases coming from many different fields, and then dove into the API design questions, using the current code in the datarray repository as a starting point. It has become very clear that one key piece of functionality we can't ignore is allowing for axis labels to be integers (without any assumption of ordering, continuity or monotonicity). This proves to be surprisingly tricky to accomodate, because the assumption that integers are indices for array ranging from 0 to n-1 in any dimension goes very deep in all slicing operations, so allowing for integers with different semantics requires a fair amount of API gymnastics.

Not everything is fully settled (and I missed some of the details because I could not stay until the very end), but it's clear that we will have integer labels, and that the main entry point for all axis slicing will be a .axes attribute. This will likely provide the basic named axis attribute-based access, as well as dictionary-style access to computed axes. We will also probably have one method ('slice' was the running name when I left) for slicing with more elaborate semantics, so that we can resolve the ambiguities of integer labeling without resorting to obscure magic overloads (numpy already has enough of those with things like ogrid[10:20:3j], where that "complex slice step" is always fun to explain to newcomers).

We just completed an episode of the inSCIght podcast that contains some more discussion about this with Travis and Wes. I really hope we won't lose the momentum we seem to have picked up and that over the next few months this will start taking shape into production code. I know we need it, badly.

Python goes to Reno: SIAM CSE 2011

noreply@blogger.com (Fernando Perez) — Wed, 06 Apr 2011 06:37:00 +0000

In what's becoming a bit of a tradition, Simula's Hans-Petter Langtangen, U. Washington's Randy LeVeque and I co-organized yet another minisymposium on Python for Scientific computing at a SIAM conference.

At the Computational Science and Engineering 2011 meeting, held in Reno February 28-March 4, we had 2 sessions with 4 talks each (part I and II). I have put together a page with all the slides I got from the various speakers, that also includes slides from python-related talks in other minisymposia. I have also posted some pictures from our sessions and from the minisymposium on reproducible research that my friend and colleague Jarrod Millman organized during the same conference.

We had great attendance, with a standing-room-only crowd for the first session, something rather unusual during the parallel sessions of a SIAM conference. But more importantly, this year there were three other sessions entirely devoted to Python in scientific computing at the conference, organized completely independently from ours. One focused on PDEs and the other on optimization. Furthermore, there were scattered talks at several other sessions where Python was explicitly discussed in the title or abstract. For all of these, I have collected the slides I was able to get; if you have slides for one such talk I failed to include, please contact me and I'll be happy to post them there.

Unfortunately for our audience, we had last-minute logistical complications that prevented Robert Bradshaw and John Hunter from attending, so I had to deliver the Cython and matplotlib talks (in addition to my IPython one). Having a speaker give three back-to-back talks isn't ideal, but both of them kindly prepared all the materials and "delivered" them to me over skype the day before, so hopefully the audience got a reasonable simile of their original intent. It's a shame, since I know first-hand how good both of them are as speakers, but canceling talks on these two key tools would really have been a disservice to everyone; my thanks go to the SIAM organizers who were flexible enough to allow for this to happen. Given how packed the room was, I'm sure we made the right choice.

It's now abundantly clear from this level of interest that Python is being very successful in solving real problems in scientific computing. We've come a long way from the days when some of us (I have painful memories of this) had to justify to our colleagues/advisors why we wanted to 'play' with this newfangled 'toy' instead of just getting on with our job using the existing tools (in my case it was IDL, a hodgepodge of homegrown shell/awk/sed/perl scripting, custom C and some Gnuplot thrown in the mix for good measure). Things are by no means perfect, and there's plenty of problems to solve, but we have a great foundation, a number of good quality tools that continue to improve as well as our most important asset: a rapidly growing community that is solving new problems, creating new libraries and coming up with innovative approaches to computational and mathematical questions, often facilitated by Python's tremendous flexibility. It's been a fun ride so far, but I suspect the next decade is going to be even more interesting. If you missed this, try to make it to SciPy 2011 or EuroSciPy 2011!

Link summary

Slides for all Python talks at the conference.
Pictures from our MS and the reproducible research one.
Slides for the reproducible research MS.

IPython and scientific Python go to Sage Days 29

noreply@blogger.com (Fernando Perez) — Wed, 30 Mar 2011 08:22:00 +0000

Last week I was in Seattle, attending part of of the Sage Days 29 workshop, which had a strong focus on the more numerical/applied topics and the 'scipy ecosystem', as it's funded by William Stein's Sage: Unifying Mathematical Software for Scientists, Engineers, and Mathematicians grant.

I gave a talk that covered topics that are fairly familiar to many in the scipy community, about using Python for numerical work, but was intended to address many from the Sage group who come from a pure mathematics/number theory background. I have posted the PDF of my slides; William recorded the talk and posted it online, as well as posting some pictures from the last day (the anecdote about how I unplugged Colombia from the internet when I was a physics undergrad is from 0:11:20 to 0:14:12):

For me, the bulk of the workshop's focus was to make progress on IPython. Thomas Kluyver, a new core IPython developer, was able to take a week off from his studies and attend from his home in Sheffield; it was great to meet Thomas in person, given the fantastic work he has done on the project in the recent months. He's been extremely productive and got us out from a number of areas were we had been somewhat stuck, so I'm looking forward to a lot more collaboration with him in the future. Min Ragan-Kelley was also present, and Brian Granger unfortunately couldn't make it but joined us for a number of long skype-based discussions. Paul Ivanov, who was officially on matplotlib duty, also helped us a lot with a number of bugs and new pull requests.

In the end, we had a tremendously productive three days. We closed 66 tickets in total. Many of these were triage work, but that's still very useful, and we also did major code review to merge most of the outstanding pull requests, as well as having detailed design discussions on the newparallel branch that will provide zeromq-based high-level parallel tools. We now see light at the end of the tunnel, and I think we'll be able to release the massive amount of work we've been calling IPython 0.11 in a matter of weeks. I'll write in more detail later on that topic, but at least I think those willing to run code from git master can start playing with it now. Things are shaping up quickly, and we'd love to get feedback from early adopters to solidify the APIs.

I'm extremely pleased with this workshop: for IPython the progress was massive, and I think the outcome was similar for numpy, matplotlib and the others. These highly focused development meetings, when held in a good environment (and the UW facilities we had, in their gorgeous new PACCAR hall, were spectacular), can really be amazingly productive. A big thank you to William for the funding and all the organization/logistical work!

Now I just have to get caught up with the other million things that piled up on my inbox/todo in the meantime...

Reproducible Research at the AAAS 2011 meeting in Washington, DC

noreply@blogger.com (Fernando Perez) — Sun, 20 Feb 2011 07:45:00 +0000

Update: added links to other related posts, significantly expanded the section on Git and Github for scientific work.

Link summary: Page with abstracts and slide links, Victoria Stodden's blog, Mark Liberman's blog, my slides and extended abstract, audio (my talk is at time 53:25 to 1:10:47).

At this year's AAAS meeting, currently taking place in DC (in unseasonably warm and sunny weather), Victoria Stodden from the statistics department at Columbia, organized a symposium titled The Digitization of Science: Reproducibility and Interdisciplinary Knowledge Transfer that was very well attended.

Lessons from the Open Source software world

I have tagged this post with "Python" because my take on the matter was to contrast the world of classic research/academic publishing with the practices of open source software development, and what little I know about that (as well as some specific tools I mentioned, like Sphinx), I picked up from the world of open source scientific Python projects I'm involved with, from IPython onwards. My argument is that the tools and practices from the open source community in fact come much closer to the scientific ideals of reproducibility than much of what is published in scientific journals today.

The OSS world is basically forced to do this, because people across the world are collaborating on developing a project from different computing environments, operating systems, library versions, compilers, etc. Without very strong systems for provenance tracking (aka version control), automatic testing and good quality documentation, this task would be simply impossible. But many of these tools can be adapted for use in everyday scientific work; for some use cases they work extremely well, for others there's still room for improvement, but overall we can and should take these lessons into everyday scientific practice.

In my talk, I spent a fair amount of time discussing the Git version control system, not in terms of its technical details, but rather trying to point out how it should not be viewed just as a tool for software development, but instead as something that can be an integral part of all aspects of the research process. Git is a powerful and sophisticated system for provenance tracking that automatically validates data integrity by design: Linus Torvalds wanted to ensure that every commit operation is signed with a hash of its contents plus the hash of its dependencies (for details on this, his sometimes abrasive Google Tech Talk about Git is an excellent reference). This simple idea ensures that a single byte change anywhere in the entire repository can be detected automatically. I keep an informal page of Git resources for those looking tot get started.

I use Git for just about all my activities at the computer that require manually creating content, with repositories not only for research projects that involve writing standalone libraries, but also for papers, grant proposals, data analysis research, and even teaching. Its distributed nature (every copy of the repository has all the project's history) makes it automatically much more resilient to failures than a more limited legacy tool like Subversion and its strong branching and merging capabilities make it great for exploratory work (something that is painful to achieve with SVN). Git is also the perfect way to collaborate on projects: all members have full versioning control, can commit work as they need it, and can make visible to collaborators only what they deem ready for sharing (this is impossible to do with SVN). Writing a multiauthor paper or grant proposal with Git is a far saner, more efficient and less error prone process than the common madness of emailing dozens or hundreds of attachments every which way between multiple people (for those who think Dropbox suffices for collaborative writing: it's like using a wood saw for brain surgery; Dropbox is great for many things and I love it, but it's not the tool for this problem). I have also used Git for teaching, by creating a public repository for all course content and individual repositories for each student that only the student, the teaching assistants and myself can access. This enables students to fetch all new class content with a simple:

git pull

instead of clicking on tens of files in some web-based interface (the typical system used by many universities). A single clone operation can reconstruct the entire class repository on another computer if they need to use it in more than one place or lose their old copy. And when it's time to submit the homework, instead of emailing or uploading anything, all they need to do is:

git push

and the TAs have immediate access to all their work, including its development history. In this manner, not only is the process vastly smoother and simpler for all involved, but the students learn to use version control as a natural tool that is simply part of their daily workflow.

I also tried to highlight the role played by the GitHub service as an enabler of collaboration. While Git can be used (and it is extremely useful in this mode) on a single computer without any server support, the moment several people want to share their repositories for collaborative work, some kind of persistent server is the best solution. GitHub, a service that is free for Open Source projects and that offers paid plans for non-public work, has a number of brilliant features that make the collaboration process amazingly useful. Github makes it trivial for new contributors to begin participating in a project by forking it (i.e. getting their personal copy to work on), and if they want their work to be incorporated into the project, they can make a pull request. The original authors then review the proposed changes, comment on them (including making line-specific comments with a single click), and once all are satisfied with the outcome, integrate them. This is effectively a public peer review system that, while created for software development, can be equally useful for collaborative authorship of a research project.

I should add, however, that I think there's still room for improvement regarding Git as a tool for pervasive use in the scientific workflow. As much as I absolutely love Git, it's a tool tailored for source code tracking and its atomic unit of change is the line of code. As such, it doesn't work as conveniently when tracking for example changes in a paper (even if written in TeX), where a small change can reflow a whole paragraph, showing a diff that is much larger than the real change. In this case, the "track changes" features of word processors actually work better at showing the specific changes made (despite the fact that I think they make for a horrible interface for the overall workflow) [Note: in the comments below, a reader indicates that the --word-diff option solves this problem, though I think it requires a very new version of Git, 1.7.2 at least. But it's fantastic to see this kind of improvement being already available]. And for tracking changes to binary files, there's simply no meaningful diff available. It would be interesting to see new ideas for improving something like git for these kinds of use cases.

I wrapped things up with a short mention of the new Open Research Computation journal, where Victoria and I are members of the editorial board, as well as several well-known contributors to the scientific Python ecosystem, including Titus Brown, Hans-Petter Langtangen, Jarrod Millman, Prabhu Ramachandran and Gaël Varoquaux.

Scientific Python tools use rising in education

noreply@blogger.com (Fernando Perez) — Wed, 16 Dec 2009 07:12:00 +0000

It's pretty clear to me that Python is rapidly growing in acceptance as a computational platform at universities everywhere. I recently heard from Josh Bloom at UC Berkeley astronomy that his proposal for a short 'boot camp' course at the beginning of the Fall semester was approved. This is excellent news, last year I taught something similar for neuroscience students and postdocs, and I'm glad to see the campus adopting python further as a key component of the computational training the science students receive.

John Hunter and I just completed a few days ago teaching another such workshop at the Claremont Colleges, supported by an NSF grant that John Milton, (J. Hunter's PhD advisor) has for exposing undergraduates to a number of research-related experiences. This grant supports summer research internships where two undergrads visit together a research lab away from their home campus to work independently on a project, as well as our teaching of scientific python tools (we were also there in 2008 and hopefully will continue next year).

I have to say that I really enjoy teaching this type of workshop, especially now that the core tools are fairly mature, installation isn't the problem it used to be, and we have a chance of presenting interesting examples to the students from the very beginning. By now we've taught a number of these at various labs and universities, and I think we've found a good workflow. John is a phenomenal lecturer, with a real knack for illustrating interesting concepts on the fly, in a very natural manner that is honestly similar to how exploratory work is actually done in real research, where you run a bit of code, plot some results, print a few things, code some more, etc. In the picture above, he was working through some of the matplotlib image tutorial, which gave us an opportunity to find and fix a small bug in how the luminance histogram was being computed (the current form is correct). Every one of these students probably has a digital camera these days (if nothing else, in a cell phone), so an example like this is a great way to connect something they are used to with mathematical and programming concepts.

Here, John gave a great illustration of random numbers and simple statistics. He built up interactively a simulation of random walks, working up from a single (one-dimensional) walker to a group of them, comparing the simulation results with the analytical predictions for various quantities (like mean and variance), and also explaining to the students how these squiggly lines could be considered a model of price fluctuations over time. In this manner, he connected the somewhat abstract statistical concepts to something the students could relate to, namely the risk of an investment making a profit or a loss after a certain period.

We also talked about FFTs, dynamical systems, error analysis, data formats, and a few other things. It was very encouraging to have most of the students return on the second day, considering how this was a completely optional activity for them, covering an entire weekend (with morning lectures both days) right before they had to start diving into their finals. But they were a smart and enthusiastic bunch, and I hope that the workshop gave them some useful starting points they can then develop on their own as they get involved in research projects.

These are just two examples of how we are now seeing Python's acceptance in university computing growing. We have a lot of work still ahead of us, but it's really encouraging to see how far we've come from the days when building scipy was a black art, IPython was little more than a different prompt for the python shell and matplotlib could only do basic line plots in a GTK window. We now have tools that provide a complete computational environment for teaching and research, and things are only getting better...

Guido van Rossum at UC Berkeley's Py4Science

noreply@blogger.com (Fernando Perez) — Fri, 06 Nov 2009 11:46:00 +0000

Update: Quick links

Video of the entire session. The talks cover the first 50 minutes, and Guido's part starts at the 54 minute mark.
A page with all the slides on my site.
Guido also blogged his impressions.
Blog posts by Jarrod Millman and Matthew Brett.

On November 4 2009, we had a special session of our informal Py4Science seminar where Guido van Rossum visited for an open discussion regarding the uses of the Python language in scientific research. Guido had expressed his interest in discussing the work that various scientists do with Python but mentioned that instead of a formal talk, he would prefer a format that allowed for more interaction with the audience. We agreed that a good plan would be for us to present a rapid-fire sequence of very short talks highlighting multiple projects so that he could get a good "high altitude" view of the scientific python landscape, leaving then ample time for discussions with the audience.

Guido has already posted his impressions of the visit, and so have my colleagues Jarrod Millman and Matthew Brett, so I'll try to provide a complementary view here without too much repetition.

We gathered for lunch first with a small group and had a very interesting discussion on various topics; we had a chance to talk in some detail about the transition to Python 3 for Numpy, something a number of people have started to think about seriously. Numpy is pretty much a 'root dependency' for so many scientific projects, that until it makes the jump it will be very difficult for anyone else in science to seriously consider Python 3. Understandably, Guido would like to see some movement from our community in this direction, and he offered useful guidance. In particular, he said that in the core Python-dev team there might be enough interest that if we ask for help there, we might find developers willing to pitch in and provide some assistance. He also expressed some disappointment that PEP 3118, which was accepted with our interests in mind, still hadn't been fully implemented. Limited manpower is the simple reason for this situation, but fortunately Jarrod mentioned that there's a good plan to address this in the near future.

I had a chance to quiz Guido about something I've wondered for a while: Python has unusually good number types in its core (arbitrary length integers, extended precision decimals and complex numbers), but the integers divide either into the integers (the truncating behavior of Python 2.x) or into the floats (in 3.x). While the 3.x division is an improvement, I would have really liked to see Python go to native rationals; for one thing, this would make the Sage 'language' (by which I mean the extensions Sage makes to pure Python) behave like Python in algorithms involving integers, eliminating a recurring source of confusion between the two. I also happen to think it would be a 'better' behavior, though there are valid reasons also for someone to expect a more 'calculator-like' answer to divisions like 1/2, who might be annoyed if they get 1/2 back instead of 0.5. While obviously such changes will not be on the table for a long while (and I should say here that I am very happy with the planned moratorium to core language changes, as I hope that will allow a more focused effort on the needs of the standard library), it was interesting to hear Guido's approach to this question as one that could be handled via overloadable literals rather than a change of integer semantics. I'd never thought of that, but it's an intriguing idea... Something to think about for when we start looking at Python 4000 :)

We then headed over to the official presentation, where we managed to cram 14 talks in 50 minutes and then had a full hour of open conversation with Guido, where the audience asked him questions on a number of topics. You can see a complete video of the entire session: after 50 minutes of talks we have a transition, and Guido's section starts at the 54 minute mark. On my website I have posted a page with all the slides for these mini-talks.

I presented an overview introduction and material on behalf of 4 others who were not present locally, but coincidentally, William Stein of Sage fame was on campus to give a talk in the same building almost at the same time, and he could present the Sage slides directly. Ondrej Certik from SymPy was able to make the trip from Reno, completing our out-of-town speakers. The other 7 presentations were from a number of local speakers (from various departments at UC Berkeley and Lawrence Berkeley National Laboratory, just up the hill from us).

I have received very good feedback from several people, and I am really thankful to all the speakers for being so attentive to the time constraints, which let us pack a lot of material while leaving ample time for the discussion with Guido. My intention with this was to really provide Guido with a broad understanding of how significant Python's penetration has been in scientific computing, where many different projects from disciplines ranging from computer science to astronomy are relying heavily on his creation. I wanted both to thank him for creating and shepherding such a high-quality language for us scientists, and to establish a good line of communication with him (and indirectly the core python development group) so that he can understand better what are some of the use patterns, concerns and questions we may have regarding the language.

I have the impression that in this we were successful, especially as we had time after the open presentations for a more detailed discussion of how we use and develop our tools. Most of us in scientific computing end up spending an enormous amount of time with open interpreter sessions, typically IPython ones (I started the project in the first place because I wanted a very good interactive environment, beyond Python's default one), and in this work mode the key source of understanding for code are good docstrings. This is an area where I've always been unhappy about the standard library, whose docstrings are typically not very good (and often they are non-existent). We showed Guido the fabulous Numpy/Scipy docstring editor by Pauli Virtanen and Emmanuelle Gouillart, as well as the fact that Numpy has an actual docstring standard that is easy to read yet fairly complete. I hope that this may lead in the future to an increase in the quality of the Python docstrings, and perhaps even to the adoption of a more detailed docstring standard as part of PEP 8, which I think would be very beneficial to the community at large.

In the end, putting all this together took me a lot more time than I'd originally planned (I think I've had this same problem before...), but I am very pleased with the results. Python has become a central tool for the work many of us do, and I am really happy to establish a good dialogue with Guido (and hopefully other core developers), which I'm sure will have benefits in both directions.

CSE'09 article in SIAM News

noreply@blogger.com (Fernando Perez) — Mon, 07 Sep 2009 08:30:00 +0000

CSE'09 article in SIAM News

This isn't fresh-off-the-oven news, but it's still on the main SIAM News page, so I'll mention it as I am trying to do a better job of keeping up with this blog...

As I mentioned on a previous post, our three part minisymposium on Python at the SIAM CSE09 conference was fairly well received. But even better, SIAM asked us to write up an article for publication in the SIAM News bulletin. I was really pleased with this, as SIAM News reaches a very broad international audience and is actually read by people. I often find very interesting material in it, as the publication hits a very good balance of quality and not being overly specialized (as we all drown in work, we tend to focus only on very narrow publication lists for everyday reading,
unfortunately).

Randy, Hans-Petter and I drafted up an article, and we received great feedback from all of the presenters at the minisymposium, including figure contributions by John Hunter and Hank Childs (who was at LLNL at the time, and I'm delighted to see is just up the hill from me at LBL).

Our article is now available in HTML online at the SIAM site, and I also have a PDF version that is more suitable for printing in case you are interested (note that my version is missing the very last corrections from the SIAM editor, but the differences should be minor).

Scipy advanced tutorials results

noreply@blogger.com (Fernando Perez) — Wed, 01 Jul 2009 06:35:00 +0000

We recently conducted a poll on Doodle, soliciting feedback on the preferred topics for the advanced track, which is meant to contain 2 days with 8 2-hour sessions focusing on one specific topic at a time. The table below shows the complete results, which I've only sorted for convenient viewing and anonymized (the raw Doodle output contains the names given by each person voting). If anyone would like the raw spreadsheet, just drop me a line.

The score was computed as #yes-#no (i.e., yes=+1, neutral=0, no=-1), from a total of 30 responses, and the results are in the table below, ranked from highest to lowest score. In my personal opinion, all the topics offered would have made for very good and interesting tutorials, but the point of asking for feedback is obviously to follow it to some degree, which we will now do.

I think it's worth noting --though not particularly surprising-- that the ranking roughly follows the generality of the tools: matplotlib and numpy are at the top, with finite elements and graph theory at the bottom. While I personally use NetworkX and love it, it's a specialized tool that for many probably offers no compelling reason to learn it, while pretty much every single numerical python user needs numpy and matplotlib. We are now in the process of contacting possible speakers for the top topics, and will communicate on the mailing list a final list of topics once we have confirmed speakers for all.

	Yes	Neutral	No	Score	Rank
Advanced topics in matplotlib use	18	10	2	16	1
Advanced numpy	18	10	2	16	2
Designing scientific interfaces with Traits	15	11	4	11	3
Mayavi/TVTK	13	11	6	7	4
Cython	14	8	8	6	5
Symbolic computing with sympy	15	6	9	6	6
Statistics with Scipy	9	15	6	3	7
Using GPUs with PyCUDA	13	7	10	3	8
Testing strategies for scientific codes	11	11	8	3	9
Parallel computing in Python and mpi4py	12	8	10	2	10
Sparse Linear Algebra with Scipy	9	12	9	0	11
Structured and record arrays in numpy	8	14	8	0	12
Design patterns for efficient iterator-based scientific codes	9	7	14	-5	13
Sage	8	6	16	-8	14
The TimeSeries scikit	4	13	13	-9	15
Hermes: high order Finite Element Methods	6	9	15	-9	16
Graph theory with NetworkX	5	9	16	-11	17

Python at the SIAM CSE'09 meeting

noreply@blogger.com (Fernando Perez) — Mon, 09 Mar 2009 20:48:00 +0000

After the success of last year's Python minisymposium at the annual SIAM meeting, this year we had a repeat: Simula's Hans-Petter Langtangen (author of the well-known Python Scripting for Computational Science), U. Washington's Randy LeVeque and I co-organized another minisymposium on Python for Scientific computing.

At the Computational Science and Engineering 2009 meeting, held in downtown Miami March 2-6, we had again 3 sessions with 4 talks each (part I, II and III), with a different mix of speakers and focus than last year. While last year we spent some effort introducing the language and to a certain extent justyfing its use in real-world scientific work, we felt that this time, the growth of the many python projects out there speaks for itself and that we should instead turn our attention to actual tools and projects useful for specific work. Thus, we had no 'why python for science' talk, although obviously most speakers spent some time providing this kind of information in the context of their own projects.

I think that this is the right path to follow: for a number of years, many of us have been developing tools and justyfing Python as a viable alternative to tools such as Matlab or IDL, but we need to start moving away from that mode. There is no doubt in my mind now that, while still immature in certain areas, Python is a credible, production alternative to said packages. In many contexts it can actually be far superior to its propietary and expensive counterparts, as many of us have found in practice. So hopefully as we move forward, we will do less of 'you can use python for scientific work' and more of 'here is a great scientific project/tool that happens to be implemented using Python'. Eventually, I hope we will not have any Python-specific sessions at scientific meetings, just like we don't see any "Fortran for scientific computing" special sessions.

This was a smaller meeting than the annual one, but attendance for our sessions was good. There were interesting discussions and questions from audience members who were obviously new to the tools and curious to learn more. Personally, I had a great time both catching up with some familiar faces and meeting some new ones, all of whom I hope will become more regular contributors to the greater ecosystem of scientific python tools. Ondrej Certik, always full of energy, promptly posted his own report, along with some video from where we were staying.

An aside: when I tried to book the conference hotel all available rooms at the conference rates were taken, so a plan B was in order. I found online a room across the street that was cheaper than the Miami Hilton's conference rate and rented it. It turned out to be a great condo on the 50th floor of this tower, with free high quality internet, lots of work space we could use and a gorgeous view. Lesson learned: in this economic crisis, shop around: we got a great place and still saved money over the normal conference rate... The conference hotel is just barely visible, in the lower right hand corner of this shot: I am currently in the process of collecting slides from the speakers; check that page later if you are interested in any talk for which I haven't posted them yet: I'll continue to update it as I receive them. Until the next meeting... [ Image credits: John D. Hunter and F.P., full album available here ]

Book review: Expert Python Programming

noreply@blogger.com (Fernando Perez) — Mon, 09 Mar 2009 19:06:00 +0000

Update: I've slightly modified the language of this review, which as my wife correctly pointed out to me, was unnecessarily harsh. While I stand by my previous evaluation of the book, I think the same things can be said in a more constructive tone.

While this isn't strictly a SciPy post, I've already have a few questions about this book, so I guess I'll tag it as 'scipy' as well, for those interested. I recently reviewed the book Expert Python Programming by Tarek Ziadé. While not aimed at a scientific audience, the book covers a number of topics that we frequently discuss on the Numpy and Scipy lists (such as documentation and testing, workflows, API decisions, etc). Since I really prefer to write longer text in reST using Emacs than in a blog editor, I've posted the review over at my static site. Feel free to head over there if you are interested in the full review, I've only reproduced here the summary:

Summary Expert Python Programming covers a list of very interesting topics regarding real-world development using Python. It assumes a reader who already knows the basics of the language and covers a number of important topics, both in the more advanced parts of the language and in terms of developing applications using Python. There is a strong emphasis on agile development practices, testing, documentation, application life cycle management and other aspects of 'real world' work. The list of topics covered is excellent, and the book is well printed and bound.

However, unfortunately it suffers from rather poor editing throughout, with a broken idea flow that makes for choppy reading. Very few ideas are properly developed, as the book relies excessively on code snippets, bullet lists and stand-out info boxes. Ultimately, this gives it more the feel of a set of notes than that of a coherent volume. This should not be read as an indictment of the book: the table of contents alone is a list of 'right things to do' when using Python, and there is a great deal of useful material in all of the chapters.

If you are looking for reference material, links and starting points for further reading, Expert Python Programming can be an excellent resource and well worth your investment. However, if you are searching for a text that develops complex concepts at length, delving into details and subtleties, it might not be the ideal tool for you. I hope this provides a useful picture of the qualities of this book.

Python tools for science go to SIAM: a trip report

noreply@blogger.com (Fernando Perez) — Mon, 28 Jul 2008 19:22:00 +0000

Update (2008.08.06): I have posted the slides I have available from the minisymposium described below.

The 2008 SIAM annual meeting was held July 7-11 in San Diego, and we had a healthy presence of Python-based tools for scientific computing. Randy LeVeque from U. Washington and I co-organized a 3-part minisymposium entitled Python and Sage: Open Source Scientific Computing. You can see the list of talks for parts I, II and III online. I should add that for the talks where we had cancellations, we ended up with excellent impromptu replacements by Bill Hart from Sandia National Labs and Travis Oliphant from Enthought.

Our sessions were well attended (especially considering the very large number of parallel sessions at the conference) and generated lively discussion. There's clearly a need in the scientific and mathematical community for open source, high level tools that integrate well with existing high performance codes written in Fortran, C or C++, while providing interactivity, visualization, access to the network, etc. This is the kind of idea that many of us in the SciPy and Sage worlds have been preaching for a while, but now the response from the wider community is much more positive than before (admittedly, years ago the tools were immature enough that they were very much of the "user assembly required" variety).

I am extremely pleased to report that we were chosen for the conference highlights page (many thanks to John Hunter, of matplotlib fame, for the great screenshot)! Given that this page mostly lists the invited plenaries and other special events, I am very happy about this (SIAM had literally hundreds of talks from parallel sessions to choose from and only two of those were highlighted). I think it is both a recognition of the work done by all our speakers, and of the relevance of the topic today. Thanks again to all who contributed. We should take this level of interest as both a recognition an a challenge: as more scientists become interested in these tools for their everyday research, the pressure on high quality documentation, easy installation and deployment, good tutorials, etc, will only increase.

Everyone is working hard on this, and in particular it's worth mentioning the excellent work that is being done this summer as part of the NumPy/SciPy Documentation Marathon 2008, where a lot of new documentation has been added to NumPy. This is an effort where anyone can contribute! The infrastructure is just a wiki where you can register to edit docs, and developers will later review and merge your contributions into the core project for the next release.

So let's all keep up the momentum going, and I hope to see many more interested faces, both new and old, at the upcoming SciPy 2008 conference at Caltech in August!

Starting off...

noreply@blogger.com (Fernando Perez) — Mon, 28 Jul 2008 18:46:00 +0000

Well, at some point I suppose I had to get a blog... This will be mostly a way to post occasional updates on projects I'm involved with for others to pick up via aggregators. This blog is registered as part of Planet SciPy, and I'll also keep updates here on IPython, neuroimaging in Python, and we'll see what else develops. So, welcome!

Fernando Perez

The IPython notebook: a historical retrospective

Austin trip: IPython at TACC and DataArray summit at Enthought

TACC Software Days

Datarray summit at Enthought

Python goes to Reno: SIAM CSE 2011

IPython and scientific Python go to Sage Days 29

Reproducible Research at the AAAS 2011 meeting in Washington, DC

Lessons from the Open Source software world

Other presentations

Scientific Python tools use rising in education

Guido van Rossum at UC Berkeley's Py4Science

CSE'09 article in SIAM News

CSE'09 article in SIAM News

Scipy advanced tutorials results

Python at the SIAM CSE'09 meeting

Book review: Expert Python Programming

Python tools for science go to SIAM: a trip report

Starting off...