Alright, an update on HyperKitty’s status is looooong overdue. Actually, I should have written this post a week or two ago, but I had hosting issues which forced me to migrate my server at the last minute, and getting it back on was a bit painful… Done now. So, about Mailman3.
There’s been a lot of activity during the Christmas holidays. Barry, the Mailman project leader, decided to merge a branch porting Mailman 3 to Python 3. And since it’s a large amount of code, Mailman3 will not be bilingual: it won’t run on Python 2.X. So yeah, in a couple days, we went from “Python3 compatibility would be nice for Mailman 3 but it’s not a blocker for the release” to “Mailman3 will only run on Python 3”.
I understand Barry’s motives, Python 3 is really superior, especially in the email and unicode handling area which are crucial to Mailman 3. However, this means that a part of HyperKitty’s structure has to change. In the Mailman 3 architecture, an archiver is a Python plugin which runs in its own subprocess, but with the same interpreter. Our plugin calls KittyStore, the HyperKitty backend. As a result, KittyStore must be Python3-compatible. Then, HyperKitty (the Django app) also makes programmatic calls to KittyStore to retrieve the content, update the votes, etc. The API between the two is somewhat isolated, but it would not make a lot of sense to have a fully separated REST protocol between the two, because the requests are just very diverse and complex. So KittyStore is used as a library, which means that if KittyStore is Python 3, HyperKitty must be Python 3 too.
Porting HyperKitty to Python3 is not really an option, because we want it to be installable on an existing Django server, with various other applications that may not be ported yet. So we decided to introduce a level of protocol isolation between Mailman and KittyStore, since that’s where the API is the simplest. REST is what is most logical here, it’s used elsewhere in the Mailman architecture and it’s very flexible. As a consequence, KittyStore has to grow a REST server, be run as an independant process, etc.
After working on that a bit, I realized that the separation between HyperKitty and KittyStore made less and less sense. It was originally a design decision that kept a lot of possibilities opened, for example having other applications (like an NNTP server) access KittyStore, even maybe merging KittyStore into Mailman itself, etc. These things haven’t happened and the chances for them to happen are pretty slim: it would be more logical to access the archive store via REST calls to HyperKitty, which holds more information, and would be more stable and scalable than an internal REST server.
Merging HyperKitty and KittyStore also makes a lot of things easier, as I have explained on HyperKitty’s development mailing-list. Unfortunately, it’s a large amount of refactoring work. But it’s “only” refactoring, and it will make things simpler, thus easier to understand for newcomers and admins. It also means that my automated installer script, mailman-bundler, is broken at the moment, and can’t be fixed until Mailman3 and HyperKitty are compatible again.
Some good news now. Last summer, the two big changes that were under consideration for Mailman 3 were the change of ORM (Storm to SQLAlchemy) and Python 3 porting. Only the ORM change was deemed necessary for the release at that time, but now that I think of it, I should have seen it coming… As a consequence, only a couple of minor bugs now block Mailman 3 from being released, and the team is aiming for a full release of the Mailman 3 suite at PyCon in April. I’ll be there to lend, so meet us if you can. This is the last mile!