Aller au contenu | Aller au menu | Aller à la recherche

Informatique › Développement

Mes petits programmes faits main :)

Fil des billets - Fil des commentaires

mer. 03 août '16

Flock 2016, day 1

The Fedora users and developers conference, called Flock, is our annual gathering to update everyone on the direction the distribution is going, to talk and hack together, and have fun together. This year it was held in Krakow, Poland. I had the chance to go there with most of the Fedora Infrastructure team, and gave a talk about the HyperKitty deployment we had this year.

Lire la suite...

jeu. 21 mai '15

Mailman 3 is out

….aaaaaaand it happened. The Mailman team just released version 3.0. Three weeks ago, OK I’m late, but that’s nothing on a Mailman-scale ;-)

Here’s the official announcement. You’ll learn there that the first alpha release for 3.0 was published a little over 7 years ago. That’s a “seven”, not a “one”. I won’t paraphrase the announcement, but I’d like to thank all the people who made this release possible over the years. Mailman 3 is complete rewrite of the popular mailing-list server that is used all over the world to create communities. The new software architecture is really sound and clean, and makes it possible to create great addons. Such as HyperKitty. HyperKitty 1.0 was released at the same time as Mailman 3.0, and is part of the Mailman suite.

People are already starting to install this new version and think about migrating their old list servers. There’s a piece of software called Mailman-bundler to help you setup the whole Mailman suite quickly and test it out, but for a real production server you may want to rely on individual packages and custom configuration.

Come talk to us on IRC (#mailman on freenode) if you have any issue setting it up!

Thanks again to the wonderful Mailman team and all those who contributed over the years. Mailman 3 is going to be legendary :-)

sam. 11 avr. '15

First day at PyCon 2015 in Montréal

The day started with a very interesting keynote about volunteer citizen development solving problems where the government falls short.

Then I met some people of the Mailman team, Florian first, then Abhilash (for the first time), and Barry. Barry had just pushed some code in his subpolicy branch that he wanted me to review, and since I didn’t find a talk that inspired me a lot this morning, I spent a couple hours reviewing his new code and making small changes.

Afterwards I realized there were two talks I would have liked to attend, but the good thing with PyCon is that all talks are recorded, so I’ll just watch the videos later.

I had lunch with Luke and Ralph of Fedora engineering fame, and then went to a talk on Python 2 & 3 compatibility, that went over the differences and the tooling we have to overcome them. Since Mailman3 is Python 3 and HyperKitty is Python 2, there are a few intermediate libraries where knowing this will be useful.

Then I went to talks on services and concurrency, which I think will be useful in a Fedmsg-enabled Fedora infrastructure.

After the break, I went to two talks on REST APIs. HyperKitty has a REST API, and it will prove very useful for integration when it’s deployed in Fedora and elsewhere. I want to make sure the API is useful and follows best practices.

The last talk, by Jeff Schenck of, was really interesting. With the dawn of full UI Javascript libraries like Angular.js and Backbone.js, REST APIs today are commonly used to serve data to your own application running in the client’s browser, and not to external developers only. As a consequence, an interface that used to be internal and run in a trusted environment (the server) is now callable by any web client. As a result, we need a powerful authorization models for REST API.

Permissions usually apply to table-level information (may I look at users?), column-level informations (I can look at users but may I look at user’s passwords?), row-level information (I can’t look at user’s private info except if the user is myself), and read/write permissions (I can only write my user info). And obviously a combination of all that.
Right now it’s way too complicated to implement, at least in django-rest-framework. You need to write your permissions twice: one for object listing and one for object access. Some libraries make it a bit easier but we’re far from where we should/could be.

Since I’m using django-rest-framework for HyperKitty’s API, I do intend to work with Jeff and all those interested to solve that issue.

In my very humble opinion, we should create permission classes that would utimately translate into ORM filters. This would apply nicely to object listings, and can work with single object access using another SQL request specifying the object’s primary key. For example, running this type of query: “SELECT COUNT(*) FROM the_table WHERE id=42 AND <permission-filters>”. If you get 0, you’re not allowed, if you get 1, you’re allowed. This model won’t work perfectly for read/write permissions, but they can be a bit different anyway, because they work with HTTP verbs.

Well, that’s just from the top of my head, and it my be obvious (or obviously wrong) to those who have been fighting with this for months/years.

Anyway, that was a very interesting and productive first day. Let’s see what tomorrow brings (and what I’ll be able to bring to tomorrow ;-) )

Also: Montréal is a wonderful city. It really is. And the people there are charming. Plus their language is wonderful :-D

jeu. 19 mar. '15

HyperKitty is now compatible with Mailman3 again

A couple days ago, I released a new version of HyperKitty on PyPI, which is now compatible again with the current Mailman3 branch (and beta tarball releases).

Lots of big changes and refactoring in this version:

  • KittyStore was merge into HyperKitty, which makes the whole suite much easier to install and maintain. This move was announced and discussed on the mailing-list. All the unit tests were ported to HyperKitty, so there shouldn’t be new bugs ;-)
  • The Mailman archiver plugin was split off in its own project: mailman-hyperkitty. This is necessary because this plugin runs within the Mailman process and thus must be Python3-compatible. It makes HTTP calls to HyperKitty to get URL locations and send emails to archive.
  • HyperKitty is now compatible with Django 1.6 and 1.7, with automatic testing using tox. A lot of tests were added, and a couple mistakes were discovered by PyLint.
  • The full-text search engine is now driven by the Haystack library, which allows us to switch to different backends depending on our needs. HyperKitty ships with the Whoosh backend enabled by default, but I’m using the Xapian backend for my tests in the Fedora infrastructure, and it’s significantly faster.

If you want to install HyperKitty now, just use the documentation. If you want to install Mailman 3 + Postorious + HyperKitty at the same time, then wait, there’s more :-)

At PyCon US 2014, I wrote a tool to easily install Mailman 3 + Postorious + HyperKitty in the same instance. This tool used Buildout to compose the install from the releases on PyPI, and was called Mailman-Bundler. Life was good. Unfortunately, when Mailman moved to a Python3-only project at the end of 2014, this tool stopped working for two reasons: HyperKitty was no longer compatible, and Mailman needed its own Python3-based virtualenv to run.

I am pleased to announce that the Mailman bundler tool is working again. If you want to easily install Mailman + HyperKitty + Postorius, just follow the tutorial. It is not very suitable for development though, because I get the code from the PyPI releases, not from bzr/git. Please test it and report bugs! If you want a development setup with fresh code from the VCS, there’s this tutorial that Mo wrote a few months ago, and that I updated.

So you may be wondering: when are we finally going to see HyperKitty in action for real? A just question. I have written a quick wiki page to summarize the steps needed to deploy HyperKitty in the Fedora infrastructure. Basically, the next big step is to package Mailman’s dependencies for EPEL7, which includes the whole Python 3.4 stack, since RHEL7 only ships 3.3 and Mailman requires 3.4. Fortunately I’m not the only one with this requirement, and a Fedora contributor has already drafted a packaging plan. There’s no code yet, but I will contribute to it.

The Mailman team is aiming for a final release at PyCon in April (this year, yes), and we’re aiming for a HyperKitty deployment around the F22 timeframe.

Also, the perspective of the incoming GSoC program this summer is bringing a lot of interested students to Mailman and HyperKitty these days, this is very cool :-)

ven. 23 janv. '15

News of the Mailman3 front

Alright, an update on HyperKitty’s status is looooong overdue. Actually, I should have written this post a week or two ago, but I had hosting issues which forced me to migrate my server at the last minute, and getting it back on was a bit painful… Done now. So, about Mailman3.

There’s been a lot of activity during the Christmas holidays. Barry, the Mailman project leader, decided to merge a branch porting Mailman 3 to Python 3. And since it’s a large amount of code, Mailman3 will not be bilingual: it won’t run on Python 2.X. So yeah, in a couple days, we went from “Python3 compatibility would be nice for Mailman 3 but it’s not a blocker for the release” to “Mailman3 will only run on Python 3”.

I understand Barry’s motives, Python 3 is really superior, especially in the email and unicode handling area which are crucial to Mailman 3. However, this means that a part of HyperKitty’s structure has to change. In the Mailman 3 architecture, an archiver is a Python plugin which runs in its own subprocess, but with the same interpreter. Our plugin calls KittyStore, the HyperKitty backend. As a result, KittyStore must be Python3-compatible. Then, HyperKitty (the Django app) also makes programmatic calls to KittyStore to retrieve the content, update the votes, etc. The API between the two is somewhat isolated, but it would not make a lot of sense to have a fully separated REST protocol between the two, because the requests are just very diverse and complex. So KittyStore is used as a library, which means that if KittyStore is Python 3, HyperKitty must be Python 3 too.

Porting HyperKitty to Python3 is not really an option, because we want it to be installable on an existing Django server, with various other applications that may not be ported yet. So we decided to introduce a level of protocol isolation between Mailman and KittyStore, since that’s where the API is the simplest. REST is what is most logical here, it’s used elsewhere in the Mailman architecture and it’s very flexible. As a consequence, KittyStore has to grow a REST server, be run as an independant process, etc.

After working on that a bit, I realized that the separation between HyperKitty and KittyStore made less and less sense. It was originally a design decision that kept a lot of possibilities opened, for example having other applications (like an NNTP server) access KittyStore, even maybe merging KittyStore into Mailman itself, etc. These things haven’t happened and the chances for them to happen are pretty slim: it would be more logical to access the archive store via REST calls to HyperKitty, which holds more information, and would be more stable and scalable than an internal REST server.

Merging HyperKitty and KittyStore also makes a lot of things easier, as I have explained on HyperKitty’s development mailing-list. Unfortunately, it’s a large amount of refactoring work. But it’s “only” refactoring, and it will make things simpler, thus easier to understand for newcomers and admins. It also means that my automated installer script, mailman-bundler, is broken at the moment, and can’t be fixed until Mailman3 and HyperKitty are compatible again.

Some good news now. Last summer, the two big changes that were under consideration for Mailman 3 were the change of ORM (Storm to SQLAlchemy) and Python 3 porting. Only the ORM change was deemed necessary for the release at that time, but now that I think of it, I should have seen it coming… ;-) As a consequence, only a couple of minor bugs now block Mailman 3 from being released, and the team is aiming for a full release of the Mailman 3 suite at PyCon in April. I’ll be there to lend, so meet us if you can. This is the last mile! :-)


jeu. 31 juil. '14

HyperKitty at Flock

As most Fedora contributors know, Flock 2014 is just around the corner. I’ll have the pleasure to meet you in the beautiful city of Prague, and I’m sure we’ll have an excellent time together at the talks, the hackfests and the evening events :-)

I’ll be giving a talk about HyperKitty on Wednesday 6th at 17:00 (5PM) in room B286, so make sure you attend if you’re interested in how the Fedora folks are going to talk to each other in the future (and then, the whole Free Software world! Mwahahahaha!).

If you want to be a part of that, if you crave for it and want it done faster, if you have a pet feature you’d like to have, come to the workshop that will take place on Friday 8 August at 17:00 (5PM) in room T9:302. I’ll show you the basics so you can be productive quickly.

See you there next week!

jeu. 13 juin '13

HyperKitty update

Hey y’all !

It’s been a long time since you last heard about HyperKitty, don’t you think? Here are the main new features that landed recently:

  • Unread tracking! This is something very useful in a web forum context that many users rely on. When you’re logged into HyperKitty, it will remember which threads you’ve read and when you’ve read them. When you come back to the month view, the threads that have new replies will be highlighted. When you read those threads, the new messages themselves will be highlighted.
  • When viewing a thread with unread messages, you get a mini navigation bar in the bottom right corner which allows you to jump through the unread messages. There are even hotkeys available! (hover the links to see them)
  • Search engine! HyperKitty now uses the Whoosh indexing library to give you full-text search results. You can even search across multiple mailing-lists in a single swoop.
  • Fixes across the board, especially in the timezone handling. When you have an international community such as Fedora, you have to get those right.

You can try out HyperKitty on this mirror of the Fedora-devel mailing-list. It’s a development server so it may be broken when you look at it; if so just come back later.

Hope you’ll like it!

- page 1 de 5