I am glad to let you know that NLNet will support my project of improving GNU Guix continuous integration system. The project will be funded through the NGI0 PET Fund, a fund established by NLnet with financial support from the European Commission's Next Generation, as explained here.
GNU Guix is using Cuirass as continuous integration software. It's a general purpose build automation server written in GNU Guile that checks out sources from VCS repositories, execute build jobs and store build results in a database. Cuirass also provides a web interface to monitor the build results.
Cuirass is supposed to build and provide binary substitutes for every package or system image definition. Without those substitutes available in time, the user experience is really degraded, as the user has to allocate important time and computation power resources into package building.
Cuirass has been supported by the Google summer of code project (2016, 2017 and 2018). It is right now developed voluntarily mostly by the GNU Guix community. This grant will allow me to dedicate 6 months of my time to work on improving Cuirass.
I plan to achieve the following tasks.
Add dynamic offloading support
Implement a dynamic offloading mode. This mode will allow an offloading server to distribute work among different build machines.
The network discovery between the offloading server and the build machines will be done using the Guile-Avahi library.
The build machines must be able to come an go dynamically. They will pick build tasks from the offloading server using a work stealing scheduling strategy.
The communication between the server and the build machines will be implemented using Guile-ZMQ library.
Improve the test suite
Implement different dynamic offloading scenarios in the Cuirass test suite to secure the development and prevent the risks of regressions.
Improve Cuirass build monitoring interface
Add build monitoring tools to Cuirass so that system administrators and users can closely monitor the health of the whole continuous integration system and promptly report any issue or slowdown.
Add a web page allowing to monitor the builds that are currently performed on the build farm, machine per machine. The build machine status (CPU load, RAM, disk usage) will also appear on that page.
Add a live build logging feature, so that the build logs can be read in realtime through the web interface.
Add new related metrics such as "build speed per machine" and "idle time per machine" to the UI metrics reports.
Improve Cuirass build reporting interface
Create a user account section to setup customized monitoring dashboards, subscribe to build failures notifications and get notified of the availability of new substitutes.
Add support to report build failures by email and RSS.
Add a user account section to setup customized monitoring dashboards and subscribe to build failures notifications.
Add a substitutes availability service, so that users can be notified when the substitutes coverage of their manifest is passing a threshold.
Enforce accessibility guidelines after the WCAG audit.
Starting to work!
As a first step, I have already migrated Cuirass from SQLite to PostgreSQL. I might write something about that later on. I also started working on the dynamic offloading mechanism with some encouraging results.
I will of course report the results periodically on this website and on the GNU Guix blog.
Don't hesitate to contact me by email or on #guix to discuss your impressions as well as the new features you would like to see in Cuirass.