Part 2: how Combodo has organized itself to adapt to the STS-LTS model

In the previous article, you could understand the many benefits of combining Long-term and Short-term support releases: being flexible and scalable while guaranteeing stability to our customers, minimizing support and development costs while ensuring the best quality of service.

But before the release of this first LTS version, the 2.7, we had to reconsider a number of things in our internal organization. This project made the Combodo teams take a step forward, and we will see in this article how we organized ourselves to meet the challenge.

Ambitious development project

How to build the first LTS version in the history of Combodo? This is a very ambitious project, and as we explained in the previous part, subject to contradictory constraints…

This project took a little over a year, mobilizing several teams within Combodo. A few figures summarize this year of development:

• Development start : March 2019
• Backlog : 275 requests in March 2019, around 400 by July 2019… to finally end at 240 requests taken into account by the end of March 2020… it evolved very quickly, and we had to adapt.
• Release date :
  • LTS release initial target : December 2019
  • Release date first postponed until end of February 2020 (Decision in December 2019)
  • New shift to the end of March 2020 (Decision in January 2020)
  • Actual release of version 2.7.0 on April 1, 2020

Modify our organization to optimize quality!

Faced with our ambitions, we revised our way of working as the project progressed.

Project management

The retrospectives carried out on the previous versions of iTop showed us the increase in requests over time. Given the size of the project, we decided to set up a weekly monitoring of the backlog: this allowed us to quickly identify any slippage and act accordingly to limit delays.

Confinement and Covid-19

Like the whole world at the beginning of 2020, we suffered the COVID-19 pandemic and suddenly our whole organization had to be changed. We like to do pair programming and code reviews, and we discovered the pros and cons of remote work:

• ➕ We were less subject to office-life distractions so we could focus on the code.
• ➖ Moving from oral to written communication (and therefore to asynchronous communication) generated tensions between the teams, particularly between the development and product teams: availability, requests consideration, response time, etc.
• ➖ Working remotely, it is much more difficult to ensure the alignment of everyone on the same understanding… When everyone is in the same office, it is often enough to simply move around to talk to all the members of the other team.

🤔 This work change undergone by the teams, added to the delay with respect to the initial planning, required flexibility in order to find the best way to operate.

So we looked for the right tools (chat and visio: finally Combodo adopted Rocket Chat) and agreed on how to use them, in order to guarantee a way of working validated by all.

Pair programming

In order to improve the application’s code quality, pair programming (or peer programming) has been generalized. There are several advantages to this practice:

• Since the two developers have a different view, the code is of better quality (better structured, more maintainable): one (at the keyboard) has a tactical view of the code, and the other has a strategic view.
• The code review that is performed when coding is more relevant than the one performed afterwards.
• Knowledge sharing: maintenance is easier as two developers are familiar with the part of the code they have designed and modified.

This practice also has some drawbacks:

• It always takes a little longer to get started, since both developers have to agree on the development process and learn how the part of the code they have to modify works.
• You need to find pairs of developers who are complementary and trust each other (not necessarily easy in small teams)
• Having two developers working on the same topic reduces the amount of projects the team can run in parallel.

Nevertheless, for us, the benefits have largely outweighed these few drawbacks. This method was therefore definitely adopted.

Code reviews

We have also set up post-project code reviews. This is not necessarily an easy exercise.

As iTop is a complex application, it is difficult for the developers to know the whole code base, or simply to evaluate the impact of a change on the whole application. However, as the whole team has been working on this version at the same time, the emulation allowed to find connections between related topics (functional as well as technical).

This was the opportunity to start creating internal pull requests to facilitate reviews, and to better integrate features in the Git history. In iTop 2.7.0, 16 internal pull requests have been integrated!

Automated testing

Thanks to the development of iTop 2.7 release, PHPUnit tests, which were in their first stages of development at Combodo, have grown significantly since we went from a few dozen tests to several thousands!
Moreover, all tests have been automated using Jenkins and are launched at each commit for a part of them, and daily for the whole.

In addition to these PHPUnit tests (which essentially cover iTop’s core functions), GUI tests have been also performed using Behat and also executed daily.

This much larger range of testing made life much easier and gave confidence to the quality team in charge of our product validation.

Plan the future

Once this organization is in place, it must be sustained, and it is also necessary to anticipate the future and the technological evolutions which are part of the important challenges of an LTS.

PHP compatibility range

Because the schedule of supported PHP versions is evolving rapidly, contradictory aspects must be reconciled:

• Allow a majority of customers to switch to our LTS version without having to upgrade their infrastructure, and therefore support a low minimum version
• Do not choose a minimum version that is too low, considering that given the pace of development of PHP, many of the libraries that we embed in iTop were likely to increase their minimum supported version quickly
• Support the most recent version of PHP

That’s why we have chosen a range of versions from PHP 5.6 minimum to PHP 7.4 maximum.

MySQL 8

The release of MySQL 8.0 also raised doubts about iTop’s performance since one of the major changes in this version was the removal of the query cache. We’ve considered that implementing a SQL query cache in iTop was out of scope, but we’ve made iTop compatible with this version anyway!

Thus, we had to work on several aspects:

• Column names always returned in uppercase starting with MySQL 8 (previously with the case used in the SELECT clause)
• Bad detection of mysqldump: we were checking the version of the program to validate the support of utf8mb4 (generalized charset in iTop 2.5.0) and this code was not functional with MySQL 8… We now use the database version rather than the tool version
• Generation of duplicate values in Ticket.ref : the system relied on the auto-increment value, but this one is stored in a cache in MySQL8… So we set up a new system with a generic counter ItopCounter

As incompatibilities had been corrected, we’ve conducted performance tests which confirmed the very negative impact of the removal of the query cache.
As a result, a specific communication was made to warn our users about these performance issues with MySQL 8, and advise them to use an earlier version of the database server.

Dependencies management

iTop embeds several PHP libraries. These libraries were previously managed manually. The LTS version being planned to last longer, we needed an efficient mechanism to manage security holes and bugs in these libraries. We naturally chose Composer, a very common tool in the PHP world which has proven itself.
We then carried out an inventory, and for each library we checked if an update was required.

Having the file composer.json in our repository brings us efficiency thanks to the support of our development tool PHPStorm, and also guarantees that we do not miss important information thanks to the Dependabot integrated to GitHub.

We have also adapted our processes and added, before each major release, a new “foundations” phase including librairies control and their update.

Silex to Symfony

We had to replace Silex librairy (out of support since June 2018) by Symfony. Cette nouvelle librairie a aussi un calendrier de support calqué sur PHP. Nous étions donc limité sur la version Symfony utilisable dans iTop 2.7.0.

Silex has therefore been replaced by Symfony 3 while foreseeing the transition to Symfony 4 and 5.

API depreciation and removal policy

A large part of iTop’s code is made of APIs: indeed, the extension creators use a lot of our classes and methods! But if the Combodo development team needs to be able to restructure the iTop code from time to time, it is also essential to guarantee stability to these developers, and to simplify the migrations of the extensions.

So we worked on different projects :

• clearly identify classes and methods representing APIs (using tag phpdoc @api) and those which should not be used elsewhere than in the iTop code (@internal)
• automate those APIs documentation thanks to scripts and publish it on the wiki
• Define a strict API depreciation and removal policy in case of update: the former code remains in version N but is flagged as deprecated, and will be removed in a later version

Deprecated methods are flagged with phpdoc @deprecated tag, the iTop version and as much as possible an indication of the code to use instead. For each deprecation, a deletion ticket is entered and scheduled for one of the next iTop releases. Everything is documented in the migration logs (e.g. for iTop 2.7.0 in a chapter of What’s new, for iTop 3.0.0 in a dedicated page)

For example the documentation of the method \DBObject::DBInsertTracked has been modified for iTop 2.7.0:

    /**
     * @internal
     *
     * @deprecated 2.7.0 N°2361 simply use {@see DBObject::DBInsert()} instead, that will automatically create and persist a CMDBChange object.
     *    If you need to persist your own, call {@see CMDBObject::SetCurrentChange()} before.
     *
     * @param CMDBChange $oChange
     *
     * @return int|null
     * @throws CoreException
     */
    public function DBInsertTracked(CMDBChange $oChange)

And this method DBObject::DBInsertTracked has been removed in 3.0.0 (commit 80161b9).

Improve security, now and in the future!

Security audit

Several of our customers have requested a security audit performed by an external agency for their own version of iTop and have pointed out some flaws.

We have decided a security audit performed on the beta version of 2.7.0 to identify and help fixing as many flaws as possible before the official release of iTop 3.0.0.

Public security policy

So that flaws are reported as quickly as possible and corrected as well as possible, we have also defined together and published our security policy on our GitHub repositories (e.g. on iTop).

Vulnerabilities are published 3 months after the release of the first version containing the fix to allow our users and customers to update their instances. The publication is done through GitHub security advisories and CVE.

The security audit and the reporting of vulnerabilities thanks to the implemented policy have allowed to correct 30 security issues on version 2.7.0.

Brute force protection extension

This extension has been added to the subscription packages (iTop Professional and Essential) to protect the application from brute force password attacks. This is indeed a known attack on web applications accessible on the internet.

A Brute-force attack is defined as below by Wikipedia :

In cryptography, a brute-force attack consists of an attacker submitting many passwords or passphrases with the hope of eventually guessing correctly. The attacker systematically checks all possible passwords and passphrases until the correct one is found.

.Source https://en.wikipedia.org/wiki/Brute-force_attack

The extension limits the number of possible attempts for the connection, which prevents this type of attack. It also offers many other protections (IP filtering, captcha, …) as indicated in the documentation.

Password policy

To further complicate the work of hackers, users of the application should not be allowed to have passwords that are too short or too simple. The local authentication system has been modified to control passwords and avoid the validation of too simple and insecure passwords.

More details in What’s new iTop 2.7.0 : Password Policy

Mandatory features

Some features definetely had to be included in this LTS. Indeed we needed :

• ease maintenance and support as much as possible,
• guarantee the best possible integration with users’ information systems,
• offer all APIs for extension developpers.

Here are some examples of features that have been implemented to meet these objectives:

Login extensibility

Since this version is intended to last, it was necessary to consider the scalability of some parts. The login page was one of the improvements made in version 2.7.0.

We added the following features:

• The login page is now in a Twig format, which allows to modify the visual aspect.
• Connection logic redesign to allow the addition of new connection types (SSO in particular with OAuth)
• Enable new security controls (as Brute Force Protection against Brute-force attacks)
• Allow to add later connection controls like 2FA or captchas

Performance

SQL generation

The iTop application uses, to handle client data, a query language specific to iTop, the OQL (Object Query Language). This language gives access to data by making queries on objects in the way of the SQL language. iTop core then converts OQL into SQL to process what expected by the user.

Before iTop 2.7, the mechanism used to generate these SQL queries guaranteed the consistency of the data returned in the results, which was reassuring for the user, but at the cost of degraded performance.

This part of the application has been reviewed with the objective of improving maintainability, thanks to a more explicit architecture, and the performance of SQL query generation, with a gain in the number of joins (which can amount to several dozen for enumeration operations!)

The optimization choice was made at the detriment of the guarantee of coherence of the results, since the queries will no longer exhaustively search all the tables composing the objects of the client’s data model , but only the tables necessary for the answer (lazy mode). Nevertheless, even if the optimizations depend greatly on the complexity of the OQL queries, an big improvement on the queries execution in the database has been noticed.

This compromise between performance and results consistency was made considering that a tool checking the database consistency called “Database integrity” was provided as part of the application.

This change was made possible by the addition of unit tests validating that the result of the queries was consistent with iTop 2.6.

Conclusion

By listing here all the evolutions both on our organization, our methods, but also the technical choices concerning the solution we’ve made for the Long-term Support, we easily realize the impact of such a project on Combodo teams and iTop solution.

iTop 2.7 has been running for 2 years now, only LTS in parallel with STS versions like iTop 3.0, and used by more than half of our customers. After all this time, we thought it would be useful to end this series of articles with a feedback on the deployment and operation of iTop, in order to prepare the next releases.

The last article of this series will allow you to take stock as we did and put the results of this project in perspective for the future.

Eric Espié

Plus de publications

Développeur dans de nombreux domaines comme les télécoms, le kernel Linux et AIX (IBM), la sécurité, la détection embarquée de pannes matérielles, les ERP, et l'ITSM.

• "Long-term" - sustainable, scalable and maintainable
• Need stability? Long-Term Support is for you