[Erp5-dev] ERP5 Release System - proposal

[Łukasz Nowak]

Hello,

We present our idea how to organise release management for ERP5. We
would like to know your opinion about this idea. Any constructive
comments as well as criticism are welcome. We are making this proposal
in good faith - to provide the best possible solution in implementing
and development of ERP5 system, to give as much as possible possibility
to arise community of skilled developers around such great project as
ERP5 is itself.

Miko?aj Antoszkiewicz
Tomasz Bochenek
Bartek G?rny
Jacek M?drzycki
?ukasz Nowak

= Proposal =

== Rationale ==

While working on ERP5 system we've been hit with many problems when
upgrading the system. Additionally it is very hard to choose '''a
good''' revision. Having in mind fact, that nowadays ERP5 is developed
and used by developers with different knowledge and skills associated
with ERP5 system it is good to provide them quite stable version of
ERP5.

That's why we are proposing a draft of stabilisation releases of ERP5.

== Releases Advantages ==
* Releases will accelerate, facilitate and lower cost of learning ERP5
system and maintenance of running ERP5 sites.
* Releases will not disturb ability to commit new
fixes/features/functionality to HEAD - upports will force developer
using release to synchronise fixes into HEAD and short release time
will force them to use almost HEAD version.
* Releases will create contributing community of lightly related users
and developers of ERP5 system.
* Only current release is supported, with RELEASE-1 to RELEASE upgrade
path. Every question from person using former releases shall be
answered with '''Upgrade to current release.'''
* this will drastically lower cost of releases' maintenance - cost
of release + cost of '''one''' upgrade path This also will force
developers using release to be in sync with HEAD version of code.
* Release Manager do not have to be highly skilled ERP5 developer,
which will lower cost of generating new release.
* Releases aren't treated as fully tested, stable version - they are
just ''enough'' tested snapshots - good enough to be used by less
experienced users and developers and current enough for bleeding edge
developers.
* Experimental developers will not feel any disadvantages when
releases arrive - they still will be using HEAD version, and
synchronised upporting system will give them all fixes generated in
releases.
* Releases may be triggered on upstream request, which will force
horde of upstream related and non-related developers to help with
release.
* Release will be created in less than 8 working days.
* Core developer would be able to focus on developing bleeding edge
code - synchronisation and backward compatibility might be suspended to
next release cycle.
* Having so often releases is guaranteeing users and developers short
times of getting freshest code.
* Having releases won't stop users and developers from fetching HEAD
version of code to be able to participate in very current stage of ERP5
code development.
* Proposed release scenario kept in mind minimum release's maintenance
cost and is trying to have minimal impact on current development way of
ERP5 (this release scenario as much as possible tries to accelerate
development of ERP5). Also it kept in mind keeping minimal cost at side
of upstream.
* Releases are very positive on marketing - many users and future
clients will be happy if they see "ERP5 Release 200804".
* As implementing ERP is quite complex problem and ERP5 installation
is not simple (it couldn't be - this is _complex_ and generic system)
releases will provide kind of good feeling to newcomers - releases will
provide them good starting point.

=== Summary ===
Releases will lower running cost of sites, will not stop developers
from fixing HEAD and will make initial installation of ERP5 sites
easier.

== Releases ==
First release will be created after 1 full month after this draft
publication.

Releases time windows '''might''' be consulted with upstream and/or
large contributors/users of ERP5 - if needed.

Release Manager (or Release Manager Team) ('''RM''') is responsible for
release, everyone needs to support him in his work.

Releases shall be made 4 times per year (every quarter). Upgrade from
one release to the next one '''has to''' be unproblematic - every
additional step, which might create problems, have to be described in
'''Upgrade Notes'''.

Specific sites are allowed to have problems with upgrade - it might be
something with API changes etc, but release have to guarantee that it
is OK. API changes shall be described in '''API Changes'''.

== How to create Release ==

1. RM is choosing revision with little errors and fails in unit tests
2. RM is removing backports created on older releases
3. RM is checking validity and the need for upports for RELEASE
4. RM is tagging SVN repository on chosen revision, naming it
RELEASE-RC
5. RM is announcing RELEASE-RC and welcomes experienced community
members to participate in RELEASE scenario
6. RM is upgrading system from previous RELEASE to RELEASE-RC
7. Other developers do test upgrades from previous RELEASE to
RELEASE-RC:
1. When needed they are sending backports/upports to RELEASE-RC
2. If something important will happen while doing RELEASE-RC it is
added as backport
8. RM writes and publicise ''What's new'', ''Upgrade Notes'' and ''API
Changes'' for RELEASE-1 to RELEASE-RC
9. RM is changing tag name from RELEASE-RC to RELEASE and announces
new release

Then is "feature freeze" - nothing more is added - every backport or
upport is only for critical bugfixes.

== Upport/Backport ==

Because creating a RELEASE has side effect of being a little behind
after HEAD, it is necessary to create a backport system (on HEAD there
is a critical bugfix) or upport (something is corrected on RELEASE,
that should be committed).

Every upport shall be proposed to be committed into CORE - if it is
accepted, it would become backport on RELEASE.

Release maintainers '''WILL NOT''' accept any upports, which weren't
send for acceptance to HEAD. No special cases will be claimed.

== Products / Business Templates ==

Business Templates shall have such skin priority:
1. erp5_trade_upport
2. erp5_trade_backport
3. erp5_trade

Products are monkey-patched by creating directories/files
upports/backports, which have monkey-patches with fixes to upport or
backports took from HEAD.

Release maintainers will be obliged to minimise upport/backport size to
minimum - only critical bugfixes or features which were introduced
while release was creating, and are treated as important or essential.

Site maintenance will thus be limited to regularly updating
backports/upports of products and BT's, and every quarter or so
updating everything to the new RELEASE - with upgrade notes etc.

Developers are guaranteed that updates in same release are flawless.

If there would be need to introduce greater changes, new release shall
come.

== Unit tests ==

RELEASE and its iterations are unit tested. It is assumed that upgrade
to next iteration of same RELEASE is non-problematic - almost always it
will be small steps, only critical bugfixes after -RC.

RELEASE-RC is unit tested.

Additional unit test scenario:
1. save site with previous RELEASE
2. unit test
3. '''upgrade''' site to RELEASE (this same site)
4. unit test
5. comparison of effects

== Release technical notes ==

Release will be named by time, when release creation happened, eg:
* 200711 for release in November 2007

Release names might have special codenames, which will follow official
release name, eg:
* 200804-SPRINGY
Release codenames have to be unique.

Release iteration won't have any additional numbers. Users and
developers have to upgrade to newest current release iteration.

It would be preferred to have special BT downloadable repository for BT
for current (and only current) release.

No upgrade path will be provided to upgrade for more then one release.

As release will be based on tags it is possible to provide full (and
almost never ending) history of release tags in SVN repository. That
way users of former releases would be able (on their own risk) to
upgrade one by one. (Tagging === copying in sense of SVN, which is
"cheap" operation.)

Release might provide possibility to create packages in RPM, DEB or any
suitable form for quick ERP5 installation on target systems.

[Yoshinori Okuji]

Hello ?ukasz,

Post by Łukasz Nowak
We present our idea how to organise release management for ERP5. We
would like to know your opinion about this idea. Any constructive
comments as well as criticism are welcome. We are making this proposal
in good faith - to provide the best possible solution in implementing
and development of ERP5 system, to give as much as possible possibility
to arise community of skilled developers around such great project as
ERP5 is itself.

First of all, I would like to thank you for writing this proposal. I highly
appreciate that you have made significant effort towards a better maintenance
process.

Let me describe how Nexedi has been maintaining the code, and succeeding in
implementing customers' projects so far:

- All the code base, including Business Templates, unit tests, functional
tests, and many other types of tools, required for generic ERP5, has been
always published on the subversion repository. No permission is required to
access the source code at any time. Everything is immediately open to anybody
in the world freely. This way, more people can take part in the reviewing
process, and facilitating contribution. Also, developers recognize more
responsibility for the quality, because of the potential risk.

- All the test results are sent to the public mailing list erp5-report every
day. People are free to check for the current status in the cutting-edge
development, thus it is easy to determine the latest stable revision from the
history of reports. If a result is good, but the system is still unstable,
this only means that we lack tests. Such tests should be complemented as soon
as possible, and contribution is very welcome.

- Chanages are, in principle, peer-reviewed. Only trivial changes, such as
fixing a typo, do not require such a review. Some people are not allowed to
directly check in changes, when we still don't know the quality level of code
produced by them. Once a person proves that she has a good skill, we give
more permissions to her, so that we will not prevent her work. However, even
experienced developers, such as me and Jean-Paul, need a peer-review, before
committing a significant change. This way, although it may not be perfect,
the quality of software is kept acceptally good all the time. Note that I
have been using the same policy in another Free Software project for many
years successfully.

- Backward compatibility is respected all the time. We break backward
compatibility, only if it is inevitable. Incompatibility is an evil, because
it produces too much work at a time when upgrading a system. We have been
publishing the information on incompatibility we made in the public wiki at
www.erp5.org, so that others will not get troubled. Still, we are always
trying to reduce incompatibility, and only deprecate old usage.

- Code base used for a customer's project is taken out of the public
subversion repository, under the control by a project manager. A decision is
made mostly based on the status of unit tests and functional tests, but also
with reviwing recent changes of the code, because some kind of problems may
not be very visible only with automatic testing. Then, the code base is
upgraded from time to time (the frequency depends on the constraints in a
project) so that bug fixes and feature enchancement are incorporated. Again,
the decision of which revision should be taken is based on the same criteria
as above, as well as project conditions in some cases. This way, we can
implement a stable system without much trouble.

As the technical leader of ERP5, I have been thinking of how to make releases
for some years, in a way that the maintenance cost would be negligible, and
woouldn't disturb development and contribution. The important questions are:
When and How. I consider that there is no question about "Why". As you
pointed out, releases make things more apparent and visible, so they are
definitely nice to have.

Here are some of my ideas:

- When

In my understanding, there are three different ways to manage the timings of
making releases.

One of them is feature-basis. When something significant is made, a new
release is published. This is often used in proprietary software products for
marketing purpose. Then, bug fixes are provided only in the form of patches.
In Open Source development, this does not make sense very much, since patches
can be included from the beginning. Instead, feature enhancement increments a
major version. For now, I am not talking about how to name versions, so I
don't take this way into consideration any more.

Another is regularity-basis. If I understand correctly, this is the same as
your suggestion. In this scheme, releases are made in the same frequency, no
matter whatever is happening. The advantage is that it is guaranteed that the
realtime characteristics of releases against the head is quite good, that is,
the distribution of delays behind the head is minimized. However, this way
has a critical drawback that it does not reflect the reality of development.
Sometimes a significant amount of development is performed in a short term,
and sometimes not. As the nature of development, changes, in particular,
which address new problems, tend to make software less stable. I will discuss
this issue more below.

The other is quality-basis. A new release is made when the head has good
quality of code. This is rather the reverse of the regularity-basis, so the
pros/cons are also reversed. This proceeds with the reality of development,
thus a delay can be potentially huge, but the quality of a release is well
maintained.

Of course, in most projects, these ways are more or less mixed. Even if the
quality-basis approach is taken, it does not make sense to make a new release
every day, only because the quality is high. Even with the regularity-basis
one, if the quality cannot meet criteria on time, a delay must be inserted to
prevent catastrophy. Many factors should be taken into consideration to keep
good release management.

As you might know, I have +10-year experience in many other Open Source/Free
Software projects. And, I have tried both the regularity-basis and
quality-basis ways. Overall, I have found that the quality-basis works far
better, because, for example, when a problem can be resolved appropriately by
a certain few people, and if they are not available for some reason for a
while, the regularity gets broken so easily. Enforcing a release quickly,
simply because of the policy, only produces unusable software.

Also, I have found that release management is far from trivial. Of course, as
you suggested, we use the result of tests for making estimate on stability,
but tests may not cover everything. It is sometimes because not all tests are
implemented yet. It is sometimes because managers are not capable of
determining whether software is stable enough or not. Especially in the
preparation of a new release, bug fixes must be encouraged, but contributors
have their own time constraints, and may not always follow a release cycle.
In this case, managers must be able to define what bug fixes can be postponed
with the idea of possible danger, and address problems by themselves,
whenever necessary.

Another perspective is how to facilitate people to contribute. Usually,
excellent developers want good feedback from managers. If they feel that
managers fully understand their changes, and evaluate the contribution
appropriately, they tend to help managers. Otherwise, they stop. This is not
related to whether they are paid or not. For instance, although workers at
Nexedi are paid, they are not always paid to contribute publicly. This part
really depends on their feeling and will. If they think that managers would
not understand their effort, they will just put changes locally and hide
them. This has been always the same as in volunteer-based projects, so I am
rather sure. Therefore, I believe that highly skilled persons must be
selected as release managers for ERP5. Otherwise, release management will
corrupt.

- How

There are two matters to consider: how many branches should be maintained or
developed, and how long a branch should be maintained.

The simplest approach is to use a single trunk, and no other branch. This is
the current way. The benefit is that all the effort heads towards one place,
so the efficiency of development is very high. We do not need to worry about
forking. The drawback is that the quality can be random along the time, no
matter however much effort is put into stabilization.

The semi-simple approach is, as you proposed, to use a main trunk, and one
branch to keep stability. And, to abandone older branches, once a new branch
is created. This is somehow easy and realistic. The benefit is that, without
having a lot of maintenance cost, stability is more or less ensured on a
branch. The drawback is that this way additionally requires porting changes
between a trunk and a branch, thus the development energy is scattered, and
extra effort and time are required for the maintenance.

For now, I compare only these two. Again, I have experience with both
approaches in the past. In short, I met difficulty in the latter approach,
because the cost of porting patches was quite higher than I expected. After
all, I gave it up, and I chose a single trunk method.

The way I did was the following: when I started to feel that I wanted to make
a new version, for example, because we had a good set of new functionality, I
forced people to stop writing a new feature, but encouraged people to test
the head, and fix it. In the meantime, all patches sent to me were waited,
and kept pending. Once I was satisified with the quality of code, I put a tag
and release it. Then, I restarted the normal cycle of development, and
integrating pending contribution.

Actually, this was so effective. All contributors understood that this was
required, and accepted that patches were in a pending state. Besides stopping
people to modify the head brutally, no extra cost was raised.

Surely, we must analyze if this way is effective for ERP5 as well, but I feel
that this is more appropriate.

The reason is really the cost. Unfortuantely, nobody is a superman. I don't
think it is a bad thing to have a stable branch, but at the same time, I
doubt if it is feasible. If we use branching, another team for a branch would
be necessary to be formed, just like the security team in Debian, because the
backporting is surprisingly hard and time-consuming.

If anybody wants to perform the backporting job, I would appreciate very much.
But be prepared that it is not trivial at all. One must be able to say what
is a pure fix, and must be able to segregate safe parts of patches from a
trunk, and adapt API usage, and so on. For now, I don't think this is
feasible with the current size of our community, but I would be very happy if
I am wrong.

So, at the moment, my suggestion is to use a single trunk. We have proven that
writing tests for a good coverage makes projects successful, because tests
always notify us of the current status, and what must be fixed, without any
branching. Release managers should promote people to write more tests for
missing parts, and negotiate with developers when next release should be
made, so that others can collaborate.

Now, this mail is getting too long, but let me describe some more additional
notes.

- It is a very good idea to describe API changes. We already do upgrade notes,
but this is a nice complement.

- Upgrade tests must be automated very well, otherwise it is too costly. Not
only unit tests, but also functional tests must be performed. Currently, the
number of functional tests is not good enough, in particular about
configuration-related stuff. Contribution is very welcome.

- Release management may not be independent of development management. Both
are very tightly coupled, because releasing depends on the status of
development. Only experts can perform this kind of job, so that other
developes will listen to their advise.

- I myself prefer co-maintenance, that is, having multiple maintainers. This
has a significant advantage in that even if one is not available (sick, busy,
or whatever), the project will not be stalled. I was proposed to be a
co-maintainer in a project, and it worked very well, so I'm proposing others
to do the same with me these days. I like it. For ERP5, 2-3 persons will be
good. Having too many is no good.

I hope my message would be useful for further discussion. Please let me know
what you think.

Regards,
YO

[Shrenik Bhura]

Hello,

It would be beneficial if this valuable discussion could be moved and
continued at the Discussions section of the wiki.

I shall do it 24 hrs from now if all are fine with it. In case of any
objections/thoughts please let me know prior to that.

Thanks,
SB

Post by Yoshinori Okuji
Hello ?ukasz,

Post by Łukasz Nowak
We present our idea how to organise release management for ERP5. We
would like to know your opinion about this idea. Any constructive
comments as well as criticism are welcome. We are making this proposal
in good faith - to provide the best possible solution in implementing
and development of ERP5 system, to give as much as possible possibility
to arise community of skilled developers around such great project as
ERP5 is itself.

First of all, I would like to thank you for writing this proposal. I highly
appreciate that you have made significant effort towards a better maintenance
process.
Let me describe how Nexedi has been maintaining the code, and succeeding in
- All the code base, including Business Templates, unit tests, functional
tests, and many other types of tools, required for generic ERP5, has been
always published on the subversion repository. No permission is required to
access the source code at any time. Everything is immediately open to anybody
in the world freely. This way, more people can take part in the reviewing
process, and facilitating contribution. Also, developers recognize more
responsibility for the quality, because of the potential risk.
- All the test results are sent to the public mailing list erp5-report every
day. People are free to check for the current status in the cutting-edge
development, thus it is easy to determine the latest stable revision from the
history of reports. If a result is good, but the system is still unstable,
this only means that we lack tests. Such tests should be complemented as soon
as possible, and contribution is very welcome.
- Chanages are, in principle, peer-reviewed. Only trivial changes, such as
fixing a typo, do not require such a review. Some people are not allowed to
directly check in changes, when we still don't know the quality level of code
produced by them. Once a person proves that she has a good skill, we give
more permissions to her, so that we will not prevent her work. However, even
experienced developers, such as me and Jean-Paul, need a peer-review, before
committing a significant change. This way, although it may not be perfect,
the quality of software is kept acceptally good all the time. Note that I
have been using the same policy in another Free Software project for many
years successfully.
- Backward compatibility is respected all the time. We break backward
compatibility, only if it is inevitable. Incompatibility is an evil, because
it produces too much work at a time when upgrading a system. We have been
publishing the information on incompatibility we made in the public wiki at
www.erp5.org, so that others will not get troubled. Still, we are always
trying to reduce incompatibility, and only deprecate old usage.
- Code base used for a customer's project is taken out of the public
subversion repository, under the control by a project manager. A decision is
made mostly based on the status of unit tests and functional tests, but also
with reviwing recent changes of the code, because some kind of problems may
not be very visible only with automatic testing. Then, the code base is
upgraded from time to time (the frequency depends on the constraints in a
project) so that bug fixes and feature enchancement are incorporated. Again,
the decision of which revision should be taken is based on the same criteria
as above, as well as project conditions in some cases. This way, we can
implement a stable system without much trouble.
As the technical leader of ERP5, I have been thinking of how to make releases
for some years, in a way that the maintenance cost would be negligible, and
When and How. I consider that there is no question about "Why". As you
pointed out, releases make things more apparent and visible, so they are
definitely nice to have.
- When
In my understanding, there are three different ways to manage the timings of
making releases.
One of them is feature-basis. When something significant is made, a new
release is published. This is often used in proprietary software products for
marketing purpose. Then, bug fixes are provided only in the form of patches.
In Open Source development, this does not make sense very much, since patches
can be included from the beginning. Instead, feature enhancement increments a
major version. For now, I am not talking about how to name versions, so I
don't take this way into consideration any more.
Another is regularity-basis. If I understand correctly, this is the same as
your suggestion. In this scheme, releases are made in the same frequency, no
matter whatever is happening. The advantage is that it is guaranteed that the
realtime characteristics of releases against the head is quite good, that is,
the distribution of delays behind the head is minimized. However, this way
has a critical drawback that it does not reflect the reality of development.
Sometimes a significant amount of development is performed in a short term,
and sometimes not. As the nature of development, changes, in particular,
which address new problems, tend to make software less stable. I will discuss
this issue more below.
The other is quality-basis. A new release is made when the head has good
quality of code. This is rather the reverse of the regularity-basis, so the
pros/cons are also reversed. This proceeds with the reality of development,
thus a delay can be potentially huge, but the quality of a release is well
maintained.
Of course, in most projects, these ways are more or less mixed. Even if the
quality-basis approach is taken, it does not make sense to make a new release
every day, only because the quality is high. Even with the regularity-basis
one, if the quality cannot meet criteria on time, a delay must be inserted to
prevent catastrophy. Many factors should be taken into consideration to keep
good release management.
As you might know, I have +10-year experience in many other Open Source/Free
Software projects. And, I have tried both the regularity-basis and
quality-basis ways. Overall, I have found that the quality-basis works far
better, because, for example, when a problem can be resolved appropriately by
a certain few people, and if they are not available for some reason for a
while, the regularity gets broken so easily. Enforcing a release quickly,
simply because of the policy, only produces unusable software.
Also, I have found that release management is far from trivial. Of course, as
you suggested, we use the result of tests for making estimate on stability,
but tests may not cover everything. It is sometimes because not all tests are
implemented yet. It is sometimes because managers are not capable of
determining whether software is stable enough or not. Especially in the
preparation of a new release, bug fixes must be encouraged, but contributors
have their own time constraints, and may not always follow a release cycle.
In this case, managers must be able to define what bug fixes can be postponed
with the idea of possible danger, and address problems by themselves,
whenever necessary.
Another perspective is how to facilitate people to contribute. Usually,
excellent developers want good feedback from managers. If they feel that
managers fully understand their changes, and evaluate the contribution
appropriately, they tend to help managers. Otherwise, they stop. This is not
related to whether they are paid or not. For instance, although workers at
Nexedi are paid, they are not always paid to contribute publicly. This part
really depends on their feeling and will. If they think that managers would
not understand their effort, they will just put changes locally and hide
them. This has been always the same as in volunteer-based projects, so I am
rather sure. Therefore, I believe that highly skilled persons must be
selected as release managers for ERP5. Otherwise, release management will
corrupt.
- How
There are two matters to consider: how many branches should be maintained or
developed, and how long a branch should be maintained.
The simplest approach is to use a single trunk, and no other branch. This is
the current way. The benefit is that all the effort heads towards one place,
so the efficiency of development is very high. We do not need to worry about
forking. The drawback is that the quality can be random along the time, no
matter however much effort is put into stabilization.
The semi-simple approach is, as you proposed, to use a main trunk, and one
branch to keep stability. And, to abandone older branches, once a new branch
is created. This is somehow easy and realistic. The benefit is that, without
having a lot of maintenance cost, stability is more or less ensured on a
branch. The drawback is that this way additionally requires porting changes
between a trunk and a branch, thus the development energy is scattered, and
extra effort and time are required for the maintenance.
For now, I compare only these two. Again, I have experience with both
approaches in the past. In short, I met difficulty in the latter approach,
because the cost of porting patches was quite higher than I expected. After
all, I gave it up, and I chose a single trunk method.
The way I did was the following: when I started to feel that I wanted to make
a new version, for example, because we had a good set of new functionality, I
forced people to stop writing a new feature, but encouraged people to test
the head, and fix it. In the meantime, all patches sent to me were waited,
and kept pending. Once I was satisified with the quality of code, I put a tag
and release it. Then, I restarted the normal cycle of development, and
integrating pending contribution.
Actually, this was so effective. All contributors understood that this was
required, and accepted that patches were in a pending state. Besides stopping
people to modify the head brutally, no extra cost was raised.
Surely, we must analyze if this way is effective for ERP5 as well, but I feel
that this is more appropriate.
The reason is really the cost. Unfortuantely, nobody is a superman. I don't
think it is a bad thing to have a stable branch, but at the same time, I
doubt if it is feasible. If we use branching, another team for a branch would
be necessary to be formed, just like the security team in Debian, because the
backporting is surprisingly hard and time-consuming.
If anybody wants to perform the backporting job, I would appreciate very much.
But be prepared that it is not trivial at all. One must be able to say what
is a pure fix, and must be able to segregate safe parts of patches from a
trunk, and adapt API usage, and so on. For now, I don't think this is
feasible with the current size of our community, but I would be very happy if
I am wrong.
So, at the moment, my suggestion is to use a single trunk. We have proven that
writing tests for a good coverage makes projects successful, because tests
always notify us of the current status, and what must be fixed, without any
branching. Release managers should promote people to write more tests for
missing parts, and negotiate with developers when next release should be
made, so that others can collaborate.
Now, this mail is getting too long, but let me describe some more additional
notes.
- It is a very good idea to describe API changes. We already do upgrade notes,
but this is a nice complement.
- Upgrade tests must be automated very well, otherwise it is too costly. Not
only unit tests, but also functional tests must be performed. Currently, the
number of functional tests is not good enough, in particular about
configuration-related stuff. Contribution is very welcome.
- Release management may not be independent of development management. Both
are very tightly coupled, because releasing depends on the status of
development. Only experts can perform this kind of job, so that other
developes will listen to their advise.
- I myself prefer co-maintenance, that is, having multiple maintainers. This
has a significant advantage in that even if one is not available (sick, busy,
or whatever), the project will not be stalled. I was proposed to be a
co-maintainer in a project, and it worked very well, so I'm proposing others
to do the same with me these days. I like it. For ERP5, 2-3 persons will be
good. Having too many is no good.
I hope my message would be useful for further discussion. Please let me know
what you think.
Regards,
YO

[Jean-Paul Smets]

Hi,

I created http://www.erp5.org/Discussion/ReleaseProcess
and will summarize discussion.

Regards,

JPS.

Post by Łukasz Nowak
Hello,
It would be beneficial if this valuable discussion could be moved and
continued at the Discussions section of the wiki.
I shall do it 24 hrs from now if all are fine with it. In case of any
objections/thoughts please let me know prior to that.
Thanks,
SB

Post by Yoshinori Okuji
Hello ?ukasz,

Post by Łukasz Nowak
We present our idea how to organise release management for ERP5. We
would like to know your opinion about this idea. Any constructive
comments as well as criticism are welcome. We are making this proposal
in good faith - to provide the best possible solution in implementing
and development of ERP5 system, to give as much as possible possibility
to arise community of skilled developers around such great project as
ERP5 is itself.

First of all, I would like to thank you for writing this proposal. I highly
appreciate that you have made significant effort towards a better maintenance
process.
Let me describe how Nexedi has been maintaining the code, and succeeding in
- All the code base, including Business Templates, unit tests, functional
tests, and many other types of tools, required for generic ERP5, has been
always published on the subversion repository. No permission is required to
access the source code at any time. Everything is immediately open to anybody
in the world freely. This way, more people can take part in the reviewing
process, and facilitating contribution. Also, developers recognize more
responsibility for the quality, because of the potential risk.
- All the test results are sent to the public mailing list erp5-report every
day. People are free to check for the current status in the cutting-edge
development, thus it is easy to determine the latest stable revision from the
history of reports. If a result is good, but the system is still unstable,
this only means that we lack tests. Such tests should be complemented as soon
as possible, and contribution is very welcome.
- Chanages are, in principle, peer-reviewed. Only trivial changes, such as
fixing a typo, do not require such a review. Some people are not allowed to
directly check in changes, when we still don't know the quality level of code
produced by them. Once a person proves that she has a good skill, we give
more permissions to her, so that we will not prevent her work. However, even
experienced developers, such as me and Jean-Paul, need a peer-review, before
committing a significant change. This way, although it may not be perfect,
the quality of software is kept acceptally good all the time. Note that I
have been using the same policy in another Free Software project for many
years successfully.
- Backward compatibility is respected all the time. We break backward
compatibility, only if it is inevitable. Incompatibility is an evil, because
it produces too much work at a time when upgrading a system. We have been
publishing the information on incompatibility we made in the public wiki at
www.erp5.org, so that others will not get troubled. Still, we are always
trying to reduce incompatibility, and only deprecate old usage.
- Code base used for a customer's project is taken out of the public
subversion repository, under the control by a project manager. A decision is
made mostly based on the status of unit tests and functional tests, but also
with reviwing recent changes of the code, because some kind of problems may
not be very visible only with automatic testing. Then, the code base is
upgraded from time to time (the frequency depends on the constraints in a
project) so that bug fixes and feature enchancement are incorporated. Again,
the decision of which revision should be taken is based on the same criteria
as above, as well as project conditions in some cases. This way, we can
implement a stable system without much trouble.
As the technical leader of ERP5, I have been thinking of how to make releases
for some years, in a way that the maintenance cost would be negligible, and
When and How. I consider that there is no question about "Why". As you
pointed out, releases make things more apparent and visible, so they are
definitely nice to have.
- When
In my understanding, there are three different ways to manage the timings of
making releases.
One of them is feature-basis. When something significant is made, a new
release is published. This is often used in proprietary software products for
marketing purpose. Then, bug fixes are provided only in the form of patches.
In Open Source development, this does not make sense very much, since patches
can be included from the beginning. Instead, feature enhancement increments a
major version. For now, I am not talking about how to name versions, so I
don't take this way into consideration any more.
Another is regularity-basis. If I understand correctly, this is the same as
your suggestion. In this scheme, releases are made in the same frequency, no
matter whatever is happening. The advantage is that it is guaranteed that the
realtime characteristics of releases against the head is quite good, that is,
the distribution of delays behind the head is minimized. However, this way
has a critical drawback that it does not reflect the reality of development.
Sometimes a significant amount of development is performed in a short term,
and sometimes not. As the nature of development, changes, in particular,
which address new problems, tend to make software less stable. I will discuss
this issue more below.
The other is quality-basis. A new release is made when the head has good
quality of code. This is rather the reverse of the regularity-basis, so the
pros/cons are also reversed. This proceeds with the reality of development,
thus a delay can be potentially huge, but the quality of a release is well
maintained.
Of course, in most projects, these ways are more or less mixed. Even if the
quality-basis approach is taken, it does not make sense to make a new release
every day, only because the quality is high. Even with the regularity-basis
one, if the quality cannot meet criteria on time, a delay must be inserted to
prevent catastrophy. Many factors should be taken into consideration to keep
good release management.
As you might know, I have +10-year experience in many other Open Source/Free
Software projects. And, I have tried both the regularity-basis and
quality-basis ways. Overall, I have found that the quality-basis works far
better, because, for example, when a problem can be resolved appropriately by
a certain few people, and if they are not available for some reason for a
while, the regularity gets broken so easily. Enforcing a release quickly,
simply because of the policy, only produces unusable software.
Also, I have found that release management is far from trivial. Of course, as
you suggested, we use the result of tests for making estimate on stability,
but tests may not cover everything. It is sometimes because not all tests are
implemented yet. It is sometimes because managers are not capable of
determining whether software is stable enough or not. Especially in the
preparation of a new release, bug fixes must be encouraged, but contributors
have their own time constraints, and may not always follow a release cycle.
In this case, managers must be able to define what bug fixes can be postponed
with the idea of possible danger, and address problems by themselves,
whenever necessary.
Another perspective is how to facilitate people to contribute. Usually,
excellent developers want good feedback from managers. If they feel that
managers fully understand their changes, and evaluate the contribution
appropriately, they tend to help managers. Otherwise, they stop. This is not
related to whether they are paid or not. For instance, although workers at
Nexedi are paid, they are not always paid to contribute publicly. This part
really depends on their feeling and will. If they think that managers would
not understand their effort, they will just put changes locally and hide
them. This has been always the same as in volunteer-based projects, so I am
rather sure. Therefore, I believe that highly skilled persons must be
selected as release managers for ERP5. Otherwise, release management will
corrupt.
- How
There are two matters to consider: how many branches should be maintained or
developed, and how long a branch should be maintained.
The simplest approach is to use a single trunk, and no other branch. This is
the current way. The benefit is that all the effort heads towards one place,
so the efficiency of development is very high. We do not need to worry about
forking. The drawback is that the quality can be random along the time, no
matter however much effort is put into stabilization.
The semi-simple approach is, as you proposed, to use a main trunk, and one
branch to keep stability. And, to abandone older branches, once a new branch
is created. This is somehow easy and realistic. The benefit is that, without
having a lot of maintenance cost, stability is more or less ensured on a
branch. The drawback is that this way additionally requires porting changes
between a trunk and a branch, thus the development energy is scattered, and
extra effort and time are required for the maintenance.
For now, I compare only these two. Again, I have experience with both
approaches in the past. In short, I met difficulty in the latter approach,
because the cost of porting patches was quite higher than I expected. After
all, I gave it up, and I chose a single trunk method.
The way I did was the following: when I started to feel that I wanted to make
a new version, for example, because we had a good set of new functionality, I
forced people to stop writing a new feature, but encouraged people to test
the head, and fix it. In the meantime, all patches sent to me were waited,
and kept pending. Once I was satisified with the quality of code, I put a tag
and release it. Then, I restarted the normal cycle of development, and
integrating pending contribution.
Actually, this was so effective. All contributors understood that this was
required, and accepted that patches were in a pending state. Besides stopping
people to modify the head brutally, no extra cost was raised.
Surely, we must analyze if this way is effective for ERP5 as well, but I feel
that this is more appropriate.
The reason is really the cost. Unfortuantely, nobody is a superman. I don't
think it is a bad thing to have a stable branch, but at the same time, I
doubt if it is feasible. If we use branching, another team for a branch would
be necessary to be formed, just like the security team in Debian, because the
backporting is surprisingly hard and time-consuming.
If anybody wants to perform the backporting job, I would appreciate very much.
But be prepared that it is not trivial at all. One must be able to say what
is a pure fix, and must be able to segregate safe parts of patches from a
trunk, and adapt API usage, and so on. For now, I don't think this is
feasible with the current size of our community, but I would be very happy if
I am wrong.
So, at the moment, my suggestion is to use a single trunk. We have proven that
writing tests for a good coverage makes projects successful, because tests
always notify us of the current status, and what must be fixed, without any
branching. Release managers should promote people to write more tests for
missing parts, and negotiate with developers when next release should be
made, so that others can collaborate.
Now, this mail is getting too long, but let me describe some more additional
notes.
- It is a very good idea to describe API changes. We already do upgrade notes,
but this is a nice complement.
- Upgrade tests must be automated very well, otherwise it is too costly. Not
only unit tests, but also functional tests must be performed. Currently, the
number of functional tests is not good enough, in particular about
configuration-related stuff. Contribution is very welcome.
- Release management may not be independent of development management. Both
are very tightly coupled, because releasing depends on the status of
development. Only experts can perform this kind of job, so that other
developes will listen to their advise.
- I myself prefer co-maintenance, that is, having multiple maintainers. This
has a significant advantage in that even if one is not available (sick, busy,
or whatever), the project will not be stalled. I was proposed to be a
co-maintainer in a project, and it worked very well, so I'm proposing others
to do the same with me these days. I like it. For ERP5, 2-3 persons will be
good. Having too many is no good.
I hope my message would be useful for further discussion. Please let me know
what you think.
Regards,
YO

[Łukasz Nowak]

Hello Yoshinori,
On 2007-09-27, 23:49:40
Yoshinori Okuji <yo at nexedi.com> wrote:

(...)

Post by Yoshinori Okuji
I hope my message would be useful for further discussion. Please let
me know what you think.

From my (community user and developer) point of view *any* release
system will be cost cut-down.

What we proposed was semi-release scenario - after reading your email I
feel, that you want to provide releases as good as possible - that's
great. What driven us for such release process was cutting down costs
of releases from Nexedi and put it on community - so it would be more
like community driven releases - and it would be stated on release
related information. But (if I understood you well) you'd like to
Nexedi have full control of release process - then community will be
happy (I hope), because releases quality will be much higher.

I didn't know the state of discussion about releases in ERP5 - they
weren't publicised for now. So I'm quite happy to read that much about
ERP5 development. I hope that current development model of ERP5 is able
to accept any release model.

On related discussion page[1] Jean-Paul put nice time constraints on
releases, which are quite acceptable from community point of view (what
*community* thinks about it?)

As I'm pushing the idea of releases I'd like to know:

* how may we - the community - help Nexedi to start release
process
* how may we - the community - participate in maintenance of release
process when it begun
* when first release appear (in one week, month, quarter. year)

I hope that releases will become early enough. Right now I'm again
attached to quite old revision - that's because again my knowledge
about ERP5, related stuff, analysing erp5-report reports is not good
enough to give me warm feeling saying me "that revision is good, consume
few days to make successful upgrade to it". Release would give me (and
community, and even non-community implementers) such feeling (it would
be based on release quality).

I'm not impatient - when I'll know how long I'd have to wait for
releases, I'll try to synchronise my work with your plans.

Regards,
Luke

[1] http://www.erp5.org/Discussion/ReleaseProcess

PS. It would be great if other community members would activate on this
topic - I think releases that's what community lack most, so it is
up to community to have them. And *help* with it.

[bartek]

Post by Łukasz Nowak
Hello Yoshinori,
On 2007-09-27, 23:49:40
(...)

Post by Yoshinori Okuji
I hope my message would be useful for further discussion. Please let
me know what you think.

From my (community user and developer) point of view *any* release
system will be cost cut-down.

[cut]

Post by Łukasz Nowak
PS. It would be great if other community members would activate on this
topic - I think releases that's what community lack most, so it is
up to community to have them. And *help* with it.

I'm not sure who I am - member of community or part of Nexedi, I guess
I'm something in between ;)

What I can say is that I'd like to help - I'll be more than happy to
invest a significat amount of man-days to make it happen, I believe
it'll pay back very soon, to me and to others.

Bartek

[Yoshinori Okuji]

Post by bartek
What I can say is that I'd like to help - I'll be more than happy to
invest a significat amount of man-days to make it happen, I believe
it'll pay back very soon, to me and to others.

Thank you very much for your offer. Your contribution would be quite helpful
in whatever ways.

YO

[Yoshinori Okuji]

Post by Łukasz Nowak
What we proposed was semi-release scenario - after reading your email I
feel, that you want to provide releases as good as possible - that's
great. What driven us for such release process was cutting down costs
of releases from Nexedi and put it on community - so it would be more
like community driven releases - and it would be stated on release
related information. But (if I understood you well) you'd like to
Nexedi have full control of release process - then community will be
happy (I hope), because releases quality will be much higher.

No, this is _not_ what I meant. I mentioned that release management should not
be independent of development processes, and a few (not only single, but also
not too many) highly skilled people should collaborate. I do not care about
whether release managers are working for Nexedi or not, as long as they are
capable of handling the management gracefully, just as the Zope2 maintainer
is not working for Zope Corporation, but a leading developer.

My belief comes from my experience in other projects. If you look at some
Linux distributions, such as Debian and Red Hat, the package maintainers, who
can be viewed as release managers in a sense, very often developers in the
upstream as well, and contribute tremendously. When they don't, sometimes
happening in other distributions, no collaboration exists, and their
relationship with real developers is extremely bad.

Also, note that I regard you as a candidate of a release manager. You have
been contributing significantly, and you have proven that you have good
skills. Naturally, it depends on what you would like to do, but I would feel
very good if you would be a member of such release managemet.

Post by Łukasz Nowak
I didn't know the state of discussion about releases in ERP5 - they
weren't publicised for now. So I'm quite happy to read that much about
ERP5 development. I hope that current development model of ERP5 is able
to accept any release model.

I am sorry that I hadn't before, but it was only because my idea was still
very vague. Perhaps, nearly all people in Nexedi even didn't know that I had
been thinking about this for a long time.

YO

[bartek]

Post by Yoshinori Okuji

Post by Łukasz Nowak
What we proposed was semi-release scenario - after reading your email I
feel, that you want to provide releases as good as possible - that's
great. What driven us for such release process was cutting down costs
of releases from Nexedi and put it on community - so it would be more
like community driven releases - and it would be stated on release
related information. But (if I understood you well) you'd like to
Nexedi have full control of release process - then community will be
happy (I hope), because releases quality will be much higher.

No, this is _not_ what I meant. I mentioned that release management should not
be independent of development processes, and a few (not only single, but also
not too many) highly skilled people should collaborate. I do not care about
whether release managers are working for Nexedi or not, as long as they are

Your are absolutely right. However, as for now all core developers are
within Nexedi, the development process is coordinated by Nexedi, and
also Nexedi is in charge of the svn repository. If we choose a
single-trunk way, as you suggested, and there is to be a feature freeze
for a week or two, then it has to be managed and coordinated by Nexedi.
Which is not a bad thing, after all :)

That said, I agree that non-Nexedi people can take part in the process,
also as members of release management team.

Bartek

Post by Yoshinori Okuji
capable of handling the management gracefully, just as the Zope2 maintainer
is not working for Zope Corporation, but a leading developer.
My belief comes from my experience in other projects. If you look at some
Linux distributions, such as Debian and Red Hat, the package maintainers, who
can be viewed as release managers in a sense, very often developers in the
upstream as well, and contribute tremendously. When they don't, sometimes
happening in other distributions, no collaboration exists, and their
relationship with real developers is extremely bad.
Also, note that I regard you as a candidate of a release manager. You have
been contributing significantly, and you have proven that you have good
skills. Naturally, it depends on what you would like to do, but I would feel
very good if you would be a member of such release managemet.

Post by Łukasz Nowak
I didn't know the state of discussion about releases in ERP5 - they
weren't publicised for now. So I'm quite happy to read that much about
ERP5 development. I hope that current development model of ERP5 is able
to accept any release model.

I am sorry that I hadn't before, but it was only because my idea was still
very vague. Perhaps, nearly all people in Nexedi even didn't know that I had
been thinking about this for a long time.
YO

[bartek]

Yoshinori Okuji wrote:
[cut]

Post by Yoshinori Okuji
The way I did was the following: when I started to feel that I wanted to make
a new version, for example, because we had a good set of new functionality, I
forced people to stop writing a new feature, but encouraged people to test
the head, and fix it. In the meantime, all patches sent to me were waited,
and kept pending. Once I was satisified with the quality of code, I put a tag
and release it. Then, I restarted the normal cycle of development, and
integrating pending contribution.
Actually, this was so effective. All contributors understood that this was
required, and accepted that patches were in a pending state. Besides stopping
people to modify the head brutally, no extra cost was raised.
Surely, we must analyze if this way is effective for ERP5 as well, but I feel
that this is more appropriate.

This sounds very nice. My question would be, what do we do then if an
important bugfix is made some time after the release? Seems to me some
backporting/patching job will still be needed, because there is no way
to do a bugfix release since we are in single-trunk and the development
has already restarted.

Bartek

[Yoshinori Okuji]

Post by bartek
This sounds very nice. My question would be, what do we do then if an
important bugfix is made some time after the release? Seems to me some
backporting/patching job will still be needed, because there is no way
to do a bugfix release since we are in single-trunk and the development
has already restarted.

My preference is, not to keep the trunk unstable too long anyway. In
principle, the trunk should be - ideally - usable at any time. I know this is
impossible in practice, but it should be feasible to make things stable again
in a few days or a few weeks, and we have been doing so.

Then, backporting would not be really required. Let's say, we have made a
release which contained a serious bug. If the head is stable enough, we can
publish a new release immediately. If not, we make the head stable as soon as
possible, hopefully in some days, and release it.

This way has been effective in my other projects, so I think there is no
reason that it does not work in ERP5.

However, as I mentioned, if we can allocate people enough for backporting, I
don't object to backport bugfixes.

YO

[bartek]

Post by Yoshinori Okuji

Post by bartek
This sounds very nice. My question would be, what do we do then if an
important bugfix is made some time after the release? Seems to me some
backporting/patching job will still be needed, because there is no way
to do a bugfix release since we are in single-trunk and the development
has already restarted.

My preference is, not to keep the trunk unstable too long anyway. In
principle, the trunk should be - ideally - usable at any time. I know this is
impossible in practice, but it should be feasible to make things stable again
in a few days or a few weeks, and we have been doing so.

So, back to your classification, it would be purely "quality-based"
releasing?

Post by Yoshinori Okuji
Then, backporting would not be really required. Let's say, we have made a
release which contained a serious bug. If the head is stable enough, we can
publish a new release immediately. If not, we make the head stable as soon as
possible, hopefully in some days, and release it.
This way has been effective in my other projects, so I think there is no
reason that it does not work in ERP5.
However, as I mentioned, if we can allocate people enough for backporting, I
don't object to backport bugfixes.

Some of it seems inevitable - like, if a bugfix is made after the
developers have already begun work on a major new feature.

B.

Post by Yoshinori Okuji
YO