[Erp5-dev] expand method performance, ZEO questions

[Łukasz Nowak]

Hello,

I've been hit by performance problem.

We have transformation with many resources (we have plenty of them in
our case). Using this transformation generates applied rule
with one simulation movement, again applied rule, then more than 50
simulation movements which has AR/SM pair. Supply chain is quite simple
- two nodes.

Invoking expand method on root of this tree lasts for about 80 seconds
on not loaded system, with hardware RAID0 on PCI-X and is consuming
100% of core of Opteron 265.

My questions:
- ZEO won't help in this case directly, as expand is one method in
activites? (we are going to use ZEO anyway)
- does expand need to be so CPU intensive?
- is there any way/idea to split expand into few smaller expands and
use clustering for them?

And another question: are there any updated
guidelines/documentation/ideas/experience docs/tips/anything about using
ZEO with ERP5?

I didn't suppose to find HowToUseZeo on ERP5 wiki ;), but there is not
too much info about this on wiki. Well - maybe it is so simple, that is
not worth mentioning it?

Regards,
Luke

[Yoshinori Okuji]

Hello,

Post by Łukasz Nowak
We have transformation with many resources (we have plenty of them in
our case). Using this transformation generates applied rule
with one simulation movement, again applied rule, then more than 50
simulation movements which has AR/SM pair. Supply chain is quite simple
- two nodes.
Invoking expand method on root of this tree lasts for about 80 seconds
on not loaded system, with hardware RAID0 on PCI-X and is consuming
100% of core of Opteron 265.
- ZEO won't help in this case directly, as expand is one method in
activites? (we are going to use ZEO anyway)

Without customization, no.

Post by Łukasz Nowak
- does expand need to be so CPU intensive?

expand is always CPU-bounded (unless MySQL becomes a bottleneck).

Post by Łukasz Nowak
- is there any way/idea to split expand into few smaller expands and
use clustering for them?

It is difficult to suggest an appropriate way without looking at your system.

Basically, the simulation system in ERP5 is created with flexibility and power
in mind. This is very good in most cases. But as in your case, it can be slow
in some cases, thus a sort of optimization is required.

The best way is to first profile it and understand what the bottleneck is (as
written in erp5.org). Intuitively, as you have many applied rules, the
selections of rules might be the bottleneck, because this part is completely
dynamic, so that we can specify very complex criteria to select an
appropriate rule. But this is sometimes overkill, and it could be acceptable
to "hardcode" rules in many cases. However, it is very dangerous to assume
something is a bottleneck without verifying it, since it can be a premature
optimization, and can rather decrease the performance / flexibility

Post by Łukasz Nowak
And another question: are there any updated
guidelines/documentation/ideas/experience docs/tips/anything about using
ZEO with ERP5?
I didn't suppose to find HowToUseZeo on ERP5 wiki ;), but there is not
too much info about this on wiki. Well - maybe it is so simple, that is
not worth mentioning it?

As you guess, it is so simple. How to set up ZEO is described in the Zope
Book, so I didn't feel that it would be worth writing in the wiki.

YO

[Łukasz Nowak]

Hello,

On 2007-12-22, 17:26:21
Yoshinori Okuji <yo at nexedi.com> wrote:

(...)

Post by Yoshinori Okuji

Post by Łukasz Nowak
- is there any way/idea to split expand into few smaller expands
and use clustering for them?

It is difficult to suggest an appropriate way without looking at your system.
Basically, the simulation system in ERP5 is created with flexibility
and power in mind. This is very good in most cases. But as in your
case, it can be slow in some cases, thus a sort of optimization is
required.
The best way is to first profile it and understand what the
bottleneck is (as written in erp5.org). Intuitively, as you have many
applied rules, the selections of rules might be the bottleneck,
because this part is completely dynamic, so that we can specify very
complex criteria to select an appropriate rule. But this is sometimes
overkill, and it could be acceptable to "hardcode" rules in many
cases. However, it is very dangerous to assume something is a
bottleneck without verifying it, since it can be a premature
optimization, and can rather decrease the performance / flexibility

So *before* starting any optimisation I'll have to have full unit test
scenarios. I'll play with it as much as possible and share my ideas
here.

Post by Yoshinori Okuji

Post by Łukasz Nowak
And another question: are there any updated
guidelines/documentation/ideas/experience docs/tips/anything about
using ZEO with ERP5?
I didn't suppose to find HowToUseZeo on ERP5 wiki ;), but there is
not too much info about this on wiki. Well - maybe it is so simple,
that is not worth mentioning it?

As you guess, it is so simple. How to set up ZEO is described in the
Zope Book, so I didn't feel that it would be worth writing in the
wiki.

Yes - it was so obvious - I left just few notes about ZEO usage on mine
wikipage.

Thanks,
Luke