[Erp5-dev] Status of ZSQLCatalog

[Racinet Georges]

Hi there,

First post on this list, feel free to redirect If this isn't the right
place !

Let me introduce myself : I am currently the main maintainer of CPS,
the content management framework developped by Nuxeo on top of Zope 2
and CMF prior to their switch to the Java world.

I've never had the opportunity to really work myself with ZSQLCatalog,
although I've seen a few in the wild being used as a ZCatalog
replacement in CPS installations, but today might be interested with
that option.
Depending on the environment and actual usage, such a solution can be
more effective and/or manageable than the Lucene based ZCatalog
replacement we have (NXLucene server and CPSLuceneCatalog).

Now the questions :

- is ZSQLCatalog considered an active project ? There are recent
changesets in svn, but that doesn't really look like active development.
- what is the transactional behaviour ? Am I right to think it's
simply not part of the overall transaction ? Any plans to leverage
something like z3c.sqlalchemy (provides two-phase commit out of the
box) ?
- what are the currently supported Zope versions ? For the record,
right now, CPS runs on Zope 2.9.x (x >= 7)

Needless to say, if we were to make ZSQLAlchemy a standard option in
CPS, we would be glad to contribute.

Thanks for your attention
--
Georges Racinet, http://www.racinet.fr
Zope/CPS/Plone expertise, assistance & development
GPG: 0x4862FFF7

[Leonardo Rochael Almeida]

Post by Racinet Georges
Hi there,

Hi Racinet,

Post by Racinet Georges
[...]
- is ZSQLCatalog considered an active project ? There are recent
changesets in svn, but that doesn't really look like active development.

We're actively using it all the time. The lack of recent commits reflect
it's somewhat unchanging requirements.

Post by Racinet Georges
- what is the transactional behaviour ? Am I right to think it's
simply not part of the overall transaction ? Any plans to leverage
something like z3c.sqlalchemy (provides two-phase commit out of the
box) ?

It is transactional in as much as the ZSQLMethods involved rely on the
transactional behaviour of ZMySQLDA.

Post by Racinet Georges
- what are the currently supported Zope versions ? For the record,
right now, CPS runs on Zope 2.9.x (x >= 7)

We work with Zope 2.8 and Zope 2.12 (and nothing in between. Don't ask),
so these are the platforms we intend to support. We have tests running
on Zope 2.12, but no production deployment yet. We should have one at
some point in this year.

Post by Racinet Georges
Needless to say, if we were to make ZSQLAlchemy a standard option in
CPS, we would be glad to contribute.

I take it you meant ZSQLCatalog. That would be very welcome, though if
you happen create some sort of ZSQLAlchemyMethod and get ZSQLCatalog to
work with it, that would be also very cool :-)

Cheers,

Leo

[Racinet Georges]

Hi Leonardo,

thanks for the quick reply.

Post by Leonardo Rochael Almeida

Post by Racinet Georges
- is ZSQLCatalog considered an active project ? There are recent
changesets in svn, but that doesn't really look like active
development.

We're actively using it all the time. The lack of recent commits reflect
it's somewhat unchanging requirements.

Thanks, that's exactly what we needed to know.

Post by Leonardo Rochael Almeida

Post by Racinet Georges
- what is the transactional behaviour ? Am I right to think it's
simply not part of the overall transaction ? Any plans to leverage
something like z3c.sqlalchemy (provides two-phase commit out of the
box) ?

It is transactional in as much as the ZSQLMethods involved rely on the
transactional behaviour of ZMySQLDA.

I see, still have to dig a bit in the design. In particular I didn't
realize that ZSQLMethods were so deeply involved in the process.

Post by Leonardo Rochael Almeida

Post by Racinet Georges
- what are the currently supported Zope versions ? For the record,
right now, CPS runs on Zope 2.9.x (x >= 7)

We work with Zope 2.8 and Zope 2.12 (and nothing in between. Don't ask),
so these are the platforms we intend to support. We have tests running
on Zope 2.12, but no production deployment yet. We should have one at
some point in this year.

Best wishes for that big jump !

Post by Leonardo Rochael Almeida

Post by Racinet Georges
Needless to say, if we were to make ZSQLAlchemy a standard option in
CPS, we would be glad to contribute.

I take it you meant ZSQLCatalog.

Oops, indeed.

Post by Leonardo Rochael Almeida
That would be very welcome, though if
you happen create some sort of ZSQLAlchemyMethod and get ZSQLCatalog to
work with it, that would be also very cool :-)

For the time being, we're still evaluating options, so that's a bit
early to say, but we'd get back to you anyway
Best,
--
Georges Racinet, http://www.racinet.fr
Zope/CPS/Plone expertise, assistance & development
GPG: 0x4862FFF7

[Santi Camps]

Post by Racinet Georges
Hi there,
First post on this list, feel free to redirect If this isn't the right
place !
Let me introduce myself : I am currently the main maintainer of CPS,
the content management framework developped by Nuxeo on top of Zope 2
and CMF prior to their switch to the Java world.
I've never had the opportunity to really work myself with ZSQLCatalog,
although I've seen a few in the wild being used as a ZCatalog
replacement in CPS installations, but today might be interested with
that option.
Depending on the environment and actual usage, such a solution can be
more effective and/or manageable than the Lucene based ZCatalog
replacement we have (NXLucene server and CPSLuceneCatalog).
- is ZSQLCatalog considered an active project ? There are recent
changesets in svn, but that doesn't really look like active development.
- what is the transactional behaviour ? Am I right to think it's
simply not part of the overall transaction ? Any plans to leverage
something like z3c.sqlalchemy (provides two-phase commit out of the
box) ?
- what are the currently supported Zope versions ? For the record,
right now, CPS runs on Zope 2.9.x (x >= 7)
Needless to say, if we were to make ZSQLAlchemy a standard option in
CPS, we would be glad to contribute.
Thanks for your attention
--
Georges Racinet, http://www.racinet.fr
Zope/CPS/Plone expertise, assistance & development
GPG: 0x4862FFF7

Hi Georges

I'm the main developer of KMKey (based on CPS 3). We have had similar
ideas, I think. We already have a version of ZSQLCatalog adapted to work
with Zope 2.9.4, and using postgresql instead of mysql. It needs some
more fixes, but is still working in testing environments

You can obtain it using subversion at
https://svn.forge.osor.eu/svn/kmkey/vendors/ZSQLCatalog/branches/kmkey

We also have a branch of CPSSchemas that writes to / reads from postgresql
tables data of portal_schemas market with an specific SQL attribute. This
is working in production environments and we are planning to write adapters
for other databases

svn co https://svn.forge.osor.eu/svn/kmkey/vendors/CPSSchemas/branches/kmkey

All this works within the overall transaction, the only problem is that the
Undo feature is broken

If you find something interesting, it will be a pleasure to see some of this
features back to CPS

Best regards

[Racinet Georges]

Post by Santi Camps
I'm the main developer of KMKey (based on CPS 3). We have had
similar ideas, I think. We already have a version of ZSQLCatalog
adapted to work with Zope 2.9.4, and using postgresql instead of
mysql. It needs some more fixes, but is still working in testing
environments
You can obtain it using subversion at https://svn.forge.osor.eu/svn/kmkey/vendors/ZSQLCatalog/branches/kmkey

Hi Santi,

thanks for the pointer. I'll take a look ASAP.

Post by Santi Camps
We also have a branch of CPSSchemas that writes to / reads from
postgresql tables data of portal_schemas

(...)

Post by Santi Camps
If you find something interesting, it will be a pleasure to see some
of this features back to CPS

Interesting, and thanks for the proposal !
I'll take a look at this as well, but probably later. Public follow-up
on cps-devel at lists.nuxeo.com, of course :-)

Cheers,
--
Georges Racinet, http://www.racinet.fr
Zope/CPS/Plone expertise, assistance & development
GPG: 0x4862FFF7

[Jean-Paul Smets]

Hi,

ZSQLCatalog was ported ages ago to be used in CPS, together with
ERP5SyncML and CMFActivity. We could, at that time (2004 ?), demonstrate
- fast queries in CPS
- replication of CPS documents

It was working really well but... CPS developers did not really have a
look at it and kept on using ZCatalog.

It is very nice to see that Santi understood how ZSQLCatalog can be
useful. Porting to Plone would be also very easy. I think we stated a
few years ago that we are happy that anyone ports ZSQLCatalog to be used
in other frameworks. We will be happy to incorporate any patches which
do not degrade performance. Any dependency to ERP5 can be removed or
catched if needed to make ZSQLCatalog friendly to other frameworks.

We also wrote some years ago a document about parts of ERP5 which can be
used in other frameworks to make them as scalable as ERP5 is:
http://www.erp5.com/kb/enterprise-High.Performance.Zope?format=pdf

FYI, ERP5 and ZSQLCatalog are handling everyday somewhere in a company
in Europe 100,000 scanned PDF mails, processed by OCR, searchable in
full text or by SQL queries and accessed by 300 concurrent users.
Handling 1,000,000 would not be an issue.

Regards,

JPS.

On Wed, Jan 20, 2010 at 12:17 PM, Racinet Georges <georges at racinet.fr
Hi there,
First post on this list, feel free to redirect If this isn't the right
place !
Let me introduce myself : I am currently the main maintainer of CPS,
the content management framework developped by Nuxeo on top of Zope 2
and CMF prior to their switch to the Java world.
I've never had the opportunity to really work myself with ZSQLCatalog,
although I've seen a few in the wild being used as a ZCatalog
replacement in CPS installations, but today might be interested with
that option.
Depending on the environment and actual usage, such a solution can be
more effective and/or manageable than the Lucene based ZCatalog
replacement we have (NXLucene server and CPSLuceneCatalog).
- is ZSQLCatalog considered an active project ? There are recent
changesets in svn, but that doesn't really look like active development.
- what is the transactional behaviour ? Am I right to think it's
simply not part of the overall transaction ? Any plans to leverage
something like z3c.sqlalchemy (provides two-phase commit out of the
box) ?
- what are the currently supported Zope versions ? For the record,
right now, CPS runs on Zope 2.9.x (x >= 7)
Needless to say, if we were to make ZSQLAlchemy a standard option in
CPS, we would be glad to contribute.
Thanks for your attention
--
Georges Racinet, http://www.racinet.fr
Zope/CPS/Plone expertise, assistance & development
GPG: 0x4862FFF7
Hi Georges
I'm the main developer of KMKey (based on CPS 3). We have had
similar ideas, I think. We already have a version of ZSQLCatalog
adapted to work with Zope 2.9.4, and using postgresql instead of
mysql. It needs some more fixes, but is still working in testing
environments
You can obtain it using subversion at
https://svn.forge.osor.eu/svn/kmkey/vendors/ZSQLCatalog/branches/kmkey
We also have a branch of CPSSchemas that writes to / reads from
postgresql tables data of portal_schemas market with an specific SQL
attribute. This is working in production environments and we are
planning to write adapters for other databases
svn co
https://svn.forge.osor.eu/svn/kmkey/vendors/CPSSchemas/branches/kmkey
All this works within the overall transaction, the only problem is
that the Undo feature is broken
If you find something interesting, it will be a pleasure to see some
of this features back to CPS
Best regards
--
Santi Camps
KMKey hacker (http://www.kmkey.com)
Earcon S.L. (http://www.earcon.com)
------------------------------------------------------------------------
_______________________________________________
Erp5-dev mailing list
Erp5-dev at erp5.org
http://mail.nexedi.com/mailman/listinfo/erp5-dev

[Racinet Georges]

Hi Jean-Paul,

this sounds overall very promising, we'll definitely do a few
experiments and get back to you in a few weeks.

Post by Jean-Paul Smets
ZSQLCatalog was ported ages ago to be used in CPS, together with
ERP5SyncML and CMFActivity. We could, at that time (2004 ?),
demonstrate
- fast queries in CPS
- replication of CPS documents
It was working really well but... CPS developers did not really have a
look at it and kept on using ZCatalog.

Point taken. I wasn't an active CPS developer at that time, and have
no clue of the technical reasons for that, if any.
The current active developers don't care much about history either, so
please consider this as a fresh start.

Post by Jean-Paul Smets
It is very nice to see that Santi understood how ZSQLCatalog can be
useful. Porting to Plone would be also very easy. I think we stated a
few years ago that we are happy that anyone ports ZSQLCatalog to be used
in other frameworks. We will be happy to incorporate any patches which
do not degrade performance. Any dependency to ERP5 can be removed or
catched if needed to make ZSQLCatalog friendly to other frameworks.
We also wrote some years ago a document about parts of ERP5 which can be
http://www.erp5.com/kb/enterprise-High.Performance.Zope?format=pdf

Interesting read, thanks.

Post by Jean-Paul Smets
FYI, ERP5 and ZSQLCatalog are handling everyday somewhere in a
company in Europe 100,000 scanned PDF mails, processed by OCR,
searchable in full text or by SQL queries and accessed by 300
concurrent users. Handling 1,000,000 would not be an issue.

This surely fits our current needs.

Best,

--
GR

Post by Jean-Paul Smets

On Wed, Jan 20, 2010 at 12:17 PM, Racinet Georges
Hi there,
First post on this list, feel free to redirect If this isn't the right
place !
Let me introduce myself : I am currently the main maintainer of CPS,
the content management framework developped by Nuxeo on top of Zope 2
and CMF prior to their switch to the Java world.
I've never had the opportunity to really work myself with
ZSQLCatalog,
although I've seen a few in the wild being used as a ZCatalog
replacement in CPS installations, but today might be interested with
that option.
Depending on the environment and actual usage, such a solution can be
more effective and/or manageable than the Lucene based ZCatalog
replacement we have (NXLucene server and CPSLuceneCatalog).
- is ZSQLCatalog considered an active project ? There are recent
changesets in svn, but that doesn't really look like active development.
- what is the transactional behaviour ? Am I right to think it's
simply not part of the overall transaction ? Any plans to leverage
something like z3c.sqlalchemy (provides two-phase commit out of the
box) ?
- what are the currently supported Zope versions ? For the record,
right now, CPS runs on Zope 2.9.x (x >= 7)
Needless to say, if we were to make ZSQLAlchemy a standard
option in
CPS, we would be glad to contribute.
Thanks for your attention
--
Georges Racinet, http://www.racinet.fr
Zope/CPS/Plone expertise, assistance & development
GPG: 0x4862FFF7
Hi Georges
I'm the main developer of KMKey (based on CPS 3). We have had
similar ideas, I think. We already have a version of ZSQLCatalog
adapted to work with Zope 2.9.4, and using postgresql instead of
mysql. It needs some more fixes, but is still working in testing
environments
You can obtain it using subversion at https://svn.forge.osor.eu/svn/kmkey/vendors/ZSQLCatalog/branches/kmkey
We also have a branch of CPSSchemas that writes to / reads from
postgresql tables data of portal_schemas market with an specific
SQL attribute. This is working in production environments and we
are planning to write adapters for other databases
svn co https://svn.forge.osor.eu/svn/kmkey/vendors/CPSSchemas/branches/kmkey
All this works within the overall transaction, the only problem is
that the Undo feature is broken
If you find something interesting, it will be a pleasure to see
some of this features back to CPS
Best regards
--
Santi Camps
KMKey hacker (http://www.kmkey.com)
Earcon S.L. (http://www.earcon.com)
------------------------------------------------------------------------
_______________________________________________
Erp5-dev mailing list
Erp5-dev at erp5.org
http://mail.nexedi.com/mailman/listinfo/erp5-dev

--
Jean-Paul Smets-Solanes, Nexedi CEO - Tel. [phone]
ERP5 Enterprise: Open Source ERP/CRM for Mission Critical Applications
http://www.erp5.com
TioLive SaaS: run your business online, with more freedom
http://www.tiolive.com
Nexedi: Consulting and Development of Free / Open Source Software
http://www.nexedi.com

--
Georges Racinet, http://www.racinet.fr
Zope/CPS/Plone expertise, assistance & development
GPG: 0x4862FFF7