Hi,

I have programmatically imported a directory tree of text files, with a
directory tree depth of 4, up to 200 folders total, and more than 23
thousand text files into my Plone 4.1.2 site. More than 25 thousand tags are
also imported from those documents.

Everything is cool, navigation and load times of pages is excellent, tags
are working nicely, search form works great. But when I click on the edit
link on the green edit bar to edit any of those pages, it takes 5 minutes to
see the edit page with tinymce, and python executable goes crazy with %100
cpu during this 5 minutes.

Without chaning anything, I click cancel and then re-edit, and the same
happens.

Any ideas? What could be happening?

Note: Debug console does not show anything (except the usual messages, the
last being "INFO Zope Ready to handle requests")

-----
ikken-tobi is never wrong

What does the ZODB cache stats tell you? (Control Panel in the ZMI)?
How large is your zodb cache? Is this a cold cache problem?

-aj

while trying to see cache details, I think I found a bug:

I restart my plone and visit Control_Panel/Database/cache_detail and I get
the following traceback (cache_extreme_detail link works, these links are at
Control_Panel/DebugInfo/manage_main):

  Module ZPublisher.Publish, line 126, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module Products.PDBDebugMode.runcall, line 70, in pdb_runcall
  Module ZPublisher.Publish, line 46, in call_object
  Module App.CacheManager, line 105, in cache_detail
  Module App.CacheManager, line 105, in <genexpr>
TypeError: %d format: a number is required, not str
-> return '\n'.join('%6d %s'%(count, name) for count, name in detail)

-----
ikken-tobi is never wrong

Please file a bug report with the Zope 2 bug tracker.

-aj

That is, here:  https://bugs.launchpad.net/zope2/

----- http://du-alter-schwede.de

https://bugs.launchpad.net/zope2/+bug/889024

-----
ikken-tobi is never wrong

A profiler may be able to provide you with details about what is
happening.

Apparently, Python has fixed " http://bugs.python.org/issue7372".

Depending on how the bug was fixed, "Products.ZopeProfiler"
may work again for Python 2.6/2.7 (it will when the bug fix has
not changed the "profile" data structures but enhanced "pstats" to
work with both "cProfile" and "profile" data).

I will look at "Products.ZopeProfiler" in the near future
and try to make sure that it will work with current Python versions
(potentially the patch from " http://bugs.python.org/issue7372". needs
to be applied to a local Python installation).

When it works again, you could use "Products.ZopeProfiler" to
learn what your time has been spent for.

--
Dieter

The cache details will not show you the primary cause for the problem --
it may only show you that a far too large number of objects (compared
to the cache size) is needed.

But editing a single of your documents should not involve a large
number of objects. If it does, something is wrong with the persistency
design.
Formerly (versions before Plone 4), such a problem could have been
caused by using a simple "Folder" to maintain a huge number of objects.
But, with Plone 4 and up, this should no longer happen (without
3rd party components that maybe are not yet intelligent enough).

--
Dieter

hm, i'm using the system's (ubuntu 11.04) python which reports itself as
version 2.7.1+. Chances are this fix is already in it. I'll try the profiler
and see what happens..
Thanks..

-----
ikken-tobi is never wrong

Note that we recommend to use Python 2.6.x for Plone 4.
While Python 2.7.x may work it is not officially supported.

Not using the system but a custom compiled Python is
generally recommended as well.

Raphael

Chances are this fix is already in it. I'll try the profiler

Your problem might be related to  http://dev.plone.org/ticket/8792:   the
current version of the keyword widget in Products.Archetypes is very
inefficient when dealing with a large number of tags due to the usage of
restricted python. Using the provided diff to patch the keyword.pt could
solve your original performance problem.

The patch very much improved the load time of the edit page, it is about a
minute now. This time my browser (javascript i guess) goes %100 cpu during
this one minute. Still much better and managable, Thanks a bunch..

-----
ikken-tobi is never wrong

I uploaded a new version of "Products.ZopeProfiler" to PyPI which works
with Zope 2.10 to 2.13, provided you are using a Python version
where " http://bugs.python.org/issue7372"  is fixed.

--
Dieter

quick update, i think google chome (15.0.874.120) under ubuntu has some
issues with javascript, i realized it is much faster with Firefox (7.0.1).
Chrome also has problems with showign the results of multi-part searches
(like body contains "abc" and title contains "123"), whereas Firefox is
still fine.

-----
ikken-tobi is never wrong