Team Chat Logs

coderanger: disk at first, but when writing my mail i had a vague idea about how to implement a mixture of memory and disk-based caching (to make things more complex ;))

[01:40:16]

hmm, there is a python API for memcached

[01:40:27]

disk-based makes it hard to scale the system

[01:40:51]

and re my question above: no second e-mail confirmation to be expected, just looked at that confirmation webpage again

[01:41:55]

coderanger: hmm, memcached sounds interesting...

[01:42:27]

otaku42: I had been pondering making a plugin for soemthing similar to that

[01:43:31]

coderanger: that would be great... even more since it would take me ages to get something like that up and running, given my rather bad python skills :)

[01:44:04]

they seem to be sufficient for simple mods, but i guess they are not good enough for something like this caching stuff

[01:45:07]

Are you guys using 0.10 or 0.11?

[01:45:34]

coderanger: we currently use 0.9 and will switch to 0.10 along with moving the site to the new server.

[01:46:20]

hrmm

[01:46:44]

coderanger: i'd even consider to move to 0.11 instead, given that:

[01:46:56]

coderanger: a. it's stable enough for production use

[01:47:13]

coderanger: b. not too many changes are needed to get 0.10-plugins running.

[01:48:04]

Not sure on a., but b. is pretty true

[01:48:20]

this should be simple enough to do for 0.10 and then port though

[01:48:39]

coderanger: then why not writing the cache plugin for 0.10 instead? :)

[01:48:46]

heh, you see :)

[01:49:00]

That was my point ;-)

[01:50:25]

i'd expect a pretty performance boost at least on slow machines

[01:50:54]

think of changesets: they never change. most wiki pages should be pretty static. milestones as well.

[01:51:08]

tickets will get more frequent updates, but it should still be notable.

[01:51:15]

well changesets are probably a bad thing to cache

[01:51:24]

as they aren't each accessed much

[01:52:14]

but wiki pages are a good starting place

[01:53:18]

maybe changeset caching should be optional... we have some pretty large changesets (resulting in 30mb html - although that probably could be addressed in a different way)

[01:53:46]

otaku42: I am thinking of a modular ICachingStrategy

[01:53:59]

So it can decide based on log data what to cache or something like that

[01:54:07]

coderanger: aye.

[01:54:18]

you can limit the max size shows for changesets

[01:54:41]

coderanger: is that a feature of 0.10, or available in 0.9 as well?

[01:55:08]

not sure

[01:55:59]

ah, teo says "since 0.10" for max_diff_bytes

[01:56:33]

coderanger: thanks for the hint :)

[01:59:45]

otaku42: Another thing that could use some caching is the permissions system

[02:00:00]

A lot of the permissions computations are very DB intensive

[02:00:21]

Thats one of the ways I got trac.adiumx.com to be a little faster

[02:00:39]

I take it you guys are already using postgres and psycopg2?

[02:03:28]

coderanger: nope, we're running on sqlite

[02:03:35]

You should fix that

[02:04:58]

coderanger: is sqlite performance that bad?

[02:05:23]

its not an issue of performance so much as efficiency

[02:16:45]

coderanger: any experiences regarding efficiency/performance regarding mod_python against other options (fast_cgi, ...)?

[02:17:18]

I can say trac.adiumx is running FCGI on Lighty and its holding up pretty well

[02:17:30]

it took 100k requests in about 2 days

[02:18:27]

I like mod_python, but you do have to punt apache periodically because it tends to gobble RAM

[02:19:18]

coderanger: madwifi.org had about 200000 requests during the last 2 days... on the old server

[02:19:51]

You guys can't find someone to donate a nicer server?

[02:20:57]

coderanger: good question... :)

[02:21:59]

i think the "new" server (the dedicated one which has been donated) should be able to handle the requests and stuff quite nicely... unless the processes need to much cpu power for content rendering ;)

[02:26:57]

okay, got cmemcache compiled

[02:27:00]

here goes nothing

[02:36:23]

coderanger: here goes nothing?

[02:36:55]

otaku42: Its a colloquialism, "Time to try <x>"

[02:37:15]

coderanger: i see :)

[02:48:41]

coderanger: does the above mean that you really consider (again) to work on such a plugin/mod?

[02:48:53]

otaku42: I currently am

[02:49:58]

coderanger++ :)

[02:56:44]

coderanger: please let me know if i can be of any help (testing, for example)

[02:57:07]

otaku42: It'l take me another hour or so before its functional enough to test

[02:57:11]

but I'll get back to you then

[02:57:22]

coderanger: great

[02:57:48]

coderanger: i'll be around for the next few hours for sure, at least for the next 5h.

[03:26:14]

otaku42: Well I've got soemthing that is working

[03:26:22]

But I'm not clear its any faster

[03:26:38]

But then again this is very powerful computer under no load

[03:27:13]

coderanger: we could try it on our server...

[03:27:20]

otaku42: One sec

[03:27:44]

You will need to install memcached, libmemcache, and the cmemcache python library

[03:28:54]

It should bring processing to near 0 for cached pages though

[03:29:00]

coderanger: ok, i'm about to do that

[03:29:07]

Just needs to reload the HDF from a string, and shove it out

[03:29:40]

oh right, I need to add stuff for cache invalidation

[03:29:55]

coderanger: is that a plugin for 0.10?

[03:30:03]

This? yep

[03:30:09]

Won't work on 0.9

[03:30:35]

0.10 added a request filter system that I am using to hijack cached requests

[03:30:52]

coderanger: that won't be a problem, the "new" server is up and running with trac 0.10.3 (plus some minor mods)

[03:30:59]

Okay

[03:31:46]

http://dev.coderanger.net/src/cachesystem/dist/TracCacheSystem-1.0-py2.4.egg

[03:33:15]

coderanger: what is libmemcache used for in this equation? is it a dependancy of cmemcache?

[03:33:46]

It is a C library to interface to memcached, used by cmemcache

[03:34:16]

If you want to get fancy you can even install memcached on a different server ;-)

[03:35:42]

coderanger: i guess it wouldn't make too much sense to have memcached on the current production server for madwifi.org (which is the box that suffers from the performance impacts described in my posting ;))

[03:36:03]

otaku42: It depends on if it is limited on RAM or CPI

[03:36:06]

er, CPU

[03:36:29]

memcached uses very little CPU time, mostly just a hash table in memory with an IP interface to it

[03:36:51]

coderanger: the "old" server has 512mb, which is pretty much used up already

[03:36:58]

ahh

[03:37:21]

coderanger: the new one has 1gb instead, and i guess i will try to throw in another gb soon

[03:38:05]

otaku42: Are you going to keep the old machine as well?

[03:38:13]

You could convert that into just a caching box

[03:39:14]

(if this ends up working that is)

[03:39:19]

coderanger: for a while it will be available, but not for too long. the replacement server that takes the rest of the sites currently hosted on the "old" server will be in another colo and won't have too much bandwidth available, so that won't be an option here, i fear

[03:39:28]

ahh

[03:39:30]

quite complicated situation :)

[03:39:33]

it was worth a shot :)

[03:40:01]

hehe, yeah

[03:40:30]

memcached up and running, now for libmemcache...

[03:41:18]

done. let's google for cmemcache...

[03:43:56]

http://gijsbert.org/cmemcache/index.html#download

[03:44:59]

ew... gcc needed for installation. hmm...

[03:47:22]

hmm. cmemcache throws a bunch of errors... let's see how i can fix that.

[03:47:58]

Double check that it found memcache.h

[03:48:00]

near the top

[03:48:41]

I put a default cache timeout of 10 minutes, sound reasonable?

[03:48:54]

coderanger: it fails to find limits.h, and i have a vague idea what the reason is. but that won't be easy to fix, i fear.

[03:49:12]

otaku42: What platform is this?

[03:49:20]

maybe I can just compile it here :P

[03:49:38]

coderanger: x86, debian etch.

[03:50:51]

coderanger: i wonder if it works when i compile it on sarge and the swap it over to etch...

[03:51:04]

http://dev.coderanger.net/_cmemcache.so thats build on OS X

[03:51:13]

probably unlikely to work

[03:51:27]

coderanger: ah, no, since sarge comes with python 2.3, while etch uses 2.4 (at least here)

[03:54:38]

dummdidumm... that sucks :) the thing is, the server uses linux-vserver.org. the host is still on sarge, the vhost for the webserver is already on etch. and installing cmemcache inside the etch vserver ends up with the mentioned issue.

[03:55:11]

ouch

[03:55:27]

There is a pure python library

[03:55:34]

I was hoping to not use it as its somewhat slower

[03:55:50]

coderanger: read about that in cmemcache's README :)

[03:55:56]

Though itl still probably help somewhat

[03:56:23]

coderanger: give me a few minutes, i'll try to get that other issue fixed first.

[03:56:31]

Okay

[03:56:37]

I'm going to try and add it as a fallback

[03:56:45]

the same way pypgsql and psycopg work

[04:01:05]

Okay, new version should work with either library I think

[04:09:23]

<the_lalelu>

remoep!

[04:09:44]

coderanger: will need an hour or so... i set up another vserver for compiling (it's easier to clean up the mess i made that way) :)

[04:10:03]

No worries

[04:10:20]

The sun is coming up, I should probably try getting some more sleep

[04:10:27]

coderanger: heh :)

[04:10:38]

coderanger: i will let you know if and how it worked out for me

[04:11:31]

Okay, I'm post this to trac-hacks

[04:11:37]

so you can grab the source from there

[04:12:26]

coderanger: oki

[04:17:39]

<cboos>

... and now it's about time to go to bed, coderanger ;)

[04:17:55]

bah

[04:18:38]

<cboos>

+ I might suggest that hacking on Trac is not a good way to find sleep, as this stimulates your brain activity ;)

[04:20:34]

cboos: I slept for ~5 hours earlier, just woke up and couldn't fall back asleep

[04:20:50]

anyone know of an old problem where the "last change" setting is wrong?

[04:21:12]

<cboos>

ah, ok ;) Your mail on trac-dev got me worrying, that's why I urged you to find some rest ;)

[04:21:33]

it seems to pick up any change done to the "repository_dir" .....

[04:22:37]

<cboos>

angrymike: not sure I understand... do you mean the last changeset is not the expected one?

[04:22:47]

cboos: yep

[04:23:00]

<cboos>

that should be fixed in 0.10.3

[04:23:03]

I have made a commit on a "branch" but it shows up when I browse on trunk

[04:23:06]

cboos: yea

[04:23:14]

cboos: but I am using the old workflow branch

[04:23:42]

cboos: I tried to find the changeset that fixed it by browsing the versioncontrol part of 10.3 ....

[04:23:48]

cboos: but I couldn't find the commit

[04:24:11]

<cboos>

well, that fix is spread around different area

[04:24:23]

<cboos>

but anyway that old workflow didn't have that bug, iirc

[04:24:28]

hmmmm

[04:24:33]

cboos: it seems to now. :D

[04:24:45]

<cboos>

... unless ... can you tell me the changeset you're using?

[04:25:00]

3682

[04:25:10]

<cboos>

angrymike: > I have made a commit on a "branch" but it shows up when I browse on trunk

[04:25:19]

<cboos>

can you explain the above a bit more in detail?

yea

root/trunk/folder

root/branch/folder

commit to /root/branch/folder

[04:26:21]

and that commit will show up in root/trunk/folder under the "last change" in the source browser

[04:26:37]

but if you click on "revision log" all is well

[04:26:58]

<cboos>

hm... that would be a different bug

[04:27:09]

cboos: dang, any pointers of where I should start to look ?

[04:27:40]

cboos: and I should tell you we are using the repository_dir to restrict this instance of trac to a specific part of the repository

[04:28:20]

<cboos>

Well, what about trying a merge between 0.10.3 and your current version, for the trac/versioncontrol part only?

[04:28:38]

cboos: would that be safe?

[04:28:42]

<cboos>

shouldn't present too many conflicts,if at all

[04:28:56]

cboos: I mean did any of the API change, so that other code would fail ?

[04:29:01]

<cboos>

well, no, not really

[04:29:11]

<cboos>

forgot about the trac/web related changes

[04:29:13]

<cboos>

ok

[04:29:29]

<cboos>

then a merge with 0.10.0 itself should do the trick

[04:29:33]

<cboos>

that should be safer

[04:29:45]

cboos: merge straight from workflow to 0.10.0 ?

[04:29:57]

<cboos>

only the trac/versioncontrol part

[04:30:00]

cboos: or just version control ?

[04:30:04]

<cboos>

wait ...

[04:32:23]

I think we need a tranquilizer gun for coderanger

[04:33:25]

<cboos>

angrymike: the diffs are visible here: http://trac.edgewall.org/changeset?old_path=%2Fsandbox%2Fworkflow%2Ftrac%2Fversioncontrol&old=3378&new_path=%2Ftags%2Ftrac-0.10%2Ftrac%2Fversioncontrol&new=

[04:34:04]

<cboos>

but at a quick glance, this doesn't seem to be able to address the problem you mentioned ... so maybe it's a new bug ;)

[04:34:30]

cboos: yea I browsed through the bugs and commits, but didn't find any references to it. :|

[04:34:49]

<cboos>

what I'd do in this case is first try to see if 0.10 would be enough to fix it (by running a quick test install of tags/trac-0.10)

[04:35:16]

<cboos>

and if everything is ok with that, then try the merge above (minus the s/markup/html/ changes, probably)

[04:35:30]

cboos: is it easy to install two versions of trac at the same time?

[04:35:50]

<cboos>

yes, just use tracd for testing

[04:36:14]

<cboos>

and do something like: python setup.py install --prefix=/opt/trac-0.10

[04:36:20]

<cboos>

for the install itself

[04:36:26]

ahh ok

[04:36:32]

<cboos>

then adjust your PYTHONPATH accordingly

[04:36:54]

<cboos>

and create a new empty env, as you're only interested in checking the TracBrowser part

[04:37:49]

cboos: when you say adjust PYTHONPATH what does that mean? :)

[04:38:27]

<cboos>

PYTHONPATH=/opt/trac-0.10 python scripts/tracd $ENV

ahh

great

will give it a go

<cboos>

if you're PYTHONPATH is not empty, though, do rather

[04:39:11]

<cboos>

PYTHONPATH=/opt/trac-0.10:$PYTHONPATH python scripts/tracd $ENV

[04:39:13]

<cboos>

of course

[04:39:25]

cboos: will do

[04:39:52]

<cboos>

oh, and it should rather be something like /opt/trac-0.10/lib/python-2.3/site-lib

[04:40:21]

<cboos>

(forgot the detail, look into your install to find the folder containing the trac/ package, i.e. the folder containing the *.py files)

[04:40:54]

ok

[04:41:29]

<cboos>

ok, checked, it should be: /opt/trac-0.10/lib/python2.4/site-packages/

[04:41:42]

<cboos>

(replace 2.4 by 2.3 if needed of course)

[04:41:48]

as long as it matches my other site-packages/trac

[04:41:57]

where the actual trac is installed. :)

[04:42:12]

<cboos>

yes, leave this one alone for now ;)

[04:42:21]

will do

[04:45:27]

cboos: it won't be a problem if both trac instances read the subversion repository right?

[04:45:40]

<cboos>

no pb there

[04:45:57]

<cboos>

but of course, you shouldn't share the trac env itself

[04:46:06]

right

[04:49:04]

cboos: what port does tracd open ?

[04:49:17]

<cboos>

80, but use -p <something>

[04:49:25]

thx

[04:49:36]

<cboos>

or -h even ;)

[04:53:23]

cboos: yea its there

[04:53:27]

cboos: the bug I mean

[04:53:34]

<cboos>

uh oh

[04:53:46]

<cboos>

... would you mind doing the same for 0.10.3?

[04:54:01]

do I need to recreate the environment ?

[04:54:05]

or just upgrade ?

[04:54:10]

<cboos>

no, just upgrade

[04:54:14]

ok

[04:54:16]

will try

[04:54:50]

<cboos>

thx, you'll get premium support if the bug is still there ;)

[04:56:28]

<cboos>

noah, err coderanger, still around?

[04:56:40]

cboos: Yeah

[04:56:45]

<cboos>

I was looking at your cachesystem plugin

[04:56:50]

decided to make breakfast

[04:56:54]

<cboos>

very interesting stuff ;)

[04:57:07]

<cboos>

(the memcache, but I bet the breakfeast as well ;) )

[04:57:34]

cboos: it clearly would need more work, but as a proof-of-concept it does actually work at least

[04:57:49]

<cboos>

sure... so for now you just memcache the parsed html of the wiki page?

[04:58:17]

It takes the entrite HDF tree under "wiki" and caches that

[04:58:18]

cboos: doh!

[04:58:22]

cboos: still there....

[04:58:42]

cboos: the welcome page still says 0.10

[04:58:50]

cboos: is that correct ?

[04:58:57]

<cboos>

coderanger: I forgot a bit about the HDF api, why req.hdf.getObj() and req.hdf.copy() instead of the req.hdf['wiki'] and req.hdf['wiki'] = ... ? is there anything special going on there?

[04:59:21]

cboos: Because the wrapper doesn't work like that

[04:59:24]

<cboos>

angrymike: no, you're still using 0.10 obviously

[04:59:30]

hmmm

[04:59:32]

ok

[04:59:39]

hdf['wiki'] is a single key, not a dict or something

[05:00:08]

<cboos>

yes, so in this case, what is it? not a string?

[05:00:14]

cboos: Its just ""

[05:00:21]

because nothing is stored there

[05:00:44]

I could walk the HDF from that point, but it was just as easy to use the built-in functions to do it

[05:01:59]

<cboos>

oh, I understand now... you put a HDFWrapper inside the HDFWrapper ...

[05:02:08]

kind of

[05:02:11]

<cboos>

evil :)

[05:02:15]

it just copies out from one into the other

[05:02:38]

The caches holds the serialized, string version of the HDF

[05:03:07]

So the only processing on cache hits is reload the HDF from the serialized copy, and render the template

[05:03:27]

<cboos>

ok...

[05:03:45]

<cboos>

but what about the simpler option to cache the whole HTML for that page?

[05:03:51]

<cboos>

did you consider that?

[05:04:45]

cboos: I did, but things like the nav bar would hard to do

[05:04:51]

since they vary from one user to another

[05:05:31]

<cboos>

yeah, we would probably need "roles" for being able to handle that efficiently

[05:05:48]

And whole-page caching is probably better handled by something like mod_cache or squid

[05:06:22]

<cboos>

even for the invalidation part?

[05:06:54]

<cboos>

i.e. how would you signal mod_cache or squid that there's a new version of the page available?

[05:06:56]

Thats the hard part ;-)

[05:07:30]

You can request that the proxy reHEAD the page every <time period>

[05:07:38]

and tell it in there if it needs to reload

[05:08:20]

<cboos>

ok, but that's somewhat heavier configuration wise

[05:08:33]

Yeah

[05:08:37]

<cboos>

what I like in this memcache stuff is that this can be done from within trac

[05:09:05]

<cboos>

so we have a fine control over what invalidates the cache or not (and what to cache)

[05:09:12]

Exactly

[05:09:40]

But its just a general purpose cache, so we need to write all the cache control code

[05:10:16]

<cboos>

ok, I'll try to find a free corner in my mind to store that for later ... think about how this would translate to 0.11 and beyond ;)

[05:10:39]

Yeah, in 0.11 the storage gets easier

[05:10:54]

since it can all just be pickled objects floating around

[05:11:42]

<cboos>

mh, not sure about the pickled object... for several reasons

[05:11:56]

cboos: The other thing to think about is how to do the invalidation policies

[05:12:09]

For example, a page with a TOC instance that depends on other pages

[05:12:32]

right now you would have to wait for it to time out of the cache

[05:12:43]

<cboos>

coderanger: of course, that's the interesting part ;)

[05:13:23]

<cboos>

with the fine grained permission, you'd end up with very different pages depending on each user ...

[05:14:06]

<cboos>

e.g. for a ticket link, some will be able to see the ticket summary, others who don't have the access right to that ticket wont

[05:15:37]

<cboos>

about the pickling: 1/ you can't pickle objects from extensions I think, and we have at least the swig/svn ones for the changeset related pages (ok, that's not normally an issue for the wiki pages)

[05:16:02]

cboos: We would have to write custom picklers for them, but thats pretty easy

[05:16:16]

<cboos>

and 2/, what's /mainly/ time consuming is the Genshi processing, so this is what should be cached in the first place

[05:17:37]

<cboos>

I see very little benefit if you just cache the objects... first you have to unpickle them, then you have to generate the template (and filter them) once again -> the real killer

[05:18:29]

<cboos>

so the real benefit - and something that would counter-balance the slow-down introduced by Genshi - would be to memcache the generated HTML

[05:19:38]

<cboos>

In 0.10, you can basically ignore the template rendering time, because Clearsilver is fast, but in 0.11, the pattern is reversed

[05:19:49]

Hmm, yeah

[05:20:11]

There would need to be some hooks added in the render process so that the output could be captured

[05:20:13]

cboos: am I checking out the right tag? svn co http://svn.edgewall.org/repos/trac/tags/trac-0.10.3

[05:20:23]

it always seems to end at 4709

[05:20:32]

<cboos>

yes, that's OK

[05:20:44]

but the trac browser says the revision is 4446

[05:21:06]

<cboos>

(the changeset you get is the "global" one, not the latest of the branch/tag)

[05:21:19]

cboos: ahh

[05:21:47]

<cboos>

in Trac we show 4446, because you that's the interesting information ;) (was the last change on the branch)

[05:21:53]

When I use self.log.debug inside my plugin code, nothing seems to appear in the log. I have set log_level to DEBUG, and debug messages are appearing for the loader though. Do I need to do something extra to get logging to work?

[05:21:55]

cboos: right

[05:22:35]

<cboos>

mallero: what does the loader say about your plugin? is it loaded at all?

[05:23:55]

cboos: ah yes, it's loading from the wrong directory

[05:25:34]

I'm really confused about the plugin loading system. It is trying to load a plugin from a different project's plugin directory (from a different environment).

[05:25:40]

Why would it do that?

[05:25:59]

mallero: Are they loading in the same memory space?

[05:26:05]

And do they have the same module nam,e

[05:26:21]

cboos:I think it is taking the default wiki stuff from /usr/share

[05:26:39]

cboos: instead of /opt/trac-0.10.3 when I am using trac-admin to create the env

[05:27:18]

<cboos>