Continuing further down this rabbit hole, it appears that a "webskin" is NOT the same as an "object". Although you can limit the amount of "objects" being cached, there's presently no means to limit the amount of "webskins".
What does that mean?
Basically, object broker is aware of my HTML objects, like my search page. It is NOT keeping track of the various views that object might have, meaning my search page has dynamic variables that changes what's displayed. Since that search page changes based on a set of variables (keyphrase, page number in the result set, etc), it's possible to have more than one webskin cached. In my case, things started going to hell after about 3000 cached pages or 400 megs of memory cache.
As much as I'd like to cache search page results, this particular client isn't going to get that goodness without forking over money for more memory at this point.
It doesn't look like this is high on the priority list for change for Daemon, but I think it'd be beneficial to limit the amount of cached webskins (or size) as well. If I can scrape together enough free time, I'll see if I can pull that off.
Also you could consider using varnish web proxy to increase our performance and have added several webskin decorators to farcry to enable us to give the proxy directives (e.g proxy cache timeout etc)
Part of the issue was without caching, I was seeing load times in the 5 second range. After the search cache, it was under a second. I'll have to just do some more DB indexing to see if it can't be improved further.