2005/01/18
IRC [01:09] *** holycow parted the chat.
IRC [03:10] *** booyaa joined the chat.
IRC [03:37] *** frankie joined the chat.
IRC [03:39] *** booyaa parted the chat.
IRC [04:29] *** tekbasse_ joined the chat.
IRC [05:25] *** frankie parted the chat.
IRC [06:33] *** chubby joined the chat.
IRC [07:43] *** frankie joined the chat.
IRC [09:30] *** bartt joined the chat.
IRC [10:48] *** iharding joined the chat.
IRC [12:56] *** iharding parted the chat.
IRC [13:19] *** rubick joined the chat.
IRC [13:33] *** frankie parted the chat.
IRC [14:02] *** tekbasse_ parted the chat.
IRC [15:27] <Dossy> think I'm going to roll AOLserver 4.0.10 today.
IRC [15:28] <Dossy> anyone have requests for must-fix bugs for 4.0.10? Andrew? :)
IRC [16:20] *** frankie joined the chat.
IRC [17:36] *** frankie parted the chat.
IRC [18:06] <rubick> Dossy: does that include the fix for the recent memory leak?
IRC [18:53] <Dossy> heh.
IRC [18:53] <Dossy> no.
IRC [18:53] <Dossy> because the memory leak is caused by a bug that may likely not get fixed (the connsperthread != 0 one)
IRC [18:53] <Dossy> if/when we can reproduce Janine's actual leak, I'll work on fixing it
IRC [18:56] <rubick> I'm having an issue that's not related to the connsperthread issue.
IRC [18:56] <rubick> I'm having difficulty tracking it down though.
IRC [18:56] <rubick> About once a day, my site becomes completely unresponsive.
IRC [18:58] <rubick> Or, one of my many sites. :)
IRC [18:59] <Dossy> not the same site?
IRC [18:59] <rubick> the same as what?
IRC [18:59] <Dossy> it would help if you took actual measurements and had hard numbers to do empirical analysis
IRC [19:00] <Dossy> is it the same site that becomes unersponsive? or only one at random out of many on one host?
IRC [19:00] <rubick> Perhaps we should put up a page on the Wiki describing what to do in order to help track down issues like this.
IRC [19:00] <rubick> The same site, every day.
IRC [19:00] <Dossy> is it at the exact same time every day? are all other sites getting traffic at that time, and remaining responsive?
IRC [19:00] <rubick> Infrequently.
IRC [19:00] <rubick> Or rather, sporadically, not predictibly
IRC [19:00] <Dossy> Every day? Or infrequently?
IRC [19:01] <Dossy> yeah, I could start putting together an AOLserver troubleshooting guide, but it's not really so specifically AOLserver as opposed to general sysadmin troubleshooting skill.
IRC [19:01] <rubick> All the other sites get traffic, but not as much as this one.
IRC [19:01] <Dossy> does the site use ns_db ?
IRC [19:01] <rubick> Yes. All the sites are OpenACS sites, with similar or almost identical configurations.
IRC [19:01] <Dossy> Ah.
IRC [19:01] <rubick> Postgres, nsopenssl
IRC [19:02] <Dossy> All pointing at the same DB? Postgres? Oracle?
IRC [19:02] <rubick> This one is mostly static webpages, though.
IRC [19:02] <rubick> All at Postgres.
IRC [19:02] <rubick> Very little database access, though.
IRC [19:02] <Dossy> minthreads? maxthreads?
IRC [19:02] <rubick> minthreads 5
IRC [19:02] <rubick> maxthreads 5
IRC [19:03] <Dossy> oh, ouch.
IRC [19:03] <rubick> bad?
IRC [19:03] <Dossy> how many hits per second does the site get?
IRC [19:03] <rubick> not too many. It's low to medium volume.
IRC [19:03] <Dossy> during the time it hangs, what's the min/max/avg for that hour in req/sec?
IRC [19:03] <rubick> quite low, always. I don't know offhand, but I could look at the logs.
IRC [19:04] <Dossy> i should write a access log processing script for this kinda stuff.
IRC [19:04] <rubick> analog?
IRC [19:04] <Dossy> yeah. for the 30 min before and after the hang, chunk the log into 5 minute intervals and do a hits count
IRC [19:04] <Dossy> you'll have 12 buckets (6 before and 6 after) ... that'll potentially be an interesting graph
IRC [19:05] <Dossy> also, you should configure nslog to log the request processing time
IRC [19:05] <rubick> We're only getting 100-150K hits a month, so not very many hits.
IRC [19:05] <Dossy> and look for anything that takes >0.5 sec...
IRC [19:06] <rubick> I'm not really sure how to do that, but I'd be happy to follow any instructions you point me to, or anything else you recommend.
IRC [19:06] <rubick> is the minthreads maxthreads unrealistic?
IRC [19:08] <rubick> wow, 50 is recommended in http://www.aolserver.com/docs/admin/config-detailed.html
IRC [19:08] <rubick> I wonder if that would hold true for OpenACS sites, which load the threads so heavily with procs, etc..
IRC [19:08] <Dossy> well, you need to figure out what your peak concurrent requests is. maxthreads must = that number, otherwise you'll see "hangs"
IRC [19:09] <Dossy> http://cvs.sourceforge.net/viewcvs.py/*checkout*/aolserver/aolserver/nscgi/nscgi.html?rev=HEAD
IRC [19:09] <Dossy> that's nscgi doc
IRC [19:09] <Dossy> http://panoptic.com/wiki/aolserver/nscgi
IRC [19:09] <Dossy> the wiki page links to the SF page
IRC [19:09] <Dossy> wtf am I doing
IRC [19:09] <rubick> I suspect that this is similar to what was happening with the nsopenssl issue. I don't really have any evidence to support that except that the behavior is exactly the same. I'm on Aolserver 4.09 + nsopenssl head now.
IRC [19:10] <Dossy> http://panoptic.com/wiki/aolserver/nslog
IRC [19:10] <Dossy> look at "logreqtime" param.
IRC [19:11] <rubick> oh, that's cool.
IRC [19:11] <rubick> and easy.
IRC [19:11] <Dossy> nod
IRC [19:11] <Dossy> and VERY useful.
IRC [19:11] <Dossy> it's cool to graph the max reqtime per 5 minute period for a 24 hr period and watch hotspots jump out
IRC [19:12] <rubick> no real performance penalty to doing this?
IRC [19:12] <Dossy> like "oh, wow, requests take 0.3sec longer for this 15 min period ... oh, backups are running then"
IRC [19:12] <rubick> that's really interesting and cool.
IRC [19:12] <Dossy> nope, we track the req time anyway and it's only adding another 5-10 bytes to each log entry :)
IRC [19:13] <Dossy> what nsopenssl issue are you talking about/
IRC [19:13] <Dossy> i think there's only one nsopenssl bug open now dealing with a memory leak w/ ns_openssl_sock* commands.
IRC [19:15] <rubick> I'm referring to the bug where nsopenssl eventually would eat up all the connections and the site would stop taking connections.
IRC [19:15] <rubick> But maybe that wasn't the core problem.
IRC [19:15] *** chubby parted the chat.
IRC [19:15] * rubick watches the response times on the log file.
IRC [19:15] <rubick> sweet!
IRC [19:17] <Dossy> i thought that bug w/ nsopenssl got fixed ...
IRC [19:17] <rubick> I think it was.
IRC [19:18] <rubick> I thought that would fix the problem I was having, but either I was having both problems, or that wasn't the issue I was having, or the problem wasn't completely fixed.
IRC [19:18] <rubick> I love the Aolserver wiki. Very useful!
IRC [19:24] <rubick> analog has a parameter for processing time.
IRC [19:32] <Dossy> heh! cool
IRC [19:32] <Dossy> :)
IRC [19:33] <rubick> The problem is that because your log file format changes, then your old log files can't be processed :)
IRC [19:34] <Dossy> oh, yea :)
IRC [19:34] <Dossy> so, gotta start out with the config set that way :)
IRC [19:35] <rubick> I'm going to put that in the default OpenACS startup
IRC [19:35] <rubick> Do you think I should change my min or maxthreads?
IRC [19:35] <rubick> Bump them up?
IRC [20:13] *** rubick parted the chat.
IRC [20:26] *** booyaa joined the chat.
IRC [20:41] *** booyaa parted the chat.
IRC [21:32] <Dossy> OK, AOLserver 4.0.10 is released. :-)
IRC [21:32] <Dossy> i'd set minthreads 5, maxthreads 20