2007/08/22
IRC [10:45] <daguz> so... I got another hung nsd process. Before I restart it... how to troubleshoot it?
IRC [10:45] <Dossy> gdb backtrace?
IRC [10:45] <daguz> Ok. I'll look into that.
IRC [10:46] * Dossy nods.
IRC [10:46] <Dossy> gdb, then "attach <pid>" ... then "thread apply all bt"
IRC [10:47] <Spark> you can also attach strace for fun and profit but gdb is probably more helpful
IRC [10:49] <daguz> It looks like I have to be smart to figure out what is going on.
IRC [10:53] <Spark> nah, just persiistant :)
IRC [10:54] <daguz> I'm trying to remember where to post code so I can beg someone to look at it and help me. ??
IRC [11:11] <daguz> http://pastebin.com/m3f1bc377
IRC [11:11] <daguz> anyone care to peruse this and see if something is obvious?
IRC [12:01] <Dossy> looking now...
IRC [12:03] <Dossy> Looks like you ran out of DB pool handles.
IRC [12:03] <Dossy> And things were hung waiting for a free handle.
IRC [12:04] <Dossy> Most threads were blocking in pthread_cond_wait following a Ns_DbPoolTimedGetMultipleHandles ...
IRC [13:13] <daguz> well, that is more obvious than I thought. Thanks Dossy.
IRC [13:21] <Dossy> Your DB pools, how many connections are they configured with? This is OpenACS, I'm guessing, right?
IRC [13:22] <daguz> sure is (openacs)... lemme check.
IRC [13:27] <daguz> you mean ns_param connections? they say 10 with 4 pools to one database... Then I have another set of two pools each with 2 connections.
IRC [13:28] <daguz> (the other set of two pools with two connections go to another database)
IRC [13:30] <daguz> then postgres has max_connections at 200
IRC [13:31] <daguz> I have two servers connecting to this database server (as spec'd above)
IRC [13:47] <Dossy> 10 connections? how many threads do you have?
IRC [13:51] <daguz> max threads 40
IRC [13:54] <Dossy> So, having less than 40 connections per pool could result in a hang like you've observed.
IRC [13:56] <daguz> Since I did not configure this, I think that we made an improper assumption: ie: 4pools of 10 hence 40 max threads. So making sure I understand what you say: I need 40 connections per pool?
IRC [13:57] <daguz> Or should I down the max threads?
IRC [13:57] <Dossy> Depends on how those pools are used.
IRC [13:58] <Dossy> I don't know OpenACS well enough to say.
IRC [13:58] <daguz> Ok. I'll search on that..
IRC [13:58] <daguz> BTW: Dossy - you rock.
IRC [13:59] <Dossy> Heh, thanks. :-)
IRC [13:59] <Dossy> Glad I could lend a hand, at least.
IRC [13:59] <daguz> I just need someone to PULL me in the right direction.
IRC [14:14] <Dossy> Right ... if your server is configured for 40 threads = 40 concurrent connections, then if each request uses 1 handle from a particular DB pool, then at peak you will need 40 handles/connections in that pool.
IRC [14:15] <Dossy> If your database/etc. can't handle 40 connections, then you should lower max threads to match, or expect times at peak when requests are going to block, waiting for a free DB handle.
IRC [14:54] <daguz> My boss is still convinced that the threads spread the connections to all four pools. I'm not so sure.
IRC [14:56] <daguz> He says that is the way oacs works. How could oacs be different from what you're saying?
IRC [14:56] <daguz> It can't, I'm sure.
IRC [14:57] <Dossy> Depends on how the pools are used.
IRC [14:58] <Dossy> But, still--suppose a particular page request, say "/abc", uses pool XYZ. Pool XYZ is configured for 10 connections. The server is configured for 40 threads. Imagine 40 simultaneous requests for "/abc" ... the first 10 will get a handle and execute ... while the remaining 30 block and wait.
IRC [15:51] *** jhavard parted the chat.
IRC [16:50] *** ryan-g2 joined the chat.
IRC [16:50] <ryan-g2> My 4.0.10 server crashes after evaluating some Tcl code. Could this be caused by bad code?
IRC [17:14] *** ryan-g2 parted the chat.
IRC [17:21] *** CIA-16 parted the chat.
IRC [17:34] *** CIA-16 joined the chat.
IRC [17:43] *** partymola joined the chat.
IRC [17:45] <Dossy> ryan-g2: Can you show us the code that's causing the problem?
IRC [17:47] <partymola> hi :)
IRC [18:41] *** CIA-16 parted the chat.
IRC [18:49] *** holycow parted the chat.
IRC [18:49] *** CIA-16 joined the chat.
IRC [19:58] *** partymola parted the chat.
IRC [21:09] *** CIA-16 parted the chat.
IRC [21:19] *** CIA-16 joined the chat.
IRC [23:09] *** holycow joined the chat.
IRC [23:22] *** jhavard joined the chat.