2004/08/10
IRC [00:30] <tekbasse> I don't see maxconnections used in the aolserver4 config.tcl file. Is it not used anymore?
IRC [00:33] <tekbasse> hmmm.. helps if I spell it correctly. Just did grep -R maxconnections /usr/local/src/aolserver and found it in sample-config.tcl
IRC [10:53] *** rubick joined the chat.
IRC [11:21] *** aolserver-scribe joined the chat.
IRC [11:21] * aolserver-scribe is logging
IRC [11:59] *** frodoroot joined the chat.
IRC [12:00] <frodoroot> morning all
IRC [12:13] <bartt> morning frodoroot
IRC [12:13] <bartt> For what it is worth, I'm using openssl 0.9.6b too
IRC [12:13] <frodoroot> my nsopenssl problems are solved when I bind to 8443 instead of 443; wtf?
IRC [12:13] <bartt> I'm using port 443 w/o problems.
IRC [12:13] <frodoroot> you would
IRC [12:14] <bartt> brb
IRC [12:16] <martinh> are you binding to port 80?
IRC [12:16] <frodoroot> nope just 8443
IRC [12:17] <frodoroot> before I was binding to both 80 and 443 so maybe the combination causes the error
IRC [12:17] <martinh> so, ssl only?
IRC [12:17] <frodoroot> that's right
IRC [12:17] <martinh> are you starting as root?
IRC [12:17] <frodoroot> yes
IRC [12:17] <martinh> and you're doing prebinding with a file?
IRC [12:17] <frodoroot> yeah I am -b 0.0.0.0:8443
IRC [12:18] <frodoroot> brb
IRC [12:22] <martinh> what's the error when you do :443? and, what's the exact startup command line?
IRC [12:23] <martinh> sorry, but i obviously missed your first discussion of this. :-)
IRC [12:23] <frodoroot> are you on the list? it's the one with the subject /.*nsopenssl.*/
IRC [12:30] <frodoroot> I can describe it here if you desire
IRC [12:30] <martinh> no i'm not. :->
IRC [12:30] <martinh> is it archived? i'll go look.
IRC [12:32] <frodoroot> basically I get a ton of errors related to nsopenssl in the server.log and the server crashes almost every 30 seconds
IRC [12:32] <frodoroot> when I bind to 443, and 80 on a production site
IRC [12:32] <frodoroot> when I perform tests on 8443 things are ok, but then traffic is low because I'm the only one testing it
IRC [12:33] <frodoroot> that's the summary
IRC [12:36] <martinh> aah. that's a strange one. . .
IRC [12:37] <martinh> which version of openssl?
IRC [12:39] <frodoroot> :) 0.9.6b
IRC [12:52] <martinh> and nsopenssl?
IRC [12:52] <martinh> b is a bit old, actually. . .
IRC [12:55] <bartt> Yet, I'm running nsopenssl successfully on RH7.3 w/ openssl 0.9.6b on port 443.
IRC [12:57] <martinh> which nsopenssl?
IRC [12:58] <frodoroot> bartt: can I see your nsd.tcl (config) file?
IRC [12:58] <bartt> Sure
IRC [12:59] <frodoroot> tinyurl 4rjzs
IRC [12:59] <frodoroot> paste it in there
IRC [12:59] <bartt> ok
IRC [12:59] <frodoroot> thanks
IRC [13:02] <bartt> frodoroot: It's there
IRC [13:03] <frodoroot> roger that
IRC [13:03] <bartt> Maybe these lines help:
IRC [13:03] <bartt> ns_param MaxKeepAlive 0 ;# Not used in AOLserve. BART 2004-04-23 ns_param KeepaliveTimeout 0 ;# Don't use keep alive b/c IE 6 + SSL + keepalive error.
IRC [13:03] <bartt> Oops
IRC [13:04] <bartt> ns_param MaxKeepAlive 0 ;# Not used in AOLserve. BART 2004-04-23
IRC [13:04] <bartt> ns_param KeepaliveTimeout 0 ;# Don't use keep alive b/c IE 6 + SSL + keepalive error.
IRC [13:04] <bartt> That's better.
IRC [13:06] <frodoroot> ok I'll try that
IRC [13:06] <frodoroot> I'm checking the ssl section now
IRC [13:08] <frodoroot> mine doesn't have anything on SessionCache
IRC [13:11] <martinh> hmmm. the ssl_error_ssl is an indication of a protocol issue. like, you're not enabling ssl2 or something of that nature. . .
IRC [13:14] <frodoroot> holy cow!
IRC [13:14] <frodoroot> SSLv2 is not in the list of Protocols
IRC [13:15] <martinh> check your protocols line and ciphersuite line.
IRC [13:16] <frodoroot> SSLv2 is not in the list of Protocols, but it was in the CipherSuite list
IRC [13:16] <frodoroot> I'm going to try again
IRC [13:17] <frodoroot> now
IRC [13:17] <martinh> for the users context. especially.
IRC [13:35] <Dossy> my spam filter seems to only let through these 0-byte spams now. interesting.
IRC [13:35] <Dossy> nate, have you figured out more about the nsopenssl issue?
IRC [13:35] <Dossy> your message about prebound sockets is interesting -- anything come from that?
IRC [13:37] <martinh> i think he's testing a different set of protocols now. cuz the ssl_error_ssl error is a protocol issue.
IRC [13:38] <bartt> I removed SSLv2 from the Protocols on Scott's advise. Can't remember why. Need to dig through notes.
IRC [13:41] <bartt> As far as I can remember it had to do with IE and SSL errors.
IRC [13:42] <Dossy> hmmm
IRC [13:46] <bartt> But that was in the past.
IRC [13:46] <bartt> My current config uses SSLv2
IRC [13:48] <frodoroot> hey Dossy
IRC [13:48] <frodoroot> bartt showed me his config file and I realized that mine didn't have SSLv2 listed as a protocol in the users context
IRC [13:48] <frodoroot> which is the only context I have since I'm not interested in making outboudn connections
IRC [13:48] <frodoroot> is that bad?
IRC [13:49] <frodoroot> so I added SSLv2 and I still get errors, but a lots less frequently
IRC [13:49] <frodoroot> the server also crashes about 1.5 minutes
IRC [13:50] <frodoroot> every 1.5 minutes
IRC [13:50] <frodoroot> is there a way I can find out what browser the user on conn::6 is using?
IRC [13:51] <frodoroot> because he/she caused the error
IRC [13:51] <bartt> Yes, you could look at the HTTP headers
IRC [13:51] <frodoroot> in tcl?
IRC [13:51] <frodoroot> ns_info or something
IRC [13:51] <bartt> Yes, you could register a filter.
IRC [13:51] <martinh> find the ip and check the logs for the non ssl stuff. if it's not recording user-agent make it record it.
IRC [13:52] <bartt> ns_conn headers
IRC [13:52] <bartt> will give you a ns_set of all headers
IRC [13:52] <bartt> http://aolserver.com/docs/devel/tcl/api/conn.html#ns_conn
IRC [13:54] <bartt> ns_log notice "[ns_conn driver] - [ns_set iget [ns_conn headers] User-Agent]"
IRC [13:54] <bartt> This would give you the drive and the User-Agent
IRC [13:54] <bartt> s/drive/driver/
IRC [13:56] <bartt> See http://aolserver.com/docs/devel/tcl/api/general.html#ns_register_filter how to register this as a filter.
IRC [14:01] <frodoroot> man... this thing is crashing all the time now
IRC [14:01] <frodoroot> mmm
IRC [14:26] <frodoroot> I'm going to try 4.0.5 to see if it has the crashing problem
IRC [15:03] <tekbasse> frodoroot, I made those ssl errors go away by changing the order of the ns_section defitions to match http://aolserver.com/docs/admin/config-detailed.html
IRC [15:04] <tekbasse> however, aolserver still crashing.. I'm trying some new things today, after reading the docs =)
IRC [15:06] <frodoroot> It's still crasing for me too
IRC [15:06] <tekbasse> ..crashing without errors to be specific.
IRC [15:06] <frodoroot> yes, that is correct
IRC [15:10] <Dossy> is it exiting, or dropping core?
IRC [15:10] <tekbasse> thinking maxthreads too high? so trying back at 5, and adding ConnsPerThread 100 which "helps control mem leaks in tcl code" according to docs.
IRC [15:11] <tekbasse> i don't see any core files, so guess that means exiting?
IRC [15:11] <Dossy> not necessarily
IRC [15:11] <Dossy> is this production or dev?
IRC [15:11] <tekbasse> no quiting msgs in the error.log
IRC [15:11] <frodoroot> production for me
IRC [15:11] <Dossy> ugh
IRC [15:11] <Dossy> was hoping you could run nsd under gdb and see how it "exits"
IRC [15:11] <tekbasse> production, just switching an aolserver3.x to 4.x
IRC [15:12] <frodoroot> same here
IRC [15:12] <tekbasse> for openacs4.6.3
IRC [15:12] <frodoroot> 3.x => 4.x
IRC [15:12] <Dossy> hmm
IRC [15:12] <Dossy> would it be possible to run under gdb and see how it exits?
IRC [15:12] <tekbasse> gdb?
IRC [15:12] <frodoroot> so far I can't produce the error out of production
IRC [15:13] <Dossy> hmm
IRC [15:13] <frodoroot> I'll go for it
IRC [15:13] <Dossy> ok
IRC [15:13] <frodoroot> what exactly should I do ?
IRC [15:13] <frodoroot> gdb nsd ....
IRC [15:14] <Dossy> gdb --args bin/nsd -ft config.tcl -u whoever
IRC [15:14] <Dossy> then when you get to the (gdb) prompt, type "run"
IRC [15:14] <Dossy> then watch it crash back out to the (gdb) prompt and let me know what it says
IRC [15:14] <Dossy> or, can i get a ssh login into your production box? :P
IRC [15:15] <frodoroot> yeah the password is Dossy
IRC [15:15] <frodoroot> :)
IRC [15:15] <Dossy> heh
IRC [15:15] <Dossy> user root? :)
IRC [15:15] <frodoroot> I'm not telling
IRC [15:15] <frodoroot> I'm the non-controlling type of nerd
IRC [15:15] <frodoroot> but the controlling ones would totally freak out
IRC [15:16] <frodoroot> so I'll just do it
IRC [15:16] <tekbasse> ah. man gdb. I'll try that tonight (when server not busy).. boss doesn't like it when I lose sales ;)
IRC [15:17] <Dossy> tekbasse: haha, ouch
IRC [15:17] <frodoroot> it will go farily quickly
IRC [15:17] <frodoroot> are you getting 100 hits a minute, tekkbasse?
IRC [15:17] * tekbasse checks
IRC [15:19] <frodoroot> Dossy: it's not a debug compile. Do I need to recompile?
IRC [15:22] <tekbasse> looks like it maxes at about 56 per min
IRC [15:23] <Dossy> nate- no
IRC [15:23] <Dossy> just to get the error cause, no
IRC [15:23] <Dossy> gdb will tell you if it died because of a signal or if it exited gracefully, etc.
IRC [15:25] <frodoroot> ok I'm scripting this for minimum downtime
IRC [15:26] <Dossy> how long will it take for the server to crash?
IRC [15:26] <Dossy> it sounded like it was pretty quickly repeatable
IRC [15:27] <frodoroot> yes it shouldn't take too long at most a minute I would say
IRC [15:29] <frodoroot> I'll be back with the output
IRC [15:29] *** frodoroot parted the chat.
IRC [15:32] *** NatetheGr8 joined the chat.
IRC [15:32] <NatetheGr8> hrm it's not loggin
IRC [15:32] <NatetheGr8> I can't tell if it has crashed or not
IRC [15:32] <NatetheGr8> 3 nsds running
IRC [15:33] <NatetheGr8> I can't hit the site
IRC [15:33] <NatetheGr8> hmmm
IRC [15:33] <Dossy> you can't script it --
IRC [15:33] <Dossy> you have to run gdb from a prompt
IRC [15:34] <NatetheGr8> http://tinyurl.com/4rjzs
IRC [15:34] <NatetheGr8> that's the output
IRC [15:35] <Dossy> sig32?
IRC [15:35] <Dossy> wtf is that
IRC [15:35] <Dossy> http://sources.redhat.com/ml/gdb/2004-03/msg00179.html
IRC [15:35] <Dossy> ah
IRC [15:35] <Dossy> you on some old RH?
IRC [15:36] <NatetheGr8> 7.3 yes
IRC [15:36] <NatetheGr8> same as bartt
IRC [15:36] <NatetheGr8> who is running it successfully I might add
IRC [15:37] <Dossy> what version of gdb?
IRC [15:38] <Dossy> http://www.mail-archive.com/bug-gdb@gnu.org/msg01102.html
IRC [15:39] <NatetheGr8> 5.2-2
IRC [15:40] <Dossy> hmm
IRC [15:40] <Dossy> guess gdb is out then
IRC [15:40] <Dossy> ugh
IRC [15:40] <Dossy> can you star tthe nsd then run strace -p? maybe it'll be able to listen for the error
IRC [15:44] <NatetheGr8> ok after lunch
IRC [16:21] *** cnk joined the chat.
IRC [16:30] <tekbasse> where does the ns_section ns/threads belong? I see a different place everywhere I look.
IRC [16:32] <AndyPiskorski> Why would the order of the ns_sections sections in the config file matter?
IRC [16:35] <tekbasse> it matters according to http://aolserver.com/docs/admin/config-detailed.html as I read them, and confirmed by bartt, but don't know why =)
IRC [16:36] <tekbasse> oops wrong link, have to backup one. 1min
IRC [16:37] <tekbasse> http://aolserver.com/docs/admin/config.html see "configuration file hierarchy"
IRC [16:39] <tekbasse> I guess the anotated config reference takes precedence: http://aolserver.com/docs/admin/config-reference.tcl.txt
IRC [17:16] <tekbasse> bbl
IRC [17:57] <NatetheGr8> tekbasse: and it's not crashing now that the config is in order?
IRC [17:58] <Dossy> ugh
IRC [17:59] <Dossy> ns_sections matter only with regard to where the modules are defined
IRC [17:59] <Dossy> remember that the config.tcl is just another tcl script and lines get executed in the order they appear.
IRC [18:08] <frodoroot> ok so I should define the modules in the order specified here http://aolserver.com/docs/admin/config-detailed.html
IRC [18:08] <frodoroot> scottg has been too busy at nasa with his 200gb disks
IRC [18:08] <frodoroot> harumph
IRC [18:09] <jhavard> if they didn't buy him all those low end machines, they wouldn't have a budget problem.
IRC [18:23] <Dossy> heh
IRC [18:25] <Dossy> whine.
IRC [18:25] <Dossy> i've identified what I think is the bug in HEAD, but can't come up with the right fix.
IRC [19:13] *** jcollins_ joined the chat.
IRC [19:29] *** jcollins parted the chat.
IRC [20:18] <jhavard> I apparently was the recipient of a spider bite sometime on sunday.
IRC [20:19] <jhavard> fortunately, it wasn't anything evil, so it's getting better and not moving towards death.
IRC [20:33] <Dossy> yay!
IRC [20:33] <Dossy> CVS HEAD might be stable enough to test in dev. now
IRC [20:35] <bartt> Yay!
IRC [20:39] <jhavard> have you comitted the changes yet?
IRC [20:39] <Dossy> yes
IRC [20:39] <Dossy> but you wont' see them in anonymous CVS for at least a few hours
IRC [20:39] <Dossy> want me to send you a patch?
IRC [20:39] <jhavard> nah, I'll just wait.
IRC [20:40] <Dossy> ok
IRC [20:40] <Dossy> i'm heading home now :)
IRC [20:40] <Dossy> see yall later.
IRC [20:40] <jhavard> awesome.
IRC [20:40] <jhavard> later
IRC [20:42] <bartt> later Dossy
IRC [20:55] <tekbasse> frodoroot, fixing the order fixed the nsopenssl errors (like those you reported). But oddly, doesn't stop the crashing.
IRC [20:57] <tekbasse> frodoroot, fixing the order fixed the nsopenssl errors (like those you reported). But doesn't stop the crashing.
IRC [20:57] <tekbasse> oops, sorry for the repeat... my screen wasn't scrolling!
IRC [21:07] *** cnk parted the chat.
IRC [22:15] *** bartt parted the chat.