2005/01/27
IRC [00:09] *** tekbasse joined the chat.
IRC [01:26] *** direwolf joined the chat.
IRC [01:26] *** direwolf parted the chat.
IRC [03:34] *** frankie joined the chat.
IRC [04:45] *** holycow parted the chat.
IRC [05:36] *** booyaa joined the chat.
IRC [05:56] *** booyaa parted the chat.
IRC [07:06] *** martinh parted the chat.
IRC [07:06] *** martinh joined the chat.
IRC [09:52] <Dossy> /names *
IRC [11:02] *** frankie parted the chat.
IRC [11:25] *** bartt joined the chat.
IRC [12:30] *** rubick joined the chat.
IRC [12:31] *** rubick parted the chat.
IRC [12:31] *** rubick joined the chat.
IRC [13:46] *** iharding joined the chat.
IRC [13:49] *** bartt parted the chat.
IRC [14:42] <rubick> Makefile:38: ../include/Makefile.module: No such file or directory
IRC [14:42] <rubick> make: *** No rule to make target `../include/Makefile.module'. Stop.
IRC [14:42] <rubick> lost the first line
IRC [14:42] <iharding> AOLSERVER=/usr/local/aolserver
IRC [14:42] <rubick> yeah I did that.
IRC [14:43] <rubick> :/usr/local/src/aolserver40r10/nsencrypt root# make AOLSERVER=/usr/local/aolserver40r10/
IRC [14:43] <rubick> Makefile:38: ../include/Makefile.module: No such file or directory
IRC [14:43] <rubick> make: *** No rule to make target `../include/Makefile.module'. Stop.
IRC [14:43] <Dossy> ./configure :)
IRC [14:43] <rubick> There is no ./configure script in nsencrypt
IRC [14:43] <Dossy> looks like the aolserver you're pointing to isn't the source and isn't ./configure'ed.
IRC [14:43] <rubick> ah I did point to the compiled version.
IRC [14:44] <rubick> I did do ./configure however.
IRC [14:44] <Dossy> at one point, you could do it because the installed version included the include/ directory.
IRC [14:44] <Dossy> on win32, I think we still do this so folks can build modules using the binary release.
IRC [14:44] <rubick> hmmm.
IRC [14:44] <Dossy> I'm not sure when/why this changed on the unix platforms.
IRC [14:44] <rubick> both:
IRC [14:44] <rubick> make AOLSERVER=/usr/local/src/aolserver40r10/aolserver/
IRC [14:44] <rubick> and
IRC [14:44] <rubick> make AOLSERVER=/usr/local/src/aolserver40r10/
IRC [14:45] <rubick> don't work
IRC [14:45] <Dossy> try NSHOME instead of AOLSERVER. nsencrypt's makefile might be lame.
IRC [14:47] <rubick> Closer..
IRC [14:47] <Dossy> ?
IRC [14:47] <rubick> make NSHOME=/usr/local/src/aolserver40r10/aolserver
IRC [14:47] <rubick> make: *** No rule to make target `/include/Makefile.global'. Stop.
IRC [14:47] <rubick> oops, damn slashes
IRC [14:47] <rubick> make NSHOME=/usr/local/src/aolserver40r10/aolserver
IRC [14:47] <rubick> xxx /usr/local/src/aolserver40r10/aolserver/include/Makefile.module:108: /include/Makefile.global: No such file or directory
IRC [14:47] <rubick> make: *** No rule to make target `/include/Makefile.global'. Stop.
IRC [14:48] <Dossy> set AOLSERVER= and NSHOME= ?
IRC [14:48] <rubick> There we go!
IRC [14:48] <Dossy> nsencrypt's Makefile wants NSHOME (dumb), and AOLserver uses AOLSERVER.
IRC [14:48] <Dossy> nsencrypt needs fixoring.
IRC [14:48] *** holycow joined the chat.
IRC [14:49] <rubick> Hmm, I get an error, but the .so file is there.
IRC [14:49] <Dossy> what error?
IRC [14:50] <rubick> make: *** [all] Error 1
IRC [14:50] <rubick> not very informative?
IRC [14:50] * rubick edits the wiki with some install instructions for nsencrypt.
IRC [14:50] <Dossy> what are the lines before that?
IRC [14:51] <rubick> _Ns_ConfigGetPath
IRC [14:51] <rubick> _Tcl_DStringInit
IRC [14:51] <rubick> _RAND_load_file
IRC [14:51] <rubick> _OpenSSL_add_all_digests
IRC [14:51] <rubick> _OpenSSL_add_all_ciphers
IRC [14:51] <rubick> _OPENSSL_add_all_algorithms_noconf
IRC [14:51] <rubick> _ERR_load_crypto_strings
IRC [14:51] <rubick> make: *** [all] Error 1
IRC [14:51] <rubick> a lot of lines like that.
IRC [14:51] <Dossy> "It would be really nice to have some install instructions here :)" <-- was that you? :)
IRC [14:51] <rubick> No errors that I see.
IRC [14:51] <rubick> Yes :)
IRC [14:51] <rubick> Just added some.
IRC [14:51] <Dossy> those lines are linker errors. that .so is busted.
IRC [14:51] <rubick> ag
IRC [14:52] <Dossy> i'm going to check out nsencrypt HEAD and see what's up
IRC [14:52] <rubick> Wow, Mac OS X doesn't have ldd?
IRC [14:52] <rubick> Thanks, Dossy.
IRC [14:59] <Dossy> holy cow the compile is awfully noisy.
IRC [15:00] <rubick> yeah
IRC [15:07] <Dossy> OK, just committed a changed Makefile.
IRC [15:07] <Dossy> tagged as nsencrypt_v0_r4
IRC [15:07] <rubick> thanks!
IRC [15:07] <Dossy> and I'm uploading to SF files section (files section still had 0.1)
IRC [15:13] <Dossy> OK, wiki page is edited.
IRC [15:13] *** cnk joined the chat.
IRC [15:13] <Dossy> hi cnk.
IRC [15:13] <cnk> Hi Dossy
IRC [15:14] <cnk> more performance tuning fun!
IRC [15:14] <Dossy> the link to the readme.txt will be "outdated" until anonymous CVS (cvsweb) syncs up with project CVS.
IRC [15:14] <cnk> but this time we have data - but still not really clueful
IRC [15:14] <Dossy> yay!
IRC [15:14] <Dossy> data is good.
IRC [15:15] <cnk> so aolserver without the ACS screams - but with the ACS (I am still running 3.4.10) sucks
IRC [15:15] <cnk> and so far the few tuning parameters we have played with have not shifted our threashold much
IRC [15:16] <Dossy> yeah, i don't think it's a tuning problem so much as a coding problem at this point :)
IRC [15:16] <cnk> What are the params that folks usually start tuning first? and what are reasonable values? Things that I have seen mentioned include tuning the fastpath cache and adjusting thread params
IRC [15:16] <Dossy> ACS, or OpenACS?
IRC [15:16] <cnk> I am on ACS - as is Janine - but different version thereof
IRC [15:16] <Dossy> as you said, vanilla AOLserver is *quite* fast right?
IRC [15:17] <Dossy> things I'd tune: minthreads, maxthreads.
IRC [15:17] <cnk> very - even with the untuned param
IRC [15:17] <Dossy> outside of that, you don't have too many knobs. nsvbuckets, maybe. depends on where your cnotention is
IRC [15:17] <Dossy> er contention is.
IRC [15:17] <Dossy> if it's ACS, you're liekly hitting Oracle?
IRC [15:18] <Dossy> make sure your Oracle is tuned. turn SQL logging/auditing on in Oracle and look at all the queries executed in the oracle trace log. make sure they're all FAST (queries complete in less than 100 msec)
IRC [15:19] <Dossy> i'm betting the bottleneck isn't in AOLserver (as you've observed, vanilla AOLserver is fast).
IRC [15:19] <cnk> The server that is giving me the most trouble doesn't do much querying - but does do inserts for the session stuff
IRC [15:19] <Dossy> yeah. if those inserts are slow ... well ...
IRC [15:19] <cnk> just hitting Oracle for simple things did not do a lot to slow down AOLserver
IRC [15:20] <Dossy> it's easy to find out: just turn the trace log on in Oracle and then look at all the queries executed. anything that's slow (>100ms, IMHO) needs to be fixed.
IRC [15:22] <cnk> I'll look. What I would kind of like to test is a server with thread sizes about the same as my w/ACS server, but that is not hitting the database at all.
IRC [15:23] <rubick> I wonder if that's what's happening on my OpenACS site: usbakery.com It's serving almost completely html files, and is serving a few logins as well. But perhaps the sessions are what's doing it in?
IRC [15:23] <rubick> it's restarting a couple of times a day now.
IRC [15:24] <cnk> http://salinger.caltech.edu/autobench/output/ACSvsAOLServer.jpg
IRC [15:25] <cnk> so that is a graph of our autobench results (autobench is a wrapper for httperf)
IRC [15:26] <cnk> the purple is AOLServer with the ORacle driver loaded, but hitting a static page. The red/green line is the same setup but hitting a page that does "select sysdate from dual" every hit
IRC [15:26] <cnk> and the short test is ACS hitting a static html page
IRC [15:27] <Dossy> cnk: it's hard to make out the ilnes
IRC [15:27] <Dossy> what's the x and y axis?
IRC [15:27] <cnk> the y axis is requests per second
IRC [15:28] <cnk> the x axis is response time in ? milliseconds
IRC [15:29] <cnk> bottom line is that with the ACS loaded - but only using the session stuff, no real database backed pages - I can only serve about 400 requests per second
IRC [15:31] <cnk> on a dual Xeon 2 Ghz machine w 4 G memory
IRC [15:33] <Dossy> "only" :)
IRC [15:34] <cnk> I could start hacking up the application code to see if I can find where it is spending its time. This is a staging setup so I could comment out all the security stuff, for example. That did not give much of a change when I tried a preliminary test - but that was before I had this testing setup worked out
IRC [15:34] <Dossy> okay, I still don't understand the graph - wtf?
IRC [15:35] <Dossy> the purple line is vanilla AOLserver and it can't do more than 0 req/sec?
IRC [15:35] <Dossy> and response time goes up to 8000 ms?
IRC [15:35] <cnk> well high values on the y scale = slow (bad), high values on the x axix = good (more reqeusts per sec)
IRC [15:35] <Dossy> oh, y = msec?
IRC [15:36] <Dossy> you said y axis = req/sec
IRC [15:36] <cnk> sorry x = req/sec
IRC [15:36] <Dossy> right - ok. so vanilla aolserver, 8000 concurrent req all serve under 100 msec
IRC [15:36] <cnk> exactly
IRC [15:37] <Dossy> stockACS_static at 1000 req/sec takes 1200 msec to respond?
IRC [15:37] <cnk> and vanilla aolserver hitting Oracle every request can do 4000 req/sec before it starts to slow down
IRC [15:37] <cnk> dossy, yes
IRC [15:37] <Dossy> what's minthreads/maxthreads set to? and db pool size?
IRC [15:37] <Dossy> that might be oracle bottlenecking at 4000 queries/sec. where's the oracle db?
IRC [15:38] <cnk> defaults on min and max threads
IRC [15:38] <cnk> db pool size 4
IRC [15:38] <cnk> same machine
IRC [15:39] <Dossy> pool size 4? yeah - at 1000 req/sec per db conn, that's pretty good :)
IRC [15:39] <Dossy> bump pool size = 16
IRC [15:39] <Dossy> and min/max threads up to 50
IRC [15:39] <Dossy> and see what the NoACS_dbtcl bench looks like
IRC [15:40] <cnk> OK we'll give that one a whirl as soon as our running test finishes. It will proabably be mid-afternoon before we can plot that
IRC [15:40] <Dossy> oh? wow - how long do the tests take?
IRC [15:40] <cnk> a couple of hours
IRC [15:41] <Dossy> it'd be cool if you'd publish your benchmarks, too
IRC [15:41] <Dossy> at least the vanilla AOLserver ones. :)
IRC [15:41] <Dossy> a couple of *hours* ? yipes.
IRC [15:41] <cnk> hey all the data!
IRC [15:41] <Dossy> heh. can't you run just the one test?
IRC [15:41] <Dossy> and it's "AOLserver" not "AOLServer" :)
IRC [15:41] <cnk> but yes, I would like to write up our tuning set up and benchmarks
IRC [15:42] <cnk> No. there seems to be some hysteresis in our set up - probably the unlimited memory growth we are seeing. If we don't run the tests with the same step size, # of reps, etc. Then the results are not comparable
IRC [15:46] <Dossy> interesting
IRC [15:46] <Dossy> you scientists... :)
IRC [15:46] <cnk> well we tried the "down and dirty" and just got irreproducable junk. Very frustrating
IRC [15:47] <cnk> by the way, what is the deal with Nathan's script to look at pools? He said something about "Zippy memory allocator"
IRC [15:48] <cnk> tried his script and we don't have anything for ns_info pools
IRC [15:50] <Dossy> oh?
IRC [15:50] <Dossy> oh, you're in 3.x
IRC [15:50] <Dossy> that's all 4.x stuff.
IRC [15:50] <cnk> OK
IRC [16:20] <Dossy> cnk: when you publish, please include the full hardware details of the server(s) involved
IRC [16:20] <Dossy> and OS details, etc.
IRC [16:20] * Dossy bounces. this is good :)
IRC [17:09] *** iharding parted the chat.
IRC [17:17] *** holycow parted the chat.
IRC [17:20] *** iharding joined the chat.
IRC [17:22] <iharding> \join #netbsd
IRC [17:35] *** holycow joined the chat.
IRC [17:46] *** bartt joined the chat.
IRC [17:46] *** iharding parted the chat.
IRC [17:56] *** holycow parted the chat.
IRC [18:36] *** holycow joined the chat.
IRC [19:21] *** holycow parted the chat.
IRC [20:42] *** bartt parted the chat.