AOLserver Cookbook

A collection of questions, and code examples to answer them.

Add your questions here. Answers should be placed below the question. Place a horizontal line ("----") between questions. Please keep this text before the first question.

---

Limiting bandwidth

Is there a simple way to limit bandwidth per connection/ip???

There's no simple way of doing it, i.e., no configuration option you can set. If I had to implement this, I'd use NSV's and ns_register_filter to keep track of bandwidth consumed by each client IP which can act as an all-or-nothing mechanism. However, what might be interesting to implement in the socket driver is a setting to control the rate at which bytes are written back to the socket, perhaps configurable per request, so that large HTTP responses don't soak up your upstream bandwidth. We should keep this in mind for a future version of AOLserver. -- Dossy 20:06, 4 November 2005 (EST)

One more question... How can I read output buffer inside function registered as postauth filter? And how can I stream it? ns_adp_puts cannot be used inside filters... ns_return would return all data at once. I assume that per connection limit would look like this:

proc ..... {
ns_adp_stream
set data [get output content buffer]
set data_length [output buffer size]
set block_size 4096
for {set i 0} {$i < $data_length} {incr i $block_size} {
   send_to_client [string range $data [expr $block_size*$i] [expr $block_size*[expr$i+1]]]
   sleep 1
}
ns_returnok
return filter_return
}

I want to limit only one vhost so nssock limit would be too much;)

First, you don't want to turn ns_adp_stream'ing on. Getting the size of the ADP output buffer can be done using ns_adp_tell and getting the contents with ns_adp_dump. I guess you could ns_adp_trunc the output buffer to clear it, then slowly write the contents back using ns_write. You'd also then need to look at ns_conn outputheaders to make sure that you wrote the proper HTTP response ... ah, but I don't know if you can get the HTTP response status code somehow. I still think doing it in nssock is probably the best place, even with vhosts -- I'd make it a preauth or postauth filter that sets the configuration for the response and then have nssock be responsible for ensuring the response is written at that rate. -- Dossy 07:24, 8 November 2005 (EST)

ns_adp_tell inside postauth filter:

  Error: This function cannot be used outside of an ADP

what's more, i use ns_returnfile inside some adp code and if i use ns_adp_tell after it, it returns 0 (this adp only returns file, no adp_puts or write). Unfortunetly i think there is no other solution then implement it inside nssock or implement some tcl code to manipulate all output buffer.

I use this code for big files... some time I'll add per ip limit, now it's for connection.

proc returnfile { file } {                                                      
    set filefp [open $file r]                                                   
    fconfigure $filefp -translation binary                                      
    set data_length [file size $file]                                           
    set block_size 4096                                                         
                                                                                
    ns_write "HTTP/1.0 200 OK                                                   
MIME-Version: 1.0                                                               
Content-Type: [ns_guesstype $file]                                              
Content-Length: $data_length                                                    
                                                                                
"                                                                               
    while { ! [eof $filefp] } {                                                 
        ns_writefp $filefp $block_size                                          
        ns_sleep 1                                                              
    }                                                                           
                                                                                
    ns_conn close                                                               
}

Paul Bukowski

---

Virtual server configuration

Is it possible to use wildcard '*' in domain name inside "ns/module/nssock/servers" section of aolserver4 config file? I would like to map all requests to *.somedomain.org to one virtual host... Something like this doesn't work:

ns_section "ns/module/nssock/servers"
ns_param   forum           somedomain.org                                           
ns_param   forum           somedomain.org:80                                        
ns_param   forum           *.somedomain.org                                       
ns_param   forum           *.somedomain.org:80
ns_param   forum           "*.somedomain.org"                                       
ns_param   forum           "*.somedomain.org:80"

-Paul Bukowski

2005aug15 Dossy: Unfortunately, no -- you currently can't use wildcards the way you want. Not yet, at least.

Why won't you get some code for mass virtual hosts from http://naviserver.sourceforge.net/ ?? -PB

Right now, to map a request to its (virtual) server, there's an O(n) search being done, but it's a "cheap" exact string comparison. To implement glob-style wildcard support, it'll become a more expensive (but possibly still "cheap" by most people's standards) glob-match comparison. When you're serving "n" requests per second (where "n" is small) the difference in cost is probably immeasurable, but when "n" is large, the cost is additive and possibly significant. It might be good if someone could benchmark the two and see what the real cost is. -- Dossy 20:12, 4 November 2005 (EST)

---

Server-side Includes

Does anyone have some code to implement apache-compatible SSI (Server-side includes)? I have some old pages that use SSI that I'd like to move to my aolserver install but I don't have the time to rewrite them as ADPs.

Dossy 20:20, 4 November 2005 (EST): I make no representations or warranties about the safety (!) or correctness of the code I'm about to share, but it might get folks started on a proper implementation in the future. In particular, I didn't implement any commands other than "include" and I even implemented it "wrong" (not according to the specification). If there's real interest in having a complete SSI implementation for AOLserver, I'd be willing to improve this code.

ns_log Notice "ssi: registering SSI ADP tags"

proc ssi_include_tag {params} {
        set param_count [ns_set size $params]
        for {set i 0} {$i < $param_count} {incr i} {
                set ssi_var_[ns_set key $params $i] [ns_set value $params $i]
        }

        if [info exists ssi_var_file] {
                if {[ns_adp_argc] > 1} {
                        set basedir [lindex [ns_adp_argv] 1]
                } else {
                        set basedir [ns_adp_dir]
                }
                if [regexp {^/} $ssi_var_file] {
                        set filename $ssi_var_file
                } else {
                        set filename $basedir/$ssi_var_file
                }
                return [ns_adp_parse -file $filename $basedir]
        }
}

ns_register_adptag {!--#include} ssi_include_tag

ns_log Notice "ssi: registering SSI ADP tags done"

---

Is there a way to see only the errors (for example - 'variable' xxx not found? I seem to remember 2 logs - servername.log and servername-error.log? The version I have now - aolserver 3.2/ad1.2 only seems to give us servername.log - and all the error messages are mixed with the normal msgs - its very hard to debug! dannyl50

The default/example config file did not have entries for both logs. The two sections you need look like (in old style notation):

[ns/parameters]
User=nsadmin 
Group=nsadmin 
ServerLog=/var/log/aolserver/servername-error.log 
PidFile=/var/log/aolserver/nspid.servername
Home=/usr/bin/aolserver-3.3ad13 

and

[ns/server/servername/module/nslog]
File=/var/log/aolserver/emp.log 
LogCombined=On 
MaxBackup=3 
RollFmt=%Y-%m-%d-%H:%M 
RollHour=0 
RollOnSignal=On 
RollLog=On

---

Howdy! Can someone tell me, in general terms, how they have implemented nsperm? It looks cool, but I am using a plain old database with names and passwords, and it is very easy. I would like the allow/deny functionality, can I have that without the username/password part? -- Ian Harding

I've implemented nsperm to restrict access to specific URLs using HTTP authentication (the username/password dialog) and cookie-based (if they have the correct cookie, otherwise URL redirect them to a login page that may set the cookie). The latter is done using a registered preauth filter. I chose to use nsperm over a database since nsperm (presumably) manages the authentication information in memory better than repeatedly hitting the database with authentication requests, but there's no reason I couldn't have used a database to store username/password information. Does this answer your question? -- Dossy

Pretty much, although I started writing my app before I understood AOLServer, so I have a rather strange design that is good in some ways, bad in others. Keep in mind that this is a low activity app, no more than 20 concurrent users right now.

What I did was to create an ADP that has a big case statement. It calls functions to generate pages based on the 'task' value it is passed through POST. I have created a pretty complex app this way, not using ns_register_proc at all. I know now I should have. As I understand it, ns_register_proc lets you create a 'virtual' directory structure that actually just calls certain procedures based on the GET or POST requested URL. This is far more graceful than hitting my main adp which opens up the form data, looks at the 'task' value, and decides which procedures to call.

Given this lame structure, each time someone requests the main adp, I just check for a valid cookie and generate a login page if it's not there. After that, I have to set another cookie that is their 'access rights' which is then checked from within the procs that generate the pages. If I was doing it right, I could have registered a proc that would fire for all URLs in my virtual tree, checking your rights to the particular URL you reqested, and redirecting you if you are not allowed, right?

Thanks!!! -- Ian Harding

I probably would've created an ADP per task, and then registered a preauth filter that checked for the cookie and redirected the user to the login page (outside the directory that the preauth filter covers -- avoid funny redirection loops) if the cookie wasn't set properly. If you don't want to create a separate ADP per task, you could have used ns_register_proc to create what you describe as "virtual pages" and still have the preauth filter kind of "sit in front" and handle the autehntication, keeping that logic seperate from your actual page generation code. -- Dossy

I started with a separate adp for each task, but I thought it would get out of hand quickly. I do think I will re-work the thing to use ns_register_proc and even do away with the few adp I do have. It seems far easier to maintain. In the meantime, I have to re-write all my SQL Server procedures in PostgreSQL PL/Tcl. That's why I want the nsfreetds module so badly, it buys me a little time!

Thanks for all your work! -- Ian Harding

---

Speaking of nsfreetds, did you try nsfreetds-0.1pre? Any luck? Any problems? -- Dossy

I just now tried it. I installed freetds with no trouble, compiled nsfreetds, modified my nsd.tcl config file, added the datasource to interfaces, and all seems fine, except for that I always get "no access to pool "mypool" while executing "ns_db gethandle mypool" ..."

I must have missed something. I saw something in the README about setting $env(SYBASE) before starting nsd. Is this the same as the SYBASE=/usr/pkg/freetds I had to do even before the build of nsfreetds?

Thanks again... I feel like I am close.. -- Ian Harding

Yes, the $env(SYBASE) means setting the SYBASE environment variable. To be safe, you can try this:

  $ cd /path/to/aolserver/install
  $ SYBASE=/usr/pkg/freetds; export SYBASE
  $ bin/nsd ...args...

Check log/server.log to make sure that the nsfreetds driver is actually loading. You should see a lines like this at start-up:

  [30/May/2001:13:42:37][11838.1024][-main-] Notice: modload: loading '/home/aolserver-dev/bin/nsfreetds.so'
  [30/May/2001:13:42:37][11838.1024][-main-] Notice: Ns_FreeTDS_DriverInit(freetds):  Loaded Panoptic FreeTDS Driver v0.1-pre, built on May 21 2001 at 22:23:01.

Are you getting an error in the server log instead? -- Dossy

I don't get an error, it seems to load nicely. I can post or give you access to whatever files or output you would like to see, or an ssh login... -- Ian Harding

---

Another question: Global variables (nsv*). I am re-writing my app to use the ns_register_proc functionality versus my kludge where everything goes through an ADP that acts as a filter and traffic cop. I have figured out register filter and register proc, but seem not to be able to use ns_puts from within the procs. Is this right? Anyway, I had been using ns_adp_include in my filter/traffic cop page to put consistent headers/footers on my pages. I gather the more correct way (and only possible way with registered procs?) is to use global variables.

When do I initialize them? How do I use them? What are the caveats? W

Thanks, -- Ian Harding

Have you started by reading the NSV documentation? -- Dossy

Of course not 8^0 That would be too easy! I will... -- Ian Harding

OK, I read it. It's too easy. I use it for header, footer, and client side script html, which I stuff into html variable before returning it. I would like the filter option we talked about in the chat which would prepend or append html automatically, but this will do. -- Ian Harding

---

How hard would it be to add an HTTPS equivalent of Ns_FetchURL?

If you can build shared libraries on your target platform with OpenSSL, it probably wouldn't be too hard. The trick would be writing enough stub code that can be loaded via the Tcl "load" proc so that you could access the OpenSSL-based https client from your Tcl code. Is there a lot of demand for this? I could probably bang something out as an initial attempt. -- Dossy

Scott Goodwin's nsopenssl now contains client side https functionality.

---

Am I a brick, or does ns_register_proc just plain not work with more than one argument for the proc? I have tried the following code on both aolserver 3.2 running on NT4 and aolserver 3.4 on NetBSD 1.5. It fails to register the proc even once, if I try to register it with more than one argument anywhere else.

ns_register_proc GET /foo1 foo 1
ns_register_proc GET /foo2 foo 1 2
ns_register_proc GET /foo3 foo 1 2 3

proc foo { conn one {two 22} {three 333}} {
ns_return 200 text/html "one is $one, two is $two, three is $three"
}

This must be something simple. I have already posted to the aolserver list but the listserv seems to have gone mad.

Ian Harding

What are you trying to do? It appears you want to pass constant parameters to your procedures. If they are constants, why pass them? Since ns_register_proc calls are generally in the global scope (and sourced when nsd starts) even a parameter that appears to be a variable probably can/will not vary.

Presumably you need to get information from one page to another. That means formvalues unless your page is stateful (usually a bad idea). Check out the ns_conn form command to get yourself started. -- Jason Kane

---

OK, I'm using Windows 2000 SP2 and 3.4.1, and trying in vain to get Perl CGIs (legacy) to work. The scripts work outside of Aolserver, and work on AOLserver on Linux, but when I try inside the script to GET from myself (eg, GET http://localhost:8000/foo?bar=bas), I'm told "unknown protocol TCP", which is quite aggravating.

Does anyone have any experience administering AOLServer on Windows?

Thanks in advance!

-Bill

---

This is interesting, wy do you let me change your page? I'm trying to find an example of how to get my data using:

set formdata [ns_conn form $conn]

-Jason

Because it's a wiki! A proc that gets posted variables includes the line you wrote there... then when you want to actually get the values, you can use something like

set myvar [ns_set get $formdata myvar]

where myvar is the name of the variable. There is also ns_queryget which looks like

set myvar [ns_queryget myvar defaultval]

This is handy because if there is no value, it will assign a default value.

Check out aolserver.com, they have extensive (although sometimes misleading) docs. If something the docs says should work doesn't, ask. It may be out of date.

Ian Harding

---

I don't know where else to share my code fragments, so here seems like as good a place as any. This one implements user public_html directories. It's based on someone else's stuff (I forget whose tho). It should really do something with unknown mime types, and should do a better job with errors like unknown users. It does honor directoryFile and adp maps in the config file.

It works by registering a procedure to /users that interprets the username and finds their directory. ns_register_proc doesn't allow wildcards other than in the last path element, so /~* wouldn't work. However that's not a problem for a filter, so there's a filter to take your /~user url and redirect it to /users/user

-JR

proc homedir { conn ctx } {
  set url [ns_conn url $conn]
  set user {}
  set file {}
  regexp {^/users/([^/]*)(/.*)?$} $url dummy user file
  # ns_returnnotice 200 "url is $url user is $user, file is $file"
  set userfile [getUserFile $user $file]
  set type [ns_guesstype $userfile]
  if {[string match "*[ns_config ns/server/[ns_info server]]/adp map]" $userfile]} {
    # ns_log notice "adp parsing $userfile"
    set cwd [pwd]
    cd [file dirname $userfile]
    # ns_return 200 text/html [ns_adp_parse -file $userfile]
    set result [catch {ns_adp_parse -file $userfile} error]
    if {$result} {
      ns_return 200 text/html $error
    } else {
      ns_return 200 text/html $error
    }
    cd $cwd
  } else {
    switch $type {
      "*/*" {
      }
      default {
       ns_returnfile 200 $type $userfile
      }
    }
  }
      
  if {[catch {set fp [open $userfile]}]} {
    ns_returnnotfound
    return
  }
}

proc getUserFile {user path} {
  foreach tail [split ,[ns_config ns/server/[ns_info server] directoryfile] ,] {
    set userfile "[glob ~$user]/public_html$path$tail"
    if {[file exists $userfile] && ![file isdirectory $userfile]} {
      return $userfile
    }
  }
  ns_returnnotfound
  # break
}

ns_register_proc GET /users/ homedir
ns_register_proc POST /users/ homedir
ns_register_proc HEAD /users/ homedir

proc userRedir {conn arg why} {
  set url [ns_conn url]
  regsub {^/~(.*)} $url {/users/\1} redir
  ns_log notice "url is $url, redir is $redir"
  ns_returnredirect $redir
  return filter_break
}

ns_register_filter preauth GET /~* userRedir

---

Hi!

Does anybody know a graphing module for Aolserver/TCL, with e.g. bargraphs, linegraphs,... ?

-wiwo

Yes: nsgds, nschartdir.

---

Can I detect if the user pushes the Stop button in their browser in my TCL/ADP page?

Yes, although it's a kludge. ns_write will return zero if it cannot write because the connection has been lost. However, ns_write is not stupid enough to try writing when you give it nothing to write, and so you must send something to the client. To prevent this being rendered use HTML comments, e.g.

if {[ns_write "<!-- STOP DETECTOR -->"] == 0} {
    # user hit the stop button, browser page was closed, or connection to client was lost.
}

Ximon Eighteen

---

Generating random passwords

The following method can be useful to generate random passwords and/or password salts.

 # desired length of random string
  set iStringLength 8

 # characters allowed to appear in the string
  set sAllowableCharacters "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"

 # get length one time
  set iAllowableCharacters [string length $sAllowableCharacters]

 # begin random string as empty
  set sRandomString ""

 # loop until desired length is achieved
  for {set ii 0} {$ii < $iStringLength} {incr ii} {

   # get next random between 0 and length of allowed characters
    set iRandomInt [ns_rand $iAllowableCharacters]

   # retrieve this random character
    set sRandomCharacter [string index $sAllowableCharacters $iRandomInt]

   # append the character to our random string
    append sRandomString $sRandomCharacter

  }

  ns_adp_puts "sRandomString: $sRandomString"

---

Customizing error pages

Edit your AOLServer's config file, and add the paths to the error pages as follows:

#
# Internal redirects
#
ns_section "ns/server/${servername}/redirects"
ns_param   404 "/notfound.html"      ;# Not Found error page
ns_param   500 "/servererror.html"   ;# Server Error page

---

Fast caching of generated pages

You can use an adp page to handle a 404 (not found) error that generated exactly the page it was looking for. Conceptually this is similar to how tcl's autoloading using unknown works. From Bas Sheffers, via John Buckman.

from config.tcl:

ns_section "ns/server/$server1/redirects"
ns_param   404           /404.adp

404.adp:

<% handle_404 %>

Support procs:

proc handle_404 {} {
    set myurl [myurl]
    set parts [split $myurl /]
    set root [lindex $parts 1]

    if {$root == "photo"} {
        return [404_photo $parts]
    } else {
        ns_returnredirect "[myhost]/"
        return
    }
    
}

proc 404_photo {parts} {
    
    set userid [file rootname [lindex $parts 2]]
    set d [user_photo_data $userid]
    if {$d == ""} {
        set fn "/b/photo/unknown.jpg"
    } else {
        set fn "/b/photo/${userid}.jpg"
        write_binary_file $fn $d
    }
    ns_returnfile 200 "image/jpeg" $fn
}

proc myurl {} {
    return [lindex [split [ns_conn request]] 1]   
}

proc photo_cache_dirty {userid} {
    set fn "/b/photo/${userid}.jpg"
    file delete $fn
}

Changing maximum upload file size

For aolserver 4.5 and higher, to change the maximum upload file size from the default value to a custom one, add a line like this to the server configuration file:

ns_limits set default -maxupload [expr 25 * 1024 * 1024]  ;# Maximum file size set to 25Mb

Return a file generated live

Example using the db_foreach function from OpenACS, for the sake of code readability:

db_foreach get_mailing_addrs "

   select company_name,
          address,
          city,
          province_abbrev,
          postal_code
     from ce_members
    order by company_name" {

         append csv_text "\"$company_name\",\"$address\",\"$city\",\"$province_abbrev\",\"$postal_code\"\r\n"
     }
 
ns_set update [ns_conn outputheaders] content-disposition "attachment; filename=\"member-addresses.csv\""
 
ns_return 200 application/octet-stream $csv_text