What's new in 4.5

This is a fairly comprehensive list of new features and changes in AOLserver 4.5 and 4.5.1 (compared to 4.0). If you know of anything that is missing, please add it.

Connection Management:

These two commands can be used together to map slow running requests to a specific pool and to set limits to avoid overload.

• New ns_pools command to map method/URL's to specific thread pools.
• New ns_limits command control the number of threads executing and/or waiting for execution by method/URL.
• Note: The new scheme does not pick up connection thread pool configurations from earlier versions of AOLServer. Please see the new example configuration files, and the documentation for ns_pools and ns_limits, for instructions in upgrading your configuration.
  • As of 1 August 2007, the HEAD revision contains "pools.tcl", which will handle many pre-4.5 configurations.
  • There are known issues when using this feature in conjunction with virtual servers; see the SourceForge bug list (and development mailing list archive) for details.
  • Also see AOLserver Tuning Configuration Parameters for a complete list of key tuning configuration parameter changes from 4.0 to 4.5

I/O Features:

• New Ns_QueueWait API to enable event-driven callbacks in the driver thread before dispatching to pools for processing.

The allows drivers to augment data received from the client (headers, request, content) with additional data fetched over the network (likely stored in the new Ns_Cls "connection local storage" API's) before dispatching to the connection threads. An example would be to add certain personalization data received via a web service. The rationale here is that I/O events are cheap so do those upfront instead of having expensive connection threads burdened with wasteful blocking I/O. This is a somewhat obscure and technical interface.

• Added ability to manage larger file uploads by spooling to a temporary file (currenty not for Windows).

• New nsproxy module which provides the ns_proxy command which enables sending Tcl scripts to a proxy process connected via a pipe. The proxy includes core Tcl commands as well as AOLserver commands from the libnsd library. Evaluating scripts in a proxy process can be used to isolate and/or timeout thread-unsafe or otherwise unsafe 3rd party code. Currently this module does not compile under Windows.

ADP Improvements:

• Added a new execution caching technique at the ns_adp_include level which allows you to save the results of execution of an included ADP and includes below that for reuse on subsequent connections up to a specified time.
• Added singlescript config option which turns ADP pages into a single script enabling syntax such as "<% foreach e $list { %> element <%= $e %> here <% } %>".
• Added support for nested adp tags, i.e. you can now do <% ns_adp_puts [ns_adp_eval {<% ... %>}] %> or use <% %> script within the body of a fancy tag. Note that there is a known issue in that only a single inner instance of an adp tag is supported - see Bug #2958550.
• Introduced a better command name to clear long-standing confusion between the previous registertag commands: ns_adp_registerscript instead of ns_register_adptag.
• Added ns_adp_registerproc which is similar to ns_adp_registerscript but the procedure receives individual arguments instead of an ns_set.
• Added output buffer improvements via new Ns_ConnFlush. See ns_adp_close and ns_adp_flush
• Added automatic UTF-8 to output charset encoding.
• Added gzip output compression. KNOWN ISSUE: When streaming is enabled or when the content exceeds the buffer size (see ns_adp_ctl) and is therefore flushed to the browser in multiple pieces, gzip compression is not performed even if you enable it. However, the last piece to be flushed ends up compressed and is therefore seen as garbage by the browser. This issue is fixed in version 4.5.2. You can workaround by explicitly disabling gzip compression when using streaming and for large content either also disabling gzip compression or increasing the buffer size to only have a single flush.
• Added streaming output in chunked-encoding format instead of the previous "response with no length" HTTP/1.0 method.
• Enhanced ADP error handling and reporting.
• Added ability to trace ADP pages. Trace output is written to the server log. See ns_adp_ctl, which is a new command for configuring the ADP execution environment at runtime.

Other Improvements:

• New Ns_TclRegisterTrace API to enable callbacks at key state transition points in a much more natural way. The ns_ictl command, which now has a second and better name that should be used going forward - ns_interp_ctl, has been updated to support script-level traces.
• New Ns_Task API designed to replace the old Ns_SockCallback API which didn't do a very good job at managing timeouts along with I/O events.
• New ns_returnmoved command to return 301 http status code and redirect to a new URL.
• New ns_internalredirect command to restart connection processing with a new URL.
• 4.5.1 - Added ns_conn channel and ns_conn contentsentlength subcommands to ns_conn to allow raw communication via Tcl I/O to the remote client.
• New ns_cache command based on ideas from the nscache module. This is semi-backward-compatible with the ns_cache module. The new command does not support the nscache module's -thread functionality, nor can create virtual-server-specific caches - all caches are now serverwide. More importantly, -timeout is no longer sliding. For details of all the differences see ns_cache.
• New ns_loop_ctl command to monitor and manage for, while, and foreach loops.
• Added ns_ictl once subcommand to ns_ictl / ns_interp_ctl command to execute a script exactly once for a virtual server (i.e. at startup).
• Added ns_ictl package subcommand to ns_ictl / ns_interp_ctl command to handle loading of Tcl packages.
• Added ns_ictl cancel subcommand to ns_ictl / ns_interp_ctl command to provide the ability to sendr async-cancel messages to any interpter listed in ns_ictl threads (another new subcommand)
• New ns_register_fastpath command to re-enable fastpath after a previous call to ns_register_proc.
• New ns_register_encoding command to enable mapping of method/URL combinations to specific charset encodings used to decode the request.
• 4.5.1 - New ns_register_cgi command to dynamically register a CGI executable to a url.
• New ns_driver command to get some stats on socket drivers.
• nszlib module is now integrated into the core and provides for on-the-fly gzip compression and other gzip facilities in ns_zlib
• -all switch added to ns_addrbyhost command.
• Added ability to access more AOLserver functionality from a tclsh (nsv_*, ns_thread, etc).
• Stack checking has returned. In a multi-threaded application like AOLserver, stack checking is important for detecting buffer overruns and deep call trees blowing your thread stack, etc.
• More detailed information when logging uncaught Tcl errors. Especially for connection threads, details of the HTTP request are logged to make diagnosing the root cause of errors easier.

Changes in Behavior from 4.0:

• Interpreter deallocate/cleanup process is different from 4.0 - in 4.0 global variables get unset and database handles get closed prior to running the rest of the oncleanup traces. In 4.5, that happens last.
• Conn ids are no longer allocated/counted per pool per virtual server and only for queued connections - conn ids are now process-wide global for all pools and provide a counter of all requests, including those that did not get to be queued and processed.
• Global conn variable used to only be defined when serving a request for an adp file, now it's always defined.
• ns_adp_dir returns the value of [ns_info pageroot] when called outside of adp, whereas in 4.0 it returned "/" and in versions before, it threw an error.
• ns_adp_mimetype will work outside of adps as well now.
• ns_startcontent -type now requires encoding specification in addition to mime type - to workaround, use a version redefined in Tcl as described under ns_startcontent
• AOLserver 4.5 has a newer version of TclX Keyed List implementation which enforces certain requirements of keyed list that were never enforced before. Specifically, keylkeys will throw an error in AOLserver 4.5 if you attempt to give it a table-like list of lists that has more than 2 columns.
• Connection thread names no longer include the virtual server name and then the pool name - now it's just the pool name instead (thread names are visible in the server log and in /_stats). Scheduled threads don't list their virtual server name in 4.0 either. Both these issues can be fixed as follows (this is included in the ampools module):

   ns_interp_ctl trace allocate {
       namespace eval ::ns {}
       set ::ns::thread_name [ns_thread name]
       ns_thread name "$::ns::thread_name[ns_info server]"
   }
   ns_interp_ctl trace deallocate {
       if {[info exists ::ns::thread_name]} {
           ns_thread name $::ns::thread_name
       }
   }

• In early versions of AOLserver 4.0, ns_tmpnam on Windows used to return names without a path (they were relative to current directory). In later builds, Jamie Rasmussen changed it to use a temp directory using GetTempPath and GetTempFileName with a prefix which resulted in names like %TEMP%/AOL1234.tmp, where 1234 is derived from time. In 4.5.1, ns_tmpnam uses _tempnam but without a prefix and also has no extension and names are just sequential integers with sequence apparently maintained per process, thus making them not very safe. This issue is fixed in version 4.5.2.
• Failure to call ns_adp_abort in an adp page after calling ns_return* will now result in a Tcl error being thrown and logged. On the other hand, without a patch from Tom Jackson (portions of which are included in version 4.5.2), ns_adp_abort itself would throw an error. In addition, adp flush errors (i.e. due to the client no longer being connected) now also result in Tcl errors being thrown and logged. These benign errors can be suppresed using the IgnoreFinalFlushErrors setting in version 4.5.2. See ns_adp_ctl for more details.
• Also see AOLserver Tuning Configuration Parameters for a complete list of key tuning configuration parameter changes from 4.0 to 4.5