Implementing a Mutex Pool

In a multi-threaded application, there may be a need to "dynamically" create mutexes for short-lived operations and the natural desire to destroy them once you are done. Creating and destroying mutexes may be a relatively cheap operation, it's still not the best way to do this. Instead, we can simply implement mutex pools and allocate mutexes out of it.

---

Note that this is not just about performance. Because of the way the underlying POSIX thread API works, there are strong correctness reasons for avoiding the regular use of "ns_mutex destroy". TODO: Add more details about that. --atp@piskorski.com, 2004/05/28 19:02 EDT

---

First, we have to determine the size of our mutex pool. We must decide that "we will never handle more than X concurrent requests", where X may be 1000. At server start-up, create 1001 mutexes. Store 1000 of them in a single NSV, where the key is the mutex ID, value an empty string. Store the 1001th mutex in another NSV, with the key being some constant string and the mutex ID being the value. The code may look something like this:

 nsv_set guardMutex guard ns_mutex create
 for {set i 0} {$i < 1000} {incr i} {
   nsv_set mutexPool ns_mutex create {}
 }

Whenever you need to grab a mutex, you use the NSV to get the mutex ID of that 1001th (guard) mutex, and lock it. Then, you pop a mutex out of the pool, then release the guard mutex. You can then use that popped mutex for the particular request you want to perform:

 ns_mutex lock nsv_get guardMutex guard
 set mutex [[lindex [nsv_array names mutexPool]] 0]
 nsv_unset mutexPool $mutex
 ns_mutex unlock nsv_get guardMutex guard

(Obviously, you want to add some error checking to make sure that [llength nsv_array names mutexPool] is not 0 -- in which case, you've exhausted your mutex pool and should either spinwait for a mutex to be returned to the pool, or handle the error however appropriate.)

When the request is done and you're done with the mutex, instead of destroying it, return it to the pool:

 ns_mutex lock nsv_get guardMutex guard
 nsv_set mutexPool $mutex {}
 ns_mutex unlock nsv_get guardMutex guard

No destroying of mutexes. Except, at server shutdown, you may want to grab the guard mutex and foreach over all the mutexes in mutexPool and destroy them all. You might want to check the size of the pool and if it's not 1000, you know that not all mutexes have been returned to the pool and you can handle that however you please.

Doing this, we no longer have to create and destroy mutexes all the time. We serialize the mutex pool through a single global guard mutex, but that's not as bad as constantly creating and destroying mutexes.