A programmer's guide to thread programming on FreeBSD

Bryan K. Ogawa <bko_public@unobvious.com>

$Id: freebsd-threads.html,v 1.2 2003/07/15 03:55:57 bkogawa Exp $

Threads Resources

http://www.humanfactor.com/pthreads/
Overview of POSIX threads in general
http://www.kegel.com/c10k.html
An overview of the "10000 connections" problem, including a ton of links to information about threads on a lot of platforms.

Update for FreeBSD 5.x

This section was last updated on July 1, 2003. Thanks to Julian Eichler for this information -- this section is entirely based on his email.

In FreeBSD 5.x, there are 3 ABI compatible thread libraries which may be swapped with each other at run time.

libc_r
the old user space thread library.
libthr
1:1 kernel threads (1 kernel thread per process thread)
libkse
N kernel threads (1 per CPU) for M user threads, with async syscalls to allow multiple user threads to share a single kernel thread. libkse will eventually be called libpthread.

Both libkse and libthr are getting to a useable state. It is possible to run most programs with them. Signals seem to be the last "difficult" part of the picture. and that's under active development.

In addition there is libmap (man libmap.conf) that allows the loader to dynamically (at run time) select which thread library any particular program should run with.

I will update this with more information when I get time. Thanks, Julian, for the information!

WARNING: The rest of this document does not describe thread support in FreeBSD 5.x . You have been warned.

Thread types

There are two major classes of threads for FreeBSD. The first is the default FreeBSD thread package, referred to henceforth as "uthreads" or "user threads". The other is the Linuxthreads port, which is a port of Linux's kernel threads based on the clone() call (or, in FreeBSD's case, the rfork() call).

FreeBSD native threads (libc_r)

FreeBSD's threads library, libc_r, has had many features and performance improvements added, which makes it almost completely POSIX-compliant.

Documentation for FreeBSD's native threads is available via the pthread(3) man page.

The libc_r threads are user-space threads.

The libc_r threads are preemptive. (Define preemptive.)

Known problems:

Known problems w/ the pthreads (from grepping 4.0's man pages):

pthread_rwlock_init / pthread_rwlockattr_setpshared
The PTHREAD_PROCESS_SHARED attribute is not supported.

Linuxthreads

In addition, Linux's kernel-supported LinuxThreads library is now available as a port (ports/devel/linuxthreads), which can be used for native FreeBSD programs.

Linuxthreads are kernel-space threads,

LinuxThreads is an POSIX pthreads implementation using "kernel threads". In this FreeBSD port, a kernel thread is started using rfork (whereas in the original Linux implementation a kernel thread is started using the Linux clone call). This implementaion provides a so-called one-to-one mapping of threads to kernel schedulable entities. For more information see about the original LinuxThreads implementation see:

http://pauillac.inria.fr/~xleroy/linuxthreads/

Note that LinuxThreads has been integrated with the GNU C library (glibc) since version 2.0, so the above URL points to dated information.

See also http://lt.tar.com , and the README file for the linuxthreads port.

on a system with up to date ports, this file appears in

/usr/ports/devel/linuxthreads/files/README.FreeBSD

this file includes information about usage and limitations. The limitations information is here:

  1. Not all libc calls are thread safe. In particular gmtime, localtime, etc are not thread safe. In general, where the pthreads spec calls for "_r" functions, these are either not provided, or if provided are not thread safe (in most cases) and the related libc calls are not thread safe. This differs somewhat from the FreeBSD libc_r library, where some, but not all, of these functions are both thread safe and have "_r" versions.
  2. Not all of the libc calls that are supposed to be cancellation points are implemented as such. While linux threads has the cancel functions implemented, deferred cancellation will not work as required by POSIX 1003.1c-1995, since the co-operation needed from libc is not complete.

Compiling with threads:

User-mode:
0. No installation; included with base OS.
1. #include <pthread.h>
2. Compile with -pthread

Kernel-mode:
0. Install linuxthreads package.
1. #include <linuxthreads/pthread.h>
2. Compile with -L /usr/local/lib -llthread -llgcc_r (?)

Other thread libraries

GNU pth

Ralf Engeshall (sp) has written GNU pth, an LGPL'ed cooperatively multitasking thread library. This library is NOT preemptive, so using it has slightly different behavior than FreeBSD's threads.

ACE

Common reentrant extensions (e.g. gethostbyname_r)?

A number of common reentrant functions are not specified by POSIX, but are in common use, including gethostbyname_r(). Since they are not standardized, various platforms have varying implementations, syntax, and semantics for these functions.

Some of these functions are available, some (notably the gethost* and other name resolver functions) are not. They are implemented as part of Linuxthreads. Note that the Linuxthreads implementations are "wrappers" around the non-reentrant forms of these functions. This has two implications. First, the performance will not be as good as if they were "natively" implemented (only one call to gethostbyname_r will actually do a lookup at any given time), and second, the Linuxthreads implementation can be used with the user threads library (as they depend on POSIX mutexes to lock the critical section).

In the future, a more general solution would be to use the newer resolver calls, e.g. getaddrinfo(3), which also support IPv6. These calls are preferable but may not yet be available by default in other operating systems. Note, however, that these functions are currently NOT thread safe, due to a bug in KAME.
http://orange.kame.net/dev/query-pr.cgi?pr=277

Debugging threaded programs

As of FreeBSD 4.1, Doug Rabson's <dfr@nlsystems.com> patches to gdb which make gdb thread-aware have been integrated into the base.

I do not know what the status of using a debugger with Linuxthreads is at the moment.

Profiling threaded programs

Should threaded programs should profile fine? I believe that Linuxthreads does not generally profile well.

What about tracing (ktrace / strace) threaded programs?

unknown.

What about threading and C++?

According to the mailing lists, multithreaded C++ exceptions have been broken since about August '99. Unknown whether recent versions (4.1) of FreeBSD have been fixed.

Cuurent issues/limitations w/ the current implementation

User threads

There are a number of places where the current thread implementation will block:

trying to select() on a file (vs. a socket).

If one thread tries to close a file descriptor from which another thread is reading, that the thread trying to close will block until the read completes.

Note that the current user threads are preemptive -- they will preempt a running thread.

Linux threads

Due to limitations in the underlying Linuxthreads library, the Linuxthreads port does not handle thread cancellation in all cases according to POSIX.

Choosing between Linuxthreads and native (userspace) threads

Since Linuxthreads produces 1 kernel process (context) per thread, does the number of threads affect performance?

Short answer: it depends.

Long answer: Matthew Dillon indicates that since they are rfork(RF_MEM)'ed process, most of the context, including page tables are shared. "So if you up the kernel resources it should work reasonably well."

Jason Evans warns about the greater overhead due to the large number of processes in cases where there are many threads (e.g. 1 thread per connection in a server), and indicates that his experience is that the default user threads implementation works well in that sort of case.

What about running with SMP on multiple processors?

The native user threads cannot concurrently run more than one thread on separate processors at one time. If you are running a single process with multiple CPU-intensive threads in hopes of utilizing multiple processors, you must use Linuxthreads to get this performance. Note that Linuxthreads will not work correctly prior to 4.0, so you will also need to upgrade to 4.0 for this to work.

What about threaded IO?

This post is enlightening:

"Zhihui Zhang"  wrote:
> (1) Some people say "For I/O bound activity, kernel threads are a really
> bad idea".  But I read the following passage from else where:
>
> Kernel threads perform better in I/O-intensive applications because system
> call can be blocking for kernel threads (so there is only one
> user-to-kernel and kernel-to-user transition per I/O operation rather
than
> two).
>
> So which one is the correct answer?  I know there is only userland thread
> in FreeBSD, but I would like to know the answer.

Both.  All I/O is not created equal.  For blockable I/O (sockets, both TCP
and UNIX domain, ttys, pipes, etc), userland pthreads are excellent.  For
disk I/O, they're less excellent because disk I/O is always ready to go,
and you'll always block waiting for it.  Not having multiple pending disk
I/O requests can be killer.

OTOH, that doesn't apply if your working set fits in memory.

> (2) User threads are supposed to be faster than kernel threads in context
> switch.  But since we poll devices during each context switch, it is
> actually slower (poll() is the extra system call).  Is this correct?

That's hard to say.  If you use a "standard" thread-per-connection, then
only the threads waiting for input contribute to the poll().  Threads that
are working do not.  Furthermore, a system call is not necessarily a cause
for alarm - yet, that's a prime location for the kernel to decide to switch
to another process, but it's not _required_ to do so.

Later,
scott

Contributor's guide to threads on FreeBSD:

Current state of affairs:

Future plans:

I have heard that there is work on hybrid threads for CURRENT. Who is working on this?

Date:      Fri, 16 Jun 2000 03:19:56 +0800
From:      Chia-liang Kao

Hi,

Alright, people brought up discussion about kernel thread once or more every
year since '97 but not this year, so I'd like to bring it up again.

I spent some time do some kernel hacking and go through the discussion last 
winter and the SA paper these days. (I didn't find the discussion till recent 
because in I thought -arch had been quiet for years) And, half year had it 
been silent after the active discussion.

  Is the model for the scenario described in 
  http://www.freebsd.org/~julian/threads/
  the final design decision? (then I think it shall be well documented.)

  shouldn't KSE to be renamed to something else? It is somewhat confusing,
  I thought KSEs meant the sub-processes(Q) at a glance.
  (I think the terms should be frozen ASAP to ease understanding for
   new developers on this)

  What's the current status of implementation (or implementation design)?
  I suppose the works to be done are:

  1. libc threadsafication
  2. kernel runqueue runs subproc (orig proc becomes subproc and the grouping 
     concept goes to the new proc)
  3. (so called) kse sleep queue(and some modification in syscall stuff?)
  4. userland lib support

  any timeline on this? I would really like to see the development to be
  active(perhaps in a branch?)

ps. Some time-saving references for late comers to catch up:

  Terry's summary about thread programming, and when we really need threads:
      <199911041804.LAA18253@usr06.primenet.com>

      http://www.FreeBSD.org/cgi/getmsg.cgi?fetch=512113+521470+/usr/local/www/db/text/1999/freebsd-arch/19991107.freebsd-arch

  The design goal:
        <Pine.BSF.4.05.9910311908080.8816-100000@home.elischer.org>

        http://www.freebsd.org/cgi/getmsg.cgi?fetch=117210+0+/usr/local/www/db/text/1999/freebsd-arch/19991107.freebsd-arch

  is there anything good to summarize the model we'll be using?
  links:
        http://www.freebsd.org/~julian/threads/
        http://people.FreeBSD.org/~deischen/

Please let me know if anything above is wrong, I just swallowed -arch
these days.

Cheers,
CLK

On Wed, May 31, 2000 at 08:20:26PM -0700, Jason Evans wrote:
> There was a lengthy discussion last winter on -arch about improving FreeBSD's
> threads support, and the result of the discussion is that we're pursuing
> scheduler activations (SAs) rather than LWPs.  I'm currently working out
> the design on paper, and hope to be coding it soon.

Jason's reply:

Date:      Fri, 16 Jun 2000 14:42:59 -0700
From:      Jason Evans 

On Fri, Jun 16, 2000 at 03:19:56AM +0800, Chia-liang Kao wrote:
> o Is the model for the scenario described in
> 
>   http://www.freebsd.org/~julian/threads/
> 
>   the final design decision? (then I think it shall be well documented.)

We have made some refinements to the model, but in general, it is the same
idea.  Daniel Eischen and I are working on a paper that gives a reasonable
overview of the current design.  Hopefully we can get it to a postable form
in the near future.  However, we're probably going to concern ourselves
primarily with actually doing the work, rather than telling everybody
beforehand exactly how we're doing it.

> o shouldn't KSE to be renamed to something else? It is somewhat confusing,
>   I thought KSEs meant the sub-processes(Q) at a glance.
>   (I think the terms should be frozen ASAP to ease understanding for
>    new developers on this)

Pay no attention to the naming used in the -arch discussion.  It will
change. =)

> o What's the current status of implementation (or implementation design)?

The design is basically done.  The scheduler activations implementation has
a number of interdependencies with the SMP changes that are currently being
worked on, so the kernel support for the new threads library is going to
take a bit longer than I would like.  Still, I think we'll have scheduler
activations done for FreeBSD 5.0.

Jason