$Id: freebsd-threads.html,v 1.2 2003/07/15 03:55:57 bkogawa Exp $
This section was last updated on July 1, 2003. Thanks to Julian Eichler for this information -- this section is entirely based on his email.
In FreeBSD 5.x, there are 3 ABI compatible thread libraries which may be swapped with each other at run time.
Both libkse and libthr are getting to a useable state. It is possible to run most programs with them. Signals seem to be the last "difficult" part of the picture. and that's under active development.
In addition there is libmap (man libmap.conf) that allows the loader to dynamically (at run time) select which thread library any particular program should run with.
I will update this with more information when I get time. Thanks, Julian, for the information!
WARNING: The rest of this document does not describe thread support in FreeBSD 5.x . You have been warned.
There are two major classes of threads for FreeBSD. The first is the default FreeBSD thread package, referred to henceforth as "uthreads" or "user threads". The other is the Linuxthreads port, which is a port of Linux's kernel threads based on the clone() call (or, in FreeBSD's case, the rfork() call).
FreeBSD's threads library, libc_r, has had many features and performance improvements added, which makes it almost completely POSIX-compliant.
Documentation for FreeBSD's native threads is available via the pthread(3) man page.
The libc_r threads are user-space threads.
The libc_r threads are preemptive. (Define preemptive.)
Known problems:
Known problems w/ the pthreads (from grepping 4.0's man pages):
In addition, Linux's kernel-supported LinuxThreads library is now available as a port (ports/devel/linuxthreads), which can be used for native FreeBSD programs.
Linuxthreads are kernel-space threads,
LinuxThreads is an POSIX pthreads implementation using "kernel threads". In this FreeBSD port, a kernel thread is started using rfork (whereas in the original Linux implementation a kernel thread is started using the Linux clone call). This implementaion provides a so-called one-to-one mapping of threads to kernel schedulable entities. For more information see about the original LinuxThreads implementation see:
http://pauillac.inria.fr/~xleroy/linuxthreads/
Note that LinuxThreads has been integrated with the GNU C library (glibc) since version 2.0, so the above URL points to dated information.
See also http://lt.tar.com , and the README file for the linuxthreads port.
on a system with up to date ports, this file appears in
/usr/ports/devel/linuxthreads/files/README.FreeBSD
this file includes information about usage and limitations. The limitations information is here:
Compiling with threads:
User-mode:
0. No installation; included with base OS.
1. #include <pthread.h>
2. Compile with -pthread
Kernel-mode:
0. Install linuxthreads package.
1. #include <linuxthreads/pthread.h>
2. Compile with -L /usr/local/lib -llthread -llgcc_r (?)
Common reentrant extensions (e.g. gethostbyname_r)?
A number of common reentrant functions are not specified by POSIX, but are in common use, including gethostbyname_r(). Since they are not standardized, various platforms have varying implementations, syntax, and semantics for these functions.
Some of these functions are available, some (notably the gethost* and other name resolver functions) are not. They are implemented as part of Linuxthreads. Note that the Linuxthreads implementations are "wrappers" around the non-reentrant forms of these functions. This has two implications. First, the performance will not be as good as if they were "natively" implemented (only one call to gethostbyname_r will actually do a lookup at any given time), and second, the Linuxthreads implementation can be used with the user threads library (as they depend on POSIX mutexes to lock the critical section).
In the future, a more general solution would be to use the newer
resolver calls, e.g. getaddrinfo(3), which also support IPv6. These
calls are preferable but may not yet be available by default in other
operating systems. Note, however, that these functions are currently
NOT thread safe, due to a bug in KAME.
http://orange.kame.net/dev/query-pr.cgi?pr=277
As of FreeBSD 4.1, Doug Rabson's <dfr@nlsystems.com> patches to gdb which make gdb thread-aware have been integrated into the base.
I do not know what the status of using a debugger with Linuxthreads is at the moment.
Should threaded programs should profile fine? I believe that Linuxthreads does not generally profile well.
What about tracing (ktrace / strace) threaded programs?
unknown.
According to the mailing lists, multithreaded C++ exceptions have been broken since about August '99. Unknown whether recent versions (4.1) of FreeBSD have been fixed.
There are a number of places where the current thread implementation will block:
trying to select() on a file (vs. a socket).
If one thread tries to close a file descriptor from which another thread is reading, that the thread trying to close will block until the read completes.
Note that the current user threads are preemptive -- they will preempt a running thread.
Due to limitations in the underlying Linuxthreads library, the Linuxthreads port does not handle thread cancellation in all cases according to POSIX.
Since Linuxthreads produces 1 kernel process (context) per thread, does the number of threads affect performance?
Short answer: it depends.
Long answer: Matthew Dillon indicates that since they are rfork(RF_MEM)'ed process, most of the context, including page tables are shared. "So if you up the kernel resources it should work reasonably well."
Jason Evans warns about the greater overhead due to the large number of processes in cases where there are many threads (e.g. 1 thread per connection in a server), and indicates that his experience is that the default user threads implementation works well in that sort of case.
The native user threads cannot concurrently run more than one thread on separate processors at one time. If you are running a single process with multiple CPU-intensive threads in hopes of utilizing multiple processors, you must use Linuxthreads to get this performance. Note that Linuxthreads will not work correctly prior to 4.0, so you will also need to upgrade to 4.0 for this to work.
This post is enlightening:
"Zhihui Zhang" wrote: > (1) Some people say "For I/O bound activity, kernel threads are a really > bad idea". But I read the following passage from else where: > > Kernel threads perform better in I/O-intensive applications because system > call can be blocking for kernel threads (so there is only one > user-to-kernel and kernel-to-user transition per I/O operation rather than > two). > > So which one is the correct answer? I know there is only userland thread > in FreeBSD, but I would like to know the answer. Both. All I/O is not created equal. For blockable I/O (sockets, both TCP and UNIX domain, ttys, pipes, etc), userland pthreads are excellent. For disk I/O, they're less excellent because disk I/O is always ready to go, and you'll always block waiting for it. Not having multiple pending disk I/O requests can be killer. OTOH, that doesn't apply if your working set fits in memory. > (2) User threads are supposed to be faster than kernel threads in context > switch. But since we poll devices during each context switch, it is > actually slower (poll() is the extra system call). Is this correct? That's hard to say. If you use a "standard" thread-per-connection, then only the threads waiting for input contribute to the poll(). Threads that are working do not. Furthermore, a system call is not necessarily a cause for alarm - yet, that's a prime location for the kernel to decide to switch to another process, but it's not _required_ to do so. Later, scott
Current state of affairs:
Future plans:
I have heard that there is work on hybrid threads for CURRENT. Who is working on this?
Date: Fri, 16 Jun 2000 03:19:56 +0800
From: Chia-liang Kao
Hi,
Alright, people brought up discussion about kernel thread once or more every
year since '97 but not this year, so I'd like to bring it up again.
I spent some time do some kernel hacking and go through the discussion last
winter and the SA paper these days. (I didn't find the discussion till recent
because in I thought -arch had been quiet for years) And, half year had it
been silent after the active discussion.
Is the model for the scenario described in
http://www.freebsd.org/~julian/threads/
the final design decision? (then I think it shall be well documented.)
shouldn't KSE to be renamed to something else? It is somewhat confusing,
I thought KSEs meant the sub-processes(Q) at a glance.
(I think the terms should be frozen ASAP to ease understanding for
new developers on this)
What's the current status of implementation (or implementation design)?
I suppose the works to be done are:
1. libc threadsafication
2. kernel runqueue runs subproc (orig proc becomes subproc and the grouping
concept goes to the new proc)
3. (so called) kse sleep queue(and some modification in syscall stuff?)
4. userland lib support
any timeline on this? I would really like to see the development to be
active(perhaps in a branch?)
ps. Some time-saving references for late comers to catch up:
Terry's summary about thread programming, and when we really need threads:
<199911041804.LAA18253@usr06.primenet.com>
http://www.FreeBSD.org/cgi/getmsg.cgi?fetch=512113+521470+/usr/local/www/db/text/1999/freebsd-arch/19991107.freebsd-arch
The design goal:
<Pine.BSF.4.05.9910311908080.8816-100000@home.elischer.org>
http://www.freebsd.org/cgi/getmsg.cgi?fetch=117210+0+/usr/local/www/db/text/1999/freebsd-arch/19991107.freebsd-arch
is there anything good to summarize the model we'll be using?
links:
http://www.freebsd.org/~julian/threads/
http://people.FreeBSD.org/~deischen/
Please let me know if anything above is wrong, I just swallowed -arch
these days.
Cheers,
CLK
On Wed, May 31, 2000 at 08:20:26PM -0700, Jason Evans wrote:
> There was a lengthy discussion last winter on -arch about improving FreeBSD's
> threads support, and the result of the discussion is that we're pursuing
> scheduler activations (SAs) rather than LWPs. I'm currently working out
> the design on paper, and hope to be coding it soon.
Jason's reply:
Date: Fri, 16 Jun 2000 14:42:59 -0700 From: Jason Evans On Fri, Jun 16, 2000 at 03:19:56AM +0800, Chia-liang Kao wrote: > o Is the model for the scenario described in > > http://www.freebsd.org/~julian/threads/ > > the final design decision? (then I think it shall be well documented.) We have made some refinements to the model, but in general, it is the same idea. Daniel Eischen and I are working on a paper that gives a reasonable overview of the current design. Hopefully we can get it to a postable form in the near future. However, we're probably going to concern ourselves primarily with actually doing the work, rather than telling everybody beforehand exactly how we're doing it. > o shouldn't KSE to be renamed to something else? It is somewhat confusing, > I thought KSEs meant the sub-processes(Q) at a glance. > (I think the terms should be frozen ASAP to ease understanding for > new developers on this) Pay no attention to the naming used in the -arch discussion. It will change. =) > o What's the current status of implementation (or implementation design)? The design is basically done. The scheduler activations implementation has a number of interdependencies with the SMP changes that are currently being worked on, so the kernel support for the new threads library is going to take a bit longer than I would like. Still, I think we'll have scheduler activations done for FreeBSD 5.0. Jason