In the rush to personal workstations, though, some of their weaknesses were overlooked. First, the operating system they run, UNIX, is itself an old timesharing system and has had trouble adapting to ideas born after it. Graphics and networking were added to UNIX well into its lifetime and remain poorly integrated and difficult to administer. More important, the early focus on having private machines made it difficult for networks of machines to serve as seamlessly as the old monolithic timesharing systems. Timesharing centralized the management and amortization of costs and resources; personal computing fractured, democratized, and ultimately amplified administrative problems. The choice of an old timesharing operating system to run those personal machines made it difficult to bind things together smoothly.
Plan 9 began in the late 1980[aq]s as an attempt to have it both ways: to build a system that was centrally administered and cost-effective using cheap modern microcomputers as its computing elements. The idea was to build a time-sharing system out of workstations, but in a novel way. Different computers would handle different tasks: small, cheap machines in people[aq]s offices would serve as terminals providing access to large, central, shared resources such as computing servers and file servers. For the central machines, the coming wave of shared-memory multiprocessors seemed obvious candidates. The philosophy is much like that of the Cambridge Distributed System [NeHe82]. The early catch phrase was to build a UNIX out of a lot of little systems, not a system out of a lot of little UNIXes.
The problems with UNIX were too deep to fix, but some of its ideas could be brought along. The best was its use of the file system to coordinate naming of and access to resources, even those, such as devices, not traditionally treated as files. For Plan 9, we adopted this idea by designing a network-level protocol, called 9P, to enable machines to access files on remote systems. Above this, we built a naming system that lets people and their computing agents build customized views of the resources in the network. This is where Plan 9 first began to look different: a Plan 9 user builds a private computing environment and recreates it wherever desired, rather than doing all computing on a private machine. It soon became clear that this model was richer than we had foreseen, and the ideas of per-process name spaces and file-system-like resources were extended throughout the system[em]to processes, graphics, even the network itself.
By 1989 the system had become solid enough that some of us began using it as our exclusive computing environment. This meant bringing along many of the services and applications we had used on UNIX. We used this opportunity to revisit many issues, not just kernel-resident ones, that we felt UNIX addressed badly. Plan 9 has new compilers, languages, libraries, window systems, and many new applications. Many of the old tools were dropped, while those brought along have been polished or rewritten.
Why be so all-encompassing? The distinction between operating system, library, and application is important to the operating system researcher but uninteresting to the user. What matters is clean functionality. By building a complete new system, we were able to solve problems where we thought they should be solved. For example, there is no real [oq]tty driver[aq] in the kernel; that is the job of the window system. In the modern world, multi-vendor and multi-architecture computing are essential, yet the usual compilers and tools assume the program is being built to run locally; we needed to rethink these issues. Most important, though, the test of a system is the computing environment it provides. Producing a more efficient way to run the old UNIX warhorses is empty engineering; we were more interested in whether the new ideas suggested by the architecture of the underlying system encourage a more effective way of working. Thus, although Plan 9 provides an emulation environment for running POSIX commands, it is a backwater of the system. The vast majority of system software is developed in the [oq]native[aq] Plan 9 environment.
There are benefits to having an all-new system. First, our laboratory has a history of building experimental peripheral boards. To make it easy to write device drivers, we want a system that is available in source form (no longer guaranteed with UNIX, even in the laboratory in which it was born). Also, we want to redistribute our work, which means the software must be locally produced. For example, we could have used some vendors[aq] C compilers for our system, but even had we overcome the problems with cross-compilation, we would have difficulty redistributing the result.
This paper serves as an overview of the system. It discusses the architecture from the lowest building blocks to the computing environment seen by users. It also serves as an introduction to the rest of the Plan 9 Programmer[aq]s Manual, which it accompanies. More detail about topics in this paper can be found elsewhere in the manual.
A large Plan 9 installation has a number of computers networked together, each providing a particular class of service. Shared multiprocessor servers provide computing cycles; other large machines offer file storage. These machines are located in an air-conditioned machine room and are connected by high-performance networks. Lower bandwidth networks such as Ethernet or ISDN connect these servers to office- and home-resident workstations or PCs, called terminals in Plan 9 terminology. Figure 1 shows the arrangement.
The modern style of computing offers each user a dedicated workstation or PC. Plan 9[aq]s approach is different. The various machines with screens, keyboards, and mice all provide access to the resources of the network, so they are functionally equivalent, in the manner of the terminals attached to old timesharing systems. When someone uses the system, though, the terminal is temporarily personalized by that user. Instead of customizing the hardware, Plan 9 offers the ability to customize one[aq]s view of the system provided by the software. That customization is accomplished by giving local, personal names for the publicly visible resources in the network. Plan 9 provides the mechanism to assemble a personal view of the public space with local names for globally accessible resources. Since the most important resources of the network are files, the model of that view is file-oriented.
The client[aq]s local name space provides a way to customize the user[aq]s view of the network. The services available in the network all export file hierarchies. Those important to the user are gathered together into a custom name space; those of no immediate interest are ignored. This is a different style of use from the idea of a [oq]uniform global name space[aq]. In Plan 9, there are known names for services and uniform names for files exported by those services, but the view is entirely local. As an analogy, consider the difference between the phrase [oq]my house[aq] and the precise address of the speaker[aq]s home. The latter may be used by anyone but the former is easier to say and makes sense when spoken. It also changes meaning depending on who says it, yet that does not cause confusion. Similarly, in Plan 9 the name /dev/cons always refers to the user[aq]s terminal and /bin/date the correct version of the date command to run, but which files those names represent depends on circumstances such as the architecture of the machine executing date. Plan 9, then, has local name spaces that obey globally understood conventions; it is the conventions that guarantee sane behavior in the presence of local names.
The 9P protocol is structured as a set of transactions that send a request from a client to a (local or remote) server and return the result. 9P controls file systems, not just files: it includes procedures to resolve file names and traverse the name hierarchy of the file system provided by the server. On the other hand, the client[aq]s name space is held by the client system alone, not on or with the server, a distinction from systems such as Sprite [OCDNW88]. Also, file access is at the level of bytes, not blocks, which distinguishes 9P from protocols like NFS and RFS. A paper by Welch compares Sprite, NFS, and Plan 9[aq]s network file system structures [Welc94].
This approach was designed with traditional files in mind, but can be extended to many other resources. Plan 9 services that export file hierarchies include I/O devices, backup services, the window system, network interfaces, and many others. One example is the process file system, /proc, which provides a clean way to examine and control running processes. Precursor systems had a similar idea [Kill84], but Plan 9 pushes the file metaphor much further [PPTTW93]. The file system model is well-understood, both by system builders and general users, so services that present file-like interfaces are easy to build, easy to understand, and easy to use. Files come with agreed-upon rules for protection, naming, and access both local and remote, so services built this way are ready-made for a distributed system. (This is a distinction from [oq]object-oriented[aq] models, where these issues must be faced anew for every class of object.) Examples in the sections that follow illustrate these ideas in action.
Each window is created in a separate name space. Adjustments made to the name space in a window do not affect other windows or programs, making it safe to experiment with local modifications to the name space, for example to substitute files from the dump file system when debugging. Once the debugging is done, the window can be deleted and all trace of the experimental apparatus is gone. Similar arguments apply to the private space each window has for environment variables, notes (analogous to UNIX signals), etc.
Each window is created running an application, such as the shell, with standard input and output connected to the editable text of the window. Each window also has a private bitmap and multiplexed access to the keyboard, mouse, and other graphical resources through files like /dev/mouse, /dev/bitblt, and /dev/cons (analogous to UNIX[aq]s /dev/tty). These files are provided by 8[12], which is implemented as a file server. Unlike X windows, where a new application typically creates a new window to run in, an 8[12] graphics application usually runs in the window where it starts. It is possible and efficient for an application to create a new window, but that is not the style of the system. Again contrasting to X, in which a remote application makes a network call to the X server to start running, a remote 8[12] application sees the mouse, bitblt, and cons files for the window as usual in /dev; it does not know whether the files are local. It just reads and writes them to control the window; the network connection is already there and multiplexed.
The intended style of use is to run interactive applications such as the window system and text editor on the terminal and to run computation- or file-intensive applications on remote servers. Different windows may be running programs on different machines over different networks, but by making the name space equivalent in all windows, this is transparent: the same commands and resources are available, with the same names, wherever the computation is performed.
The command set of Plan 9 is similar to that of UNIX. The commands fall into several broad classes. Some are new programs for old jobs: programs like ls, cat, and who have familiar names and functions but are new, simpler implementations. Who, for example, is a shell script, while ps is just 95 lines of C code. Some commands are essentially the same as their UNIX ancestors: awk, troff, and others have been converted to ANSI C and extended to handle Unicode, but are still the familiar tools. Some are entirely new programs for old niches: the shell rc, text editor sam, debugger acid, and others displace the better-known UNIX tools with similar jobs. Finally, about half the commands are new.
Compatibility was not a requirement for the system. Where the old commands or notation seemed good enough, we kept them. When they didn[aq]t, we replaced them.
The file server has three levels of storage. The central server in our installation has about 100 megabytes of memory buffers, 27 gigabytes of magnetic disks, and 350 gigabytes of bulk storage in a write-once-read-many (WORM) jukebox. The disk is a cache for the WORM and the memory is a cache for the disk; each is much faster, and sees about an order of magnitude more traffic, than the level it caches. The addressable data in the file system can be larger than the size of the magnetic disks, because they are only a cache; our main file server has about 40 gigabytes of active storage.
The most unusual feature of the file server comes from its use of a WORM device for stable storage. Every morning at 5 o[aq]clock, a dump of the file system occurs automatically. The file system is frozen and all blocks modified since the last dump are queued to be written to the WORM. Once the blocks are queued, service is restored and the read-only root of the dumped file system appears in a hierarchy of all dumps ever taken, named by its date. For example, the directory /n/dump/1995/0315 is the root directory of an image of the file system as it appeared in the early morning of March 15, 1995. It takes a few minutes to queue the blocks, but the process to copy blocks to the WORM, which runs in the background, may take hours.
There are two ways the dump file system is used. The first is by the users themselves, who can browse the dump file system directly or attach pieces of it to their name space. For example, to track down a bug, it is straightforward to try the compiler from three months ago or to link a program with yesterday[aq]s library. With daily snapshots of all files, it is easy to find when a particular change was made or what changes were made on a particular date. People feel free to make large speculative changes to files in the knowledge that they can be backed out with a single copy command. There is no backup system as such; instead, because the dump is in the file name space, backup problems can be solved with standard tools such as cp, ls, grep, and diff.
The other (very rare) use is complete system backup. In the event of disaster, the active file system can be initialized from any dump by clearing the disk cache and setting the root of the active file system to be a copy of the dumped root. Although easy to do, this is not to be taken lightly: besides losing any change made after the date of the dump, this recovery method results in a very slow system. The cache must be reloaded from WORM, which is much slower than magnetic disks. The file system takes a few days to reload the working set and regain its full performance.
Access permissions of files in the dump are the same as they were when the dump was made. Normal utilities have normal permissions in the dump without any special arrangement. The dump file system is read-only, though, which means that files in the dump cannot be written regardless of their permission bits; in fact, since directories are part of the read-only structure, even the permissions cannot be changed.
Once a file is written to WORM, it cannot be removed, so our users never see [oq][oq]please clean up your files[aq][aq] messages and there is no df command. We regard the WORM jukebox as an unlimited resource. The only issue is how long it will take to fill. Our WORM has served a community of about 50 users for five years and has absorbed daily dumps, consuming a total of 65% of the storage in the jukebox. In that time, the manufacturer has improved the technology, doubling the capacity of the individual disks. If we were to upgrade to the new media, we would have more free space than in the original empty jukebox. Technology has created storage faster than we can use it.
Perhaps the most remarkable file server in Plan 9 is 8[12], the window system. It is discussed at length elsewhere [Pike91], but deserves a brief explanation here. 8[12] provides two interfaces: to the user seated at the terminal, it offers a traditional style of interaction with multiple windows, each running an application, all controlled by a mouse and keyboard. To the client programs, the view is also fairly traditional: programs running in a window see a set of files in /dev with names like mouse, screen, and cons. Programs that want to print text to their window write to /dev/cons; to read the mouse, they read /dev/mouse. In the Plan 9 style, bitmap graphics is implemented by providing a file /dev/bitblt on which clients write encoded messages to execute graphical operations such as bitblt (RasterOp). What is unusual is how this is done: 8[12] is a file server, serving the files in /dev to the clients running in each window. Although every window looks the same to its client, each window has a distinct set of files in /dev. 8[12] multiplexes its clients[aq] access to the resources of the terminal by serving multiple sets of files. Each client is given a private name space with a different set of files that behave the same as in all other windows. There are many advantages to this structure. One is that 8[12] serves the same files it needs for its own implementation[em]it multiplexes its own interface[em]so it may be run, recursively, as a client of itself. Also, consider the implementation of /dev/tty in UNIX, which requires special code in the kernel to redirect open calls to the appropriate device. Instead, in 8[12] the equivalent service falls out automatically: 8[12] serves /dev/cons as its basic function; there is nothing extra to do. When a program wants to read from the keyboard, it opens /dev/cons, but it is a private file, not a shared one with special properties. Again, local name spaces make this possible; conventions about the consistency of the files within them make it natural.
8[12] has a unique feature made possible by its design. Because it is implemented as a file server, it has the power to postpone answering read requests for a particular window. This behavior is toggled by a reserved key on the keyboard. Toggling once suspends client reads from the window; toggling again resumes normal reads, which absorb whatever text has been prepared, one line at a time. This allows the user to edit multi-line input text on the screen before the application sees it, obviating the need to invoke a separate editor to prepare text such as mail messages. A related property is that reads are answered directly from the data structure defining the text on the display: text may be edited until its final newline makes the prepared line of text readable by the client. Even then, until the line is read, the text the client will read can be changed. For example, after typing
C] % make rm * R]
There is no ftp command in Plan 9. Instead, a user-level file server called ftpfs dials the FTP site, logs in on behalf of the user, and uses the FTP protocol to examine files in the remote directory. To the local user, it offers a file hierarchy, attached to C]/n/ftpR] in the local name space, mirroring the contents of the FTP site. In other words, it translates the FTP protocol into 9P to offer Plan 9 access to FTP sites. The implementation is tricky; ftpfs must do some sophisticated caching for efficiency and use heuristics to decode remote directory information. But the result is worthwhile: all the local file management tools such as cp, grep, diff, and of course ls are available to FTP-served files exactly as if they were local files. Other systems such as Jade and Prospero have exploited the same opportunity [Rao81, Neu92], but because of local name spaces and the simplicity of implementing 9P, this approach fits more naturally into Plan 9 than into other environments.
One server, exportfs, is a user process that takes a portion of its own name space and makes it available to other processes by translating 9P requests into system calls to the Plan 9 kernel. The file hierarchy it exports may contain files from multiple servers. Exportfs is usually run as a remote server started by a local program, either import or cpu. Import makes a network call to the remote machine, starts exportfs there, and attaches its 9P connection to the local name space. For example,
C] import helix /net R]
C] import helix /proc R]
The cpu command connects the local terminal to a remote CPU server. It works in the opposite direction to import: after calling the server, it starts a local exportfs and mounts it in the name space of a process, typically a newly created shell, on the server. It then rearranges the name space to make local device files (such as those served by the terminal[aq]s window system) visible in the server[aq]s C]/devR] directory. The effect of running a cpu command is therefore to start a shell on a fast machine, one more tightly coupled to the file server, with a name space analogous to the local one. All local device files are visible remotely, so remote applications have full access to local services such as bitmap graphics, C]/dev/consR], and so on. This is not the same as rlogin, which does nothing to reproduce the local name space on the remote system, nor is it the same as file sharing with, say, NFS, which can achieve some name space equivalence but not the combination of access to local hardware devices, remote files, and remote CPU resources. The cpu command is a uniquely transparent mechanism. For example, it is reasonable to start a window system in a window running a cpu command; all windows created there automatically start processes on the CPU server.
The system software is portable and the same operating system runs on all hardware. Except for performance, the appearance of the system on, say, an SGI workstation is the same as on a laptop. Since computing and file services are centralized, and terminals have no permanent file storage, all terminals are functionally identical. In this way, Plan 9 has one of the good properties of old timesharing systems, where a user could sit in front of any machine and see the same system. In the modern workstation community, machines tend to be owned by people who customize them by storing private information on local disk. We reject this style of use, although the system itself can be used this way. In our group, we have a laboratory with many public-access machines[em]a terminal room[em]and a user may sit down at any one of them and work.
Central file servers centralize not just the files, but also their administration and maintenance. In fact, one server is the main server, holding all system files; other servers provide extra storage or are available for debugging and other special uses, but the system software resides on one machine. This means that each program has a single copy of the binary for each architecture, so it is trivial to install updates and bug fixes. There is also a single user database; there is no need to synchronize distinct C]/etc/passwdR] files. On the other hand, depending on a single central server does limit the size of an installation.
Another example of the power of centralized file service is the way Plan 9 administers network information. On the central server there is a directory, C]/lib/ndbR], that contains all the information necessary to administer the local Ethernet and other networks. All the machines use the same database to talk to the network; there is no need to manage a distributed naming system or keep parallel files up to date. To install a new machine on the local Ethernet, choose a name and IP address and add these to a single file in C]/lib/ndbR]; all the machines in the installation will be able to talk to it immediately. To start running, plug the machine into the network, turn it on, and use BOOTP and TFTP to load the kernel. All else is automatic.
Finally, the automated dump file system frees all users from the need to maintain their systems, while providing easy access to backup files without tapes, special commands, or the involvement of support staff. It is difficult to overstate the improvement in lifestyle afforded by this service.
Plan 9 runs on a variety of hardware without constraining how to configure an installation. In our laboratory, we chose to use central servers because they amortize costs and administration. A sign that this is a good decision is that our cheap terminals remain comfortable places to work for about five years, much longer than workstations that must provide the complete computing environment. We do, however, upgrade the central machines, so the computation available from even old Plan 9 terminals improves with time. The money saved by avoiding regular upgrades of terminals is instead spent on the newest, fastest multiprocessor servers. We estimate this costs about half the money of networked workstations yet provides general access to more powerful machines.
The Plan 9 C dialect has some minor extensions, described elsewhere [Pike95], and a few major restrictions. The most important restriction is that the compiler demands that all function definitions have ANSI prototypes and all function calls appear in the scope of a prototyped declaration of the function. As a stylistic rule, the prototyped declaration is placed in a header file included by all files that call the function. Each system library has an associated header file, declaring all functions in that library. For example, the standard Plan 9 library is called libc, so all C source files include <libc.h>. These rules guarantee that all functions are called with arguments having the expected types [em] something that was not true with pre-ANSI C programs.
Another restriction is that the C compilers accept only a subset of the preprocessor directives required by ANSI. The main omission is #if, since we believe it is never necessary and often abused. Also, its effect is better achieved by other means. For instance, an #if used to toggle a feature at compile time can be written as a regular if statement, relying on compile-time constant folding and dead code elimination to discard object code.
Conditional compilation, even with #ifdef, is used sparingly in Plan 9. The only architecture-dependent #ifdefs in the system are in low-level routines in the graphics library. Instead, we avoid such dependencies or, when necessary, isolate them in separate source files or libraries. Besides making code hard to read, #ifdefs make it impossible to know what source is compiled into the binary or whether source protected by them will compile or work properly. They make it harder to maintain software.
The standard Plan 9 library overlaps much of ANSI C and POSIX [POSIX], but diverges when appropriate to Plan 9[aq]s goals or implementation. When the semantics of a function change, we also change the name. For instance, instead of UNIX[aq]s creat, Plan 9 has a create function that takes three arguments, the original two plus a third that, like the second argument of open, defines whether the returned file descriptor is to be opened for reading, writing, or both. This design was forced by the way 9P implements creation, but it also simplifies the common use of create to initialize a temporary file.
Another departure from ANSI C is that Plan 9 uses a 16-bit character set called Unicode [ISO10646, Unicode]. Although we stopped short of full internationalization, Plan 9 treats the representation of all major languages uniformly throughout all its software. To simplify the exchange of text between programs, the characters are packed into a byte stream by an encoding we designed, called UTF-8, which is now becoming accepted as a standard [FSSUTF]. It has several attractive properties, including byte-order independence, backwards compatibility with ASCII, and ease of implementation.
There are many problems in adapting existing software to a large character set with an encoding that represents characters with a variable number of bytes. ANSI C addresses some of the issues but falls short of solving them all. It does not pick a character set encoding and does not define all the necessary I/O library routines. Furthermore, the functions it does define have engineering problems. Since the standard left too many problems unsolved, we decided to build our own interface. A separate paper has the details [Pike93].
A small class of Plan 9 programs do not follow the conventions discussed in this section. These are programs imported from and maintained by the UNIX community; tex is a representative example. To avoid reconverting such programs every time a new version is released, we built a porting environment, called the ANSI C/POSIX Environment, or APE [Tric95]. APE comprises separate include files, libraries, and commands, conforming as much as possible to the strict ANSI C and base-level POSIX specifications. To port network-based software such as X Windows, it was necessary to add some extensions to those specifications, such as the BSD networking functions.
To avoid byte order problems, data is communicated between programs as text whenever practical. Sometimes, though, the amount of data is high enough that a binary format is necessary; such data is communicated as a byte stream with a pre-defined encoding for multi-byte values. In the rare cases where a format is complex enough to be defined by a data structure, the structure is never communicated as a unit; instead, it is decomposed into individual fields, encoded as an ordered byte stream, and then reassembled by the recipient. These conventions affect data ranging from kernel or application program state information to object file intermediates generated by the compiler.
Programs, including the kernel, often present their data through a file system interface, an access mechanism that is inherently portable. For example, the system clock is represented by a decimal number in the file /dev/time; the time library function (there is no time system call) reads the file and converts it to binary. Similarly, instead of encoding the state of an application process in a series of flags and bits in private memory, the kernel presents a text string in the file named status in the /proc file system associated with each process. The Plan 9 ps command is trivial: it prints the contents of the desired status files after some minor reformatting; moreover, after
C] import helix /proc R]
Each supported architecture has its own compilers and loader. The C and Alef compilers produce intermediate files that are portably encoded; the contents are unique to the target architecture but the format of the file is independent of compiling processor type. When a compiler for a given architecture is compiled on another type of processor and then used to compile a program there, the intermediate produced on the new architecture is identical to the intermediate produced on the native processor. From the compiler[aq]s point of view, every compilation is a cross-compilation.
Although each architecture[aq]s loader accepts only intermediate files produced by compilers for that architecture, such files could have been generated by a compiler executing on any type of processor. For instance, it is possible to run the MIPS compiler on a 486, then use the MIPS loader on a SPARC to produce a MIPS executable.
Since Plan 9 runs on a variety of architectures, even in a single installation, distinguishing the compilers and intermediate names simplifies multi-architecture development from a single source tree. The compilers and the loader for each architecture are uniquely named; there is no cc command. The names are derived by concatenating a code letter associated with the target architecture with the name of the compiler or loader. For example, the letter [oq]8[aq] is the code letter for Intel x86 processors; the C compiler is named 8c, the Alef compiler 8al, and the loader is called 8l. Similarly, the compiler intermediate files are suffixed .8, not .o.
The Plan 9 build program mk, a relative of make, reads the names of the current and target architectures from environment variables called $cputype and $objtype. By default the current processor is the target, but setting $objtype to the name of another architecture before invoking mk results in a cross-build:
C] % objtype=sparc mk R]
There is a trend in new operating systems to implement two classes of processes: normal UNIX-style processes and light-weight kernel threads. Instead, Plan 9 provides a single class of process but allows fine control of the sharing of a process[aq]s resources such as memory and file descriptors. A single class of process is a feasible approach in Plan 9 because the kernel has an efficient system call interface and cheap process creation and scheduling.
Parallel programs have three basic requirements: management of resources shared between processes, an interface to the scheduler, and fine-grain process synchronization using spin locks. On Plan 9, new processes are created using the rfork system call. Rfork takes a single argument, a bit vector that specifies which of the parent process[aq]s resources should be shared, copied, or created anew in the child. The resources controlled by rfork include the name space, the environment, the file descriptor table, memory segments, and notes (Plan 9[aq]s analog of UNIX signals). One of the bits controls whether the rfork call will create a new process; if the bit is off, the resulting modification to the resources occurs in the process making the call. For example, a process calls rfork(RFNAMEG) to disconnect its name space from its parent[aq]s. Alef uses a fine-grained fork in which all the resources, including memory, are shared between parent and child, analogous to creating a kernel thread in many systems.
An indication that rfork is the right model is the variety of ways it is used. Other than the canonical use in the library routine fork, it is hard to find two calls to rfork with the same bits set; programs use it to create many different forms of sharing and resource allocation. A system with just two types of processes[em]regular processes and threads[em]could not handle this variety.
There are two ways to share memory. First, a flag to rfork causes all the memory segments of the parent to be shared with the child (except the stack, which is forked copy-on-write regardless). Alternatively, a new segment of memory may be attached using the segattach system call; such a segment will always be shared between parent and child.
The rendezvous system call provides a way for processes to synchronize. Alef uses it to implement communication channels, queuing locks, multiple reader/writer locks, and the sleep and wakeup mechanism. Rendezvous takes two arguments, a tag and a value. When a process calls rendezvous with a tag it sleeps until another process presents a matching tag. When a pair of tags match, the values are exchanged between the two processes and both rendezvous calls return. This primitive is sufficient to implement the full set of synchronization routines.
Finally, spin locks are provided by an architecture-dependent library at user level. Most processors provide atomic test and set instructions that can be used to implement locks. A notable exception is the MIPS R3000, so the SGI Power series multiprocessors have special lock hardware on the bus. User processes gain access to the lock hardware by mapping pages of hardware locks into their address space using the segattach system call.
A Plan 9 process in a system call will block regardless of its [oq]weight[aq]. This means that when a program wishes to read from a slow device without blocking the entire calculation, it must fork a process to do the read for it. The solution is to start a satellite process that does the I/O and delivers the answer to the main program through shared memory or perhaps a pipe. This sounds onerous but works easily and efficiently in practice; in fact, most interactive Plan 9 applications, even relatively ordinary ones written in C, such as the text editor Sam [Pike87], run as multiprocess programs.
The kernel support for parallel programming in Plan 9 is a few hundred lines of portable code; a handful of simple primitives enable the problems to be handled cleanly at user level. Although the primitives work fine from C, they are particularly expressive from within Alef. The creation and management of slave I/O processes can be written in a few lines of Alef, providing the foundation for a consistent means of multiplexing data flows between arbitrary processes. Moreover, implementing it in a language rather than in the kernel ensures consistent semantics between all devices and provides a more general multiplexing primitive. Compare this to the UNIX select system call: select applies only to a restricted set of devices, legislates a style of multiprogramming in the kernel, does not extend across networks, is difficult to implement, and is hard to use.
Another reason parallel programming is important in Plan 9 is that multi-threaded user-level file servers are the preferred way to implement services. Examples of such servers include the programming environment Acme [Pike94], the name space exporting tool exportfs [PPTTW93], the HTTP daemon, and the network name servers cs and dns [PrWi93]. Complex applications such as Acme prove that careful operating system support can reduce the difficulty of writing multi-threaded applications without moving threading and synchronization primitives into the kernel.
The mount call attaches a new hierarchy to the existing name space. The bind system call, on the other hand, duplicates some piece of existing name space at another point in the name space. The unmount system call allows components to be removed.
Using either bind or mount, multiple directories may be stacked at a single point in the name space. In Plan 9 terminology, this is a union directory and behaves like the concatenation of the constituent directories. A flag argument to bind and mount specifies the position of a new directory in the union, permitting new elements to be added either at the front or rear of the union or to replace it entirely. When a file lookup is performed in a union directory, each component of the union is searched in turn and the first match taken; likewise, when a union directory is read, the contents of each of the component directories is read in turn. Union directories are one of the most widely used organizational features of the Plan 9 name space. For instance, the directory /bin is built as a union of /$cputype/bin (program binaries), /rc/bin (shell scripts), and perhaps more directories provided by the user. This construction makes the shell $PATH variable unnecessary.
One question raised by union directories is which element of the union receives a newly created file. After several designs, we decided on the following. By default, directories in unions do not accept new files, although the create system call applied to an existing file succeeds normally. When a directory is added to the union, a flag to bind or mount enables create permission (a property of the name space) in that directory. When a file is being created with a new name in a union, it is created in the first directory of the union with create permission; if that creation fails, the entire create fails. This scheme enables the common use of placing a private directory anywhere in a union of public ones, while allowing creation only in the private directory.
By convention, kernel device file systems are bound into the /dev directory, but to bootstrap the name space building process it is necessary to have a notation that permits direct access to the devices without an existing name space. The root directory of the tree served by a device driver can be accessed using the syntax #c, where c is a unique character (typically a letter) identifying the type of the device. Simple device drivers serve a single level directory containing a few files. As an example, each serial port is represented by a data and a control file:
C] % bind -a [aq]#t[aq] /dev % cd /dev % ls -l eia* --rw-rw-rw- t 0 bootes bootes 0 Feb 24 21:14 eia1 --rw-rw-rw- t 0 bootes bootes 0 Feb 24 21:14 eia1ctl --rw-rw-rw- t 0 bootes bootes 0 Feb 24 21:14 eia2 --rw-rw-rw- t 0 bootes bootes 0 Feb 24 21:14 eia2ctl R]
It is the universal use of the 9P protocol that connects Plan 9[aq]s components together to form a distributed system. Rather than inventing a unique protocol for each service such as rlogin, FTP, TFTP, and X windows, Plan 9 implements services in terms of operations on file objects, and then uses a single, well-documented protocol to exchange information between computers. Unlike NFS, 9P treats files as a sequence of bytes rather than blocks. Also unlike NFS, 9P is stateful: clients perform remote procedure calls to establish pointers to objects in the remote file server. These pointers are called file identifiers or fids. All operations on files supply a fid to identify an object in the remote file system.
The 9P protocol defines 17 messages, providing means to authenticate users, navigate fids around a file system hierarchy, copy fids, perform I/O, change file attributes, and create and delete files. Its complete specification is in Section 5 of the Programmer[aq]s Manual [9man]. Here is the procedure to gain access to the name hierarchy supplied by a server. A file server connection is established via a pipe or network connection. An initial session message performs a bilateral authentication between client and server. An attach message then connects a fid suggested by the client to the root of the server file tree. The attach message includes the identity of the user performing the attach; henceforth all fids derived from the root fid will have permissions associated with that user. Multiple users may share the connection, but each must perform an attach to establish his or her identity.
The walk message moves a fid through a single level of the file system hierarchy. The clone message takes an established fid and produces a copy that points to the same file as the original. Its purpose is to enable walking to a file in a directory without losing the fid on the directory. The open message locks a fid to a specific file in the hierarchy, checks access permissions, and prepares the fid for I/O. The read and write messages allow I/O at arbitrary offsets in the file; the maximum size transferred is defined by the protocol. The clunk message indicates the client has no further use for a fid. The remove message behaves like clunk but causes the file associated with the fid to be removed and any associated resources on the server to be deallocated.
9P has two forms: RPC messages sent on a pipe or network connection and a procedural interface within the kernel. Since kernel device drivers are directly addressable, there is no need to pass messages to communicate with them; instead each 9P transaction is implemented by a direct procedure call. For each fid, the kernel maintains a local representation in a data structure called a channel, so all operations on files performed by the kernel involve a channel connected to that fid. The simplest example is a user process[aq]s file descriptors, which are indexes into an array of channels. A table in the kernel provides a list of entry points corresponding one to one with the 9P messages for each device. A system call such as read from the user translates into one or more procedure calls through that table, indexed by the type character stored in the channel: procread, eiaread, etc. Each call takes at least one channel as an argument. A special kernel driver, called the mount driver, translates procedure calls to messages, that is, it converts local procedure calls to remote ones. In effect, this special driver becomes a local proxy for the files served by a remote file server. The channel pointer in the local call is translated to the associated fid in the transmitted message.
The mount driver is the sole RPC mechanism employed by the system. The semantics of the supplied files, rather than the operations performed upon them, create a particular service such as the cpu command. The mount driver demultiplexes protocol messages between clients sharing a communication channel with a file server. For each outgoing RPC message, the mount driver allocates a buffer labeled by a small unique integer, called a tag. The reply to the RPC is labeled with the same tag, which is used by the mount driver to match the reply with the request.
The kernel representation of the name space is called the mount table, which stores a list of bindings between channels. Each entry in the mount table contains a pair of channels: a from channel and a to channel. Every time a walk succeeds in moving a channel to a new location in the name space, the mount table is consulted to see if a [oq]from[aq] channel matches the new name; if so the [oq]to[aq] channel is cloned and substituted for the original. Union directories are implemented by converting the [oq]to[aq] channel into a list of channels: a successful walk to a union directory returns a [oq]to[aq] channel that forms the head of a list of channels, each representing a component directory of the union. If a walk fails to find a file in the first directory of the union, the list is followed, the next component cloned, and walk tried on that directory.
Each file in Plan 9 is uniquely identified by a set of integers: the type of the channel (used as the index of the function call table), the server or device number distinguishing the server from others of the same type (decided locally by the driver), and a qid formed from two 32-bit numbers called path and version. The path is a unique file number assigned by a device driver or file server when a file is created. The version number is updated whenever the file is modified; as described in the next section, it can be used to maintain cache coherency between clients and servers.
The type and device number are analogous to UNIX major and minor device numbers; the qid is analogous to the i-number. The device and type connect the channel to a device driver and the qid identifies the file within that device. If the file recovered from a walk has the same type, device, and qid path as an entry in the mount table, they are the same file and the corresponding substitution from the mount table is made. This is how the name space is implemented.
The version field of the qid is changed whenever the file is modified, which makes it possible to do some weakly coherent forms of caching. The most important is client caching of text and data segments of executable files. When a process execs a program, the file is re-opened and the qid[aq]s version is compared with that in the cache; if they match, the local copy is used. The same method can be used to build a local caching file server. This user-level server interposes on the 9P connection to the remote server and monitors the traffic, copying data to a local disk. When it sees a read of known data, it answers directly, while writes are passed on immediately[em]the cache is write-through[em]to keep the central copy up to date. This is transparent to processes on the terminal and requires no change to 9P; it works well on home machines connected over serial lines. A similar method can be applied to build a general client cache in unused local memory, but this has not been done in Plan 9.
This example illustrates the structure of the TCP device:
C] % ls -lp /net/tcp d-r-xr-xr-x I 0 bootes bootes 0 Feb 23 20:20 0 d-r-xr-xr-x I 0 bootes bootes 0 Feb 23 20:20 1 --rw-rw-rw- I 0 bootes bootes 0 Feb 23 20:20 clone % ls -lp /net/tcp/0 --rw-rw---- I 0 rob bootes 0 Feb 23 20:20 ctl --rw-rw---- I 0 rob bootes 0 Feb 23 20:20 data --rw-rw---- I 0 rob bootes 0 Feb 23 20:20 listen --r--r--r-- I 0 bootes bootes 0 Feb 23 20:20 local --r--r--r-- I 0 bootes bootes 0 Feb 23 20:20 remote --r--r--r-- I 0 bootes bootes 0 Feb 23 20:20 status % R]
A call is initiated by writing a connect message with a network-specific address as its argument; for example, to open a Telnet session (port 23) to a remote machine with IP address 135.104.9.52, the string is:
C] connect 135.104.9.52!23 R]
C] announce 23 R]
A uniform structure for network devices cannot hide all the details of addressing and communication for dissimilar networks. For example, Datakit uses textual, hierarchical addresses unlike IP[aq]s 32-bit addresses, so an application given a control file must still know what network it represents. Rather than make every application know the addressing of every network, Plan 9 hides these details in a connection server, called cs. Cs is a file system mounted in a known place. It supplies a single control file that an application uses to discover how to connect to a host. The application writes the symbolic address and service name for the connection it wishes to make, and reads back the name of the clone file to open and the address to present to it. If there are multiple networks between the machines, cs presents a list of possible networks and addresses to be tried in sequence; it uses heuristics to decide the order. For instance, it presents the highest-bandwidth choice first.
A single library function called dial talks to cs to establish the connection. An application that uses dial needs no changes, not even recompilation, to adapt to new networks; the interface to cs hides the details.
The uniform structure for networks in Plan 9 makes the import command all that is needed to construct gateways.
We designed a new protocol, called IL (Internet Link), to transmit 9P messages over IP. It is a connection-based protocol that provides reliable transmission of sequenced messages between machines. Since a process can have only a single outstanding 9P request, there is no need for flow control in IL. Like TCP, IL has adaptive timeouts: it scales acknowledge and retransmission times to match the network speed. This allows the protocol to perform well on both the Internet and on local Ethernets. Also, IL does no blind retransmission, to avoid adding to the congestion of busy networks. Full details are in another paper [PrWi95].
In Plan 9, the implementation of IL is smaller and faster than TCP. IL is our main Internet transport protocol.
Each Plan 9 user has an associated DES [NBS77] authentication key; the user[aq]s identity is verified by the ability to encrypt and decrypt special messages called challenges. Since knowledge of a user[aq]s key gives access to that user[aq]s resources, the Plan 9 authentication protocols never transmit a message containing a cleartext key.
Authentication is bilateral: at the end of the authentication exchange, each side is convinced of the other[aq]s identity. Every machine begins the exchange with a DES key in memory. In the case of CPU and file servers, the key, user name, and domain name for the server are read from permanent storage, usually non-volatile RAM. In the case of terminals, the key is derived from a password typed by the user at boot time. A special machine, known as the authentication server, maintains a database of keys for all users in its administrative domain and participates in the authentication protocols.
The authentication protocol is as follows: after exchanging challenges, one party contacts the authentication server to create permission-granting tickets encrypted with each party[aq]s secret key and containing a new conversation key. Each party decrypts its own ticket and uses the conversation key to encrypt the other party[aq]s challenge.
This structure is somewhat like Kerberos [MBSS87], but avoids its reliance on synchronized clocks. Also unlike Kerberos, Plan 9 authentication supports a [oq]speaks for[aq] relation [LABW91] that enables one user to have the authority of another; this is how a CPU server runs processes on behalf of its clients.
Plan 9[aq]s authentication structure builds secure services rather than depending on firewalls. Whereas firewalls require special code for every service penetrating the wall, the Plan 9 approach permits authentication to be done in a single place[em]9P[em]for all services. For example, the cpu command works securely across the Internet.
CPU servers have an equivalent user name that allows administrative access to resources on that server such as the control files of user processes. Such permission is necessary, for example, to kill rogue processes, but does not extend beyond that server. On the other hand, by means of a key held in protected non-volatile RAM, the identity of the administrative user is proven to the authentication server. This allows the CPU server to authenticate remote users, both for access to the server itself and when the CPU server is acting as a proxy on their behalf.
Finally, a special user called none has no password and is always allowed to connect; anyone may claim to be none. None has restricted permissions; for example, it is not allowed to examine dump files and can read only world-readable files.
The idea behind none is analogous to the anonymous user in FTP services. On Plan 9, guest FTP servers are further confined within a special restricted name space. It disconnects guest users from system programs, such as the contents of /bin, but makes it possible to make local files available to guests by binding them explicitly into the space. A restricted name space is more secure than the usual technique of exporting an ad hoc directory tree; the result is a kind of cage around untrusted users.
The speaks for relation [LABW91] is kept in a table on the authentication server. To simplify the management of users computing in different authentication domains, it also contains mappings between user names in different domains, for example saying that user rtm in one domain is the same person as user rtmorris in another.
C] pjw:pjw: sys::pjw,ken,philw,presotto R]
Regular files are owned by the user that creates them. The group name is inherited from the directory holding the new file. Device files are treated specially: the kernel may arrange the ownership and permissions of a file appropriate to the user accessing the file.
A good example of the generality this offers is process files, which are owned and read-protected by the owner of the process. If the owner wants to let someone else access the memory of a process, for example to let the author of a program debug a broken image, the standard chmod command applied to the process files does the job.
Another unusual application of file permissions is the dump file system, which is not only served by the same file server as the original data, but represented by the same user database. Files in the dump are therefore given identical protection as files in the regular file system; if a file is owned by pjw and read-protected, once it is in the dump file system it is still owned by pjw and read-protected. Also, since the dump file system is immutable, the file cannot be changed; it is read-protected forever. Drawbacks are that if the file is readable but should have been read-protected, it is readable forever, and that user names are hard to re-use.
The program tests the time to do a context switch (rendezvous on Plan 9, blockproc on IRIX); a trivial system call (rfork(0) and nap(0)); and lightweight fork (rfork(RFPROC) and sproc(PR_SFDS|PR_SADDR)). It also measures the time to send a byte on a pipe from one process to another and the throughput on a pipe between two processes. The results appear in Table 1.
Table 1. Performance comparison.
Although the Plan 9 times are not spectacular, they show that the kernel is competitive with commercial systems.
Plan 9[aq]s focus on files and naming is central to its expressiveness. Particularly in distributed computing, the way things are named has profound influence on the system [Nee89]. The combination of local name spaces and global conventions to interconnect networked resources avoids the difficulty of maintaining a global uniform name space, while naming everything like a file makes the system easy to understand, even for novices. Consider the dump file system, which is trivial to use for anyone familiar with hierarchical file systems. At a deeper level, building all the resources above a single uniform interface makes interoperability easy. Once a resource exports a 9P interface, it can combine transparently with any other part of the system to build unusual applications; the details are hidden. This may sound object-oriented, but there are distinctions. First, 9P defines a fixed set of [oq]methods[aq]; it is not an extensible protocol. More important, files are well-defined and well-understood and come prepackaged with familiar methods of access, protection, naming, and networking. Objects, despite their generality, do not come with these attributes defined. By reducing [oq]object[aq] to [oq]file[aq], Plan 9 gets some technology for free.
Nonetheless, it is possible to push the idea of file-based computing too far. Converting every resource in the system into a file system is a kind of metaphor, and metaphors can be abused. A good example of restraint is /proc, which is only a view of a process, not a representation. To run processes, the usual fork and exec calls are still necessary, rather than doing something like
C] cp /bin/date /proc/clone/mem R]
What would we do differently next time? Some elements of the implementation are unsatisfactory. Using streams to implement network interfaces in the kernel allows protocols to be connected together dynamically, such as to attach the same TTY driver to TCP, URP, and IL connections, but Plan 9 makes no use of this configurability. (It was exploited, however, in the research UNIX system for which streams were invented.) Replacing streams by static I/O queues would simplify the code and make it faster.
Although the main Plan 9 kernel is portable across many machines, the file server is implemented separately. This has caused several problems: drivers that must be written twice, bugs that must be fixed twice, and weaker portability of the file system code. The solution is easy: the file server kernel should be maintained as a variant of the regular operating system, with no user processes and special compiled-in kernel processes to implement file service. Another improvement to the file system would be a change of internal structure. The WORM jukebox is the least reliable piece of the hardware, but because it holds the metadata of the file system, it must be present in order to serve files. The system could be restructured so the WORM is a backup device only, with the file system proper residing on magnetic disks. This would require no change to the external interface.
Although Plan 9 has per-process name spaces, it has no mechanism to give the description of a process[aq]s name space to another process except by direct inheritance. The cpu command, for example, cannot in general reproduce the terminal[aq]s name space; it can only re-interpret the user[aq]s login profile and make substitutions for things like the name of the binary directory to load. This misses any local modifications made before running cpu. It should instead be possible to capture the terminal[aq]s name space and transmit its description to a remote process.
Despite these problems, Plan 9 works well. It has matured into the system that supports our research, rather than being the subject of the research itself. Experimental new work includes developing interfaces to faster networks, file caching in the client kernel, encapsulating and exporting name spaces, and the ability to re-establish the client state after a server crash. Attention is now focusing on using the system to build distributed applications.
One reason for Plan 9[aq]s success is that we use it for our daily work, not just as a research tool. Active use forces us to address shortcomings as they arise and to adapt the system to solve our problems. Through this process, Plan 9 has become a comfortable, productive programming environment, as well as a vehicle for further systems research.
[ANSIC] American National Standard for Information Systems - Programming Language C, American National Standards Institute, Inc., New York, 1990.
[Duff90] Tom Duff, [oq][oq]Rc - A Shell for Plan 9 and UNIX systems[aq][aq], Proc. of the Summer 1990 UKUUG Conf., London, July, 1990, pp. 21-33, reprinted, in a different form, in this volume.
[Fra80] A.G. Fraser, [oq][oq]Datakit - A Modular Network for Synchronous and Asynchronous Traffic[aq][aq], Proc. Int. Conf. on Commun., June 1980, Boston, MA.
[FSSUTF] File System Safe UCS Transformation Format (FSS-UTF), X/Open Preliminary Specification, 1993. ISO designation is ISO/IEC JTC1/SC2/WG2 N 1036, dated 1994-08-01.
[ISO10646] ISO/IEC DIS 10646-1:1993 Information technology - Universal Multiple-Octet Coded Character Set (UCS) [em] Part 1: Architecture and Basic Multilingual Plane.
[Kill84] T.J. Killian, [oq][oq]Processes as Files[aq][aq], USENIX Summer 1984 Conf. Proc., June 1984, Salt Lake City, UT.
[LABW91] Butler Lampson, Mart['i]n Abadi, Michael Burrows, and Edward Wobber, [oq][oq]Authentication in Distributed Systems: Theory and Practice[aq][aq], Proc. 13th ACM Symp. on Op. Sys. Princ., Asilomar, 1991, pp. 165-182.
[MBSS87] S. P. Miller, B. C. Neumann, J. I. Schiller, and J. H. Saltzer, [oq][oq]Kerberos Authentication and Authorization System[aq][aq], Massachusetts Institute of Technology, 1987.
[NBS77] National Bureau of Standards (U.S.), Federal Information Processing Standard 46, National Technical Information Service, Springfield, VA, 1977.
[Nee89] R. Needham, [oq][oq]Names[aq][aq], in Distributed systems, S. Mullender, ed., Addison Wesley, 1989
[NeHe82] R.M. Needham and A.J. Herbert, The Cambridge Distributed Computing System, Addison-Wesley, London, 1982
[Neu92] B. Clifford Neuman, [oq][oq]The Prospero File System[aq][aq], USENIX File Systems Workshop Proc., Ann Arbor, 1992, pp. 13-28.
[OCDNW88] John Ousterhout, Andrew Cherenson, Fred Douglis, Mike Nelson, and Brent Welch, [oq][oq]The Sprite Network Operating System[aq][aq], IEEE Computer, 21(2), 23-38, Feb. 1988.
[Pike87] Rob Pike, [oq][oq]The Text Editor sam[aq][aq], Software - Practice and Experience, Nov 1987, 17(11), pp. 813-845; reprinted in this volume.
[Pike91] Rob Pike, [oq][oq]8[12], the Plan 9 Window System[aq][aq], USENIX Summer Conf. Proc., Nashville, June, 1991, pp. 257-265, reprinted in this volume.
[Pike93] Rob Pike and Ken Thompson, [oq][oq]Hello World or [*K][*a][*l][*y][*m][u03AD][*r][*a] [*k][u03CC][*s][*m][*e] or [u3053][u3093][u306B][u3061][u306F] [u4E16][u754C][aq][aq], USENIX Winter Conf. Proc., San Diego, 1993, pp. 43-50, reprinted in this volume.
[Pike94] Rob Pike, [oq][oq]Acme: A User Interface for Programmers[aq][aq], USENIX Proc. of the Winter 1994 Conf., San Francisco, CA,
[Pike95] Rob Pike, [oq][oq]How to Use the Plan 9 C Compiler[aq][aq], Plan 9 Programmer[aq]s Manual, Volume 2, AT&T Bell Laboratories, Murray Hill, NJ, 1995.
[POSIX] Information Technology[em]Portable Operating System Interface (POSIX) Part 1: System Application Program Interface (API) [C Language], IEEE, New York, 1990.
[PPTTW93] Rob Pike, Dave Presotto, Ken Thompson, Howard Trickey, and Phil Winterbottom, [oq][oq]The Use of Name Spaces in Plan 9[aq][aq], Op. Sys. Rev., Vol. 27, No. 2, April 1993, pp. 72-76, reprinted in this volume.
[Presotto] Dave Presotto, [oq][oq]Multiprocessor Streams for Plan 9[aq][aq], UKUUG Summer 1990 Conf. Proc., July 1990, pp. 11-19.
[PrWi93] Dave Presotto and Phil Winterbottom, [oq][oq]The Organization of Networks in Plan 9[aq][aq], USENIX Proc. of the Winter 1993 Conf., San Diego, CA, pp. 43-50, reprinted in this volume.
[PrWi95] Dave Presotto and Phil Winterbottom, [oq][oq]The IL Protocol[aq][aq], Plan 9 Programmer[aq]s Manual, Volume 2, AT&T Bell Laboratories, Murray Hill, NJ, 1995.
[RFC768] J. Postel, RFC768, User Datagram Protocol, DARPA Internet Program Protocol Specification, August 1980.
[RFC793] RFC793, Transmission Control Protocol, DARPA Internet Program Protocol Specification, September 1981.
[Rao91] Herman Chung-Hwa Rao, The Jade File System, (Ph. D. Dissertation), Dept. of Comp. Sci, University of Arizona, TR 91-18.
[Rit84] D.M. Ritchie, [oq][oq]A Stream Input-Output System[aq][aq], AT&T Bell Laboratories Technical Journal, 63(8), October, 1984.
[Tric95] Howard Trickey, [oq][oq]APE [em] The ANSI/POSIX Environment[aq][aq], Plan 9 Programmer[aq]s Manual, Volume 2, AT&T Bell Laboratories, Murray Hill, NJ, 1995.
[Unicode] The Unicode Standard, Worldwide Character Encoding, Version 1.0, Volume 1, The Unicode Consortium, Addison Wesley, New York, 1991.
[UNIX85] UNIX Time-Sharing System Programmer[aq]s Manual, Research Version, Eighth Edition, Volume 1. AT&T Bell Laboratories, Murray Hill, NJ, 1985.
[Welc94] Brent Welch, [oq][oq]A Comparison of Three Distributed File System Architectures: Vnode, Sprite, and Plan 9[aq][aq], Computing Systems, 7(2), pp. 175-199, Spring, 1994.
[Wint95] Phil Winterbottom, [oq][oq]Alef Language Reference Manual[aq][aq], Plan 9 Programmer[aq]s Manual, Volume 2, AT&T Bell Laboratories, Murray Hill, NJ, 1995.
| /man/10/plan9(10) | Rev: Wed Feb 22 04:14:06 GMT 2023 |