Yesterday I got to thinking about the file system, like you do. I’m not talking about any kind of complicated, systems engineering subject here. I’m talking about basic terminology, like what the relationship is between a disk, a partition, a volume, a file system, and the act of mounting.
Now you might be thinking, shouldn’t you know this stuff? You’re a programmer, by gum, and you wrote a book on Mac OS X. Thing is, it’s possible to know something without really knowing it. You know enough to get by, possibly via several conflicting models, but when you actually sit down and try to map everything out in your head, you find out you’re not really sure you know what you know.
For example, it’s common to say one mounts a disk, yet everyday experience shows that to be inaccurate. Disk Utility, to say nothing of the Finder, refers to volumes, of which there may be several on a single disk, due to partitioning. So, one might get the impression that a volume is the same thing as a partition.
However, disk images are also mounted, meaning they must have volumes, but clearly they don’t have partitions, do they? The man pages for mount (either the UNIX command or the BSD function) refer to “file systems” as their target, as in “the system maintains a list of currently mounted file systems.”
So wait, a disk contains one or more partitions, which may or may not also be volumes, which may or may not be file systems?
See one, do one, teach one
That phrase, which I learned when I was a pre-med, are the three steps to mastery. It’s pretty intuitive that in order to know something you have to learn it, be it from a book, or by seeing someone else do it. Experience teaches us that in order to lock that knowledge in, you have to practice doing something over and over again, until you’re good at it. Someone who’s been programming for a year is going to know more than someone who’s been reading about programming for a year, just because they’re going to remember more.
It’s the teaching requirement that’s new. Why should teaching be requisite to learning? Having to teach something means having to come up with a unified model of that thing. Reviewing that thing in preparation of telling others about it may reveal gaps in knowledge or previously unappreciated facets. Moreover, saying something out loud, then hearing yourself say it, stimulates different parts of the brain, further committing it to long-term memory.
But there’s something else in here, something I first learned about in programming, but that I’ve only recently come to understand: the value of assertions.
In programming, an assertion is a statement that is tested at runtime and that, should it fail, will cause an exception. In other words, for no good reason, you tell the compiler, “I assert that the following is true, and if I’m wrong, crash.” When I first learned about assertions I thought, what a terrible idea. Why would you want to do something that could cause your program to crash?
The answer, of course, is that it’s better to crash now while there’s still time to examine why your assertion failed, than it is to continuing being wrong, which may lead to worse complications further down the road. In real life terms, one might say, it’s better to be embarrassed than to be wrong.
Which is not to say it’s any fun to be embarrassed. It’s the fear of embarrassment that prevents people from making assertions in the first place. Nobody wants to admit, hey, I’m not entirely sure I know what a volume is. I think maybe it’s the same thing as a partition? The funny thing is, you’d be surprised how many people don’t really know either, including some people who really think they do.
For example, yesterday I twittered that I was annoyed by people who dropped trivia that turned out to be wrong. The example I gave was “the UNIX command sudo stands for “superuser do.” A lot of people responded, “doesn’t it? Wikipedia says it does.” It turns out, I’m not the only one susceptible to not knowing things as well as I thought I did.
OK, J.J. Abrams, how about some answers?
The UNIX command sudo stands for “su do,”* because it is a safer replacement for the older command su, which itself stands for “substitute user.”* The confusion arises because most of the time when you are substituting a user, you’re doing so because you want to escalate privileges to root (the superuser) in order to accomplish some system-level task. Indeed, the default user for both su and sudo is root. However, it’s possible, by using an argument with su or the -u flag with sudo, to act as any user.
For my file system answers, I turned to Amit Singh’s excellent volume, “Mac OS X Internals.”
-
A disk may be logically divided into one or more partitions, which are sets of contiguous blocks that can be thought of as subdisks. A partition may contain an instance of a file system. Such an instance — a volume — is essentially a structured file residing on the partition.
A-ha! So a disk has partitions, and a partition may or may not contain an instance of a file system. (The alternative is for the partition to merely contain free space.) It is the instance of that file system that we call a “volume” in Macintosh land, and simply a “file system” in UNIX land.
To look at it another way, a volume is a file system object, which is to say, software. It has an optional, one-to-one relationship with a partition, which is hardware. Partitions may exist without containing volumes, and volumes may exist in other than the confines of a partition. Brilliant!
On RAIDs and ZFS
The situation is further complicated by RAIDs, which are arrangements of multiple disks, and ZFS, which is a a new file system that, among other things, spans multiple disks. I had actually had this in mind from the very beginning, but left it as an exercise for the reader, if you will. However, it wasn’t long before someone brought it up with Alison, so I figured I should go ahead and address it.
It helps to understand the difference between a logical device and a physical device. A device is a piece of hardware, such as a disk. In order for your computer to communicate with a device, it uses a very low-level piece of software called a driver. The driver provides an abstraction of the device to higher level software.
So, for any device, there exists two aspects. There’s the physical aspect, which is what we, as humans, can see. Then there’s the logical aspect, which is what the software sees. The difference between a partition and a volume is exactly such a distinction.
A partition is a physical phenomenon (even if its boundaries are defined in on-disk software). It literally consists of blocks of (say) magnetized particles on spinning platters. However, all your computer sees are the ones and zeroes that are neatly addressed by the file system and are accessible via the mounting process.
Well, what if the driver or some other piece of software were to lie to the computer? What if the driver were to take several physical disks with their myriad partitions and present it to the computer as if it were one disk? Suddenly the logical device and the physical device would be very different.
This is exactly what happens with a RAID, thanks to a sneaky piece of software called a RAID controller, which can either be implemented in the disk’s own software (a hardware controller) or on the computer itself (a software controller).
Thus, a RAID is a single logical disk, which may be split into one or more logical partitions, which may in turn contain file system objects, also known as volumes. The volumes’ bits may be scattered across the physical globe, but as long as the logical partitions remain intact, all is well.
For ZFS, the relationship between disks and volumes is further blurred, because the file system object appears above the device controller, rather than below it. You can think of ZFS as being like a software RAID controller, except instead of presenting a single logical disk, it presents a single logical volume.
In other words, the relationship between a disk partition and a file system object is irrelevant. As long as the volume is visible, it will be mountable. The presence of any number of disks, partitions, or whatever are no more an issue than the lack of physical hardware seen in a disk image.
So what’s the take away from all this?
If you are a person who loves knowledge, recognize when you may not be 100% certain of something and, possibly in the confines of a safe place, make an assertion. If your assertion fails, find out why, and fix it. If nothing else, when you think you know something pretty well, teach it to someone else. Write a blog entry explaining the whole thing, or make a presentation at your local MUG or developer group.
If you’re lucky, someone will raise their hand and you’ll both end up a little bit smarter.
Addenda 
It occurred to me after I posted this that I had basically re-invented peer review, one the foundations of something called “science.” Maybe you’ve heard of it? So be it — let us, as computer scientists, submit ourselves to peer review just like those hoity-toity real scientists. Hell, let’s just do that as people. Anyway, some comments:
uv
Tip of the iceberg - what about RAIDs? What about partitions spanning multiple disks? Why would you want to have multiple partitions on the same disk?
Truth is, filesystems have a rather old design which includes a lot of overlapping terms. Check out Sun’s ZFS for a new (and final?) look at this old problem:
http://opensolaris.org/os/community/zfs/demos/basics/
Tip of the iceberg - what about RAIDs? What about partitions spanning multiple disks? Why would you want to have multiple partitions on the same disk?
Truth is, filesystems have a rather old design which includes a lot of overlapping terms. Check out Sun’s ZFS for a new (and final?) look at this old problem:
http://opensolaris.org/os/community/zfs/demos/basics/
Mike Lee
That’s a good enough question to warrant an edit. Check out the new paragraphs I’ve added about RAIDs and ZFS.
That’s a good enough question to warrant an edit. Check out the new paragraphs I’ve added about RAIDs and ZFS.
Stripes
You said a partition can have a filesystem or free space. On other Unixish systems (say FreeBSD) you could have a swap partition (OS X uses swap files inside a regular filesystem).
Swap partitons are somewhat faster to access then swap files, and in theory have fewer issues with user-level filesystems (because you will never have a swap file on the user-level filesystem, so you won’t swap part of the file-system out and then need that very part to get itself back).
Sometime partitons are used directly by applications (I think Oracle and Sybase can directly use them to store tables and transaction logs and such). Again you get a slight speed bump.
However in all those cases you have “more crap” to manage, and it is very seldom worth it (who wants to figure out how much VM they will _ever_ use when they first set up their system?)
You said a partition can have a filesystem or free space. On other Unixish systems (say FreeBSD) you could have a swap partition (OS X uses swap files inside a regular filesystem).
Swap partitons are somewhat faster to access then swap files, and in theory have fewer issues with user-level filesystems (because you will never have a swap file on the user-level filesystem, so you won’t swap part of the file-system out and then need that very part to get itself back).
Sometime partitons are used directly by applications (I think Oracle and Sybase can directly use them to store tables and transaction logs and such). Again you get a slight speed bump.
However in all those cases you have “more crap” to manage, and it is very seldom worth it (who wants to figure out how much VM they will _ever_ use when they first set up their system?)
Mike Lee
True story, though I would argue that those examples are not in opposition to the idea that a partition without a file system object is free, assuming you understand “free” to mean “free to use for other things, like swap space, direct access, and so forth.”
True story, though I would argue that those examples are not in opposition to the idea that a partition without a file system object is free, assuming you understand “free” to mean “free to use for other things, like swap space, direct access, and so forth.”
The day after Christmas a tree appeared outside an apartment building. A few days ago a second tree appeared. Something tells me there will soon be many, many more trees.
Tuesday, January 8, 2008
Mounting
