Live from Austin!

One of the things I found confusing about bash was its startup scripts: there were so many of them. Eventually I snapped and sat down with a terminal and the man pages, and worked out how it actually behaves. Here’s a summary.

On startup, bash executes any script labeled A in the table above, followed by the first script B it finds. On exit, it executes any script labeled C above.

Let’s look at the column headings in a little more detail.

• An interactive login shell is a shell that you are typing into, that is the first such shell you execute on the machine. Typically you will have had to log in immediately before the shell starts. For example, when you SSH to a remote system and type commands to that system, you are typing into an interactive login shell.

• An interactive non-login shell is a new shell started once you have already logged in; one which doesn’t require that you log in again.For example, if you open a new terminal window in your graphical user interface and get a shell prompt, that’s an interactive non-login shell. Another example of an interactive non-login shell would be a sub-shell started from inside a text editor; for example, typing :sh in vi.
• A non-interactive shell is a shell which doesn’t prompt you; it just runs a program and then exits. The most common example of this is any program written in shell script, such as a configure script, a startup script in /etc/init.d, or any other file marked as executable that has #!/bin/bash on the first line.
• A remote shell is a shell started by a program such as SSH or rsh in order to run a command on a remote machine.For example, the rsync and scp commands use SSH remote shells in order to copy files between machines.

So looking at the second column, an interactive login shell will execute /etc/profile always. It then looks for ~/.bash_profile, ~/.bash_login, and ~/.profile in turn, and executes the first of those it finds. On logout, it executes ~/.bash_logout.

The /etc/bash.bashrc A† item is special; whether bash searches for it is dependent on a compile-time option.

BASH_ENV is an environment variable which allows you to make non-interactive non-login shells (such as shell scripts) execute a startup script. Set BASH_ENV to the filename of a script and then invoke a sub-shell, and the script will be executed when the sub-shell starts up.

Problems with bash’s behavior

Given the above table, the short summary is:

• If you want something executed only when you first log in, put it in ~/.bash_profile
• If you want something executed only for additional shells (such as OS X terminal windows and xterms), put it in ~/.bashrc

But there are a couple of problems with this arrangement, problems which suggest that bash’s startup behavior wasn’t really thought out with users in mind.

Firstly, if you are anything like me, most of the things you want to put in shell startup scripts are things you always want executed. Command aliases, for example; or environment variables that tell pieces of software where to find their bits (JAVA_HOME, ECLIPSE_HOME).

You could put those in both .bashrc and .bash_login, but that represents a maintenance problem: if you change something, you have to remember to change it in both places. So, you might set up a third file for global stuff, and use the shell command source to read it in from both .bashrc and .bash_login. I’ve seen some Linux distributions set this up as the default. I don’t like it, however, because it means you now have 3 different startup files floating around, and when you want to change something you have to remember which file it’s in (or sit and work it out).

The second issue with bash startup scripts is that the distinction between login shell and non-login shell isn’t a very useful one these days. Most of us use graphical user interfaces, so we never see a login shell on the machine we’re using. (For example, any terminal window you open on OS X is a non-login one.) Even when I use SSH to shell into a remote system, I don’t generally want that first login to behave differently to any other shell I start (such as shells inside screen).

What I do care about, on the other hand, is whether the shell is interactive. I don’t want my reminder program printing stuff when rsync is trying to connect and transfer files. I don’t want all my custom commands and aliases getting in the way when running scripts to configure or build software. And I don’t want to slow things down loading cdargs unless I’m actually going to be maneuvering around the directory structure by typing.

So what I want is to have a single customization script, and be able to split it into stuff that is always run, and stuff that is only run when I’m using the shell session interactively. Here’s how to do that.

Simple all-purpose bash initialization script

Start off by moving all your current bash startup scripts into a temporary directory, so you have a clean slate. Then, create a skeleton ~/.bashrc that looks like this:

### Start of universal section ###
# Commands in this section will be executed by both interactive and
# non-interactive shells.
# Commands here must produce no output, or they will break commands
# like scp and rsync.

### End of universal section ###
[ -z "$PS1" ] && return
### Start of interactive section ###
# Commands in this section will be executed only by interactive shells.

### End of interactive section ###

Next, cd ~ if you’re not already in your home directory, then ln -s .bashrc .bash_login

Now you have a single customization file for all your shell sessions, called ~/.bashrc. You can copy in each command from your old customization files, placing them in the appropriate section according to whether you need them all the time, or just in shells that you’re typing in to.

If you really care about login shells

If for some reason you do want to have login shells behave differently from non-login, that’s pretty simple too. Instead of the ln -s command above, create the following ~/.bash_login file:

if [ -f ~/.bashrc ]; then
  source ~/.bashrc
fi
# Commands for login shells only go under here

Now you have two places customization commands may be placed, but you get the option of having login-specific stuff.

Dealing with multiple systems

Another trick I use is to examine the host name of the machine. This lets me use the same .bashrc everywhere; my Mac’s .bashrc is the same as the one I use on my Linux box and the System z mainframe at work. Here’s the code:

if [ "$HOSTNAME" = "T41p" ]; then
  # Customizations specific to the ThinkPad laptop go in here
fi

You can use code like this in either the interactive or non-interactive section of the .bashrc above.

One feature the Unix shell offers is customizing prompts. Most ‘power users’ make use of the feature, and it is indeed very handy. However, it’s easy to go completely overboard and end up with a prompt like this:

[21:52:15] [fred@webhost:/var/log/apache] $

The problem with a long prompt is that you quickly hit the right hand edge of the screen and your command starts wrapping. If you use KDE, however, there’s a better way.

Konsole

The xterm program introduced an escape sequence to set the window title. That can help a bit, because now you can put some of the boring info up in the window title, and reserve the prompt for path information.

I use KDE, and the Konsole terminal program in KDE goes further than xterm. As well as letting you change the window title, you can also have multiple terminal sessions in tabs (like Firefox web page tabs), and chance their titles too. The purpose of this posting is to explain the extra functionality in Konsole, and how to make use of it with the bash shell (default in Linux).

Continue reading »

Hi, I’m mathew, and this is my web site. Jakob Nielsen believes that omitting a photo of the author is one of the top 10 mistakes in weblog usability, so who am I to argue?

I started using the Internet around 1987; I remember Jakob’s Hypercard stack, as it was one of the first cool things I downloaded. I was introduced to Unix the following year, accidentally typed rn instead of rm one day (true story), and the rest is history.

I’ve been doing my best to gather together the worthwhile content I’ve written since then. It’s an ongoing process, but the archives genuinely go back to 1988. Of course, what I consider worthwhile may look like crap to you, but it’s all categorized and searchable so hopefully you can find something of interest.

Over the years I’ve done all kinds of work, most of it involving computers in some way—telephone technical support, data recovery, system administration, a bit of sales and marketing, application development, hardware maintenance, networking, web design, and so on. I’ve written code in well over a dozen different programming languages. I’m something of a generalist, a term I borrow from Ted Nelson, inventor of hypertext. His ideas inspired my choice of career—I built a primitive network hypertext system around 1985, wrote a browser in 1989, and wrote my first web page back in the days of HTML 1.0. I was rather startled when the rest of the world suddenly took an interest.

I currently live in Austin, Texas. Since being opinionated on the web hasn’t led to fame and riches, I work for IBM as a web architect.

Outside of computers, I’m interested in electronic music, photography, politics, design, video, and small fluffy animals. I also find physics and mathematics very interesting, but because my knowledge is broad rather than deep I tend to get lost soon after integrals get involved.

Consider the following entries in auth.log:

Oct 30 01:55:47 T41p sshd[15554]: (pam_unix) session opened for user paul by (uid=0) Oct 30 02:05:48 T41p sshd[15554]: (pam_unix) session closed for user paul

Quick quiz: How long was paul logged on for?

If you answered “about 10 minutes”, you’re wrong. In fact, the correct answer is “either 10 minutes or 1 hour 10 minutes, there’s no way to tell”.

The problem is that BSD syslog writes its timestamps modified by whatever the local time zone offset is; but unfortunately, it doesn’t bother to record the offset anywhere. At the end of October, the time zone offset changes for most US systems, as daylight saving time ends.

In other words: There are going to be two 01:55s on October 30th, because at 2am the time zone switches and we go back to 01:00. In the example above, we don’t know which 01:55 was the one paul logged in at, so we can’t tell how long he was logged in for.

In the real world, there will often be extra information that will allow us to work out what time the syslog entry must have been written. For example:

Oct 30 01:55:47 T41p sshd[15554]: (pam_unix) session opened for user paul by (uid=0) Oct 30 01:30:47 T41p sshd[15554]: (pam_unix) session opened for user mike by (uid=0) Oct 30 02:05:48 T41p sshd[15554]: (pam_unix) session closed for user paul

Because there are enough events in this example, we can see the discontinuity in the log, and deduce that the first entry must have been written at the earlier 01:55.

Still, that’s hardly an acceptable approach, especially given that there are many log files where events are expected to occur less often than once an hour. (e.g. failed su attempts!)

Given that syslog is one of the major security tools on every single Unix box worldwide, I find it remarkable that this problem has either been overlooked until now, or has been deemed too unimportant to fix.

Of course, you can set up a central syslog server, and run it under UTC. End of problem, right? Well, not quite–because you still need to worry about the time zone configuration of the clients.

I’ve taken a look at the various syslog implementations out there for Linux, and it appears that every single one of them is broken by default; however, there are three that allow you to configure them to always record the exact time of events.

1. multilog records timestamps as 64 bit hexadecimal TAI atomic clock timestamps. Admirably precise, but a pain in the ass for humans. Also, it requires that you drink the djb daemontools Kool-Aid, and I’m not ready for that.

2. rsyslog is a fork of sysklogd. Amongst other things, it adds support for RFC 3339 timestamps, i.e. timestamps in a useful format. Unfortunately, there’s no Debian package yet.

3. syslog-ng is an allegedly very popular replacement for sysklogd. It is possible to configure it to generate useful timestamps by replacing all the file directives in syslog-ng.conf. You’ll need to change

   { file( ... ); };

   to something like

   { file( ... template("$ISODATE $FACILITY@$HOST $PRIORITY $MSG\n")); };

   everywhere. A pain, but you only have to do it once. Supposedly syslog-ng version 2 has a template system for doing this, but it’s not stable yet (i.e. it’s not even in debian-unstable).

So, there we have it. If you run a Unix box with the time zone set to something other than UTC, chances are it is throwing away information and not always recording the exact time of log events. Take a look and see. If it is, you now know how to fix it.

My web hosting provider exploded. The company who supposedly bought their customer lists has failed to get things going after a week or two. So, I need a new web host.

Requirements:

• Linux or UNIX based
• SSH access and rsync for uploading my site
• Low monthly fees
• No price gouging for extra bandwidth
• Low or zero setup fee
• One domain, at least 3 subdomains
• At least 2 POP3/IMAP mailboxes
• A reasonable amount of space (50 MB or more)
• SpamAssassin

Nice-to-have features:

• Usenet/NNTP access and server
• Jabber server
• Logs and stats
• Search engine support for sites

I don’t need PHP, JSP, ASP, SSI, SQL, CGI, … Just static web pages, cheap. Also:

• I don’t need DNS hosting or domain registration, just the servers to point my existing domain at and a working MX or two.
• High uptime isn’t all that important; the odd day or so of downtime is fine.
• Phone support isn’t important.
• Technical hand-holding won’t be needed, obviously.

I’ve found:

• WebIntellects (attraction: unmetered bandwidth)
• DreamHost (attraction: Samba, WebDAV, streaming QuickTime)
• 123CheapHosting (attraction: cheap)
• Micfo.com (generous allocations yet cheap)

Frankly, I’d be tempted to do it myself if we weren’t planning on moving in the near future…

The SCO-IBM-Microsoft-Linux lawsuit just keeps getting weirder.

Now Novell have pointed out that they didn’t actually sell the UNIX System V intellectual property to SCO in the first place. They have also asked SCO to document its claim that Novell’s UNIX code has been stolen and incorporated into Linux.

Meanwhile, Microsoft have categorically denied that they paid licensing fees to SCO just to help pay for the lawsuit. Which means that in fact, they paid to license code from SCO that SCO doesn’t actually own. So, will they get angry at SCO, or will they do nothing and hence tacitly admit that they’re backing the lawsuit?

Oh, given that SCO’s shares went up when Microsoft thought it licensed UNIX from them, the outcome of their shareholder earnings call this morning should be interesting…

Recently SCO launched a lawsuit alleging that Linux contained SCO UNIX source code, illegally copied. SCO is pointing fingers at IBM, suggesting that hackers working on AIX lifted code and added it to Linux. In addition, SCO has written to 1,500 companies telling them they might be next if they don’t stop using Linux.

The bizarre thing about the lawsuit is that SCO themselves are Linux vendors. For years they have been shipping Caldera OpenLinux, which contains a number of modifications to the standard Linux kernel. Clearly if the kernel does contain SCO code, SCO themselves have shipped that code under the GPL. Since code shipped under the GPL cannot be pulled out of free distribution, and SCO clearly had the legal right to put their code under the GPL, it appears to me that Linux should be pretty safe.

It’s going to be interesting to see how they play this one in court… “Yes, your honor, we did do a lot of work on the Linux kernel code, but we had no idea the code we were working on was ours when we shipped it. I mean, how were we supposed to recognize it?”

Last week SCO belatedly realized that they weren’t exactly doing themselves any favors, and yanked OpenLinux from sale.

Now there’s been another interesting development. Microsoft has licensed SCO UNIX. This has provided them with an opportunity to take a swipe at Linux by talking about their “ongoing commitment to respecting intellectual property”. (Yeah, tell that to Stac Electronics and the inventor of the wheel mouse.)

Nobody knows whether SCO threatened Microsoft with a lawsuit, or whether Microsoft actually has any products that might use UNIX source code. What is pretty clear is that SCO have had dismal financial results for the last six years, and without a sudden injection of unknown amounts of cash from Microsoft, they might not have had the money to fight their lawsuit.

Is this Microsoft shoring up SCO after the fact, or is this payback for a plan suggested to SCO by Microsoft in the first place? Take your pick, depending on how smart and devious you think Steve Ballmer is. It’s quite possible that SCO came up with the idea themselves; I’m sure they’ve been very unhappy with the way IBM seems to have placed Linux, rather than AIX, at the center of its plans for the future.

Though the lawsuit seems to me to be pretty meritless, SCO has timed it carefully: IBM’s UNIX license comes up for renewal on June 13th. If IBM doesn’t cave, SCO might force IBM to stop distributing AIX. From a practical point of view, Linux still isn’t ready to take over from AIX at the high end, so IBM probably can’t afford to let SCO kill off AIX. Many have speculated that SCO really wants to get bought out by IBM as their exit strategy from a hostile marketplace.

I’ve always been more of a SYSV than a BSD UNIX guy. I just don’t like the cruftiness of a lot of the BSD stuff; SYSV always seemed to have the cleaner design. (terminfo vs termcap, runlevels vs startup scripts, lp vs lpr, ksh vs csh and sh, …) However, one thing this lawsuit has done is to drive me solidly into the BSD camp. I never want to use a SYSV system again. Farewell, Solaris.

[I should point out that I have no knowledge of IBM’s official position regarding the lawsuit. The above comments are obviously not official in any way. I don’t use AIX, and I don’t know any AIX developers. I do know someone who worked for SCO, but he jumped ship a few years ago when the company started going down the toilet.]

Porn star Asia Carrera has posted some pictures of herself in Doc Martens and shorts, disassembling and reassembling PCs in her (rather untidy) computer room, and snuggling her cats. For the record, she also played Carnegie Hall at 13, is a member of MENSA with an IQ of 156, knows UNIX, and does her own Photoshop work. Oh, plus she’s made over 250 hardcore porn movies…

(The link probably isn’t safe for work, of course. The pictures on that page are all innocent stuff, but the URL will set off alarm bells.)

I guess I’m just amused by the idea of UNIX hackers as porn stars. Also, sara can attest to the fact that I find ability to master a command line to be a definite turn-on.

I was in Best Buy today, looking to see if they had any cheap DVD-Rs, when a guy walked up and asked me if I knew about laptops. I said that yes, I did, but that I didn’t work for Best Buy.

To my surprise he said that he knew that, that the people who worked at Best Buy didn’t know what they were talking about, and that I looked like I might be the kind of person who did know stuff about laptops.

I ended up chatting to him for quite a while about PC laptops, features to look for, and tradeoffs between factors like price, size, speed, and battery life. I mentioned that I used Macs at home, and he said that he had an old Mac, but hadn’t been using it much recently. I told him about OS X, and he was really excited to hear that it was based on UNIX. Turned out he was keen to ditch Windows, so long as he could continue to do what he needed to do on Mac or Linux. He was also interested in web development and multimedia, so we talked for a while about what came with Mac OS X in the way of development tools and media applications.

Anyway, I told him where he could look at laptops (both Windows and Mac), told him that he really didn’t want to run XP or OS X in 128MB of RAM no matter what they said, explained the problem of not being able to buy laptops with Linux from the manufacturers, mentioned that it installed pretty well on ThinkPads, explained what he needed to get to set up a secure firewalled home network, and gave him my personal card so he could e-mail me if he had any more questions…

I sometimes wonder if I could pay the rent this way.

1. Installation was totally painless.

2. It actually manages to do what I thought was impossible—it puts a cohesive Mac-like UI on top of a UNIX box. It’s convincingly graphical and feels solid and friendly, something Linux and GNOME fail to achieve. Yet there’s all the juicy BSD goodness underneath the skin, if you open up a telnet window.

3. The e-mail client is usable. I’ve tried to wean myself off of the Eudora habit many times over the years; the big E has been showing its age in many ways. OS X Mail is good enough that I’m saying goodbye to Qualcomm’s offering, particularly as they haven’t bothered to produce an OS X version yet. In particular, Mail’s IMAP support works, whereas Eudora’s doesn’t.

4. Built in FTP, HTTP, WebDAV, SMB, SSH, et al. No more problems getting my files from machine A to machine B.

5. I had problems with a program that locked up hard—so I did ps aux | grep meta then a kill -9. Woo. (There’s also a graphical process control program, of course.)

6. It hasn’t crashed yet. Last night I was simultaneously reading my e-mail via IMAP, browsing two different web sites via SSL connections, had a couple of telnet sessions open, was backing up files to DVD, and also playing some MP3s.

7. It’s faster than OS 9 at quite a few things. Movie playback is smoother. DivX playback works. iTunes is way faster.

8. I plugged the digital camera in and it worked. Built in OS software to rotate and download images. Drag the slider and the icons in the view smoothly enlarge to 128 pixels across so you can see the images as thumbnails.

9. All the developer tools are on a CD in the box, for free.