IMAP file switch

This page discusses moving your IMAP mail files to the Nevis mail server. If you use POP instead of IMAP, you can ignore this page.

Note that this is a one-time procedure. After you do this, you'll never have to do it again.

Procedure

19-Feb-2009: Edited to reflect what you have to do now that the switch has been made.

In the following instructions, $user is the name of your Nevis login account.

1. Follow the directions in IMAP mail files to see the location where your IMAP folders are stored.

2. If this location already begins with /a/mail/folders, /mail/folders, or is blank, you're set up. Don't do any of the remaining steps; they were already done.

3. Otherwise, copy the location that you see. If it's blank, the location is your home directory. If location does not begin with a /, then it's relative to your home directory; e.g., "mail" means ~/mail.

   Don't forget to copy this! If you make a mistake, you can go back and type in the old location.

4. Replace the IMAP location with a blank, that is, erase any text you see in the field.

5. Quit your mail program.

6. Login to the Nevis Linux cluster.

7. Create your new mail directory with your account name within the automount directory /a/mail/folders and copy your mail files to that new directory.
   For example, if your old IMAP mail location was ~/mail:
   mkdir /a/mail/folders/$user
cd ~/mail
cp -arv * /a/mail/folders/$user
   1. Be careful if your default mail location was your home directory. You don't want to copy your entire home directory to the mail server, only the mail files. You'll have to copy them one-by-one. If you aren't sure which files are mail files, look at your ~/.subscriptions file (you might have to edit this in the next step anyway).
   2. Why /mail/folders in the mail reader, but /a/mail/folders in the above step? It's because of automount. You have to refer to the directory using the automount name, but the mail server (on which IMAP is running) can use the direct name. If this confuses you, use /a/mail/folders everywhere; it won't make too much difference.

8. The next step is to tell your mail reader which IMAP mail files contain your mail. You have a choice of how to do this:
   1. Start your mail reader. If you don't see your IMAP folders come up immediately, subscribe to them again. In Thunderbird, you do this by selecting "Subscribe..." from the "File" menu.
   2. If you have a large number of IMAP mail files, or a complex folder hierarchy, the above step may be tedious. The alternative is for you to edit your /a/mail/folders/$user/.subscriptions file.
      1. Make a backup copy first!
      2. Make sure that every IMAP file name has no special prefixes, including utility folders like Sent, Trash, and Junk. For example, if your home directory is /a/home/amsterdam/jsmith, you might change a line like this:
         /a/home/amsterdam/jsmith/mail/DetectorDescription
         to
         DetectorDescription
      3. If you've used several mail readers over the years, .subscriptions have many different sets of prefixes to the same mail files. A given mail file only has to be in .subscriptions once; you can remove duplicate entries.
      4. Start your mail reader after you've finished editing /a/mail/folders/$user/.subscriptions.

9. Move (not copy!) your old mail files in such a way that you won't think that they're still in use. Don't laugh; people have been confused before. Something like this is sufficient:
   mv ~/mail ~/old-mail

10. Make sure everything works. You should see your IMAP folders displayed normally by your mail reader.

11. (Optional, but valuable) Send WilliamSeligman a quick message to let me know that you've made this change.

By when do we have to do this?

The mail server configuration was changed on 18-Feb-2009, at about 6:30PM. I also copied any ~/.subscriptions file to /a/mail/folders/$user/.subscriptions so you didn't have to do it yourself.

The change moved the "default mail location" from each user's home directory to /mail/folders/$user. New accounts will automatically have their mail folders and .subscriptions file stored in this directory.

Webmail

If your mail files are in /a/mail/folders/$user, then make sure the folder path in webmail is blank. You'll be able to (re-)subscribe to your folders.

Background

Why do we have to make this change?

We've all seen it happen: one of the Nevis workgroup servers goes down, and the mail server slows to the point that no one can read or send mail.

The reason why this happens is the IMAP server. When someone tries to read their mail, the IMAP server dovecot tries to get write access to the mail files in the user's home directory. If that user's home directory isn't there (because the server it's on is down), then the IMAP server keeps trying forever. If more than one user with a home directory on a down server is trying to read their mail, then the mail server becomes clogged with IMAP server processes.

One solution to this problem is to tell users, "If your server is down, don't try to read your mail." This is not a practical solution.

Another solution is to put all our mail files on the mail server, so if a workgroup server goes down, it will affect that group (e.g., they can't log in) but everyone else can still read their mail.

The safest way to do this is a two-step procedure:

• Everyone individually moves their mail files to the mail server.

• I revise the mail server configuration to change the location of the .subscriptions file, and move everyone's ~/.subscriptions to the new location /mail/folders/$user/.subscriptions.

Why wasn't it done this way before?

Actually, it was. Before 2002, all of Nevis' computer services were stored on one computer, nevis1. It was both our central login system and our mail server. Each user's mail files were stored in their home directory.

When we de-centralized and moved to the Nevis Linux cluster, the mail services and home directories became separated. The mail files were still kept in home directories, in part to minimize all the changes but mainly because no one realized the potential problems.

Now it is time to organize our mail files the same way that sites like Columbia do: put the mail on the server.

What if the mail server goes down?

If it does, then you can't read your mail... but that will always be true, whether we make this change or not.

In the current environment, the mail server becomes unusable if riverside, karthur, kolya, shang, or hogwarts go down; in other words, the mail server depends on at least five other boxes being up. I think it's better to go through the pain of making this switch, have the mail server be independent of the workgroup servers, and having more reliable access to mail.

Can't you do this automatically, without getting getting each user involved?

I wish I could. It would be much simpler for both me and everyone else if I could come up with an automated procedure for moving the files.

Unfortunately, there are reasons why an automated solution is too complicated:

• The change relies on the value of where your IMAP mail files are stored. That's stored in the program that reads your mail (e.g., Thunderbird, Outlook, Apple Mail), not on the mail server. I can't change that remotely; you have to do that yourself.

• Take a look at your file /a/mail/folders/$user/.subscriptions file. Maybe it looks simple; I hope it does. Or perhaps, as you scroll through the file, you'll see lots of different file prefixes. This can happen if you ever have read your mail using webmail, or if you've ever switched programs to read your mail. It's difficult to write an automated procedure to recognize which files in .subscriptions are current mail files and which are entries left over from years ago.

Can't you do this for me?

I will manually go through the procedure above with those people who do not have the technical skill to login to their Nevis accounts: the administrative staff and the staff who work in the electronics building.

If you know how to login, I ask that you make the change yourself.

What will happen if I don't go through all these instructions?

On the day I change the default IMAP location, your IMAP folders will probably "disappear." Actually, the files won't go anywhere, it's just that your mail reader won't know where to look for them. You can go through the process of re-subscribing to them, but that's equivalent to the procedure described above.

What about other mail files?

If you look at the page on mail-related files, you'll see that there are other files in addition to .subscriptions and your IMAP files that affect the mail server. Do these files have to be moved as well?

It turns out that the answer is no. Files like ~/.forward, ~/.procmailrc, and ~/.spamassassin/user_prefs are accessed by the mail server in read-only mode. If the mail server can't find them because your home directory isn't available, there are no significant delays. The only difference is that, if your home directory isn't there, those files can't affect mail processing; e.g., your mail won't be forwarded or filtered as you prefer.

However, you have the option of moving ~/.forward to /a/mail/forward/$user and ~/.procmailrc to /a/mail/procmailrc/$user, to make sure your mail is handled properly even if your server goes down. See the page on mail-related files.

alpine

If you're using alpine, then:

• If you use IMAP folders from within alpine, then you must have set up folder collections. In that case, I assume you're sophisticated enough to adapt the above instructions for your own use.

• The main benefit of making the above change is that if a workgroup server goes down, users can still work with e-mail. That doesn't apply to alpine users; the program depends on configuration files that are stored in your home directory. Or to put it another way: if you can't log in, you can't use alpine; that is true now, and will continue to be true after this change.

Some more notes - Added 17-Feb-2009

After the switch, my folders have vanished!

Don't panic. Your mail hasn't been lost. it's just that, for some reason, your mail reader has lost track of where they are.

If you've gone through the above procedure, and your mail folders have disappeared, then:

1. Follow the instructions in IMAP mail files and set your mail directory to blank; that is, erase any text that's there.
2. Re-subscribe to your mail folders (File -> Subscribe in Thunderbird).

If you haven't gone through the above procedure, better late than never.

If you want to keep your mail files in your home directory for some reason, then you'll have to follow the instructions in IMAP mail files but set the directory to the full path of your home directory; e.g., /a/home/amsterdam/jsmith/mail if your account name is jsmith, your home directory is on amsterdam.nevis.columbia.edu, and you keep your mail in ~/mail. You'll still have to edit your /a/mail/folders/$user/.subscriptions file or re-subscribe to your folders.

Sent and Drafts folders

It's possible that even if you've gone through the above procedure, your "Sent" and "Drafts" folders may appear to lose messages. This is because your mail reader may have associated the paths ~/Sent and ~/Drafts with the location of those folders, even if your IMAP directory was set to something else. If that happens, don't panic. You haven't lost any messages; they're just in a file that you didn't copy.

There are a couple of things you can do:

If you don't care about whether or not you have duplicate mail messages, quit your mail reader and do:

cat $HOME/Sent >> /a/mail/folders/$user/Sent
cat $HOME/Drafts >> /a/mail/folders/$user/Drafts

If you want to use the drag-n-drop features of your mail reader to sort out the messages, do:

cp $HOME/Sent /a/mail/folders/$user/Sent.save
cp $HOME/Drafts /a/mail/folders/$user/Drafts.save

Then use File -> Subscribe (in Thunderbird) to subscribe to Sent.save and Drafts.save. When you're done dragging the messages, you can unsubscribe from the files you're not using.

procmailrc files

If you use procmail, it's a good idea to move (not copy!) your procmailrc file (as noted in the page on mail-related files:

mv $HOME/.procmailrc /a/mail/procmailrc/$user

If you have a line like this in your procmailrc file:

MAILDIR=$HOME/mail

remember to change it to the new location of your mail files, as seen by the mail server:

MAILDIR=/mail/folders/$LOGNAME

And one last note on webmail:

At this point, unless you're going to use webmail in the next couple of days, don't bother to fiddle with its settings. When the IMAP server switch is made, you'd probably have to re-do any webmail settings anyway.

If you've made the IMAP server switch, the new settings are simple: set your IMAP folder directory (see IMAP mail files for how to do this) to blank; erase any text that's there. After saving the changes, click on the "Folders" link at the top of the page; use the panels at the bottom of the page to re-subscribe to your folders.