SiKing

August 10, 2008

GMail on Thunderbird: what you might not know

Filed under: google,tech — SiKing @ 12:25 pm

Google Mail is one of the most popular free (as well as paid) e-mail providers. Thunderbird is a very popular free e-mail client. In this article I will discuss what are some of the features of this duo that you might not expect or know about. I will not be discussing why you would want to chose either of these over any of their competition, nor will I be discussing why you would want to use Thunderbird over the GMail web interface. The discussion here will focus on features, not pros and cons.

The nicest thing about standards is that there are so many of them to choose from.” Ken Olsen

E-mail protocols

Let me start with some background: what you need to know in order to set everything up and to be able to follow the rest of the discussion as well as some of the provided links.

There are three main mail protocols, at least two of which must be configured so you to be able to access a mail server (such as GMail) from a desktop client (such as Thunderbird). I will give only a very simplified description, a quick overview, of each protocol; for a more detailed discussion see any number of websites on the Internet, Wikipedia being a very good starting point.

The Post Office Protocol 3rd version – POP3 – is a simple protocol used for retrieving mail to your machine. A connection between your machine and the server holding your mail is established only for the actual retrieval of mail and then disconnected. Mail is always downloaded and stored in plain-text on the local machine. This may be a potential security / privacy risk (this risk is normally reduced with user-level file access and can be even further reduced with encrypted file systems, but both of these topics are very much outside of the scope of this discussion). POP3 is a one way communication protocol, specifically from the server to your machine; mail cannot be sent from your machine to the server using the POP3 mechanism. Besides retrieving mail, the protocol also offers some simple server management commands like: delete mail after download, mark mail as read after download, etc. GMail ignores most of these commands from the client. They have instead created their own tasks that are automatically performed after the mail is retrieved. The three tasks are: keep mail in the Inbox (effectively do nothing), archive the mail, delete the mail. This is usually the protocol of choice for portable computers.

The Internet Message Access Protocol – IMAP – is a more modern protocol, compared to POP3, and certainly more feature rich. The main difference from POP3 is that your mail is always stored on the mail server, and is downloaded to your machine only for reading. Your local machine keeps only metadata about the mail – things such as message headers, message count, messages read, message location, etc. The mail is physically present on your machine only in memory and only while you are reading it, which lowers the potential security / privacy risk. However, the server and the client must be in constant contact. For this reason, IMAP is not normally a choice for portable computers. IMAP allows for two way communication; you can use the IMAP protocol to upload a mail message from your client to the server, but only for storage purposes and not for mailing! Obviously to accomplish all of this, the IMAP protocol defines significantly more commands. GMail also traps some of these commands and does their own thing – complete discussion is further below. This is often the protocol of choice for desktop computers, and especially for users that want to be able to access their mail from different computers (such as Internet cafes).

Only one protocol is available for sending mail – SMTP – the Simple Mail Transfer Protocol. It is the counterpart to POP3 in that it was developed at about the same time. Just as POP3 it only establishes a connection for the transfer of the mail, and then disconnects. This protocol is what is used on all servers all over the Internet to pass your mail from one point to the next.

Each of these protocols is handled by a piece of software on your mail provider’s computer, and is often referred to as a server. The mail provider, depending on their needs, can have any number of hardware servers (often and unfortunately also referred to as just a server) from one (a small personal mail server) to possibly even hundreds (for somebody like GMail). Each of these will usually have a unique server name. In order to keep each of the services from colliding with each other, this is especially important in the case of one server systems, each service is assigned a unique port number. This is a virtual software identifier that allows multiple services to live on one physical machine. Normally the end user is only concerned with the machine name and port number for each protocol, and this must be provided by the mail provider.

A simple world

The easiest setup the user can have is POP3 / SMTP. In this case the user will configure their mail client to receive mail from their provider from a server, for convenience, often named something like pop3.provider.com, and the default port is 110. The user then defines that outgoing mail should be sent to their provider to a server called, again for simplicity, something like smtp.provider.com, and the default port is 25. Normally all the default setting are predefined in the client software. Historically the users were given small storage quotas, usually enough to keep a few hundred mails on the server. Once the quota was reached, the user had to either delete mail or downloaded and remove it from the server – all this was managed from the mail client. Today storage space is no longer an issue and users are often given extremely high, effectively unlimited storage quotas.

The other combination that a user can have is IMAP / SMTP. In this case, the only difference is the incoming server will be probably called something like imap.provider.com, and the port that it will be expecting by default will be 143. There is actually one more slight difference, but this is completely transparent to the user. Remember that in this case, all the mail actually lives on the server; even copies of the mail that the user sends out. When the user sends a mail, it will actually go out twice: once to the SMTP server to be mailed off to the intended party, and a second time over to the IMAP server for storage so that the user can have a local copy of the mail.

Of course the user is free to use all three servers, assuming the provider is offering all three services.

A more complicated world

In today’s world, due to privacy and security issues, almost all communication is done over secure encrypted channels. This is accomplished quite simply by another software service, installed on the same machine as each of the above discussed protocols, but using a different port. Secured POP3 connection is normally on port 995, and 465 for secure SMTP. Secure IMAP normally runs on port 993. All of the defaults, secured or not, can be changed by the provider if they chose to do so. Again, all the details must be given to you by the mail provider; they are mentioned here however so that you have some idea of what to look for when you are searching for the setting on your provider’s help site.

Thunderbird features

Thunderbird logoA list of Thunderbird features is online; the ones that are relevant to the discussion here are:

  • Support for all the mail protocols discussed above.
  • Allows grouping of mail into effectively unlimited number of folders and subfolders.
  • Support for “staring”, which is a special mark used to highlight a mail message. Staring is supported by the IMAP protocol, and thereby transferable to other machines. Remember that POP3 is a one-way protocol; if you connect to your mail server using POP3 and want to mark a message with a star, Thunderbird will allow it but the mark will not be available on the server and therefore not visible on any other machine that you use to access your mail.
  • Support for labels, which are additional marks you can assign to mails that can subsequently be used in display filters or searches. Labels do not have any kind of hierarchy. It is important to note that mail labels are not part of any of the standard protocols. The labels that you define in Thunderbird will become part of the metadata and will be stored only on the local machine. This means that even if you use the IMAP protocol (where all the mail is stored on the server), the labels will not be available there and therefore not visible on any other machine that you use to access your mail.

Updated 2008/12/21: I am beginning to suspect that I am wrong about the Thunderbird labels. I just reinstalled my kompy, and somewhere in the process I managed to screw up my backups. Once I got everything set up in TBird and downloaded all my mail through IMAP, all my old labels – mails that were marked in TBird with a particular label – were all still there! However, I still do not see any way to get at these labels through the GMail online interface.

GMail offerings

Gmail logoGMail’s list of features can be found here; again, only the ones that are relevant to this discussion are:

  • Support for all the mail protocols discussed above, even with the free accounts. Each protocol provides some additional functionality.
  • GMail has a different definition of folders. Their philosophy is that they give the user effectively unlimited disk space, they remove the ability (hassle?) for the user to sort their mail, and thereby force the user to use their search engine to find past mails. Each account comes with several “views” (as GMail calls them), each of which have a special function, that are presented to the user as folders. The user is not able to create additional views or to remove existing views. The views are:
    • Inbox – holds all incoming mail. Using only the GMail web interface, the user does not get any copy or move buttons as they might expect. Mail is moved or copied into the other views through various actions. To remove mail from the Inbox, the user can Archive the mail (see All Mail below) or Delete the mail, effectively move it to the Bin view. If you connect via POP3, you will be able to access only mail that is in this view; all the others are effectively invisible to the POP3 protocol.
    • Starred – message that have been marked with a star. As mentioned above, this mark is supported by the IMAP protocol and therefore visible to the client. The star cannot be transfered over the POP3 protocol.
    • Chats – history of discussions from Google Talk.
    • Sent Mail – stores copies of all mail sent by the user. Special note: any mail sent through smtp.gmail.com is automatically copied to this view. This is a feature that significant majority of other providers do not have. If you configure your client to place a copy of each outgoing message here, GMail automatically will delete the duplicates.
    • Drafts – stores a copy of mails that you started to create but have not yet sent out.
    • All Mail – holds an apparent copy of every single mail from every single view above; the two views below are not duplicated here. I use the word apparent for several reasons. If, for example, a mail arrives into your Inbox it will also be replicated in All Mail. However, when you Delete the mail in the Inbox, the copy here is deleted as well – you do not have to explicitly go and delete this copy also. The Archive button, mentioned above, is used to remove the mail from the Inbox only and preserve the copy in the All Mail view. The link works the other way as well: if you delete a mail in the All Mail view, it will be deleted in all other views where it may exist, except for the Drafts view – it is not clear to me if this is a feature or a bug.
    • Spam – any mail that GMail determines to be a scam of some kind. It is possible to turn this feature off, although not advised; the GMail spam filter is very good. The user is also able to mark any mail in any view as Spam, in which case that mail will be moved into this view and automatically removed from all other views. The user can subsequently unmark any mail in this view, whether marked by GMail automatically or by the user, thereby moving the mail from this view into the All Mail view.
    • Bin – any mail that is scheduled for deletion; this view is emptied periodically by GMail or can be emptied by the user. It is possible for a user to subsequently undelete any mail in this view, thereby moving from this view into the All Mail view.
  • Labels are also handled differently at GMail. As was already mentioned labels are not supported by any of the protocols; GMail has an interesting workaround for this: they expose all labels as folders to the client. Also, any folders created by the client will show up as labels at GMail.

Putting it all together

When you first create a GMail account on the web, all the protocols are disabled. You need to go into your account’s Settings > Forwarding and POP3/IMAP and enable POP3 and/or IMAP; SMTP will get enabled automatically if either of the previous two is turned on. While you are there, I already mentioned that GMail ignores the commands from your client regarding what should be done with the mail on the server after it is retrieved. By default GMail will do nothing, that is, it will leave the mail in your Inbox. In all likelihood, you will want to change this to Archive all mail; this is done on the same Settings page where you activate POP3.

Thunderbird has a wizard to help you set up a GMail account. Answering all the questions will configure Thunderbird for POP3 and SMTP. If you want to also set up IMAP, you will have to do that manually. When you connect for the first time with each of these (receive messages for POP3 or IMAP, send messages for SMTP), you will be asked to enter your password each time. As I already mentioned, each of the protocols is actually a service running on some server somewhere. It is theoretically possible, although not common, that you can have a different username / password for each server. That is why Thunderbird will ask you for the password three times, even though on GMail they are actually one and the same.

When you connect to GMail using IMAP, the first thing Thunderbird does is to retrieve the list of folders. First time, you will notice a new folder appear in Thunderbird called “[Google Mail]” (without the quotes, but with the square brackets). All the views discussed above will now show up as subfolders under this, except for Inbox which will be correctly mapped to your Thunderbird Inbox. By default, Thunderbird creates a local folder called Trash. If you look at your GMail account on the web, you will see a corresponding label “[IMAP]/Trash”.

Ready to go? Not quite. Deleting mail now becomes a little tricky. All mail you retrieve from POP3 will get archived – moved to the All Mail view. If you delete anything in your POP3 account in Thunderbird, GMail will not be notified of it, even if you turned on the “Leave messages on server Until I delete them” options – remember GMail ignores all the POP3 management commands. With IMAP, it’s even worse. If you click on the Delete button in Thunderbird (remember none of the mail in IMAP is actually on your machine) the mail will just get moved to the label [IMAP]/Trash but will still remain in the All Mail view on GMail. In order to delete it, you would actually have drag the mail from [Google Mail]/All Mail to [Google Mail]/Bin. Like I said: GMail is actually pushing the fact that you never have to delete mail as a feature, and therefore they did not put a lot of thought to the delete process.

One more small item; this is independent of GMail. When you delete a mail in a POP3 account on Thunderbird, the mail is not actually deleted but only marked for deletion. This means that the mail is no longer displayed for you, but it is still stored on the disk. This has two implications. First, if you just leave it there thinking nobody will find it, you are wrong. Go to $HOME/.mozilla-thunderbird/*.default/Mail/pop.gmail.com (on Windows machines, $HOME is going to be something like C:\Documents and Settings\<login name>\Application Data) and have a look at any of the files in a plain text editor. You will still see mails that have been apparently deleted. This could be a security / privacy issue. Second, even if you delete a mail it is not gone, you can still change your mind. You should be able to retrieve if from the disk with a plain text editor. To remove deleted mail for good, select File > Compact Folders.

Personal habits

On my laptop I have both POP3 and IMAP setup. POP3 is configured to retrieve mail on startup and every 15 minutes, IMAP is does not retrieve mail automatically at all. I use the POP3 account to retrieve / reply to mail, and the IMAP account to do any kind of maintenance – deleting mail (err: moving mail to the Bin view), labelling mail (moving mail to folders), marking mail with stars, etc.

As I already mentioned, any mail that is sent through smtp.gmail.com is automatically copied into the Sent Mail view. Therefore, under Account Settings > Copies and Folders, you can turn off the “When sending messages …” thing for POP3. Under IMAP, first I set a copy of the sent mail to be placed in [Google Mail]/Sent Mail on IMAP, and then I turn this feature off. This may seem strange, but what actually happens is that Thunderbird will start to treat this folder as a proper sent message folder: the icon of the folder will change, but the bigger point is that when you are viewing the list of mails, the Sender column will be changed to the Recipient. Next, the Drafts and the Templates: I leave these as they are for POP3, however under my IMAP account I redirect both of these to their equivalents under the POP3 account. That way if I am creating either I do not have to be online (remember IMAP needs a constant connection), as my POP3 folders are actually physical folders on the machine’s disk.

Also, I turn off the junk mail feature in Thunderbird for the IMAP account, but leave it for the POP3 account. Any junk mail filtering logic will not be passed from Thunderbird to GMail. Having Thunderbird move junk mail round in the IMAP account will only replicate the junk. But if anything gets past the GMail filters (which are quite good), it will get caught by Thunderbird in my POP3 account, without polluting the server with copies of the junk mail. To mark this mail properly as junk at GMail, unfortunately can only be done using the web interface.

If you have more than one IMAP server defined and you try to move messages between the servers, GMail will allow you to only copy not move (that is delete) from GMail to somewhere else. This applies even if you explicitly use the Move function of Thunderbird, or if you use a mouse drag and drop. However, if you have two different GMail accounts, both setup on your machine as IMAP, then in this case GMail will allow a move from one account to another.

There is one last item of interest. I use to keep an archive of all my mail from all past sources on a CD. Over the years I changed my mind a few times on exactly how best to accomplish this. When creating the archives, every once in a while I accidentally duplicated some mails (and once in a while delete some, but GMail will not help me with that now). When I opened my GMail account, which has over 6GB of disk space, I decided to upload all of my old mail archives. Import the mails into Thunderbird as fictitious POP3 account, create the appropriate folders on GMail, and Move the mail. I discovered two interesting things. First that GMail actually filters out (deletes) duplicate mails in any view. Second is that if try to drag and drop the mails to GMail, and something happens with the connection, Thunderbird will not retry it. However, if you use the Move feature in Thunderbird, it actually has a retry, and I was able to move much larger blocks of mail that way.

Advertisements

11 Comments »

  1. Shouldn’t it be called “NSA-mail” ?
    Those of you still using anything by Google… beware! They are between the sheets with our very own NSA to spy on every single thing you do

    Do your own research.. find the truth yourselves! I stopped using Google anything long ago and couldn’t be happier.

    Comment by PoorWhiteMan@knowhere.net — December 15, 2014 @ 5:50 am | Reply

  2. This is a great overview of the details and basics of Gmail and client side email use! Doesn’t help me with getting Sent emails to delete off the Gmail side or help me quit getting every sent email i send from http side to stop showing up in my Thunderbird Inbox, but apparently nothing helps that except complicated and stupid filters that don’t resolve the issue. If anyone knows of a better email provider than Gmail please let us know! Now that Google is rumored to be a major CIA/NSA front it would be nice to ditch them anyway. Thanks. 🙂

    Comment by Anonymous — January 25, 2012 @ 12:29 pm | Reply

  3. Has anyone ever had the problem where Thunderbird will stop retrieving the messages from GMail for a period of time? This has happened to me in the past and is happening now. I have Google Talk installed so I am notified whenever there’s a new message on my GMail account. Most of the time Thunderbird will retrieve the message a minute or so later, but sometimes it doesn’t. Even if I force Thunderbird to retrieve messages (using the Get Mail button) it won’t. If I completely shut down Thunderbird and restart it still no messages. Then at some point (anywhere from 15-30 minutes after I’ve seen the Google Talk notification) Thunderbird will retrieve a bunch of messages. The most frustrating part of this is that no settings have changed at all. Thunderbird is set to retrieve messages for my GMail account on startup and every 1 minute. Sorry for the long winded comment, but this is driving me nuts.

    Comment by Darryl — April 7, 2009 @ 12:45 pm | Reply

    • I have my TB set to remember all passwords, and sometimes the thing comes back that I have the wrong password and I need to renter it. I do and it still tells me that it is wrong. I restart TB, reenter the password, and everything is fine. I usually attribute this to network problems. I would not even put it past ISPs to throttle down any IMAP/POP3 connections outside of their own domain, in an attempt to force customers to use their mail services, thereby locking them in…
      Now where did I put my tinfoil hat? 🙂

      Comment by siking — April 7, 2009 @ 4:32 pm | Reply

  4. I have a question. I like using the gmail web interface, but I want to store all incoming messages and sent messages (I want to keep a record of all mail received and sent- exactly as it appears the first time I receive/send it) on my computer. An important part of this would be for a copy of the messages to remain where they are (either in the inbox or sent mail box) even if I delete it on Gmail. I would like to use Thunderbird for this. Do you know how this can be achieved?

    Thanks!

    Comment by Doug — March 24, 2009 @ 6:54 am | Reply

    • Doug. It sounds as if you do not want to synchronize your Thunderbird and mail server; IMAP synchronizes the two, so that is not for you. You want to set up your Thunderbird to use POP3 to access GMail. POP3 will download your incoming messages, and after that it will not care what you do on either end – you can delete the message on one, without the other knowing or caring about it. HTH.

      Comment by siking — March 24, 2009 @ 7:23 am | Reply

  5. Thanks Siking.

    I’ve discovered your point the hard way. Emails reappear in my
    “All mail” folder after I’ve moved them to another Gmail acct.

    Now my Tbird “All mail” folder has many more emails than “All mail”
    online. It’s very confusing. I’m afraid though that unsubscribing
    to “All mail” I’m going to lose some items.
    I’m now trying to find a good way of comparing various folders to
    discover dups to delete by moving to Gmail trash. All the programs
    I’ve found only compare folders nested together.

    Comment by Rob — February 18, 2009 @ 10:47 am | Reply

  6. Rob, GMail (on their end) will not allow you to copy+delete (aka: move) your mail. You have to do it yourself as a two step process: 1) copy your mail, followed by 2) move the mail to the [Google Mail]/Bin folder. Remember, if a piece of mail is in [Google Mail]/Bin, that is the _only_ way for GMail to consider it deleted, and it will clean up that mail from everywhere else.

    Comment by siking — February 18, 2009 @ 10:37 am | Reply

  7. This is a great summary.

    I am still having issues with deleting Gmail in Thunderbird.
    More specifically, moving mail between two different Gmail
    accounts. I can’t figure out how to delete the originals.
    They keep re-appearing in All mail no matter what I do.

    Comment by Rob — February 16, 2009 @ 2:45 pm | Reply

  8. You are a smart man, really. Thank you for sharing your experience.

    Comment by Wunderbar — February 13, 2009 @ 2:56 pm | Reply

  9. This post apparently got its 500th hit yesterday (by far the highest on this blog), and not one single comment. If you’re reading this, why not say (at least) hi? 🙂

    Comment by siking — January 22, 2009 @ 9:05 pm | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: