Introduction
Email Chains are important to keep track of email conversations and properly represent them in Jira as issues and its comments.
By default, messages being part of a particular mail thread are identified and the corresponding email thread is maintained based on the Jira issue key appearing in the messages' subject.
Even if the issue key is not quoted in the subject, there is still another option to rely on a globally unique attribute of a message, namely the Message-ID.
A Message-ID is a globally unique message identifier that refers to a particular version of a particular message. It is generated and set in the header part of the email by the email server that dispatches it.
Although - according to the RFC-5322 standard - Message-IDs are not mandatory (header) attributes of an email, it is highly recommmended to provide one and most service providers comply with this.
Typical use cases
To better understand the need for managing mail threads, consider the following, quite usual situation:
Anne sends an email to Bill and Corp, where Corp is the mail account that is processed by Jira. The email does not contain any reference to an existing issue in Jira.
The issue is created in Jira by our Mail Handler.
Bill receives the email and uses the Reply-All function of his mailer app to send a reply to Anne and implicitly to Corp as well, since Corp was on the original email.
The reply email reaches Anne and Jira, but as it lacks any valid issue key, a new (kind of duplicate) issue is created again by the Mail Handler.
However, the preferred (or even the expected) behavior was to save Bill’s response as a comment in the issue created from Anne’s original email.
Let’s see a bit more complex example:
Anne sends an email to Bill and Corp, where Corp is the mail account that is processed by Jira. The email does not contain any reference to an existing issue in Jira.
The issue is created in Jira by our Mail Handler.
Bill receives the email and replies to Anne and Daniel (using the Reply or the Forward option of his mailer app), but does not include Corp in the recipient list (i.e. explicitly removes it).
Daniel receives the email and replies to Anne, but not to Bill and Corp.
Anne receives the email and replies to Daniel’s response, but at the same time she decides to send the message also to Corp (she adds it to the recipients again).
The reply email reaches Daniel and Jira, but as it lacks any valid issue key, a new (kind of duplicate) issue is created again by the Mail Handler.
However, the preferred (or even the expected) behavior was to save Anne’s response from the 5th step as a comment in the issue created from Anne’s original email.
It is worth to note that in contrast to the previous example, the mail sub-thread in the 6th step is completely detached from the original (main) thread started in step 1 and 2. If - in the meantime - somebody replies from the Jira to the original message (to Anne or Bill or both), a completely independent sub-thread is created possibly containing several replies of its own. On the other hand, these two sub-threads will never be interleaved again (even if there are some common recipients every now and then), the only common point is the original root message sent by Anne in the 1st step. This is the root of the message graph/tree.
Assuming that the initial subject stated in the original email sent by Anne in the 1st step is likely valid and topically relevant in any message of any sub-thread, it is a justified expectation to have them assigned to the very same Jira issue.
At last, let’s consider the following use case:
Anne sends an email to Bill and Daniel. The email does not contain any reference to an existing issue in Jira.
Bill receives the email and decides to have a Jira issue created from the topic, therefore he replies both to Anne and to Corp (added as CC recipient). Besides, he removes Daniel from the recipient list.
The issue is created in Jira by our Mail Handler.
Anne receives the email, but replies to Bill only (instead of Reply-All, she uses the Reply function). As a consequence, Jira is not notified about the update, it does not know about the reply message.
Daniel also receives the original email from Anne, then at his discretion he also decides to have a Jira issue created from the topic (he does not know about the already created issue as he was omitted from the replies in the main thread) and thus replies both to Anne, Bill and Corp (added as CC recipient).
The reply email reaches Anne, Bill and Jira, but as it lacks any valid issue key, a new (kind of duplicate) issue is created again by the Mail Handler.
However, the preferred (or even the expected) behavior was to save Daniel's response from the 5th step as a comment in the issue created from Bill’s reply in the 2nd (and 3rd) step.
In this last example, we saw an incident, when Jira was not involved into the conversion from the very beginning, but it was notified from two different sub-threads at different stages during the flow of conversion. On the other hand, it is expected to get those replies appear as comments in the very same Jira issue.
The occurrence of such (effectively) duplicate, but seemingly “independently living” Jira issues is a recurring problem - especially when managing lots of customer requests - and causes confusion in many cases, as cohesive messages and their history are administered in separate Jira issues.
In addition to the propositions above, it is extremely unlikely that mail threads have a linear history. There are almost unlimited number of combinations for creating branches, that is, sub-threads in a mail thread, where some of the original recipients are omitted or other recipients are added (or re-added) on the way. And the essential point is that Jira (Corp) can get posted (i.e. involved into the conversation) at any stage again and again, while missing some messages sent by other participants in the meantime (in any sub-thread).
The main purpose of mail chain management is to identify such mail threads correctly and preserve the cohesion of the messages belonging virtually together, also in lack of issue keys in the emails' subject. In other words, to avoid creating practically duplicate issues, as mentioned above.
To get around the problem of ensuring the cohesion of email threads, please follow the instructions below.
Configuration
To support email thread management based on Message-IDs, you just have to switch on the Enable issue lookup by mail headers option on the Find issue tab on the Edit Mail Handler page. After this, your email chains will be automatically maintained.
As a result, the handler will attempt to find issues by extracting Message-IDs from the References or In-Reply-To email headers of the reply emails. If found, the reply mail’s body is added as a comment which is the desired behaviour in most use cases.
Please note that in order to take this rule effect, the messages' subjects must not contain any existing issue key, otherwise the issues will be matched by the issue key in subject rule due to its precedence. This may lead to confusion in particular situations, if the two rules would otherwise result in different issue resolutions (that should not happen in an ideal setting).
Operation principles
General
If possible, for all incoming emails, all available Message-IDs that can be extracted from the headers of the current incoming email are saved to the respective Jira issue, even if this particular lookup is switched off (i.e. if the matching issue is found based on the “issue key in subject” rule or based on a JQL filter condition).
By implementing this kind of retroactive Message-ID storage, the Jira issue will also know about the preceding part of the mail thread, even if Jira was get into the loop in a later stage (practically, it can learn about the Message-ID of the root message).
By turning off the Enable issue lookup by mail headers option, the matching issue will no longer be found based on the Message-IDs. This may result in creating new issues, as described in the use cases section above. However, re-enabling this lookup will then possibly return multiple issues for the given search, which can be problematic by nature.
Handling multiple matches
Provided that the mail chaining feature is turned on, multiple matches are not possible any more (following a phasing-out mechanism).
However, if either the mail chaining feature is temporarily turned off or if duplicate issues already exist for a particular mail thread in the system (at the time of introduction of this feature), the problem of multiple matches must be addressed.
As email processing runs in the background (without user interaction), our primary goal is to provide a systematic approach to an unambiguous resolution of the matching issue in most possible cases, that is, we try to identify the issue, which the message shall be assigned to.
In case of multiple hits, the issue having the lowest issue ID gets automatically selected. The reason behing this logic is that most probably the oldest Jira issue - that is, the issue with the lowest issue ID - was created first in time, as the original entry point for the given topic, and all other issues can be considered as duplicates.
Search issues manually by Message-ID
Technically speaking, the list of the associated Message-IDs is stored as a JSON array in a Jira entity property of the given issue. It is possible to check the saved data from the Jira’s user interface, too, as this entity property was made publicly searchable. In order to to this, navigate to the advanced issue search page (Filters / Advanced issue search menu) and enter any of the following expressions by substituting with the appropriate data.
To search for issues based on a known Message-ID (e.g. “messageId1“) that appears in the associated mail thread, enter the following JQL search query:
issue.property[com.metainf.emailthisissue.entityproperty.messageids].messageIds = "messageId1"
To search for issues based on a known Message-ID (e.g. “messageId2“) that was the entry point of the given mail thread for the mail handler, enter the following JQL search query:
issue.property[com.metainf.emailthisissue.entityproperty.messageids].initialMessageId = "messageId2"
The expressions above can also be combined, e.g. to search for issues that have any of the Message-IDs (“messageId1“ or “messageId2“) in their mail threads, use the following JQL search query:
issue.property[com.metainf.emailthisissue.entityproperty.messageids].messageIds = "messageId1" OR issue.property[com.metainf.emailthisissue.entityproperty.messageids].messageIds = "messageId2"
This manual search feature is a great tool to double-check your issue metadata and start troubleshooting, if you encounter some weird behavior (e.g. not the expected issue is being processed by the mail handler), possibly due to multiple matches. In case of “mail thread equivalent duplicates”, you need to do the clean up among the issues to ensure the correct behavior.
Normally, an ever-increasing record is built from the Message-ID list of a continued mail thread. However, if the size of the Message-ID list exceeds 32 kB, only the most relevant Messsage-IDs will be preserved, the less relevant ones will be discarded. Usually, this won’t bring the mail chaining feature to a halt, as a mail thread can also be identified typically based on one of the “most relevant” Message-IDs only. One of such special purpose Message-IDs is the initialMessageId, i.e. the entry point of an email thread from the mail handler’s point of view, which is stored in a dedicated field. The other very important Message-ID is that of the the “root message” (the orignal message of the entire email thread), which is usually the first item in the Message-ID list featured in the References header.
Message-IDs in outgoing emails
The outgoing emails sent by the Email This Issue app has a Message-ID specific for Jira (as Message-IDs are generated by the service provider’s own choice). This Message-ID contains the numeric issueId to easily identify the associated issue of any message sent by Jira, later on.
Since we use this feature to identify the associated issue for reply messages, the References header is currently not populated by the app (i.e. all the previous Message-IDs of the thread are not copied over). This may lead to confusion in certain email clients, as such outgoing mails and replies to them may be misconceived by them and regarded as separate email threads (they might lose track). However, this won’t cause any malfunction at the Jira side: as soon as any reply is posted to the mail handler again, the respective issue will be found and commented, as expected.