Duplicate messages and SMTP
Why SMTP can deliver the same message twice and how to prevent it.
Partridge, C., "Duplicate messages and SMTP", RFC 1047, DOI 10.17487/RFC1047, February 1988, <https://www.rfc-editor.org/info/rfc1047>.
@misc{rfc1047,
series = {Request for Comments},
number = 1047,
howpublished = {RFC 1047},
publisher = {RFC Editor},
doi = {10.17487/RFC1047},
url = {https://www.rfc-editor.org/info/rfc1047},
author = {C. Partridge},
title = {{Duplicate messages and SMTP}},
pagetotal = 3,
year = 1988,
month = feb,
} RFC text
Contents
Network Working Group Craig Partridge
Request for Comments: 1047 CIC at BBN Labs
February 1988
DUPLICATE MESSAGES AND SMTP
STATUS OF THIS MEMO
An examination of a synchronization problem in the Simple Mail
Transfer Protocol (SMTP) is presented. This synchronization problem
can cause a message to be delivered multiple times. A method for
avoiding this problem is suggested. Nodding familiarity with the
SMTP specification, RFC-821, is required. Distribution of this memo
is unlimited.
INTRODUCTION
Over the past few years, the staff of the CSNET Coordination and
Information Center (CIC) has often been asked to help determine why a
single mail message is being delivered multiple times to its
recipients. In the process of tracing the problems of multiple
delivery, we have discovered that many duplicate messages are the
result of a synchronization problem in SMTP. There is a point in the
process of delivering a message where the receiving mailer knows it
has accepted the message but the sending mailer is still not sure the
message has been reliably delivered. If the SMTP conversation is
broken at this point, the sending mailer will be forced to re-deliver
the message, even though the message has already been received and
delivered by the receiving mailer.
DESCRIPTION OF THE PROBLEM
The synchronization problem occurs at the end of delivering a
message. When the sending mailer has finished sending the text of a
message, it is required to send a line containing a single dot or
period. When the receiving mailer receives this final dot, it is
expected to do its final message processing and either confirm
receipt of the message (with a 250 reply) or reject the message with
any one of several error codes.
Observe that there is a potential synchronization gap here. During
the period between the time the receiving mailer has determined that
it will accept the message, and the time that sending mailer gets the
250 reply, the message is active at both the sending and receiving
mailer. Until the sending mailer gets the 250 reply, it must assume
the message was not delivered. After the receiving mailer has
decided to accept the message, it must assume the message has been
delivered to it. If the communications link fails during this
synchronization gap, then the message has been duplicated. Both
mailers have active copies of the message that they will try to
deliver.
It may be hard to believe that this problem is the cause of many
duplicate messages. Intuitively, one might expect that the time
spent in the state between the final dot and its accepting 250 reply
is quite small. In practice, however, this period is often quite
long; long enough that timeouts by the sending mailer (or possibly
network failures) are quite common. Observations by the author
suggest that this synchronization problem may be the second leading
cause of duplicate messages on the Internet (second to mail loops).
Many mailers delay responding to the final dot because they are doing
sophisticated processing of the message, in an attempt to confirm
that they can deliver the message. For example, the mailers may
expand an entire mailing list to confirm that it can reach all
addressees or may attempt to physically deposit the message into the
mailboxes of local users, before confirming receipt of the final dot.
These practices are not unreasonable, but they often cause the
synchronization gap to continue for several minutes, and increase the
likelihood that the sending mailer will timeout or the network will
fail before the accepting 250 reply is sent.
AVOIDING SYNCHRONIZATION PROBLEMS
The best way to avoid the synchronization problem is to minimize the
length of the synchronization gap. In other words, receiving mailers
should acknowledge the final dot as soon as possible and do more
complex processing of the message later.
RFC-821 (on page 22) states that unless the receiving mailer is
completely unable to process a message it should accept the message
and acknowledge any errors in processing in a separate message or
messages sent back to the originator of the message. As a result,
receiving mailers should be able to acknowledge the final dot as soon
as the message has been safely put in a non-volatile (e.g., disk)
queue for further processing. Fast acceptance of a message does not
violate RFC-821.
Some mailers can be configured to do more or less processing upon
receipt of the final dot. In such situations, the mailer should
always be configured to do less processing.
Finally, some mailers allow remote mailers only a minute or two to
acknowledge the final dot before timing out and trying again. Given
the increasing round-trip times on the Internet, and that some
processing after the final dot is required, the timeout for reply to
the final dot should probably be at least 5 minutes and a timeout of
10 minutes would not be unreasonable.
AI ELI5
On this page
§1The Duplicate Message Problem
Mail administrators were often asked to investigate why a single message had been delivered multiple times to its recipients. Investigation revealed a pattern: many duplicates were caused by a synchronization problem in SMTP itself, not mail loops or retransmission queues gone haywire.
The CSNET Coordination and Information Center (CIC) studied this extensively and found that the synchronization gap at the end of message delivery was the second leading cause of duplicate messages on the Internet (after mail loops). This seemed counterintuitive because the gap appeared brief, but in practice it was often long enough for timeouts or network failures.
When you put a letter in the mail, there's a moment between when the post office takes it and when you know for sure they took it. If you assume they didn't and send another copy, the recipient gets two.
§2The Critical Moment
In SMTP, the sender ends the message by transmitting a line containing only a single dot (period). The receiver is then expected to perform final processing and send either a 250 reply (success) or an error code. SMTP defines this as the boundary: until the receiver sends the 250, the sender must assume delivery failed.
But here is the problem: the receiver may have already written the message to disk and decided to accept it, yet the 250 reply has not been sent. If the network link breaks at this exact moment, the sender will timeout and resend the message, not knowing it was already accepted. Both mailers now have the message queued for delivery.
The package is on the shelf, but you haven't heard 'received'. You knock again, nobody answers, so you leave another copy.
§3Minimizing the Gap
The RFC cites RFC 821, which states that receiving mailers should accept messages and queue them for processing, acknowledging errors later in separate messages if needed. This allows mailers to send the 250 reply as soon as the message is on disk (non-volatile storage), without waiting for delivery to local users, mailing list expansion, or other intensive processing.
Additionally, sending mailers should use longer timeouts for the final dot reply. Many mailers use 1 to 2 minutes, which is too short given modern network latencies and legitimate processing delays. A timeout of at least 5 minutes, and preferably 10 minutes, is recommended to avoid spurious retransmissions.
The post office should say 'got it' immediately after filing it away, not after reading it, sorting it, and delivering it. And you should wait a full 10 minutes before assuming they didn't get it.