Introduction
Testing email functionality during the pre-production phase is critical to ensuring that emails are delivered correctly and with the necessary content. However, test-users or test data frameworks might create existing email addresses and it is not always easy to setup send-restrictions on the existing SMTP infrastructure. To ensure that no external emails are sent from the system while testing emails and layouts in as many actual email clients as possible, a few unique issues arise throughout this process. In this blog article, we will examine traditional methods for testing email functionality and present a novel strategy that we have found to be very successful.
Common Approaches
Using email sinks like cloud or on-premises fakeSMTP services is one frequent method for testing email functionality. The drawback of these solutions is that you do not actually get a real email preview, despite how simple they are to set up. Another strategy is to use custom code to take out any external mail or send to shared mailboxes in the mail delivering application. This method, meanwhile, has several drawbacks, including changing the original content and being prone to mistakes because of frequent changes in the application.
New Approach
The best configuration for testing email functionality, according to our research, is to send emails to both specialized test-email addresses as well as central test-mailboxes using a bespoke SMTP proxy server that makes sure no non-whitelisted external emails spread. We used a small custom-coded microservice to do this, which we would like to provide in this blog.
Setup
The mail-sending program, the SMTP proxy server, and the test mailboxes make up the configuration. The mail-sending program sends emails to the SMTP proxy server, which filters out any external emails and forwards the filtered emails to the test-mailboxes.
A whitelist is used by the SMTP proxy server, a specially programmed microservice, to guarantee that only authorized email addresses may be transmitted. Every email traffic is intercepted, and any emails not on the whitelist are filtered out. Then, test-mailboxes get the filtered emails.
A group of email addresses called the test-mailboxes are used to get the filtered emails from the SMTP proxy server. To ensure that emails are being sent successfully and that they display as expected in various email clients, these mailboxes may be examined.
Code
To implement the custom SMTP proxy server approach, we used the subethasmtp library, which is a lightweight SMTP server written in Java. Here is how to set up the dependency in Gradle:
dependencies {
// SMTP handling
implementation("com.github.davidmoten:subethasmtp:6.0.5")
// Logging
implementation("ch.qos.logback:logback-classic:1.4.5")
implementation("org.slf4j:slf4j-api:2.0.6")
implementation("org.apache.logging.log4j:log4j-api-kotlin:1.2.0")
// optional ECS encoder for structured ELK logging
implementation("co.elastic.logging:logback-ecs-encoder:1.5.0")
// Test SMTP Setup to verify the sink
testImplementation("com.icegreen:greenmail:2.0.0")
// Test framework
testImplementation("org.junit.jupiter:junit-jupiter-engine:5.9.2")
}
The SMTP server must then be configured and started. Here is the Kotlin code to accomplish it:
class Application
private val logger =
logger("ApplicationKt") // Needs to be defined this way, as Application.kt is a file and not a class
// CSV of email accounts to forward each message to.
private const val CATCH_ALL_MAILBOX_LIST = "CATCH_ALL_MAILBOX_LIST"
// CSV of whitelisted email accounts that should continue to receive messages if they are already adressed
private const val WHITELIST_MAILBOX_PATTERN_LIST = "WHITELIST_MAILBOX_PATTERN_LIST"
private const val PROXY_MAILS_TO_SERVER = "PROXY_MAILS_TO_SERVER"
private const val PROXY_MAILS_TO_SERVER_PORT = "PROXY_MAILS_TO_SERVER_PORT"
fun main(args: Array<String>) {
val port = 1025 // Port our SMTP is listening on, we use docker so no need to make this configurable here.
val catchAllMailbox: List<String>? = System.getenv(CATCH_ALL_MAILBOX_LIST)?.split(",")
val whitelistMailboxes: List<String>? = System.getenv(WHITELIST_MAILBOX_PATTERN_LIST)?.split(",")
val mailToServer: String = System.getenv(PROXY_MAILS_TO_SERVER)
val mailToServerPort: String = System.getenv(PROXY_MAILS_TO_SERVER_PORT) ?: "25"
val myFactory =
ReplacingRecipientsMessageHandlerFactory(mailToServer, mailToServerPort, catchAllMailbox, whitelistMailboxes)
val smtpServer = SMTPServer.port(port).messageHandlerFactory(myFactory).build()
logger.info("Starting Basic SMTP Server on port $port...")
smtpServer.start()
}
As suggested by the 12 factor microservice methodology, all configuration is kept from the environment in this example. We also provide a ReplacingRecipientsMessageHandlerFactory object, which will do tasks like filtering out emails that are not on the whitelist.
The ReplacingRecipientsMessageHandlerFactory code is as follows:
class ReplacingRecipientsMessageHandlerFactory(
val mailToServer: String,
val mailToServerPort: String,
val catchAllMailbox: List<String>?,
val whitelistMailboxes: List<String>?
) : MessageHandlerFactory {
private val logger = LoggerFactory.getLogger(javaClass)
override fun create(ctx: MessageContext?): MessageHandler {
return Handler(ctx, mailToServer, mailToServerPort, catchAllMailbox, whitelistMailboxes);
}
internal class Handler(
var ctx: MessageContext?,
val mailToServer: String,
val mailToServerPort: String,
val catchAllMailbox: List<String>?,
val whitelistMailboxes: List<String>?
) : MessageHandler {
private val logger = LoggerFactory.getLogger(javaClass)
var from = ""
var to = ArrayList<String>()
@Throws(RejectException::class)
override fun from(from: String) {
logger.info("FROM: $from")
this.from = from
}
@Throws(RejectException::class)
override fun recipient(recipient: String) {
logger.info("RECIPIENT: $recipient")
this.to.add(recipient)
}
// ...
override fun done() {
logger.info("Finished")
}
}
}
For each email received, the ReplacingRecipientsMessageHandlerFactory produces a new instance of the Handler. All original recipients will be stored initially by the Handler.
Lastly, the email data must be processed. We check the recipients list against the set whitelist, eliminating any non-whitelisted emails, then add the test mailboxes and forward the original email to the next SMTP server. For demonstration purposes, we additionally edit the email by adding all original recipients to the subject – most of them remain in the email header, but BCC may also be confirmed in this manner. Here is the Kotlin code to accomplish it:
@Throws(IOException::class)
override fun data(data: InputStream?): String {
logger.info("MAIL DATA")
logger.info("= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =")
val remodeledEmail = modifyMessageSubjectAddOriginalRecipients(data)
logger.info(String(remodeledEmail))
logger.info("= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =")
val filteredToSend = filterWhiteListAndAddDefault(to)
if (filteredToSend?.isEmpty()) {
logger.info("no forwarding mails found - assume just used to log mail.")
return "OK message not forwarded."
}
val forwardingSmtp = SmartClient.createAndConnect(mailToServer, mailToServerPort.toInt(), "testproxy.local")
forwardingSmtp.from(from)
filteredToSend.forEach { forwardingSmtp.to(it) }
forwardingSmtp.dataStart()
forwardingSmtp.dataWrite(remodeledEmail, remodeledEmail.size)
forwardingSmtp.dataEnd()
return "OK message processed."
}
private fun modifyMessageSubjectAddOriginalRecipients(data: InputStream?): ByteArray {
val session = Session.getInstance(Properties())
val message = MimeMessage(session, data)
message.setSubject("[TESTMAIL] " + message.getSubject() + " [original RCPT TOs (including bcc): ${this.to}]")
val outputStream = ByteArrayOutputStream()
message.writeTo(outputStream)
val remodeledEmail = outputStream.toByteArray()
return remodeledEmail
}
private fun filterWhiteListAndAddDefault(to: ArrayList<String>): ArrayList<String> {
val ret = ArrayList<String>()
if (catchAllMailbox != null) {
ret.addAll(catchAllMailbox)
}
logger.debug("added $catchAllMailbox to $ret")
ret.addAll(to.filter { email ->
whitelistMailboxes?.any {
it.toRegex().matchEntire(email)?.range == IntRange(
0,
email.length - 1
)
} == true
})
logger.debug("add Filtered list of $to to ret, result: $ret")
return ret
}
The MimeMessage class is used in this example to parse the email data and handle the message as needed.
Let us write a little test for this configuration. Greenmail, an integrated email testing framework with SMTP, POP3 and IMAP support, was installed, and two test accounts were created:
private fun greenMailServerSetupPair(): Pair<GreenMail, ServerSetup> {
val greenMailWithDynamicPort = GreenMail(ServerSetupTest.SMTP_POP3_IMAP)
greenMailWithDynamicPort.start()
// We have 2 potential Mailboxes to verify
UserUtil.createUsers(greenMailWithDynamicPort, InternetAddress("to@interhyp.de"))
UserUtil.createUsers(greenMailWithDynamicPort, InternetAddress("catchAllMailbox@testing.de"))
val serverSetup = greenMailWithDynamicPort.smtp.serverSetup
return Pair(greenMailWithDynamicPort, serverSetup)
}
Add two helper functions at this point: one to confirm that a mailbox has received a specific number of emails, the other to send the email to a particular server.
@Throws(MessagingException::class, IOException::class)
private fun retrieveAndCheck(server: AbstractServer, login: String, expectedMessages: Int) {
Retriever(server).use { retriever ->
val messages: Array<Message> = retriever.getMessages(login)
assertEquals(expectedMessages, messages.size)
}
}
private fun sendTestMail(smtpServer: SMTPServer) {
logger.info("called smtpTest with Params: $smtpServer")
val localServerSetup = ServerSetup(smtpServer.portAllocated, "127.0.0.1", "smtp")
GreenMailUtil.sendTextEmail(
"to@interhyp.de",
"from@test.de",
"Connectivity TestMail Subject",
"Just a technical minimal test email.",
localServerSetup
)
}
In our test, we will send the mail to our proxy, which is configured to only forward to one email, to observe how it arrives at greenmail. We will send the mail to „to@interhyp.de“ but expect it to be present only in the forwarding email „catchAllMailbox@testing.de.“
@Test
fun test() {
val (greenMailWithDynamicPort, greenMailServerSetup) = greenMailServerSetupPair()
val testFactory = ReplacingRecipientsMessageHandlerFactory(
greenMailServerSetup.getBindAddress(),
greenMailServerSetup.port.toString(),
listOf("catchAllMailbox@testing.de"), null
)
val smtpServer = SMTPServer.port(0).messageHandlerFactory(testFactory).build()
smtpServer.start()
logger.info("Starting Basic SMTP Server on port ${smtpServer.port} / ${smtpServer.portAllocated}...")
try {
sendTestMail(smtpServer)
assertEquals(greenMailWithDynamicPort.receivedMessages.first().content)
assertEquals(
1,
greenMailWithDynamicPort.receivedMessages.first().allRecipients.size,
"recipients not as expected"
)
assertEquals(
// message still holds recipient but should not be delivered
"to@interhyp.de",
(greenMailWithDynamicPort.receivedMessages.first().allRecipients.first().toString()),
"only should forward to catchAll Account in this setup"
)
retrieveAndCheck(greenMailWithDynamicPort.imap, "to@interhyp.de", 0)
retrieveAndCheck(greenMailWithDynamicPort.imap, "catchAllMailbox@testing.de", 1)
retrieveAndCheck(greenMailWithDynamicPort.pop3, "to@interhyp.de", 0)
retrieveAndCheck(greenMailWithDynamicPort.pop3, "catchAllMailbox@testing.de", 1)
} finally {
greenMailWithDynamicPort.stop()
}
}
As a result, we see that this test turns green as expected. It is important to be cautious about where to expect what, as Greenmail delivers the mail correctly but appears to lack direct verifications for the true „RCPT TO“ recipients.
Advantages
We see many benefits of this strategy. Initially it ensures that no external emails are sent from the system while testing emails and layouts in as many actual email clients as possible. Second, it sends the email to designated test-email addresses as well as central test-mailboxes while maintaining the precise email structure. It doesn’t require any special coding in the mail sending program and is simple to set up. So, it fits our need to make sure emails are sent accurately and with the necessary information.
Conclusion
It is crucial that emails are sent out successful and with the appropriate content is testing email functionality during the pre-production phase. While there are a number of widely used methods for testing email functionality, we use a fresh method that so far has proven to be very successful. We can make sure that we send emails correctly and with the appropriate content by utilizing a customized SMTP proxy server that filters out all external emails or sends the filtered emails to central mailboxes. We hope that this will also help other developers and we welcome criticism of this setup.
Next Steps
One advantage of using a custom microservice for this purpose is that we can easily extend it for non-functional testing. For example, we can now introduce slow response times or protocol errors to test how our system handles these situations.
It is also important to implement rate throttling by external domains to prevent load-test spamming and thus avoid negative reputation on the SMTP servers. This will help us to maintain a good sender score other email providers and prevent our emails from being marked as SPAM.
Another area we plan to look into is sending and receiving Delivery Status Notification (DSN) messages. These are automated messages that notify senders of delivery problems, such as a non-delivery notification (NDN). Ignoring those might also lower the server SPAM score. By processing these messages, we can verify that our email functionality is working correctly and that our system is able to automatically handle delivery problems.
Finally, we will continue to add tests to ensure that our custom SMTP proxy server is robust and reliable.
By continually improving and expanding our custom SMTP proxy server, we can be confident in the quality and reliability of our email functionality and ensure that our users are able to send and receive emails without issues.