State: Draft

Introduction

This Testing Guide provides a plan for how one would test that their implementation is compliant with the Direct Project Specification. This Testing Plan consists of validating that your implementation can interoperate with at least three other implementations of various Deployment Models. The goal is to prove heuristically compliance with the Direct Project Specification.

Testing Overview

The system being tested is referred to as the "System Under Test" (SUT). It is expected that this is a complete system capable of Sending and Receiving using the Direct Project Specification. We use 'system' as the deployment models show any Sending side or any Receiving side is often made up of multiple distinct components. It is the whole set of components working together from end-user experience to the point at which the Direct Project Specifications that is collectively referred to as the System Under Test (SUT).

This Test Guide instructs the tester to test the SUT against at least three other implementations of a different type. These other implementations will be referred to as the "Other Test System" (OTS). By testing against three other systems we get a good heuristic to show that the Direct Project Specification was implemented correctly.

Test Plan

Pre-Conditions: Prior to testing the following must be available.

  • Endpoint Addresses must be known and recorded
  • Endpoint Digital Certificates must be issued
  • Endpoint Digital Certificates must be published in a way that the SUT can discover (sometimes this is manual, LDAP, e-mail, or DNS)
  • SUT must have basic network connectivity to the OTS (this can be over the internet, or in a lab)


Test Scenarios


Preconditions


Trust anchors: A0, B0,
Valid certificate paths:
A0->A1->A2
A2 and A1 have AIA extensions providing the chain, A1 is retrievable via HTTP, chain not precomputed at SUT
A0->A3
B0->B1

Invalid certificates:
E1 (has AIA but path not retrievable)
E2 (has AIA, path retrievable)
A0->E3 (expired, via CRL)
A0->E4 (expired, via OCSP)

Certificate bindings (published in DNS or available in local trust store)
sunny.example.org, bound to A3
happy.example.org, bound to A2
bob@valley.example.org, bound to B1

Positive Tests

Test
Scenario
Expected behavior
Same address
Organizational certificate
user1@sunny.example.org receives message[1] from user1@sunny.example.org
Success, delivery, MDN
Same certificate path
| user1@sunny.example.org receives valid signed and encrypted message from user2@sunny.example.org
Success, delivery, MDN
Different mutually trusted certificate paths
Address bound certificate
user1@sunny.example.org receives valid signed and encrypted message from bob@valley.example.org
Success, delivery, MDN
AIA certificate path building
user1@sunny.example.org receives valid signed and encrypted message from user_a@happy.example.org
Success, delivery, MDN

Negative Receipt Tests


Please note that many of these tests will intentionally require the system to receive a message that should be impossible to generate by the same system. Nothing in these these tests should imply that the SUT should create the negative test messages.

Test
Scenario
Expected behavior
Unencrypted payload
user1@sunny.example.org receives unencrypted message
No delivery, no MDN, Audit, error retrievable at SUT
Encrypted, no signature
user1@sunny.example.org receives encrypted message without a signature
No delivery, no MDN, Audit, error retrievable at SUT
Encrypted, bad signature
user1@sunny.example.org receives encrypted message with corrupt signature
No delivery, no MDN, Audit, error retrievable at SUT
Message not verified
user1@sunny.example.org receives encrypted message where content does not match signature
No delivery, no MDN, Audit, error retrievable at SUT
Corrupt encryption
user1@sunny.example.org receives corrupted encrypted message
No delivery, no MDN, Audit, error retrievable at SUT
Chain can not be built
user1@sunny.example.org receives valid signed and encrypted message signed by E1
No delivery, no MDN, Audit, error retrievable at SUT
Untrusted trust anchor
user1@sunny.example.org receives valid signed and encrypted message signed by E2
No delivery, no MDN, Audit, error retrievable at SUT
Expired certificate
user1@sunny.example.org receives valid signed and encrypted message signed by E3
No delivery, no MDN, Audit, error retrievable at SUT
Expired certificate
user1@sunny.example.org receives valid signed and encrypted message signed by E4
No delivery, no MDN, Audit, error retrievable at SUT

Negative Send Tests


Test
Scenario
Expected behavior
Chain can not be built
user1@sunny.example.org attempts to send to address bound to E1
No send, notification to sender, audit, error retrievable at SUT
Untrusted trust anchor
user1@sunny.example.org attempts to send to address bound to E2
No send, notification to sender, audit, error retrievable at SUT
Expired certificate
user1@sunny.example.org attempts to send to address bound to E3
No send, notification to sender, audit, error retrievable at SUT
Expired certificate
user1@sunny.example.org r attempts to send to address bound to E4
No send, notification to sender, audit, error retrievable at SUT

Additional Negative Send Tests for Service Model


These tests deal with an organization operating as a Service Model Security User Agent.

Test
Scenario
Expected behavior
Unauthenticated Sender
Unauthenticated user attempts to send using SUT
No send, no notification to sender, audit, error retrievable at SUT
Receiver not managed by SUT
SUT receives encrypted payload for list of receivers none of whom are managed by SUT
No receipt, audit, error retrievable at SUT
SUT receives unencrypted payload
SUT receives unencrypted payload over an unencrypted channel (e.g., meant for other security user agents to send encrypted content)
No receipt, audit, error retrievable at SUT


Procedure

For each pairing of the SUT against each OS the overall procedure is:
  1. Test the success scenarios where address, digital certificate, S/MIME, and content packaging are all functioning as expected
    1. send a document using the SUT to the OTS
    2. send a document using the OTS to the SUT
  2. Test the failure scenarios to assure that they do not expose the document inappropriately and fail gracefully. Since some 'systems' will not allow the user to do some of these tests, it is acceptable to indicate that the system prevented the test
    1. If possible try to send a document from the SUT to the OTS without using the security provided.
    2. If possible try to send a document from a non-S/MIME e-mail client so that it is delivered as an attachment, but without security (S/MIME)
  3. Test the options
    1. Configure the SUT to use TLS when connecting to the remote or internet mail server (this applies to both outbound SMTP as well as inbound SMTP/IMAP/POP as appropriate to the SUT deployment model).
    2. send a document using XDM zip file from the SUT to the OTS
    3. send a document using XDM zip file from the OTS to the SUT

Candidate Other Test system

The system under test will be tested against at least three of the following, preferably from different deployment models:

Test Report


The Direct Project does not record success or failure of testing, but the test results should be documented and provided with any distribution or statements about the System Under Test (SUT).

  • What Other Test Systems was it tested against?
  • What modes of digital certificate distribution were tested?
  • Was XDM option tested?
  • Was TLS option tested?
  • Document any test failures

  1. ^ Implementations need not sign and encrypt same-domain messages if they have local mechanisms for providing the same services that signing and encryption provides