Sending, receiving, and organizing email in Outlook.com
I found the solution. There are two issues with Microsoft's implementation of the MTAs for Outlook.com
I am using Letsencrypt certificates and DANE. DANE supports four "usages" for certificates: PKIX TA (0), PKIX EE (1), DANE TA (2) and DANE EE (3) (see RFC 6698, RFC 7671). A relying party should accept a certificate, if the relying party find any usable and validating TLSA record (in case there are multiple). This is necessary for smooth roll-overs. Additionally, RFC 7672, Sec. 3.1 recommends usage type 2 or 3 for SMTP applications (i.e. those which do not require a PKIX trust anchor). The reason for that recommendation is that with the PKIX-based usage type, the TA must additionally be installed in the trusted certificate store of the relying party.
Issue #1
In order to only manage my TLSA record once, I have been using a setup with CNAME records (for all of my services) which all pointed to the same set of TLSA records, i.e. my DNS records were
_25._tcp.mail.my-domain.tld. 14400 IN CNAME letsencrypt._dane.my-domain.tld.
_443._tcp.www.my-domain.tld. 14400 IN CNAME letsencrypt._dane.my-domain.tld.
letsencrypt._dane.my-domain.tld. 14400 IN TLSA 0 1 1 0B9FA5A59EED715C26C1020C711B4F6EC42D58B0015E14337A39DAD301C5AFC3 # ISRG Root X1 as PKIX-TA
letsencrypt._dane.my-domain.tld. 14400 IN TLSA 0 1 1 762195C225586EE6C0237456E2107DC54F1EFC21F61A792EBD515913CCE68332 # ISRG Root X2 as PKIX-TA
letsencrypt._dane.my-domain.tld. 14400 IN TLSA 2 1 1 2BBAD93AB5C79279EC121507F272CBE0C6647A3AAE52E22F388AFAB426B4ADBA # Lets Encrypt Intermediate R10 as DANE-TA
letsencrypt._dane.my-domain.tld. 14400 IN TLSA 2 1 1 6DDAC18698F7F1F7E1C69B9BCE420D974AC6F94CA8B2C761701623F99C767DC7 # Lets Encrypt Intermediate R11 as DANE-TA
letsencrypt._dane.my-domain.tld. 14400 IN TLSA 2 1 1 3586D4ECF070578CBD27AEDCE20B964E48BC149FAEB9DAD72F46B857869172B8 # Lets Encrypt Intermediate E5 as DANE-TA
letsencrypt._dane.my-domain.tld. 14400 IN TLSA 2 1 1 D016E1FE311948ACA64F2DE44CE86C9A51CA041DF6103BB52A88EB3F761F57D7 # Lets Encrypt Intermediate E6 as DANE-TA
Microsoft's MTAs refuse to communicate, but immediately closes the SMPT session again, if they encounter a TLSA record with usage type 0.
This means Microsoft over-interprets the recommendation to not use usage type 0 as a strict "shall not" and actively closes the connection again. If one disables DANE altogether (for testing purposes only), the MTA are able to establish a secure connection, hence, the ISRG root certificate are actually trusted by Microsofts MTAs which make things even more surprising, because it defeats the only reason for the recommendation not to use PKIX-TA. Moreover, in doing so, Microsoft MTAs violate the the rule that a single matching TLSA record is sufficient, because the MTA are also able to validate one of the intermediates as DANE-TA.
The solution is to separate the TLSA records for WWW and all mail services like this:
_25._tcp.mail.my-domain.tld. 14400 IN CNAME mail._dane.my-domain.tld.
_443._tcp.www.my-domain.tld. 14400 IN CNAME www._dane.my-domain.tld.
www._dane.my-domain.tld. 14400 IN TLSA 0 1 1 0B9FA5A59EED715C26C1020C711B4F6EC42D58B0015E14337A39DAD301C5AFC3 # ISRG Root X1 as PKIX-TA
www._dane.my-domain.tld. 14400 IN TLSA 0 1 1 762195C225586EE6C0237456E2107DC54F1EFC21F61A792EBD515913CCE68332 # ISRG Root X2 as PKIX-TA
mail._dane.my-domain.tld. 14400 IN TLSA 2 1 1 2BBAD93AB5C79279EC121507F272CBE0C6647A3AAE52E22F388AFAB426B4ADBA # Lets Encrypt Intermediate R10 as DANE-TA
mail._dane.my-domain.tld. 14400 IN TLSA 2 1 1 6DDAC18698F7F1F7E1C69B9BCE420D974AC6F94CA8B2C761701623F99C767DC7 # Lets Encrypt Intermediate R11 as DANE-TA
mail._dane.my-domain.tld. 14400 IN TLSA 2 1 1 3586D4ECF070578CBD27AEDCE20B964E48BC149FAEB9DAD72F46B857869172B8 # Lets Encrypt Intermediate E5 as DANE-TA
mail._dane.my-domain.tld. 14400 IN TLSA 2 1 1 D016E1FE311948ACA64F2DE44CE86C9A51CA041DF6103BB52A88EB3F761F57D7 # Lets Encrypt Intermediate E6 as DANE-TA
Issue #2
Currently, Letsencrypt has four active intermediate certificates (E5, E6, R10, R11) which are randomly used to issue subscriber certificates. Moreover, there are six backup certificates (E7, E8, E9 and R12, R13, R14) in case one of the active certificates needs to be revoked. As a safety measure I already had added TLSA records for all intermediates, not only the four active certificates as shown above. Hence, I had 12 TLSA records originally. It appears that there seems to be an upper limit in Microsoft's MTAs on how many TLSA records they parse. After I trimmed down the list the the currently relevant, four certificates, it started working.