Initially, it was just a guess, based on the reported error message.
Then I started to suspect that was the problem.  (Knowing that the host
names used in the config were in the SAN list, and that the canonical
name wasn't.)  Then, one of our grad students found these forum posts...

Issue: https://support.google.com/mail/thread/38789651?hl=en
Recommended answer:
https://support.google.com/mail/thread/38336515?msgid=39890656

... which confirmed my suspicions.  Adding the PTR record fixed the
problem, and further confirmed my suspicions and what was claimed in the
forum posts.

Incidentally, cranking the sendmail LogLevel past 11 (and all the way up
to 14) didn't shed any extra light.  All we saw was that connections
from google.com servers would start TLS negotiations, and then just
disconnect before issuing any SMTP commands.  I had to rely on users to
send me error messages from their bounce messages to see what was
failing on the client side, i.e. Google's side:

TLS Negotiation failed, the certificate doesn't match the host.

Not much to go on!

Gilbert

On 2020-04-20 10:03 a.m., Hartmut W Sager wrote:
> How did you (and your work colleagues) come to realize that the RDNS
> name is what Google cert is trying to match?
>
> Hartmut W Sager - Tel +1-204-339-8331
>
>
> On Mon, 20 Apr 2020 at 09:20, Gilbert E. Detilllieux
> <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>> wrote:
>
>     Yeah, we got bitten by this one at work last week (actually started
>     noticing problems around Wednesday or Thursday of the previous week,
>     but
>     only resolved it last week).  We had some users within the department
>     who wanted to manage their CS e-mail via Gmail, and had set up their
>     Gmail to send out via our SMTP server, which had worked fine until
>     Google unilaterally decided on this new restriction, which most likely
>     violates several standards.
>
>     Since we control reverse DNS on our domain, I was able to fudge things
>     up to get Google to accept our cert (a legitimate cert, issued by
>     Globalsign, that has multiple generic aliases in the SAN list, but
>     intentionally avoided the canonical host name, since these generic
>     aliases should be allowed to migrate to different physical servers,
>     transparently).  I found out that you can have multiple PTR records on
>     one IP address, which is completely legal in DNS, but not usually
>     considered good practice (or so I thought).  Of course, this second PTR
>     record caused some things to fail in a non-deterministic way, since
>     lookups on the IP address gave the PTR records in pseudo-random order,
>     causing code that only looked at the first answer to get inconsistent
>     results.  Grrr!
>
>     Thanks, Google, for once again messing with standards, and forcing
>     everyone else to bend to your will!
>
>     Gilbert
>
>     On 2020-04-18 3:14 p.m., Hartmut W Sager wrote:
>     [... deleted ...]

--
Gilbert E. Detillieux        E-mail:  <gedetil@cs.umanitoba.ca>
Dept. of Computer Science    Web:     http://www.cs.umanitoba.ca/~gedetil/
University of Manitoba       Phone:   (204)474-8161
Winnipeg MB CANADA  R3T 2N2  Fax:     (204)474-7609