How to authenticate for any VPN Connection by certificates with the built-in MacOS VPN-Client?
I'm using a Zyxel ZyWall 110 and I want to establish a client-to-side VPN connection to the ZyWall by using the built-in VPN-Client from MacOS 10.14.6 (Mojave).
So far I was able to get successful connections with IKEv2 and L2TP/IPSec, but all of them use a username/password client authentication. That's not what I want since such authentications are vulnerable by dictionary and brute force attacks. My goal is to use client certificates issued by a self-created certification authority to authenticate the clients.
IKEv2, which is preferred by me to use in the further network configuration, works and the server is able to authenticate itself by a certificate. For client authentication I have to use EAP-MSCHAPv2, because the ZyWall do not supports EAP-TLS. If I do not activate EAP, other clients such Linux strongSwan are able to connect by a certification based authentication but not the MacOS built-in VPN-Client.
After that I tried to use a RADIUS server to authenticate the EAP-TLS request from the MacOS VPN-Client to bypass the not supported EAP-TLS. The RADIUS server successful authenticate the client and give that response to the ZyWall but after that, the ZyWall does not anything with that, so the client get no response and no connection can be established. That behaviour of the ZyWall is the same with the Linux Strongswan VPN-Client. But if the VPN server is a Linux strongSwan too, the RADIUS server EAP-TLS authentication works perfectly.
I also get a woking L2TP/IPSec connection, but I was not able to implement a certificate based authentication for server nor clients as well. Both, the machine authentication and the user authentication, does not work with the certificates. It is only possible to establish that connection while using the PSK for machine authentication and username/password for user authentication. In the ZyWall-logs you can read "Authentication mismatch" and the connection will not be established.
- ZyWall firmware: 4.33(AAAA.0)C0
- used as PPPoE access to ISP with dynamic public IP (updated by DDNS from ZyWall)
- Different MacBooks with MacOS 10.14.6 Mojave and iPhones with iOS 12.4
- all clients were in a ZyWall independent network at the time of the connection attempts but behind a NAT router of course.
- freeRADIUS: 3.0.17 (on Raspberry Pi, Raspbian Buster)
- while using the RADIUS server I have set the authentication server
- strongSwan: U5.5.1/K4.19.57-v7l+ (on Raspberry Pi, Raspbian Stretch)
- Certificates issued by OpenSSL
- Certificates from ROOT CA and Intermediate CA are installed on all machines and marked as trusted, so that verifying the certificates were never a problem
- ZyWall certificate was created as CSR on the Zywall and signed by the Intermediate CA:
- CN = hostname.domain.tld
- X509v3 Key Usage: critical: Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
- X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication, iKEIntermediate
- All DNS-Names are in the X509v3 Subject Alternative Name listet starting with the DDNS-Name
- Client certificates are also signed by the same Intermediate CA:
- CN = first and last name of the user
- X509v3 Key Usage: critical: Digital Signature, Non Repudiation, Key Encipherment
- X509v3 Extended Key Usage: TLS Web Client Authentication, E-mail Protection (since the client certificates are also used for S/MIME)
- At this point I have to say that I tried a lots of different settings for the certificates to rule out that the problems are the certificates itself such as giving TLS Web Server Authentication and iKEIntermediate to the client certificates too and using self-signed root certificates from the ZyWall. In the End I do not think that the certificates are the reason for the authentication problems but I am ready for all ideas.
Since this is the first time I need to use this forum because I am really stuck with that, I hope you can help me. After 4 weeks, I have no idea how to proceed. The last resort is that I continue to use the currently working OpenVPN infrastructure, which I actually wanted to replace with the ZyWall.