Have you tried to grant connect permission to the endpoints on the primary and the mirror?
GRANT CONNECT ON ENDPOINT ::Mirroring TO [NT AUTHORITY\SYSTEM] --use yours here
GRANT CONNECT ON ENDPOINT ::Mirroring TO [NT SERVICE\MSSQLSERVER]
GRANT CONNECT ON ENDPOINT ::Mirroring TO [NT SERVICE\SQLSERVERAGENT]