Closing the loop here- After almost a year of mostly unproductive back-and-forth on a MS ticket, we finally got the straight answer we suspected (paraphrasing MS response )

• ...by design availability...

We just pushed CU22 and are still experiencing the problem.  We had 3 dbs that stopped synchronizing even after doubling CPU and RAM per Microsoft recommendations.  We are uploading logs...

We upgraded to CU21 in mid-July and we are still having the issue.  Fingers crossed for CU22 which we will apply in early/mid September.

MS is recommending we double our CPU...

Interesting article, thank you.  The error returned 1 for the state.

Our dev team updated the code that generates the EF query, and that has resolved the problem (so I won't...

Thanks Jeff for the reply.  Cardinality Estimation is set the same on both databases and changing it doesn't resolve the issue.  All server and database options that you mention (and...

Just an update that we are still at a dead-end on this with Microsoft.  They claim that it's due to our primary and secondary nodes being on different patch levels,...

Thanks Per4inka, I'll share this in our DBA team meeting tomorrow.

BTW we STILL have an open ticket with MS on this so I'll be sure to post back with whatever...

We are on CU19 and will go to CU20 in about 3 weeks.

Alex S - thanks for the suggestion.  It will be difficult to test as we don't have a...

Yes, we are using Automatic seeding for our AGs.

Our reboots most often are due to patching, so we reboot many secondaries at the same time and yes, we always have...

Yes we have combed the logs pretty thoroughly and can't find anything definitive.  We have a ticket open with MS so I'll post back here with whatever they report.

No they are not Sharepoint DBs.   50 DBs/AG is relatively arbitrary, that's just where we landed with our capacity testing when we first implemented AGs about 5 years ago.

Rebooting the...

No, 300 dbs on a server.  About 50 dbs/AG.

Yes, we have ~300 dbs/server and indeed we do see thread exhaustion errors.  We are working on an initiative to reduce the number of databases but it's a slow process.

Thanks...

Apologies that I am slow in replying, thanks to those who replied.  I did query the default trace files, unfortunately they did not go back far enough to capture the...

Just an update... CHECKDB found 0 allocation errors and 0 consistency errors in database and we are unable to reproduce the error on a copy of the db.  I guess we'll...