Hi Hyperlance,
I had the same issue.
I also discovered a change in behaviour after moving to SQL 2022.
Our jobs have multiple steps, and I don't want to modify all of those.
Luckily the change is simple:
Replace the 'raiserror'-part by 'throw', just like this:
IF sys.fn_hadr_is_primary_replica('<dbname>') <> 1
THROW 50000, 'This is not the primary replica.', 1;
Add this line of script in the first step of the job and set that to 'fail the job showing success'.
Good luck!