SQL Server Always On provides a fairly robust environment for disaster recovery; however you can never be too careful when it comes to technology and protecting your data. Regular maintenance routines (i.e. backups, etc.) need to be set up and performed on an ongoing basis, even when you have your databases in an Always On Availability Group. 

Always On allows some maintenance routines to be performed on the secondary replicas, but determining which maintenance to perform on what replica can be somewhat confusing. We will discuss the following basic maintenance routines and where to execute them.

• Backups
• Integrity Checks
• Index Maintenance
• Update Statistics

Backups

Even though there is a copy of your database on the secondary replica(s), you should still continue to back up your databases and transaction logs on a regular basis. Backups can be very resource intensive by putting strain on I/O and CPU especially when using compression. Because of this, offloading your backup operations to a secondary replica can greatly improve the performance of your primary production replica.   

With an Always On group, you have several options of where to back up your databases as shown below. You can choose your backup preference when you set up your Always On group, or you can go back later and set your backup preferences through the AG properties.

Prefer Secondary - backups should occur on a secondary replica except when the primary replica is the only replica online. In that case, the backup should occur on the primary replica. This is the default behavior. 

Secondary Only - backups will happen only on a replica that is designated as a secondary, and will execute in the order of the backup priority you set. Backups will not occur on the primary replica.

Primary – backups will only occur on the replica currently designated as primary.

Any Replica – backups will occur on the replica with the highest backup priority setting.

If you perform backups on your primary replica, Full, Differential, and Tlog backups can be run normally. If you perform backups on a secondary replica, there are some differences to be aware of:

• You must configure your Always On group to specify where you want your backups to occur. You can do this through the New Availability Group setup GUI (as shown in picture above), or by using TSQL script (for more info click here https://msdn.microsoft.com/en-us/library/ff878399.aspx).  These backup preferences will apply to both Full and Tlog backups.
• The automated backup preference setting has no impact on ad-hoc backups. Manual ad-hoc backups will occur on whatever replica you perform the backup.
• If you create your backup jobs through a Maintenance Plan, create the jobs on all replicas that have any intention of potentially performing backups. The job will automatically include the scripting logic that calls and checks the sys.fn_hadr_backup_is_preferred_replica function, and will not execute if the replica is not the preferred backup location. The job will not fail, it simply won’t execute. And in the event of failover the job will already exist on the replica the Always On Group has failed over to, and will execute accordingly.
• Full Backups run on secondary replicas are copy only backups. Regular backups are not allowed. Copy-only backups do not impact the log chain or clear the differential bitmap.
• Differential backups are not allowed on secondary replicas.
• Tlog Backups only support regular backups on a secondary replica (copy only is not allowed). Tlog backups can be taken on any replica regardless of synchronization mode, and the log chain will remain consistent.
• If databases exist in your SQL instance that are not part of the Always On Group, create separate maintenance jobs for those databases apart from the Always On databases so that the sys.fn_hadr_backup_is_preferred_replica function is not invoked. This way your non-Always On databases will not get lumped in with the Always On databases. Having separate maintenance routines for separate groups of databases can get a little hairy, as well as managing the jobs that need to run on each replica…for a great article on this click here https://blog.sqlrx.com/2015/10/08/Always On-availability-groups-enable-or-disable-jobs-on-failover/
• Since the instances on the Always On replicas do not share an msdb database backups will not show up in the msdb backup tables.   This is important to know in the event you have to perform audits, or need to research backup history.

Integrity Checks

Perform Integrity Checks on all replicas if possible.

There are many differing opinions in the IT world on where to run the DBCC CHECKDB in an Always On Group. Some believe integrity checks should be run on the secondary only, some believe it should be run wherever your database backups are happening, and some believe it should be run on all replicas.

SQL Server allows integrity checks to be offloaded to any of the secondary replicas in order to prevent resource contention on the primary replica, which keeps performance optimized. This is a good idea in theory, however if there is corruption in the data files of the primary database (assuming you have offloaded your integrity checks to a secondary) it will not be detected by the integrity check. Even though data in the databases is the same across all replicas, the actual data files reside on the separate disk subsystems of each replica. The integrity check runs against the data files on each disk subsystem, and if your hardware or a file is corrupt on one of your replicas the integrity check run on another replica will not catch it. 

Corruption on a primary does not transfer to the secondary because complete database pages are not sent, only the tlog records that only describe the physical changes made to the pages…they don’t contain the actual pages.  Performing an integrity check on a primary replica doesn’t detect corruption on a secondary and vise-versa. They are separate databases with their own data files kept in sync through transaction logs that describe the physical changes.

If your organization is extremely sensitive to performance for your Always On group, we recommend you restore a copy of your database backups to a server not in your Always On group (like a test server) and run the integrity checks there. This will provide a consistency check of your data, will ensure your backups are viable, and will eliminate overhead of running CHECKDB on the primary.

Index Maintenance

Perform Index Maintenance on the primary replica

Because secondary databases are read-only, index maintenance must be performed on the primary replica. Index maintenance generates large amounts of logged changes by design and consequently will result in large transaction logs being replicated to the secondaries. Also index maintenance can cause the primary log files to grow large because the log cannot be truncated until redo has completed the changes in all secondary replicas.  

Some ways to reduce resource contention for index maintenance is:

1. Schedule during off peak times if possible
2. Break up maintenance into smaller phases, spread out over time period
3. Use MAXDOP to limit the number of processors used for the operation
4. Rebuild/reorganize based upon specified criteria:
   1. Ignore heaps or small tables
   2. Rebuild/reorganize based upon fragmentation levels acceptable to your environment

If you are fortunate enough to have a large maintenance window to be able to rebuild your indexes on a daily basis, there is not much need to update statistics as rebuilding indexes does this by design. The frequency of your maintenance routines will depend on each individual SQL environment. As a general rule with our clients, we rebuild indexes once per week and update statistics every day…but again, your environment may require something different.

Update Statistics

Update Statistics on the primary replica.

SQL Server creates statistics on database tables and indexes that gives the query optimizer information about the distribution of data. This allows SQL to compile the most efficient query plan in order to improve query performance and run the queries as quickly as possible.  

In an Always On Group of databases, statistics that are created and updated on the primary replica are persisted to the secondary replicas as part of the data replication of the transaction logs records. 

But what if you’re using your secondaries for read-only operations and the secondary databases need different statistics than the primary?  Databases on the secondary replicas are restricted to read-only access so statistics cannot be created on them!  If the secondary replicas cannot create statistics, then performance of read-only workloads can suffer.  

Thankfully SQL will create and maintain statistics on the secondary replicas in tempdb by using the suffix _readonly_database_statistic appended to the name of the temporary statistic. This keeps it separate from the permanent statistics that have been sent over from the primary. These temporary statistics can only be created by SQL Server, and you need to allow auto update and auto creation of statistics. 

For more information on SQL statistics, click here https://msdn.microsoft.com/en-us/library/ms190397.aspx

Conclusion

This article covers a high-level overview for performing some basic maintenance on your Always On databases.  These are suggestions based on a simple Always On environment. There are always exceptions to these rules and circumstances where a system is set up in a more complex manner. For example, you may want Tlog backups to occur on a different replica than the Full backups. You may have your Always On group also involved in log shipping or some other functionality that requires more or different maintenance routines. There are methods to accommodate every environment, but is beyond the scope of this article.  Hopefully these topics have given you a basic foundation for maintaining your Always On environment.