Cannot bring the Windows Server Failover Clustering (WSFC) resource (ID '38e4748b-1b06-4017-9c3a-e7eb3f45cb2c') online (Error code 5018).

Question

Cannot bring the Windows Server Failover Clustering (WSFC) resource (ID '38e4748b-1b06-4017-9c3a-e7eb3f45cb2c') online (Error code 5018).

databases.specialist

SSC Veteran

Points: 235
More actions
January 13, 2019 at 11:34 pm

#373546

I am getting below error messages...granted the sysadmin to the respective login and granted the privileges but still getting below error, attached the cluster log.
TITLE: Microsoft SQL Server Management Studio
------------------------------
Create failed for Availability Group 'OST-MOD-DBS-AG1'. (Microsoft.SqlServer.Management.HadrModel)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=14.0.17119.0+((SSMS_Rel).170517-0412)&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Create+AvailabilityGroup&LinkId=20476
------------------------------
ADDITIONAL INFORMATION:
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
------------------------------
Cannot bring the Windows Server Failover Clustering (WSFC) resource (ID '38e4748b-1b06-4017-9c3a-e7eb3f45cb2c') online (Error code 5018). The WSFC service may not be running or may not be accessible in its current state, or the WSFC resource may not be in a state that could accept the request. For information about this error code, see "System Error Codes" in the Windows Development documentation.
Failed to designate the local availability replica of availability group 'OST-MOD-DBS-AG1' as the primary replica. The operation encountered SQL Server error 41066 and has been terminated. Check the preceding error and the SQL Server error log for more details about the error and corrective actions.
Failed to create availability group 'OST-MOD-DBS-AG1'. The operation encountered SQL Server error 41160 and has been rolled back. Check the SQL Server error log for more details. When the cause of the error has been resolved, retry CREATE AVAILABILITY GROUP command. (Microsoft SQL Server, Error: 41066)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%20SQL%20Server&ProdVer=13.00.4001&EvtSrc=MSSQLServer&EvtID=41066&LinkId=20476
------------------------------
BUTTONS:
OK
------------------------------
000022a4.000021f0::2019/01/14-06:15:55.470 INFO [RES] Network Name: Agent: Sending request Netname/RecheckConfig to NN:071916e4-38c1-4e44-a2ec-dadf25cd11ca:Netbios
00002514.00000ed8::2019/01/14-06:15:56.915 INFO [API] s_ApiGetQuorumResource final status 0.
00002514.00000ed8::2019/01/14-06:15:56.915 INFO [API] s_ApiGetQuorumResource final status 0.
00002514.00000ed8::2019/01/14-06:15:56.915 WARN [API] s_ApiOpenResourceEx: Resource not found, status = 5007
00002514.00002d30::2019/01/14-06:15:57.960 INFO [RCM] rcm::RcmApi::CreateGroup: (OST-MOD-DBS-AG1, a0bbb6d1-2ea3-4b86-badb-1fa4e414a7ba)
00002514.00002d30::2019/01/14-06:15:57.960 INFO [GUM] Node 1: executing request locally, gumId:302, my action: /rcm/gum/CreateGroup, # of updates: 1
00002514.00002d30::2019/01/14-06:15:57.960 INFO [RCM] rcm::RcmGum::CreateGroup(OST-MOD-DBS-AG1,1,a0bbb6d1-2ea3-4b86-badb-1fa4e414a7ba)
00002514.00000ed8::2019/01/14-06:15:57.960 INFO [RCM] moved 0 tasks from staging set to task set. TaskSetSize=0
00002514.00000ed8::2019/01/14-06:15:57.960 INFO [RCM] rcm::RcmPriorityManager::StartGroups: [RCM] done, executed 0 tasks
00002514.00002d30::2019/01/14-06:15:57.961 INFO Received notification on group that was not findable: a0bbb6d1-2ea3-4b86-badb-1fa4e414a7ba
00002514.00002d30::2019/01/14-06:15:57.961 INFO Received notification on group that was not findable: a0bbb6d1-2ea3-4b86-badb-1fa4e414a7ba
00002514.00002d30::2019/01/14-06:15:57.961 INFO Received notification on group that was not findable: a0bbb6d1-2ea3-4b86-badb-1fa4e414a7ba
00002514.00002d30::2019/01/14-06:15:57.961 INFO [RCM] CreateGroup GUM for group OST-MOD-DBS-AG1 (id=a0bbb6d1-2ea3-4b86-badb-1fa4e414a7ba) succeeded
00002514.00000ee0::2019/01/14-06:15:57.962 INFO [NM] Received request from client address 10.145.247.135.
00002514.00002a30::2019/01/14-06:15:57.963 INFO [NM] Received request from client address ::1.
00002514.00001760::2019/01/14-06:15:57.963 INFO [NM] Received request from client address ::1.
00002514.00002a30::2019/01/14-06:15:57.964 INFO [GUM] Node 1: executing request locally, gumId:303, my action: /dm/update, # of updates: 1
00002514.00002a30::2019/01/14-06:15:57.966 INFO [RCM] rcm::RcmApi::CreateResource: (OST-MOD-DBS-AG1, OST-MOD-DBS-AG1, 15f1bb3a-e124-4020-8fe8-c196ec89e155, SQL Server Availability Group)
00002514.00002a30::2019/01/14-06:15:57.966 INFO [GUM] Node 1: executing request locally, gumId:304, my action: /rcm/gum/CreateResource, # of updates: 1
00002514.00002a30::2019/01/14-06:15:57.966 INFO [RCM] rcm::RcmGum::CreateResource(OST-MOD-DBS-AG1,15f1bb3a-e124-4020-8fe8-c196ec89e155,OST-MOD-DBS-AG1)
000025c8.00001a8c::2019/01/14-06:15:57.966 INFO [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] Open request
00002514.00001760::2019/01/14-06:15:57.967 INFO [RCM] HandleMonitorReply: OPENRESOURCE for 'OST-MOD-DBS-AG1', gen(0) result 0/0.
00002514.00000ed8::2019/01/14-06:15:57.968 INFO [NM] Received request from client address 10.145.247.135.
00002514.00001760::2019/01/14-06:15:57.968 INFO [NM] Received request from client address ::1.
00002514.00002a30::2019/01/14-06:15:57.969 INFO [GUM] Node 1: executing request locally, gumId:305, my action: /dm/update, # of updates: 1
00002514.00002a30::2019/01/14-06:15:57.972 INFO [GUM] Node 1: executing request locally, gumId:306, my action: /dm/update, # of updates: 1
00002514.00002a30::2019/01/14-06:15:57.974 INFO [GUM] Node 1: executing request locally, gumId:307, my action: /dm/update, # of updates: 1
00002514.00000ed8::2019/01/14-06:15:57.976 INFO [GUM] Node 1: executing request locally, gumId:308, my action: /dm/update, # of updates: 1
00002514.00000ed8::2019/01/14-06:15:57.981 INFO [GUM] Node 1: executing request locally, gumId:309, my action: /dm/update, # of updates: 1
00002514.00000ed8::2019/01/14-06:15:57.983 INFO [GUM] Node 1: executing request locally, gumId:310, my action: /dm/update, # of updates: 1
00002514.00000ed8::2019/01/14-06:15:57.986 INFO [RCM] rcm::RcmApi::AddPossibleOwner: (OST-MOD-DBS-AG1, 1)
00002514.00000ed8::2019/01/14-06:15:57.986 INFO [GUM] Node 1: executing request locally, gumId:311, my action: /rcm/gum/AddPossibleOwner, # of updates: 1
00002514.00000ed8::2019/01/14-06:15:57.986 INFO [RCM] rcm::RcmGum::AddPossibleOwner(OST-MOD-DBS-AG1,1)
00002514.00000ed8::2019/01/14-06:15:57.986 ERR mscs::GumAgent::ExecuteHandlerLocally: (5010)' because of 'The specified node is already a possible owner.'
00002514.00000ed8::2019/01/14-06:15:57.986 WARN [DM] Aborting group transaction 2:2:268+1
00002514.00000ed8::2019/01/14-06:15:57.986 ERR [RCM] rcm::RcmApi::AddPossibleOwner: (5010)' because of 'Gum handler completed as failed'
00002514.00000ed8::2019/01/14-06:15:57.986 INFO [RCM] rcm::RcmApi::MoveGroup: (OST-MOD-DBS-AG1, 1, 0, MoveType::Manual )
00002514.00000ed8::2019/01/14-06:15:57.987 INFO [RCM] rcm::RcmApi::OnlineResource: (OST-MOD-DBS-AG1, 0)
00002514.00000ed8::2019/01/14-06:15:57.987 INFO [RCM-rbtr] giving default token to group OST-MOD-DBS-AG1
00002514.00000ed8::2019/01/14-06:15:57.987 INFO [RCM-rbtr] giving default token to group OST-MOD-DBS-AG1
00002514.00000ed8::2019/01/14-06:15:57.987 INFO [RCM] Res OST-MOD-DBS-AG1: Offline -> OnlineCallIssued( StateUnknown )
00002514.00000ed8::2019/01/14-06:15:57.987 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) Offline-->OnlineCallIssued.
00002514.00000ed8::2019/01/14-06:15:57.987 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (OST-MOD-DBS-AG1, Offline --> Pending)
00002514.00000ed8::2019/01/14-06:15:57.987 INFO [RCM-rbtr] giving default token to group OST-MOD-DBS-AG1
00002514.000021c0::2019/01/14-06:15:57.987 INFO [RCM] ignored non-local state Pending for group OST-MOD-DBS-AG1
00002514.00002d24::2019/01/14-06:15:57.987 INFO rcm::RcmResource::OnlineWorker[RCM] Issuing Online(OST-MOD-DBS-AG1) to RHS.
00002514.00002a30::2019/01/14-06:15:57.987 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'OST-MOD-DBS-AG1', gen(0) result 997/0.
00002514.00002a30::2019/01/14-06:15:57.987 INFO [RCM] Res OST-MOD-DBS-AG1: OnlineCallIssued -> OnlinePending( StateUnknown )
00002514.00002a30::2019/01/14-06:15:57.987 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) OnlineCallIssued-->OnlinePending.
000025c8.00000810::2019/01/14-06:15:57.988 INFO [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] The DeadLockTimeout property has a value of 300000
000025c8.00000810::2019/01/14-06:15:57.988 INFO [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] The PendingTimeout property has a value of 180000
000025c8.00000810::2019/01/14-06:15:57.988 ERR [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] Unable to enumerate SqlInstToNodeMap key. Error: ea.
000025c8.00000810::2019/01/14-06:15:57.988 ERR [RHS] Online for resource OST-MOD-DBS-AG1 failed.
00002514.00001d34::2019/01/14-06:15:57.988 WARN [RCM] HandleMonitorReply: ONLINERESOURCE for 'OST-MOD-DBS-AG1', gen(0) result 5018/0.
00002514.00001d34::2019/01/14-06:15:57.988 INFO [RCM] Res OST-MOD-DBS-AG1: OnlinePending -> ProcessingFailure( StateUnknown )
00002514.00001d34::2019/01/14-06:15:57.988 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) OnlinePending-->ProcessingFailure.
00002514.00001d34::2019/01/14-06:15:57.988 ERR [RCM] rcm::RcmResource::HandleFailure: (OST-MOD-DBS-AG1)
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] resource OST-MOD-DBS-AG1: failure count: 0, restartAction: 0 persistentState: 1.
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] numDependents is zero, auto-returning true
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] Will queue immediate restart (500 milliseconds) of OST-MOD-DBS-AG1 after terminate is complete.
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] Res OST-MOD-DBS-AG1: ProcessingFailure -> WaitingToTerminate( DelayRestartingResource )
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) ProcessingFailure-->[WaitingToTerminate to DelayRestartingResource].
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] Res OST-MOD-DBS-AG1: [WaitingToTerminate to DelayRestartingResource] -> Terminating( DelayRestartingResource )
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) [WaitingToTerminate to DelayRestartingResource]-->[Terminating to DelayRestartingResource].
000025c8.00002c50::2019/01/14-06:15:57.989 INFO [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] Online operation was not issued for the availability group. Simply returning from the Terminate call without the offline operation
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] HandleMonitorReply: TERMINATERESOURCE for 'OST-MOD-DBS-AG1', gen(1) result 0/0.
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] Res OST-MOD-DBS-AG1: [Terminating to DelayRestartingResource] -> DelayRestartingResource( StateUnknown )
00002514.00001d34::2019/01/14-06:15:57.989 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
00002514.00001d34::2019/01/14-06:15:57.989 WARN [RCM] Queueing immediate delay restart of resource OST-MOD-DBS-AG1 in 500 ms.
00002514.00000ed8::2019/01/14-06:15:58.009 INFO [GUM] Node 1: executing request locally, gumId:311, my action: /dm/update, # of updates: 1
00002514.00000ed8::2019/01/14-06:15:58.012 INFO [RCM] rcm::RcmApi::DeleteGroup: (OST-MOD-DBS-AG1)
00002514.00000ed8::2019/01/14-06:15:58.012 ERR [RCM] rcm::RcmApi::DeleteGroup: (5023)' because of 'Group OST-MOD-DBS-AG1: is in a pending state and cannot be deleted.'
00002514.00002d24::2019/01/14-06:15:58.489 INFO [RCM] Delay-restarting OST-MOD-DBS-AG1 and any waiting dependents.
00002514.00002d24::2019/01/14-06:15:58.489 INFO [RCM-rbtr] giving default token to group OST-MOD-DBS-AG1
00002514.00002d24::2019/01/14-06:15:58.489 INFO [RCM-rbtr] giving default token to group OST-MOD-DBS-AG1
00002514.00002d24::2019/01/14-06:15:58.489 INFO [RCM] Res OST-MOD-DBS-AG1: DelayRestartingResource -> OnlineCallIssued( StateUnknown )
00002514.00002d24::2019/01/14-06:15:58.489 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) DelayRestartingResource-->OnlineCallIssued.
00002514.000014c0::2019/01/14-06:15:58.489 INFO rcm::RcmResource::OnlineWorker[RCM] Issuing Online(OST-MOD-DBS-AG1) to RHS.
00002514.000014c0::2019/01/14-06:15:58.489 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'OST-MOD-DBS-AG1', gen(1) result 997/0.
00002514.000014c0::2019/01/14-06:15:58.489 INFO [RCM] Res OST-MOD-DBS-AG1: OnlineCallIssued -> OnlinePending( StateUnknown )
00002514.000014c0::2019/01/14-06:15:58.489 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) OnlineCallIssued-->OnlinePending.
000025c8.0000096c::2019/01/14-06:15:58.490 INFO [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] The DeadLockTimeout property has a value of 300000
000025c8.0000096c::2019/01/14-06:15:58.490 INFO [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] The PendingTimeout property has a value of 180000
000025c8.0000096c::2019/01/14-06:15:58.490 ERR [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] Did not find the instance to connect in SqlInstToNodeMap key.
000025c8.0000096c::2019/01/14-06:15:58.490 ERR [RHS] Online for resource OST-MOD-DBS-AG1 failed.
00002514.00001d34::2019/01/14-06:15:58.490 WARN [RCM] HandleMonitorReply: ONLINERESOURCE for 'OST-MOD-DBS-AG1', gen(1) result 5018/0.
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] Res OST-MOD-DBS-AG1: OnlinePending -> ProcessingFailure( StateUnknown )
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) OnlinePending-->ProcessingFailure.
00002514.00001d34::2019/01/14-06:15:58.491 ERR [RCM] rcm::RcmResource::HandleFailure: (OST-MOD-DBS-AG1)
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] resource OST-MOD-DBS-AG1: failure count: 1, restartAction: 2 persistentState: 1.
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] numDependents is zero, auto-returning true
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] Resource OST-MOD-DBS-AG1 is causing group OST-MOD-DBS-AG1 to failover.
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] rcm::RcmGroup::Failover: (OST-MOD-DBS-AG1)
00002514.00001d34::2019/01/14-06:15:58.491 WARN [RCM] Failing over group OST-MOD-DBS-AG1, failoverCount 1, last time 1601/01/01-00:00:00.000.
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] This node is director, making placement decision myself
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] applying filter NodeDownFilter to group OST-MOD-DBS-AG1 moveType=MoveType::Failover
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] applying filter NodeShuttingDownFilter to group OST-MOD-DBS-AG1 moveType=MoveType::Failover
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] applying filter CurrentNodeFilter to group OST-MOD-DBS-AG1 moveType=MoveType::Failover
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] removing banned candidate: NodeCandidate(1) banCode: CurrentNodeFilter (5016)
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] applying filter PausedNodeFilter to group OST-MOD-DBS-AG1 moveType=MoveType::Failover
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] applying filter PossibleOwnerFilter to group OST-MOD-DBS-AG1 moveType=MoveType::Failover
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] Group OST-MOD-DBS-AG1: done going through resources, returning true
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] applying filter QueuingPerviouslyRejectedFilter to group OST-MOD-DBS-AG1 moveType=MoveType::Failover
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] applying filter PreferredOwnerWaitFilter to group OST-MOD-DBS-AG1 moveType=MoveType::Failover
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] applying filter AntiAffinityFilter to group OST-MOD-DBS-AG1 moveType=MoveType::Failover
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] applying filter StmFilter to group OST-MOD-DBS-AG1 moveType=MoveType::Failover
00002514.00000ee0::2019/01/14-06:15:58.491 INFO [RCM] applying timely STM connectivity snapshot for group OST-MOD-DBS-AG1
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] filters reduced candidate list for group OST-MOD-DBS-AG1 to size 1, no need to run rankers
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM-plcmt] placement manager result: grp=OST-MOD-DBS-AG1 moveType=MoveType::Failover, node=2
00002514.00001d34::2019/01/14-06:15:58.491 INFO MTimer(GetPlacementAsDirector): [Total: 0 ms ( 0 s )]
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] Res OST-MOD-DBS-AG1: ProcessingFailure -> WaitingToTerminate( Failed )
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) ProcessingFailure-->[WaitingToTerminate to Failed].
00002514.00000ed8::2019/01/14-06:15:58.491 INFO [RCM] rcm::RcmGroup::FailoverWorker: (OST-MOD-DBS-AG1)
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] Res OST-MOD-DBS-AG1: [WaitingToTerminate to Failed] -> Terminating( Failed )
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) [WaitingToTerminate to Failed]-->[Terminating to Failed].
000025c8.00002c50::2019/01/14-06:15:58.491 INFO [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] Online operation was not issued for the availability group. Simply returning from the Terminate call without the offline operation
00002514.000014c0::2019/01/14-06:15:58.491 INFO [RCM] HandleMonitorReply: TERMINATERESOURCE for 'OST-MOD-DBS-AG1', gen(2) result 0/0.
00002514.000014c0::2019/01/14-06:15:58.491 INFO [RCM] Res OST-MOD-DBS-AG1: [Terminating to Failed] -> Failed( StateUnknown )
00002514.000014c0::2019/01/14-06:15:58.491 INFO [RCM] TransitionToState(OST-MOD-DBS-AG1) [Terminating to Failed]-->Failed.
00002514.000014c0::2019/01/14-06:15:58.491 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (OST-MOD-DBS-AG1, Pending --> Failed)
00002514.000021c0::2019/01/14-06:15:58.491 INFO [RCM] ignored non-local state Failed for group OST-MOD-DBS-AG1
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] moved 0 tasks from staging set to task set. TaskSetSize=0
00002514.00001d34::2019/01/14-06:15:58.491 INFO [RCM] rcm::RcmPriorityManager::StartGroups: [RCM] done, executed 0 tasks
00002514.00000ed8::2019/01/14-06:15:58.542 INFO [RCM] ordering failover of group OST-MOD-DBS-AG1 to node 2
00002514.00000ed8::2019/01/14-06:15:58.542 INFO [RCM] rcm::RcmGroup::Move: (OST-MOD-DBS-AG1, 2, MoveType::Failover)
00002514.00000ed8::2019/01/14-06:15:58.542 INFO [RCM] No need to bring group 'OST-MOD-DBS-AG1' offline before move; state Failed, owner 1.
00002514.00001d34::2019/01/14-06:15:58.542 INFO [RCM] rcm::QueuedMovesHolder::RemoveGroup: (OST-MOD-DBS-AG1) GroupBeingMoved: true AllowMoveCancel: false NotifyMoveFailure: false
00002514.00001d34::2019/01/14-06:15:58.542 INFO [GUM] Node 1: executing request locally, gumId:312, my action: /rcm/gum/GroupMoveOperation, # of updates: 1
00002514.00001d34::2019/01/14-06:15:58.542 INFO [RCM] rcm::RcmGum::GroupMoveOperation(1)
00002514.00001d34::2019/01/14-06:15:58.542 INFO [RCM] move of group OST-MOD-DBS-AG1 from ost-clb-iss-dbs-c01-data(1) to ost-cla-iss-dbs-c01-data(2) of type MoveType::Failover is about to succeed, failoverCount=1, lastFailoverTime=2019/01/14-17:15:58.491 targeted=false
00002514.00000ed8::2019/01/14-06:15:58.542 INFO [RCM] moved 0 tasks from staging set to task set. TaskSetSize=0
00002514.00000ed8::2019/01/14-06:15:58.542 INFO [RCM] rcm::RcmPriorityManager::StartGroups: [RCM] done, executed 0 tasks
00002514.00001d34::2019/01/14-06:15:58.542 INFO MTimer(OneGroupMovetimer): [Total: 0 ms ( 0 s )]
00002514.00001d34::2019/01/14-06:15:58.542 INFO [RCM] processed 1 group moves, 0 failures
00002514.00001d34::2019/01/14-06:15:58.542 INFO [DM] An empty single transaction is cancelled 2:2:269+1::0
00002514.00000ee0::2019/01/14-06:15:59.019 INFO [RCM] rcm::RcmApi::DeleteGroup: (OST-MOD-DBS-AG1)
00002514.00000ee0::2019/01/14-06:15:59.020 WARN [RCM] rcm::RcmApi::DeleteGroup: retrying: OST-MOD-DBS-AG1, 5908.
00002514.00002380::2019/01/14-06:15:59.545 INFO [GEM] Node 1: Deleting [2:229 , 2:242] (both included) as it has been ack'd by every node
00002514.00002380::2019/01/14-06:16:00.021 INFO [GUM] Node 1: Processing RequestLock 2:57
00002514.00002380::2019/01/14-06:16:00.021 INFO [GUM] Node 1: Processing GrantLock to 2 (sent by 1 gumid: 312)
00002514.00000ed8::2019/01/14-06:16:00.023 INFO [GUM] Node 1: Executing locally gumId: 313, updates: 1, first action: /rcm/gum/SetGroupPersistentState
00002514.00000ed8::2019/01/14-06:16:00.023 INFO [RCM] rcm::RcmGum::SetGroupPersistentState(OST-MOD-DBS-AG1,0)
00002514.000014c0::2019/01/14-06:16:00.035 INFO [GUM] Node 1: Executing locally gumId: 314, updates: 1, first action: /rcm/gum/DeleteGroup
00002514.000014c0::2019/01/14-06:16:00.035 INFO [RCM] rcm::RcmGum::DeleteGroup(OST-MOD-DBS-AG1,true)
00002514.00001d34::2019/01/14-06:16:00.035 WARN [RCM] rcm::RcmApi::GetResourceState: retrying: OST-MOD-DBS-AG1, 5007.
00002514.00000ee0::2019/01/14-06:16:00.041 INFO [GUM] Node 1: Processing RequestLock 1:265
00002514.00002380::2019/01/14-06:16:00.042 INFO [GUM] Node 1: Processing GrantLock to 1 (sent by 2 gumid: 314)
00002514.00000ee0::2019/01/14-06:16:00.042 INFO [GUM] Node 1: executing request locally, gumId:315, my action: /dm/update, # of updates: 1
000022a4.000021f0::2019/01/14-06:16:00.470 INFO [RES] Network Name: Agent: Sending request Netname/RecheckConfig to NN:071916e4-38c1-4e44-a2ec-dadf25cd11ca:Netbios
00002514.00002380::2019/01/14-06:16:01.023 INFO [GEM] Node 1: Deleting [2:243 , 2:248] (both included) as it has been ack'd by every node
00002514.00001d34::2019/01/14-06:16:01.037 ERR [RCM] rcm::RcmApi::GetResourceState: (5006)' because of 'Resource has been deleted.'
00002514.00001d34::2019/01/14-06:16:01.037 ERR [API] s_ApiGetResourceState: (5006)' because of 'GetResourceState failed.'
00002514.00001d34::2019/01/14-06:16:01.037 WARN [API] s_ApiOpenGroupEx: Group OST-MOD-DBS-AG1 failed, status = 5013
000022a4.000021f0::2019/01/14-06:16:05.471 INFO [RES] Network Name: Agent: Sending request Netname/RecheckConfig to NN:071916e4-38c1-4e44-a2ec-dadf25cd11ca:Netbios
000022a4.000021f0::2019/01/14-06:16:10.472 INFO [RES] Network Name: Agent: Sending request Netname/RecheckConfig to NN:071916e4-38c1-4e44-a2ec-dadf25cd11ca:Netbios
00002514.00001d34::2019/01/14-06:16:12.793 INFO [DCM] HandleSweeperRecheck
00002514.00001d34::2019/01/14-06:16:12.793 INFO [CLI] LsaCallAuthenticationPackage: 0, 0 size: 4, buffer: HDL( fa84740000 )
000022a4.000021f0::2019/01/14-06:16:15.393 INFO [RES] Network Name <Cluster Name>: Dns: HealthCheck: OST-MOD-DBS-CL1
000022a4.000021f0::2019/01/14-06:16:15.393 INFO [RES] Network Name <Cluster Name>: Dns: End of Slow Operation, state: Initialized/Reading, prevWorkState: Reading
000022a4.000021f0::2019/01/14-06:16:15.466 INFO [RES] Network Name: Agent: Sending request Netname/RecheckConfig to NN:071916e4-38c1-4e44-a2ec-dadf25cd11ca:Netbios

Viewing 5 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply

debasishbhat SSC Enthusiast Points: 191 More actions · Answer 1

From your log I can find below error

000025c8.0000096c::2019/01/14-06:15:58.490 ERR [RES] SQL Server Availability Group <OST-MOD-DBS-AG1>: [hadrag] Did not find the instance to connect in SqlInstToNodeMap key.

SqlInstToNodeMap is a registry key which should have the same information as sys.dm_hadr_instance_node_map.

There is some metadata mismatch between information in cluster and information in SQL Server. Even both replicas are having a mismatch of information about availability group. Ran below command on secondary to remove information about AG. You may not able to delete using use UI aa can give an error

1	`DROP` `AVAILABILITY` `GROUP` AGNAME

As soon as you executed, the databases were in restoring state and AG information was cleared from all DVMs and cluster also. Then we recreated the availability group using the AG wizard and we were back in business

databases.specialist SSC Veteran Points: 235 More actions · Answer 2

select * from sys.availability_groups;
giving zero rows selected.

group_id name resource_id resource_group_id failure_condition_level health_check_timeout automated_backup_preference automated_backup_preference_desc version basic_features dtc_support db_failover is_distributed

drop availability giving as it does not exist..

DROP AVAILABILITY GROUP "OST-MOD-DBS-AG1";
Msg 15151, Level 16, State 2, Line 3
Cannot drop the availability group 'OST-MOD-DBS-AG1', because it does not exist or you do not have permission.

debasishbhat SSC Enthusiast Points: 191 More actions · Answer 3

databases.specialist - Monday, January 14, 2019 6:30 PM
select * from sys.availability_groups;
giving zero rows selected.
group_id name resource_id resource_group_id failure_condition_level health_check_timeout automated_backup_preference automated_backup_preference_desc version basic_features dtc_support db_failover is_distributed
drop availability giving as it does not exist..
DROP AVAILABILITY GROUP "OST-MOD-DBS-AG1";
Msg 15151, Level 16, State 2, Line 3
Cannot drop the availability group 'OST-MOD-DBS-AG1', because it does not exist or you do not have permission.

Is it giving same for Primary & Secondary Instances?

Rick-153145 SSCrazy Points: 2823 More actions · Answer 4

In AD, have you given rights from the cluster container to the nodes? If not, then you will not be able to find either node and therefore creating an availability group will fail. If you have done this correctly and think it should be able to manage both nodes, check your nodes firewall settings, ensure you create a rule on 1434 for UDP as well as the required TCP settings (Port for SQL, Endpoints and whatever the nodes are running under).