User Profile Service and Synchronization services are showing running but profile synchronization not getting started: SharePoint 2010 & 2013

Yesterday our user were complaining that current changes in AD not getting replicated in SharePoint after initial inspection I found user profile not getting sync from last 2 days. I also tried to restart UPS and User profile synchronization service but no luck. I so tried to clean cache and restart timer services but it not helped.

When I checked log I am getting  below log event (Event ID 8301)

An exception occurred while updating addresses for connected app {7e90e5d3-268a-44c3-8020-d28b022b7d09_775ca572-a48f-4af9-9257-791e0129aa82}. The uri endpoint information may be stale. System.InvalidOperationException: The requested application could not be found at Microsoft.SharePoint.SPTopologyWebServiceApplicationProxy.ProcessCommonExceptions(Uri endpointAddress, String operationName, Exception ex, SPServiceLoadBalancerContext context)at Microsoft.SharePoint.SPTopologyWebServiceApplicationProxy.ExecuteOnChannel(StringoperationName, CodeBlock codeBlock) at Microsoft.SharePoint.SPTopologyWebServiceApplicationProxy.GetEndPoints(GuidserviceId) at Microsoft.SharePoint.SPConnectedServiceApplicationAddressesRefreshJob.Execute(Guid targetInstanceId)

Log files are showing below entry  with High Level in User profile category

User Profile Application: SynchronizeMIIS encounters an exception: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Server.UserProfiles.UserProfileImportJob.<>c__DisplayClass2.<IsTimerJobRunning>b__1() at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()

at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)

at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)

at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)

at Microsoft.Office.Server.UserProfiles.UserProfileImportJob.IsTimerJobRunning(UserProfileApplicationJob timerJob)

at Microsoft.Office.Server.Administration.UserProfileApplication.SynchronizeMIIS()

at Microsoft.Office.Server.Administration.ILMProfileSynchronizationJob.Execute()

 

It means we are getting null reference exception when FIM management trying to connect to SharePoint. Most of time this is related to timer job that will be created when we set UPSA to Active Directory import . But in our case user profile sync was running fine and when I read about this it show this property can put UPSA in unstable condition so if some how it toggle it could be create issue. I checked using the below command but it showed it is already set to false

$upsa = Get-SPServiceApplication -name "User Profile Service Application"
$upsa.NoILMUsed
But since we have custom code in our farm so thought it to reset by appending below command $upsa.NoILMUsed = $true
$upsa.Update()
$upsa.NoILMUsed = $false
$upsa.Update()


After that i hit user profile sync and it work well.

References

https://support.microsoft.com/en-us/kb/2624198

http://blogs.msdn.com/b/ronalg/archive/2015/02/17/sharepoint-2013-external-fim-issue-missing-manager-property.aspx