Sitecore .Net File Change Notification Shut down issues

 

Problem Statement

Not sure what is causing file change in webapps whether it is appinsight or for that matter logging handler that keep adding debug, info and error traces, cache or something else. For some reason there is something going under the hood which causes this file change notification. This take me back to the basics of file change notification causes app pool recycle when there is change in web.config or global.asax. Now as best practice it is always mentioned the logging has to be done outside your webroot and for some reason this is not the case with webapp. I'm naïve in webapp world and file and drive structure in azure webapp paas environment. Those days are gone when you use to have full control of VMs and you can deep dive into solving and identifying issues.

KB: Sitecore stability issues when using Azure Web Apps

https://kb.sitecore.net/articles/983839

It talks about dynamic cache- WEBSITE_DYNAMIC_CACHE

FCN Cause and Effect

Now this file change notification itself is a problem which as ripple effect on few things. This is how it works.

Chain and sequences of events due to FCN- Filewatcher/File change notification

1. File change notification, filewatcher notification triggers
2. Sitecore shut down
3. Solr or lucene Indexing lookup failed. Empty Index found.
4. Session State, if sql , Redis or Mongo used: Failure of session state, results in session leaks and session_end events, and all other subsequent calls.
5. XDb: Analytics current tracking failed or current tracking is null
6. Eventually spike in CPU utilization and memory leaks.

How it goes?

FCN Cause -> Affected--> Sitecore shut down-> Session --> Indexing--> Analytics-> CPU utilization--> Memory Leaks

It is very important to understand how we approach to the problem and how we look up error logs. It is not that straight forward instead need problem occurrence chart[Fish bone technique to lay down the cause and effect] to note down each events. In fact I see this as solving mystery or identify suspects. Like above chain of sequence. As rule of thumb memory leaks and CPU utilization always leads you to bad code or DI leaks etc. I insist more lateral thinking than primitive way of identifying problem.

.Net Way Short term Workaround: Brute force Fixes

http://www.dnnsoftware.com/community-blog/cid/154980/aspnet-file-change-notifications-and-dnn

https://shazwazza.com/post/all-about-aspnet-file-change-notification-fcn/

https://blogs.msdn.microsoft.com/tmarq/2007/11/01/asp-net-file-change-notifications-exactly-which-files-and-directories-are-monitored/

https://stackoverflow.com/questions/37039727/azure-web-app-worker-process-recycling-unexpectedly

As a work-around I have added fcnMode = "Disabled" to the httpRuntime element in the web.config. The root cause remains unknown.

httpruntime fcnmode="Disabled"/