Instrument time spent by fiber in running state and log it for SR eventbase threads
Summary: We often run into cases where system CPU util is low but SR eventbase threads are backing up. During S213134 debugging, it took us a while to figure out that requests with http transport were expensive and causing issues on intern-rpc clients and later realized it was a known issue for http transport to be CPU intensive. We got lucky that http transport requests used different stack we could see in strobelight, but this will likely not be the case for other similar issues. For ease of debugging these issues, one idea is to log SR Eventbase cost of each request to SR scuba. **The closest proxy for this is the wall clock time spent by SR's request-dispatch task in fiber, excluding all the preemptions.** This metric does have couple of drawbacks: - It wont capture all the work a request does via runInMainContext. This should be Ok since SR tries not to do lot of work in mainContext of fiber manager. - If system context switching the threads a lot and if eventbase thread is preempted by OS, we will count that time as time spent in task. I am hoping that new metric will still be useful even with this caveat. Design: Allow fiberManager.AddTask() to optionally take in a bool `logRunningTime` to decide whether it can log the running time of the task. If `logRunningTime` is set, fiber will keep track of elapsed time since most recent run in `currStartTime_` and track the running time of all previous runs together in `prevDuration_`. I looked into using ExecutionObserver for this purpose. I did not go that route because 1. ExecutionObserver's callbacks return fiber IDs, we want a callback to return function/task ID that we can tie back to Sr request. See here https://fburl.com/diffusion/qh9ioeq3 2. Caller (SR) will need to maintain the separate cache of all outstanding tasks with their running_times and manage their lifetime, which feels more complex than current approach. I have a way to turn off this change using new killswitch I added in SR. Reviewed By: yfeldblum Differential Revision: D24823396 fbshipit-source-id: 62f79b8c340cc48a22684c9cb3cdddadc260a0ab
Showing
Please register or sign in to comment