CMSSW I/O in synchronous mode

• Running PAT tuple creation.
• With the CMS I/O patches found here.
• cacheSize set to 20MB.
• readHint mode set to "application-only".
• Reading out 40 events; after the first 20 events, the TTreeCache is activated.

Below are a few graphs showing the I/O activity, with the zoomed-in picture focusing on the area where TTreeCache fills its buffer (TTreeCache::!FillBuffer):

• I/O activity on analyzed file over the whole run of the application:
  
  There are four main bands of activity - first, when ROOT opens the file. Second, when ROOT starts opening and processing the TTree itself (figuring out what it needs to deserialize, event metadata). Third is the read of the first event. After this, there's a long pause in I/O when the conditions data is loaded. Finally, there is the run through the file itself. The first 20 events are read using "normal" ROOT I/O. The TTreeCache buffer is then filled, and you see no more stalls for the rest of the file (as CMSSW has almost all the I/O it needs for awhile).
• Focus on the time where ROOT was filling the TTreeCache buffers:
  
  This solid band of green represents ROOT synchronously reading data in from the file. Notice there are a lot of "stalls", as the I/O pattern appears completely random to the operating system - the OS completely fails to predict any future reads.
• Notice that the disk-level parallelism is pretty low - at the greatest, it splits a single ~128KB I/O request into ~30 block requests, which get coalesced into 4 distinct requests out to the block device.

My claim would be that synchronous mode, while a clear improvement is still not optimal because it assumes ROOT knows best about how to organize requests. We have no knowledge of whether or not two pieces of non-sequential data in the file are actually sequential at the block layer. The point behind asynchronous I/O is to allow the block layer - which has a much better, if not complete, idea of the layout - determine what reads to coalesce or split.