The following is a demonstration of the iopending tool,

Here we run it with a sample interval of 1 second,

   # iopending 1
   Tracing... Please wait.
   2006 Jan  6 20:21:59,  load: 0.02,  disk_r:      0 KB,  disk_w:      0 KB
   
              value  ------------- Distribution ------------- count
                < 0 |                                         0
                  0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1010
                  1 |                                         0
   
   2006 Jan  6 20:22:00,  load: 0.03,  disk_r:      0 KB,  disk_w:      0 KB
   
              value  ------------- Distribution ------------- count
                < 0 |                                         0
                  0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1000
                  1 |                                         0
   
   2006 Jan  6 20:22:01,  load: 0.03,  disk_r:      0 KB,  disk_w:      0 KB
   
              value  ------------- Distribution ------------- count
                < 0 |                                         0
                  0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1000
                  1 |                                         0
   
   ^C

The iopending tool samples at 1000 Hz, and prints a distribution of how many
disk events were "pending" completion. In the above example the disks are
quiet - for all the samples there are zero disk events pending.



Now iopending is run with no arguments. It will default to an interval of 5
seconds,

   # iopending
   Tracing... Please wait.
   2006 Jan  6 19:15:41,  load: 0.03,  disk_r:   3599 KB,  disk_w:      0 KB
   
              value  ------------- Distribution ------------- count
                < 0 |                                         0
                  0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@     4450
                  1 |@@@                                      390
                  2 |@                                        80
                  3 |                                         40
                  4 |                                         20
                  5 |                                         30
                  6 |                                         0
   
   ^C

In the above output there was a little disk activity. For 390 samples there
was 1 I/O event pending; for 80 samples there was 2, and so on. 




In the following example iopending is run during heavy disk activity. We
print output every 10 seconds,

   # iopending 10
   Tracing... Please wait.
   2006 Jan  6 20:58:07,  load: 0.03,  disk_r:  25172 KB,  disk_w:  33321 KB
   
              value  ------------- Distribution ------------- count
                < 0 |                                         0
                  0 |@@@@@@@@@                                2160
                  1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@              6720
                  2 |@@@@                                     1000
                  3 |                                         50
                  4 |                                         30
                  5 |                                         20
                  6 |                                         10
                  7 |                                         10
                  8 |                                         10
                  9 |                                         0
   
   2006 Jan  6 20:58:17,  load: 0.05,  disk_r:   8409 KB,  disk_w:  12449 KB
   
              value  ------------- Distribution ------------- count
                < 0 |                                         0
                  0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@            7260
                  1 |@@@@@@@                                  1700
                  2 |@                                        300
                  3 |                                         0
                  4 |                                         10
                  5 |                                         10
                  6 |                                         10
                  7 |                                         20
                  8 |                                         0
                  9 |                                         0
                 10 |                                         0
                 11 |                                         0
                 12 |                                         0
                 13 |                                         0
                 14 |                                         0
                 15 |                                         0
                 16 |                                         0
                 17 |                                         10
                 18 |                                         20
                 19 |                                         0
                 20 |                                         0
                 21 |                                         0
                 22 |                                         0
                 23 |                                         0
                 24 |                                         0
                 25 |                                         0
                 26 |                                         0
                 27 |                                         0
                 28 |                                         0
                 29 |                                         0
                 30 |                                         0
                 31 |                                         10
              >= 32 |@@@                                      650
   
   ^C

In the first output, most of the time (67%) there was 1 event pending, 
and for a short time there were 8 events pending. In the second output we
see many samples were off the scale - 650 samples at 32 or more pending
events. For this sample I had typed "sync" in another window, which 
queued many disk events immediately which were eventually completed.