client4
2016 January 13, 17:05
fio-2.1.10
CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
60 s ramp + 7200 s run, per test; s between tests

# 
# SR-71mach5 Configuration
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 

Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz   Status: Populated, Enabled Core Enabled: 4
Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz   Status: Populated, Enabled Core Enabled: 4
DIMM_A1: 4096 MB | 64/72 bits | 1333 MHz | Undefined        |  | SN: 00000000  
DIMM_B1: 4096 MB | 64/72 bits | 1333 MHz | Undefined        |  | SN: 00000000  
DIMM_C1: 4096 MB | 64/72 bits | 1333 MHz | Undefined        |  | SN: 00000000  
DIMM_D1: 4096 MB | 64/72 bits | 1333 MHz | Undefined        |  | SN: 00000000  
DIMM_E1: 4096 MB | 64/72 bits | 1333 MHz | Undefined        |  | SN: 00000000  
DIMM_F1: 4096 MB | 64/72 bits | 1333 MHz | Undefined        |  | SN: 00000000  
DIMM_G1: 4096 MB | 64/72 bits | 1333 MHz | Undefined        |  | SN: 00000000  
DIMM_H1: 4096 MB | 64/72 bits | 1333 MHz | Undefined        |  | SN: 00000000  
____________________________________________________________________
MemTotal:       32673036 kB


# 
# actual mount paramters
# 

10.0.5.144:/control on /root/sr71/control type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.5.135,local_lock=none,addr=10.0.5.144)
10.10.6.100:/sr71/C0P4 on /root/sr71/SATA0 type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=100,retrans=2,sec=sys,clientaddr=10.10.6.10,local_lock=none,addr=10.10.6.100)
10.10.7.100:/sr71/C1P4 on /root/sr71/SATA1 type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=100,retrans=2,sec=sys,clientaddr=10.10.7.10,local_lock=none,addr=10.10.7.100)
10.10.6.100:/sr71/NVMeR5V0P4 on /root/sr71/NVMe0 type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=100,retrans=2,sec=sys,clientaddr=10.10.6.10,local_lock=none,addr=10.10.6.100)
10.10.7.100:/sr71/NVMeR5V1P4 on /root/sr71/NVMe1 type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=100,retrans=2,sec=sys,clientaddr=10.10.7.10,local_lock=none,addr=10.10.7.100)



# 
# fio job file
fio-2.1.10
# 

 seconds between fio tests
[global]
ioengine=libaio
direct=1
time_based
ramp_time=${RAMPTIME}
runtime=${RUNTIME}
bs=${BS}
rw=${RW}
rwmixread=${RWMIXREAD}
group_reporting

[SATA-R5-01]
new_group
size=1536g
iodepth=${IODEPTHS}
numjobs=${NUMJOBSS}
filename=/root/sr71/SATA0/fio.tst

[SATA-R5-11]
new_group
size=1536g
iodepth=${IODEPTHS}
numjobs=${NUMJOBSS}
filename=/root/sr71/SATA1/fio.tst

[NVMe-R5-01]
new_group
size=200g
iodepth=${IODEPTHN}
numjobs=${NUMJOBSN}
filename=/root/sr71/NVMe0/fio.tst

[NVMe-R5-11]
new_group
size=200g
iodepth=${IODEPTHN}
numjobs=${NUMJOBSN}
filename=/root/sr71/NVMe1/fio.tst




# 
# 201601131705 test details
# 



# 
# Wed 13 Jan 2016 05:05:00 PM EST
# 8kB randwrite with SATA 1 jobs and 1 I/Os Outstanding
# 8kB randwrite with NVMe 1 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 2961 IOPS | 23 MBps | 333 μsec Average Latency 
SATA-R5-1: 2765 IOPS | 22 MBps | 357 μsec Average Latency 
NVMe-R5-0: 5096 IOPS | 40 MBps | 391 μsec Average Latency 
NVMe-R5-1: 5477 IOPS | 43 MBps | 363 μsec Average Latency 

Totals
IOPS: 16299
MBPS: 128
Average Latency (μsec): 361

SATA-R5-01: (g=0): rw=randwrite, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=1
SATA-R5-11: (g=1): rw=randwrite, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=1
NVMe-R5-01: (g=2): rw=randwrite, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
NVMe-R5-11: (g=3): rw=randwrite, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=108411: Wed Jan 13 19:06:01 2016
  write: io=166603MB, bw=23695KB/s, iops=2961, runt=7200001msec
    slat (usec): min=2, max=245, avg=10.06, stdev= 4.98
    clat (usec): min=33, max=200090, avg=323.64, stdev=328.55
     lat (usec): min=89, max=200102, avg=333.83, stdev=328.05
    clat percentiles (usec):
     |  1.00th=[  105],  5.00th=[  112], 10.00th=[  193], 20.00th=[  245],
     | 30.00th=[  342], 40.00th=[  358], 50.00th=[  362], 60.00th=[  362],
     | 70.00th=[  366], 80.00th=[  366], 90.00th=[  370], 95.00th=[  390],
     | 99.00th=[  418], 99.50th=[  458], 99.90th=[  516], 99.95th=[  628],
     | 99.99th=[ 1304]
    bw (KB  /s): min=    0, max=43840, per=100.00%, avg=23702.69, stdev=2826.31
    lat (usec) : 50=0.01%, 100=0.24%, 250=20.35%, 500=79.27%, 750=0.10%
    lat (usec) : 1000=0.02%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.52%, sys=1.97%, ctx=43191246, majf=0, minf=1162
  IO depths    : 1=100.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=21325158/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=108412: Wed Jan 13 19:06:01 2016
  write: io=155570MB, bw=22126KB/s, iops=2765, runt=7200001msec
    slat (usec): min=2, max=223, avg= 9.25, stdev= 3.84
    clat (usec): min=39, max=85714, avg=348.40, stdev=60.67
     lat (usec): min=89, max=85738, avg=357.79, stdev=59.61
    clat percentiles (usec):
     |  1.00th=[  129],  5.00th=[  233], 10.00th=[  290], 20.00th=[  350],
     | 30.00th=[  358], 40.00th=[  362], 50.00th=[  362], 60.00th=[  366],
     | 70.00th=[  366], 80.00th=[  366], 90.00th=[  370], 95.00th=[  394],
     | 99.00th=[  422], 99.50th=[  470], 99.90th=[  516], 99.95th=[  628],
     | 99.99th=[ 1128]
    bw (KB  /s): min=    0, max=40464, per=100.00%, avg=22132.17, stdev=971.62
    lat (usec) : 50=0.01%, 100=0.05%, 250=7.70%, 500=92.10%, 750=0.11%
    lat (usec) : 1000=0.02%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.39%, sys=1.83%, ctx=40299210, majf=0, minf=1493
  IO depths    : 1=100.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=19912961/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=108413: Wed Jan 13 19:06:01 2016
  write: io=286662MB, bw=40770KB/s, iops=5096, runt=7200001msec
    slat (usec): min=2, max=1978, avg= 9.06, stdev= 3.52
    clat (usec): min=52, max=201072, avg=381.87, stdev=1831.64
     lat (usec): min=90, max=201085, avg=391.07, stdev=1831.61
    clat percentiles (usec):
     |  1.00th=[  165],  5.00th=[  235], 10.00th=[  278], 20.00th=[  350],
     | 30.00th=[  362], 40.00th=[  362], 50.00th=[  366], 60.00th=[  366],
     | 70.00th=[  366], 80.00th=[  370], 90.00th=[  374], 95.00th=[  398],
     | 99.00th=[  422], 99.50th=[  470], 99.90th=[  652], 99.95th=[16512],
     | 99.99th=[92672]
    bw (KB  /s): min=    0, max=51552, per=100.00%, avg=40873.03, stdev=5533.88
    lat (usec) : 100=0.03%, 250=7.66%, 500=92.10%, 750=0.12%, 1000=0.02%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.47%, sys=2.93%, ctx=73230787, majf=0, minf=1670
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=36692796/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=108414: Wed Jan 13 19:06:01 2016
  write: io=308115MB, bw=43821KB/s, iops=5477, runt=7200001msec
    slat (usec): min=2, max=190, avg= 8.96, stdev= 3.43
    clat (usec): min=55, max=200287, avg=354.64, stdev=423.92
     lat (usec): min=88, max=200294, avg=363.73, stdev=423.80
    clat percentiles (usec):
     |  1.00th=[  173],  5.00th=[  239], 10.00th=[  302], 20.00th=[  354],
     | 30.00th=[  362], 40.00th=[  366], 50.00th=[  366], 60.00th=[  366],
     | 70.00th=[  366], 80.00th=[  370], 90.00th=[  374], 95.00th=[  398],
     | 99.00th=[  422], 99.50th=[  462], 99.90th=[  524], 99.95th=[  676],
     | 99.99th=[ 1432]
    bw (KB  /s): min=    0, max=48416, per=100.00%, avg=43832.76, stdev=1819.97
    lat (usec) : 100=0.02%, 250=6.63%, 500=93.18%, 750=0.12%, 1000=0.02%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.59%, sys=3.16%, ctx=78800983, majf=0, minf=1823
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=39438712/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
  WRITE: io=166603MB, aggrb=23694KB/s, minb=23694KB/s, maxb=23694KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
  WRITE: io=155570MB, aggrb=22125KB/s, minb=22125KB/s, maxb=22125KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
  WRITE: io=286662MB, aggrb=40769KB/s, minb=40769KB/s, maxb=40769KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
  WRITE: io=308115MB, aggrb=43820KB/s, minb=43820KB/s, maxb=43820KB/s, mint=7200001msec, maxt=7200001msec



# 
# Wed 13 Jan 2016 07:06:03 PM EST
# 64kB randwrite with SATA 1 jobs and 1 I/Os Outstanding
# 64kB randwrite with NVMe 4 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 1497 IOPS | 95 MBps | 663 μsec Average Latency 
SATA-R5-1: 1417 IOPS | 90 MBps | 701 μsec Average Latency 
NVMe-R5-0: 608 IOPS | 38 MBps | 13151 μsec Average Latency 
NVMe-R5-1: 656 IOPS | 42 MBps | 12183 μsec Average Latency 

Totals
IOPS: 4178
MBPS: 265
Average Latency (μsec): 6674

SATA-R5-01: (g=0): rw=randwrite, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=1
SATA-R5-11: (g=1): rw=randwrite, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=1
NVMe-R5-01: (g=2): rw=randwrite, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randwrite, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 10 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=113567: Wed Jan 13 21:07:04 2016
  write: io=673812MB, bw=95827KB/s, iops=1497, runt=7200321msec
    slat (usec): min=8, max=1709, avg=28.75, stdev= 9.44
    clat (usec): min=4, max=3933.8K, avg=635.03, stdev=8435.20
     lat (usec): min=191, max=3933.8K, avg=663.85, stdev=8435.22
    clat percentiles (usec):
     |  1.00th=[  294],  5.00th=[  314], 10.00th=[  326], 20.00th=[  338],
     | 30.00th=[  342], 40.00th=[  346], 50.00th=[  350], 60.00th=[  350],
     | 70.00th=[  366], 80.00th=[  394], 90.00th=[  462], 95.00th=[  498],
     | 99.00th=[  628], 99.50th=[  876], 99.90th=[84480], 99.95th=[142336],
     | 99.99th=[346112]
    bw (KB  /s): min=    1, max=168576, per=100.00%, avg=99344.93, stdev=30384.46
    lat (usec) : 10=0.01%, 250=0.03%, 500=95.19%, 750=4.14%, 1000=0.22%
    lat (msec) : 2=0.11%, 4=0.03%, 10=0.02%, 20=0.01%, 50=0.11%
    lat (msec) : 100=0.06%, 250=0.07%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=1.33%, sys=2.23%, ctx=21787890, majf=0, minf=3468
  IO depths    : 1=100.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10780984/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=113568: Wed Jan 13 21:07:04 2016
  write: io=637840MB, bw=90711KB/s, iops=1417, runt=7200321msec
    slat (usec): min=8, max=454, avg=29.38, stdev=10.24
    clat (usec): min=116, max=3933.1K, avg=671.98, stdev=8646.75
     lat (usec): min=188, max=3933.1K, avg=701.49, stdev=8646.77
    clat percentiles (usec):
     |  1.00th=[  294],  5.00th=[  310], 10.00th=[  326], 20.00th=[  338],
     | 30.00th=[  342], 40.00th=[  346], 50.00th=[  350], 60.00th=[  350],
     | 70.00th=[  366], 80.00th=[  402], 90.00th=[  466], 95.00th=[  502],
     | 99.00th=[  636], 99.50th=[  892], 99.90th=[95744], 99.95th=[150528],
     | 99.99th=[350208]
    bw (KB  /s): min=    1, max=168576, per=100.00%, avg=93956.88, stdev=30782.28
    lat (usec) : 250=0.04%, 500=94.72%, 750=4.58%, 1000=0.22%
    lat (msec) : 2=0.11%, 4=0.02%, 10=0.01%, 20=0.02%, 50=0.12%
    lat (msec) : 100=0.08%, 250=0.08%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=1.27%, sys=2.18%, ctx=20652403, majf=0, minf=3810
  IO depths    : 1=100.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10205435/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=113569: Wed Jan 13 21:07:04 2016
  write: io=273674MB, bw=38923KB/s, iops=608, runt=7200001msec
    slat (usec): min=6, max=306, avg=34.21, stdev=22.52
    clat (usec): min=75, max=3999.9K, avg=13117.52, stdev=52912.49
     lat (usec): min=208, max=3999.1K, avg=13151.85, stdev=52912.22
    clat percentiles (usec):
     |  1.00th=[  302],  5.00th=[  362], 10.00th=[  402], 20.00th=[  446],
     | 30.00th=[  474], 40.00th=[  498], 50.00th=[  532], 60.00th=[  572],
     | 70.00th=[  620], 80.00th=[  700], 90.00th=[ 1064], 95.00th=[122368],
     | 99.00th=[199680], 99.50th=[276480], 99.90th=[464896], 99.95th=[610304],
     | 99.99th=[1019904]
    bw (KB  /s): min=    1, max=122368, per=25.48%, avg=9919.02, stdev=7017.71
    lat (usec) : 100=0.01%, 250=0.16%, 500=40.46%, 750=42.92%, 1000=5.86%
    lat (msec) : 2=2.68%, 4=0.28%, 10=0.12%, 20=0.05%, 50=0.59%
    lat (msec) : 100=1.49%, 250=4.80%, 500=0.51%, 750=0.05%, 1000=0.02%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=0.11%, sys=0.28%, ctx=8260616, majf=0, minf=23983
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=4378783/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=113573: Wed Jan 13 21:07:04 2016
  write: io=295414MB, bw=42013KB/s, iops=656, runt=7200291msec
    slat (usec): min=6, max=326, avg=45.24, stdev=28.36
    clat (usec): min=106, max=3964.4K, avg=12137.97, stdev=50589.33
     lat (usec): min=228, max=3964.5K, avg=12183.53, stdev=50589.01
    clat percentiles (usec):
     |  1.00th=[  294],  5.00th=[  362], 10.00th=[  402], 20.00th=[  450],
     | 30.00th=[  482], 40.00th=[  516], 50.00th=[  548], 60.00th=[  588],
     | 70.00th=[  628], 80.00th=[  700], 90.00th=[ 1012], 95.00th=[94720],
     | 99.00th=[199680], 99.50th=[259072], 99.90th=[452608], 99.95th=[602112],
     | 99.99th=[1019904]
    bw (KB  /s): min=    1, max=109184, per=25.61%, avg=10761.20, stdev=7343.37
    lat (usec) : 250=0.25%, 500=36.21%, 750=47.25%, 1000=6.21%
    lat (msec) : 2=2.51%, 4=0.27%, 10=0.11%, 20=0.05%, 50=0.68%
    lat (msec) : 100=1.65%, 250=4.29%, 500=0.46%, 750=0.05%, 1000=0.02%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=0.15%, sys=0.41%, ctx=8890757, majf=0, minf=22916
  IO depths    : 1=0.1%, 2=101.1%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=4726616/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
  WRITE: io=673812MB, aggrb=95826KB/s, minb=95826KB/s, maxb=95826KB/s, mint=7200321msec, maxt=7200321msec

Run status group 1 (all jobs):
  WRITE: io=637840MB, aggrb=90710KB/s, minb=90710KB/s, maxb=90710KB/s, mint=7200321msec, maxt=7200321msec

Run status group 2 (all jobs):
  WRITE: io=273674MB, aggrb=38922KB/s, minb=38922KB/s, maxb=38922KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
  WRITE: io=295414MB, aggrb=42012KB/s, minb=42012KB/s, maxb=42012KB/s, mint=7200291msec, maxt=7200291msec



# 
# Wed 13 Jan 2016 09:07:08 PM EST
# 8kB randrw with SATA 4 jobs and 8 I/Os Outstanding
# 8kB randrw with NVMe 4 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 5901 IOPS | 31 MBps | 7274 μsec Average Latency 
SATA-R5-1: 5798 IOPS | 31 MBps | 7403 μsec Average Latency 
NVMe-R5-0: 9987 IOPS | 53 MBps | 4226 μsec Average Latency 
NVMe-R5-1: 10651 IOPS | 57 MBps | 3937 μsec Average Latency 

Totals
IOPS: 32337
MBPS: 172
Average Latency (μsec): 5710

SATA-R5-01: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
SATA-R5-11: (g=1): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
NVMe-R5-01: (g=2): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
NVMe-R5-11: (g=3): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
fio-2.1.10
Starting 16 processes

SATA-R5-01: (groupid=0, jobs=4): err= 0: pid=118701: Wed Jan 13 23:08:08 2016
  read : io=222434MB, bw=31635KB/s, iops=3954, runt=7200002msec
    slat (usec): min=1, max=5358, avg=14.33, stdev=15.33
    clat (usec): min=9, max=1625.1K, avg=1806.54, stdev=4730.42
     lat (usec): min=153, max=1625.1K, avg=1821.02, stdev=4729.89
    clat percentiles (usec):
     |  1.00th=[  366],  5.00th=[  446], 10.00th=[  486], 20.00th=[  556],
     | 30.00th=[  604], 40.00th=[  644], 50.00th=[  700], 60.00th=[  772],
     | 70.00th=[  908], 80.00th=[ 1576], 90.00th=[ 3056], 95.00th=[ 9664],
     | 99.00th=[16768], 99.50th=[18304], 99.90th=[25216], 99.95th=[28032],
     | 99.99th=[74240]
    bw (KB  /s): min=    0, max=26176, per=25.28%, avg=7997.45, stdev=8493.95
  write: io=109555MB, bw=15581KB/s, iops=1947, runt=7200002msec
    slat (usec): min=2, max=5300, avg=18.29, stdev=17.60
    clat (usec): min=30, max=1628.7K, avg=12708.61, stdev=33253.93
     lat (usec): min=126, max=1628.7K, avg=12727.06, stdev=33252.90
    clat percentiles (usec):
     |  1.00th=[  366],  5.00th=[  580], 10.00th=[ 1464], 20.00th=[ 2768],
     | 30.00th=[ 3376], 40.00th=[ 3856], 50.00th=[ 4384], 60.00th=[ 4896],
     | 70.00th=[ 5728], 80.00th=[ 7456], 90.00th=[16768], 95.00th=[63232],
     | 99.00th=[187392], 99.50th=[209920], 99.90th=[261120], 99.95th=[301056],
     | 99.99th=[610304]
    bw (KB  /s): min=    0, max=12592, per=25.28%, avg=3939.13, stdev=4184.45
    lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.05%
    lat (usec) : 500=9.09%, 750=31.89%, 1000=10.76%
    lat (msec) : 2=7.63%, 4=16.76%, 10=16.10%, 20=4.38%, 50=1.47%
    lat (msec) : 100=0.67%, 250=1.14%, 500=0.04%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=0.45%, sys=1.14%, ctx=71601866, majf=0, minf=18948
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=28471586/w=14023018/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=4): err= 0: pid=118705: Wed Jan 13 23:08:08 2016
  read : io=218542MB, bw=31082KB/s, iops=3885, runt=7200022msec
    slat (usec): min=1, max=6449, avg=14.63, stdev=15.66
    clat (usec): min=40, max=1585.1K, avg=1836.87, stdev=4801.99
     lat (usec): min=157, max=1585.1K, avg=1851.65, stdev=4801.43
    clat percentiles (usec):
     |  1.00th=[  366],  5.00th=[  450], 10.00th=[  490], 20.00th=[  556],
     | 30.00th=[  604], 40.00th=[  644], 50.00th=[  708], 60.00th=[  772],
     | 70.00th=[  916], 80.00th=[ 1640], 90.00th=[ 3088], 95.00th=[ 9792],
     | 99.00th=[17024], 99.50th=[18816], 99.90th=[25216], 99.95th=[27776],
     | 99.99th=[54016]
    bw (KB  /s): min=    0, max=26272, per=25.28%, avg=7855.85, stdev=8453.46
  write: io=107648MB, bw=15310KB/s, iops=1913, runt=7200022msec
    slat (usec): min=2, max=6702, avg=18.74, stdev=18.24
    clat (usec): min=1, max=1777.1K, avg=12937.68, stdev=34157.69
     lat (usec): min=125, max=1777.1K, avg=12956.56, stdev=34156.63
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[  588], 10.00th=[ 1448], 20.00th=[ 2736],
     | 30.00th=[ 3344], 40.00th=[ 3856], 50.00th=[ 4320], 60.00th=[ 4896],
     | 70.00th=[ 5792], 80.00th=[ 7520], 90.00th=[17792], 95.00th=[64256],
     | 99.00th=[191488], 99.50th=[216064], 99.90th=[264192], 99.95th=[292864],
     | 99.99th=[593920]
    bw (KB  /s): min=    0, max=12512, per=25.28%, avg=3869.52, stdev=4163.17
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.05%
    lat (usec) : 500=8.84%, 750=31.90%, 1000=10.78%
    lat (msec) : 2=7.67%, 4=17.08%, 10=15.80%, 20=4.47%, 50=1.54%
    lat (msec) : 100=0.67%, 250=1.16%, 500=0.04%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=0.45%, sys=1.16%, ctx=70222077, majf=0, minf=18895
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=27973434/w=13778889/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=118709: Wed Jan 13 23:08:08 2016
  read : io=376400MB, bw=53532KB/s, iops=6691, runt=7200007msec
    slat (usec): min=1, max=4463, avg=17.33, stdev=17.69
    clat (usec): min=1, max=1734.2K, avg=1194.60, stdev=3736.67
     lat (usec): min=96, max=1734.3K, avg=1212.11, stdev=3736.98
    clat percentiles (usec):
     |  1.00th=[  326],  5.00th=[  406], 10.00th=[  454], 20.00th=[  502],
     | 30.00th=[  564], 40.00th=[  612], 50.00th=[  676], 60.00th=[  780],
     | 70.00th=[ 1096], 80.00th=[ 1928], 90.00th=[ 2672], 95.00th=[ 3376],
     | 99.00th=[ 4832], 99.50th=[ 5408], 99.90th=[ 7520], 99.95th=[ 8512],
     | 99.99th=[12352]
    bw (KB  /s): min=    0, max=35680, per=25.15%, avg=13461.09, stdev=9545.38
  write: io=185423MB, bw=26371KB/s, iops=3296, runt=7200007msec
    slat (usec): min=2, max=4332, avg=22.05, stdev=20.06
    clat (usec): min=11, max=1768.5K, avg=7218.43, stdev=12012.54
     lat (usec): min=115, max=1768.5K, avg=7240.67, stdev=12013.14
    clat percentiles (usec):
     |  1.00th=[  354],  5.00th=[  532], 10.00th=[  916], 20.00th=[ 1816],
     | 30.00th=[ 2256], 40.00th=[ 2640], 50.00th=[ 3056], 60.00th=[ 3696],
     | 70.00th=[ 6560], 80.00th=[13376], 90.00th=[20608], 95.00th=[25472],
     | 99.00th=[34048], 99.50th=[37120], 99.90th=[44288], 99.95th=[49920],
     | 99.99th=[481280]
    bw (KB  /s): min=    0, max=17616, per=25.15%, avg=6631.42, stdev=4700.78
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.16%
    lat (usec) : 500=14.11%, 750=27.49%, 1000=7.41%
    lat (msec) : 2=12.84%, 4=23.78%, 10=6.01%, 20=4.64%, 50=3.54%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=0.88%, sys=2.35%, ctx=119062056, majf=0, minf=17374
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.7%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=48179155/w=23734086/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=118713: Wed Jan 13 23:08:08 2016
  read : io=401403MB, bw=57088KB/s, iops=7136, runt=7200009msec
    slat (usec): min=1, max=3317, avg=18.23, stdev=18.36
    clat (usec): min=1, max=1611.4K, avg=1166.02, stdev=3601.48
     lat (usec): min=99, max=1611.6K, avg=1184.43, stdev=3601.78
    clat percentiles (usec):
     |  1.00th=[  322],  5.00th=[  402], 10.00th=[  450], 20.00th=[  506],
     | 30.00th=[  564], 40.00th=[  612], 50.00th=[  684], 60.00th=[  788],
     | 70.00th=[ 1112], 80.00th=[ 1880], 90.00th=[ 2576], 95.00th=[ 3120],
     | 99.00th=[ 4576], 99.50th=[ 5088], 99.90th=[ 7264], 99.95th=[ 8256],
     | 99.99th=[11968]
    bw (KB  /s): min=    0, max=36112, per=25.15%, avg=14355.49, stdev=9256.09
  write: io=197743MB, bw=28123KB/s, iops=3515, runt=7200009msec
    slat (usec): min=2, max=3248, avg=23.17, stdev=20.76
    clat (usec): min=15, max=1617.3K, avg=6668.48, stdev=10669.65
     lat (usec): min=123, max=1617.3K, avg=6691.83, stdev=10670.31
    clat percentiles (usec):
     |  1.00th=[  346],  5.00th=[  532], 10.00th=[  964], 20.00th=[ 1880],
     | 30.00th=[ 2352], 40.00th=[ 2736], 50.00th=[ 3184], 60.00th=[ 3952],
     | 70.00th=[ 7072], 80.00th=[12096], 90.00th=[17792], 95.00th=[21632],
     | 99.00th=[28800], 99.50th=[31360], 99.90th=[37632], 99.95th=[42240],
     | 99.99th=[428032]
    bw (KB  /s): min=    0, max=17680, per=25.15%, avg=7071.96, stdev=4554.72
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.18%, 500=14.07%, 750=26.91%, 1000=7.56%
    lat (msec) : 2=13.18%, 4=23.58%, 10=6.56%, 20=5.68%, 50=2.27%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=0.98%, sys=2.66%, ctx=126883049, majf=0, minf=19164
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.7%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=51379523/w=25311102/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=222434MB, aggrb=31635KB/s, minb=31635KB/s, maxb=31635KB/s, mint=7200002msec, maxt=7200002msec
  WRITE: io=109555MB, aggrb=15581KB/s, minb=15581KB/s, maxb=15581KB/s, mint=7200002msec, maxt=7200002msec

Run status group 1 (all jobs):
   READ: io=218542MB, aggrb=31081KB/s, minb=31081KB/s, maxb=31081KB/s, mint=7200022msec, maxt=7200022msec
  WRITE: io=107648MB, aggrb=15309KB/s, minb=15309KB/s, maxb=15309KB/s, mint=7200022msec, maxt=7200022msec

Run status group 2 (all jobs):
   READ: io=376400MB, aggrb=53532KB/s, minb=53532KB/s, maxb=53532KB/s, mint=7200007msec, maxt=7200007msec
  WRITE: io=185423MB, aggrb=26371KB/s, minb=26371KB/s, maxb=26371KB/s, mint=7200007msec, maxt=7200007msec

Run status group 3 (all jobs):
   READ: io=401403MB, aggrb=57088KB/s, minb=57088KB/s, maxb=57088KB/s, mint=7200009msec, maxt=7200009msec
  WRITE: io=197743MB, aggrb=28123KB/s, minb=28123KB/s, maxb=28123KB/s, mint=7200009msec, maxt=7200009msec



# 
# Wed 13 Jan 2016 11:08:11 PM EST
# 64kB randrw with SATA 5 jobs and 4 I/Os Outstanding
# 64kB randrw with NVMe 5 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 1862 IOPS | 79 MBps | 14499 μsec Average Latency 
SATA-R5-1: 1795 IOPS | 77 MBps | 15093 μsec Average Latency 
NVMe-R5-0: 1522 IOPS | 65 MBps | 7860 μsec Average Latency 
NVMe-R5-1: 1628 IOPS | 69 MBps | 7376 μsec Average Latency 

Totals
IOPS: 6807
MBPS: 290
Average Latency (μsec): 11207

SATA-R5-01: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
SATA-R5-11: (g=1): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
NVMe-R5-01: (g=2): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 20 processes

SATA-R5-01: (groupid=0, jobs=5): err= 0: pid=124020: Thu Jan 14 01:09:12 2016
  read : io=561852MB, bw=79908KB/s, iops=1248, runt=7200017msec
    slat (usec): min=5, max=3048, avg=39.09, stdev=26.74
    clat (usec): min=153, max=4363.2K, avg=3398.95, stdev=31565.94
     lat (usec): min=211, max=4363.2K, avg=3438.31, stdev=31566.07
    clat percentiles (usec):
     |  1.00th=[  756],  5.00th=[  900], 10.00th=[  988], 20.00th=[ 1096],
     | 30.00th=[ 1192], 40.00th=[ 1272], 50.00th=[ 1368], 60.00th=[ 1480],
     | 70.00th=[ 1672], 80.00th=[ 2608], 90.00th=[ 5088], 95.00th=[ 7776],
     | 99.00th=[17536], 99.50th=[42752], 99.90th=[203776], 99.95th=[618496],
     | 99.99th=[1630208]
    bw (KB  /s): min=    1, max=49876, per=23.33%, avg=18645.08, stdev=12601.51
  write: io=276373MB, bw=39306KB/s, iops=614, runt=7200017msec
    slat (usec): min=7, max=1704, avg=61.76, stdev=32.67
    clat (usec): min=1, max=4451.6K, avg=25498.25, stdev=100049.26
     lat (usec): min=238, max=4451.6K, avg=25560.30, stdev=100049.20
    clat percentiles (usec):
     |  1.00th=[  516],  5.00th=[ 1128], 10.00th=[ 4512], 20.00th=[ 7136],
     | 30.00th=[ 8512], 40.00th=[ 9408], 50.00th=[10432], 60.00th=[11584],
     | 70.00th=[13632], 80.00th=[18304], 90.00th=[46848], 95.00th=[69120],
     | 99.00th=[268288], 99.50th=[528384], 99.90th=[1630208], 99.95th=[2007040],
     | 99.99th=[2801664]
    bw (KB  /s): min=    1, max=24320, per=23.36%, avg=9182.16, stdev=6154.45
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=0.30%, 750=1.06%, 1000=7.12%
    lat (msec) : 2=45.24%, 4=7.74%, 10=18.80%, 20=13.23%, 50=3.18%
    lat (msec) : 100=2.08%, 250=0.85%, 500=0.20%, 750=0.06%, 1000=0.04%
    lat (msec) : 2000=0.09%, >=2000=0.02%
  cpu          : usr=0.46%, sys=0.93%, ctx=25464050, majf=0, minf=53324
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8989626/w=4421955/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
SATA-R5-11: (groupid=1, jobs=5): err= 0: pid=124025: Thu Jan 14 01:09:12 2016
  read : io=541625MB, bw=77031KB/s, iops=1203, runt=7200007msec
    slat (usec): min=4, max=4586, avg=28.44, stdev=22.38
    clat (usec): min=173, max=4145.7K, avg=3441.52, stdev=31559.03
     lat (usec): min=212, max=4145.7K, avg=3470.13, stdev=31559.11
    clat percentiles (usec):
     |  1.00th=[  716],  5.00th=[  860], 10.00th=[  948], 20.00th=[ 1064],
     | 30.00th=[ 1160], 40.00th=[ 1240], 50.00th=[ 1336], 60.00th=[ 1464],
     | 70.00th=[ 1688], 80.00th=[ 2864], 90.00th=[ 5472], 95.00th=[ 8160],
     | 99.00th=[16768], 99.50th=[41216], 99.90th=[203776], 99.95th=[618496],
     | 99.99th=[1630208]
    bw (KB  /s): min=    1, max=52015, per=23.35%, avg=17983.27, stdev=12493.88
  write: io=266421MB, bw=37891KB/s, iops=592, runt=7200007msec
    slat (usec): min=8, max=799, avg=48.70, stdev=30.16
    clat (usec): min=13, max=4225.5K, avg=26667.18, stdev=101865.43
     lat (usec): min=233, max=4225.6K, avg=26716.07, stdev=101865.42
    clat percentiles (usec):
     |  1.00th=[  490],  5.00th=[ 1080], 10.00th=[ 4768], 20.00th=[ 7264],
     | 30.00th=[ 8512], 40.00th=[ 9408], 50.00th=[10304], 60.00th=[11584],
     | 70.00th=[14144], 80.00th=[20096], 90.00th=[52480], 95.00th=[72192],
     | 99.00th=[276480], 99.50th=[561152], 99.90th=[1646592], 99.95th=[2023424],
     | 99.99th=[2834432]
    bw (KB  /s): min=    1, max=24271, per=23.35%, avg=8848.93, stdev=6096.29
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.36%
    lat (usec) : 750=1.44%, 1000=9.13%
    lat (msec) : 2=41.96%, 4=7.67%, 10=19.71%, 20=12.58%, 50=3.39%
    lat (msec) : 100=2.49%, 250=0.85%, 500=0.20%, 750=0.06%, 1000=0.05%
    lat (msec) : 2000=0.09%, >=2000=0.02%
  cpu          : usr=0.31%, sys=0.68%, ctx=24804683, majf=0, minf=52481
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8665992/w=4262726/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-01: (groupid=2, jobs=5): err= 0: pid=124030: Thu Jan 14 01:09:12 2016
  read : io=459320MB, bw=65325KB/s, iops=1020, runt=7200008msec
    slat (usec): min=4, max=707, avg=33.35, stdev=24.58
    clat (usec): min=1, max=4135.8K, avg=4003.89, stdev=36160.41
     lat (usec): min=184, max=4135.8K, avg=4037.47, stdev=36160.47
    clat percentiles (usec):
     |  1.00th=[  556],  5.00th=[  708], 10.00th=[  804], 20.00th=[  980],
     | 30.00th=[ 1240], 40.00th=[ 1848], 50.00th=[ 2576], 60.00th=[ 3056],
     | 70.00th=[ 3632], 80.00th=[ 4320], 90.00th=[ 5152], 95.00th=[ 5920],
     | 99.00th=[ 8512], 99.50th=[10944], 99.90th=[280576], 99.95th=[806912],
     | 99.99th=[1810432]
    bw (KB  /s): min=    1, max=71808, per=23.48%, avg=15340.61, stdev=10358.33
  write: io=226187MB, bw=32169KB/s, iops=502, runt=7200008msec
    slat (usec): min=7, max=4006, avg=54.01, stdev=31.89
    clat (usec): min=14, max=4152.2K, avg=11628.96, stdev=66949.59
     lat (usec): min=225, max=4152.3K, avg=11683.22, stdev=66949.55
    clat percentiles (usec):
     |  1.00th=[  442],  5.00th=[  756], 10.00th=[ 1004], 20.00th=[ 1656],
     | 30.00th=[ 2512], 40.00th=[ 4256], 50.00th=[ 6816], 60.00th=[ 9152],
     | 70.00th=[11072], 80.00th=[13120], 90.00th=[15552], 95.00th=[17536],
     | 99.00th=[92672], 99.50th=[171008], 99.90th=[1187840], 99.95th=[1597440],
     | 99.99th=[2473984]
    bw (KB  /s): min=    1, max=34304, per=23.49%, avg=7557.26, stdev=5088.12
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=0.84%, 750=5.52%, 1000=10.88%
    lat (msec) : 2=18.86%, 4=27.28%, 10=24.41%, 20=11.12%, 50=0.45%
    lat (msec) : 100=0.10%, 250=0.33%, 500=0.06%, 750=0.03%, 1000=0.03%
    lat (msec) : 2000=0.06%, >=2000=0.01%
  cpu          : usr=0.32%, sys=0.67%, ctx=21188579, majf=0, minf=29692
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=7349120/w=3618995/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=5): err= 0: pid=124035: Thu Jan 14 01:09:12 2016
  read : io=491103MB, bw=69846KB/s, iops=1091, runt=7200010msec
    slat (usec): min=4, max=2865, avg=27.03, stdev=21.01
    clat (usec): min=1, max=4152.2K, avg=3708.24, stdev=34429.23
     lat (usec): min=180, max=4152.2K, avg=3735.46, stdev=34429.24
    clat percentiles (usec):
     |  1.00th=[  548],  5.00th=[  700], 10.00th=[  804], 20.00th=[  980],
     | 30.00th=[ 1208], 40.00th=[ 1688], 50.00th=[ 2352], 60.00th=[ 2832],
     | 70.00th=[ 3312], 80.00th=[ 3952], 90.00th=[ 4768], 95.00th=[ 5408],
     | 99.00th=[ 7904], 99.50th=[10048], 99.90th=[240640], 99.95th=[733184],
     | 99.99th=[1728512]
    bw (KB  /s): min=   24, max=72960, per=23.47%, avg=16392.54, stdev=10303.07
  write: io=241694MB, bw=34374KB/s, iops=537, runt=7200010msec
    slat (usec): min=7, max=2911, avg=45.92, stdev=29.14
    clat (usec): min=1, max=4155.5K, avg=10972.44, stdev=65428.67
     lat (usec): min=226, max=4155.6K, avg=11018.55, stdev=65428.59
    clat percentiles (usec):
     |  1.00th=[  430],  5.00th=[  764], 10.00th=[ 1048], 20.00th=[ 1784],
     | 30.00th=[ 2640], 40.00th=[ 4192], 50.00th=[ 6368], 60.00th=[ 8384],
     | 70.00th=[10176], 80.00th=[11968], 90.00th=[14272], 95.00th=[16192],
     | 99.00th=[81408], 99.50th=[154624], 99.90th=[1155072], 99.95th=[1564672],
     | 99.99th=[2408448]
    bw (KB  /s): min=    1, max=37248, per=23.48%, avg=8070.26, stdev=5056.88
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.02%
    lat (usec) : 500=0.93%, 750=5.51%, 1000=10.76%
    lat (msec) : 2=20.09%, 4=29.62%, 10=22.47%, 20=9.77%, 50=0.23%
    lat (msec) : 100=0.10%, 250=0.32%, 500=0.06%, 750=0.03%, 1000=0.03%
    lat (msec) : 2000=0.05%, >=2000=0.01%
  cpu          : usr=0.26%, sys=0.60%, ctx=22845306, majf=0, minf=27743
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=7857655/w=3867096/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=561852MB, aggrb=79907KB/s, minb=79907KB/s, maxb=79907KB/s, mint=7200017msec, maxt=7200017msec
  WRITE: io=276373MB, aggrb=39306KB/s, minb=39306KB/s, maxb=39306KB/s, mint=7200017msec, maxt=7200017msec

Run status group 1 (all jobs):
   READ: io=541625MB, aggrb=77030KB/s, minb=77030KB/s, maxb=77030KB/s, mint=7200007msec, maxt=7200007msec
  WRITE: io=266421MB, aggrb=37890KB/s, minb=37890KB/s, maxb=37890KB/s, mint=7200007msec, maxt=7200007msec

Run status group 2 (all jobs):
   READ: io=459320MB, aggrb=65325KB/s, minb=65325KB/s, maxb=65325KB/s, mint=7200008msec, maxt=7200008msec
  WRITE: io=226187MB, aggrb=32168KB/s, minb=32168KB/s, maxb=32168KB/s, mint=7200008msec, maxt=7200008msec

Run status group 3 (all jobs):
   READ: io=491103MB, aggrb=69845KB/s, minb=69845KB/s, maxb=69845KB/s, mint=7200010msec, maxt=7200010msec
  WRITE: io=241694MB, aggrb=34374KB/s, minb=34374KB/s, maxb=34374KB/s, mint=7200010msec, maxt=7200010msec



# 
# Thu 14 Jan 2016 01:09:15 AM EST
# 1mB write with SATA 1 jobs and 2 I/Os Outstanding
# 1mB write with NVMe 1 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 336 IOPS | 344 MBps | 5940 μsec Average Latency 
SATA-R5-1: 308 IOPS | 316 MBps | 6470 μsec Average Latency 
NVMe-R5-0: 140 IOPS | 143 MBps | 57100 μsec Average Latency 
NVMe-R5-1: 137 IOPS | 141 MBps | 58090 μsec Average Latency 

Totals
IOPS: 921
MBPS: 944
Average Latency (μsec): 31900

SATA-R5-01: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=2
SATA-R5-11: (g=1): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=2
NVMe-R5-01: (g=2): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
NVMe-R5-11: (g=3): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=130013: Thu Jan 14 03:10:16 2016
  write: io=2367.5GB, bw=344778KB/s, iops=336, runt=7200002msec
    slat (usec): min=87, max=1354, avg=180.39, stdev=43.01
    clat (usec): min=881, max=558365, avg=5757.27, stdev=14334.50
     lat (msec): min=1, max=558, avg= 5.94, stdev=14.33
    clat percentiles (usec):
     |  1.00th=[ 1368],  5.00th=[ 1576], 10.00th=[ 1816], 20.00th=[ 1928],
     | 30.00th=[ 1992], 40.00th=[ 2064], 50.00th=[ 2128], 60.00th=[ 2224],
     | 70.00th=[ 2448], 80.00th=[ 3024], 90.00th=[ 6880], 95.00th=[37632],
     | 99.00th=[69120], 99.50th=[84480], 99.90th=[138240], 99.95th=[181248],
     | 99.99th=[284672]
    bw (KB  /s): min=   17, max=870706, per=100.00%, avg=346944.00, stdev=119148.23
    lat (usec) : 1000=0.01%
    lat (msec) : 2=30.15%, 4=55.15%, 10=8.16%, 20=0.70%, 50=2.87%
    lat (msec) : 100=2.68%, 250=0.27%, 500=0.02%, 750=0.01%
  cpu          : usr=1.39%, sys=2.80%, ctx=11719727, majf=0, minf=67765
  IO depths    : 1=0.1%, 2=100.6%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=2424218/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=130014: Thu Jan 14 03:10:16 2016
  write: io=2171.3GB, bw=316208KB/s, iops=308, runt=7200002msec
    slat (usec): min=91, max=1322, avg=265.80, stdev=72.85
    clat (msec): min=1, max=657, avg= 6.21, stdev=15.54
     lat (msec): min=1, max=657, avg= 6.47, stdev=15.54
    clat percentiles (usec):
     |  1.00th=[ 1448],  5.00th=[ 1720], 10.00th=[ 1832], 20.00th=[ 1928],
     | 30.00th=[ 1992], 40.00th=[ 2064], 50.00th=[ 2128], 60.00th=[ 2224],
     | 70.00th=[ 2416], 80.00th=[ 2960], 90.00th=[ 7264], 95.00th=[41216],
     | 99.00th=[74240], 99.50th=[89600], 99.90th=[146432], 99.95th=[199680],
     | 99.99th=[288768]
    bw (KB  /s): min=   17, max=821652, per=100.00%, avg=318227.55, stdev=110113.03
    lat (msec) : 2=30.06%, 4=54.78%, 10=7.96%, 20=0.74%, 50=2.86%
    lat (msec) : 100=3.27%, 250=0.32%, 500=0.02%, 750=0.01%
  cpu          : usr=1.94%, sys=3.45%, ctx=10446598, majf=0, minf=67242
  IO depths    : 1=0.1%, 2=100.6%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=2223334/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=130015: Thu Jan 14 03:10:16 2016
  write: io=985.14GB, bw=143466KB/s, iops=140, runt=7200189msec
    slat (usec): min=110, max=1630, avg=412.83, stdev=93.93
    clat (msec): min=1, max=1376, avg=56.69, stdev=71.71
     lat (msec): min=1, max=1377, avg=57.10, stdev=71.71
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   11], 60.00th=[   57],
     | 70.00th=[   77], 80.00th=[  105], 90.00th=[  155], 95.00th=[  200],
     | 99.00th=[  285], 99.50th=[  371], 99.90th=[  553], 99.95th=[  619],
     | 99.99th=[  824]
    bw (KB  /s): min=   17, max=697536, per=100.00%, avg=144850.24, stdev=53599.41
    lat (msec) : 2=0.36%, 4=8.02%, 10=40.69%, 20=4.54%, 50=3.99%
    lat (msec) : 100=21.14%, 250=19.61%, 500=1.53%, 750=0.11%, 1000=0.02%
    lat (msec) : 2000=0.01%
  cpu          : usr=2.46%, sys=1.69%, ctx=4578544, majf=0, minf=69336
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1008765/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=130016: Thu Jan 14 03:10:16 2016
  write: io=991494MB, bw=141010KB/s, iops=137, runt=7200102msec
    slat (usec): min=105, max=1729, avg=406.20, stdev=93.74
    clat (msec): min=1, max=1297, avg=57.69, stdev=72.16
     lat (msec): min=1, max=1298, avg=58.09, stdev=72.15
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   12], 60.00th=[   58],
     | 70.00th=[   79], 80.00th=[  109], 90.00th=[  159], 95.00th=[  202],
     | 99.00th=[  289], 99.50th=[  359], 99.90th=[  529], 99.95th=[  611],
     | 99.99th=[  807]
    bw (KB  /s): min=   17, max=710656, per=100.00%, avg=142146.23, stdev=51799.43
    lat (msec) : 2=0.37%, 4=8.30%, 10=39.94%, 20=4.68%, 50=4.04%
    lat (msec) : 100=20.61%, 250=20.33%, 500=1.61%, 750=0.10%, 1000=0.02%
    lat (msec) : 2000=0.01%
  cpu          : usr=2.43%, sys=1.64%, ctx=4500468, majf=0, minf=69861
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=991487/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: io=2367.5GB, aggrb=344777KB/s, minb=344777KB/s, maxb=344777KB/s, mint=7200002msec, maxt=7200002msec

Run status group 1 (all jobs):
  WRITE: io=2171.3GB, aggrb=316207KB/s, minb=316207KB/s, maxb=316207KB/s, mint=7200002msec, maxt=7200002msec

Run status group 2 (all jobs):
  WRITE: io=985.14GB, aggrb=143466KB/s, minb=143466KB/s, maxb=143466KB/s, mint=7200189msec, maxt=7200189msec

Run status group 3 (all jobs):
  WRITE: io=991494MB, aggrb=141010KB/s, minb=141010KB/s, maxb=141010KB/s, mint=7200102msec, maxt=7200102msec



# 
# Thu 14 Jan 2016 03:10:18 AM EST
# 1mB read with SATA 1 jobs and 8 I/Os Outstanding
# 1mB read with NVMe 1 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 690 IOPS | 706 MBps | 11580 μsec Average Latency 
SATA-R5-1: 689 IOPS | 706 MBps | 11590 μsec Average Latency 
NVMe-R5-0: 321 IOPS | 328 MBps | 6220 μsec Average Latency 
NVMe-R5-1: 320 IOPS | 328 MBps | 6240 μsec Average Latency 

Totals
IOPS: 2020
MBPS: 2068
Average Latency (μsec): 8907

SATA-R5-01: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
SATA-R5-11: (g=1): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
NVMe-R5-01: (g=2): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=2
NVMe-R5-11: (g=3): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=2
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=135146: Thu Jan 14 05:11:18 2016
  read : io=4854.5GB, bw=706979KB/s, iops=690, runt=7200009msec
    slat (usec): min=65, max=10568, avg=213.13, stdev=250.29
    clat (usec): min=999, max=35731, avg=11370.05, stdev=2921.00
     lat (msec): min=1, max=35, avg=11.58, stdev= 2.91
    clat percentiles (usec):
     |  1.00th=[ 5280],  5.00th=[ 6880], 10.00th=[ 8032], 20.00th=[ 9280],
     | 30.00th=[ 9920], 40.00th=[10432], 50.00th=[10816], 60.00th=[11456],
     | 70.00th=[12480], 80.00th=[13760], 90.00th=[15424], 95.00th=[16768],
     | 99.00th=[19328], 99.50th=[20352], 99.90th=[23424], 99.95th=[24704],
     | 99.99th=[27776]
    bw (KB  /s): min=   17, max=796672, per=100.00%, avg=707594.29, stdev=34740.49
    lat (usec) : 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.08%, 10=31.30%, 20=68.00%, 50=0.62%
  cpu          : usr=0.37%, sys=8.34%, ctx=24522886, majf=0, minf=73528
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=4970946/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=135147: Thu Jan 14 05:11:18 2016
  read : io=4850.8GB, bw=706442KB/s, iops=689, runt=7200007msec
    slat (usec): min=64, max=10016, avg=213.20, stdev=242.33
    clat (msec): min=1, max=42, avg=11.38, stdev= 2.93
     lat (msec): min=1, max=43, avg=11.59, stdev= 2.92
    clat percentiles (usec):
     |  1.00th=[ 5280],  5.00th=[ 6880], 10.00th=[ 8032], 20.00th=[ 9280],
     | 30.00th=[ 9920], 40.00th=[10432], 50.00th=[10816], 60.00th=[11456],
     | 70.00th=[12480], 80.00th=[13760], 90.00th=[15424], 95.00th=[16768],
     | 99.00th=[19328], 99.50th=[20352], 99.90th=[23424], 99.95th=[24704],
     | 99.99th=[27776]
    bw (KB  /s): min=   17, max=806912, per=100.00%, avg=707065.27, stdev=35359.39
    lat (msec) : 2=0.01%, 4=0.08%, 10=31.44%, 20=67.85%, 50=0.63%
  cpu          : usr=0.38%, sys=8.28%, ctx=24493562, majf=0, minf=76155
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=4967170/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=135148: Thu Jan 14 05:11:18 2016
  read : io=2257.5GB, bw=328769KB/s, iops=321, runt=7200006msec
    slat (usec): min=68, max=7222, avg=207.60, stdev=85.90
    clat (usec): min=135, max=28613, avg=6016.05, stdev=2973.89
     lat (msec): min=1, max=28, avg= 6.22, stdev= 2.98
    clat percentiles (usec):
     |  1.00th=[ 1464],  5.00th=[ 1944], 10.00th=[ 2512], 20.00th=[ 3696],
     | 30.00th=[ 4512], 40.00th=[ 4960], 50.00th=[ 5344], 60.00th=[ 5792],
     | 70.00th=[ 6624], 80.00th=[ 8512], 90.00th=[10688], 95.00th=[11840],
     | 99.00th=[14016], 99.50th=[15040], 99.90th=[17536], 99.95th=[18816],
     | 99.99th=[21376]
    bw (KB  /s): min=   17, max=407968, per=100.00%, avg=329085.57, stdev=26062.61
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=5.49%, 4=17.70%, 10=63.50%, 20=13.28%, 50=0.02%
  cpu          : usr=0.27%, sys=3.71%, ctx=11540346, majf=0, minf=67253
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2311656/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=135149: Thu Jan 14 05:11:18 2016
  read : io=2253.6GB, bw=328193KB/s, iops=320, runt=7200005msec
    slat (usec): min=72, max=8235, avg=208.08, stdev=83.95
    clat (usec): min=291, max=30004, avg=6026.65, stdev=2966.48
     lat (msec): min=1, max=30, avg= 6.24, stdev= 2.97
    clat percentiles (usec):
     |  1.00th=[ 1480],  5.00th=[ 1976], 10.00th=[ 2544], 20.00th=[ 3696],
     | 30.00th=[ 4512], 40.00th=[ 4960], 50.00th=[ 5344], 60.00th=[ 5856],
     | 70.00th=[ 6624], 80.00th=[ 8512], 90.00th=[10688], 95.00th=[11840],
     | 99.00th=[14016], 99.50th=[15040], 99.90th=[17536], 99.95th=[18560],
     | 99.99th=[21376]
    bw (KB  /s): min=   17, max=398222, per=100.00%, avg=328508.87, stdev=26082.93
    lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=5.15%, 4=18.00%, 10=63.58%, 20=13.25%, 50=0.02%
  cpu          : usr=0.26%, sys=3.71%, ctx=11522692, majf=0, minf=67382
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2307605/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=4854.5GB, aggrb=706979KB/s, minb=706979KB/s, maxb=706979KB/s, mint=7200009msec, maxt=7200009msec

Run status group 1 (all jobs):
   READ: io=4850.8GB, aggrb=706442KB/s, minb=706442KB/s, maxb=706442KB/s, mint=7200007msec, maxt=7200007msec

Run status group 2 (all jobs):
   READ: io=2257.5GB, aggrb=328768KB/s, minb=328768KB/s, maxb=328768KB/s, mint=7200006msec, maxt=7200006msec

Run status group 3 (all jobs):
   READ: io=2253.6GB, aggrb=328192KB/s, minb=328192KB/s, maxb=328192KB/s, mint=7200005msec, maxt=7200005msec



# 
# Thu 14 Jan 2016 05:11:21 AM EST
# 256kB write with SATA 1 jobs and 8 I/Os Outstanding
# 256kB write with NVMe 1 jobs and 1 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 1427 IOPS | 365 MBps | 5600 μsec Average Latency 
SATA-R5-1: 1401 IOPS | 358 MBps | 5706 μsec Average Latency 
NVMe-R5-0: 490 IOPS | 125 MBps | 2034 μsec Average Latency 
NVMe-R5-1: 495 IOPS | 126 MBps | 2016 μsec Average Latency 

Totals
IOPS: 3813
MBPS: 974
Average Latency (μsec): 3839

SATA-R5-01: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=8
SATA-R5-11: (g=1): rw=write, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=8
NVMe-R5-01: (g=2): rw=write, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=1
NVMe-R5-11: (g=3): rw=write, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=140352: Thu Jan 14 07:12:22 2016
  write: io=2509.9GB, bw=365525KB/s, iops=1427, runt=7200002msec
    slat (usec): min=22, max=3335, avg=79.33, stdev=26.91
    clat (usec): min=376, max=527138, avg=5520.57, stdev=15016.08
     lat (usec): min=566, max=527218, avg=5600.19, stdev=15015.74
    clat percentiles (usec):
     |  1.00th=[  756],  5.00th=[ 1004], 10.00th=[ 1208], 20.00th=[ 1464],
     | 30.00th=[ 1592], 40.00th=[ 1672], 50.00th=[ 1768], 60.00th=[ 1832],
     | 70.00th=[ 1928], 80.00th=[ 2160], 90.00th=[ 3920], 95.00th=[37120],
     | 99.00th=[67072], 99.50th=[94720], 99.90th=[201728], 99.95th=[203776],
     | 99.99th=[209920]
    bw (KB  /s): min=    4, max=742400, per=100.00%, avg=366814.40, stdev=88534.81
    lat (usec) : 500=0.01%, 750=0.91%, 1000=3.96%
    lat (msec) : 2=70.29%, 4=14.96%, 10=2.04%, 20=0.25%, 50=5.23%
    lat (msec) : 100=1.94%, 250=0.42%, 500=0.01%, 750=0.01%
  cpu          : usr=2.80%, sys=5.39%, ctx=18166858, majf=0, minf=67900
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10280390/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=140353: Thu Jan 14 07:12:22 2016
  write: io=2463.9GB, bw=358817KB/s, iops=1401, runt=7200003msec
    slat (usec): min=22, max=3275, avg=46.24, stdev=14.10
    clat (usec): min=326, max=441881, avg=5659.81, stdev=15551.75
     lat (usec): min=496, max=441921, avg=5706.14, stdev=15551.39
    clat percentiles (usec):
     |  1.00th=[  748],  5.00th=[  980], 10.00th=[ 1176], 20.00th=[ 1448],
     | 30.00th=[ 1592], 40.00th=[ 1704], 50.00th=[ 1800], 60.00th=[ 1848],
     | 70.00th=[ 1960], 80.00th=[ 2192], 90.00th=[ 3888], 95.00th=[39680],
     | 99.00th=[71168], 99.50th=[95744], 99.90th=[201728], 99.95th=[203776],
     | 99.99th=[214016]
    bw (KB  /s): min=    4, max=783603, per=100.00%, avg=360114.74, stdev=88377.48
    lat (usec) : 500=0.01%, 750=1.03%, 1000=4.42%
    lat (msec) : 2=68.69%, 4=16.04%, 10=2.23%, 20=0.29%, 50=4.43%
    lat (msec) : 100=2.42%, 250=0.44%, 500=0.01%
  cpu          : usr=1.68%, sys=3.56%, ctx=19324092, majf=0, minf=66854
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10091714/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=140354: Thu Jan 14 07:12:22 2016
  write: io=882160MB, bw=125462KB/s, iops=490, runt=7200017msec
    slat (usec): min=27, max=2860, avg=94.58, stdev=19.90
    clat (usec): min=358, max=401639, avg=1939.98, stdev=8746.48
     lat (usec): min=499, max=401714, avg=2034.90, stdev=8745.76
    clat percentiles (usec):
     |  1.00th=[  510],  5.00th=[  540], 10.00th=[  548], 20.00th=[  564],
     | 30.00th=[  588], 40.00th=[  596], 50.00th=[  620], 60.00th=[  660],
     | 70.00th=[  788], 80.00th=[ 1560], 90.00th=[ 1880], 95.00th=[ 2128],
     | 99.00th=[47360], 99.50th=[66048], 99.90th=[109056], 99.95th=[142336],
     | 99.99th=[203776]
    bw (KB  /s): min=    4, max=290304, per=100.00%, avg=125990.22, stdev=35530.79
    lat (usec) : 500=0.33%, 750=68.56%, 1000=4.34%
    lat (msec) : 2=20.55%, 4=4.08%, 10=0.36%, 20=0.07%, 50=0.80%
    lat (msec) : 100=0.78%, 250=0.14%, 500=0.01%
  cpu          : usr=1.27%, sys=2.02%, ctx=7110654, majf=0, minf=9308
  IO depths    : 1=100.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=3528641/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=140355: Thu Jan 14 07:12:22 2016
  write: io=891444MB, bw=126782KB/s, iops=495, runt=7200051msec
    slat (usec): min=23, max=1791, avg=71.15, stdev=20.99
    clat (usec): min=363, max=400132, avg=1945.35, stdev=8863.10
     lat (usec): min=466, max=400205, avg=2016.69, stdev=8862.48
    clat percentiles (usec):
     |  1.00th=[  502],  5.00th=[  532], 10.00th=[  540], 20.00th=[  556],
     | 30.00th=[  572], 40.00th=[  580], 50.00th=[  604], 60.00th=[  636],
     | 70.00th=[  740], 80.00th=[ 1464], 90.00th=[ 1880], 95.00th=[ 2128],
     | 99.00th=[48896], 99.50th=[67072], 99.90th=[109056], 99.95th=[142336],
     | 99.99th=[203776]
    bw (KB  /s): min=    4, max=305152, per=100.00%, avg=127330.14, stdev=36423.81
    lat (usec) : 500=0.91%, 750=69.32%, 1000=4.24%
    lat (msec) : 2=19.24%, 4=4.12%, 10=0.35%, 20=0.07%, 50=0.79%
    lat (msec) : 100=0.82%, 250=0.14%, 500=0.01%
  cpu          : usr=0.83%, sys=2.12%, ctx=7206417, majf=0, minf=9584
  IO depths    : 1=100.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=3565774/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: io=2509.9GB, aggrb=365525KB/s, minb=365525KB/s, maxb=365525KB/s, mint=7200002msec, maxt=7200002msec

Run status group 1 (all jobs):
  WRITE: io=2463.9GB, aggrb=358816KB/s, minb=358816KB/s, maxb=358816KB/s, mint=7200003msec, maxt=7200003msec

Run status group 2 (all jobs):
  WRITE: io=882160MB, aggrb=125462KB/s, minb=125462KB/s, maxb=125462KB/s, mint=7200017msec, maxt=7200017msec

Run status group 3 (all jobs):
  WRITE: io=891444MB, aggrb=126782KB/s, minb=126782KB/s, maxb=126782KB/s, mint=7200051msec, maxt=7200051msec



# 
# Thu 14 Jan 2016 07:12:25 AM EST
# 256kB read with SATA 1 jobs and 8 I/Os Outstanding
# 256kB read with NVMe 1 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 2201 IOPS | 563 MBps | 3631 μsec Average Latency 
SATA-R5-1: 2199 IOPS | 562 MBps | 3635 μsec Average Latency 
NVMe-R5-0: 1914 IOPS | 490 MBps | 2086 μsec Average Latency 
NVMe-R5-1: 1899 IOPS | 486 MBps | 2103 μsec Average Latency 

Totals
IOPS: 8213
MBPS: 2101
Average Latency (μsec): 2863

SATA-R5-01: (g=0): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=8
SATA-R5-11: (g=1): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=8
NVMe-R5-01: (g=2): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=4
NVMe-R5-11: (g=3): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=4
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=145475: Thu Jan 14 09:13:25 2016
  read : io=3870.6GB, bw=563687KB/s, iops=2201, runt=7200003msec
    slat (usec): min=15, max=3836, avg=56.95, stdev=40.41
    clat (usec): min=1, max=28919, avg=3573.85, stdev=1055.72
     lat (usec): min=853, max=28942, avg=3631.05, stdev=1054.14
    clat percentiles (usec):
     |  1.00th=[ 1832],  5.00th=[ 2352], 10.00th=[ 2608], 20.00th=[ 2896],
     | 30.00th=[ 3056], 40.00th=[ 3216], 50.00th=[ 3344], 60.00th=[ 3536],
     | 70.00th=[ 3760], 80.00th=[ 4128], 90.00th=[ 4768], 95.00th=[ 5408],
     | 99.00th=[ 7584], 99.50th=[ 8512], 99.90th=[11072], 99.95th=[12224],
     | 99.99th=[15168]
    bw (KB  /s): min=    4, max=617472, per=100.00%, avg=564026.79, stdev=25656.89
    lat (usec) : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=1.80%, 4=75.03%, 10=22.99%, 20=0.19%, 50=0.01%
  cpu          : usr=0.84%, sys=7.64%, ctx=28424012, majf=0, minf=69452
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=15853708/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=145476: Thu Jan 14 09:13:25 2016
  read : io=3865.8GB, bw=562988KB/s, iops=2199, runt=7200003msec
    slat (usec): min=14, max=3363, avg=60.95, stdev=40.23
    clat (usec): min=1, max=24836, avg=3574.41, stdev=1076.89
     lat (usec): min=827, max=24870, avg=3635.55, stdev=1075.18
    clat percentiles (usec):
     |  1.00th=[ 1800],  5.00th=[ 2320], 10.00th=[ 2608], 20.00th=[ 2864],
     | 30.00th=[ 3056], 40.00th=[ 3184], 50.00th=[ 3344], 60.00th=[ 3536],
     | 70.00th=[ 3792], 80.00th=[ 4128], 90.00th=[ 4768], 95.00th=[ 5472],
     | 99.00th=[ 7648], 99.50th=[ 8640], 99.90th=[11200], 99.95th=[12480],
     | 99.99th=[15296]
    bw (KB  /s): min=    4, max=619008, per=100.00%, avg=563326.35, stdev=26566.56
    lat (usec) : 2=0.01%, 10=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
    lat (usec) : 1000=0.01%
    lat (msec) : 2=2.00%, 4=74.51%, 10=23.29%, 20=0.20%, 50=0.01%
  cpu          : usr=0.85%, sys=8.22%, ctx=28163576, majf=0, minf=67886
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=15834034/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=145477: Thu Jan 14 09:13:25 2016
  read : io=3366.2GB, bw=490227KB/s, iops=1914, runt=7200003msec
    slat (usec): min=16, max=3127, avg=65.48, stdev=43.04
    clat (usec): min=1, max=22106, avg=2020.65, stdev=886.51
     lat (usec): min=367, max=22184, avg=2086.33, stdev=886.17
    clat percentiles (usec):
     |  1.00th=[  596],  5.00th=[  916], 10.00th=[ 1144], 20.00th=[ 1416],
     | 30.00th=[ 1608], 40.00th=[ 1736], 50.00th=[ 1880], 60.00th=[ 2024],
     | 70.00th=[ 2192], 80.00th=[ 2480], 90.00th=[ 3056], 95.00th=[ 3568],
     | 99.00th=[ 5088], 99.50th=[ 6176], 99.90th=[ 8640], 99.95th=[ 9792],
     | 99.99th=[12608]
    bw (KB  /s): min=    4, max=548864, per=100.00%, avg=490533.04, stdev=21022.08
    lat (usec) : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=0.37%, 750=2.23%, 1000=4.00%
    lat (msec) : 2=51.98%, 4=38.65%, 10=2.72%, 20=0.04%, 50=0.01%
  cpu          : usr=0.83%, sys=7.39%, ctx=25442384, majf=0, minf=35299
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=13787646/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=145478: Thu Jan 14 09:13:25 2016
  read : io=3338.5GB, bw=486198KB/s, iops=1899, runt=7200003msec
    slat (usec): min=16, max=4966, avg=64.85, stdev=42.53
    clat (usec): min=1, max=25205, avg=2038.58, stdev=898.23
     lat (usec): min=364, max=25344, avg=2103.62, stdev=897.68
    clat percentiles (usec):
     |  1.00th=[  596],  5.00th=[  916], 10.00th=[ 1160], 20.00th=[ 1432],
     | 30.00th=[ 1608], 40.00th=[ 1752], 50.00th=[ 1896], 60.00th=[ 2040],
     | 70.00th=[ 2224], 80.00th=[ 2512], 90.00th=[ 3088], 95.00th=[ 3632],
     | 99.00th=[ 5088], 99.50th=[ 6304], 99.90th=[ 8768], 99.95th=[10048],
     | 99.99th=[12864]
    bw (KB  /s): min=    4, max=557056, per=100.00%, avg=486508.51, stdev=22407.48
    lat (usec) : 2=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.37%
    lat (usec) : 750=2.23%, 1000=3.80%
    lat (msec) : 2=51.07%, 4=39.62%, 10=2.86%, 20=0.05%, 50=0.01%
  cpu          : usr=0.84%, sys=7.34%, ctx=25236437, majf=0, minf=34899
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=13674335/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=3870.6GB, aggrb=563687KB/s, minb=563687KB/s, maxb=563687KB/s, mint=7200003msec, maxt=7200003msec

Run status group 1 (all jobs):
   READ: io=3865.8GB, aggrb=562987KB/s, minb=562987KB/s, maxb=562987KB/s, mint=7200003msec, maxt=7200003msec

Run status group 2 (all jobs):
   READ: io=3366.2GB, aggrb=490227KB/s, minb=490227KB/s, maxb=490227KB/s, mint=7200003msec, maxt=7200003msec

Run status group 3 (all jobs):
   READ: io=3338.5GB, aggrb=486198KB/s, minb=486198KB/s, maxb=486198KB/s, mint=7200003msec, maxt=7200003msec



# 
# Thu 14 Jan 2016 09:13:28 AM EST
# 64kB write with SATA 1 jobs and 4 I/Os Outstanding
# 64kB write with NVMe 1 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 6100 IOPS | 390 MBps | 654 μsec Average Latency 
SATA-R5-1: 6371 IOPS | 407 MBps | 626 μsec Average Latency 
NVMe-R5-0: 2016 IOPS | 129 MBps | 1982 μsec Average Latency 
NVMe-R5-1: 1938 IOPS | 124 MBps | 2061 μsec Average Latency 

Totals
IOPS: 16425
MBPS: 1050
Average Latency (μsec): 1330

SATA-R5-01: (g=0): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
SATA-R5-11: (g=1): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
NVMe-R5-01: (g=2): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
NVMe-R5-11: (g=3): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=150638: Thu Jan 14 11:14:29 2016
  write: io=2680.9GB, bw=390421KB/s, iops=6100, runt=7200058msec
    slat (usec): min=6, max=780, avg=23.77, stdev= 9.36
    clat (usec): min=85, max=468632, avg=630.64, stdev=3931.34
     lat (usec): min=187, max=468654, avg=654.56, stdev=3931.32
    clat percentiles (usec):
     |  1.00th=[  266],  5.00th=[  310], 10.00th=[  334], 20.00th=[  350],
     | 30.00th=[  366], 40.00th=[  390], 50.00th=[  422], 60.00th=[  450],
     | 70.00th=[  474], 80.00th=[  494], 90.00th=[  564], 95.00th=[  652],
     | 99.00th=[ 1384], 99.50th=[ 3920], 99.90th=[47360], 99.95th=[87552],
     | 99.99th=[195584]
    bw (KB  /s): min=    1, max=595968, per=100.00%, avg=392629.64, stdev=98495.26
    lat (usec) : 100=0.01%, 250=0.54%, 500=80.72%, 750=15.61%, 1000=1.55%
    lat (msec) : 2=0.88%, 4=0.20%, 10=0.09%, 20=0.05%, 50=0.26%
    lat (msec) : 100=0.06%, 250=0.04%, 500=0.01%
  cpu          : usr=2.68%, sys=6.00%, ctx=75713859, majf=0, minf=9533
  IO depths    : 1=0.1%, 2=0.1%, 4=101.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=43922747/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=150639: Thu Jan 14 11:14:29 2016
  write: io=2799.9GB, bw=407750KB/s, iops=6371, runt=7200001msec
    slat (usec): min=6, max=1275, avg=23.97, stdev= 9.44
    clat (usec): min=81, max=465601, avg=602.58, stdev=3299.60
     lat (usec): min=186, max=465642, avg=626.70, stdev=3299.56
    clat percentiles (usec):
     |  1.00th=[  274],  5.00th=[  314], 10.00th=[  338], 20.00th=[  354],
     | 30.00th=[  370], 40.00th=[  394], 50.00th=[  430], 60.00th=[  454],
     | 70.00th=[  478], 80.00th=[  498], 90.00th=[  564], 95.00th=[  652],
     | 99.00th=[ 1336], 99.50th=[ 2768], 99.90th=[41216], 99.95th=[72192],
     | 99.99th=[154624]
    bw (KB  /s): min=    1, max=583680, per=100.00%, avg=408790.74, stdev=70475.17
    lat (usec) : 100=0.01%, 250=0.40%, 500=80.41%, 750=16.16%, 1000=1.48%
    lat (msec) : 2=0.92%, 4=0.21%, 10=0.09%, 20=0.03%, 50=0.23%
    lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%
  cpu          : usr=2.81%, sys=6.22%, ctx=78928910, majf=0, minf=9823
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=45871823/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=150640: Thu Jan 14 11:14:29 2016
  write: io=907230MB, bw=129028KB/s, iops=2016, runt=7200001msec
    slat (usec): min=6, max=747, avg=36.71, stdev=22.16
    clat (usec): min=114, max=451750, avg=1945.28, stdev=12792.83
     lat (usec): min=201, max=451786, avg=1982.21, stdev=12792.75
    clat percentiles (usec):
     |  1.00th=[  290],  5.00th=[  330], 10.00th=[  358], 20.00th=[  398],
     | 30.00th=[  434], 40.00th=[  454], 50.00th=[  478], 60.00th=[  502],
     | 70.00th=[  540], 80.00th=[  596], 90.00th=[  700], 95.00th=[  884],
     | 99.00th=[50944], 99.50th=[93696], 99.90th=[199680], 99.95th=[199680],
     | 99.99th=[268288]
    bw (KB  /s): min=    1, max=406016, per=100.00%, avg=130295.55, stdev=49506.27
    lat (usec) : 250=0.21%, 500=58.77%, 750=33.37%, 1000=3.69%
    lat (msec) : 2=1.61%, 4=0.32%, 10=0.15%, 20=0.06%, 50=0.80%
    lat (msec) : 100=0.57%, 250=0.44%, 500=0.02%
  cpu          : usr=1.13%, sys=3.40%, ctx=24096178, majf=0, minf=9387
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=14515683/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=150641: Thu Jan 14 11:14:29 2016
  write: io=872384MB, bw=124071KB/s, iops=1938, runt=7200090msec
    slat (usec): min=6, max=7133, avg=36.74, stdev=22.55
    clat (usec): min=114, max=421699, avg=2024.54, stdev=13240.06
     lat (usec): min=210, max=421723, avg=2061.51, stdev=13239.98
    clat percentiles (usec):
     |  1.00th=[  294],  5.00th=[  338], 10.00th=[  366], 20.00th=[  406],
     | 30.00th=[  438], 40.00th=[  462], 50.00th=[  482], 60.00th=[  506],
     | 70.00th=[  548], 80.00th=[  596], 90.00th=[  692], 95.00th=[  876],
     | 99.00th=[53504], 99.50th=[99840], 99.90th=[199680], 99.95th=[201728],
     | 99.99th=[276480]
    bw (KB  /s): min=    1, max=419456, per=100.00%, avg=125198.15, stdev=47636.31
    lat (usec) : 250=0.16%, 500=57.52%, 750=34.87%, 1000=3.46%
    lat (msec) : 2=1.59%, 4=0.33%, 10=0.15%, 20=0.05%, 50=0.77%
    lat (msec) : 100=0.61%, 250=0.48%, 500=0.02%
  cpu          : usr=1.08%, sys=3.44%, ctx=23136673, majf=0, minf=9439
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=13958137/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: io=2680.9GB, aggrb=390421KB/s, minb=390421KB/s, maxb=390421KB/s, mint=7200058msec, maxt=7200058msec

Run status group 1 (all jobs):
  WRITE: io=2799.9GB, aggrb=407749KB/s, minb=407749KB/s, maxb=407749KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
  WRITE: io=907230MB, aggrb=129028KB/s, minb=129028KB/s, maxb=129028KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
  WRITE: io=872384MB, aggrb=124070KB/s, minb=124070KB/s, maxb=124070KB/s, mint=7200090msec, maxt=7200090msec



# 
# Thu 14 Jan 2016 11:14:31 AM EST
# 64kB read with SATA 1 jobs and 10 I/Os Outstanding
# 64kB read with NVMe 1 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 7078 IOPS | 453 MBps | 1411 μsec Average Latency 
SATA-R5-1: 7032 IOPS | 450 MBps | 1420 μsec Average Latency 
NVMe-R5-0: 8992 IOPS | 575 MBps | 888 μsec Average Latency 
NVMe-R5-1: 8855 IOPS | 566 MBps | 902 μsec Average Latency 

Totals
IOPS: 31957
MBPS: 2044
Average Latency (μsec): 1155

SATA-R5-01: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=10
SATA-R5-11: (g=1): rw=read, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=10
NVMe-R5-01: (g=2): rw=read, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8
NVMe-R5-11: (g=3): rw=read, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=155796: Thu Jan 14 13:15:31 2016
  read : io=3110.9GB, bw=453054KB/s, iops=7078, runt=7200001msec
    slat (usec): min=5, max=6999, avg=21.81, stdev=20.90
    clat (usec): min=1, max=43650, avg=1389.65, stdev=510.70
     lat (usec): min=167, max=43748, avg=1411.60, stdev=509.53
    clat percentiles (usec):
     |  1.00th=[  506],  5.00th=[  764], 10.00th=[  908], 20.00th=[ 1048],
     | 30.00th=[ 1160], 40.00th=[ 1240], 50.00th=[ 1336], 60.00th=[ 1432],
     | 70.00th=[ 1528], 80.00th=[ 1672], 90.00th=[ 1880], 95.00th=[ 2096],
     | 99.00th=[ 3056], 99.50th=[ 3696], 99.90th=[ 6048], 99.95th=[ 8032],
     | 99.99th=[10944]
    bw (KB  /s): min=    1, max=504192, per=100.00%, avg=453199.11, stdev=15570.10
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.02%
    lat (usec) : 500=0.92%, 750=3.58%, 1000=11.03%
    lat (msec) : 2=77.53%, 4=6.56%, 10=0.34%, 20=0.02%, 50=0.01%
  cpu          : usr=1.26%, sys=8.86%, ctx=76438726, majf=0, minf=22302
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=50968597/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=10
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=155797: Thu Jan 14 13:15:31 2016
  read : io=3090.7GB, bw=450100KB/s, iops=7032, runt=7200001msec
    slat (usec): min=5, max=6551, avg=21.63, stdev=20.49
    clat (usec): min=26, max=20896, avg=1399.10, stdev=559.65
     lat (usec): min=170, max=20922, avg=1420.87, stdev=558.63
    clat percentiles (usec):
     |  1.00th=[  498],  5.00th=[  748], 10.00th=[  884], 20.00th=[ 1032],
     | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1320], 60.00th=[ 1416],
     | 70.00th=[ 1528], 80.00th=[ 1688], 90.00th=[ 1912], 95.00th=[ 2192],
     | 99.00th=[ 3472], 99.50th=[ 4016], 99.90th=[ 6368], 99.95th=[ 8384],
     | 99.99th=[11072]
    bw (KB  /s): min=    1, max=500352, per=100.00%, avg=450245.15, stdev=15816.08
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.02%, 500=1.02%, 750=3.97%
    lat (usec) : 1000=11.80%
    lat (msec) : 2=75.29%, 4=7.39%, 10=0.49%, 20=0.03%, 50=0.01%
  cpu          : usr=1.25%, sys=8.83%, ctx=76061467, majf=0, minf=24176
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=50636199/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=10
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=155798: Thu Jan 14 13:15:31 2016
  read : io=3951.7GB, bw=575504KB/s, iops=8992, runt=7200001msec
    slat (usec): min=5, max=2936, avg=21.51, stdev=20.53
    clat (usec): min=0, max=18362, avg=867.03, stdev=385.02
     lat (usec): min=171, max=18373, avg=888.67, stdev=383.68
    clat percentiles (usec):
     |  1.00th=[  350],  5.00th=[  458], 10.00th=[  510], 20.00th=[  596],
     | 30.00th=[  668], 40.00th=[  732], 50.00th=[  796], 60.00th=[  876],
     | 70.00th=[  980], 80.00th=[ 1096], 90.00th=[ 1288], 95.00th=[ 1464],
     | 99.00th=[ 1880], 99.50th=[ 2128], 99.90th=[ 4384], 99.95th=[ 6304],
     | 99.99th=[10048]
    bw (KB  /s): min=    1, max=635648, per=100.00%, avg=575633.31, stdev=23538.96
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.11%
    lat (usec) : 500=8.88%, 750=33.69%, 1000=29.67%
    lat (msec) : 2=26.95%, 4=0.58%, 10=0.10%, 20=0.01%
  cpu          : usr=1.50%, sys=10.92%, ctx=94825294, majf=0, minf=18692
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=64744155/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=155799: Thu Jan 14 13:15:31 2016
  read : io=3891.5GB, bw=566730KB/s, iops=8855, runt=7200001msec
    slat (usec): min=4, max=4879, avg=21.49, stdev=20.23
    clat (usec): min=0, max=20068, avg=880.80, stdev=400.11
     lat (usec): min=175, max=20089, avg=902.42, stdev=398.79
    clat percentiles (usec):
     |  1.00th=[  346],  5.00th=[  450], 10.00th=[  506], 20.00th=[  596],
     | 30.00th=[  668], 40.00th=[  740], 50.00th=[  804], 60.00th=[  892],
     | 70.00th=[  996], 80.00th=[ 1128], 90.00th=[ 1320], 95.00th=[ 1512],
     | 99.00th=[ 1928], 99.50th=[ 2160], 99.90th=[ 4448], 99.95th=[ 6432],
     | 99.99th=[10304]
    bw (KB  /s): min=    1, max=633088, per=100.00%, avg=566860.51, stdev=24335.15
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.12%
    lat (usec) : 500=9.31%, 750=32.44%, 1000=28.39%
    lat (msec) : 2=28.94%, 4=0.69%, 10=0.10%, 20=0.01%, 50=0.01%
  cpu          : usr=1.51%, sys=10.88%, ctx=93690347, majf=0, minf=18278
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=63757141/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=3110.9GB, aggrb=453054KB/s, minb=453054KB/s, maxb=453054KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=3090.7GB, aggrb=450099KB/s, minb=450099KB/s, maxb=450099KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
   READ: io=3951.7GB, aggrb=575503KB/s, minb=575503KB/s, maxb=575503KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=3891.5GB, aggrb=566730KB/s, minb=566730KB/s, maxb=566730KB/s, mint=7200001msec, maxt=7200001msec



# 
# Thu 14 Jan 2016 01:15:34 PM EST
# 64kB randread with SATA 2 jobs and 2 I/Os Outstanding
# 64kB randread with NVMe 4 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 3714 IOPS | 237 MBps | 1074 μsec Average Latency 
SATA-R5-1: 3937 IOPS | 252 MBps | 1013 μsec Average Latency 
NVMe-R5-0: 8704 IOPS | 557 MBps | 916 μsec Average Latency 
NVMe-R5-1: 7601 IOPS | 486 MBps | 1050 μsec Average Latency 

Totals
IOPS: 23956
MBPS: 1532
Average Latency (μsec): 1013

SATA-R5-01: (g=0): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
SATA-R5-11: (g=1): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
NVMe-R5-01: (g=2): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 12 processes

SATA-R5-01: (groupid=0, jobs=2): err= 0: pid=160925: Thu Jan 14 15:16:35 2016
  read : io=1632.3GB, bw=237706KB/s, iops=3714, runt=7200001msec
    slat (usec): min=5, max=1681, avg=31.55, stdev=26.12
    clat (usec): min=16, max=20503, avg=1042.52, stdev=350.72
     lat (usec): min=201, max=20524, avg=1074.28, stdev=349.07
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[  756], 10.00th=[  812], 20.00th=[  868],
     | 30.00th=[  916], 40.00th=[  964], 50.00th=[ 1004], 60.00th=[ 1048],
     | 70.00th=[ 1096], 80.00th=[ 1160], 90.00th=[ 1256], 95.00th=[ 1352],
     | 99.00th=[ 1848], 99.50th=[ 3024], 99.90th=[ 5728], 99.95th=[ 6816],
     | 99.99th=[11200]
    bw (KB  /s): min=    1, max=127488, per=50.03%, avg=118927.56, stdev=3068.00
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.03%
    lat (usec) : 750=4.46%, 1000=44.65%
    lat (msec) : 2=49.95%, 4=0.69%, 10=0.20%, 20=0.02%, 50=0.01%
  cpu          : usr=0.76%, sys=3.50%, ctx=51731326, majf=0, minf=14152
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=26741909/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=2): err= 0: pid=160927: Thu Jan 14 15:16:35 2016
  read : io=1730.6GB, bw=252032KB/s, iops=3937, runt=7200002msec
    slat (usec): min=5, max=1090, avg=19.00, stdev=18.58
    clat (usec): min=18, max=23088, avg=994.54, stdev=344.56
     lat (usec): min=188, max=23102, avg=1013.69, stdev=344.02
    clat percentiles (usec):
     |  1.00th=[  612],  5.00th=[  700], 10.00th=[  748], 20.00th=[  820],
     | 30.00th=[  868], 40.00th=[  916], 50.00th=[  964], 60.00th=[ 1004],
     | 70.00th=[ 1064], 80.00th=[ 1128], 90.00th=[ 1224], 95.00th=[ 1320],
     | 99.00th=[ 1672], 99.50th=[ 2704], 99.90th=[ 5600], 99.95th=[ 6816],
     | 99.99th=[11328]
    bw (KB  /s): min=    1, max=136448, per=50.03%, avg=126092.30, stdev=3535.04
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.06%
    lat (usec) : 750=9.87%, 1000=48.99%
    lat (msec) : 2=40.37%, 4=0.51%, 10=0.18%, 20=0.02%, 50=0.01%
  cpu          : usr=0.57%, sys=2.30%, ctx=55514562, majf=0, minf=12719
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=28353551/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=160929: Thu Jan 14 15:16:35 2016
  read : io=3825.4GB, bw=557101KB/s, iops=8704, runt=7200001msec
    slat (usec): min=5, max=12302, avg=33.94, stdev=27.01
    clat (usec): min=0, max=27669, avg=882.39, stdev=368.11
     lat (usec): min=164, max=27682, avg=916.53, stdev=366.46
    clat percentiles (usec):
     |  1.00th=[  330],  5.00th=[  486], 10.00th=[  588], 20.00th=[  676],
     | 30.00th=[  732], 40.00th=[  788], 50.00th=[  836], 60.00th=[  892],
     | 70.00th=[  956], 80.00th=[ 1048], 90.00th=[ 1208], 95.00th=[ 1368],
     | 99.00th=[ 1816], 99.50th=[ 2128], 99.90th=[ 5408], 99.95th=[ 6688],
     | 99.99th=[10816]
    bw (KB  /s): min=    1, max=174592, per=25.01%, avg=139347.85, stdev=6191.85
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.20%, 500=5.28%, 750=27.72%, 1000=41.78%
    lat (msec) : 2=24.38%, 4=0.46%, 10=0.15%, 20=0.02%, 50=0.01%
  cpu          : usr=0.84%, sys=4.54%, ctx=120362583, majf=0, minf=25450
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=62673919/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=160933: Thu Jan 14 15:16:35 2016
  read : io=3340.6GB, bw=486506KB/s, iops=7601, runt=7200001msec
    slat (usec): min=4, max=933, avg=19.72, stdev=19.44
    clat (usec): min=0, max=23800, avg=1030.57, stdev=652.70
     lat (usec): min=160, max=23818, avg=1050.45, stdev=652.33
    clat percentiles (usec):
     |  1.00th=[  314],  5.00th=[  478], 10.00th=[  580], 20.00th=[  668],
     | 30.00th=[  732], 40.00th=[  788], 50.00th=[  844], 60.00th=[  916],
     | 70.00th=[ 1020], 80.00th=[ 1208], 90.00th=[ 1720], 95.00th=[ 2288],
     | 99.00th=[ 3632], 99.50th=[ 4320], 99.90th=[ 6112], 99.95th=[ 7264],
     | 99.99th=[11456]
    bw (KB  /s): min=    1, max=160128, per=25.01%, avg=121698.43, stdev=11662.58
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.26%, 500=5.45%, 750=28.26%, 1000=34.27%
    lat (msec) : 2=24.61%, 4=6.48%, 10=0.65%, 20=0.02%, 50=0.01%
  cpu          : usr=0.53%, sys=2.34%, ctx=107528445, majf=0, minf=23622
  IO depths    : 1=0.1%, 2=101.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=54731939/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=1632.3GB, aggrb=237705KB/s, minb=237705KB/s, maxb=237705KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=1730.6GB, aggrb=252031KB/s, minb=252031KB/s, maxb=252031KB/s, mint=7200002msec, maxt=7200002msec

Run status group 2 (all jobs):
   READ: io=3825.4GB, aggrb=557101KB/s, minb=557101KB/s, maxb=557101KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=3340.6GB, aggrb=486506KB/s, minb=486506KB/s, maxb=486506KB/s, mint=7200001msec, maxt=7200001msec



# 
# Thu 14 Jan 2016 03:16:37 PM EST
# 64kB randread with SATA 6 jobs and 4 I/Os Outstanding
# 64kB randread with NVMe 4 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 5915 IOPS | 378 MBps | 4054 μsec Average Latency 
SATA-R5-1: 6672 IOPS | 427 MBps | 3594 μsec Average Latency 
NVMe-R5-0: 8509 IOPS | 544 MBps | 3758 μsec Average Latency 
NVMe-R5-1: 7120 IOPS | 455 MBps | 4491 μsec Average Latency 

Totals
IOPS: 28216
MBPS: 1804
Average Latency (μsec): 3974

SATA-R5-01: (g=0): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
SATA-R5-11: (g=1): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
NVMe-R5-01: (g=2): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8
...
NVMe-R5-11: (g=3): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8
...
fio-2.1.10
Starting 20 processes

SATA-R5-01: (groupid=0, jobs=6): err= 0: pid=166103: Thu Jan 14 17:17:37 2016
  read : io=2599.6GB, bw=378587KB/s, iops=5915, runt=7200007msec
    slat (usec): min=4, max=5006, avg=27.03, stdev=57.45
    clat (usec): min=1, max=26053, avg=4027.53, stdev=1888.64
     lat (usec): min=242, max=26062, avg=4054.77, stdev=1887.88
    clat percentiles (usec):
     |  1.00th=[ 1128],  5.00th=[ 1752], 10.00th=[ 2320], 20.00th=[ 2832],
     | 30.00th=[ 3056], 40.00th=[ 3248], 50.00th=[ 3408], 60.00th=[ 3760],
     | 70.00th=[ 4384], 80.00th=[ 5216], 90.00th=[ 6496], 95.00th=[ 7904],
     | 99.00th=[10688], 99.50th=[11712], 99.90th=[13888], 99.95th=[14784],
     | 99.99th=[16768]
    bw (KB  /s): min=    1, max=76032, per=16.68%, avg=63147.38, stdev=2883.74
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.42%
    lat (msec) : 2=6.54%, 4=57.68%, 10=33.78%, 20=1.55%, 50=0.01%
  cpu          : usr=0.34%, sys=1.65%, ctx=75944699, majf=0, minf=86847
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=42591065/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
SATA-R5-11: (groupid=1, jobs=6): err= 0: pid=166109: Thu Jan 14 17:17:37 2016
  read : io=2932.5GB, bw=427061KB/s, iops=6672, runt=7200006msec
    slat (usec): min=4, max=6643, avg=29.47, stdev=54.76
    clat (usec): min=1, max=27233, avg=3564.68, stdev=1844.91
     lat (usec): min=211, max=27246, avg=3594.37, stdev=1844.18
    clat percentiles (usec):
     |  1.00th=[ 1128],  5.00th=[ 1592], 10.00th=[ 1896], 20.00th=[ 2256],
     | 30.00th=[ 2512], 40.00th=[ 2736], 50.00th=[ 2992], 60.00th=[ 3344],
     | 70.00th=[ 3920], 80.00th=[ 4704], 90.00th=[ 5984], 95.00th=[ 7328],
     | 99.00th=[10176], 99.50th=[11200], 99.90th=[13504], 99.95th=[14400],
     | 99.99th=[16512]
    bw (KB  /s): min=    1, max=101556, per=16.68%, avg=71233.01, stdev=6514.26
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.41%
    lat (msec) : 2=11.80%, 4=58.68%, 10=27.94%, 20=1.11%, 50=0.01%
  cpu          : usr=0.37%, sys=1.89%, ctx=90145554, majf=0, minf=83431
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=48044427/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=166115: Thu Jan 14 17:17:37 2016
  read : io=3739.6GB, bw=544608KB/s, iops=8509, runt=7200003msec
    slat (usec): min=5, max=6804, avg=31.89, stdev=85.36
    clat (usec): min=0, max=29651, avg=3726.14, stdev=1890.87
     lat (usec): min=208, max=29671, avg=3758.24, stdev=1889.63
    clat percentiles (usec):
     |  1.00th=[  772],  5.00th=[ 1272], 10.00th=[ 1832], 20.00th=[ 2512],
     | 30.00th=[ 2800], 40.00th=[ 2992], 50.00th=[ 3184], 60.00th=[ 3536],
     | 70.00th=[ 4128], 80.00th=[ 4896], 90.00th=[ 6176], 95.00th=[ 7456],
     | 99.00th=[10304], 99.50th=[11328], 99.90th=[13632], 99.95th=[14656],
     | 99.99th=[16768]
    bw (KB  /s): min=    1, max=160640, per=25.02%, avg=136238.84, stdev=7462.50
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.16%, 750=0.71%, 1000=1.83%
    lat (msec) : 2=9.02%, 4=56.17%, 10=30.90%, 20=1.20%, 50=0.01%
  cpu          : usr=0.60%, sys=3.31%, ctx=110840453, majf=0, minf=93100
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=61268361/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=166119: Thu Jan 14 17:17:37 2016
  read : io=3128.2GB, bw=455692KB/s, iops=7120, runt=7200010msec
    slat (usec): min=4, max=15366, avg=31.33, stdev=72.34
    clat (usec): min=0, max=75909, avg=4460.44, stdev=3588.72
     lat (usec): min=178, max=75923, avg=4491.99, stdev=3588.27
    clat percentiles (usec):
     |  1.00th=[  732],  5.00th=[ 1096], 10.00th=[ 1448], 20.00th=[ 1896],
     | 30.00th=[ 2224], 40.00th=[ 2512], 50.00th=[ 3024], 60.00th=[ 4048],
     | 70.00th=[ 5216], 80.00th=[ 6880], 90.00th=[ 9152], 95.00th=[11200],
     | 99.00th=[17792], 99.50th=[20096], 99.90th=[25472], 99.95th=[28032],
     | 99.99th=[34560]
    bw (KB  /s): min=    1, max=146395, per=25.02%, avg=114000.15, stdev=14432.49
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.18%, 750=0.96%, 1000=2.63%
    lat (msec) : 2=19.17%, 4=36.68%, 10=32.88%, 20=6.99%, 50=0.50%
    lat (msec) : 100=0.01%
  cpu          : usr=0.55%, sys=2.85%, ctx=94310951, majf=0, minf=105883
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=101.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=51265352/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=2599.6GB, aggrb=378587KB/s, minb=378587KB/s, maxb=378587KB/s, mint=7200007msec, maxt=7200007msec

Run status group 1 (all jobs):
   READ: io=2932.5GB, aggrb=427061KB/s, minb=427061KB/s, maxb=427061KB/s, mint=7200006msec, maxt=7200006msec

Run status group 2 (all jobs):
   READ: io=3739.6GB, aggrb=544607KB/s, minb=544607KB/s, maxb=544607KB/s, mint=7200003msec, maxt=7200003msec

Run status group 3 (all jobs):
   READ: io=3128.2GB, aggrb=455691KB/s, minb=455691KB/s, maxb=455691KB/s, mint=7200010msec, maxt=7200010msec



# 
# Thu 14 Jan 2016 05:17:40 PM EST
# 32kB randread with SATA 5 jobs and 2 I/Os Outstanding
# 32kB randread with NVMe 3 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 9815 IOPS | 314 MBps | 1016 μsec Average Latency 
SATA-R5-1: 9936 IOPS | 317 MBps | 1004 μsec Average Latency 
NVMe-R5-0: 14001 IOPS | 448 MBps | 855 μsec Average Latency 
NVMe-R5-1: 12923 IOPS | 413 MBps | 926 μsec Average Latency 

Totals
IOPS: 46675
MBPS: 1492
Average Latency (μsec): 950

SATA-R5-01: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=2
...
SATA-R5-11: (g=1): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=2
...
NVMe-R5-01: (g=2): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=4
...
NVMe-R5-11: (g=3): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=4
...
fio-2.1.10
Starting 16 processes

SATA-R5-01: (groupid=0, jobs=5): err= 0: pid=171293: Thu Jan 14 19:18:41 2016
  read : io=2156.9GB, bw=314111KB/s, iops=9815, runt=7200001msec
    slat (usec): min=2, max=3203, avg=17.21, stdev=20.77
    clat (usec): min=0, max=21718, avg=999.39, stdev=542.63
     lat (usec): min=143, max=21728, avg=1016.77, stdev=542.23
    clat percentiles (usec):
     |  1.00th=[  596],  5.00th=[  692], 10.00th=[  732], 20.00th=[  788],
     | 30.00th=[  844], 40.00th=[  876], 50.00th=[  924], 60.00th=[  972],
     | 70.00th=[ 1020], 80.00th=[ 1096], 90.00th=[ 1224], 95.00th=[ 1352],
     | 99.00th=[ 3088], 99.50th=[ 4896], 99.90th=[ 8384], 99.95th=[10304],
     | 99.99th=[12480]
    bw (KB  /s): min=    0, max=70656, per=20.01%, avg=62860.23, stdev=2508.93
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.14%, 750=13.13%, 1000=52.79%
    lat (msec) : 2=32.49%, 4=0.66%, 10=0.74%, 20=0.06%, 50=0.01%
  cpu          : usr=0.56%, sys=2.16%, ctx=122158288, majf=0, minf=30021
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=70675060/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=5): err= 0: pid=171298: Thu Jan 14 19:18:41 2016
  read : io=2183.5GB, bw=317981KB/s, iops=9936, runt=7200001msec
    slat (usec): min=2, max=6215, avg=20.49, stdev=22.49
    clat (usec): min=0, max=26967, avg=983.59, stdev=552.69
     lat (usec): min=178, max=26977, avg=1004.29, stdev=552.06
    clat percentiles (usec):
     |  1.00th=[  564],  5.00th=[  652], 10.00th=[  708], 20.00th=[  764],
     | 30.00th=[  820], 40.00th=[  868], 50.00th=[  908], 60.00th=[  956],
     | 70.00th=[ 1012], 80.00th=[ 1080], 90.00th=[ 1208], 95.00th=[ 1352],
     | 99.00th=[ 3152], 99.50th=[ 4896], 99.90th=[ 8640], 99.95th=[10560],
     | 99.99th=[12608]
    bw (KB  /s): min=    0, max=72192, per=20.01%, avg=63634.24, stdev=2424.58
    lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.31%, 750=16.56%, 1000=51.34%
    lat (msec) : 2=30.31%, 4=0.68%, 10=0.73%, 20=0.06%, 50=0.01%
  cpu          : usr=0.60%, sys=2.40%, ctx=135093821, majf=0, minf=35766
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=71545733/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=3): err= 0: pid=171303: Thu Jan 14 19:18:41 2016
  read : io=3076.7GB, bw=448064KB/s, iops=14001, runt=7200001msec
    slat (usec): min=3, max=15218, avg=19.25, stdev=24.91
    clat (usec): min=0, max=24754, avg=835.69, stdev=525.61
     lat (usec): min=134, max=24761, avg=855.12, stdev=524.84
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[  494], 10.00th=[  564], 20.00th=[  620],
     | 30.00th=[  676], 40.00th=[  724], 50.00th=[  756], 60.00th=[  804],
     | 70.00th=[  868], 80.00th=[  948], 90.00th=[ 1096], 95.00th=[ 1256],
     | 99.00th=[ 2288], 99.50th=[ 4576], 99.90th=[ 8032], 99.95th=[ 9920],
     | 99.99th=[12224]
    bw (KB  /s): min=    0, max=170816, per=33.34%, avg=149402.08, stdev=7558.52
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.07%, 500=5.32%, 750=42.73%, 1000=36.29%
    lat (msec) : 2=14.43%, 4=0.50%, 10=0.61%, 20=0.05%, 50=0.01%
  cpu          : usr=1.13%, sys=4.49%, ctx=180342809, majf=0, minf=28838
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=100814308/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=3): err= 0: pid=171306: Thu Jan 14 19:18:41 2016
  read : io=2839.7GB, bw=413552KB/s, iops=12923, runt=7200002msec
    slat (usec): min=3, max=9621, avg=21.09, stdev=24.34
    clat (usec): min=0, max=26829, avg=905.37, stdev=626.46
     lat (usec): min=133, max=26841, avg=926.65, stdev=625.73
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[  498], 10.00th=[  564], 20.00th=[  636],
     | 30.00th=[  692], 40.00th=[  740], 50.00th=[  780], 60.00th=[  836],
     | 70.00th=[  908], 80.00th=[ 1012], 90.00th=[ 1240], 95.00th=[ 1544],
     | 99.00th=[ 3792], 99.50th=[ 4896], 99.90th=[ 8512], 99.95th=[10432],
     | 99.99th=[12480]
    bw (KB  /s): min=    0, max=174208, per=33.34%, avg=137896.83, stdev=12507.73
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.08%, 500=5.10%, 750=38.23%, 1000=35.40%
    lat (msec) : 2=18.38%, 4=1.91%, 10=0.84%, 20=0.06%, 50=0.01%
  cpu          : usr=1.14%, sys=4.80%, ctx=165841324, majf=0, minf=33397
  IO depths    : 1=0.1%, 2=0.1%, 4=101.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=93049228/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=2156.9GB, aggrb=314111KB/s, minb=314111KB/s, maxb=314111KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=2183.5GB, aggrb=317981KB/s, minb=317981KB/s, maxb=317981KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
   READ: io=3076.7GB, aggrb=448063KB/s, minb=448063KB/s, maxb=448063KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=2839.7GB, aggrb=413552KB/s, minb=413552KB/s, maxb=413552KB/s, mint=7200002msec, maxt=7200002msec



# 
# Thu 14 Jan 2016 07:18:44 PM EST
# 32kB randread with SATA 4 jobs and 16 I/Os Outstanding
# 32kB randread with NVMe 3 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 16643 IOPS | 532 MBps | 3843 μsec Average Latency 
SATA-R5-1: 16725 IOPS | 535 MBps | 3824 μsec Average Latency 
NVMe-R5-0: 8904 IOPS | 284 MBps | 2693 μsec Average Latency 
NVMe-R5-1: 8880 IOPS | 284 MBps | 2700 μsec Average Latency 

Totals
IOPS: 51152
MBPS: 1635
Average Latency (μsec): 3265

SATA-R5-01: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=16
...
SATA-R5-11: (g=1): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=16
...
NVMe-R5-01: (g=2): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=8
...
NVMe-R5-11: (g=3): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=8
...
fio-2.1.10
Starting 14 processes

SATA-R5-01: (groupid=0, jobs=4): err= 0: pid=176435: Thu Jan 14 21:19:45 2016
  read : io=3657.2GB, bw=532604KB/s, iops=16643, runt=7200003msec
    slat (usec): min=3, max=12228, avg=25.89, stdev=75.21
    clat (usec): min=0, max=146417, avg=3817.38, stdev=1724.38
     lat (usec): min=194, max=146437, avg=3843.46, stdev=1722.72
    clat percentiles (usec):
     |  1.00th=[ 1352],  5.00th=[ 1912], 10.00th=[ 2224], 20.00th=[ 2544],
     | 30.00th=[ 2832], 40.00th=[ 3088], 50.00th=[ 3408], 60.00th=[ 3728],
     | 70.00th=[ 4192], 80.00th=[ 4768], 90.00th=[ 5984], 95.00th=[ 7328],
     | 99.00th=[ 9920], 99.50th=[11072], 99.90th=[13376], 99.95th=[14272],
     | 99.99th=[16512]
    bw (KB  /s): min=    0, max=167040, per=25.01%, avg=133216.56, stdev=5667.82
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.14%
    lat (msec) : 2=5.94%, 4=60.12%, 10=32.80%, 20=0.98%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.92%, sys=4.38%, ctx=192510642, majf=0, minf=133859
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=119835845/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
SATA-R5-11: (groupid=1, jobs=4): err= 0: pid=176439: Thu Jan 14 21:19:45 2016
  read : io=3674.1GB, bw=535202KB/s, iops=16725, runt=7200003msec
    slat (usec): min=3, max=14031, avg=26.05, stdev=76.35
    clat (usec): min=0, max=116134, avg=3798.57, stdev=1723.78
     lat (usec): min=186, max=116149, avg=3824.81, stdev=1722.05
    clat percentiles (usec):
     |  1.00th=[ 1320],  5.00th=[ 1896], 10.00th=[ 2192], 20.00th=[ 2544],
     | 30.00th=[ 2800], 40.00th=[ 3088], 50.00th=[ 3376], 60.00th=[ 3728],
     | 70.00th=[ 4192], 80.00th=[ 4768], 90.00th=[ 5920], 95.00th=[ 7328],
     | 99.00th=[ 9920], 99.50th=[11072], 99.90th=[13376], 99.95th=[14272],
     | 99.99th=[16512]
    bw (KB  /s): min=    0, max=160892, per=25.01%, avg=133867.40, stdev=5596.60
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.16%
    lat (msec) : 2=6.23%, 4=60.21%, 10=32.40%, 20=0.97%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.92%, sys=4.39%, ctx=193598120, majf=0, minf=121366
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=120420413/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
NVMe-R5-01: (groupid=2, jobs=3): err= 0: pid=176443: Thu Jan 14 21:19:45 2016
  read : io=1956.5GB, bw=284931KB/s, iops=8904, runt=7200003msec
    slat (usec): min=2, max=12017, avg=22.94, stdev=63.65
    clat (usec): min=0, max=118874, avg=2670.47, stdev=1580.60
     lat (usec): min=180, max=118884, avg=2693.60, stdev=1579.80
    clat percentiles (usec):
     |  1.00th=[  724],  5.00th=[ 1064], 10.00th=[ 1320], 20.00th=[ 1608],
     | 30.00th=[ 1784], 40.00th=[ 1976], 50.00th=[ 2192], 60.00th=[ 2480],
     | 70.00th=[ 2928], 80.00th=[ 3568], 90.00th=[ 4448], 95.00th=[ 5984],
     | 99.00th=[ 8640], 99.50th=[ 9664], 99.90th=[11968], 99.95th=[12864],
     | 99.99th=[14912]
    bw (KB  /s): min=    0, max=115904, per=33.35%, avg=95031.52, stdev=5430.22
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.10%, 750=1.08%, 1000=2.94%
    lat (msec) : 2=37.41%, 4=44.30%, 10=13.75%, 20=0.41%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.70%, sys=3.02%, ctx=109618998, majf=0, minf=46693
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=64109585/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=3): err= 0: pid=176446: Thu Jan 14 21:19:45 2016
  read : io=1951.3GB, bw=284170KB/s, iops=8880, runt=7200002msec
    slat (usec): min=3, max=10565, avg=23.09, stdev=64.46
    clat (usec): min=0, max=102145, avg=2677.49, stdev=1585.50
     lat (usec): min=174, max=102157, avg=2700.76, stdev=1584.68
    clat percentiles (usec):
     |  1.00th=[  732],  5.00th=[ 1064], 10.00th=[ 1320], 20.00th=[ 1608],
     | 30.00th=[ 1800], 40.00th=[ 1976], 50.00th=[ 2192], 60.00th=[ 2480],
     | 70.00th=[ 2928], 80.00th=[ 3568], 90.00th=[ 4448], 95.00th=[ 5984],
     | 99.00th=[ 8640], 99.50th=[ 9664], 99.90th=[11968], 99.95th=[12864],
     | 99.99th=[15040]
    bw (KB  /s): min=    0, max=112768, per=33.35%, avg=94777.58, stdev=5509.44
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.10%, 750=1.07%, 1000=2.93%
    lat (msec) : 2=37.18%, 4=44.44%, 10=13.86%, 20=0.41%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.71%, sys=3.01%, ctx=109558625, majf=0, minf=52750
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=63938226/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=3657.2GB, aggrb=532603KB/s, minb=532603KB/s, maxb=532603KB/s, mint=7200003msec, maxt=7200003msec

Run status group 1 (all jobs):
   READ: io=3674.1GB, aggrb=535201KB/s, minb=535201KB/s, maxb=535201KB/s, mint=7200003msec, maxt=7200003msec

Run status group 2 (all jobs):
   READ: io=1956.5GB, aggrb=284931KB/s, minb=284931KB/s, maxb=284931KB/s, mint=7200003msec, maxt=7200003msec

Run status group 3 (all jobs):
   READ: io=1951.3GB, aggrb=284169KB/s, minb=284169KB/s, maxb=284169KB/s, mint=7200002msec, maxt=7200002msec



# 
# Thu 14 Jan 2016 09:19:47 PM EST
# 16kB randread with SATA 5 jobs and 16 I/Os Outstanding
# 16kB randread with NVMe 4 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 28104 IOPS | 449 MBps | 2844 μsec Average Latency 
SATA-R5-1: 28009 IOPS | 448 MBps | 2854 μsec Average Latency 
NVMe-R5-0: 6521 IOPS | 104 MBps | 2451 μsec Average Latency 
NVMe-R5-1: 6710 IOPS | 107 MBps | 2382 μsec Average Latency 

Totals
IOPS: 69344
MBPS: 1108
Average Latency (μsec): 2632

SATA-R5-01: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=16
...
SATA-R5-11: (g=1): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=16
...
NVMe-R5-01: (g=2): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=4
...
NVMe-R5-11: (g=3): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=4
...
fio-2.1.10
Starting 18 processes

SATA-R5-01: (groupid=0, jobs=5): err= 0: pid=181614: Thu Jan 14 23:20:48 2016
  read : io=3087.7GB, bw=449675KB/s, iops=28104, runt=7200002msec
    slat (usec): min=2, max=15201, avg=18.77, stdev=28.57
    clat (usec): min=1, max=111911, avg=2825.75, stdev=599.58
     lat (usec): min=269, max=111919, avg=2844.69, stdev=598.22
    clat percentiles (usec):
     |  1.00th=[ 1848],  5.00th=[ 2128], 10.00th=[ 2288], 20.00th=[ 2448],
     | 30.00th=[ 2544], 40.00th=[ 2672], 50.00th=[ 2768], 60.00th=[ 2864],
     | 70.00th=[ 2992], 80.00th=[ 3152], 90.00th=[ 3376], 95.00th=[ 3632],
     | 99.00th=[ 4704], 99.50th=[ 5664], 99.90th=[ 8256], 99.95th=[ 9152],
     | 99.99th=[12096]
    bw (KB  /s): min=    0, max=123968, per=20.01%, avg=89959.43, stdev=1807.20
    lat (usec) : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
    lat (usec) : 1000=0.02%
    lat (msec) : 2=2.28%, 4=95.38%, 10=2.29%, 20=0.03%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.29%, sys=4.75%, ctx=343161776, majf=0, minf=83352
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=202353803/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
SATA-R5-11: (groupid=1, jobs=5): err= 0: pid=181619: Thu Jan 14 23:20:48 2016
  read : io=3077.2GB, bw=448145KB/s, iops=28009, runt=7200002msec
    slat (usec): min=2, max=11469, avg=18.54, stdev=28.30
    clat (usec): min=72, max=159410, avg=2835.72, stdev=596.70
     lat (usec): min=227, max=159471, avg=2854.45, stdev=595.35
    clat percentiles (usec):
     |  1.00th=[ 1848],  5.00th=[ 2160], 10.00th=[ 2288], 20.00th=[ 2448],
     | 30.00th=[ 2576], 40.00th=[ 2672], 50.00th=[ 2768], 60.00th=[ 2864],
     | 70.00th=[ 2992], 80.00th=[ 3152], 90.00th=[ 3376], 95.00th=[ 3632],
     | 99.00th=[ 4704], 99.50th=[ 5600], 99.90th=[ 8256], 99.95th=[ 9152],
     | 99.99th=[11968]
    bw (KB  /s): min=    0, max=116928, per=20.01%, avg=89653.45, stdev=1779.36
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=2.23%, 4=95.48%, 10=2.25%, 20=0.03%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.26%, sys=4.67%, ctx=342868011, majf=0, minf=81071
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=201665067/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=181624: Thu Jan 14 23:20:48 2016
  read : io=733685MB, bw=104346KB/s, iops=6521, runt=7200001msec
    slat (usec): min=2, max=3154, avg= 9.66, stdev=14.48
    clat (usec): min=15, max=112905, avg=2441.64, stdev=579.10
     lat (usec): min=220, max=112909, avg=2451.44, stdev=578.75
    clat percentiles (usec):
     |  1.00th=[ 1480],  5.00th=[ 1768], 10.00th=[ 1912], 20.00th=[ 2064],
     | 30.00th=[ 2192], 40.00th=[ 2288], 50.00th=[ 2384], 60.00th=[ 2480],
     | 70.00th=[ 2608], 80.00th=[ 2736], 90.00th=[ 2992], 95.00th=[ 3216],
     | 99.00th=[ 4192], 99.50th=[ 5152], 99.90th=[ 7712], 99.95th=[ 8640],
     | 99.99th=[11456]
    bw (KB  /s): min=    0, max=50240, per=25.02%, avg=26103.26, stdev=667.44
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
    lat (usec) : 750=0.03%, 1000=0.09%
    lat (msec) : 2=14.95%, 4=83.71%, 10=1.19%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.48%, sys=1.46%, ctx=54710057, majf=0, minf=26798
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=46955816/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=181628: Thu Jan 14 23:20:48 2016
  read : io=754904MB, bw=107364KB/s, iops=6710, runt=7200001msec
    slat (usec): min=2, max=3139, avg= 9.33, stdev=14.06
    clat (usec): min=23, max=118621, avg=2373.00, stdev=577.83
     lat (usec): min=216, max=118626, avg=2382.49, stdev=577.51
    clat percentiles (usec):
     |  1.00th=[ 1384],  5.00th=[ 1688], 10.00th=[ 1832], 20.00th=[ 1992],
     | 30.00th=[ 2128], 40.00th=[ 2224], 50.00th=[ 2320], 60.00th=[ 2416],
     | 70.00th=[ 2544], 80.00th=[ 2704], 90.00th=[ 2928], 95.00th=[ 3152],
     | 99.00th=[ 4048], 99.50th=[ 4960], 99.90th=[ 7648], 99.95th=[ 8640],
     | 99.99th=[11072]
    bw (KB  /s): min=    0, max=57728, per=25.02%, avg=26858.27, stdev=816.04
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%
    lat (usec) : 1000=0.11%
    lat (msec) : 2=20.18%, 4=78.60%, 10=1.05%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.49%, sys=1.46%, ctx=56050743, majf=0, minf=32135
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=48313846/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=3087.7GB, aggrb=449675KB/s, minb=449675KB/s, maxb=449675KB/s, mint=7200002msec, maxt=7200002msec

Run status group 1 (all jobs):
   READ: io=3077.2GB, aggrb=448144KB/s, minb=448144KB/s, maxb=448144KB/s, mint=7200002msec, maxt=7200002msec

Run status group 2 (all jobs):
   READ: io=733685MB, aggrb=104346KB/s, minb=104346KB/s, maxb=104346KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=754904MB, aggrb=107364KB/s, minb=107364KB/s, maxb=107364KB/s, mint=7200001msec, maxt=7200001msec



# 
# Thu 14 Jan 2016 11:20:51 PM EST
# 8kB randread with SATA 6 jobs and 2 I/Os Outstanding
# 8kB randread with NVMe 4 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 19507 IOPS | 156 MBps | 613 μsec Average Latency 
SATA-R5-1: 19354 IOPS | 154 MBps | 618 μsec Average Latency 
NVMe-R5-0: 14250 IOPS | 114 MBps | 559 μsec Average Latency 
NVMe-R5-1: 14147 IOPS | 113 MBps | 563 μsec Average Latency 

Totals
IOPS: 67258
MBPS: 537
Average Latency (μsec): 588

SATA-R5-01: (g=0): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
...
SATA-R5-11: (g=1): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
...
NVMe-R5-01: (g=2): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 20 processes

SATA-R5-01: (groupid=0, jobs=6): err= 0: pid=186763: Fri Jan 15 01:21:52 2016
  read : io=1071.6GB, bw=156061KB/s, iops=19507, runt=7200001msec
    slat (usec): min=1, max=15037, avg=13.14, stdev=13.97
    clat (usec): min=1, max=74811, avg=599.93, stdev=200.53
     lat (usec): min=140, max=74818, avg=613.26, stdev=199.76
    clat percentiles (usec):
     |  1.00th=[  394],  5.00th=[  454], 10.00th=[  474], 20.00th=[  494],
     | 30.00th=[  516], 40.00th=[  548], 50.00th=[  580], 60.00th=[  604],
     | 70.00th=[  620], 80.00th=[  652], 90.00th=[  724], 95.00th=[  804],
     | 99.00th=[ 1192], 99.50th=[ 1416], 99.90th=[ 3120], 99.95th=[ 3536],
     | 99.99th=[ 6240]
    bw (KB  /s): min=    0, max=28576, per=16.67%, avg=26019.56, stdev=464.95
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=23.54%, 750=69.04%, 1000=5.11%
    lat (msec) : 2=1.99%, 4=0.29%, 10=0.03%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=0.85%, sys=2.26%, ctx=267367618, majf=0, minf=17749
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=140454986/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=6): err= 0: pid=186769: Fri Jan 15 01:21:52 2016
  read : io=1063.2GB, bw=154838KB/s, iops=19354, runt=7200001msec
    slat (usec): min=1, max=9727, avg=15.85, stdev=15.96
    clat (usec): min=2, max=16757, avg=602.00, stdev=192.50
     lat (usec): min=145, max=16763, avg=618.03, stdev=191.30
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  450], 10.00th=[  470], 20.00th=[  494],
     | 30.00th=[  524], 40.00th=[  556], 50.00th=[  588], 60.00th=[  604],
     | 70.00th=[  628], 80.00th=[  660], 90.00th=[  732], 95.00th=[  812],
     | 99.00th=[ 1176], 99.50th=[ 1352], 99.90th=[ 3056], 99.95th=[ 3504],
     | 99.99th=[ 6176]
    bw (KB  /s): min=    0, max=29184, per=16.67%, avg=25815.59, stdev=448.83
    lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=22.22%, 750=69.96%, 1000=5.58%
    lat (msec) : 2=1.97%, 4=0.23%, 10=0.03%, 20=0.01%
  cpu          : usr=0.93%, sys=2.76%, ctx=265187397, majf=0, minf=17201
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=139354194/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=186775: Fri Jan 15 01:21:52 2016
  read : io=801567MB, bw=114001KB/s, iops=14250, runt=7200001msec
    slat (usec): min=1, max=2967, avg=14.09, stdev=14.81
    clat (usec): min=0, max=34355, avg=545.30, stdev=180.03
     lat (usec): min=100, max=34371, avg=559.57, stdev=178.82
    clat percentiles (usec):
     |  1.00th=[  318],  5.00th=[  378], 10.00th=[  422], 20.00th=[  462],
     | 30.00th=[  482], 40.00th=[  494], 50.00th=[  510], 60.00th=[  540],
     | 70.00th=[  580], 80.00th=[  612], 90.00th=[  668], 95.00th=[  748],
     | 99.00th=[ 1080], 99.50th=[ 1224], 99.90th=[ 2768], 99.95th=[ 3376],
     | 99.99th=[ 6112]
    bw (KB  /s): min=    0, max=31728, per=25.01%, avg=28509.21, stdev=438.93
    lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.16%, 500=43.93%, 750=50.85%, 1000=3.57%
    lat (msec) : 2=1.32%, 4=0.15%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=0.91%, sys=2.40%, ctx=194302704, majf=0, minf=10353
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=102600633/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=186779: Fri Jan 15 01:21:52 2016
  read : io=795779MB, bw=113177KB/s, iops=14147, runt=7200001msec
    slat (usec): min=1, max=7062, avg=16.17, stdev=16.32
    clat (usec): min=1, max=18006, avg=547.20, stdev=178.91
     lat (usec): min=109, max=18015, avg=563.56, stdev=177.41
    clat percentiles (usec):
     |  1.00th=[  318],  5.00th=[  378], 10.00th=[  422], 20.00th=[  462],
     | 30.00th=[  482], 40.00th=[  494], 50.00th=[  516], 60.00th=[  548],
     | 70.00th=[  588], 80.00th=[  620], 90.00th=[  676], 95.00th=[  756],
     | 99.00th=[ 1080], 99.50th=[ 1224], 99.90th=[ 2768], 99.95th=[ 3376],
     | 99.99th=[ 6048]
    bw (KB  /s): min=    0, max=33040, per=25.01%, avg=28303.40, stdev=459.77
    lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.18%, 500=42.50%, 750=52.11%, 1000=3.73%
    lat (msec) : 2=1.30%, 4=0.14%, 10=0.03%, 20=0.01%
  cpu          : usr=1.00%, sys=2.90%, ctx=192772924, majf=0, minf=11867
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=101859753/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=1071.6GB, aggrb=156061KB/s, minb=156061KB/s, maxb=156061KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=1063.2GB, aggrb=154837KB/s, minb=154837KB/s, maxb=154837KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
   READ: io=801567MB, aggrb=114000KB/s, minb=114000KB/s, maxb=114000KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=795779MB, aggrb=113177KB/s, minb=113177KB/s, maxb=113177KB/s, mint=7200001msec, maxt=7200001msec



# 
# Fri 15 Jan 2016 01:21:54 AM EST
# 8kB randread with SATA 4 jobs and 8 I/Os Outstanding
# 8kB randread with NVMe 4 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 23817 IOPS | 190 MBps | 1341 μsec Average Latency 
SATA-R5-1: 23813 IOPS | 190 MBps | 1341 μsec Average Latency 
NVMe-R5-0: 12928 IOPS | 103 MBps | 1235 μsec Average Latency 
NVMe-R5-1: 13074 IOPS | 104 MBps | 1222 μsec Average Latency 

Totals
IOPS: 73632
MBPS: 587
Average Latency (μsec): 1284

SATA-R5-01: (g=0): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
SATA-R5-11: (g=1): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
NVMe-R5-01: (g=2): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=4
...
NVMe-R5-11: (g=3): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=4
...
fio-2.1.10
Starting 16 processes

SATA-R5-01: (groupid=0, jobs=4): err= 0: pid=192608: Fri Jan 15 03:22:55 2016
  read : io=1308.4GB, bw=190539KB/s, iops=23817, runt=7200001msec
    slat (usec): min=1, max=13645, avg=16.33, stdev=17.05
    clat (usec): min=54, max=78342, avg=1325.24, stdev=417.87
     lat (usec): min=208, max=78350, avg=1341.76, stdev=416.90
    clat percentiles (usec):
     |  1.00th=[  764],  5.00th=[  932], 10.00th=[ 1004], 20.00th=[ 1096],
     | 30.00th=[ 1144], 40.00th=[ 1208], 50.00th=[ 1256], 60.00th=[ 1320],
     | 70.00th=[ 1368], 80.00th=[ 1480], 90.00th=[ 1672], 95.00th=[ 1896],
     | 99.00th=[ 2992], 99.50th=[ 3760], 99.90th=[ 5472], 99.95th=[ 6816],
     | 99.99th=[ 8384]
    bw (KB  /s): min=    0, max=50528, per=25.01%, avg=47646.85, stdev=910.47
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.03%, 750=0.82%, 1000=8.92%
    lat (msec) : 2=86.61%, 4=3.23%, 10=0.39%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.54%, sys=4.98%, ctx=294369939, majf=0, minf=22064
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=171485016/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=4): err= 0: pid=192612: Fri Jan 15 03:22:55 2016
  read : io=1308.2GB, bw=190507KB/s, iops=23813, runt=7200002msec
    slat (usec): min=1, max=6795, avg=13.20, stdev=15.54
    clat (usec): min=35, max=93161, avg=1328.43, stdev=411.31
     lat (usec): min=206, max=93169, avg=1341.81, stdev=410.63
    clat percentiles (usec):
     |  1.00th=[  780],  5.00th=[  948], 10.00th=[ 1012], 20.00th=[ 1096],
     | 30.00th=[ 1160], 40.00th=[ 1224], 50.00th=[ 1256], 60.00th=[ 1320],
     | 70.00th=[ 1384], 80.00th=[ 1480], 90.00th=[ 1672], 95.00th=[ 1880],
     | 99.00th=[ 2896], 99.50th=[ 3760], 99.90th=[ 5472], 99.95th=[ 6816],
     | 99.99th=[ 8384]
    bw (KB  /s): min=    0, max=50736, per=25.01%, avg=47639.26, stdev=904.57
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.02%, 750=0.74%
    lat (usec) : 1000=8.33%
    lat (msec) : 2=87.41%, 4=3.12%, 10=0.38%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.33%, sys=3.46%, ctx=304951211, majf=0, minf=16688
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=171456569/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=192616: Fri Jan 15 03:22:55 2016
  read : io=727218MB, bw=103427KB/s, iops=12928, runt=7200001msec
    slat (usec): min=1, max=8677, avg=15.36, stdev=16.29
    clat (usec): min=20, max=93836, avg=1220.13, stdev=381.39
     lat (usec): min=139, max=93843, avg=1235.69, stdev=380.51
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[  844], 10.00th=[  916], 20.00th=[  996],
     | 30.00th=[ 1064], 40.00th=[ 1112], 50.00th=[ 1160], 60.00th=[ 1224],
     | 70.00th=[ 1272], 80.00th=[ 1368], 90.00th=[ 1544], 95.00th=[ 1752],
     | 99.00th=[ 2352], 99.50th=[ 3248], 99.90th=[ 5280], 99.95th=[ 6624],
     | 99.99th=[ 8160]
    bw (KB  /s): min=    0, max=27792, per=25.01%, avg=25867.69, stdev=527.38
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.10%, 750=2.11%
    lat (usec) : 1000=18.07%
    lat (msec) : 2=77.59%, 4=1.83%, 10=0.30%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=0.92%, sys=2.80%, ctx=168583690, majf=0, minf=14103
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=93083875/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=192620: Fri Jan 15 03:22:55 2016
  read : io=735435MB, bw=104595KB/s, iops=13074, runt=7200001msec
    slat (usec): min=1, max=3036, avg= 8.39, stdev=10.62
    clat (usec): min=25, max=61324, avg=1213.54, stdev=376.69
     lat (usec): min=159, max=61328, avg=1222.07, stdev=376.41
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[  844], 10.00th=[  908], 20.00th=[  996],
     | 30.00th=[ 1064], 40.00th=[ 1112], 50.00th=[ 1144], 60.00th=[ 1224],
     | 70.00th=[ 1272], 80.00th=[ 1368], 90.00th=[ 1528], 95.00th=[ 1736],
     | 99.00th=[ 2320], 99.50th=[ 3184], 99.90th=[ 5152], 99.95th=[ 6560],
     | 99.99th=[ 8160]
    bw (KB  /s): min=    0, max=27920, per=25.01%, avg=26160.07, stdev=517.92
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.10%, 750=2.20%
    lat (usec) : 1000=18.80%
    lat (msec) : 2=76.90%, 4=1.71%, 10=0.29%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=0.82%, sys=2.03%, ctx=130707828, majf=0, minf=11401
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=94135674/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=1308.4GB, aggrb=190538KB/s, minb=190538KB/s, maxb=190538KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=1308.2GB, aggrb=190507KB/s, minb=190507KB/s, maxb=190507KB/s, mint=7200002msec, maxt=7200002msec

Run status group 2 (all jobs):
   READ: io=727218MB, aggrb=103426KB/s, minb=103426KB/s, maxb=103426KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=735435MB, aggrb=104595KB/s, minb=104595KB/s, maxb=104595KB/s, mint=7200001msec, maxt=7200001msec



# 
# Fri 15 Jan 2016 03:22:57 AM EST
# 4kB randread with SATA 4 jobs and 2 I/Os Outstanding
# 4kB randread with NVMe 4 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 15378 IOPS | 61 MBps | 518 μsec Average Latency 
SATA-R5-1: 15114 IOPS | 60 MBps | 527 μsec Average Latency 
NVMe-R5-0: 17035 IOPS | 68 MBps | 467 μsec Average Latency 
NVMe-R5-1: 16851 IOPS | 67 MBps | 472 μsec Average Latency 

Totals
IOPS: 64378
MBPS: 256
Average Latency (μsec): 496

SATA-R5-01: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2
...
SATA-R5-11: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2
...
NVMe-R5-01: (g=2): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 16 processes

SATA-R5-01: (groupid=0, jobs=4): err= 0: pid=197759: Fri Jan 15 05:23:58 2016
  read : io=432512MB, bw=61513KB/s, iops=15378, runt=7200001msec
    slat (usec): min=1, max=5858, avg=10.51, stdev=10.36
    clat (usec): min=12, max=59189, avg=507.66, stdev=159.32
     lat (usec): min=125, max=59198, avg=518.35, stdev=158.82
    clat percentiles (usec):
     |  1.00th=[  334],  5.00th=[  370], 10.00th=[  402], 20.00th=[  450],
     | 30.00th=[  470], 40.00th=[  482], 50.00th=[  490], 60.00th=[  498],
     | 70.00th=[  516], 80.00th=[  548], 90.00th=[  604], 95.00th=[  644],
     | 99.00th=[ 1020], 99.50th=[ 1224], 99.90th=[ 2480], 99.95th=[ 2960],
     | 99.99th=[ 6112]
    bw (KB  /s): min=    0, max=16000, per=25.01%, avg=15382.91, stdev=260.34
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.04%, 500=60.76%
    lat (usec) : 750=36.81%, 1000=1.31%
    lat (msec) : 2=0.93%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=0.97%, sys=2.14%, ctx=207094384, majf=0, minf=6906
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=110723037/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=4): err= 0: pid=197763: Fri Jan 15 05:23:58 2016
  read : io=425094MB, bw=60458KB/s, iops=15114, runt=7200001msec
    slat (usec): min=1, max=14707, avg=15.98, stdev=14.58
    clat (usec): min=16, max=76694, avg=511.07, stdev=174.13
     lat (usec): min=121, max=76711, avg=527.27, stdev=173.03
    clat percentiles (usec):
     |  1.00th=[  322],  5.00th=[  370], 10.00th=[  402], 20.00th=[  446],
     | 30.00th=[  466], 40.00th=[  478], 50.00th=[  490], 60.00th=[  502],
     | 70.00th=[  524], 80.00th=[  556], 90.00th=[  612], 95.00th=[  652],
     | 99.00th=[ 1064], 99.50th=[ 1288], 99.90th=[ 2736], 99.95th=[ 3056],
     | 99.99th=[ 6560]
    bw (KB  /s): min=    0, max=15936, per=25.01%, avg=15119.01, stdev=270.58
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.08%, 500=58.48%
    lat (usec) : 750=38.83%, 1000=1.39%
    lat (msec) : 2=1.00%, 4=0.20%, 10=0.02%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.16%, sys=3.38%, ctx=204113240, majf=0, minf=7797
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=108824068/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=197767: Fri Jan 15 05:23:58 2016
  read : io=479122MB, bw=68142KB/s, iops=17035, runt=7200001msec
    slat (usec): min=1, max=4271, avg=12.10, stdev=11.89
    clat (usec): min=1, max=62902, avg=455.57, stdev=140.63
     lat (usec): min=81, max=62949, avg=467.85, stdev=139.57
    clat percentiles (usec):
     |  1.00th=[  270],  5.00th=[  326], 10.00th=[  346], 20.00th=[  370],
     | 30.00th=[  390], 40.00th=[  430], 50.00th=[  458], 60.00th=[  478],
     | 70.00th=[  490], 80.00th=[  506], 90.00th=[  548], 95.00th=[  612],
     | 99.00th=[  892], 99.50th=[ 1064], 99.90th=[ 1448], 99.95th=[ 1928],
     | 99.99th=[ 5216]
    bw (KB  /s): min=    0, max=17728, per=25.01%, avg=17040.02, stdev=238.85
    lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.59%, 500=77.25%, 750=20.43%, 1000=1.08%
    lat (msec) : 2=0.61%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.04%, sys=2.39%, ctx=228943061, majf=0, minf=6608
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=122655299/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=197771: Fri Jan 15 05:23:58 2016
  read : io=473948MB, bw=67406KB/s, iops=16851, runt=7200001msec
    slat (usec): min=1, max=15350, avg=16.97, stdev=15.46
    clat (usec): min=1, max=74340, avg=455.56, stdev=143.64
     lat (usec): min=90, max=74349, avg=472.74, stdev=141.91
    clat percentiles (usec):
     |  1.00th=[  258],  5.00th=[  318], 10.00th=[  342], 20.00th=[  366],
     | 30.00th=[  394], 40.00th=[  430], 50.00th=[  458], 60.00th=[  474],
     | 70.00th=[  490], 80.00th=[  510], 90.00th=[  556], 95.00th=[  612],
     | 99.00th=[  884], 99.50th=[ 1048], 99.90th=[ 1416], 99.95th=[ 1800],
     | 99.99th=[ 5920]
    bw (KB  /s): min=    0, max=18056, per=25.01%, avg=16855.91, stdev=265.71
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.76%, 500=75.36%, 750=22.15%, 1000=1.10%
    lat (msec) : 2=0.59%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.28%, sys=3.81%, ctx=226310228, majf=0, minf=8479
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=121330579/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=432512MB, aggrb=61512KB/s, minb=61512KB/s, maxb=61512KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=425094MB, aggrb=60457KB/s, minb=60457KB/s, maxb=60457KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
   READ: io=479122MB, aggrb=68141KB/s, minb=68141KB/s, maxb=68141KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=473948MB, aggrb=67405KB/s, minb=67405KB/s, maxb=67405KB/s, mint=7200001msec, maxt=7200001msec



# 
# Fri 15 Jan 2016 05:24:01 AM EST
# 4kB randread with SATA 5 jobs and 16 I/Os Outstanding
# 4kB randread with NVMe 4 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 30680 IOPS | 122 MBps | 2605 μsec Average Latency 
SATA-R5-1: 30624 IOPS | 122 MBps | 2610 μsec Average Latency 
NVMe-R5-0: 6725 IOPS | 26 MBps | 2377 μsec Average Latency 
NVMe-R5-1: 6866 IOPS | 27 MBps | 2328 μsec Average Latency 

Totals
IOPS: 74895
MBPS: 297
Average Latency (μsec): 2480

SATA-R5-01: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
SATA-R5-11: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
NVMe-R5-01: (g=2): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
...
NVMe-R5-11: (g=3): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
...
fio-2.1.10
Starting 18 processes

SATA-R5-01: (groupid=0, jobs=5): err= 0: pid=202993: Fri Jan 15 07:25:03 2016
  read : io=862882MB, bw=122721KB/s, iops=30680, runt=7200003msec
    slat (usec): min=1, max=7178, avg=12.43, stdev=13.38
    clat (usec): min=133, max=1723.2K, avg=2593.28, stdev=1567.24
     lat (usec): min=187, max=1723.2K, avg=2605.87, stdev=1567.09
    clat percentiles (usec):
     |  1.00th=[ 1848],  5.00th=[ 2064], 10.00th=[ 2160], 20.00th=[ 2288],
     | 30.00th=[ 2384], 40.00th=[ 2480], 50.00th=[ 2544], 60.00th=[ 2608],
     | 70.00th=[ 2736], 80.00th=[ 2864], 90.00th=[ 3056], 95.00th=[ 3248],
     | 99.00th=[ 3792], 99.50th=[ 4128], 99.90th=[ 5600], 99.95th=[ 7200],
     | 99.99th=[10560]
    bw (KB  /s): min=    0, max=33400, per=20.01%, avg=24554.47, stdev=566.68
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=3.21%, 4=96.16%, 10=0.60%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, 2000=0.01%
  cpu          : usr=1.42%, sys=3.33%, ctx=386234078, majf=0, minf=21520
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=220897606/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
SATA-R5-11: (groupid=1, jobs=5): err= 0: pid=202998: Fri Jan 15 07:25:03 2016
  read : io=861306MB, bw=122497KB/s, iops=30624, runt=7200003msec
    slat (usec): min=1, max=13880, avg=16.18, stdev=15.49
    clat (usec): min=112, max=1744.1K, avg=2594.31, stdev=1577.53
     lat (usec): min=210, max=1744.1K, avg=2610.68, stdev=1577.32
    clat percentiles (usec):
     |  1.00th=[ 1832],  5.00th=[ 2064], 10.00th=[ 2192], 20.00th=[ 2320],
     | 30.00th=[ 2384], 40.00th=[ 2480], 50.00th=[ 2544], 60.00th=[ 2640],
     | 70.00th=[ 2736], 80.00th=[ 2864], 90.00th=[ 3056], 95.00th=[ 3216],
     | 99.00th=[ 3728], 99.50th=[ 4048], 99.90th=[ 5600], 99.95th=[ 7136],
     | 99.99th=[10816]
    bw (KB  /s): min=    0, max=47008, per=20.01%, avg=24510.50, stdev=566.78
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=3.35%, 4=96.08%, 10=0.53%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, 2000=0.01%
  cpu          : usr=1.60%, sys=5.19%, ctx=376189341, majf=0, minf=25894
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=220494285/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=203003: Fri Jan 15 07:25:03 2016
  read : io=189162MB, bw=26903KB/s, iops=6725, runt=7200003msec
    slat (usec): min=1, max=3113, avg= 4.56, stdev= 5.44
    clat (usec): min=37, max=1689.9K, avg=2372.67, stdev=1491.60
     lat (usec): min=124, max=1689.9K, avg=2377.36, stdev=1491.61
    clat percentiles (usec):
     |  1.00th=[ 1608],  5.00th=[ 1848], 10.00th=[ 1960], 20.00th=[ 2096],
     | 30.00th=[ 2160], 40.00th=[ 2256], 50.00th=[ 2320], 60.00th=[ 2384],
     | 70.00th=[ 2512], 80.00th=[ 2640], 90.00th=[ 2864], 95.00th=[ 3024],
     | 99.00th=[ 3504], 99.50th=[ 3792], 99.90th=[ 5216], 99.95th=[ 6752],
     | 99.99th=[10176]
    bw (KB  /s): min=    0, max=27432, per=25.02%, avg=6732.10, stdev=182.87
    lat (usec) : 50=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.03%
    lat (msec) : 2=14.07%, 4=85.55%, 10=0.32%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, 2000=0.01%
  cpu          : usr=0.46%, sys=1.01%, ctx=47031641, majf=0, minf=8340
  IO depths    : 1=0.1%, 2=0.1%, 4=100.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=48425463/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=203007: Fri Jan 15 07:25:03 2016
  read : io=193132MB, bw=27468KB/s, iops=6866, runt=7200001msec
    slat (usec): min=1, max=6389, avg= 8.66, stdev=10.38
    clat (usec): min=15, max=1719.1K, avg=2319.55, stdev=1477.01
     lat (usec): min=167, max=1719.1K, avg=2328.38, stdev=1476.93
    clat percentiles (usec):
     |  1.00th=[ 1512],  5.00th=[ 1768], 10.00th=[ 1880], 20.00th=[ 2008],
     | 30.00th=[ 2128], 40.00th=[ 2192], 50.00th=[ 2256], 60.00th=[ 2352],
     | 70.00th=[ 2480], 80.00th=[ 2608], 90.00th=[ 2800], 95.00th=[ 2992],
     | 99.00th=[ 3472], 99.50th=[ 3728], 99.90th=[ 5152], 99.95th=[ 6624],
     | 99.99th=[10304]
    bw (KB  /s): min=    0, max=29608, per=25.02%, avg=6872.04, stdev=226.33
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.02%
    lat (usec) : 750=0.03%, 1000=0.05%
    lat (msec) : 2=18.87%, 4=80.74%, 10=0.28%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01%
  cpu          : usr=0.56%, sys=1.44%, ctx=57529741, majf=0, minf=15964
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=49441653/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=862882MB, aggrb=122720KB/s, minb=122720KB/s, maxb=122720KB/s, mint=7200003msec, maxt=7200003msec

Run status group 1 (all jobs):
   READ: io=861306MB, aggrb=122496KB/s, minb=122496KB/s, maxb=122496KB/s, mint=7200003msec, maxt=7200003msec

Run status group 2 (all jobs):
   READ: io=189162MB, aggrb=26903KB/s, minb=26903KB/s, maxb=26903KB/s, mint=7200003msec, maxt=7200003msec

Run status group 3 (all jobs):
   READ: io=193132MB, aggrb=27467KB/s, minb=27467KB/s, maxb=27467KB/s, mint=7200001msec, maxt=7200001msec