client1
2016 January 13, 17:05
fio-2.1.10
CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
60 s ramp + 7200 s run, per test; s between tests

# 
# SR-71mach5 Configuration
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 

Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz   Status: Populated, Enabled Core Enabled: 8
Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz   Status: Populated, Enabled Core Enabled: 8
DIMM_A1: 4096 MB | 64/72 bits | 1600 MHz | Hynix            | HMT351R7CFR8C-PB | SN: 38B42B25  
DIMM_B1: 4096 MB | 64/72 bits | 1600 MHz | Hynix            | HMT351R7CFR8C-PB | SN: 38642B21  
DIMM_C1: 4096 MB | 64/72 bits | 1600 MHz | Hynix            | HMT351R7CFR8C-PB | SN: 38A42B25  
DIMM_D1: 4096 MB | 64/72 bits | 1600 MHz | Hynix            | HMT351R7CFR8C-PB | SN: 38942B16  
DIMM_E1: 4096 MB | 64/72 bits | 1600 MHz | Hynix            | HMT351R7CFR8C-PB | SN: 2C637F25  
DIMM_F1: 4096 MB | 64/72 bits | 1600 MHz | Hynix            | HMT351R7CFR8C-PB | SN: 2C137F27  
DIMM_G1: 4096 MB | 64/72 bits | 1600 MHz | Hynix            | HMT351R7CFR8C-PB | SN: 38A42B2B  
DIMM_H1: 4096 MB | 64/72 bits | 1600 MHz | Hynix            | HMT351R7CFR8C-PB | SN: 38442B23  
____________________________________________________________________
MemTotal:       32673120 kB


# 
# actual mount paramters
# 

10.0.5.144:/control on /root/sr71/control type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.5.132,local_lock=none,addr=10.0.5.144)
10.10.0.100:/sr71/C0P1 on /root/sr71/SATA0 type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=100,retrans=2,sec=sys,clientaddr=10.10.0.10,local_lock=none,addr=10.10.0.100)
10.10.1.100:/sr71/C1P1 on /root/sr71/SATA1 type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=100,retrans=2,sec=sys,clientaddr=10.10.1.10,local_lock=none,addr=10.10.1.100)
10.10.0.100:/sr71/NVMeR5V0P1 on /root/sr71/NVMe0 type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=100,retrans=2,sec=sys,clientaddr=10.10.0.10,local_lock=none,addr=10.10.0.100)
10.10.1.100:/sr71/NVMeR5V1P1 on /root/sr71/NVMe1 type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=100,retrans=2,sec=sys,clientaddr=10.10.1.10,local_lock=none,addr=10.10.1.100)



# 
# fio job file
fio-2.1.10
# 

 seconds between fio tests
[global]
ioengine=libaio
direct=1
time_based
ramp_time=${RAMPTIME}
runtime=${RUNTIME}
bs=${BS}
rw=${RW}
rwmixread=${RWMIXREAD}
group_reporting

[SATA-R5-01]
new_group
size=1536g
iodepth=${IODEPTHS}
numjobs=${NUMJOBSS}
filename=/root/sr71/SATA0/fio.tst

[SATA-R5-11]
new_group
size=1536g
iodepth=${IODEPTHS}
numjobs=${NUMJOBSS}
filename=/root/sr71/SATA1/fio.tst

[NVMe-R5-01]
new_group
size=200g
iodepth=${IODEPTHN}
numjobs=${NUMJOBSN}
filename=/root/sr71/NVMe0/fio.tst

[NVMe-R5-11]
new_group
size=200g
iodepth=${IODEPTHN}
numjobs=${NUMJOBSN}
filename=/root/sr71/NVMe1/fio.tst




# 
# 201601131705 test details
# 



# 
# Wed 13 Jan 2016 05:05:00 PM EST
# 8kB randwrite with SATA 1 jobs and 1 I/Os Outstanding
# 8kB randwrite with NVMe 1 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 2937 IOPS | 23 MBps | 338 μsec Average Latency 
SATA-R5-1: 2803 IOPS | 22 MBps | 355 μsec Average Latency 
NVMe-R5-0: 5072 IOPS | 40 MBps | 392 μsec Average Latency 
NVMe-R5-1: 5407 IOPS | 43 MBps | 367 μsec Average Latency 

Totals
IOPS: 16219
MBPS: 128
Average Latency (μsec): 363

SATA-R5-01: (g=0): rw=randwrite, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=1
SATA-R5-11: (g=1): rw=randwrite, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=1
NVMe-R5-01: (g=2): rw=randwrite, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
NVMe-R5-11: (g=3): rw=randwrite, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=101027: Wed Jan 13 19:06:01 2016
  write: io=165250MB, bw=23502KB/s, iops=2937, runt=7200001msec
    slat (usec): min=4, max=219, avg=11.92, stdev= 5.36
    clat (usec): min=0, max=199494, avg=326.43, stdev=305.14
     lat (usec): min=96, max=199510, avg=338.51, stdev=304.48
    clat percentiles (usec):
     |  1.00th=[  106],  5.00th=[  115], 10.00th=[  215], 20.00th=[  247],
     | 30.00th=[  346], 40.00th=[  362], 50.00th=[  362], 60.00th=[  362],
     | 70.00th=[  366], 80.00th=[  366], 90.00th=[  370], 95.00th=[  386],
     | 99.00th=[  418], 99.50th=[  458], 99.90th=[  516], 99.95th=[  644],
     | 99.99th=[ 1336]
    bw (KB  /s): min=    0, max=42208, per=100.00%, avg=23510.14, stdev=2448.86
    lat (usec) : 2=0.01%, 50=0.01%, 100=0.18%, 250=20.37%, 500=79.30%
    lat (usec) : 750=0.12%, 1000=0.02%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.95%, sys=2.41%, ctx=42640647, majf=0, minf=1450
  IO depths    : 1=100.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=21152027/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=101028: Wed Jan 13 19:06:01 2016
  write: io=157700MB, bw=22428KB/s, iops=2803, runt=7200001msec
    slat (usec): min=4, max=1022, avg=11.13, stdev= 4.18
    clat (usec): min=1, max=177664, avg=343.81, stdev=74.23
     lat (usec): min=101, max=177693, avg=355.06, stdev=72.98
    clat percentiles (usec):
     |  1.00th=[  159],  5.00th=[  225], 10.00th=[  245], 20.00th=[  338],
     | 30.00th=[  358], 40.00th=[  362], 50.00th=[  362], 60.00th=[  366],
     | 70.00th=[  366], 80.00th=[  366], 90.00th=[  374], 95.00th=[  386],
     | 99.00th=[  414], 99.50th=[  446], 99.90th=[  516], 99.95th=[  628],
     | 99.99th=[ 1128]
    bw (KB  /s): min=    0, max=35920, per=100.00%, avg=22435.10, stdev=853.45
    lat (usec) : 2=0.01%, 100=0.01%, 250=11.10%, 500=88.74%, 750=0.11%
    lat (usec) : 1000=0.02%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.80%, sys=2.04%, ctx=40717450, majf=0, minf=1231
  IO depths    : 1=100.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=20185556/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=101029: Wed Jan 13 19:06:01 2016
  write: io=285331MB, bw=40580KB/s, iops=5072, runt=7200001msec
    slat (usec): min=2, max=498, avg=10.78, stdev= 3.98
    clat (usec): min=1, max=201250, avg=381.93, stdev=1860.25
     lat (usec): min=99, max=201261, avg=392.85, stdev=1860.21
    clat percentiles (usec):
     |  1.00th=[  185],  5.00th=[  235], 10.00th=[  286], 20.00th=[  350],
     | 30.00th=[  362], 40.00th=[  362], 50.00th=[  366], 60.00th=[  366],
     | 70.00th=[  366], 80.00th=[  366], 90.00th=[  374], 95.00th=[  394],
     | 99.00th=[  430], 99.50th=[  478], 99.90th=[  668], 99.95th=[16768],
     | 99.99th=[95744]
    bw (KB  /s): min=    0, max=50352, per=100.00%, avg=40676.62, stdev=5562.94
    lat (usec) : 2=0.01%, 20=0.01%, 100=0.01%, 250=7.26%, 500=92.48%
    lat (usec) : 750=0.16%, 1000=0.02%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.29%, sys=3.89%, ctx=72512427, majf=0, minf=1400
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=36522383/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=101030: Wed Jan 13 19:06:01 2016
  write: io=304145MB, bw=43256KB/s, iops=5407, runt=7200001msec
    slat (usec): min=3, max=2575, avg=23.68, stdev=11.12
    clat (usec): min=17, max=200393, avg=343.36, stdev=458.72
     lat (usec): min=105, max=200429, avg=367.53, stdev=458.51
    clat percentiles (usec):
     |  1.00th=[  195],  5.00th=[  245], 10.00th=[  302], 20.00th=[  330],
     | 30.00th=[  338], 40.00th=[  346], 50.00th=[  346], 60.00th=[  350],
     | 70.00th=[  362], 80.00th=[  362], 90.00th=[  366], 95.00th=[  378],
     | 99.00th=[  434], 99.50th=[  466], 99.90th=[  532], 99.95th=[  692],
     | 99.99th=[ 1480]
    bw (KB  /s): min=    0, max=46848, per=100.00%, avg=43268.46, stdev=1687.00
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=5.51%, 500=94.32%
    lat (usec) : 750=0.13%, 1000=0.02%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.41%, sys=7.21%, ctx=77391884, majf=0, minf=1828
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=38930574/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
  WRITE: io=165250MB, aggrb=23502KB/s, minb=23502KB/s, maxb=23502KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
  WRITE: io=157700MB, aggrb=22428KB/s, minb=22428KB/s, maxb=22428KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
  WRITE: io=285331MB, aggrb=40580KB/s, minb=40580KB/s, maxb=40580KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
  WRITE: io=304145MB, aggrb=43256KB/s, minb=43256KB/s, maxb=43256KB/s, mint=7200001msec, maxt=7200001msec



# 
# Wed 13 Jan 2016 07:06:03 PM EST
# 64kB randwrite with SATA 1 jobs and 1 I/Os Outstanding
# 64kB randwrite with NVMe 4 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 1240 IOPS | 79 MBps | 800 μsec Average Latency 
SATA-R5-1: 1389 IOPS | 88 MBps | 716 μsec Average Latency 
NVMe-R5-0: 599 IOPS | 38 MBps | 13351 μsec Average Latency 
NVMe-R5-1: 648 IOPS | 41 MBps | 12339 μsec Average Latency 

Totals
IOPS: 3876
MBPS: 246
Average Latency (μsec): 6801

SATA-R5-01: (g=0): rw=randwrite, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=1
SATA-R5-11: (g=1): rw=randwrite, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=1
NVMe-R5-01: (g=2): rw=randwrite, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randwrite, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 10 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=106146: Wed Jan 13 21:07:04 2016
  write: io=558381MB, bw=79414KB/s, iops=1240, runt=7200001msec
    slat (usec): min=11, max=741, avg=58.55, stdev=13.86
    clat (usec): min=3, max=3933.5K, avg=741.73, stdev=9251.03
     lat (usec): min=212, max=3933.6K, avg=800.62, stdev=9251.09
    clat percentiles (usec):
     |  1.00th=[  298],  5.00th=[  306], 10.00th=[  314], 20.00th=[  330],
     | 30.00th=[  342], 40.00th=[  406], 50.00th=[  430], 60.00th=[  446],
     | 70.00th=[  470], 80.00th=[  540], 90.00th=[  564], 95.00th=[  588],
     | 99.00th=[  700], 99.50th=[  988], 99.90th=[99840], 99.95th=[158720],
     | 99.99th=[395264]
    bw (KB  /s): min=    1, max=164352, per=100.00%, avg=82366.02, stdev=25733.49
    lat (usec) : 4=0.01%, 250=0.02%, 500=73.37%, 750=25.79%, 1000=0.33%
    lat (msec) : 2=0.14%, 4=0.03%, 10=0.02%, 20=0.01%, 50=0.11%
    lat (msec) : 100=0.07%, 250=0.08%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=1.53%, sys=3.94%, ctx=18037889, majf=0, minf=3362
  IO depths    : 1=100.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=8934095/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=106147: Wed Jan 13 21:07:04 2016
  write: io=625429MB, bw=88950KB/s, iops=1389, runt=7200001msec
    slat (usec): min=11, max=292, avg=34.50, stdev=10.27
    clat (usec): min=85, max=3900.6K, avg=682.24, stdev=8732.54
     lat (usec): min=193, max=3900.6K, avg=716.99, stdev=8732.57
    clat percentiles (usec):
     |  1.00th=[  294],  5.00th=[  310], 10.00th=[  326], 20.00th=[  338],
     | 30.00th=[  342], 40.00th=[  342], 50.00th=[  346], 60.00th=[  354],
     | 70.00th=[  374], 80.00th=[  442], 90.00th=[  470], 95.00th=[  540],
     | 99.00th=[  668], 99.50th=[  916], 99.90th=[96768], 99.95th=[150528],
     | 99.99th=[354304]
    bw (KB  /s): min=    1, max=168192, per=100.00%, avg=92172.07, stdev=30343.44
    lat (usec) : 100=0.01%, 250=0.03%, 500=93.34%, 750=5.93%, 1000=0.26%
    lat (msec) : 2=0.11%, 4=0.02%, 10=0.01%, 20=0.02%, 50=0.12%
    lat (msec) : 100=0.08%, 250=0.08%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=1.11%, sys=2.63%, ctx=20194870, majf=0, minf=3332
  IO depths    : 1=100.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10006860/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=106148: Wed Jan 13 21:07:04 2016
  write: io=269564MB, bw=38337KB/s, iops=599, runt=7200192msec
    slat (usec): min=7, max=2063, avg=49.42, stdev=30.52
    clat (usec): min=87, max=3984.1K, avg=13302.00, stdev=52955.30
     lat (usec): min=240, max=3984.1K, avg=13351.68, stdev=52954.92
    clat percentiles (usec):
     |  1.00th=[  298],  5.00th=[  370], 10.00th=[  410], 20.00th=[  458],
     | 30.00th=[  490], 40.00th=[  524], 50.00th=[  564], 60.00th=[  604],
     | 70.00th=[  652], 80.00th=[  724], 90.00th=[ 1064], 95.00th=[132096],
     | 99.00th=[199680], 99.50th=[276480], 99.90th=[468992], 99.95th=[610304],
     | 99.99th=[1003520]
    bw (KB  /s): min=    1, max=141044, per=25.46%, avg=9759.05, stdev=6833.08
    lat (usec) : 100=0.01%, 250=0.25%, 500=32.17%, 750=49.61%, 1000=7.31%
    lat (msec) : 2=2.66%, 4=0.29%, 10=0.12%, 20=0.05%, 50=0.58%
    lat (msec) : 100=1.46%, 250=4.88%, 500=0.53%, 750=0.06%, 1000=0.02%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=0.15%, sys=0.39%, ctx=8057429, majf=0, minf=20849
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=4313025/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=106152: Wed Jan 13 21:07:04 2016
  write: io=291659MB, bw=41479KB/s, iops=648, runt=7200181msec
    slat (usec): min=8, max=391, avg=50.69, stdev=31.65
    clat (usec): min=59, max=3983.3K, avg=12289.02, stdev=50805.07
     lat (usec): min=239, max=3983.4K, avg=12339.99, stdev=50804.63
    clat percentiles (usec):
     |  1.00th=[  294],  5.00th=[  370], 10.00th=[  410], 20.00th=[  462],
     | 30.00th=[  498], 40.00th=[  532], 50.00th=[  572], 60.00th=[  604],
     | 70.00th=[  652], 80.00th=[  724], 90.00th=[ 1020], 95.00th=[95744],
     | 99.00th=[199680], 99.50th=[264192], 99.90th=[448512], 99.95th=[602112],
     | 99.99th=[1028096]
    bw (KB  /s): min=    1, max=126208, per=25.62%, avg=10626.48, stdev=7134.68
    lat (usec) : 100=0.01%, 250=0.29%, 500=30.77%, 750=51.13%, 1000=7.59%
    lat (msec) : 2=2.55%, 4=0.26%, 10=0.11%, 20=0.05%, 50=0.68%
    lat (msec) : 100=1.69%, 250=4.32%, 500=0.47%, 750=0.05%, 1000=0.02%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=0.15%, sys=0.43%, ctx=8727063, majf=0, minf=20707
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=4666536/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
  WRITE: io=558381MB, aggrb=79414KB/s, minb=79414KB/s, maxb=79414KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
  WRITE: io=625429MB, aggrb=88949KB/s, minb=88949KB/s, maxb=88949KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
  WRITE: io=269564MB, aggrb=38337KB/s, minb=38337KB/s, maxb=38337KB/s, mint=7200192msec, maxt=7200192msec

Run status group 3 (all jobs):
  WRITE: io=291659MB, aggrb=41479KB/s, minb=41479KB/s, maxb=41479KB/s, mint=7200181msec, maxt=7200181msec



# 
# Wed 13 Jan 2016 09:07:08 PM EST
# 8kB randrw with SATA 4 jobs and 8 I/Os Outstanding
# 8kB randrw with NVMe 4 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 6053 IOPS | 32 MBps | 7031 μsec Average Latency 
SATA-R5-1: 5863 IOPS | 31 MBps | 7297 μsec Average Latency 
NVMe-R5-0: 10271 IOPS | 55 MBps | 4071 μsec Average Latency 
NVMe-R5-1: 10735 IOPS | 57 MBps | 3887 μsec Average Latency 

Totals
IOPS: 32922
MBPS: 175
Average Latency (μsec): 5571

SATA-R5-01: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
SATA-R5-11: (g=1): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
NVMe-R5-01: (g=2): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
NVMe-R5-11: (g=3): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
fio-2.1.10
Starting 16 processes

SATA-R5-01: (groupid=0, jobs=4): err= 0: pid=111307: Wed Jan 13 23:08:09 2016
  read : io=228169MB, bw=32451KB/s, iops=4056, runt=7200011msec
    slat (usec): min=2, max=3047, avg=19.52, stdev=20.77
    clat (usec): min=1, max=1579.3K, avg=1870.78, stdev=4861.50
     lat (usec): min=143, max=1579.3K, avg=1890.51, stdev=4862.03
    clat percentiles (usec):
     |  1.00th=[  378],  5.00th=[  462], 10.00th=[  506], 20.00th=[  572],
     | 30.00th=[  620], 40.00th=[  676], 50.00th=[  732], 60.00th=[  820],
     | 70.00th=[  980], 80.00th=[ 1768], 90.00th=[ 3152], 95.00th=[ 9664],
     | 99.00th=[16768], 99.50th=[18304], 99.90th=[25216], 99.95th=[28288],
     | 99.99th=[78336]
    bw (KB  /s): min=    0, max=27264, per=25.28%, avg=8204.67, stdev=8687.94
  write: io=112371MB, bw=15982KB/s, iops=1997, runt=7200011msec
    slat (usec): min=2, max=664, avg=25.23, stdev=23.58
    clat (usec): min=11, max=1661.5K, avg=12147.35, stdev=32395.42
     lat (usec): min=155, max=1661.5K, avg=12172.79, stdev=32395.85
    clat percentiles (usec):
     |  1.00th=[  390],  5.00th=[  604], 10.00th=[ 1304], 20.00th=[ 2512],
     | 30.00th=[ 3120], 40.00th=[ 3632], 50.00th=[ 4128], 60.00th=[ 4640],
     | 70.00th=[ 5472], 80.00th=[ 7136], 90.00th=[16192], 95.00th=[57600],
     | 99.00th=[185344], 99.50th=[209920], 99.90th=[259072], 99.95th=[296960],
     | 99.99th=[602112]
    bw (KB  /s): min=    7, max=13536, per=25.29%, avg=4040.84, stdev=4277.70
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.05%, 500=7.14%, 750=30.18%, 1000=12.89%
    lat (msec) : 2=9.03%, 4=18.47%, 10=14.69%, 20=4.29%, 50=1.47%
    lat (msec) : 100=0.71%, 250=1.05%, 500=0.04%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=0.61%, sys=1.66%, ctx=72457231, majf=0, minf=15283
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=29205565/w=14383453/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=4): err= 0: pid=111311: Wed Jan 13 23:08:09 2016
  read : io=221000MB, bw=31431KB/s, iops=3928, runt=7200011msec
    slat (usec): min=2, max=802, avg=15.23, stdev=15.58
    clat (usec): min=13, max=1557.7K, avg=1863.02, stdev=5031.65
     lat (usec): min=122, max=1557.7K, avg=1878.41, stdev=5032.78
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[  454], 10.00th=[  494], 20.00th=[  564],
     | 30.00th=[  604], 40.00th=[  652], 50.00th=[  716], 60.00th=[  788],
     | 70.00th=[  948], 80.00th=[ 1720], 90.00th=[ 3120], 95.00th=[ 9792],
     | 99.00th=[17280], 99.50th=[18816], 99.90th=[25472], 99.95th=[28032],
     | 99.99th=[55552]
    bw (KB  /s): min=    0, max=26576, per=25.28%, avg=7947.31, stdev=8543.19
  write: io=108851MB, bw=15481KB/s, iops=1935, runt=7200011msec
    slat (usec): min=2, max=718, avg=19.65, stdev=18.36
    clat (usec): min=1, max=1563.6K, avg=12697.35, stdev=33812.45
     lat (usec): min=141, max=1563.6K, avg=12717.16, stdev=33813.88
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[  588], 10.00th=[ 1384], 20.00th=[ 2640],
     | 30.00th=[ 3248], 40.00th=[ 3728], 50.00th=[ 4192], 60.00th=[ 4768],
     | 70.00th=[ 5664], 80.00th=[ 7456], 90.00th=[17280], 95.00th=[61184],
     | 99.00th=[191488], 99.50th=[216064], 99.90th=[264192], 99.95th=[296960],
     | 99.99th=[643072]
    bw (KB  /s): min=    0, max=12736, per=25.29%, avg=3914.48, stdev=4205.38
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.05%
    lat (usec) : 500=8.44%, 750=31.38%, 1000=11.09%
    lat (msec) : 2=8.17%, 4=17.87%, 10=15.15%, 20=4.46%, 50=1.54%
    lat (msec) : 100=0.69%, 250=1.11%, 500=0.04%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=0.49%, sys=1.44%, ctx=72126928, majf=0, minf=14146
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=28288057/w=13932852/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=111315: Wed Jan 13 23:08:09 2016
  read : io=387087MB, bw=55052KB/s, iops=6881, runt=7200010msec
    slat (usec): min=2, max=2052, avg=20.62, stdev=20.90
    clat (usec): min=1, max=1588.3K, avg=1230.09, stdev=3494.77
     lat (usec): min=109, max=1588.4K, avg=1250.92, stdev=3495.57
    clat percentiles (usec):
     |  1.00th=[  334],  5.00th=[  418], 10.00th=[  466], 20.00th=[  524],
     | 30.00th=[  580], 40.00th=[  636], 50.00th=[  708], 60.00th=[  820],
     | 70.00th=[ 1176], 80.00th=[ 1992], 90.00th=[ 2736], 95.00th=[ 3440],
     | 99.00th=[ 4896], 99.50th=[ 5472], 99.90th=[ 7584], 99.95th=[ 8512],
     | 99.99th=[12224]
    bw (KB  /s): min=    0, max=36784, per=25.13%, avg=13834.65, stdev=9626.64
  write: io=190694MB, bw=27121KB/s, iops=3390, runt=7200010msec
    slat (usec): min=2, max=1545, avg=26.42, stdev=23.45
    clat (usec): min=1, max=1597.2K, avg=6865.77, stdev=11222.10
     lat (usec): min=146, max=1597.2K, avg=6892.42, stdev=11224.14
    clat percentiles (usec):
     |  1.00th=[  366],  5.00th=[  548], 10.00th=[  876], 20.00th=[ 1704],
     | 30.00th=[ 2160], 40.00th=[ 2544], 50.00th=[ 2960], 60.00th=[ 3600],
     | 70.00th=[ 6368], 80.00th=[12608], 90.00th=[19328], 95.00th=[24192],
     | 99.00th=[32384], 99.50th=[35584], 99.90th=[42752], 99.95th=[47872],
     | 99.99th=[399360]
    bw (KB  /s): min=    0, max=18224, per=25.13%, avg=6815.92, stdev=4741.22
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.14%
    lat (usec) : 500=12.05%, 750=27.50%, 1000=8.63%
    lat (msec) : 2=13.95%, 4=23.60%, 10=6.15%, 20=4.90%, 50=3.06%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=1.12%, sys=2.98%, ctx=122203162, majf=0, minf=14357
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.7%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=49547165/w=24408798/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=111319: Wed Jan 13 23:08:09 2016
  read : io=404581MB, bw=57540KB/s, iops=7192, runt=7200010msec
    slat (usec): min=2, max=1624, avg=20.24, stdev=20.18
    clat (usec): min=1, max=1633.6K, avg=1191.51, stdev=3671.82
     lat (usec): min=109, max=1633.6K, avg=1211.94, stdev=3672.38
    clat percentiles (usec):
     |  1.00th=[  330],  5.00th=[  414], 10.00th=[  462], 20.00th=[  516],
     | 30.00th=[  580], 40.00th=[  628], 50.00th=[  700], 60.00th=[  820],
     | 70.00th=[ 1160], 80.00th=[ 1928], 90.00th=[ 2608], 95.00th=[ 3152],
     | 99.00th=[ 4576], 99.50th=[ 5088], 99.90th=[ 7264], 99.95th=[ 8256],
     | 99.99th=[12096]
    bw (KB  /s): min=    0, max=36336, per=25.14%, avg=14465.69, stdev=9210.37
  write: io=199303MB, bw=28345KB/s, iops=3543, runt=7200010msec
    slat (usec): min=2, max=2382, avg=25.99, stdev=23.18
    clat (usec): min=2, max=1636.4K, avg=6537.96, stdev=10550.79
     lat (usec): min=133, max=1636.4K, avg=6564.16, stdev=10552.27
    clat percentiles (usec):
     |  1.00th=[  358],  5.00th=[  548], 10.00th=[  956], 20.00th=[ 1848],
     | 30.00th=[ 2320], 40.00th=[ 2704], 50.00th=[ 3152], 60.00th=[ 3920],
     | 70.00th=[ 6880], 80.00th=[11840], 90.00th=[17280], 95.00th=[21376],
     | 99.00th=[28288], 99.50th=[30848], 99.90th=[37120], 99.95th=[42240],
     | 99.99th=[432128]
    bw (KB  /s): min=    0, max=18080, per=25.14%, avg=7125.96, stdev=4541.52
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.14%, 500=12.55%, 750=27.14%, 1000=8.26%
    lat (msec) : 2=13.73%, 4=23.71%, 10=6.66%, 20=5.69%, 50=2.09%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=1.11%, sys=3.04%, ctx=128356061, majf=0, minf=15835
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.7%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=51786346/w=25510703/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=228169MB, aggrb=32450KB/s, minb=32450KB/s, maxb=32450KB/s, mint=7200011msec, maxt=7200011msec
  WRITE: io=112371MB, aggrb=15981KB/s, minb=15981KB/s, maxb=15981KB/s, mint=7200011msec, maxt=7200011msec

Run status group 1 (all jobs):
   READ: io=221000MB, aggrb=31431KB/s, minb=31431KB/s, maxb=31431KB/s, mint=7200011msec, maxt=7200011msec
  WRITE: io=108851MB, aggrb=15480KB/s, minb=15480KB/s, maxb=15480KB/s, mint=7200011msec, maxt=7200011msec

Run status group 2 (all jobs):
   READ: io=387087MB, aggrb=55052KB/s, minb=55052KB/s, maxb=55052KB/s, mint=7200010msec, maxt=7200010msec
  WRITE: io=190694MB, aggrb=27120KB/s, minb=27120KB/s, maxb=27120KB/s, mint=7200010msec, maxt=7200010msec

Run status group 3 (all jobs):
   READ: io=404581MB, aggrb=57540KB/s, minb=57540KB/s, maxb=57540KB/s, mint=7200010msec, maxt=7200010msec
  WRITE: io=199303MB, aggrb=28345KB/s, minb=28345KB/s, maxb=28345KB/s, mint=7200010msec, maxt=7200010msec



# 
# Wed 13 Jan 2016 11:08:12 PM EST
# 64kB randrw with SATA 5 jobs and 4 I/Os Outstanding
# 64kB randrw with NVMe 5 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 1862 IOPS | 79 MBps | 14536 μsec Average Latency 
SATA-R5-1: 1823 IOPS | 78 MBps | 14807 μsec Average Latency 
NVMe-R5-0: 1539 IOPS | 66 MBps | 7754 μsec Average Latency 
NVMe-R5-1: 1632 IOPS | 70 MBps | 7296 μsec Average Latency 

Totals
IOPS: 6856
MBPS: 293
Average Latency (μsec): 11098

SATA-R5-01: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
SATA-R5-11: (g=1): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
NVMe-R5-01: (g=2): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 20 processes

SATA-R5-01: (groupid=0, jobs=5): err= 0: pid=116586: Thu Jan 14 01:09:12 2016
  read : io=561928MB, bw=79918KB/s, iops=1248, runt=7200024msec
    slat (usec): min=5, max=1040, avg=26.19, stdev=16.82
    clat (usec): min=180, max=3532.1K, avg=3341.00, stdev=30337.70
     lat (usec): min=206, max=3533.2K, avg=3367.38, stdev=30337.89
    clat percentiles (usec):
     |  1.00th=[  732],  5.00th=[  876], 10.00th=[  964], 20.00th=[ 1080],
     | 30.00th=[ 1160], 40.00th=[ 1256], 50.00th=[ 1336], 60.00th=[ 1448],
     | 70.00th=[ 1640], 80.00th=[ 2576], 90.00th=[ 5088], 95.00th=[ 7840],
     | 99.00th=[18048], 99.50th=[43264], 99.90th=[201728], 99.95th=[561152],
     | 99.99th=[1564672]
    bw (KB  /s): min=    1, max=51353, per=23.22%, avg=18557.65, stdev=12631.79
  write: io=276423MB, bw=39313KB/s, iops=614, runt=7200024msec
    slat (usec): min=10, max=1463, avg=44.70, stdev=23.34
    clat (usec): min=19, max=4217.1K, avg=25660.12, stdev=98863.09
     lat (usec): min=235, max=4217.1K, avg=25705.01, stdev=98863.49
    clat percentiles (usec):
     |  1.00th=[  494],  5.00th=[ 1128], 10.00th=[ 5024], 20.00th=[ 7264],
     | 30.00th=[ 8384], 40.00th=[ 9408], 50.00th=[10304], 60.00th=[11328],
     | 70.00th=[13632], 80.00th=[18560], 90.00th=[47872], 95.00th=[71168],
     | 99.00th=[272384], 99.50th=[509952], 99.90th=[1630208], 99.95th=[1990656],
     | 99.99th=[2768896]
    bw (KB  /s): min=    1, max=23680, per=23.25%, avg=9140.45, stdev=6159.66
    lat (usec) : 20=0.01%, 50=0.01%, 250=0.01%, 500=0.35%, 750=1.16%
    lat (usec) : 1000=8.19%
    lat (msec) : 2=44.03%, 4=7.52%, 10=19.39%, 20=12.76%, 50=3.19%
    lat (msec) : 100=2.13%, 250=0.88%, 500=0.21%, 750=0.06%, 1000=0.04%
    lat (msec) : 2000=0.08%, >=2000=0.02%
  cpu          : usr=0.27%, sys=0.77%, ctx=25809955, majf=0, minf=41663
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8990850/w=4422753/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
SATA-R5-11: (groupid=1, jobs=5): err= 0: pid=116591: Thu Jan 14 01:09:12 2016
  read : io=550214MB, bw=78252KB/s, iops=1222, runt=7200032msec
    slat (usec): min=5, max=798, avg=38.22, stdev=27.59
    clat (usec): min=152, max=4144.4K, avg=3471.53, stdev=31791.26
     lat (usec): min=218, max=4144.4K, avg=3510.04, stdev=31791.34
    clat percentiles (usec):
     |  1.00th=[  732],  5.00th=[  868], 10.00th=[  956], 20.00th=[ 1064],
     | 30.00th=[ 1160], 40.00th=[ 1256], 50.00th=[ 1352], 60.00th=[ 1480],
     | 70.00th=[ 1720], 80.00th=[ 2896], 90.00th=[ 5472], 95.00th=[ 8256],
     | 99.00th=[16768], 99.50th=[41728], 99.90th=[203776], 99.95th=[643072],
     | 99.99th=[1581056]
    bw (KB  /s): min=    1, max=52352, per=23.32%, avg=18251.71, stdev=12702.53
  write: io=270631MB, bw=38490KB/s, iops=601, runt=7200032msec
    slat (usec): min=10, max=1003, avg=61.80, stdev=35.93
    clat (usec): min=1, max=4230.2K, avg=26043.44, stdev=100892.31
     lat (usec): min=252, max=4230.2K, avg=26105.54, stdev=100892.27
    clat percentiles (usec):
     |  1.00th=[  498],  5.00th=[ 1048], 10.00th=[ 3568], 20.00th=[ 6816],
     | 30.00th=[ 8096], 40.00th=[ 9152], 50.00th=[10048], 60.00th=[11328],
     | 70.00th=[13888], 80.00th=[19584], 90.00th=[50944], 95.00th=[72192],
     | 99.00th=[268288], 99.50th=[544768], 99.90th=[1630208], 99.95th=[2023424],
     | 99.99th=[2899968]
    bw (KB  /s): min=    1, max=24960, per=23.34%, avg=8984.49, stdev=6207.01
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=0.35%, 750=1.35%, 1000=8.85%
    lat (msec) : 2=42.36%, 4=8.08%, 10=20.16%, 20=11.81%, 50=3.37%
    lat (msec) : 100=2.41%, 250=0.84%, 500=0.19%, 750=0.06%, 1000=0.04%
    lat (msec) : 2000=0.09%, >=2000=0.02%
  cpu          : usr=0.42%, sys=0.92%, ctx=25002544, majf=0, minf=47139
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=8803420/w=4330079/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-01: (groupid=2, jobs=5): err= 0: pid=116596: Thu Jan 14 01:09:12 2016
  read : io=464294MB, bw=66033KB/s, iops=1031, runt=7200004msec
    slat (usec): min=6, max=3993, avg=42.41, stdev=29.67
    clat (usec): min=1, max=4150.2K, avg=3988.06, stdev=36092.40
     lat (usec): min=202, max=4150.3K, avg=4030.83, stdev=36092.44
    clat percentiles (usec):
     |  1.00th=[  564],  5.00th=[  716], 10.00th=[  820], 20.00th=[ 1004],
     | 30.00th=[ 1256], 40.00th=[ 1848], 50.00th=[ 2544], 60.00th=[ 3056],
     | 70.00th=[ 3600], 80.00th=[ 4320], 90.00th=[ 5088], 95.00th=[ 5856],
     | 99.00th=[ 8384], 99.50th=[10944], 99.90th=[276480], 99.95th=[806912],
     | 99.99th=[1810432]
    bw (KB  /s): min=    1, max=69248, per=23.51%, avg=15527.40, stdev=10262.01
  write: io=228653MB, bw=32519KB/s, iops=508, runt=7200004msec
    slat (usec): min=8, max=967, avg=67.14, stdev=39.23
    clat (usec): min=1, max=4160.7K, avg=11411.55, stdev=66919.38
     lat (usec): min=236, max=4160.8K, avg=11479.09, stdev=66919.19
    clat percentiles (usec):
     |  1.00th=[  442],  5.00th=[  748], 10.00th=[  988], 20.00th=[ 1608],
     | 30.00th=[ 2448], 40.00th=[ 4128], 50.00th=[ 6560], 60.00th=[ 8896],
     | 70.00th=[10816], 80.00th=[12736], 90.00th=[15296], 95.00th=[17280],
     | 99.00th=[87552], 99.50th=[166912], 99.90th=[1187840], 99.95th=[1581056],
     | 99.99th=[2506752]
    bw (KB  /s): min=    1, max=37632, per=23.53%, avg=7651.00, stdev=5047.09
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.02%, 500=0.84%, 750=5.12%, 1000=10.69%
    lat (msec) : 2=19.68%, 4=27.59%, 10=24.35%, 20=10.69%, 50=0.39%
    lat (msec) : 100=0.10%, 250=0.33%, 500=0.06%, 750=0.03%, 1000=0.03%
    lat (msec) : 2000=0.06%, >=2000=0.01%
  cpu          : usr=0.40%, sys=0.84%, ctx=21499193, majf=0, minf=28724
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=7428700/w=3658441/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=5): err= 0: pid=116601: Thu Jan 14 01:09:12 2016
  read : io=492674MB, bw=70069KB/s, iops=1094, runt=7200031msec
    slat (usec): min=5, max=5269, avg=39.25, stdev=27.94
    clat (usec): min=1, max=4151.2K, avg=3789.04, stdev=35269.29
     lat (usec): min=210, max=4151.2K, avg=3828.60, stdev=35269.31
    clat percentiles (usec):
     |  1.00th=[  564],  5.00th=[  724], 10.00th=[  828], 20.00th=[ 1020],
     | 30.00th=[ 1256], 40.00th=[ 1736], 50.00th=[ 2384], 60.00th=[ 2864],
     | 70.00th=[ 3344], 80.00th=[ 3984], 90.00th=[ 4768], 95.00th=[ 5408],
     | 99.00th=[ 7904], 99.50th=[10304], 99.90th=[252928], 99.95th=[765952],
     | 99.99th=[1761280]
    bw (KB  /s): min=    1, max=68992, per=23.50%, avg=16468.29, stdev=10252.71
  write: io=242452MB, bw=34482KB/s, iops=538, runt=7200031msec
    slat (usec): min=9, max=1444, avg=62.34, stdev=36.36
    clat (usec): min=1, max=4154.1K, avg=10703.01, stdev=64677.43
     lat (usec): min=221, max=4155.4K, avg=10765.68, stdev=64677.33
    clat percentiles (usec):
     |  1.00th=[  442],  5.00th=[  756], 10.00th=[ 1012], 20.00th=[ 1656],
     | 30.00th=[ 2480], 40.00th=[ 3920], 50.00th=[ 6112], 60.00th=[ 8096],
     | 70.00th=[ 9920], 80.00th=[11712], 90.00th=[14016], 95.00th=[15936],
     | 99.00th=[80384], 99.50th=[146432], 99.90th=[1122304], 99.95th=[1531904],
     | 99.99th=[2408448]
    bw (KB  /s): min=    1, max=35712, per=23.50%, avg=8103.18, stdev=5044.35
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.02%, 500=0.84%, 750=4.78%, 1000=10.38%
    lat (msec) : 2=21.45%, 4=29.65%, 10=22.75%, 20=9.31%, 50=0.22%
    lat (msec) : 100=0.09%, 250=0.33%, 500=0.06%, 750=0.03%, 1000=0.03%
    lat (msec) : 2000=0.05%, >=2000=0.01%
  cpu          : usr=0.39%, sys=0.86%, ctx=22848422, majf=0, minf=27615
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=7882786/w=3879231/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=561928MB, aggrb=79918KB/s, minb=79918KB/s, maxb=79918KB/s, mint=7200024msec, maxt=7200024msec
  WRITE: io=276423MB, aggrb=39313KB/s, minb=39313KB/s, maxb=39313KB/s, mint=7200024msec, maxt=7200024msec

Run status group 1 (all jobs):
   READ: io=550214MB, aggrb=78252KB/s, minb=78252KB/s, maxb=78252KB/s, mint=7200032msec, maxt=7200032msec
  WRITE: io=270631MB, aggrb=38489KB/s, minb=38489KB/s, maxb=38489KB/s, mint=7200032msec, maxt=7200032msec

Run status group 2 (all jobs):
   READ: io=464294MB, aggrb=66032KB/s, minb=66032KB/s, maxb=66032KB/s, mint=7200004msec, maxt=7200004msec
  WRITE: io=228653MB, aggrb=32519KB/s, minb=32519KB/s, maxb=32519KB/s, mint=7200004msec, maxt=7200004msec

Run status group 3 (all jobs):
   READ: io=492674MB, aggrb=70068KB/s, minb=70068KB/s, maxb=70068KB/s, mint=7200031msec, maxt=7200031msec
  WRITE: io=242452MB, aggrb=34481KB/s, minb=34481KB/s, maxb=34481KB/s, mint=7200031msec, maxt=7200031msec



# 
# Thu 14 Jan 2016 01:09:14 AM EST
# 1mB write with SATA 1 jobs and 2 I/Os Outstanding
# 1mB write with NVMe 1 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 302 IOPS | 309 MBps | 6610 μsec Average Latency 
SATA-R5-1: 284 IOPS | 291 MBps | 7020 μsec Average Latency 
NVMe-R5-0: 134 IOPS | 137 MBps | 59630 μsec Average Latency 
NVMe-R5-1: 131 IOPS | 134 MBps | 60700 μsec Average Latency 

Totals
IOPS: 851
MBPS: 871
Average Latency (μsec): 33490

SATA-R5-01: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=2
SATA-R5-11: (g=1): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=2
NVMe-R5-01: (g=2): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
NVMe-R5-11: (g=3): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=122563: Thu Jan 14 03:10:15 2016
  write: io=2128.3GB, bw=309944KB/s, iops=302, runt=7200003msec
    slat (usec): min=102, max=1589, avg=210.85, stdev=47.04
    clat (msec): min=1, max=753, avg= 6.39, stdev=16.58
     lat (msec): min=1, max=753, avg= 6.61, stdev=16.58
    clat percentiles (usec):
     |  1.00th=[ 1384],  5.00th=[ 1640], 10.00th=[ 1832], 20.00th=[ 1912],
     | 30.00th=[ 1976], 40.00th=[ 2040], 50.00th=[ 2096], 60.00th=[ 2192],
     | 70.00th=[ 2384], 80.00th=[ 3024], 90.00th=[ 7520], 95.00th=[41728],
     | 99.00th=[79360], 99.50th=[103936], 99.90th=[164864], 99.95th=[207872],
     | 99.99th=[309248]
    bw (KB  /s): min=   17, max=784862, per=100.00%, avg=311619.46, stdev=97802.13
    lat (msec) : 2=33.39%, 4=51.04%, 10=8.20%, 20=0.85%, 50=2.88%
    lat (msec) : 100=3.08%, 250=0.52%, 500=0.03%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.37%, sys=2.91%, ctx=10404730, majf=0, minf=63728
  IO depths    : 1=0.1%, 2=100.6%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=2179291/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=122564: Thu Jan 14 03:10:15 2016
  write: io=2001.4GB, bw=291472KB/s, iops=284, runt=7200001msec
    slat (usec): min=102, max=3530, avg=267.52, stdev=65.25
    clat (msec): min=1, max=783, avg= 6.75, stdev=17.49
     lat (msec): min=1, max=783, avg= 7.02, stdev=17.49
    clat percentiles (usec):
     |  1.00th=[ 1432],  5.00th=[ 1704], 10.00th=[ 1816], 20.00th=[ 1912],
     | 30.00th=[ 1976], 40.00th=[ 2040], 50.00th=[ 2096], 60.00th=[ 2160],
     | 70.00th=[ 2352], 80.00th=[ 2992], 90.00th=[ 7648], 95.00th=[44800],
     | 99.00th=[83456], 99.50th=[107008], 99.90th=[173056], 99.95th=[207872],
     | 99.99th=[313344]
    bw (KB  /s): min=   17, max=793549, per=100.00%, avg=293172.42, stdev=97512.70
    lat (msec) : 2=33.70%, 4=50.42%, 10=8.05%, 20=0.86%, 50=2.82%
    lat (msec) : 100=3.55%, 250=0.58%, 500=0.03%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.47%, sys=3.18%, ctx=9634628, majf=0, minf=66327
  IO depths    : 1=0.1%, 2=100.7%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=2049410/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=122565: Thu Jan 14 03:10:15 2016
  write: io=965909MB, bw=137370KB/s, iops=134, runt=7200208msec
    slat (usec): min=105, max=1351, avg=334.09, stdev=90.29
    clat (msec): min=1, max=1109, avg=59.30, stdev=74.60
     lat (msec): min=1, max=1109, avg=59.63, stdev=74.59
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   13], 60.00th=[   59],
     | 70.00th=[   79], 80.00th=[  113], 90.00th=[  161], 95.00th=[  204],
     | 99.00th=[  306], 99.50th=[  396], 99.90th=[  578], 99.95th=[  627],
     | 99.99th=[  848]
    bw (KB  /s): min=   17, max=567296, per=100.00%, avg=138762.19, stdev=52664.81
    lat (msec) : 2=0.49%, 4=7.26%, 10=40.01%, 20=4.90%, 50=4.19%
    lat (msec) : 100=20.35%, 250=20.72%, 500=1.93%, 750=0.12%, 1000=0.02%
    lat (msec) : 2000=0.01%
  cpu          : usr=1.02%, sys=1.73%, ctx=4404341, majf=0, minf=64208
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=965902/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=122566: Thu Jan 14 03:10:15 2016
  write: io=948937MB, bw=134959KB/s, iops=131, runt=7200036msec
    slat (usec): min=102, max=1340, avg=319.47, stdev=92.99
    clat (msec): min=1, max=1208, avg=60.38, stdev=74.90
     lat (msec): min=1, max=1208, avg=60.70, stdev=74.90
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   13], 60.00th=[   61],
     | 70.00th=[   82], 80.00th=[  116], 90.00th=[  163], 95.00th=[  206],
     | 99.00th=[  306], 99.50th=[  379], 99.90th=[  537], 99.95th=[  611],
     | 99.99th=[  807]
    bw (KB  /s): min=   17, max=598943, per=100.00%, avg=136160.98, stdev=50504.83
    lat (msec) : 2=0.48%, 4=7.82%, 10=38.91%, 20=5.01%, 50=4.11%
    lat (msec) : 100=20.18%, 250=21.32%, 500=2.03%, 750=0.11%, 1000=0.02%
    lat (msec) : 2000=0.01%
  cpu          : usr=1.01%, sys=1.70%, ctx=4339319, majf=0, minf=66838
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.6%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=948930/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: io=2128.3GB, aggrb=309943KB/s, minb=309943KB/s, maxb=309943KB/s, mint=7200003msec, maxt=7200003msec

Run status group 1 (all jobs):
  WRITE: io=2001.4GB, aggrb=291471KB/s, minb=291471KB/s, maxb=291471KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
  WRITE: io=965909MB, aggrb=137369KB/s, minb=137369KB/s, maxb=137369KB/s, mint=7200208msec, maxt=7200208msec

Run status group 3 (all jobs):
  WRITE: io=948937MB, aggrb=134959KB/s, minb=134959KB/s, maxb=134959KB/s, mint=7200036msec, maxt=7200036msec



# 
# Thu 14 Jan 2016 03:10:18 AM EST
# 1mB read with SATA 1 jobs and 8 I/Os Outstanding
# 1mB read with NVMe 1 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 745 IOPS | 763 MBps | 10730 μsec Average Latency 
SATA-R5-1: 743 IOPS | 761 MBps | 10750 μsec Average Latency 
NVMe-R5-0: 360 IOPS | 369 MBps | 5540 μsec Average Latency 
NVMe-R5-1: 360 IOPS | 368 MBps | 5550 μsec Average Latency 

Totals
IOPS: 2208
MBPS: 2261
Average Latency (μsec): 8142

SATA-R5-01: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
SATA-R5-11: (g=1): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
NVMe-R5-01: (g=2): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=2
NVMe-R5-11: (g=3): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=2
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=127720: Thu Jan 14 05:11:18 2016
  read : io=5242.4GB, bw=763466KB/s, iops=745, runt=7200010msec
    slat (usec): min=68, max=10947, avg=217.55, stdev=132.77
    clat (msec): min=1, max=37, avg=10.51, stdev= 2.44
     lat (msec): min=1, max=37, avg=10.73, stdev= 2.43
    clat percentiles (usec):
     |  1.00th=[ 5344],  5.00th=[ 7200], 10.00th=[ 8160], 20.00th=[ 9024],
     | 30.00th=[ 9408], 40.00th=[ 9792], 50.00th=[10048], 60.00th=[10432],
     | 70.00th=[10944], 80.00th=[11968], 90.00th=[13888], 95.00th=[15552],
     | 99.00th=[18048], 99.50th=[19072], 99.90th=[21888], 99.95th=[23168],
     | 99.99th=[26496]
    bw (KB  /s): min=   17, max=856399, per=100.00%, avg=764106.87, stdev=30719.81
    lat (msec) : 2=0.01%, 4=0.08%, 10=49.15%, 20=50.49%, 50=0.28%
  cpu          : usr=0.37%, sys=9.48%, ctx=26467661, majf=0, minf=69329
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=5368118/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=127721: Thu Jan 14 05:11:18 2016
  read : io=5230.1GB, bw=761804KB/s, iops=743, runt=7200008msec
    slat (usec): min=73, max=10466, avg=222.55, stdev=143.85
    clat (usec): min=488, max=36304, avg=10527.61, stdev=2455.33
     lat (msec): min=1, max=36, avg=10.75, stdev= 2.45
    clat percentiles (usec):
     |  1.00th=[ 5408],  5.00th=[ 7200], 10.00th=[ 8160], 20.00th=[ 8896],
     | 30.00th=[ 9408], 40.00th=[ 9792], 50.00th=[10048], 60.00th=[10432],
     | 70.00th=[10944], 80.00th=[11968], 90.00th=[14016], 95.00th=[15552],
     | 99.00th=[18048], 99.50th=[19072], 99.90th=[21888], 99.95th=[23168],
     | 99.99th=[26240]
    bw (KB  /s): min=   17, max=845824, per=100.00%, avg=762440.95, stdev=30929.19
    lat (usec) : 500=0.01%
    lat (msec) : 2=0.01%, 4=0.08%, 10=49.09%, 20=50.54%, 50=0.28%
  cpu          : usr=0.36%, sys=9.42%, ctx=26506459, majf=0, minf=68804
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=5356436/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=127722: Thu Jan 14 05:11:18 2016
  read : io=2534.2GB, bw=369060KB/s, iops=360, runt=7200007msec
    slat (usec): min=81, max=5556, avg=254.28, stdev=104.60
    clat (usec): min=559, max=32718, avg=5289.25, stdev=2329.17
     lat (msec): min=1, max=32, avg= 5.54, stdev= 2.33
    clat percentiles (usec):
     |  1.00th=[ 1496],  5.00th=[ 2192], 10.00th=[ 3024], 20.00th=[ 3824],
     | 30.00th=[ 4256], 40.00th=[ 4512], 50.00th=[ 4768], 60.00th=[ 5088],
     | 70.00th=[ 5472], 80.00th=[ 6240], 90.00th=[ 8768], 95.00th=[10560],
     | 99.00th=[12864], 99.50th=[13760], 99.90th=[16064], 99.95th=[17024],
     | 99.99th=[19840]
    bw (KB  /s): min=   17, max=445572, per=100.00%, avg=369405.40, stdev=26119.34
    lat (usec) : 750=0.01%, 1000=0.01%
    lat (msec) : 2=3.90%, 4=19.57%, 10=70.15%, 20=6.37%, 50=0.01%
  cpu          : usr=0.31%, sys=4.31%, ctx=13002535, majf=0, minf=66213
  IO depths    : 1=0.1%, 2=100.7%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2594957/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=127723: Thu Jan 14 05:11:18 2016
  read : io=2532.3GB, bw=368779KB/s, iops=360, runt=7200007msec
    slat (usec): min=81, max=6456, avg=256.22, stdev=106.98
    clat (usec): min=352, max=30179, avg=5291.75, stdev=2334.19
     lat (msec): min=1, max=30, avg= 5.55, stdev= 2.34
    clat percentiles (usec):
     |  1.00th=[ 1528],  5.00th=[ 2224], 10.00th=[ 3024], 20.00th=[ 3824],
     | 30.00th=[ 4256], 40.00th=[ 4512], 50.00th=[ 4768], 60.00th=[ 5088],
     | 70.00th=[ 5472], 80.00th=[ 6304], 90.00th=[ 8768], 95.00th=[10560],
     | 99.00th=[12992], 99.50th=[13760], 99.90th=[16064], 99.95th=[17024],
     | 99.99th=[19840]
    bw (KB  /s): min=   17, max=446464, per=100.00%, avg=369130.88, stdev=26370.88
    lat (usec) : 500=0.01%, 1000=0.01%
    lat (msec) : 2=3.66%, 4=20.22%, 10=69.68%, 20=6.43%, 50=0.01%
  cpu          : usr=0.30%, sys=4.28%, ctx=12994155, majf=0, minf=67904
  IO depths    : 1=0.1%, 2=100.7%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2592976/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=5242.4GB, aggrb=763465KB/s, minb=763465KB/s, maxb=763465KB/s, mint=7200010msec, maxt=7200010msec

Run status group 1 (all jobs):
   READ: io=5230.1GB, aggrb=761804KB/s, minb=761804KB/s, maxb=761804KB/s, mint=7200008msec, maxt=7200008msec

Run status group 2 (all jobs):
   READ: io=2534.2GB, aggrb=369060KB/s, minb=369060KB/s, maxb=369060KB/s, mint=7200007msec, maxt=7200007msec

Run status group 3 (all jobs):
   READ: io=2532.3GB, aggrb=368778KB/s, minb=368778KB/s, maxb=368778KB/s, mint=7200007msec, maxt=7200007msec



# 
# Thu 14 Jan 2016 05:11:21 AM EST
# 256kB write with SATA 1 jobs and 8 I/Os Outstanding
# 256kB write with NVMe 1 jobs and 1 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 1485 IOPS | 380 MBps | 5382 μsec Average Latency 
SATA-R5-1: 1470 IOPS | 376 MBps | 5437 μsec Average Latency 
NVMe-R5-0: 511 IOPS | 131 MBps | 1948 μsec Average Latency 
NVMe-R5-1: 505 IOPS | 129 MBps | 1973 μsec Average Latency 

Totals
IOPS: 3971
MBPS: 1016
Average Latency (μsec): 3685

SATA-R5-01: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=8
SATA-R5-11: (g=1): rw=write, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=8
NVMe-R5-01: (g=2): rw=write, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=1
NVMe-R5-11: (g=3): rw=write, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=132885: Thu Jan 14 07:12:22 2016
  write: io=2611.2GB, bw=380269KB/s, iops=1485, runt=7200043msec
    slat (usec): min=26, max=2170, avg=82.03, stdev=26.95
    clat (usec): min=370, max=471594, avg=5300.59, stdev=13680.74
     lat (usec): min=546, max=471659, avg=5382.86, stdev=13680.26
    clat percentiles (usec):
     |  1.00th=[  756],  5.00th=[  996], 10.00th=[ 1192], 20.00th=[ 1448],
     | 30.00th=[ 1576], 40.00th=[ 1672], 50.00th=[ 1752], 60.00th=[ 1816],
     | 70.00th=[ 1896], 80.00th=[ 2064], 90.00th=[ 3760], 95.00th=[37120],
     | 99.00th=[65280], 99.50th=[89600], 99.90th=[126464], 99.95th=[138240],
     | 99.99th=[175104]
    bw (KB  /s): min=    4, max=833024, per=100.00%, avg=381140.14, stdev=84968.77
    lat (usec) : 500=0.01%, 750=0.91%, 1000=4.13%
    lat (msec) : 2=72.65%, 4=12.64%, 10=1.94%, 20=0.25%, 50=5.20%
    lat (msec) : 100=1.94%, 250=0.34%, 500=0.01%
  cpu          : usr=2.97%, sys=5.47%, ctx=18941480, majf=0, minf=67370
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10695125/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=132886: Thu Jan 14 07:12:22 2016
  write: io=2585.2GB, bw=376490KB/s, iops=1470, runt=7200043msec
    slat (usec): min=26, max=491, avg=54.62, stdev=14.71
    clat (usec): min=402, max=543689, avg=5383.14, stdev=14038.42
     lat (usec): min=532, max=543758, avg=5437.97, stdev=14038.01
    clat percentiles (usec):
     |  1.00th=[  748],  5.00th=[ 1004], 10.00th=[ 1208], 20.00th=[ 1480],
     | 30.00th=[ 1592], 40.00th=[ 1704], 50.00th=[ 1784], 60.00th=[ 1832],
     | 70.00th=[ 1928], 80.00th=[ 2064], 90.00th=[ 3632], 95.00th=[39168],
     | 99.00th=[66048], 99.50th=[88576], 99.90th=[129536], 99.95th=[142336],
     | 99.99th=[177152]
    bw (KB  /s): min=    4, max=768495, per=100.00%, avg=377370.29, stdev=85378.75
    lat (usec) : 500=0.01%, 750=1.01%, 1000=3.96%
    lat (msec) : 2=72.57%, 4=13.07%, 10=1.98%, 20=0.23%, 50=4.53%
    lat (msec) : 100=2.32%, 250=0.34%, 500=0.01%, 750=0.01%
  cpu          : usr=1.89%, sys=4.39%, ctx=20019974, majf=0, minf=66849
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10588841/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=132887: Thu Jan 14 07:12:22 2016
  write: io=921373MB, bw=131039KB/s, iops=511, runt=7200044msec
    slat (usec): min=32, max=701, avg=102.47, stdev=21.02
    clat (usec): min=363, max=549419, avg=1845.17, stdev=7905.05
     lat (usec): min=504, max=549530, avg=1948.03, stdev=7904.18
    clat percentiles (usec):
     |  1.00th=[  506],  5.00th=[  532], 10.00th=[  540], 20.00th=[  556],
     | 30.00th=[  580], 40.00th=[  588], 50.00th=[  612], 60.00th=[  644],
     | 70.00th=[  764], 80.00th=[ 1528], 90.00th=[ 1848], 95.00th=[ 2040],
     | 99.00th=[45824], 99.50th=[63744], 99.90th=[101888], 99.95th=[115200],
     | 99.99th=[148480]
    bw (KB  /s): min=    4, max=284672, per=100.00%, avg=131421.12, stdev=33160.89
    lat (usec) : 500=0.45%, 750=69.05%, 1000=4.14%
    lat (msec) : 2=21.01%, 4=3.30%, 10=0.32%, 20=0.06%, 50=0.80%
    lat (msec) : 100=0.76%, 250=0.11%, 500=0.01%, 750=0.01%
  cpu          : usr=1.26%, sys=2.30%, ctx=7425473, majf=0, minf=9307
  IO depths    : 1=100.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=3685492/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=132888: Thu Jan 14 07:12:22 2016
  write: io=909492MB, bw=129349KB/s, iops=505, runt=7200044msec
    slat (usec): min=29, max=671, avg=103.62, stdev=19.59
    clat (usec): min=297, max=352189, avg=1869.67, stdev=8000.52
     lat (usec): min=504, max=352297, avg=1973.72, stdev=7999.94
    clat percentiles (usec):
     |  1.00th=[  510],  5.00th=[  532], 10.00th=[  548], 20.00th=[  564],
     | 30.00th=[  580], 40.00th=[  596], 50.00th=[  620], 60.00th=[  652],
     | 70.00th=[  748], 80.00th=[ 1528], 90.00th=[ 1848], 95.00th=[ 2024],
     | 99.00th=[47360], 99.50th=[65280], 99.90th=[100864], 99.95th=[114176],
     | 99.99th=[146432]
    bw (KB  /s): min=    4, max=281600, per=100.00%, avg=129715.61, stdev=33945.10
    lat (usec) : 500=0.34%, 750=69.75%, 1000=4.16%
    lat (msec) : 2=20.40%, 4=3.25%, 10=0.32%, 20=0.06%, 50=0.80%
    lat (msec) : 100=0.81%, 250=0.10%, 500=0.01%
  cpu          : usr=1.18%, sys=2.28%, ctx=7331824, majf=0, minf=9434
  IO depths    : 1=100.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=3637967/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: io=2611.2GB, aggrb=380269KB/s, minb=380269KB/s, maxb=380269KB/s, mint=7200043msec, maxt=7200043msec

Run status group 1 (all jobs):
  WRITE: io=2585.2GB, aggrb=376490KB/s, minb=376490KB/s, maxb=376490KB/s, mint=7200043msec, maxt=7200043msec

Run status group 2 (all jobs):
  WRITE: io=921373MB, aggrb=131038KB/s, minb=131038KB/s, maxb=131038KB/s, mint=7200044msec, maxt=7200044msec

Run status group 3 (all jobs):
  WRITE: io=909492MB, aggrb=129349KB/s, minb=129349KB/s, maxb=129349KB/s, mint=7200044msec, maxt=7200044msec



# 
# Thu 14 Jan 2016 07:12:24 AM EST
# 256kB read with SATA 1 jobs and 8 I/Os Outstanding
# 256kB read with NVMe 1 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 2390 IOPS | 612 MBps | 3343 μsec Average Latency 
SATA-R5-1: 2406 IOPS | 616 MBps | 3322 μsec Average Latency 
NVMe-R5-0: 2154 IOPS | 551 MBps | 1853 μsec Average Latency 
NVMe-R5-1: 2133 IOPS | 546 MBps | 1872 μsec Average Latency 

Totals
IOPS: 9083
MBPS: 2325
Average Latency (μsec): 2597

SATA-R5-01: (g=0): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=8
SATA-R5-11: (g=1): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=8
NVMe-R5-01: (g=2): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=4
NVMe-R5-11: (g=3): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=4
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=138028: Thu Jan 14 09:13:25 2016
  read : io=4202.9GB, bw=612088KB/s, iops=2390, runt=7200004msec
    slat (usec): min=17, max=3132, avg=66.27, stdev=43.48
    clat (usec): min=1, max=23098, avg=3277.14, stdev=901.82
     lat (usec): min=923, max=23208, avg=3343.64, stdev=900.22
    clat percentiles (usec):
     |  1.00th=[ 2008],  5.00th=[ 2416], 10.00th=[ 2576], 20.00th=[ 2768],
     | 30.00th=[ 2896], 40.00th=[ 2992], 50.00th=[ 3088], 60.00th=[ 3216],
     | 70.00th=[ 3344], 80.00th=[ 3568], 90.00th=[ 4048], 95.00th=[ 4832],
     | 99.00th=[ 7136], 99.50th=[ 8096], 99.90th=[10560], 99.95th=[11840],
     | 99.99th=[14784]
    bw (KB  /s): min=    4, max=664576, per=100.00%, avg=612402.70, stdev=16255.36
    lat (usec) : 2=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.94%, 4=88.62%, 10=10.29%, 20=0.14%, 50=0.01%
  cpu          : usr=0.97%, sys=8.97%, ctx=32081174, majf=0, minf=68539
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=17214964/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=138029: Thu Jan 14 09:13:25 2016
  read : io=4229.1GB, bw=616022KB/s, iops=2406, runt=7200004msec
    slat (usec): min=18, max=3151, avg=44.13, stdev=23.19
    clat (usec): min=191, max=23719, avg=3278.49, stdev=919.40
     lat (usec): min=770, max=23749, avg=3322.75, stdev=918.78
    clat percentiles (usec):
     |  1.00th=[ 1960],  5.00th=[ 2384], 10.00th=[ 2576], 20.00th=[ 2768],
     | 30.00th=[ 2864], 40.00th=[ 2992], 50.00th=[ 3088], 60.00th=[ 3216],
     | 70.00th=[ 3376], 80.00th=[ 3568], 90.00th=[ 4080], 95.00th=[ 4896],
     | 99.00th=[ 7200], 99.50th=[ 8160], 99.90th=[10560], 99.95th=[11840],
     | 99.99th=[14784]
    bw (KB  /s): min=    4, max=664576, per=100.00%, avg=616341.28, stdev=16078.26
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=1.14%, 4=87.99%, 10=10.72%, 20=0.14%, 50=0.01%
  cpu          : usr=0.96%, sys=8.00%, ctx=33102565, majf=0, minf=67492
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=17325633/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=138030: Thu Jan 14 09:13:25 2016
  read : io=3787.7GB, bw=551612KB/s, iops=2154, runt=7200002msec
    slat (usec): min=17, max=2519, avg=66.87, stdev=44.70
    clat (usec): min=0, max=21321, avg=1786.87, stdev=669.89
     lat (usec): min=371, max=21362, avg=1853.99, stdev=668.11
    clat percentiles (usec):
     |  1.00th=[  756],  5.00th=[ 1064], 10.00th=[ 1224], 20.00th=[ 1400],
     | 30.00th=[ 1512], 40.00th=[ 1608], 50.00th=[ 1704], 60.00th=[ 1800],
     | 70.00th=[ 1928], 80.00th=[ 2064], 90.00th=[ 2320], 95.00th=[ 2608],
     | 99.00th=[ 4640], 99.50th=[ 5856], 99.90th=[ 8256], 99.95th=[ 9408],
     | 99.99th=[12096]
    bw (KB  /s): min=    4, max=608256, per=100.00%, avg=551917.03, stdev=16243.80
    lat (usec) : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=0.11%, 750=0.82%, 1000=2.75%
    lat (msec) : 2=71.98%, 4=23.04%, 10=1.26%, 20=0.04%, 50=0.01%
  cpu          : usr=0.92%, sys=7.99%, ctx=29257554, majf=0, minf=34639
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=15514082/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=138031: Thu Jan 14 09:13:25 2016
  read : io=3750.2GB, bw=546154KB/s, iops=2133, runt=7200002msec
    slat (usec): min=17, max=2648, avg=70.56, stdev=51.85
    clat (usec): min=0, max=25522, avg=1801.76, stdev=686.17
     lat (usec): min=376, max=25584, avg=1872.60, stdev=684.01
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[ 1048], 10.00th=[ 1208], 20.00th=[ 1384],
     | 30.00th=[ 1512], 40.00th=[ 1608], 50.00th=[ 1720], 60.00th=[ 1816],
     | 70.00th=[ 1944], 80.00th=[ 2096], 90.00th=[ 2352], 95.00th=[ 2672],
     | 99.00th=[ 4704], 99.50th=[ 5920], 99.90th=[ 8256], 99.95th=[ 9408],
     | 99.99th=[12224]
    bw (KB  /s): min=    4, max=613888, per=100.00%, avg=546453.26, stdev=16256.53
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.14%, 750=0.90%, 1000=2.87%
    lat (msec) : 2=69.48%, 4=25.27%, 10=1.29%, 20=0.04%, 50=0.01%
  cpu          : usr=0.88%, sys=8.12%, ctx=29017883, majf=0, minf=37168
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=15360573/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=4202.9GB, aggrb=612087KB/s, minb=612087KB/s, maxb=612087KB/s, mint=7200004msec, maxt=7200004msec

Run status group 1 (all jobs):
   READ: io=4229.1GB, aggrb=616022KB/s, minb=616022KB/s, maxb=616022KB/s, mint=7200004msec, maxt=7200004msec

Run status group 2 (all jobs):
   READ: io=3787.7GB, aggrb=551611KB/s, minb=551611KB/s, maxb=551611KB/s, mint=7200002msec, maxt=7200002msec

Run status group 3 (all jobs):
   READ: io=3750.2GB, aggrb=546153KB/s, minb=546153KB/s, maxb=546153KB/s, mint=7200002msec, maxt=7200002msec



# 
# Thu 14 Jan 2016 09:13:27 AM EST
# 64kB write with SATA 1 jobs and 4 I/Os Outstanding
# 64kB write with NVMe 1 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 5566 IOPS | 356 MBps | 717 μsec Average Latency 
SATA-R5-1: 6159 IOPS | 394 MBps | 647 μsec Average Latency 
NVMe-R5-0: 1916 IOPS | 122 MBps | 2085 μsec Average Latency 
NVMe-R5-1: 1851 IOPS | 118 MBps | 2158 μsec Average Latency 

Totals
IOPS: 15492
MBPS: 990
Average Latency (μsec): 1401

SATA-R5-01: (g=0): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
SATA-R5-11: (g=1): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
NVMe-R5-01: (g=2): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
NVMe-R5-11: (g=3): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=143153: Thu Jan 14 11:14:28 2016
  write: io=2446.3GB, bw=356260KB/s, iops=5566, runt=7200001msec
    slat (usec): min=8, max=1363, avg=28.47, stdev= 9.49
    clat (usec): min=79, max=399720, avg=688.48, stdev=4547.25
     lat (usec): min=202, max=399746, avg=717.16, stdev=4547.22
    clat percentiles (usec):
     |  1.00th=[  282],  5.00th=[  322], 10.00th=[  338], 20.00th=[  350],
     | 30.00th=[  378], 40.00th=[  410], 50.00th=[  438], 60.00th=[  466],
     | 70.00th=[  478], 80.00th=[  506], 90.00th=[  580], 95.00th=[  668],
     | 99.00th=[ 1464], 99.50th=[ 5472], 99.90th=[77312], 99.95th=[111104],
     | 99.99th=[199680]
    bw (KB  /s): min=    1, max=564096, per=100.00%, avg=357986.55, stdev=88670.64
    lat (usec) : 100=0.01%, 250=0.29%, 500=78.23%, 750=18.17%, 1000=1.61%
    lat (msec) : 2=0.97%, 4=0.20%, 10=0.09%, 20=0.05%, 50=0.25%
    lat (msec) : 100=0.08%, 250=0.07%, 500=0.01%
  cpu          : usr=2.49%, sys=6.79%, ctx=68233908, majf=0, minf=9525
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=40079269/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=143154: Thu Jan 14 11:14:28 2016
  write: io=2706.7GB, bw=394187KB/s, iops=6159, runt=7200001msec
    slat (usec): min=9, max=1430, avg=39.68, stdev=18.52
    clat (usec): min=94, max=368212, avg=607.83, stdev=3390.79
     lat (usec): min=207, max=368241, avg=647.71, stdev=3390.68
    clat percentiles (usec):
     |  1.00th=[  282],  5.00th=[  314], 10.00th=[  330], 20.00th=[  354],
     | 30.00th=[  386], 40.00th=[  418], 50.00th=[  438], 60.00th=[  462],
     | 70.00th=[  478], 80.00th=[  516], 90.00th=[  588], 95.00th=[  676],
     | 99.00th=[ 1320], 99.50th=[ 2352], 99.90th=[43776], 99.95th=[89600],
     | 99.99th=[132096]
    bw (KB  /s): min=    1, max=550656, per=100.00%, avg=396325.11, stdev=115605.34
    lat (usec) : 100=0.01%, 250=0.24%, 500=76.80%, 750=19.68%, 1000=1.70%
    lat (msec) : 2=1.00%, 4=0.22%, 10=0.08%, 20=0.03%, 50=0.17%
    lat (msec) : 100=0.05%, 250=0.04%, 500=0.01%
  cpu          : usr=3.27%, sys=10.60%, ctx=73509867, majf=0, minf=9378
  IO depths    : 1=0.1%, 2=0.1%, 4=100.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=44346010/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=143155: Thu Jan 14 11:14:28 2016
  write: io=862595MB, bw=122679KB/s, iops=1916, runt=7200081msec
    slat (usec): min=8, max=414, avg=35.87, stdev=22.23
    clat (usec): min=107, max=514635, avg=2048.95, stdev=13495.86
     lat (usec): min=207, max=514701, avg=2085.05, stdev=13495.79
    clat percentiles (usec):
     |  1.00th=[  294],  5.00th=[  334], 10.00th=[  358], 20.00th=[  406],
     | 30.00th=[  438], 40.00th=[  462], 50.00th=[  482], 60.00th=[  506],
     | 70.00th=[  548], 80.00th=[  596], 90.00th=[  692], 95.00th=[  876],
     | 99.00th=[55552], 99.50th=[109056], 99.90th=[199680], 99.95th=[199680],
     | 99.99th=[226304]
    bw (KB  /s): min=    1, max=438912, per=100.00%, avg=123666.44, stdev=51307.83
    lat (usec) : 250=0.15%, 500=58.29%, 750=34.21%, 1000=3.45%
    lat (msec) : 2=1.61%, 4=0.31%, 10=0.13%, 20=0.05%, 50=0.67%
    lat (msec) : 100=0.55%, 250=0.56%, 500=0.01%, 750=0.01%
  cpu          : usr=1.02%, sys=3.05%, ctx=22898072, majf=0, minf=9309
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=13801512/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=143156: Thu Jan 14 11:14:28 2016
  write: io=833304MB, bw=118514KB/s, iops=1851, runt=7200001msec
    slat (usec): min=8, max=1023, avg=37.80, stdev=23.65
    clat (usec): min=48, max=410329, avg=2120.33, stdev=13803.52
     lat (usec): min=211, max=410378, avg=2158.34, stdev=13803.42
    clat percentiles (usec):
     |  1.00th=[  286],  5.00th=[  334], 10.00th=[  366], 20.00th=[  410],
     | 30.00th=[  446], 40.00th=[  470], 50.00th=[  490], 60.00th=[  516],
     | 70.00th=[  564], 80.00th=[  612], 90.00th=[  708], 95.00th=[  884],
     | 99.00th=[58112], 99.50th=[113152], 99.90th=[199680], 99.95th=[199680],
     | 99.99th=[224256]
    bw (KB  /s): min=    1, max=438400, per=100.00%, avg=119469.48, stdev=49837.74
    lat (usec) : 50=0.01%, 250=0.29%, 500=53.52%, 750=38.52%, 1000=3.68%
    lat (msec) : 2=1.64%, 4=0.32%, 10=0.13%, 20=0.05%, 50=0.65%
    lat (msec) : 100=0.59%, 250=0.59%, 500=0.01%
  cpu          : usr=1.00%, sys=3.09%, ctx=22042252, majf=0, minf=9508
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=13332861/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: io=2446.3GB, aggrb=356260KB/s, minb=356260KB/s, maxb=356260KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
  WRITE: io=2706.7GB, aggrb=394186KB/s, minb=394186KB/s, maxb=394186KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
  WRITE: io=862595MB, aggrb=122678KB/s, minb=122678KB/s, maxb=122678KB/s, mint=7200081msec, maxt=7200081msec

Run status group 3 (all jobs):
  WRITE: io=833304MB, aggrb=118514KB/s, minb=118514KB/s, maxb=118514KB/s, mint=7200001msec, maxt=7200001msec



# 
# Thu 14 Jan 2016 11:14:30 AM EST
# 64kB read with SATA 1 jobs and 10 I/Os Outstanding
# 64kB read with NVMe 1 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 7222 IOPS | 462 MBps | 1383 μsec Average Latency 
SATA-R5-1: 7182 IOPS | 459 MBps | 1391 μsec Average Latency 
NVMe-R5-0: 9278 IOPS | 593 MBps | 861 μsec Average Latency 
NVMe-R5-1: 9268 IOPS | 593 MBps | 862 μsec Average Latency 

Totals
IOPS: 32950
MBPS: 2107
Average Latency (μsec): 1124

SATA-R5-01: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=10
SATA-R5-11: (g=1): rw=read, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=10
NVMe-R5-01: (g=2): rw=read, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8
NVMe-R5-11: (g=3): rw=read, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8
fio-2.1.10
Starting 4 processes

SATA-R5-01: (groupid=0, jobs=1): err= 0: pid=148302: Thu Jan 14 13:15:31 2016
  read : io=3174.5GB, bw=462248KB/s, iops=7222, runt=7200001msec
    slat (usec): min=6, max=2986, avg=24.24, stdev=23.43
    clat (usec): min=1, max=21166, avg=1358.99, stdev=485.06
     lat (usec): min=189, max=21180, avg=1383.39, stdev=483.93
    clat percentiles (usec):
     |  1.00th=[  548],  5.00th=[  788], 10.00th=[  908], 20.00th=[ 1048],
     | 30.00th=[ 1144], 40.00th=[ 1224], 50.00th=[ 1304], 60.00th=[ 1384],
     | 70.00th=[ 1496], 80.00th=[ 1624], 90.00th=[ 1816], 95.00th=[ 2024],
     | 99.00th=[ 2992], 99.50th=[ 3632], 99.90th=[ 5856], 99.95th=[ 7520],
     | 99.99th=[10432]
    bw (KB  /s): min=    1, max=530304, per=100.00%, avg=462383.14, stdev=14171.16
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.65%, 750=3.30%, 1000=12.19%
    lat (msec) : 2=78.40%, 4=5.13%, 10=0.31%, 20=0.01%, 50=0.01%
  cpu          : usr=1.41%, sys=9.23%, ctx=76749584, majf=0, minf=23495
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=52002889/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=10
SATA-R5-11: (groupid=1, jobs=1): err= 0: pid=148303: Thu Jan 14 13:15:31 2016
  read : io=3156.3GB, bw=459661KB/s, iops=7182, runt=7200001msec
    slat (usec): min=6, max=2973, avg=25.02, stdev=23.74
    clat (usec): min=0, max=22097, avg=1366.00, stdev=533.66
     lat (usec): min=183, max=22111, avg=1391.19, stdev=532.62
    clat percentiles (usec):
     |  1.00th=[  532],  5.00th=[  764], 10.00th=[  892], 20.00th=[ 1032],
     | 30.00th=[ 1128], 40.00th=[ 1208], 50.00th=[ 1288], 60.00th=[ 1384],
     | 70.00th=[ 1480], 80.00th=[ 1624], 90.00th=[ 1848], 95.00th=[ 2096],
     | 99.00th=[ 3408], 99.50th=[ 3952], 99.90th=[ 6112], 99.95th=[ 7648],
     | 99.99th=[10688]
    bw (KB  /s): min=    1, max=513408, per=100.00%, avg=459794.22, stdev=15065.68
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.76%, 750=3.74%, 1000=13.18%
    lat (msec) : 2=75.80%, 4=6.04%, 10=0.45%, 20=0.02%, 50=0.01%
  cpu          : usr=1.40%, sys=9.46%, ctx=77238722, majf=0, minf=22638
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=51711911/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=10
NVMe-R5-01: (groupid=2, jobs=1): err= 0: pid=148304: Thu Jan 14 13:15:31 2016
  read : io=4077.7GB, bw=593851KB/s, iops=9278, runt=7200001msec
    slat (usec): min=6, max=1417, avg=24.68, stdev=23.29
    clat (usec): min=0, max=20725, avg=836.20, stdev=362.13
     lat (usec): min=176, max=20738, avg=861.03, stdev=360.49
    clat percentiles (usec):
     |  1.00th=[  350],  5.00th=[  454], 10.00th=[  506], 20.00th=[  580],
     | 30.00th=[  652], 40.00th=[  716], 50.00th=[  772], 60.00th=[  844],
     | 70.00th=[  932], 80.00th=[ 1048], 90.00th=[ 1224], 95.00th=[ 1400],
     | 99.00th=[ 1816], 99.50th=[ 2064], 99.90th=[ 4048], 99.95th=[ 5920],
     | 99.99th=[ 9664]
    bw (KB  /s): min=    1, max=651008, per=100.00%, avg=593967.08, stdev=18433.55
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.11%, 500=9.28%, 750=36.81%, 1000=29.98%
    lat (msec) : 2=23.24%, 4=0.48%, 10=0.09%, 20=0.01%, 50=0.01%
  cpu          : usr=1.76%, sys=11.94%, ctx=95253758, majf=0, minf=17868
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=66808251/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=1): err= 0: pid=148305: Thu Jan 14 13:15:31 2016
  read : io=4073.2GB, bw=593198KB/s, iops=9268, runt=7200001msec
    slat (usec): min=5, max=1882, avg=17.68, stdev=16.79
    clat (usec): min=1, max=19143, avg=844.25, stdev=375.24
     lat (usec): min=170, max=19161, avg=862.08, stdev=374.34
    clat percentiles (usec):
     |  1.00th=[  338],  5.00th=[  442], 10.00th=[  494], 20.00th=[  572],
     | 30.00th=[  644], 40.00th=[  716], 50.00th=[  772], 60.00th=[  860],
     | 70.00th=[  948], 80.00th=[ 1080], 90.00th=[ 1256], 95.00th=[ 1432],
     | 99.00th=[ 1864], 99.50th=[ 2096], 99.90th=[ 4080], 99.95th=[ 5920],
     | 99.99th=[ 9664]
    bw (KB  /s): min=    1, max=647296, per=100.00%, avg=593323.11, stdev=19253.81
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.10%, 500=10.50%, 750=35.46%, 1000=28.32%
    lat (msec) : 2=24.97%, 4=0.55%, 10=0.09%, 20=0.01%
  cpu          : usr=1.66%, sys=9.52%, ctx=98515369, majf=0, minf=18145
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=66734806/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=3174.5GB, aggrb=462247KB/s, minb=462247KB/s, maxb=462247KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=3156.3GB, aggrb=459661KB/s, minb=459661KB/s, maxb=459661KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
   READ: io=4077.7GB, aggrb=593851KB/s, minb=593851KB/s, maxb=593851KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=4073.2GB, aggrb=593198KB/s, minb=593198KB/s, maxb=593198KB/s, mint=7200001msec, maxt=7200001msec



# 
# Thu 14 Jan 2016 01:15:34 PM EST
# 64kB randread with SATA 2 jobs and 2 I/Os Outstanding
# 64kB randread with NVMe 4 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 3662 IOPS | 234 MBps | 1089 μsec Average Latency 
SATA-R5-1: 3686 IOPS | 235 MBps | 1081 μsec Average Latency 
NVMe-R5-0: 8706 IOPS | 557 MBps | 916 μsec Average Latency 
NVMe-R5-1: 7281 IOPS | 466 MBps | 1095 μsec Average Latency 

Totals
IOPS: 23335
MBPS: 1492
Average Latency (μsec): 1045

SATA-R5-01: (g=0): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
SATA-R5-11: (g=1): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
NVMe-R5-01: (g=2): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 12 processes

SATA-R5-01: (groupid=0, jobs=2): err= 0: pid=153421: Thu Jan 14 15:16:34 2016
  read : io=1609.5GB, bw=234398KB/s, iops=3662, runt=7200001msec
    slat (usec): min=6, max=2251, avg=24.19, stdev=22.50
    clat (usec): min=19, max=17698, avg=1065.43, stdev=338.38
     lat (usec): min=188, max=17725, avg=1089.79, stdev=337.34
    clat percentiles (usec):
     |  1.00th=[  716],  5.00th=[  796], 10.00th=[  844], 20.00th=[  900],
     | 30.00th=[  948], 40.00th=[  988], 50.00th=[ 1032], 60.00th=[ 1064],
     | 70.00th=[ 1112], 80.00th=[ 1176], 90.00th=[ 1272], 95.00th=[ 1384],
     | 99.00th=[ 1880], 99.50th=[ 3024], 99.90th=[ 5664], 99.95th=[ 6624],
     | 99.99th=[10304]
    bw (KB  /s): min=    1, max=123392, per=50.03%, avg=117273.58, stdev=2785.18
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.04%
    lat (usec) : 750=2.18%, 1000=41.49%
    lat (msec) : 2=55.37%, 4=0.70%, 10=0.20%, 20=0.01%
  cpu          : usr=0.73%, sys=3.00%, ctx=51257050, majf=0, minf=11890
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=26369787/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=2): err= 0: pid=153424: Thu Jan 14 15:16:34 2016
  read : io=1620.9GB, bw=235942KB/s, iops=3686, runt=7200001msec
    slat (usec): min=7, max=3755, avg=37.13, stdev=31.09
    clat (usec): min=17, max=23426, avg=1044.35, stdev=339.87
     lat (usec): min=203, max=23451, avg=1081.75, stdev=337.60
    clat percentiles (usec):
     |  1.00th=[  620],  5.00th=[  732], 10.00th=[  796], 20.00th=[  868],
     | 30.00th=[  924], 40.00th=[  972], 50.00th=[ 1012], 60.00th=[ 1064],
     | 70.00th=[ 1112], 80.00th=[ 1176], 90.00th=[ 1272], 95.00th=[ 1384],
     | 99.00th=[ 1736], 99.50th=[ 2736], 99.90th=[ 5600], 99.95th=[ 6624],
     | 99.99th=[10688]
    bw (KB  /s): min=    1, max=128000, per=50.03%, avg=118046.22, stdev=3344.22
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.09%
    lat (usec) : 750=6.08%, 1000=40.96%
    lat (msec) : 2=52.13%, 4=0.54%, 10=0.19%, 20=0.01%, 50=0.01%
  cpu          : usr=0.96%, sys=3.99%, ctx=50873282, majf=0, minf=10835
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=26543480/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=153426: Thu Jan 14 15:16:34 2016
  read : io=3826.2GB, bw=557228KB/s, iops=8706, runt=7200001msec
    slat (usec): min=5, max=2651, avg=31.73, stdev=29.27
    clat (usec): min=0, max=21225, avg=884.36, stdev=358.05
     lat (usec): min=173, max=21247, avg=916.32, stdev=356.57
    clat percentiles (usec):
     |  1.00th=[  330],  5.00th=[  494], 10.00th=[  596], 20.00th=[  676],
     | 30.00th=[  740], 40.00th=[  788], 50.00th=[  836], 60.00th=[  892],
     | 70.00th=[  964], 80.00th=[ 1048], 90.00th=[ 1208], 95.00th=[ 1368],
     | 99.00th=[ 1816], 99.50th=[ 2096], 99.90th=[ 5280], 99.95th=[ 6432],
     | 99.99th=[10048]
    bw (KB  /s): min=    1, max=179584, per=25.01%, avg=139380.48, stdev=6735.37
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.19%, 500=5.04%, 750=27.32%, 1000=42.21%
    lat (msec) : 2=24.61%, 4=0.47%, 10=0.15%, 20=0.01%, 50=0.01%
  cpu          : usr=0.88%, sys=4.23%, ctx=120313877, majf=0, minf=23309
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=62688143/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=153430: Thu Jan 14 15:16:34 2016
  read : io=3199.1GB, bw=466023KB/s, iops=7281, runt=7200001msec
    slat (usec): min=6, max=6439, avg=37.23, stdev=31.20
    clat (usec): min=0, max=19846, avg=1058.00, stdev=657.51
     lat (usec): min=173, max=19886, avg=1095.50, stdev=656.60
    clat percentiles (usec):
     |  1.00th=[  334],  5.00th=[  502], 10.00th=[  596], 20.00th=[  692],
     | 30.00th=[  748], 40.00th=[  812], 50.00th=[  876], 60.00th=[  948],
     | 70.00th=[ 1048], 80.00th=[ 1240], 90.00th=[ 1768], 95.00th=[ 2320],
     | 99.00th=[ 3696], 99.50th=[ 4384], 99.90th=[ 6112], 99.95th=[ 7008],
     | 99.99th=[11072]
    bw (KB  /s): min=    1, max=169344, per=25.01%, avg=116575.47, stdev=13727.53
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.19%, 500=4.78%, 750=24.82%, 1000=35.77%
    lat (msec) : 2=26.91%, 4=6.80%, 10=0.71%, 20=0.02%
  cpu          : usr=0.93%, sys=4.02%, ctx=100789425, majf=0, minf=23328
  IO depths    : 1=0.1%, 2=101.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=52427618/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=1609.5GB, aggrb=234398KB/s, minb=234398KB/s, maxb=234398KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=1620.9GB, aggrb=235942KB/s, minb=235942KB/s, maxb=235942KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
   READ: io=3826.2GB, aggrb=557227KB/s, minb=557227KB/s, maxb=557227KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=3199.1GB, aggrb=466023KB/s, minb=466023KB/s, maxb=466023KB/s, mint=7200001msec, maxt=7200001msec



# 
# Thu 14 Jan 2016 03:16:37 PM EST
# 64kB randread with SATA 6 jobs and 4 I/Os Outstanding
# 64kB randread with NVMe 4 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 6443 IOPS | 412 MBps | 3721 μsec Average Latency 
SATA-R5-1: 7521 IOPS | 481 MBps | 3188 μsec Average Latency 
NVMe-R5-0: 9420 IOPS | 602 MBps | 3394 μsec Average Latency 
NVMe-R5-1: 7901 IOPS | 505 MBps | 4047 μsec Average Latency 

Totals
IOPS: 31285
MBPS: 2000
Average Latency (μsec): 3587

SATA-R5-01: (g=0): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
SATA-R5-11: (g=1): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
...
NVMe-R5-01: (g=2): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8
...
NVMe-R5-11: (g=3): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8
...
fio-2.1.10
Starting 20 processes

SATA-R5-01: (groupid=0, jobs=6): err= 0: pid=158602: Thu Jan 14 17:17:38 2016
  read : io=2831.6GB, bw=412379KB/s, iops=6443, runt=7200003msec
    slat (usec): min=5, max=4079, avg=33.49, stdev=41.79
    clat (usec): min=0, max=25837, avg=3688.26, stdev=1648.18
     lat (usec): min=243, max=25858, avg=3721.99, stdev=1647.30
    clat percentiles (usec):
     |  1.00th=[ 1368],  5.00th=[ 2160], 10.00th=[ 2512], 20.00th=[ 2768],
     | 30.00th=[ 2896], 40.00th=[ 3024], 50.00th=[ 3120], 60.00th=[ 3280],
     | 70.00th=[ 3568], 80.00th=[ 4320], 90.00th=[ 5984], 95.00th=[ 7392],
     | 99.00th=[ 9792], 99.50th=[10816], 99.90th=[13120], 99.95th=[14016],
     | 99.99th=[16064]
    bw (KB  /s): min=    1, max=74752, per=16.68%, avg=68781.80, stdev=1836.69
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.13%
    lat (msec) : 2=3.71%, 4=73.22%, 10=22.03%, 20=0.89%, 50=0.01%
  cpu          : usr=0.49%, sys=2.10%, ctx=89124010, majf=0, minf=68614
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=46392676/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
SATA-R5-11: (groupid=1, jobs=6): err= 0: pid=158608: Thu Jan 14 17:17:38 2016
  read : io=3305.4GB, bw=481370KB/s, iops=7521, runt=7200004msec
    slat (usec): min=5, max=5251, avg=28.73, stdev=40.11
    clat (usec): min=0, max=24763, avg=3159.50, stdev=1645.36
     lat (usec): min=241, max=24783, avg=3188.46, stdev=1644.96
    clat percentiles (usec):
     |  1.00th=[ 1224],  5.00th=[ 1656], 10.00th=[ 1880], 20.00th=[ 2128],
     | 30.00th=[ 2320], 40.00th=[ 2480], 50.00th=[ 2640], 60.00th=[ 2832],
     | 70.00th=[ 3152], 80.00th=[ 3760], 90.00th=[ 5408], 95.00th=[ 6880],
     | 99.00th=[ 9280], 99.50th=[10304], 99.90th=[12608], 99.95th=[13504],
     | 99.99th=[15424]
    bw (KB  /s): min=    1, max=99584, per=16.68%, avg=80287.33, stdev=4898.74
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.24%
    lat (msec) : 2=13.93%, 4=68.18%, 10=16.98%, 20=0.64%, 50=0.01%
  cpu          : usr=0.50%, sys=2.31%, ctx=104033051, majf=0, minf=65002
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=54154175/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=158614: Thu Jan 14 17:17:38 2016
  read : io=4139.1GB, bw=602915KB/s, iops=9420, runt=7200002msec
    slat (usec): min=5, max=4746, avg=32.92, stdev=38.56
    clat (usec): min=0, max=27968, avg=3361.30, stdev=1629.38
     lat (usec): min=198, max=27996, avg=3394.44, stdev=1628.56
    clat percentiles (usec):
     |  1.00th=[  916],  5.00th=[ 1624], 10.00th=[ 2064], 20.00th=[ 2448],
     | 30.00th=[ 2608], 40.00th=[ 2736], 50.00th=[ 2864], 60.00th=[ 3024],
     | 70.00th=[ 3344], 80.00th=[ 4016], 90.00th=[ 5536], 95.00th=[ 6944],
     | 99.00th=[ 9408], 99.50th=[10432], 99.90th=[12736], 99.95th=[13632],
     | 99.99th=[15808]
    bw (KB  /s): min=    1, max=169984, per=25.01%, avg=150816.00, stdev=5444.98
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.07%, 750=0.35%, 1000=0.96%
    lat (msec) : 2=7.46%, 4=70.83%, 10=19.66%, 20=0.67%, 50=0.01%
  cpu          : usr=0.92%, sys=4.24%, ctx=121237692, majf=0, minf=76308
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=67827912/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=158618: Thu Jan 14 17:17:38 2016
  read : io=3472.4GB, bw=505691KB/s, iops=7901, runt=7200004msec
    slat (usec): min=5, max=4032, avg=26.86, stdev=38.83
    clat (usec): min=0, max=76138, avg=4020.61, stdev=3432.37
     lat (usec): min=191, max=76151, avg=4047.69, stdev=3432.23
    clat percentiles (usec):
     |  1.00th=[  708],  5.00th=[ 1112], 10.00th=[ 1400], 20.00th=[ 1736],
     | 30.00th=[ 1976], 40.00th=[ 2192], 50.00th=[ 2480], 60.00th=[ 3056],
     | 70.00th=[ 4576], 80.00th=[ 6496], 90.00th=[ 8512], 95.00th=[10560],
     | 99.00th=[17024], 99.50th=[19328], 99.90th=[24704], 99.95th=[27264],
     | 99.99th=[34048]
    bw (KB  /s): min=    1, max=168064, per=25.02%, avg=126503.19, stdev=11771.60
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.27%, 750=0.99%, 1000=2.37%
    lat (msec) : 2=27.38%, 4=36.09%, 10=26.94%, 20=5.55%, 50=0.40%
    lat (msec) : 100=0.01%
  cpu          : usr=0.71%, sys=3.39%, ctx=105659680, majf=0, minf=78938
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=101.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=56890267/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=2831.6GB, aggrb=412379KB/s, minb=412379KB/s, maxb=412379KB/s, mint=7200003msec, maxt=7200003msec

Run status group 1 (all jobs):
   READ: io=3305.4GB, aggrb=481370KB/s, minb=481370KB/s, maxb=481370KB/s, mint=7200004msec, maxt=7200004msec

Run status group 2 (all jobs):
   READ: io=4139.1GB, aggrb=602914KB/s, minb=602914KB/s, maxb=602914KB/s, mint=7200002msec, maxt=7200002msec

Run status group 3 (all jobs):
   READ: io=3472.4GB, aggrb=505691KB/s, minb=505691KB/s, maxb=505691KB/s, mint=7200004msec, maxt=7200004msec



# 
# Thu 14 Jan 2016 05:17:40 PM EST
# 32kB randread with SATA 5 jobs and 2 I/Os Outstanding
# 32kB randread with NVMe 3 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 9546 IOPS | 305 MBps | 1045 μsec Average Latency 
SATA-R5-1: 9925 IOPS | 317 MBps | 1005 μsec Average Latency 
NVMe-R5-0: 13627 IOPS | 436 MBps | 878 μsec Average Latency 
NVMe-R5-1: 13037 IOPS | 417 MBps | 918 μsec Average Latency 

Totals
IOPS: 46135
MBPS: 1475
Average Latency (μsec): 961

SATA-R5-01: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=2
...
SATA-R5-11: (g=1): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=2
...
NVMe-R5-01: (g=2): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=4
...
NVMe-R5-11: (g=3): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=4
...
fio-2.1.10
Starting 16 processes

SATA-R5-01: (groupid=0, jobs=5): err= 0: pid=163782: Thu Jan 14 19:18:41 2016
  read : io=2097.6GB, bw=305476KB/s, iops=9546, runt=7200001msec
    slat (usec): min=3, max=2330, avg=26.74, stdev=29.30
    clat (usec): min=1, max=24693, avg=1018.02, stdev=540.03
     lat (usec): min=170, max=24703, avg=1045.01, stdev=538.75
    clat percentiles (usec):
     |  1.00th=[  580],  5.00th=[  684], 10.00th=[  740], 20.00th=[  804],
     | 30.00th=[  852], 40.00th=[  900], 50.00th=[  948], 60.00th=[  996],
     | 70.00th=[ 1048], 80.00th=[ 1128], 90.00th=[ 1256], 95.00th=[ 1400],
     | 99.00th=[ 3152], 99.50th=[ 4896], 99.90th=[ 8256], 99.95th=[ 9920],
     | 99.99th=[12352]
    bw (KB  /s): min=    0, max=67968, per=20.01%, avg=61133.18, stdev=2521.43
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.26%, 750=11.64%, 1000=49.53%
    lat (msec) : 2=37.06%, 4=0.71%, 10=0.76%, 20=0.05%, 50=0.01%
  cpu          : usr=0.79%, sys=3.01%, ctx=131275780, majf=0, minf=19353
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=68732048/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=5): err= 0: pid=163787: Thu Jan 14 19:18:41 2016
  read : io=2180.1GB, bw=317621KB/s, iops=9925, runt=7200001msec
    slat (usec): min=3, max=3487, avg=21.66, stdev=25.87
    clat (usec): min=0, max=24903, avg=983.58, stdev=538.78
     lat (usec): min=167, max=24918, avg=1005.41, stdev=537.97
    clat percentiles (usec):
     |  1.00th=[  564],  5.00th=[  652], 10.00th=[  708], 20.00th=[  764],
     | 30.00th=[  820], 40.00th=[  860], 50.00th=[  908], 60.00th=[  956],
     | 70.00th=[ 1012], 80.00th=[ 1096], 90.00th=[ 1224], 95.00th=[ 1368],
     | 99.00th=[ 3088], 99.50th=[ 4832], 99.90th=[ 8160], 99.95th=[ 9920],
     | 99.99th=[12352]
    bw (KB  /s): min=    0, max=70912, per=20.01%, avg=63561.90, stdev=2344.60
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.32%, 750=16.81%, 1000=50.88%
    lat (msec) : 2=30.51%, 4=0.69%, 10=0.73%, 20=0.05%, 50=0.01%
  cpu          : usr=0.66%, sys=2.57%, ctx=137553516, majf=0, minf=18800
  IO depths    : 1=0.1%, 2=100.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=71464696/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=3): err= 0: pid=163792: Thu Jan 14 19:18:41 2016
  read : io=2994.3GB, bw=436068KB/s, iops=13627, runt=7200001msec
    slat (usec): min=4, max=2320, avg=24.41, stdev=27.44
    clat (usec): min=0, max=24351, avg=854.06, stdev=522.05
     lat (usec): min=156, max=24362, avg=878.69, stdev=520.45
    clat percentiles (usec):
     |  1.00th=[  378],  5.00th=[  502], 10.00th=[  564], 20.00th=[  636],
     | 30.00th=[  692], 40.00th=[  732], 50.00th=[  780], 60.00th=[  828],
     | 70.00th=[  884], 80.00th=[  972], 90.00th=[ 1128], 95.00th=[ 1288],
     | 99.00th=[ 2352], 99.50th=[ 4576], 99.90th=[ 7840], 99.95th=[ 9536],
     | 99.99th=[12096]
    bw (KB  /s): min=    0, max=176512, per=33.34%, avg=145402.22, stdev=7390.13
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.09%, 500=4.87%, 750=38.53%, 1000=39.12%
    lat (msec) : 2=16.19%, 4=0.53%, 10=0.63%, 20=0.04%, 50=0.01%
  cpu          : usr=1.37%, sys=5.83%, ctx=168696643, majf=0, minf=17511
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=98115308/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=3): err= 0: pid=163795: Thu Jan 14 19:18:41 2016
  read : io=2864.8GB, bw=417203KB/s, iops=13037, runt=7200001msec
    slat (usec): min=3, max=1609, avg=16.69, stdev=20.68
    clat (usec): min=1, max=24746, avg=901.90, stdev=610.86
     lat (usec): min=134, max=24757, avg=918.73, stdev=610.29
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[  498], 10.00th=[  564], 20.00th=[  636],
     | 30.00th=[  692], 40.00th=[  732], 50.00th=[  780], 60.00th=[  836],
     | 70.00th=[  900], 80.00th=[ 1012], 90.00th=[ 1240], 95.00th=[ 1544],
     | 99.00th=[ 3728], 99.50th=[ 4832], 99.90th=[ 8096], 99.95th=[ 9920],
     | 99.99th=[12352]
    bw (KB  /s): min=    0, max=170048, per=33.34%, avg=139115.41, stdev=11507.65
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=0.08%, 500=4.97%, 750=38.76%, 1000=35.18%
    lat (msec) : 2=18.25%, 4=1.88%, 10=0.83%, 20=0.05%, 50=0.01%
  cpu          : usr=1.12%, sys=4.42%, ctx=166121963, majf=0, minf=18327
  IO depths    : 1=0.1%, 2=0.1%, 4=100.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=93870720/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=2097.6GB, aggrb=305475KB/s, minb=305475KB/s, maxb=305475KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=2180.1GB, aggrb=317620KB/s, minb=317620KB/s, maxb=317620KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
   READ: io=2994.3GB, aggrb=436068KB/s, minb=436068KB/s, maxb=436068KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=2864.8GB, aggrb=417203KB/s, minb=417203KB/s, maxb=417203KB/s, mint=7200001msec, maxt=7200001msec



# 
# Thu 14 Jan 2016 07:18:44 PM EST
# 32kB randread with SATA 4 jobs and 16 I/Os Outstanding
# 32kB randread with NVMe 3 jobs and 8 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 17704 IOPS | 566 MBps | 3613 μsec Average Latency 
SATA-R5-1: 17646 IOPS | 564 MBps | 3625 μsec Average Latency 
NVMe-R5-0: 9588 IOPS | 306 MBps | 2501 μsec Average Latency 
NVMe-R5-1: 9455 IOPS | 302 MBps | 2536 μsec Average Latency 

Totals
IOPS: 54393
MBPS: 1738
Average Latency (μsec): 3068

SATA-R5-01: (g=0): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=16
...
SATA-R5-11: (g=1): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=16
...
NVMe-R5-01: (g=2): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=8
...
NVMe-R5-11: (g=3): rw=randread, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=8
...
fio-2.1.10
Starting 14 processes

SATA-R5-01: (groupid=0, jobs=4): err= 0: pid=168963: Thu Jan 14 21:19:44 2016
  read : io=3890.2GB, bw=566549KB/s, iops=17704, runt=7200003msec
    slat (usec): min=3, max=3923, avg=24.40, stdev=36.32
    clat (usec): min=1, max=119275, avg=3588.61, stdev=1594.09
     lat (usec): min=203, max=119285, avg=3613.18, stdev=1593.40
    clat percentiles (usec):
     |  1.00th=[ 1528],  5.00th=[ 2024], 10.00th=[ 2224], 20.00th=[ 2512],
     | 30.00th=[ 2736], 40.00th=[ 2960], 50.00th=[ 3184], 60.00th=[ 3440],
     | 70.00th=[ 3760], 80.00th=[ 4256], 90.00th=[ 5472], 95.00th=[ 7136],
     | 99.00th=[ 9536], 99.50th=[10688], 99.90th=[12992], 99.95th=[13888],
     | 99.99th=[16064]
    bw (KB  /s): min=    0, max=164928, per=25.01%, avg=141698.87, stdev=4885.53
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.06%
    lat (msec) : 2=4.53%, 4=71.31%, 10=23.34%, 20=0.74%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.19%, sys=5.34%, ctx=206990887, majf=0, minf=78790
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=127473486/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
SATA-R5-11: (groupid=1, jobs=4): err= 0: pid=168967: Thu Jan 14 21:19:44 2016
  read : io=3877.4GB, bw=564684KB/s, iops=17646, runt=7200007msec
    slat (usec): min=3, max=6763, avg=21.81, stdev=42.07
    clat (usec): min=0, max=119983, avg=3603.08, stdev=1596.95
     lat (usec): min=264, max=119994, avg=3625.08, stdev=1596.19
    clat percentiles (usec):
     |  1.00th=[ 1512],  5.00th=[ 2008], 10.00th=[ 2224], 20.00th=[ 2512],
     | 30.00th=[ 2736], 40.00th=[ 2960], 50.00th=[ 3184], 60.00th=[ 3440],
     | 70.00th=[ 3760], 80.00th=[ 4256], 90.00th=[ 5536], 95.00th=[ 7136],
     | 99.00th=[ 9536], 99.50th=[10688], 99.90th=[12992], 99.95th=[13888],
     | 99.99th=[15936]
    bw (KB  /s): min=    0, max=165568, per=25.01%, avg=141232.44, stdev=4855.69
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.08%
    lat (msec) : 2=4.66%, 4=70.71%, 10=23.80%, 20=0.75%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.14%, sys=5.10%, ctx=210694186, majf=0, minf=82110
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=127053933/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
NVMe-R5-01: (groupid=2, jobs=3): err= 0: pid=168971: Thu Jan 14 21:19:44 2016
  read : io=2106.1GB, bw=306841KB/s, iops=9588, runt=7200003msec
    slat (usec): min=3, max=4358, avg=24.54, stdev=38.70
    clat (usec): min=0, max=89965, avg=2476.40, stdev=1441.92
     lat (usec): min=188, max=89976, avg=2501.12, stdev=1441.34
    clat percentiles (usec):
     |  1.00th=[  828],  5.00th=[ 1224], 10.00th=[ 1416], 20.00th=[ 1608],
     | 30.00th=[ 1752], 40.00th=[ 1896], 50.00th=[ 2064], 60.00th=[ 2256],
     | 70.00th=[ 2544], 80.00th=[ 2992], 90.00th=[ 3824], 95.00th=[ 5792],
     | 99.00th=[ 8256], 99.50th=[ 9280], 99.90th=[11712], 99.95th=[12608],
     | 99.99th=[14528]
    bw (KB  /s): min=    0, max=120128, per=33.35%, avg=102333.36, stdev=4403.86
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.05%, 750=0.55%, 1000=1.63%
    lat (msec) : 2=44.51%, 4=44.53%, 10=8.42%, 20=0.32%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=0.94%, sys=3.88%, ctx=118174098, majf=0, minf=34012
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=69039343/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-11: (groupid=3, jobs=3): err= 0: pid=168974: Thu Jan 14 21:19:44 2016
  read : io=2077.7GB, bw=302578KB/s, iops=9455, runt=7200003msec
    slat (usec): min=3, max=7703, avg=22.13, stdev=41.58
    clat (usec): min=0, max=95718, avg=2514.04, stdev=1454.20
     lat (usec): min=186, max=95726, avg=2536.38, stdev=1453.73
    clat percentiles (usec):
     |  1.00th=[  820],  5.00th=[ 1224], 10.00th=[ 1432], 20.00th=[ 1624],
     | 30.00th=[ 1784], 40.00th=[ 1928], 50.00th=[ 2096], 60.00th=[ 2288],
     | 70.00th=[ 2576], 80.00th=[ 3056], 90.00th=[ 3888], 95.00th=[ 5856],
     | 99.00th=[ 8384], 99.50th=[ 9408], 99.90th=[11712], 99.95th=[12608],
     | 99.99th=[14528]
    bw (KB  /s): min=    0, max=125184, per=33.35%, avg=100911.83, stdev=4475.39
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.01%, 500=0.04%, 750=0.57%, 1000=1.70%
    lat (msec) : 2=42.27%, 4=46.19%, 10=8.89%, 20=0.33%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=0.89%, sys=3.72%, ctx=118510886, majf=0, minf=32459
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=68079964/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=3890.2GB, aggrb=566548KB/s, minb=566548KB/s, maxb=566548KB/s, mint=7200003msec, maxt=7200003msec

Run status group 1 (all jobs):
   READ: io=3877.4GB, aggrb=564683KB/s, minb=564683KB/s, maxb=564683KB/s, mint=7200007msec, maxt=7200007msec

Run status group 2 (all jobs):
   READ: io=2106.1GB, aggrb=306841KB/s, minb=306841KB/s, maxb=306841KB/s, mint=7200003msec, maxt=7200003msec

Run status group 3 (all jobs):
   READ: io=2077.7GB, aggrb=302577KB/s, minb=302577KB/s, maxb=302577KB/s, mint=7200003msec, maxt=7200003msec



# 
# Thu 14 Jan 2016 09:19:47 PM EST
# 16kB randread with SATA 5 jobs and 16 I/Os Outstanding
# 16kB randread with NVMe 4 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 28253 IOPS | 452 MBps | 2829 μsec Average Latency 
SATA-R5-1: 27944 IOPS | 447 MBps | 2861 μsec Average Latency 
NVMe-R5-0: 6519 IOPS | 104 MBps | 2452 μsec Average Latency 
NVMe-R5-1: 6622 IOPS | 105 MBps | 2413 μsec Average Latency 

Totals
IOPS: 69338
MBPS: 1108
Average Latency (μsec): 2638

SATA-R5-01: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=16
...
SATA-R5-11: (g=1): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=16
...
NVMe-R5-01: (g=2): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=4
...
NVMe-R5-11: (g=3): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=4
...
fio-2.1.10
Starting 18 processes

SATA-R5-01: (groupid=0, jobs=5): err= 0: pid=174101: Thu Jan 14 23:20:48 2016
  read : io=3104.8GB, bw=452063KB/s, iops=28253, runt=7200003msec
    slat (usec): min=2, max=4832, avg=15.19, stdev=20.25
    clat (usec): min=55, max=190204, avg=2814.51, stdev=594.06
     lat (usec): min=305, max=190214, avg=2829.84, stdev=593.33
    clat percentiles (usec):
     |  1.00th=[ 1864],  5.00th=[ 2128], 10.00th=[ 2288], 20.00th=[ 2448],
     | 30.00th=[ 2544], 40.00th=[ 2640], 50.00th=[ 2736], 60.00th=[ 2864],
     | 70.00th=[ 2960], 80.00th=[ 3120], 90.00th=[ 3376], 95.00th=[ 3600],
     | 99.00th=[ 4640], 99.50th=[ 5600], 99.90th=[ 8256], 99.95th=[ 9280],
     | 99.99th=[12096]
    bw (KB  /s): min=    0, max=96960, per=20.01%, avg=90439.77, stdev=1832.35
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=2.19%, 4=95.60%, 10=2.16%, 20=0.03%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.30%, sys=4.86%, ctx=342353321, majf=0, minf=53641
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=203428396/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
SATA-R5-11: (groupid=1, jobs=5): err= 0: pid=174106: Thu Jan 14 23:20:48 2016
  read : io=3070.5GB, bw=447108KB/s, iops=27944, runt=7200003msec
    slat (usec): min=2, max=15458, avg=20.36, stdev=24.16
    clat (usec): min=37, max=210203, avg=2840.59, stdev=601.65
     lat (usec): min=309, max=210215, avg=2861.14, stdev=600.30
    clat percentiles (usec):
     |  1.00th=[ 1864],  5.00th=[ 2160], 10.00th=[ 2288], 20.00th=[ 2448],
     | 30.00th=[ 2576], 40.00th=[ 2672], 50.00th=[ 2768], 60.00th=[ 2896],
     | 70.00th=[ 2992], 80.00th=[ 3152], 90.00th=[ 3376], 95.00th=[ 3632],
     | 99.00th=[ 4640], 99.50th=[ 5600], 99.90th=[ 8256], 99.95th=[ 9152],
     | 99.99th=[11968]
    bw (KB  /s): min=    0, max=97152, per=20.01%, avg=89448.87, stdev=1801.81
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
    lat (usec) : 1000=0.01%
    lat (msec) : 2=2.12%, 4=95.62%, 10=2.22%, 20=0.03%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.49%, sys=6.01%, ctx=332933802, majf=0, minf=54745
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=201198501/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=174111: Thu Jan 14 23:20:48 2016
  read : io=733500MB, bw=104320KB/s, iops=6519, runt=7200003msec
    slat (usec): min=2, max=2602, avg=14.38, stdev=19.69
    clat (usec): min=1, max=101605, avg=2437.55, stdev=577.39
     lat (usec): min=257, max=101630, avg=2452.07, stdev=576.87
    clat percentiles (usec):
     |  1.00th=[ 1496],  5.00th=[ 1784], 10.00th=[ 1912], 20.00th=[ 2064],
     | 30.00th=[ 2192], 40.00th=[ 2288], 50.00th=[ 2384], 60.00th=[ 2480],
     | 70.00th=[ 2608], 80.00th=[ 2736], 90.00th=[ 2992], 95.00th=[ 3216],
     | 99.00th=[ 4192], 99.50th=[ 5088], 99.90th=[ 7776], 99.95th=[ 8768],
     | 99.99th=[11456]
    bw (KB  /s): min=    0, max=28621, per=25.02%, avg=26097.47, stdev=664.90
    lat (usec) : 2=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
    lat (usec) : 750=0.03%, 1000=0.07%
    lat (msec) : 2=15.12%, 4=83.58%, 10=1.17%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.48%, sys=1.46%, ctx=84759868, majf=0, minf=15946
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=46943977/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=174115: Thu Jan 14 23:20:48 2016
  read : io=745083MB, bw=105967KB/s, iops=6622, runt=7200003msec
    slat (usec): min=2, max=4476, avg=22.38, stdev=26.21
    clat (usec): min=1, max=123490, avg=2390.83, stdev=584.90
     lat (usec): min=272, max=123509, avg=2413.44, stdev=583.75
    clat percentiles (usec):
     |  1.00th=[ 1416],  5.00th=[ 1720], 10.00th=[ 1848], 20.00th=[ 2008],
     | 30.00th=[ 2128], 40.00th=[ 2224], 50.00th=[ 2320], 60.00th=[ 2448],
     | 70.00th=[ 2544], 80.00th=[ 2704], 90.00th=[ 2928], 95.00th=[ 3184],
     | 99.00th=[ 4080], 99.50th=[ 5024], 99.90th=[ 7712], 99.95th=[ 8640],
     | 99.99th=[11072]
    bw (KB  /s): min=    0, max=29536, per=25.02%, avg=26509.35, stdev=777.24
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=0.01%, 750=0.03%, 1000=0.09%
    lat (msec) : 2=18.90%, 4=79.89%, 10=1.08%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.66%, sys=2.04%, ctx=90664168, majf=0, minf=15549
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=47685316/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=3104.8GB, aggrb=452063KB/s, minb=452063KB/s, maxb=452063KB/s, mint=7200003msec, maxt=7200003msec

Run status group 1 (all jobs):
   READ: io=3070.5GB, aggrb=447107KB/s, minb=447107KB/s, maxb=447107KB/s, mint=7200003msec, maxt=7200003msec

Run status group 2 (all jobs):
   READ: io=733500MB, aggrb=104319KB/s, minb=104319KB/s, maxb=104319KB/s, mint=7200003msec, maxt=7200003msec

Run status group 3 (all jobs):
   READ: io=745083MB, aggrb=105967KB/s, minb=105967KB/s, maxb=105967KB/s, mint=7200003msec, maxt=7200003msec



# 
# Thu 14 Jan 2016 11:20:51 PM EST
# 8kB randread with SATA 6 jobs and 2 I/Os Outstanding
# 8kB randread with NVMe 4 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 18855 IOPS | 150 MBps | 634 μsec Average Latency 
SATA-R5-1: 19351 IOPS | 154 MBps | 618 μsec Average Latency 
NVMe-R5-0: 13869 IOPS | 110 MBps | 574 μsec Average Latency 
NVMe-R5-1: 14216 IOPS | 113 MBps | 560 μsec Average Latency 

Totals
IOPS: 66291
MBPS: 527
Average Latency (μsec): 596

SATA-R5-01: (g=0): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
...
SATA-R5-11: (g=1): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
...
NVMe-R5-01: (g=2): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 20 processes

SATA-R5-01: (groupid=0, jobs=6): err= 0: pid=179287: Fri Jan 15 01:21:51 2016
  read : io=1035.8GB, bw=150841KB/s, iops=18855, runt=7200002msec
    slat (usec): min=2, max=9273, avg=19.68, stdev=18.93
    clat (usec): min=13, max=74896, avg=614.36, stdev=205.83
     lat (usec): min=170, max=74903, avg=634.27, stdev=204.17
    clat percentiles (usec):
     |  1.00th=[  374],  5.00th=[  450], 10.00th=[  474], 20.00th=[  502],
     | 30.00th=[  532], 40.00th=[  572], 50.00th=[  596], 60.00th=[  612],
     | 70.00th=[  644], 80.00th=[  684], 90.00th=[  740], 95.00th=[  836],
     | 99.00th=[ 1208], 99.50th=[ 1464], 99.90th=[ 3120], 99.95th=[ 3568],
     | 99.99th=[ 6304]
    bw (KB  /s): min=    0, max=26640, per=16.67%, avg=25149.52, stdev=445.63
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.02%, 500=19.33%
    lat (usec) : 750=71.33%, 1000=6.84%
    lat (msec) : 2=2.16%, 4=0.30%, 10=0.03%, 20=0.01%, 100=0.01%
  cpu          : usr=1.08%, sys=3.61%, ctx=258120804, majf=0, minf=16775
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=135757171/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=6): err= 0: pid=179293: Fri Jan 15 01:21:51 2016
  read : io=1063.5GB, bw=154811KB/s, iops=19351, runt=7200001msec
    slat (usec): min=2, max=3517, avg=11.45, stdev=11.30
    clat (usec): min=1, max=30989, avg=606.67, stdev=192.17
     lat (usec): min=113, max=31002, avg=618.28, stdev=191.62
    clat percentiles (usec):
     |  1.00th=[  406],  5.00th=[  458], 10.00th=[  478], 20.00th=[  498],
     | 30.00th=[  524], 40.00th=[  564], 50.00th=[  596], 60.00th=[  612],
     | 70.00th=[  628], 80.00th=[  660], 90.00th=[  732], 95.00th=[  812],
     | 99.00th=[ 1176], 99.50th=[ 1352], 99.90th=[ 3024], 99.95th=[ 3504],
     | 99.99th=[ 6240]
    bw (KB  /s): min=    0, max=26896, per=16.67%, avg=25811.19, stdev=379.65
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=21.65%, 750=70.36%, 1000=5.73%
    lat (msec) : 2=1.99%, 4=0.23%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=0.85%, sys=2.40%, ctx=266687795, majf=0, minf=12795
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=139330170/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=179299: Fri Jan 15 01:21:51 2016
  read : io=780133MB, bw=110952KB/s, iops=13869, runt=7200001msec
    slat (usec): min=2, max=3159, avg=19.56, stdev=19.52
    clat (usec): min=1, max=32384, avg=554.88, stdev=183.62
     lat (usec): min=117, max=32472, avg=574.67, stdev=181.46
    clat percentiles (usec):
     |  1.00th=[  318],  5.00th=[  378], 10.00th=[  426], 20.00th=[  466],
     | 30.00th=[  486], 40.00th=[  502], 50.00th=[  524], 60.00th=[  556],
     | 70.00th=[  596], 80.00th=[  628], 90.00th=[  692], 95.00th=[  764],
     | 99.00th=[ 1096], 99.50th=[ 1240], 99.90th=[ 2800], 99.95th=[ 3408],
     | 99.99th=[ 6176]
    bw (KB  /s): min=    0, max=29488, per=25.01%, avg=27747.06, stdev=448.84
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.22%
    lat (usec) : 500=38.64%, 750=55.49%, 1000=4.07%
    lat (msec) : 2=1.39%, 4=0.15%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=1.18%, sys=3.75%, ctx=189024078, majf=0, minf=10145
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=99857044/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=179303: Fri Jan 15 01:21:51 2016
  read : io=799684MB, bw=113733KB/s, iops=14216, runt=7200001msec
    slat (usec): min=2, max=993, avg=11.18, stdev=10.86
    clat (usec): min=1, max=53776, avg=549.64, stdev=177.86
     lat (usec): min=113, max=53783, avg=560.97, stdev=177.18
    clat percentiles (usec):
     |  1.00th=[  326],  5.00th=[  382], 10.00th=[  434], 20.00th=[  470],
     | 30.00th=[  482], 40.00th=[  494], 50.00th=[  510], 60.00th=[  548],
     | 70.00th=[  588], 80.00th=[  620], 90.00th=[  668], 95.00th=[  756],
     | 99.00th=[ 1080], 99.50th=[ 1224], 99.90th=[ 2736], 99.95th=[ 3344],
     | 99.99th=[ 6112]
    bw (KB  /s): min=    0, max=32176, per=25.01%, avg=28442.45, stdev=410.16
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.14%
    lat (usec) : 500=43.32%, 750=51.37%, 1000=3.71%
    lat (msec) : 2=1.29%, 4=0.14%, 10=0.03%, 20=0.01%, 100=0.01%
  cpu          : usr=0.91%, sys=2.50%, ctx=194708422, majf=0, minf=8457
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=102359542/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=1035.8GB, aggrb=150841KB/s, minb=150841KB/s, maxb=150841KB/s, mint=7200002msec, maxt=7200002msec

Run status group 1 (all jobs):
   READ: io=1063.5GB, aggrb=154811KB/s, minb=154811KB/s, maxb=154811KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
   READ: io=780133MB, aggrb=110952KB/s, minb=110952KB/s, maxb=110952KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=799684MB, aggrb=113732KB/s, minb=113732KB/s, maxb=113732KB/s, mint=7200001msec, maxt=7200001msec



# 
# Fri 15 Jan 2016 01:21:54 AM EST
# 8kB randread with SATA 4 jobs and 8 I/Os Outstanding
# 8kB randread with NVMe 4 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 23998 IOPS | 191 MBps | 1331 μsec Average Latency 
SATA-R5-1: 23882 IOPS | 191 MBps | 1338 μsec Average Latency 
NVMe-R5-0: 12957 IOPS | 103 MBps | 1233 μsec Average Latency 
NVMe-R5-1: 12830 IOPS | 102 MBps | 1244 μsec Average Latency 

Totals
IOPS: 73667
MBPS: 587
Average Latency (μsec): 1286

SATA-R5-01: (g=0): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
SATA-R5-11: (g=1): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
...
NVMe-R5-01: (g=2): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=4
...
NVMe-R5-11: (g=3): rw=randread, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=4
...
fio-2.1.10
Starting 16 processes

SATA-R5-01: (groupid=0, jobs=4): err= 0: pid=185134: Fri Jan 15 03:22:55 2016
  read : io=1318.3GB, bw=191987KB/s, iops=23998, runt=7200001msec
    slat (usec): min=2, max=2017, avg=11.02, stdev=11.50
    clat (usec): min=50, max=119858, avg=1320.64, stdev=415.52
     lat (usec): min=160, max=119866, avg=1331.80, stdev=415.10
    clat percentiles (usec):
     |  1.00th=[  764],  5.00th=[  932], 10.00th=[  996], 20.00th=[ 1096],
     | 30.00th=[ 1144], 40.00th=[ 1208], 50.00th=[ 1240], 60.00th=[ 1304],
     | 70.00th=[ 1368], 80.00th=[ 1480], 90.00th=[ 1672], 95.00th=[ 1880],
     | 99.00th=[ 2992], 99.50th=[ 3760], 99.90th=[ 5472], 99.95th=[ 6816],
     | 99.99th=[ 8384]
    bw (KB  /s): min=    0, max=50912, per=25.01%, avg=48008.62, stdev=905.36
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.03%, 750=0.86%, 1000=9.34%
    lat (msec) : 2=86.24%, 4=3.15%, 10=0.38%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.35%, sys=3.94%, ctx=298114704, majf=0, minf=14875
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=172788302/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SATA-R5-11: (groupid=1, jobs=4): err= 0: pid=185138: Fri Jan 15 03:22:55 2016
  read : io=1311.1GB, bw=191062KB/s, iops=23882, runt=7200003msec
    slat (usec): min=2, max=6614, avg=17.82, stdev=19.22
    clat (usec): min=53, max=71960, avg=1320.09, stdev=414.38
     lat (usec): min=244, max=72030, avg=1338.12, stdev=413.09
    clat percentiles (usec):
     |  1.00th=[  764],  5.00th=[  924], 10.00th=[  996], 20.00th=[ 1096],
     | 30.00th=[ 1144], 40.00th=[ 1208], 50.00th=[ 1256], 60.00th=[ 1304],
     | 70.00th=[ 1368], 80.00th=[ 1480], 90.00th=[ 1672], 95.00th=[ 1880],
     | 99.00th=[ 2928], 99.50th=[ 3760], 99.90th=[ 5472], 99.95th=[ 6816],
     | 99.99th=[ 8384]
    bw (KB  /s): min=    0, max=50784, per=25.01%, avg=47776.92, stdev=902.40
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.03%, 750=0.86%, 1000=9.37%
    lat (msec) : 2=86.23%, 4=3.13%, 10=0.38%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.62%, sys=5.75%, ctx=293481540, majf=0, minf=14713
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=171956200/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=185142: Fri Jan 15 03:22:55 2016
  read : io=728843MB, bw=103658KB/s, iops=12957, runt=7200003msec
    slat (usec): min=2, max=1703, avg=11.28, stdev=12.36
    clat (usec): min=13, max=65113, avg=1221.71, stdev=378.76
     lat (usec): min=165, max=65123, avg=1233.14, stdev=378.32
    clat percentiles (usec):
     |  1.00th=[  684],  5.00th=[  844], 10.00th=[  924], 20.00th=[  996],
     | 30.00th=[ 1064], 40.00th=[ 1112], 50.00th=[ 1160], 60.00th=[ 1224],
     | 70.00th=[ 1272], 80.00th=[ 1368], 90.00th=[ 1544], 95.00th=[ 1752],
     | 99.00th=[ 2352], 99.50th=[ 3216], 99.90th=[ 5280], 99.95th=[ 6560],
     | 99.99th=[ 8160]
    bw (KB  /s): min=    0, max=27712, per=25.01%, avg=25924.87, stdev=513.50
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.10%
    lat (usec) : 750=2.06%, 1000=18.07%
    lat (msec) : 2=77.66%, 4=1.81%, 10=0.29%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=0.79%, sys=2.24%, ctx=174922242, majf=0, minf=9566
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=93291882/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=185146: Fri Jan 15 03:22:55 2016
  read : io=721691MB, bw=102640KB/s, iops=12830, runt=7200001msec
    slat (usec): min=2, max=1985, avg=18.77, stdev=19.80
    clat (usec): min=1, max=89279, avg=1225.96, stdev=380.89
     lat (usec): min=175, max=89286, avg=1244.95, stdev=379.66
    clat percentiles (usec):
     |  1.00th=[  684],  5.00th=[  844], 10.00th=[  916], 20.00th=[ 1004],
     | 30.00th=[ 1064], 40.00th=[ 1112], 50.00th=[ 1176], 60.00th=[ 1224],
     | 70.00th=[ 1288], 80.00th=[ 1384], 90.00th=[ 1560], 95.00th=[ 1752],
     | 99.00th=[ 2352], 99.50th=[ 3248], 99.90th=[ 5216], 99.95th=[ 6624],
     | 99.99th=[ 8256]
    bw (KB  /s): min=    0, max=27584, per=25.01%, avg=25670.54, stdev=526.88
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
    lat (usec) : 500=0.08%, 750=2.02%, 1000=17.35%
    lat (msec) : 2=78.40%, 4=1.85%, 10=0.30%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.08%, sys=3.44%, ctx=171641764, majf=0, minf=10347
  IO depths    : 1=0.1%, 2=0.1%, 4=100.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=92376400/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=1318.3GB, aggrb=191987KB/s, minb=191987KB/s, maxb=191987KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=1311.1GB, aggrb=191062KB/s, minb=191062KB/s, maxb=191062KB/s, mint=7200003msec, maxt=7200003msec

Run status group 2 (all jobs):
   READ: io=728843MB, aggrb=103657KB/s, minb=103657KB/s, maxb=103657KB/s, mint=7200003msec, maxt=7200003msec

Run status group 3 (all jobs):
   READ: io=721691MB, aggrb=102640KB/s, minb=102640KB/s, maxb=102640KB/s, mint=7200001msec, maxt=7200001msec



# 
# Fri 15 Jan 2016 03:22:58 AM EST
# 4kB randread with SATA 4 jobs and 2 I/Os Outstanding
# 4kB randread with NVMe 4 jobs and 2 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 14828 IOPS | 59 MBps | 537 μsec Average Latency 
SATA-R5-1: 15183 IOPS | 60 MBps | 525 μsec Average Latency 
NVMe-R5-0: 16554 IOPS | 66 MBps | 481 μsec Average Latency 
NVMe-R5-1: 16967 IOPS | 67 MBps | 469 μsec Average Latency 

Totals
IOPS: 63532
MBPS: 252
Average Latency (μsec): 503

SATA-R5-01: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2
...
SATA-R5-11: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2
...
NVMe-R5-01: (g=2): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2
...
NVMe-R5-11: (g=3): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2
...
fio-2.1.10
Starting 16 processes

SATA-R5-01: (groupid=0, jobs=4): err= 0: pid=190273: Fri Jan 15 05:23:59 2016
  read : io=417059MB, bw=59315KB/s, iops=14828, runt=7200001msec
    slat (usec): min=2, max=4788, avg=17.73, stdev=16.69
    clat (usec): min=14, max=77937, avg=519.39, stdev=163.94
     lat (usec): min=126, max=77945, avg=537.35, stdev=162.21
    clat percentiles (usec):
     |  1.00th=[  322],  5.00th=[  374], 10.00th=[  410], 20.00th=[  450],
     | 30.00th=[  470], 40.00th=[  482], 50.00th=[  498], 60.00th=[  516],
     | 70.00th=[  540], 80.00th=[  580], 90.00th=[  620], 95.00th=[  676],
     | 99.00th=[ 1032], 99.50th=[ 1240], 99.90th=[ 2512], 99.95th=[ 2960],
     | 99.99th=[ 6048]
    bw (KB  /s): min=    0, max=15712, per=25.01%, avg=14833.26, stdev=271.77
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.14%, 500=52.22%
    lat (usec) : 750=44.97%, 1000=1.52%
    lat (msec) : 2=0.99%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.23%, sys=3.89%, ctx=200209882, majf=0, minf=7120
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=106766980/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
SATA-R5-11: (groupid=1, jobs=4): err= 0: pid=190277: Fri Jan 15 05:23:59 2016
  read : io=427037MB, bw=60734KB/s, iops=15183, runt=7200001msec
    slat (usec): min=1, max=2002, avg= 9.83, stdev= 8.55
    clat (usec): min=19, max=64811, avg=515.26, stdev=170.23
     lat (usec): min=108, max=64817, avg=525.24, stdev=169.77
    clat percentiles (usec):
     |  1.00th=[  334],  5.00th=[  374], 10.00th=[  414], 20.00th=[  454],
     | 30.00th=[  474], 40.00th=[  486], 50.00th=[  490], 60.00th=[  502],
     | 70.00th=[  516], 80.00th=[  556], 90.00th=[  612], 95.00th=[  652],
     | 99.00th=[ 1064], 99.50th=[ 1288], 99.90th=[ 2736], 99.95th=[ 3056],
     | 99.99th=[ 6304]
    bw (KB  /s): min=    0, max=15808, per=25.01%, avg=15188.15, stdev=253.62
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.04%, 500=59.08%
    lat (usec) : 750=38.27%, 1000=1.39%
    lat (msec) : 2=1.00%, 4=0.20%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=0.91%, sys=2.50%, ctx=205610388, majf=0, minf=6486
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=109321549/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=190281: Fri Jan 15 05:23:59 2016
  read : io=465596MB, bw=66218KB/s, iops=16554, runt=7200001msec
    slat (usec): min=1, max=8032, avg=18.90, stdev=18.12
    clat (usec): min=1, max=61886, avg=461.99, stdev=144.42
     lat (usec): min=102, max=61892, avg=481.13, stdev=141.69
    clat percentiles (usec):
     |  1.00th=[  249],  5.00th=[  318], 10.00th=[  342], 20.00th=[  370],
     | 30.00th=[  398], 40.00th=[  434], 50.00th=[  462], 60.00th=[  478],
     | 70.00th=[  494], 80.00th=[  516], 90.00th=[  572], 95.00th=[  628],
     | 99.00th=[  900], 99.50th=[ 1064], 99.90th=[ 1448], 99.95th=[ 1944],
     | 99.99th=[ 5152]
    bw (KB  /s): min=    0, max=17552, per=25.01%, avg=16558.89, stdev=255.31
    lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=1.04%, 500=71.47%, 750=25.64%, 1000=1.18%
    lat (msec) : 2=0.63%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.37%, sys=4.36%, ctx=222678453, majf=0, minf=6915
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=119192576/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=190285: Fri Jan 15 05:23:59 2016
  read : io=477208MB, bw=67870KB/s, iops=16967, runt=7200001msec
    slat (usec): min=1, max=1623, avg=10.28, stdev= 8.79
    clat (usec): min=1, max=74353, avg=459.41, stdev=139.29
     lat (usec): min=100, max=74368, avg=469.84, stdev=138.49
    clat percentiles (usec):
     |  1.00th=[  274],  5.00th=[  330], 10.00th=[  350], 20.00th=[  370],
     | 30.00th=[  394], 40.00th=[  438], 50.00th=[  466], 60.00th=[  482],
     | 70.00th=[  490], 80.00th=[  506], 90.00th=[  556], 95.00th=[  620],
     | 99.00th=[  884], 99.50th=[ 1048], 99.90th=[ 1400], 99.95th=[ 1768],
     | 99.99th=[ 5280]
    bw (KB  /s): min=    0, max=17544, per=25.01%, avg=16971.85, stdev=228.05
    lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.60%
    lat (usec) : 500=76.19%, 750=21.49%, 1000=1.09%
    lat (msec) : 2=0.59%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.02%, sys=2.77%, ctx=228818803, majf=0, minf=6708
  IO depths    : 1=0.1%, 2=100.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=122165344/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: io=417059MB, aggrb=59314KB/s, minb=59314KB/s, maxb=59314KB/s, mint=7200001msec, maxt=7200001msec

Run status group 1 (all jobs):
   READ: io=427037MB, aggrb=60734KB/s, minb=60734KB/s, maxb=60734KB/s, mint=7200001msec, maxt=7200001msec

Run status group 2 (all jobs):
   READ: io=465596MB, aggrb=66218KB/s, minb=66218KB/s, maxb=66218KB/s, mint=7200001msec, maxt=7200001msec

Run status group 3 (all jobs):
   READ: io=477208MB, aggrb=67869KB/s, minb=67869KB/s, maxb=67869KB/s, mint=7200001msec, maxt=7200001msec



# 
# Fri 15 Jan 2016 05:24:01 AM EST
# 4kB randread with SATA 5 jobs and 16 I/Os Outstanding
# 4kB randread with NVMe 4 jobs and 4 I/Os Outstanding
# CentOS Linux release 7.2.1511  Linux 3.10.0-327.3.1.el7.x86_64
# 


Summary
SATA-R5-0: 30872 IOPS | 123 MBps | 2589 μsec Average Latency 
SATA-R5-1: 30580 IOPS | 122 MBps | 2614 μsec Average Latency 
NVMe-R5-0: 6689 IOPS | 26 MBps | 2389 μsec Average Latency 
NVMe-R5-1: 6825 IOPS | 27 MBps | 2342 μsec Average Latency 

Totals
IOPS: 74966
MBPS: 298
Average Latency (μsec): 2483

SATA-R5-01: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
SATA-R5-11: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
NVMe-R5-01: (g=2): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
...
NVMe-R5-11: (g=3): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
...
fio-2.1.10
Starting 18 processes

SATA-R5-01: (groupid=0, jobs=5): err= 0: pid=195490: Fri Jan 15 07:25:03 2016
  read : io=868277MB, bw=123488KB/s, iops=30872, runt=7200003msec
    slat (usec): min=2, max=14391, avg=16.70, stdev=16.53
    clat (usec): min=95, max=1741.3K, avg=2572.99, stdev=1508.95
     lat (usec): min=252, max=1741.3K, avg=2589.91, stdev=1508.70
    clat percentiles (usec):
     |  1.00th=[ 1816],  5.00th=[ 2040], 10.00th=[ 2160], 20.00th=[ 2288],
     | 30.00th=[ 2352], 40.00th=[ 2448], 50.00th=[ 2512], 60.00th=[ 2608],
     | 70.00th=[ 2704], 80.00th=[ 2832], 90.00th=[ 3024], 95.00th=[ 3216],
     | 99.00th=[ 3760], 99.50th=[ 4080], 99.90th=[ 5600], 99.95th=[ 7008],
     | 99.99th=[10688]
    bw (KB  /s): min=    0, max=30768, per=20.01%, avg=24705.21, stdev=588.15
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=3.73%, 4=95.67%, 10=0.57%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=1.70%, sys=5.67%, ctx=374624098, majf=0, minf=21080
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=222278964/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
SATA-R5-11: (groupid=1, jobs=5): err= 0: pid=195495: Fri Jan 15 07:25:03 2016
  read : io=860074MB, bw=122322KB/s, iops=30580, runt=7200003msec
    slat (usec): min=1, max=4477, avg=12.09, stdev=11.27
    clat (usec): min=167, max=1731.2K, avg=2602.71, stdev=1584.37
     lat (usec): min=231, max=1731.4K, avg=2614.94, stdev=1584.27
    clat percentiles (usec):
     |  1.00th=[ 1832],  5.00th=[ 2064], 10.00th=[ 2192], 20.00th=[ 2320],
     | 30.00th=[ 2384], 40.00th=[ 2480], 50.00th=[ 2576], 60.00th=[ 2640],
     | 70.00th=[ 2736], 80.00th=[ 2864], 90.00th=[ 3056], 95.00th=[ 3248],
     | 99.00th=[ 3760], 99.50th=[ 4048], 99.90th=[ 5536], 99.95th=[ 7072],
     | 99.99th=[10816]
    bw (KB  /s): min=    0, max=25928, per=20.01%, avg=24474.98, stdev=548.22
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=3.18%, 4=96.25%, 10=0.54%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=1.34%, sys=4.26%, ctx=375723170, majf=0, minf=21573
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=220178995/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
NVMe-R5-01: (groupid=2, jobs=4): err= 0: pid=195500: Fri Jan 15 07:25:03 2016
  read : io=188143MB, bw=26758KB/s, iops=6689, runt=7200002msec
    slat (usec): min=1, max=4451, avg=18.58, stdev=18.06
    clat (usec): min=11, max=1757.8K, avg=2370.78, stdev=1472.92
     lat (usec): min=172, max=1757.8K, avg=2389.61, stdev=1472.73
    clat percentiles (usec):
     |  1.00th=[ 1592],  5.00th=[ 1832], 10.00th=[ 1944], 20.00th=[ 2064],
     | 30.00th=[ 2160], 40.00th=[ 2256], 50.00th=[ 2320], 60.00th=[ 2416],
     | 70.00th=[ 2512], 80.00th=[ 2640], 90.00th=[ 2832], 95.00th=[ 3024],
     | 99.00th=[ 3536], 99.50th=[ 3792], 99.90th=[ 5216], 99.95th=[ 6688],
     | 99.99th=[10304]
    bw (KB  /s): min=    0, max= 8232, per=25.02%, avg=6694.82, stdev=163.81
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
    lat (usec) : 750=0.01%, 1000=0.03%
    lat (msec) : 2=14.11%, 4=85.51%, 10=0.32%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, 2000=0.01%
  cpu          : usr=0.68%, sys=1.80%, ctx=92761111, majf=0, minf=9401
  IO depths    : 1=0.1%, 2=0.1%, 4=100.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=48164554/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4
NVMe-R5-11: (groupid=3, jobs=4): err= 0: pid=195504: Fri Jan 15 07:25:03 2016
  read : io=191953MB, bw=27300KB/s, iops=6825, runt=7200001msec
    slat (usec): min=1, max=2676, avg=11.52, stdev=11.76
    clat (usec): min=28, max=1716.5K, avg=2331.16, stdev=1440.16
     lat (usec): min=120, max=1716.5K, avg=2342.81, stdev=1440.08
    clat percentiles (usec):
     |  1.00th=[ 1544],  5.00th=[ 1784], 10.00th=[ 1880], 20.00th=[ 2024],
     | 30.00th=[ 2128], 40.00th=[ 2224], 50.00th=[ 2288], 60.00th=[ 2384],
     | 70.00th=[ 2480], 80.00th=[ 2608], 90.00th=[ 2800], 95.00th=[ 2992],
     | 99.00th=[ 3472], 99.50th=[ 3760], 99.90th=[ 5088], 99.95th=[ 6624],
     | 99.99th=[10304]
    bw (KB  /s): min=    0, max=12200, per=25.02%, avg=6830.31, stdev=179.43
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.02%
    lat (usec) : 1000=0.03%
    lat (msec) : 2=18.03%, 4=81.61%, 10=0.29%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01%
  cpu          : usr=0.49%, sys=1.25%, ctx=91065879, majf=0, minf=8596
  IO depths    : 1=0.1%, 2=0.1%, 4=100.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=49140025/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: io=868277MB, aggrb=123488KB/s, minb=123488KB/s, maxb=123488KB/s, mint=7200003msec, maxt=7200003msec

Run status group 1 (all jobs):
   READ: io=860074MB, aggrb=122321KB/s, minb=122321KB/s, maxb=122321KB/s, mint=7200003msec, maxt=7200003msec

Run status group 2 (all jobs):
   READ: io=188143MB, aggrb=26758KB/s, minb=26758KB/s, maxb=26758KB/s, mint=7200002msec, maxt=7200002msec

Run status group 3 (all jobs):
   READ: io=191953MB, aggrb=27300KB/s, minb=27300KB/s, maxb=27300KB/s, mint=7200001msec, maxt=7200001msec