Shell's Home

Jan 23, 2014 - 7 minute read - Comments

lxc和virtualbox和物理机的简单性能测试和对比

说明

测试各种虚拟化系统下的虚拟机性能。

测试使用sysbench。

CPU采用如下指令测试。

sysbench --test=cpu --num-threads=2 --cpu-max-prime=50000 run

文件IO采用如下指令测试。

sysbench --test=fileio --file-total-size=10G prepare
sysbench --test=fileio --file-total-size=10G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run

内存采用如下指令测试。

sysbench --test=memory --num-threads=2 --memory-access-mode=seq run
sysbench --test=memory --num-threads=2 --memory-access-mode=rnd run

线程采用如下指令。

sysbench --test=threads --num-threads=2 run
sysbench --test=mutex --num-threads=2 --mutex-locks=1000000 run

裸硬盘测试采用如下指令。

hdparm -tT <dev>

物理机上有三个文件系统,ext4/xfs/btrfs,前两者仅做fileio测试以对比性能。

另外做两个特殊文件系统对比,aufs带复制和aufs无复制。前者在只读层上准备好测试文件,而后进行随机读写测试。其中就附带了文件复制开销。后者在aufs建立后初始化测试文件,因此消除了文件复制开销。

所有测试都是测试数次,取最高者(因为低者可能受到各种干扰)。一般是2-3次。

物理机是一台DELL Intel 64位桌面系统,支持硬件虚拟化,有4G内存。系统采用debian jessie,测试于2014年1月17日-20日执行,内核3.12.6-2 (2013-12-29) x86_64。

虚拟机lxc是使用lxc切分的一台虚拟机,没有做资源限制。

虚拟机vbox是使用virtualbox切分的一台虚拟机,分配了所有CPU,打开了硬件虚拟化,分配了1G内存。

文件系统

ext4

Operations performed: 21311 Read, 14207 Write, 45440 Other = 80958 Total Read 332.98Mb Written 221.98Mb Total transferred 554.97Mb (1.8499Mb/sec) 118.39 Requests/sec executed

Test execution summary: total time: 300.0044s total number of events: 35518 total time taken by event execution: 168.4761 per-request statistics: min: 0.00ms avg: 4.74ms max: 118.67ms approx. 95 percentile: 12.48ms

Threads fairness: events (avg/stddev): 35518.0000/0.00 execution time (avg/stddev): 168.47610.00

xfs

Operations performed: 20789 Read, 13859 Write, 44288 Other = 78936 Total Read 324.83Mb Written 216.55Mb Total transferred 541.38Mb (1.8046Mb/sec) 115.49 Requests/sec executed

Test execution summary: total time: 300.0018s total number of events: 34648 total time taken by event execution: 172.0475 per-request statistics: min: 0.00ms avg: 4.97ms max: 96.11ms approx. 95 percentile: 12.30ms

Threads fairness: events (avg/stddev): 34648.0000/0.00 execution time (avg/stddev): 172.0475/0.00

btrfs

Operations performed: 6180 Read, 4120 Write, 13105 Other = 23405 Total Read 96.562Mb Written 64.375Mb Total transferred 160.94Mb (549.23Kb/sec) 34.33 Requests/sec executed

Test execution summary: total time: 300.0556s total number of events: 10300 total time taken by event execution: 65.8914 per-request statistics: min: 0.00ms avg: 6.40ms max: 337.28ms approx. 95 percentile: 17.01ms

Threads fairness: events (avg/stddev): 10300.0000/0.00 execution time (avg/stddev): 65.89140.00

aufs透明

Operations performed: 5340 Read, 3560 Write, 11279 Other = 20179 Total Read 83.438Mb Written 55.625Mb Total transferred 139.06Mb (474.65Kb/sec) 29.67 Requests/sec executed

Test execution summary: total time: 300.0084s total number of events: 8900 total time taken by event execution: 32.3634 per-request statistics: min: 0.00ms avg: 3.64ms max: 1037.04ms approx. 95 percentile: 0.02ms

Threads fairness: events (avg/stddev): 8900.0000/0.00 execution time (avg/stddev): 32.36340.00

aufs非透明

Operations performed: 20320 Read, 13546 Write, 43264 Other = 77130 Total Read 317.5Mb Written 211.66Mb Total transferred 529.16Mb (1.7638Mb/sec) 112.88 Requests/sec executed

Test execution summary: total time: 300.0054s total number of events: 33866 total time taken by event execution: 170.7252 per-request statistics: min: 0.00ms avg: 5.04ms max: 143.86ms approx. 95 percentile: 12.62ms

Threads fairness: events (avg/stddev): 33866.0000/0.00 execution time (avg/stddev): 170.72520.00

物理机

hdparm

Timing cached reads: 11980 MB in 2.00 seconds = 5992.56 MB/sec Timing buffered disk reads: 366 MB in 3.01 seconds = 121.52 MB/sec

cpu

Test execution summary: total time: 51.4463s total number of events: 10000 total time taken by event execution: 102.8828 per-request statistics: min: 9.93ms avg: 10.29ms max: 36.11ms approx. 95 percentile: 11.29ms

Threads fairness: events (avg/stddev): 5000.0000/30.00 execution time (avg/stddev): 51.44140.00

memory

Operations performed: 104857600 (4545662.48 ops/sec)

102400.00 MB transferred (4439.12 MB/sec)

Test execution summary: total time: 23.0676s total number of events: 104857600 total time taken by event execution: 34.1991 per-request statistics: min: 0.00ms avg: 0.00ms max: 18.05ms approx. 95 percentile: 0.00ms

Threads fairness: events (avg/stddev): 52428800.0000/69790.00 execution time (avg/stddev): 17.0995/0.01

Operations performed: 104857600 (5407739.22 ops/sec)

102400.00 MB transferred (5281.00 MB/sec)

Test execution summary: total time: 19.3903s total number of events: 104857600 total time taken by event execution: 26.8579 per-request statistics: min: 0.00ms avg: 0.00ms max: 22.46ms approx. 95 percentile: 0.00ms

Threads fairness: events (avg/stddev): 52428800.0000/7211.00 execution time (avg/stddev): 13.42890.14

threads

Test execution summary: total time: 1.0112s total number of events: 10000 total time taken by event execution: 2.0210 per-request statistics: min: 0.15ms avg: 0.20ms max: 11.19ms approx. 95 percentile: 0.24ms

Threads fairness: events (avg/stddev): 5000.0000/3.00 execution time (avg/stddev): 1.0105/0.00

Test execution summary: total time: 0.1665s total number of events: 2 total time taken by event execution: 0.3238 per-request statistics: min: 157.39ms avg: 161.90ms max: 166.41ms approx. 95 percentile: 10000000.00ms

Threads fairness: events (avg/stddev): 1.0000/0.00 execution time (avg/stddev): 0.16190.00

nginx

Concurrency Level: 10000 Time taken for tests: 5.745 seconds Complete requests: 100000 Failed requests: 0 Write errors: 0 Total transferred: 172100000 bytes HTML transferred: 160000000 bytes Requests per second: 17404.93 [#/sec] (mean) Time per request: 574.550 [ms] (mean) Time per request: 0.057 [ms] (mean, across all concurrent requests) Transfer rate: 29251.85 [Kbytes/sec] received

lxc

cpu

Test execution summary: total time: 51.4368s total number of events: 10000 total time taken by event execution: 102.8619 per-request statistics: min: 9.92ms avg: 10.29ms max: 35.08ms approx. 95 percentile: 11.68ms

Threads fairness: events (avg/stddev): 5000.0000/5.00 execution time (avg/stddev): 51.43100.00

fileio

Operations performed: 5548 Read, 3698 Write, 11776 Other = 21022 Total Read 86.688Mb Written 57.781Mb Total transferred 144.47Mb (493.07Kb/sec) 30.82 Requests/sec executed

Test execution summary: total time: 300.0294s total number of events: 9246 total time taken by event execution: 84.4687 per-request statistics: min: 0.01ms avg: 9.14ms max: 394.13ms approx. 95 percentile: 36.20ms

Threads fairness: events (avg/stddev): 9246.0000/0.00 execution time (avg/stddev): 84.46870.00

memory

Operations performed: 104857600 (4456398.83 ops/sec)

102400.00 MB transferred (4351.95 MB/sec)

Test execution summary: total time: 23.5297s total number of events: 104857600 total time taken by event execution: 34.8417 per-request statistics: min: 0.00ms avg: 0.00ms max: 20.22ms approx. 95 percentile: 0.00ms

Threads fairness: events (avg/stddev): 52428800.0000/155952.00 execution time (avg/stddev): 17.42080.06

Operations performed: 104857600 (5327923.43 ops/sec)

102400.00 MB transferred (5203.05 MB/sec)

Test execution summary: total time: 19.6808s total number of events: 104857600 total time taken by event execution: 27.3010 per-request statistics: min: 0.00ms avg: 0.00ms max: 16.48ms approx. 95 percentile: 0.00ms

Threads fairness: events (avg/stddev): 52428800.0000/297738.00 execution time (avg/stddev): 13.65050.09

threads

Test execution summary: total time: 1.2490s total number of events: 10000 total time taken by event execution: 2.4954 per-request statistics: min: 0.21ms avg: 0.25ms max: 7.39ms approx. 95 percentile: 0.28ms

Threads fairness: events (avg/stddev): 5000.0000/7.00 execution time (avg/stddev): 1.24770.00

Test execution summary: total time: 0.1222s total number of events: 2 total time taken by event execution: 0.2275 per-request statistics: min: 107.53ms avg: 113.77ms max: 120.02ms approx. 95 percentile: 10000000.00ms

Threads fairness: events (avg/stddev): 1.0000/0.00 execution time (avg/stddev): 0.11380.01

nginx

Concurrency Level: 10000 Time taken for tests: 16.976 seconds Complete requests: 100000 Failed requests: 0 Write errors: 0 Total transferred: 1551500000 bytes HTML transferred: 1539400000 bytes Requests per second: 5890.69 [#/sec] (mean) Time per request: 1697.594 [ms] (mean) Time per request: 0.170 [ms] (mean, across all concurrent requests) Transfer rate: 89252.02 [Kbytes/sec] received

vbox

hdparm

Timing cached reads: 10122 MB in 1.99 seconds = 5088.70 MB/sec Timing buffered disk reads: 300 MB in 3.00 seconds = 99.87 MB/sec

cpu

Test execution summary: total time: 54.0469s total number of events: 10000 total time taken by event execution: 108.0595 per-request statistics: min: 9.03ms avg: 10.81ms max: 61.39ms approx. 95 percentile: 14.87ms

Threads fairness: events (avg/stddev): 5000.0000/87.00 execution time (avg/stddev): 54.0297/0.00

fileio

Operations performed: 68153 Read, 45435 Write, 145280 Other = 258868 Total Read 1.0399Gb Written 709.92Mb Total transferred 1.7332Gb (5.916Mb/sec) 378.62 Requests/sec executed

Test execution summary: total time: 300.0045s total number of events: 113588 total time taken by event execution: 177.2314 per-request statistics: min: 0.01ms avg: 1.56ms max: 579.66ms approx. 95 percentile: 13.10ms

Threads fairness: events (avg/stddev): 113588.0000/0.00 execution time (avg/stddev): 177.23140.00

memory

一直报错,测试不出来。

threads

Test execution summary: total time: 16.0377s total number of events: 10000 total time taken by event execution: 32.0421 per-request statistics: min: 1.19ms avg: 3.20ms max: 38.39ms approx. 95 percentile: 5.74ms

Threads fairness: events (avg/stddev): 5000.0000/15.00 execution time (avg/stddev): 16.0210/0.00

Test execution summary: total time: 0.3023s total number of events: 2 total time taken by event execution: 0.5990 per-request statistics: min: 297.52ms avg: 299.50ms max: 301.47ms approx. 95 percentile: 10000000.00ms

Threads fairness: events (avg/stddev): 1.0000/0.00 execution time (avg/stddev): 0.29950.00

nginx

Concurrency Level: 5000 Time taken for tests: 41.758 seconds Complete requests: 50000 Failed requests: 0 Write errors: 0 Keep-Alive requests: 0 Total transferred: 890350000 bytes HTML transferred: 884250000 bytes Requests per second: 1197.38 [#/sec] (mean) Time per request: 4175.799 [ms] (mean) Time per request: 0.835 [ms] (mean, across all concurrent requests) Transfer rate: 20821.94 [Kbytes/sec] received

vbox on lvm

在物理机上开辟一个lvm卷,然后用vmdk引用物理卷的功能挂到vbox上使用,格式化为ext4文件格式。

hdparm

Timing cached reads: 10034 MB in 1.99 seconds = 5044.30 MB/sec Timing buffered disk reads: 270 MB in 3.01 seconds = 89.73 MB/sec

fileio

Operations performed: 22765 Read, 15176 Write, 48512 Other = 86453 Total Read 355.7Mb Written 237.12Mb Total transferred 592.83Mb (1.976Mb/sec) 126.47 Requests/sec executed

Test execution summary: total time: 300.0080s total number of events: 37941 total time taken by event execution: 286.5402 per-request statistics: min: 0.01ms avg: 7.55ms max: 336.67ms approx. 95 percentile: 20.49ms

Threads fairness: events (avg/stddev): 37941.0000/0.00 execution time (avg/stddev): 286.54020.00

vbox on vdi

基本同vbox on lvm,不过使用vdi作为存储。和vbox测试相比,内部系统换为主系统同样的debian。

hdparm

Timing cached reads: 9152 MB in 1.99 seconds = 4598.80 MB/sec Timing buffered disk reads: 352 MB in 3.00 seconds = 117.16 MB/sec

fileio

Operations performed: 151020 Read, 100680 Write, 322060 Other = 573760 Total Read 2.3044Gb Written 1.5363Gb Total transferred 3.8406Gb (13.109Mb/sec) 839.00 Requests/sec executed

Test execution summary: total time: 300.0011s total number of events: 251700 total time taken by event execution: 60.6296 per-request statistics: min: 0.01ms avg: 0.24ms max: 106.94ms approx. 95 percentile: 0.26ms

Threads fairness: events (avg/stddev): 251700.0000/0.00 execution time (avg/stddev): 60.62960.00

分析

计算

从CPU上说,vbox的消耗大约是5%,而lxc的消耗基本是0%。

内存测试上,vbox无法测量。lxc的性能和物理机十分相近,两者的差异在1.5%-2%之间。

在threads测试和mutex测试上,vbox显示出远远低于lxc的性能。这些基本都是内核陷入类的事务,也是预期vbox会发生性能下降的地方。

IO

从硬件设备IO上来分析,lxc和主机是共用一套物理设备的。vbox的裸设备IO比物理机低了15-18%。但是文件系统IO则表现出完全相反的景象。

lxc底层使用的是btrfs,因此性能和物理机上的btrfs性能十分相近,误差在10%以内。这一性能比物理机上的ext4/xfs低了70%以上。这个表现出btrfs的性能和ext4/xfs性能的差异(注:怎么会差这么多?)。

如果使用aufs的话,则要视复制特性而定。如果引发复制,性能会跌到和btrfs相近。而不引发复制的话,则和aufs下面的文件系统相近(误差在5%以内)。

而vbox内只有ext4,其性能高达物理机的220%。怎么可能?

network

从nginx的rps来说,lxc的性能只有物理机的1/3,vbox的更只有7%。而且vbox连10000并发都无法支撑,只能用5000并发测试。

这里固然有因为物理机目录中文件比较少的原因,但是vbox和lxc的文件数是一样的,两者性能比高达1:5是个不争的事实。

加测

因为vbox内的ext4性能太好,怀疑有鬼,所以加测了一下。果然,使用vdi后fileio性能比物理机还好(我用的是新的vdi)。这个结论在大规模读写和老的vdi上很可能退化成一点都不靠谱,否则所有的设备都应该先装一堆虚拟机做vdi。

结论

虚拟化层级

从虚拟化的深度来说,CPU虚拟化最深,完全虚拟化次之,半虚拟化其后,硬件虚拟化和半虚拟化难分伯仲,操作系统虚拟化已经很浅了。下面是简单解说。

  1. CPU虚拟化:使用CPU仿真另一块CPU的每个指令。速度最慢(大约是真实CPU的1/60),可以仿真其他CPU。bochs为其代表,qemu亦支持。
  2. 完全虚拟化:对另一个系统的部分核心调用进行处理。速度次之(大约一半略快)。必须是同种CPU,但可以是不同系统。vmware早期的虚拟机都是此种。
  3. 半虚拟化:修改另一个系统的核心代码,使其与hypervisor交互以提高性能(大约5-10%)。必须同种CPU,但是guest系统必须可以修改(排除了windows)。Xen为其代表。
  4. 硬件虚拟化:利用硬件加速完全虚拟化,使其性能接近半虚拟化,但是不需要修改内核。必须同种CPU,可以为不同系统。目前大部分虚拟机都支持。
  5. 系统虚拟化:在系统上完成另一个系统的特性。必须是同种CPU同种内核,但可以是不同系统(例如debian和centos)。linux上的lxc/openvz,bsd上的jail,windows上的Virtuozzo为其代表。
  6. 用户隔离:在同一个系统上,利用用户分割权限和限制配额。必须在同一个系统内。DAE为其代表。
  7. 沙盒:在语言内部搭建隔离平台,对API进行鉴定和抽象。GAE为其代表。

性能和隔离性的取舍

上面显然可见,隔离性越好,性能越差。使用硬件虚拟化后,nginx的rps只有原本的1/10。lxc的overload明明很低,但是因为用了虚拟网卡,所以rps下降到1/3。从效率上说,当然不希望为了虚拟化而消耗大量的资源。所以,如果服务原本可以用户隔离,就不要用系统虚拟化,如果可以系统虚拟化,就不要硬件虚拟化。所以在大规模使用虚拟化之前,不妨考虑一下是不是先好用户权限级隔离比较合适。

文件系统

从上面可以看出,对于lxc这种小消耗的虚拟方案,与其在意虚拟机的性能消耗,不如更在意文件系统的性能消耗。但是比较倒霉的是,lxc是不能在虚拟机内自行配置文件系统的,需要从主机内分配挂载。

  • ext4:适用于大部分情况,iops很高,小文件下吞吐量很不错。大部分虚拟化场景都是小系统简单服务的时候适合。
  • xfs:适用于大设备内部的大虚拟机。每个虚机的服务复杂,或者是有大文件。
  • btrfs:这个东西性能很低,但是写时复制的能力对快速clone很重要。适合用于产生一堆临时生成的环境,用于程序员测试或者测试工程师搭环境,测试完了就删除的。
  • aufs:和btrfs情况差不多,快速clone不错。不过安全起见,clone后原image不可以启动实例或者修改image。在没有COW写入时性能很高,这点比btrfs好。而在COW时瞬时开销很高。