
Results overview
As we know about the number of requests we can expect from a single WordPress front server, it’s time to try to scale out.
Load balancer configuration
I will use haproxy, and as the post HAProxy – Experimental evaluation of the performance shows it, I can setup a small VM with 1vcpu and 512MB.
As WordPress is a stateless product, I don’t need to manage session persistence so a very basic configuration can be used (see Software Installation)
Rem:
In a following post, I will show how to share sessions on an NFS server. For the need of sessions, I will install a e-commerce plugin: WooCommerce
I also deactivated the cache but kept APC.
Test #19, APC activated, 2 Fronts, 4Vcpu, 2GB on servers ade-esxi-02 and ade-esxi-01 (Intel(R) Core(TM) Quad CPU Q6600 @ 2.40GHz; No Hyperthreading; 8 GB DDR2 800) :
I commented out server3 and server4 in haproxy configuration as followed
server tu-web-01 192.168.100.210:80 check
server tu-web-02 192.168.100.218:80 check
#server tu-web-03 192.168.100.172:80 check
#server tu-web-04 192.168.100.167:80 check
$ ab -kc 10 -n 100 http://tu-lb-01/mysite/
Requests per second: 18.89 [#/sec] (mean)
This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking tu-lb-01 (be patient).....done
Server Software: Apache/2.2.22
Server Hostname: tu-lb-01
Server Port: 80
Document Path: /mysite/
Document Length: 47979 bytes
Concurrency Level: 10
Time taken for tests: 5.295 seconds
Complete requests: 100
Failed requests: 93
(Connect: 0, Receive: 0, Length: 93, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 0
Total transferred: 4853880 bytes
HTML transferred: 4798680 bytes
Requests per second: 18.89 [#/sec] (mean)
Time per request: 529.450 [ms] (mean)
Time per request: 52.945 [ms] (mean, across all concurrent requests)
Transfer rate: 895.29 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 331 504 86.7 512 691
Waiting: 220 372 73.9 383 561
Total: 331 504 86.7 512 691
Percentage of the requests served within a certain time (ms)
50% 512
66% 564
75% 578
80% 584
90% 619
95% 634
98% 661
99% 691
100% 691 (longest request)
CPU usage (sum of user and system usage) on the 3 servers are shown here
server |
Max CPU |
load balancer |
4% |
front1 |
86% |
front2 |
80% |
Rem:
I guess that I didn’t hit the 100% on the front because I used only 10 concurrent access. Testing 3 fronts with 30 simultaneous connections gives almost 100% on each front
dstat result on load balancer
-------cpu0-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---load-avg--- ---procs--- ----swap--- ----system----
usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free| 1m 5m 15m |run blk new| used free| time
0 0 98 0 0 2: 0 0 98 0 0 2|1079k 1048k| 0 0 | 776 476 |63.5M 21.2M 67.2M 340M| 0 0.02 0.05| 0 0 0| 0 508M|23-08 16:07:12
1 0 98 0 0 1: 1 0 98 0 0 1|1027k 997k| 0 0 | 852 456 |63.5M 21.2M 67.2M 340M| 0 0.02 0.05| 0 0 0| 0 508M|23-08 16:07:13
0 0 99 0 0 1: 0 0 99 0 0 1|1093k 1063k| 0 0 | 911 470 |63.5M 21.2M 67.2M 340M| 0 0.02 0.05| 0 0 0| 0 508M|23-08 16:07:14
1 2 96 0 0 1: 1 2 96 0 0 1|1094k 1066k| 0 0 | 894 472 |63.5M 21.2M 67.2M 340M| 0 0.02 0.05|1.0 0 0| 0 508M|23-08 16:07:15
1 1 98 0 0 0: 1 1 98 0 0 0| 815k 799k| 0 12k| 653 401 |63.5M 21.2M 67.2M 340M| 0 0.02 0.05| 0 0 0| 0 508M|23-08 16:07:16
1 0 99 0 0 0: 1 0 99 0 0 0| 260B 1012B| 0 0 | 29 46 |63.5M 21.2M 67.2M 340M| 0 0.02 0.05| 0 0 0| 0 508M|23-08 16:07:17
dstat result on front1
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---load-avg--- ---procs--->
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free| 1m 5m 15m |run blk new>
31 3 64 0 0 2: 22 1 77 0 0 0: 21 3 76 0 0 0: 20 3 77 0 0 0: 23 3 74 0 0 0| 456k 65k| 0 0 | 808 780 | 280M 27.1M 158M 1538M|0.99 0.42 0.43|5.0 0 0>
85 9 6 0 0 0: 76 9 14 1 0 0: 63 8 28 0 0 0: 73 11 15 0 0 0: 75 9 16 0 0 0|1138k 692k| 0 0 |1747 1419 | 288M 27.1M 158M 1530M|1.39 0.51 0.46|6.0 0 0>
78 9 10 1 0 2: 70 5 25 0 0 0: 63 10 27 0 0 0: 73 9 17 0 0 0: 71 8 20 0 0 0| 996k 569k| 0 0 |1671 1375 | 285M 27.1M 158M 1533M|1.39 0.51 0.46|5.0 0 0>
82 7 9 0 0 2: 73 11 17 0 0 0: 71 7 21 0 0 0: 71 11 17 1 0 0: 75 9 16 0 0 1|1135k 682k| 0 0 |1910 1405 | 281M 27.1M 158M 1537M|1.39 0.51 0.46|5.0 0 1.0>
81 10 6 0 0 3: 73 9 16 1 0 0: 68 15 15 1 0 0: 75 10 15 0 0 0: 74 11 14 1 0 1|1161k 634k| 0 20k|2406 1546 | 289M 27.1M 158M 1528M|1.39 0.51 0.46|6.0 0 0>
71 6 19 1 0 3: 45 10 44 1 0 0: 49 7 43 0 0 0: 57 5 37 1 0 0: 56 7 36 1 0 1| 713k 550k| 0 16k|1794 1046 | 250M 27.1M 158M 1568M|1.39 0.51 0.46| 0 0 0>
dstat result on front2
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---procs--->
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free|run blk new>
40 4 55 0 0 1: 35 4 61 0 0 0: 38 4 58 0 0 0: 35 3 62 0 0 0: 37 4 59 0 0 0| 656k 305k| 0 0 |1023 805 | 268M 27.1M 208M 1500M|3.0 0 0>
81 11 4 0 0 4: 45 4 51 0 0 0: 76 6 17 1 0 0: 78 9 13 0 0 0: 70 8 22 0 0 1|1117k 607k| 0 0 |1857 1546 | 283M 27.1M 208M 1484M|4.0 0 0>
76 11 10 0 0 2: 74 6 19 1 0 0: 73 6 21 0 0 0: 49 6 44 0 0 0: 68 7 23 0 0 1|1056k 609k| 0 16k|1813 1454 | 293M 27.1M 208M 1475M|4.0 0 0>
83 7 6 0 0 4: 71 11 18 0 0 0: 56 6 38 0 0 0: 60 6 34 0 0 0: 67 8 24 0 0 1|1122k 618k| 0 0 |2644 1480 | 292M 27.1M 209M 1476M|5.0 0 0>
82 7 6 0 0 5: 68 7 25 0 0 0: 60 5 34 0 0 1: 71 13 15 0 0 0: 71 8 20 0 0 1|1133k 648k| 0 8192B|2535 1574 | 280M 27.1M 209M 1488M|4.0 0 0>
47 5 46 1 0 1: 31 3 66 0 0 0: 39 5 56 0 0 0: 14 1 85 0 0 0: 33 4 63 0 0 0| 505k 382k| 0 0 |1454 829 | 251M 27.1M 209M 1517M| 0 0 0>
Test #20, APC activated, 3 Fronts
- 2 fronts 4Vcpu, 2GB on servers ade-esxi-02 and ade-esxi-01 (Intel(R) Core(TM) Quad CPU Q6600 @ 2.40GHz; No Hyperthreading; 8 GB DDR2 800)
- 1 front 2Vcpu, 2GB on server ade-esxi-03 (Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz; Hyperthreading active; 8 GB DDR3 1066)
I commented out server4 in haproxy configuration as followed
server tu-web-01 192.168.100.210:80 check
server tu-web-02 192.168.100.218:80 check
server tu-web-03 192.168.100.172:80 check
#server tu-web-04 192.168.100.167:80 check
$ ab -kc 30 -n 200 http://tu-lb-01/mysite/
Requests per second: 28.51 [#/sec] (mean)
This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking tu-lb-01 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests
Server Software: Apache/2.2.22
Server Hostname: tu-lb-01
Server Port: 80
Document Path: /mysite/
Document Length: 47980 bytes
Concurrency Level: 30
Time taken for tests: 7.016 seconds
Complete requests: 200
Failed requests: 187
(Connect: 0, Receive: 0, Length: 187, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 0
Total transferred: 9707290 bytes
HTML transferred: 9596890 bytes
Requests per second: 28.51 [#/sec] (mean)
Time per request: 1052.358 [ms] (mean)
Time per request: 35.079 [ms] (mean, across all concurrent requests)
Transfer rate: 1351.22 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.3 0 3
Processing: 376 976 478.6 766 2201
Waiting: 280 691 319.9 562 1490
Total: 376 976 478.6 766 2201
Percentage of the requests served within a certain time (ms)
50% 766
66% 1082
75% 1327
80% 1507
90% 1741
95% 1900
98% 2068
99% 2189
100% 2201 (longest request)
CPU usage (sum of user and system usage) on the 4 servers are shown here
server |
Max CPU |
load balancer |
16% |
front1 |
98% |
front2 |
99% |
front2 |
100% |
dstat result on load balancer
-------cpu0-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---load-avg--- ---procs--- ----swap--- ----system----
usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free| 1m 5m 15m |run blk new| used free| time
0 2 97 0 0 1: 0 2 97 0 0 1|1435k 1402k| 0 0 |1287 617 |67.2M 21.7M 67.3M 335M|0.19 0.15 0.10| 0 0 0| 0 508M|23-08 16:36:11
0 0 99 0 0 1: 0 0 99 0 0 1|1385k 1346k| 0 0 |1299 601 |67.2M 21.7M 67.3M 335M|0.19 0.15 0.10| 0 0 0| 0 508M|23-08 16:36:12
1 1 96 0 0 2: 1 1 96 0 0 2|1521k 1483k| 0 0 |1367 722 |67.4M 21.7M 67.3M 335M|0.19 0.15 0.10| 0 0 1.0| 0 508M|23-08 16:36:13
0 4 84 9 1 3: 0 4 84 9 1 3|1603k 1565k| 0 24k|1421 724 |67.4M 21.7M 67.3M 335M|0.19 0.15 0.10| 0 0 0| 0 508M|23-08 16:36:14
1 4 91 0 0 4: 1 4 91 0 0 4|1503k 1470k| 0 0 |1351 620 |67.4M 21.7M 67.3M 335M|0.17 0.15 0.10| 0 0 0| 0 508M|23-08 16:36:15
1 4 92 0 0 3: 1 4 92 0 0 3|1368k 1336k| 0 0 |1080 527 |67.3M 21.7M 67.3M 335M|0.17 0.15 0.10| 0 0 0| 0 508M|23-08 16:36:16
dstat result on front1
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---load-avg--- ---procs--->
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free| 1m 5m 15m |run blk new>
79 15 4 0 0 2: 79 13 8 0 0 0: 78 15 7 0 0 0: 83 13 4 0 0 0: 80 14 6 0 0 1| 950k 635k| 0 16k|1789 1522 | 264M 27.4M 160M 1552M|3.70 1.68 0.92|4.0 0 1.0>
82 15 1 0 0 2: 81 16 3 0 0 0: 92 7 1 0 0 0: 85 13 2 0 0 0: 85 13 2 0 0 1|1232k 692k| 0 0 |1966 1707 | 305M 27.4M 160M 1511M|3.97 1.77 0.95|8.0 0 2.0>
88 8 2 0 0 2: 86 10 4 0 0 0: 84 11 5 0 0 0: 87 9 4 0 0 0: 86 10 4 0 0 1|1226k 677k| 0 0 |2220 1627 | 324M 27.4M 160M 1491M|3.97 1.77 0.95|8.0 0 0>
87 6 6 0 0 1: 79 11 9 1 0 0: 80 9 10 1 0 0: 78 10 12 0 0 0: 81 9 9 1 0 0|1182k 682k| 0 0 |2372 1474 | 305M 27.4M 160M 1510M|3.97 1.77 0.95|5.0 0 0>
81 10 5 0 0 4: 82 10 8 0 0 0: 80 9 11 0 0 0: 81 10 9 0 0 0: 81 10 8 0 0 1|1164k 701k| 0 0 |2374 1470 | 297M 27.4M 160M 1518M|3.97 1.77 0.95|7.0 0 0>
77 12 8 0 0 3: 76 10 14 0 0 0: 78 12 9 1 0 0: 78 9 12 1 0 0: 77 11 11 1 0 1|1114k 681k| 0 16k|2336 1527 | 292M 27.4M 160M 1524M|3.97 1.77 0.95|4.0 0 0>
22 5 71 1 0 1: 11 1 88 0 0 0: 4 1 95 0 0 0: 13 0 87 0 0 0: 13 2 85 0 0 0| 159k 186k| 0 0 | 579 274 | 254M 27.4M 160M 1562M|3.81 1.77 0.96| 0 0 0>
dstat result on front2
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---procs--->
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free|run blk new>
71 7 20 0 0 2: 70 6 23 0 0 0: 71 8 21 0 0 0: 67 11 22 0 0 0: 70 8 22 0 0 1|1159k 540k| 0 0 |1740 1422 | 296M 27.4M 210M 1470M|8.0 1.0 1.0>
82 8 1 0 0 9: 89 10 1 0 0 0: 86 12 2 0 0 0: 88 10 2 0 0 0: 86 10 1 0 0 2|1233k 630k| 0 0 |2657 1666 | 294M 27.4M 210M 1472M|7.0 0 2.0>
78 13 0 0 0 9: 83 16 1 0 0 0: 79 19 2 0 0 0: 84 15 1 0 0 0: 81 15 1 0 0 2|1128k 636k| 0 0 |2616 1567 | 320M 27.4M 210M 1446M|6.0 0 0>
84 5 1 0 0 10: 76 12 12 0 0 0: 71 7 22 0 0 0: 86 13 1 0 0 0: 79 10 9 0 0 2|1179k 682k| 0 0 |2463 1489 | 326M 27.4M 210M 1439M|5.0 1.0 0>
83 8 0 0 0 9: 91 6 3 0 0 0: 85 12 3 0 0 0: 84 10 6 0 0 0: 86 9 3 0 0 2|1204k 740k| 0 0 |2491 1583 | 302M 27.4M 210M 1463M|5.0 0 0>
83 8 5 0 0 4: 84 6 9 1 0 0: 84 8 8 0 0 0: 72 10 17 1 0 0: 81 8 10 0 0 1|1182k 674k| 0 16k|2072 1626 | 312M 27.4M 211M 1454M|5.0 0 0>
41 7 49 1 0 1: 34 5 61 0 0 0: 17 4 79 0 0 0: 21 2 77 0 0 0: 29 5 66 0 0 0| 418k 343k| 0 0 | 962 717 | 260M 27.4M 211M 1505M| 0 0 0>
dstat result on front3
-------cpu0-usage--------------cpu1-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---load-avg--- ---procs--- ----swap--- ----system----
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free| 1m 5m 15m |run blk new| used free| time
45 1 51 1 0 2: 43 2 55 0 0 0: 45 2 53 0 0 1| 562k 77k| 24k 16k| 739 966 |1185M 560k 53.8M 764M|26.4 8.52 3.02| 0 0 0| 44M 464M|23-08 16:36:50
93 3 3 0 0 1: 94 2 4 0 0 0: 93 3 4 0 0 0|1105k 613k| 128k 0 |1758 1441 |1165M 560k 53.9M 784M|26.4 8.52 3.02|7.0 0 0| 44M 464M|23-08 16:36:51
97 3 0 0 0 0: 98 2 0 0 0 0: 98 2 0 0 0 1|1160k 611k| 0 0 |1745 1641 |1178M 560k 54.0M 771M|26.4 8.52 3.02| 13 0 0| 44M 464M|23-08 16:36:52
93 5 0 0 0 2: 93 7 0 0 0 0: 93 6 0 0 0 1|1125k 533k| 0 0 |1655 1575 |1202M 560k 54.0M 747M|25.3 8.58 3.07| 15 1.0 0| 44M 464M|23-08 16:36:53
94 5 0 0 0 1: 95 5 0 0 0 0: 94 5 0 0 0 0|1051k 619k| 168k 0 |1904 1639 |1184M 560k 54.1M 765M|25.3 8.58 3.07| 15 0 0| 41M 467M|23-08 16:36:54
95 4 0 0 0 1: 93 7 0 0 0 0: 94 6 0 0 0 0|1083k 543k| 0 0 |1952 1579 |1187M 560k 54.1M 762M|25.3 8.58 3.07| 16 0 0| 41M 467M|23-08 16:36:55
95 4 0 0 0 1: 94 6 0 0 0 0: 94 5 0 0 0 1| 893k 715k| 0 16k|1996 1514 |1161M 568k 54.1M 788M|25.3 8.58 3.07| 13 0 0| 41M 467M|23-08 16:36:56
45 3 52 0 0 0: 45 2 53 0 0 0: 45 2 53 1 0 0| 402k 521k| 0 0 | 691 667 |1024M 568k 54.2M 924M|25.3 8.58 3.07| 0 0 0| 41M 467M|23-08 16:36:57
Test #21, APC activated, 4 Fronts
- 2 fronts 4Vcpu, 2GB on servers ade-esxi-02 and ade-esxi-01 (Intel(R) Core(TM) Quad CPU Q6600 @ 2.40GHz; No Hyperthreading; 8 GB DDR2 800)
- 2 fronts 2Vcpu, 2GB on server ade-esxi-03 (Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz; Hyperthreading active; 8 GB DDR3 1066)
haproxy configuration:
server tu-web-01 192.168.100.210:80 check
server tu-web-02 192.168.100.218:80 check
server tu-web-03 192.168.100.172:80 check
server tu-web-04 192.168.100.167:80 check
$ ab -kc 40 -n 200 http://tu-lb-01/mysite/
Requests per second: 36.28 [#/sec] (mean)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking tu-lb-01 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests
Server Software: Apache/2.2.22
Server Hostname: tu-lb-01
Server Port: 80
Document Path: /mysite/
Document Length: 47881 bytes
Concurrency Level: 40
Time taken for tests: 5.513 seconds
Complete requests: 200
Failed requests: 180
(Connect: 0, Receive: 0, Length: 180, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 0
Total transferred: 9687247 bytes
HTML transferred: 9577047 bytes
Requests per second: 36.28 [#/sec] (mean)
Time per request: 1102.531 [ms] (mean)
Time per request: 27.563 [ms] (mean, across all concurrent requests)
Transfer rate: 1716.09 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 2
Processing: 422 1015 335.4 965 1817
Waiting: 327 722 221.7 710 1254
Total: 422 1015 335.4 965 1817
Percentage of the requests served within a certain time (ms)
50% 965
66% 1179
75% 1269
80% 1348
90% 1508
95% 1581
98% 1701
99% 1741
100% 1817 (longest request)
Rem:
I used 40 simultaneous connections.
CPU usage (sum of user and system usage) on the 5 servers are shown here
server |
Max CPU |
load balancer |
5% |
front1 |
96% |
front2 |
100% |
front3 |
100% |
front4 |
100% |
dstat result on load balancer
-------cpu0-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---load-avg--- ---procs--- ----swap--- ----system----
usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free| 1m 5m 15m |run blk new| used free| time
1 0 99 0 0 0: 1 0 99 0 0 0| 746B 1338B| 0 0 | 43 55 |67.3M 22.5M 67.4M 334M|0.40 0.13 0.08| 0 0 0| 0 508M|23-08 17:44:24
0 1 98 0 0 1: 0 1 98 0 0 1| 438k 427k| 0 0 | 407 186 |67.4M 22.5M 67.4M 334M|0.40 0.13 0.08|1.0 0 0| 0 508M|23-08 17:44:25
1 1 97 0 0 1: 1 1 97 0 0 1|2308k 2258k| 0 0 |1841 905 |67.4M 22.5M 67.4M 334M|0.40 0.13 0.08|1.0 0 0| 0 508M|23-08 17:44:26
1 0 98 0 0 1: 1 0 98 0 0 1|2129k 2078k| 0 0 |1808 846 |67.4M 22.5M 67.4M 334M|0.40 0.13 0.08| 0 0 0| 0 508M|23-08 17:44:27
1 0 98 0 0 1: 1 0 98 0 0 1|2002k 1956k| 0 20k|1646 851 |67.3M 22.5M 67.4M 334M|0.40 0.13 0.08| 0 0 0| 0 508M|23-08 17:44:28
0 1 95 0 0 4: 0 1 95 0 0 4|2007k 1959k| 0 0 |1598 757 |67.3M 22.5M 67.4M 334M|0.37 0.13 0.07| 0 0 0| 0 508M|23-08 17:44:29
1 1 97 0 0 1: 1 1 97 0 0 1|1519k 1485k| 0 0 |1050 560 |67.3M 22.5M 67.4M 334M|0.37 0.13 0.07| 0 0 0| 0 508M|23-08 17:44:30
dstat result on front1
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---load-avg--- ---procs--->
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free| 1m 5m 15m |run blk new>
74 12 9 0 0 5: 81 8 11 0 0 0: 77 13 10 0 0 0: 85 10 5 0 0 0: 79 11 9 0 0 1|1362k 632k| 0 0 |2289 1866 | 297M 29.3M 184M 1493M|1.19 1.94 1.50|8.0 0 1.0>
75 14 3 0 0 9: 83 12 5 0 0 0: 82 13 5 0 0 0: 83 13 4 0 0 0: 81 13 4 0 0 2|1194k 653k| 0 0 |1999 1570 | 288M 29.3M 184M 1502M|1.19 1.94 1.50|6.0 0 2.0>
71 13 8 0 0 8: 80 8 12 0 0 0: 79 15 6 0 0 0: 74 16 10 0 0 0: 76 13 9 0 0 2|1191k 576k| 0 0 |1729 1503 | 324M 29.3M 184M 1467M|1.19 1.94 1.50|6.0 0 4.0>
75 14 9 0 0 2: 68 11 19 1 0 0: 67 18 14 1 0 0: 64 15 20 0 0 0: 68 15 16 1 0 1| 891k 533k| 0 28k|1614 1380 | 359M 29.3M 184M 1431M|1.19 1.94 1.50|4.0 0 0>
80 8 9 0 0 3: 69 8 22 0 0 0: 79 10 11 0 0 0: 71 10 19 0 0 0: 75 9 15 0 0 1|1057k 707k| 0 0 |1521 1276 | 329M 29.3M 184M 1461M|1.18 1.92 1.50|3.0 0 0>
19 3 78 0 0 0: 0 0 100 0 0 0: 0 3 97 0 0 0: 7 1 92 0 0 0: 7 2 92 0 0 0| 73k 93k| 0 0 | 302 158 | 288M 29.3M 184M 1502M|1.18 1.92 1.50| 0 0 0>
dstat result on front2
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---procs--->
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| recv send| read writ| int csw | used buff cach free|run blk new>
18 2 80 0 0 0: 18 4 78 0 0 0: 16 4 80 0 0 0: 17 5 78 0 0 0: 17 4 79 0 0 0| 524k 24k| 0 16k| 531 481 | 275M 29.9M 228M 1470M| 11 0 0>
83 11 0 0 0 6: 88 11 1 0 0 0: 88 12 0 0 0 0: 87 12 1 0 0 0: 86 12 0 0 0 2|1104k 652k| 0 0 |2478 1695 | 287M 29.9M 228M 1459M|3.0 0 1.0>
84 11 1 0 0 4: 86 14 0 0 0 0: 88 12 0 0 0 0: 87 13 0 0 0 0: 86 12 0 0 0 1|1199k 707k| 0 0 |2192 1632 | 304M 29.9M 228M 1442M|6.0 0 2.0>
81 9 1 0 0 9: 88 11 1 0 0 0: 80 17 2 1 0 0: 84 14 2 0 0 0: 83 13 2 0 0 2|1185k 638k| 0 0 |2334 1528 | 311M 29.9M 228M 1434M|6.0 0 0>
87 8 2 0 0 3: 88 9 2 1 0 0: 73 9 17 0 0 0: 75 12 13 0 0 0: 81 9 9 0 0 1|1130k 665k| 0 0 |2260 1420 | 300M 29.9M 228M 1445M|6.0 0 0>
63 3 32 0 0 2: 39 6 55 0 0 0: 50 5 45 0 0 0: 43 6 51 0 0 0: 49 5 45 0 0 0| 614k 481k| 0 0 |1444 933 | 249M 29.9M 228M 1496M| 0 0 0>
dstat result on front3
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---procs--->
70 5 24 0 0 1: 68 7 25 0 0 0: 70 6 24 0 0 1| 973k 253k| 0 0 |1333 1288 | 415M 5572k 86.7M 1496M|5.28 6.17 4.07| 10 0 0| 16M 492M|23-08 17:47:40
95 5 0 0 0 0: 96 4 0 0 0 0: 95 5 0 0 0 0|1210k 601k| 0 16k|1305 1501 | 417M 5580k 86.7M 1494M|5.28 6.17 4.07| 11 0 0| 16M 492M|23-08 17:47:41
94 6 0 0 0 0: 95 5 0 0 0 0: 94 6 0 0 0 0| 995k 608k| 0 0 |1189 1345 | 417M 5580k 86.8M 1495M|5.66 6.23 4.10| 13 0 0| 16M 492M|23-08 17:47:42
95 5 0 0 0 0: 95 5 0 0 0 0: 95 5 0 0 0 0|1166k 664k| 0 0 |1245 1492 | 403M 5580k 86.8M 1508M|5.66 6.23 4.10| 12 0 0| 16M 492M|23-08 17:47:43
94 5 0 0 0 1: 94 6 0 0 0 0: 95 5 0 0 0 0|1004k 576k| 0 0 |1229 1477 | 401M 5580k 86.9M 1510M|5.66 6.23 4.10|9.0 0 0| 16M 492M|23-08 17:47:44
52 1 47 0 0 0: 44 1 55 0 0 0: 48 1 51 1 0 0| 402k 492k| 0 0 | 596 720 | 305M 5580k 86.9M 1606M|5.66 6.23 4.10| 0 0 0| 16M 492M|23-08 17:47:45
dstat result on front4
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage-----------total-cpu-usage---- -net/total- -dsk/total- ---system-- ------memory-usage----- ---procs--->
21 2 77 0 0 0: 15 4 81 0 0 0: 18 3 79 0 0 1| 524k 30k| 0 16k| 490 476 | 333M 6632k 91.8M 1572M|4.81 8.31 6.64| 10 0 0| 12M 496M|23-08 17:44:19
91 6 0 0 0 3: 97 3 0 0 0 0: 94 4 0 0 0 1|1024k 583k| 0 0 |1585 1433 | 387M 6632k 91.8M 1518M|5.15 8.32 6.66| 13 1.0 0| 12M 496M|23-08 17:44:20
87 6 0 0 0 7: 95 5 0 0 0 0: 91 6 0 0 0 4| 984k 541k| 0 0 |1848 1514 | 410M 6632k 91.9M 1495M|5.15 8.32 6.66| 13 1.0 0| 12M 496M|23-08 17:44:21
93 5 0 0 0 2: 95 5 0 0 0 0: 95 5 0 0 0 1|1068k 501k| 0 0 |1788 1591 | 437M 6632k 91.9M 1469M|5.15 8.32 6.66| 13 0 0| 12M 496M|23-08 17:44:22
96 3 0 0 0 1: 94 6 0 0 0 0: 95 4 0 0 0 1|1067k 617k| 0 0 |1458 1477 | 408M 6632k 91.9M 1497M|5.15 8.32 6.66| 11 0 1.0| 12M 496M|23-08 17:44:23
96 4 0 0 0 0: 96 4 0 0 0 0: 96 5 0 0 0 0| 911k 660k| 0 16k|1517 1521 | 391M 6640k 92.0M 1514M|5.15 8.32 6.66|7.0 0 0| 12M 496M|23-08 17:44:24
24 1 75 0 0 0: 19 1 79 1 0 0: 22 1 78 0 0 0| 173k 267k| 0 0 | 347 294 | 306M 6640k 92.0M 1599M|5.46 8.33 6.67| 0 0 0| 12M 496M|23-08 17:44:25
Conclusion on influence of the number of fronts
Is it linear ?
The following graph shows that it is pretty much linear in function of the number of fronts for this particular workload.

Number of fronts influence on req/sec
In the graph above, the red line is the theoretical throughput (sum of req/sec of the fronts).
We can notice that the obtained line is roughly linear (the regression line has a slope of 9.33, which is close to the average req/s 10.8) and close to the theoretical line.
I suppose that we don’t have exactly the theoretical throughput for different reasons:
- The algorithm of the load-balancer could not dispatch exactly in the same way on the fronts
- The front1 runs on an the same host as the mysql server (so they share some resources even if the mysql server load is low for this test)
- The virtualized layer could induce some decrease in performance when servers are running on the same host (context switchs, shared memory bus, etc…)
- The front3 and front4 are not really equivalent to the front1 and 2
But anyway, the result is not so bad when you consider the conditions of the tests.
Which is cool with this configuration is that the service can be delivered to the clients even if some fronts die. But it is another story because if you want a fail-over configuration, you have also to have a redundant load-balancer and a redundant database system.
There is something more annoying when you stress an application on many servers, you have to play with a lot of windows to run the commands that give you the performance counters and it is not so easy to see in real time all the counters on all the servers at one glance.
I need a graphical dashboard for that. Here comes Graphite. It will be the subject of the next post



