I am not a fan of NFS for production information.  NFS is great for aggregating data from across multiple different machines, storing deployment files and other such administrative things.  Serving static content?  No.  I haven’t blogged about it but I have talked about it several times at conferences.  NFS, as a static content distribution mechanism is horribly slow.  Here’s the chart to prove it.

Never mind the issues that you need to take into consideration.

I know a lot of Magento deployments use NFS so they can have static content accessible from multiple different servers and have it all “up to date”.  But, like I said, I am not a fan of using NFS for production and so while it works, in my opinion, it is not optimal.

So I wanted to test GlusterFS to see provided a more optimal solution.  So I wrote a quick test script to try it out.  I tried creating multiple files, reading from one file multiple times, reading from the sequence of files multiple times and then deleting the files.  The Gluster configuration is vanilla, as is NFS.  Here’s the code I used to test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

$fs = array('/kschroeder/gfstest', '/mnt/gfstest', '/var/tmp/gfstest');
 
foreach ($fs as $f) {
  @mkdir($f, 0755, true);
  $startTime = microtime(true);
  for ($i = 0; $i < 500; $i++) {
    file_put_contents($f . '/test-' . $i, str_pad('', 0, 10240));
  }
  $elapsed = microtime(true) - $startTime;
  echo "$f write $elapsed\n";
 
  $startTime = microtime(true);
  $files = glob("$f/*");
  $file = array_shift($files);
  for ($i = 0; $i < 10000; $i++) {
    file_get_contents($file);
  }
  $elapsed = microtime(true) - $startTime; 
  echo "$f read-one $elapsed\n"; 
 
  $startTime = microtime(true);
  foreach (glob("$f/*") as $file) {
    file_get_contents($file);
  }
  $elapsed = microtime(true) - $startTime; 
  echo "$f read-all $elapsed\n";
 
  $startTime = microtime(true);
  foreach (glob("$f/*") as $file) {
    unlink($file);
  }
  $elapsed = microtime(true) - $startTime;
  echo "$f delete $elapsed\n";
 
}

/kschroeder/gfstest was an NFS mount, /mnt/gfstest was a GlusterFS mount and /var/tmp/gfstest was local.

Here are the results.

This is the elapsed time to conduct the individual tests so lower is better.  GlusterFS did better than NFS on all the different operations.  However, when compared with local speeds there was no comparison.

GlusterFS is definitely better, IMHO, than NFS.  It does a lot of things like clustering, striping and replication out of the box and is very easy to configure all of those options without having to worry about magic or other components to make it work.  For that reason I would definitely put it on my list of preferred solutions for handling static web content.

But what I want is a distributed, replicated static file storage mechanism that does client-side caching so my static file read performance is at least almost as good as local.  And if my main file server goes down I don’t want my entire web site to go down.  And it needs to be easy to install and manage so that merchants, who often do not have a lot of system administration experience, are able to do basic SA tasks.  As much of an improvement that GlusterFS is, out of the box it doesn’t seem to be solve the problem fully.

The search continues (and I do have some other possibilities I want to test).

Having started working at Magento I have been making myself more familiar with many of the different parts of the community.  I have spent a fair amount of time over the past several weeks trying to understand how people work with Magento and what their problems are.

One of the things that often comes up is speed.  And there are lists of things that people can do to try and make Magento faster.  But there’s something that bugs me in many of these lists.  Often people say that in order to make your Magento installation run faster you need to put certain things on tmpfs or RAM.

Makes sense, right?  The disk is slow, RAM is fast and so a RAM drive must be fast.  Right?

Nope.

Yes.  The disk is slow.  But that does not mean that the file system is.  The disk is only part of the file system.  The file system includes this nifty thing called a disk block cache.  The file system will cache often-used disk blocks in RAM.

What?  The file system uses RAM?  Yep.  When you type in “top” and you see that value for “cached”?  That is RAM that the operating system is using to store disk blocks.

The result of that caching is this chart.  It measures the throughput in number of requests per second of a static resource via Nginx over HTTP load tested from a remote host on the same network.

Server Throughput (HTTP Requests per Second)

To conduct the test I did 3 test runs for each of the types of file systems.  A physical ext3 disk, tmpfs and a RAM drive.  I ran 10,000 iterations via 100 concurrent requests.  The results of the test show that the physical media was actually the fastest!

I was actually a little surprised by that.  I was expecting that the physical (local) file system would only be keeping up, instead it was faster.  I would be willing to chalk that win up to entropy in the system but the assertion that a RAM drive or tmpfs is faster than the file system is clearly not true.

Don’t get me wrong.  RAM drives are great if you want to explicitly define a file system which WILL stay in RAM.  However, I side with Linus Torvalds (when he was talking about O_DIRECT) that the purpose of an operating system is to manage a lot of this for you.  You might be able to get some better results for RAM or tmpfs from some tuning but it would seem to be a micro-optimization at best or a giant waste of time at worst.

Commonly known as .htaccess, AllowOverride is a neat little feature that allows you to tweak the server’s behavior without modifying the configuration file or restarting the server.  Personally, I think this is great for development purposes.  It allows you to quickly test various server configurations without needing to mess with restarting the server.  It helps you be more (buzzword alert!) agile.

Beyond the obvious security problems of allowing configuration modifications in a public document root there is also a performance impact.  What happens with AllowOverride is that Apache will do an open() call on each parent directory from the requested file onward.

To demonstrate this I used a program called strace which checks for system calls and gives you a list of each system call that is made.

First we’ll take a look at the strace with AllowOverride set to None.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

semop(1638426, {{0, -1, SEM_UNDO}}, 1) = 0
epoll_wait(42, {{EPOLLIN, {u32=3507213864, u64=139813282633256}}}, 2, 10000) = 1
accept4(4, {sa_family=AF_INET6, sin6_port=htons(55755), inet_pton(AF_INET6, "::ffff:192.168.0.212", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 43
semop(1638426, {{0, 1, SEM_UNDO}}, 1) = 0
getsockname(43, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "::ffff:192.168.0.212", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
fcntl(43, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(43, F_SETFL, O_RDWR|O_NONBLOCK) = 0
read(43, "GET /test.txt HTTP/1.0\r\nHost: ma"..., 8000) = 87
gettimeofday({1361542861, 683952}, NULL) = 0
stat("/var/www/magento.loc/test.txt", {st_mode=S_IFREG|0644, st_size=12, ...}) = 0
open("/var/www/magento.loc/test.txt", O_RDONLY|O_CLOEXEC) = 44
fcntl(44, F_GETFD) = 0x1 (flags FD_CLOEXEC)
fcntl(44, F_SETFD, FD_CLOEXEC) = 0
mmap(NULL, 12, PROT_READ, MAP_SHARED, 44, 0) = 0x7f28cfc74000
writev(43, [{"HTTP/1.1 200 OK\r\nDate: Fri, 22 F"..., 267}, {"hello world\n", 12}], 2) = 279
munmap(0x7f28cfc74000, 12) = 0
write(12, "192.168.0.212 - - [22/Feb/2013:0"..., 79) = 79
shutdown(43, 1 /* send */) = 0
poll([{fd=43, events=POLLIN}], 1, 2000) = 1 ([{fd=43, revents=POLLIN|POLLHUP}])
read(43, "", 512) = 0
close(43) = 0
read(6, 0x7fff79e2d7cf, 1) = -1 EAGAIN (Resource temporarily unavailable)
close(44) = 0
semop(1638426, {{0, -1, SEM_UNDO}}, 1

Now let’s take a look at the strace results with AllowOverride set to All.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

semop(1736730, {{0, -1, SEM_UNDO}}, 1) = 0
epoll_wait(42, {{EPOLLIN, {u32=3392874024, u64=140410168747560}}}, 2, 10000) = 1
accept4(4, {sa_family=AF_INET6, sin6_port=htons(55795), inet_pton(AF_INET6, "::ffff:192.168.0.212", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 43
semop(1736730, {{0, 1, SEM_UNDO}}, 1) = 0
getsockname(43, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "::ffff:192.168.0.212", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
fcntl(43, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(43, F_SETFL, O_RDWR|O_NONBLOCK) = 0
read(43, "GET /test.txt HTTP/1.0\r\nHost: ma"..., 8000) = 87
gettimeofday({1361543373, 140117}, NULL) = 0
stat("/var/www/magento.loc/test.txt", {st_mode=S_IFREG|0644, st_size=12, ...}) = 0
open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/magento.loc/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/magento.loc/test.txt/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOTDIR (Not a directory)
open("/var/www/magento.loc/test.txt", O_RDONLY|O_CLOEXEC) = 44
fcntl(44, F_GETFD) = 0x1 (flags FD_CLOEXEC)
fcntl(44, F_SETFD, FD_CLOEXEC) = 0
mmap(NULL, 12, PROT_READ, MAP_SHARED, 44, 0) = 0x7fb3c8bf9000
writev(43, [{"HTTP/1.1 200 OK\r\nDate: Fri, 22 F"..., 267}, {"hello world\n", 12}], 2) = 279
munmap(0x7fb3c8bf9000, 12) = 0
write(12, "192.168.0.212 - - [22/Feb/2013:0"..., 79) = 79
shutdown(43, 1 /* send */) = 0
poll([{fd=43, events=POLLIN}], 1, 2000) = 1 ([{fd=43, revents=POLLIN|POLLHUP}])
read(43, "", 512) = 0
close(43) = 0
read(6, 0x7fff95abfc1f, 1) = -1 EAGAIN (Resource temporarily unavailable)
close(44) = 0
semop(1736730, {{0, -1, SEM_UNDO}}, 1

You can clearly see the additional open() calls being made to try and discover the .htaccess file.  In this case the calls are completely superfluous because we have nothing there.  But even so we have a significant impact on static file throughput.

AllowOverride None

1
2
3
4
5
6
7
8
9
10
11

Concurrency Level: 10
Time taken for tests: 2.441 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 2790279 bytes
HTML transferred: 120012 bytes
Requests per second: 4096.02 [#/sec] (mean)
Time per request: 2.441 [ms] (mean)
Time per request: 0.244 [ms] (mean, across all concurrent requests)
Transfer rate: 1116.12 [Kbytes/sec] received

AllowOverride All

1
2
3
4
5
6
7
8
9
10
11

Concurrency Level: 10
Time taken for tests: 3.922 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 2790558 bytes
HTML transferred: 120024 bytes
Requests per second: 2549.42 [#/sec] (mean)
Time per request: 3.922 [ms] (mean)
Time per request: 0.392 [ms] (mean, across all concurrent requests)
Transfer rate: 694.76 [Kbytes/sec] received

The requests where AllowOverride was turned off were executed at 60% of the time of the ones where AllowOverride was turned on.

And remember, this is just the impact of file operations and does not take into account the time to reconfigure Apache during the course of these requests.

So the data would clearly show that there is a negative impact to having AllowOverride turned on in a production environment.  Instead it will generally be better to take those changes in .htaccess and place them in your httpd configuration file.

[UPDATE]

In fact Mike Willbanks says you should never do it.  I  agree with him, but I wouldn’t make as big a stink in dev as I would in prod.

@kpschrade or ever for that matter; modifying the configuration directive is just as fast as .htaccess.

— Mike Willbanks (@mwillbanks) February 22, 2013