Specifically, I ran into this using the pyFilesystem library. While not a failing of the library itself, the library makes use of __del__() methods, file systems (the major object type in this library) also reference one another. This makes it very easy to create a reference cycle as described above.

The solution is to be diligent about close()ing the file systems. Each fs has a close() method, but being naive to this problem, and used to the Python gc having my back, I was not diligent. I eventually discovered the problem because our FTP server was exhausting it’s memory. This problem never turned up in testing because frankly our testing focuses on correctness and does not generate a huge amount of load. By the time this problem made it to production, I had to troubleshoot it there, and fast.

I started by looking for advice from other pyfs users, I turned to the mailing list and received some great information from Ryan Kelly.

However, in the end, the solution was just good old fashioned hard work. I produced a series of patches that corrected my misuse of the library, try…finally and with … as … solved all the simple issues. Some more sophisticated approaches solved the complex ones. But in the end, ensuring the fs instance is closed when you are done with it ensures it frees all it’s references to other fs instances, breaking the cycle.

The only interesting part, and the reason for this post is that I ended up writing a tool to help troubleshoot this problem. As I mentioned, it made it to production, which is not a good place for debuggers, especially in an FTP server under load. Therefore, I needed a tool that I could invoke against the running daemon, then perform offline analysis without impacting the server. When I looked, I did not find anything, so I created caulk (for plugging memory leaks).

The main feature of this library and cli tool is that it makes it easy to set up a signal handler. Then you can `kill -usr1` your Python process and analyze the dump file using the caulk command. One late addition (added today) is integration with objgraph (if it is installed), which will also dump some graphs showing the reference to the uncollectable object instances. You can view the graph using the xdot command. This last part is what finally led me to the few remaining code paths that were not properly closing the fs.

I selected the VPN client option to use a remote DNS server. This allows me to access remote hosts by name instead of by IP address or using an /etc/hosts file. It is convenient, but totally broke DNS resolution in my guest VMs.

I did not try to figure out the root cause, but whenever I am connected to the VPN, the guest cannot resolve any hosts. My guests use NAT networking in VirtualBox. The DHCP client on the guest is configuring /etc/resolv.conf with the IP address of my host machine’s interface on the NAT network.

The quick fix was to add PEERDNS=”no” to the guest OS /etc/sysconfig/network-scripts/ifcfg-eth0 file (the guest OS being CentOS). And then hard-coding the DNS server address to something like 8.8.8.8 in /etc/resolv.conf, which if connected to the Internet should always work. I made this change in the VM that I created my basebox from, repackaged it and now any Vagrant VMs derived from it have this change. At least Vagrant made it easy to propagate the change to the other VMs.

I am sure there is a bug somewhere in the DNS resolver or proxy that VirtualBox ships with that is the root cause, but I have been too busy to spend any time digging further.

Besides my use of both projects, Pyftpdlib and PyFilesystem have another thing in common: great maintainers. I could not have pulled off everything I have with these two projects without the great oversight and direction of Giampaolo Rodola (pyftpdlib) and Will McGugan (pyfilesystem). So a big thanks to both of them is due for their excellent projects and leadership!

Pyftpdlib consists of a library and tools for building FTP servers in Python. It is based on asyncore and as such provides a very efficient system which can scale to many thousands of simultaneous clients. It also ships with ready made FTP servers that can simply be executed to start serving files over FTP. Pyftpdlib is used by many players to bring FTP into myriad Python based systems. The main strength of Pyftpdlib is that is provides an easy method to extend the base FTP server, one can plug in their own file access classes and authentication classes to customize the FTP server. Since the library is written in Python, even beyond the simple plugin system, one can customize the FTP server to any purpose. There are also a number of optional additions provided such as SSL support, and throttling to name two.

PyFilesystem is a library and tools which abstracts storage interaction. The system consists of a common API that can be utilized for all file interactions, a number of backends which implement this API and store data in myriad systems, as well as a bunch of tools for interacting with the API. For example, using PyFilesystem, one can interact with Amazon’s S3, Riak, FTP, SFTP, WebDAV, and many other storage systems. All using the same API. Some backends even provide transformation of data, some examples are encrypting or compressed file systems. Some file system backends exist purely for merging multiple other backends together, examples of this are MultiFS, which can “stack” multiple file systems into one unified view, and MountFS which can merge multiple file systems into a tree. Many of this functionality can be had using existing tools, but PyFilesystem allows greater flexibility in that all of this functionality is available within a Python application.

The way I use these two projects together should be obvious by now. But I have gone a step further and decided to contribute my code to the PyFilesystem project. The code I contributed is another in a growling list of “expose” wrappers that glue PyFilesystem into other Python projects (Paramiko (SFTP) being another notable one). This should allow others to easily create FTP servers to serve their favorite storage medium.

I found an undocumented feature in the latest HAProxy, log-send-hostname which changes this behaviour.

http://haproxy.1wt.eu/git?p=haproxy.git;a=commitdiff;h=df5b38fac1788e6a134095459170a618a1c23388

service smartd stop && chkconfig smartd off
service avahi-daemon stop && chkconfig avahi-daemon off
service cups stop && chkconfig cups off
service autofs stop && chkconfig autofs off
service iscsid stop && chkconfig iscsid off
service iscsi stop && chkconfig iscsi off
service bluetooth stop && chkconfig bluetooth off
service kudzu stop && chkconfig kudzu off
service mdmonitor stop && chkconfig mdmonitor off
service hidd stop && chkconfig hidd off
service haldaemon stop && chkconfig haldaemon off
service gpm stop && chkconfig gpm off
service auditd stop && chkconfig auditd off
service cpuspeed stop && chkconfig cpuspeed off
service messagebus stop && chkconfig messagebus off
service pcscd stop && chkconfig pcscd off

These do things like manage devices or file systems. Things a VM won’t need done. This stopped about 20 processes and freed up a few megabytes of memory. Every little bit helps when you run tens or hundreds of VMs.

However, util-linux provides a program named logger that writes to syslog. It can be used directly with apache to proxy log entries to syslog. You just need to configure it via the CustomLog directive.

CustomLog " |/usr/bin/logger -t 'apache'" combined

I have also seen suggestions to use a dedicated socket to receive Apache log entries. I am not sure what the reason for this is, I would assume that logger uses internal syslog calls by default which would perform better than a UNIX socket. However, if you want to test this method as well, you will need to add a new source to /etc/syslog-ng/syslog-ng.conf:

source s_apache {
	unix-stream ("/dev/log_apache" max-connections(512) keep-alive(yes));
};

Then the CustomLog directive in the httpd.conf changes slightly:

CustomLog " |/usr/bin/logger -t 'apache' -u /dev/log_apache" combined

Previously, I was using a python script to do the same job. I am glad to retire my python script :-).

When you are done remixing, you can download a new PDF file that contains the pages you selected in the order you defined.

This allows you to quickly combine a cover letter with another document, or remove some pages from a document before email it etc.

The site is pdfremix.com.

My associate wanted to use the latest version of lftp, which is not available using either one of these methods. Therefore I built version 4.3.0 (lastest as of now) for him on Cygwin. Below are the simple steps required to do so.

1. Download cygwin’s setup.exe.
2. Install your toolchain: bison, autoconf, gcc, gcc-c++, make.
3. Install the provided version of lftp (4.2.3-1) this will pull in all the required dependencies.
4. Install GNUTLS: gnutls & gnutls-devel.
5. You can now extract the lftp sources and build using:

   $ ./configure && make

6. Once the build was successful, I simply determined the dependencies, then gathered the .exe file and all required .dll files into a folder and zipped them up. The following command will show all the non-Windows (cygwin) dependencies:

   $ ldd lftp.exe | grep -v WINDOWS

– OR –

You can download the version I built for him and save some time :-).

If you are using Apache as a webserver, then one choice is to use mod_xsendfile, which allows you to set a header, instructing Apache to efficiently send the file to the client. This gives you the best of both worlds.

However, when dealing with an international application where filenames are determined by customers, you will often have file names that contain non-ASCII characters. In this case, Django will raise an exception when you set the X-SendFile header:

UnicodeEncodeError: 'ascii' codec can't encode character u'\uf026' in position 74:
ordinal not in range(128), HTTP response headers must be in US-ASCII format

This is because HTTP headers must be ASCII only. This problem is well documented. So, what is the solution?

My solution was to patch mod_xsendfile so that it can accept an additional header, X-SendFile-Encoding, instructing it to decode the file name before use. In this manner, the Django application can encode the filename into ASCII, send it to the module, which will then decode it and send the file. The encoding scheme I selected is url encoding. Given that my file system encoding is UTF8, the full solution is:

response['X-SendFile-Encoding'] = 'url'
response['X-SendFile'] = urllib.quote(path.encode('utf8'))

The patch is available at: http://ben.timby.com/pub/mod_xsendfile-url_encoding.patch
I have also built an RPM for CentOS at: http://dagobah.ftphosting.net/yum/mod_xsendfile-0.11.1-5.x86_64.rpm
And of course the SRPM: http://dagobah.ftphosting.net/yum/SRPMS/mod_xsendfile-0.11.1-5.x86_64.rpm

I hope this helps somebody!

Update

The author of mod_xsendfile implemented a better version of my patch. Now by default the header’s value will be decoded (causing no problems for non-encoded values). This behavior can be disabled using an optional configuration flag (XSendFileUnescape off).

https://github.com/nmaier/mod_xsendfile/commit/b98d2d1df9f7acd720bf082e32b1392188a23379

https://github.com/nmaier/mod_xsendfile/commit/0efcd03ac196930da6b139b77972c0d430e0225c

Thank you Nils!

I am working on an client/server application that transfers huge amounts of data over a network. The server produces data and then it passes through a processing pipeline before being transmitted. The client receives the data and then pulls it through a similar processing pipeline before parsing and saving the result.

The transport is HTTP and the processing pipeline consists of encoders such as a chunked transfer encoder/decoder and a deflate/inflate filter. While building this project, I had need of an efficient FIFO buffer to be used by each of the links in the processing pipeline. For example, the deflate filter buffers data until it has received a defined ‘block size’ of data before compressing the block and passing it down the chain. This is done for efficiency reasons, compressing 100 small buffers takes longer than compressing one large buffer. The chunked transfer decoder also needed a FIFO buffer, as it parses the data block by block. When reading from the previous link in the pipeline, sometimes the read would end in the middle of a structure. In that case, the read data must be buffered until the remainder of the structure is available upstream.

Because I am working with multi-gigabyte data streams, I can’t afford to buffer the entire thing, especially on the client. As data arrives, I want to feed it into one end of a FIFO buffer, and then consume from the other end. The obvious choice here is to use StringIO (cStringIO) however, this is not suitable for one simple reason: StringIO does not provide a means of truncating the BEGINNING of the buffer. You can write data to the tail, consume it from the head, but you cannot discard data after you read it.

The simple solution is to simply append data blocks to a list when writing, and pop them from the beginning when reading. This simple approach works, but is very slow. I present below the solution I am using now which is about 10x faster. It works using the same idea, but the list contains StringIO instances rather than large data blocks in the form of strings.

The Solution.

The Buffer object starts off with an empty buffer list (self.buffers = []). When data arrives, the buffer will append a new StringIO instance to this list. As more data arrives, it is appended to this StringIO instance. Once the StringIO instance reaches a threshold, another StringIO instance is appended to the list and receives subsequent data. My testing uses a 4MB threshold for the StringIO size limit.

As data is read, it is read from the oldest StringIO instance. Once that instance is exhausted, it is removed from the list, freeing up the memory it consumed. Subsequent reads are then satisfied by the next oldest StringIO instance until all of them are exhausted. Once all StringIO’s are exhausted the list is cleared and the whole process starts again.

This provides for an elastic FIFO buffer that only keeps data resident until it is consumed. Once consumed, the data is eventually freed (in increments of 4MB). My implementation also uses a Lock to create a critical section when reading/writing the buffers. This is required in my project since the buffer is used to move data between threads (some processing pipeline elements have dedicated threads).

import threading
try:
    from cStringIO import StringIO
except ImportError:
    from StringIO import StringIO

MAX_BUFFER = 1024**2*4

class Buffer(object):
    def __init__(self, max_size=MAX_BUFFER):
        self.buffers = []
        self.max_size = max_size
        self.lock = threading.Lock()
        self.closing = False
        self.eof = False
        self.read_pos = 0
        self.write_pos = 0

    def write(self, data):
        self.lock.acquire()
        try:
            if not self.buffers:
                self.buffers.append(StringIO())
                self.write_pos = 0
            buffer = self.buffers[-1]
            buffer.seek(self.write_pos)
            buffer.write(data)
            if buffer.tell() >= self.max_size:
                buffer = StringIO()
                self.buffers.append(buffer)
            self.write_pos = buffer.tell()
        finally:
            self.lock.release()

    def read(self, length=-1):
        self.lock.acquire()
        read_buf = StringIO()
        try:
            remaining = length
            while True:
                if not self.buffers:
                    break
                buffer = self.buffers[0]
                buffer.seek(self.read_pos)
                read_buf.write(buffer.read(remaining))
                self.read_pos = buffer.tell()
                if length == -1:
                    # we did not limit the read, we exhausted the buffer, so delete it.
                    # keep reading from remaining buffers.
                    del self.buffers[0]
                    self.read_pos = 0
                else:
                    #we limited the read so either we exhausted the buffer or not:
                    remaining = length - read_buf.tell()
                    if remaining > 0:
                        # exhausted, remove buffer, read more.
                        # keep reading from remaining buffers.
                        del self.buffers[0]
                        self.read_pos = 0
                    else:
                        # did not exhaust buffer, but read all that was requested.
                        # break to stop reading and return data of requested length.
                        break
        finally:
            self.lock.release()
        return read_buf.getvalue()

    def __len__(self):
        len = 0
        self.lock.acquire()
        try:
            for buffer in self.buffers:
                buffer.seek(0, 2)
                if buffer == self.buffers[0]:
                    len += buffer.tell() - self.read_pos
                else:
                    len += buffer.tell()
            return len
        finally:
            self.lock.release()

    def close(self):
        self.eof = True