Pyftpdlib + PyFilesystem

The title of this post mentions two great Python libraries which I use heavily. Actually it is a disingenuous to call them libraries, because they are both in fact systems. I use these systems as part of the foundation of SmartFile, which is a Cloud storage provider with FTP, SFTP as well as HTTP and API file access methods.

Besides my use of both projects, Pyftpdlib and PyFilesystem have another thing in common: great maintainers. I could not have pulled off everything I have with these two projects without the great oversight and direction of Giampaolo Rodola (pyftpdlib) and Will McGugan (pyfilesystem). So a big thanks to both of them is due for their excellent projects and leadership!

Pyftpdlib consists of a library and tools for building FTP servers in Python. It is based on asyncore and as such provides a very efficient system which can scale to many thousands of simultaneous clients. It also ships with ready made FTP servers that can simply be executed to start serving files over FTP. Pyftpdlib is used by many players to bring FTP into myriad Python based systems. The main strength of Pyftpdlib is that is provides an easy method to extend the base FTP server, one can plug in their own file access classes and authentication classes to customize the FTP server. Since the library is written in Python, even beyond the simple plugin system, one can customize the FTP server to any purpose. There are also a number of optional additions provided such as SSL support, and throttling to name two.

PyFilesystem is a library and tools which abstracts storage interaction. The system consists of a common API that can be utilized for all file interactions, a number of backends which implement this API and store data in myriad systems, as well as a bunch of tools for interacting with the API. For example, using PyFilesystem, one can interact with Amazon’s S3, Riak, FTP, SFTP, WebDAV, and many other storage systems. All using the same API. Some backends even provide transformation of data, some examples are encrypting or compressed file systems. Some file system backends exist purely for merging multiple other backends together, examples of this are MultiFS, which can “stack” multiple file systems into one unified view, and MountFS which can merge multiple file systems into a tree. Many of this functionality can be had using existing tools, but PyFilesystem allows greater flexibility in that all of this functionality is available within a Python application.

The way I use these two projects together should be obvious by now. But I have gone a step further and decided to contribute my code to the PyFilesystem project. The code I contributed is another in a growling list of “expose” wrappers that glue PyFilesystem into other Python projects (Paramiko (SFTP) being another notable one). This should allow others to easily create FTP servers to serve their favorite storage medium.


About this entry