Yesterday I found myself puzzling over a problem that I've been wanting to solve ever since I started teaching myself Docker.
One of the big problems I have when working with Drupal module development is that I need an environment in which to test the modules. I've tried a number of different things over the years. I've installed LAMP on Linux or Acquia Dev Desktop on Mac OS. I eventually started using a few different Vagrant-managed Virtual Machines before switching to Docker entirely. While I do have a Docker environment I can spin up to run Drupal core and test modules, it's currently no better than those other solutions. I need to initialize the environment, pull down a new version of Drupal core, and then copy my module into the Drupal installation to test.
Usually I only have an hour or two to work on modules between other responsibilities. Given that it can sometimes take 20-30 minutes to init and stand up the environment, my productivity has been stymied. What would be best is if I can run a script that will do all the init, pulling and everything else on the Drupal side. Docker would mount my module code so that I can apply and review patches as quickly as possible.
How hard could that be?
After thinking about this for a few hours, the first step is to gather some requirements:
- A minimal Docker environment focused on testing modules, not running sites.
- No global tools to install, or weird additional requirements on the host system.
- Configured to run SimpleTest out of the box, in the web UI or on the CLI.
- A minimal footprint in the module repository, if any.
- As little workstation configuration as possible. Ideal: Pull the module repo and run the init script.
The first few requirements are pretty easy, since I've spent months reviewing and building what I consider a minimal Docker environment, it doesn't need any global tools installed apart from what's needed to run Docker, and it already runs SimpleTest pretty well. It's those last two that makes things really complicated.
Module Repositories are Inflexibly Structured
Currently, module repositories on Drupal.org are set up to mirror the module directory contents. Drupal.org's packaging script zips up the repository contents minus the .git/ directory, adds version information to the *.info or *.info.yml file, and then makes them available for download. This works, but it also makes things a bit inflexible.
Most Drupal-site repositories I've worked with now do not dump the site docroot in the parent directory of the repository. Instead, they keep it in a www/, docroot/, or some other directory in the root. This allows developers to add support files, project-related assets, Behat tests, git-based documentation, and other things in the same repository cleanly. Drupal module, repositories, however, do not have that option. For example, if I want to include the project logo in the Flag module repository, I have to "hide" it in an images/ or assets/ directory. This is the only way to include the image in the repository, even if the module itself never uses the image! With our existing repository directory = module directory system, however, this means that this "useless" file just takes up space without providing any use on every Drupal site in which Flag is installed.
The situation improved slightly with Drupal 8. The bulk of a module's code is in the src/ directory. There are still configuration directories, *.inc files, *.yml files, and other things in the module root directory. There's still no easy way to include files related to the module project needed for developers, but not needed when running the module on the site.
All of this makes creating a Docker environment for our module difficult. We need to find a way to mount the module files in the container. Ideally, we'd do so as a volume, so any changes on the host file system would immediately be copied to the container without the need to rebuild.
Option 1: Fix D.O's Packaging Script
The best solution would be for Drupal.org to allow a directory within the module repo be used for the module directory. For Flag module, this could be <root>/module or <root>/flag. Since the repository directory usually is named after the module machine name, flag/module seems more sensible. Then we can include whatever support files and assets we need in the repository root, outside of what's packaged into tarballs on D.O. Then, it would an easy matter to include the docker-compose.yml, associated support scripts, and to mount the directory containing module code as a volume in the container.
Obviously, this (probably) won't happen. It would require a lot of work by the infrastructure team, as well as tremendous re-education of module developers. Even if a Drupal.org were to allow the use of a .travis-like file for packaging configuration, it would open a can of worms everywhere.
Option 2: Mount the Parent Directory
First, we create a .docker/ directory in the root of our module repository. This will contain all the Docker files, support scripts, and everything. For our VOLUME statement, we actually mount the parent directory. This will mount all the module files, but also the .docker/ directory too. This is a bit of an odd hack, but there doesn't seem to be any reason why you can't do this. This has the advantage that everything needed to support the Docker image is also in the project repository, but it's hidden away in a distinct directory that doesn't "mess up" the repository.
Furthermore, when we configure our IDE, we can simply point it to the root of the repository. No git submodules or anything odd. We can even open a terminal within our IDE and rebuild the application easily by calling a script inside the .docker/ directory. Furthermore, the advantage is that all developers who work on the module could rely on the same Docker image. After all, the configuration is already in the repository.
Even if the Docker configuration isn't added to the repository, it wouldn't be hard to copy it once into the repo, and set a gitignore on the .docker/ directory.
Option 3: Rely on Local Setup
If we can't restructure our packaging script, and we don't want to change our include anything in our module repository, we throw requirement 5 out the window. Instead, developers need to download and maintain an environment harness locally. The environment harness would need to be downloaded once, and the git repo for the module cloned inside. To init or rebuild the environment, terminal commands would need to be executed outside the module root. It would be slightly more cumbersome, but it would work.
I have been doing a version of this so far already. On Linux, you can point the VOLUME statement at a symlink and it will resolve the correct directory. I have Drupal cloned elsewhere on my system and symlink to the environment harness' docroot/ directory. This doesn't work on Mac OS or Windows, sadly. You can also modify the VOLUME statement and mount the target directory directly. The only change here is that my docker-rebuild script would also need to clone Drupal, and the VOLUME statement would point at the module repository.
I really don't like my options above, but 2 feels like the best compromise for now. At least, until I find something that makes that option unviable. Option 3 is a decent fallback position, but by far not my favorite choice. It would require additional management on my end, and I'd rather keep things as self-contained as I possible.