Cleaning up after Docker

Posted on by

We use Docker at work for pretty much all our deployments, and have a local build/testing system that uses Docker locally. Recently, I started getting weird intermittent failures in the integration tests, and some digging showed a lot of errors around running out of space like no space left on device.

I found that there are 3 ways to clean up lingering Docker materials to get rid of this message and get my tests to run properly so I can find the actual failure.

  1. Docker images: View the images with the docker images command, and delete them with docker rmi.

  2. Docker containers: Look at the docker containers with docker ps -a command. The -a flag makes it list all the containers, not just currently running ones. To clean up containers, use the docker rm command.

  3. Docker volumes: List all the docker volumes with docker volume ls, remove them with docker volume rm.

Some helpful commands

StackOverflow was very helpful on this one. I took a ‘burn it all down’ approach to cleaning up after Docker, so I just deleted everything. Links to further info on each command in the footnotes.

  • docker rm $(docker ps -aq) The -q flag limits the output so containers gets quietly deleted instead of getting a bunch of error messages over the extraneous text. This will remove everything1. Another potentially useful flag that I did not use this time, but will keep a note of is the -v flag. It is supposed to remove the associated volume as well, allowing the volume removal step to be skipped 2 3.

  • docker rmi $(docker images -q) Same as above, but for images. Again, removes everything 4.

  • docker rmi $(docker images -q -f dangling=true) For a less scorched earth approach to removing images, this allows only the removal of images that are no longer needed 5.

  • docker volume rm $(docker volume ls -qf dangling=true) Similar to the above, remove only unused volumes 6.

Once I knew what to look for, the documentation from Docker was also very helpful about how to move through the clean-up quickly.

  1. StackOverflow: Docker error no space left on device ↩
  2. StackOverflow: Docker on Mac error no space left on device ↩
  3. Docker Docs: rm ↩
  4. StackOverflow: Docker error no space left on device ↩
  5. StackOverflow: Docker on Mac error no space left on device ↩
  6. StackOverflow: Docker error no space left on device ↩

Fixing docker error message on new terminal open

Posted on by

Since we use docker for just about everything at work, I got started with the very helpful ‘Getting Started’ guide. Shortly after, I noticed an issue where every time I opened a new terminal window, I was getting an error message.

Error checking TLS connection: Host is not running

This was because I was attempting to be clever and have my bash profile get docker set up for me, except it didn’t work when I didn’t have a docker-machine running already.

The fix: run docker-machine start default (substitute whatever name you want for default if you’re fancy) before running any other docker-machine commands.

The real fix: Don’t try to be quite that clever. I’ll get more sensible error messages that way.

The real real fix: Why do all the helpers for docker have a freaking - in the middle!? It is very annoying to type.

Fixing Sublime Text 2 and GoSublime

Posted on by

I’m still using Sublime Text 2 because it works well for me. I also use the GoSublime plugin to trigger go fmt and goimports on every save. This broke when GoSublime updated recently.

ETA: As of 2016/05/18 this broke again, still working out why.

** ETA2: I never got Sublime Text 2 to work properly with GoSublime. However, upgrading to Sublime Text 3 was painless and GoSublime works properly again!**

The fix: Instead of using the Package Manager, manually clone down the git repo and check out and older version (hat tip these folks).

Step 1: Uninstall GoSublime through the Package Manager.

Step 2: Rename or delete the GoSublime folder in /Users/<USERNAME>/Library/Application\ Support/Sublime\ Text\ 2/Packages.

Step 3: Pull down the GoSublime repo.

Step 4: Check out commit 852a15fb933

Notes on Go, part 7 of ∞

Posted on by

Coming to Go from Ruby has been an experience and a half. A lot of the things I’ve stubbed my toes on are either vocabulary things, or things that I don’t know about because I’ve never really used a statically typed language before.

A map is a hash table is a ruby hash

Lets start by paying a visit to Wikipedia, the repository of all modern knowledge.

Default values are unexpected

In Go, some types have a default value (probably to deal with the whole static typing thing???). For example, the default value of a boolean is false.

Similarly, ints also have a default value of 0.

So if I initialize a map of strings and ints, I get a number back, even if that key doesn’t exist in the map.

Multiple returns strike again

To solve the default value problem, Go offers optional multiple returns when querying a map/hash.

Now, when I query the map, I can then check the value of ok, which will return false if the key didn’t actually exist in the map.

The multiple return collection is optional. If I remove the ok assignment, Go will not complain.

make() vs literal syntax

There are two ways to initialize a map. You can use the make() syntax:

Or you can use the ‘literal’ syntax, which I’ve used the the previous examples. Arbitrarily, I like the literal syntax. Maybe I just like curly braces.

Resources

A Tour of Go
Go maps in action
Go By Example
Effective Go

Notes on Go, part 6 of ∞

Posted on by

To deal with the idiosyncratic failing test issues from my last go post, I got to do a bunch of stuff with our monorepo and learn a bit more about the Godep tool, and now I know more about why the stuff at the end of this post works.

The solution to my problem was to update the version of sqlmock that I was using. One of the potential issues with monorepos is that if you update a package, you have to update it across the entire monorepo, because of how Go dependencies work. One solution to this problem is to make whoever wants to change versions do all the updates across the repo. Another (less ‘Go’) solution is to have two versions of the dependency in your project. Which is what we did.

It took me a few tries to get everything down in a way that made both Godep and git happy, so here is my recommendation on the order of steps to take.

  1. Checkout your master branch and make sure it is up to date.

  2. Make sure your local Godeps/_workspace is up to date by running the godep restore command. I think this is the one I was having the most trouble with, because ‘restore’ is not what I think I want to do, conceptually. The restore command is (I think) making your local Godep/_workspace match what is in the Godep.json file. So if you’ve messed added ore removed a whole bunch of stuff, and want to start over, the ‘restore’ command is an accurate reflection of what you are doing, but it also works as an ‘update’, which is conceptually what I thought I wanted to do.

  3. Check your git status and make sure you don’t see anything unexpected. In my case, I didn’t expect to see really anything in my status because our Godep folder is checked into git.

  4. If you haven’t already, do a godep get PACKAGE/I/WANT.

  5. Update the import statement(s) in your actual *.go file(s).

  6. To save this into your dependencies, use godep save -r ./.... The -r flag takes care of re-writing any import statements to now point to your Godep/_workspace folder. The ./... makes sure it checks the current folder and all folders inside of it to catch any new imports.

  7. Again, check your git diff. At this point, I expected to see:

– the import changes
– the new package in Godep/_workspace

If it all looks good, commit!

  1. If this update was a breaking version change, now go back and fix your code that it compiles and runs again.

  2. Finish the thing that sent you down this rabbit hole in the first place!

Notes on Go, part 5 of ∞

Posted on by

Problem: We’re working in a Go monorepo. I’ve been writing tests for one package in one service in said monorepo. When I run
godep go test ./directoryofinterest,
I get the following (positive) message:
ok github.com/path/to/my/directoryofinterest 0.019s.

When I run
godep go test ./directoryofinterest -v,
I get several failure messages in the main part of the output, and the last two lines are
FAIL
and
ok github.com/path/to/my/directoryofinterest 0.021.

This is definitely some unexpected behavior.

Working through some of the 11th chapter of The Go Programming Language has surfaced a few idiosyncrasies.

  1. Running go test from the directory where the tests are returns 2 lines, whereas running go test ./directoryofinterest from one directory up returns only one line.

  2. When you get the 2 lines, one is a PASS or FAIL message, and the second is the ok message.

BUT WHYYYY???

Fun fact: it’s because i”m using an old version of Data Dog’s sqlmock.

After updating sqlmock to the current version, I get the expected behavior when running the tests.

Running the go test ./directoryofinterest now returns 1 line when tests pass with the ‘ok’ message. Running the same command with failing tests returns multiple lines with the details of the failure, and the final line contains the FAIL message.

Notes on Go, part 4 of ∞

Posted on by

I learned a fun thing about the html/template package in Go. When compiling your program, it does not check the validity of your template.

I was working through the Writing Web Applications tutorial, and I made the following mistake:

<h1>{{.Title</h1>

There is supposed to be a set of closing brackets after Title.

<h1>{{.Title}}</h1>

However, when I compiled and ran my program, I didn’t get any error messages until I tried to access /view/test. Then, it blew up with a panic message. Reading through the stack trace pretty quickly implicated the Template package, and it was a relatively quick problem to solve.

Tips and Tricks: Intermediate Git configuration

Posted on by

There are several ‘levels’ at which you can configure git. The three I know about are:

  • repository-level
  • repository branch-level
  • global

They are slightly self explanatory.

  • repository branch-level means that the configurations/aliases are applied ONLY to a specific branch in that repository. I haven’t had a need to use this one yet, so I can’t tell you a ton about it.
  • repository-level means that the configurations/aliases are applied ONLY to that repository, but to all branches on that repository. This can be configured to use different email addresses to different projects (ex: work email vs personal email), among other things.
  • global means that the configurations/aliases are applied to ALL git repositories on your system.

So, how do we set this up?

From the command line

Most of the tutorials on how to set up aliases give the command line version.

In your git repository, you can type git config XXX (where XXX is filler for whatever configuration you want to set). To set the configurations globally, you can type git config --global XXX. Adding the --global flag sets it for all git repositories. Some examples:

git config --global user.name "Clare Glinka"
– This sets my name as “Clare Glinka” for all repos.

git config --global user.email myemail@example.com
– This sets my email as myemail@example.com for all repos.

git config merge.ff false
– disables fast-forward merges by default at the repository level. Fast-forward merging will be disabled for all branches in that repo. Adding the --global flag will disable fast-forward merging for ALL repos.

You can check out other configuration settings in the Git documentation on customizing your git configuration and the git-config manual page.

Now that we’ve set some configs, how do we look at them? (because nothing is real till we see the output, right?)

git config --list will list all the configuration settings for your current directory/branch, including things like your remotes.

git config --list --global will list all the configurations settings for your system. For me, this is a much shorter list, since it does not include things like remotes.

Edit .gitconfig

There is another way to set configurations that I find a little easier to look at sometimes. That is checking out the .gitconfig file. For your global settings, .gitconfig lives in your ~ directory, and you can open it up and edit it with your favorite text editor. You can check out your repository-level configuration file by looking at REPO/.git/config.

References

First-Time Git Setup
Customizing Git Configuration
git-config Manual Page
StackOverflow: Can I make fast forwarding be off by default in git?
Git Tip: git config user name and email for local (not global) config

“In the context of the Single Responsibility Principle (SRP) we define a responsibility to be “a reason for change.” If you can think of more than one motive for changing a class, then that class has more than one responsibility.”

— Bob Martin, The Principles of OOD

“A class should do the smallest possible useful thing; that is, it should have a single responsibility.”

— Sandi Metz, Practical Object-Oriented Design in Ruby

Saving dotfiles with symlinks on OSX

Posted on by

I’ve always been impressed with folks who have their dotfiles all backed up to git somewhere, and I’ve finally worked out how to do it myself. The magic answer is symlinks!

STEP 1: DOTFILES

So, dotfiles are basically ‘hidden’ files that contain information about settings you don’t want to keep setting every time you try to do something. Lots of programs use dotfiles to keep track of user-specific configurations. For example. in ~/.gitconfig, Git stores information like your username and user email (and much, MUCH more).

STEP 2: Collect dotfiles into a Git repo

Mostly, dotfiles live in your home directory. But, you don’t really want to put everything in your home directory under git’s watchful eye, so the easier step is to collect all the dotfiles into a git repo that lives elsewhere. Pick a folder, ANY FOLDER! For best experience, I suspect it should live somewhere under your home directory. I’m currently keeping mine in ~/code/dotfiles.

Dotfiles you might want to keep under version control:
– Git config (~/.gitconfig)
– terminal config (~/.bash_profile, ~/.bashrc, ~/.zshrc, etc)
– basically anything were you’ve set prefs that are stored!

Hint The First

In order for the next step to work, you can’t have an existing copy of your dotfiles in your home directory. So if you want to put your .gitconfig under version control, for example, it is better to MOVE it instead of COPYING it.

Hint The Second

The ultimate point of this whole exercise (for me) was to have all my dotfiles backed up somewhere. If (like me) you are not yet ready to publicly publish your dotfiles on Github, there are a number of other Git-based services that will provide private repos for free.

STEP 3: Symlink!

The ln command is used to create links in your (unix-based) system. Use the --help command to get more info on the myriad of options available to you. Just using the ln command creates something called a ‘hard link’, which is not what we want. We want a ‘symbolic’ link instead. The syntax is ln -s + actual location of the file + name and location you want to see that file under.

Example:

ln -s ~/code/dotfiles/gitconfig ~/.gitconfig

will result in the gitconfig in the dotfiles directory to be accessible from the ~/.gitconfig location, which is where Git is expecting to see all the Git preferences you’ve set.

STEP 4: PROFIT!

RESOURCES:

I stumbled across this great article called How to Create and Use Symlinks on a Mac that made the lightbulb go off and instigated this whole song and dance.