Tag Archives: command line

Cleaning up after Docker

Posted on by

We use Docker at work for pretty much all our deployments, and have a local build/testing system that uses Docker locally. Recently, I started getting weird intermittent failures in the integration tests, and some digging showed a lot of errors around running out of space like no space left on device.

I found that there are 3 ways to clean up lingering Docker materials to get rid of this message and get my tests to run properly so I can find the actual failure.

  1. Docker images: View the images with the docker images command, and delete them with docker rmi.

  2. Docker containers: Look at the docker containers with docker ps -a command. The -a flag makes it list all the containers, not just currently running ones. To clean up containers, use the docker rm command.

  3. Docker volumes: List all the docker volumes with docker volume ls, remove them with docker volume rm.

Some helpful commands

StackOverflow was very helpful on this one. I took a ‘burn it all down’ approach to cleaning up after Docker, so I just deleted everything. Links to further info on each command in the footnotes.

  • docker rm $(docker ps -aq) The -q flag limits the output so containers gets quietly deleted instead of getting a bunch of error messages over the extraneous text. This will remove everything1. Another potentially useful flag that I did not use this time, but will keep a note of is the -v flag. It is supposed to remove the associated volume as well, allowing the volume removal step to be skipped 2 3.

  • docker rmi $(docker images -q) Same as above, but for images. Again, removes everything 4.

  • docker rmi $(docker images -q -f dangling=true) For a less scorched earth approach to removing images, this allows only the removal of images that are no longer needed 5.

  • docker volume rm $(docker volume ls -qf dangling=true) Similar to the above, remove only unused volumes 6.

Once I knew what to look for, the documentation from Docker was also very helpful about how to move through the clean-up quickly.

  1. StackOverflow: Docker error no space left on device ↩
  2. StackOverflow: Docker on Mac error no space left on device ↩
  3. Docker Docs: rm ↩
  4. StackOverflow: Docker error no space left on device ↩
  5. StackOverflow: Docker on Mac error no space left on device ↩
  6. StackOverflow: Docker error no space left on device ↩

Fixing docker error message on new terminal open

Posted on by

Since we use docker for just about everything at work, I got started with the very helpful ‘Getting Started’ guide. Shortly after, I noticed an issue where every time I opened a new terminal window, I was getting an error message.

Error checking TLS connection: Host is not running

This was because I was attempting to be clever and have my bash profile get docker set up for me, except it didn’t work when I didn’t have a docker-machine running already.

The fix: run docker-machine start default (substitute whatever name you want for default if you’re fancy) before running any other docker-machine commands.

The real fix: Don’t try to be quite that clever. I’ll get more sensible error messages that way.

The real real fix: Why do all the helpers for docker have a freaking - in the middle!? It is very annoying to type.

Notes on Go, part 6 of ∞

Posted on by

To deal with the idiosyncratic failing test issues from my last go post, I got to do a bunch of stuff with our monorepo and learn a bit more about the Godep tool, and now I know more about why the stuff at the end of this post works.

The solution to my problem was to update the version of sqlmock that I was using. One of the potential issues with monorepos is that if you update a package, you have to update it across the entire monorepo, because of how Go dependencies work. One solution to this problem is to make whoever wants to change versions do all the updates across the repo. Another (less ‘Go’) solution is to have two versions of the dependency in your project. Which is what we did.

It took me a few tries to get everything down in a way that made both Godep and git happy, so here is my recommendation on the order of steps to take.

  1. Checkout your master branch and make sure it is up to date.

  2. Make sure your local Godeps/_workspace is up to date by running the godep restore command. I think this is the one I was having the most trouble with, because ‘restore’ is not what I think I want to do, conceptually. The restore command is (I think) making your local Godep/_workspace match what is in the Godep.json file. So if you’ve messed added ore removed a whole bunch of stuff, and want to start over, the ‘restore’ command is an accurate reflection of what you are doing, but it also works as an ‘update’, which is conceptually what I thought I wanted to do.

  3. Check your git status and make sure you don’t see anything unexpected. In my case, I didn’t expect to see really anything in my status because our Godep folder is checked into git.

  4. If you haven’t already, do a godep get PACKAGE/I/WANT.

  5. Update the import statement(s) in your actual *.go file(s).

  6. To save this into your dependencies, use godep save -r ./.... The -r flag takes care of re-writing any import statements to now point to your Godep/_workspace folder. The ./... makes sure it checks the current folder and all folders inside of it to catch any new imports.

  7. Again, check your git diff. At this point, I expected to see:

– the import changes
– the new package in Godep/_workspace

If it all looks good, commit!

  1. If this update was a breaking version change, now go back and fix your code that it compiles and runs again.

  2. Finish the thing that sent you down this rabbit hole in the first place!

Tips and Tricks: Intermediate Git configuration

Posted on by

There are several ‘levels’ at which you can configure git. The three I know about are:

  • repository-level
  • repository branch-level
  • global

They are slightly self explanatory.

  • repository branch-level means that the configurations/aliases are applied ONLY to a specific branch in that repository. I haven’t had a need to use this one yet, so I can’t tell you a ton about it.
  • repository-level means that the configurations/aliases are applied ONLY to that repository, but to all branches on that repository. This can be configured to use different email addresses to different projects (ex: work email vs personal email), among other things.
  • global means that the configurations/aliases are applied to ALL git repositories on your system.

So, how do we set this up?

From the command line

Most of the tutorials on how to set up aliases give the command line version.

In your git repository, you can type git config XXX (where XXX is filler for whatever configuration you want to set). To set the configurations globally, you can type git config --global XXX. Adding the --global flag sets it for all git repositories. Some examples:

git config --global user.name "Clare Glinka"
– This sets my name as “Clare Glinka” for all repos.

git config --global user.email myemail@example.com
– This sets my email as myemail@example.com for all repos.

git config merge.ff false
– disables fast-forward merges by default at the repository level. Fast-forward merging will be disabled for all branches in that repo. Adding the --global flag will disable fast-forward merging for ALL repos.

You can check out other configuration settings in the Git documentation on customizing your git configuration and the git-config manual page.

Now that we’ve set some configs, how do we look at them? (because nothing is real till we see the output, right?)

git config --list will list all the configuration settings for your current directory/branch, including things like your remotes.

git config --list --global will list all the configurations settings for your system. For me, this is a much shorter list, since it does not include things like remotes.

Edit .gitconfig

There is another way to set configurations that I find a little easier to look at sometimes. That is checking out the .gitconfig file. For your global settings, .gitconfig lives in your ~ directory, and you can open it up and edit it with your favorite text editor. You can check out your repository-level configuration file by looking at REPO/.git/config.

References

First-Time Git Setup
Customizing Git Configuration
git-config Manual Page
StackOverflow: Can I make fast forwarding be off by default in git?
Git Tip: git config user name and email for local (not global) config

Saving dotfiles with symlinks on OSX

Posted on by

I’ve always been impressed with folks who have their dotfiles all backed up to git somewhere, and I’ve finally worked out how to do it myself. The magic answer is symlinks!

STEP 1: DOTFILES

So, dotfiles are basically ‘hidden’ files that contain information about settings you don’t want to keep setting every time you try to do something. Lots of programs use dotfiles to keep track of user-specific configurations. For example. in ~/.gitconfig, Git stores information like your username and user email (and much, MUCH more).

STEP 2: Collect dotfiles into a Git repo

Mostly, dotfiles live in your home directory. But, you don’t really want to put everything in your home directory under git’s watchful eye, so the easier step is to collect all the dotfiles into a git repo that lives elsewhere. Pick a folder, ANY FOLDER! For best experience, I suspect it should live somewhere under your home directory. I’m currently keeping mine in ~/code/dotfiles.

Dotfiles you might want to keep under version control:
– Git config (~/.gitconfig)
– terminal config (~/.bash_profile, ~/.bashrc, ~/.zshrc, etc)
– basically anything were you’ve set prefs that are stored!

Hint The First

In order for the next step to work, you can’t have an existing copy of your dotfiles in your home directory. So if you want to put your .gitconfig under version control, for example, it is better to MOVE it instead of COPYING it.

Hint The Second

The ultimate point of this whole exercise (for me) was to have all my dotfiles backed up somewhere. If (like me) you are not yet ready to publicly publish your dotfiles on Github, there are a number of other Git-based services that will provide private repos for free.

STEP 3: Symlink!

The ln command is used to create links in your (unix-based) system. Use the --help command to get more info on the myriad of options available to you. Just using the ln command creates something called a ‘hard link’, which is not what we want. We want a ‘symbolic’ link instead. The syntax is ln -s + actual location of the file + name and location you want to see that file under.

Example:

ln -s ~/code/dotfiles/gitconfig ~/.gitconfig

will result in the gitconfig in the dotfiles directory to be accessible from the ~/.gitconfig location, which is where Git is expecting to see all the Git preferences you’ve set.

STEP 4: PROFIT!

RESOURCES:

I stumbled across this great article called How to Create and Use Symlinks on a Mac that made the lightbulb go off and instigated this whole song and dance.

Tips and Tricks: Gem pristine

Posted on by

A while ago, some friends and I were doing some code exploration inside gems, trying to figure out how they worked. All was well and good until we had changed everything and broke the gem. Here are a few great commands for playing around with this stuff, and then resetting it when you are done so everything on your system still works the way it is supposed to.

To open a gem in your default text editor and poke around it’s squishy inner bits:

bundle open GEMNAME

EX:

bundle open rake

Once you are done playing around and just want things to go back to the way they were, there are two good commands to know.

For when you have bundler set up to install gems to a local vendor directory, and that is the gem you were playing with:

bundle exec gem pristine GEMNAME

Or if you want to restore all the possible versions of a gem:

gem pristine GEMNAME

Happy hacking!

Tips and Tricks: JSON in IRB or Pry

Posted on by

Trick one: Parse your raw JSON objects when you pull them in.

For our Ruby Gem project, we were pulling down JSON-formatted data from the Wikipedia  API. To successfully work with JSON in IRB, I imported the object with an HTTP gem and then parsed the JSON data with the JSON gem. EX:

require 'json' 
require 'rest_client' 
JSON.parse(RestClient.get <URL>)

Which gives us a nice Hash output that looks like this:

{"query-continue"=&gt;
    {"images"=&gt;
        {"gimcontinue"=&gt;"736|Citizen-Einstein.jpg"}},
    "query"=&gt;
        {"pages"=&gt;
            {"-1"=&gt;{"ns"=&gt;6, "title"=&gt;"File:1919 eclipse positive.jpg", "missing"=&gt;""},
             "-2"=&gt;{"ns"=&gt;6, "title"=&gt;"File:Albert Einstein's exam of maturity grades (color2).jpg", "missing"=&gt;""}}}}

Without parsing, the information was still in JSON format, a thing that looked like:

"{\"query-continue\":{\"images\":{\"gimcontinue\":\"736|Citizen-Einstein.jpg\"}},\"query\":  ...

with extra quotation marks and all the backslashes, and Ruby got pretty cranky about trying to work with that string.

Trick Two: ‘puts’ your JSON

For another project, I was converting hashes into JSON, and I was getting a bit frustrated. I was quite certain that I was converting the hash into JSON correctly, but I kept getting extraneous backslash-escaped quotation marks in my JSON returns like before.

"{\"query-continue\":{\"images\":{\"gimcontinue\":\"736|Citizen-Einstein.jpg\"}}, ...

Thanks to some StackOverflow googling, I realized/remembered that this was because I was directly calling the JSON in the console, instead of puts-ing it from within the script I was running. When I used a puts statement inside the script, I could see that my output was actually formatted correctly, as I expected. Using puts in the console also worked to show me the JSON with it’s correct formatting.

puts <JSON>

instead of

<JSON>

Github repo confusion

Posted on by

Last week, I accidentally cloned down the wrong repo from GitHub. Whoops! I forked the repo for one of our class assignments, but I grabbed the URL for the MakerSquare repo, not mine. Fortunately, there was Google to the rescue!

Turns out, this is actually a little bit easy, though I read through the first 5 or so google search results before I did anything. This blog post was the most helpful because the commands were explained very nicely. It probably also helped that it was the 4th or so thing I read, so I was beginning to understand what I needed to do.

I did the following:

1. Check the current remote to make sure it is actually what I think it is

git origin -v

2. Remove the current origin

git remote rm origin

3. Add the new (correct!) origin

git remote add origin https://github.com/my_username/correct_repo.git

And tah-dah! All fixed! And in even nicer news, there is actually an even easier way of doing this that might be newer than said blog post. On GitHub Help, there is some information about using

git remote set-url

to do the same thing as the above step 3 and allows you to skip step 2 entirely.

Since this was an early mistake in a very simple single-branch repo, I didn’t have to take any other steps to make sure the HEAD was in the right spot or that any changes were correctly synced.