Replacing Git Submodules with GitMan

Lots of languages offer dependency managers (pip, gem, npm), but in many situations, that’s not enough. Sometimes you need to:

• use a language without a dependency manager
• include code from multiple languages
• explicitly control the installation location

Git Submodules

When using Git for version control, the obvious choice is to use submodules to include the source from another repository. However, in practice, submodules can often be a pain to use, as they:

• require extra information to meaningfully identify the submodule’s SHA
• cause confusing merge conflicts (one SHA vs. another)
• show confusing status changes when switching branches

And while submodules can be used to track a branch (rather than a SHA), this will:

• show confusing status changes if the branch head moves
• require a new commit by whomever updates submodules first

An Alternative

GitMan avoids these issues and adds the ability to:

• track a specific tag in a source dependency’s repository
• checkout by rev-parse dates (e.g. 'develop@{2015-06-18 10:30:59}')

Installation

To install GitMan, first install Python 3 and it’s dependency manager, pip:

• Windows: python.org/downloads
• Mac: $ brew install python3
• Ubuntu: $ sudo apt-get install python3-pip

Then, install gitman using pip3:

$ pip3 install --upgrade gitman

Version and help information are available on the command-line

$ gitman --version
$ gitman --help

Mimicking Submodule

While GitMan, provides additional capabilities, it can also directly replace the behavior of submodules. To mimic a working tree containing a submodule:

<root>/vendor/my_dependency  # submodule at: a5fe3d...

create a gitman.yml file in the root of your working tree:

location: gitman_sources
sources:
- repo: <URL of my_dependency's repository>
  dir: my_dependency
  rev: a5fe3d
  link: vendor/my_depenendy

and run:

$ gitman install

To display the specific versions of source dependencies:

$ gitman list

See a typo? Help me edit this post.

Find a problem with gitman? Please submit an issue or contribute!

YORM v0.4 Released

Yesterday, I released an important milestone of my file-based object relational mapper for Python, YORM. This release provides support for unlimited nesting of container-like attributes. This is a feature I’ve wanted for a while, but was actually quite difficult to implement. The API for YORM is also starting to stabilize after some breaking changes from the previous release.

Some Background

Lately, I’ve been running into many situations where I’d like to store program configuration and/or data in version control. YORM was born to provide automatic, bidirectional, and human-friendly mappings of Python object attributes to YAML files.

Traditional object serializes don’t provide output fit for human modification and ORM databases aren’t fit for storage in version control. YORM supports additional uses beyond typical object serialization and mapping including:

• bidirectional conversion between basic YAML and Python types
• attribute creation and type inference for new attributes
• storage of content in text files optimized for version control
• extensible converters to customize formatting on complex classes

An Example

Given an existing class:

class Student:
    def __init__(self, name, school, number, year=2009):
        self.name = name
        self.school = school
        self.number = number
        self.year = year
        self.gpa = 0.0

an attribute mapping is defined mapping attributes to converter classes and instances to a file pattern:

import yorm
from yorm.converters import String, Integer, Float

@yorm.attr(name=String, year=Integer, gpa=Float)
@yorm.sync("students/{self.school}/{self.number}.yml")
class Student:
    ...

Modifications to each object’s mapped attributes:

>>> s1 = Student("John Doe", "GVSU", 123)
>>> s2 = Student("Jane Doe", "GVSU", 456, year=2014)
>>> s1.gpa = 3

are automatically reflected on the filesytem:

$ cat students/GVSU/123.yml
name: John Doe
gpa: 3.0
school: GVSU
year: 2009

Modifications and new content in each mapped file:

$ echo "name: John Doe
> gpa: 1.8
> year: 2010
> expelled: true
" > students/GVSU/123.yml

are automatically reflected in their corresponding object:

>>> s1.gpa
1.8
>>> s1.expelled
True

Current Uses

Right now I’m using YORM to:

• store program state in Dropbox: mine
• simplify configuration file loading: GitMan
• prototype a RESTful game API: GridCommand

See a typo? Help me edit this post.

Find a problem with yorm? Please submit an issue or contribute!

Syncing iTunes using Dropbox with Mine

Many applications provide their own synchronization methods to enable usage on multiple computers, but what about those that don’t? It turns out that lots of programs are perfectly happy to have their files stored inside Dropbox rather than their typical location.

Storing iTunes in Dropbox

Many guides exist showing you how to do this with iTunes:

• digiwonk.wonderhowto.com/how-to/sync-your-itunes-library-with-several-computers-using-dropbox
• macstories.net/tutorials/how-to-sync-your-entire-itunes-library-with-dropbox

Unfortunately, the shared caveat in all these guides is that only one instance of iTunes is to be running at any given time. That’s where mine comes in.

Installing and Configuring Mine

mine is a daemon and command-line Python program that starts and stops remote applications using a configuration file in Dropbox. After setting up iTunes and Dropbox using one of the above guides, install mine:

$ pip3 install --upgrade mine

If you don’t have pip3, install python3 with your system’s package manager (on OSX with Homebrew: $ brew install python3).

Additional configuration instructions are found in the project’s README.

Using Mine to Manage Remote Applications

Once installed and configured, let it run in the background on each computer:

$ mine --daemon

Applications can be killed remotely and started on the current computer:

$ mine switch

To kill all local applications and start them on another computer:

$ mine switch <name>

where <name> is part of the name of another computer with mine running.

See a typo? Help me edit this post.

Find a problem with mine? Please submit an issue or contribute!

Measuring Coverage with XcodeCoverage, xctool, & Make

The Pieces

XcodeCoverage

The XodeCoverage project is a set of shell scripts bundled with lcov to measure lines of code coverage during execution of instrumented test builds.

In this example, the scripts are used to generate an HTML coverage report (with a few modifications made in my fork to customize report location).

xctool

Facebook created the xctool command-line program to provide an easier way to build and test Xcode projects.

In this example, the tool is used to build and run Objective-C unit tests from the command line.

Make

Make is usually my default entry point for creating builds, running tests, and generating reports. I like putting this sort of automation in a Makefile because, for basic tasks, the syntax is fairly minimal and make is ubiquitous on most platforms.

Putting Them Together

The Makefile

First, we define few shared variables that can be common to all projects:

WORKSPACE_NAME:=<???>
PROJECT_NAME:=<???>
SOURCE_NAME:=<???>
APP_NAME:=<???>

# Common
ROOT_DIR:=.
PROJECT_DIR:=$(ROOT_DIR)/$(SOURCE_NAME)
SOURCE_DIR:=$(PROJECT_DIR)/$(SOURCE_NAME)
SOURCES:=Makefile $(SOURCE_DIR)/*

# Xcode
XCODE_SCHEME?=$(APP_NAME)
XCODE_CONFIGURATION?=Debug

# xctool
XCTOOL:=xctool
XCTOOL_RESULTS_REPORTER?=pretty
XCTOOL_ARGS_SHARED:=-scheme $(XCODE_SCHEME) -configuration \
    $(XCODE_CONFIGURATION) -reporter user-notifications
XCTOOL_ARGS_TEST:=-reporter $(XCTOOL_RESULTS_REPORTER)

# XcodeCoverage
XCODECOVERAGE_DIR:=$(PROJECT_DIR)/XcodeCoverage
XCODECOVERAGE_GETCOV:=$(XCODECOVERAGE_DIR)/getcov
XCODECOVERAGE_CLEANCOV:=$(XCODECOVERAGE_DIR)/cleancov

and a few more variables dictating where we’d like coverage output to go:

COVERAGE_DIR:=$(ROOT_DIR)/coverage
COVERAGE_LOG:=$(COVERAGE_DIR)/getcov.log
COVERAGE_REPORT:=$(COVERAGE_DIR)/index.html

The test target is defined as:

.PHONY: test
test: $(COVERAGE_LOG)
$(COVERAGE_LOG): $(SOURCES)
    $(XCODECOVERAGE_CLEANCOV)
    $(XCTOOL) test $(XCTOOL_ARGS_SHARED) $(XCTOOL_ARGS_TEST)
    mkdir -p $(COVERAGE_DIR) && \
        $(XCODECOVERAGE_GETCOV) $(PROJECT_NAME) $(COVERAGE_DIR) > \
            $(COVERAGE_LOG)
    tail -n 3 $(COVERAGE_LOG)

which will:

1. Delete the old coverage data
2. Build and run the unit tests
3. Parse the generated coverage data
4. Generate an HTML coverage report
5. Display the percentage of lines coverage

A shortcut to open the coverage report is defined as:

.PHONY: read-cov
read-cov: $(COVERAGE_INDEX)
    open $(COVERAGE_INDEX)

$(COVERAGE_INDEX): $(COVERAGE_LOG)

Example Output

Running $ make test displays something like:

./MyProject/XcodeCoverage/cleancov
Deleting all .da files in /Users/Browning/Library/Developer/Xcode/DerivedData/MyWorkspace/Build/Intermediates/MyProject.build/Debug-iphonesimulator/MyProject.build/Objects-normal/x86_64 and subdirectories
Done.

xctool test -scheme MyProject -configuration Debug -reporter user-notifications -reporter pretty
[Info] Loading settings for scheme 'MyProject' ... (1950 ms)

=== TEST ===

  xcodebuild build build
    MyProject / MyProject (Debug)
      ✓ Check dependencies (111 ms)
      ✓ Write auxiliary files (0 ms)
      ✓ Compile BatterySensor.m (574 ms)
      ...
      ✓ Compile Platform.m (68 ms)
      0 errored, 0 warning (1315 ms)

  [Info] Collecting info for testables... (1196 ms)
  run-test MyProjectTests.xctest (iphonesimulator8.2, iPad Air, application-test)
    [Info] Installed 'MyProject'. (1392 ms)
    [Info] Launching test host and running tests ... (0 ms)
    ✓ -[BatterySensorCellTestCase testSensorLevel] (5 ms)
    ...
    ✓ -[GroupManagerTestCase testCanCreateGroup] (0 ms)
    99 passed, 0 failed, 0 errored, 99 total (9999 ms)

** TEST SUCCEEDED: 99 passed, 0 failed, 0 errored, 99 total ** (99999 ms)

mkdir -p ./coverage && ./MyProject/XcodeCoverage/getcov MyProject ./coverage > ./coverage/getcov.log

tail -n 3 ./coverage/getcov.log
Overall coverage rate:
  lines......: 17.6% (3676 of 20867 lines)
  functions..: 19.5% (817 of 4192 functions)

Running $ make read-cov launches a report similar to:

See a typo? Help me edit this post.

My New Site

I used to have a really old site built with iWeb. It’s time I started blogging and sharing interesting or useful bits of knowlede outside of Twitter and Speaker Deck.

Until I add more content, I’ll leave you with my resume.