Deploying a Jekyll Blog to GitHub Pages using Travis CI

Jekyll is a “blog-aware static site generator” written in Ruby that will generate a responsive website from Markdown and YAML files.

GitHub Pages offers you a free way to host static websites. And while they will automatically publish files in a repository named <your-github-id>.github.io, sometimes you need a little more control over the process.

Project Tooling

After running $ jekyll new, the base project structure will be generated for you. To simplify setup on additional machines and enable continuous deployment, I like to add additional tooling to this project.

Gemfile

A Gemfile is used to specify the Ruby dependencies (“Gems”) required to generate the website. In addition to jekyll, I’m using:

• rouge - to provide syntax highlighting in code examples
• html-proofer - to confirm that external links are valid

Gemfile.lock

The Gemfile.lock is automatically generated when installing Gems. It’s purpose is to record of the exact version of each dependency the last time the project was successfully deployed.

Bundler

bundler is a tool that can install and run particular versions of Gems for a project. I prefer to store the Gems locally via $ bundler install --path vendor so that I’m not polluting my system directories and can completely delete all files a project creates.

Makefile

A Makefile puts everything together. I like using make (rather than rake) because it doesn’t depend on Ruby itself, does a good of tracking when files have changed, and provides a standard interface between projects.

I’ll highlight the important parts of this file. This reinstalls the dependencies whenever they change:

VENDOR_DIR := vendor
INSTALLED_FLAG := $(VENDOR_DIR)/.installed

.PHONY: install
install: $(INSTALLED_FLAG)
$(INSTALLED_FLAG): Gemfile* Makefile
    bundle install --path vendor
    @ touch $(INSTALLED_FLAG)

This builds the site and validates the generated HTML:

.PHONY: build
build: install
    bundle exec jekyll build --quiet
    bundle exec htmlproof _site --only-4xx

And these targets provide a way to run the site locally:

.PHONY: run
run: install
    bundle exec jekyll serve --future --drafts

.PHONY: launch
launch: install
    eval "sleep 5; open http://localhost:4000" & make run

See the source code that generates this site as an example of how these files are used together in practice.

Developing Locally

Using the above tooling, my only system dependencies are ruby, bundler, and make. To work on a new blog entry, I simply run:

$ make launch

to bring up a local instance of the site that is regenerated whenever I edit content.

When I am done editing, running $ make ci will confirm that the site is ready to be published.

Deploying with Travis CI

Travis CI offers free continuous integration for open source projects that can be used to deploy software after running a number of checks.

Adding a .travis.yml to your project tells Travis CI how to build and deploy your site. This specifies which commands to run to install and validate each commit:

install:
- make install

script:
- make ci

If a new commit passes those checks, the following shell script is run:

# Generate HTML
make build ;
# Configure Git with Travis CI information
git config --global user.email "[email protected]" ;
git config --global user.name "travis-ci" ;
# Delete the current repository
rm -rf .git ;
# Rebuild the repository from the generated files and push to GitHub pages
cd _site ;
git init ;
git add . ;
git commit -m "Deploy Travis CI build $TRAVIS_BUILD_NUMBER to GitHub pages" ;
git push -f https://${GH_TOKEN}@github.com/${TRAVIS_REPO_SLUG} main:gh-pages ;

which will publish the generated files to the gh-pages branch on GitHub.

GH_TOKEN is an encrypted access token to grant Travis CI permissions to modify files in your repository. This site provides a good overview on generating these tokens.

After deployment, you should now see your Jekyll blog live at:

https://<your-github-id>.github.io/<your-repository-name>

See a typo? Help me edit this post.

Replacing Git Submodules with GitMan

Lots of languages offer dependency managers (pip, gem, npm), but in many situations, that’s not enough. Sometimes you need to:

• use a language without a dependency manager
• include code from multiple languages
• explicitly control the installation location

Git Submodules

When using Git for version control, the obvious choice is to use submodules to include the source from another repository. However, in practice, submodules can often be a pain to use, as they:

• require extra information to meaningfully identify the submodule’s SHA
• cause confusing merge conflicts (one SHA vs. another)
• show confusing status changes when switching branches

And while submodules can be used to track a branch (rather than a SHA), this will:

• show confusing status changes if the branch head moves
• require a new commit by whomever updates submodules first

An Alternative

GitMan avoids these issues and adds the ability to:

• track a specific tag in a source dependency’s repository
• checkout by rev-parse dates (e.g. 'develop@{2015-06-18 10:30:59}')

Installation

To install GitMan, first install Python 3 and it’s dependency manager, pip:

• Windows: python.org/downloads
• Mac: $ brew install python3
• Ubuntu: $ sudo apt-get install python3-pip

Then, install gitman using pip3:

$ pip3 install --upgrade gitman

Version and help information are available on the command-line

$ gitman --version
$ gitman --help

Mimicking Submodule

While GitMan, provides additional capabilities, it can also directly replace the behavior of submodules. To mimic a working tree containing a submodule:

<root>/vendor/my_dependency  # submodule at: a5fe3d...

create a gitman.yml file in the root of your working tree:

location: gitman_sources
sources:
- repo: <URL of my_dependency's repository>
  dir: my_dependency
  rev: a5fe3d
  link: vendor/my_depenendy

and run:

$ gitman install

To display the specific versions of source dependencies:

$ gitman list

See a typo? Help me edit this post.

Find a problem with gitman? Please submit an issue or contribute!

YORM v0.4 Released

Yesterday, I released an important milestone of my file-based object relational mapper for Python, YORM. This release provides support for unlimited nesting of container-like attributes. This is a feature I’ve wanted for a while, but was actually quite difficult to implement. The API for YORM is also starting to stabilize after some breaking changes from the previous release.

Some Background

Lately, I’ve been running into many situations where I’d like to store program configuration and/or data in version control. YORM was born to provide automatic, bidirectional, and human-friendly mappings of Python object attributes to YAML files.

Traditional object serializes don’t provide output fit for human modification and ORM databases aren’t fit for storage in version control. YORM supports additional uses beyond typical object serialization and mapping including:

• bidirectional conversion between basic YAML and Python types
• attribute creation and type inference for new attributes
• storage of content in text files optimized for version control
• extensible converters to customize formatting on complex classes

An Example

Given an existing class:

class Student:
    def __init__(self, name, school, number, year=2009):
        self.name = name
        self.school = school
        self.number = number
        self.year = year
        self.gpa = 0.0

an attribute mapping is defined mapping attributes to converter classes and instances to a file pattern:

import yorm
from yorm.converters import String, Integer, Float

@yorm.attr(name=String, year=Integer, gpa=Float)
@yorm.sync("students/{self.school}/{self.number}.yml")
class Student:
    ...

Modifications to each object’s mapped attributes:

>>> s1 = Student("John Doe", "GVSU", 123)
>>> s2 = Student("Jane Doe", "GVSU", 456, year=2014)
>>> s1.gpa = 3

are automatically reflected on the filesytem:

$ cat students/GVSU/123.yml
name: John Doe
gpa: 3.0
school: GVSU
year: 2009

Modifications and new content in each mapped file:

$ echo "name: John Doe
> gpa: 1.8
> year: 2010
> expelled: true
" > students/GVSU/123.yml

are automatically reflected in their corresponding object:

>>> s1.gpa
1.8
>>> s1.expelled
True

Current Uses

Right now I’m using YORM to:

• store program state in Dropbox: mine
• simplify configuration file loading: GitMan
• prototype a RESTful game API: GridCommand

See a typo? Help me edit this post.

Find a problem with yorm? Please submit an issue or contribute!

Syncing iTunes using Dropbox with Mine

Many applications provide their own synchronization methods to enable usage on multiple computers, but what about those that don’t? It turns out that lots of programs are perfectly happy to have their files stored inside Dropbox rather than their typical location.

Storing iTunes in Dropbox

Many guides exist showing you how to do this with iTunes:

• digiwonk.wonderhowto.com/how-to/sync-your-itunes-library-with-several-computers-using-dropbox
• macstories.net/tutorials/how-to-sync-your-entire-itunes-library-with-dropbox

Unfortunately, the shared caveat in all these guides is that only one instance of iTunes is to be running at any given time. That’s where mine comes in.

Installing and Configuring Mine

mine is a daemon and command-line Python program that starts and stops remote applications using a configuration file in Dropbox. After setting up iTunes and Dropbox using one of the above guides, install mine:

$ pip3 install --upgrade mine

If you don’t have pip3, install python3 with your system’s package manager (on OSX with Homebrew: $ brew install python3).

Additional configuration instructions are found in the project’s README.

Using Mine to Manage Remote Applications

Once installed and configured, let it run in the background on each computer:

$ mine --daemon

Applications can be killed remotely and started on the current computer:

$ mine switch

To kill all local applications and start them on another computer:

$ mine switch <name>

where <name> is part of the name of another computer with mine running.

See a typo? Help me edit this post.

Find a problem with mine? Please submit an issue or contribute!

Measuring Coverage with XcodeCoverage, xctool, & Make

The Pieces

XcodeCoverage

The XodeCoverage project is a set of shell scripts bundled with lcov to measure lines of code coverage during execution of instrumented test builds.

In this example, the scripts are used to generate an HTML coverage report (with a few modifications made in my fork to customize report location).

xctool

Facebook created the xctool command-line program to provide an easier way to build and test Xcode projects.

In this example, the tool is used to build and run Objective-C unit tests from the command line.

Make

Make is usually my default entry point for creating builds, running tests, and generating reports. I like putting this sort of automation in a Makefile because, for basic tasks, the syntax is fairly minimal and make is ubiquitous on most platforms.

Putting Them Together

The Makefile

First, we define few shared variables that can be common to all projects:

WORKSPACE_NAME:=<???>
PROJECT_NAME:=<???>
SOURCE_NAME:=<???>
APP_NAME:=<???>

# Common
ROOT_DIR:=.
PROJECT_DIR:=$(ROOT_DIR)/$(SOURCE_NAME)
SOURCE_DIR:=$(PROJECT_DIR)/$(SOURCE_NAME)
SOURCES:=Makefile $(SOURCE_DIR)/*

# Xcode
XCODE_SCHEME?=$(APP_NAME)
XCODE_CONFIGURATION?=Debug

# xctool
XCTOOL:=xctool
XCTOOL_RESULTS_REPORTER?=pretty
XCTOOL_ARGS_SHARED:=-scheme $(XCODE_SCHEME) -configuration \
    $(XCODE_CONFIGURATION) -reporter user-notifications
XCTOOL_ARGS_TEST:=-reporter $(XCTOOL_RESULTS_REPORTER)

# XcodeCoverage
XCODECOVERAGE_DIR:=$(PROJECT_DIR)/XcodeCoverage
XCODECOVERAGE_GETCOV:=$(XCODECOVERAGE_DIR)/getcov
XCODECOVERAGE_CLEANCOV:=$(XCODECOVERAGE_DIR)/cleancov

and a few more variables dictating where we’d like coverage output to go:

COVERAGE_DIR:=$(ROOT_DIR)/coverage
COVERAGE_LOG:=$(COVERAGE_DIR)/getcov.log
COVERAGE_REPORT:=$(COVERAGE_DIR)/index.html

The test target is defined as:

.PHONY: test
test: $(COVERAGE_LOG)
$(COVERAGE_LOG): $(SOURCES)
    $(XCODECOVERAGE_CLEANCOV)
    $(XCTOOL) test $(XCTOOL_ARGS_SHARED) $(XCTOOL_ARGS_TEST)
    mkdir -p $(COVERAGE_DIR) && \
        $(XCODECOVERAGE_GETCOV) $(PROJECT_NAME) $(COVERAGE_DIR) > \
            $(COVERAGE_LOG)
    tail -n 3 $(COVERAGE_LOG)

which will:

1. Delete the old coverage data
2. Build and run the unit tests
3. Parse the generated coverage data
4. Generate an HTML coverage report
5. Display the percentage of lines coverage

A shortcut to open the coverage report is defined as:

.PHONY: read-cov
read-cov: $(COVERAGE_INDEX)
    open $(COVERAGE_INDEX)

$(COVERAGE_INDEX): $(COVERAGE_LOG)

Example Output

Running $ make test displays something like:

./MyProject/XcodeCoverage/cleancov
Deleting all .da files in /Users/Browning/Library/Developer/Xcode/DerivedData/MyWorkspace/Build/Intermediates/MyProject.build/Debug-iphonesimulator/MyProject.build/Objects-normal/x86_64 and subdirectories
Done.

xctool test -scheme MyProject -configuration Debug -reporter user-notifications -reporter pretty
[Info] Loading settings for scheme 'MyProject' ... (1950 ms)

=== TEST ===

  xcodebuild build build
    MyProject / MyProject (Debug)
      ✓ Check dependencies (111 ms)
      ✓ Write auxiliary files (0 ms)
      ✓ Compile BatterySensor.m (574 ms)
      ...
      ✓ Compile Platform.m (68 ms)
      0 errored, 0 warning (1315 ms)

  [Info] Collecting info for testables... (1196 ms)
  run-test MyProjectTests.xctest (iphonesimulator8.2, iPad Air, application-test)
    [Info] Installed 'MyProject'. (1392 ms)
    [Info] Launching test host and running tests ... (0 ms)
    ✓ -[BatterySensorCellTestCase testSensorLevel] (5 ms)
    ...
    ✓ -[GroupManagerTestCase testCanCreateGroup] (0 ms)
    99 passed, 0 failed, 0 errored, 99 total (9999 ms)

** TEST SUCCEEDED: 99 passed, 0 failed, 0 errored, 99 total ** (99999 ms)

mkdir -p ./coverage && ./MyProject/XcodeCoverage/getcov MyProject ./coverage > ./coverage/getcov.log

tail -n 3 ./coverage/getcov.log
Overall coverage rate:
  lines......: 17.6% (3676 of 20867 lines)
  functions..: 19.5% (817 of 4192 functions)

Running $ make read-cov launches a report similar to:

See a typo? Help me edit this post.