Using git-notes for marking test suite successes

The libinput test suite takes somewhere around 35 minutes now for a full run. That's annoying, especially as I'm running it for every commit before pushing. I've tried optimising things, but attempts at making it parallel have mostly failed so far (almost all tests need a uinput device created) and too many tests rely on specific timeouts to check for behaviours. Containers aren't an option when you have to create uinput devices so I started out farming out into VMs.

Ideally, the test suite should run against multiple commits (on multiple VMs) at the same time while I'm working on some other branch and then accumulate the results. And that's where git notes come in. They're a bit odd to use and quite the opposite of what I expected. But in short: a git note is an object that can be associated with a commit, without changing the commit itself. Sort-of like a post-it note attached to the commit. But there are plenty of limitations, for example you can only have one note (per namespace) and merge conflicts are quite easy to trigger. Look at any git notes tutorial to find out more, there's plenty out there.

Anyway, dealing with merge conflicts is a no-go for me here. So after a bit of playing around, I found something that seems to work out well. A script to run make check and add notes to the commit, combined with a repository setup to fetch those notes and display them automatically. The core of the script is this:

make check
rc=$?
if [ $? -eq 0 ]; then
    status="SUCCESS"
else
    status="FAIL"
fi

if [ -n "$sha" ]; then
    git notes --ref "test-$HOSTNAME" append \
        -m "$status: $HOSTNAME: make check `date`" HEAD
fi
exit $rc

Then in my main repository, I add each VM as a remote, adding a fetch path for the notes:

[remote "f22-libinput1"]
        url = f22-libinput1.local:/home/whot/code/libinput
        fetch = +refs/heads/*:refs/remotes/f22-libinput1/*
        fetch = +refs/notes/*:refs/notes/f22-libinput1/*

Finally, in the main repository, I extended the glob that displays notes to 'everything':

$ git config notes.displayRef "*" 

Now git log (and by extension tig) displays all notes attached to a commit automatically. All that's needed is a git fetch --all to fetch everything and it's clear in the logs which commit fails and which one succeeded.

:: whot@jelly:~/code/libinput (master)> git log
commit 6896bfd3f5c3791e249a0573d089b7a897c0dd9f
Author: Peter Hutterer 
Date:   Tue Jul 14 14:19:25 2015 +1000

    test: check for fcntl() return value
    
    Mostly to silence coverity complaints.
    
    Signed-off-by: Peter Hutterer 

Notes (f22-jelly/test-f22-jelly):
    SUCCESS: f22-jelly: make check Tue Jul 14 00:20:14 EDT 2015

Whenever I look at the log now, I immediately see which commits passed the test suite and which ones didn't (or haven't had it run yet). The only annoyance is that since a note is attached to a commit, amending the commit message or rebasing makes the note "go away". I've copied notes manually after this, but it'd be nice to find a solution to that.

Everything else has been working great so far, but it's quite new so there'll be a bit of polishing happening over the next few weeks. Any suggestions to improve this are welcome.

Using git - the next level

There's a million tutorials out there how to learn git. This isn't one of them. I'm going to assume that you learned git a while ago, you've been using it a bit and you're generally familiar with its principles. I'm going to show is a couple of things that improved my workflow. Chances are, it will improve yours too. This isn't a tutorial though. I'm just pointing you in the direction of things, you'll have to learn how to use them yourself.

Use tig

Seriously. Don't tell me you use gitk or git log is good enough for you. Use tig. tig is to git log what mutt is to mail(1). It has been the source of the biggest efficiency increase for me. Screenshots don't do it justice because the selling point is that it is interactive. But anyway, here are some official screenshots: tig blame shows you the file and the commits, you just need to select the line, hit enter and you see the actual commit. The main view by default shows you tags, branch names, remote branch names, etc. So not only do you immediately know which branch you're on, you will see local branches that have been merged, tags that have been applied, etc. It gives you an awareness that git log doesn't. Do yourself a favour, install it, use it for a day or two and I'm pretty sure you won't go back.

tig also supports custom configurations. Here is my $HOME/.tigrc:

bind generic X !git cherry-pick -x %(commit)
bind generic C !git cherry-pick %(commit)
bind generic R !git revert %(commit)
bind generic E !git format-patch -1 %(commit)
bind generic 0 !git checkout %(commit)
bind generic 9 !git checkout %(commit)~
bind generic A !git commit --amend -s
bind generic S !git show %(commit)

So with a couple of key strokes I can cherry-pick, export patches, revert, check out a single tree, etc. Especially cherry-picking is extremely efficient: check out the target branch, run "tig master", then simply select each commit, it "C" or "X" and done.

Use branches

Anytime it takes you more than 5 minutes to fix an issue, create a new branch. I'm getting torn between multiple things all the time. I may spend a day or two on one bug, then it's back to another, unrelated issue. With the review requirements on some projects I may have multiple patches waiting for feedback, but I can't push them yet. Hence - a branch for each feature/bugfix. master is reserved for patches that can be pushed immediately.

This approach becomes particularly useful for fixes that may need some extra refacturing. You start on a feature-based branch, but halfway through realise you need a few extra patches to refactor things. Those are easy to review so you send them out to gather reviews, then cherry-pick them to master and push. Back to your feature branch, rebase and you're done - you've managed two separate streams of fixes without interference. And most importantly, you got rid of a few patches that you'd otherwise have to carry in your feature branch.

Of course, it takes a while to get used this and it takes discipline. It took me a few times before I really managed to always work like this but the general rule for me is now: if I'm hacking on the master branch, something is off. Remember: there's no real limit to how many branches you can create - just make sure you clean them up when you're done to keep things easy for your brain.

Use the branch names to help you. You can rename branches (git branch -m), so I tend to name anything that's a bigger rewrite with "wip/somefeature" whereas normal bug fixes go on branches with normal names. And because I rebase local feature branches it doesn't matter what I name them anyway, the branches are deleted once I merge them. Branches where I do care about the branch history (i.e. those I pull them into master with a merge commit) I rename before pulling to get rid of the "wip" prefix.

Use branch descriptions

Hands up if you have a "devel" branch from 4 months ago. Hands up if you still remember what the purpose of that branch was. Right, I didn't think so. git branch --edit-description fires up an editor and lets you add a description for the branch. Sometimes a single sentence is enough to refresh your memory. Most importantly: when you task-switch to a different feature, edit the description to note where you left off, what the plan was, etc. This reduces the time to get back to work. git config branch.<branchname>.description shows you the description for the matching branch.

I even have a git hook to nag me when I check out a branch without a description. Note that branch descriptions are local only, they are not pushed to the remote.

Amend and rebase until the cows come home

The general rule: what is committed, doesn't get lost. At least not easily, it is still in the git reflog. So commit when you think you're done. Then review, test, add, and git commit --amend. That typo you made in line 4 - edit and amend. I have shell aliases for amend, rbs (git rebase -i) and rbc (git rebase --continue), and almost every commit goes through at least 3 amends (usually one because I missed something, one for that typo, one for commit log message editing). Importantly: it doesn't matter how often you amend. Really. This is local only, no-one cares. The important thing is that you get to a good patch set, not that you get there with one commit.

git commit --amend only modifies the last commit, to go back and edit the past, you need to rebase. So, you need to

Learn how to rebase

Not just the normal git rebase, the tutorials cover that. Make sure you know how to use git rebase --interactive. Make sure you know how to change the ordering of a commit, how to delete commits, how to abort a rebase. Make sure you know how to squash two commits together and what the difference is between squash and fixup. I'm not going to write a tutorial on that, because you can find the documentation is easy enough to find. Simply take this as a hint that the time you spend learning how to rebase pays off. Also, you may find git squash interesting.

And remember: even if a rebase goes bad, the previous state is still in the reflog. Which brings me to:

Learn how to use the reflog

The git reflog is the list of changes in reverse chronological order of how they were applied to the repository, regardless what branch you're on. So HEAD@{0} is always "whatever we have now", HEAD@{1} is always "the repository before the last command". This doesn't just mean commits, it remembers any change. So if you switch from branch A to branch B, commit something, then switch to branch C, HEAD@{3} is A. git reflog helpfully annotates everything with the type, so you know what actually happened. So for example, if you accidentally dropped a patch during a rebase, you can look at the reflog, figure out when the rebase started. Then you either reset to that commit, or you just tig it and cherry-pick the missing commits back onto the current branch. Create yourself up with a test git repository and learn how to do exactly that now, it'll save you some time in the future.

Note that the reflog is local only. And remember, if it hasn't been committed, it's not in the reflog.

Use a git push hook

Repeat after me: echo make > .git/hooks/pre-push. And no more embarrassment for pushing patches that don't compile. I've made that mistake too many times, so now I even use my own git patch-set command that will run a hook for me when I'm generating a patch set to send to a list. You might want to make the hooks executable btw.

git-branch-tools: creating patch sets

git-branch-tools is my little repo for git scripts to make a few things easier. I first talked about it here. The repository is available on https://github.com/whot/git-branch-tools, the latest addition is git patch-set. I used to create git patch sets with just git format-patch, but too often I found some minor change on the last review and had to re-generate it. So ended up with multiple patch files in the directory, or worse, a combination of old and new ones in danger of being sent by git send-email later. git-patch-set fixes this for me:

$> git patch-set HEAD~2
patches/patches-201309130933-HEAD~2/0001-test-provide-wrapper-for-fetching-the-devnode-from-a.patch
patches/patches-201309130933-HEAD~2/0002-wrap-EVIOCSCLOCKID-into-an-API-call.patch

So my patches are in the $GIT_DIR/patches/ directory, named after the current date + time and the refs used for the list. This makes them identifiable and sortable (to some degree anyway). And, to make things easier, $GIT_DIR/patches/latest is a symlink to the latest patch set, so usually the workflow is

$> git patch-set HEAD~2
patches/patches-201309130933-HEAD~2/0001-test-provide-wrapper-for-fetching-the-devnode-from-a.patch
patches/patches-201309130933-HEAD~2/0002-wrap-EVIOCSCLOCKID-into-an-API-call.patch
$> git send-email patches/latest/*.patch

That's not all though. I've added two hooks, pre-patch-set and post-patch-set to be run before/after the actual patch generation.

$> cat .git/hooks/pre-patch-set
#!/bin/bash -e
echo "running make check"
make check
$> git patch-set HEAD~2
running make check
Making check in doc
doxygen libevdev.doxygen
Making check in libevdev
make  check-am
make[2]: Nothing to be done for `check-am'.
Making check in tools
make[1]: Nothing to be done for `check'.
Making check in test
make  check-TESTS check-local
PASS: test-libevdev
make[4]: Nothing to be done for `all'.
============================================================================
Testsuite summary for libevdev 0.3
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
  GEN      gcov-report.txt
========== coverage report ========
libevdev-uinput.c: total lines: 172 not tested: 28 (83%)
libevdev.c: total lines: 689 not tested: 78 (88%)
========== =============== ========
patches/patches-2013091309:33-HEAD~2/0001-test-provide-wrapper-for-fetching-the-devnode-from-a.patch
patches/patches-2013091309:33-HEAD~2/0002-wrap-EVIOCSCLOCKID-into-an-API-call.patch

I've been using that script for quite a while now and it did make sending patch sets a bit easier. Plus, now I'm not in danger of sending out patch sets that don't pass make check :)

Making git reset less dangerous

If you've ever fat-fingered a git reset HEAD~2 into a git reset HEAD~23 then this may be of interest to you. If you have, add this to your .gitconfig:

[alias]

	r = !sh -c \"git log -1 --format=oneline\" && git reset

Using git r instead of git reset now prints the current HEAD before resetting.

$ git r --hard HEAD~23

9d09ffc3ba1a65fc7feefd21abd5adacf3274628 dix: NewCurrentScreen must work on pointers where possible

HEAD is now at 5083b56 Revert "mi: move switching screen on dequeue to separate helper function"

Whoops. Fat-fingered. A git r --hard 9d09ffc3ba1a65fc7feefd21abd5adacf3274628 will undo the above and get you back to your previous HEAD.

Note: git fsck --lost-found will also help you to find the commit again, but it'll take more time and effort.

Update: as mjg59 pointed out to me, git reflog records git resets, thus making it easy to find the previous HEAD too. Wish I'd known that command sooner ;)

Using environment git trees

Despite the confusing title, this is not tree-hugger territory ;)

I have two main machines and several test boxes. More often than not, I end up installing a new machine and get new home directory. This is annoying, because I've gotten quite used to my zsh setup, my vim customisations and all the other little tweaks my systems have accumulated over the years.

Somewhen around last year, this really started pissing me off so I set up a git tree for all those configurations, but I have now split that git tree into two trees: one called environment and one called shellenv.

Here's the current list of files in shellenv:

:: whot@barra:~/shellenv (master)> git ls-files
.gitconfig
.screenrc
.ssh/authorized_keys2
.tigrc
.zsh/10-exports
.zsh/20-aliases
.zsh/30-chpwd
.zsh/30-compctl
.zsh/30-keybindings
.zsh/40-xinput-compctl
.zsh/90-prompt
.zsh/func.tmp
.zsh/func/_fedpkg
.zshrc
setup.sh

The list of files in environment includes my mutt setup, my $HOME/scripts directory with all scripts to automate various stuff, my irssi setups, etc. Both repositories have a setup.sh:

#!/bin/bash

pwd=$PWD
filename=`basename $0`
excludes=". .. .git $filename"

for file in `ls -a`; do
    skip=0
    for exclude in $excludes; do
        if test $file = $exclude; then
            skip=1
            break
        fi
    done

    if test $skip -eq 1; then
        continue
    fi

    if ! test -e "$HOME/$file"; then
        echo ln -s "$pwd/$file" "$HOME/$file"
        ln -s "$pwd/$file" "$HOME/$file"
    elif test -d "$file"; then
        echo "$file already exists"
    else
        echo "$file already exists"
    fi
done

All this does is to symlink the target file with the current file (provided if the target does not exist). The script used in my environment tree is a tad more complicated since it takes hostnames into account to populate the various /etc directories as well but you get the gist.

The result of all that? I can install a new test machine, run git clone and ./setup.sh and the machine has my shell environment. I get a new actual machine (or I update any of my local configuration) and I can run git clone or git pull and all my main boxes have the identical setup. Do try this at home.


Note: I know there are a few bugs in the symlink scripts. I always run them the same way though, they work for me and it's not really worth my time right now to try to make them perfect

Adding Reviewed-by tags to patches

I merge a lot of patches and (thankfully!) I get a lot of Reviewed-by, Acked-by, etc. comments on both my patches and others that I end up merging. Copying those is painful and quiet annoying. Somewhen last year, I stumbled across snipMate which allows you to add autocompletion snippets to vim. My $HOME/.vim/snippets/gitcommit file now looks like this:

snippet ack
        Acked-by: ${1}
snippet rev
        Reviewed-by: ${1}
snippet sob
        Signed-off-by: ${1}
snippet tested
        Tested-by: ${1}
snippet me
        Peter Hutterer <peter.hutterer@...>

and the names of the common patch reviewers. So whenever I comment on a patch or I amend a commit after receiving reviews, just typing "rev<space>name alias<space>" is enough to add the Reviewed-by: line.

snipMate is quite powerful but so far I haven't used it much beyond the above. But even just that saved me numerous hours.

On commit messages

In the last few weeks, I've had a surprising number of discussions about commit messages. Many of them were with developers new to a project, trying to get them started. So here's a list of things you should do when committing, and why you should do it. Hint: the linux kernel mailing list gets it right, go there to learn.

Any software project is a collaborative project. It has at least two developers, the original developer and the original developer a few weeks or months later when the train of thought has long left the station. This later self needs to reestablish the context of a particular piece of code each time a new bug occurs or a new feature needs to be implemented.

Re-establishing the context of a piece of code is wasteful. We can't avoid it completely, so our efforts should go to reducing it to as small as possible. Commit messages can do exactly that and as a result, a commit message shows whether a developer is a good collaborator.

A good commit message should answer three questions about a patch:

• Why is it necessary? It may fix a bug, it may add a feature, it may improve performance, reliabilty, stability, or just be a change for the sake of correctness.


• How does it address the issue? For short obvious patches this part can be omitted, but it should be a high level description of what the approach was.


• What effects does the patch have? (In addition to the obvious ones, this may include benchmarks, side effects, etc.)

These three questions establish the context for the actual code changes, put reviewers and others into the frame of mind to look at the diff and check if the approach chosen was correct. A good commit message also helps maintainers to decide if a given patch is suitable for stable branches or inclusion in a distribution.

A patch without these questions answered is mostly useless. The burden for such a patch is on each and every reviewer to find out what the patch does and how it fixes a given issue. Given a large number of reviewers and a sufficiently complex patch, this means many man-hours get wasted just because the original developer did not write a good commit message. Worse, if the maintainers of the project enforce SCM discipline, they will reject the patch and the developer needs to spend time again to rewrite the patch, reviewers spend time reviewing it again, etc. The time wasted quickly multiplies and given that a commit message only takes a few minutes to write, it is simply not economically viable to omit them or do them badly.

Consider this is a hint for proprietary software companies too - not having decent SCM discipline costs money!

How to do it better

There's no strict definition of the ideal commit message, but some general rules have emerged.
A commit should contain exactly one logical change. A logical change includes adding a new feature, fixing a specific bug, etc. If it's not possible to describe the high level change in a few words, it is most likely too complex for a single commit. The diff itself should be as concise as reasonably possibly and it's almost always better to err on the side of too many patches than too few. As a rule of thumb, given only the commit message, another developer should be able to implement the same patch in a reasonable amount of time.

If you're using git, get familiar with "git add -p" (or -i) to split up changes into logical commits.

The git commit format

If you're submitting patches for git, the format is mostly standardised. A short one-line summary of the change (the maximum length of the line differs between projects, it's usually somewhere between 50 and 78 characters). This is the line that'll be seen most often, make it count. Many git tools are in one way or another optimised for this format. After that one-line summary, an empty line, then multiple paragraphs explaining the patch in detail (if needed). Don't describe the code, describe the intent and the approach. And keep the log in a present tense.

Learn to love the log

I have used CVS (and SVN to a lesser extent) in the past and log was a tool that was hardly ever used. Mostly because it was pretty useless, both the tool and the information available. These days I look at git logs more often than at code. The git log tool is vastly superior to CVS log and the commit discipline in the projects I'm working on now is a lot better. I grep git logs more often than code files and I use git blame all the time to figure out why a particular piece of code looks the way it does. It's certainly saving me a lot of time and effort. It's come to the point where the most annoying X server bugs are the ones where the git history stops at the original import from XFree86. If you're not using your SCM's log tool yet, I recommend to get more familiar with it.

How not to do it

There's a bunch of common sins that are committed (yay, a pun!) regularly.

• SCM is not a backup system! My personal pet hate. Developers who use it as such tend to do end-of-day commits, checking in everything at the end of the day. The result is useless, a random diff across the code with changes that are impossible to understand by anyone including the original author once a few months have passed. (On this note: universities, please stop teaching this crap).


• Per-file commit. More often than not a logical change affects more than one file and it should not be split up into two commits.


• Lazy commit messages, any commit labelled as "misc fixes and cleanups" or similar. I've seen my fair share of those on non-FOSS projects and they always come back to bite you. Impossible to find when a bug was introduced, hard to bisect and makes it harder for anyone else to keep track of what's happening in the project.


• Two changes in one patch. Something like "Fixed bug 2345 and renamed all foo to bar". Unless bug 2345 required the renaming, fixes whould be split it up into multiple patches. Others may have to take one of those bug fixes and apply it to a stable branch but not the other one. Picking bad patches apart into useful chunks is one of the most time-consuming and frustrating things I've done since it doesn't actually add any value to the project.
  


• Whitespace changes together with code changes. Needle in a haystack is a fun game, but not when you're looking at patches. It's a great way to introduce bugs, though because almost no-one will spot the bug hidden in hundreds of lines that got reindented for fun and profit.


• The ever-so-lovely code drops. Patches with hundreds of lines of code to dump a new feature into the code while at the same time rewriting half the existing infrastructure to support this feature. As a result, those hundreds of lines of code need to be reviewed every time a bug is discovered that is somehow related to that area of code.
  It's easier and less time consuming to first rework the infrastructure one piece at a time, then plug the new feature on top. As a side-effect, if a project relies on code dumps too often it's discouraging outside developers. Would you like to contribute to a project where the time spent filtering the signal from the noise outweighs the actual contribution to the code?
  


• Unrelated whitespace changes in patches. A reviewer needs to get the big picture of a patch into their brains. Whitespace-only hunks just confuse, a reviewer has to look extra hard to check if there's a real change or whether it can be ignored. That's not so bad for empty lines added or removed,it's really bad for indentation changes.
  

There's plenty of excuses for the above, with the favourite one being "but it works!". It may work, but code is not a static thing. In a few weeks time, that code may have moved, been rewritten, may be called in a different manner or may have a bug in it. At the same time, the original developer may have moved on and no-one knows why the code is that way. In the worst case, everyone is afraid of touching it because nobody knows how it actually works.

Another common excuse is the "but I'm the only one working on it". Not true, any software project is a collaborative project (see above). Assuming that there's no-one else is simply short-sighted. In FOSS projects specifically we rely on outside contributors, be it testers, developers, triagers, users, etc. The harder it becomes for them to join, the more likely the project will fail.

Another, less common excuse these days is that the SCM used is too slow. Distributed SCMs avoid this issue, saving time and by inference money.

Bisecting your favourite bug.

Ran into a new bug? Wasn't there with the last version? Want a developer to fix it? Too easy, just bisect it. And here's how:

First of all, we assume you've been running whatever your distribution ships. Find out that version and find the upstream repository. Now clone that repository and check out the matching version. For the example below, let's assume 1.2 is fine and 1.3 is broken.

$> git clone git://host/path/to/project
$> cd project
$> git tag
# find tag name for the broken version
$> git checkout project-1.3 # the broken version
$> ./configure $CONFIGUREFLAGS
$> make && make install
# test

You can get the configure flags from your distribution's build system. e.g. Fedora's koji. Look up the build of the version, go into the build.log and copy the configure flags from there. After that, the project will be installed in the same location, with the same flags as the rpm/deb/whatever file.
Verify it's still broken with this version, if not it might be distribution-specific patches. If it's still broken, check out the working version and verify that one too.

$> git checkout project-1.2 # the working version
$> make && make install
# test

So, you've just verified that vanilla 1.2 works and 1.3 doesn't? Great, off to bisecting.

$> git bisect start
$> git bisect good project-1.2
$> git bisect bad project-1.3

git now gives you a version in between those two, simply build and install it, test it and depending on whether it works or not, run "git bisect good" or "git bisect bad". After a number of test runs, git will tell you which commit introduced the bug. And you're done, send this commit info to the developers and they'll have a much easier time fixing the bug.

Now, of course nothing is as easy as it seems so you may see versions that you can't build (git bisect skip) or you'll run into dependency issues between versions so you can't bisect further. I've had some reasonable success with the process above and the smaller the range of commits you can provide to the developers the better.

You can use the same process for distribution-specific patches too. After checking out a a working version, just apply all patches from that distro in-order and then bisect between the working version and HEAD. Works like a treat.

git patches from tarballs

Generating patches with git is easy if you clone upstream. Many users don't run their software from repositories. They work from either distribution packages or tarballs. So a number of times I've heard something along the lines of "Sorry, it's not a git patch because I'm working from the tarball". It is quite easy though to create git patches from your tarballs. Simply run the following command in the extracted directory:

git init && git add --ignore-errors .; git commit -m "`basename $PWD`"

Explanation:

• "git init" initialises a new git repository.


• "git add ." adds all existing files to the repo. The --ignore-errors is there so git skips over files that can't be added, I've had that happen in a few tarballs that had their permissions busted, etc. If there are errors, you obviously need to check whether they affect files you want to hack on. If not, ignore the errors. (btw. you want to run git add before compiling everything, having all object files in the git index is painful)


• "git commit" commits all newly added files with the name of the current directory as commit message.

Now the directory is basically the same as upstream when they released the tarball, without the history. Either way, you can just hack, commit, rebase, etc. and then create a patch with git-format-patch and submit it to upstream. Assuming that upstream hasn't diverged too much from the tarball, chances are the maintainers can just apply the patch as-is.

Disclaimer: I learned this workflow from the Fedora X11 packages where this method is used to apply upstream patches to the tarballs. So the credit goes to ajax (or maybe someone else).

[edits]
"git init-db" replaced with "git init".

My workflow

After posting about the X.Org supermodule I was asked for more details about how to get everything up and running. So here's my usual workflow. I'm open for improvements in the (moderated) comments.

Prerequisites

My workflow currently includes two patches to git, the second of which is still under review.
Install git from git and apply these patches. They add rebasing support to git submodule update and make the process a lot easier.

Set up a bunch of environment variables that you'll need each time you build:

$> export PKG_CONFIG_PATH=/opt/xorg/lib/pkgconfig
$> export PATH=/opt/xorg/bin:$PATH
$> export LD_LIBRARY_PATH=/opt/xorg/lib
$> export ACLOCAL="aclocal -I /opt/xorg/share/aclocal"

and some more that are just handy:

$> export CFLAGS="-Wall -O0 -ggdb"
$> export CC="ccache gcc"
$> export MAKEFLAGS="-j3"

Building X.Org from git in 5 steps

$> git clone git://people.freedesktop.org/~whot/xorg.git
$> cd xorg
$> git submodule init
$> git submodule update
$> ./util/modular/build.sh -f built.modules /opt/xorg

The supermodule is set up for automatic rebasing and you'll end up with a tree running each module on master. The last command builds everything in the right order and - with the "-f" flag - echos the modules being built into the built.modules file. If it fails (usually due to missing packages) you can resume from the last to-be-built component (the last one in built.modules).

$> ./util/modular/build.sh -f built.modules -r `tail -n 1 built.modules` /opt/xorg

Once that is done, you're left with an X tree in /opt/xorg, most important of which is the binary in /opt/xorg/bin/Xorg.

Working with the tree

I tend to have three or more branches in most repos. The branches that matter are "master", "queue" and "devel". Then I have additional branches for features that result in a patch series (e.g. "xi2").

"master" is always as close to upstream master as I can get. Anything that lands on master will likely be rebased and pushed upstream. Day-to-day bugfixing also happens on master.

"queue" is for patches I sent to the xorg-devel list. The workflow here is usally development on some other branch, then git-format-patch + email, then cherry-picking from the other branch to queue. Patches in "queue" get cherry-picked to master and pushed, and once master is pushed, "queue" is rebased onto master.

"devel" usually happens when I realize that the patch series on master is more than it should be. This is when I branch master into devel, reset master to the previous state and continue on devel. devel is heavily rebased and sometimes doesn't lead anywhere. If it does, intermediate patches are cherry-picked onto master and pushed when they're ready. devel is deleted as soon as I finish with it.

Feature branches (e.g. "xi2") tend to be the same as devel but with a specific feature in mind. Anything that isn't related to it (bugs that I find in code around that feature) is cherry-picked to master and queue, and the feature branch then rebased.

So a single patch may wander from xi2 to devel (when xi2 is branched for some reason) to queue to master before being pushed.

I've been using this workflow for months now, and one of the main reasons why it works fine for me is tig. Tig shows other branches heads and tags in the history list, so by rebasing often I always have a visual marker where the new patches start. Tig also makes cherry-picking easy, so I use it more often than git pull.

The other thing that is incredibly helpful is the zsh git prompt I got from here and modified a bit. My current output reads:

:: whot@dingo:~/xorg/xserver (xi2*+)> 

Where xi2 is the branch name, * shows I have changes not added to the index, + shows I have changes added to the index that will be committed. I cannot recommend this prompt for the kernel though, it takes to long. For the repos I work with it's fine.

A word about backups

Since most of the work is local, it's important to have a backup, especially for devel branches that live longer than a day or two. I added a "backup" remote and force-push the branch I've been working on at the end of each day to this remote. In the worst case, I can just clone from there and resume where I left off.

$> git remote add backup user@host:~/repository.git # only do this once
$> git push -f backup branchname

The big fat X.Org supermodule

I've been using a git supermodule for nearly 2 months now and it works great. So here it is, do with it what you will.

git://people.freedesktop.org/~whot/xorg.git

It includes all the modules I care about to get a server running.

The advantages of git submodule are simply that I have a known working tree I can easily share between my test machines. Whenever I update a component, I can test it and easily revert back to the previous working version if needed.

Advantages for you: if you pull from that tree, you're running a version that at least compiles and runs on my hardware. That doesn't mean it's bug-free of course, but it's a start if you want to get into X server testing.

Disadvantages: I update input stuff often, other stuff when needed. So if you care about the latest and greatest graphics patches, you will need to maintain your own tree. Such is life.

Updating is easy: git pull and git submodule update.

The git submodule interface is missing one important feature to make this workflow better (automatic rebasing), but that'll hopefully be fixed in a future git version.

git-format-patch for a single commit

One thing that always annoyed me was the weirdness of getting a patch from a single commit somewhen back in the history. Turns out I was just ignorant and reading the man page actually helps, git-format-patch accepts a -<n> option, where <n> is the number of patches you need since the commit (inclusive).

So, because I'm a big fan of tig, add this to your $HOME/.tigrc:

bind generic E !git format-patch -1 %(commit)

Start tig, mark the patch you want, hit "E" to get a nice patch file. Doesn't get much simpler than that.

[update Jun 05 2012]
As SEJeff and Michael point out in the comments

git show $sha

shows a single commit. The output of git show is different to git format-patch though (it cannot be applied via git-am).