The amazing adventures and final toolchain of Pro Git, 2nd Edition
Almost exactly six years ago I was approached by Apress to help out on a book that was being written but was falling behind schedule called Pro Git. Eventually the original author decided not to keep writing and I rewrote the book from scratch, finally being published around August of 2009. For the first three chapters or so, I wrote the book in Word, sending documents to my editor and getting redline versions back some time later.
After about three chapters of this I was about to give up when I suggested that we switch to doing the book in Markdown for the writing and technical editing phase and then move back to Word only for copy editing which was agreed to. Once the book was completed, I again moved all the content back to Markdown so I could publish it online at a website I created. Luckily for me, the original author had gotten Apress to agree to Creative Commons license the work as well.
Over a year later, Apress also released a Kindle version of the book, though people had produced Kindle compatible versions from the open source content almost as soon as the book was released.
Pro Git, Second Edition
Since the book has done reasonably well, after about four years of being in print, Apress approached me to do another edition of the book. In the course of that time, my company GitHub had grown from being 4 guys working on a site for a few thousand users to a giant online presence hosting millions of repositories and employing over 200 people. Not only had the GitHub community changed that much, but the tool itself had also had some significant changes. I agreed that it was probably time to revamp the content.
While you can now read the second edition (as well as the first edition) online or download it in PDF, ePub or Mobi (Kindle) formats to read on your favorite device, I thought an interesting story to share would be the differences in how this version was produced from the first version about five years ago.
The Writing of Pro Git v2
I had actually started revamping Pro Git a few times in the last few years. Mostly I was interested in the toolchain aspects of it. At some point I started a project called Git Scribe which was meant to demonstrate the type of toolchain I wished I could have used to write Pro Git. The scribe tool provided a semi-easy way to install a toolchain that would take a specifically formatted Asciidoc book and produce nice looking HTML, PDF, ePub and Mobi ebooks from it. Though the project has since died and we ended up using O’Reilly’s Atlas platform instead, the Scribe project got me into Asciidoc in the first place.
Asciidoc
So I switched large portions of Pro Git to Asciidoc. The issue with Markdown was that it was too simple. It didn’t specify things like table formatting, cross references, indexing, callouts, source code examples, etc. All of which Asciidoc does in a format that is just as easy to write.
So I moved everything from Markdown into Asciidoc and noticed that some publishers were doing so as well. O’Reilly started allowing books to be written in Asciidoc since it generates Docbook easily and I believe Pragmatic Programmers had been using a simple markup language for a while before that.
Most importantly for all of this was the rise of Asciidoctor. When I went to redo my Git website (git-scm.com) I wanted to take the Asciidoc man-page sources for Git and make nice online versions of them and I wanted to do it with Ruby. My friend and GitHub coworker Nick Hengeveld wrote a nice parser and formatter for Asciidoc to help do this. Eventually some other GitHubbers (Ryan Waldron and Jeremy McAnally) joined in and turned Asciidoc.rb into Asciidoctor, which was eventually taken over by the amazing Dan Allen, who has taken the project to a whole new level.
Now that Asciidoctor is around, I can do truly amazing things with my Asciidoc source book that were not nearly as simple before.
GitHub
One of the biggest changes to the writing workflow was that GitHub is a lot better now than it was before. Now we had Pull Requests for review and commenting, Issue milestones for chapter planning, prose diffs for easier review and spot-checking, and the Atom text editor for writing and Asciidoc preview.
The workflow we got to use is one that everyone who has ever written a technical book, especially with a co-author, will probably be insanely jealous of.
We wrote out everything we wanted to do, added all the major points as issues attached to milestones for each chapter and then each decided what we were going to do and started writing. We could make a branch for the unit of work we were doing (normally a chapter) and start writing and pushing to the GitHub branch. We talked in Slack and got instant updates whenever one of us pushed some change. We opened a Pull Request almost immediately, usually with a checklist of things we wanted to complete before merging so you could tell what the status of each branch was (example).
When a branch ran long, we could merge the master branch into it to keep it up to date and make sure that conflicts were never a huge problem even through massive structural changes like splitting up chapters into multiple files.
Review was done through the Pull Requests — we could leave comments on any line of introduced text and have a conversation either there or in chat about it.
I was able to install the Asciidoc Atom preview plugin and get live rendered views of the text I was writing as I was writing it, including sidebar hints about what I had added, modified or deleted, what branch I was on, merge conflict helping tools, zen mode, etc. It was an amazing writing environment.
Not only this, but we did most of this collaboration while 9 time zones apart and our editor could at any time look at the milestones page to see what our progress was through the book.
If you’ve written a technical book and didn’t do it in Asciidoc on modern GitHub, you have done ten times more work than you needed to.
Atlas
Not only was writing and collaborating on the book way easier than in the first edition, but seeing what it would look like was infinitely easier.
The O’Reilly Atlas build system. Point and click for beautiful ebooks in minutes.
Thanks in part to conversations I had with Tim O’Reilly, Laura Baldwin and others over at O’Reilly Media years ago about how much better this toolchain could be, I was invited to try out the platform they were developing to make it all easier. We all agreed how great it would be to go from writing in Asciidoc to ‘git push’ to PDF. They ended up actually building it and thankfully they let me try it out.
You can see me being pretty active on their forum for several months and every time I mentioned something they had it fixed within days or even hours. Not only that, but the builds produced such beautiful output that for chapter reviews I ended up building a PDF, downloading it and reading it in Preview while making notes and changes in the Asciidoc source.
Luckily, they also have an API for the service and you can build different branches separately. This means I was able to write a service which listens for post-receive hooks on the Pro Git repository, pulls down the changes, pushes them to Atlas and automatically starts a build. It then polls for when all the builds are done and moves them to my own AWS S3 bucket and updates the new progit.org website and the git-scm.com website content and links.
Not only that, but I can do that for every language that Pro Git is translated into. This means that in the near future, if you go to either the Pro Git website or the git-scm.com website, you can download professional-grade ebooks in any format of the latest version of Pro Git as of minutes after the last errata merge. In any language. With zero work from me or the language maintainers after hitting the ‘merge’ button.
I basically now have both the simplest and most sophisticated multi-language ebook production and distribution setup on the planet.
Open Source
Another interesting difference between the first and second editions is that since I didn’t want to go through the Markdown to Word to Markdown process again, we agreed that I could write the book in its entirety and then “throw it over the wall” to Apress who would do the Word conversion and copy editing and then send us the changes. This means that Ben and I were able to open source the book much earlier in the process than I was able to before.
The book is currently entirely available online, including in all the ebook formats, but is still probably months away from going to print. In this time we’re sure to get a ton of small copy edits and technical corrections (we already got a handful in the first 24 hours) and will be able to merge them in before Apress goes to print. This means that we can find and fix errata before actually printing the book, which is fairly amazing.
We actually debated writing the entire book in the open or waiting until it was mostly complete before re-opening it. In the end, we opted for the latter, though I’m still not sure the former wouldn’t have been the better option. We felt it might be too much of a hassle to deal with opinions from the whole community during the entire process and that we preferred to own the content a little more fundamentally, but it may have been interesting to have gone the other route. Perhaps next time.
Translations
The final thing that’s dramatically different this time around is the handling of translations of the work. When open sourcing the first edition I don’t remember thinking of translations at all. I don’t remember getting my first translated work, but it had to have been fairly quickly after I released it because I restructured the directories to put everything in an ‘en’ directory shortly after the release.
In the five years that the book has been out, it’s been fully translated into at least 10 languages (Deutsch, 简体中文, 正體中文, Français, 日本語, Nederlands, Русский, 한국어, Português and Čeština) and partially translated into probably another 20. Since I decided to store them in subdirectories of the main repository and took Pull Requests for them, that meant that someone who had write access to the repository had to merge them. Which meant that I basically had to merge everything, even though I can’t read almost any of these languages. Eventually, after a few years, I got pretty bad at keeping up with the Pull Requests and the amazing Jean-Noël Avila took over and has managed the translations (and even the English errata) ever since.
This time though, we can learn from the issues we’ve had before and try to make it easier on everyone. This time we’re making every translation a separate repository under the progit organization and adding language specific maintainers to each one. This means that you can open an issue on the ZH version in Mandarin and not have us be totally confused. If maintainers stop being responsive and someone else is interested, we can add them as a new maintainer.
Thanks to Atlas, each translation will also have its own automated build process instead of making everyone run their own builds locally as has been the case until now.
In Summary
I believe this process is at least the short to medium term future of technical publishing. Writing your book in a simpler format like Asciidoc on GitHub, easily collaborating with your editors and co-authors, open sourcing the content, embracing a robust translation community and self publishing with tools like Asciidoc or Atlas.
I also believe that all of this has very serious and interesting implications for the publishing industry as a whole, which I’ll write about in depth in a future post.