18 December 2014

Our approach to manufacturing is as quaint as punchcards

Recently, I had the privilege to hang out with some septuagenarian programmers. I was showing off the bicycle shifter I've been working on. I complained to the veterans that I was shocked by the amount of time required to bring a physical design to implementation. It took two weeks to come up with a CAD model; a month and 30 failed attempts to make a usable 3D print; 4 months to manufacture a part out of aluminum.

"That's what programming was like back in the day" was their reply. Really? "You'd turn in your punch cards and hope to get the output a week later — sooner if you were lucky". Can you imagine writing a program like that? I had never considered the analogy to physical design.

It's unimaginable how anyone prototyped physical objects before 3D printing. Debugging a program with one execution attempt per week is similarly inconceivable. We're spoiled these days. We grew up reliant on read-eval-print loops (REPLs), interactive debuggers, quick compilers, and instant gratification. Though we've progressed beyond punchcards, many people think our tools still aren't fast enough. Our generation is obsessed with speed and convenience (perhaps for good reason).

The question is: What will bring physical manufacturing up to parity with software? Earlier this year I would have said it's 3D printing. But 3D printing is slow. Even though laser sintering can produce precision parts like rocket engines, it doesn't scale. SpaceX isn't printing 1,000,000 engines, they're printing 100.

To build cars, cell phones, and soda cans you need to produce high volumes quickly. Waiting for a 3D printer to finish a job isn't cost effective for most terrestrial products. Instead, you need to design for manufacturing with mills in factories where the machine time is measured in seconds, not hours. But today designing for factories is too way too difficult. Our approach to manufacturing is as quaint as running assembly code on punchcards once a week.

Though there are high-level tools to turn CAD models into milling toolpaths, that's only a small portion of the problem. What we need is a way to click a button and launch a manufacturing process: a generalized, automated workflow from CAD to produced parts. Maybe affordable milling tools will help us get there. Maybe cheap, inflatable robots will enable scale.

As much as I loved the Diamond Age, I don't think 3D printers will make it possible. But I think it's a goal worth pursuing. It would be wonderful to tell our grandchildren the same line: "That's what manufacturing was like back in the day". Who knows what they'll be creating then.

You can read Effective Python today

The rough cut of Effective Python is now available on Safari Books Online. This is an early preview of the full content of the book before editing has been completed. I've also published the table of contents on the website and all the example source code in the GitHub project.

17 December 2014

Fun time tonight at the Homebrew #indieweb club at Mozilla SF with Tantek, Ryan, Katie, Pius, Nick, Ari, Jon, Kyle, and more! Thanks!

16 December 2014

It feels surreal to see proofs of the final layout of Effective Python.

Compare the shunting-yard algorithm implemented in Rust vs. Samizdat. Samizdat has built-in syntax for defining PEG parsers.

15 December 2014

danfuzz released the source to Samizdat today! "Samizdat is a high-level programming language somewhere down the family lineage from all of ALGOL, Lisp, and Smalltalk." I'm excited to try it out.

12 December 2014

10 December 2014

Excited to be giving a talk at PyCon 2015: "How to Be More Effective with Functions". Plus, Effective Python will be published by then! Hope to see you in Montréal.

09 December 2014

brew install npm
npm install bower
bower install

08 December 2014

I signed up for LinkedIn on Saturday to test running ads (I'm curious if they work better than AdWords or Twitter). Even though I didn't give it access to my address book, I think it emailed everyone I've ever known. Embarrassing! But I've heard that's par for the course with them. Anyways, if you got an email with my face in it, I'm sorry about it.
tracemalloc alone is a good enough reason to prefer Python 3.

06 December 2014

Another I missed: "7 Common Mistakes In Go And When To Avoid Them". It's a good list.

Effective Python is now available for preorder

Exciting! You can get it on Amazon by using this link.

Kindle, Safari, and eBook editions will be available once we're closer to the publishing date.

04 December 2014

Cool overview that I missed: "Go 1.4+ Garbage Collection (GC) Plan and Roadmap"

01 December 2014

Docker vs. Rocket vs. Vagrant — Still anyone's game

In case you missed it, today CoreOS introduced its own container format called Rocket (to compete with Docker). It's hard to decide who actually knows what they're talking about here. The CoreOS folks have proven they have good judgement when it comes to reimplementing existing infrastructure. Docker's CEO posted a rebuttal that's encouraging, but it rings hollow to me.

Separately, Vagrant is barely mentioned these days. The lines between all three options (Docker, Rocket, Vagrant) blur for most people. For example, check out this Stack Overflow thread from about a year ago where the creator of Docker and the creator of Vagrant go head-to-head on the question, "Should I use Vagrant or Docker?" They're both a bit confused, but Docker has captured mindshare since then.

My take on this is things are moving so fast nobody knows where it'll end up. Docker was first demoed at PyCon in March 2013 — so recently! Yet when you see both Amazon and Google announcing Docker container services within weeks of each other, it's easy to assume that Docker has won. The beauty of open source and high-level APIs is that it's still anyone's game. Confident broadsides like Rocket cast doubt on our assumptions. I applaud CoreOS for keeping everyone alert.

Meanwhile, Solomon Hykes (Docker's creator and CTO) still knows what's up. Here's his Tweet stack on the matter. His words ring true to me. Why isn't Docker responding on their blog with strong technical leadership like that? After Twitter and others destroyed their ecosystems, we're all a bit skeptical of rhetoric about "open" these days.

22 November 2014

And here's even more detail on Facebook's datacenter networking (previously). This is what it takes to compete in infrastructure. I'd rather write code that runs in these environments than build them.
Some pretty cool developments in JavaScript recently.

I wonder if these approaches could be extended to Python?

16 November 2014

You only get 1,000 lines of code a week

This week you're going to write a huge amount of code. You're going to get a lot done. You're going to ship something big. If something goes wrong, you'll be there to fix it. You'll root-cause any issue. You'll deploy at midnight to save the day. The only thing holding you back is the problem you're trying to solve.

Such an approach to building and launching works for quite a long time. When you're first starting as a programmer you're tasked with implementations that seem like "a simple matter of typing" in hindsight. Once you hit your stride, it feels like nothing can stand in your way. As time goes on, the complexity of the problems you confront grows exponentially. The balance between time spent thinking vs. implementing shifts towards design. The difficulty of shipping becomes more social than technical.

My advice for senior developers: Spend your time wisely. Imagine you can only write 1,000 lines of code this week. Is what you're doing today the most important thing? Are you the only person who could write the code you're writing? If not, why isn't someone else doing the work? You need to make your 1,000 lines count as much as possible. Your work must have leverage. Your 1,000 lines should be the essential core that enables a team of programmers to write another 10,000 lines effortlessly.

Eventually the same idea applies to everything else. You can only have 4 meetings this week. You can only send 20 emails. Make them count. Maximize the time/impact tradeoff. Let go of the things you know you can do that someone else could do just as well. Choose to do only the things that nobody else can do.

The exception to this is solidarity. Just because your time is valuable doesn't mean you're above it all. You need to carry your own weight just like anyone else. If you don't know the nitty gritty of your team first-hand you're useless.
The final draft of Effective Python is done. Stay tuned for a sneak peak of the book's contents before publication.

10 November 2014

"MetricsGraphics.js is a library built on top of D3 that is optimized for visualizing and laying out time-series data."

06 November 2014

One outcome of writing Effective Python (which covers both Python 2 and Python 3) is I'm now enamored with Python 3. It's the same familiar language, but with so much more polish.

21 October 2014

The second draft of Effective Python done!

Thanks to Brett Cannon, Tavis Rudd, and Bear for their technical guidance.

Now: one last polish for the final draft.
"Learn to do something yourself first before having someone do it for you." -Paul McDonald

19 October 2014


Why is it called a Mix-in? Because they had those at "Steve's Ice Cream, a favorite ice cream shop of MIT students". That and 1000 other answers are on c2. I never realized how important c2 is for new programmers to gain context.

07 October 2014

The Peripheral

Look what I scrounged up! A bit worn by the last reader, but totally worth it. Gibson's traveling around on his book tour now, so check your local bookstores to see if he's coming through your area. Can't wait to read it!

06 October 2014

Python is #8 in the TIOBE index (October 2014), #4 in the Redmonk ranking (June 2014), and #3 in the Gartner index (October 2014).

Go is #42, #21, and #24, respectively. Rust is only on the Redmonk list at #58. Swift is #19, #68, and #29. Dart is #17, #38, and #22.

The methodology of this is always questionable but it's still fun to see languages grow and others fade.

What'd be fun is a relative measure of rate of change. Then you'd at least have something to normalize to so you could compare the rates of change or direction of change across indexes.

02 October 2014

Armin Ronacher shares some of his experience with Rust.
Wow RAW is awesome! It's a wysiwyg UI in front of D3.

Custom faces for smart watches

After some initial misgivings about Android watches, I've been having a lot of fun with Facer, a $1 app that lets you customize your watch face. There's a built-in library of faces:

And Facer allows you to create your own. People are getting really deep into this. There's a whole site called FaceRepo dedicated to hosting people's creations. There's also an official and very active subreddit just for Facer users. People are doing silly things like recreating the Mickey Mouse watch and Rolexes.

Of course I had to try this out, even though I'm not a big watch person (this is the other watch I own). I made a bunch of middling faces to learn how to use the tool and figure out what's possible. For a few days there I was obsessed with tweaking my designs. It's pretty fun! Here's the best minimalist face I could come up with. It's called "Monoid". You can get it here.

Facer is totally worth the $1!

Trying an Android watch

I bought an LG G Watch after they were launched at the Google I/O conference this year (June 25th).

There were a lot of reasons not to. At the time the reviews said the Samsung Galaxy Gear Live was better. Months before that, in March, the Moto 360 teaser went out and everyone got excited about round faces. The rumors of the Apple Watch release were already solidified (and indeed, they've since pre-announced it). And generally the whole ideal of a smart watch seems stupid, excessive, and wasteful.

But I wanted to give it a try! Wearable computing is supposed to be the future (?). I wore the watch for a few weeks.

What I liked:
  • I could stop checking my phone for notifications
  • I could send text messages by voice from my wrist at a stop light while biking
  • The watch could reliably tell time

What I didn't like:
  • It was illegible in the sun
  • Once you dismissed a notification there's no way to get it back
  • The microphone didn't deal well with wind or noise

I put it down for a couple months. The watch was reasonably useful, but I felt like it's still a version one product. I anticipated the Moto 360 would be awesome and the Apple Watch would redefine the category. Now both of those are here and they're not as good as I expected. The Moto 360 has a fatal flaw (the "flat tire" at the bottom of the screen) and it's round face causes status message to be cut off in dumb ways (like the HD TV effect).

The Apple Watch is absurdly complicated: it's got buttons; it's got a wheelie thing; the touch screen has a sense of pressure — Steve Jobs is rolling in his grave. I found that the Apple Watch and LG G are about the same size and thickness. The LG G Watch has no buttons, only a touch screen and contacts to charge it.

After seeing all this I put the LG G Watch back on. It's still a version one product, but at least it's improved since I last used it. The latest Android Wear software is better. Strava works well with the watch now. And there are more apps to try, such as Facer. I'll give it another month and see if it sticks.

01 October 2014

Metrics from the first draft

It took me 683 commits to write the first draft of Effective Python over the course of 132 days (~4 months). That includes all the work for tools I built, the website, notes, and code examples. I've been storing the content in a private GitHub repo. Here are some fun metrics from it.

Commit density by day:

The majority of my commits are on the weekends.

Commits per week:

I was offline quite a lot in September.

Commits by hour of day / day of week:

Clearly I have been staying up way too late working on this thing.

What will be interesting is how many commits it takes to do revisions and get the book into a publishable state.

30 September 2014

Microsoft launches Prediction Lab, a follow-up to their Xbox election work in 2012.
Cool example of Generics in Go that works by dynamically providing remote repositories for import. Quite a hack!

29 September 2014

The first draft of Effective Python is done. 55,000 words, 8 chapters, 58 items. Now begins the revision process. Hopefully I can polish this into something worth reading.

22 September 2014

Returning from a vacation

28 August 2014

24 August 2014

I want the Kardashian game, but starring Elon Musk.

How to Internet

  1. Read a cool blog post you find interesting
  2. Leave a supportive comment and link to related ideas
  3. Receive an email from the post's author saying your comment is spam
  4. They delete your comment

Ahh... wondrous delight. Just another day.

Tweet storms

1/ Tweets that are numbered are annoying. Please stop being lazy and use a blog.

2/ If it's not important enough for you to edit, it's not important enough for us to read.

Just stop! You're wasting everyone's time.

19 August 2014

The Data Triumvirate

There's a huge focus right now on building products and services that do data analysis. Developing these systems involves three distinct groups of stakeholders that have opposing viewpoints.

  • The product managers are trying to sell something. They want the data to show what they're selling is working for someone (themselves, customers, end-users). They want impact.
  • The statisticians are trying to ensure correctness. They want the data to be unbiased. They want the methodology for finding results in the data to be defensible to their peers.
  • The engineers are trying to ship the simplest thing possible. They want to minimize the complexity of analyzing the data. They want a data pipeline that is maintainable and extensible.

The tension between these roles is crucial. If one outlook dominates a joint effort you're setting yourself up for failure.

  • If the product managers always get their way you're letting a fox guard the hen house. They'll find significance in the data at the expense of bias and methodological validity. You'll be selling snake oil.
  • If the statisticians get their way you'll never ship your product. Compensating for every bias in a dataset is nearly impossible. You'll never have the 99% confidence they want for every measure.
  • If the engineers get their way your product will be too simplistic. The most maintainable implementation will undermine the statistical methods. The impact measured won't be compelling enough to sell.

If you're doing analysis, make sure you're part of a data triumvirate. It's similar to the relationship between product managers and tech leads. You need a balance of power to build the right thing.

16 August 2014

Damn! If only there were more static site generators to choose from.

07 August 2014

It's "Temps Universel Coordonné" or "Coordinated Universal Time" — Why do we call it UTC?

Python, C++, and Go as bicycles

I'm still trying to find the boundaries between Python, C++, and Go when building something new. The split between Python and C++ is a clear tradeoff of bare-metal performance for developer productivity. The split between Go and C++ is easy for me because of scars from templates and the difficulty of concurrent programming in C++.

What's been difficult is finding the dividing line between Python and Go. What I've been able to come up with so far is a trite analogy involving types of bicycles.

Python is a touring bicycle. It's approachable and easy for anyone to learn how to ride. It has features like a basket, fenders, lights, and a pump that make it practical for almost every situation. Its limited gears mean it's dependable, but slow unless you pedal hard.

C++ is a race bike. It's difficult to ride and easy to crash. It has multiple sets of handlebars and every other feature a bike could offer for maximum speed. Its fragility makes it impractical for simple riding, but it can get you there faster than anything else.

Go is a modern cyclo-cross bike. It's simpler and safer than a race bike, but still fast. It has most of the features you want (light-weight, aerodynamic) and some that are uniquely surprising (durability, knobby tires). It's a whole new category of riding.

Continuing with the bad joke, Java would be the bicycle factory factory. Okay — the analogy breaks down quickly. But it's not so awful.

For an unexpected adventure I'd choose the touring bike (Python): quickly building something new, handling all environments easily, everything I need built-in. For competing on a more well-defined course with treacherous obstacles I'd choose the modern cyclo-cross bike (Go): speed, versatility, safety. For a time trial on smooth roads with no obstacles I may even risk riding the race bike (C++).

That's the best assessment I've got for now. Where do you draw the line?

06 August 2014

from turtle import *

Did you know Python has turtle built-in?

05 August 2014

Quilla now uses HTTPS by default for everything.

04 August 2014

My last deploy of Quilla was 528 days, 0:39:22 ago. I have done zero maintenance in that time. This is the dream of PaaS.

From plastic 3D print to milled aluminum — Making a bicycle gear shifter

Earlier this year I spent 30 days designing and 3D-printing a bicycle gear shifter for the Shimano 8 speed internal hub. The goal was to make one for my Mission Bicycle. I wrote about the experience here. After posting that story I worked with the team to revise the design. Here's the result as a plastic prototype. I've been pedaling around with this for the past few months.

My next goal was to find a machinist to turn this part into 6061 aluminum for strength and durability. I searched around the SF Bay Area and found a few machine shops. A couple folks I contacted never responded. One replied but declined because they couldn't open my 3D design files. Another met me in person, but wasn't excited about the project (his primary business is building things that go into space!).

Finally, a friend of mine connected me to someone whose day job is running a machine shop. For the sake of privacy I'll refer to him as "the machinist". It's important to note that there's no way any of what follows would have happened without their work. I'm grateful for all of their effort on the project. I've just been along for the ride.

Design for manufacturing

The machinist also had problems opening my CAD files (from 123D Design — $10/month). The issue is the DWG files exported by one company's software can't always be imported by another company's software. There are tools out there for format conversions (like Teigha) but they aren't good enough. Professionals seem to use Solidworks (MSRP: $5000) over Autodesk. It's the Cadillac of CAD. I can't afford it.

Luckily, I found a free tool by the makers of Solidworks called Draftsight that can be used to open and annotate the DWG files from Autodesk products. As long as it could be opened in Draftsight it could also be opened in Solidworks. When it didn't work I knew something in my model was funky and needed to be fixed.

Once the machinist could open my files we went through another cycle of design tweaking. There were a lot of things I thought were simple to mill out of aluminum that turned out to be impossible. Machining is really an art form like blacksmithing. It's full of arcane knowledge. The only way to learn is to actually use the tools and gain experience from your mistakes. Here's the final design for manufacturing, which looks better than the original.

Milling the part

With the final design in hand I bought all of the parts to build and assemble a prototype. I got the screws, washers, and ball springs from McMaster-Carr. I bought aluminum from TCI Aluminum. Machinists call raw uncut aluminum blocks "blanks". A blank is the starting material that is cut down to the final part by the CNC machine. Here's what my blanks looked like.

You normally buy 12 foot long bars of aluminum that best fit the profile of your part. Then you cut it yourself to size. Here's what that process looks like. It's quite a saw.

Here is the inside of the Haas VF-2SSYT 3-axis mill (MSRP: $72,000 — as configured $100,000+) that we used.

The control panel on the right lets you modify the program before or while it runs. What you see in the middle on a spindle is the business end of the machine. Here's a picture of the 3 inch diameter facing endmill. It's hardened steel and extremely heavy. It has carbide inserts so it doesn't wear down too quickly.

While the mill is doing its work it needs to change tools for particular cuts. To do this it has an enormous cache of end mills that can be changed automatically by the machine. This lets the machine cut, drill, face, and tap without human intervention. The machine uses a robot arm and compressed air fittings to swap the bits. It takes less than a second to swap (insane!). Here's a picture of the revolving cache, about 10 feet off the ground.

The mill is a dumb machine like a computer. It runs programs in G-Code, which describe where to move the cutting bits in XYZ coordinates. The mill does little to ensure the G-Code is valid. If the G-Code said to bash the bit into the vise the machine would do it. It's similar to how a bad set of instructions can crash a computer program.

The G-Code comes from CAD software that's the machinist's equivalent of a compiler (Mastercam and HSMWorks are popular tools). The CAD lets you build a set of "operations". An operation is all of the various cuts you need to make in one side of an aluminum blank. This can include multiple passes by many different types of cutting tools. The software doesn't automatically convert your 3D model into operations. You need to manually look at the outlines of the part and come up with a series of "tool paths" for the cutting blades to follow. The hope is the cuts will result in your design by subtracting from the aluminum blank. Machinists call this conversion process "programming".

One of the coolest things the design software can do is simulate the mill in action. It's like a highly sophisticated interactive debugger. Here's what it looks like.

Once you have the G-Code you put it on a USB stick and upload it to the mill. Then you put the blank in the machine and clamp it down with a vise. You bang it with a dead blow hammer to make sure it's seated properly. Then you use the mill control panel and it's 1000 buttons to tell the machine to measure the XYZ coordinate of one of the blank's corners. This tells the mill where to start cutting so you get exactly the physical object that you expect. Here's one blank in the vise.

You can see aluminum shavings all over the inside of the mill. Those are the "chips" from the end mill. There's also this weird fluid on everything. That's lubricating coolant to ensure the end mills don't overheat while they're spinning at 8000+ RPM and cutting. When the machine's actually going you can't see anything because coolant is everywhere. But the cutting sound is extremely loud. Here's a video of the machine making a cut.

The bike shifter ended up being 7 operations in all. 3 were for the base that goes over the steering tube. 4 were for the knob that pulls the shifter cable. The design also required 3 different pairs of "soft jaws", which are essentially jigs that hold the aluminum blanks in place to handle mating surfaces that aren't square.

I was surprised by how important it is to hold the aluminum blank properly in the mill. It significantly affects how you choose the order of your operations. Each operation must leave on enough "fixture stock" to let you grip the part from the opposing side until everything is square or fits in a soft jaw. Here you can see each step in cutting the shifter's base piece and the various fixtures.

The shifter knob provided an additional challenge because it was important that all of the cuts for the center hole and ball springs were concentric. Here's what the blank looked like after each operation for the knob.

Built and biking

At last! It's all done. Here's the shifter actually mounted on my bicycle.

And here's what it looks like from the top when you're riding it.

For the next month we'll try these prototypes and see how they feel. We'll probably make some design revisions and update the G-Code programs accordingly. We'll get some anodized in black. Once that's done I'm hoping we'll do a manufacturing run at a local machine shop. I'd like to start putting these on Mission Bikes and selling them to anyone who likes the Nexus 8 hub. Let me know if you're interested in getting one!

03 August 2014

Don't fall in love with your tools.

31 July 2014

How I'm writing a programming book

Update: Visit the official book website to get the book!

I've been working on Effective Python for just over two months now. The plan is to have 8 chapters in total. I've written a first draft of 5 so far. Chapter 3, the first one I wrote, was the hardest for many reasons. I had to establish a consistent voice for talking about Python. I took on the most difficult subjects first (objects and metaclasses) to get that work out of the way. I also had to build tools to automate my workflow for writing.

Each chapter consists of a number of short items that are about 2-4 pages in length. The title of an item is the "what", the shortest possible description of its advice (e.g., "Prefer Generators to Returning Lists"). The text of the item is the "why", the explanation that justifies following the advice. It's important to make the core argument for each item using Python code. But it's also important to surround the code with detailed reasoning.

Before I started I read a nice retrospective on how one author wrote their programming book. They had separate source files for the example code and used special comments to stitch it into the book's text at build time. That's a great idea because it ensures the code that's in the book definitely compiles and runs. There's nothing worse in programming tutorials than typing in code from the text and having it barf.

I wanted to go a step further. I wanted my examples to be very short. I wanted to intermix code and prose more frequently, so the reader could focus on smaller pieces one step at a time. I wanted to avoid huge blocks of code followed by huge blocks of prose. I needed a different approach.

Writing workflow

After some experimenting what I landed on is a script that processes GitHub Flavored Markdown. It incrementally reads the input Markdown, finds blocks that are Python code, runs them dynamically, and inserts the output into a following block.

Here's an example of what the input Markdown looks like:

The basic form of the slicing syntax is `list[start:end]`
where `start` is inclusive and `end` is exclusive.

a = [1, 2, 3, 4, 5, 6, 7, 8]
print('First four:', a[:4])
print('Last four: ', a[-4:])
print('Middle two:', a[3:-3])

First four: [1, 2, 3, 4]
Last four:  [5, 6, 7, 8]
Middle two: [4, 5]

When slicing from the start of a list you should leave
out the zero index to reduce visual noise.

assert a[:5] == a[0:5]

I write the files in Sublime Text. When I press Command-B it builds the Markdown by running my script, which executes all the Python, inserts the output back into the text, and then overwrites the original file in-place. This makes it easy to develop the code examples at the same time I'm writing the explanatory prose. It feels like the read/eval/print loop of an interactive Python shell.

My favorite part is how I made Python treat the Markdown files as input source code. That means when there's an error in my examples and an exception is raised, I'll get a traceback into the Markdown file at exactly the line where the issue occurred.

Here's an example of what that looks like in the Sublime build output:

Traceback (most recent call last):
  File ".../Slicing.md", line 29, in 
IndexError: list index out of range

It's essentially iPython Notebook, but tuned for my specific needs and checked into a git repo as Markdown flat files. Update: A couple people mentioned that this is a variation of Knuth's Literate Programming. Indeed it is!

Publishing workflow

Unfortunately, my deliverable for each chapter must be a Microsoft Word document. As a supporter of open source software and open standards this requirement made me wince when I first heard it. But the justification is understandable. The publisher has a technical book making system that uses Word-based templates and formatting. They have their own workflow for editing and preparing the book for print. This is the reality of desktop publishing. More modern tools like O'Reilly Atlas exist, but they are new and still in beta.

There is no way I'm going to manually convert my Markdown files into Word files. The set of required paragraph and character styles is vast and complicated. These styles are part of why the published book will look good, but it's tedious work that's easy to get wrong. Sounds like the perfect job for automation!

I have a second script that reads the input Markdown (using mistune) and spits out a Word .docx file (using python-docx). The script has a bunch of rules to map Markdown syntax to Word formatting. The script also passes all of the Python code examples through the Python lexer to generate syntax highlighting in the resulting document.

The other important thing the publishing script does is post-process the source code. Often times in an example there are only two lines out of 20 I need to show to the reader to demonstrate my point. The other 18 lines are for setup and ensuring the example actually demonstrates the right thing (testing). So I have special directives in the code as comments that can hide lines or collapse them with ellipses.

Here's an example of what this looks like in the Markdown file:

class MissingPropertyDB(object):
    def __getattr__(self, name):
        if name == 'missing':
            raise AttributeError('That property is missing!')
        # COMPRESS
        value = 'Value for %s' % name
        setattr(self, name, value)
        return value
        # END

data = MissingPropertyDB()
data.foo  # Test the success case
except AttributeError as e:

The actual output in the published book would look like this:

class MissingPropertyDB(object):
    def __getattr__(self, name):
        if name == 'missing':
            raise AttributeError('That property is missing!')
        # ...

data = MissingPropertyDB()
except AttributeError as e:

AttributeError('That property is missing!',)


If you have any interest in using these tools let me know! Writing a book is already hard enough. Having a good workflow helps a lot. I'd like to save you the trouble. Otherwise, if you have any suggestions on what I should put in the book, please email me here.
© 2009-2014 Brett Slatkin