This year, the first day of Hanukkah coincides with Thanksgiving Day. Amazingly, this is the first time it's happened since President Lincoln established Thanksgiving in 1863 and it is also the last time it'll happen until the year 79,811. I'll say that again: after this year, Hanukkah and Thanksgiving Day won't overlap for another 77,798 years.

(kottke.org)

Today I gave a talk at Velocity about a recent side project of mine called dpxdt. "Depicted" makes continuous deployment safe by comparing before and after webpage screenshots for each release. It shows when any visual, perceptual differences are found. This is the ultimate, automated end-to-end test.

Here's a screenshot of the UI in action with my site:



More specifically, Depicted is:

• An API server for capturing webpage screenshots and automatically generating visual, perceptual difference images ("pdiffs").
• A workflow for teams to coordinate new releases using pdiffs.
• A client library for integrating with existing continuous integration processes.
• Built for portability; API server runs on App Engine, behind the firewall, etc.
• A wrapper of PhantomJS for screenshotting.
• Open source, Apache 2.0 licensed.
• Not a framework, not a religion.

If you'd like to know more about the motivation behind Depicted, be sure to check out this video presentation. I'm looking forward to adding many features to this tool. I need help. Webhook callbacks, Firefox screenshots, IE screenshots in a VM, email notifications, and much more functionality is missing. Contributions are welcome on GitHub or discuss ideas on the mailing list.

Tantek and I sat down to talk about all things indie for GDL last week:

"whoever came up with this *blonk* idea should be *blonk*-ing fired!" -- Google Hangouts discussion forum

"Legislation is managing failure." - Jane Frost

"Data is the pollution of the information age." - Bruce Schneier

On the one hand, I am furious. On the other, I am thankful to not lose all productivity for the rest of the summer.

With the new Mac Pro, we’re assembling the entire product and machining several of its high-precision components in the United States. By leveraging the innovative power of industry-leading companies in Texas, Florida, Illinois, Kentucky, and over a dozen other states across America, we're able to build a product that's impeccably constructed and beautiful in every detail. In other words, exactly as it was envisioned by our designers and engineers in California.

Details here. Looks pretty rad.

Gave a talk for Google Developers Live. See all the details when I present at Velocity next week.

Every time I migrate a schema I wonder why I have a schema at all.

I had no recognition of the name Giorgio Moroder from Daft Punk's new album. Then realized he did "Chase" from Midnight Express in 1978. That song, Axel F. (1984, by his protégé), and Blue Monday (1983) were what got me into techno—oh!—but I first heard them as MODs downloaded over xmodem.

PS: The "Chase" is more proof that 1978-1982 were the best years in the history of modern music.

Going to NewsBlur.

I finally had a chance to read George Packer's article for the New Yorker, entitled Change the World: Silicon Valley transfers its slogans—and its money—to the realm of politics. It's actually less politically focused than I expected. I think he gets most of it right. It's a good critique of the most well-known and self-congratulatory parts of Silicon Valley, the same people who have private transit fleets, "won" against PIPA and SOPA, elected Ed Lee, disrupted industry, etc.

Packer's thesis is this:

One question for technology boosters—maybe the crucial one—is why, during the decades of the personal computer and the Internet, the American economy has grown so slowly, average wages have stagnated, the middle class has been hollowed out, and inequality has surged. Why has a revolution that is supposed to be as historically important as the industrial revolution coincided with a period of broader economic decline?

While visiting the Valley, he asked everyone this question. The answers Packer got were embarrassing.

It was Steve Jobs, after all, who told President Obama that Apple's manufacturing jobs would not be coming back from China. Apple's position on issues like inequality was expressed last year by an executive who said, "We don't have an obligation to solve America's problems. Our only obligation is making the best product possible."

My answer for Packer would be: The economy has declined because of outsourcing and globalization. The economy has grown because of technology. If we could return manufacturing jobs to the US wages would increase broadly, inequality would be reduced, democracy would improve.

I'm not surprised that the free-market-minded folks that he interviewed, in the upper echelons of the Valley, dismiss this idea or are blind to it. It's fundamentally incompatible with their worldview. But maybe they're coming around. The Apple executive quote is from January 2012. By December 2012, Apple had decided to move some manufacturing back to the US for political reasons.

There's still hope. I think the majority of us in the Valley don't buy into the bullshit.

Taught someone about Django template inheritance today. I forgot how mind-blowing override blocks are to newbies.

class Memo(dict): \
get = lambda s, k: (len(s) > 1000 and s.clear()) or \
  (k in s and s[k]) or s.setdefault(k, s.f(k)); \
f = lambda s, k: # your code here

Thankful for Google's reversal:

In response to those requests, we are keeping the CalDAV API public. And in the spirit of openness, today we’re also making CardDAV – an open standard for accessing contact information across the web – available to everyone for the first time.

Sent an email at work entitled "Ninja competition". My job is strange sometimes.

"[The 850 billion dollar bank bailout] is greater than the entire 50 year running budget of NASA".

- Neil deGrasse Tyson

If I have to fill out a form with my name/address/phone required to read your product's case studies, you're doing it wrong.

"Google Abandons Open Standards for Instant Messaging"

Sigh.

I went to AAPOR for the first time this year, a conference covering public opinion polling and survey research. It's hardcore. Using "data" as a singular noun there is gauche. My goal was to see the panel session on Pew's use of Google Surveys. I had a great time and would go again.

A highlight was when I used triangulation to correct the data in an economist's presentation (never done that before!). It made me realize that beyond academia, entrepreneurs and startups should also be using triangulated research to validate their product plans and business models.


What is triangulation?

Triangulation is when you ask the same question many different ways and compare the results. You'll see agreement or disagreement between the questions. If they don't agree, something is happening that you don't understand. This lets you self-validate or corroborate your findings. Think of it like running an A/B test on survey question correctness, except that you want zero separation. It's similar to a fundamental part of the scientific method.

To show what I mean, I ran four different survey questions about cat ownership in the US. Here are the results (after some simple math):

#	Question	Cat ownership	Dog ownership	Pet ownership
1.	What kind of pets do you have in your household?	23.9% (+2.2/-2.0)	41.0% (+2.4/-2.4)	~52%
2.	How many cats do you have in your household?	28.2% (+2.3/-2.2)	-	>= ~28%
3.	Do you have one or more cats in your household?	24.0% (+2.1/-2.0)	-	>= ~24%
4.	Are cats or dogs not present in your household? Or do you have both types of pet?	30.5% (+2.4/-2.3)	44.0% (+2.5/-2.5)	~58%

The results converge on the same numbers (cat ownership between 22% and 28%) and agree with each other. Most differences are within the margin of error. The min/max span is 4 percentage points. The numbers also agree with data from the humane society and the AVMA sourcebook. I'm confident I know how many people online have cats.


What happens without triangulation

At AAPOR, one of the researchers from NORC / University Chicago presented test results on how representative Google Surveys are. Their original, less accurate finding was that Google Survey data does not closely agree with benchmarks for telephone ownership. We ran a follow-up survey to find out why. The problem was modal bias: Asking a question over the phone introduces errors that are different than the errors from a microsurvey.

By tweaking the question slightly we were able to reproduce the Pew Internet data within 3 percentage points (our results are here in Q1/2; Q3/4 demonstrate the modal bias).

The NORC folks were happy to hear our data was better than they thought. Had they triangulated their results themselves, they would have seen disagreement and known that something else (modal bias) was happening.


Why not triangulate?

Surprisingly, nobody I asked about triangulation at AAPOR had employed it in their own research. Maybe I missed somebody, but it makes sense:

1. Most polling that exists today is extremely rigorous and proven.
2. But it's slipping away because:
   
   • It's illegal to robo-call cellphones.
   • Phone response rates are very low.
   • Your current location and cell phone area code are often unrelated.
3. This makes traditional market research and opinion polling expensive and introduces bias.

So it's plausible that traditional researchers don't triangulate because they can't afford to. And why would they triangulate if the existing measures and techniques work well? The problem is when old measures are applied to new situations, like the NORC example above.


Now it's easy

With new methods it's cheap to do triangulation. I've seen startups triangulating decisions using Google Surveys and the results are great. I presented one such case in London, recently. Anyone can do it.

So: Whenever you make an important decision about a product, business, or research you should triangulate the data used in your conclusion. Try multiple approaches and find agreement between many measures of the same idea. This will give you confidence in your conclusions. It will provide defense against detractors. It will bring consensus to your team.

And you'll know that you're right.

“When you wake up in the morning, Pooh," said Piglet at last, "what's the first thing you say to yourself?"

"What's for breakfast?" said Pooh. "What do you say, Piglet?"

"I say, I wonder what's going to happen exciting today?" said Piglet.

Pooh nodded thoughtfully. "It's the same thing," he said.

I enjoyed this slide deck from Nick Galbreath, especially slide 25, which states the following hypothesis:

• It is impossible to simulate the production environment in development, either due to operational differences or data differences.
• No amount of QA or Security Testing can prove you don't have bugs, vulnerabilities, or cause severe operational problems.
• You have bugs and vulnerabilities, right now, in your application.

And the conclusion is that the only solution is continuous deployment. Indeed! I'm happy to see this viewpoint taking hold.

Here are the slides:

The Isley Brothers – Footsteps In The Dark - Part 1 & 2

The Google Ventures Startup Lab posted the video of my talk about Cohort Analysis. The slides are here.


Enjoy!

I've talked about my preference for first world goods previously. The gist of it is, I want to buy things made by people who have the same rights and freedoms that I do. I've managed to find great first world sneakers, shoes, tees, hoodies, shirts, and more. The missing bit has been a decent pair of jeans.

There's a place in SF called Self Edge that, for no real reason, I always assumed was snooty and bullshit. They also have a store in NYC, and I stopped in with a friend on a recent trip, putting my skepticism aside. That day, Kiya Babzani, the store's founder, happened to be in the NYC shop. We talked for a while and it turns out he's a rad guy. It made me realize that Self Edge isn't snooty, it's border-line Otaku. What they sell is mostly made in Japan, where the car/motorcycle/rockabilly subculture has co-opted classic American manufacturing and brought it back to life.

Which brings us to the pair of pants I bought. Check out the well-executed, overly descriptive, not-quite-but-still-is-Engrish on this tag (e.g., "vintage sleek"):



Anyways, these pants are great. They're sturdy enough that they identify them by weight more than anything. I figure they'll last five years, probably more, justifying the cost to me (it helps that the Yen is super weak right now). Contrary to popular belief, you're supposed to wash them frequently.

If you're still skeptical, try reading this two part interview with the founder. I enjoy things that people are passionate about, like bikes, to the point of being obsessive. The Flat Head and Self Edge pass the test. I stand corrected. And now I have a nice pair of first world pants.

I actually sit down and open my postal mail maybe four times a year. Wish I could stop it.

The past few summers I've attempted the near-impossible in San Francisco: Growing tomatoes. Every day it's some combination of cold, foggy, and windy. But I've managed.

What's surprising about growing tomatoes is how something so small can grow so big. I start with a tiny sapling in a thin plastic pot. I move it to a larger pot and then water it every couple of days. By the end of a week its mass has doubled. At a month it's over a foot high and blossoming. After two months the first tomatoes are growing fast and the plant is chest-high.



Each day I water the plant it looks the same as the day before. I can't notice daily changes because the differences are subtle. I went away for a week and when I got back the plant looked enormous compared to before. When I hadn't seen the incremental changes, the plant's growth was astonishing. Growing tomatoes has made me see the value solely in time passing.

How would I be if I spent a tiny bit of time cultivating myself every day? It wouldn't seem like much to me, since I'd witness the small differences. It'd be hard to stick with it. But after two months or a year, I may look back and realize how far I've come. I think this is how I developed as a programmer. I wonder how else I could improve this way.

Bad marketing email. Not even trying to be experimental about it. No call to action. Too much text.

From this article: "The people you want to hire aren’t applying and interviewing, they’re running their own companies." Or at Google, Facebook, Twitter, etc.

All useful code must be rewritten at least twice.

"Today we can officially confirm the release date for the Mac version of SimCity is June 11". Sweet. Goodbye summer!

John Carmack's magical sqrt(). Came up in conversation again tonight. It's amazing!

The feeling when the domain you want for your project is available.

I dislike dubstep because it's so desperate for attention.

Woah, end of an era.

subprocess.CalledProcessError: Command '['java', '-version']' returned non-zero exit status 1

I don't even have it installed.

The other day I saw a tweet go by that made me think twice:



The idea that this issue is big enough to win the popular vote for President seemed questionable.

I wondered:

• Who would want to use electronics during takeoff/landing?
• How many people actually own a smartphone or tablet at all?
• And how many people fly on an airplane regularly anyways?

If you are reading this, chances are you fall in these categories and so do your peers. But the real answers are surprising.


The data

Baseline for perspective: 81% of US adults use the Internet. 67% of them are on Facebook. (Pew)



87% of Americans have a cellphone, but only 45% have a smartphone. 31% have a tablet. (Pew)



The kicker: 56% of US Internet users have not been on an airplane this past year. 18% never have. (Google Surveys)



Let that sink in

• Only 43% of the US Internet population has been on an airplane in the past 12 months
• More people have used Facebook than have flown on an airplane this year
• More people own a smartphone than have flown on an airplane this year

So don't assume that everyone's experience is just like yours. As we see in the data, outside of your circle the world is very different.

I don't think we should feel ashamed for living in a bubble and having privileges. Products like Dopplr and TripIt are very valuable to their niche of frequent travelers. As long as we acknowledge how lucky we are, it's fine. The problem is when people don't realize they're a niche and assume everyone thinks like they do.


In the end

Perhaps Ben's tweet was deadpan, a hilarious critique of Senator Claire McCaskill, Democrat of Missouri, who really should be focusing on the welfare of the majority of Americans, not the privileged few.


(Discuss on YC HN)

HN is so far gone now that someone felt compelled to put "Time for a technical article" in their submission title.

Shit happens

Had a nightmare about the shape of our funnel. Not based in reality, just exposes my anxieties.

I hope the big winner in this is Mozilla. Hooray diversity! Boo monoculture!

Chromium has forked WebKit. An important nut:

"Nevertheless , we believe that having multiple rendering engines—similar to having multiple browsers—will spur innovation and over time improve the health of the entire open web ecosystem."

Missed this one!

So Tufte decided it was time to out himself as an ex-phone phreaker. He wrote Bowen an email. If AT&T could decline to ruin Tufte's life, maybe JSTOR could find the courage to make sure that Swartz's abilities would not be wasted. Tufte was phreaking before Captain Crunch, before Steves Wozniak and Jobs. Those guys turned out to have lots more to contribute.

JSTOR did the right thing. Not only did it tell the Federal prosecutors that it had come to a satisfactory arrangement with Swartz, but it took significant steps to advance Swartz's (and JSTOR's!) agenda af making information more accessible to everyone.

Best one I've seen all day: Google Chrome’s Broken Image Icon Heals Broken Hearts