Your stupid app made my screen all black and I could not uninstall it do to the fact that I could not see anything on my damn screen!!! I had to take my phone back to the store to get it reset and was charged $30 which is exactly what you need to reimburst me! I HATE YOU!!!Thank you, fan, for reminding me why I made my app free.
Sent from my Wireless Phone
29 January 2012
Fan mail for my app, Screen Filter, which has about 350,000 active installs.
1 reply
03 December 2011
Shortest explanation I've seen of why we're here today:
"This put AT&T in a bind. In 1956, AT&T had agreed to a U.S government consent decree that prevented the company from selling products not directly related to telephones and telecommunications, in return for its legal monopoly status in running the country's long-distance phone service. So Unix could not be sold as a product. Instead, AT&T released the Unix source code under license to anyone who asked, charging only a nominal fee. The critical wrinkle here was that the consent decree prevented AT&T from supporting Unix. Indeed, for many years Bell Labs researchers proudly displayed their Unix policy at conferences with a slide that read, "No advertising, no support, no bug fixes, payment in advance."
With no other channels of support available to them, early Unix adopters banded together for mutual assistance, forming a loose network of user groups all over the world. They had the source code, which helped. And they didn't view Unix as a standard software product, because nobody seemed to be looking after it. So these early Unix users themselves set about fixing bugs, writing new tools, and generally improving the system as they saw fit."
From The Strange Birth and Long Life of Unix
→ reply
"This put AT&T in a bind. In 1956, AT&T had agreed to a U.S government consent decree that prevented the company from selling products not directly related to telephones and telecommunications, in return for its legal monopoly status in running the country's long-distance phone service. So Unix could not be sold as a product. Instead, AT&T released the Unix source code under license to anyone who asked, charging only a nominal fee. The critical wrinkle here was that the consent decree prevented AT&T from supporting Unix. Indeed, for many years Bell Labs researchers proudly displayed their Unix policy at conferences with a slide that read, "No advertising, no support, no bug fixes, payment in advance."
With no other channels of support available to them, early Unix adopters banded together for mutual assistance, forming a loose network of user groups all over the world. They had the source code, which helped. And they didn't view Unix as a standard software product, because nobody seemed to be looking after it. So these early Unix users themselves set about fixing bugs, writing new tools, and generally improving the system as they saw fit."
From The Strange Birth and Long Life of Unix
Instagram Engineering's blog post is interesting, but I can't overcome the feeling: I never even consider 80% of what they describe because it's taken care of for me. I'm spoiled by being high up on a good stack. I spend all day building features.
→ reply
07 October 2011
Google Cloud SQL's coolest feature:
→ reply
"your data is replicated synchronously to multiple data centers. Machine, rack and data center failures are handled automatically to minimize end-user impact."-- Exciting stuff!
27 September 2011
18 July 2011
Samplez is a statistical sampling utility for App Engine, open sourced today as part of the App Engine Afterburner project. I originally built this to track stats in the PubSubHubbub hub codebase. The new module's API is much better and it's optimized for speed. Patches welcome!
→ reply
11 July 2011
Another post where I disagree with the author's outlook on engineering culture at Google. (the other one). The contents otherwise are interesting observations about Google+.
Some highlights:
"As part of induction into Emerald Sea, my team got the 30-minute pitch from the Circles team. I listened politely, all the while rolling-my-eyes in secret at their seemingly implausible naivete."
"I laughed, disbelieving. Facebook has a hacker culture, they're only a handful of engineers, and they develop with quick, adaptable tools like PHP. Especially when compared with the slow moving mammoths we were using at Google."
"On my side of the fence, engineers were increasingly frustrated. Some leaving Emerald Sea for other projects and some even leaving for facebook. I had the impression that Paul Adams was not being heard (if you're not an engineer at Google, you often aren't)."
"Engineers outside Emerald Sea--a cynical bunch at the best of times--were making snide comments and writing off the project as a dismal failure before it even launched."
I'll say that in my experience it's not like this. People don't act this way. It strikes me that the author's perspective is the crucial factor here, not the environment (which I continue to thrive in). My advice in general:
"I think you need to ask yourself why nobody is agreeing with you."
— Hilary Clinton
→ reply
Some highlights:
"As part of induction into Emerald Sea, my team got the 30-minute pitch from the Circles team. I listened politely, all the while rolling-my-eyes in secret at their seemingly implausible naivete."
"I laughed, disbelieving. Facebook has a hacker culture, they're only a handful of engineers, and they develop with quick, adaptable tools like PHP. Especially when compared with the slow moving mammoths we were using at Google."
"On my side of the fence, engineers were increasingly frustrated. Some leaving Emerald Sea for other projects and some even leaving for facebook. I had the impression that Paul Adams was not being heard (if you're not an engineer at Google, you often aren't)."
"Engineers outside Emerald Sea--a cynical bunch at the best of times--were making snide comments and writing off the project as a dismal failure before it even launched."
I'll say that in my experience it's not like this. People don't act this way. It strikes me that the author's perspective is the crucial factor here, not the environment (which I continue to thrive in). My advice in general:
"I think you need to ask yourself why nobody is agreeing with you."
— Hilary Clinton
09 July 2011
Proposal for working around the Python GIL
First off, an acknowledgement: Attempting to remove the Global Interpreter Lock from Python is its own form of alchemy. Here is my attempt to satisfy the practitioners of the Art.
My goal is to add new functionality to Python to enable multi-threading as a first-class component of the language. My motivation is being able to consume N CPU cores on a modern machine in a single Python process, taking advantage of all of the caching locality, concurrency, and throughput multi-threaded processes have to offer on modern operating systems.
This gist of the idea is to allow new Python code to opt-in to thread-aware behavior. I believe the core requirements are:
Isolated threads and owned objects
Every Python function has its own scope, accessible via the locals() built-in function. The core of my proposal is to allow code to indicate that the function's locals dictionary is strictly thread-local. That means every object in the locals dict must be owned by the current thread. The isolated thread will manage reference counts and do its own garbage collection. As soon as the thread tries to access a symbol outside a thread-safe locals dict-- normally symbol resolution traverses up function scopes, eventually to the global scope-- the thread will have to acquire the GIL.
Code will run in two different scopes: global context, while holding the GIL; and isolated context, where all interactions are with thread-local data. Immutable built-in objects will be thread-safe by default (strings, tuples, numbers). When created or accessed in the global context their reference counting will be behind the GIL. When created or accessed in the isolated thread context, their reference counting will be local to the owning thread. Functions declare their thread-safe status with the
User-defined classes and modules declare thread-safety with the
Isolated threads can allocate objects of thread-safe classes without holding the GIL. This gives them a reference to the object in their locals dict. They know that they are the single owner of that object and the only one mutating its attribute dictionary. Isolated threads may pass that object to any other thread-safe code without acquiring the GIL. Isolated threads may transfer ownership of the object via a special type of producer/consumer queue (described later).
Interactions between GILed and thread-safe code
Isolated threads need to be able to call non-thread-safe methods using owned objects, even if it means acquiring the GIL. The risk here is the non-thread-safe code may keep a reference to an owned object after the GILed method completes (e.g., a memoization cache). This opens us up to terrible race conditions like accessing deleted memory.
My proposed solution borrows heavily from the style of TCMalloc. Three parameters would be associated with each PyObject: The
Example synchronization behaviors
1. Isolated thread orphans owned objects: Say an isolated thread called a GILed function with an owned object, which in turn incremented the
2. GILed code accesses owned objects: Say an isolated thread calls a GILed function with an owned object and the GILed function keeps a reference to that object. Sometime later, other code calls the GILed function, causing it to try to access the owned object. Before fetching an attribute, the runtime will look at the object's
3. Isolated thread accesses global objects: Say an isolated thread calls a GILed function that returns a global object as a result. This object's
4. Global thread orphans owned objects: Say an isolated thread called a GILed function with an owned object, which in turn incremented the
5. One isolated thread gets a reference to another isolated thread's owned object: Say isolated thread A calls a GILed function, which holds a reference to an object owned by thread A. Then isolated thread B calls the GILed function and gets back the object owned by thread A. This would immediately raise an
6. GILed code calls thread-safe code: Say a GILed function calls thread-safe code. This will cause all reference increments and decrements to affect the
Transferring object ownership
There will be a new bidirectional producer/consumer queue type. Go's Channels might be good inspiration for the API. Its role is to transfer object ownership between threads. When a producer puts an object into a queue, the Python runtime will verify that the
A lot more to figure out
This is just a sketch. There's a lot more to figure out about specific integration with the CPython API, specific architecture issues like memory fences, etc. I've been sitting on this post since PyCon this year but haven't found the time to figure those things out. I could use some help.
Meanwhile, I saw this recent post about PyPy trying to use STM and it seems totally abstract (no offense intended). I highly doubt we can have our cake and eat it too. The language semantics will have to change. My hope is to identify the minimal changes necessary to make it an easy migration from the GIL to multi-threading.
Let me know what you think!
2 replies
First off, an acknowledgement: Attempting to remove the Global Interpreter Lock from Python is its own form of alchemy. Here is my attempt to satisfy the practitioners of the Art.
My goal is to add new functionality to Python to enable multi-threading as a first-class component of the language. My motivation is being able to consume N CPU cores on a modern machine in a single Python process, taking advantage of all of the caching locality, concurrency, and throughput multi-threaded processes have to offer on modern operating systems.
This gist of the idea is to allow new Python code to opt-in to thread-aware behavior. I believe the core requirements are:
- Tag each object with its thread state ("owned" or "global")
- Thread-local ownership of objects (memory allocation, reference counting, GC)
- Decorators for flagging functions/classes/modules as thread-safe
- Producer/consumer queues for transfering object ownership between threads
- Compatibility mode for acquiring the GIL when necessary
- Ability to see warnings and/or exceptions on accidental GIL acquiring
Isolated threads and owned objects
Every Python function has its own scope, accessible via the locals() built-in function. The core of my proposal is to allow code to indicate that the function's locals dictionary is strictly thread-local. That means every object in the locals dict must be owned by the current thread. The isolated thread will manage reference counts and do its own garbage collection. As soon as the thread tries to access a symbol outside a thread-safe locals dict-- normally symbol resolution traverses up function scopes, eventually to the global scope-- the thread will have to acquire the GIL.
Code will run in two different scopes: global context, while holding the GIL; and isolated context, where all interactions are with thread-local data. Immutable built-in objects will be thread-safe by default (strings, tuples, numbers). When created or accessed in the global context their reference counting will be behind the GIL. When created or accessed in the isolated thread context, their reference counting will be local to the owning thread. Functions declare their thread-safe status with the
@threadsafe decorator.
User-defined classes and modules declare thread-safety with the
__threadsafe__ special attribute. This changes the behavior of a class or module by freezeing its attribute __dict__, making it immutable after definition (similar to __slots__). This enables multiple threads to make concurrent look-ups in the class's immutable attribute dictionary and ensures they do not conflict through mutation. This also means that isolated threads can skip reference counting on a thread-safe class's attributes altogether. Incrementing the reference count on the class itself is good enough to ensure the attribute dictionary and everything it points to are not deallocated.
Isolated threads can allocate objects of thread-safe classes without holding the GIL. This gives them a reference to the object in their locals dict. They know that they are the single owner of that object and the only one mutating its attribute dictionary. Isolated threads may pass that object to any other thread-safe code without acquiring the GIL. Isolated threads may transfer ownership of the object via a special type of producer/consumer queue (described later).
Interactions between GILed and thread-safe code
Isolated threads need to be able to call non-thread-safe methods using owned objects, even if it means acquiring the GIL. The risk here is the non-thread-safe code may keep a reference to an owned object after the GILed method completes (e.g., a memoization cache). This opens us up to terrible race conditions like accessing deleted memory.
My proposed solution borrows heavily from the style of TCMalloc. Three parameters would be associated with each PyObject: The
owner_thread_id of the isolated thread that owns the object (-1 for global objects); the local_ref_count for this object in the isolated thread context; and the global_ref_count for this object from GILed code.
Example synchronization behaviors
1. Isolated thread orphans owned objects: Say an isolated thread called a GILed function with an owned object, which in turn incremented the
global_ref_count for that object. Now the isolated thread is done with the object, and decrements it local_ref_count to zero. The isolated thread knows it cannot garbage collect the object because the global_ref_count is non-zero (synchronized when the GIL was acquired for the original function call). Instead, the isolated thread puts the object on a thread-local list of garbage. The next time the isolated thread acquires the GIL, it will add these garbage objects to the global garbage collector, transferring cleanup responsibility.
2. GILed code accesses owned objects: Say an isolated thread calls a GILed function with an owned object and the GILed function keeps a reference to that object. Sometime later, other code calls the GILed function, causing it to try to access the owned object. Before fetching an attribute, the runtime will look at the object's
owner_thread_id; if it does not match the current execution context, the runtime will raise a ConcurrencyError at the point of access. The reason for this behavior is the GILed code has no way to preempt the isolated code to force synchronization (but the other way around works).
3. Isolated thread accesses global objects: Say an isolated thread calls a GILed function that returns a global object as a result. This object's
owner_thread_id would be -1 (global), its global_reference_count would be positive, and its local_reference_count would be incremented (while the GIL was locked) by the isolated thread that holds the reference. Sometime later, the isolated thread releases the GIL and does more processing; it then tries to access the global object. Before fetching an attribute, the runtime will look at the object's owner_thread_id; it will see that the value is -1 (global), meaning the GIL must be acquired before proceeding. The isolated thread will acquire the GIL and then proceed. Similarly, when an isolated thread needs to decrement a reference to a global object, it will acquire the GIL first.
4. Global thread orphans owned objects: Say an isolated thread called a GILed function with an owned object, which in turn incremented the
global_ref_count for that object. Although the GILed function can never access this object (unless the calling thread context matches), it may create new local references or delete its local references to the object. This is safe because the only context that the global_ref_count can be modified is when the GIL has been acquired. When the global_ref_count is decremented to zero, the global context can assume that the object will either be cleaned up by the isolated thread in the future or the object is already in the global garbage list.
5. One isolated thread gets a reference to another isolated thread's owned object: Say isolated thread A calls a GILed function, which holds a reference to an object owned by thread A. Then isolated thread B calls the GILed function and gets back the object owned by thread A. This would immediately raise an
OwnershipError upon the return of the GILed function. When a symbol is updated in a thread-safe code's locals dict (upon function return), the runtime in the thread-safe context will check that the owner_thread_id of the object either matches the current context or is a global object. Isolated threads may never access to each other's objects. The only way to have cross-isolated-thread communication is through special producer/consumer queues (described below).
6. GILed code calls thread-safe code: Say a GILed function calls thread-safe code. This will cause all reference increments and decrements to affect the
global_ref_count field, not the local_ref_count, since the execution context is global. If the thread-safe code needs to access global symbols, the GIL will already be acquired and the code can move forward without any special handling. Put simply: GILed code can always call thread-safe code.
Transferring object ownership
There will be a new bidirectional producer/consumer queue type. Go's Channels might be good inspiration for the API. Its role is to transfer object ownership between threads. When a producer puts an object into a queue, the Python runtime will verify that the
local_ref_count for that object is 1, otherwise the channel will raise an OwnershipError exception. Putting the object on the queue will remove its symbol from the locals dict. When a consumer thread gets an object from a queue, the owner_thread_id will be updated to the caller context and that thread will take over memory management and GC responsibilities.
A lot more to figure out
This is just a sketch. There's a lot more to figure out about specific integration with the CPython API, specific architecture issues like memory fences, etc. I've been sitting on this post since PyCon this year but haven't found the time to figure those things out. I could use some help.
Meanwhile, I saw this recent post about PyPy trying to use STM and it seems totally abstract (no offense intended). I highly doubt we can have our cake and eat it too. The language semantics will have to change. My hope is to identify the minimal changes necessary to make it an easy migration from the GIL to multi-threading.
Let me know what you think!
27 June 2011
23 June 2011
20 June 2011
Dion nails it: "To bring it to your field Bill..... would you rather have one Shakespeare or 100 Bill Taylor's?" ✄
1 reply
Great "tell" today from an article in the HBR: The author doesn't know that programming is hard. Great hackers are hard to find.
1 reply
18 June 2011
10 June 2011
06 June 2011
I disagree with almost the entire contents of this post from an engineer leaving Google. I wonder why? Maybe it's a tell?-- Mine? Theirs?-- Good luck~
→ reply
27 May 2011
Mission Bicycle is hiring:
→ reply
Our head mechanic Sal is irreplaceable. But we're gonna replace him anyway. He's relocating this summer so we're looking for a personable perfectionist. And because we continue to grow, we've added a new part time position.
If you know someone that's right for the job, please spread the word. We'll be accepting applications through the end of the month.
keep an eye out for two new smiling faces on June 15th.
25 May 2011
Google Correlate launched today. Awesome and impressive stuff.
My favorites so far (just food); look at the correlated words too when you click on the links:
"Hoagie"
"Chitlins"
"Poutine"
"Buffet"
→ reply
My favorites so far (just food); look at the correlated words too when you click on the links:
"Hoagie"
"Chitlins"
"Poutine"
"Buffet"
Volume: I mark conversations in my work Gmail with a label "Me" if I'm in the To or CC fields directly. I did the math tonight: I've received 20 direct emails I've been directly addressed in 20 new email threads per day (on average) for the past 5 and 1/2+ years*. I despise email so much. Fun times, though~
* Note that this is not meant to be a contest. It's just an interesting data point in personal metrics.
1 reply
* Note that this is not meant to be a contest. It's just an interesting data point in personal metrics.
23 May 2011
22 May 2011
Finally figured out how to upload AVCHD .mts files from my Panasonic GF1 to Picasa Web Albums. This has plagued me forever. Picasa and the Uploader for Mac both won't recognize .mts files. The website interface also rejects them. I've actually transcoded .mts to .mp4 with Handbrake just to get these things online.
The solution is bone-headed: Change the file extension tonow with drag-and-drop! You have to use the old file browser to select them; drag & drop doesn't work). Duh~
2 replies
The solution is bone-headed: Change the file extension to
.m2ts and upload them via the browser (21 May 2011
What does it feel like when you're #2 in the Android Market for the word "brightness"?
Approaching 130,000 active installs since I released Screen Filter on November 17th, 2010. Wow!
I gave an update at the two week mark, which seems silly now-- only 2,500 installs then. App Brain's hot apps showing got me more. Now I'm seeing awesome, Japanese reviews of my app. Supposedly it's even covered in this Zine.
Thanks everyone~
1 reply
Approaching 130,000 active installs since I released Screen Filter on November 17th, 2010. Wow!
I gave an update at the two week mark, which seems silly now-- only 2,500 installs then. App Brain's hot apps showing got me more. Now I'm seeing awesome, Japanese reviews of my app. Supposedly it's even covered in this Zine.
Thanks everyone~
20 May 2011
The App Engine Pricing FAQ (published Tuesday) is helpful. They changed parts and highlighted details. Especially relevant to Python programmers. Further feedback appreciated.
→ reply
19 May 2011
Finished The Baroque Cycle at last. Epic. Worth it. I suggest reading Anathem first; it's still sci-fi but gets you in the mood for the historical speculative fiction that starts with Quicksilver. That order's also fun because you can see inspiration for Anathem in the pages of The System of the World. Next up: Avoiding reading for a while~
1 reply
13 May 2011
11 May 2011
10 May 2011
Plans to leave preview is the biggest App Engine news of today. My take on the most important parts of this announcement:
1. "In order to become an official Google product we must restructure our pricing model to obtain sustainable revenue."
2. "[Provide] all paid users a 99.95% uptime service level agreement, operational and developer support, billing via invoice, a new Terms of Service agreement geared towards businesses..."
This means App Engine is here to stay. Build your business on it.
→ reply
1. "In order to become an official Google product we must restructure our pricing model to obtain sustainable revenue."
2. "[Provide] all paid users a 99.95% uptime service level agreement, operational and developer support, billing via invoice, a new Terms of Service agreement geared towards businesses..."
This means App Engine is here to stay. Build your business on it.
App Engine 1.5: Backends, i.e., resident instances with no deadlines, lots of RAM, and continuous CPU burn. Use-cases: Custom in-memory datastructures, Game servers, Chat servers, etc. Also, tons of bug fixes.
→ reply
07 May 2011
06 May 2011
I am not an audiophile. I do not want some huge receiver. So how do I power a good set of speakers? I'm into so-called "Class-T" amplifiers, such as this. Can't recommend these things enough.
→ reply
Royal Wedding bounty of statistics:
"The Official Royal Wedding web site generated over 2,000 requests per second, serving 15 million page views from 5.6 million visitors on the day of the wedding. Despite this load, the web site ran smoothly, serving off Google’s shared platform without disturbing the other 200,000+ apps, which collectively serve over 1.5 billion views per day.
However, from a load perspective, the really busy Google App Engine app was the YouTube Royal Channel site. This app's traffic peaked at 32,000 requests per second (!) - with an additional 10,000 requests per second during the 10 seconds around the kiss(es)."
→ reply
"The Official Royal Wedding web site generated over 2,000 requests per second, serving 15 million page views from 5.6 million visitors on the day of the wedding. Despite this load, the web site ran smoothly, serving off Google’s shared platform without disturbing the other 200,000+ apps, which collectively serve over 1.5 billion views per day.
However, from a load perspective, the really busy Google App Engine app was the YouTube Royal Channel site. This app's traffic peaked at 32,000 requests per second (!) - with an additional 10,000 requests per second during the 10 seconds around the kiss(es)."
01 May 2011
30 April 2011
Historical tale of the day:
"When Germany invaded Denmark in World War II, the Hungarian chemist George de Hevesy dissolved the gold Nobel Prizes of the German physicists Max von Laue (1914) and James Franck (1925) in aqua regia to prevent the Nazis from confiscating them. The German government had prohibited Germans from accepting or keeping any Nobel Prize after the jailed peace activist Carl von Ossietzky had received the Nobel Peace Prize in 1935. De Hevesy placed the resulting solution on a shelf in his laboratory at the Niels Bohr Institute. It was subsequently ignored by the Nazis who thought the jar—one of perhaps hundreds on the shelving—contained common chemicals. After the war, de Hevesy returned to find the solution undisturbed and precipitated the gold out of the acid. The gold was returned to the Royal Swedish Academy of Sciences and the Nobel Foundation who recast the medals and again presented them to Laue and Franck."
-- Aqua Regina
→ reply
"When Germany invaded Denmark in World War II, the Hungarian chemist George de Hevesy dissolved the gold Nobel Prizes of the German physicists Max von Laue (1914) and James Franck (1925) in aqua regia to prevent the Nazis from confiscating them. The German government had prohibited Germans from accepting or keeping any Nobel Prize after the jailed peace activist Carl von Ossietzky had received the Nobel Peace Prize in 1935. De Hevesy placed the resulting solution on a shelf in his laboratory at the Niels Bohr Institute. It was subsequently ignored by the Nazis who thought the jar—one of perhaps hundreds on the shelving—contained common chemicals. After the war, de Hevesy returned to find the solution undisturbed and precipitated the gold out of the acid. The gold was returned to the Royal Swedish Academy of Sciences and the Nobel Foundation who recast the medals and again presented them to Laue and Franck."
-- Aqua Regina
19 April 2011
18 April 2011
07 April 2011
Facebook's Open Compute Project is really cool! I hope companies will take advantage of it. But it's important to call out a tell: Networking is not mentioned anywhere. I wonder why?
→ reply
06 April 2011
31 March 2011
29 March 2011
You should be afraid of what you're doing. It means you're expanding your scope into the unknown. Never being scared means you are not growing.
1 reply