introduction |
current events |
projects |
stuff
Archives of this public on-line diary: 2001 | 2002 | 2003 | 2004 | 2005
All hyperlinks which lead to a different web site are in this pretty green colorRead my new blog!
(My blog is sometimes called a "klog", which is an abbreviation of "hack log".)
My new blog is the first known instance of a new class of software: the "gridapp" or "cloudapp". All of the computation is done client-side, in your web browser, when you look at it (it is written in Java Script). All of the storage is in a secure, decentralized cloud storage grid. The bytes that make up my blog are stored on ten separate servers, and as long as any three of them serve up the cryptographically-verified correct bytes when you do your HTTP GET request, then my blog loads and starts running in your web browser.
I imagine that in the future a lot of software could be written this way -- as JavaScript running in a web browser whenever someone looks at it, and stored, encrypted, in a decentralized cloud of storage servers.
Read my new blog!
(My new blog is the same thing as my klog.)
My new blog is the first known instance of a new class of software: the decentralized web app. All of the computation is done client-side, in your web browser, when you look at it (it is written in Java Script). All of the storage is in a secure, decentralized storage grid. The bytes that make up my blog are stored on ten separate servers, and as long as any three of them serve up the cryptographically-verified correct bytes when you do your HTTP GET request, then my blog loads and starts running in your web browser.
I imagine that in the future a lot of software could be written this way -- as JavaScript running in a web browser whenever someone looks at it, and stored, encrypted, in a decentralized grid of storage servers.
READ MY KLOG.
My klog is back! Hooray!
ACM CCS was great. My old friend Nikita Borisov didn't make it, but I did get to hang around and chat about decentralized secure networks with some of his students or collaborators.
My presentation on Tahoe went well. At the end a fellow named Zhuowei Li, who works on Microsoft Azure, raised his hand to ask a question. It was a bit difficult for me to understand him, as English isn't his first language. "How do you keep track of all the users's files, to see if they have some bad files like pornography or something in case the police ask for it?".
"Well, uh..." I hesitated for a second, "in general, we don't. Many of our users do ask us to keep a copy of their root cap so that we can give it back to them if they forget their password. If we have their root cap, then in theory we can traverse all their files and inspect them, but we have a policy of not looking at users' files, and if they didn't share their root cap with us in the first place then there is no way for us to see their files even if we wanted to."
"But you have to!", he exclaimed.
"What? Why?"
"Legal tells me that we have to keep a catalog of files such as pornography so for example if the FBI comes."
"Ah," I said, "Well, I believe that the Microsoft legal department told you that, but if so then that is a Microsoft policy, not a law in the United States. There's no requirement for my company to have the same policy."
Later I spoke to him in person, and added that if Microsoft wanted to buy http://allmydata.com and use Tahoe in Azure, they could easily implement that policy just by making sure they had copies of all the customer's read-caps.
I went to "BarCamp Boulder #3" this weekend. It was fun. I wore my "BarCamp Palo Alto 2005" t-shirt the first night, but I don't think people recognized it as the founding BarCamp. "BarCamp" is a tradition or a practice nowadays -- there are hundreds of such events all around the globe every year. This one (like the first one) had a lot of "Web 2.0" business model talk, although I don't think anyone called it "Web 2.0".
When preparing my presentation about Tahoe (LAFS), I resolved to come clean about the secret political motivations behind its unusual technical architecture. "I want to live in a world where it is easy for people without a lot of power," I said, "to share web pages, either publicly or privately, which certain powerful people really hate, but which they can't censor or spy on.". To my surprise and pleasure, several people in the audience understood and approved of this motivation. I had half-expected them to say "So, uh, what's your business model?".
(Note: there actually is a business model behind Tahoe, and I actually do enjoy chatting about business at parties, but I didn't want to do so at this one.)
The level of interest among the attendees at this little event is another indication that there is a resurgence of cypherpunk sentiment in the world.
P.S. I guess the guy on the right in this comic is Eric S. Tiedemann (1966-2008). I think he would have approved of my secret political agenda, and his knowledge and insight would have helped with the execution of it. I'll miss him.
After reading about FairplayMP, I was reminded of the existence of VIFF, the Virtual Ideal Functionality Framework.
I'm planning to attend the ACM security conference this year.
Here are some papers due to be presented there that might be interesting:
Here are some more:
In addition, some of these before-the-conference workshops look interesting. I wonder if I can meander in to more than one of them?
And here are the most interesting-looking of the after-the-conference workshops:
And of course the after-the-conference workshop that I really have to attend:
Bah! My new blog broke.
I've been updating my work blog. It may also be interesting to folks (Hi, Mom!) who don't understand the work parts because I'm also writing about my family on my work blog. Because, you know, everything is deeply intertwingled.
My new hlog -- everything you could possibly want to know about the hacking work that I'm up to today. Also, it is a decentralized web app -- the failure of any single server or even any half a dozen servers cannot destroy the contents nor make the contents unavailable.
allmydata.org has announced the Hack Tahoe! contest.
Many bright minds in computer security are already hard at work trying to earn the coolest "Thank You" t-shirt. (I can tell, because some of them have been posting to the tahoe-dev mailing list or the IRC channel.) Join in! If you can find an important security issue in Tahoe, the Least-Authority Filesystem, then we'll give you a specialized t-shirt and add you to the Hall Of Fame!
(This is the text of a letter I wrote to Bruce Schneier. He asked me to post it as a comment on his blog.)
From: zooko@zooko.com Subject: zfone vs. MITM, security & human behavior, polite devices, Re: CRYPTO-GRAM, July 15, 2008 Date: July 15, 2008 16:10:51 PM MDT To: schneier@SCHNEIER.COM Cc: prz@mit.edu
Dear Bruce Schneier:
Thank you for the mention of Zfone in your recent CRYPTO-GRAM:
On Jul 15, 2008, at 1:21 AM, Bruce Schneier wrote:
Man-in-the-Middle Attacks
...
Zfone, a secure VoIP system, protects against MITM attacks with a short authentication string. After two Zfone terminals exchange keys, both computers display a four-character string. The users are supposed to manually verify that both strings are the same -- "my screen says 5C19; what does yours say?" -- to ensure that the phones are communicating directly with each other and not with an MITM. The AT&T TSD-3600 worked similarly.
This is correct, but it omits two other pieces of the design which together make Zfone stronger and more convenient than traditional encrypting phones.
You can see it on the image on this page:
http://zfoneproject.com/prod_zfone.html
This user interface combines traditional voice check authentication, the key-continuity defense against MITM attack, and "sticky-note identification".
This combination is intended to make it risky and difficult to launch a MITM attack on Zfone users even given that most users don't spend a lot of effort on security.
I helped Phil design the Zfone protocol in 2006, and I occasionally consult with him nowadays, as he is getting it standardized and deployed. (I've also, with a certain well-known cryptographer, invented with a newfangled crypto trick to defeat MITM, but that idea hasn't gone anywhere yet. If you're interested I'd be delighted to explain it -- the rest of this letter is not about my new crypto trick, but instead about Zfone.)
To see how we have tried to make the MITM's job as uncomfortable as possible, consider things from the attacker's perspective:
Alice and Bob are dialing one another using their new Zfones. You, as a potential Man-In-The-Middle, have to decide right now whether to do a MITM attack on the Diffie-Hellman key agreement or not. If you don't, then you will be unable to listen in on this phone call, but worse you will also be unable to listen in on future phone calls without triggering both of their Zfone UIs to change dramatically -- removing their friend's name from the sticky-note, resetting the "secure-since" date from the date of their first call to today's date, and unsetting the "Compare with partner" field.
On the other hand, if you do launch the MITM attack now, then if Alice and Bob later make a call which you do not intercept (for example because one of them travels and connects from a different network), or if Alice and Bob, on this or on subsequent calls, do a voice-authentication check then you run the risk of being discovered.
I'm aware of research showing that web browser users are mostly oblivious to security signals on the edges of the web browser, but the Zfone UI is different from those and it may turn out to be effective in the hands of normal users:
To run a full MITM attack with reasonable stealth would be difficult -- in addition to the normal requirements of intercepting and modifying all packets, it would also probably require voice recognition in the loop to detect if the users eventually do the voice-check authentication and abort the attack in a way that looks like normal software bugginess. Even if you did implement this attack, you might be forced to abort right when the users are doing the voice-check authentication, which would tend to arouse suspicion.
If you instead launch a cheap MITM attack without voice recognition in the loop and without comprehensive interception of all of their packets (even on subsequent calls from different locations), then you run the risk of being exposed by the users performing a voice authentication check and finding that the check words differ.
There are many potential attackers who would be deterred by this risk. There is no safe state that the attacker can reach where they can continue to eavesdrop without fear of detection. As long as they are running the attack they are incurring an ongoing risk of being detected or at least of arousing the suspicions of their victims.
This risk of subsequent detection is incurred even if most users are careless most of the time -- it only takes one voice-check authentication to retroactively expose the earlier MITM attack on earlier calls. They might do this for example because they've decided to talk about something sensitive. (I remember when I switched to using hushmail to exchange letters with the girl who is now my wife. It was when our correspondance started to become a little more intimate.)
Caveat: actually there is a safe state that the attacker can reach, if they can reliably, in near-realtime, fake the victim's voices doing the voice-authentication check. This is what Phil calls "The Rich Little Attack". I believe that such an attack is currently expensive (i.e. the most cost-effective way to do it is probably to hire a talented human or several). The crypto trick that I invented on has the potential to defeat even this attack, but as I mentioned I couldn't figure out how to make it practical.
Okay, this letter was way longer than I intended, but I hope it shows why Zfone has the potential to be substantially stronger, even in the hands of users with imperfect security practices, than previous (and current) alternatives.
By the way, good job on organizing the workshop on security and human behavior. Like behavioural economics and "experimental philosophy" (as someone recently termed it), human-behavior-oriented security is a much needed perspective.
I liked your rechristening of "polite devices" into "selective device jamming". I would point out that politeness among humans is voluntary. (When it becomes involuntary it is no longer a matter of politeness.)
Most people, including me, would be happy to configure their devices to voluntarily do things like silence themselves in theatres. The capability security folks have a paradigm for the implementation of such behavior, called "Voluntary Oblivious Compliance".
Regards,
Zooko O'Whielacronx
P.S. Feel free to publish or re-use this letter as you see fit, with one caveat: a certain well-known cryptographer is, as you probably know, jealous of his privacy, and it might be polite to remove his name before publishing this letter.
P.P.S. Dear Phil: writing the above has made me think that Zfone would be more secure if it didn't have those "Secure" and "Clear" buttons on the UI. There would be less cognitive clutter to distract the users from the important stuff. Maybe the "Secure" and "Clear" buttons could be moved into the "Advanced Usage" window, if not removed altogether.
(This is the text of a comment that I posted on the excellent Linux-loving web magazine LWN.net. Both the readers of and the writers of LWN.net are Linux kernel hackers and/or Linux lovers, so I expect this post to generate some heated responses. See also my next post, in which I point out that malicious actors can exploit bugs which are so rare that they never occur in the non-malicious case.)
"That said, I didn't actually _test_ my patch. That's what users are for!"
-- Linus Torvalds
Sigh. I know Linus likes to be brash, but this is, for me, a "Ha ha only serious" moment.
This really is the Linux development paradigm in a nutshell -- the core developers generate new code and update existing code as fast as possible and without too much time and effort spent on quality control, because they are relying on a vast and ill-organized community of other hackers, distributions, users, etc. test it and report bugs. (And then, as Andrew Morton and others have repeatedly stated, most of those bug reports go nowhere.)
From a systems/evolutionary viewpoint, this process should be expected to be very efficient at quickly generating lots of valuable new functionality and fixing what I'll call the "99% percentile" flaws -- the ones that show up frequently enough and are analyzable enough that this process will fix them. This is ESR's notion that "to enough eyeballs all bugs are shallow".
Except that it doesn't apply to all bugs, only to the ones that are frequent enough and analyzable enough that they can be noticed by users, communicated effectively to developers, and reproduced or analyzed by developers.
So by the same token, this process should be expected to generate more and more of the rare bugs, the three-nines or four-nines bugs, i.e. the ones that are visible only in 1 run out of 10,000 runs or only for 1 user out of 1000 users.
And indeed, there is some circumstantial evidence (nicely chronicled by LWN.net's reporting on the concerns of Andrew Morton and other Linux core developers) that this is exactly what it is doing.
Now obviously there is great value in a tool which is very featureful and flexible and performant and widely supported and swarming with rare, subtle, small bugs. Linux is very widely used and provides great value to its users. But this kind of tool is not the right tool for every job.
I really hope to find a Free and Open operating system that I can use whose core developers are a little more... careful. Fortunately the new open source version of Solaris has worked fine for me so far. Thank goodness for diversity and competition. Now I've got to figure out how to gain insight into the Solaris core development practices and see if it is any different.
Last night I played "Groundies" in Martin Park. In Groundies, It has to keep its eyes closed. The other players have to keep their feet off the ground usually, because if It shouts "Groundies" then anyone touching the ground becomes It. Also if It touches you, you become It, and It stops being It, if you see what I mean. So if you are not It then you have to climb and swing all over the play structure to stay out of Its reach, ideally while also being quiet, or at least quieter than the other players. If you are It then you have to hear or guess where the players are going and then suddenly flail out in a random direction in the hopes of tagging one of them before they can scurry away. This sounds difficult, not to mention dangerous.
Also It has a kickball that they can throw at you, and if it hits you (even after rebounding multiple times), then you're It. Therefore this variant is sometimes called "Dodgies", which according to Nat sounds like a really sketchy brand of breakfast cereal.
At first I sat out because my knee is sore, but it looked like so much fun that I finally decided to try it in spite of my knee, and it turned out to be great fun and also easier than I thought -- for some reason you soon find yourself swinging around the outside of a play structure, utterly blind, flailing and lunging at your invisible targets. The others congratulated me on my clever, dramatic, and occasionally successful manuevers, and nominated me for Most Valuable Player afterward, which felt very good, because I had feared that I was too old and overweight for it.
(This blog entry brought to you by Danny O'Brien, who is now following me on Twitter.)
Yesterday a couple of people thanked me for contributing to open source software (Python packaging tools). It felt so good! I am highly motivated by gratitude, including a simple "Thank you.".
I forgot to mention that at the end of the Hack Fest the other night Sebastian played a video on one of the projectors and we all giggled and giggled. He uploaded it to the Tahoe Test Grid. It's the one named "How-To-Irrigate-Your-Nasal-Passages.flv".
The First Boulder Hack Fest was a success. In attendance were Sebastian Kuzminsky, Adam Boggs, Marek Sotola, Tom Tromey, Christian Grothoff, Neal McBurnett, Maciej Fijalkowski, and myself. allmydata.com, my employer, bought us all pizza, beer and snacks. Thanks to Seb, the University of Colorado gave us nice meeting space with power, networking, comfy chairs, and projectors.
Maciej demonstrated an interactive graphical, zooming, graph-layout visualizer/debugger for PyPy, which made people expostulate enthusiastically. Tom asked many questions about PyPy's design (Tom's day job is hacking on gcc and gdb). Seb showed movies of big machines making physical things, controlled by Free Software programs and tools, and of smaller machines built by "homebrew fabricators" in their garages and basements. The Free Software hackers in attendance got excited to learn about this promising new technology and about the tiny but growing culture of Free Software hackers who are part of it. Christian explained a few things about GNUnet and Seb asked several questions about it, which questions seemed to be related to Seb's day job of "Delay Tolerant Networking". Neal powered up an OLPC XO-1 and started playing Pong in Forth (note: there was no "booting" step involved in this process.), which was cause for amused celebration and a discussion of how Forth is like the ancient abstract strategy game Go. Or maybe the discussion was about how Go is like Forth. There were many other discussions of tools (compilers, build tools, revision control tools, OLPC, wireless networking cards) and other topics, not all of which I overheard or took notes on, so this report is incomplete.
Maciej showed me how to plug in a new implementation of the builtin dict type in PyPy, and we started implementing such a thing using Judy Trees, but we stopped after implementing __new__() and __getitem__() and __len__() because he had to go and also because it is awfully inconvenient for C code to think that it can hold on to RPython objects -- what if the RPython garbage collector wants to move those objects around, for starters. Maciej added our experiment to a PyPy SVN sandbox, but we agreed as he left that a quicker experiment would be to make a type that doesn't serve as the standard dict for all of RPython, but is instead a separate dict-like object which simply maps ints to ints. That will be very easy to implement using Judy Trees (as soon as Maciej sends me the boilerplate for creating your own foreign-function-interface-implemented objects in RPython) and will serve to measure the basic performance of such a Judy Tree-based data structure in RPython.
Everyone was eager to see the 3-D printers and other automated fabrication tools that CU Boulder owns, but they were currently locked up for the night, so Seb agreed to arrange a Field Trip for Boulder Hack Festers during business hours when the machines are observable.
That's it for now! I'll be out of town for the next four weeks, mostly in San Francisco, so if there are any Boulder Hack Fests in that period they'll have to be organized without me.
I've argued with Brian Warner -- an exceedingly good engineer -- about the advisability of relying on synchronized clocks (as implemented by e.g. NTP) for secure decentralized systems.
Brian mentioned how DJB -- another exceedingly good engineer -- disapproves of the widespread use of UTC, since UTC is adjusted every few years to add a leap second, in order to maintain the property that noon UTC is when the sun is highest in the sky over some spot in England. This property is not important to computers, DJB correctly observes, and fiddling about with leap seconds can cause problems for the properties that are important to computers, such as your clock monotonically increasing, and your clock's rate not diverging significantly from the rate of time experienced by your human user, from the rate of time indicated by the clocks in other computers that you talk to, or from the proper time of a machine that you are observing or operating. DJB recommends that computers use TAI -- Temps Atomique International -- instead. That is just like UTC except that it doesn't insert leap seconds to keep its "noon" synchronized with the Earth's rotation.
In light of this engineering controversy, I was fascinated by the accounts spread out around the net of how the most recent UTC leap second -- the one at the dawn of 2005 -- caused various kinds of misbehavior in various computers. The highlights (low-lights?) include some clocks jumping backwards (violating the monotonically non-decreasing property), various kinds of unnecessary rate changes, and in one case the clock completely losing sync with the NTP masters until it was reset by a human.
Well, in light of all that, I am hoping that by the next time a leap second comes around, the secure decentralized systems that I am responsible for or rely upon are not using UTC and attempting to stay in sync with it by mechanisms such as NTP. When will that be? Well, nobody actually knows. The politicians, bureaucrats and scientists responsible for deciding when to insert leap-seconds are having a big political squabble about it and haven't made up their minds. One expert predicts that it will be the dawn of 2009.
Hopefully by then the systems that I rely on will use TAI. Actually, I would be even more satisfied if they used local monotonically-increasing clocks and did not rely on having their clocks synchronized with other clocks at all.
(This blog entry brought to you by my mom, who found something lacking in the previous blog entry.)
For the last couple of days it was beautiful, warm and sunny here. Irby and Elliot would play outside in the back yard (they are feeding bits of leftover food to ants and building bridges and structures for the ants to walk on) after dinner, since the sun has grown so tardy in setting that it is perfect backyard weather at seven P.M. (which was formerly bed-time for boys). Elliot, seeking some leftovers to distribute, would politely ask in his little three-year-old voice "Can I feed my ants?".
Today it is snowing! Everyone walks around shaking their heads affectionately about their crazy weather. Weather is one of those experiences that make people feel an affiliation with their community. The snow melts as soon as it lands on sidewalk or street.
Amber and I arranged a "household administration date" at Caffe Sole this morning, but we got distracted and talked about the properties of hash functions instead, because I was excited that I had just dared to post the following message to a mailing list populated by Serious Cryptographers:
Dear Rene Peralta:
On May 1, 2008, at 8:11 AM, Rene Peralta wrote: > At 09:14 AM 5/1/2008, you wrote: > > (By the way, it is interesting to note that while second pre-image > > resistance (ideally, collision resistance) is required for this use, > > pre-image resistance isn't. The data being input to the hash > > function is not secret.) > ... > Since hashing is a many-to-one mapping, pre-image resistance is a > precondition for second pre-image and collision resistance.Thank you for this argument. I've heard this argument before (in personal conversation with John Kelsey), and I think there must be some disconnect between theory and practice here. Perhaps the disconnect is between information-theoretic and computational conceptions of "pre-image resistance".
Consider the function SHAstupid:
SHAstupid(x) = SHA-256(x) || first_256_bits_of(x)Where "||" is string concatenation, and first_256_bits_of() is equal to the first 256 bits of its argument.
To me, it seems intuitive that SHAstupid has collision resistance better than SHA-256 has, but has very poor pre-image resistance.
Perhaps I should be using the term "partial pre-image resistance" instead of "pre-image resistance"?
This distinction seems to me to be important in practice. For one thing, it suggests that a combination like
SHAboth(x) = SHA-256(x) || Tiger(x)Probably has better collision resistance than either SHA-256 or Tiger has, and probably has worse (partial, computational) pre-image resistance than either hash function alone.
If your application requires collision resistance, but does not require to preserve secrecy of the inputs (as do all of the applications in the "bulk data use case" [1]), then SHAboth is probably a more secure (if more expensive) alternative to SHA-256. Some of the applications in the bulk data use case use constructions like these.
Regards,
Zooko
(This blog entry brought to you by some stranger I've never heard of before and who is mostly invisible to web stalking.)
Yay! Donald Knuth has called "Emperor Has No Clothes" on the multicore CPU trend:
To me, it looks more or less like the hardware designers have run out of ideas, and that they’re trying to pass the blame for the future demise of Moore’s Law to the software writers by giving us machines that work faster only on a few key benchmarks!
...
The machine I use today has dual processors. I get to use them both only when I’m running two independent jobs at the same time; that’s nice, but it happens only a few minutes every week. If I had four processors, or eight, or more, I still wouldn’t be any better off, considering the kind of work I do—even though I’m using my computer almost every day during most of the day. So why should I be so happy about the future that hardware vendors promise? They think a magic bullet will come along to make multicores speed up my kind of work; I think it’s a pipe dream.
It may be too much to hope that the vast and rich market in computer components will pay attention to Knuth's criticism. If you are a consumer of CPUs then you should try to measure the actual effect of multiple CPUs or multiple cores on your work. Chances are that a multi-core CPU like this one will be less stable, use more power and generate more heat and noise, and provide absolutely no actual performance benefit to you, while costing hundreds of dollars more, compared to single-core CPU like this one. (There are a few technical or scientific uses where multiple cores are worth their cost. If you are a consumer of CPUs for one of those uses, I would like to hear about your benchmarks, too...)
Here's the full interview, which includes discussion of open source, literate programming, MMIX, TeX, and whether the legend is true that he once won a programming contest by writing a program which worked correctly on the first compile. He also describes his current development environment, from pencil and paper and big wastebasket to Ubuntu and backupfs.
(This blog entry brought to you by Réjane.)
I'm still excited about my work: allmydata.org -- a secure, decentralized filesystem. I'm travelling to Chicago this weekend for a conference (Pycon) and then returning before the conference is over in order to be here on Irby's seventh birthday. It's really hard to believe that he's so big.
Irby is in First Grade. His favorite activity is having play-dates with his friends, in which they play computer games or make-believe. He reads from chapter books to himself every night at bedtime.
Elliot is three and a half years old, and he is excited about the prospect that when he has a birthday, he might be as big as Irby. He also loves wearing Irby's clothes, and refuses to acknowledge that they might be a bit baggy on him. Fortunately for Elliot, his big brother usually enjoys playing with him, and when he doesn't want to play with him he's usually kind about it.
I occasionally listen to this podcast about economics called econtalk. It has a broad definition of "economics", which is as it should be. This week's episode is an interview with Prof. Deborah Gordon, an entomologist who studies ants. She mentions that the recent interest in figuring out how ant colonies self-organize was inspired in part by the growing realization that self-organization was a useful technique for building computer systems.
Earlier I posted an explanation of how you can install Python modules which were packaged with either distutils or setuptools, using GNU stow to limit the authority available to the installer/install scripts and to simply and uniformly managed all packages that you've installed. That post is still worth reading, as it explains exactly what GNU stow does (which is easy since it is so simple) and why you might want to use it.
However, it turns out that using the --root option to distutils or setuptools messes up the paths stored in .pyc files, which can cause subtle problems. Therefore, there is unfortunately no single process that will install either a distutils-packaged or a setuptools-packaged module in a stow-compatible way. But fortunately there is a straightforward process that will install a distutils-packaged module and another straightforward process that will install a setuptools-packaged module.
Run:
python ./setup.py install --single-version-externally-managed
If it says "error: option --single-version-externally-managed not recognized" then it is distutils. If it says "error: You must specify --record or --root when building system packages" then it is setuptools.
In the following, let us use "$PKG_NAME" to denote the name of the package that you want to install. A convenient trick is to set a shell environment variable to equal the name of the current directory, like this:
PKG_NAME=`basename ${PWD}`
How To Make A Place To Install Into (without executing unknown code as root)
sudo mkdir -p /usr/local/stow/$PKG_NAME &&
sudo chown `whoami` /usr/local/stow/$PKG_NAME
How To Install Into That Place With Distutils
python ./setup.py install --prefix=/usr/local/stow/$PKG_NAME
How To Install Into That Place With Setuptools
python ./setup.py install --single-version-externally-managed --record=/usr/local/stow/$PKG_NAME/$PKG_NAME-install.log --prefix=/usr/local/stow/$PKG_NAME
How To Make Symlinks From Your System Directories Pointing Into That Place (using GNU stow)
cd /usr/local/stow && sudo stow $PKG_NAME
How To Delete The Package
rm -rf /usr/local/stow/$PKG_NAME
How To Remove Any Symlinks From Your System Directories Pointing Into That Place (using GNU stow)
cd /usr/local/stow && sudo stow -D $PKG_NAME
That's it! Now you can safely install setuptools-packaged and distutils-packaged Python modules (along with pretty much all other Unix-compatible software) in a uniform way that leaves packaging under your control by using your filesystem and your familiar tools such as "rm" and "mv".
How To Do This All At Once
Addendum: I put this all into a script (including the part about figuring out whether it is setuptools or distutils), which I can now invoke to spare myself typing effort. Here it is:
STOW_DIR=/usr/local/stow
PKG_NAME=`basename ${PWD}`
PKG_DIR=${STOW_DIR}/${PKG_NAME}
if python ./setup.py install --single-version-externally-managed 2>&1 | grep -q "You must specify --record" ; then
SETUPTOOLS_OPTS=" --single-version-externally-managed --record=${PKG_DIR}/${PKG_NAME}-install.log"
fi
sudo mkdir -p ${PKG_DIR} &&
sudo chown `whoami` ${PKG_DIR} &&
${PYTHON:-python} ./setup.py install${SETUPTOOLS_OPTS} --prefix=${PKG_DIR} &&
pushd ${STOW_DIR} &&
sudo stow -v ${PKG_NAME} &&
popd
2007-05-21 dura-link v1.0.1
Announcing Allmydata Tahoe dura-link v1.0.0
The company that I work for, Allmydata, Inc. has released a free software project -- Tahoe -- a distributed storage grid. The code we're releasing is in fact the working prototype which will grow into our next product, but in addition to that, we hope that it will be useful to other people who want to use or extend a decentralized storage grid. The rules of the licence are standard GPL (which basically says that you can use our source code if we can use yours) but it also has the added feature that you may keep your derived work proprietary for up to twelve months. I hope that this will make it a little bit easier for people who are considering using our source code in a commercial endeavour. You can, of course, always contact us and ask for other licensing terms.
For more information, please see the release announcement, the licence, and the web site
2007-04-27 dura-link v1.3.1
Brought To You By dura-link v1.0.1
This blog entry brought to you by Tav (he told me he reads my blog, which motivated me to update it).
distutils or setuptools + GNU stow dura-link v1.3.1
Update: thanks to PJE, the author of setuptools. for pointing out that with a slight change to this recipe it will work the same with both distutils and setuptools.
It turns out that there is a straightforward way to tell setuptools to emit the built package contents in a nice Unixy layout in a diretory you specify. This makes it easy to put the entire contents of the package into a separate directory which you can later rm in order to delete the package from your system.
You should also learn to automate the process with GNU stow. GNU stow is extremely simple. It is extremely simple to learn, simple to use, and in fact if all copies of GNU stow were to disappear from the earth tomorrow, you could reimplement it in about a hundred lines of Python or bash. It uses no database, config, file, or metadata file of any kind. There are, in my experience no options that you need to pass to it on the command line. It is 100% compatible with your existing packaging tools (such as apt-get, rpm, BSD ports, fink, etc. and it is 98% compatible with all of the software that you want to install. The 2% exception is those apps which think that they need to know the fully qualified path from the root directory to their current location, such that if you mv them or access them through a symlink, they break. I have installed somewhere between one hundred and two hundred packages with GNU stow, and only two of them broke in this way, and both of them were configurable so that they stopped doing that and became stowable.
Okay, so the way to tell setuptools to produce a package entirely within a given directory is:
./setup.py install --prefix="." --root=$DESIREDLOCATION
Note that you should not use sudo to do this, since setuptools or the package's setup.py file might have bugs so that it attempts to write files outside of the prescribed directory. Instead, create the $DESIREDLOCATION and set permission bits on it so that you can write into it without root power.
For example, when using GNU stow you can tell setuptools to put the package into /usr/local/stow/$packagename, so to install zfec, I would do this:
sudo mkdir -p /usr/local/stow/zfec-1.0 &&
sudo chown `whoami` /usr/local/stow/zfec-1.0 &&
./setup.py install --prefix="." --root=/usr/local/stow/zfec-1.0
(The name of the package -- here zfec-1.0 is entirely up to you. You could call it just zfec if you like, or that_new_erasure_code_tool_from_zooko.
Then you tell GNU stow to make symlinks that live in /usr/local/lib, /usr/local/bin, /usr/local/man, and so on, which point into /usr/local/stow/zfec-1.0/lib, /usr/local/stow/zfec-1.0/bin, /usr/local/stow/zfec-1.0/man, and so on, like this:
cd /usr/local/stow && stow ./zfec-1.0
That's all GNU stow does -- make those symlinks. And the rule it uses to make the symlinks is dead simple: it finds all files and directories in the ./zfec-1.0, and then it makes a symlink from the same relative path rooted at /usr/local to the relative path rooted at /usr/local/stow/zfec-1.0. Make sense?
Now when you want to uninstall zfec, you can always just
/bin/rm -rf /usr/local/stow/zfec-1.0
But that will, of course, leave a bunch of dangling symlinks in /usr/local pointing at the place where zfec-1.0 used to live. So (either before or after rm'ing that directory), you can ask GNU stow to remove all such symlinks:
cd /usr/local/stow && stow -D ./zfec-1.0
/bin/rm -rf /usr/local/stow/zfec-1.0
Done! Now you can easily and safely install arbitrary Unix source from source, including setuptools-packaged source code.
2007-04-04 dura-link v1.0.0
Brought To You By dura-link v1.0.0
This blog entry brought to you by Mom.
The Magic Words dura-link v1.0.0
For the last few years I've practiced suppressing my tendency to be nice and polite with telemarketers. Not that I speak rudely or angrily or anything -- just that I'm willing to hang up the phone without first explaining to them what I'm about to do or trying to get a polite "good-bye" from them first. I consider this to be good practice -- suppressing my natural social urges in the service of higher values such as spending a few more minutes with my family.
Even though I've gotten fairly good at it, dealing with telemarketer calls remains an interruption and struggle. Recently I learned from the estimable Brian Warner that there is, at least in the United States, a magic incantation which causes the telemarketer to jump straight to the end of the script, with no more sales attempts! The magic incantation is "Please put me on your Do Not Call list.".
2007-03-21 dura-link v1.0.0
Songfight Live in San Francisco dura-link v1.0.0
I'm going to be in San Francisco next week, and I intend to see this live performance by songfight.org, a bunch of amateur rock bands that compete to produce new tracks every week and publish them freely on the net. You can see the bands chattering with each other about organizing the shows on this web forum.
2007-02-21 dura-link v1.0.0
Downhill in the Sunshine Forever dura-link v1.0.0
Yesterday Irby and I ran home from school. It was the first beautiful sunny day of spring and he took off his sweater and we laughed and held hands as we ran. As we rounded the corner of our block Irby said "I wish we could run downhill in the sunshine forever!". Me too, Irby.
2007-02-20 dura-link v1.0.0
Brought To You By dura-link v1.0.0
This update brought to you by AaronSw and Peter Eckersley who visited the Allmydata Crow's Nest all too briefly.
John Robb and Riverbend dura-link v1.0.0
John Robb has an impressive track record both for the acuity of his analyses and for the prescience of his predictions. His blog is well worth browsing if you want to understand or predict the evolution of global warfare and its impact on society. His most recent entry forecasts a major attack by Al Qaeda, targetting either the fragile global oil infrastructure or the (also fragile?) psyches of United States voters.
The blog of a young Iraqi woman named Riverbend is heartbreaking and unforgettable. Start at the beginning. Or start at the current day if you don't feel you have time, but later start at the beginning, which is more of a personal narrative and less of a political rant. If you find yourself expressing an opinion to other people about the situation in Iraq, and you haven't read Riverbend's diary yet, then you're probably full of shit.
2007-01-12 dura-link v1.2.1
Brought To You By dura-link v1.0.0
This update brought to you by Seth Schoen, who told me at Thanksgiving that he reads my blog.
Why Habeas Corpus Matters dura-link v1.1.1
I previously posted my outrage that the Military Commissions Act of 2006 violated the writ of habeas corpus. Some people that I've talked to didn't understand why this abstract legal theory with a Latin name is so important, so I'll spell it out:
The writ of habeas corpus means that the government is not allowed to make people "just disappear".
It should be obvious that adherence to this principle is necessary for the protection of our other freedoms. A little reflection on the regimes of recent history which did not honor this principle should drive the point home. The writ of habeas corpus was important enough to The Framers of The United States Constitution that they enshrined it with the following rule (in the article prescribing the powers of the legislative branch):
The privilege of the Writ of Habeas Corpus shall not be suspended, unless when in Cases of Rebellion or Invasion the public Safety may require it.
[The United States Constitution, Article I]
According to Nick Szabo's article, the Constitution specifies that only Congress may suspend the writ of habeas corpus, and only in case of rebellion or invasion (that is -- not in case of a foreign war), and even then only if public safety demands it.
usconstitution.net describes the writ of habeas corpus this way:
The basic premise behind habeas corpus is that you cannot be held against your will without just cause. To put it another way, you cannot be jailed if there are no charges against you. If you are being held, and you demand it, the courts must issue a writ or habeas corpus, which forces those holding you to answer as to why. If there is no good or compelling reason, the court must set you free. It is important to note that of all the civil liberties we take for granted today as a part of the Bill of Rights, the importance of habeas corpus is illustrated by the fact that it was the sole liberty thought important enough to be included in the original text of the Constitution.
[usconstitution.net]
Meta-Slash dura-link v1.0.0
When typing some text in (X)Emacs, try hitting M-/. (X)Emacs will finish the current word you were typing in the style of tab-completion. In fact, it's a little-known feature that you can write the first line of your Python script and then just hit M-/ repeatedly and (X)Emacs will finish your program for you. (Thanks to Brian Warner. (Didn't know he was an Emacs and Python hacker, did you?))
2006-10-15 dura-link v1.4.1
The Ides of October dura-link v1.4.1
Here in Boulder, Colorado the trees have already changed to a beautiful set of reds, golds, yellows, browns and also green needles. Certain of the needles smell strongly sweet when you break them off in your hands while walking to school. The weather is vascillating between summer and fall.
My private blog remains quiescent. Maybe I'll upload a picture of the beautiful boys? Or maybe a thousand words.
Although I normally disdain to talk about politics, the Military Commissions Act of 2006 cannot be ignored. This un-Constitutional law strikes at the legal and moral foundations of our nation. No terrorist could ever hope to damage America as deeply as this law does. The erudite and creative scholar Nick Szabo (who is no Democratic Party sympathizer) has posted an interesting explanation of habeas corpus -- what it means, and why it matters. Liberty and habeas corpus part 1 , Liberty and habeas corpus part 2
Please observe carefully how your senators and representatives voted on this shameful bill. The process of restoring America's glory starts by throwing these irresponsible people out of office.
2006-09-17 dura-link v1.2.2
ZimmermannRTP dura-link v1.2.2
Attention: It was a joke that Phil Zimmermann named ZRTP after me. In truth, the Z stands for "Zimmermann". Phil conceived and produced the original PGPfone in the mid-1990's. In 2004, I was jobless and looking for work so I talked to Phil and he said he had been thinking about starting a project to bring some of PGPfone's ideas to the modern world of VoIP, and to leverage standards such as SIP, RTP and SRTP. He paid me to help him design the protocol and to implement it. Although I contributed substantially to the design of the secure network protocol which became known as ZRTP, Phil was the one taking all of the financial risks, the final decision maker on all issues of cryptography, network protocol, and user-interaction, the primary inventor of the unique Man-In-The-Middle defenses, the sole holder of copyright and trademark, and the man solely responsible for the business development and the product. I'm very proud of my contribution (look for my name in the credits), and I was amused at the coincidence of the initial "Z", but make no mistake -- Zfone and ZRTP are Phil's babies and he deserves all the credit.
By the way, Phil has pursued a strategy of openness with Zfone, making the source code publically visible for peer review, documenting the protocol and submitting it to the IETF for standardization and even, according to the Zfone FAQ, currently considering GPL'ing the implementation! This is a risky business strategy, and Phil deserves praise and support for his civic-minded decision to make the protocol and the implementation open and easily implementable. You can show your support (and get a cheap, convenient, and highly secure voice tool at the same time) by trying out the beta release from the Zfone web site and sending feedback to Phil about how it performs.
2006-09-01 dura-link v1.1.0
Hi! We made it to beautiful Boulder, Colorado.
Google Books now has downloadable PDFs! Here's a link. I learned this by way of Adam Langley, who is partially responsible for having implemented it. I visited google campus a couple of weeks ago and happened to see Adam walking by and hailed him. It was nice to see him.
2006-05-24 dura-link v1.0.0
We're moving to beautiful Boulder, Colorado! ETA June. Send me e-mail if you would like more details, possibly including digital pictures.
2006-03-10 dura-link v2.0.2
ZRTP dura-link v2.0.2
I'm very glad that Phil Zimmermann, Alan Johnston and Jon Callas have submitted an Internet Draft for the ZRTP protocol for securing Internet phones. I poured my heart into this project in 2004 and 2005, encouraging Phil to restart his old "PGP Phone" project and working with him to design and implement ZRTP. I'm very pleased that he is now taking it public, publishing standards and hopefully in the near future releasing an open source implementation (hopefully the one I wrote). ZRTP uses some innovative and clever crypto techniques to make your phone calls safe, even against very powerful and sophisticated eavesdroppers, without any centralized key management. That last clause is the important point, because would-be eavesdroppers have more leverage over centralized key managers than you do.
I'm also quite pleased that he has chosen to name it after me. "ZRTP", for "Zooko's Real-Time Protocol".
Joke alert: caveat lector. See ZimmermannRTP
2006-03-09 dura-link v1.0.0
tools for working on Microsoft Windows dura-link v1.0.0
I just got a new laptop with Microsoft Windows on it. Here are tools that I install to take more control of the system and to empower myself to do more:
- Install FireFox web browser; download and install
- Install Cygwin toolset; download and run the "install Cygwin now" setup.exe packager, and then clickety-clickety-click to get all the tools you want. You can always execute Cygwin's setup.exe again later to get more tools or to uninstall some that you currently have installed. Most important tools:
- bash
- XEmacs (requires X-windows, which will be automatically installed when you select XEmacs)
- rxvt -- the best terminal app for w32 that I've yet found
- gcc
- GNU stow -- for simply and easily tracking what locally-compiled software you have installed
- Install darcs -- simple, flexible, and painless revision control tool; Click on the link entitled "Precompiled binaries" and then on "Microsoft Windows" and then on the one labelled "with Cygwin". I use darcs on pretty much all text files, configuration files, source code (mine or other people's), etc.
- Configure cygwin.bat to launch rxvt.exe instead of cmd.exe. I do this by using darcs to fetch my configuration files from another system. You can do it by copying my cygwin.bat.
- I would include GNU Screen in that list of most-important tools, but alas it still doesn't work with Cygwin.
Okay! Now I have a toolset that gives me control over the system and the power to do what I want.
Depending on your needs, you might also be interested in the non-Free Software vmware workstation tool. It can give you added control over the system, for example the ability to rewind the state of your virtual machine to an earlier snapshot, and better control over the network activity of the virtual machine. It also allows you to run multiple virtual machines at the same time, to install other operating systems on virtual machines, etc. On the other hand, the virtual machines will run dramatically slower than the real machine, and since vmware isn't Free Software it doesn't give you complete control.
2006-02-19 dura-link v1.0.0
I'm happy to see that I'm getting e-mails requesting access to the precious Irby Christmas Morning video.
2006-02-18 dura-link v1.0.2
I made a web page to hold the 2005 archive of this public on-line diary. It is much smaller than the previous years. I've become more protective of the privacy of my family.
On the other hand, an old friend wrote to us and said she reads this web page, and that she misses Irby and wishes to see more pictures and stories about him. So I've started a private on-line diary where I feel free to post such things. The first episode features a video of Irby on Christmas morning 2005! If you would like access to it, just e-mail me.
2006-01-30 dura-link v1.0.0
censorship.google.com dura-link v1.0.0
Google.com is now telling people in China that this is what the world thinks of Tianenmen Square: http://images.google.cn/images?q=tiananmen (hint: replace "google.cn" with "google.com" and try again). I'm hoping that this is the wake-up call that persuades people of goodwill that Google's motto of "Don't be evil." is bullshit. Of course for-profit corporations will deploy catchy marketing slogans, but google's slogan is more offensive because of its hypocrisy and sanctimony.
I wonder if google.com will tweak this search result in order to deflect criticism. Perhaps they could just start returning different search results depending on IP address, so that those meddlesome foreigners won't be able to see what they are telling people in China and raise objections to it.
2006-01-20 dura-link v1.2.0
reads his mails dura-link v1.0.0
The Good Reader is very interested in reading his e-mails. (I'm telling you, he is phenomenally good at reading.) He is only sporadically interested in composing replies. Maybe I should just make a video or an audio recording of him reading the mails and send those back to the correspondent. :-)
darcs devel community dura-link v1.0.0
A few times now I've seen people write that the darcs revision control system is hobbled by being written in an obscure language (Haskell) and thus depriving itself of contributors. However it appears to be the case that darcs enjoys a large and active community of contributors -- at least as large and active, and possibly more so, than the other Free Software revision control tools.
Not all of these contributors are fluent in Haskell. Those who aren't do things like integration with other tools (Tailor, Trac+Darcs+Bazaar-NG), add-on tools (darcs.cgi, darcsweb, a patch-dependency grapher, a xml-to-ascii translator for annotate results), unit tests (written in bash and Perl), testing and bug reporting.
To get an idea of the current development community around darcs, look at the darcs-devel mailing list or the issue tracker.
Zooko
Last modified: Fri Mar 20 12:28:30 MDT 2009