Author: Christie Koehler

software engineer, geek, yoga practitioner, bike commuter, zen buddhist, queer, vegan, legion of tech board member, osbridge planner, engineer@ShopIgniter

Python’s os.fork, subprocess, and the case of the missing output

One of my current projects is helping to improve contributor documentation and tools for Zulip, an open source group chat platform. We recommend that first-time contributors use the Vagrant method of setting up the Zulip development environment.

One current downside of this method, in the context of Zulip, is that many of the helper tools are written to be used with a Zulip development environment setup directly on a host system running Ubuntu. An example of this is the pre-commit hook, which looked like this when I first started working on Zulip:

./tools/lint-all $(git diff --cached --name-only --diff-filter=ACM) || true

This code is intended to be run in the zulip directory, with a Python venv activated and all dependencies installed and configured. So, if I run it in my Mac terminal, I see the following:

ImportError: No module named typing
You need to run the Zulip linters inside a Zulip dev environment.
If you are using Vagrant, you can `vagrant ssh` to enter the Vagrant guest.

That’s a helpful enough message and if I ssh into my zulip dev environment, I can run the linters:

christie@Olamina [~/Work/zulip] $ vagrant ssh vagrant ssh
...
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./tools/lint-all

Which is good, but doesn’t align with how I actually work. Recall that this command is part of a git pre-commit hook. It’s intended to be run by git when I run git commit, which I do on my Mac terminal, not within the Zulip dev environment. I like to use git on my Mac terminal because that’s where I have git configured and where I have all my fancy oh-my-zshell terminal integrations. I could set this up inside the Zulip dev environment, but because that’s designed to be encapsulated and generalized for anyone contributing to Zulip, I’d have to do it every time I re-created or re-provisioned the environment. Not ideal.

Knowing that you can pass commands through vagrant ssh just like you can through regular ssh, I figured it would be easy to update the pre-commit script to work for both types of dev setup. I tried this:

vagrant ssh -c '/srv/zulip/tools/lint-all $(cd /srv/zulip && git diff --cached --name-only --diff-filter=ACM) || true'

The only change I made was to add a cd to the zulip directory before running git diff. This is necessary because ssh’ing doesn’t start you in the /srv/zulip directory.

It seemed to work. I saw some output from the linter:

Fix trailing whitespace at README.prod.md line 4:
This documents the process for installing Zulip in a production environment.

Until this point I’d been testing the hook by invoking the script directly by running .git/hooks/pre-commit in Mac my terminal. Now it was time to try with an actual git commit. The command took a long while to complete, so it seemed as if the linter was running, but I didn’t see any output. WTF?

I vagrant ssh’d into the dev environment and tried a git commit there (after setting the bare minimum config values that git requires). The linter ran and generated output. I was back at square one.

Let’s recap the situation:

I saw linter output when I:

  • Ran the .git/hooks/pre-commit script directly via Mac terminal
  • Ran the vagrant ssh -c command directly via Mac terminal
  • Ran .git/hooks/pre-commit, ./tools/lint-all, or git commit inside vagrant virtual machine

I do not see linter output when I:

  • Ran git commit via Mac terminal outside of the virtual machine.

Because the hook intentionally returns true, and therefore allows git to continue with your commit, no matter the output of the linter, having output is essential. It’s the only way you know you have code style issues to address.

First I investigated the bash/ssh aspect of the situation. I spend a couple of hours reviewing Bash documentation, including a side tangent about psuedoterminals that was both frustrating and fascinating. This led no where. In fact, all I ended up figuring out was that the Bash conditions under which the linters were evoked were essentially identical. (I did this by putting a shoptat the beginning of .git/hooks/pre-commit and comparing the output from running it on my Mac terminal and inside the Zulip dev environment side-by-side.)

At this point it was well into dinner time and I called it a night.

By the time I returned to my keyboard the next morning, fellow Zulip contributor Steve Howell had been working on the issue and eliminated vagrant as the issue by reproducing the missing output using plain old ssh. This and my work from the previous evening shifted my focus to the ./tools/lint-all python code itself. I dove in, determined the understand each and every line.

This took a while. lint-all is almost 500 lines. It looks like one of those things that started simple, but has grown more complex over time as its parent project grew more complex. It has very few inline comments.

Here’s what I figured out:

  • lint-all runs several linter scripts in parallel using Python’s os.fork(), subprocess.call(), and subprocess.Popen().
  • The linter scripts are a combination of other stand alone commands (e.g. jslint, puppet parser, etc.) and inline Python code. Each linter wrapper is its own function and is cataloged by the lint_functions dictionary. The run_parallel() function loops through lint_functions, creating a new child process for each with os.fork() and then executing it. It also checks the status of each of the child processes and returns failed == True if any of them exit with a failed status.
  • Most of the linter wrappers use subprocess.call() to invoke their respective command line scripts. The exception is the function that runs the pyflakes linter. Pyflakes writes some messages to stdout and others to stderr. So, in order to capture both kinds of messages and then loop through them to determine what to output, subprocess.Popen is used with arguments stdout = subprocess.PIPE and stderr = subprocess.PIPE

(If all of that is really confusing, feel free to take a look at this highly simplified version I wrote to accompany this post.)

While I was learning how lint-all worked, I asked other Zulip contributors as I went along. At one point, Steve Howell asked what the redirected output looked like and I ran this within the virtual environment:

 ./tools/lint-all > out.log 2>&1 

And it created a file without any output. Wait, what? > out.log 2>&1 should have redirected both stdout and stderr to the file out.log.

So, when I ran the linter, I saw output on screen. But when trying to redirect that output to a file, I got nothing. Within the virtual environment.

At this point I knew the problem had something to do with these child processes and pipes, but I was getting tired. So I did what any good programmer does when they’re stuck, which is start asking questions on Twitter:

To which Adam Lowry replies with this hint:

And so I added sys.stderr.flush() and sys.stdout.flush() prior to the call to os._exit() and that fixed it! I could now see complete output when I ran the linter under all conditions.

For whatever reason, output that was in the stdout and stderr buffers was getting lost when child processes were terminated. The python docs for os._exit() does give a hit about this:

os._exit(n)
Exit the process with status n, without calling cleanup handlers, flushing stdio buffers, etc.

Lesson learned? If in doubt, .flush() your output. Also, Zulip’s ./tools/lint-all could use some re-factoring. I haven’t tackled it yet, but I’d try getting rid of that os.fork() and using subprocess methods functions entirely, or another concurrency method.

Again, if you’d like to try or see a simplified example of the problem for yourself, take a look here.

And here’s the final git pre-commit hook:

#!/bin/bash

# This hook runs the Zulip code linter ./tools/lint-all and returns true
# regardless of linter results so that your commit may continue.

# Messages from the linter will be printed out to the screen.
#
# If you are running this one machine hosting a Vagrant guest that
# contains your provisioned Zulip development environment, the linter
# will automatically be run through `vagrant ssh`.

if [ -z "$VIRTUAL_ENV" ] && `which vagrant > /dev/null` && [ -e .vagrant ]; then
    vcmd='/srv/zulip/tools/lint-all $(cd /srv/zulip && git diff --cached --name-only --diff-filter=ACM) || true'
    echo "Running lint-all using vagrant..."
    vagrant ssh -c "$vcmd"
else
    ./tools/lint-all $(git diff --cached --name-only --diff-filter=ACM) || true
fi
exit 0

Update 27 August, 2016 17:10 PDT: Brendan asked me about the use of os._exit() so I took another look at the docs. That led me to wonder about using sys.exit() instead, which led me to this post. Using sys.exit (code) appears to obviate the need for the .flush() statements.

Life one year after leaving Mozilla

A year ago tomorrow was my last day as a Mozilla employee. Quitting was the one of the best decisions I’ve ever made. I’m working for myself and I love it.

Here are some projects that I’ve been working on:

Building a wood bookcase from scratch.

Bookcase I built from scratch.
Bookcase I built from scratch.

Watching and photographing the birds that visit our yard.

Pine Siskins have words at the feeder.
Pine Siskins have words at the feeder.

Recruiting and on-boarding several new Stumptown Syndicate board members.

Driving to California (accidentally during a snowstorm) to visit family. One of the things we did on the trip was visit the Monterey Bay Aquarium, which I had never been to.

23774600566_edd8ecf29f_z

Starting a consulting practice, Authentic Engine. I have a few client projects now, and am looking to book more work for the Fall. Know an open source project wanting expert help with participation, leadership, or governance issues? Get in touch. I’m also available for contract programming (python, php). If you like stickers, I have those for sale too.

No endeavor is truly launched until it has stickers.
No endeavor is truly launched until it has stickers. And so it is with Authentic Engine. Buy some!

Learning audio recording and engineering and launching the Recompiler Podcast.

recompiler-podcast

Handing off the organizing of Open Source Bridge to two new co-chairs!

Taking Bertie to the beach for the first time. He loved it. Although, we learned the hard way bulldogs really can’t exercise for very long because Bertie needed several days of recovery afterward.

Bertie's first trip to the beach.
Bertie’s first trip to the beach.

Seeing my favorite band, The Cure. Twice! Once in Los Angeles and again closer to home in Ridgefield, Washington. We had much better seats at the Ridgefield show and had a fantastic time.

The Cure performing Just Like Heaven, Ridgefield, Washington, May 2016
The Cure performing Just Like Heaven, Ridgefield, Washington, May 2016

Touring SpaceX. While we were in LA to see The Cure, a friend of ours arranged a tour of SpaceX. It was amazing. Couldn’t take any pictures inside, unsurprisingly, but managed to get a goofy selfie outside.

Smiling because we just toured a fraking rocket facility!
Smiling because we just toured a fraking rocket facility!

Attending Allied Media Conference, including the excellent Growing our Souls tour (my photos) of Detroit.

Me, at Project Heidelberg in Detroit, Mi
Me, at Project Heidelberg in Detroit, Mi

Shopping for individual health insurance plans three times. Yes, three times in one year. The first was before I left Mozilla because applying COBRA would have been prohibitively expensive (~$1,400 per month). The second was during 2016 open enrollment because our rates had been raised over $100/month. The third was last month when the State of Oregon abruptly put Oregon Health Co-op into receivership. Fun times! But, hey, at least thanks to the ACA, we can actually sorta find health insurance outside of a group plan. We’re with Providence now and we hope they stay affordable and in business for a while.

Gardening. Lots and lots of gardening. This year we planted lots of vegetables and added several new flower beds, populated mostly with plants I started from seed. It turns out I have a bit of a green thumb! Who knew?

An evening's harvest from our garden.
An evening’s harvest from our garden.

 

Flower beds in bloom. Most of these I grew from seed.
Flower beds in bloom. Most of these I grew from seed.

Photographing the flowers I’ve been growing. I don’t have a macro lens, but am faking it well, I think, with my 35mm and some close-up lenses.

Bee on Shirley Poppy, one of the many flower macros I've taken this season.
Bee on Shirley Poppy, one of the many flower macros I’ve taken this season.

Watching hot air balloons launch at the Tigard Festival of Balloons. I first heard about this festival shortly after I moved to Portland in 2007 and realized just before my birthday that it’s practically in our neighborhood, so Sherri got us tickets. It was challenging to get up at 5am to get ourselves over there in time for the sunrise launches, but it was so worth it.

Hot air balloons launching at the Tigard Festival of Balloons!
Hot air balloons launching at the Tigard Festival of Balloons!

Speaking at Open Source & Feelings on a really tough topic.

Reading and more reading. Soon I will have read all of Walter Mosley’s Easy Rawlins‘ books and this will be bittersweet. I take solace in knowing I still have plenty left to read of Mosley’s Fearless Jones and Leonid McGill series and I’m only a little over half-way through Laurie King’s Mary Russell and Sherlock Holmes books. Plus you never know when you’re going to stumble across a great new detective series such as M.J. McGrath’s Edie Kiglatuk or Cara Black’s Aimee Leduc Investigations.

Feeling better about myself and being less stressed than anytime during the previous 4+ years. Our income isn’t steady yet and dealing with health insurance is obnoxious. But we’re making it work. I can say now that leaving a job that was steadily grinding me down was absolutely the right call, even if it felt totally wrong at the time.

Thanks everyone who’s supported me along the way and continues to do so! You make all the difference. If there’s anything I can do for you, please let me know.

 

When the Worst Happens (OS Feels 2016)

What follows is the text I used as my guide during my 2016 Open Source and Feelings talk When the Worst Happens. You can also view a video recording of the talk. [Link added 26 August, 2016]

Some housekeeping to get us started. First, I have cough-variant asthma. I can usually get through a shorter talk like this, but just in case I want you to know that if I’m over-taken by what sounds like a very bad cough, I’m okay. I’ll just step to the side here, cough my way through it, and then return. Hopefully if it happens we can crowd-source some jokes or maybe Utah could dance a jig.

Second, a few content warnings. I’ll be mentioning suicide. I’ll also touch upon issues around burnt out, mental health, housing instability, and other ills of capitalism. Please do what you need to do to take care of yourself and feel free to step out at any time. I’ll be sharing my slides and a transcript afterwards, so you’ll have a chance to catch up if and when you’d like.

And, I’ll take questions at the end, if time allows, or during the break.

OS Feels 2016.003

In Portland, we run a conference, some of you may have heard of it, Open Source Bridge. We call it the conference for open source citizens. We focus on participation, connection, and inclusiveness across the open source community. I co-chaired the conference for five years. This year was my first having no official job other than letting everyone else do theirs without butting in. It was wonderful. For me, it was incredibly rewarding to be able to step back from something I’ve invested much of my heart into and see others continue the good work without me.

OS Feels 2016.004

I also helped start a non-profit called Stumptown Syndicate. The Syndicate works to bring tech and maker skills to communities in ways that enable positive social change. Through my work with Stumptown Syndicate, I’ve also helped organize many other tech gatherings, including BarCamp Portland. I am also co-author of the Citizen Code of Conduct.

Somewhere in the middle of all this organizing a few colleagues and I thought it would be a good idea to share what we’d been learning. So, we put together a book and workshop on how to organize community tech events.

Like all good workshop materials, ours included a bingo card of things that could go wrong.

OS Feels 2016.005

Each of these squares describes a problem we have directly experienced as organizers or as attendees. If you don’t already know about the branded toilet paper at OSCON that one year, all you need to know is: DON’T. Just no.

You can tell I haven’t updated this card in a while because otherwise it would have a square for “food-borne illness outbreak”, which is something that happened at Open Source Bridge last year.

The point of this card is to show that yes, things will go wrong. And that often the things that go wrong won’t be what you expected or prepared for. By putting all these things that can go wrong on one card, we wanted to demonstrate that you can survive the bad stuff and become stronger for it.

Or, at the very least you can shout BINGO at some point.

There’s another square is missing from this card. I’m not sure I will add it because it turns out it’s probably the hardest thing to deal with both as an individual organizer and as a community. If it were to be included, it would be an entire bingo card on its own. And probably in too poor of taste. I’m speaking of the unexpected death of a key organizer.

In 2013, a few days before we were to send speaker acceptances and finalize our schedule for Open Source Bridge, Igal Koshevoy, one of our key organizers and someone who’d worked on Open Source Bridge since its beginning, took his own life.

OS Feels 2016.009

Those closest to Igal knew about his struggles with depression and social anxiety. He had been withdrawn from the community for months. Welfare checks had been requested and performed.

And yet, you are never prepared for this kind of loss.

Igal died in early April, 10 weeks before Open Source Bridge was scheduled to begin. Contracts had been signed. Tickets had been sold. We had to notify speakers and publish our schedule.

OS Feels 2016.010

We had no choice but to move forward. To make the conference happen. To help settle the affairs of our friend. To navigate our grief. And, most of us had day jobs and families to attend to as well.

We managed to keep the conference more or less on schedule. Open Source Bridge year 5 happened. It went as well as it ever had. We had a big party at the Oregon Museum of Science and Industry where we celebrated our 5th anniversary and we celebrated Igal.

OS Feels 2016.011

While writing this talk, I tried to figure out exactly how we kept everything moving. I couldn’t. I looked through the archives for our email planning list. The messages don’t look substantially different from any other year. We cancelled exactly one planning meeting, the week of Igal’s death.

I think we were able to do this because we had a lot going for us, relatively speaking. It was year five of the event, not year one or even two. We weren’t in a significant transition year in terms of team members. And our core team is really dedicated.

We also were supported by the Portland tech community at large.

While we planned our conference, we also organized a\celebration of life for Igal. This is how I learned that it turns out a lot of the skills you use to organize tech events also apply to memorials. As they often did with so many of our events, the Portland tech community came together to help. The folks who we’d been convening with for years at barcamps, user groups, and other events pitched in here and there, in small and big ways to help make Igal’s celebration happen.

It turns out that for a lot of people, Igal was one of their first connections to the Portland tech community. It was his warmth, his presence, his dedication, that helped bring people into the community and feel welcome.

These are just a few of the things folks said about Igal upon his passing:

OS Feels 2016.013

This was certainly true for me. I moved to Portland in Fall of 2007 and Igal is one of the first folks I distinctly recall meeting. Actually, I encountered Igal first on twitter and I was kind of intimidated by him because of his avatar:

OS Feels 2016.014

Do you ever have this experience? You meet encounter someone, either on twitter, or in person, and your impression of them is, “too cool for me!” or something similar? It makes me wonder what people’s first impression of me is…Anyway…

I saw Igal’s avatar and, this is embarrassing, I thought, damn, this dude can bench press hundreds of pounds. And can probably light objects ablaze with a single glance!

Once I met him though I quickly realized my impression was a little off. Although very resourceful as a former eagle scout, Igal wasn’t at all scary or intimidating. He was warm, friendly, and dedicated to serving the community. He loved cats. He went out of his way to see that people had what they needed. He spoke several languages. He used to take and share the most amazingly detailed, useful notes.

OS Feels 2016.021

Once he explained to me his process for creating these notes, I never again took note-taking for granted again. For Igal, it was complicated and time consuming because the language of his thoughts, the one his mind used to process, was not English. So, he had to translate twice, once upon hearing the information and then again upon writing it. That was Igal, though. He would do the work even when it took twice the effort if he though it would be helpful.

OS Feels 2016.023

Igal also created beautiful art. Although some of it was a little weird, in a funny way.

And Igal struggled, the extent to which is something I learned only after he was gone.

My impression of Igal changed significantly from when I first met him to after I got to know him. And it evolved further when I learned things about him after he passed.

I call attention to these disparities for a reason.

We often project ideas about the organizers in our communities that aren’t necessarily true. We project that they feel empowered by the power we place in them. We project they feel like they can ask for help because we are willing to provide it if asked. We project that they know they are appreciated because we feel appreciative of all that they do. We project that they are doing okay because we see them getting shit done. We project that they feel loved because we love them and all they do for us. We project that they can set objects ablaze with a single glance because they have a stern twitter avatar.

OS Feels 2016.029

While holding leaders accountable is important we also have to recognize that we need to build systems and communities that allows leaders access to self-care and respite and appreciation and support as they need it.

OS Feels 2016.030

In addition to planning our conference, and a memorial for Igal, we worked to safe guard our community’s assets that had been in Igal’s possession at the time of his death.

Because Igal was so involved in the activities of Portland Tech, he actually own or controlled a lot of our communal assets too. Mostly this included things like domains, code, and hosted applications.

In our case, we were lucky, if you can call it that, on multiple counts. Although Igal attempted to leave a will, it was not legally valid. We were able to transfer all of the community’s assets that Igal had been hosting because: a) Igal left us with al his passwords and login details, and b) By this time we had already created Stumptown Syndicate and so we had somewhere appropriate and uncontroversial to migrate things to. And his next of kin was cooperative.

Igal didn’t host all of these things for us to be controlling or to concentrate power, but rather he did it out of a sense of service and to provide convenience. I think this is true for a lot of our community work. You get together to write an app to do a thing. When it’s time to deploy someone creates a droplet or a vm or a vps or whatever’s en vogue at the time and you get up and running.

I think if we were to take a census, the majority of our small group work product in open source is probably technically owned or at least controlled by individuals.

Think about your own community’s work. Who owns the accounts under which things are setup? Who has access to these accounts? Does they have access because of shared login credentials, or have they been made an authorized accessor? What would happen to these accounts and whatever assets they hold if the owner were to die unexpectedly?

OS Feels 2016.031

The laws on this vary by location, but generally if the person hasn’t left a valid will — and many people do not have valid wills — then ownership/control falls to the person’s next of kin. Is that person going to be available? Accessible? Cooperative? how many of you have tried to explain open source to your relatives?

This issue is relevant not just to community assets, but to you personal body of work as well. If you care about how your work is used and where any proceeds generated by your work should be directed after you’ve passed, you need to account for that in your estate planning.

A while ago Neil Gaiman noticed that authors had a similar situation in that it was common for authors to pass on without having a will in place that explicitly outlined their wishes with regard to their intellectual property. Not having this document quite often created conflict among surviving friends and family members, and/or an outcome the author clearly would not have wanted. When I read Neil’s post I immediately thought of Stieg Larsson.

OS Feels 2016.032

I’m guessing we have some Girl with the Dragon Tattoo fans here, so some of you may know about this. Larsson’s wildly popular Millennium trilogy was published after his sudden death from a heart attack at age 50. Like Igal, Larsson left a will but it was not legally valid. As a result, Larsson’s estate, including control of his literary works, fell to his family members, and not his long-time partner Eva Gabrielsson. And that’s why we have a fourth Lisbeth Salander book not written by Larsson.

And so Neil worked with an attorney friend of his to create a boiler plate will that explicitly covered an author’s intellectual property.

OS Feels 2016.034

I think we need something like that for open source work.

It’s been a little over three years since Igal passed. I no longer feel the same acute pangs of grief that I did then, although I occasionally think I’ve spotted Igal out of the corner of my eye and a split-second of joy gives way to a dull ache of sadness and regret.

Looking back, what I regret most is not spending more time just hanging out with Igal while he was with us. Outside of the few annual holidays that Igal celebrated with us in our home, most of my interaction with Igal happened in the content of our open source community work.

This pattern isn’t specific to my friendship with Igal. I’ve built a lot of friendships based around work. Actually, until recently, I’ve built most of them around some kind of “work.”

Some of this is Protestant culture and work ethic. I grew up in a family where work was paramount and play was discouraged and sometimes even punished. I learned early on that in my family, volunteering was one of the few permissable way to socialize with others. And, it often made accessible to me activities that were otherwise out of reach because of our economic situation.

Open source greatly benefits from people like me who are conditioned to seek out extra curricular work projects and volunteer opportunities and so it is rewarded and encouraged. Indeed, open source thrives on the line between vocation and hobby being as blurry as possible, because open source is dependent on a steady source of free labor. This is a problem because there are very few, if any, mechanisms for monitoring contributor well-being.

OS Feels 2016.036

Why is this so and what does it mean for our movement?

Well, what do our FOSS forebearers have to say about this? It turns out, not much. They have focused, for the most part, on the intellectual property rights, the licensing aspect, of open source.

For example, the Free Software Foundation is primarily concerned with user freedom and does much of its work via GPL enforcement. GPL being the “GNU Public License,” which they created.

OS Feels 2016.037

The Open Source Initiative has made it its business to determine what licenses even qualify as open-source.

OS Feels 2016.038

The few organizations, such as Apache Foundation, that are focused on individual developers are concerned mostly with infrastructure and with ensuring the production of open source software continues, rather than contributor and community well-being.

OS Feels 2016.039

What these approaches lack, and it is a serious omission, is recognition that open source is as much a mode of production, as it is a matter of intellectual property rights. By “mode of production” I mean the combination of labor and the materials, infrastructure, and knowledge required to make and distribute things. In the case of open source, we’re generally talking about software, and sometimes hardware.

OS Feels 2016.046

What are unique characteristics of open source software production?

The labor is a mix of paid and unpaid. Labor tends to me geographically distributed, with many working in nontraditional office environments during non-traditional work hours. There is no professional body governing entrance into open-source. Or ethics, or continuing education. For the most part if you have the skills, or the gumption to learn them, and access to a computer and Internet, you can start producing. The materials and infrastructure are provided by mix of individuals, foundations, and for-profit companies. Material input and work product is shared by partners and competitors alike. Because a major advantage is being able to use the prior work of others, Open source tends to favor interoperability and standards compliance. And because contributions of code are among the easiest to track and therefore make visible, “merit” is assigned disproportionately to technical contributions and technical roles tend to be valued above all others.

All of this sounds like open source as a mode of production greatly favors the individual worker. And there are many aspects of open-source culture that is very empowering. You can contribute from nearly anywhere. You can generally take share your portfolio as you move through your career. People with no formal training whatsoever can become prolific open source contributors. With an inexpensive laptop and access to the Internet you can create and share software without prohibitive licensing fees.

OS Feels 2016.051

This last reason is why got into open source. It wasn’t ideology. It was economics. What mattered to me at the time was “free as in beer” not “free as in freedom.” As time went on and I gained some financial stability and started participating in open-source I got more into the ideology. It seemed to explain why I felt so good being part of the open source community. Not only was helping to build something greater than myself, but I was helping a righteous cause. Free software felt righteous.

OS Feels 2016.052

And for a time, I thought contributing to open source giving me a unique career advantage. For several years, as I became more and more involved and visible and open-source, I got better, higher paying jobs. Eventually I even got my open source dream job, at Mozilla.

OS Feels 2016.053

But it turned out my open-source dream job wasn’t all that dreamy. While a highly visible and often looked up to the organization, Mozilla is highly dysfunctional. And it’s socially regressive particularly when compared to many of its peers in the tech Industry.

During the time I struggled at my dream job I watched friend after friend hit their own glass walls of frustration and burnout. I watched them run out of patience waiting to be appreciated and recognized for their technical acumen. I watched friends struggle over and over again to meet their basic needs for housing, food, and healthcare.

I observed in myself a struggle to balance my increasingly complicated health and family needs with my open source community obligations. I struggled to understand why I was unable to effectively apply my extensive open source experience at my dream job Mozilla. All the while I was growing weary and really wanting to break from unpaid volunteer work in order to explore other life pursuits.

Loosing Igal, watching people I care about struggle, and facing my own disallusionment, prompted me to start thinkingcritically about open source and how current norms and practices affects individual contributors and our communities. I’m not the alone in this. Many of us are starting to have these conversations.

Here’s where I am in my analysis.

There are two significant downsides to open source as practiced today.

The first is that organizations disproportionately benefit. How? Three ways.

OS Feels 2016.057

ONE: They profit from the aggregate of our free labor. The sum of the free labor that we collectively give to organizations reduces their cost of production, and thereby increases their profit margin. And they are under no obligation to redistribute any of that profit or to recognize the economic value of our labor in any other way whatsoever.

TWO: Organizations have more power than individuals working alone. There are a few exceptions to this, like if you’re Linus Torvolds and you hold the copyright to something like Linux. But generally speaking, large open source projects, or even medium-sized ones, face little to no consequences for dysfunctional behvior because they easily withstand it when folks stop contributing here and there. At worst, the organization will actually have to hire someone and pay them a salary to do the work of an unpaid contributor who has left.

THREE: Organizations have a better chance at sustainability because they have greater access to capital to invest in product research, development, and distribution, which in turn generates greater profit. They have greater access to capital because they have access to investors and because they have a cheap labor source (us).

The second significant disadvantage to open source today is that individuals, on aggregate, disproportionately suffer. How? Three ways.

OS Feels 2016.061

ONE: The economic value of our labor is made invisible. This is bad because it contributes to underemployment and underpayment. We might think we’re getting good resume-building experience — and for folks brand new to tech this might be true. But it has significantly diminishing returns. Most companies won’t pay for things they know they can get for free.

TWO: Open source promotes building community and social relationships that are fundamentally fragile, unaffirming and extractive. This is because open source transforms what should be our third spaces — places that are anchors of community life like the neighborhood coffee shop, the community center, the local church, and so on, into work places. The problem with this is that it brings work-type stress into our personal relationships. It keeps those relationships one-sided, bound if not to professionalism directly, but to a directive to always be getting things done. It squeezes out the space to simply play together. Relax together. Be together.

THREE: An empasis on consistent technical contribution as a requirement for inclusion reinforces the idea that we are only as valuable in so much as we are able to contribute. If we become unable to contribute, either in the short term or the long term, of if we were never able to fully participate in the first place, it can leave us feeling worthless. Our value as human beings and our right to be included in our communities shouldn’t be contingent on an on-going ability to consistently make technical contributions. We are so much more than that.
The title of this talk is, “When this worst happens.” On a micro-level, it’s above the challenges our community faced when loosing one of our core organizers. On a macro-level, it’s about how we’re all participating in a system dependent on an ever-steady stream of contributors who give as much as they can, often without any monetary compensation for their labor, until they can’t give any more. Not being able to give any more means different things for different people. Some face emotional and physical burnout. Some have to take paying jobs outside of open source. Some can’t find paying work even with their extensive open source experience. Some decide that the pain of this world is too much and decide to leave it like Igal did.

OS Feels 2016.062

When I criticize open source culture, I’m not arguing in favor of proprietary licenses. I think the less restrictive intellectual property rights are, the better society is as a whole. Rather, I’m arguing that we need a better model of free and open source software. One that prioritizes our humanity. One that helps meet our needs for food, shelter, and security. One that helps us build communities that realize everyone’s potential, whatever that may be. One that contributes towards the building of a better future for us all. The four freedoms are meaningless if people don’t have enough to eat, clean water to drink, a place to live, or adequate healthcare for mind and body. They are meaningless where trans folks don’t have a safe place to go to the bathroom and where black lives don’t matter.

“When machines and computers, profit motives and property rights are considered more important than people,…the giant triplets of racism, materialism, and militarism are incapable of being conquered.”

I came across this quote while reading the other weekend and it shocked me how contemporary it felt. Does anyone recognize it? Martin Luther King said this in 1967 about the Vietnam War.

OS Feels 2016.064

What do I want you to take away from this talk?

One, ensure your community’s assets are safeguarded. At the very least, document where everything is and how to access it. For bonus points, set up something to handle significant transitions.

Two, ensure personal body of work is safe guarded by creating a will, one that specifically covers your digital and creative work.

Three, build community in well-rounded ways and help restore our third spaces. Think of yourself more like an ecumenical community center than a user group or open source project.

Finally, join me in building a better, more sustainable model. One that recognizes and fairly compensates everyone’s labor and reinvests into our communities. Think of starting co-ops and benefit corps, rather than vc-funded 10x startups or even traditional foundations.

OS Feels 2016.066

During the last couple of weeks I’ve been asking for the names of folks we’ve lost before their time.

OS Feels 2016.068

They are listed here so that we may honor them now. I know this list isn’t complete, though. Please add any name that’s missing now, either by speaking it silently to yourself, or out-loud as you like.

I feel like this has been a heavy, dreary talk. So I want to leave you with some words of hope.

OS Feels 2016.069

Angels In America: A Gay Fantasia on National Themes, is one of my all-time favorite plays. It’s a massive work about AIDS and queerness and 1980s America. One of the issues it grapples with is how to make sense of loss. This quote is from the monologue that closes the play, it goes like this:

“Night flight to San Francisco. Chase the moon across America. God! It’s been years since I was on a plane. When we hit 35,000 feet we’ll have reached the tropopause, the great belt of calm air. As close as I’ll ever get to the ozone. I dreamed we were there. The plane leapt the tropopause, the safe air and attained the outer rim, the ozone which was ragged and torn, patches of it threadbare as old cheesecloth and that was frightening. But I saw something only I could see because of my astonishing ability to see such things. Souls were rising, from the earth far below, souls of the dead of people who’d perished from famine, from war, from the plague and they floated up like skydivers in reverse, limbs all akimbo, wheeling and spinning. And the souls of these departed joined hands, clasped ankles and formed a web, a great net of souls. And the souls were three atom oxygen molecules of the stuff of ozone and the outer rim absorbed them and was repaired. Nothing’s lost forever. In this world, there is a kind of painful progress. Longing for what we’ve left behind and dreaming ahead. At least I think that’s so.”

I believe we are working towards that kind of painful progress. And, I like to believe the souls of those we lost are helping us to do that, still. I ask you now, let’s dream up a better future and start building it together today.

OS Feels 2016.070

Thank You.

 

I forgot to include acknowledgements at the time I gave me talk, so I’ll do so now:

Thank you Sumana Harihareswara, Ed Finkler, and Amy Farrell for reviewing my proposal for this talk and encouraging me to submit it.

Thank you Chris Ball, Chris Koerner, Andromeda Yelton, Federico Mena Quintero, Chris Swenson, Damien McKenna, Floren F, Dan Paulston, and Davey Shafik for helping me compile names for the remembrance portion of my talk.

Thank you Sherri Koehler, my lovely wife, for staying up late on Skype to hear me reverse drafts of it.

Thank you Jeremey Flores and Kerri Miller for being super supportive of me giving this tough talk and for putting on an awesome conference.

Lastly, thank you Igal Koshevoy and Nóirín Plunkett for your friendship. Miss you both.

 

Free speech and its many meanings

The last two weeks I’ve been spending way too much time reading, thinking and tweeting about LambdaConf. If you’re unfamiliar with the situation and want to catch up, read this excellent post by @codepaintsleep.

One thing observing the situation has made me realize is that some folks have a very different conception of free speech than I do.

In my mind, our right to “free speech” prohibits the government from restricting or punishing speech. I do not view it as anti-free speech when a group of private citizens denies you the opportunity to speak in their spaces. Nor do I believe our constitutionally protected (in the United States) right to free speech guarantees anyone the right to unfettered expression via commercial platforms such as Twitter, Facebook, etc.

That’s not to say that I don’t have concerns about how companies and commercial bodies regulate expression within their realms. They often do it badly, enabling abuse, or reinforcing existing inequalities. Some entities have sufficient control in how business is done in their industries that their suppression of speech can have adverse affects similar to government censorship.

I also have complicated and unresolved feelings about the use of boycotts. Particularly this is true when it’s a call to boycott a vendor who provides a lot of things to the general public. I think even the most offensive material, should be available for people to learn from, to understand, to be aware of, should they choose to investigate. I think it’s possible in certain contexts to sell something without endorsing it.

Furthermore, I believe that the right to free speech must be balanced with the right to choose with whom we spend our time and what we do with our bodies. This doesn’t make me anti-free speech. This makes me pro-free speech, pro-free association, and pro-bodily autonomy. None of those rights make sense on their own without the others. If your right to free speech means that I have to listen to you no matter what, my right to free association and bodily autonomy is diminished.

Prior to this year’s LambdaConf debate I never thought there was a real debate about this. I’ve been around open source for a while, so I’m familiar with the idea that some think any restriction on speech, regardless of context, is censorship.

What’s new to me is that there are folks who think any mechanism which enables someone not to listen to you violates free speech and is censorship. By this logic, if Twitter were a noisy cafe, then putting on your headphones so you could concentrate would be censorship. And, by this logic, any tool that allow others to reduce their exposure to harassment (blocking, muting, etc.) is censorship. It’s even more egregious censorship if these tools are provided at scale.

Furthermore, some of these folks seem to think ‘free speech’ is an exemption from civil behavior and social norms.

I’m still processing what this means. If nothing else, it helps me understand better what’s going on when we debate about this stuff. Being weary of centralization and government interference is different from being weary of systems which empower individuals to choose what they consume and how they communicate with each other online.

I firmly believe we can create tools and systems which mitigate harassment and abuse while enabling free expression and exchange of information based on choice and autonomy. I believe we can design these systems in a ways that preserve anonymity when desired and that don’t necessarily further government surveillance and intrusion.

But we probably aren’t going to build these systems with the help of folks who are fully committed to prioritizing free speech at the expense of free association and bodily autonomy.

 

 

A product development epiphany

At the beginning of the year, I announced Authentic Engine, my new consultancy.

I’m going to share a rather honest update about how things are going, and I plan to do so with some regularity. This gives me some trepidation. Visibility nearly always makes me feel vulnerable, even if the attention is subjectively positive. And general business wisdom favors secrecy and possessiveness above transparency and sharing. Instead, I’m making a deliberate choice to muster courage to favor intimacy (which requires vulnerability) and community.

Perhaps some of you reading are potential clients or customers and in reading about my uncertainties and missteps you’ll decide not to work with me. I’m okay with that, for if my writing here turns you off then we were never going to be a good match. What’s gained, I believe, will be far greater than whatever’s lost. Some will be enriched reading about my experience. Others will appreciate my candor and that I’m willing to do the scary work of making mistakes, sharing those mistakes, and trying again. Some of you will be motivated to work with me explicitly because of this writing.

So here goes it!

A raison d’etre

While I have a great deal of open source tech experience (my first tech job was in 1997 at UC Davis’ network operations center), I have almost no experience consulting in the way I’ve set out to do now. I’ve worked at start-ups and well-established companies. I’ve started and ran non-profits. I ran my own programming shop under CK Web Development. All those experiences have been immensely valuable, but my experience with them has limited applicability to what I want to build now.

I’ve gravitated towards consulting because I want to apply my experience as a programmer, a technical project manager, a developer advocate / technical evangelist, a non-profit manager, a small business person, and an open source community organizer to help folks working in those fields have better work experiences.

By better work experiences, I mean a few things:

  • authenticity, presence, mindfulness, and empathy in day-to-day work;
  • empowered, adaptive leadership;
  • effective navigation of organizational life; and,
  • the building of meaningful and fulfilling careers without burning out.

This isn’t just about making people feel better at work. Whatever your current role in tech, these are practices which will make you better at your job. The production of software, the governing of open source projects, the evangelizing of technology all requires building and maintaining relationships with people. The better your people skills, the better you can effectively apply your technical skills.

By focusing my work on individuals, I intend to contribute to a cumulative effect of improving our technical and open source communities. The more the folks that make up our communities practice and cultivate the above “people” skills, the more inclusive, sustainable, resilient, and commons-serving they will be.

So that’s my raison d’etre for starting Authentic Engine, but how am I actually going to make a living?

What appeals to me about “consulting”

Before I announced Authentic Engine, I did a lot of reading on what it mean to be a consultant and how to run a consulting practice. Some things I read resonated with me deeply: consulting as a calling, humble inquiry, being a trusted adviser, process consultation and the helping relationship. Other things I couldn’t connect with at all, especially the nature of the marketing and sales processes many consulting business books (such as this one) espouse. I don’t want to have a ton of annoying pop-ups soliciting your email for watered-down content, or resell someone’s proprietary organizational trait assessment. Moreover, many books are scant on details about how to actually develop customers if you don’t already have a base established.

Finally it occurred to me to start thinking in terms of answering the question, “what is my product?” The consulting books say you are your product. Okay, I get that to an extent. But it doesn’t feel very sustainable or scalable to me. Nor does it have the depth of experience I want Authentic Engine to bring to others. I want many teachers, and coaches, and trusted advisers, with all their varied histories, experiences and backgrounds to inform and create what we bring to others.

If I am not my product, what is?

So now that I’ve rejected that I am my product, I have to find out what is. That has me researching and learning all about product development. Working in technology as a programmer, project manager, and developer advocate means that I’ve been on the periphery of product management and development most of my career. Now I’m embracing the role officially, which means learning and practicing a new way of thinking.

I’m in the middle of Steve Blank’s The Four Steps to the Epiphany, which focuses on a kind of lean product development for startups that Blank calls Customer Development. So far the framework is making a lot of sense to me. It centers upon developing a deep understanding of your customer and their needs and using that to inform your product development.

Learning deeply from my customers

So that’s what I’ll be doing next. I’ll be thinking about my customers — starting with folks who hold the jobs I’ve had before: technical evangelists (aka developer advocates) and their managers, software developer and their managers, technical product marketing managers (which I’ve worked for, producing events), and open source project stewards.

Actually I’ll be doing more than just thinking about them. I’ll be reaching out to as many of them–as many of you–as possible to listen and learn what their pain points are, what those pain points are costing them, and how they’d like to solve them. I’ll use this information to validate and improve my hypotheses about the Authentic Engine products that will help them.

Exciting and scary

I’m finding this approach very exciting. I love connecting with people, learning about them, building connections, and figuring out ways . But it’s also terrifying because I know this will take time and our financial reserves are dwindling. While I’m developing my customers and products, I’m also going to have to find income, either through consulting or contracting.

How you can help

So, if you or someone you know is needing help with planning a community-focused technical event (like Open Source Bridge, which I co-chaired for 5 years), or managing a technical project (like creating a localization platform for Firefox OS App developers, which I did at Mozilla), or revitalizing a community contributor platform (like I did with MozillaWiki), or improving open source governance (Stumptown Syndicate), or anything else about which I have subject area expertise, please get in touch or book a meeting.

A promise to continue sharing and connecting

Meanwhile, I’ll keep sharing here what I’m working on and what I’m learning.

Soon I’ll post about:

  • realizing I need to be talking to lots more people and how I’m following through on that despite how weird and shameful it can feel;
  • what I’ve learned about content-based marketing and how I’m putting it into practice;
  • how Sherri and I are figuring how how to support one another and keep our household functional while we both build new businesses;
  • what I’m doing to manage my workload, replenish my “spoons,” and avoid burn-out; and,
  • learning to be okay with uncertainty, imperfection, and iterative improvement.

Keeping in touch

As I intimated above, I promise to make more, better connections with all of you. And I invite you to do the same. Leave a comment below, send me an email, or a tweet, or give me a call (go to authenticengine.com for phone number).

The complex reality of adopting a meaningful code of conduct

A number of prominent, globally distributed open source projects are debating the adoption of a Code of Conduct: Ruby, PHP (rfc, discussion), WordPress. There are probably others. Additionally, thousands of smaller projects have adopted codes of conduct as have many conferences.

Why are some communities able to quickly and effortlessly adopt a code of conduct while others become mired in conflict and division whenever the topic arises?

In this post I explore what I see as the main reasons we experience conflict when talking about adopting codes of conduct in our communities:

  1. misalignment of perceived shared values
  2. the relative difficulty of facilitating organizational change
  3. lack of governance infrastructure and non-technical leadership

What I hope people take away from this post is a greater appreciation for the potential complexities of FOSS communities meaningfully adopting a code of conduct and some ideas for confronting these challenges constructively, including:

  1. closing the non-technical leadership gap in our communities
  2. embracing multiple viewpoints and integrating conflict productively
  3. employing compassion and unconditional positive regard whenever possible

Some Background

Why the current momentum around adopting CoCs?

Many FLOSS communities have been around for years, yet the push for codes of conduct is relatively recent, picking up steam about 5 years ago.

What is the reason for this momentum?

First, let’s examine what a code of conduct is for. The purpose of a code of conduct is to make explicit the agreed upon social norms of group interaction within the community. There are many ways to accomplish this, but generally a code of conduct should document:

  1. expected behavior;
  2. unacceptable (transgressive) behavior,
  3. a mechanism for reporting problematic conduct and grievances, and
  4. consequences for unacceptable behavior.

All communities already have an implicit set of social norms. This is important, so I’m going to repeat: All communities already have a set of norms that govern group interactions, whether they are written or not. What a code of conduct does is make a subset of those norms explicit by putting them in writing where everyone can see them.

Over the last few years there has been a significant push for tech communities to adopt codes of conduct that make explicit a specific set of norms that strive to make these communities more equitable, welcoming, and safer for individuals from groups generally underrepresented in tech. This includes explicitly defining the destructive behavior that those from underrepresented groups are disproportionately subjected to and specifically labeling that behavior as unacceptable. The reason this is important is that the opposite standard of behavior generally remains the status quo across our communities. We may want to believe that harassment and other problematic behavior rooted in racism, classism, homophobia, sexism, transphobia, misogyny, ableism, etc., does not happen in our communities because we have never personally witnessed or been subject to it, but it happens nevertheless.

Making explicit a set of social norms that so clearly strives to re-balance power dynamics is one reason why some have such a vehement reaction against adopting a code of conduct. Those previously in the dominant social group will, on average, have to give up some of their power. Everyone will have to learn new ways of interacting. Change is scary.

Let’s not forget intersectionality

Before I continue, I want to make the intersectionality of my approach clear. Those who generally belong to the group with the dominant social power can suffer abuse and injustice too. Everyone in a community needs to abide by the community norms. Everyone is deserving of compassion and unconditional positive regard. And, many who, because of their relative privilege, are not accustomed to doing so will have to yield power, tolerate some loss, and stretch their emotional muscles further than they have generally been required to do. Some will have to do this while learning how to heal from their own experiences of abuse and injustice.

Two Personal Examples

Case Study 1: Citizen Code of Conduct

We wrote the Citizen Code of Conduct for use at Open Source Bridge in early 2011. We’ve modified it a few times since then and now use it for all Stumptown Syndicate events. Starting with text from a sample conference anti-harassment policy written by members of the Geek Feminism community, we modified and added text as needed to represent and embody the values of our community.  For example, for us, it was important to include not just a list of prohibitions, but also to set positive expectations for community interaction.

We’ve fielded a handful of reports or otherwise acted on our our code of conduct since we adopted it in 2011. None of them matched what I had imagined. Often they were more mundane yet more complicated to respond to than I had anticipated. Adopting our code of conduct did not stir up controversy, though at least one of our responses did. Generally the feedback we’ve received is that our code of conduct makes the conference more welcoming to underrepresented groups and this has been reflected in our changing demographics (more women, more PoC, more queer folks). A small number of people have expressed their discomfort or have stayed away entirely.

We’re a relatively small, contained community. In a given year, about 500-800 people are involved in Syndicate events and we operate almost exclusively in Portland. This has made adopting a code of conduct and responding to it a relatively manageable thing.

But we still have missing stairs. We have a mechanism in place for responding to code of conduct reports, but it’s almost entirely implicit. That works in the short-term, but it’s not scaleable and doesn’t ensure stability and adaptability over time. Communities need ways to transfer critical institutional operating knowledge as new leaders come aboard. Stumptown Syndicate just elected six new board members and Open Source Bridge is looking for new co-chairs, so we’re figuring out how to do this right now.

Another thing that made adopting and responsibly using a code of conduct possible was the reason Stumptown Syndicate was founded in the first place. We created Stumptown to be a trusted holding institution for the open source projects we cared most about. From the beginning we were about providing important governance and other structures that help ensure the long-term health of open source communities.

Case Study 2: Mozilla Participation Guidelines

In late 2012, Mozilla adopted their Participation Guidelines after a months-long and highly contentious process that was kicked off by the Planet Mozilla controversy. I was heavily invested in seeing Mozilla adopt a code of conduct. This cost me a lot emotionally — I got a threat from a co-worker (still employed by Mozilla and now a manager) as well as an anonymous death threat. Looking back it almost certainly burned a good deal of my social capital, too.

I suppose all that would have been worth it if I could say now with confidence that the Participation Guidelines have been useful for improving community interactions and improving diversity and inclusion.

But I can’t. (I would love to hear from you if they’ve been useful to you.)

For the most part, the weird, uncomfortable, blocking, and transgressive behavior I encountered while involved with Mozilla wasn’t (and still isn’t) addressed clearly by the Participation Guidelines. And in the few cases where you’d think the Participation Guidelines would be helpful, they weren’t. One involved a co-worker and was addressed via our employee anti-harassment/discrimination policy through HR channels (to a less than satisfactory end, but that’s another story). The others were from anonymous sources and thus weren’t easily actionable.

What are actionable events according to Mozilla’s participation guidelines are by no means clear to me. What are “exclusionary practices” in this context? The guidelines say

“Intentional efforts to exclude people from Mozilla activities are not acceptable and will be dealt with appropriately.”

But “intentional efforts” aren’t defined or exemplified.

And then the guidelines includes this bit, which to me signals a fundamental misunderstanding of how institutional oppression manifests in individual behavior:

“It’s hard to imagine how one might unintentionally do this, but if this happens we will figure out how to make it not happen again.”

It’s not hard for some of us to imagine how others can unintentionally make spaces unwelcoming because it happens all the time.

Most people who engage in behavior that makes others uncomfortable or otherwise transgresses a social norm do not do so intentionally. And these are the people who benefit most from explicit norm setting and compassionate intervention. The group that engages in transgressive behavior intentionally is much smaller and does so for a varied, complex set of reasons, some of which is more easily addressed by community governance than others.

If the participation guidelines are getting invoked more than I realize, I wonder, by what mechanism are issues being resolved?

Early on the guidelines mention “groups for escalation and dispute resolution” but what are these groups? Later on, the guidelines instruct you to do the following if you experience conflict, you’re to engage with:

  1. the person with whom you have conflict,
  2. other “trusted Mozillians,” or
  3. Conductors.

To my knowledge, Conductors is completely self-selected (no training,  qualifications,  vetting, or standard of conduct) and…mostly defunct. Defunct as in many of the people listed are no longer employees and may or may not even still be involved in Mozilla projects.

Furthermore, the values indicated by the Participation Guidelines conflict with my direct experience of the community and its capacities.

This line would seem to indicate Mozilla values differing perspectives and spending energy to surface and integrate these different perspectives:

“Try to understand different perspectives. Our goal should not be to “win” every disagreement or argument. A more productive goal is to be open to ideas that make our own ideas better. “Winning” is when different perspectives make our work richer and stronger.”

However, Mozilla seems to have no functional, consistent mechanism for doing this. Many times during my four years with Mozilla I saw people who voiced differing perspectives ignored or outright silenced. (Example: Those of us who calmly, clearly, and respectfully voiced concerns about migrating to Gmail were invisibly silenced — our managers were instructed privately to make us stop.)

This line would seem to indicate we value working through conflict respectfully:

“Be respectful. We may not always agree, but disagreement is no excuse for poor manners. We will all experience some frustration now and then, but we don’t allow that frustration to turn into a personal attack. A community where people feel uncomfortable or threatened is not a productive one.”

But we have no effective channels for resolving conflict so we vent publicly and in back-channels, or simply stew in silence.

And how does the value of “respect” as alluded to in Mozilla’s Participation Guidelines apply to the co-worker who proselytizes to people in project spaces without their consent? Or the contributors of religious faith who worry that private demonstrations of their faith could be used to expel them from the project? Or the queer employees who wonder why their company remains silent while nearly every other tech company celebrates the SCOTUS ruling on gay marriage?

Who is getting use from this document? Did the protracted and divisive process the Mozilla community endured to get this document create any lasting, useful change? I don’t know what percentage of the community buys-in to the Participation Guidelines or even knows about them. Does the mere existence of the document make some more comfortable knowing they are there even if they’ve never invoked it?

A comparison to frame the issue

Is adopting a code of conduct like adopting a FOSS license?

Reinventing the wheel bugs me a lot. In tech, I think we waste a lot of resources and make a lot of unnecessary blunders not building upon and learning from what has come before. It’s why we’ve put effort into the Citizen Code of Conduct and making it easy for other communities to adapt and use. It’s why I’m glad others have engaged in similar work, like the Contributor Covenant, the Geek Feminism Wiki sample policy, and others.

Let’s look at a similar activity that should be familiar to a lot of FOSS contributors: licensing.

In the same way that adding a license to your project is “easy” — especially now that Github includes a drop-down selection of them during repository creation — it’s also easy to add a code of conduct.

But how many project owners who have added the GPL, or MIT, or any other open source license actually qualified, capable, and willing to enforce these licenses? And how do they determine, if they’re not well-versed in IP law, which license is really best for their project? Or what the long-term ramifications will be of their selection? Project leads already give a lot of their free time to open source development and community organizing; Do they really have more time to learn the skills required and then respond to licensing questions and concerns? Is that a fair request to make of our technical leaders?

Let’s say a project is five, ten, or fifteen years old by the time someone suggests adopting a license. How is consensus achieved across long-term contributors when some of them are very MIT-leaning and others are very-GPL leaning? Not just that, but when some folks already assumed everyone in the project was pro-GPL and are astounded the topic is even up for discussion.

This analogy has its limits, so I ask you not to over analyze it or take it too far. I don’t think it makes sense to have a third-party body doing code of conduct enforcement, for example. I think enforcement needs to stay within communities. Though I do think we need third-party experts providing training — and there’s a handful of us working on that now.

(Though if you do want to explore the idea of copyleft licensing and codes of conduct further, I suggest reading Sumana Harihareswara‘s excellent essay Codes of conduct and the trade-offs of copyleft.)

The licensing analogy demonstrates two main issues related to meaningful code of conduct adoption:

  1. The complexity of selecting and adopting a code of conduct especially in larger, already established and highly distributed projects/communities.
  2. That code of conduct response requires skills and resources many project leaders don’t already have.

Misalignment of shared values and the painful process of re-alignment

The role of shared values in conflict resolution.

Ultimately, a code of conduct is one part of a community’s conflict resolution strategy.

Making a plan for how to resolve conflict is one of the first things any community needs to do, long before any conflict arises. And it starts with reaching agreement and making explicit what your shared values are. Why? Because the values you agree as a community to prioritize in your work together need to drive decision-making about that work. Do we value unconstrained free expression of speech, or do we value inclusion of underrepresented groups? Do we value “free as in beer” or “free as in freedom”? Do we value shared public resources or do we value private ownership? Do we value adherence to using open source software, or do we value promoting the open web? In cases where are shared values fall somewhere in the middle of two extremes, where do we draw the line and which side is most important?

When communities are young, values alignment is usually implicit. This makes sense. A small number of folks get together to work on a project or advance a particular mission. Often times these folks already know each other. They may know intuitively and from past experience what their shared values are and so they don’t think about spending energy to make them explicit.

If the shared values that were implicit when a community forms are not made explicit as the community grows, you end up with divergent thinking about what the shared community values are. People who where there at the beginning have one thing in mind. People who joined at different points in time think another thing, based on what attracted them to the project and their own experience of interacting with the project/community. It’s really easy to assume the shared values of our FLOSS communities are what we want them to be. Because why else would we be there?

Misalignment of perceived shared values is at the heart of conflict

I see this misalignment of perceived shared values to be at the heart of numerous conflicts in open source communities. I see it pop up in every discussion about whether and which code of conduct to adopt. I saw it over and over again at Mozilla (migrating to Gmail, supporting EME, integrating Pocket, appointing Brendan as CEO, how community is included, etc.). I noticed it each time we struggled with a decision around Open Source Bridge / Stumptown Syndicate, which is why we spent a whole weekend last year debating what our shared principles are and then made them explicit and public.

Adding a code of conduct often requires the painful process of values re-alignment

So, one reason it is so much more complex to add a code of conduct to long-standing projects is because a code of conduct is an operating document born from a community’s shared values and in many of our large open source communities we do not have an agreed upon set of shared values. Adopting a code of conduct is a forcing function that brings that values misalignment out in to the open.

Values re-alignment requires organizational change capacity beyond what we have

To realign on values, a community needs to go through the process of agreeing what its shared values should be (not are — because people have been operating under mixed assumptions) and then put them in writing and make them public. This will inevitably require some change in the community. It’s likely that everyone will have to shift their position at least a bit. If not, some will need to leave the community. The larger and more complex the community, the longer this change process will take and the more facilitation it will require.

Organizational change is extremely difficult and most FOSS communities do not have the capacity and experience to manage the change. For the change to be navigated successfully, great care needs to be applied and community leaders, as well as everyone participating in the discussion really need to step up their game. There needs to be room for anger, for disagreement, for being weary of change, for being ravenously hungry for change. People have to be willing to change their minds. There need to be mechanisms for dealing with the inevitable flood of disruptive outsiders and blocking/sabotaging insiders. Leaders need to be able to hold the environment well enough to drive change but not let their communities implode. Everyone needs to understand and be patient with the reality that the process will take time. Months, not days or weeks.

Adopting a code of conduct in larger communities is not a technical problem with clear, templated solutions. Rather, it is an adaptive one that requires the community learn how to change itself.

The role of governance

Having a code of conduct is part of having a conflict resolution strategy, which is, in turn, essential to good governance.

What do I mean by governance? Governance includes everything about how a project carries out its work and engages with its community.

Key governance questions are:

  • how are decisions made and communicated?
  • how is conflict resolved?
  • how are conflicts of interest handled?
  • who are our leaders, and how are they selected, evaluated, and held accountable?
  • how is community membership decided?
  • how are tasks tracked and delegated?
  • what is our mission, vision and how are we planning to achieve that?
  • what are our values and are we living up to them?
  • what is our reputation within our own community and outside of it?
  • do ensure we have the resources we need to carry out our mission both in the short-term and the long-term?
  • how are we developing our community members and leaders?
  • are we serving our mission or just ourselves?
  • are we in compliance with with local laws? are we up to date on our taxes?
  • are we doing what our community needs us to do?
  • are we hearing all the feedback we need to be? from whom we need to hear it?

Having mechanisms that answer these questions in an explicit, transparent way is key for long-term sustainability and success.

In the same way that shared values are implicit when communities first form, so are the ways in which community members work together. And so must they be made explicit as a community grows. Many communities have not made these governance structures explicit. Or if they have, they are incomplete, out of date or simply don’t match how the community actually operates.

In some situations, generally when communities are small and relatively contained (e.g. to a geographic locality like a conference), it works to start with a code of conduct and make explicit the other governance structures as you go. But that becomes much more difficult the larger and more complex a community becomes.

In the same way that a large community will have misalignment in their perceived shared values, they will also have misalignment about how they think the community works together. Re-aligning is a process that takes time and care.

Does a community need to have all the above questions answered and documented before adopting a code of conduct?

No. But any community, especially large and highly distributed ones need to answer these governance-related questions when figuring out how to meaningfully adopt a code of conduct:

  • Who’s going to be responsible for holding the community accountable to the standards documented in the code of conduct?
  • How will those people be chosen, what training & resources will they need, and by what mechanism will they be held accountable?
  • How will code of conduct reports be collected? Will their resolution be communicated to the larger community? How?
  • What are the legal/privacy/etc. implications of our chosen method of reporting?
  • Have we selected a code of conduct that accurately reflects our community’s shared values, as they are actually practiced?
  • Have we selected a code of conduct we are actually willing and able to enforce?

I can’t stress the importance of the last two enough. No community should adopt a code of a conduct they think people want to see if it’s not a true representation of their community’s values and one they are willing and have the capacity to enforce. It’s immensely damaging for a community to have explicit policies that it doesn’t, for whatever reason, live out in practice. Integrity is essential for healthy community.

There’s also a danger is trying to make the perfect plan before acting. This is impossible. Sketch out a reasonable starting policy, practice it for a while and work with your community to adapt as needed over time.

If you’re trying to adopt a code of conduct and members of your community seem overly focused on legalistic analysis or want to negotiate every conceivable yet hypothetical situation before moving forward, it’s a sign that trust in governance and leadership is low.

The role of non-technical leadership

Community leaders, both those with formal authority and those without it, are critical to a community’s ability to successfully navigate change. Those with power in a community need to be on board with the change required be willing to do the work otherwise meaningful change is very unlikely to happen.

It’s entirely possible some of the FOSS communities that we have so much invested in aren’t willing or able to go through the change we need of them in this moment. For our  well-being and to avoid burn out, it’s important we develop the wisdom to be able to identify when that’s the case so we use our energy to build alternative communities.

(I do think there are strategies for bringing leaders on-board when change is needed and they are reticent to productively engage. But discussion of them is out of scope for this post.)

Some strategies for making things better

Close the leadership gap in our FLOSS communities

By now it should be clear that we have a non-technical leadership gap in our FLOSS communities and it’s harming our ability to navigate change and thrive.

To close this gap we need to develop, recognize, support, and elevate non-technical leaders. Non-technical leaders need to be recognized as valuable experts in the same way we recognize technical leaders for their expertise.

Embrace multiple viewpoints and integrate conflict productively

We need to avoid denigrating and silencing those who are reticent of change or raise concerns over adopting a code of conduct. I’m not talking about folks who show up for the sole purpose of trolling, derailing, and abusing. Let’s not increase the damage those people inflict by casting everyone who’s not automatically on board as one of them.

Conflicting points of view are crucial tool for learning. Ronald A. Heifetz in Leadership Without Easy Answers says:

“the mix of values in a society provides multiple vantage points from which to view reality. Conflict and heterogeneity are resources for social learning. Although people may not come to share one another’s values, they may learn vital information that would ordinarily be lost to view without engaging the perspectives of those who challenge them”

Respond compassionately to people’s sense of loss brought about by change

Some may respond with concern or be against a code of conduct because of the loss of power and privilege it represents to them (whether they are conscious of this or not). While we may feel that these folks have long enjoyed what they did not rightfully earn and that a redistribution of power as might occur with a code of conduct is fair, just, and necessary, we still need to respond to their sense of loss with compassion, not derision. Some will never adjust to the changing world in which they live, but many are capable of doing so and will, but they need help doing so.

Good governance and leadership prevent abuses of power

There are a handful of folks (example) who keep playing the dog whistle of impending fascism in response to their community’s proposal of adopting a code of conduct. This assertion simply has no merit and serves mostly to distract and derail. It also serves as a rallying call to those who feel disenfranchised by the changing social, political and economic environment.

Fascism and other abuses of power can happen with or without a code of conduct. Good governance, of which an explicit, written code of conduct is a part, is the antidote to abuses of power. In fact, I believe that keeping the ways your community works implicit and unwritten is more fertile ground for abuses of power than having written, communally available policies you actually follow.

Well-written codes of conduct are enforceable yet flexible enough to adapt to a community’s changing needs and circumstances over time. They are one mechanism by which conflicts can be resolved judiciously (as in, with good judgement, not related to a court of law) and consistently. Without a code of conduct or similar operating procedure, conflicts still arise but a community has no way to consistently approach or resolve them.

Some words of caution: A code of conduct won’t enable abuse of power where previously none was possible, but it might shift the the targets of that abuse or create unintended ones. It may also provide a false sense of security if there are not robust mechanisms in place to receive and respond fairly to reports. This is why a code of conduct needs to be backed up by skilled, ethical leadership and good governance infrastructure.

One last thought about compassion, unconditional positive regard, and emotional labor

Several times in this post I’ve stated we should show everyone compassion and unconditional positive regard.

When I say that, I’m not asking every individual and every given time to show compassion and unconditional positive regard for everyone else, including those who are or have engaged in abusive behavior or are members of social groups that have relatively greater power and privilege. Instead, what I ask is that people cultivate compassion and practice unconditional positive regard as they are able and willing.

Community is a shared burden, as a well as a shared resource. At any given time, some of us will be more capable of and willing to do certain things than others. Sometimes we need to concentrate on our own needs and healing.

Cultivating compassion, understanding, and unconditional positive regard is a communal activity, a communal responsibility. It’s how we’re going to move forward together towards a brighter future.

Announcing Authentic Engine, my Empathy-Based Leadership Development Consultancy

Since I last posted on the subject, I’ve made quite a bit of progress towards starting the next phase of my career. I’m excited to share that with you all now.

One of the things I’ve learned through my 4 years at Mozilla and 8+ organizing in the tech community is that most everyone wants to contribute towards making stronger, healthier, more inclusive and resilient communities. The problem is that they don’t know how and the opportunities to learn and practice the requisite skills are few and far between.

So that’s what I’m going to concentrate on: Providing the leadership learning we need and hunger for.

logo-horizontal-250pxAs such, I’ve founded a consultancy, Authentic Engine, which will focus on providing empathy-based leadership learning and practice opportunities.

There’s a huge emphasis now in our community on improving Diversity & Inclusion, as well as safety and incident response (including Code of Conduct adoption and enforcement). These are important skills our community organizers need help developing and honing. Talented folks are leading efforts here and I look forward to working with them.

I have chosen something slightly different, although complementary. I want to help develop leaders, stewards, and organizers who cultivate wisdom over time and who will be equipped to adapt to the changing needs, stresses, and resources of their communities. I want to teach people how to lead and steward authentically, utilizing their unique combination of experiences, talents, and interests.

With Authentic Engine, I’ll help people develop into the leaders and stewards our communities need now and into the future.

Here’s what I hope to accomplish this year:

  1. Launch 4 (roughly one per quarter) 3-4 hour workshops, each on a specific leadership/stewardship topic. The first will be a guided exploration of what leadership is, what kind of leader you want to be and how to become that leader. The workshops will likely debut in Portland and I’ll bring them to other cities as there is demand. The workshops will not be free, but they will be affordable and have scholarship slots.
  2. Facilitate an on-going community cohort so that folks engaged in improving themselves as leaders and stewards have a support and peer-learning network.
  3. Provide at least 100 hours of one-on-one coaching and mentoring. Anyone who takes one of my workshops will get some amount of free coaching.
  4. Help 2-3 organizations improve or develop their leadership capacity. This could take many forms from counseling an ad-hoc group of conference organizers about how best to take the next step toward formal incorporation, to helping an open source project create a training program for its contributors, to customizing one of my workshops for in-house presentation. (More ideas further down.)

I’m excited about this work because it will bring greatly needed leadership resources to our communities, helping us all to improve sustainability and resiliency. I’m also thrilled because it will allow me to draw upon and apply the varied skills and experiences I’ve developed over my entire career. This includes:

  • managing technical teams and products
  • running a small business
  • serving on the working board of a trade association (501(c)(6)
  • founding and serving on the working board of a tax-exempt non-profit (501(c)(3)
  • developing software
  • writing, publishing, and podcasting
  • organizing communities around open source projects
  • coaching and teaching others
  • documenting and improving systems & processes
  • designing, organizing, and running participatory learning events
  • managing and responding to crises, including: embezzlement, intimate partner violence, food-borne illness, unexpected budget shortfalls, and more.

This is a huge venture to undertake and I’m excited about it. I hope you’ll be part of making my journey successful.

Here are some ways you can help:

  1. Sign-up for the Authentic Engine mailing list so you’ll be notified when I launch new workshops. Register to attend a workshop (once they are launched).
  2. Let me know which leadership/stewardship topics you’re most interested by completing this interest form.
  3. If we’ve worked together and you found the experience worthwhile, write and send me a testimonial I can post to my website (and/or recommend me via LinkedIn).
  4. Recommend me to folks who could benefit from my leadership development consulting services (see above for the list of things I have experience doing).

Most of all, continue to send me your encouraging words. These mean so much to me and really help carry me through the days that feel completely overwhelming.

Wisdom is lived out in community

This Odd and Wondrous Calling (cover)
I’m in the middle of reading This Odd and Wondrous Calling: The Public and Private Lives of Two Ministers, by Lillian Daniel and Martin B. Copenhaver.

In a chapter called Expertise and Wisdom, Copenhaver provides a definition of wisdom that resonates with me deeply. He says:

Before going further with this, I need to pause to say a word about what I mean by wisdom. It has been called the woolly mammoth of ideas — big, shaggy, and elusive. Philosophers, theologians, and social scientists have all found wisdom notoriously difficult to define. In part, this is because wisdom is more than a single attribute. It is more like a cluster of attributes, including a clear-eyed view of human behavior, coupled with a keen self-understanding; a certain tolerance for ambiguity and what might be called the messiness of life; emotional resiliency; an ability to think clearly in circumstance of conflict or stress; a tendency to approach a crisis as an intriguing puzzle to be solved; an inclination to forgive and move on; humility enough to know that it is not all about you; a gift for seeing how smaller facts fit in within a larger picture; a mix of empathy and detachment; a knack for learning from lifetime experiences; a way of suspending judgement long enough to achieve greater clarity; an ability to act coupled with a willingness to embrace judicious inaction.

A bit later, he continues, explaining the importance of community in cultivating wisdom:

Unlike expertise, wisdom is lived out in community. One can become an expert by solitary study. One could, for instance, become an expert in the mating habits of turtles by reading every published study on the subject and doing one’s own field study. Wisdom, by contrast, is not a solitary activity. Wisdom is formed in the ongoing life of a community and it is exercised in community. One cannot speak of wisdom without reference to human community.

I’m sharing this here because these words speak so eloquently about the nature of the community work I strive to do, and the role I strive to fulfill.