Apple's Open Source efforts with the Darwin project have been a failure. Apple failed to build a community around Darwin in the 7 years since its original release because it was not a corporate direction, but rather a marketing stunt. Culturally, Apple did not and does not understand what it means to be open source or to build a working community.

When Apple started shipping Mac OS X Server, it included some GPL'd projects in addition to many other open source projects. A few engineers working on these projects were able to convince some marketing people and some managers that since they were required to release some of the source anyway, why not be good citizens and release modifications to all of their open source software? From there it was one short marketing leap away from releasing the low level components of Mac OS X as open source, and one large, consistent project to release the entire base of Mac OS X as a standalone OS. During the heyday of the dot com days, open source was good. Everyone wanted a piece of the open source pie, and this was Apple's way of getting it. This was great all the way around, the marketing department wins, the engineers win, and the open source world (appears to) win. However, just throwing some source out there and calling it good isn't enough. Darwin 0.3 was source only and didn't attract much attention. Several Apple engineers worked, largely spending more time and effort that they needed, to build a standalone OS based on the sources. They came up with a build system that was compatible with Apple's internal build system, built up a dependency graph for the projects, and ultimately were able to release an OS. This was something concrete that they could show. This is what Apple is offering the open source community, a new, free operating system.

With something to show for their efforts, one Apple engineer in particular went above and beyond calling attention to and attracting developers for their new operating system. This was great, they now had something to build a community around, they just needed to get the word out. The word definitely got out, and people started arriving to take a look. Darwin was brand new, this was its first release, and it wasn't as polished as the established OS' and certainly wouldn't be mistaken as a corporate product. It isn't easy to take a collection of code that doesn't build, and create an OS out of it, but a small number of Apple engineers did so, often on their own time, and it showed. To install the OS, you needed a Mac with OS 9 to dump a backup image onto a partition. The source required click through registration to acquire, and when you did finally download it, it was impossible to build. However, some people did make it through all the hurdles, understood it was a brand new project, and were excited by the possibility of running a free UNIX based OS on their Mac. Some were just excited to be part of a brand new project, getting in on the ground level. Everyone involved could feel the potential of project. Apple was a large company with lots of engineers, and if they got behind an open source effort, this could be great.

The potential was slow to be realized, however. Darwin 1.2 came, as did Mac OS X Public Beta. With the release of Mac OS X Public Beta, the Darwin community got a hint that Apple doesn't care about Darwin and it's community, but rather the product Mac OS X. More people came to the Darwin lists that were not running Darwin, but Mac OS X Public Beta. Some of the Darwin source started having dependencies on non-open source components, and became increasingly difficult to use on Darwin. Darwin improved slightly between 1.0 and 1.2, but for each step forward, there was a step back. When Darwin 1.3 was release along with Mac OS X 10.0, we found Darwin had lost hardware support that was in Darwin 1.0 and Darwin 1.2. This was highly unusual for an open source project to remove old hardware support. This of course was done to Mac OS X as a business decision, and Darwin got what Mac OS X got. Several drivers became binary only, and we no longer had the complete source to Darwin. This clearly established that Apple as a whole worked on Mac OS X. Darwin got engineering leftovers. However, with Darwin 1.3, x86 became a "supported" platform. Not only that, Darwin/x86 1.3 booted from a CD and installed rather than needing to be restored from a back on OS 9. This opened up possibilities of doing upgrade installs, and potential freedom from our Mac OS X oppressors, by being able to work on a platform Mac OS X didn't have. Additionally, it opened Darwin up to a whole new world of contributors: those without Macs. Clearly, if you had a Mac and were able to run Mac OS X (Darwin and Mac OS X had the same hardware requirements), you'd run Mac OS X. But now, people without Macs could run Darwin, and it gave us hope.

However, there was a problem. Already, over the course of a year, Apple was starting to alienate some of it's users and contributors. Apple was developing a reputation for not taking contributions, particularly not new features or additions. Small bug fixes which were easily reviewed might be accepted back, but medium sized features, such as SysV IPC support, were not getting traction. People were getting upset at spending all of this time working on their project which was ultimately supposed to benefit Apple, and having all of their efforts ignored. None of these contributions got a "sorry, we're not interested" or even an acknowledgment from Apple engineers. These contributions were almost universally met with silence, as if they went to /dev/null.

During this time period, Apple ended up hiring over half of their most active contributors, which amounted to roughly 3-4 people. This drained the contributor pool significantly, and effectively muffled the ones that got hired. Once these contributors became employees, they signed NDA agreements, intellectual property agreements, and were otherwise restrained from much of the unrestricted participation they enjoyed prior to employment. They discovered the reason for the lack of communication, being a culture of secrecy. Even being perceived as someone who might leak information, even accidentally, was a liability, and it was best to just not communicate with those outside Apple, or even outside your own group. Additionally, they discovered the reason why no one wanted to accept large patches: whoever submitted them to the build became personally responsible for the feature, and everyone had more than enough to work on with the features given to them by managers, let alone people not in their management hierarchy. Engineers were rated based on how they met their management's goals, and management's goals did not include anything related to Darwin or an external community.

As time went on, Darwin proper did not improve. It had the same crappy shell script installer, poor terminal support, and poor hardware support on non-Mac OS X supported systems. The major problems of Darwin did not seem to be getting any attention, source started disappearing, first from leaf projects, then from increasingly important projects. The source that was there was getting increasingly difficult to build as "integration" happened and formerly standalone projects started having more and more dependencies, many of which on closed source components. By this time, pretty much anyone that was interested in Darwin enough to overcome these obstacles had either been hired by Apple, or left the community. The Darwin lists were almost entirely populated by people running Mac OS X and had questions about UNIX problems.

For what it was worth, Apple recognized this problem and tried to act to correct it. Some people were hired specifically to create a better Darwin community. For over a year this problem was discussed, solutions were proposed and rejected, and countless meetings were held. All the while, less source was available, and what was there was harder to build. In the end, it was decided an independent organization to promote Darwin and it's development was in order. It was clear Apple its self was incapable of creating and sustaining the community it desired. The end result was the OpenDarwin project, jointly created by Apple and ISC. Apple provided the hardware, ISC provided connectivity and power. Much of OpenDarwin's initial content came from a site called darwinfo.org, which I ran until I became employed by Apple and was asked to remove much of the functionality.

OpenDarwin was off to a confusing start, but it was going to be the solution to all our problems. OpenDarwin started with no clear objective. Was OpenDarwin to repackage Apple's standalone OS and extend it in ways Apple was unwilling to spend the resources to do? Was OpenDarwin to provide the necessary tools and environment to allow people to hack up their Mac OS X systems? Or was OpenDarwin simply there to provide hosting and resources to projects that were built on or around Darwin and Mac OS X? Over time, this became a process of elimination. Providing tools and an environment to allow people to hack up their Mac OS X systems was largely unsustainable since any system we came up with would be blown away by the next Software Update or Dev Tools/XCode Update. Repackaging Apple's standalone OS became increasingly difficult and time consuming. It didn't help that most of OpenDarwin's initial members were the former Darwin contributors now working for Apple on Darwin, and didn't have the time to devote to such projects. Not to mention having just worked on Darwin all day, and wanting a distraction when off the clock. Even just maintaining the infrastructure for hosting projects was burdensome. The OpenDarwin project was able to attract a marginal community, but few in the community had the time or dedication to make any meaningful contributions. Those that did, ended up taking everything upon themselves and quickly getting burnt out. The OpenDarwin Project continues to exist in a state of limbo.

With the creation of the OpenDarwin Project, Apple was able to push open source to the back burner. They were able to check off their open source agenda item with hardware donations instead of cultural shift and true interaction. Source was pushed out when binary forms of Mac OS X were made available to the public, the need to provide buildable source was diminished, since that was OpenDarwin's problem, and the engineers were generally happy to have one less thing to be harassed about.

Just one problem: over the course of several years, the original goal of creating a free standalone OS and a community around it had failed. There was no community, and the free standalone OS had become composed of binary packages and source no one could build. Even worse, instead of acknowledging the problem, Apple and most of the world simply forgot about Darwin.

So, how did it fail? What was the problem? The problem was Darwin was a marketing ploy, and nothing more. Marketing thought that by throwing some source files over the wall, they could increase their market share and revenue. And they were right. You go to almost any technical conference these days, and there is an astonishing number of Mac laptops in the crowd. Prior to Mac OS X, people would have been laughed out of the building for having a Mac, but now, with Mac OS X and the magic of open source, it was suddenly OK or even cool. Managerially and culturally, Apple had no intention of becoming involved with any community outside its own walls. There were some well intentioned engineers and even a few low level managers that were duped by their own marketing department, and that was all. These people had given substantial amounts of their free time under the illusion they were doing something Right, something Good, but in the end were simply used by their own company's marketing department.

Feb. 23 2006