Julia Contribution Organisation: Problems and Solutions

If I am to talk about the contributor experience, I think it makes sense to start with my own: the development of my concerns around the experience, and my conviction that there is improvement to be had. After working in the ecosystem for a while, I had a conversation on Slack around frustrations with styling in strings, and a solution I had in mind. I was encouraged to spin my proposed solution into a pull request (PR), and did so. In early May this year, I started the “Styled Strings” (later renamed to “Annotated Strings”) PR. It was merged half a year later (170 days), but it was a slog to get there. Along the way, I presented a vision for how we could build on this to improve the Julia REPL experience, which was largely well received. This lead to me becoming the maintainer of a new standard library, along with a further 14 PRs to Julia itself from me, all made within a week or so of the “Annotated Strings” PR being merged. While three of those PRs have been merged, the rest are in the “waiting for a response” stage of existence. When a chain of PRs depend on each other and take months to merge this stifles motivation, saps momentum, and makes it harder to get other people involved in the effort. All of the Julia contributors have been fantastic, and I don’t blame anyone in particular for the frustration I’m currently experiencing. The suggestions to make bump messages, raise the PRs in slack, etc. are helpful, but I’ve been left with the impression that if you have a good contribution to Julia, creating a PR is often step 1 in an arduous campaign for merging, and it’s easy for un-championed PRs to fall through the cracks. While I’m sure it’s unintentional, this ongoing experience has left me with the nagging feeling that my contributions to Julia are more tolerated than welcomed.

As a result, I can’t help but feel I’ll sooner stop contributing to Julia because I run out of motivation to go through the contribution process, rather than when I run out of ideas or time.

I didn’t want to make a fuss if I’ve just been unlucky with how my recent contributions have played out though. With our different experiences in the Julia project (being part of the core team, a contributor, a package maintainer, or someone active in the social spaces) I think the parable of four blind people and the elephant applies — we all see different parts of the situation as a whole.

To gain a more complete picture of the situation, and in particular, see if other contributors have experienced similar pain points to me, I’ve had a look through the open PRs. A clear pattern emerges of PRs that seem to have “slipped through the cracks”, collectively painting a picture of inefficiency and prolonged frustration. Examples of these PRs stretch across six years, and involve many individuals (none of whom are the problem in any of the interactions I am about to mention). I hope that going through some of these will help starkly illustrate the broader issues at hand.

Last year, @mcabbot created a nicely formatted ∼40 line PR (#47354) extending Iterators.cycle with justification, examples, docstring updates, and tests. It has does not seem to have received any attention, as there are no comments or any action taken other than tagging it with iteration 😕. One can try bumping PRs, with mixed results. A 2021 PR to the Downloads stdlib (Downloads.jl#130) is still receiving semi-annual bumps, and a 2022 PR adding new rad2deg/deg2rad methods (#45947) has received eight bump comments since March this year.

Small PRs seem particularly prone to being forgotten about. In 2017 a PR adding a new sortperm signature that opened the door to the use of specialised algorithms (#24772) was opened. It was asked why it was needed, a response was given … and that’s it. PRs from this year adding a new SubVector type (#50537), and adding Zen4 support (#50226) also seem to be part of the “burst of interaction then nothing” club, as does a PR from last year (#46209) adding a small optimisation to inline small strings. Sometimes, even small quickly reviewed and approved PRs aren’t merged (#50925). At least these PRs have received some interaction, unlike #48891 and the others with zero interactions beyond tagging.

Limbo is also a potential destination for PRs fixing issues. #49669 is a small fix to nextpow/prevpow which has been reviewed as good to merge in August, after being opened in May, but hasn’t seen any activity since. Last year an inconsistency in logical indexing was picked up (#45869), and after Viral asked Stefan who would be good to review it, and Stefan nominated Matt, nothing has happened. In a similar manner, a Julia 1.6 MacOS threading segfault backport (#47053) has no activity past Viral asking about CI setup.

One set of PRs that were particularly disheartening to come across was #48851, #48859, and #49938. @stahta01 seems to have started a concerted effort to improve the Windows build of Julia in February, making 13 PRs this year. Four were merged, but these three have been open for months. Viral asks if the first one can be merged, and there’s a single point of feedback raised that’s acted on, but the PR isn’t merged. The second PR has zero comments or direction. The third PR is approved by Viral, but not merged. The final interactions in March are the last we’ve seen of @stahta01.

These examples paint a fairly grim picture, but is important to note that they are not representative of the overall situation — the Julia project has merged 5670 PRs from 1373 people to date, with a median merge time of one and a half days.The examples show there is a cohort of PRs that “slip through the cracks”, and the absence of safeguards to prevent this from happening. These figures represent a dynamic and productive community, underlying the potential of targeted improvements to the aspects that are currently falling short.

The absence of safeguards around the contribution process affects more than just whether PRs are given attention. #48059 is an example where two members of the core team discussed the PR, and the next steps to take. Unfortunately, neither the details of the discussion, nor the fact there was a discussion in the first place were recorded in the PR at the time — we know this because after the fact one party asked the other for a reminder. While this is an entirely different kind of issue to the sidelined PRs used as examples previously, it fits into the pattern of a contribution process that mostly works, but lacks safeguards.

It is with this in mind, that I think it is fair to say there are systematic problems with the current contribution process.

Something that quickly became apparent when looking at data on merged PRs to Julia is the very different experience that “core” contributors (those with merge permissions) have to everybody else. The average PR from a core contributor is merged in half the time (14 days instead of 28 days), and 57% of PRs made by core contributors are merged by the PR author themselves.

Representing the most frequent mergers as a graph offers a visual representation of the interactions an relationships between core contributors. It reveals patterns of collaboration and areas of expertise, showcasing how informal structures have evolved.

While this informal arrangement appears to work well for core contributors, it also creates an implicit division in the experience core and non-core contributors have when interacting with the Julia project. This is neither surprising, nor unique to the Julia project. While these patterns are a natural outcome of community evolution, they also underscore the need for more explicit structures. Jo Freeman’s seminal essay The Tyranny of Structurelessness examines exactly this situation, the difficulties inherent in the growth of an (initially) unstructured group, and strategies to mitigate undesired emergent behaviour.

Should you read Freeman’s essay (and I highly recommend doing so), parallels between her experience with feminist movements and the Julia project jump out. She talks about how groups that appear to have no structure may seem open and democratic, but often end up forming invisible hierarchies. We see this in the Julia project, where while having embraced an open-source ethos, certain contributors, especially those who have been around longer or are more active, inadvertently form an informal power structure. As Freeman notes, this has (unintended) insidious effects — such as leading to a lack of transparency in decision-making, and contributing to an uneven playing field for new or less frequent contributors.

The issues of communication and transparency that Freeman highlights are crucial. She points out that in groups with informal structures, the decision-making rules are often only known (in full) to a few. This seems to reflect experiences within the Julia project, where processes like PR reviews and feedback can be opaque, leading to delays and uncertainty. By not having clear formalised communication channels, the project risks alienating contributors who feel left off-side of these critical processes.

Transparency is a particularly thorny aspect of the problem, and is something both the Apache foundation and Rust explicitly acknowledge when talking about project governance. Julia has taken some steps to open up decision making discussions — namely in the form of the open bi-weekly triage meetings, which now occur at alternating times to be accessible to a broader range of time zones. However, even at a Julia conference (such as the recent local JuliaCon Eindhoven) in conversation one finds a number of attendees aren’t aware of the existence of triage meetings.

A central theme in Freeman’s arguments is the need for democratic structuring as a group grows, and explicit delegation of tasks and responsibilities. Implementing more formal structures and processes helps ensure that decision-making is democratic, accountable, and doesn’t rely on informal, unspoken rules — making the process more transparent and equitable.

This is particularly relevant to open source projects such as Julia, and complements a preeminent essay on open source development — Eric Raymond’s The Cathedral and the Bazaar. Raymond advocates for openness in the development process, not less structure but rather a different structure — a more publically transparent, collaborative approach. It is worth noting that Raymond’s work is largely based on observations on Linux kernel development, which is both open and explicitly hierarchical, with maintainers assigned particular areas, and patches emailed to those maintainers, who then pass accepted patches “up the chain” until they are merged into the main tree. Much of the kernel’s success lies in its blend of structured, accountable stewardship with the dynamic energy of a “bazaar-like” collaborative environment.

Despite springing from completely different arenas, Freeman and Raymond’s perspectives together provide a compelling insight on necessary ingredients for a highly successful open-source project. In any community, ensuring everybody has a fair chance to contribute, and that their contributions are valued and considered is vital to maintaining a vibrant and productive environment. This is especially true in open-source projects like Julia, where the community’s strength lies in its diverse and engaged pool of contributors.

Given its 24-year history, the Apache Software Foundation has a rich background in open-source project management. They’ve nurtured projects, like Airflow, Arrow, Cassandra, Hadoop, and Kafka, into well-known tech staples. This track record suggests a wealth of relevant experience we can draw upon.

However, it’s important to temper our expectations. While Apache’s successes offer valuable lessons, it’s equally important to acknowledge that they, like any large organisation or project, grapple with their own challenges. Keeping this in mind helps us extract applicable insights without overlooking the complexity and diversity of different project environments.

Indeed, the Apache foundation has published a page on how the foundation and its projects operate, which makes a number of salient points around effective management. These insights provide an interesting comparison to the way Julia currently operates, offering a means by which we can examine and refine the strategies employed in the Julia project.

For starters, each project is led by a Project Management Committee (PMC), who’s role is not to code but oversight — to safeguard the health of the community. Apache gives legal compliance as the first role of PMCs, but we’re more interested in the second.

The second role of the PMC is to further the long-term development and health of the community as a whole, and to ensure that balanced and wide scale peer review and collaboration takes place. Within the ASF we worry about any community which centers around a few individuals who are working virtually without review. We believe that this is detrimental to quality, stability, and robustness of both code and long-term social structures.

Each Apache project comes under a group who’s primary purpose is to foster an environment conducive to the long-term health of the project and a highly collaborative culture, with rampant peer-review. Encouraging more community members to participate in code reviews not only enhances the quality of the code but also fosters a sense of collective ownership and responsibility. We also note that Apache has taken a leaf from Freeman’s essay, and cares about the democratic accountability of those in positions of authority “Your role at the ASF is one assigned to you personally, and is bestowed on you by your peers. It is not tied to your job or current employer or company”. Distributing authority among many people also helps guard against the formation of power monopolies and shows the project recognises the importance of diverse participation in decision-making processes — when a variety of viewpoints are considered it is harder for a small group to become dominant.

This gives us a sense of how contributions should be received, but how do they actually occur? Examining their contribution process, we see particular care has been taken to consider how it contributes to the environment and culture of a project.

Projects are normally auto governing and driven by the people who volunteer for the job. This is sometimes referred to as “do-ocracy” — power of those who do. This functions well in most cases.

When coordination is required, projects make decisions with a lazy consensus approach: a few positive votes with no negative vote is enough to get going.

The rules require that a PMC member registering a negative vote must include an alternative proposal or a detailed explanation of the reasons for the negative vote.

The community then tries to gather consensus on an alternative proposal that can resolve the issue. In the great majority of cases, the concerns leading to the negative vote can be addressed.

This process is called “consensus gathering” and we consider it a very important indication of a healthy community.

For the most part, this is rather standard: people say they’ll do things and present a contribution, it is reviewed, discussed, and likely incorporated. The details however, are worth taking note of. For starters, the explicit “do-ocracy” focus and need for only a small amount of approval for a decision stops small improvements from being mired in bureaucracy, streamlining the contributor experience. The handling of negative opinions is also crafted to encourage constructive criticism and shape objections into opportunities for improvement rather than roadblocks. Explicitly writing about this process also helps set clear expectations around contribution, which is valuable in its own right. Currently Julia’s CONTRIBUTING.md details the technical process of contributing, but not the social. This leads onto the importance of openness and transparency, another common ground between Freeman’s writing and the Apache structure.

We endeavour to conduct as much discussion in public as possible. This encourages openness, provides a public record, and stimulates the broader community.

Making information widely accessible and keeping discussions public supports an environment where the community members are well-informed, empowered to participate, and mitigates the risks of insular groups forming. Apache’s commitment to public discussions not only aligns with Freeman’s and Raymond’s principles but provides a blueprint for other projects seeking to foster an open and welcoming environment.

The ZeroMQ messaging library is a highly successful open source project started in 2007 by Pieter Hintjens and Martin Sustrik, with a large community and hundreds of projects — such as bindings to 53 programming languages. This is “merely” a library not a programming language, but Pieter Hintjens has dedicated particular effort to contemplating the organisation of open-source projects. He makes many suggestions that could be applied to the Julia project to enhance the approach to community engagement and contributor experience.

In 2015, Pieter Hintjens gave a keynote at DomCode on building open source communities. It’s an excellent talk (well worth the 20 minutes, as is the question time), and can be seen as a companion piece to his blog-post (spun off from an audience question at that talk) on Optimistic Merging, the ZeroMQ C4 “Collective Code Construction Contract”, and his book Social Architecture. This collection of resources provides an insightful view into oft-overlooked social dynamics involved in software development, and measures that can be taken to substantially improve the contributor experience and (largely as a result) project health.

Hintjens’ central thesis is the primacy of social architecture over technical architecture in open-source projects — by carefully and consciously shaping a project’s culture, a good interaction model (social guidelines, contribution process, etc.) will lead to good technical outcomes, and trying to operate in reverse (putting technical details before social ones) is putting the cart before the horse. An open-source project lives and dies by its ability to attract and retain skilled contributors. A project’s culture thus forms the social bedrock upon which technical excellence is built. Focusing on a conducive social environment — characterised by clear guidelines, rapid integration of contributions, and a welcoming atmosphere — technical success naturally follows.

Remember that open source is competitive, it’s like a sport and you want the best people to be in your team. Not best in terms of their insanely smart brains, best in terms of their character, ability to work with others, ability to see problems on a wide scale. If they like your project, and they like your process, and they like the way that you treat them, they will come back. They will do more than just give you patches they will also sell your product, they will talk about it in conferences, they will push it — just from merging a pull request quickly. It’s insane, and the other way around is if you put someone’s pull request on hold and say “yes we’re going to do a code review next month we have our monthly code review cycle and you’re a number 25 in the list”, “we’ll get to you possibly next month, maybe the month after” how do you feel? You feel rejected, with good reason, and it’s right away you’re like “this project sucks”.

Hintjens’ articulation of the value in keeping highly-effective contributors won’t be news to anybody watching the Julia repo, where the value of particular individuals is clear — however Hintjens’ emphatic argument that this concern should drive the contribution model is striking.

The ZeroMQ C4 process is reflection of Hintjens’ belief in the paramount importance of social structure and culture in open-source projects. By making the rights and obligations of each role within the project explicit and formalised, Hintjens neutralises the potential for informal and opaque power structures, and with the removal of unwritten rules reduces the friction when a strangers starts collaborating. The principles of C4 emphasise several critical aspects, which largely seem underdeveloped in Julia’s current processes. Coarsely summarised, these aspects are:

• The prioritisation of community building
• Clear guidelines and protocols
• Role separation, and collective ownership
• Diversity in decision-making
• Encouraging experimentation between stable releases
• Minimising complexity
• Specific requirements for patches and issues, and rapid merging

Reading C4 and contrasting it to Julia’s development model is a thought-provoking experience. There are considerable similarities between C4’s explicit social model and what occurs informally with Julia, yet differences in both the broad strokes and the details. C4’s focus on providing both structure and speed is an overarching aspect that seems particularly valuable.

Interestingly, Hintjens’ writing has substantial overlap with Freeman’s essay and the Apache project guidelines, presenting clear points where Julia could improve its organisation. They collectively underscore the critical role of community and the necessity for transparency in social structures and decision-making processes. The emphasis on distributing authority among contributors, clearly definition of roles, and the importance of a welcoming and inclusive environment are common threads. This convergence of ideas across different sources highlights a shared understanding of what makes open-source communities robust and sustainable.

Rust provides a particularly interesting point of comparison, as it features an RFC process largely absent in Julia’s organisational model. This process features prominently on Rust’s Governance page, along with a clear demarkation of the various different teams within the project (the leadership council, language team, moderation team, etc. along with working groups).

Under Rust’s governance structure, minor changes (rephrasing, refactoring, etc.) and quality improvements (test coverage, speedups, bugfixes) are made directly as PRs, but larger changes (semantic/syntactic changes, new language features, interface changes, etc.) must start off as an RFC. This structure serves several key purposes:

Documentation and visibility

Since RFCs have a particular structure (with sections on: summary, motivation, explanation, rationale, and alternatives), they clearly communicate the proposal in a way that’s visible to the whole community — helping attract a diverse range of viewpoints in considering the RFC

Collaborative (community-lead) design

By having a dedicated place for design efforts, it provides a clear way to gather feedback and refine ideas before significant development effort is invested.

Structured decision-making

Laying out a clear path from an initial idea to incorporation into the Rust language brings clarity and predictability to the process, making it more approachable

The Julia project has played with a somewhat similar idea with Juleps: Julia Enhancement Proposals, however it has only been tried tentatively as an “optional extra step” (taken nine times in total) rather than the integral part of the development process that RFCs are to Rust.

There’s a particular RFC from early 2015 that I want to take a closer look at. RFC#1068 was created in response to a perception that the project was running into limitations from the governance structure at the time. Curiously, there seem to be substantial parallels between the issues mentioned in this RFC and issues facing Julia today.

At a high level, we need to improve:

• Process scalability.
• Stakeholder involvement.
• Clarity/transparency.
• Moderation processes.

A particular focus of this RFC is the problem of scalability, and its commentary reminds me of comments I’ve heard about the “not enough maintainers to review PRs” problem we hear about in Julia.

In some ways, the RFC process is a victim of its own success: as the volume and depth of RFCs has increased, it’s harder for the entire core team to stay educated and involved in every RFC. The shepherding process has helped make sure that RFCs don’t fall through the cracks, but even there it’s been hard for the relatively small number of shepherds to keep up (on top of the other work that they do).

Rust is doing a little better than Julia with the inclusion of a “shepherding process” (someone is assigned to the RFC, to guide it and reach out to stakeholders) to help avoid some of the issues we saw examples of in Julia at the start, however they clearly struggle with the same general problem — insufficient bandwidth to accommodate all the hopeful contributors.

This RFC established the idea of Rust subteams to address this problem. Creating smaller, focused groups on specific areas like language design, tooling, and documentation —then giving them agency to manage those aspects of the project— ensures that specialised groups handle relevant contributions, improving efficiency and focus.

An extra step Julia might want to take is to turn the Juleps process into a more central component of the project, by adopting guidelines for when it is required and how they should be reviewed and acted on.

That RFC also touches on other themes that overlap with Freeman, Apache, and ZeroMQ — such as the need for a diverse team of people holding authority, the need to involve the community in decisions, decision/process formalism, and transparency, but I don’t think the Rust RFC adds anything new to this discussion other than reinforcing the importance of those aspects.

---

There are some other aspects of the Rust project that I think bear briefly mentioning. One is the existence of a “contribution walkthrough” that sets expectations for what the process of contributing looks like, and who is responsible for what.

When PRs are actually made, there are a few other social design elements that encourage a positive experience. The first is making sure that every PR is assigned to an (initial) reviewer, and so someone is accountable for handling the contribution.

When you open a PR on the rust-lang/rust, a bot will assign your PR to a reviewer. If there is a particular Rust team member you are working with, you can request that reviewer by leaving a comment on the thread with r? @reviewer-github-id (e.g. r? @eddyb). If you don’t know who to request, don’t request anyone; the bot will assign someone automatically based on which files you changed.

The automation around PRs goes a bit further than this though, as discussed in Emily Durnham’s talk Life is better with Rust’s community automation. At a glance, we can see some basic positive differences in the Rust repo’s PRs compared to Julia’s current set that are likely in part a result of this PR automation tooling: more than double as many PRs made, Rust actually has less open PRs (by about a third), and the oldest open PR is from 2020 (just one), while Julia has six open PRs from 2015 (not tagged with Potential 2.0, and eleven from 2016). Furthermore, looking at the oldest open PRs in Rust it is clear from the tags why they are still open, with tags like S-waiting-on-author or S-blocked.

Another social design element is the existence of notification-groups, to automatically inform interested parties of changes in areas they care about. A clever aspect to the zero-commitment interest-groups is that they provide a ramp onto becoming a member of a subteam and helping spread the load of PR review.

There is much more that could be said about Rust’s governance model, including a discussion on the problems with it (some of which served as stepping stones on the way to the current design). However, I think it makes sense to focus on the ways in which it works well, and the aspects most relevant to the issues Julia is currently facing.

In addition to other languages and libraries, Julia can also examine current best practices in its ecosystem. SciML often comes up in conversation as an a well-managed project, and in November last year Chris Rackuckas delivered a talk on the management of it at the JuliaHEP workshop.

While the points Chris raises have already been introduced in prior examples, this overlap underscores the value of these principles. In particular, I think it’s worth drawing attention to the “Principle 4” slide:

Principle 4: Your Contributors are your BFFs

Always find the time to respond to and validate your contributors.

• Attempt to respond to a PR within the same day
  • If the feature isn’t quite what you expected, explain the greater vision
  • If the code could be improved, use GitHub suggestions to make the improvements in a teachable way
  • Fix the code yourself the first few times and aim for faster merging
  • Help with rebasing and merge conflicts when you can
  • Don’t aim for perfection: merge and open issues with next steps
• If you trust your tests, doing quick merges will be easier!
• You never know who your next maintainer will be

At the conclusion of her essay, Freeman outlines seven principles essential to democratic structuring, all of which are relevant to varying extents. Of these seven, there are three principles in particular I would like to draw attention to.

1. Delegation of specific authority to specific individuals for specific tasks by democratic procedures. Letting people assume jobs or tasks only by default means they are not dependably done. If people are selected to do a task, preferably after expressing an interest or willingness to do it, they have made a commitment which cannot so easily be ignored.

3. Distribution of authority among as many people as is reasonably possible. This prevents monopoly of power and requires those in positions of authority to consult with many others in the process of exercising it. It also gives many people the opportunity to have responsibility for specific tasks and thereby to learn different skills.

6. Diffusion of information to everyone as frequently as possible. Information is power. Access to information enhances one’s power. When an informal network spreads new ideas and information among themselves outside the group, they are already engaged in the process of forming an opinion — without the group participating. The more one knows about how things work and what is happening, the more [politically] effective one can be.

Considering the problems identified, and the good examples we can draw from, a number of particular steps that seem highly beneficial spring to mind.