Preamble

I joined Ruby Central’s Open Source Committee on October 22nd, 2025, after the GitHub access changes. I was adamant internally and externally from day one about performing a retrospective to try to wrap my head around the full, true picture of what happened and why.

In the pursuit of this task, I’ve spent 20+ hours interviewing and chasing up leads, easily quadrupling that time spent reviewing other artifacts such as chats and raw GitHub access logs. For any fact learned verbally, I’ve cross-referenced it with either another independent (important) account or hard evidence, such as a document or video, etc. This incident involved many people over a rather long time scale, and it was important to detangle how people perceived events from how they actually unfolded. The subject matter is deeply subjective, and multiple failed attempts at writing this doc came as a result of aiming for objectivity, for blameless representation. Therefore, those named in this report are:

• Full-time employees of Ruby Central
• Part-time consultants who were involved in access discussions
• Anyone who made an access change from September 10th-18th, 2025
• Those who have already been publicly identified in the discourse

Volunteer groups, including the Ruby Central Board and the Open Source Software (OSS) Committee, are listed, but their actions are represented as a group. Individual quotes from the OSS Committee are used without direct attribution when they represent a general consensus.

This document attempts to paint an accurate representation of what Ruby Central (staff and members, especially in the OSS Committee) experienced and did in the events leading up to and including GitHub access removal. It is not the only lens to view these events, and it’s not exhaustive, but the hope is that it will provide transparency and hopefully closure.

Summary

Two engineers, André Arko and Samuel Giddins, were working on RV together. Each announced that they were leaving Ruby Central. Ruby Central, in turn, wanted to cleanly offboard them and sever ties with RubyGems.org production access, which was tightly coupled to GitHub access. However, Ruby Central lacked the structural ability to make this change directly (did not have admin controls on the GitHub Business/Enterprise). The resulting process was drawn out and poorly communicated internally and to the general public.

This led to the GitHub access changes between September 10th-18th, 2025, that resulted in the walkout of paid contributors: André Arko (indirect), David Rodríguez (deivid-rodriguez), Ellen Dash (duckinator), Josef Šimánek (simi), Martin Emde (martinemde), and Samuel Giddins (segiddins). A group that refers to itself as the maintainers.

This group asserted that they controlled administrative access to the github.com/rubygems organization via the Business/Enterprise permissions and that Ruby Central, the operator of the RubyGems.org service, does not have a right to those abilities or access. When Ruby Central’s Open Source Director, Marty Haught, gained access to the GitHub Business/Enterprise and would not relinquish this control, they quit in protest.

Incident Lessons

Here are some of the lessons from the timeline below:

Policies and procedures are important: Runbooks and other operational documents, technical and non-technical, were neglected for a long time. Efforts to document and standardize these efforts began shortly before this incident in July/August 2025. At the time of the incident, there were no documented offboarding policies or checklists. There are now.

Many companies have an “Outside Business Activities” declaration process for full-time employees to formally let their employer know when they’re working with a foundation or pursuing an activity with compensation, such as writing a technical book. Ruby Central does not have a policy like this in place, but I’ve suggested we add one.

Distinguish remarks from requests: The initial access loss timing was accidental. This could have been prevented by clearly labeling requests for action.

Access changes should always come with explanations: When an access change event happens, the person affected should always know:

• What access was changed?
• Why?
• What can they do about it going forward? Is it permanent or temporary? Who can they talk to if they have questions or problems?

Ruby Central has reached out to some of the developers who were removed due to inactivity to let them know why. We will reach out to all of them.

Platforms with permissions management, such as GitHub and RubyGems.org, should consider adding an option for users to include messages when access changes occur.

Teams need to know why access is changing: Beyond letting the people affected know, those who work with them need to know. This is important not just for access removals but additions as well.

• Whose access changed?
• What is it now?
• Who should they talk to if they have questions or problems? It’s not always appropriate to share specifics with everyone, but say as much as you can.

Impacts of access changes should be clear: Platforms with Role-Based Access Control (RBAC, such as team membership) and hierarchical access systems (such as a GitHub Business/Enterprise containing many Organizations) make it harder for the user making access changes to know the end result of their change. It is better to place the burden of understanding the impact on the system than on the user.

I suggest that these platforms consider adding the ability to preview the effect of access changes to reduce mistakes. A preview could help ensure that the Principle of Least Privilege (PoLP) is being followed without accidentally removing too much.

Perform sensitive operations “out loud”: The Japanese have a concept translated to “pointing and calling” for avoiding and recognizing mistakes. This can be repurposed for many contexts, such as access changes.

An example could be announcing “I’m going to change access now” in a new Slack thread, then following up with how you plan on changing it (such as posting a screenshot, hovering over the new status) before actually making the change. This gives a log so the person can remember individual actions and when they were taken. This is useful even if the system has a log to separate intentions from actions. Done in front of others, it also gives them time to react if the planned action is not desired. It won’t stop every mistake, but it will stop some and make doing a retro on others easier.

Decouple access from personal identity: Access changes in open source are especially emotional experiences. Beyond losing capabilities, access reduction can affect perceived credit for and earned positions. For example, removing ownership on RubyGems.org has the side effect of removing gem download metrics from a user’s profile page.

Platforms that couple metrics with permissions management should consider ways to extend attribution that are decoupled from direct access. For example, preserving gem downloads for “alumni” of package owners even after they’re removed.

Access shouldn’t gatekeep contributions or getting paid: The way that the maintainers explained financial compensation came from early days of paid work on Bundler and Ruby Gems, where someone would make good contributions, be recognized with commit access, eventually promoted to “maintainer” status, and this would open the door for getting paid for maintenance or operations. The implication is that if commit access is taken away, it is perceived as removing both status and income. This pipeline also introduced an incentive problem where new members gaining status would be perceived as possible income competition.

Public work and funding require public accountability: The business of open source is not just doing the work, but making sure it’s done in a clear and appropriate way. That means more time and care need to be spent on drafting appropriate policies and communicating them internally and externally. It also means that foundations need to be fluent in engaging with their communities and communicating clearly.

Timeline leading up to the Incident

January 13, 2025

• Marty Haught, the Ruby Central Open Source (OSS) Director, created a 2025 Roadmap document with funding ideas, including “Ruby one-line installer” and “Make RubyGems faster” proposals, both of which referenced UV, the Python package manager written in Rust.

January 17, 2025

• André Arko edited the 2025 Roadmap to add detail and change the title to “Ruby one-line installer/manager”.

Ultimately, neither project received funding (no grants or sponsors were found to directly hire for the project through Ruby Central), but the item was still in the foundation’s technical roadmap of projects it was interested in.

March 5, 2025

• Marty Haught introduced policy pages to RubyGems.org, including Acceptable Use Policy (AUP), Copyright Policy, Privacy Notice, and Terms of Service (TOS). This included policies such as how RubyGems.org shared personal information with 3rd parties and what personal information (PII) it collects.

June 3, 2025

• André Arko announced he was leaving Ruby Central with a message to the #general channel:

“After sitting with the DHH news for a few days, I’m sorry to say that I’m not going to be able to attend RailsConf, and I am resigning from my advisory role at Ruby Central.

I plan to continue participating in and advising the RubyGems open source projects, but platforming DHH at RailsConf means I cannot continue as an official member or representative of Ruby Central. […]”

André announced that he is leaving his paid, part-time advisory role with Ruby Central. The OSS Committee read this post as indicating he desired to cut all financial ties with Ruby Central. However, that was not the case. Andre did not leave his secondary on-call rotation, which is also a paid part-time position.

The Ruby Central OSS Committee is a group of volunteers who have oversight of Ruby Central’s OSS budget, as proposed by the Ruby Central OSS Director, Marty Haught. At the time, the committee consisted of Gabi Stefanini, Mike Dalessio, and Ufuk Kayserilioglu (who was also a board member). These members have since moved on, and today the committee is composed of one member, Richard Schneeman, the author of this document.

June 27, 2025

• Ruby Central OSS Committee meeting program updates. Raw notes from the meeting:

• Maintenance budget
  • Cutting back from $22k to $12k per month to extend maintenance budget
    • Eliminates secondary on call ($4k per mo)
    • Cuts back maintenance 50% ($6k)
• RubyGems.org supporter membership
  • $2-5k per year for businesses
  • All goes to maintenance and on call (after admin cut)
  • Criteria
    • Allows easy credit card signup with recurring monthly payments.
    • Fully self-service for managing their subscriptions.
    • Internationally friendly
    • No contract to sign
    • Minimal sign up data collected: name, email, organization, payment info.
    • API so we can programmatically generate a list of members on RubyGems and RubyCentral websites.
• New contractor: […]
  • Former […] - Used to work with [Ruby Central paid bundler maintainer]

In this meeting, they discussed balancing the maintenance budget given a sponsorship gap by eliminating secondary on-call, as well as onboarding a new contractor. They planned to use this new contractor to support initiatives that can bring in more revenue to fund OSS maintenance, as spelled out in the middle bullet.

July 7, 2025

• Ruby Central published a blog post, RubyGems.org Funding Model & A New Path For Community-Led Growth.

This blog post tried to drum up additional funds and diversify sponsors:

“With roughly 110 supporters, we would be able to fully fund our annual goals for operations and maintenance. In addition to our other funding sources, such as corporate sponsors, this level of community funding would enable us to expand beyond maintenance and focus on new features and enhancements that will benefit developers and gem creators.”

July 8, 2025

• First day of “The last RailsConf”.

July 9, 2025

• First commit on Ruby Central “runbooks” (private) repository by the new contractor.

There was no documentation on how to run the RubyGems.org service (a.k.a “runbooks”), so in addition to doing scoped work, the new contractor was also tasked with documenting their onboarding process. This also means there was no documentation on how to revoke someone’s access or offboard them from the RubyGems.org service.

July 10th, 2025

• The last day of “The last RailsConf”.

July 11, 2025

• Ruby Central “RubyGems maintainer offsite” was held in the same city as RailsConf, at a different venue. Travel for Marty Haught, Samuel Giddins (the full-time Security Engineer on staff for Ruby Central), André Arko, and one more were covered by Ruby Central.
• The first commit on RV, a Rust tool for managing Ruby dependencies, was created by André Arko and Samuel Giddins.

July 20, 2025

• The spinel-coop org where the RV project lives got a description:

Spinel maintains the Ruby language packaging ecosystem, and acts as maintainer of last resort for the Ruby ecosystem. Our portfolio includes:

- rv, the ultimate Ruby version manager and gem tool

A Spinel retainer offers organizations the opportunity to ensure the sustainability of their foundational Ruby dependencies, and direct access to the expertise of the maintainers.

If you’re betting your business on a critical open source technology, you

1. want it to be sustainably and predictably maintained; and
2. need occasional access to expertise that would be blisteringly expensive to acquire and retain.

Getting maintainers on retainer solves both problems for a fraction of the cost of a fully-loaded full-time engineer. From the maintainers’ point of view, it’s steady income to keep doing what they do best. It’s a great deal for both sides.

Ruby Central was not aware of RV or either Sam or André’s participation in this new Co-Op at this time. When they learned of it, the structure seemed similar to RubyTogether, which was created in 2015 by André Arko and merged with Ruby Central in 2021. RubyTogether funded bundler and later RubyGems.org codebase maintenance and merged with Ruby Central in October 2021. In the merger, the Ruby Central OSS Director position was created, and André Arko became the first acting director.

While he was only paid part-time to work with Ruby Central, he had intimate knowledge of the foundation roadmap and knew Ruby Central was interested in undertaking a similar project. He did not tell anyone in Ruby Central about this work until it launched later, on August 26, 2025.

July 28, 2025

• André registers Spinel Cooperative Corporation in the state of California.

August 1, 2025

• Shan Cureton, Ruby Central’s Executive Director, had a meeting with the Ruby Central board. Board members are: Ben Greenberg, David Corson-Knowles, Freedom Dumlao, Kinsey Ann Durham, Naijeria Toweett, Ufuk Kayserilioglu, and Valerie Woolard.
• Shan shared that a pledge to fund OSS at Ruby Central for $250,000 was withdrawn. Contributed Systems (Sidekiq) later publicly shared that they withdrew a pledge due to DHH speaking at the last RailsConf.

The forecast from June 27, 2025, included this pledge, meaning an already tight OSS budget got tighter. However, there was already a declared interest in reducing secondary on-call to slow the burn rate.

August 4-22, 2025

• OSS Committee Slack message from Marty in Slack regarding making money from the RubyGems.org server logs:

“André suggested a way that his consultancy could cover the cost of secondary on call by analyzing gem download access logs to provide usage data by companies. Here’s a quick write-up of the proposal in this thread. […]”

The proposal has André acting as a middleman between Ruby Central and a third party who would pay for the logs. It is unclear at that time who that third party was, how they would make money from the logs, or how much money those logs would be worth to them.

A OSS Committee member responds:

“[…] Do we really, truly need a secondary shift? I think I remember Marty saying that in that last few years, the secondary shift has never been escalated to. […] It’s not stated explicitly in André’s message, but my understanding is that he will want to own any derived works based on the HTTP logs. If that’s the case, then we need to make sure we’re comfortable losing control of community PII in a way that may be unpleasantly surprising down the road. […] Let’s please make sure we’re imagining the worst-case scenarios before going much further.”

The cost of secondary on-call is $48,000 USD annually.

Marty responds:

“[…] Legally, we’ll need to investigate this to be clear on what we can do here. It is PII. We do care about GDPR. Our Privacy Policy specifically mentions this data: […] The transparency and ownership points you bring up are the biggest in my mind. In the end, it may not be worth allowing a third party to do this sort of thing. RC may want to provide more visibility in how gems are consumed to publishers but that is separate from this discussion. […] Though I am still keen to [hear] the committee’s thoughts. I don’t think we can proceed with this. Long-term, we are better off not involving André with the operation of the platform. Signing a deal with his consultancy to build a product with PII from the service isn’t worth all the reputational risk we are likely to incur regardless if legal signs off on it. While the short-term gains are attractive when we’re tight on funding, it’s not worth it in many aspects.”

August 18, 2025

• Samuel Giddins gives Ruby Central his two-week notice that he will be terminating his full-time security position.
• André Arko creates an access token named rubygems-github-backup with access to all repos in the github.com/rubygems organization, including private repos. This is the only access token of its kind.

August 25, 2025

• André releases RV, and it is announced to the public. The post mentions Samuel Giddins as a team member.
• André follows up with Marty on his proposal to cover secondary on-call with access to the RubyGems.org logs.

August 26, 2025 18:26 UTC

• OSS Committee Slack message by Marty:

“I’m back and catching up after my mountain trip. I’d love to hear any updates from the committee meeting.

So the other new thing is André’s post about RV. I saw this mentioned in two slack channels. I did not know of this until I read the post, which is disappointing. Looks like Sam is leaving to work on that. Both of which I learned through the post and not directly from either of them. I’d love your thoughts on how we should respond.

I had debated internally if we should make a public announcement about Sam’s departure. Does this push us more in a direction?

I was already planning on accelerating removing André but this seems to put that at a higher priority for when I’m back from Rails World. It would be easier if I had another 2 engineers to help with on call.”

This message is the first recorded artifact that mentions a desire to offboard André. The framing “accelerating removing” also shows that the intent came prior to this message and the RV announcement.

August 26, 2025 (follow-up)

• OSS Committee Slack, a member responds:

“yes, agreed. we can add [a previous RubyGems contributor] to the rotation if [they are] ready for it, and i think we should accelerate the adoption of people from other companies that can do similar work (like [another contributor from a different company]).”

Another member responds:

“I’d cut Sam and André’s ties with the organization as soon as possible, announce the departures, and wish them the best of luck on their next endeavor.”

The conversation quickly turns to operations of the RubyGems.org service, where on-call staffing is a concern. Previously, Samuel Giddins was part of the rotation; with his departure, they would need time to find a replacement.

August 27, 2025

• RV adds a license. It is now dual MIT/Apache-2 open source licensed.

August 29, 2025

• Message from Marty in the OSS Slack

“To follow up on the on call, the team discussed how to handle on call with Sam’s departure. We’ll have André cover that in September. We’re going to put together onboarding and improved documentation in September to train up several folks to be on call ready. [new consultant] and [another developer] has volunteered to be part of that group. I’m looking to get 3-4 total people per time zone block (emea, americas, apac) so we can have a sustainable rotation.”

At this point, the path to offboarding everyone cleanly was uncertain. The new consultant made progress on some runbook documentation, but still did not have server access.

At this time, Marty, as the Ruby Central OSS Director, did not have admin permissions on GitHub. Those permissions are held by Colby Swandale, Hiroshi Shibata, André Arko, Samuel Giddins, and Martin Emde. That means in order to offboard anyone, he needs to ask someone else to make a change.

September 4, 2025

• First day of RailsWorld conference.

There was a meeting at RailsWorld between Rails Core, including David Henimier Hanson (DHH), and Ruby Central representatives at the conference. I’ve interviewed five of the attendees independently. I believe the intent to offboard came from Marty, prior to this meeting on August 26th. Coordinating operators to take over on-call also started on August 26th.

I believe that if this meeting hadn’t happened, some details may have changed, but the outcome would have been the same. However, not all present at the meeting would have known all of those pieces or come to the same conclusion.

Hiroshi was not at that meeting, but spoke to Marty at the conference. Marty did not request GitHub access at RailsWorld.

Overall, RV and André’s involvement was a very popular “hallway track” topic, as it was released so recently. I had not yet joined Ruby Central, but I saw Marty, and I asked him about RV at the conference. Marty confirmed that RV was not a Ruby Central project. He didn’t share any information regarding André’s resignation from Ruby Central or access to RubyGems.org.

September 5, 2025

• Samuel Giddins’ employment notice took effect, and he was no longer employed at Ruby Central.
• Last day of RailsWorld conference.

September 8, 2025

• André follows up with Marty on his proposal to cover secondary on-call with access to the RubyGems.org logs.

September 9, 2025

• Marty talked to Colby Swandale and Deivid Rodriguez about Sam and André leaving Ruby Central and their access permissions.

Both Colby and Deivid were paid part-time by Ruby Central. Colby is a paid part-time contractor who primarily works on the RubyGems.org service and codebase, while Deivid was the number one contributor to Bundler by commits. All of the self-described the maintainers have a financial relationship with Ruby Central in addition to their unpaid volunteer contributions.

Marty reported back to the committee:

“I have a quick update. Both Colby and David are not supportive of pushing André out on the OSS side of RubyGems without his consent. Removing André’s access to the service does not seem to be an issue, so I’m proceeding with that planning.”

A committee member clarified on the “pushing out” framing:

“We are not “pushing André out on the OSS side of RubyGems”. I think that framing is wrong. André can continue to be a maintainer of RubyGems/Bundler as an open source contributor/committer, I have no problems about that. However, him having ownership of the organization and repos is not acceptable for the organization that is ultimately responsible for the security and reliability of those tools. In that sense, we are trying to make sure the repos have better homes in ruby and rubycentral orgs respectively. [September 10, 17:01 UTC out of order, but wanted to mention it in case people stopped reading too soon]

The risk of handling operations in a world where André and Samuel don’t have access to our ops is a risk I am willing to take, considering we can bring in people like [contributor], [another contributor], etc, into the fold if/when necessary. IMO, it is more important to swiftly and publicly cut ties with the folks that have already committed (semi-)publicly that they want to have nothing to do with RC, than worry about incidents.”

The link between GitHub access and production access is not enumerated. As Deivid Rodriguez does not operate the RubyGems.org service, he wouldn’t have known about the link between GitHub access controls and production server admin. Colby is more familiar with the service and would have known.

Another OSS Committee member responds:

“Now that you have already discussed with Colby & David […], I can guarantee you that André & Samuel already know. You should expect and be prepared for retaliation, be it a blog post that might post or that they remove your access from the repos. [September 9, 2025].”

At this point, it was clear to the OSS Committee that even though André and Sam had “left Ruby Central,” they did not want to reduce their permission levels. A sentiment that is consistent with most Ruby Open Source projects, where access is granted and rarely revoked. For example, https://rubygems.org/gems/resque still has the founders of GitHub as gem owners on it, even though they’ve not pushed a new release for a very long time. For a community library, access is usually considered a “reward” and “earned.”

RubyGems.org and other package registries are faced with an increased surface area of supply chain attacks. An extremely public NPM supply chain attack happened at roughly the same time as these access changes were happening (September 15, 2025). While this attack was not known to Ruby Central when the first access change occurred, the attack vector is one they were worried about: A targeted phishing campaign compromised a maintainer’s access tokens. Ruby Central is actively engaged with openssf.org and specifically their Principles for Package Repository Security, and is currently working towards level 3 security maturity.

With that context, Ruby Central (OSS Committee and Director) desired an access model closer to that of a professional organization, like the one I work for at my day job. I hold admin rights of a repository that I maintain, but do not have admin rights over the whole organization. With this strategy, the foundation can audit access and fully own completing offboarding from the RubyGems.org service. This strategy was a change from how things previously worked under other directors. The OSS committee believed that reducing Sam and Andre’s access levels (at all, even temporarily) would be perceived as a demotion that risks retaliation.

The timeline of access changes is below, but to understand them fully, you need to understand what a GitHub Business/Enterprise is.

GitHub Business/Enterprise explanation

Most GitHub repositories live under an organization. For example, github.com/zombocom/rack-timeout is the rack-timeout repository that lives in the zombocom organization. There is another hierarchical level that can contain many organizations, which is known as either a Business or an Enterprise. Most GitHub users aren’t familiar with this level. I’ve been writing Ruby code since 2006, and I’ve never encountered it. It looks like this:

GitHub Business/Enterprise
	└── Organization(s)
	     └── Repositorie(s)

For the RubyGems GitHub Business/Enterprise, it holds a nearly empty bundler organization and the rubygems organization with many codebases:

RubyGems GitHub Business/Enterprise
	└── github.com/bundler [Organization]
	|   └── github.com/bundler/.github
	└── github.com/rubygems [Organization]
	    ├── github.com/rubygems/rubygems.org
	    ├── github.com/rubygems/shipit
	    ├── github.com/rubygems/terraform
	    ├── github.com/rubygems/rubygems-mirror
	    ├── [...]
	    └── github.com/rubygems/bundler-site

The RubyGems Business/Enterprise account was created by Hiroshi Shibata, hsbt, who promoted the then current admins on the RubyGems organization to admins on the enterprise.

Confusingly, GitHub has another product called “GitHub Enterprise,” which is different; that’s a product that allows you to run a self-hosted version of GitHub on your own infrastructure. When “Enterprise” is used, it is in the “Business” context. These changes will show up in access logs with a prefix of business.

In addition to these levels, organizations also have teams as a way to control permissions.

Incident timeline

This section was reconstructed based on audit logs from the enterprise account. The ability to audit access logs was not previously available for the OSS Director and the OSS committee. This was resolved when Marty gained access to the enterprise/business account.

September 9, 2025 (continued)

• Marty requested Hiroshi Shibata hsbt make GitHub access changes on behalf of Ruby Central.

September 10, 2:23:49 UTC by hsbt

• business.remove_admin: indirect, segiddins, martinemde
• business.invite_admin: mghaught

September 10, 2025, 3:32:27 UTC by hsbt

• org.update_member: indirect, segiddins, martinemde before: admin, after: read
• org.update_member: deivid-rodriguez, before: read, after: admin (note that access increased)

André and Samuel were removed from the business, and their organization permissions were downgraded at the request of Ruby Central. Marty (OSS Director) was added to the business.

Hiroshi stated that Martin was granted this permission in 2023 by André, but he didn’t know why at the time. So, he reverted Martin’s Business/Enterprise owner status.

At this time, they could not transfer repositories outside of the organization, but they still had admin access to the RubyGems and Bundler repositories. They also still had the capacity to run RubyGems.org operations (such as deploying the service) through their team access.

Hiroshi believed that Deivid Rodriguez’s access level was too low for the level of work he was performing, and increased his organization’s access to admin.

This removal of indirect, segiddins, and martinemde was characterized externally as “a mistake,” but the only mistake was the timing, which was not properly communicated by Marty to Hiroshi.

September 10, 2025

• Message shared in Slack by Hiroshi with the RubyGems developers:

“I’m reviewing account permissions for rubygems now. I’ve assigned enterprise and org owner roles to only Marty, Colby, Deivid and me. The repository admin roles remains unchanged. Please let me know if you encounter any problems.”

André asked why his permissions were downgraded in a DM, and Hiroshi shares:

“[…] Because you are [leaving] Ruby Central now. Owner can access billing and account control. I would like to align to minimum permission around that.”

September 11, 2025

• OSS Committee Slack, Marty shared an update:

“[…], Colby, David Rodriguez, and hsbt are the owners in RubyGems. hsbt, Colby, and me are the enterprise owners. Team access is still the same but with a quick check André only has member access in the places I’ve checked.”

Committee members asked for clarification on the commit status. Marty responds:

“Let’s discuss this tomorrow. Forcing André out is likely to cause the team to defect. Removing his administrative abilities less so. I can share my thoughts on ways to keep him from blocking changes.”

September 11, 2025

• André follows up with Marty on his proposal to cover secondary on-call with logs.

September 14, 2025

• Martin proposes a Governance RFC via PR https://github.com/rubygems/rfcs/pull/61 aimed at restoring access.

That RFC creates an “owner” definition, described as “A person with enterprise or organization owner permissions on GitHub for RubyGems/Bundler projects,” which would mirror the GitHub enterprise/billing owner. The high-level idea was to leave permissions as they were, but create a voting mechanism for removing permissions outside of “inactivity.”

While they didn’t know all the specifics of what Ruby Central desired (when they first drafted the RFC), they knew it had something to do with limiting access and thought this document would afford them the ability to have an official way to ask for permission/access removal, such that the OSS Director wouldn’t need it directly.

Ruby Central wanted to make access control finer-grained and split RubyGems.org production server access controls out from the rest of the repositories. The top suggestion within the OSS committee was to move bundler/rubygems into the github.com/ruby organization.

September 14, 2025

• Andre stated he would seek someone else to restore his permissions, whether Ruby Central approved it or not.
• Upon hearing this ultimatum, the OSS director was alarmed. He expressed concerns internally about the possibility of not knowing who had access, or being removed in retaliation (or both). Ruby Central was not told who Andre expected to restore his access. This left a lingering doubt on all who held that structural access of whether or not they would assist Andre.
• For lack of a better word, there was a “truce” between the parties.
• Those who had been previously removed were asked not to remove Marty or Hiroshi.

September 15, 2025, 8:59:39 UTC by hsbt

• The ability for enterprise members to delete repositories was turned off. Members cannot delete or transfer repositories in any organization in an enterprise at this time.

September 16, 2025, 1:51:29 UTC by hsbt at the request of Ruby Central

• org.update_member: indirect, martinemde, segiddins before: read, after: admin
• business.invite_admin: indirect, martinemde, segiddins
• business.invite_admin: deivid-rodriguez

All access was returned to indirect, segiddins, and martinemde. Permissions for mghaught, the current OSS Director, and deivid-rodriguez gained on September 10, 2025, remained.

A meeting was scheduled.

September 17, 2025

• There was a Zoom meeting with Marty.
• The meeting was recorded by Ruby Central, a practice consistent with other “maintainer sync” meetings, for the purpose of privately sharing with maintainers who could not be present.
• André and Martin produced a list of thirteen developers who were invited. Five attend, and four speak: André (indirect), Josef (simi), Ellen (duckinator), Martin (martinemde).
• Two of these five had access changed (indirect and martinemde).
• Sam Giddins and Deivid Rodriguez did not attend.
• These four developers make it clear that they reject a Ruby Central employee retaining any access to the github.com/rubygems GitHub organization or enterprise.
• They state their expectation that someone must gain administrative control via performing code contribution, and then, based on an internal selection criterion, only those who already have it will decide whose individual merit deserves administrative access to the github.com/rubygems organization or not.
• They express that they’re unhappy that Ruby Central hired a new contractor instead of offering work to one of them.
• Many alternatives are floated, and the developers acknowledge that private code and controls are intertwined in the public GitHub organization. They suggest that Ruby Central can fork the Ruby Gems repos that are needed for private control and access.
• Overall, the conversation is awkward and strained.

Context not present in the meeting: André and Martin were former acting OSS Directors and had github.com/rubygems organization and enterprise controls at the time. Prior to that, it was held by Evan Phoenix on behalf of Ruby Central until February 28, 2025 when it was removed by André Arko. So, they are not opposed to “Ruby Central” or the OSS Director having that control; they are opposed to someone getting that control without their input or buy-in as current Business/Enterprise admins.

September 17, 2025 (cont.)

• After the public call, André met with Marty one-on-one and stated that he would quietly leave in exchange for a license to access and resell the RubyGems.org logs.

September 17, 2025 (cont.)

• Marty released the video of the developer call to the OSS Committee members.
• The RFC from Martin was shared with the OSS Committee members.
• An OSS Committee member responded to the video privately in Slack:

“This call is mind-boggling to me. The lines between open-source work and paid work are blurry as hell and we need to fix this ASAP. […]”

September 18, 2025

• Ruby Central Board meeting

On September 18th, there was a board meeting where an official decision was made on offboarding Sam and André. They decided to remove production access, including control of the github.com/rubygems organization, and their ability to commit. At the time, Ruby Central lacked legal agreements with all operators regarding their access to production servers and data.

It was thought that after offboarding efforts were finalized, they could work together to disentangle GitHub access. Ruby Central’s plan was to eventually present them with legal “operator agreements,” which would be enough to alleviate Ruby Central’s concerns and restore commit access. Followed by resolving proper homes for all repositories.

Internally in Ruby Central, the decision to remove commit access was met with dissension and debate. GitHub team access and Shipit access were explicitly talked about in the September 18th board session, but the full implication of that access (that it effectively meant that GitHub access and production infrastructure access were connected) was not fully spelled out.

This would mean that those with prior knowledge of those systems believe everyone understood the full links between access changes to github.com/rubygems and the RubyGems.org server, while those who were new to the information wouldn’t have intuitively understood all of the connections. Some felt that this was clearly a line too far, and some argued that their concerns merited an extended loss of access.

Due to the lack of a prior documented onboarding or offboarding procedure (runbooks), there was uncertainty around the minimum acceptable access changes to remove all RubyGems.org production server access.

September 18, 2025, 17:56:19 UTC by mghaught

• business.remove_member: indirect, segiddins, martinemde, deivid-rodriguez

André and Sam are removed from the business again. From prior conversations, there was a worry that others would add their permissions back without warning, so while Martin is not being offboarded, his access is reduced. Deivid did not have business admin access prior to September 10th, and this change was an attempt to revert to that state.

When I came to Ruby Central, I was unfamiliar with the business/enterprise access level. So I did not know, as Marty didn’t, that this action of removing a member here would remove them entirely. This total loss of access included all teams and repositories. This was a mistake. This action cannot be undone. Someone removed from a business must be invited back, and that person must accept.

While Ruby Central intended to remove commit access from André and Samuel temporarily, they did not intend to remove them completely from the business. Now, before any access can be given back, they need to be invited back to the org and accept. Their complete removal from the org was a mistake.

Further, Deivid was not supposed to have his access reduced below the September 10, 2025, levels. This was a mistake.

September 18, 2025 18:04 UTC

• Off-boarding emails sent to André and Sam from Marty.
• Both are similar. Here is the body of one:

“After consultation with the OSS Committee and the Ruby Central board, we have removed your RubyGems.org production access, given your departure from Ruby Central. We’re also pausing the on call rotations while we work through this transition. Please send a prorated invoice for on call services.

I believe there are two remaining service accounts I never transferred, PagerDuty and HelpScout. I’d appreciate it if you’d transfer ownership to my email at your convenience.

I’m deeply grateful for all you’ve done for the RubyGems and our community. I will be meeting with the OSS Committee shortly so I can resolve any open conversations.”

Both emails contained a rationale for the access loss, but it was limited. Neither message acknowledged the GitHub access changes. Changes to commit access should have been listed along with the intention that those changes were intended to be temporary.

A strong concern from the start was that Ruby Central would lose developers beyond the two who were offboarded, due to a walkout. Rather than being direct and clear with actions and their motivations, Ruby Central tried to avoid the impression of conflict in hopes that others of the maintainers would not resign.

No other emails were prepared or sent to the other developers whose access was affected or their peers who still retained access. There were messages in Slack, but this communication was not pre-prepared.

September 18, 2025 18:47 UTC

• Email sent to indirect, martinemde, segiddins, deivid-rodriguez, by Marty:

“I’m terribly sorry about the GitHub removal. I messed up, and I accidentally removed all org access instead of downgrading. This is temporary as I work to fix the permissions structure. Martin’s been helping me with this.

I was forced by the board to take this action due to legal risk to Ruby Central. We’re actively exploring how to move the specific repos (terraform, shipit, GitHub team control) out and adjust team control so that RubyGems, the code base, and GitHub organization can be controlled by least privilege by the agreed governance.

I’ll follow up more on this and engage with the governance rfc in good faith.”

An email was sent apologizing for the mistake of removing developers from the GitHub organization. This apology was for the mechanics of the access removal, but not the intent.

Martin requested that Marty restore his permissions, and he would assist with restoring the rest. Marty inexplicably didn’t see the controls he expected to invite someone to the github.com/rubygems organization. Martin was unable to identify the problem either. While being removed from a business/enterprise is inherited by the organization, being added as an admin is not. The issue was that he had to grant himself organization access, which he had not done.

September 18, 2025, 21:45 UTC

• A message from Marty to Colby asking for help restoring access to Samuel, Martin, and Deivid.

Colby was not directly involved in any of the GitHub removals and received the same communications as other team members. He was asked to help correct the accidental removal from the GitHub Business/Enterprise. Colby is in the AEST (Australia) time zone, so this was very early in his day (7:45 am). This request took some time to respond to.

September 18, 2025, 23:17:32 UTC taken by hsbt

• business.invite_admin: paracycle

A Ruby Central board and OSS Committee member, Ufuk, was granted enterprise and organization access. He was asked to assist with access changes. This occurred roughly five hours after the business.remove_member changes went into effect.

September 18, 2025, 23:19:49 UTC taken by mghaught

• org.update_member: mghaught before: read, after: admin

The permissions issue with the Enterprise/Business was diagnosed and resolved. With this change, Marty could send out invites to the organization. However, communication in this time period between business.remove_member, and now could be described as a conflict cycle where attempts to avoid conflict by one side have the opposite reaction and can perpetuate the cycle. While both sides share some similar goals and objectives, the built-up tension, lack of trust, and unclear communications prevented closure or repair.

Various grievances around how things unfolded would be added. The talks for resolution were counterproductive, and they ultimately pushed both parties further apart.

From this point forward, communication or requests/attempts to “restore all access” are interpreted by Ruby Central as including production access and therefore as attempts to reverse off-boarding measures and effectively take control of the service. From the point of view of those removed, Ruby Central had just kicked them out for a second time; whether the specific mechanics involved were intended or not wasn’t important, or wasn’t even believable, given the earlier hiding of motivations.

September 18, 2025, 23:21:19 UTC taken by colby-swandale

• org.invite_member: martinemde

Colby responded to Marty’s earlier message by inviting Martin back to the organization. He then requested more details from Marty and was asked to pause the task. Martin would have received the invitation at 4:21 pm (16:21) his local time.

September 18, 2025, 23:31:42 UTC taken by paracycle

• team.remove_member(s): rubygems/maintainers

These changes were based on the previously stated risk that someone would add offboarded members back unexpectedly. Other changes were made based on inactivity.

Ellen Dash, duckinator, lost admin access to repositories in the github.com/rubygems organization due to removal from rubygems/maintainers. In addition, four other members were removed from the rubygems/maintainers team with the most recent commits to the github.com/rubygems organization in 2022, 2021, 2018, and 2016 (due to inactivity).

Notably, Josef Šimánek, simi, retained rubygems/maintainers team membership at this time.

September 18, 2025 23:35:15 UTC by hsbt

• org.cancel_invitation: martinemde

September 18, 2025, 23:38:02 UTC by paracycle

• team.remove_member(s): rubygems/infrastructure
• team.remove_member(s): rubygems/rubygems-org
• team.remove_member(s): rubygems/rubygems-org-deployers
• team.remove_member(s): rubygems/security

The rubygems/rubygems-org-deployers team gates access to deploy RubyGems.org via Shipit. The rubygems/rubygems-org team gates access to the RubyGems.org admin panel. Users with admin access to that panel have significant capabilities such as: modifying owners on gems, blocking users, yanking gems, resetting credentials, managing feature flags, running arbitrary SQL queries, and running maintenance tasks.

All of these team-level access changes affected a total of fifteen developers, including duckinator. Aside from duckinator, who had been recently active in 2025, most changes were made due to inactivity. The remaining fourteen developers last contributed to github.com/rubygems in the year:

• 2022 (3 developers)
• 2021 (3 developers)
• 2019 (1 developer)
• 2018 (1 developer)
• 2016 (3 developers)
• 2015 (1 developer)
• Never, zero commits (2 developers)

You can see a list of the github.com/rubygems organization members and their activity levels in Mike McQuaid’s post, RubyGems Contribution Data with Homebrew’s Tooling (September 24, 2025).

September 19th, 2025 1:53:25 UTC by hsbt

• org.invite_member: deivid-rodriguez

While most access changes have been about removal, the access of deivid-rodriguez has been consistently increased. However, removal from an organization cannot be reversed; you must re-invite them, and they must accept. Deivid was invited back again.

September 19, 2025, 5:01:10 UTC by hsbt

• repo.remove_member rubygems/bundler-site

This included one bot and three developers who last contributed to the bundler site in 2023, 2020, and 2016.

September 19, 2025, 8:59:40 UTC by hsbt

• org.remove_member: duckinator

This was the last organization removal via Ruby Central.

At this time, simi retained access, but the rest of the maintainers are no longer in the organization. None of those affected were contacted with an explanation of why the changes had been made.

September 19, 2025

• One of the earliest timestamps of a social post linking to “Ruby Central’s Attack on RubyGems” (originally posted Sep 19, 2025, 05:57 AM CST and updated over time)
• Wayback link to “Ruby Central’s Attack on RubyGems”.

The first public description of the incident is published by duckinator, though they did not yet name a concrete group or adopt the label the maintainers. At the time, Ruby Central had not delivered any description of the situation internally beyond those who played a part in it directly.

Ruby Central found itself in a similar situation to the September 10th, 2025, enterprise/business access changes, where they struggled to explain their motivations without discussing personnel matters. Later that day, Ruby Central would release a blog post that did not directly address the duckinator sequence of events laid out or the “takeover” claims.

A key point in the post said:

“In the near term we will temporarily hold administrative access to these projects […].”

Ruby Central would be unable to return any of the administrative access (or any access) to the maintainers without their participation. At this time, there have been two invitations delivered, and one canceled, which leaves one outstanding (deivid-rodriguez). Zero invitations have been accepted. In addition, Ruby Central would not relinquish administrative access to the github.com/rubygems organization.

In this time period, there were public blog posts by many of the maintainers. There were limited and sporadic private communications. Like before, these conversations were unproductive at best, counterproductive at worst. Ruby Central would not be restoring access controls to their state prior to September 10th, and the maintainers would not accept anything less.

September 23, 2025, 20:28:03 UTC by simi

• org.remove_member: simi

This removal of simi was self-imposed as a protest after continued talks proved to be unproductive. Prior to this time, none of his access had been changed by Ruby Central.

September 24, 2025 1:07:58 UTC by hsbt

• org.cancel_invitation: deivid-rodriguez

Deivid indicated in Slack that he would not be accepting the invitation, so it was canceled. The sentiment inside of Ruby Central was that they had “walked away.” This was later validated by a conversation with one of the maintainers.

“just reflecting a bit, I’m a little surprised you didn’t know that we all walked out on [Ruby Central]. That’s the whole situation. This was “you f[…]d up that bad and you want us to come groveling back to you, no” [January 2, 2026]”.

As of September 24, two of the original community owners retained organization controls: hsbt and colby-swandale. Two added members, the Ruby Central Director of Open Source, mghaught, and a then Ruby Central Board member, paracycle, retained control on behalf of Ruby Central. None of the maintainers had membership in the github.com/rubygems organization nor any outstanding invitations.

Ufuk’s access was later passed to another board member when he left the Ruby Central board.

Conclusion

Some execution failures and mistakes are individual, but the purpose of having a foundation and having an institution is that it can rise above individual limitations and provide robust, fault-tolerant systems. Therefore, these are our mistakes, collectively. And collectively we’ll learn from them, but only if we face what happened, what we meant to do, and where we fell short.

The hope is that by sharing this, we can provide some closure to the community and increase transparency. It’s also been a time to reflect internally and understand deeper issues that led up to this situation. You’ve likely been witness to some effects of this process, even if they seem mundane or unrelated. We have been going through a period of structural change, and that process will continue. It will not happen overnight. We want to face this and learn from it. You’re welcome to judge us by our actions, and we hope you keep calling us in and calling us out when we don’t live up to expectations.

Updates

• March 31, 2026 16:33 UTC - Typo fix: Samuel’s name was spelled incorrectly. It is now corrected to “Samuel Giddins.”

I was listening to “The Nvidia Way” recently, as recommended by “Oxide and Friends” Books in the Box episode. It mentions “The Innovator’s Dilemma” a LOT. So I picked up that audiobook too. I was surprised to find out that the forward (of this edition) of The Innovator’s Dilemma was written by Marc Benioff, CEO of Salesforce (which has owned and operated Heroku since 2011).

I’m still working through the book, but some of the terminology we’re now being thrust into seems to come from the book. A “sustaining” business. https://online.hbs.edu/blog/post/sustaining-vs-disruptive-innovation. In that context, “sustaining” isn’t a bad thing; it’s basically short for “predictable.” If you’re in the business of selling hammers, there are incremental improvements to materials or processes. You can find efficiency through forecasting and prior market knowledge. But it’s not a market where new upstarts are coming in with radically different things and taking over.

When I worked for GE as an intern in their Appliance Park (made refrigerators, etc.) there was an organizational strategy for delivering new products to market (such as French door refrigerators, or meeting new energy-star guidelines). It was called NPI (New Product Introduction), where those engineers got good at optimizing for speed of delivery to the market. And another org called PCTO (Product Cost Take Out), where they would take products already delivered and find ways to increase the margins on them (like using a cheaper compressor in exchange for more expensive insulation if that allowed you to still meet regulation and energy-star targets).

Even in such an old field as home appliances, there’s still work to be done. There are still competitive edges to be found and optimized. To me, that’s an example of a sustaining business.

No GE intern wanted to work in PCTO. It wasn’t “cool” or “glamorous.” But it’s the bulk of the work. It’s how the company stays competitive. It would have been “better” to “design it right the first time,” but that would lead to longer product introduction cycles, which not only means your competitors deliver before you, but it also means they’re learning what works and what doesn’t before you. The PCTO org brought value not just by having a cost-competitive product. It is also what enabled NPI to exist at all.

When I hear Heroku say it is moving to a “sustaining” engineering model, it doesn’t mean features stop. Heck, the first commercial fridge was introduced in 1913, and we’re still finding ways to add bells and whistles, like the water pitcher in the door and quad-door design of my most recent fridge. But those innovations aren’t disruptive; they’re iterative and relatively predictable. Those innovations are only possible because worse is better. i.e., GE figured out what mattered (shipping fast is more important than perfect), but it did it in a way that it didn’t stop there, once it’s shipped, it’s shipped again over and over until the kinks are worked out and the margins are competitive.

Sustaining is Focus and Predictable Growth

In “The Innovator’s Dilemma,” a “sustaining” innovation would be increasing the density of iron on a disk platter to achieve incrementally more storage density or going from one spindle to two. A “disruptive” technology would be the digital camera. It originally produced worse images than film and was very expensive. Kodak didn’t invest in it because it didn’t give their current customers what they needed. By the time digital cameras disrupted the film camera industry, Kodak was too late to make a difference.

In the context of Heroku, a transition to a “sustaining engineering model” means (to me) we’ve got to examine our sacred cows and focus on the most important pieces of the company. Some of this is a continuation of what we were already doing. There’s already been a push for toil reduction and increased automation. In my personal role, I want to move building Ruby binaries to be a completely automated process (right now it is semi-automated), with a balance between speed and automation, and security via checksum validation. This is an engineering investment in engineering. Spending hours now to save minutes for a thing that is predictable and recurring will not only free me up to work on other automations, but it will also reduce interruptions and toil. Previously, this was on the roadmap, but it’s been a lower priority. Other teams have other examples. For example, Next Generation Postgres is already in pilot and still moving ahead.

To me, “sustaining” means “focus.” Focus inward on what we’re doing today and what our customers need today. It might mean missing out on disruptive changes, but focusing on what the “next big thing” could be can cause you to miss out on the incremental progress improvements right in front of your nose.

More doesn’t always mean better. More processes, more ways to track work, and more inboxes to check all slow us down. Less doesn’t always mean worse. Sustaining isn’t just keeping lights on. It’s keeping your product and your customers nourished, sustained, not frozen in time or starved. When Heroku first came out, it was a mind-blowing, disruptive change from how people spun up servers before it. Sustaining means aligning on what not to do, beyond just what we would like to do. We can’t afford to be fat and lazy. We can’t afford to aim for 100% in 100% of everything we do. Frankly, that model wasn’t serving our customers very well.

TLDR: Be clear in your communications what you support and “whose side are you on.” There’s a bullet point list of suggestions at the bottom.

Knot-quite-right help

Before connecting my thoughts back to tech and to “drama”, I’m going to start with a made-up example: Imagine you’re at school and a classmate comes to you and says:

“Hey idiot, tie your shoes.”

What do you do next? Do you say, “Hey, thanks! I appreciate you helping me not trip over myself,” or do you get defensive?

The schoolmate clearly led with an attack: “Hey, idiot.” So the rest of it: “tie your shoes” is likely to also be interpreted as an attack too. Maybe you guess they’ll say “made you look” and get the rest of your classmates to laugh too when you glance down at your shoes. Who knows. It’s not logical, it’s emotional.

Let’s break it down. In any “cause” or “drama,” there might be two sides, there’s at least three actors (bodies): Those involved, the speaker, and the listener.

In this case, the actors are:

• Those involved: You, with Schrodinger’s shoelaces
• The speaker: Classmate
• The listener: You || The rest of class

Note: Here || is a logical “or”. So read this as “You or the rest of the class”.

Every conflict has two sides. Here sides for/against are:

• For taking action: This person is looking out for me, and we have a shared goal
• Against taking action: This person wants to make me look bad for some reason

Now, instead, imagine they said, “Hey, I saw your shoes are untied. I don’t want you to trip over them, get hurt, and delay getting to lunch for everyone.” Do you think you would be more or less likely to check your shoes and take action?

The difference between the two approaches is: Calling out, versus calling in. In the second example, the classmate didn’t just raise awareness of the issue, but also worked to make it clear they were on the side of “not delaying lunch” (which falls on the side of “please tie your shoes”). They didn’t just point out a problem; they worked to ensure the listener could hear their message and take the desired action.

Comment confusion

Recently, someone posted on Reddit, asking, “Why does tool X suck?” The actors involved are:

• Those involved: Authors of the tool || Users of the tool
• The speaker: Person asking the question on Reddit
• The listener: Users of the tool || Community at large || Authors of the tool

It was a genuine question, and kicked off a great discussion. But, it was made without awareness that the authors are active in the sub-reddit and would 100% for sure see the post. Whether the poster meant it this is the conflict inherent in the question:

• For: The author should feel good about making something people use even if it has rough edges.
• Against: The author is bad and should feel bad.

If the intent of “Why does tool X suck?” was to say “I am mad at this tool and want to attack the authors,” then those would be good words to choose. If the intent is to understand the software, identify strengths and weaknesses, and possibly help the authors understand and make it better, the first step is not alienating them.

Three body problem

Now that you understand the for/against and involved/speaker/listener frame, it’s going to get complicated by getting recursive. As everyone involved can switch perspectives based on the flow of the conversation. Like the real “three body problem,” this switching of places of involved/speaker/listener makes real-world conflicts chaotic and unpredictable.

Take the prior case of “why does tool X suck?” After the post is made, someone sees it they comment something like “hey, that’s a really crappy thing to say…” as the original person “calling out” the tool, becomes called out. My suggested fix would be to recursively apply “calling in,” including “calling in” about “calling out.”

It also gets tricky, as a post on Mastodon (or other social media platforms) is read by individuals as if it were written “to” them, but “them” might be a random bystander or the leader of an organization. A statement, “I hate this tool,” might be true, but it might not be helpful if it’s missing the context and qualifiers needed to let people know what exactly you meant by making that statement.

Humans are really good at pattern matching and applying meaning, even when none exists. Others will read your communications and implicitly try to assign an “us” or a “them” side, whether it was meant that way or not.

For more info on this topic I like this PDF on “Calling In”, starting on page 2 for more examples.

Non Violent Comments

All of that is well and good. Here’s some other rapid-fire suggestions for being able to both hear and be heard:

• No dog piles, please: Admitting a weakness is being vulnerable, not an invitation to attack and pile on. Agree with it and help strengthen it. If someone states, “here’s my bias on this issue,” it doesn’t invalidate their words, but helps frame them with context.
• Don’t accept framing at face value, don’t dismiss it either: When people speak, they do so with bias and agenda. There’s always at least 2 sides to any story and normally many more. It’s okay to explore and question alternative viewpoints and interpretations. But no one likes to exclusively talk to “devil’s advocates.” Also, make sure you’re not dismissing lived experiences. Another way to express this is “you can acknowledge you heard someone without agreeing with their words.”
• Stay hydrated, but don’t be thirsty: Being “thirsty” is being desperate for attention. Sometimes a comment thread isn’t about you, or the other person just doesn’t want to talk about what you want to talk about. Try to make conversations give-and-take and not take-and-take-and-take.
• Some people just want to chat: Not everyone has an agenda, or a fully resolved set of viewpoints. Many people are working through how they feel on things and they need a place to talk through it. If this is you, I suggest looking for smaller communities like a friend’s five-person discord instead of the open internet. Even if you’re just “asking questions,” it will look like you’re trying to frame things, and you’ll be pushed into a position. If you are genuinely asking questions, try to be sensitive of the optics and be willing to have people respond defensively.
• Disagree productively: From the r/ruby rules sidebar here are some high level tips:
  • YES: Read comments fully before responding
  • YES: Practice active listening. Let the other person know what you heard.
  • YES: Distinguish acknowledgment from agreement.
  • NO: Willful misrepresentation of someone’s stated position.
  • NO: Sexualized language or imagery
  • NO: Trolling, insulting or derogatory comments, and personal or political attacks.
  • NO: Conduct which could reasonably be considered inappropriate in a professional setting.
• When in doubt, use Non-Violent Communication (NVC): If you’re struggling to say what you have to say without others getting defensive check your communication has the following parts:
  • Observation: What did you experience that led to the comment? This is where you build shared context
  • Emotion: How did that make you feel are you supportive or against that observation? Note that “I feel that you…” or “You are making me feel…” is not an emotion. It’s okay to be mad but also sad or frustrated or curious or a billion other characters that will be in inside out 3
  • Need (general): Is it clear what you want from the world? What are your values? Do you care about justice or safety? Give the reader an anchor to know where you’re coming from
  • Request (to the reader): Is it clear what you want from the person you’re speaking to? It could be “to hear me.” If it’s not clear, people often default to thinking you want them to change their position or opinion, which, if you didn’t know, is … not common on the internet, therefore people fight back. Also note that a request is not a demand.

The above NVC is reworded as a bit of a checklist of questions, because if you formulaically apply it like an SAT essay, then it will sound robotic and condescending. Also, a big part of the real NVC practice is working to actively defuse perceived slights and perceived attacks from the speaker. NVC is not a magic bullet, and people can use it to harm. But I like thinking of it as structural de-composition. Every communication you send has all four parts in it. If something you say is missing a part, you’re asking the reader to do that work for you and fill in the blank. Not only is this a bit rude, but there’s also no guarantee they’ll do it correctly.

This section got longer than I was anticipating, but please don’t call me out on it. Instead, call me in.

Update: you can also use this technique to disallow unwrap()! There’s also unwrap_used which you use by adding #![deny(clippy::unwrap_used)] to your main.rs.

std lib enhancer

I use the fs_err crate in my projects, which provides the same filesystem API as std::fs but with one crucial difference: error messages it produces have the name of the file you’re trying to modify. Recently, while I was skimming the issues, someone mentioned using clippy.toml to deny std::fs usage. I thought the idea was neat, so I tried it in my projects, and it worked like a charm. With this in the clippy.toml file:

disallowed-methods = [
    # Use fs_err functions, so the filename is available in the error message
    { path = "std::fs::canonicalize", replacement = "fs_err::canonicalize" },
    { path = "std::fs::copy", replacement = "fs_err::copy" },
    { path = "std::fs::create_dir", replacement = "fs_err::create_dir" },
    # ...
]

Someone running cargo clippy will get an error:

$ cargo clippy
    Checking jruby_executable v0.0.0 (/Users/rschneeman/Documents/projects/work/docker-heroku-ruby-builder/jruby_executable)
    Checking shared v0.0.0 (/Users/rschneeman/Documents/projects/work/docker-heroku-ruby-builder/shared)
warning: use of a disallowed method `std::fs::canonicalize`
   --> ruby_executable/src/bin/ruby_build.rs:169:9
    |
169 |         std::fs::canonicalize(Path::new("."))?;
    |         ^^^^^^^^^^^^^^^^^^^^^ help: use: `fs_err::canonicalize`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.91.0/index.html#disallowed_methods
    = note: `#[warn(clippy::disallowed_methods)]` on by default

Running cargo clippy --fix will now automatically update the code. Neat!

CWD protector

Why was I skimming issues in the first place? I suggested adding a feature to allow enhancing errors with debugging information, so instead of:

failed to open file `file.txt`: The system cannot find the file specified. (os error 2)

The message could contain a lot more info:

failed to open file `file.txt`: The system cannot find the file specified. (os error 2)

Path does not exist `file.txt`
- Absolute path `/path/to/dir/file.txt`
- Missing `file.txt` from parent directory:
  `/path/to/dir`
    └── `file.md`
    └── `different.txt`

To implement that functionality, I wrote path_facts, a library that provides facts about your filesystem (for debugging purposes). And since the core value of the library is around producing good-looking output, I wanted snapshot tests that covered all my main branches. This includes content from both relative and absolute paths. A naive implementation might look like this:

let temp = tempfile::tempdir().unwrap();
std::env::set_current_dir(temp.path()).unwrap(); // <= Not thread safe

std::fs::write(Path::new("exists.txt"), "").unwrap();

insta::assert_snapshot!(
    PathFacts::new(path)
        .to_string()
        .replace(&temp.path().canonicalize().unwrap().display().to_string(), "/path/to/directory"),
    @r"
    exists `exists.txt`
     - Absolute: `/path/to/directory/exists.txt`
     - `/path/to/directory`
         └── `exists.txt` file [✅ read, ✅ write, ❌ execute]
    ")

In the above code, the test changes the current working directory to a temp dir where it is then free to make modifications on disk. But, since Rust uses a multi-threaded test runner and std::env::set_current_dir affects the whole process, this approach is not safe ☠️.

There are a lot of different ways to approach the fix, like using cargo-nextest, which executes all tests in their own process (where changing the CWD is safe). Though this doesn’t prevent someone from running cargo test accidentally. There are other crates that use macros to force non-concurrent test execution, but they require you to remember to tag the appropriate tests. I wanted something lightweight that was hard to mess up, so I turned to clippy.toml to fail if anyone used std::env::set_current_dir for any reason:

disallowed-methods = [
    {
        path = "std::env::set_current_dir",
        reason = "Use `crate::test_help::SetCurrentDirTempSafe` to safely set the current directory for tests"
    },
]

Then I wrote a custom type that used a mutex to guarantee that only one test body was executing at a time:

impl<'a> SetCurrentDirTempSafe<'a> {
    pub(crate) fn new() -> Self {
        // let global_lock = ...
        // ...

        #[allow(clippy::disallowed_methods)]
        std::env::set_current_dir(tempdir.path()).unwrap();

You might call my end solution hacky (this hedge statement brought to you by too many years of being ONLINE), but it prevents anyone (including future-me) from writing an accidentally thread-unsafe test:

$ cargo clippy --all-targets --all-features -- --deny warnings
    Checking path_facts v0.2.1 (/Users/rschneeman/Documents/projects/path_facts)
error: use of a disallowed method `std::env::set_current_dir`
   --> src/path_facts.rs:395:9
    |
395 |         std::env::set_current_dir(temp.path()).unwrap();
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: Use `crate::test_help::SetCurrentDirTempSafe` to safely set the current directory for tests
    = help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.91.0/index.html#disallowed_methods
    = note: `-D clippy::disallowed-methods` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::disallowed_methods)]`

clippy.toml

Those are only two quick examples showing how to use clippy.toml to enhance a common API, and how to safeguard against incorrect usage. There’s plenty more you can do with that file, including:

• disallowed-macros
• disallowed-methods
• disallowed-names
• disallowed-types

You wouldn’t want to use this technique of annotating your project with clippy.toml if the thing you’re trying to prevent would be actively malicious for the system if it executes, since clippy.toml rules won’t block your cargo build. You’ll also need to make sure to run cargo clippy --all-targets in your CI so some usage doesn’t accidentally slip through.

And that clippy lint work has paid off, my latest PR to fs_err was merged and deployed in version 3.2.0, and you can use it to speed up your development debugging by turning on the debug feature:

[dev-dependencies]
fs-err = { features = ["debug"] }

Clip cautiously, my friends.

Keep reading in the Heroku blog post.

Keep reading in the heroku blog post.

Let’s set the scene: Our protagonist is burning the midnight oil. They’ve got a great idea for an open source RFC, but they want to do some due-diligence first with some co-workers so they open up their employee encouraged document sharing solution, Slack Canvas and write a beautiful bit of carefully annotated markdown. It looks … mostly right (why on earth do H3s look nearly identical to H2??) but it’s good enough. They got some great feedback, made some edits and now it’s time to go contribute to the great commons we call open source. They select all (CMD+a), they copy the text (CMD+c), they paste it into GitHub and hit enter. Slowly the blood drains from their face. It’s all gone. All the backticks, all the bolds, all the links. It kinda looks like the original document, but is quietly, subtly, horrifically mangled.

In desperation, they reach first for the Edit menu, then File. Flashes of the crappy https://quip.com/ markdown export feature flash through their memory. It had it’s own problems, but at least it was there. Silence covers the room. An almost imperceptible evil laugh of Google doc’s bastardized markdown support can be heard of the dawning realization: the app takes markdown synatax in. But it devours it. Destroys it. Vomiting rich text in in the place of where carefully currated backticks and astericks once shone brightly. The document is dead. Our hero weeps.

The whole point of markdown (as witnessed and confirmed in my headcannon as a developer since 2006) is that it provides a format that can be consumed as EITHER rendered text, OR as plain text. Markdown is not a format for styling a text document, it is a format that requires that the writer, can read the original text (not literally, you pedants, there’s not exactly a spec that says that, but perhaps there should be). Without this guarantee we get a sea of “almost markdown” formats. We are mark-drowning in them. They all behave slightly differently. For apps that do a decently good job of letting developers “read your write” there’s product-driven feature’s like Notion’s desire to embed rich objects such as spreadsheets and to add custom inter-notion document linking syntax that encourage users to subtly poison the portability of their original text.

I don’t begrudge them extending the syntax to meet their needs, but wish an export story was better thought out. Perhaps rich objects should be supported via iframes and exporting documents to markdown should support some kind of a “deep” export (i.e. it doesn’t matter if the markdown it exports is perfect, if it links to a bunch of gated and internal documents. Such thoughts probably get your cyber-security sense tingling (as they should) and likely bore investors and product managers to tears, therefore it’s a usecase that is sorely neglected given the millions/billions of dollors of “innovation” in the “markdown as a shared mangled RTF doc” apps that we’re collectively burried under.

So forget “hard mode” all I’m asking for is simple. If you take markdown in, you should allow me to take my original markdown out CHARACTER FOR CHARACTER. So far the best tools I’ve found for this are: Vim, GitHub, and Obsidian. (Yes, your favorite IDE too). These are all hacker tools, but why can’t we live in a world where ALL tools respect our input enough to let us read it back? Perhaps you’ve been toiling like our hero. These document inflicted wounds aren’t large, but they’re never-ending. They create toil, and burn trust in tools. Perhaps like me, you’ve not had the words to enumerate what exactly is missing. Hopefully, now you do. If a company asks for some feedback, please let them know that to “read my markdown write” is a fundamental and inalienable right.

Special bonus rant: Please add a newline after your markdown headers

Oh and while I’m ranting. I beg of you, please (please, please, please, please) add a newline after markdown headers. I.e. Don’t do this:

## I really don't like when
People do this. Hard to read.

Instead, please do this:

## Everyone who does this is amazing

They are the coolest people I know.

Why? Remember when I said markdown was supposed to optimize for rendering and plain text? When someone is reading your plain text, the first example without the vertical whitespace provides no visual pause. It’s notexactlythesamethingasreadingasentencewithnospaces. But you get how important white space can be when it comes to reading and comprehension speed. Yes, your favorite markdown (and markdown-ish) tools will render both of them the same, but to someone not viewing those docs in that same tool, they will appreciate you and star your github repos more and other vauge promises of things developers presumably want.

For some reason, I don’t entirely comprehend, the first style is much more popular on GitHub. To the point that the vast majority of LLM produced markdown does not have vertical whitespace after headers. We’re in a world now where “do what I do” (as opposed to do as I say) is fed back to us one token at a time. We might be past the point of no-return when it comes self-reinforcing behavior (as AI is now trained on the output that developers are committing as their own), but when skynet takes over and you’re looking for a shibboleth to prove you’re not a robot with an Austrian accent, you might remember this post and send a PR with some glorious, human, hand-crafted markdown.

If I say to a friend, “I’m hungry, let’s go to McDonald’s” (or wherever), they’re not allowed to block me without making a counter-suggestion. They can’t just say “No,” they have to say something like “How about Arby’s” instead. This simple rule changes the dynamic of the suggester/blocker to one of the proposer/counter-proposer. If someone is simply refusing to be involved, they McBlocked me.

In practice, though, it’s hard to always have a suggestion you’re willing to run with, so a relaxed version of the rule is that the other person has to AT LEAST specify why not. Instead of “no” it must be “no, because”. For example, it could be “I had a burger for lunch” or “I’m banned for life after jumping on a table and demanding Szechuan dipping sauce.” This helps show that you’re not just blocking things, you understand the goal and want to move the conversation forward. It gives the other person something to work with. Easy for eats, but what about tech?

I work for Heroku, and recently, there was a stack EOL where customers were asked to migrate off of Ubuntu 20.04 (heroku-20). In this (many-month-long) deprecation process, I saw a lot of people make a lot of absolute statements. One of them was:

“You cannot run Rails 4 on heroku-22.”

Which, as you’ll guess, is only half the story. What they meant was:

“Rails 4.2 saw its last release in 2020 and is quite thoroughly EOL. That version cannot run on any Ruby version 3.1. x- 3.4. x, which are present on heroku-22 or above, due to library errors. Therefore, to run Rails 4 on heroku-22, you would have to fork it and patch the security vulnerabilities yourself and update it to run on a modern Ruby version.”

Which, to be fair, sounds a lot like “cannot be done,” but with more words. But, as you’ll also have likely guessed, once you know about the possible path forwards, however impractical, it might give you other ideas.

You might start asking questions like “if we have to fork and maintain it, anyone else would have to also, I wonder if someone else already did.” This could send you down a quick search where you might discover that Rails LTS is a thing and basically provides a managed fork of Rails 4.2 for a fee that runs with the latest Ruby versions.

I wrote about the existence of this service previously:

• Heroku blog: Migrating Your Ruby Apps to the Latest Stack
• Reddit thread: On using an old Ruby version on a newer stack

Now, that new thing could still be a bad idea, and you might still not end up doing it, but the key here is that you’re not saying “no,” you’re saying “here are the barriers I know about.” A good way to test if you’re just using more words to say “no” or not is if your statement is falsifiable or satisfiable in some way.

A “no, because” statement instead of a plain “no” moves the problem from a blocker into an opportunity. You can see this in a really good open source conversation. Instead of “this can’t be done,” someone can send a PR. Instead of “I won’t merge your PR” they can comment: “I agree/disagree with the problem/opportunity you’ve raised, I’m uncomfortable merging this because of <specific reason>.”

A quick story, and you can go. Before writing this post, I pitched the word-smithing of “McBlocker” to my wife at the dinner table (where you can tell we are very cool and fun people). My kids, age 7 and 9, were there. My 9 y/o asked me to take him to the library after dinner (did I mention how cool we are?), where I was talking to him about types of non-fiction that he might like. I was talking about biographies when he blurted out, “I don’t like biographies.” To which I responded, “Hey, don’t McBlock me,” and when I got a laugh of recognition in return, I figured the phrase was worth a blog post.

If you enjoyed this, you might enjoy my service for helping people contribute to open source (free) or my book How to Open Source (paid). Now, go McRepost this to your favorite federated social network!

TLDR: What’s a duplicate duck?

A “duplicate duck” is a type that implements a subset of traits of a popular type with the same results. In my case I wrote a type, MultiError, that I later realized was identically duck typed to syn::Error and that my struct added nothing. I deleted my type with no loss in functionality and the world was better for it.

I saved my code before throwing it away. The following is the story of my design process and eventual epiphany.

Quick whoami: I write Rust for Heroku where I maintain the Ruby Cloud Native Buildpack. I also maintain a free service CodeTriage and wrote a book, How to Open Source, for turning coders into contributors.

Story version - Context

I’ve been hacking on proc macros recently, you can read about a recent investigation “A Daft proc-macro trick: How to Emit Partial-Code + Errors”. I want proc macro authors to emit as many accumulated errors as possible (versus stopping on the first one), I’m also a fan of unit testing. I wanted to add a return type from my functions that said, “I return many accumulated errors,” and I wanted that return type to be unit-testable.

In my code, I’ve been accumulating errors with VecDeque<syn::Error>. This makes it easy to combine them into a single syn::Error :

if let Some(mut error) = errors.pop_front() {
    for e in errors {
        error.combine(e);
    }
    Some(error)
} else {
    None
}

However, I don’t want to return a result of Result<T, VecDeque<syn::Error>> from my functions as the error state isn’t guaranteed to be non-empty. A good type should make invalid state impossible to represent.

Start with the data

To guarantee my type always had at least one error, I separated out the first error from the rest of the collection. Even if this container is empty, the type definition guarantees we can always turn this into a syn::Error

/// Guaranteed to hold at least one [`syn::Error`]
///
/// The [`syn::Error`] can hold multiple errors
/// through [`syn::Error::combine()`], however it
/// does not allow the receiver to distinguish
/// between the two cases, which makes testing
/// less precise. Using this type is a stronger
/// hint that the function accumulates errors.
///
#[derive(Debug, Clone)]
pub(crate) struct MultiError {
    first: syn::Error,
    rest: VecDeque<syn::Error>,
}

impl MultiError {
    pub(crate) fn from(mut errors: VecDeque<syn::Error>) -> Option<Self> {
        if let Some(first) = errors.pop_front() {
            let rest = errors;
            Some(Self { first, rest })
        } else {
            None
        }
    }
}

Warning: Just because the docs state something, doesn’t mean it’s true.

Note the visibility, by default I use pub(crate) for the struct and associated functions but not for the fields (first and rest). When I’m unsure of my design, it’s easier to change them later if all access goes through functions.

This type allowed me to introduce helper functions like this:

pub(crate) fn parse_attrs<T>(
        attrs: &[syn::Attribute]
    ) -> Result<Vec<T>, MultiError>
where
    T: syn::parse::Parse,
{
    let mut errors = VecDeque::new();
    // ...
    if let Some(error) = MultiError::from(errors) { // <== HERE
        Err(error)
    } else {
        Ok(
        // ...
        )
    }
}

This code says “I take in any slice of syn::Attribute and then parse that attribute into a vector of T or return one or more syn errors”. So far, so good.

But my macro needs a syn::Error to generate error tokens and my function returns a MultiError. So I needed a way to convert my type into a syn::Error.

Add into behavior

Based on the properties of the type, we know we can always convert into a syn::Error infallibly, so I can expose that via implementing Into<syn::Error>:

impl From<MultiError> for syn::Error {
    fn from(value: MultiError) -> Self {
        let MultiError { mut first, rest } = value;
        for e in rest {
            first.combine(e);
        }
        first
    }
}

As a bonus, the try operator (?) will implicitly call into() which allows us to do things like this:

fn check_logic(...) -> Result<(), syn::Error> {
  // ...
  let result: Result<(), MultiError> = logic();
  let _ = result?; // <=== Convert MultiError to syn::Error implicitly
  // ...
}

With that added, I needed a way to test my logic to ensure I was capturing multiple errors.

Add Display

To render the error on failure it needs to implement std::fmt::Display:

impl std::fmt::Display for MultiError {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        Into::<syn::Error>::into(self.clone()).fmt(f)
    }
}

It’s not pretty, but it worked and was easy. This code path is only ever called under test.

Add iteration

To expose multiple errors for testing, I chose to implement the IntoIterator trait:

impl IntoIterator for MultiError {
    type Item = syn::Error;
    type IntoIter = <VecDeque<syn::Error> as IntoIterator>::IntoIter;

    fn into_iter(self) -> Self::IntoIter {
        let MultiError { first, mut rest } = self;
        rest.push_front(first);
        rest.into_iter()
    }
}

This code says that we can now convert our struct into something that produces a series of syn::Error-s. Since we’ve already got a VecDeque lying around, and I knew that it implemented the same trait, I piggybacked my logic on top. This allowed me to do things like this test:

    #[test]
    fn test_captures_many_field_errors() {
        let field: syn::Field = syn::parse_quote! {
            #[cache_diff(unknown)]
            #[cache_diff(unknown)]
            version: String,
        };
        let result: Result<Vec<ParseAttribute>, MultiError> =
            crate::shared::parse_attrs::<ParseAttribute>(&field.attrs);

        assert!(
            result.is_err(),
            "Expected {result:?} to be err but it is not"
        );
        let error = result.err().unwrap();
        assert_eq!(2, error.into_iter().count()); // <== into_iter() HERE
    }

    enum ParseAttribute {
        //...
    }
    impl syn::parse::Parse for ParseAttribute {
        // ...
    }

This code parses a single field with multiple syn::Attribute-s on it. In this case, cache_diff(unknown) is an invalid attribute, and I want to assert that it does not stop after the first one it sees. The code converts my result into an iterator and then asserts that there are two elements. Great!

Iter Oops

While the above code example worked fine, I kept applying this pattern, bubbling up errors until I hit a failure in my code:

    #[test]
    fn test_captures_many_field_errors() {
        let result = ParseContainer::from_derive_input(&syn::parse_quote! {
            struct Metadata {
                #[cache_diff(unknown)]
                #[cache_diff(unknown)]
                version: String,

                #[cache_diff(unknown)]
                #[cache_diff(unknown)]
                name: String
            }
        });

        assert!(
            result.is_err(),
            "Expected {result:?} to be err but it is not"
        );
        let error = result.err().unwrap();
        assert_eq!(4, error.into_iter().count()); // <== FAILED here
    }

The error said that I was returning only two errors instead of 4. Which was confusing. I moved the code into a trybuild integration test and saw 4 errors. At this point it dawned on me, that at some time I was storing multiple errors into a single syn::Error and then placing that combined error in my MultiError. Basically I had a multi MultiError.

If that was hard to follow, here’s some pseudo code:

let mut errors: VecDeque<syn::Error> = VecDeque::new();

match call_fun() { // Returns a MultiError
    Ok(_) => todo!(),
    // Combines it into a single `syn::Error`
    Error(error) => errors.push_back(error.into())
}
// ...

if let Some(error) = MultiError::from(errors) {
    Err(error)
} else {
    Ok(
    // ...
    )
}

Essentially, my MultiError type allowed for what I thought was uninspectable-state. Each syn::Error could hold N errors.

A Fowl Epiphany

As I went through the stages of grief for my beautiful type that had a fundamental flaw, I hit on the idea that perhaps I could upstream a change to expose the internal combined errors from syn::Error. I thought that the IntoIterator interface was a good candidate to add. But to my shock, when I opened the docs the impl IntoIterator for syn::Error was right there this whole time. I just missed it.

When I realized that syn::Error already implemented every trait that I needed, I was able to change every MultiError into syn::Error and replace every MultiError::from_error with a function that returns Option<syn::Error>. Then, with zero other logic changes, my code compiled. That confirmed my suspicions that I had written a duck-typed duplicate of a commonly available struct.

The only value my MultiError type brought was that it hinted that the function was written with error accumulation in mind, but could not guarantee that the accumulation logic was correct. It didn’t seem like this minor social hint was enough to justify the extra code. I could achieve similar goals with a type alias.

Bad duck

If a type doesn’t introduce new capabilities or constraints and can be replaced by an existing, stable type, it should probably be deleted in favor of the more common type.

Good duck

Just because a type starts to smell a little foul (or rather “fowl”) does that mean you need to get rid of it? Producing a new type guarantees that there are no mix-up between your type and the common type. New typing could also allow you to restrict operations to a subset of the common type. Both of these things are about adding constraints.

A third reason to keep a duck around would be the stability of the interface. If you’re going to expose your type via a library and you’re worried it might change, then it could be helpful to wrap the type so your downstream user don’t have to change their code even if the underlying logic or implementation changes.

Duck documentation

When in doubt, consider documenting your duck and explaining what constraints the new type adds over the original. After writing them down, search for an already existing type that has the same behaviors. Perhaps go so far as to document why those types don’t meet your needs. If you cannot enumerate those differences well, then perhaps it’s a sign you should ditch your duck.

In my case I had explicitly called out syn::Error and even went as far as implementing Into<syn::Error>. Those are two strong signs that I should have investigated my claims and looked for features provided by trait implementations.

Practice your duck calls

One of the reasons I missed that syn::Error already met my needs that I didn’t stop to consider why certain traits were implemented on the struct or think about how they might be used to expose the data that I needed. Over time I’ve been better at internalizing and mentally mapping trait names to the behaviors they provide. Still, I’ve got some more work to do. Hopefully after this experience, with strong hints that I’m re-implementing an existing type as a duck, I won’t forget to check trait implementations for what I need.

Beyond “trying harder” and “writing a blog post as penitence so I don’t do it again,” I thought that it would be nice if this behavior was also shown via an example, so I sent a PR to syn to add some examples to syn::Error::combine. I don’t think we need to clutter all code with documenting every possible use case of every possible trait, but this very useful iteration functionality its in nicely in demonstrating how the combine behavior works. Hopefully, the addition of these docs will bre received well and not as an albatross

I would like to encourage everyone to pay attention to your types and the pain you’re feeling around them. If you find you’ve written a type that you later refactored away, consider pausing and capturing why it was written and why the world is better off without it. What other “bad type” patterns are out there and how can we make it easier for newcomers to spot and avoid them?

Update (2025/04/02): The change I suggested below was merged in PR #64. It’s pretty neat I went from knowing nothing about this project to contributing to it in the span of a single blog post.

A recent Oxide and Friends podcast episode, “A crate is born,” detailed the creation of a proc macro for deriving “diffable” data structures with a trick I want to tell you about. To help rust-analyzer as much as possible, @rain explained that the macro should always emit as much valid source code as possible, even when an error is emitted. They didn’t go into detail, so I looked into the internals that made this code + error emitting behavior possible and wanted to share.

Podcast link: A Crate is Born

This post covers:

• Why does macro output matter to rust-analyzer ?
• What mechanics are used to emit code + errors?
• When does this macro emit code + errors versus when does it just emit code?
• How does this relate to best practices in future or existing Rust macros?
• What is error accumulation, and why should every proc macro use it?

Who am I? I write Rust code for Heroku, mainly on the Ruby Cloud Native Buildpack (CNB). CNBs are an alternative to Dockerfile for building OCI images. You can learn more by following a language-specific tutorial you can run locally. I also wrote a book on Open Source contribution (paid) and I maintain an Open Source contribution app - CodeTriage.com (free).

Why does macro output matter to rust-analyzer?

Skip this if you already understand the problem statement

The Rust compiler will stop when it hits code that cannot compile. However, rust-analyzer (the Language Server Protocol implementation that powers IDEs like vscode) tries to resume after an error because it can’t just stop rendering type hints.

Intuitively, it makes sense that if you have an invalid function in your code, it shouldn’t break syntax highlighting (or other features) in your valid code:

fn invalid_wrong_return() -> String {
  ()
}

fn valid() -> String {
  "I am valid".to_string()
}

Daft (v0.1.2), emits trait implementations and sometimes generates new data structures. From the snapshot tests, an input of something like this:

#[derive(Debug, Eq, PartialEq, Diffable)]
struct Basic {
    a: i32,
    b: BTreeMap<Uuid, BTreeSet<usize>>,
}

Will generate code like this:

struct BasicDiff<'__daft> {
    a: <i32 as ::daft::Diffable>::Diff<'__daft>,
    b: <BTreeMap<Uuid, BTreeSet<usize>> as ::daft::Diffable>::Diff<'__daft>,
}
// ...
impl ::daft::Diffable for Basic {
    type Diff<'__daft> = BasicDiff<'__daft> where Self: '__daft;
    fn diff<'__daft>(&'__daft self, other: &'__daft Self) -> BasicDiff<'__daft> {
        Self::Diff {
            a: ::daft::Diffable::diff(&self.a, &other.a),
            b: ::daft::Diffable::diff(&self.b, &other.b),
        }
    }
}

If the macro does not emit this information (possibly due to some hypothetical error not present in this example), then rust-analyzer wouldn’t know that the BasicDiff struct was expected to exist, what its fields were, or that Basic::diff() returned a BasicDiff struct. In short, the IDE would be generally less helpful.

Now that you understand the goal, how do we emit code when there’s an error?

What mechanics are used to emit code + errors?

The short version is that macros don’t output code or errors; they emit tokens. The daft crate collects errors and continues when possible. If it can generate code, it will emit that generated code as tokens before turning the errors into tokens and then emitting both. An earlier version of the code looked like this:

let errors = error_store
    .into_inner()
    .into_iter()
    .map(|error| error.into_compile_error());

quote! {
    #out
    #(#errors)*
}

Where quote! produces tokens from both the code (#out) and the errors #(#errors)*.

But don’t take my word for it, read the source: The entry point for the Daft derive macro is internal::derive_diffable. This function returns a DeriveDiffableOutput. The DeriveDiffableOutput holds Option<TokenStream> for valid code that was generated and Vec<syn::Error> for errors

The DeriveDiffableOutput implements quote::ToTokens that emits the valid code followed by the errors (if they exist). code.

Put it all together, and you have a crate that emits partially generated code, even with errors. Neat.

When to emit code + errors?

Now that I knew how Daft implemented this feature, I wanted to understand when they chose to apply this pattern. I reviewed the snapshot tests and developed my own classifications.

There are three classes of failures in the snapshot tests:

• Emit code and errors (Warning error)
• Emit code only, no errors (Compile error)
• Emit errors only, no code (Error)

First, proc-macros cannot emit warnings. If the coder entered slightly off information that wouldn’t affect compilation, the macro author must choose between letting it slide or raising an error. There’s no in-between.

When there’s a problem but daft can determine programmer intent, it will emit code and errors. I call this a “warning error.” The primary example in snapshot testing is when the #[daft(leaf)] attribute is used on an enum that is already a leaf by default (this is an internal concept to the crate).

Note that the daft crate does not use “warning error” as terminology. I am making that distinction based on my analysis of the snapshot test. Notes are here..

Second, the program can fail to compile even when code is emitted without error, for example, if a trait bound is not satisfied. The macro author cannot detect the problem because the reflection tools don’t expose the necessary information. They must rely on the compiler errors to guide their user.

Finally, there are situations where the author cannot safely emit code because an input is ambiguous or wrong. With these “plain” errors, if the macro author tried to guess and got it incorrect, they’re feeding rust-analyzer incorrect information, which might confuse the end user more. For example, if an attribute that doesn’t exist, such as #[daft(unknown)], is found, the macro author has no idea what was intended there and shouldn’t guess.

While working on this classification exercise I found two snapshot tests where code isn’t emitted but could be.

Should all proc macros emit code + errors?

Obligatory: “It depends.”

If you find your rust-analyzer horribly broken due to a proc-macro problem, then this is a great trick to suggest. However, it isn’t a critical feature that every macro should have. Instead, libraries should focus on improving error accumulation (talked about later).

This code + error functionality requires a lot of plumbing, and ultimately, there’s only one code path (or two if they like my suggestion) that generates code + errors. Looking at how this code path came to be, it seems more that it was added because the plumbing already existed and the opportunity presented itself. From that lens, it’s easy to see why Daft goes this extra mile. The cost to implement was (comparatively) low:

• Error store introduced in PR #41
• Emit Code + Errors added in PR #42

While the lede: emitting code + errors is likely too much of an ask for most crates, every proc macro should accumulate errors. Let’s look at that now.

More ! for your $: Accumulate proc macro errors

What do I mean by accumulated errors? In a Python (or Ruby) program that raises an error, you have to fix that to find out if there’s another error lurking that also needs to be fixed. Rust programmers don’t like playing that game. They want as many errors upfront as possible:

error: #[daft(leaf)] specified multiple times
 --> tests/fixtures/invalid/field-specified-multiple-times.rs:5:18
  |
5 |     #[daft(leaf, leaf, leaf)]
  |                  ^^^^

error: #[daft(leaf)] specified multiple times
 --> tests/fixtures/invalid/field-specified-multiple-times.rs:5:24
  |
5 |     #[daft(leaf, leaf, leaf)]
  |                        ^^^^

error: #[daft(ignore)] specified multiple times
 --> tests/fixtures/invalid/field-specified-multiple-times.rs:8:12
  |
8 |     #[daft(ignore)]
  |            ^^^^^^

error: #[daft(ignore)] specified multiple times
 --> tests/fixtures/invalid/field-specified-multiple-times.rs:9:12
  |
9 |     #[daft(ignore)]
  |            ^^^^^^

This daft output says there are two errors on line 5. One error on lines 8 and 9. This code generated the following errors:

use daft::Diffable;

#[derive(Diffable)]
struct MyStruct {
    #[daft(leaf, leaf, leaf)] // line 5
    a: i32,
    #[daft(ignore)]
    #[daft(ignore)] // line 8
    #[daft(ignore)] // line 9
    b: String,
}

Fields and their attributes are parsed iteratively, so it’s common for a macro to stop iterating on the first problem (line 5) before returning. Instead, Daft stores the errors and continues parsing until it longer can.

I don’t think it’s the end of the world if a proc macro only emits a single error at a time, but it’s a requirement if you’re aiming for a “Michelin star proc-macro” experience.

How does Daft implement error accumulation?

Instead of using a Result<T, syn::Error> return, daft passes an accumulator that holds a Vec<syn::Error> to every fallible function. If there’s an error, it’s added to the accumulator. This pattern also means that instead of having to choose between emitting T or syn::Error (via a Result<T, syn::Error>), the programmer can do both by returning a T while mutating the accumulator. That would indicate the problem is more of a “warning error” if the data structure can still be safely returned. Functions that return Option<T> indicate they’re likely holding one or more plain errors that would prevent code generation when a None is returned.

Beyond affording the ability to return code + errors, not using a Result means that the try operator (?) cannot be used accidentally for an early/eager return. This property encourages the macro author to capture as many errors as possible and emit them all instead of only emitting the first error. It’s a neat pattern, but it’s not the only way to accumulate errors.

Alternative pattern for syn::Error accumulation

The syn::Error struct has the capability of combining multiple errors without an accumulator by using syn::Error::combine. For example:

let mut errors = VecDeque::<syn::Error>::new();
// ...
if let Some(mut first) = errors.pop_front() {
    for e in errors.into_iter() {
        first.combine(e);
    }
    Err(first)
} else {
    Ok(
    // ...
    )
}

This pattern is useful when the function signature isn’t changeable. For example, the syn::parse::Parse trait is commonly used by proc macros as a building block, and it has a fixed signature:

fn parse(input: syn::parse::ParseStream<'_>) -> Result<T, syn::Error>

With this pattern, the error behavior of the function is encoded in its return type:

• Errors that block code generation should return Result<T, syn::Error>
• Errors that do not block code generation should return (T, Option<syn::Error>)
• Errors that may or may not block code generation should return Result<(T, Option<syn::Error>), syn::Error>.
  • Ok((T, None)) indicates no errors
  • Ok((T, Some())) indicates an error that did not block code generation.
  • Err() indicates that code could not be generated due to error

That last one is verbose, but it prevents representing an invalid state when None code and None errors are returned simultaneously.

The downside of this technique is that nothing prevents an early return on error with try (?).

I was curious how this pattern would look implemented in place of the daft one, so I experimented with a draft (not daft) PR.

Note: The PR is to my own main branch, not theirs. I don’t think any maintainer loves waking up to a giant PR with the “refactoring” in it.

Wrap up

We learned why rust-analyzer is sensitive to macro output. We explored the mechanics that Daft uses to emit code + errors, and accumulate errors. I introduced an alternative error accumulation method and I made some strong statements. Namely that emitting code + errors is a nice-to-have while accumulating and emitting all errors is an achievable best practice.

Coming from Ruby, proc macros are wonderful things that allow Rust developers to write powerful and expressive DSLs, and I love them. With the power to meta-program, there’s also the possibility to meta-confuse your end user or toolchain (like rust-analyzer). I love that the Daft maintainers put as much work and care into the failure modes as the rest of their logic in addition to accumulating and presenting as many errors as possible.

I hoped you enjoyed learning about these patterns as much as I did.

FYI I’m working on a proc-macro tutorial and would love to hear from readers on Mastodon or Reddit about what real-world patterns you’ve seen around improving the end-user experience, especially around errors.