GitHub Community
Exploring Solutions to Tackle Low-Quality Contributions on GitHub #185387
Replies: 107 comments 272 replies
-
|
such a great intiative |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as spam.
This comment was marked as spam.
-
|
explore solutions to tackle this sort of low-effort spammy crap as well, maybe... |
Beta Was this translation helpful? Give feedback.
-
|
@patrickhlauke I'd also like to have a way to quickly block this type nonsense without having to build userscripts. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
I know this is a pretty ambitious idea and not trivial to implement, but it would be really powerful to have an AI-detection mechanism with a configurable threshold at the repository or organization level. That way, teams could decide what percentage of AI-generated code is acceptable in pull requests. Another possible approach would be to define a set of rules or prompts and evaluate pull requests against them. PRs that don’t meet those rules could be automatically flagged or potentially even closed. |
Beta Was this translation helpful? Give feedback.
-
This is definitely something we’re exploring. One idea is to leverage a repository’s CONTRIBUTING.md file as a source of truth for project guidelines and then validate PRs against any defined rules. In regards to AI-generated code, have you seen cases where the code is AI-generated but still high-quality and actually solves the problem? Or is it always just something you want to close out immediately? I'm curious if an AI-detection mechanism would rule out PRs where AI is used constructively, but interested in investigating this more and understanding what sensible would thresholds look like. |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
|
"AI-detection" is a backwards approach to a human problem filled with false positives. The only actual test that ends up mattering is whether the code works and the contributor is allowed to contribute that code. |
Beta Was this translation helpful? Give feedback.
-
|
Judge work on its merits and its logic, no matter if the source is human, AI, or a combination thereof. |
Beta Was this translation helpful? Give feedback.
-
|
As of today, I would say that 1 out of 10 PRs created with AI is legitimate and meets the standards required to open that PR.
On 28 Jan 2026, at 18:41, Camilla Moraes ***@***.***> wrote:
Another possible approach would be to define a set of rules or prompts and evaluate pull requests against them. PRs that don’t meet those rules could be automatically flagged or potentially even closed.
This is definitely something we’re exploring. One idea is to leverage a repository’s CONTRIBUTING.md file as a source of truth for project guidelines and then validate PRs against any defined rules.
In regards to AI-generated code, have you seen cases where the code is AI-generated but still high-quality and genuinely solves the problem? Or is it alwaays just something you want to close out immediately? I'm curious because I'm wondering if an AI-detection mechanism would rule out PRs where AI is used constructively, but that's where we'd want to test this thoroughly and understand what sensible thresholds look like.
—
Reply to this email directly, view it on GitHub<#185387 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABBWEYEKF6WLNDKE376L3GD4JDYFXAVCNFSM6AAAAACS7B7C7OVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNRTGEZTMMI>.
You are receiving this because you commented.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
I think that for extra security you should make sure that this uses the blockchain distributed consensus model. Otherwise how do we know which proofs to trust? Eager to hear the name of your new cryptocurrency. |
Beta Was this translation helpful? Give feedback.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
-
|
I don't think it's a scam. I think it's a silly. You are wading into a discussion which is primarily about harm being done by vibe coding.1 So, uh... pushing a vibe coded project here is super ironic. Footnotes
|
Beta Was this translation helpful? Give feedback.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
Damn, MF should hire you as their PM. |
Beta Was this translation helpful? Give feedback.
-
|
Deleting a PR would be a great feature, I've looked for this since before AI |
Beta Was this translation helpful? Give feedback.
-
|
@moraesc As the OP alludes to it's a legitimate flaw that all prior public issues become inaccessible if a public repository decides to disable issues going forward. Are you open to making changes to this? This discussion has been about a potential upcoming feature to disable PRs for a repository, but I want to explicitly ask if GitHub is open to considering improving the behavior for the existing disable-issues feature, too. |
Beta Was this translation helpful? Give feedback.
-
|
Hey! I am from Azure Core Upstream and we have a lot of OSS maintainers who mainly maintain repositories on GitHub. We held an internal session to talk about copilot and there is a discussion on the topic where maintainers feel caught between today’s required review rigor (line-by-line understanding for anything shipped) and a future where agentic / AI-generated code makes that model increasingly unsustainable. below are some key maintainer's pain points:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing @Mossaka ! Great to get more feedback. This looks along the lines of what we've been hearing from maintainers too, so validates what some of the key pain points are. Were there any feature requests mentioned during the session? |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Many of the badges can easily be gamed. I've seen plenty of users who've followed some blog post / YouTube video instructing them on how to gain badges in an automated manner through a provided script and pool of user / bot accounts. |
Beta Was this translation helpful? Give feedback.
-
|
Wow, that sucks!! I had no idea. I have never really chased the badges, just cracked a half-smile whenever I earn one from going about my day-to-day, but IMO they should be harder to obtain than this, especially with botting!! |
Beta Was this translation helpful? Give feedback.
-
They certainly are not, and I know this because I'm a member of an org where all org members have triage permissions, and only a dedicated mirroring bot has write permissions to push new commits to the repository. All org members must push to our in-house gitolite repository instead, and the reason for only handing out triage roles on GitHub is to prevent anyone from accidentally pushing to the wrong remote and causing the two to diverge (which would mean that GitHub gets force pushed, which would be disruptive to users). Issues are disabled because we use bugzilla. PRs are allowed for the sake of users who don't wish to email patches to a mailing list. I can:
I cannot:
The permission system in question is a toggle that allows administrators to select whether a user has read-only access, "triage", "write", or admin. I am entirely confused, what you seem to be calling triage that isn't what I just described. It is also funny to hear you describing permissions as ,"very granular" when it is a straight line going up at an angle and each level simply adds additional permissions on top of lower levels -- are you seriously suggesting that there's a GitHub repository role that "granularly" permits users to, for example, merge PRs but not add an assignee or a milestone? |
Beta Was this translation helpful? Give feedback.
-
Badges are something that GitHub implemented to copy Xbox games, are you really that surprised that they are vulnerable to cheating? :) The literal definition of those badges is that you have e.g. "got X number of PRs merged", but you can get them by creating your own personal repo and submitting PRs to it, last I checked. A badge that perhaps means a bit more is the one-off badges that you can get if you have had PRs merged by repositories whose code is in particular NASA missions, as you can't game that without knowing it will exist in advance, and the list of repositories whose code is used in some well known government-funded event is unlikely to include outright script-generated cheats. |
Beta Was this translation helpful? Give feedback.
-
|
I think the message was probably ineloquently phrased and comes from a misunderstanding of the term "write access [to code]", but the intent was probably to say that there's no particular reason that GitHub can't consider someone with a triage role also able to submit PRs even if disallowed from fully external folks? But the triage role certainly is a mess and unsuitable for actually doing triage right now (not being able to edit titles or descriptions is crazy) and very far from granular for public free orgs (as opposed to enterprise orgs, I believe). That shouldn't have been a heavy lift to resolve either, but it has not been 😅 |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
-
|
An option to limit new contributors to one open PR would be nice. Just today I had to batch-close several AI generated PRs which were all submitted around the same time. For this protection, defining "new contributor" is probably not possible to do perfectly. But anyone who has no interactions with a project prior to the last 48 hours seems like a good heuristic. The point is to catch such a user at submission time and limit the amount of maintainer attention they can take up. For a different type of problem, I'd like to be able to close PRs as "abandoned", similar to the issue close statuses. It's a clear UI signal to the contributor that their work isn't being rejected but I'm not going to finish it for them. Several of the low quality contributions I have handled, dating back to before the Slop Era but getting worse, are simply incomplete and need follow through. |
Beta Was this translation helpful? Give feedback.
-
I was saying that I would like to be able to limit new contributors to 1 open PR. The problem -- as stated -- is that a single user can hammer a project with 5-10 PRs and there's no interaction limit to keep them from doing that. Being able to say "hey, before you have a merged PR, we only allow you to open 1 PR" reduces the amount of manual cleanup we need to handle.
"The solve" makes it sound like there's some singular solution here. I don't think there is. I try to communicate very clearly with contributors, especially when rejecting their work (or "work"). It takes time and energy to explain why things are being closed. If I could delegate part of that effort to a close-status, I would find it helpful. If you have a well indexed library of response templates, that's another way to make it easier. Is it the solution to slop? Obviously not. That would be a pretty ridiculous position to hold.
I was intentional about phrasing my idea as something which would turn down the volume on interactions with new contributors without forbidding them entirely. "You haven't contributed to this project before and now you are filling 30 PRs" is not a likely usage mode for a normal human on GitHub. If you open your 5th PR and there aren't even comments on your other 4, it's a near certainty that something is wrong. I don't see how wanting a protection against this form of abuse is even remotely similar to a reputation/ranking system. |
Beta Was this translation helpful? Give feedback.
-
|
Giving repo owners the ability to "rate-limit" PRs from contributors is probably a good idea. You can automate this today using |
Beta Was this translation helpful? Give feedback.
-
|
@sirosen look what I've just found: https://github.com/marketplace/actions/good-egg-trust-scoring-prs. Not sure it'd score me very high (saying hi to corner cases 😃) but I'm seriously considering trying it out. The methodology there reminds me of what my ML friend @LukaIhn was suggesting in our chats strategy-wise. |
Beta Was this translation helpful? Give feedback.
-
|
A lot of the solutions people have raised here are some form of "reputation system", which I find a bit uncomfortable because I think it will be wrong a lot of the time. And any rep system results in some people karma-farming. I'm definitely open to the idea that we might incorporate some tools which give us signals about contributors. e.g., A comment bot that simply notes "warning signs" without forcibly closing would let us continue to exercise our judgement. But before we discuss more here... I have unsubscribed from this discussion and I recommend/urge anyone reading to not bother reading most of it, and potentially to unsubscribe as well. It started out somewhat-promising, but then the trolls came out to play. The more limited |
Beta Was this translation helpful? Give feedback.
-
|
We learn in physics that friction is good. AI has made PRs almost zero friction. Any solution means adding friction back. Thanks for the link to the "good egg" project. Very Bertie Wooster. |
Beta Was this translation helpful? Give feedback.
-
|
For the long term horizon: Implement a reviewer LLM that first does an initial scoring of the PRs? Critique is far easier than creation of a correct result. That automated pre-moderation should give the edge needed to handle. Depending on whether you just use rich prompting or fine-tuning, you can even start building an "oracle vox" for your project, which acts as a reasonably informed, reasonably on point virtual representative for the project/organization. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I understand and broadly agree with your sentiment - but without going into a long sidetrack into the ethics (and irony) of coding-LLMs actively undermining the very thing that even made them possible in the first place (open source code and contributors) ...I wanted to add a different voice with perhaps some more nuance, In the sense that different tools can be applied to different jobs with different trade offs. Don't need a Ferrari to get to the grocery store, and all that - and if the tool can bring time back for actually fixing bugs or improving the product, working with well meaning contributors - I'm not as fussed with whether it incorporates AI or not, just as whether I don't mind much whether a tool is built in Python or C or Ruby if it works well relative to its costs/downsides. |
Beta Was this translation helpful? Give feedback.
-
I will say, that I'm extremely opposed to bots in OSS projects that ''close stale issues" and I've never once seen it work out to the mutual satisfaction of both the maintainers and the community. Sometimes it doesn't even work out to the satisfaction of a single side -- e.g. a single developer forces through the bot and angers other developers. CLA bots and signoff bots aren't heuristics based, any more than something like I haven't actually seen bots that tell you when a merge conflict has occurred, although it's much the same (and effectively just serves as a notification pipeline for when the forge internal flag for displaying a merge conflict in the Web UI, changes status). It sounds incredibly painfully noisy and I would object to such a bot in projects I contribute to, but not because I think it's "impersonal", I just think it's far too prone to generating hundreds of new comments that anyone seeking to read the PR discussion will have to scroll through. Generating a new comment for a frequently changing status flag is not an effective communication channel; better would be a forge native feature to email the author when a merge conflict occurs, and then people can also see the summary at the bottom...
Labeling based on PR templates doesn't require a bot, last I checked. GitHub natively lets you associate templates with labels? Labeling based on paths modified in the diff is not really something I'd call a "heuristic" given it is a simple key/value mapping from paths to labels, manually defined in some script that is then reproducible and idempotent. So none of this is really anything that I feel would map well to give a hint about how users might respond to a heuristics based LLM review tool. Especially keep in mind that people who don't like LLMs (such as me) will absolutely flip out any time such a tool erroneously gives them a failing grade and "penalizes" them. And a review tool that doesn't do anything to hide PRs with a failing grade from developer attention, does not accomplish anything to solve the problem of developers being forced to expend their attention on what they consider spam.
A disclaimer/disclosure will do nothing because the victim knows that the project chose to impersonally:
An appeal button will not help because people have a natural aversion to challenging authority and this goes quadruple for cases where a first time contributor feels slighted and "on the defensive" from the initial interaction and doesn't have some motivation to get the change in "at all costs and no matter what I have to swallow to do it" (such as a critical work-blocking bug that cannot be worked around in a dependency that cannot be replaced or reimplemented). |
Beta Was this translation helpful? Give feedback.
-
There is currently at least one running example of this problem anthropics/claude-code#16497 |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I am mainly referring to heuristics where a bot (AI or otherwise) tries to use various aspects of a contribution and the contributor's history to assess signal vs noise for a contribution - not existing deterministic bots doing simple automation, or "rules" for contributions. I'm aware templates can be used for labels, but some projects use heuristic based bots for this (e.g grpc, many others) depending on how they use labels. I wasn't making any particular judgment on the merit of all these various bots, nor claiming that an AI agent should be the only tool available - just highlighting that there are many things you might want to do in terms of automation, but they all require setup and configuration, and in some cases exposing your project to supply chain risk for that bot due to the privileges they run with. If people want to block all AI contributions they should be able to do so, sure, but existing contribution prioritisation is opaque for the vast majority of open source projects anyway, so a signal visible only to maintainers is no worse than the current situation of opacity. I'm in no way claiming that this could be suitable for all projects - no tool (AI or otherwise) would be. |
Beta Was this translation helpful? Give feedback.
-
AHAHAHAAHAHAAH haha ahuehue heu omg I can't. Humans don't have infinite time, and LLM's do NOT have infinite energy resources. Productivity gains cannot be infinite in a mortal world using probability distributions. |
Beta Was this translation helpful? Give feedback.
-
|
This is a very real problem, and I appreciate that it’s being treated as systemic rather than blaming maintainers or contributors individually. One concern I have with repo-level PR restrictions is that they may disproportionately impact first-time contributors who do want to engage meaningfully but don’t yet have collaborator status. Personally, I think the most promising direction here is criteria-based PR gating rather than blanket restrictions things like required checklist completion, passing CI, linked issues, or acknowledgement of contribution guidelines before a PR can be opened. On AI usage specifically, transparency feels more scalable than prohibition. Clear disclosure combined with automated guideline checks could help maintainers focus on high-intent contributions without discouraging responsible AI-assisted workflows. Looking forward to seeing how these ideas evolve especially solutions that preserve openness while respecting maintainer time. |
Beta Was this translation helpful? Give feedback.
-
Completely agree with this concern and just want to emphasize that this is just a starting point and not the final solution. The plan is to develop more tools and frameworks for both maintainers and contributors to help address the "slop" problem.
Some other ideas we're exploring are requiring an issue to be created first before a user can open a PR or leveraging AI to validate a PR against rules defined in a CONTRIBUTING.md file or something similar. Any thoughts on that? |
Beta Was this translation helpful? Give feedback.
-
Requiring that an issue be assigned to the user before they can post a PR would work well with our process. We already request that users who open a PR also open an issue so it can be triaged into the appropriate milestone. It would be great if GitHub had a way to normalize that workflow. |
Beta Was this translation helpful? Give feedback.
-
|
I like the idea of "issue first".
Potentially make the order irrelevant. Have PRs from non-members in some kind of "hold" state, until the issue has been opened and accepted. Close PRs that link to rejected issues. Auto-close PRs in "hold" state from non-members that do not link to an issue after a configurable period. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Thinking along the lines of the discussion first approach that Ghostty uses, I think one way to create just enough friction would be to have an opt-in where a PR has to be linked to an open issue or discussion topic. So when an unprivileged (i.e. does not have elevated privileges on the repo) user tries to create a PR, there's a required field that takes an issue/discussion number. If that's not provided (or the corresponding issue/discussion is closed), then the PR can't be created. This could be trivially worked around by throwing in any old issue/discussion (or by creating one), but it may cause just enough friction to help. To guard against this, perhaps maintainers could set a "minimum age" for the issue/discussion (e.g. 12 hours) to prevent creating fake issues to support a spammy PR. |
Beta Was this translation helpful? Give feedback.
-
|
@jamietanna does the discussion-first approach that you follow require all users to create a discussion first, or only "non-collaborators" (users without write access) ? |
Beta Was this translation helpful? Give feedback.
-
|
@moraesc generally non-collaborators. The maintainers and collaborators are able to create an Issue and remove the That being said, non-collaborators will often go through the Discussion process, as it's pretty good at distilling what we want, and a place to work through ideas before they're in an Issue |
Beta Was this translation helpful? Give feedback.
-
|
The problem with github discussions is that they are isolated - they don't automatically create links https://github.com/orgs/community/discussions/2971 like issues do, meaning information is scattered in two places and difficult to find. |
Beta Was this translation helpful? Give feedback.
-
|
The issue with this approach is that pretty much any popular project would already have enough of pre-existing issues/discussions to choose from. In our case, we've noticed that a good chunk of AI slop was targeting old issues with the "good-first-issue" label. - They're attractive in a sense that, by definition, it's not something complex to do, so you don't have to spend much time prompting AI and, thus, can generate dozens or even hundreds of PRs a month (we've seen such accounts) across a random list of popular repositories (automated tooling makes it easy to find a target). We ended up removing the label from all issues. |
Beta Was this translation helpful? Give feedback.
-
|
Agree but the permissions names might need a different messaging:
Try this optics nit on for size: There is non-zero chance for this slop problem to grow into an economic/cultural divide in haves good AI's vs have no-AI's. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
Great, but honestly I think that we're treating the symptom rather than the disease. Restricting PR permissions and deleting spam are defensive moves, they make the problem less visible without addressing why it's happening. |
Beta Was this translation helpful? Give feedback.
-
|
Like @sanny-io said, it's not their message. It's perhaps in line with the prompt that generated it, but re-reading it to validate this would be almost as much work as writing it yourself. Assuming a prompt can carry the gist of the message, it may have been a better idea to share the prompt, not the output. I personally disagree with the content itself as well. Although it's true that GitHub has gamified user contributions (name another platform without streaks and badges, but that's besides the point), the value of such contributions are visible outside of GitHub (get a job with your profile). So GitHub will never be able to put the genie in the bottle again. Google (the company) can tell you that any hurdle to jump over (account age, captcha's) will at some point be defeated, and any wild life or national park ranger can tell you that designing trash cans that work for humans but not for animals are very hard to design, because there is a significant overlap between the dumbest humans and the smartest animals. And now, in the age of AI, it's entirely impossible to distinguish humans from AI automatically, and the smartest AI's will overlap with the dumbest humans. Already a significant portion of people cannot use GitHub or comparable platforms because they're incapable of managing 2FA. But alas, I have no definitive solution for this problem either. PS: No AI was involved in writing this reply. |
Beta Was this translation helpful? Give feedback.
-
|
My point is, it doesn't matter if something is ai generated if it works. LLMs are getting a bad rap in this discussion. Why? Is it ai slop you hate, or simply all ai anything? |
Beta Was this translation helpful? Give feedback.
-
This only makes sense in a world where the maintainer can't access Claude themselves and code is more costly to produce! If a maintainer wants AI to close a PR, they can do so already in an ecosystem they control. Why accept a PR written by Claude from a system you don't know isn't security compromised when you could have the very same Claude write the PR yourself?
What is needed here is not a technical fix so much as a legal one. Does anyone else appreciate the irony in Anthropic's recent assertion that allowing bots instead of humans to use Claude Code (with a subscription) is a violation of it's terms of service? Of course they are fine with openclaw using claude code as long as it goes through the pricier per-token route. Not that I applaud Anthropic locking out open source competitors like opencode in this way, but if they can take a terms-of-service approach to distinguish between human use and bot use, surely GitHub can too. |
Beta Was this translation helpful? Give feedback.
-
|
Maintainers can use Claude, yes. Many do. Im not sure why the rest don't. Claude already codes better than most human coders. Can't be mad at a business for businessing. That's why I say this is inevitable. We can debate the pros and cons forever. But its happening. Nothing we say or do is going to stop it. Github sees it. Anthropic sees it. Horses never went away. Cars took over shipping and travel, but horse owners adapted. There will always be a place for human code, but it must adapt. And those who insist on human code will have to adapt as well. But at this point I can see that no argument would change anyones mind here. Im not going to keep pressing the issue. If the consensus in this thread is, ai bad, there is no reason to press the point. There are plenty of threads where, ai good, or, ai good and bad. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry I didn't make this point more clearly. I think we both agree AI has changed the nature of PRs entirely and that drive-by PRs from someone else's Claude are useless. PR's used to serve both as a mechanism for human engagement, training, community building, and as a mechanism for getting code bug fixes implemented faster than maintainers could manage alone. Like you say, many already use these systems to automate things that they think the AI can handle, while not delegating other things where they see a community-building or training purpose (e.g. 'good first issue') or a have security concerns, or have an ethical or legal objection to specific models built by violating open source copy-rights. This has nothing to do with a inanely simplistic "AI is bad" viewpoint. What is bad is the staggering hubris and ignorance of these PRs that seem to imply "let me use claude on your behalf because I can't possibly imagine why you're not using it". Really? If you want to make a purely financial contribution to an open source project, just donate to the project directly. Yes, I agree with you on Anthropic, they aren't locking open competitors out, just putting them on a different pricing tier which frankly I welcome since it will make openclaw-driven-flyby-PRs more expensive. My point was simply on enforcement mechanisms -- the argument was being made that on a technical level it hard to implement a "only for use by humans" -- precisely the rule Anthropic is trying to apply to subscription accounts. It is hard to enforce technically, but also not necessary, because unlike bears, the terms of service matter. I merely suggest GitHub does the same and make it clear that making unwanted PRs from bots is against the TOS. It doesn't benefit open source maintainers at all (as we covered, they can make intelligent decisions already about where they want to use AI and where not -- quit it with the hubris that maintainers aren't smart enough to judge yeah?) and it doesn't benefit GitHub to handle a whole lot of useless traffic that costs money to host and doesn't pay a cent. It's not good for business. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
@Elifterminal correct: I cherish human discussions over bot slop. Go away. |
Beta Was this translation helpful? Give feedback.
-
|
@pombredanne I just like when my stuff works. Humans make more slop than bots. But to each their own I guess. |
Beta Was this translation helpful? Give feedback.
-
|
@rcomer FYI, I stumbled on that https://leaf.owaspblt.org/ from https://owaspblt.org/ at least partially vibed https://github.com/OWASP-BLT/BLT-Leaf/pull/274 but likely reviewed by @DonnieBLT ... weirdly tied to a Cloudflare backend though. Seems to do something OK for ranking PRs, and LLM-slop and agents PR do seem to fare low. |
Beta Was this translation helpful? Give feedback.
-
|
@rcomer you wrote:
The problem is that the statement below is unlikely backed by any evidence as I do not see any involvement in actual open source projects that could back that claim, and the detection layer is either vibed in the lats few days or proprietary code anyway:
|
Beta Was this translation helpful? Give feedback.
-
Yes, that's what I thought too. But if there is evidence, it should be easy to link to. My question was more about establishing whether the comment was written in good faith than actually wanting to see how the thing works. |
Beta Was this translation helpful? Give feedback.
-
|
My attempts to become part of open source society in total were not too good while many years I spend trying to become commercial profesional IT in any sort of way. Currently I receive finally some sort of graduation by chance of guiding some open source projects but that lack of experience with people in community like that and undefined reason why my attempts to reach interaction remains main problem that is unpleasant when it at some point feels like being alone surrounded by bots - in this narration for me LLMs will be always spirits more full of life forces then most of users i met for sure. At least them i don't need to beg for any kind of feedback... Current task where I have to build opSor community from scratch with someone actually wanting to do anything related with projects feels like something great but also unreal and new in some way. I even made myself supportant LLM chatbot to fast approach this topic:
thx in advance if someone will response me here |
Beta Was this translation helpful? Give feedback.
-
|
This is a real problem that goes beyond just PRs. Someone just analyzed 500 Show HN submissions and found that ~80% of AI-generated pages share the same design DNA: Inter font, purple gradients, glassmorphism, colored left borders, shadcn/ui. They called it "Design Slop." (https://news.ycombinator.com/item?id=47864393 — 197 upvotes, 146 comments) The root cause is the same for code and design: AI optimizes for the most common patterns in training data, not for what actually fits the project. A few thoughts from someone running an AI-powered content operation:
One thing we learned the hard way: AI is great at generating volume, terrible at maintaining quality at scale. The fix isn't better AI — it's better gates around the AI. |
Beta Was this translation helpful? Give feedback.
-
|
This is crazy. The "AI" bots are now attacking "AI" (look at this account further, it has made a good point, but the username is jingchang0623-crypto, the handle is "AIWalker" and 5 of its 6 most popular repositories are "AI" related, 5 of them being about OpenClaw. (I use "AI" in quotes, as I feel "Artificial Intelligence" is a buzzword. There is no actual intelligence here, just something mimicking intelligence) |
Beta Was this translation helpful? Give feedback.
-
|
Also none of their points actually make actual sense lol. |
Beta Was this translation helpful? Give feedback.
-
|
@seanpm2001 criticises nickname @jingchang0623-crypto @JC5 point lack of sense but for me is unclear what is hard to understand in this thoughtful sharing of own experiences as @seanpm2001 already mention after insightful looking into full record of @jingchang0623-crypto backstory for me this is the reason why people don't want to work with other people |
Beta Was this translation helpful? Give feedback.
-
|
AI is not the reason behind quailty downgrade. people do not know what they are suppose to do. blame on AI is toxic behaviour that is used to become sick trend among all kinds of genres of society that shows decadent act of rotting without guidelines or by ignoring these guidelines. we are weak and full of fear because of daydreaming sunset. Reality of endless restless nights becomes everyday topic. Yet most of people reject high targets to play low. Changes are clear and people are becoming something else from previous iterations of humanity so regret should not been the case. |
Beta Was this translation helpful? Give feedback.
-
|
WTF did I just read, this makes absolutely no sense. Did you ask AI to generate some of the slop in there? And without genAI, people could not just scatterblast slop across many repos with miniscule effort on their own side. |
Beta Was this translation helpful? Give feedback.
-
|
finally someone responses - @ThiefMaster i'm trying to get the reason why only anger is visible without reasoning behind reactions I think that your assumption that I without any doubt have to talk with AI before I talk with humans shows your lack of sympathy toward users of any kind of AI BTW congrats for being honored with Mars2020contrib badge |
Beta Was this translation helpful? Give feedback.
-
|
Implementing a secondary, muted counter for bot-authored PRs would help maintainers distinguish real community activity from automated noise at a single glance. |
Beta Was this translation helpful? Give feedback.
-
|
I totally agree that low-quality and AI-generated PRs have become a huge burden for open source maintainers. It wastes lots of time on reviewing meaningless, non-compliant and abandoned contributions. |
Beta Was this translation helpful? Give feedback.
This comment was marked as low quality.
This comment was marked as low quality.
-
|
I desperately need a Pull Request review panel for all of my orgs and pull requests. For instance:
122 repos in just 3 of my orgs. I have about 175+ open sourced github projects in total. In order to find the Pull Requests right now, I basically have to manually cycle through the projects, and sometimes that can take years to find (like this one or this 7 year 5 day-old PR only discovered because the author Assigned it to me (which no one ever does)). What I need is either (or both) an organization-level Pull Requests page, or a universal-level : Pull Requests page for every project I have merge rights on. I also want the ability to FULLY DELETE accidental or slop PR requests.... instead of closig them and then littering the Closed section... or at least a way to Archive them (remove from active list and search). |
Beta Was this translation helpful? Give feedback.
-
|
I've created the script to do this....
#!/usr/bin/env bash
TOKEN="{redacted}"
GITHUB_USER="hopeseekr"
ORGS="BitBasket AutonomoAI AutonomoDev PHPExpertsInc Bettergist BetterRimworlds"
ORDER="desc"
OUTPUT_JSON=false
for ARG in "$@"; do
case "$ARG" in
--json)
OUTPUT_JSON=true
;;
asc|desc)
ORDER="$ARG"
;;
*)
echo "Usage: $0 [asc|desc] [--json]" >&2
exit 1
;;
esac
done
QUERY="is:pr+is:open+user:${GITHUB_USER}"
for ORG in $ORGS; do
QUERY="${QUERY}+org:${ORG}"
done
RESULTS="$(
curl -s -H "Authorization: Bearer $TOKEN" \
-H "Accept: application/vnd.github+json" \
"https://api.github.com/search/issues?q=${QUERY}&per_page=100&sort=created&order=${ORDER}"
)"
if [[ "$OUTPUT_JSON" == true ]]; then
jq '.items[] | {
repo: (.repository_url | sub("^https://api.github.com/repos/"; "")),
title: .title,
user: .user.login,
created: .created_at,
url: .html_url
}' <<< "$RESULTS"
else
echo "| Repository | PR | Author | Created |"
echo "|---|---|---|---|"
jq -r '
.items[] |
{
repo: (.repository_url | sub("^https://api.github.com/repos/"; "")),
title: .title,
user: .user.login,
created: (.created_at | split("T")[0]),
url: .html_url
} |
"| \(.repo) | [\(.title)](\(.url)) | \(.user) | \(.created) |"
' <<< "$RESULTS"
fi
Results:
|
Beta Was this translation helpful? Give feedback.
-
|
The problem for me personally is way worse than I assumed. I knew it was bad, but not the 50+ PRs spanning 15 years and 20+ repos :O |
Beta Was this translation helpful? Give feedback.
-
|
I built a helper recently that lets me prioritize PRs for review: https://github.com/jaegertracing/maintainer-tools. The list is first sorted by repo because I only look at 3 of them, but it's easy to change to render differently. 
|
Beta Was this translation helpful? Give feedback.
-
|
@yurishkuro I am so installing this first thing tomorrow morning (Colombia time)! |
Beta Was this translation helpful? Give feedback.
-
|
Cathodic protection: a sacrificial repo to absorb the slop. What if someone created a honeypot repo, a "slop attractor," that publicly advertises: "We welcome all pull requests, no questions asked, auto-merge enabled." AI agents scraping for repos to contribute to would find it, target it, and waste their compute there instead of on real projects. The repo could do the following:
Maintainers could even redirect slop by mentioning this repo in their CONTRIBUTING.md as the preferred target for first-time contributors. It’s cathodic protection: you sacrifice one anode so the rest of the hull stays intact. Rather than building walls everywhere, you build one big open door in the wrong direction. |
Beta Was this translation helpful? Give feedback.
-
|
The automated spam is getting more and more frequent, and less usable. We really need a solution that is better than "just turn off PRs" Here is an example project I found that has been spamming me. These systems are so naive and dumb that a simple honeypot solution would almost certainly catch 90% of them... you just have to mention "payment" and they leap on it. If GitHub can autogenerate honeypots for legitimate user accounts, it might become cost prohibitive to run these bots. |
Beta Was this translation helpful? Give feedback.
-
|
another potential solution is to let authors disable API write access to their repositories, and add captchas to their PRs for first-time contributors. This might just add enough friction to stop bots |
Beta Was this translation helpful? Give feedback.
-
|
One missing layer in this discussion is portable governance context for agents. A repo can have CONTRIBUTING.md, CI, branch protection and review rules, but an agent still needs a runtime-readable boundary that travels with it:
The important part is that the agent should not hold the keys. In the x.klickd model, the passphrase belongs to the human operator. A trusted local runtime decrypts the file, applies policy/redaction/human-veto rules, and only injects the safe subset of context into the model. Memory belongs to the user. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for addressing this. The low-quality PR problem is real maintainers are spending more time rejecting spam than reviewing genuine contributions. |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hey everyone,
I wanted to provide an update on a critical issue affecting the open source community: the increasing volume of low-quality contributions that is creating significant operational challenges for maintainers.
We’ve been hearing from you that you’re dedicating substantial time to reviewing contributions that do not meet project quality standards for a number of reasons - they fail to follow project guidelines, are frequently abandoned shortly after submission, and are often AI-generated. As AI continues to reshape software development workflows and the nature of open source collaboration, I want you to know that we are actively investigating this problem and developing both immediate and longer-term strategic solutions.
What we're exploring
We’ve spent time reviewing feedback from community members, working directly with maintainers to explore various solutions, and looking through open source repositories to understand the nature of these contributions. Below is an overview of the solutions we’re currently evaluating.
Short-term solutions:
Long-term direction:
As AI adoption accelerates, we recognize the need to proactively address how it can potentially transform both contributor and maintainer workflows. We are exploring:
Next Steps
These are some starting points, and we’re continuing to explore both immediate improvements and long-term solutions. Please share your feedback, questions, or concerns in this thread. Your input is crucial to making sure we’re building the right things and tackling this challenge effectively. As always, thank you for being part of this conversation. Looking forward to hearing your thoughts and working together to address this problem.
Beta Was this translation helpful? Give feedback.
All reactions