OpenAI detector

Question

Screenshot / Code Snippet

About

This script adds a link to each post on SE, revision on SE revision pages, and post Markdown views on metasmoke, to check it against Hugging Face's AI detector.

The first click will test the Markdown text using the Hugging Face detector API for the likelihood it was created by GPT and report that as a percentage likelihood that the post is fake.

Lots of users and spammers edit the text provided by ChatGPT a bit, so if you click the button a second time (or use Alt-Click) an iframe to the detector demo page will open with the Markdown text pre-filled for you, so you can try some manipulations to see if the content was edited from ChatGPT source text. For example, on metasmoke it looks like:

Download

The script can be installed via this link; source code can be found here.

Platform

Tested with Violentmonkey in Firefox on macOS, but it's likely to work on all browsers and platforms. (Please let me know if it doesn't!)

Contact

Post a comment or answer here, or create an issue/pull request on GitHub.

The detector seems to have stopped working for me, should I create a bug report? It flashes an ellipsis briefly when I click the "Detect OpenAI" link, but then nothing. The Javascript console shows "openai-detector:1 Uncaught SyntaxError: Unexpected token 'Y', "Your space"... is not valid JSON at JSON.parse () at receiveOpenAIDetectionDataForButton (:62:25) at userscript.html?name=Stack%2520Exchange%252C%2520OpenAI%2520detector.user.js&id=93dddf2d-810f-485c-aadf-f10370b0623e:604:20" — tripleee, Mar 27 '23 at 08:58
@tripleee right now, the Hugging Face website is temporarily unavailable - https://openai-openai-detector.hf.space/ gives "Your space is on error, check its status on hf.co". — Glorfindel, Mar 27 '23 at 09:01
Heh: Your space is on error ... I had that, too. Ironically, it sounds so much like an AI-generated message. ;-P — Adrian Mole, Mar 28 '23 at 13:05
A very minor complaint; the reverse tab arrow "⭰" (U+2B70 LEFTWARDS TRIANGLE-HEADED ARROW TO BAR) that you're using for the de-indent button doesn't show up for me. "⇤" (U+21E4 LEFTWARDS ARROW TO BAR) would work fine. — miken32, Apr 21 '23 at 19:09
@miken32 fixed in v0.14 - unfortunately, the detector API is down right now (the new version also shows an error message in those cases - I don't know how to show one in the proper style, but I hope someone else can improve that). — Glorfindel, Apr 27 '23 at 11:56

score 4 · Answer 1 · edited Feb 09 '23 at 15:29

4

feature-request status-completed

Based on this comment I replaced the alert message with a normal message. I created a PR (Pull Request) on GitHub.

New look:

Tested on:

Tampermonkey v4.18.1

Chrome Version 108.0.5359.125 (Official Build) (64-bit)

edited Feb 09 '23 at 15:29

doppelgreener

277
1
10

answered Dec 25 '22 at 09:43

4b0

406
2
9

6

Thank you; I thought it would be better to use the standard Stacks notice instead, so I've made an update on my own. – Glorfindel Dec 25 '22 at 11:34
3

@Glorfindel you can use StackExchange.helpers.showToast('message') for simplicity. – double-beep Dec 25 '22 at 11:58
3

@double-beep TIL ... did I miss / forget about some documentation? – Glorfindel Dec 25 '22 at 12:41
Any documented link for all StackExchange.helpers.--- ?? – 4b0 Dec 25 '22 at 12:50
4

@Glorfindel it's not documented anywhere. There are types for some SE functions. – double-beep Dec 25 '22 at 13:22

score 4 · Answer 2 · edited Jan 01 '23 at 17:48

4

feature-request status-completed

Please add the button for new answers as well or posts that have been reloaded in general (because of an edit, for example).

edited Jan 01 '23 at 17:48

Glorfindel

6,772
3
20
46

answered Dec 30 '22 at 11:04

double-beep

681
4
14

1

Fixed in version 0.4 - well, at least for edits, and I'm quite sure it works for new answers as well. Please let me know if it doesn't, and thanks for the suggestion! – Glorfindel Jan 01 '23 at 17:52

score 3 · Answer 3 · edited Jan 21 '23 at 11:35

3

bug status-completed

It seems that comments (and formatting blocks on a post?) containing a link to a post's revision also have a non-working "Detect OpenAI" link appended.

Example:

You can prepend a paragraph with >! (like this) to add a spoiler block, but it only works on questions and answers, not comments :)Detect OpenAI – Aki Tanaka♦ Jan 13 at 15:16

Screenshot

edited Jan 21 '23 at 11:35

Glorfindel

6,772
3
20
46

answered Jan 17 '23 at 15:49

Andrew T.

255
2
8

Another example: https://stackapps.com/revisions/16aeccc4-4344-45a7-aeef-ce3391de7ec3/view-source --> – Andrew T. Jan 17 '23 at 15:50
1

The link has to end in view-source then it gets picked up as a "revision link". It's supposed to only work in the revisions view. – VLAZ Jan 17 '23 at 15:54
@VLAZ ah, I see. Yeah, I guess this is really an edge case since it's very rare for someone to link to the revision's source. – Andrew T. Jan 17 '23 at 16:00
1

At least for me, using the most recent version of the userscript, the button appended to the end of the comment is functional. While this might be unintended, and the formatting could be better, I can see some situations where this could be quite useful. – Makyen Jan 17 '23 at 17:28

score 3 · Answer 4 · edited Jan 23 '23 at 12:18

3

bug status-completed

Looks like this commit broke the script on Tampermonkey on Chrome.
The current version (0.9) doesn't work.

When I click "Detect OpenAI" it throws an error

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'dispatchEvent')

and points to this line.

edited Jan 23 '23 at 12:18

Glorfindel

6,772
3
20
46

answered Jan 21 '23 at 08:29

gre_gor

131
3

I hope Makyen can fix that, my JavaScript is not advanced enough to know the details ... – Glorfindel Jan 21 '23 at 11:43
1

I've added a comment on the PR that introduced the bug. – double-beep Jan 21 '23 at 13:26
2

This should be fixed in version 0.11: https://github.com/Glorfindel83/SE-Userscripts/pull/43 – Glorfindel Jan 23 '23 at 12:18

score 1 · Answer 5 · edited May 12 '23 at 06:39

bug status-completed

I'm using macOS Ventura, Google Chrome, TamperMonkey

Stack Apps says:

OpenAI Detector encountered a problem getting data from https://openai-openai-detector.hf.space/openai-detector. The browser console may have more information.

ntent.js:8 injected: Refused to connect to "https://openai-openai-detector--5smxg.hf.space/openai-detector?%3C!--%20thumbnail%3A%20https%3A%2F%2Fi.stack.imgur.com%2F8Q61x.png%20--%3E%0A%3C!--%20version%3A%200.1%20--%3E%0A%3C!--%20tag%3A%20%5Bmoderation%5D%20--%3E%0A%3C!--%20excerpt%3A%20Adds%20a%20button%20to%20check%20the%20probability%20that%20a%20post%20was%20written%20by%20a%20bot%20--%3E%0A%0A%23%23%20Screenshot%20%2F%20Code%20Snippet%0A%0A%5B!%5B%22Detect%20OpenAI%22%20link%20in%20post%20menu%3B%20notice%20with%20results%5D%5B1%5D%5D%5B1%5D%0A%0A%23%23%20About%0A%0AThis%20script%20adds%20a%20link%20to%20each%20post%20on%20SE%2C%20revision%20on%20SE%20revision%20pages%2C%20and%20post%20Markdown%20views%20on%20metasmoke%2C%20to%20check%20it%20against%20%5BHugging%20Face%27s%20AI%20detector%5D(https%3A%2F%2Fhuggingface.co%2Fopenai-detector).%0A%0AThe%20first%20click%20will%20test%20the%20Markdown%20text%20using%20the%20Hugging%20Face%20detector%20API%20for%20the%20likelihood%20it%20was%20created%20by%20GPT%20and%20report%20that%20as%20a%20percentage%20likelihood%20that%20the%20post%20is%20fake.%0A%0ALots%20of%20users%20and%20spammers%20edit%20the%20text%20provided%20by%20ChatGPT%20a%20bit%2C%20so%20if%20you%20click%20the%20button%20a%20second%20time%20(or%20use%20%3Ckbd%3EAlt%3C%2Fkbd%3E-%3Ckbd%3EClick%3C%2Fkbd%3E)%20an%20iframe%20to%20the%20detector%20demo%20page%20will%20open%20with%20the%20Markdown%20text%20pre-filled%20for%20you%2C%20so%20you%20can%20try%20some%20manipulations%20to%20see%20if%20the%20content%20was%20edited%20from%20ChatGPT%20source%20text.%20For%20example%2C%20on%20metasmoke%20it%20looks%20like%3A%0A%0A%5B!%5BAnimation%20of%20a%20post%20on%20metasmoke.%20First%20the%20base%20%25%20FAKE%20is%20determined%20of%2024.94%25%2C%20then%20an%20iframe%20is%20opened%20showing%20that%20after%20removal%20of%20the%20spammer%27s%20added%20text%20the%20detection%20is%2099.86%25%20FAKE%5D(https%3A%2F%2Fi.stack.imgur.com%2F5wyoP.gif)%5D(https%3A%2F%2Fi.stack.imgur.com%2F5wyoP.gif)%0A%0A%0A%0A%23%23%23%20Download%0A%0AThe%20script%20can%20be%20installed%20via%20%5Bthis%20link%5D(https%3A%2F%2Fgithub.com%2FGlorfindel83%2FSE-Userscripts%2Fraw%2Fmaster%2Fopenai-detector%2Fopenai-detector.user.js)%3B%20source%20code%20can%20be%20found%20%5Bhere%5D(https%3A%2F%2Fgithub.com%2FGlorfindel83%2FSE-Userscripts%2Fblob%2Fmaster%2Fopenai-detector%2Fopenai-detector.user.js).%0A%0A%23%23%23%20Platform%0A%0ATested%20with%20Violentmonkey%20in%20Firefox%20on%20macOS%2C%20but%20it%27s%20likely%20to%20work%20on%20all%20browsers%20and%20platforms.%20(Please%20let%20me%20know%20if%20it%20doesn%27t!)%0A%0A%23%23%20Contact%0A%0APost%20a%20comment%20or%20answer%20here%2C%20or%20create%20an%20issue%2Fpull%20request%20on%20GitHub.%0A%0A%0A%20%20%5B1%5D%3A%20https%3A%2F%2Fi.stack.imgur.com%2FmDMTr.png": Request was redirected to a not whitelisted URL
Mt @ content.js:8
userscript.html?name=Stack%2520Exchange%252C%2520OpenAI%2520detector.user.js&id=16694645-97d1-4f98-983b-63d22e4f6426:1311 OpenAI Detector error response: Object
(anonymous) @ userscript.html?name=Stack%2520Exchange%252C%2520OpenAI%2520detector.user.js&id=16694645-97d1-4f98-983b-63d22e4f6426:1311
VM90:75 Uncaught TypeError: Cannot read properties of null (reading 'fake_probability')
    at receiveOpenAIDetectionDataForButton (<anonymous>:75:39)
    at userscript.html?name=Stack%2520Exchange%252C%2520OpenAI%2520detector.user.js&id=16694645-97d1-4f98-983b-63d22e4f6426:1277:20

Workaround suggested by script author:

replace

@connect     openai-openai-detector.hf.space

by

@connect     *

Confirmed this still happens today, if we could get this fixed that'd be great. — Thomas Ward, May 08 '23 at 05:56
@ThomasWard This should be resolved with version 0.15, which is now available.. — Makyen, May 10 '23 at 22:05

score 1 · Answer 6 · answered May 28 '23 at 16:54

1

feature-request

I'd like to be able to scan only part of a post for OpenAI detection, perhaps via using normal text selection on the post & hitting a scan button.

In that event, the script could use window.getSelection() to get the text to check.

A recent post was almost certainly GPT output—it had all the hallmarks—but the answer was written to lead with a paragraph representing the prompt they gave to the generator. The full answer had 66% confidence. The bit after the prompt was 98.5% confidence once I copied it over to another GPT detector, which is more what I was expecting to see.

answered May 28 '23 at 16:54

doppelgreener

277
1
10

1

If you click "Detect OpenAI" twice (once to run the regular detection, then again) underneath the post you'd get an extra area open up which allows you to edit what text gets checked. Also has some convenience buttons to strip away some formatting but you can also freely edit it and cut out any text you are not interested in. – VLAZ May 28 '23 at 17:00
@VLAZ Oh hey, I've never noticed that! That's pretty good and will do the trick here. – doppelgreener May 28 '23 at 17:03

score 0 · Answer 7 · answered Jun 08 '23 at 03:30

feature-request

Please make this user script work on election pages.

Currently, the nomination phase of the 2023 Web Applications Moderator Election is running. The link is shown, but apparently, nothing happens.

Looking at the web browser's JavaScript console, the following error message is logged:

Uncaught TypeError: Cannot read properties of undefined (reading 'href')
    at HTMLAnchorElement.handlePostMenuButtonClick (<anonymous>:393:37)
    at HTMLAnchorElement.dispatch (jquery.min.js:3:12444)
    at r.handle (jquery.min.js:3:9173)

OpenAI detector

Screenshot / Code Snippet

About

Download

Platform

Contact

7 Answers7