It’s NOT time to abandon online data pools! However, we should screen carefully

Show Post

+1. Eerke, I can only speak for Prolific, but the team here is full of smart, motivated people working extremely hard to maintain and improve the integrity and quality of our platform for running online research. The misuse of AI tools is a threat, but one that can be protected against.

Show Post

I trust you're all smart, highly motivated, and hard working. I don't believe you can reliably detect AI long term as it is an arms race with plenty of adversarial training possibilities. Some research I've seen suggests that even a fairly low false negative rate already distorts outcomes too much.

Show Post

I’m confident that this is addressable with sufficient innovation/investment personally. May require some significant changes in how we run projects though. E.g. controlled envs, multimodal tooling beyond text, high confidence auditing back to identified humans, etc

Show Post

That sounds like changing the rules of the game, and I agree you'd need to. Privacy and autonomy risks from controlled environments (like with "proctoring" in online assessment) and any other additional surveillance, identifying humans means loss of anonymity ...

Show Post