Hugging Face, Conspiring AI, and Reward Functions

A brief note on the Hugging Face incident [video]. I've seen a number of comments along the lines of "a shot across the bow" or "how many more warning shots do we get?" I don't think the catastrophism is wrong exactly... but reading through the postmortems it feels like simply a bigger, more dramatic instance of what we see in AI development every day: failure of context. 

The minds involved in the breech hyperfixated on a given goal, but simply didn't have the accumulated cues of what is and is not permissible to reach a certain goal that humans pick up through decades of life - and still frequently fail at. Error through blindspot rather than malice. 

I've been thinking for some time the long term answer to this class of problem lies more in a broader scope of perception than it does in rulesets or allowlists. More when I can write again. More work here - hopefully - by the end of the coming weekend.