Following a quip on LinkedIn about the introduction of Recall in Copilot+ and Kevin Beaumont’s great piece about why this is a really bad idea from a security point of view, I got thinking:

This is dystopian techno-fascism

Kevin posited a disconnect in Microsoft that led to the creation of this feature and whether people really wanted it. Personally, I’ve got a brain like a sieve and would not know the command line without ~/.bash_history but storing screenshots of all your interaction with a computer… I just don’t like the idea.

Connecting the dots

• Remember the McKinsey article on measuring developer productivity?
• Remember the Productivity Paranoia about returning to the office?
• Remember the Great Amazon Heist

Then I started thinking about the upper management’s “real” concern about ensuring:

• 100% utilisation of developers
• Time-tracking every move
• Ranking/Laddering people to weed out the worst performers
• Making sure that any waste is eliminated

So what if the ultimate goal is to create a system where software development (and this could be as easily be any office work) is to ensure that every ounce of freedom and autonomy is squeezed out of workers. Kind of an “Amazonification of office work”, where an AI model checks what you are doing second of the day?

Prepare to get a final written warning as soon as the model detects:

• You’re spending too much time on BBC news, or LinkedIn
• You’ve sent a message making fun of the boss over a private chat channel
• You used a non-company approved tool to make your life easier
• You chatted about your poorly kitten
• You talked about joining a union (probably instant dismissal this one)

A Control Freak’s Wet Dream

How far-fetched is this dystopian nightmare? Cory Doctorow’s Enshittification pretty much describes that the big tech companies are more than ready to throw ethics out the window time and time again.

So it’s not a good idea?

Studies have shown that Psychological Safety is the best indicator for performant teams. I liked this take too: in a nutshell, Psychological Safety isn’t about being fluffy and nice and avoiding conflict but

The belief that the work environment is safe for interpersonal risk taking

What’s the best way of removing any appetite for risk taking?

• Remove autonomy
• Ensure that people always feel like they’re being watched
• Punish/fire them

Conclusion

Am I saying that I know that Microsoft is going down this path intentionally? Well, I don’t know. I’d love to give them the benefit of the doubt. But considering how layoff-happy some tech giants have gotten in the pursuit of short-term good news for the stock market, it makes you wonder.

Science Fiction always promised that Artificial Intelligence would take the drudgery out of life. Let’s hope that it won’t turn out that AI doesn’t just ensure that the drudgery is all we have left. Who knows, maybe it won’t matter anyway, when nobody can believe anything anymore and any content is AI-generated.

Tags my-take-on security generative-ai

If you'd like to find more of my writing, why not follow me on Bluesky or Mastodon?