Why AI is a change management challenge, not a technology one 

Everything you need to know about AI right now you can learn from chocolate — specifically an ad from the Canadian arm of KitKat.  

Playing on their longstanding ‘have a break’ tagline, the company’s advertising team asked AI to complete a range of tasks and then repeated the process, this time instructing the AI to have a break before solving the problems.  

The result (at least according to the ad) was improved performance and better outputs across the board.  

Now while that might be a cute gimmick for the chocolate maker, it points to a much bigger truth facing all businesses right now: AI is weird.  

In a world where we are used to working with software that runs on rails — responding in predictable ways to actions, even when the user makes a mistake — generative AI is just a strange cat.  

It’s not human, but it is so closely based on our own quirks and characteristics, that it can act indistinguishably from us … even when we don’t want it to.  

The AI ad by KitKat was prompted by work conducted by researchers from Google’s DeepMind, who have tested a wide range of human phrases in trying to get the best out of AI systems.  

For example, when giving a range of AI tools some maths problems …  

“The researchers discovered that the prompt ‘Take a deep breath and work on this problem step by step’ was most effective with Google’s PaLM 2 language model. This phrase reached the highest accuracy score of 80.2% when tested against GSM8K, a dataset of grade-school math word problems. In comparison, PaLM 2, without any special prompting, achieved only a 34% accuracy on GSM8K, while the classic prompt ‘Let’s think step by step’ reached a 71.8% accuracy score.” 

The excellent Professor Ethan Mollick from Wharton has written extensively on his personal experiments, finding that by chivvying AI along or reminding it that things are possible, you can get it to perform far better than by just accepting its first response.  

Take this example below, where the power of a large language model can be applied to a problem that it thought defied its programming capacity, just by telling it to have a crack.

So far, so positive. The lesson you might take away from this is that you just need to sprinkle some words of encouragement through your AI prompt and hey presto.  

But sometimes AI is, frankly, just not interested.  

Last November, after regular users started to complain that ChatGPT was becoming a little lacklustre, OpenAI was forced to confront claims its flagship AI was becoming “lazy”. (It has rolled out an anti-lazy update in recent weeks).  

In response to dealing with a sulky and petulant AI, people tested a range of hypotheses, including that it was exhibiting signs of seasonal depression, dropping productivity and the quality of its work in the same way many humans do during the colder months.  

The cures were equally wacky.  

Telling ChatGPT it was May, not December, improved its performance on coding.  Telling it the user would give it a tip for its work, encouraged it to provide longer, more detailed answers. Other users have told it that they need help because “coding makes my eyes hurt” or “I have lost all my fingers”. Simply reminding it that accuracy is really, really, personally important to the user seems to have an impact as well. 

What all this mean for businesses trying to get a handle on implementing AI in 2024 is that the tech is not the challenge — it is the people in your office who need to be able to manage a co-worker who can be brilliant but inconstant, enthusiastic or apathetic, and swayed by the phrasing of its human masters.  

We say at Purple a lot that the AI revolution requires humans to be trained in better communication, and this has never been more true.  

For your team, understanding what generative AI is and how it works is only the beginning. The choice of platform is important but not the real driver of success.  

To get the best out of AI, you need everyone from the pinnacle of your organisation to the bottom to be skilled in the right way to think about AI, including an awareness of its extraordinary quirks.  

They need the management skills necessary to delegate, instruct and chivvy-along AI on tasks, the judgement and critical thinking skills needed to identify good and reject poor performance, and the feedback skills required to coach this weird but powerful technology on different tasks.  

In short, it’s a change management challenge, and organisations that assume they can get the best results simply by focusing on the tech are missing a trick.   

Can we help?  

After our successful AI Readiness webinar series last year we are now working directly with clients to train up teams in the use of AI, both on Copilot and ChatGPT, as well as supporting organisations in development of good governance policies and risk mitigation.  

If you would like a confidential discussion about what we do and how, please get in touch with Ruth on rcallaghan@canningspurple.com.au.