Who's checking the AI?
Trust runs through the whole issue: who checks the AI's output, who owns the strategy, and how far should we trust it. Plus a bank's AI lead on getting it into production.
TECH TEAMS TUESDAY · A Building Tech Teams publication
00 · BONUS INTRO
Spoiler, the theme of this newsletter is around trust. And I wanted to share that Substack this week announced a partnership with Pangram which allows users to see how much of a Substack newsletter is estimated to be written by hand or with the assistance of AI.
I’m now five weeks into my newsletter writing journey and I thought form the start, if I’m going to write a newsletter it needs to be mine, and I’m proud to share all the below is mine, it’s my opinion and writing style.
To be clear I use AI to do research each week on articles that could be relevant for me across the web and many email subscriptions I subscribe to - but the selection of them and then writing how they relate to Building Tech Teams is all me. Here’s my Pangram assessment:
01 · THE PATTERN
The theme from this issue was the same theme of a conversation I had with my daughter this week. She’s currently writing a speech for school and her topic is “Should kids be allowed to use AI for homework”.
Her opinion was an obvious yes, because “we’ll use it in the real world”, and as much as I like AI, I did question her on “what happens if the AI is wrong”, “do you think if you use AI for everything, then your brain will turn to mush”, and my belief that school is for learning how to learn, and one of the best skills she can learn at school that will continue to be valuable as she gets older is problem solving, and that shouldn’t all be outsourced to AI.
I am currently awaiting her edited re-write.
And the reason that’s relevant is three of the moves I read about this week, and the podcast conversation I recorded all covered similar themes.
What are we doing to review or test AI output?
How far should we trust AI?
What checks, reviews, testing should we implement on AI output?
Patrick McQuaid, who runs data and AI inside a bank and is on this week’s episode of the podcast, made the point; you keep experts around precisely because someone has to verify the machine is right.
It’s an important conversation as building trust is crucial to building great tech teams, and if our tech teams of the future are a blend of human and agent, the theme of trust only becomes more important between human and AI.
02 · THE MOVES
1. Writing code got cheap and fast. Now engineers are burning out on reviews and testing.
Gergely Orosz, who writes The Pragmatic Engineer newsletter, says one thing that’s top of mind for engineering leaders is no longer the speed at which their engineering teams can produce code, but the testing and reviews needed by the developers. He said better models started writing more of the code around January, and that the bottleneck in building software had moved from coding to the review phase. (The Pragmatic Engineer)
There are new review tools being released each month, they are getting better, and some companies are building their own internal tools and practices. But the worry among leaders is a common one, how far does the testing need to go? One of the most worrisome quotes from the newsletter was: “devs see others as no longer able to review code with intent, whereby, if the AI code review has no real comments, they just approve it. Meanwhile, those devs who put the same effort and energy into code review as before feel overloaded by AI slop PRs sent their way.”
Given we are still only six months on from when many people say the models got to a point where they could output quality code, and most companies are still building out their AI software development practices, I’m sure we will get to a standard of what quality reviewing and testing looks like. But like many things at the moment in AI, we are still in a testing and learning phase, and most companies don’t have this problem completely solved.
For leaders: burnout is becoming a common theme with AI and engineering. Software Engineering leaders should be staying abreast of best practices and the tools being highly reviewed when it comes to code review and testing. Fixing theis bottleneck before it becomes an issue and burning out your engineering team should be a high priority.
For tech professionals: I think the theme we are hearing more and more with software engineering with the new tooling is that good software development principles still stand, and maybe even more so in the age of AI. Commenting, code reviews, testing, all just as important now as previously.
2. AI is making people more sure of themselves but not more right.
A group of European researchers set out to see how access to AI advice affects people’s willingness to admit ignorance. The title of their paper reveals their findings: “AI advice suppresses people’s willingness to say ‘I don’t know’, even when the advice is wrong and accuracy is incentivized.”(The Register)
In the study, one group had to answer the questions without AI advice, and another group could ask the AI for advice. Some of the key findings:
People with AI won’t admit they don’t know an answer:
Group without AI: 44% of people responded that they didn’t know the answer
Group with AI: 3% of people responded that they didn’t know the answer
People trust AI more than their own judgment:
Group without AI: 27% of people gave the correct answer
Group with AI: only 9% of people gave the correct answer. So some would-be correct people asked for AI advice and became wrong.
Access to AI advice made people more confident that they were correct.
Group without AI: 30% confidence levels in their answers
Group with AI: 76% confidence levels in the answers
In a time where more and more people are using AI for knowledge work, and more companies are relying on the output of AI for decision making, the importance of the the right reviews and testing (like in Move 1) matter even more.
For leaders: the real risk in AI-heavy teams is that human judgment is no longer considered, or humans no longer have an opinion and use AI for all decision making. The ability to ask the right questions, have healthy debate, and have the right reviews and testing in place (whether building software or using in other parts of the business) is vitally important.
For tech professionals: like I mentioned in a previous newsletter, the people that are getting ahead and being offered the largest paychecks are the ones who are mixing their knowledge and the new tooling. Defaulting to AI alone won’t move you ahead on the corporate ladder.
3. Trust expands outside of just model answers to who owns the technology (and our National strategy)
A survey of just over a thousand Australians, taken the day after the government’s AI announcement this month asked Australians specifically whether we should build our own AI capability in Australia or rely on tools built overseas. 37% want us to build it ourselves, even if that means moving slower. And an additional 43% want a mix of Australian built and those from overseas.
Natalie Ashes says when you remove the undecided you get “92% of Australians who want local capability to be part of the answer. Only 7% are happy to rent it from another country.” (Startup Daily)
There are obvious financial and economic benefits for Australia to build and own its own AI tools and infrastructure but I can’t help but to think that trust is an element on many Australian’s minds when it comes to who owns their data, and who is benefiting from the data.
On the topic of trust the survey results showed only 43% of Australians trust the government to get the AI rules right. And when you look at a question around speed vs caution, the trust aspect comes up again, with Australia being very cautious, with only 12% prioritising speed over caution.
Every month we are seeing new models being delivered, AI companies are growing fastest then ever, and there is a realistic fear that if we don’t make moves that we run the risk of missing out. But trust is showing up across the board, from who’s building and owning the technology, to who’s designing and executing on the strategy. Finding the balance between staying relevant and building and maintaining trust is a very real challenge.
(Disclosure: the survey data was commissioned by WorkClone, and I wanted to acknowledge that they are a company I have invested in. The article was published in StartupDaily, with the survey being run by Primara Research - nothing changes the findings but I thought it was important to acknowledge).
For leaders: a great starting point for many companies can be an AI Policy. Name where you use AI in the company and processes, name what data is shared with AI models, getting ahead of the conversation can be a surefire way to build trust with your employees and customers.
For tech professionals: help build strong AI processes internally, help set the right strategic, security and governance frameworks. As trust and security become a bigger conversation area, those close to this space become even more valuable.
03 · FROM THE INDEX
One read a week from the AI Jobs Index, the live tracker I run at aijobsindex.com.au.
The number I built the AI Jobs Index for. It has logged 3,610 Australian jobs cut in announcements where the company itself named AI as the driver.
There are, however, a total of 11,934 workforce reductions where AI was cited alongside other factors. I have built the methodology of the site to rank these two numbers separately where there are other contributing factors, or lack of company cited reasoning.
The trust aspect has been brought up many times however with redundancies that I have tracked on the site, and those that are happening in smaller companies without announcements. Because of the negative press that can be brought about with admitting AI is a reason for redundancies there are companies that have used other excuses, including organisational restructures, to avoid the conversation. The people inside these organisations know the truth, and events like this can be the reason for distrust within the company.
04 · PEOPLE: COMPANIES ARE MOVING ON THE SENIOR AI SEAT
The trend is continuing in Australia for top companies filling senior positions with AI expertise. And the people in demand are those who have been ahead of the curve and already shipped and managed AI in enterprise environments.
This week, the listed Australian tech company SiteMinder, the Sydney-founded hotel commerce platform, named Samantha Lawson its Chief Product Officer, starting August 1st. Lawson comes from a strong AI and product background, with her current role being Chief Product Officer at an AI-powered banking platform, and previous to that she led Gen AI at Optus. SiteMinder positioned the appointment squarely around AI reshaping its business. (ASX announcement)
The read from the recruiting seat: these roles are very competitive with a small pool of talent having run significant AI programs at enterprise level. Previously I’ve been writing about the big banks and their senior AI hires, now it’s happening across other listed ASX companies. I expect the trend to continue as companies look to build strategy and execute around AI.
05 · WHAT I’M BUILDING
This week has been a lot of swearing at my Claude Code, my Cowork and general frustration. I am very honest with my level of technical ability, I know enough to be dangerous, I have spoken with tech professionals daily for ten years, I understand at a reasonably technical level what good software looks like, but I am finding my limitations.
This week has been plugging holes, fixing broken pipelines, auditing and optimising my setup. Pretty much many of the themes of this newsletter today, me putting a lot of trust in Claude, mistakes happening, me being upset, then having to rebuild skills and processes.
I also came across this post from Tayla Burrell on getting the most from Fable, and I used an edited version of her prompt to Audit and Optimise my setup. The prompt started:
Act as a senior systems auditor doing a whole-system review of my AI Operating System. I want one thing: a ranked findings report, highest-leverage first.
I believe this entire process I’m working through is called iteration, and it’s part of growth and me building better systems for myself, but Claude broke my trust this week and was threatened many times this week with me walking away to OpenAI.
06 · THIS WEEK ON THE PODCAST
Patrick McQuaid runs data, AI and analytics at NGM Group, Australia’s largest customer-owned bank. He has spent a career leading technical teams without being the most technical person in the room, across data, digital and analytics roles at NAB, Telstra and in consulting. His method is simple: get the experts around a table, ask the right questions, and let the answer fall out.
We get into the part most companies are stuck on. Moving AI out of proof-of-concepts and into production inside a highly regulated bank, where risk and privacy are paramount. With AI being the thing everyone is talking about, Patrick shares that to be successful you need to “Get the data right first. Nothing works with AI until the boring data foundations are in place”.
Some of the most interesting parts of the conversation:
- Why “if you are arguing about which model to use, you are having the wrong conversation” is his test for whether a team is serious.
- Getting AI into production in a regulated bank, and why the human in the loop turned out to be a thicker loop than anyone expected.
- Who he hires: a few people with real depth, paired with high-attitude people who are curious and will have a go, over chasing scarce AI specialists.
- The three skills he does not think AI can take: verifying the work, storytelling, and building the relationships that get anything shipped.
Why the conversation matters: it is the same thread as this whole issue, in a highly regulated bank, trust and security of data are not optional.
Listen: YouTube · Apple · Spotify
07 · THE ASK
What would you like to see more (or less) of in the newsletter?
Cheers,
James
Tech Teams Tuesday is the weekly read from Building Tech Teams. Written by James MacDonald, MD of NTP Talent. The AI Jobs Index is a Building Tech Teams publication.