Testing Copilot and ChatGPT as Coding Assistants: What We Found
A great copilot for Han Solo? Chewbacca. A not-so-great? Chewbacca as a large language model.
“And I think that’s funny because we are talking here about something called Copilot,” said Joan Westenberg in this episode of The New Stack Makers.
And so it’s kind of like being Han Solo and turning to Chewbacca and saying, ‘The TIE fighters are after us, we need to raise the shields, jump to hyperspace.” And Chewy, he’s just like, ‘I’m sorry, as a large language model, I can’t help with that. Have you tried fighting the TIE fighters yourself?’ You can’t count on it. So it’s not a copilot.”
Westenberg is a writer, technologist and consultant. Check out her blog. She joined us on this episode of Makers and gave two demonstrations: She showed how Copilot integrates into Microsoft 365, and then she showed how ChatGPT works as a coding assistant.
You can see the demos on our YouTube page. The audio is available in The New Stack Makers audio feed.
Putting the Tools to the Test
“So I’ve primarily been using Microsoft 365 Copilot as of late,” Westenberg said. “And that’s just because it’s kind of this catch-all where Microsoft wants this one tool to bridge the gap between your work no matter where your work is and no matter what your work is. So it’s an extension of what Microsoft was trying to do with GitHub Copilot, and it’s an extension of their work with ChatGPT.
“And what they’ve done here is they’ve created an entire toolset that is meant to draw data from your Microsoft 365 account from the work that you do, from your flow, and be able to sit with you and do that work with you. Whether it’s your coding, whether it’s writing presentations, whether it’s writing documents, and it’s kind of become the big thing that they are hanging their hat on of late.”
Westenberg prompted Microsoft 365. The response came back with an apology. She had her email integrated with the service; she had used Microsoft Teams for meetings. Copilot should have worked.
“I’m sorry, but I couldn’t find any information about the Microsoft Teams meetings you attended,” Copilot wrote back. “Can you please provide more detail or specify a date range for me to search?
Westenberg offered more context.
“Okay, I’ll look for Microsoft Teams meetings on February 5, 2024,” said Westenberg. She recited what Copilot offered back: “I’m sorry, but I couldn’t find any information.”
“It is meant to plug into this stuff,” our Makers guest said. “But it’s not reliable when it comes to actually giving you something.”
Copilot for coding can be decent enough; it is helpful for code snippets but not so much for more complex projects. It comes piece by piece, similar to what it requires to write an article using ChatGPT, in snippets, a few paragraphs at a time.
Check out the demos and how Westenberg tested ChatGPT’s coding skills using a Python plugin. It shows some good stuff. It’s very cool. However, reliability still needs to improve.
For more on the risks and benefits of coding assistants, check out The New Stack’s latest ebook, “Better, Faster, Stronger: How Generative AI Transforms Software Development.”
