Google’s rolling out of Gemini AI chatbot to kids is risky

GOOGLE has announced it will roll out its Gemini artificial intelligence (AI) chatbot to children under the age of 13.

Advertisement

While the launch starts within the next week in the US and Canada, it will launch in Australia later this year. The chatbot will only be available to people via Google's Family Link accounts.

But this development comes with major risks. It also highlights how, even if children are banned from social media, parents will still have to play a game of whack-a-mole with new technologies as they try to keep their children safe. A good way to address this would be to urgently implement a digital duty of care for big tech companies such as Google.

Google's Family Link accounts allow parents to control access to content and apps, such as YouTube.

To create a child's account, parents provide personal details, including the child's name and date of birth. This may raise privacy concerns for parents concerned about data breaches, but Google says children's data when using the system will not be used to train the AI system.

Chatbot access will be "on" by default, so parents need to actively turn the feature off to restrict access. Young children will be able to prompt the chatbot for text responses, or to create images, which are generated by the system.

Google acknowledges the system may "make mistakes." So, assessment of the quality and trustworthiness of content is needed. Chatbots can make up information (known as "hallucinating"); so, if children use the chatbot for homework help, they need to check facts with reliable sources.

Google and other search engines retrieve original materials for people to review. A student can read news articles, magazines and other sources when writing up an assignment.

Generative AI tools are not the same as search engines. AI tools look for patterns in source material and create new text responses (or images) based on the query — or "prompt" — a person provides. A child could ask the system to "draw a cat" and the system will scan for patterns in the data of what a cat looks like (such as whiskers, pointy ears and a long tail) and generate an image that includes those cat-like details.

Understanding the differences between materials retrieved in a Google search and content generated by an AI tool will be challenging for young children. Studies show even adults can be deceived by AI tools. And even highly skilled professionals — such as lawyers — have reportedly been fooled into using fake content generated by ChatGPT and other chatbots.

Google says the system will include "built-in safeguards designed to prevent the generation of inappropriate or unsafe content."

However, these safeguards could create new problems. For example, if particular words (such as ‘breasts’) are restricted to protect children from accessing inappropriate sexual content, this could mistakenly also exclude children from accessing age-appropriate content about bodily changes during puberty.

Many children are also very tech-savvy, often with well-developed skills for navigating apps and getting around system controls. Parents cannot rely exclusively on inbuilt safeguards. They need to review generated content and help their children understand how the system works, and assess whether content is accurate.

The eSafety Commission has issued an online safety advisory on the potential risk of AI chatbots, including those designed to simulate personal relationships, particularly for young children.

The eSafety advisory explains that AI companions can "share harmful content, distort reality and give advice that is dangerous." The advisory highlights the risks for young children, in particular, who "are still developing the critical thinking and life skills needed to understand how they can be misguided or manipulated by computer programs, and what to do about it."

My research team has recently examined a range of AI chatbots, such as ChatGPT, Replika and Tessa. We found these systems mirror people's interactions based on the many unwritten rules that govern social behaviour — or, what are known as "feeling rules". These rules are what lead us to say "thank you" when someone holds the door open for us or "I'm sorry!" when you bump into someone on the street.

By mimicking these and other social niceties, these systems are designed to gain our trust.

These human-like interactions will be confusing, and potentially risky, for young children. They may believe content can be trusted, even when the chatbot is responding with fake information. And, they may believe they are engaging with a real person, rather than a machine.

This rollout is happening at a crucial time in Australia, as children under 16 will be banned from holding social media accounts in December. Children — and parents — must be educated in how all types of digital tools can be used appropriately and safely.

As Gemini's AI chatbot is not a social media tool, it will fall outside Australia's ban. Parents must keep up with new tool developments and understand the potential risks their children face.

This highlights the urgent need to revisit Australia's proposed digital-duty-of-care legislation. While the European Union and UK launched digital-duty-of-care legislation in 2023, Australia's has been on hold since November 2024. This legislation would hold technology companies to account by legislating that they deal with harmful content, at source, to protect everyone.

Courtesy: The Conversation

Lisa M Given is Professor, Information Sciences, RMIT University.