Shadow AI: Closing the Gap Between AI Adoption and Governance

Shadow AI is no longer just a productivity or shadow IT issue. In mid-size organisations, AI-built applications and agents are creating a widening gap between adoption and oversight. Governance is the mechanism that allows organisations to close that gap without suppressing innovation.

Artificial Intelligence is now widely regarded as a driver of productivity and insight. But the rapid adoption of AI tools is also creating a growing governance and security risk. This is especially concerning in mid-size enterprises, where IT departments are often limited in the tooling, capacity and capability required to oversee AI usage. Meanwhile, departments are using Shadow AI tools to rapidly innovate. While shadow IT is not a new concept, AI is increasing the speed and proliferation of unsanctioned software solutions.

The result is a widening gap between adoption and oversight. That gap is where risk begins to concentrate. CIOs and Tech Chiefs must manage this tension carefully to promote productivity while managing risk.

In production environments, poorly governed agentic systems can contribute to privacy and data breaches, cybersecurity events, intellectual property leakage, process drift, hallucinations and quality failures. In more serious scenarios, these risks can erode customer trust, degrade products or services, and expose directors to legal and compliance concerns.

AI need not be an ungoverned black box - but closing this gap requires deliberate governance, and that's where this article lands.

When citizen development of AI-built apps becomes risk

Low-code and no-code platforms such as Microsoft Power Platform, Make and Zapier have already given business user or citizen developers the ability to build basic web and mobile-enabled applications, often without direct involvement from skilled developers. The rapid development of new digital tools abound in sales, finance and operations departments, catering to a range of data input and processing demands.

Creation of digital tools and applications using generative AI tools such as Claude or ChatGPT introduces a completely new paradigm in business user development. But where AI-assisted Software Delivery Lifecycle (AI-SDLC) processes are absent, the risk profile changes materially compared with traditional low-code and no-code development.

A lack of observability and explainability in AI-built systems presents a material risk for enterprises. If a business user or citizen developer cannot describe the functionality of an application at a code level, it should not be treated as production-ready, scalable or secure. This is not the same as the finance department building their own smart Excel tools with VB macros. In fact, the blast radius of an agentic application may be far greater due to the uncontrolled intelligence of the application, integration or digital workflow.

Recently I worked with an international organisation that tasked an employee with vibe-coding an application for international users to report and collaborate on infrastructure projects via a web portal. Built with Replit, the application was deemed not production-ready and required significant additional investment to refactor the code and remediate a significant number of code-level defects and cyber security vulnerabilities.

The AI success story delusion

When I have canvassed colleagues or customers in sub-enterprise businesses to share their successes with AI - the response is always fascinating. While many executives recognise the vast usage of often unsanctioned shadow AI applications such as ChatGPT or Claude, responses range from a lack of awareness or ambivalence through to a clear recognition of risk. It's in the conversations with the former cohort - ambivalent or unaware - where I'll often hear the most concerning stories:

• a Victorian local government representative shared that they uploaded sensitive financial data into Claude outputting "great results";
• a risk leader uploading customer bank statements into Claude to assess credit risk; and
• operational staff using ChatGPT to assist with admin-level configuration of a production ERP during a transformation project.

These examples are often presented internally as innovation success stories. In reality, they may point to serious control failures.

Take the example of sensitive financial data being uploaded into Claude. Under ISO/IEC 42001:2023, this raises immediate questions about whether the organisation has an accountable AI management system in place: who approved the use case, what data was authorised for processing, whether the supplier was assessed, whether the AI system was risk-assessed, and whether operational controls existed before sensitive information was submitted to a third-party model. If those questions cannot be answered, the issue is not merely poor judgement. It is evidence that AI use is occurring outside the organisation's management system.

The NIST AI Risk Management Framework would frame the same example through its Govern, Map, Measure and Manage functions. Before sensitive financial data is processed by an AI tool, the organisation should understand the context of use, identify affected stakeholders, map data flows, assess privacy and security impacts, measure reliability and potential harms, and manage residual risk. Uploading financial data into an external AI service without those steps creates foreseeable exposure around confidentiality, data leakage, inappropriate secondary use, inaccurate outputs and lack of auditability.

For Australian organisations, the more immediate legal exposure sits closer to home. The Privacy Act 1988 (Cth) and the Australian Privacy Principles govern how personal information is collected, used, disclosed and secured, including disclosure to overseas recipients and the obligation to take reasonable steps to protect it. Public sector entities carry further obligations under state privacy, information security and records legislation. An employee pasting sensitive financial information into a consumer AI tool may leave the organisation unable to demonstrate compliance with any of them.

The credit-risk example is even more serious. Under the EU AI Act, for example, AI systems used to evaluate creditworthiness or establish credit scores are treated as high-risk, unless used only to detect financial fraud. That classification brings obligations around risk management, data governance, technical documentation, logging, transparency, human oversight, accuracy, robustness and cybersecurity. A risk leader using a general-purpose AI tool to assess customer bank statements may therefore create risk across several dimensions: an unapproved high-impact decision process, insufficient explainability, possible unfair or discriminatory outcomes, inadequate record-keeping, and no clear evidence that the tool is suitable for the purpose.

Most Australian mid-size organisations will not be directly captured by the EU AI Act. But it matters because it signals where global expectations are heading, and because Australian customers, insurers, regulators and enterprise procurement teams are increasingly asking for the same evidence: documented risk assessment, human oversight and a clear record of how AI-assisted decisions were made.

The ERP configuration example creates a different but equally material risk. If ChatGPT is being used to guide admin-level configuration of a production ERP, the business may lose traceability over why changes were made, whether generated guidance was correct, whether sensitive configuration data was exposed, and whether standard change-management controls were bypassed.

These are not abstract compliance concerns. They go directly to privacy, confidentiality, cybersecurity, auditability, fairness, operational control and director oversight. The common failure across all three examples is not simply that AI was used. It is that AI was used without evidence of approval, risk assessment, data classification, human oversight, vendor assurance, logging, monitoring or accountability.

Managing the risk: governed innovation

With teams building their own AI-generated productivity tools, we are in the ironic position of witnessing our teams innovate faster than ever. Is this what we've always sought from our teams? Creativity? Innovation? Productivity?

But the issue isn't the act of innovation or experimentation itself. The issue is that innovation is occurring without a clear view of data flows, model behaviour, vendor terms, human oversight, security controls or accountability. The aforementioned stories should not be dismissed as harmless experimentation. They are often early warning signs that AI adoption has moved faster than governance.

Throughout my career I've sat at the intersection of technology and its translation to value generation, innovation and the natural tension with governance. With exposure to tens of business models across literally hundreds of customers, the same themes are evident in successful businesses: curiosity, empathy, creativity and attention to the detail. These are virtues we should harness and promote rather than be stifled by red tape. So, where I see the real opportunity for medium-size businesses to thrive in increasingly competitive global markets is in effective governance.

Governance should be seen as a facilitator and enabler of innovation. A clear governance framework, delivery strategy and tooling in our AI-enabled commercial world gives teams the freedom to build useful tools while operating within agreed guardrails for security, privacy, transparency and accountability.

From AI adoption to board-level assurance

The right digital governance tools can be deployed to not only provide the compliance frameworks to set these essential operating guardrails, but also to assess pre-production AI-built systems in coordination with cybersecurity tooling, integrate with observability tools (tools that observe exactly how agents work including ingress/egress of data), and deliver much-needed monitoring and transparency to IT teams to assure oversight. Add alignment with Australian, ISO or other international standards, and boards and exec teams may effectively manage legal and compliance risks.

While Microsoft offers excellent native agentic observability and monitoring tools such as Azure Application Insights, many organisations have deployed tools beyond Copilot and Copilot Studio. Further, these tools alone also fail to help IT teams gain a comprehensive view of how systems may align with new and evolving compliance standards or legal requirements.

Technology can support this governance model. AI governance platforms such as Cognitive View can help organisations assess AI systems, monitor agentic activity and map governance controls to relevant standards. Xperiens Digital partners with Cognitive View as part of its Responsible AI Governance Advisory service, combining strategic advisory with governance technology to help organisations establish practical policies, risk management practices and board-level assurance.

Importantly, platforms such as Cognitive View allow tech chiefs and boards to devise internal AI usage policies that are tailored to the organisation's needs.

If your organisation's staff and management have deployed AI tools into production environments, you may already be facing a situation where hundreds of unsanctioned agents and GPT client applications are active (eg. Claude or ChatGPT). And as these tools proliferate, control over staff and client data, intellectual property and product/service quality oversight may be diminishing while the implied risks increase in scale and complexity.

Now is the time to set the wheels in motion for AI governance. With this control, both IT departments and boards alike gain the peace of mind to safely drive value generation through AI-enabled innovation activities.