Generative AI is a double-edged sword. While it offers many benefits, it also poses costly security risks for enterprises. 

In a recent Salesforce survey, 71% of 500 senior IT leaders opine that generative AI will likely introduce new security risks to data.

Data privacy risks

The most significant risk with generative AI relates to data privacy. 

Generative AI applications such as ChatGPT store user conversations in their servers, with no way to delete them. ChatGPT’s privacy policy clarifies that the platform collects users browsing activities and IP addresses and may even share them with third-party entities. OpenAI’s standard policy is to retain conversations for 30 days to monitor possible abuse, even if the user turns off chat history. 

Companies integrating ChatGPT into business processes risk passing sensitive information to ChatGPT servers.

Employees looking to save time ask questions that include confidential data, whether they mean to or not. They may input confidential and sensitive information such as internal code, trade secrets, intellectual property, Personally Identifiable Information, API keys and more while using the generative AI tool.

Such stored conversations may become the target of breaches. It may also find its way to competitors, customers, regulators, and other unintended third parties. Even if the data is safe at ChatGPT’s end, any threat actor compromising employees’ ChatGPT accounts could access sensitive data, including users’ queries and the AI’s responses.

Group-IB, a cyber-intelligence firm, found 100,000+ compromised ChatGPT accounts advertised for sale on the darknet.

Many companies already have unsavoury experiences of data leaks.

Samsung suffered a major data leak when its engineers uploaded confidential source code and sensitive work-related reports to ChatGPT for summarization. The breach led to Samsung banning the use of generative AI in its workplaces. 

Amazon discovered examples of ChatGPT responses that resembled internal Amazon data.

Regulatory risks

Transmitting sensitive data to AI providers, such as OpenAI, may violate compliance mandates. Generative AI applications storing such data may conflict with GDPR, CCPA, and other mandates.

Several banks, including Bank of America, Citigroup, Deutsche Bank, Wells Fargo, Goldman Sachs and JPMorgan Chase, have restricted the use of ChatGPT amid concerns about regulatory risks surrounding the sharing of sensitive financial information. 

Co-opting ChatGPT or other generative AI applications may conflict with the law in some places. For instance, California’s bot disclosure law mandates companies to inform customers before deploying bots.

Security vulnerabilities 

Generative AI is, at the end of the day, software. Like any other software, it may contain vulnerabilities that expose companies to cyber threats. The best proof is OpenAI taking ChatGPT offline in March 2023 to fix a bug in the chatbot’s open-source library. The bug exposed some user conversations and payment history to other users. 

AI models remain especially vulnerable to “prompt injection” attacks. Here, the threat actors jailbreak the model and put it to malicious use. They may, for instance, instruct the model to deliver a false response.

Issues with the training model

Another big challenge is ensuring the output is not fraudulent or malevolent.

The efficacy of AI depends on training data. All generative AI models need training with a huge quantum of data. 

Most of such data comes from public sources. But there is also the risk of confidential data making its way to the training data. The training data could co-opt confidential user communications copyright materials. Using such data may expose the enterprise to legal liability.

Another big risk pertains to erroneous output. When the AI model contains out-of-date or incomplete data, it hallucinates and gives false or superficial answers. Misleading information affects reputation and may give rise to legal liability. 

Also, biased training content will amplify the bias in the output. 

How to overcome such risks

One way to overcome the security risks of generative AI is to limit its use through company policies, paywalls, or both.

Sam Altman, the co-founder of OpenAI, characterised ChatGPT as “incredibly limited, but good enough at some things to create a misleading impression of greatness.” He urged companies not to utilise ChatGPT for essential tasks. 

But, the benefits of generative AI are too tempting for enterprises to shut off its use completely. Moreover, enterprises cannot look on when competitors use it and gain competitive advantages. 

1. Establish human oversight

Never trust AI-generated content or take the output at face value. Generative AI tools cannot understand emotional context. As such, their responses may not be appropriate even when they do not hallucinate.

Establish human oversight to 

• Review and edit the generative AI responses for accuracy and bias. Ensure human reviewers are aware of industry-specific compliance requirements and data privacy laws. 
• Classify data to identify sensitive data. Limit the use of sensitive data in conversations to what is unavoidable. When unavoidable, encrypt or anonymise the data before use in AI applications.
• Keep track of compliance regulations. Compliance regulations related to AI use evolve by the day.

Generative AI does not work on a set-it-and-forget-it basis. These tools need constant oversight. 

2. Develop an AI-use policy

Giving generative AI a free run is a recipe for disaster. Identify the risks associated with using generative AI in the enterprise and devise an action plan to mitigate such risks. 

Develop an AI use policy to ensure human oversight works in a friction-free manner.. 

While it is a good idea to benchmark industry best practices, the best policies customise to suit the specifics of the enterprise. 

• Have clarity on the data allowed and disallowed in conversation. For instance, users should never feed copyrighted materials or trade secrets into AI chat feeds.
• Prioritise ethical considerations. Neglecting ethical considerations introduces unintended biases in the data.
• Establish clear and actionable guidelines around bias, privacy, IP rights, provenance and transparency.
• Train employees on generative AI security risks. Educate employees on the risks associated with the technology and how to use the technology to get the best results without side effects.

3. Establish strong data governance frameworks

Back up the AI use policy with robust data governance. Robust governance allows the enterprise to control how the workforce and other system users access and use data.

Apply the least privilege principle to restrict access to authorised personnel.

Control access rights using identity and access management tools.

4. Get the basics of security right

Successful generative AI use requires a safe environment. Get the basics of security right to run generative AI applications without the risk of security breaches.

• Embrace zero-trust.
• Use anomaly detection and monitoring over traditional antivirus software.
• Conduct security audits and penetration tests against generative AI tools to identify vulnerabilities. 
• Extend the auditing to AI vendors. Review vendor’s security controls to ensure weaknesses in the vendor’s systems do affect enterprise IT. 

5. Set up custom interfaces

Generative AI comes in two layers:

• The user interface application, such as Chat GPT 3.5 or Chat GPT 4.0, 
• The backend LLM. 

The data leakage risk is often at the application layer, not the LLM layer. To overcome the issue, create custom front-ends that replace the application interface. Such a custom application can leverage the LLM API.

Co-opting a sandbox as a gateway for LLM service consumption enables adding filters to safeguard data and reduce bias.

Augment LLMs with agent-based models enable enterprises to apply the needed guardrails or checks. Agent-based models verify and govern all LLM interactions based on the set parameter of controls. Creating automated agent-based models ensures safe and outcome-oriented interactions with the LLM. 

Another, even more secure option is building enterprise-specific LLMs. Such LLMs offer enterprises full control over the training data. But, developing and pre-training LLMs requires massive investments in computing infrastructure. It is not a viable option for most enterprises. 

6. Train the data

Most generative AI LLMs, such as OpenAI and Google LaMDA, come closed source, meaning users cannot control their work. Meta’s Llama2 makes its code publicly available. Users can develop custom applications and control what goes on inside. 

When developing custom applications, review the datasets used to train models. Remove biased, toxic, and false elements. Use optimisation techniques to keep bias, toxicity, and hallucinations in check. 

Limit the training data to zero-party data and first-party data. Zero-party data is data customers share. First-party data is data that enterprises collect. Model accuracy and reliability depend on data provenance. Depending on third-party data or external sources makes it difficult to ensure that output is accurate. In any case, be open and upfront regarding the data used to train the model. Clarity promotes trust. 

Get feedback from employees and users exposed to the generative AI system, and use such insights to fine-tune and better the system. A comprehensive security strategy that includes human oversight safeguards against potential threats.