Generative AI is a type of artificial intelligence that can create new content. Agentic AI, on the other hand, is not only responsive but also capable of making its own decisions and taking actions autonomously. In the field of cybersecurity, these systems can both detect threats quickly and defend automatically. In short, they can do everything humans can and even reduce the mistakes that humans might make. As mentioned, generative and agentic AI are highly beneficial in the field of cybersecurity; however, on the other hand, they can also be equally harmful. Now, I would like to talk about their potential risks.
1.1 Data Security Risks : Attacks that target the data used to operate artificial intelligence systems aim at compromising its confidentiality, integrity, and privacy. This, in turn, can lead to the theft of sensitive data.
Case Study – Marriott (Starwood Breach) : The Marriott breach exposed the personal data of nearly 500 million guests and shows how overlooked vulnerabilities in legacy systems can become serious data security risks. Originating in Starwood’s network years before the acquisition, the incident highlights the importance of thorough cybersecurity checks during mergers to prevent hidden risks from resurfacing later.
1.2 Data Poisoning : Data poisoning is a security threat that occurs when malicious actors deliberately alter the data used to train artificial intelligence systems. Such attacks can cause the model to make incorrect decisions or to classify harmful content as safe.
We can summarize this situation in its simplest form as follows: Data extraction occurs when attackers “trick” an AI model into revealing training data that is supposed to remain confidential.
Case Study – IKEA (2021 Internal Phishing Attack) : In 2021, IKEA faced an internal phishing attack after attackers gained access to its email server. Using real employee accounts, they sent malicious links to other staff, showing how easily trusted channels can be abused. The case highlights the limits of traditional email filters and the importance of advanced security tools that can spot unusual behavior, even when it comes from inside the organization.
1.3 Data Extraction: Attackers ask the model carefully crafted questions, pretending to be ordinary users. Their goal is to make the model “leak” information it learned during training, such as email addresses, phone numbers, or sensitive company data.
Case Study – Macy’s (Magecart Attack) Macy’s became the target of a Magecart attack, a common threat in the e‑commerce industry. Hackers injected malicious JavaScript into the company’s checkout page, allowing them to steal customers’ credit card details in real time. This incident highlights how critical the client side of web applications is and shows why continuous monitoring is essential to detect unauthorized code changes before they can cause serious damage.
1.4 Inference Attacks : Inference attacks aim to extract hidden information from a model’s training data. They include membership inference, which reveals whether a specific user’s data was part of the training set, and property inference, which exposes general characteristics such as demographics. These attacks compromise privacy and data integrity by analyzing the model’s outputs.
Case Studies – Mastercard : Mastercard developed Decision Intelligence, an AI-powered platform to fight payment fraud globally. The system analyzes thousands of data points in real time, reducing false positives and improving security for merchants and cardholders. However, such models remain vulnerable to inference attacks. Through membership inference, attackers may reveal if a user’s data was in the training set, while property inference can expose general characteristics like demographics. These threats compromise privacy and data integrity, weakening trust in the system.
1.5 Model Security Risks: Model security risks refer to threats that directly target the structure and functioning of AI models. These attacks aim to undermine the integrity and reliability of outputs, making the system less trustworthy. Addressing such risks is essential to ensure that AI remains secure and robust, especially in critical applications.
1.6 Evasion Attacks : Evasion attacks occur when attackers subtly manipulate input data to force the model into making incorrect decisions. Even highly accurate systems can be deceived by these adversarial examples.
1.7 Backdoor Attacks: These types of attacks occur when hidden triggers are injected into a model during its training phase. When activated, these triggers cause targeted misclassifications, while the model’s performance on normal inputs remains unchanged. Such attacks can be used to bypass security systems.
1.8 Model Poisoning : Model poisoning occurs when attackers manipulate a model’s parameters or architecture to introduce vulnerabilities or reduce its performance. This threat is particularly concerning in collaborative learning environments such as federated learning, where multiple participants contribute to training a shared model.
1.9 Model Stealing : Model stealing occurs when attackers repeatedly query a model to extract or replicate its architecture, parameters, or functionality. This compromises proprietary models and intellectual property, allowing attackers to create unauthorized replicas.
2.0 Denial of Service Attacks: Denial of service attacks occur when an AI system is flooded with excessive requests, making it unavailable or unable to respond to legitimate users. Such attacks can interrupt critical services and significantly reduce the overall quality of the user experience. In short, these attacks waste system resources by unnecessarily occupying traffic.
2.1 Resource Exhaustion Attacks : These attacks drain the computational power of AI systems, reducing their performance and availability. Attackers exploit system weaknesses with crafted queries or inputs, causing excessive resource consumption. As a result, legitimate requests face delays and the system operates less efficiently.
2.2 Supply Chain Attacks: Supply chain attacks target the integrity of AI development tools, libraries, or platforms. By compromising these components, attackers introduce vulnerabilities that can spread widely across multiple systems and applications. The impact is often far‑reaching, since the affected parts are commonly reused in many AI environments.
2.3 Application Risks : Application Risks Application risks refer to the challenges that arise from how AI systems are used in practice. They stem from the interaction between models and users, and from the broader ethical and societal impact of deploying AI in different contexts.
2.4 Prompt Injection Attacks: This type of attack manipulates the input given to an AI system to produce harmful outputs. Such attacks can generate malicious results that exploit data security.
2.5 Output Integrity Issues: If these issues stem from inconsistent outputs, they make the AI system unreliable. In short, they can lead to privacy violations and a decline in trust toward AI.
2.6 Ethical and Societal Risks: These risks are related to the use of AI. They can reinforce biases, violate privacy, and increase social inequality.
These risks make AI something we cannot fully trust, even to the extent we might hope to. This report, together with the research, took me 3 hours. As you can see, I created the visual below using generative AI.
Contributed by GuestPosts.biz