Using social engineering against LLMs: Additionally, GTIG found that attackers are increasingly using “social engineering-like pretexts” in their prompts to get around LLM safeguards. Notably, they have posed as participants in a “capture-the-flag” (CTF) gamified cybersecurity competition, persuading Gemini to give up information it would otherwise refuse to reveal. In one interaction, for instance, an attacker attempted to use Gemini to identify vulnerabilities on a system that had been compromised; but they were blocked by the model’s safeguards. However, after they reframed the prompt and identified as a CTF player developing phishing and exploitation skills, Gemini obliged, giving advice about the next steps in a red-teaming scenario and providing details that could be used to attack the system.Researchers underscored the importance of nuance in these types of CTF prompts, which would normally be harmless. “This nuance in AI use highlights critical differentiators in benign versus misuse of AI that we continue to analyze,” they note.They also observed an Iranian state-sponsored actor who used Gemini to conduct research to build custom malware, including web shells and a Python-based C2 server. The group was able to get past security guardrails by posing as students working on a final university project or an informational paper on cybersecurity.The attackers then used Gemini to help with a script designed to listen for and decrypt requests, and to transfer files or remotely execute tasks. However, this technique revealed “sensitive, hard-coded information” to Gemini, including the C2 domain and encryption keys, which assisted in defenders’ efforts to disrupt the campaign, the researchers said.
AI tools are hot on the cybercrime marketplace: Further investigations by the GTIG team found that the underground marketplace for illicit AI tools has “matured.” Tools for purchase on the black market include:
Malware generation: To build malware for specific cases or improve upon existing malware;Deepfake and image generation: To create “lure content” or bypass know your customer (KYC) requirements;Phishing kits and support: To craft “engaging lure content” or distribute to wider audiences;Research and reconnaissance: To quickly gather and summarize cybersecurity concepts or general topics;Vulnerability exploitation: To identify publicly-available research on pre-existing vulnerabilities;Technical support and code generation.Researchers point out that pricing models for these tools are increasingly mimicking those of conventional ones: Free versions inject ads, and subscription tier-pricing introduces more advanced technical features such as image generation or API and Discord access.”Tools and services offered via underground forums can enable low-level actors to augment the frequency, scope, efficacy, and complexity of their intrusions,” the GTIG researchers write, “despite their limited technical acumen and financial resources.”And, they add, “Given the increasing accessibility of these [AI tools], and the growing AI discourse in these forums, threat activity leveraging AI will increasingly become commonplace amongst threat actors.”
First seen on csoonline.com
Jump to article: www.csoonline.com/article/4085494/google-researchers-detect-first-operational-use-of-llms-in-active-malware-campaigns.html
