Full list of AI resources shared by Nicole Carignan (VP AI Strategy Darktrace) on recent 'whoami' podcast apppearance:

AI versus Counter AI battle
https://www.defense.gov/News/News-Stories/Article/Article/3656926/battle-looming-between-ai-and-counter-ai-says-official/

LLM-based autonomous agents
https://medium.com/the-modern-scientist/a-complete-guide-to-llms-based-autonomous-agents-part-i-69515c016792

One of our predictions for 2024 is research into the increased complexity decision-making of autonomous agents.
https://www.enterprisesecuritytech.com/post/darktrace-predicts-key-cybersecurity-trends-for-2024-generative-ai-ot-attacks-and-more

Rahul Nayak released his research on an Autonomous AI Research Agent that can answer difficult questions with deep multi-hop reasoning capabilities. This research is incredibly impactful for autonomous agents. But, as you know, adversaries are going to optimize the use of autonomous agents for on-demand attack augmentation.
https://towardsdatascience.com/the-research-agent-4ef8e6f1b741

At the same time, as adversaries double down on the use and optimization of autonomous agents for attacks, defenders will become increasingly reliant on autonomous agents for defense to contain an incident at machine speed and mitigate potential damage.

Last year, we saw the increase in jailbreaking LLMs to create autonomous agents.

Vice reported “by fine-tuning an LLM with jailbreak prompts, we demonstrate the possibility of automated jailbreak generation targeting a set of well-known commercialized LLM chatbots. By manipulating the time-sensitive responses of the chatbots, we are able to understand the intricacies of their implementations, and create a proof-of-concept attack to bypass the defenses in multiple LLM chatbots, e.g., CHATGPT, Bard, and Bing Chat,” wrote the international team of researchers.
https://www.vice.com/en/article/bvjba8/this-ai-chatbot-is-trained-to-jailbreak-other-chatbots

Fine-tuning LLMs with jailbreaking techniques is the beginning. Various AI techniques can be utilized including LLM fine-tuning, RAG, API-calls, vector databases, and optimizing iterative information collection and decision-making to learn adversarial techniques and optimizing autonomous agents.

AI Worm Research
https://www.wired.com/story/here-come-the-ai-worms/

Open source AI models containing malware
https://www.bleepingcomputer.com/news/security/malicious-ai-models-on-hugging-face-backdoor-users-machines/

Guidelines for secure AI system development - covers secure implementation considerations
https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development

NIST AI RMF - Covers the criticality of TEVV
https://www.nist.gov/itl/ai-risk-management-framework

Agentic AI Planning and Reasoning
Multi-agent AI workflows and potential for AGI
https://medium.com/the-modern-scientist/a-complete-guide-to-llms-based-autonomous-agents-part-i-69515c016792



twitter: Kirk Trychel (@Teach2Breach)