Loading...
1 results
Search Results
Now showing 1 - 1 of 1
- Analysis of the Capability and Training of Chat Bots in the Generation of Rules for Firewall or Intrusion Detection SystemPublication . Louro, Bernardo Amaral; Inácio, Pedro Ricardo Morais; Sequeiros, João Bernardo FerreiraChat bots have a lot of potential to complement human knowledge or help fill a lack of it in various technical areas, like cybersecurity, by providing a tool that could generate specialized rules for computer security systems, such as Intrusion Detection Systems (IDSs) and firewalls, from instructions in natural language. The preliminary evaluation conducted within the scope of this work proved that currently available chat bots are limited when it comes to generating correct and efficient rules and that extra attention is needed if their outputs are to be deployed in production systems, as the consequences can be severe. The chat bots evaluated were Microsoft 365 Copilot, ChatGPT 3.5, Gemini 1.5 Pro, LLaMA 2 7B, Mistral 7B, GPT4All Falcon, Nous-Hermes and Wizard. This document explores four fine-tuning approaches to address some of these limitations, with each of them achieving some degree of success on their different objectives. Approach #1 had a success rate of 89% and assessed if the knowledge obtained was still outputted when the question was arranged differently. Approach #2 had a success rate of 61% and assessed if the model could link knowledge between two different prompt-response pairs. Approach #3 had a success rate of 79% and assessed if the model could create complex rule from learning simpler and generic rules. Approach #4 assessed if the model could identify rules through a multiple choice question, in which it achieved 48% and 89% success rate depending on the order of the choices. From this work it can be concluded that fine-tuning is successful in improving the generation of firewall and IDS rules in chat bots and the results suggest that with some improvements and considerations, a specialized model can be achieved.