A Data Driven Anomaly Based Behavior Detection Method for Advanced Persistent Threats (Apt).
ABSTRACT
Advanced Persistent Threats (APTs), represent sophisticated and enduring network intrusion campaigns targeting sensitive information from targeted organizations and operating over a long period. These types of threats are much harder to detect using signature-based methods.
Anomaly based methods consist of monitoring system activity to determine whether an observed activity is normal or abnormal. This is done according to heuristic or statistical analysis, and can be used to detect unknown attacks.
Despite all significant research efforts, such techniques still suffer from a high number of false positive detections. Detecting APTs is complex because it tends to follow a “low and slow” attack profile that is very difficult to distinguish from normal, legitimate activity.
The volume of data that must be analyzed is overwhelming. One technology that holds promise for detecting this kind of attack that is nearly invisible is Big data analytics.
In this work, I propose a data-driven anomaly-based behavior detection method which aims to leverage big data methods, and capable of processing significant amounts of data from diverse or several data sources.
Big data analytics will significantly enhance or improve the detection capabilities, enabling the detection of Advanced Persistent Threats (APTs) activities that pass under the radar of traditional security solutions.
TABLE OF CONTENTS
CERTIFICATION…………………………………………………………………………………………………………………….. ii
ABSTRACT……………………………………………………………………………………………………………………………… iii
ACKNOWLEDGEMENTS ………………………………………………………………………………………………………. iv
DEDICATION…………………………………………………………………………………………………………………………… v
LIST OF ABBREVIATIONS …………………………………………………………………………………………………….. x
LIST OF FIGURES AND TABLES…………………………………………………………………………………………… xi
CHAPTER ONE ………………………………………………………………………………………………………………………..1
INTRODUCTION………………………………………………………………………………………………………………………1
1.1 Background of the study…………………………………………………………………………………………………1
1.2 Objective of the research………………………………………………………………………………………………..4
1.3 Research statement……………………………………………………………………………………………………….4
1.4 Structure of the work……………………………………………………………………………………………………..4
CHAPTER TWO ……………………………………………………………………………………………………………………….5
LITERATURE REVIEW …………………………………………………………………………………………………………..5
2.1. What is an Advanced Persistent Threat?……………………………………………………………………………….5
2.1.1. What actually differentiates APT from other non-targeted threats?……………………………..6
2.1.2. Terminology………………………………………………………………………………………………………….6
2.1.3. Common Goals of APT Attack [51]………………………………………………………………………..7
2.1.4. Other Attacks Related to APT [51] ………………………………………………………………………..8
2.1.5. The Relationship between APT, AET and Botnet [51]…………………………………………….9
2.2. Tools and Methods used by the attackers…………………………………………………………………………9
2.2.1. Malware ……………………………………………………………………………………………………………….9
2.2.1.1. Malware capabilities………………………………………………………………………………………9
2.2.1.2. How does malware infiltrate a computer?……………………………………………………..10
2.2.2. Phishing and other e-mail attacks ………………………………………………………………………..11
2.3. Traditional Security solutions ………………………………………………………………………………………..13
2.3.1. Antivirus software……………………………………………………………………………………………….13
2.3.1.1. Ways to get rid of viruses [26]……………………………………………………………………….14
2.3.1.2. Limitations of antivirus software…………………………………………………………………..15
2.3.2. Firewalls……………………………………………………………………………………………………………..16
2.3.3. Intrusion Prevention Systems………………………………………………………………………………16
2.3.4. Web filters…………………………………………………………………………………………………………..17
2.3.5. Spam filters…………………………………………………………………………………………………………17
2.4. APT Life Cycle ………………………………………………………………………………………………………………17
2.5. Model of operation of APT malware ………………………………………………………………………………21
2.6. Command & Control Channels (C&C) ……………………………………………………………………………..22
2.6.1. Malware C&C Network Protocol Usage……………………………………………………………….23
2.6.2. Detection and Reaction………………………………………………………………………………………..24
2.6.3. C&C Channel Detection Techniques ……………………………………………………………………25
2.6.3.1. Blacklisting ………………………………………………………………………………………………….25
2.6.3.2. Signature based ……………………………………………………………………………………………25
2.6.3.3. DNS protocol based………………………………………………………………………………………25
2.6.3.4. IRC protocol based ………………………………………………………………………………………25
2.6.3.5. Peer to peer protocol based …………………………………………………………………………..26
2.6.3.6. HTTP protocol based……………………………………………………………………………………26
2.6.3.7. Temporal-based……………………………………………………………………………………………26
2.6.3.8. Anomaly detection………………………………………………………………………………………..27
2.6.3.9. Correlation based…………………………………………………………………………………………27
2.7. Research Direction……………………………………………………………………………………………………….27
2.8. Related work ……………………………………………………………………………………………………………….28
CHAPTER THREE………………………………………………………………………………………………………………….29
METHODOLOGY …………………………………………………………………………………………………………………..29
3.1. Big data and Big data analytics ………………………………………………………………………………………29
3.1.1. Big Data………………………………………………………………………………………………………………29
3.1.2. Big Data Analytics……………………………………………………………………………………………….29
3.1.3. Some Big Data Technologies………………………………………………………………………………..30
3.1.3.1. Hadoop………………………………………………………………………………………………………..30
3.1.3.2. MapReduce and Distributed Computing Using Spark……………………………………31
3.1.3.3. Spark Ecosystem ………………………………………………………………………………………….32
3.1.3.4. What are the benefits of Spark? ……………………………………………………………………32
3.1.3.5. Resilient Distributed Datasets……………………………………………………………………….33
3.1.3.6. Predictive Modeling and Analytics………………………………………………………………..33
3.1.3.7. Types of Machine Learning Models………………………………………………………………34
3.1.4. Machine Learning and Big Data Analytics……………………………………………………………34
3.1.5. Benefits of Big Data Analytics in APT attack detection …………………………………………35
3.2. Methodology……………………………………………………………………………………………………………….37
3.2.1. What is Anomaly Detection?………………………………………………………………………………..37
3.2.2. The Components of a Data-driven Anomaly-based Behavior Detection method for
Advanced Persistent Threats (APT)……………………………………………………………………………………39
3.2.2.1. Data Collection…………………………………………………………………………………………….41
Data preprocessing………………………………………………………………………………………………….41
3.2.2.2…………………………………………………………………………………………………………………………….41
3.2.2.3. Model Creation via classification…………………………………………………………………..44
3.2.2.4. Model Selection ……………………………………………………………………………………………46
3.2.2.5. Model Prediction and Evaluation………………………………………………………………….46
CHAPTER FOUR…………………………………………………………………………………………………………………….49
IMPLEMENTATION AND EVALUATION……………………………………………………………………………..49
4.1. Big Data Analytics (Machine learning) based on network traces with full payloads………………49
4.2. Big Data Analytics (Machine Learning) based on HTTP traffic…………………………………………….49
4.3. Environment for the Implementation……………………………………………………………………………..50
4.4. IMPLEMENTATION STAGES……………………………………………………………………………………………50
4.4.1. Data Collection……………………………………………………………………………………………………50
4.4.2. Data Preprocessing ……………………………………………………………………………………………..53
4.4.2.1. Load and Analyze data…………………………………………………………………………………53
4.4.2.2. Feature Extraction ……………………………………………………………………………………….53
4.4.2.3. Data Cleaning ………………………………………………………………………………………………56
4.4.2.4. Feature Engineering and Transformation……………………………………………………..56
4.4.3. Model Creation via classification………………………………………………………………………….58
4.4.3.1. Create Pipeline …………………………………………………………………………………………….58
4.4.4. Model Selection …………………………………………………………………………………………………..59
4.4.4.1. Tuning the pipeline using a CrossValidator …………………………………………………..59
4.4.5. Model Prediction and Evaluation…………………………………………………………………………60
CHAPTER FIVE ……………………………………………………………………………………………………………………..65
CONCLUSIONS ………………………………………………………………………………………………………………………65
5.1. Summary …………………………………………………………………………………………………………………….65
5.2. Challenges…………………………………………………………………………………………………………………..65
5.3. Future Work………………………………………………………………………………………………………………..66
REFERENCES…………………………………………………………………………………………………………………………67
INTRODUCTION
With the rapid development of computer networks, new and sophisticated types of attacks have emerged which require novel and more sophisticated defense mechanisms. Advanced Persistent Threats (APTs) are one of the most fast-growing cyber security threats that organizations face today.
They are carried out by knowledgeable, very skilled and well-funded hackers, targeting sensitive information from specific organizations.
The objective of an APT attack is to steal sensitive data from the targeted organization, to gain access to sensitive customer data, or to access strategic or important business information that could be used for financial gain, blackmail, embarrassment, data poisoning, “illegal insider trading or disrupting an organization’s business”.
APT attackers target organizations in sectors with high-value information, such as national defense or military, manufacturing, and the financial industry.
The technologies and methods employed in APT attacks are stealthy and difficult to detect, for instance, they can employ “social engineering which involves tricking people into breaking normal security procedures”.
In addition, the APT intruders constantly change and refine their methods, including having insiders (those within the organization) who abuse legitimate access rights to manipulate and steal data. Once hacking into the targeted network is successful, the intruder installs APT malware on the victim’s system.
The attacker then is able to monitor and control the spread of malware and alsoremotely control the infected systems. This opens a channel through which they steal sensitive information from the victim’s system unknowingly to the owner, over a long period of time except if the malicious activity is detected.
After the information of interest has been found the attacker gives a command to exfiltrate the information. This is usually done through a channel separate from the Command and Control (C&C) channel.
REFERENCES
Xiaohua Yan; Joy Ying Zhang; (2013), Early Detection of Cyber Security Threats
using Structured Behavior Modeling.
Cloud Security Alliance; (September 2013); Big Data Analytics for Security
Intelligence.
Randy Franklin Smith; Brook Watson;(2013), 3 Big data security analytics
techniques you can apply now to catch advanced persistent threats
Judith S. Hurwitz, Alan F. Nugent, Fern Halper, PhD, Marcia A. Kaufman;(2013),
Big data for dummies.
Ping Chen, Lieven Desmet, and Christophe Huygens ;( 2013), A study on Advanced
Persistent Threats.
What is advanced persistent threat (APT)? Definition from whatis.com
http://searchsecurity.techtarget.com/definition/advanced-persistent-threat-APT
Intrusion detection system – Wikipedia, the free encyclopedia.htm
StudentsandScholarship Team.
Join Our Newsletter!
Don’t miss this opportunity
Enter Your Details
Be the first to comment