{
  "id": 427497,
  "title": "Variants of ensemble learning (in intrusion detection classification problems)",
  "url": "/competitions/2023-kaggle-ai-report/discussion/427497",
  "author_name": "Chung Wai Tang",
  "post_date": "2023-07-28T07:14:59.532000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<h1>Variants of ensemble learning (in <strong>intrusion detection</strong> classification problems)</h1>\n<p>Good day, everyone! Since \"ensemble learning\" came up in more than a handful of essays, I thought to share with you a quick summary of this paper that I came upon recently: <br>\n<code>Tama, B. A., &amp; Lim, S. (2021). Ensemble learning for intrusion detection systems: A systematic mapping study and cross-benchmark evaluation. *Computer Science Review*, 39. https://doi.org/10.1016/j.cosrev.2020.100357</code></p>\n<hr>\n<h2>Papers they reviewed…</h2>\n<p>The authors reviewed 124 articles published in 2015 -2020 that employed <strong>ensemble learning</strong> to address one of these sub-tasks in <strong>intrusion detection</strong> problems:</p>\n<ol>\n<li>Intrusion detection systems </li>\n<li>\"Anomaly\" detection</li>\n<li>\"Misuse\" detection</li>\n<li>\"Signature\" detection<br>\n<br><br>\nOf these 124 papers:</li>\n</ol>\n<ul>\n<li>50 were journal articles (e.g. IEEE Access, IEEE IoT J, IEEE Trans on Dependable Secure &amp; Computing, J of Computational Science)</li>\n<li>54 were conference papers (e.g. Procedia Computer Science, ACM Conference on Internet Measurement Conf.)</li>\n<li>11 were book chapters</li>\n<li>…</li>\n</ul>\n<h2>Ensemble strategies they found popular in these papers…</h2>\n<p>They tabulated frequencies of each ensemble variant in these papers:</p>\n<table>\n<thead>\n<tr>\n<th>Ensemble family</th>\n<th>Scheme</th>\n<th>Tested by n articles</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Homogeneous</strong></td>\n<td>Bagging</td>\n<td>n=33</td>\n</tr>\n<tr>\n<td></td>\n<td>Boosting</td>\n<td>35 (AdaBoost) + 16 (other kinds of boosting)</td>\n</tr>\n<tr>\n<td></td>\n<td>Random subspace</td>\n<td>3</td>\n</tr>\n<tr>\n<td></td>\n<td>Rotation forest</td>\n<td>6</td>\n</tr>\n<tr>\n<td></td>\n<td>Trees</td>\n<td>52 (random) + 21 (other kinds of trees)</td>\n</tr>\n<tr>\n<td></td>\n<td>Dagging</td>\n<td>1</td>\n</tr>\n<tr>\n<td><strong>Heterogeneous</strong></td>\n<td>Stacking (an implementation can be found in a <a href=\"https://www.kaggle.com/code/yassineghouzam/titanic-top-4-with-ensemble-modeling/notebook\" target=\"_blank\">notebook by Dr. Ghouzam ~6 years ago</a>)</td>\n<td>17</td>\n</tr>\n<tr>\n<td></td>\n<td>Voting</td>\n<td>18 (majority voting) + 8 (average probability voting) + (other kinds of voting schemes)</td>\n</tr>\n</tbody>\n</table>\n<h2>Datasets most often used by these papers…</h2>\n<p>Most of these articles performed some kind of benchmarking using public datasets, some of which were:</p>\n<table>\n<thead>\n<tr>\n<th>Dataset</th>\n<th>Info</th>\n<th>Used by n articles</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>NSL-KDD <a href=\"https://www.kaggle.com/datasets/hassan06/nslkdd\" target=\"_blank\">Kaggle data card</a></td>\n<td>125,973 x 41 features</td>\n<td>n=58 articles</td>\n</tr>\n<tr>\n<td>KDD Cup 99 <a href=\"https://www.kaggle.com/datasets/galaxyh/kdd-cup-1999-data\" target=\"_blank\">Kaggle data card</a></td>\n<td><a href=\"https://medium.com/analytics-vidhya/building-an-intrusion-detection-model-using-kdd-cup99-dataset-fb4cba4189ed\" target=\"_blank\"> 145,586 x 42 features</a></td>\n<td>28</td>\n</tr>\n<tr>\n<td>UNSW-NB15 <a href=\"https://www.kaggle.com/datasets/dhoogla/unswnb15\" target=\"_blank\">Kaggle data card</a></td>\n<td>37,000 normal + 45,332; 43 features</td>\n<td>17</td>\n</tr>\n<tr>\n<td>CICIDS 2017</td>\n<td>168+186+2180 samples; 78 features</td>\n<td>7</td>\n</tr>\n<tr>\n<td>UNSW 2018 IoT Botnet</td>\n<td>370 (normal) + 2,934,447; 16 featyres</td>\n<td>1</td>\n</tr>\n<tr>\n<td>Private</td>\n<td>Various</td>\n<td>9</td>\n</tr>\n</tbody>\n</table>\n<h2>Results from authors' own comparison of 19 ensemble variants…</h2>\n<p>The authors performed their own benchmarks using 4 datasets on 19 different stacking ensembles, some of which used homogeneous ensembles, e.g. eXtreme Gradient Boosting, Random Forests, while other ensemble variants used stacking (with generalized linear model as the classifier on the second level). Weka and H2O tools were used to execute the classification task.<br>\nTheir statistical analysis procedure:</p>\n<ol>\n<li>Calculate each method's rank for each dataset using Friedman's rank test w.r.t. their MCC scores</li>\n<li>Calculate average rank of each method over all datasets</li>\n<li>Calculate p-value using Friedman's test</li>\n<li>If Friedman's test shows p&lt;0.1, then run Finner's test whereby the best-performing method is then compared against remaining methods<br>\nTheir findings:</li>\n</ol>\n<ul>\n<li>Friedman's rank tests show that Gradient boosting machine (GBM) was superior compared with other emsemble variants</li>\n<li>Finner's tests show that GBM was significantly better than decision tree (known as J48), Credal Decision Tree, Classification and regression tree, and Random Trees<br>\n<hr></li>\n</ul>\n<h2>Possible prompts for discussion</h2>\n<ol>\n<li>Which other benchmark studies support (or refute) the superiority of GBM over other ensemble variants?  </li>\n<li>One secondary analysis question might be to stratify results by the size of training set. What other ideas should we consider when we were to replicate their analyses (for other classification problems)?  </li>\n<li>…</li>\n</ol>",
  "messages": [
    {
      "id": 2362572,
      "postDate": "2023-07-28T07:14:59.533Z",
      "content": "<h1>Variants of ensemble learning (in <strong>intrusion detection</strong> classification problems)</h1>\n<p>Good day, everyone! Since \"ensemble learning\" came up in more than a handful of essays, I thought to share with you a quick summary of this paper that I came upon recently: <br>\n<code>Tama, B. A., &amp; Lim, S. (2021). Ensemble learning for intrusion detection systems: A systematic mapping study and cross-benchmark evaluation. *Computer Science Review*, 39. https://doi.org/10.1016/j.cosrev.2020.100357</code></p>\n<hr>\n<h2>Papers they reviewed…</h2>\n<p>The authors reviewed 124 articles published in 2015 -2020 that employed <strong>ensemble learning</strong> to address one of these sub-tasks in <strong>intrusion detection</strong> problems:</p>\n<ol>\n<li>Intrusion detection systems </li>\n<li>\"Anomaly\" detection</li>\n<li>\"Misuse\" detection</li>\n<li>\"Signature\" detection<br>\n<br><br>\nOf these 124 papers:</li>\n</ol>\n<ul>\n<li>50 were journal articles (e.g. IEEE Access, IEEE IoT J, IEEE Trans on Dependable Secure &amp; Computing, J of Computational Science)</li>\n<li>54 were conference papers (e.g. Procedia Computer Science, ACM Conference on Internet Measurement Conf.)</li>\n<li>11 were book chapters</li>\n<li>…</li>\n</ul>\n<h2>Ensemble strategies they found popular in these papers…</h2>\n<p>They tabulated frequencies of each ensemble variant in these papers:</p>\n<table>\n<thead>\n<tr>\n<th>Ensemble family</th>\n<th>Scheme</th>\n<th>Tested by n articles</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Homogeneous</strong></td>\n<td>Bagging</td>\n<td>n=33</td>\n</tr>\n<tr>\n<td></td>\n<td>Boosting</td>\n<td>35 (AdaBoost) + 16 (other kinds of boosting)</td>\n</tr>\n<tr>\n<td></td>\n<td>Random subspace</td>\n<td>3</td>\n</tr>\n<tr>\n<td></td>\n<td>Rotation forest</td>\n<td>6</td>\n</tr>\n<tr>\n<td></td>\n<td>Trees</td>\n<td>52 (random) + 21 (other kinds of trees)</td>\n</tr>\n<tr>\n<td></td>\n<td>Dagging</td>\n<td>1</td>\n</tr>\n<tr>\n<td><strong>Heterogeneous</strong></td>\n<td>Stacking (an implementation can be found in a <a href=\"https://www.kaggle.com/code/yassineghouzam/titanic-top-4-with-ensemble-modeling/notebook\" target=\"_blank\">notebook by Dr. Ghouzam ~6 years ago</a>)</td>\n<td>17</td>\n</tr>\n<tr>\n<td></td>\n<td>Voting</td>\n<td>18 (majority voting) + 8 (average probability voting) + (other kinds of voting schemes)</td>\n</tr>\n</tbody>\n</table>\n<h2>Datasets most often used by these papers…</h2>\n<p>Most of these articles performed some kind of benchmarking using public datasets, some of which were:</p>\n<table>\n<thead>\n<tr>\n<th>Dataset</th>\n<th>Info</th>\n<th>Used by n articles</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>NSL-KDD <a href=\"https://www.kaggle.com/datasets/hassan06/nslkdd\" target=\"_blank\">Kaggle data card</a></td>\n<td>125,973 x 41 features</td>\n<td>n=58 articles</td>\n</tr>\n<tr>\n<td>KDD Cup 99 <a href=\"https://www.kaggle.com/datasets/galaxyh/kdd-cup-1999-data\" target=\"_blank\">Kaggle data card</a></td>\n<td><a href=\"https://medium.com/analytics-vidhya/building-an-intrusion-detection-model-using-kdd-cup99-dataset-fb4cba4189ed\" target=\"_blank\"> 145,586 x 42 features</a></td>\n<td>28</td>\n</tr>\n<tr>\n<td>UNSW-NB15 <a href=\"https://www.kaggle.com/datasets/dhoogla/unswnb15\" target=\"_blank\">Kaggle data card</a></td>\n<td>37,000 normal + 45,332; 43 features</td>\n<td>17</td>\n</tr>\n<tr>\n<td>CICIDS 2017</td>\n<td>168+186+2180 samples; 78 features</td>\n<td>7</td>\n</tr>\n<tr>\n<td>UNSW 2018 IoT Botnet</td>\n<td>370 (normal) + 2,934,447; 16 featyres</td>\n<td>1</td>\n</tr>\n<tr>\n<td>Private</td>\n<td>Various</td>\n<td>9</td>\n</tr>\n</tbody>\n</table>\n<h2>Results from authors' own comparison of 19 ensemble variants…</h2>\n<p>The authors performed their own benchmarks using 4 datasets on 19 different stacking ensembles, some of which used homogeneous ensembles, e.g. eXtreme Gradient Boosting, Random Forests, while other ensemble variants used stacking (with generalized linear model as the classifier on the second level). Weka and H2O tools were used to execute the classification task.<br>\nTheir statistical analysis procedure:</p>\n<ol>\n<li>Calculate each method's rank for each dataset using Friedman's rank test w.r.t. their MCC scores</li>\n<li>Calculate average rank of each method over all datasets</li>\n<li>Calculate p-value using Friedman's test</li>\n<li>If Friedman's test shows p&lt;0.1, then run Finner's test whereby the best-performing method is then compared against remaining methods<br>\nTheir findings:</li>\n</ol>\n<ul>\n<li>Friedman's rank tests show that Gradient boosting machine (GBM) was superior compared with other emsemble variants</li>\n<li>Finner's tests show that GBM was significantly better than decision tree (known as J48), Credal Decision Tree, Classification and regression tree, and Random Trees<br>\n<hr></li>\n</ul>\n<h2>Possible prompts for discussion</h2>\n<ol>\n<li>Which other benchmark studies support (or refute) the superiority of GBM over other ensemble variants?  </li>\n<li>One secondary analysis question might be to stratify results by the size of training set. What other ideas should we consider when we were to replicate their analyses (for other classification problems)?  </li>\n<li>…</li>\n</ol>",
      "rawMarkdown": "\n# Variants of ensemble learning (in **intrusion detection** classification problems)\n\n\nGood day, everyone! Since \"ensemble learning\" came up in more than a handful of essays, I thought to share with you a quick summary of this paper that I came upon recently: \n\n\n`Tama, B. A., & Lim, S. (2021). Ensemble learning for intrusion detection systems: A systematic mapping study and cross-benchmark evaluation. *Computer Science Review*, 39. https://doi.org/10.1016/j.cosrev.2020.100357`\n\n<hr>\n\n## Papers they reviewed... \n\nThe authors reviewed 124 articles published in 2015 -2020 that employed **ensemble learning** to address one of these sub-tasks in **intrusion detection** problems:\n1. Intrusion detection systems \n2. \"Anomaly\" detection\n3. \"Misuse\" detection\n4. \"Signature\" detection\n<br>\nOf these 124 papers:\n- 50 were journal articles (e.g. IEEE Access, IEEE IoT J, IEEE Trans on Dependable Secure & Computing, J of Computational Science)\n- 54 were conference papers (e.g. Procedia Computer Science, ACM Conference on Internet Measurement Conf.)\n- 11 were book chapters\n- ...\n\n## Ensemble strategies they found popular in these papers...\n\nThey tabulated frequencies of each ensemble variant in these papers:\n\n\n| Ensemble family | Scheme | Tested by n articles |\n|:--|:--|:--|\n| **Homogeneous** | Bagging | n=33 |\n|   | Boosting | 35 (AdaBoost) + 16 (other kinds of boosting) |\n|   | Random subspace | 3 |\n| | Rotation forest | 6 |\n| | Trees | 52 (random) + 21 (other kinds of trees) |\n| | Dagging | 1 |\n| **Heterogeneous**| Stacking (an implementation can be found in a [notebook by Dr. Ghouzam ~6 years ago](https://www.kaggle.com/code/yassineghouzam/titanic-top-4-with-ensemble-modeling/notebook)) | 17 |\n| | Voting |  18 (majority voting) + 8 (average probability voting) + (other kinds of voting schemes) |\n\n## Datasets most often used by these papers...\n\nMost of these articles performed some kind of benchmarking using public datasets, some of which were:\n\n| Dataset  | Info  | Used by n articles | \n| :-- | :-- |:-- |\n| NSL-KDD [Kaggle data card](https://www.kaggle.com/datasets/hassan06/nslkdd)|  125,973 x 41 features | n=58 articles |\n| KDD Cup 99 [Kaggle data card](https://www.kaggle.com/datasets/galaxyh/kdd-cup-1999-data) | [ 145,586 x 42 features](https://medium.com/analytics-vidhya/building-an-intrusion-detection-model-using-kdd-cup99-dataset-fb4cba4189ed) | 28 |\n| UNSW-NB15 [Kaggle data card](https://www.kaggle.com/datasets/dhoogla/unswnb15)| 37,000 normal + 45,332; 43 features | 17 |\n| CICIDS 2017 | 168+186+2180 samples; 78 features | 7 |\n| UNSW 2018 IoT Botnet | 370 (normal) + 2,934,447; 16 featyres  | 1 |\n| Private | Various | 9 |\n\n## Results from authors' own comparison of 19 ensemble variants...  \n\nThe authors performed their own benchmarks using 4 datasets on 19 different stacking ensembles, some of which used homogeneous ensembles, e.g. eXtreme Gradient Boosting, Random Forests, while other ensemble variants used stacking (with generalized linear model as the classifier on the second level). Weka and H2O tools were used to execute the classification task.\n\nTheir statistical analysis procedure:\n1. Calculate each method's rank for each dataset using Friedman's rank test w.r.t. their MCC scores\n2. Calculate average rank of each method over all datasets\n3. Calculate p-value using Friedman's test\n4. If Friedman's test shows p<0.1, then run Finner's test whereby the best-performing method is then compared against remaining methods\n\nTheir findings:\n- Friedman's rank tests show that Gradient boosting machine (GBM) was superior compared with other emsemble variants\n- Finner's tests show that GBM was significantly better than decision tree (known as J48), Credal Decision Tree, Classification and regression tree, and Random Trees\n\n<hr>\n\n## Possible prompts for discussion\n\n1. Which other benchmark studies support (or refute) the superiority of GBM over other ensemble variants?  \n2. One secondary analysis question might be to stratify results by the size of training set. What other ideas should we consider when we were to replicate their analyses (for other classification problems)?  \n3. ...\n",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2362572": "\n# Variants of ensemble learning (in **intrusion detection** classification problems)\n\n\nGood day, everyone! Since \"ensemble learning\" came up in more than a handful of essays, I thought to share with you a quick summary of this paper that I came upon recently: \n\n\n`Tama, B. A., & Lim, S. (2021). Ensemble learning for intrusion detection systems: A systematic mapping study and cross-benchmark evaluation. *Computer Science Review*, 39. https://doi.org/10.1016/j.cosrev.2020.100357`\n\n<hr>\n\n## Papers they reviewed... \n\nThe authors reviewed 124 articles published in 2015 -2020 that employed **ensemble learning** to address one of these sub-tasks in **intrusion detection** problems:\n1. Intrusion detection systems \n2. \"Anomaly\" detection\n3. \"Misuse\" detection\n4. \"Signature\" detection\n<br>\nOf these 124 papers:\n- 50 were journal articles (e.g. IEEE Access, IEEE IoT J, IEEE Trans on Dependable Secure & Computing, J of Computational Science)\n- 54 were conference papers (e.g. Procedia Computer Science, ACM Conference on Internet Measurement Conf.)\n- 11 were book chapters\n- ...\n\n## Ensemble strategies they found popular in these papers...\n\nThey tabulated frequencies of each ensemble variant in these papers:\n\n\n| Ensemble family | Scheme | Tested by n articles |\n|:--|:--|:--|\n| **Homogeneous** | Bagging | n=33 |\n|   | Boosting | 35 (AdaBoost) + 16 (other kinds of boosting) |\n|   | Random subspace | 3 |\n| | Rotation forest | 6 |\n| | Trees | 52 (random) + 21 (other kinds of trees) |\n| | Dagging | 1 |\n| **Heterogeneous**| Stacking (an implementation can be found in a [notebook by Dr. Ghouzam ~6 years ago](https://www.kaggle.com/code/yassineghouzam/titanic-top-4-with-ensemble-modeling/notebook)) | 17 |\n| | Voting |  18 (majority voting) + 8 (average probability voting) + (other kinds of voting schemes) |\n\n## Datasets most often used by these papers...\n\nMost of these articles performed some kind of benchmarking using public datasets, some of which were:\n\n| Dataset  | Info  | Used by n articles | \n| :-- | :-- |:-- |\n| NSL-KDD [Kaggle data card](https://www.kaggle.com/datasets/hassan06/nslkdd)|  125,973 x 41 features | n=58 articles |\n| KDD Cup 99 [Kaggle data card](https://www.kaggle.com/datasets/galaxyh/kdd-cup-1999-data) | [ 145,586 x 42 features](https://medium.com/analytics-vidhya/building-an-intrusion-detection-model-using-kdd-cup99-dataset-fb4cba4189ed) | 28 |\n| UNSW-NB15 [Kaggle data card](https://www.kaggle.com/datasets/dhoogla/unswnb15)| 37,000 normal + 45,332; 43 features | 17 |\n| CICIDS 2017 | 168+186+2180 samples; 78 features | 7 |\n| UNSW 2018 IoT Botnet | 370 (normal) + 2,934,447; 16 featyres  | 1 |\n| Private | Various | 9 |\n\n## Results from authors' own comparison of 19 ensemble variants...  \n\nThe authors performed their own benchmarks using 4 datasets on 19 different stacking ensembles, some of which used homogeneous ensembles, e.g. eXtreme Gradient Boosting, Random Forests, while other ensemble variants used stacking (with generalized linear model as the classifier on the second level). Weka and H2O tools were used to execute the classification task.\n\nTheir statistical analysis procedure:\n1. Calculate each method's rank for each dataset using Friedman's rank test w.r.t. their MCC scores\n2. Calculate average rank of each method over all datasets\n3. Calculate p-value using Friedman's test\n4. If Friedman's test shows p<0.1, then run Finner's test whereby the best-performing method is then compared against remaining methods\n\nTheir findings:\n- Friedman's rank tests show that Gradient boosting machine (GBM) was superior compared with other emsemble variants\n- Finner's tests show that GBM was significantly better than decision tree (known as J48), Credal Decision Tree, Classification and regression tree, and Random Trees\n\n<hr>\n\n## Possible prompts for discussion\n\n1. Which other benchmark studies support (or refute) the superiority of GBM over other ensemble variants?  \n2. One secondary analysis question might be to stratify results by the size of training set. What other ideas should we consider when we were to replicate their analyses (for other classification problems)?  \n3. ...\n"
  }
}