{
  "id": 244894,
  "title": "How to handle outliers? ",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/244894",
  "author_name": "",
  "post_date": "2021-06-08T18:50:56.685120200Z",
  "votes": 6,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I recently came across this file in the training set <strong><em>8432e134-5cb0-4dfc-8794-b5b05ac8dea3.json</em></strong>. </p>\n<p>Here is the first few lines of the file.</p>\n<p><code>[{\"section_title\": \"Abstract\", \"text\": \"The National Weather Service (NWS) h a s d e v e l o p e d a h u r r i c a n e s t o r m s u r g e model c a l l e d SLOSH ( S e a ,\\n-Lake, a n d O v e r l a n d S u r g e s f r o m g u r r i c a n e s ) t o com-\"}, {\"section_title\": \"\", \"text\": \"p u t e h u r r i c a n e s t o r m s u r g e s , g i v e n s t o r m d a t a as input. The numerical model incorporates a dynamic coastline, overland flooding, and sub-grid feat u r e s s u c h a s b a r r i e r s , c u t s b e t w e e n b a r r i e r s , and one-dimensional flow along channels of varying width. The\"}, {\"section_title\": \"SLOSH model h a s b e e n a p p l i e d t o most of t h e U n i t e d S t a t e s '\", \"text\": \"Gulf of M e x i c o a n d A t l a n t i c c o a s t l i n e s v u l n e r a b l e t o h u r r i c a n e s . A d e s c r i p t i o n o f t h e m o d e l ' s u s e a n d l i m i t a t i o n s f o r h u r r i c a n e s t o r m s u r g e f o r e c a s t i n g i s p r es e n t e d .\\nS o m e s p e c i f i c e x a m p l e s of t h e u s e of SLOSH a r e d r a w n f r o m H u r r i c a n e E l e n a , 1 9 8 5 .\"}, {\"section_title\": \"I n a d d i t i o n t o\", \"text\": \"SLOSH'S u s e f o r r e a l -t i m e s u r g e f o r e c a s t i n g , SLOSH is u s e d e x t e n s i v e l y i n h u r r icane evacuation planning. The model i s r u n w i t h s e v e r a l h u n d r e d h y p o t h e t i c a l h u r r i c a n e s , s e l e c t e d a c c o r d i n g t o a n a r e a ' s c l i m a t o l o g y .\\nT\"}, {\"section_title\": \"h e m o d e l g e n e r a t e s t h e f l o o d i n g e x p e c t e d f o r e a c h s t o r m .\", \"text\": \"Combining t h e s e f l o o d i n g p a t t e r n s h e l p s t o d e t e rmine a n a r e a ' s v u l n e r a b i l i t y t o h u r r i c a n e s .\\nAlso, model-generated winds assist p l a n n e r s\"},</code>  </p>\n<p>As you can see for some words, all the letters present in the word are separated by whitespace whereas for some words the letters don't have whitespaces in between them. Any idea on how to handle this? We can remove the whitespace between the letters, (For example: <code>w i t h s e v e r a l h u n d r e d h y p o t h e t i c a l h u r r i c a n e s ---&gt; withseveralhundredhypotheticalhurricanes</code>) but how can we make sure that the words are biforcated properly (<code>withseveralhundredhypotheticalhurricanes ----&gt; with several hundred hypothetical hurricanes</code>).</p>\n<p>Thanks in advance.  </p>",
  "messages": [
    {
      "id": "1341599",
      "postDate": "06/08/2021 18:50:56",
      "content": "<p>I recently came across this file in the training set <strong><em>8432e134-5cb0-4dfc-8794-b5b05ac8dea3.json</em></strong>. </p>\n<p>Here is the first few lines of the file.</p>\n<p><code>[{\"section_title\": \"Abstract\", \"text\": \"The National Weather Service (NWS) h a s d e v e l o p e d a h u r r i c a n e s t o r m s u r g e model c a l l e d SLOSH ( S e a ,\\n-Lake, a n d O v e r l a n d S u r g e s f r o m g u r r i c a n e s ) t o com-\"}, {\"section_title\": \"\", \"text\": \"p u t e h u r r i c a n e s t o r m s u r g e s , g i v e n s t o r m d a t a as input. The numerical model incorporates a dynamic coastline, overland flooding, and sub-grid feat u r e s s u c h a s b a r r i e r s , c u t s b e t w e e n b a r r i e r s , and one-dimensional flow along channels of varying width. The\"}, {\"section_title\": \"SLOSH model h a s b e e n a p p l i e d t o most of t h e U n i t e d S t a t e s '\", \"text\": \"Gulf of M e x i c o a n d A t l a n t i c c o a s t l i n e s v u l n e r a b l e t o h u r r i c a n e s . A d e s c r i p t i o n o f t h e m o d e l ' s u s e a n d l i m i t a t i o n s f o r h u r r i c a n e s t o r m s u r g e f o r e c a s t i n g i s p r es e n t e d .\\nS o m e s p e c i f i c e x a m p l e s of t h e u s e of SLOSH a r e d r a w n f r o m H u r r i c a n e E l e n a , 1 9 8 5 .\"}, {\"section_title\": \"I n a d d i t i o n t o\", \"text\": \"SLOSH'S u s e f o r r e a l -t i m e s u r g e f o r e c a s t i n g , SLOSH is u s e d e x t e n s i v e l y i n h u r r icane evacuation planning. The model i s r u n w i t h s e v e r a l h u n d r e d h y p o t h e t i c a l h u r r i c a n e s , s e l e c t e d a c c o r d i n g t o a n a r e a ' s c l i m a t o l o g y .\\nT\"}, {\"section_title\": \"h e m o d e l g e n e r a t e s t h e f l o o d i n g e x p e c t e d f o r e a c h s t o r m .\", \"text\": \"Combining t h e s e f l o o d i n g p a t t e r n s h e l p s t o d e t e rmine a n a r e a ' s v u l n e r a b i l i t y t o h u r r i c a n e s .\\nAlso, model-generated winds assist p l a n n e r s\"},</code>  </p>\n<p>As you can see for some words, all the letters present in the word are separated by whitespace whereas for some words the letters don't have whitespaces in between them. Any idea on how to handle this? We can remove the whitespace between the letters, (For example: <code>w i t h s e v e r a l h u n d r e d h y p o t h e t i c a l h u r r i c a n e s ---&gt; withseveralhundredhypotheticalhurricanes</code>) but how can we make sure that the words are biforcated properly (<code>withseveralhundredhypotheticalhurricanes ----&gt; with several hundred hypothetical hurricanes</code>).</p>\n<p>Thanks in advance.  </p>",
      "rawMarkdown": "I recently came across this file in the training set ***8432e134-5cb0-4dfc-8794-b5b05ac8dea3.json***. \n\nHere is the first few lines of the file.\n\n`[{\"section_title\": \"Abstract\", \"text\": \"The National Weather Service (NWS) h a s d e v e l o p e d a h u r r i c a n e s t o r m s u r g e model c a l l e d SLOSH ( S e a ,\\n-Lake, a n d O v e r l a n d S u r g e s f r o m g u r r i c a n e s ) t o com-\"}, {\"section_title\": \"\", \"text\": \"p u t e h u r r i c a n e s t o r m s u r g e s , g i v e n s t o r m d a t a as input. The numerical model incorporates a dynamic coastline, overland flooding, and sub-grid feat u r e s s u c h a s b a r r i e r s , c u t s b e t w e e n b a r r i e r s , and one-dimensional flow along channels of varying width. The\"}, {\"section_title\": \"SLOSH model h a s b e e n a p p l i e d t o most of t h e U n i t e d S t a t e s '\", \"text\": \"Gulf of M e x i c o a n d A t l a n t i c c o a s t l i n e s v u l n e r a b l e t o h u r r i c a n e s . A d e s c r i p t i o n o f t h e m o d e l ' s u s e a n d l i m i t a t i o n s f o r h u r r i c a n e s t o r m s u r g e f o r e c a s t i n g i s p r es e n t e d .\\nS o m e s p e c i f i c e x a m p l e s of t h e u s e of SLOSH a r e d r a w n f r o m H u r r i c a n e E l e n a , 1 9 8 5 .\"}, {\"section_title\": \"I n a d d i t i o n t o\", \"text\": \"SLOSH'S u s e f o r r e a l -t i m e s u r g e f o r e c a s t i n g , SLOSH is u s e d e x t e n s i v e l y i n h u r r icane evacuation planning. The model i s r u n w i t h s e v e r a l h u n d r e d h y p o t h e t i c a l h u r r i c a n e s , s e l e c t e d a c c o r d i n g t o a n a r e a ' s c l i m a t o l o g y .\\nT\"}, {\"section_title\": \"h e m o d e l g e n e r a t e s t h e f l o o d i n g e x p e c t e d f o r e a c h s t o r m .\", \"text\": \"Combining t h e s e f l o o d i n g p a t t e r n s h e l p s t o d e t e rmine a n a r e a ' s v u l n e r a b i l i t y t o h u r r i c a n e s .\\nAlso, model-generated winds assist p l a n n e r s\"}, `  \n\nAs you can see for some words, all the letters present in the word are separated by whitespace whereas for some words the letters don't have whitespaces in between them. Any idea on how to handle this? We can remove the whitespace between the letters, (For example: `w i t h s e v e r a l h u n d r e d h y p o t h e t i c a l h u r r i c a n e s ---> withseveralhundredhypotheticalhurricanes`) but how can we make sure that the words are biforcated properly (`withseveralhundredhypotheticalhurricanes ----> with several hundred hypothetical hurricanes`).\n\nThanks in advance.",
      "votes": null
    },
    {
      "id": "1343145",
      "postDate": "06/10/2021 04:35:45",
      "content": "<p>For this case, I think that the best action should be avoid this file. </p>",
      "rawMarkdown": "For this case, I think that the best action should be avoid this file.",
      "votes": null
    },
    {
      "id": "1343533",
      "postDate": "06/10/2021 09:00:01",
      "content": "<p>Okay. That sounds like a good idea. </p>",
      "rawMarkdown": "Okay. That sounds like a good idea.",
      "votes": null
    },
    {
      "id": "1345283",
      "postDate": "06/11/2021 13:15:59",
      "content": "<p>r u checking manually all 15k datasets??? </p>",
      "rawMarkdown": "r u checking manually all 15k datasets???",
      "votes": null
    },
    {
      "id": "1345288",
      "postDate": "06/11/2021 13:22:16",
      "content": "<p>Not really. I ran my model on the training set and got the predictions. The predictions were kind of unique for this document. So I opened the file and observed this. </p>",
      "rawMarkdown": "Not really. I ran my model on the training set and got the predictions. The predictions were kind of unique for this document. So I opened the file and observed this.",
      "votes": null
    },
    {
      "id": "1345432",
      "postDate": "06/11/2021 15:31:24",
      "content": "<p>Thank you for bringing this up, i will drop this file as well</p>",
      "rawMarkdown": "Thank you for bringing this up, i will drop this file as well",
      "votes": null
    },
    {
      "id": "1345436",
      "postDate": "06/11/2021 15:33:44",
      "content": "<p>You're welcome. </p>",
      "rawMarkdown": "You're welcome.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1343145,
      "author_name": "henryjavier",
      "author_url": "",
      "post_date": "06/10/2021 04:35:45",
      "content": "<p>For this case, I think that the best action should be avoid this file. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1343533,
          "author_name": "kartikbhargav",
          "author_url": "",
          "post_date": "06/10/2021 09:00:01",
          "content": "<p>Okay. That sounds like a good idea. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1345283,
      "author_name": "udaygurugubelli",
      "author_url": "",
      "post_date": "06/11/2021 13:15:59",
      "content": "<p>r u checking manually all 15k datasets??? </p>",
      "votes": null,
      "replies": [
        {
          "id": 1345288,
          "author_name": "kartikbhargav",
          "author_url": "",
          "post_date": "06/11/2021 13:22:16",
          "content": "<p>Not really. I ran my model on the training set and got the predictions. The predictions were kind of unique for this document. So I opened the file and observed this. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1345432,
      "author_name": "victorasso",
      "author_url": "",
      "post_date": "06/11/2021 15:31:24",
      "content": "<p>Thank you for bringing this up, i will drop this file as well</p>",
      "votes": null,
      "replies": [
        {
          "id": 1345436,
          "author_name": "kartikbhargav",
          "author_url": "",
          "post_date": "06/11/2021 15:33:44",
          "content": "<p>You're welcome. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1341599": "I recently came across this file in the training set ***8432e134-5cb0-4dfc-8794-b5b05ac8dea3.json***. \n\nHere is the first few lines of the file.\n\n`[{\"section_title\": \"Abstract\", \"text\": \"The National Weather Service (NWS) h a s d e v e l o p e d a h u r r i c a n e s t o r m s u r g e model c a l l e d SLOSH ( S e a ,\\n-Lake, a n d O v e r l a n d S u r g e s f r o m g u r r i c a n e s ) t o com-\"}, {\"section_title\": \"\", \"text\": \"p u t e h u r r i c a n e s t o r m s u r g e s , g i v e n s t o r m d a t a as input. The numerical model incorporates a dynamic coastline, overland flooding, and sub-grid feat u r e s s u c h a s b a r r i e r s , c u t s b e t w e e n b a r r i e r s , and one-dimensional flow along channels of varying width. The\"}, {\"section_title\": \"SLOSH model h a s b e e n a p p l i e d t o most of t h e U n i t e d S t a t e s '\", \"text\": \"Gulf of M e x i c o a n d A t l a n t i c c o a s t l i n e s v u l n e r a b l e t o h u r r i c a n e s . A d e s c r i p t i o n o f t h e m o d e l ' s u s e a n d l i m i t a t i o n s f o r h u r r i c a n e s t o r m s u r g e f o r e c a s t i n g i s p r es e n t e d .\\nS o m e s p e c i f i c e x a m p l e s of t h e u s e of SLOSH a r e d r a w n f r o m H u r r i c a n e E l e n a , 1 9 8 5 .\"}, {\"section_title\": \"I n a d d i t i o n t o\", \"text\": \"SLOSH'S u s e f o r r e a l -t i m e s u r g e f o r e c a s t i n g , SLOSH is u s e d e x t e n s i v e l y i n h u r r icane evacuation planning. The model i s r u n w i t h s e v e r a l h u n d r e d h y p o t h e t i c a l h u r r i c a n e s , s e l e c t e d a c c o r d i n g t o a n a r e a ' s c l i m a t o l o g y .\\nT\"}, {\"section_title\": \"h e m o d e l g e n e r a t e s t h e f l o o d i n g e x p e c t e d f o r e a c h s t o r m .\", \"text\": \"Combining t h e s e f l o o d i n g p a t t e r n s h e l p s t o d e t e rmine a n a r e a ' s v u l n e r a b i l i t y t o h u r r i c a n e s .\\nAlso, model-generated winds assist p l a n n e r s\"}, `  \n\nAs you can see for some words, all the letters present in the word are separated by whitespace whereas for some words the letters don't have whitespaces in between them. Any idea on how to handle this? We can remove the whitespace between the letters, (For example: `w i t h s e v e r a l h u n d r e d h y p o t h e t i c a l h u r r i c a n e s ---> withseveralhundredhypotheticalhurricanes`) but how can we make sure that the words are biforcated properly (`withseveralhundredhypotheticalhurricanes ----> with several hundred hypothetical hurricanes`).\n\nThanks in advance.",
    "1343145": "For this case, I think that the best action should be avoid this file.",
    "1343533": "Okay. That sounds like a good idea.",
    "1345283": "r u checking manually all 15k datasets???",
    "1345288": "Not really. I ran my model on the training set and got the predictions. The predictions were kind of unique for this document. So I opened the file and observed this.",
    "1345432": "Thank you for bringing this up, i will drop this file as well",
    "1345436": "You're welcome."
  },
  "source": "meta"
}