For bigger_govt_dataset_list, I collected the top 22,000 popular gov_datasets from Data.gov added to the original data_set_800 provided by @mlconsult. All labels are cleaned according to the competition requirement.
More bigger-govt-dataset: Coleridge additional_gov_datasets_22000popular
\nUsing example: External_Datasets_Matching + MLM # Literal Matching
If you find the data useful, please share your findings here. Let's find the magic ;)
", "messages": [ { "id": "1322154", "postDate": "05/25/2021 08:19:35", "content": "For bigger_govt_dataset_list, I collected the top 22,000 popular gov_datasets from Data.gov added to the original data_set_800 provided by @mlconsult. All labels are cleaned according to the competition requirement.
More bigger-govt-dataset: Coleridge additional_gov_datasets_22000popular
\nUsing example: External_Datasets_Matching + MLM # Literal Matching
If you find the data useful, please share your findings here. Let's find the magic ;)
", "rawMarkdown": "For `bigger_govt_dataset_list`, I collected the **top 22,000 popular** `gov_datasets` from [Data.gov](https://www.data.gov/) added to the original `data_set_800` provided by @mlconsult. All `labels` are cleaned according to the competition requirement.\n\nMore bigger-govt-dataset: [Coleridge additional_gov_datasets_22000popular](https://www.kaggle.com/chienhsianghung/coleridge-additional-gov-datasets-22000popular)\nUsing example: [External_Datasets_Matching + MLM # Literal Matching](https://www.kaggle.com/chienhsianghung/external-datasets-matching-mlm?cellId=18)\n\nIf you find the data useful, please share your findings here. Let's find the magic ;)", "votes": null }, { "id": "1322172", "postDate": "05/25/2021 08:42:09", "content": "Nice, we're thinking about using external data but the government websites are a mess to navigate.
", "rawMarkdown": "Nice, we're thinking about using external data but the government websites are a mess to navigate.", "votes": null }, { "id": "1322544", "postDate": "05/25/2021 13:53:52", "content": "Thanks, I created this just to try if the greater labels for str matching could somehow enhance the LB. Do you get 0.661 by only using the competition training set π²
", "rawMarkdown": "Thanks, I created this just to try if the greater labels for str matching could somehow enhance the LB. Do you get **0.661** by only using the competition training set π²", "votes": null }, { "id": "1322563", "postDate": "05/25/2021 14:06:22", "content": "So can these string matching methods be used in the final ranking?π
", "rawMarkdown": "So can these string matching methods be used in the final ranking?π", "votes": null }, { "id": "1322578", "postDate": "05/25/2021 14:15:48", "content": "I want to say YES, given these are the hottest datasets π Who knows
", "rawMarkdown": "I want to say YES, given these are the hottest datasets π Who knows", "votes": null }, { "id": "1323125", "postDate": "05/26/2021 02:44:48", "content": "nice to see, finally you crossed the base line..
", "rawMarkdown": "nice to see, finally you crossed the base line..", "votes": null }, { "id": "1323269", "postDate": "05/26/2021 05:48:15", "content": "Hi, may I ask how did you scrape these data?
", "rawMarkdown": "Hi, may I ask how did you scrape these data?", "votes": null }, { "id": "1323729", "postDate": "05/26/2021 12:17:15", "content": "Unfortunately not, my model performance w/o str matching still gets nowhere π§
", "rawMarkdown": "Unfortunately not, my model performance w/o str matching still gets nowhere π§", "votes": null }, { "id": "1323734", "postDate": "05/26/2021 12:19:51", "content": "I scrapped through Data.gov according to the popularity for the name of each dataset.
", "rawMarkdown": "I scrapped through **Data.gov** according to the *popularity* for the name of each dataset.", "votes": null }, { "id": "1323792", "postDate": "05/26/2021 12:51:29", "content": "Here is a notebook that downloads government datasets titles from the government datasets APIs - I get about 103,000 line items - many (similar) duplicates - I haven't tried straight up matching, but have played with it a bit. There are some of the APIs that are not in the same format so I do not use them, but they just need some cleaning. Below is also a link to the GOV open dashboard for dataset APIs.
\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df
\nhttps://labs.data.gov/dashboard/offices/qa
", "rawMarkdown": "Here is a notebook that downloads government datasets titles from the government datasets APIs - I get about 103,000 line items - many (similar) duplicates - I haven't tried straight up matching, but have played with it a bit. There are some of the APIs that are not in the same format so I do not use them, but they just need some cleaning. Below is also a link to the GOV open dashboard for dataset APIs. \n\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df\n\nhttps://labs.data.gov/dashboard/offices/qa", "votes": null }, { "id": "1323794", "postDate": "05/26/2021 12:53:05", "content": "Thanks for your answer. However, what I'm curious about is if you are using any scraping tool(e.g. BeautifulSoup) to collect data automatically.
", "rawMarkdown": "Thanks for your answer. However, what I'm curious about is if you are using any scraping tool(e.g. BeautifulSoup) to collect data automatically.", "votes": null }, { "id": "1323813", "postDate": "05/26/2021 13:02:38", "content": "lol @keetar yes, for tools I used request and bs4
Here is how I do it: web scraping for bigger_govt_dataset_list
", "rawMarkdown": "lol @keetar yes, for tools I used `request` and `bs4`\n\nHere is how I do it: [web scraping for bigger_govt_dataset_list](https://www.kaggle.com/chienhsianghung/web-scraping-for-bigger-govt-dataset-list)", "votes": null }, { "id": "1323833", "postDate": "05/26/2021 13:10:35", "content": "Wow, Thank you very much!
", "rawMarkdown": "Wow, Thank you very much!", "votes": null }, { "id": "1323834", "postDate": "05/26/2021 13:10:40", "content": "That's superb, I literally just scrapped through the website, didn't know they have APIs to use lol
", "rawMarkdown": "That's superb, I literally just scrapped through the website, didn't know they have APIs to use lol", "votes": null }, { "id": "1323840", "postDate": "05/26/2021 13:14:48", "content": "It's really helpful. Thank you!
", "rawMarkdown": "It's really helpful. Thank you!", "votes": null }, { "id": "1325530", "postDate": "05/27/2021 20:07:04", "content": "Wow, Thank you very much!π
", "rawMarkdown": "Wow, Thank you very much!π", "votes": null }, { "id": "1327513", "postDate": "05/29/2021 11:18:41", "content": "Thank you very much
", "rawMarkdown": "Thank you very much", "votes": null }, { "id": "1327742", "postDate": "05/29/2021 15:34:40", "content": "Thank You for the link
", "rawMarkdown": "Thank You for the link", "votes": null }, { "id": "1328055", "postDate": "05/29/2021 22:47:03", "content": "Here is a notebook that downloads government datasets titles from the government datasets APIs - I get about 103,000 line items - many (similar) duplicates - I haven't tried straight up matching, but have played with it a bit. There are some of the APIs that are not in the same format so I do not use them, but they just need some cleaning. Below is also a link to the GOV open dashboard for dataset APIs.
\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df
\nhttps://labs.data.gov/dashboard/offices/qa
", "rawMarkdown": "Here is a notebook that downloads government datasets titles from the government datasets APIs - I get about 103,000 line items - many (similar) duplicates - I haven't tried straight up matching, but have played with it a bit. There are some of the APIs that are not in the same format so I do not use them, but they just need some cleaning. Below is also a link to the GOV open dashboard for dataset APIs.\n\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df\n\nhttps://labs.data.gov/dashboard/offices/qa", "votes": null }, { "id": "1328599", "postDate": "05/30/2021 11:57:42", "content": "nice work. Thank you!
", "rawMarkdown": "nice work. Thank you!", "votes": null }, { "id": "1330089", "postDate": "05/31/2021 15:29:43", "content": "what scrapping tools are you using?
\nalso great work!
Thank you @perkymaster. I used request and bs4.
Here is how I scrapped it: web scraping for bigger_govt_dataset_list
", "rawMarkdown": "Thank you @perkymaster. I used `request` and `bs4`.\n\nHere is how I scrapped it: [web scraping for bigger_govt_dataset_list](https://www.kaggle.com/chienhsianghung/web-scraping-for-bigger-govt-dataset-list)", "votes": null }, { "id": "1330168", "postDate": "05/31/2021 16:21:02", "content": "this uses the GOV api and downloads to DF.
\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df
", "rawMarkdown": "this uses the GOV api and downloads to DF.\n\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df", "votes": null }, { "id": "1330601", "postDate": "06/01/2021 01:56:18", "content": "Thank you @chienhsianghung really great workπ.
\nDo you use any generalized data cleaning technique for all datasets?
Hey Buddy can I use Your DataSet for my submission
", "rawMarkdown": "Hey Buddy can I use Your DataSet for my submission", "votes": null }, { "id": "1335985", "postDate": "06/04/2021 15:10:04", "content": "Government websites are always so messy to work on! Great Work!
", "rawMarkdown": "Government websites are always so messy to work on! Great Work!", "votes": null }, { "id": "1343444", "postDate": "06/10/2021 07:55:10", "content": "Thanks for sharing the dataset. Is there cased(preserving capital letters) dataset available??
", "rawMarkdown": "Thanks for sharing the dataset. Is there cased(preserving capital letters) dataset available??", "votes": null } ], "comments": [ { "id": 1322172, "author_name": "suicaokhoailang", "author_url": "", "post_date": "05/25/2021 08:42:09", "content": "Nice, we're thinking about using external data but the government websites are a mess to navigate.
", "votes": null, "replies": [ { "id": 1322544, "author_name": "chienhsianghung", "author_url": "", "post_date": "05/25/2021 13:53:52", "content": "Thanks, I created this just to try if the greater labels for str matching could somehow enhance the LB. Do you get 0.661 by only using the competition training set π²
", "votes": null, "replies": [] }, { "id": 1323792, "author_name": "mlconsult", "author_url": "", "post_date": "05/26/2021 12:51:29", "content": "Here is a notebook that downloads government datasets titles from the government datasets APIs - I get about 103,000 line items - many (similar) duplicates - I haven't tried straight up matching, but have played with it a bit. There are some of the APIs that are not in the same format so I do not use them, but they just need some cleaning. Below is also a link to the GOV open dashboard for dataset APIs.
\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df
\nhttps://labs.data.gov/dashboard/offices/qa
", "votes": null, "replies": [] }, { "id": 1323834, "author_name": "chienhsianghung", "author_url": "", "post_date": "05/26/2021 13:10:40", "content": "That's superb, I literally just scrapped through the website, didn't know they have APIs to use lol
", "votes": null, "replies": [] }, { "id": 1323840, "author_name": "keetar", "author_url": "", "post_date": "05/26/2021 13:14:48", "content": "It's really helpful. Thank you!
", "votes": null, "replies": [] } ] }, { "id": 1322563, "author_name": "vanle73", "author_url": "", "post_date": "05/25/2021 14:06:22", "content": "So can these string matching methods be used in the final ranking?π
", "votes": null, "replies": [ { "id": 1322578, "author_name": "chienhsianghung", "author_url": "", "post_date": "05/25/2021 14:15:48", "content": "I want to say YES, given these are the hottest datasets π Who knows
", "votes": null, "replies": [] } ] }, { "id": 1323125, "author_name": "udaygurugubelli", "author_url": "", "post_date": "05/26/2021 02:44:48", "content": "nice to see, finally you crossed the base line..
", "votes": null, "replies": [ { "id": 1323729, "author_name": "chienhsianghung", "author_url": "", "post_date": "05/26/2021 12:17:15", "content": "Unfortunately not, my model performance w/o str matching still gets nowhere π§
", "votes": null, "replies": [] } ] }, { "id": 1323269, "author_name": "keetar", "author_url": "", "post_date": "05/26/2021 05:48:15", "content": "Hi, may I ask how did you scrape these data?
", "votes": null, "replies": [ { "id": 1323734, "author_name": "chienhsianghung", "author_url": "", "post_date": "05/26/2021 12:19:51", "content": "I scrapped through Data.gov according to the popularity for the name of each dataset.
", "votes": null, "replies": [] }, { "id": 1323794, "author_name": "keetar", "author_url": "", "post_date": "05/26/2021 12:53:05", "content": "Thanks for your answer. However, what I'm curious about is if you are using any scraping tool(e.g. BeautifulSoup) to collect data automatically.
", "votes": null, "replies": [] }, { "id": 1323813, "author_name": "chienhsianghung", "author_url": "", "post_date": "05/26/2021 13:02:38", "content": "lol @keetar yes, for tools I used request and bs4
Here is how I do it: web scraping for bigger_govt_dataset_list
", "votes": null, "replies": [] }, { "id": 1323833, "author_name": "keetar", "author_url": "", "post_date": "05/26/2021 13:10:35", "content": "Wow, Thank you very much!
", "votes": null, "replies": [] } ] }, { "id": 1325530, "author_name": "mohamedbakrey", "author_url": "", "post_date": "05/27/2021 20:07:04", "content": "Wow, Thank you very much!π
", "votes": null, "replies": [] }, { "id": 1327513, "author_name": "bibibababiba", "author_url": "", "post_date": "05/29/2021 11:18:41", "content": "Thank you very much
", "votes": null, "replies": [] }, { "id": 1327742, "author_name": "uzairzahidi", "author_url": "", "post_date": "05/29/2021 15:34:40", "content": "Thank You for the link
", "votes": null, "replies": [] }, { "id": 1328055, "author_name": "mlconsult", "author_url": "", "post_date": "05/29/2021 22:47:03", "content": "Here is a notebook that downloads government datasets titles from the government datasets APIs - I get about 103,000 line items - many (similar) duplicates - I haven't tried straight up matching, but have played with it a bit. There are some of the APIs that are not in the same format so I do not use them, but they just need some cleaning. Below is also a link to the GOV open dashboard for dataset APIs.
\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df
\nhttps://labs.data.gov/dashboard/offices/qa
", "votes": null, "replies": [] }, { "id": 1328599, "author_name": "winniy", "author_url": "", "post_date": "05/30/2021 11:57:42", "content": "nice work. Thank you!
", "votes": null, "replies": [] }, { "id": 1330089, "author_name": "perkymaster", "author_url": "", "post_date": "05/31/2021 15:29:43", "content": "what scrapping tools are you using?
\nalso great work!
Thank you @perkymaster. I used request and bs4.
Here is how I scrapped it: web scraping for bigger_govt_dataset_list
", "votes": null, "replies": [] }, { "id": 1330168, "author_name": "mlconsult", "author_url": "", "post_date": "05/31/2021 16:21:02", "content": "this uses the GOV api and downloads to DF.
\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df
", "votes": null, "replies": [] } ] }, { "id": 1330601, "author_name": "yashdixit24", "author_url": "", "post_date": "06/01/2021 01:56:18", "content": "Thank you @chienhsianghung really great workπ.
\nDo you use any generalized data cleaning technique for all datasets?
Hey Buddy can I use Your DataSet for my submission
", "votes": null, "replies": [] }, { "id": 1335985, "author_name": "shubhamzorc9", "author_url": "", "post_date": "06/04/2021 15:10:04", "content": "Government websites are always so messy to work on! Great Work!
", "votes": null, "replies": [] }, { "id": 1343444, "author_name": "jkreddy", "author_url": "", "post_date": "06/10/2021 07:55:10", "content": "Thanks for sharing the dataset. Is there cased(preserving capital letters) dataset available??
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1322154": "For `bigger_govt_dataset_list`, I collected the **top 22,000 popular** `gov_datasets` from [Data.gov](https://www.data.gov/) added to the original `data_set_800` provided by @mlconsult. All `labels` are cleaned according to the competition requirement.\n\nMore bigger-govt-dataset: [Coleridge additional_gov_datasets_22000popular](https://www.kaggle.com/chienhsianghung/coleridge-additional-gov-datasets-22000popular)\nUsing example: [External_Datasets_Matching + MLM # Literal Matching](https://www.kaggle.com/chienhsianghung/external-datasets-matching-mlm?cellId=18)\n\nIf you find the data useful, please share your findings here. Let's find the magic ;)", "1322172": "Nice, we're thinking about using external data but the government websites are a mess to navigate.", "1322544": "Thanks, I created this just to try if the greater labels for str matching could somehow enhance the LB. Do you get **0.661** by only using the competition training set π²", "1322563": "So can these string matching methods be used in the final ranking?π", "1322578": "I want to say YES, given these are the hottest datasets π Who knows", "1323125": "nice to see, finally you crossed the base line..", "1323269": "Hi, may I ask how did you scrape these data?", "1323729": "Unfortunately not, my model performance w/o str matching still gets nowhere π§", "1323734": "I scrapped through **Data.gov** according to the *popularity* for the name of each dataset.", "1323792": "Here is a notebook that downloads government datasets titles from the government datasets APIs - I get about 103,000 line items - many (similar) duplicates - I haven't tried straight up matching, but have played with it a bit. There are some of the APIs that are not in the same format so I do not use them, but they just need some cleaning. Below is also a link to the GOV open dashboard for dataset APIs. \n\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df\n\nhttps://labs.data.gov/dashboard/offices/qa", "1323794": "Thanks for your answer. However, what I'm curious about is if you are using any scraping tool(e.g. BeautifulSoup) to collect data automatically.", "1323813": "lol @keetar yes, for tools I used `request` and `bs4`\n\nHere is how I do it: [web scraping for bigger_govt_dataset_list](https://www.kaggle.com/chienhsianghung/web-scraping-for-bigger-govt-dataset-list)", "1323833": "Wow, Thank you very much!", "1323834": "That's superb, I literally just scrapped through the website, didn't know they have APIs to use lol", "1323840": "It's really helpful. Thank you!", "1325530": "Wow, Thank you very much!π", "1327513": "Thank you very much", "1327742": "Thank You for the link", "1328055": "Here is a notebook that downloads government datasets titles from the government datasets APIs - I get about 103,000 line items - many (similar) duplicates - I haven't tried straight up matching, but have played with it a bit. There are some of the APIs that are not in the same format so I do not use them, but they just need some cleaning. Below is also a link to the GOV open dashboard for dataset APIs.\n\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df\n\nhttps://labs.data.gov/dashboard/offices/qa", "1328599": "nice work. Thank you!", "1330089": "what scrapping tools are you using?\nalso great work!", "1330111": "Thank you @perkymaster. I used `request` and `bs4`.\n\nHere is how I scrapped it: [web scraping for bigger_govt_dataset_list](https://www.kaggle.com/chienhsianghung/web-scraping-for-bigger-govt-dataset-list)", "1330168": "this uses the GOV api and downloads to DF.\n\nhttps://www.kaggle.com/mlconsult/govt-dataset-json-to-df", "1330601": "Thank you @chienhsianghung really great workπ. \nDo you use any generalized data cleaning technique for all datasets?", "1334509": "Hey Buddy can I use Your DataSet for my submission", "1335985": "Government websites are always so messy to work on! Great Work!", "1343444": "Thanks for sharing the dataset. Is there cased(preserving capital letters) dataset available??" }, "source": "meta" }