\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1206143%2F15abe6d0b77669bcfc4db157c783ee34%2Fdbt_bosch_data_model.jpg?generation=1732786543827459&alt=media\")
3 years has passed, I am still interested in the Bosche huge data set and want to use it to solve quality issue, process improve.
\nToday, I use mixed data science toolset to work on it.
use dbt, duckdb for huge data process, GB level wide columns data can be transformed to desired long table and stored in parquet format
\n
By using BupaR(a process mining opensource package in R language) to create , analysis, visulize and predict the quality issue.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1206143%2Fdb3d3fa0a39de4f19851df2e8bc32de3%2Ffirst100case_pm.jpg?generation=1732785995117524&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1206143%2F0d2777e067703832c990feafa46427ea%2Ffirst5000case_fail.jpg?generation=1732786017890397&alt=media\")
2020 Year
\nI was interested in Bosch Production line Performance competition 1 years. At that tim, the big data blocked me to further analysis.
\nRecently, I was focused on combining database + python to do EDA & engineering first and then lets the refined data go to sklearn or ther machine learning libs.
\nBelow is last week study to the Bosch Production line Performance.
\n. Target
Tools
\n** Method**
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1206143%2F264543947900d929c59598d3512d8721%2Fparts.jpg?generation=1570580546384640&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1206143%2Fc69016307f0071df0a38a25f4e49664b%2Ffeature%20num.jpg?generation=1570580580511519&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1206143%2F1aed45450206d6377f557ca9582d244c%2Fstation%20map%20per%20volume.jpg?generation=1570580625129669&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1206143%2Fcb1542ffc4855a1f7ca74dc2b9421d4f%2Fvolume.jpg?generation=1570580698479850&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1206143%2F2602b13786b4e33cb80d0c4c665a12de%2Fduration.jpg?generation=1570580519874540&alt=media\")
3 years has passed, I am still interested in the Bosche huge data set and want to use it to solve quality issue, process improve.
\nToday, I use mixed data science toolset to work on it.
use dbt, duckdb for huge data process, GB level wide columns data can be transformed to desired long table and stored in parquet format
\n
By using BupaR(a process mining opensource package in R language) to create , analysis, visulize and predict the quality issue.
\n
2020 Year
\nI was interested in Bosch Production line Performance competition 1 years. At that tim, the big data blocked me to further analysis.
\nRecently, I was focused on combining database + python to do EDA & engineering first and then lets the refined data go to sklearn or ther machine learning libs.
\nBelow is last week study to the Bosch Production line Performance.
\n. Target
Tools
\n** Method**
\nIs there a possibility to get a similar data set as the one for Bosch Production Line Performance but that can be handled on a PC? I would like to further investigate some other aspects of the problem but at a smaller scale.\nIs there any other data source similar to this available on-line? Thanks, AleX
", "rawMarkdown": "Is there a possibility to get a similar data set as the one for Bosch Production Line Performance but that can be handled on a PC? I would like to further investigate some other aspects of the problem but at a smaller scale.\nIs there any other data source similar to this available on-line? Thanks, AleX", "votes": null }, { "id": "1342485", "postDate": "06/09/2021 13:48:53", "content": "Would you like to share your source code? This looks very interesting
", "rawMarkdown": "Would you like to share your source code? This looks very interesting", "votes": null }, { "id": "1343143", "postDate": "06/10/2021 04:33:31", "content": "thanks for interesting. I used to to manually write python + sqlalchmey code to run it. Now, I has switched to using DBT(Data build tool) to do feature engineering and machine learning. It can do version control, unit test and show the data feature engineering lineage.
\nI may share some code about DBT + kaggle case in git in coming months.
handle those data in PC. it might be challenge. I used to run some featuring engineering in a PC, but it took several hours.
\nRecently, I used DBT to do those similary tasks. the task can be breakdown into small module and based on DAG calculation. So, some rework could be reduced.
You can try DBT + Postgres13( install plpython extension + lightgbm) on it. As the task break down in to small modules, the work might be done in a PC.
", "rawMarkdown": "handle those data in PC. it might be challenge. I used to run some featuring engineering in a PC, but it took several hours. \nRecently, I used DBT to do those similary tasks. the task can be breakdown into small module and based on DAG calculation. So, some rework could be reduced. \n\nYou can try DBT + Postgres13( install plpython extension + lightgbm) on it. As the task break down in to small modules, the work might be done in a PC.", "votes": null }, { "id": "1349242", "postDate": "06/14/2021 15:59:36", "content": "That would be awesome thank you
", "rawMarkdown": "That would be awesome thank you", "votes": null } ], "comments": [ { "id": 1342485, "author_name": "janikbischoff", "author_url": "", "post_date": "06/09/2021 13:48:53", "content": "Would you like to share your source code? This looks very interesting
", "votes": null, "replies": [ { "id": 1343143, "author_name": "wangyong101", "author_url": "", "post_date": "06/10/2021 04:33:31", "content": "thanks for interesting. I used to to manually write python + sqlalchmey code to run it. Now, I has switched to using DBT(Data build tool) to do feature engineering and machine learning. It can do version control, unit test and show the data feature engineering lineage.
\nI may share some code about DBT + kaggle case in git in coming months.
That would be awesome thank you
", "votes": null, "replies": [] } ] }, { "id": 744312, "author_name": "petchesi", "author_url": "", "post_date": "02/12/2020 18:18:38", "content": "Is there a possibility to get a similar data set as the one for Bosch Production Line Performance but that can be handled on a PC? I would like to further investigate some other aspects of the problem but at a smaller scale.\nIs there any other data source similar to this available on-line? Thanks, AleX
", "votes": null, "replies": [ { "id": 1343147, "author_name": "wangyong101", "author_url": "", "post_date": "06/10/2021 04:36:31", "content": "handle those data in PC. it might be challenge. I used to run some featuring engineering in a PC, but it took several hours.
\nRecently, I used DBT to do those similary tasks. the task can be breakdown into small module and based on DAG calculation. So, some rework could be reduced.
You can try DBT + Postgres13( install plpython extension + lightgbm) on it. As the task break down in to small modules, the work might be done in a PC.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "644540": "# 2024 change\n\n3 years has passed, I am still interested in the Bosche huge data set and want to use it to solve quality issue, process improve.\nToday, I use mixed data science toolset to work on it.\n\n## data preprocess : \nuse dbt, duckdb for huge data process, GB level wide columns data can be transformed to desired long table and stored in parquet format\n\n\n## process analysis.\nBy using BupaR(a process mining opensource package in R language) to create , analysis, visulize and predict the quality issue. \n\n## Visulization example\n### 1. first 100 case based process map\n\n\n\n### 2. first 5000 case which has some failured case\n\n\n==============================================================\n2020 Year\n\nI was interested in Bosch Production line Performance competition 1 years. At that tim, the big data blocked me to further analysis. \n\nRecently, I was focused on combining database + python to do EDA & engineering first and then lets the refined data go to sklearn or ther machine learning libs.\n\nBelow is last week study to the Bosch Production line Performance.\n\n**. Target**\n1. Draw Total Volume based layout\n2.Draw Total worktime based layout.\n3.Draw avg worktime based layout\n4. Draw stddev worktime based layout\n layout1,2 will help us to understand how the parts was flow from begining to end in total volume and work time viewpoint. This is fundatmention map, later performance improvement discussion could start based on that. \n Regarding performance, I would like change it to profile concept. That mean,\n Profile = Selling Price - Cost. \n Assumed the selling price is fixed, so if the cost can down, the profile will be increased accordingly.\n The traditional cost lead by 7 wastes, that is TIMWOOD. Based on the Bosch produciton line data, there is some clue showing 3 wastes: Waiting, Over Processing, and Inventory Waste might exists. Of course, defect exists, that is Response 1 for quality control. \n Now, I will show my study step by step. \n\n**Input:**\n 1. Kaggle Data \n 2. Manufacturing Knowledge & Experience (LEAN and SixSigma and TOC)\n 3. Reference \n https://www.kaggle.com/gingerman/shopfloor-visualization-2-0\n \n**Tools**\n 1. Jupyter Scipy/Scientist Docker (a python docker image)\n 2. PostgresSQL V11 (a open source relation database)\n 3. superset (a data visualization dashborad tools)\n \n** Method**\n1. ETL from csv to postgresql \n2. EDA based on postgresql \n\n**Output**\n1. Parts number\n\n\n2. Feature number\n\n\n3. work duration - station vs features \n\n\n4. Layout (station based, id 999 means the work finished. the others means orginal Bosch production line station id)\n4.1 Total Volume based\n\n\n\n4.2 Total worktime based(average vs stand deviaiton)\n", "744312": "Is there a possibility to get a similar data set as the one for Bosch Production Line Performance but that can be handled on a PC? I would like to further investigate some other aspects of the problem but at a smaller scale.\nIs there any other data source similar to this available on-line? Thanks, AleX", "1342485": "Would you like to share your source code? This looks very interesting", "1343143": "thanks for interesting. I used to to manually write python + sqlalchmey code to run it. Now, I has switched to using DBT(Data build tool) to do feature engineering and machine learning. It can do version control, unit test and show the data feature engineering lineage. \nI may share some code about DBT + kaggle case in git in coming months.", "1343147": "handle those data in PC. it might be challenge. I used to run some featuring engineering in a PC, but it took several hours. \nRecently, I used DBT to do those similary tasks. the task can be breakdown into small module and based on DAG calculation. So, some rework could be reduced. \n\nYou can try DBT + Postgres13( install plpython extension + lightgbm) on it. As the task break down in to small modules, the work might be done in a PC.", "1349242": "That would be awesome thank you" }, "source": "meta" }