Traditionally this is Foxtrot's job, but they don't seem to be in this competition. It went really well in the Amazon one when a bunch of people did this, so I figured I'd try it here.
The idea behind this approach is that instead of worrying about interpolating, just let your model figure it out. It uses Ridge Regression from scikit-learn to do that so you can kind of think of it as a form of linear interpolation of all of the GEFS gird data to each station that is learned by the model instead of using some heuristic like distance. Additionally it uses all of the data instead of only the 'usrwf_sfc' data.
I've built off of this code to get my current model. You can probably squeeze a bit more out of this code by trying different methods of merging the data or trying different subsets of the files to be used but I have a feeling that you can't get much better than 2200k error using a linear model like this on the raw data and need to move to either a non-linear model to do better (polynomial/rbf regression, support vector regression, random forest, GBM, neural net, etc...) or get clever with feature engineering/selection.
It takes about 15-20 minutes to run on a Core i7 laptop and shouldn't need more than a gig or so of RAM.
Let me know if you have any questions!
Traditionally this is Foxtrot's job, but they don't seem to be in this competition. It went really well in the Amazon one when a bunch of people did this, so I figured I'd try it here.
The idea behind this approach is that instead of worrying about interpolating, just let your model figure it out. It uses Ridge Regression from scikit-learn to do that so you can kind of think of it as a form of linear interpolation of all of the GEFS gird data to each station that is learned by the model instead of using some heuristic like distance. Additionally it uses all of the data instead of only the 'usrwf_sfc' data.
I've built off of this code to get my current model. You can probably squeeze a bit more out of this code by trying different methods of merging the data or trying different subsets of the files to be used but I have a feeling that you can't get much better than 2200k error using a linear model like this on the raw data and need to move to either a non-linear model to do better (polynomial/rbf regression, support vector regression, random forest, GBM, neural net, etc...) or get clever with feature engineering/selection.
It takes about 15-20 minutes to run on a Core i7 laptop and shouldn't need more than a gig or so of RAM.
Let me know if you have any questions!
Thanks for the code!
\nOne comment, I did not run it yet and only just read it so apologies if I'm off-base, but wouldn't it be better to set a lower limit to your predictions when you run a linear regression with the GEFS data? I would imagine there would be a few predictions that are negative and below 0 doesn't make sense in this context.
", "rawMarkdown": "", "votes": null }, { "id": "28156", "postDate": "08/03/2013 21:12:10", "content": "Good point, min/max clipping the predictions resulted in another 10K(ish) drop on the CV, but I'm out of submissions so I can't verify it. Really simple to implement, just after the model.predict() line add:
preds = np.clip(preds,np.min(trainY),np.max(trainY))
Thank you Alec for sharing the code.
\nI receive the following error when I run the code.
\nIt's from the load_GEFS_file function.
\nI couldn't figure how to solve this problem. Any direction would be much appreciated.
\nThanks again!!
", "rawMarkdown": "", "votes": null }, { "id": "28170", "postDate": "08/04/2013 09:40:36", "content": "I was able to run it as is. Have you installed the netcdf4 package?
\nAlso, have you checked the files are in the correct location?
", "rawMarkdown": "", "votes": null }, { "id": "28171", "postDate": "08/04/2013 09:46:30", "content": "Yes I installed netcdf4 package.
\nAnd I put all the files in a folder called "data"...
", "rawMarkdown": "", "votes": null }, { "id": "28172", "postDate": "08/04/2013 10:09:30", "content": "I had to revert back to python 2.7 to run it (can't remember why) so I installed netcdf4 for 3.3 and 2.7
", "rawMarkdown": "", "votes": null }, { "id": "28179", "postDate": "08/04/2013 18:05:29", "content": "[quote=Adil;28168]
\nThank you Alec for sharing the code.
\nI receive the following error when I run the code.
\nIt's from the load_GEFS_file function.
\nI couldn't figure how to solve this problem. Any direction would be much appreciated.
\nThanks again!!
\n[/quote]
My guess is somehow the GEFS data you're loading doesn't follow the normal shape of (n,11,5,9,16) do you mind posting what shape X is before that line?
[quote=Alec Radford;28098]
\nThe idea behind this approach is that instead of worrying about interpolating, just let your model figure it out.
\n[/quote]
\nHello!
\nHere's what I think is an interesting paper related to this idea:
\nhttp://www.sciencedirect.com/science/article/pii/S1364815211001654
\nCheers, thanks for sharing!
", "rawMarkdown": "", "votes": null }, { "id": "28200", "postDate": "08/05/2013 10:10:00", "content": "Thanks for paper. I used a Random Forest on Alec's code but I got a worse result than his linear model and it took ages to run. Having to rethink it.
\n\n
EDIT: Though I'm new to Python so might not have done it correctly!
", "rawMarkdown": "", "votes": null }, { "id": "28212", "postDate": "08/05/2013 15:34:51", "content": "[quote=Alec Radford;28179]
\n[quote=Adil;28168]
\nThank you Alec for sharing the code.
\nI receive the following error when I run the code.
\nIt's from the load_GEFS_file function.
\nI couldn't figure how to solve this problem. Any direction would be much appreciated.
\nThanks again!!
\n[/quote]
My guess is somehow the GEFS data you're loading doesn't follow the normal shape of (n,11,5,9,16) do you mind posting what shape X is before that line?
[/quote]
\nThanks Alec.
\nI could get the code to run. The problem was in: X = nc.Dataset(path,'r+').variables.values()[-1][:]
\nThe nc.Dataset(path,'r+').variables.values() is an unordered dictionary. Therefore if you take the last element using [-1], sometimes it takes the wrong element like 'lon'.
\nI solved it by defining the write mapping between the file and the target element.
\nmapping = {'dswrf_sfc' : 'Downward_Short-Wave_Rad_Flux',
'dlwrf_sfc' : 'Downward_Long-Wave_Rad_Flux',
'uswrf_sfc' : 'Upward_Short-Wave_Rad_Flux',
'ulwrf_sfc' : 'Upward_Long-Wave_Rad_Flux_surface',
'ulwrf_tatm': 'Upward_Long-Wave_Rad_Flux',
'pwat_eatm' : 'Precipitable_water',
'tcdc_eatm' : 'Total_cloud_cover',
'apcp_sfc' : 'Total_precipitation',
'pres_msl': 'Pressure',
'spfh_2m' : 'Specific_humidity_height_above_ground',
'tcolc_eatm' : 'Total_Column-Integrated_Condensate',
'tmax_2m' : 'Maximum_temperature',
'tmin_2m' : 'Minimum_temperature',
'tmp_2m' : 'Temperature_height_above_ground',
'tmp_sfc' : 'Temperature_surface'}
X = nc.Dataset(path).variables[mapping[data_type]][:]
\n\n
Thanks again for sharing the code!!
\n", "rawMarkdown": "", "votes": null }, { "id": "28440", "postDate": "08/08/2013 15:34:46", "content": "
More of a Python question. I've been trying to run other algorithms using your data transformation. Is there a way to change the shape as a lot of the algorithms complain about it? Newbie to Python, usually use R
", "rawMarkdown": "", "votes": null }, { "id": "28466", "postDate": "08/08/2013 22:40:01", "content": "Numpy.reshape will reshape the arrays.
", "rawMarkdown": "", "votes": null }, { "id": "28719", "postDate": "08/15/2013 09:17:47", "content": "In reply to DomCastro.
\nIt is expected. Linear models in general run faster than non-linear models like Random Forest. It will be unexpected if Random Forest runs faster than Ridge provided by Alec.
", "rawMarkdown": "", "votes": null }, { "id": "28720", "postDate": "08/15/2013 09:20:35", "content": "[quote=Alec Radford;28156]
\nGood point, min/max clipping the predictions resulted in another 10K(ish) drop on the CV, but I'm out of submissions so I can't verify it. Really simple to implement, just after the model.predict() line add:
preds = np.clip(preds,np.min(trainY),np.max(trainY))
[/quote]
\nHi Alec, not sure if you have already tried it and submitted to leadership board? Did it decreased the MAE?
\nIntuitively it should too (I will generate a submission later on to try), but it seems to be performing slightly worse than before from the cross validation.
", "rawMarkdown": "", "votes": null }, { "id": "28755", "postDate": "08/15/2013 19:22:58", "content": "@Alec
\nHow to save X and the fitted values from this model into standard format? (I'm totally new to python for ML)
\n,for X, I've tried to add these lines without success:
\nwriter = csv.writer(open("C:/Users/Herimanitra/Downloads/KAGGLE COMPETITION/AMS 2013 2014 Forecasting contest/train/", 'w'))
for row in X :
writer.write(row)
ofile = open('testtrans.csv', "wb")
\nwriter = csv.writer(ofile, delimiter=',' )
\nfor row in testX:
\nwriter.writerow(row)
\nofile.close()
\n\n
and do for trainX as well
", "rawMarkdown": "", "votes": null }, { "id": "28761", "postDate": "08/15/2013 20:17:57", "content": "Thanks a lot!
", "rawMarkdown": "", "votes": null }, { "id": "29156", "postDate": "08/21/2013 15:30:04", "content": "Hi, @Alec
\nI am new to Python and I roughly read your code, do you intend to do a ridge regression on Y = XB,
\nY = 5113x98,
\nwhere 98 denotes mesonet locations of total solar energy output
\nX = 5113x2160,
\nwhere 2160 = 16*9*15, 15 different features (averaged measures) output at grid 16x9
\nB = 2160x98
\nregression weights to be used.
\nAm I right ?
", "rawMarkdown": "", "votes": null }, { "id": "29161", "postDate": "08/21/2013 16:40:20", "content": "[quote=liubenyuan;29156]
\nHi, @Alec
\nI am new to Python and I roughly read your code, do you intend to do a ridge regression on Y = XB, where
\nY = 5113x98,
\nwhere 98 denotes mesonet locations of total solar energy output
\nX = 5113x2160,
\nwhere 2160 = 16*9*15, 15 different features output at grid 16x9
\nB = 2160x98
\nregression weights to be used.
\nAm I right ?
\n[/quote]
\nSince Alec hasn't reply yet, I'll try to post my understanding here.
\nI think your understanding is almost correct, with a few clarification worth mentioning : X is a 5113 days of the mean of summing all the values of the 11 ensembles and the 5 forecasts at each day at each 16x9 GEFS data points, rather than a single measurement. You may verify this in the code where he uses np.mean(...) to "flatten" the 11*5 values, and print the values out X.shape out.
\nI hope this makes sense to you.
", "rawMarkdown": "", "votes": null }, { "id": "29200", "postDate": "08/21/2013 22:47:24", "content": "Thanks ! I finally make the code run under ipython2 by installing netCDF4 package.
\nEach X from a nc file is an averaged prediction on 9x16 grids flatterned each row to form 5113x114, and the 15 files are concatenate together to form the 5113x2160 training matrix.
\nSo there maybe more to do by extending the regression method and the formation of X :P
\nPS: I found this challenge more alike the kaggle stoke exchange one .
", "rawMarkdown": "", "votes": null }, { "id": "29201", "postDate": "08/21/2013 22:52:36", "content": "BTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
", "rawMarkdown": "", "votes": null }, { "id": "29282", "postDate": "08/23/2013 05:21:04", "content": "[quote=liubenyuan;29201]
\nBTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
\n[/quote]
\nWhat are you thinking? changing the mean by other statistics ?
", "rawMarkdown": "", "votes": null }, { "id": "29288", "postDate": "08/23/2013 08:05:25", "content": "[quote=Herimanitra;29282]
\n[quote=liubenyuan;29201]
\nBTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
\n[/quote]
\nWhat are you thinking? changing the mean by other statistics ?
\n[/quote]
\nI think he's suggesting investigating intraday weather changes in a different manner than the mean of all predictions summed together. I'm starting to look at that now too.
", "rawMarkdown": "", "votes": null }, { "id": "29289", "postDate": "08/23/2013 08:13:13", "content": "[quote=Rainman;29288]
\n[quote=Herimanitra;29282]
\n[quote=liubenyuan;29201]
\nBTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
\n[/quote]
\nWhat are you thinking? changing the mean by other statistics ?
\n[/quote]
\nI think he's suggesting investigating intraday weather changes in a different manner than the mean of all predictions summed together. I'm starting to look at that now too.
\n[/quote]
\n\n
Yes, I am doing that too !
", "rawMarkdown": "", "votes": null }, { "id": "29310", "postDate": "08/23/2013 16:17:40", "content": "[quote=liubenyuan;29289]
\n[quote=Rainman;29288]
\n[quote=Herimanitra;29282]
\n[quote=liubenyuan;29201]
\nBTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
\n[/quote]
\nWhat are you thinking? changing the mean by other statistics ?
\n[/quote]
\nI think he's suggesting investigating intraday weather changes in a different manner than the mean of all predictions summed together. I'm starting to look at that now too.
\n[/quote]
\n\n
Yes, I am doing that too !
\n[/quote]
\nLet's go!
", "rawMarkdown": "", "votes": null }, { "id": "29324", "postDate": "08/23/2013 17:12:46", "content": "I think thie competition calls **really** creative / competive methods.
\n1. I have tried ridge regression, Bayesian regression using Alec method ( 5113x144 submatrix for each file), the results are around 2200k.
\n2. I also tried interp the 98 MESONETs from the 144 GEFS grid data and using RF as a regressor, the results are around 2100k.
\n3. I googled some reference, that aside from MAE, relative MAE (rMAE) and ME can also be used as the performance metric. However, in the above two method, some rMAE for a given MESONET (i.e., idx = 2) are around 40~60%. while a acceptable rMAE should be around 12~20%, which indicates that the final ranking of MAE should be around 1600k. Well, still a long way to go !
\n\n
Liu
", "rawMarkdown": "", "votes": null }, { "id": "29335", "postDate": "08/23/2013 19:01:56", "content": "Liu,
\nWhen you mention your results, you mean in your CV sets, right?
", "rawMarkdown": "", "votes": null }, { "id": "29340", "postDate": "08/23/2013 22:06:35", "content": "[quote=BS Man;29335]
\nLiu,
\nWhen you mention your results, you mean in your CV sets, right?
\n[/quote]
\n\n
Yes, 10 fold cv test with .8 split
", "rawMarkdown": "", "votes": null }, { "id": "29447", "postDate": "08/25/2013 17:39:21", "content": "I also tried linear models but that's 2.2M seems to be topping the maximum already (w.r.t. to Alec's feature representation).
\nBy the way, I was also interpolating but so far has less fruitful results. (mean of the 11 ensembles x 5 forecasts, and mean of either axis, and even no mean). Could you share as to how you are interpolating?
\nThanks.
", "rawMarkdown": "", "votes": null }, { "id": "29459", "postDate": "08/26/2013 00:16:41", "content": "[quote=Log0;29447]
\nI also tried linear models but that's 2.2M seems to be topping the maximum already (w.r.t. to Alec's feature representation).
\nBy the way, I was also interpolating but so far has less fruitful results. (mean of the 11 ensembles x 5 forecasts, and mean of either axis, and even no mean). Could you share as to how you are interpolating?
\nThanks.
\n[/quote]
\n\n
The interpolation is uniform. However, you could append the following features :
\npenatration ratio : uswrf_sfc / dswrf_sfc, ulwrf_sfc / dlwrf_sfc
\nAnd then as all the weather forecasts are ON the MESONET locations, you could concatenate 98 mesonet into one y = (5113*98)x1, x = (5113*98)x17, then do the Bayesian regression or RF.
\nEDIT: I donnot think Interpolate might be a wise idea to use the weather forecasts ...
", "rawMarkdown": "", "votes": null }, { "id": "29466", "postDate": "08/26/2013 04:08:39", "content": "I tried both interpolating and not interpolating. So far, not interpolating has given better results for now - though if you said you are already doing 2.1M I am sure there's room of improvements on interpolating techniques... =]
", "rawMarkdown": "", "votes": null }, { "id": "29629", "postDate": "08/28/2013 08:21:31", "content": "[quote=liubenyuan;29459]
\nThe interpolation is uniform. However, you could append the following features :
\npenatration ratio : uswrf_sfc / dswrf_sfc, ulwrf_sfc / dlwrf_sfc
\nAnd then as all the weather forecasts are ON the MESONET locations, you could concatenate 98 mesonet into one y = (5113*98)x1, x = (5113*98)x17, then do the Bayesian regression or RF.
\nEDIT: I donnot think Interpolate might be a wise idea to use the weather forecasts ...
\n[/quote]
\nHello Liu.I preprocessed data almost as you the difference being I didn't include 2 additional features. I used spline interpolation taken from benchmark code. Data transformation takes huge time so I used (500*98)x15 dataframe. It took 4-5 hours to run on 2GB, single core netbook. I got 2293K cv score using linear regression with polynomial features.. I have the following questions:
\n@Davit
\nHi, my CV test is 2106K and my onboard result is 2197k.
\n griddata = fdata(:,:,hour,ens,day); % 16lon(y) x 9lat(x)
[X,Y] = meshgrid(lat,lon);
fdata_new(:,hour,ens,day) = interp2(X,Y,griddata,xi,yi);
and the regressor I am using is the matlab fitensemble method, codes are
\n NLS = 24;
tic;
% t = RegressionTree.template(...
% 'MinParent',5);
bag = fitensemble(x,y,'Bag',NLS,'tree','Type','regression');
train_time = toc;
fprintf('FINISH ensemble training (%4d): CPU hoc=%f\\n',NLS,train_time);
yt_hat = bag.predict(xt);
hope it helped.
", "rawMarkdown": "", "votes": null }, { "id": "29689", "postDate": "08/29/2013 02:48:26", "content": "Hi liubenyuan,
\nI am wondering if you can share to me your email - I thought I'd like to write to you a bit. Thanks. Alternatively, you may find my email on my profile, too.
", "rawMarkdown": "", "votes": null }, { "id": "29691", "postDate": "08/29/2013 04:28:48", "content": "@log0
\nThe attachment is my implementation using interp method in matlab.
\n\n\n
PS : I am learning python right now and I think python is a great tool for sci-computing and sharing :)
", "rawMarkdown": "", "votes": null }, { "id": "29952", "postDate": "09/01/2013 19:09:49", "content": "Thanks Liubenyuan, I was asking for your email because I'd like to discuss in depth more than just exchanging code. =]
\n", "rawMarkdown": "", "votes": null }, { "id": "30337", "postDate": "09/06/2013 16:51:56", "content": "
I've tried to use grids (3:6 , 7:12) which surround Mesonets (as depicted in the figure) but it seems to lower my score.
\nWhat about you?
", "rawMarkdown": "", "votes": null }, { "id": "30340", "postDate": "09/06/2013 16:56:54", "content": "[quote=Herimanitra;30337]
\nI've tried to use grids (3:6 , 7:12) which surround Mesonets (as depicted in the figure) but it seems to lower my score.
\nWhat about you?
\n[/quote]
\nYou mean not using those 'far away' GEFS points right? I suppose you didn't interpolate in this case but used all the GEFS points (3:6, 7:12) as features?
\nIn that case, I got poor results as well... but I did not dive into it too much, since in my case it is still rather slow for any complex models...
", "rawMarkdown": "", "votes": null }, { "id": "30346", "postDate": "09/06/2013 17:10:55", "content": "[quote=Log0;30340]
\n[quote=Herimanitra;30337]
\nI've tried to use grids (3:6 , 7:12) which surround Mesonets (as depicted in the figure) but it seems to lower my score.
\nWhat about you?
\n[/quote]
\nYou mean not using those 'far away' GEFS points right? I suppose you didn't interpolate in this case but used all the GEFS points (3:6, 7:12) as features?
\nIn that case, I got poor results as well... but I did not dive into it too much, since in my case it is still rather slow for any complex models...
\n[/quote]
\nYes, not using those 'far away' GEFS points.
\nI did not interpolate but I'm looking for a code to do it
\n", "rawMarkdown": "", "votes": null }, { "id": "30347", "postDate": "09/06/2013 17:13:32", "content": "
Not sure if you use Python. But Scipy's interpolation library here may help. So far you have not used interpolation at all to predict, but rather using the whole grid (9x16)?
", "rawMarkdown": "", "votes": null }, { "id": "30356", "postDate": "09/06/2013 17:36:05", "content": "Thanks,
\nMy implementation is mainly based on Spline interpolation and features that I extracted
\nfrom Alec's code.
\n\n
\n
", "rawMarkdown": "", "votes": null }, { "id": "30357", "postDate": "09/06/2013 17:39:31", "content": "
[quote=Herimanitra;30356]
\nThanks,
\nMy implementation is mainly based on Spline interpolation and features that I extracted
\nfrom Alec's code.
\n[/quote]
\nThanks for sharing. I'm curious to learn how to improve more, I just checked that Alec's code doesn't have any interpolation at all. You mean StormMiner's Python code?
\n", "rawMarkdown": "", "votes": null }, { "id": "30358", "postDate": "09/06/2013 17:42:19", "content": "
Spline interpolation from admin's code
\nMy question is now how to change solarSplines.py in order to use just GEFS points (3:6, 7:12) ?
", "rawMarkdown": "", "votes": null }, { "id": "30361", "postDate": "09/06/2013 17:52:29", "content": "I don't have this solarSplines.py, but this is a piece of code interpolating via Scipy. See if this helps you get started? =]
", "rawMarkdown": "", "votes": null }, { "id": "30362", "postDate": "09/06/2013 17:57:17", "content": "solarSplines.py is the benchmark provided by Admin
", "rawMarkdown": "", "votes": null }, { "id": "30367", "postDate": "09/06/2013 19:10:09", "content": "Well, I have a better LB score (closely matching my CV score) right now with a simple linear model (using also the intra-day time series as features, and not averaging/summing them out as in the sample code). Nevertheless, I agree that you cannot get much better leaderboard score than about that with a linear model (maybe a few 10k improvement can be achieved). But who knows...
", "rawMarkdown": "", "votes": null }, { "id": "30368", "postDate": "09/06/2013 19:12:46", "content": "I meant to reply to liubenyuan, but it doesn't show
\n-->
\nI also tried linear models but that's 2.2M seems to be topping the maximum already (w.r.t. to Alec's feature representation).
By the way, I was also interpolating but so far has less fruitful
results. (mean of the 11 ensembles x 5 forecasts, and mean of either
axis, and even no mean). Could you share as to how you are
interpolating?
Thanks.
Thanks for the code!
\nOne comment, I did not run it yet and only just read it so apologies if I'm off-base, but wouldn't it be better to set a lower limit to your predictions when you run a linear regression with the GEFS data? I would imagine there would be a few predictions that are negative and below 0 doesn't make sense in this context.
", "votes": null, "replies": [] }, { "id": 28156, "author_name": "alecradford", "author_url": "", "post_date": "08/03/2013 21:12:10", "content": "Good point, min/max clipping the predictions resulted in another 10K(ish) drop on the CV, but I'm out of submissions so I can't verify it. Really simple to implement, just after the model.predict() line add:
preds = np.clip(preds,np.min(trainY),np.max(trainY))
Thank you Alec for sharing the code.
\nI receive the following error when I run the code.
\nIt's from the load_GEFS_file function.
\nI couldn't figure how to solve this problem. Any direction would be much appreciated.
\nThanks again!!
", "votes": null, "replies": [] }, { "id": 28170, "author_name": "domcastro", "author_url": "", "post_date": "08/04/2013 09:40:36", "content": "I was able to run it as is. Have you installed the netcdf4 package?
\nAlso, have you checked the files are in the correct location?
", "votes": null, "replies": [] }, { "id": 28171, "author_name": "adilmouja", "author_url": "", "post_date": "08/04/2013 09:46:30", "content": "Yes I installed netcdf4 package.
\nAnd I put all the files in a folder called "data"...
", "votes": null, "replies": [] }, { "id": 28172, "author_name": "domcastro", "author_url": "", "post_date": "08/04/2013 10:09:30", "content": "I had to revert back to python 2.7 to run it (can't remember why) so I installed netcdf4 for 3.3 and 2.7
", "votes": null, "replies": [] }, { "id": 28179, "author_name": "alecradford", "author_url": "", "post_date": "08/04/2013 18:05:29", "content": "[quote=Adil;28168]
\nThank you Alec for sharing the code.
\nI receive the following error when I run the code.
\nIt's from the load_GEFS_file function.
\nI couldn't figure how to solve this problem. Any direction would be much appreciated.
\nThanks again!!
\n[/quote]
My guess is somehow the GEFS data you're loading doesn't follow the normal shape of (n,11,5,9,16) do you mind posting what shape X is before that line?
[quote=Alec Radford;28098]
\nThe idea behind this approach is that instead of worrying about interpolating, just let your model figure it out.
\n[/quote]
\nHello!
\nHere's what I think is an interesting paper related to this idea:
\nhttp://www.sciencedirect.com/science/article/pii/S1364815211001654
\nCheers, thanks for sharing!
", "votes": null, "replies": [] }, { "id": 28200, "author_name": "domcastro", "author_url": "", "post_date": "08/05/2013 10:10:00", "content": "Thanks for paper. I used a Random Forest on Alec's code but I got a worse result than his linear model and it took ages to run. Having to rethink it.
\n\n
EDIT: Though I'm new to Python so might not have done it correctly!
", "votes": null, "replies": [] }, { "id": 28212, "author_name": "adilmouja", "author_url": "", "post_date": "08/05/2013 15:34:51", "content": "[quote=Alec Radford;28179]
\n[quote=Adil;28168]
\nThank you Alec for sharing the code.
\nI receive the following error when I run the code.
\nIt's from the load_GEFS_file function.
\nI couldn't figure how to solve this problem. Any direction would be much appreciated.
\nThanks again!!
\n[/quote]
My guess is somehow the GEFS data you're loading doesn't follow the normal shape of (n,11,5,9,16) do you mind posting what shape X is before that line?
[/quote]
\nThanks Alec.
\nI could get the code to run. The problem was in: X = nc.Dataset(path,'r+').variables.values()[-1][:]
\nThe nc.Dataset(path,'r+').variables.values() is an unordered dictionary. Therefore if you take the last element using [-1], sometimes it takes the wrong element like 'lon'.
\nI solved it by defining the write mapping between the file and the target element.
\nmapping = {'dswrf_sfc' : 'Downward_Short-Wave_Rad_Flux',
'dlwrf_sfc' : 'Downward_Long-Wave_Rad_Flux',
'uswrf_sfc' : 'Upward_Short-Wave_Rad_Flux',
'ulwrf_sfc' : 'Upward_Long-Wave_Rad_Flux_surface',
'ulwrf_tatm': 'Upward_Long-Wave_Rad_Flux',
'pwat_eatm' : 'Precipitable_water',
'tcdc_eatm' : 'Total_cloud_cover',
'apcp_sfc' : 'Total_precipitation',
'pres_msl': 'Pressure',
'spfh_2m' : 'Specific_humidity_height_above_ground',
'tcolc_eatm' : 'Total_Column-Integrated_Condensate',
'tmax_2m' : 'Maximum_temperature',
'tmin_2m' : 'Minimum_temperature',
'tmp_2m' : 'Temperature_height_above_ground',
'tmp_sfc' : 'Temperature_surface'}
X = nc.Dataset(path).variables[mapping[data_type]][:]
\n\n
Thanks again for sharing the code!!
\n", "votes": null, "replies": [] }, { "id": 28440, "author_name": "domcastro", "author_url": "", "post_date": "08/08/2013 15:34:46", "content": "
More of a Python question. I've been trying to run other algorithms using your data transformation. Is there a way to change the shape as a lot of the algorithms complain about it? Newbie to Python, usually use R
", "votes": null, "replies": [] }, { "id": 28466, "author_name": "djgagne", "author_url": "", "post_date": "08/08/2013 22:40:01", "content": "Numpy.reshape will reshape the arrays.
", "votes": null, "replies": [] }, { "id": 28719, "author_name": "ericchio", "author_url": "", "post_date": "08/15/2013 09:17:47", "content": "In reply to DomCastro.
\nIt is expected. Linear models in general run faster than non-linear models like Random Forest. It will be unexpected if Random Forest runs faster than Ridge provided by Alec.
", "votes": null, "replies": [] }, { "id": 28720, "author_name": "ericchio", "author_url": "", "post_date": "08/15/2013 09:20:35", "content": "[quote=Alec Radford;28156]
\nGood point, min/max clipping the predictions resulted in another 10K(ish) drop on the CV, but I'm out of submissions so I can't verify it. Really simple to implement, just after the model.predict() line add:
preds = np.clip(preds,np.min(trainY),np.max(trainY))
[/quote]
\nHi Alec, not sure if you have already tried it and submitted to leadership board? Did it decreased the MAE?
\nIntuitively it should too (I will generate a submission later on to try), but it seems to be performing slightly worse than before from the cross validation.
", "votes": null, "replies": [] }, { "id": 28755, "author_name": "herimanitra", "author_url": "", "post_date": "08/15/2013 19:22:58", "content": "@Alec
\nHow to save X and the fitted values from this model into standard format? (I'm totally new to python for ML)
\n,for X, I've tried to add these lines without success:
\nwriter = csv.writer(open("C:/Users/Herimanitra/Downloads/KAGGLE COMPETITION/AMS 2013 2014 Forecasting contest/train/", 'w'))
for row in X :
writer.write(row)
ofile = open('testtrans.csv', "wb")
\nwriter = csv.writer(ofile, delimiter=',' )
\nfor row in testX:
\nwriter.writerow(row)
\nofile.close()
\n\n
and do for trainX as well
", "votes": null, "replies": [] }, { "id": 28761, "author_name": "herimanitra", "author_url": "", "post_date": "08/15/2013 20:17:57", "content": "Thanks a lot!
", "votes": null, "replies": [] }, { "id": 29156, "author_name": "liubenyuan", "author_url": "", "post_date": "08/21/2013 15:30:04", "content": "Hi, @Alec
\nI am new to Python and I roughly read your code, do you intend to do a ridge regression on Y = XB,
\nY = 5113x98,
\nwhere 98 denotes mesonet locations of total solar energy output
\nX = 5113x2160,
\nwhere 2160 = 16*9*15, 15 different features (averaged measures) output at grid 16x9
\nB = 2160x98
\nregression weights to be used.
\nAm I right ?
", "votes": null, "replies": [] }, { "id": 29161, "author_name": "ericchio", "author_url": "", "post_date": "08/21/2013 16:40:20", "content": "[quote=liubenyuan;29156]
\nHi, @Alec
\nI am new to Python and I roughly read your code, do you intend to do a ridge regression on Y = XB, where
\nY = 5113x98,
\nwhere 98 denotes mesonet locations of total solar energy output
\nX = 5113x2160,
\nwhere 2160 = 16*9*15, 15 different features output at grid 16x9
\nB = 2160x98
\nregression weights to be used.
\nAm I right ?
\n[/quote]
\nSince Alec hasn't reply yet, I'll try to post my understanding here.
\nI think your understanding is almost correct, with a few clarification worth mentioning : X is a 5113 days of the mean of summing all the values of the 11 ensembles and the 5 forecasts at each day at each 16x9 GEFS data points, rather than a single measurement. You may verify this in the code where he uses np.mean(...) to "flatten" the 11*5 values, and print the values out X.shape out.
\nI hope this makes sense to you.
", "votes": null, "replies": [] }, { "id": 29200, "author_name": "liubenyuan", "author_url": "", "post_date": "08/21/2013 22:47:24", "content": "Thanks ! I finally make the code run under ipython2 by installing netCDF4 package.
\nEach X from a nc file is an averaged prediction on 9x16 grids flatterned each row to form 5113x114, and the 15 files are concatenate together to form the 5113x2160 training matrix.
\nSo there maybe more to do by extending the regression method and the formation of X :P
\nPS: I found this challenge more alike the kaggle stoke exchange one .
", "votes": null, "replies": [] }, { "id": 29201, "author_name": "liubenyuan", "author_url": "", "post_date": "08/21/2013 22:52:36", "content": "BTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
", "votes": null, "replies": [] }, { "id": 29282, "author_name": "herimanitra", "author_url": "", "post_date": "08/23/2013 05:21:04", "content": "[quote=liubenyuan;29201]
\nBTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
\n[/quote]
\nWhat are you thinking? changing the mean by other statistics ?
", "votes": null, "replies": [] }, { "id": 29288, "author_name": "rainman", "author_url": "", "post_date": "08/23/2013 08:05:25", "content": "[quote=Herimanitra;29282]
\n[quote=liubenyuan;29201]
\nBTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
\n[/quote]
\nWhat are you thinking? changing the mean by other statistics ?
\n[/quote]
\nI think he's suggesting investigating intraday weather changes in a different manner than the mean of all predictions summed together. I'm starting to look at that now too.
", "votes": null, "replies": [] }, { "id": 29289, "author_name": "liubenyuan", "author_url": "", "post_date": "08/23/2013 08:13:13", "content": "[quote=Rainman;29288]
\n[quote=Herimanitra;29282]
\n[quote=liubenyuan;29201]
\nBTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
\n[/quote]
\nWhat are you thinking? changing the mean by other statistics ?
\n[/quote]
\nI think he's suggesting investigating intraday weather changes in a different manner than the mean of all predictions summed together. I'm starting to look at that now too.
\n[/quote]
\n\n
Yes, I am doing that too !
", "votes": null, "replies": [] }, { "id": 29310, "author_name": "herimanitra", "author_url": "", "post_date": "08/23/2013 16:17:40", "content": "[quote=liubenyuan;29289]
\n[quote=Rainman;29288]
\n[quote=Herimanitra;29282]
\n[quote=liubenyuan;29201]
\nBTW, we are only **average** but do not predict the weather change happened in a day, should that improves the results too. And I also think precise interpolation is not needed so long as the GEFS its self has errors.
\n[/quote]
\nWhat are you thinking? changing the mean by other statistics ?
\n[/quote]
\nI think he's suggesting investigating intraday weather changes in a different manner than the mean of all predictions summed together. I'm starting to look at that now too.
\n[/quote]
\n\n
Yes, I am doing that too !
\n[/quote]
\nLet's go!
", "votes": null, "replies": [] }, { "id": 29324, "author_name": "liubenyuan", "author_url": "", "post_date": "08/23/2013 17:12:46", "content": "I think thie competition calls **really** creative / competive methods.
\n1. I have tried ridge regression, Bayesian regression using Alec method ( 5113x144 submatrix for each file), the results are around 2200k.
\n2. I also tried interp the 98 MESONETs from the 144 GEFS grid data and using RF as a regressor, the results are around 2100k.
\n3. I googled some reference, that aside from MAE, relative MAE (rMAE) and ME can also be used as the performance metric. However, in the above two method, some rMAE for a given MESONET (i.e., idx = 2) are around 40~60%. while a acceptable rMAE should be around 12~20%, which indicates that the final ranking of MAE should be around 1600k. Well, still a long way to go !
\n\n
Liu
", "votes": null, "replies": [] }, { "id": 29335, "author_name": "bensolucky", "author_url": "", "post_date": "08/23/2013 19:01:56", "content": "Liu,
\nWhen you mention your results, you mean in your CV sets, right?
", "votes": null, "replies": [] }, { "id": 29340, "author_name": "liubenyuan", "author_url": "", "post_date": "08/23/2013 22:06:35", "content": "[quote=BS Man;29335]
\nLiu,
\nWhen you mention your results, you mean in your CV sets, right?
\n[/quote]
\n\n
Yes, 10 fold cv test with .8 split
", "votes": null, "replies": [] }, { "id": 29447, "author_name": "ericchio", "author_url": "", "post_date": "08/25/2013 17:39:21", "content": "I also tried linear models but that's 2.2M seems to be topping the maximum already (w.r.t. to Alec's feature representation).
\nBy the way, I was also interpolating but so far has less fruitful results. (mean of the 11 ensembles x 5 forecasts, and mean of either axis, and even no mean). Could you share as to how you are interpolating?
\nThanks.
", "votes": null, "replies": [] }, { "id": 29459, "author_name": "liubenyuan", "author_url": "", "post_date": "08/26/2013 00:16:41", "content": "[quote=Log0;29447]
\nI also tried linear models but that's 2.2M seems to be topping the maximum already (w.r.t. to Alec's feature representation).
\nBy the way, I was also interpolating but so far has less fruitful results. (mean of the 11 ensembles x 5 forecasts, and mean of either axis, and even no mean). Could you share as to how you are interpolating?
\nThanks.
\n[/quote]
\n\n
The interpolation is uniform. However, you could append the following features :
\npenatration ratio : uswrf_sfc / dswrf_sfc, ulwrf_sfc / dlwrf_sfc
\nAnd then as all the weather forecasts are ON the MESONET locations, you could concatenate 98 mesonet into one y = (5113*98)x1, x = (5113*98)x17, then do the Bayesian regression or RF.
\nEDIT: I donnot think Interpolate might be a wise idea to use the weather forecasts ...
", "votes": null, "replies": [] }, { "id": 29466, "author_name": "ericchio", "author_url": "", "post_date": "08/26/2013 04:08:39", "content": "I tried both interpolating and not interpolating. So far, not interpolating has given better results for now - though if you said you are already doing 2.1M I am sure there's room of improvements on interpolating techniques... =]
", "votes": null, "replies": [] }, { "id": 29629, "author_name": "davminer", "author_url": "", "post_date": "08/28/2013 08:21:31", "content": "[quote=liubenyuan;29459]
\nThe interpolation is uniform. However, you could append the following features :
\npenatration ratio : uswrf_sfc / dswrf_sfc, ulwrf_sfc / dlwrf_sfc
\nAnd then as all the weather forecasts are ON the MESONET locations, you could concatenate 98 mesonet into one y = (5113*98)x1, x = (5113*98)x17, then do the Bayesian regression or RF.
\nEDIT: I donnot think Interpolate might be a wise idea to use the weather forecasts ...
\n[/quote]
\nHello Liu.I preprocessed data almost as you the difference being I didn't include 2 additional features. I used spline interpolation taken from benchmark code. Data transformation takes huge time so I used (500*98)x15 dataframe. It took 4-5 hours to run on 2GB, single core netbook. I got 2293K cv score using linear regression with polynomial features.. I have the following questions:
\n@Davit
\nHi, my CV test is 2106K and my onboard result is 2197k.
\n griddata = fdata(:,:,hour,ens,day); % 16lon(y) x 9lat(x)
[X,Y] = meshgrid(lat,lon);
fdata_new(:,hour,ens,day) = interp2(X,Y,griddata,xi,yi);
and the regressor I am using is the matlab fitensemble method, codes are
\n NLS = 24;
tic;
% t = RegressionTree.template(...
% 'MinParent',5);
bag = fitensemble(x,y,'Bag',NLS,'tree','Type','regression');
train_time = toc;
fprintf('FINISH ensemble training (%4d): CPU hoc=%f\\n',NLS,train_time);
yt_hat = bag.predict(xt);
hope it helped.
", "votes": null, "replies": [] }, { "id": 29689, "author_name": "ericchio", "author_url": "", "post_date": "08/29/2013 02:48:26", "content": "Hi liubenyuan,
\nI am wondering if you can share to me your email - I thought I'd like to write to you a bit. Thanks. Alternatively, you may find my email on my profile, too.
", "votes": null, "replies": [] }, { "id": 29691, "author_name": "liubenyuan", "author_url": "", "post_date": "08/29/2013 04:28:48", "content": "@log0
\nThe attachment is my implementation using interp method in matlab.
\n\n\n
PS : I am learning python right now and I think python is a great tool for sci-computing and sharing :)
", "votes": null, "replies": [] }, { "id": 29952, "author_name": "ericchio", "author_url": "", "post_date": "09/01/2013 19:09:49", "content": "Thanks Liubenyuan, I was asking for your email because I'd like to discuss in depth more than just exchanging code. =]
\n", "votes": null, "replies": [] }, { "id": 30337, "author_name": "herimanitra", "author_url": "", "post_date": "09/06/2013 16:51:56", "content": "
I've tried to use grids (3:6 , 7:12) which surround Mesonets (as depicted in the figure) but it seems to lower my score.
\nWhat about you?
", "votes": null, "replies": [] }, { "id": 30340, "author_name": "ericchio", "author_url": "", "post_date": "09/06/2013 16:56:54", "content": "[quote=Herimanitra;30337]
\nI've tried to use grids (3:6 , 7:12) which surround Mesonets (as depicted in the figure) but it seems to lower my score.
\nWhat about you?
\n[/quote]
\nYou mean not using those 'far away' GEFS points right? I suppose you didn't interpolate in this case but used all the GEFS points (3:6, 7:12) as features?
\nIn that case, I got poor results as well... but I did not dive into it too much, since in my case it is still rather slow for any complex models...
", "votes": null, "replies": [] }, { "id": 30346, "author_name": "herimanitra", "author_url": "", "post_date": "09/06/2013 17:10:55", "content": "[quote=Log0;30340]
\n[quote=Herimanitra;30337]
\nI've tried to use grids (3:6 , 7:12) which surround Mesonets (as depicted in the figure) but it seems to lower my score.
\nWhat about you?
\n[/quote]
\nYou mean not using those 'far away' GEFS points right? I suppose you didn't interpolate in this case but used all the GEFS points (3:6, 7:12) as features?
\nIn that case, I got poor results as well... but I did not dive into it too much, since in my case it is still rather slow for any complex models...
\n[/quote]
\nYes, not using those 'far away' GEFS points.
\nI did not interpolate but I'm looking for a code to do it
\n", "votes": null, "replies": [] }, { "id": 30347, "author_name": "ericchio", "author_url": "", "post_date": "09/06/2013 17:13:32", "content": "
Not sure if you use Python. But Scipy's interpolation library here may help. So far you have not used interpolation at all to predict, but rather using the whole grid (9x16)?
", "votes": null, "replies": [] }, { "id": 30356, "author_name": "herimanitra", "author_url": "", "post_date": "09/06/2013 17:36:05", "content": "Thanks,
\nMy implementation is mainly based on Spline interpolation and features that I extracted
\nfrom Alec's code.
\n\n
\n
", "votes": null, "replies": [] }, { "id": 30357, "author_name": "ericchio", "author_url": "", "post_date": "09/06/2013 17:39:31", "content": "
[quote=Herimanitra;30356]
\nThanks,
\nMy implementation is mainly based on Spline interpolation and features that I extracted
\nfrom Alec's code.
\n[/quote]
\nThanks for sharing. I'm curious to learn how to improve more, I just checked that Alec's code doesn't have any interpolation at all. You mean StormMiner's Python code?
\n", "votes": null, "replies": [] }, { "id": 30358, "author_name": "herimanitra", "author_url": "", "post_date": "09/06/2013 17:42:19", "content": "
Spline interpolation from admin's code
\nMy question is now how to change solarSplines.py in order to use just GEFS points (3:6, 7:12) ?
", "votes": null, "replies": [] }, { "id": 30361, "author_name": "ericchio", "author_url": "", "post_date": "09/06/2013 17:52:29", "content": "I don't have this solarSplines.py, but this is a piece of code interpolating via Scipy. See if this helps you get started? =]
", "votes": null, "replies": [] }, { "id": 30362, "author_name": "herimanitra", "author_url": "", "post_date": "09/06/2013 17:57:17", "content": "solarSplines.py is the benchmark provided by Admin
", "votes": null, "replies": [] }, { "id": 30367, "author_name": "gdi90215", "author_url": "", "post_date": "09/06/2013 19:10:09", "content": "Well, I have a better LB score (closely matching my CV score) right now with a simple linear model (using also the intra-day time series as features, and not averaging/summing them out as in the sample code). Nevertheless, I agree that you cannot get much better leaderboard score than about that with a linear model (maybe a few 10k improvement can be achieved). But who knows...
", "votes": null, "replies": [] }, { "id": 30368, "author_name": "gdi90215", "author_url": "", "post_date": "09/06/2013 19:12:46", "content": "I meant to reply to liubenyuan, but it doesn't show
\n-->
\nI also tried linear models but that's 2.2M seems to be topping the maximum already (w.r.t. to Alec's feature representation).
By the way, I was also interpolating but so far has less fruitful
results. (mean of the 11 ensembles x 5 forecasts, and mean of either
axis, and even no mean). Could you share as to how you are
interpolating?
Thanks.