{ "id": 5288, "title": "What is the Ensemble variable?", "url": "/competitions/ams-2014-solar-energy-prediction-contest/discussion/5288", "author_name": "", "post_date": "2013-08-03T22:08:20.700Z", "votes": null, "comment_count": 7, "views": 2453, "content": "

Hi all

\n

I don't understand what the ensemble variable is/provides?  Do these represent different models used to come up with forecasts in each of the netCDF files? 

\n

If someone could please provide some information on this then it would be much appreciated.  Hopefully this is not a dumb question that is obvious to everyone except me.

\n

The "get the data" pages states the following:

\n

"The second dimension is the ensemble member that the forecast comes
from. The GEFS has 11 ensemble members with perturbed initial
conditions."  The netCDF files have the following: "0 is control, other values are perturbation numbers"

\n

I'm not sure if I should be interested in this variable or not, but if I wasn't interested then would taking the data at 0 (control) make sense?

\n

Thanks in advance.

\n

Stephen

\n

Ps I started looking up perturbation theory on wikipedia, but that made my head hurt. 

", "messages": [ { "id": "28158", "postDate": "08/03/2013 22:08:20", "content": "

Hi all

\n

I don't understand what the ensemble variable is/provides?  Do these represent different models used to come up with forecasts in each of the netCDF files? 

\n

If someone could please provide some information on this then it would be much appreciated.  Hopefully this is not a dumb question that is obvious to everyone except me.

\n

The "get the data" pages states the following:

\n

"The second dimension is the ensemble member that the forecast comes
from. The GEFS has 11 ensemble members with perturbed initial
conditions."  The netCDF files have the following: "0 is control, other values are perturbation numbers"

\n

I'm not sure if I should be interested in this variable or not, but if I wasn't interested then would taking the data at 0 (control) make sense?

\n

Thanks in advance.

\n

Stephen

\n

Ps I started looking up perturbation theory on wikipedia, but that made my head hurt. 

", "rawMarkdown": "", "votes": null }, { "id": "28174", "postDate": "08/04/2013 12:26:04", "content": "

This might help http://www.kaggle.com/c/ams-2014-solar-energy-prediction-contest/forums/t/5057/understanding-data

\n
 
", "rawMarkdown": "", "votes": null }, { "id": "28183", "postDate": "08/04/2013 20:44:05", "content": "

I'll take a crack at answering this question. First off, though formal perturbation theory is related to this, it isn't a prerequisite for understanding forecast ensembles. 

\n

Weather is a dynamic system, and the differential equations governing it are highly non-linear (and deterministic). As such, it is a chaotic system and any predictions made with the governing equations will be susceptible to any errors or inaccuracies in observed data (the initial conditions). For instance, the instrument error of a thermometer, though small, might be sufficient to cause a weather model to make a prediction with an enormous amount of error (due to non-linearity). This is where the ensemble of models comes in to play. The control is the observed data (instrument measurement with error). The other members of the ensemble are the perturbed initial conditions with a fraction of the control added or subtracted. The idea is to represent the uncertainty of the instrument data, and allow the most probable result from the deterministic model to be selected. 

", "rawMarkdown": "", "votes": null }, { "id": "28187", "postDate": "08/04/2013 22:12:41", "content": "

Thanks Bradenx1.  This helps a lot,.

\n

Cheers

\n

Stephen

", "rawMarkdown": "", "votes": null }, { "id": "28395", "postDate": "08/07/2013 19:32:26", "content": "

A follow up question to this...  Are the perturbations applied to the initial conditions for each ensemble member similar from run-to-run?  For example, does the ensemble with index '7' use the same "type" of perturbations each day? Or are the applied perturbations pretty much random and change each day? 

\n

I only ask because this information can possibly be used to decide if one ensemble member tends to be "better" than another.  

", "rawMarkdown": "", "votes": null }, { "id": "28396", "postDate": "08/07/2013 20:13:38", "content": "

The Reforecast GEFS uses the Ensemble Transform with Rescaling technique to generate its perturbations, so each ensemble member should be randomly perturbed each day and not in a consistent way. They also all use the same parameterization schemes. For more information, you can check out this paper.

", "rawMarkdown": "", "votes": null }, { "id": "28666", "postDate": "08/14/2013 05:42:15", "content": "

I want to ask if there is a resemblance in initial conditions of ensemble members in different days.Is there a relation between initial values of 3. ensemble member of 01 Jan 1994 and those of 3. ensemble member of 02 Jan 1994? Or initial values of those ensemble members in different days are random?

", "rawMarkdown": "", "votes": null }, { "id": "28689", "postDate": "08/14/2013 13:42:18", "content": "

The ensemble control run (the first ensemble member) should have the best fit initial conditions for each day. The other members have a random perturbation applied to their initial conditions, but the perturbation should be within the expected range of the instrument error. The perturbations are different each day. Theoretically there should not be an initial condition bias for individual ensemble members, but you may want to do a statistical analysis of the training data to see if one appears.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 28174, "author_name": "domcastro", "author_url": "", "post_date": "08/04/2013 12:26:04", "content": "

This might help http://www.kaggle.com/c/ams-2014-solar-energy-prediction-contest/forums/t/5057/understanding-data

\n
 
", "votes": null, "replies": [] }, { "id": 28183, "author_name": "bradenx1", "author_url": "", "post_date": "08/04/2013 20:44:05", "content": "

I'll take a crack at answering this question. First off, though formal perturbation theory is related to this, it isn't a prerequisite for understanding forecast ensembles. 

\n

Weather is a dynamic system, and the differential equations governing it are highly non-linear (and deterministic). As such, it is a chaotic system and any predictions made with the governing equations will be susceptible to any errors or inaccuracies in observed data (the initial conditions). For instance, the instrument error of a thermometer, though small, might be sufficient to cause a weather model to make a prediction with an enormous amount of error (due to non-linearity). This is where the ensemble of models comes in to play. The control is the observed data (instrument measurement with error). The other members of the ensemble are the perturbed initial conditions with a fraction of the control added or subtracted. The idea is to represent the uncertainty of the instrument data, and allow the most probable result from the deterministic model to be selected. 

", "votes": null, "replies": [] }, { "id": 28187, "author_name": "shallybu", "author_url": "", "post_date": "08/04/2013 22:12:41", "content": "

Thanks Bradenx1.  This helps a lot,.

\n

Cheers

\n

Stephen

", "votes": null, "replies": [] }, { "id": 28395, "author_name": "marksv", "author_url": "", "post_date": "08/07/2013 19:32:26", "content": "

A follow up question to this...  Are the perturbations applied to the initial conditions for each ensemble member similar from run-to-run?  For example, does the ensemble with index '7' use the same "type" of perturbations each day? Or are the applied perturbations pretty much random and change each day? 

\n

I only ask because this information can possibly be used to decide if one ensemble member tends to be "better" than another.  

", "votes": null, "replies": [] }, { "id": 28396, "author_name": "djgagne", "author_url": "", "post_date": "08/07/2013 20:13:38", "content": "

The Reforecast GEFS uses the Ensemble Transform with Rescaling technique to generate its perturbations, so each ensemble member should be randomly perturbed each day and not in a consistent way. They also all use the same parameterization schemes. For more information, you can check out this paper.

", "votes": null, "replies": [] }, { "id": 28666, "author_name": "gungoryilmaz", "author_url": "", "post_date": "08/14/2013 05:42:15", "content": "

I want to ask if there is a resemblance in initial conditions of ensemble members in different days.Is there a relation between initial values of 3. ensemble member of 01 Jan 1994 and those of 3. ensemble member of 02 Jan 1994? Or initial values of those ensemble members in different days are random?

", "votes": null, "replies": [] }, { "id": 28689, "author_name": "djgagne", "author_url": "", "post_date": "08/14/2013 13:42:18", "content": "

The ensemble control run (the first ensemble member) should have the best fit initial conditions for each day. The other members have a random perturbation applied to their initial conditions, but the perturbation should be within the expected range of the instrument error. The perturbations are different each day. Theoretically there should not be an initial condition bias for individual ensemble members, but you may want to do a statistical analysis of the training data to see if one appears.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "28158": "", "28174": "", "28183": "", "28187": "", "28395": "", "28396": "", "28666": "", "28689": "" }, "source": "meta" }