![\"Fig.](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3966/media/overview-AfSIS-kaggle.png\")
SOC, pH, Ca, P, Sand are the five target variables for predictions. The data have been monotonously transformed from the original measurements and thus include negative values.
\r\nWe have also included some potential spatial predictors from remote sensing data sources. Short variable descriptions are provided below and additional descriptions can be found at AfSIS data. The data have been mean centered and scaled.
\r\nAn example model using Bayesian Additive Regression Trees can be found here.
\r\nWhy not more data?
\r\nWe will not introduce additional data (e.g. georeference) at this stage of the competition. We think that would be confusing (to us), as we would really like to find out how predictive the spectral methods are/would be when they are applied in new places and/or at different points in time by data science experts such as yourselves. Subsequent Kaggle competitions may focus on explicitly spatial and or space-time predictions.
\r\nBackground on data set creation
\r\nThere have been a number of questions regarding why and how the data were ordered in the training and test sets. As some of you have surmised there is certainly geographical clustering in this dataset. This is due to the spatially stratified multilevel sampling design that was used to assemble the data. The following is an abbreviated version of how this came about.
\r\nWhen the Africa Soil Information Service (AfSIS) project started in 2009, we were faced with the enormous logistical task of obtaining a representative sample covering ~18.1 million km2 of the non-desert portion of Africa, including Madagascar, that could be used as a baseline for monitoring soil and other ecosystem properties.
\r\nThe way we chose to go about this was to select 60, 10 × 10 km sized “Sentinel Landscapes”, stratified by the major Koeppen-Geiger climate zones of Africa, excluding the true deserts and some of the African countries which we were not allowed to work in at the time, due to security reasons.
\r\nWithin each of the 60 Sentinel Landscapes AfSIS field teams sampled 16, 1 km2 “Sampling Clusters” (1 km2 circular areas) with 10, 1000 m2, randomly located circular “Sampling Plots”.
\r\nTopsoil (0-20 cm) and subsoil (20-50 cm) samples were subsequently recovered by physically mixing core subsamples from 4 locations within each Sampling Plot. Hence the intent was to obtain a representative multilevel/multistage sample consisting of:
\r\nMultiply those numbers and you obtain the intended number of composite soil samples (19,200) that were to be collected in the field over a 4-year period between 2009-2012.
\r\nTo achieve this target, we pre-generated appropriately randomized GPS coordinates for every Sampling Plot and, AfSIS field teams then navigated to (most) of those spots on the map to collect samples (insert n/N Sampling Plots).
\r\nAs might be expected with an exercise of this magnitude, the actual total number of soil samples in this dataset is somewhat smaller than intended, as some locations were either completely inaccessible by 4WD vehicle and/or on foot or that had soil depth restrictions that prevented the field teams from recovering physical samples.
\r\nAll physically recovered samples went into our lab (in Nairobi) to be characterized with the MIR spectral measurements that you are currently using. The potential spatial predictors, which cover the entire African continent (and beyond), were derived from NASA remote sensing data missions.
\r\nA 10% subsample of all the soils that were measured with the MIR method, subsequently went on to be characterized with more reference measurements.
\r\n“Reference measurements” are much more expensive (potentially hundreds of U$ per sample) and time-consuming. The other 90% of samples that were not characterized with “reference” methods have been physically archived, so that we can potentially retrieve those for calibrating new analytical methods and/or validating old methods.
\r\nWhat is posted for this Kaggle is the complete spectral + reference dataset that we have currently, subject to the sampling procedures described above. The training and test data have been split along Sentinel Landscape levels because we are primarily interested in predicting soil properties at new Sentinel Landscapes.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Description", "content": "Advances in rapid, low cost analysis of soil samples using infrared spectroscopy, georeferencing of soil samples, and greater availability of earth remote sensing data provide new opportunities for predicting soil functional properties at unsampled locations. Soil functional properties are those properties related to a soil’s capacity to support essential ecosystem services such as primary productivity, nutrient and water retention, and resistance to soil erosion. Digital mapping of soil functional properties, especially in data sparse regions such as Africa, is important for planning sustainable agricultural intensification and natural resources management.
\r\n
Diffuse reflectance infrared spectroscopy has shown potential in numerous studies to provide a highly repeatable, rapid and low cost measurement of many soil functional properties. The amount of light absorbed by a soil sample is measured, with minimal sample preparation, at hundreds of specific wavebands across a range of wavelengths to provide an infrared spectrum (Fig. 1). The measurement can be typically performed in about 30 seconds, in contrast to conventional reference tests, which are slow and expensive and use chemicals.
\r\nConventional reference soil tests are calibrated to the infrared spectra on a subset of samples selected to span the diversity in soils in a given target geographical area. The calibration models are then used to predict the soil test values for the whole sample set. The predicted soil test values from georeferenced soil samples can in turn be calibrated to remote sensing covariates, which are recorded for every pixel at a fixed spatial resolution in an area, and the calibration model is then used to predict the soil test values for each pixel. The result is a digital map of the soil properties.
\r\nThis competition asks you to predict 5 target soil functional properties from diffuse reflectance infrared spectroscopy measurements.
\r\nThis competition is sponsored by the Africa Soil Information Service.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "Submissions are scored on MCRMSE (mean columnwise root mean squared error):
\r\n$$
\\textrm{MCRMSE} = \\frac{1}{5}\\sum_{j=1}^{5}\\sqrt{\\frac{1}{n} \\sum_{i=1}^{n} (y_{ij} - \\hat{y}_{ij})^2},
$$
where \\\\(y\\\\) and \\\\(\\hat{y}\\\\) are the actual and predicted values, respectively.
\r\nFor each row in the dataset, the submission file should contain an identifier column (PIDN) and 5 prediction columns: Ca, P, pH, SOC, and Sand. PIDN, the sample identifier, should be copied from the first column of test data file. Ca, P, pH, SOC, and Sand are soil properties whose values you must predict.
\r\nThe file should contain a header and have the following format:
\r\nPIDN,Ca,P,pH,SOC,Sand", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Prizes", "content": "
XNhoFZW5,1.3,3.1,2.0,1.5,4.8
You cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n
\r\nKaggle and the competition host have the right to publicly disseminate any entries or models.
\r\n\r\n Winning solutions need to be made available under\r\n a popular OSI-approved license\r\n in order to be eligible for recognition and prize money.\r\n
\r\n\r\n Prizes will be awarded after the winners have posted their solutions to the competition forum.\r\n Winners must post or link to their solutions within seven (7) days of being notified of their winning status.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
The maximum size of a team is 4 participants.
\r\nYou may submit a maximum of 3 entries per day.
\r\nYou may select up to 2 final submissions for judging.
\r\nStart Date: 8/27/2014 8:44 PM UTC
\r\nMerger Deadline: 10/14/2014 11:59 PM UTC
\r\nEntry Deadline: 10/14/2014 11:59 PM UTC
\r\nEnd Date: 10/21/2014 11:59 PM UTC
\r\n\t\r\nCOMPETITION NAME (the 'Competition'): AfSIS Soil Properties Competition
\r\nCOMPETITION SPONSOR: Columbia Global Center
\r\nCOMPETITION WEBSITE: https://www.kaggle.com/c/afsis-soil-properties
\r\nPRIZES: $8,000
\r\nWINNER LICENSE TYPE: Open Source License
\r\nMAXIMUM TEAM SIZE: as listed above.
\r\nMAXIMUM SUBMISSIONS PER DAY: as listed above.
\r\nThese are the complete, official rules for the Competition (the 'Competition Rules') and incorporate by reference the contents of the Competition Website listed above.
\r\nBy downloading a dataset linked from the Competition Website, submitting an entry to this Competition, or joining a Team in this Competition, you are agreeing to be bound by these Competition Rules which constitute a binding agreement between you and the Competition Sponsor.
\r\nThe Competition is sponsored by the Competition Sponsor listed above and hosted on the Sponsor's behalf by Kaggle Inc ('Kaggle').
\r\nThe Competition will run from Start Date to End Date as listed on the Competition Website. Note that Competition deadlines are subject to change and additional hurdle deadlines may be introduced during the Competition. Any additional or altered deadlines not already described in these Competition Rules will be publicized to the Participants via the Competition Website. It is the Participant's responsibility in this Competition to check the website regularly throughout the Competition to stay informed of any new or updated deadlines. Neither Kaggle nor Competition Sponsor are responsible for any Participant's failure to do so.
\r\nEach registered individual or Team is referred to as a Participant. You may only participate using a single, unique Kaggle account registered at http://www.kaggle.com. Participating using more than one Kaggle account per individual Participant is a breach of these Competition Rules and Competition Sponsor reserves the right to disqualify any Participant (or Team including Participant) who is found to breach these Competition Rules.
\r\nThe Competition is open to all individuals over the age of 18 at the time of entry and to all validly formed legal entities that have not declared or been declared in bankruptcy.
\r\nOfficers, directors, employees and advisory board members (and their immediate families and members of the same household) of the Competition Sponsor, Kaggle and their respective affiliates, subsidiaries, contractors (with the express exception of Kaggle's authorized Kaggle Community Evangelists), agents, judges and advertising and promotion agencies are not eligible to participate in the Competition.
\r\nYou are not eligible to receive any Prize in the Competition if you are a resident of a country designated by the United States Treasury’s Office of Foreign Assets Control (see <http://www.treasury.gov/resource-center/sanctions/SDN-List/Pages/default.aspx> for additional information).
\r\n'Submission' means the material submitted by you in the manner and format specified on the Website via the Submission form on the Website. You (or if you are part of a Team, your Team) may submit up to the maximum number of Submissions per day as specified above. All Submissions must be uploaded to the Website in the manner and format specified on the Website. Submissions must be received prior to the Competition deadline and adhere to the guidelines for Submissions specified on the Website.
\r\nSubmissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.
\r\nIf the Competition is a multi-stage Competition with temporally separate training data and/or leaderboard data, one or more valid Submissions must be made and selected during each stage of the Competition in the manner described on the Competition Website.
\r\n'Data' means the Data or Datasets linked from the Competition Website for the purpose of use by Participants in the Competition. For the avoidance of doubt, Data is deemed for the purpose of these Competition Rules to include any prototype or executable code provided to Participants by Kaggle or Competition Sponsor via the Website. Participants must use the Data only as permitted by these Competition Rules and any associated data use rules specified on the Competition Website.
\r\nUnless otherwise permitted by the terms of the Competition Website, Participants must use the Data solely for the purpose and duration of the Competition, including but not limited to reading and learning from the Data, analyzing the Data, modifying the Data and generally preparing your Submission and any underlying models and participating in forum discussions on the Website. Participants agree to use suitable measures to prevent persons who have not formally agreed to these Competition Rules from gaining access to the Data and agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Data to any party not participating in the Competition. Participants agree to notify Kaggle immediately upon learning of any possible unauthorized transmission or unauthorized access of the Data and agree to work with Kaggle to rectify any unauthorized transmission. Participants agree that participation in the Competition shall not be construed as having or being granted a license (expressly, by implication, estoppel, or otherwise) under, or any right of ownership in, any of the Data.
\r\nUnless otherwise expressly stated on the Competition Website, Participants must not use data other than the Data to develop and test their models and Submissions. Competition Sponsor reserves the right in its sole discretion to disqualify any Participant who Competition Sponsor discovers has undertaken or attempted to undertake the use of data other than the Data, or who uses the Data other than as permitted according to the Competition Website and in these Competition Rules, in the course of the Competition.
\r\nParticipants are prohibited from privately sharing source or executable code developed in connection with or based upon the Data, and any such sharing is a breach of these Competition Rules and may result in disqualification.
\r\nParticipants are permitted to publicly share source or executable code developed in connection with or based upon the Data, or otherwise relevant to the Competition, provided that such sharing does not violate the intellectual property rights of any third party. By so sharing, the sharing Participant is thereby deemed to have licensed the shared code under the MIT License (an open source software license commonly described at <http://opensource.org/licenses/MIT>).
\r\nA Submission will be ineligible to win a prize if it was developed using code containing or depending on software licensed under an open source license:
\r\n* other than an Open Source Initiative-approved license (see <http://opensource.org/>); or
* an open source license that prohibits commercial use.
This Competition is a challenge of skill and the final results are determined solely by leaderboard ranking on the private leaderboard (subject to compliance with these Competition Rules). Participants' scores and ranks on the Competition Website at any given stage of the Competition will be based on the evaluation metric described on the Competition Website, as determined by applying the predictions in the Submission to the ground truth of a validation dataset whose instances were a fixed set sampled from the Data.
\r\nThe evaluation metric used for scoring and ranking Submissions will be displayed on the Competition Website.
\r\nPrize awards are subject to verification of eligibility and compliance with these Competition Rules. All decisions of the Competition Sponsor and judges will be final and binding on all matters relating to this Competition. Competition Sponsor reserves the right to examine the Submission and any associated code or documentation for compliance with these Competition Rules. In the event that the Submission demonstrates a breach of these Competition Rules, Competition Sponsor may at its discretion take either of the following actions:
\r\n* disqualify your Submission(s); or
* require that you remediate within one week all issues identified in your Submission(s) (including, without limitation, the resolution of license conflicts, the fulfillment of all obligations required by software licenses, and the removal of any software that violates the software restrictions).
A tie between two or more valid and identically ranked submissions will be resolved in favour of the tied submission that was submitted first.
\r\nA Participant may decline to be nominated as a Winner by notifying Kaggle directly within one week following the Competition deadline, in which case the declining Participant forgoes any prize or other features associated with winning the Competition. Kaggle reserves the right to disqualify a Participant who so declines at Kaggle's sole discretion if Kaggle deems disqualification appropriate.
\r\nAs a condition of receipt of the Prize, the Prize winner must deliver the final model’s software code as used to generate the winning Submission and associated documentation (consistent with the winning model documentation template available on the kaggle wiki at https://www.kaggle.com/wiki/WinningModelDocumentationTemplate) to the Competition Sponsor. The delivered software code must be capable of generating the winning Submission and contain a description of resources required to build and/or run the executable code successfully.
\r\nIf a potential winning Participant or Team member is a U.S. citizen, potential winner must also sign and return an IRS W-9 form, or if a foreign resident, an IRS W-8BEN form, within the stated time in order to claim the prize.
\r\nPotential winners will be disqualified and the prize may be awarded to an alternate winner (the next-ranked qualified Participant on the leaderboard) if
\r\n* the required documentation is not returned within 14 days after receipt of any request to provide documentation;
* prize notification letter/email or prize is returned as undeliverable; or
* potential winner or winning Team member is disqualified for any reason.
By accepting any Prize, each Participant receiving a Prize thereby agrees to use of his/her name, address, likeness and/or Prize information by Kaggle and Competition Sponsor for promotional purposes in any medium without additional compensation.
\r\nAs a further condition of receipt of a Prize, each winning Participant thereby licenses their winning Submission and the source code used to generate the Submission according to the Winner License Type specified above (note: if no Winner License Type is specified above, the Winner License Type is deemed to be Non-Exclusive License).
\r\nIf the Winner License Type for the Competition (see Winner License Type above) is a Non-Exclusive License then each Winner by accepting a Prize thereby:
\r\n* grants to Competition Sponsor and its designees a worldwide, non-exclusive, sub-licensable, transferable, fully paid-up, royalty-free, perpetual, irrevocable right to use, not use, reproduce, distribute, create derivative works of, publicly perform, publicly display, digitally perform, make, have made, sell, offer for sale and import their winning Submission and the source code used to generate the Submission, in any media now known or hereafter developed, for any purpose whatsoever, commercial or otherwise, without further approval by or payment to Participant; and
* represents that he/she/it has the unrestricted right to grant that license.
If the Winner License Type for the Competition (see Winner License Type above) is Open Source License, then each Winner by accepting a Prize thereby:
\r\n* licenses their winning Submission and the source code used to generate the Submission under the MIT License (an open source software license commonly described at <http://opensource.org/licenses/MIT>), unless another open source license is chosen explicitly from the list of those approved by the Open Source Initiative at <http://opensource.org/licenses/>; and
* represents that he/she/it has the unrestricted right to grant that license.
If the Competition is classed above as a Recruiting Competition, Participant agrees to grant and hereby grants to Competition Sponsor a perpetual, irrevocable, worldwide, royalty-free, transferable, and sublicenseable right to use, review, reproduce, and internally distribute the Submission (including any and all submitted source code) in connection with evaluating Participant's suitability for employment, provided that the Submission licensed as set forth above will not be used for Competition Host’s commercial purposes except as may otherwise be set forth in this Agreement.
\r\nParticipating using more than one Kaggle account is deemed cheating and, if discovered, will result in disqualification from the Competition and any other affected Competitions and may result in banning or deactivation of affected Kaggle accounts.
\r\nAfter verification of eligibility, each Prize winner will receive the prize in the form of a check or wire transfer made out to the Prize winner (if an individual, or to the individual Team members if a Team). Allow 30 days from final confirmation for Prize delivery. Any winners who are U.S. citizens will receive an IRS 1099 form in the amount of their prize at the appropriate time. Prize winners are responsible for any taxes, fees or other liability resulting from their receipt of a Prize.
\r\nMultiple individuals or entities may collaborate as a team (“Team”). You may not participate on more than one Team. Each Team member must be a single individual operating a separate Kaggle account. You must register individually for the Competition before joining a Team. You must confirm your Team membership to make it official by responding to the Team notification message which will be sent to your Account.
\r\nTeam membership may not exceed the Maximum Team Size.
\r\nTeams may be permitted to merge at Kaggle’s discretion, so long as the merged Team meets all requirements in these Competition Rules. Kaggle will review any merger request and will either approve or reject the request within three business days. Merger requests may be rejected if the combined number of Entries made by the merging Teams exceeds the number of Entries permissible at the date of the merger request. Team merger requests will not be permitted within seven days of any deadline listed on the Competition Website.
\r\nIf a Team wins a monetary Prize, Competition Sponsor will allocate the Prize money in even shares between the Team members unless the Team unanimously contacts Kaggle via the Competition Website within three business days following the Submission deadline to request an alternative prize distribution.
\r\nBy registering, you agree that (a) your Account is complete, correct and accurate and (b) your registration may be rejected or terminated and all Entries submitted by you and/or your Team may be disqualified if any of the information in your Account is (or Competition Sponsor has reasonable grounds to believe it is) incomplete, incorrect or inaccurate. You are solely responsible for your Account. All registration information is deemed collected in the United States.
\r\nParticipation is subject to all federal, state and local laws and regulations. Void where prohibited or restricted by law. You are responsible for checking applicable laws and regulations in your jurisdiction before participating in the Competition to make sure that your participation is legal. You are responsible for all taxes and reporting related to any award that you may receive as part of the Competition. You are responsible for abiding by your employer's policies regarding participation in the Competition. Competition Sponsor disclaims any and all liability or responsibility for disputes arising between you and your employer related to this Competition.
\r\nEach Participant is solely responsible for all equipment, including but not necessarily limited to a computer and internet connection necessary to access the Website and to develop and upload any Submission, and any telephone, data, hosting or other service fees associated with such access, as well as all costs incurred by or behalf of the Entrant in participating in the Competition.
\r\nBy entering a Submission, you represent and warrant that all information you enter on the Website is true and complete to the best of your knowledge, that you have the right and authority to make the Submission (including any underlying code and model) on your own behalf or on behalf of the persons and entities that you specify within the Submission, and that your Submission:
\r\n* is your own original work, or is used by permission, in which case full and proper credit and identify is given and the third party contributions are clearly identified within your Submission;
* does not contain confidential information or trade secrets and is not the subject of a registered patent or pending patent application;
* does not violate or infringe upon the patent rights, industrial design rights, copyrights, trademarks, rights of privacy, publicity or other intellectual property or other rights of any person or entity;
* does not contain malicious code, such as viruses, timebombs, cancelbots, worms, Trojan horses or other potentially harmful programs or other material or information;
* does not and will not violate any applicable law, statute, ordinance, rule or regulation;
* does not trigger any reporting or royalty obligation to any third party; and
* was not previously published and has not won any other prize/award.
A breach of any of these warranties will result in the corresponding Submission being invalid.
By participating in the Competition, each Participant agrees to release, indemnify and hold harmless Competition Sponsor, Kaggle, their respective affiliates, subsidiaries, advertising and promotions agencies, as applicable, and each of their respective agents, representatives, officers, directors, shareholders, and employees from and against any injuries, losses, damages, claims, actions and any liability of any kind resulting from or arising out of your participation in or association with the Competition. Competition Sponsor is not responsible for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete, late or misdirected Submissions. Competition Sponsor reserves the right to cancel, modify or suspend the Competition should any computer virus, bug or other technical difficulty or other causes beyond the control of Competition Sponsor corrupt the administration, security or proper play of the Competition, and to determine winners from among Submission not affected by the corruption, if any, in its sole discretion.
\r\nNeither Kaggle nor Competition Sponsor are responsible for (a) late, lost, stolen, damaged, garbled, incomplete, incorrect or misdirected Entries or other communications, (b) errors, omissions, interruptions, deletions, defects, or delays in operations or transmission of information, in each case whether arising by way of technical or other failures or malfunctions of computer hardware, software, communications devices, or transmission lines, or (c) data corruption, theft, destruction, unauthorized access to or alteration of Submission materials, loss or otherwise. Neither Kaggle nor Competition Sponsor are responsible for electronic communications or emails which are undeliverable as a result of any form of active or passive filtering of any kind, or insufficient space in any email account to receive email messages. Competition Sponsor disclaims any liability for damage to any computer system resulting from participation in, or accessing or downloading information in connection with, the Competition.
\r\nCompetition Sponsor reserves the right to modify the Start Date, End Date, and the right to modify, add or remove additional Competition deadlines for any reason. Any changes will be communicated through the Competition Website. Competition Sponsor also reserves the right to modify, remove or add Data to the Website upon notice via the Competition Website. NEITHER KAGGLE NOR COMPETITION SPONSOR ARE RESPONSIBLE FOR ANY FAILURE OF A PARTICIPANT TO RECEIVE DATA CHANGES.
\r\nThe invalidity or unenforceability of any provision of these Competition Rules shall not affect the validity or enforceability of any other provision. In the event that any provision is determined to be invalid or otherwise unenforceable or illegal, these Competition Rules shall otherwise remain in effect and be construed in accordance with their terms as if the invalid or illegal provision was not contained herein.
\r\nYou agree that these terms and the relationship between you and Competition Sponsor shall be governed by the laws of the State of California and the United States of America.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Timeline", "content": "All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The organizers reserve the right to update the contest timeline if they deem it necessary.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Winners", "content": "1st Place: Yasser Tabandeh
\r\n2nd Place: Charly B.
\r\n3rd Place: CodiLime.com - with this code and model documentation
\r\n