![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1314380%2F3c9c4891c3adb1d3916c2fa74ea4e469%2Finbox_2437718_890e2a2a74191e1528ba30f4e0149333_Oct-23-2019%2015-12-34.gif?generation=1581437893992836&alt=media\")
I'm trying to create my user secret and failing. I must be missing something obvious, so can you help?
\n\n\n\n\n\n\nee.Authenticate()
\n
returns
\n\n\n\n\n\n\nTo authorize access needed by Earth Engine, open the following URL in a web browser and follow the instructions:
\n
\n\n\nThe authorization workflow will generate a code, which you should paste in the box below
\n \nEnter verification code: \"the verification code\"
\n \nSuccessfully saved authorization token.
\n
Now what do I do? Creating a user secret with the verification code doesn't work. It says the authorization token is saved, but saved where?
", "messages": [ { "id": 742865, "postDate": "2020-02-11T15:51:11.317Z", "content": "I'm trying to create my user secret and failing. I must be missing something obvious, so can you help?
\n\n\n\n\n\n\nee.Authenticate()
\n
returns
\n\n\n\n\n\n\nTo authorize access needed by Earth Engine, open the following URL in a web browser and follow the instructions:
\n
\n\n\nThe authorization workflow will generate a code, which you should paste in the box below
\n \nEnter verification code: \"the verification code\"
\n \nSuccessfully saved authorization token.
\n
Now what do I do? Creating a user secret with the verification code doesn't work. It says the authorization token is saved, but saved where?
", "rawMarkdown": "I'm trying to create my user secret and failing. I must be missing something obvious, so can you help?\n\n>>ee.Authenticate()\n\nreturns\n\n\n> To authorize access needed by Earth Engine, open the following URL in a web browser and follow the instructions:\n\nhttps://accounts.google.com/o/oauth2/auth?client_id=517222506229-vsmmajv00ul0bs7p89v5m89qs8eb9359.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fearthengine+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code\n> \nThe authorization workflow will generate a code, which you should paste in the box below\n\n>Enter verification code: \"the verification code\"\n\n>Successfully saved authorization token.\n\nNow what do I do? Creating a user secret with the verification code doesn't work. It says the authorization token is saved, but saved where? ", "votes": 3 }, { "id": 742936, "postDate": "2020-02-11T16:44:42.057Z", "content": "Using Paul's kernel.
\n\nGot secret added but not doing something right because I get both an error message and congrats on being connected (which I am not).
\n\nAlso if I restart the session I get a whole new error that no secret exists - so the indication that doing it once is all that is needed is not totally correct.
", "rawMarkdown": "Using Paul's kernel.\n\nGot secret added but not doing something right because I get both an error message and congrats on being connected (which I am not).\n\nAlso if I restart the session I get a whole new error that no secret exists - so the indication that doing it once is all that is needed is not totally correct.", "replies": [ { "id": 742947, "postDate": "2020-02-11T16:54:51.543Z", "content": "Tried again - after the \nee.Authenticate()\n When I click on the link I am taken to a blank kernel page - nothing happens
User secrets should remain attached and should not become detached if you restart your notebook. You can find more instructions on setting user secrets here. Can you create an arbitrary user secret to test whether or not it is really broken? It seems to be working fine for me.
\n\nAs per clicking on the link that pops up after you run import ee; ee.Authenticate(), you should not click on that link but rather you should open it in a new tab. It only works if you open it in a new tab.
Hopefully that helps, please let us know if you get it working!
", "rawMarkdown": "User secrets should remain attached and should not become detached if you restart your notebook. You can find more instructions on setting user secrets [here](https://www.kaggle.com/product-feedback/114053). Can you create an arbitrary user secret to test whether or not it is really broken? It seems to be working fine for me.\n\nAs per clicking on the link that pops up after you run `import ee; ee.Authenticate()`, you should not click on that link but rather you should open it in a new tab. It only works if you open it in a new tab.\n\nHopefully that helps, please let us know if you get it working!", "votes": 1 }, { "id": 744484, "postDate": "2020-02-12T21:50:23.003Z", "content": "I had issues generating the token because my internet was switched off. There was issue both with ee.Authenticate() (after entering the code) and running !cat ~/.config/earthengine/credentials .
\n\nCould be good to add a comment about verifying internet in the final instructions.\nThanks for a good guidance!
", "rawMarkdown": "I had issues generating the token because my internet was switched off. There was issue both with ee.Authenticate() (after entering the code) and running !cat ~/.config/earthengine/credentials . \n\nCould be good to add a comment about verifying internet in the final instructions.\nThanks for a good guidance!", "votes": 1 }, { "id": 744495, "postDate": "2020-02-12T22:14:27.847Z", "content": "That is true, you need to enable the internet (in the settings menu of the notebook editor) in order to interact with the earthengine-api. So the updated steps would be:
\n\nHow to connect a Kaggle notebook to the earthengine-api:\n* Step 0: Register your account using both of the following two links: Link #1, Link #2 \n* Step 1: Open an internet-enabled notebook and then retrieve your token by opening the ee.Authenticate() link in a new tab. You will also need to copy/paste that value into the relevant input box, and you will need to enable the internet (in the settings menu of the notebook editor) in order for the value to get created in the first place.\n* Step 2: After completing Step 1, retrieve your refresh-token by running !cat ~/.config/earthengine/credentials\n* Step 3: Save your refresh-token as a Kaggle user secret\n* Step 4: Run the code snippet that contains ee.Initialize()
Steps #0, #1, #2, and #3 only need to be performed once (for the initial setup). Step #4 is run every time that you run your code. Hopefully that helps!
", "rawMarkdown": "That is true, you need to enable the internet (in the settings menu of the notebook editor) in order to interact with the earthengine-api. So the updated steps would be:\n\nHow to connect a Kaggle notebook to the earthengine-api:\n* Step 0: Register your account using both of the following two links: [Link #1](https://earthengine.google.com/signup/), [Link #2](https://docs.google.com/forms/d/e/1FAIpQLScFk_pkrrDDF4O8imsEBMaryLDU-Ghf44eHbgujIAl_SXJTJQ/viewform) \n* Step 1: Open an **internet-enabled** notebook and then retrieve your token by opening the ee.Authenticate() link **in a new tab**. You will also need to copy/paste that value into the relevant **input box**, and you will need to enable the internet (in the settings menu of the notebook editor) in order for the value to get created in the first place.\n* Step 2: After completing Step 1, retrieve your **refresh-token** by running `!cat ~/.config/earthengine/credentials`\n* Step 3: Save your refresh-token as a **Kaggle user secret**\n* Step 4: Run the code snippet that contains ee.Initialize()\n\nSteps #0, #1, #2, and #3 only need to be performed once (for the initial setup). Step #4 is run every time that you run your code. Hopefully that helps!" }, { "id": 764314, "postDate": "2020-03-05T10:35:35.417Z", "content": "@paultimothymooney Paul, THANKS A LOT!! Your steps saved me a lot of time. I believe your answer should be part of Kaggle knowledge base for troubleshooting issues with using Secrets addon.
\n\nI would add one step, maybe this is only in my case, but I was able to enable the Internet for the notebook by verifying my phone number. Thought I'll add this in case internet is not On for some users in their notebooks.
\n\nCheers!\nChinmoy.
", "rawMarkdown": "@paultimothymooney Paul, THANKS A LOT!! Your steps saved me a lot of time. I believe your answer should be part of Kaggle knowledge base for troubleshooting issues with using Secrets addon.\n\nI would add one step, maybe this is only in my case, but I was able to enable the Internet for the notebook by verifying my phone number. Thought I'll add this in case internet is not On for some users in their notebooks.\n\nCheers!\nChinmoy.", "votes": 1 } ] }, { "id": 742885, "postDate": "2020-02-11T16:16:51.137Z", "content": "I have attempted to answer this question at the bottom of the starter notebook. Saving the authorization token as a Kaggle user secret is the recommended method to securely connect to the Earth Engine API.
\n\n
Does it work if you use this code snippet in an internet-enabled notebook?
\n\n\nimport ee\nfrom kaggle_secrets import UserSecretsClient\nfrom google.oauth2.credentials import Credentials\nuser_secret = \"earth_engine\" # Your user secret, defined in the add-on menu of the notebook editor\nrefresh_token = UserSecretsClient().get_secret(user_secret)\ncredentials = Credentials(\n None,\n refresh_token=refresh_token,\n token_uri=ee.oauth.TOKEN_URI,\n client_id=ee.oauth.CLIENT_ID,\n client_secret=ee.oauth.CLIENT_SECRET,\n scopes=ee.oauth.SCOPES)\nee.Initialize(credentials=credentials)\n
I'm using your starter notebook, but I guess I'm confused on what exactly I enter for the value of the secret. I tried the verification code that I got from web page, but that didn't seem to work.
", "rawMarkdown": "I'm using your starter notebook, but I guess I'm confused on what exactly I enter for the value of the secret. I tried the verification code that I got from web page, but that didn't seem to work.", "votes": 1 }, { "id": 742920, "postDate": "2020-02-11T16:36:54.600Z", "content": "If you run the following code snippet
\n\n\nimport ee;\nee.Authenticate()\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1314380%2F3ec65f7f274673ddb9f6550b90386bc6%2FScreen%20Shot%202020-02-11%20at%209.33.03%20AM.png?generation=1581438912819873&alt=media\")
it should give you a URL that you can copy and paste into a new tab in your browser. Next a pop-up window should appear where you sign into your Google account and then after you do that you should copy that secret token, paste it into the relevant input box, and then retrieve the actual refresh-token by running !cat ~/.config/earthengine/credentials. This is the value that can then be saved as a user secret in your Kaggle notebook. One place where it is easy to get stuck is if you open the URL in the same tab instead of opening the URL in a new tab. It only works if you open that URL in a new tab.
Hopefully that helps, please let us know if you get it working!
", "rawMarkdown": "If you run the following code snippet\n\n`\nimport ee;\nee.Authenticate()\n`\n\n\n\nit should give you a URL that you can copy and paste into a new tab in your browser. Next a pop-up window should appear where you sign into your Google account and then after you do that you should copy that secret token, paste it into the relevant input box, and then retrieve the actual refresh-token by running `!cat ~/.config/earthengine/credentials`. This is the value that can then be saved as a user secret in your Kaggle notebook. One place where it is easy to get stuck is if you open the URL in the same tab instead of opening the URL in a new tab. It only works if you open that URL in a new tab.\n\nHopefully that helps, please let us know if you get it working!", "votes": 1 }, { "id": 742965, "postDate": "2020-02-11T17:07:38.030Z", "content": "Paul - any chance you can generate a clean flow of what code goes where.
\n\nI did manage to get my token - the need to go to a completly new brower page was the key to success - but now I crash on the next bit of code
\n\nimport ee\nfrom kaggle_secrets import UserSecretsClient\nfrom google.oauth2.credentials import Credentials\nuser_secret = \"earth_engine\" # Your user secret, defined in the add-on menu of the notebook editor\nrefresh_token = UserSecretsClient().get_secret(user_secret)\ncredentials = Credentials(\n None,\n refresh_token=refresh_token,\n token_uri=ee.oauth.TOKEN_URI,\n client_id=ee.oauth.CLIENT_ID,\n client_secret=ee.oauth.CLIENT_SECRET,\n scopes=ee.oauth.SCOPES)\nee.Initialize(credentials=credentials)
But not sure that's the right stuff
\n\nThe flow - give me the flow please
", "rawMarkdown": "Paul - any chance you can generate a clean flow of what code goes where.\n\nI did manage to get my token - the need to go to a completly new brower page was the key to success - but now I crash on the next bit of code\n\n`import ee\nfrom kaggle_secrets import UserSecretsClient\nfrom google.oauth2.credentials import Credentials\nuser_secret = \"earth_engine\" # Your user secret, defined in the add-on menu of the notebook editor\nrefresh_token = UserSecretsClient().get_secret(user_secret)\ncredentials = Credentials(\n None,\n refresh_token=refresh_token,\n token_uri=ee.oauth.TOKEN_URI,\n client_id=ee.oauth.CLIENT_ID,\n client_secret=ee.oauth.CLIENT_SECRET,\n scopes=ee.oauth.SCOPES)\nee.Initialize(credentials=credentials)`\n\nBut not sure that's the right stuff \n\nThe flow - give me the flow please" }, { "id": 742973, "postDate": "2020-02-11T17:14:54.193Z", "content": "Step 1: Retrieve your token (by opening the ee.Authenticate() link in a new tab)\nStep 2: Save your token as a Kaggle user secret\nStep 3: Run the code snippet that contains ee.Initialize()
Steps #1 and #2 only need to be performed once (for the initial setup). Step #3 is run every time that you run your code. Hopefully that helps!
\n\nStep 0: Register your account using both of the following two links: Link #1, Link #2 \nStep 1: Retrieve your first token by opening the ee.Authenticate() link in a new tab (and copying the value that is returned into the relevant input box)\nStep 2: Retrieve your refresh-token by running !cat ~/.config/earthengine/credentials\nStep 3: Save your refresh-token as a Kaggle user secret\nStep 4: Run the code snippet that contains ee.Initialize()
Steps #1, #2, and #3 only need to be performed once (for the initial setup). Step #4 is run every time that you run your code. Hopefully that helps!
", "rawMarkdown": "Step 1: Retrieve your token (by opening the ee.Authenticate() link in a new tab)\nStep 2: Save your token as a [Kaggle user secret](https://www.kaggle.com/product-feedback/114053)\nStep 3: Run the code snippet that contains `ee.Initialize()`\n\nSteps #1 and #2 only need to be performed once (for the initial setup). Step #3 is run every time that you run your code. Hopefully that helps!\n\n### **EDIT: A STEP WAS MISSING**\n\nStep 0: Register your account using both of the following two links: [Link #1](https://earthengine.google.com/signup/), [Link #2](https://docs.google.com/forms/d/e/1FAIpQLScFk_pkrrDDF4O8imsEBMaryLDU-Ghf44eHbgujIAl_SXJTJQ/viewform.) \nStep 1: Retrieve your first token by opening the ee.Authenticate() link in a new tab (and copying the value that is returned into the relevant input box)\nStep 2: Retrieve your refresh-token by running `!cat ~/.config/earthengine/credentials`\nStep 3: Save your refresh-token as a Kaggle user secret\nStep 4: Run the code snippet that contains ee.Initialize()\n\nSteps #1, #2, and #3 only need to be performed once (for the initial setup). Step #4 is run every time that you run your code. Hopefully that helps!", "votes": 1 }, { "id": 742996, "postDate": "2020-02-11T17:31:46.167Z", "content": "To clarify - after I get my token that value should be pasted into the \nVALUE box in the Secret dialog. I can give the secret any LABEL I want and the label is what goes in the code snippet.
\n\nRun the code snippet that you entered in this discussion topic rather than the code that is contained in your notebook?
\n\nEdit- must be something wrong - neither code snip from here or from your notebook works.\nIt fails at this step using either of the snips\nee.Initialize(credentials=credentials)
Yes, in my example the \"label\" is earth_engine and then the value is the secret authorization token. So if you use a different label then you will need to update the code snippet accordingly in order to reference your user secret properly.
The code snippet at the top of this thread was taken directly from the starter notebook. If you share a link to a public notebook it will be much easier to help you troubleshoot your code. Here is another quick demo to show how to use a Kaggle user secret:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1314380%2F7192ccc61c39a2fd099093b824f9c005%2FScreen%20Shot%202020-02-11%20at%2011.16.21%20AM.png?generation=1581445086237699&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1314380%2F2b694bf9ccf880928f5ad8d53bf90ca5%2FScreen%20Shot%202020-02-11%20at%2011.16.55%20AM.png?generation=1581445086321327&alt=media\")
Don't forget to click on the \"save\" button when creating your user secret.
", "rawMarkdown": "Yes, in my example the \"label\" is `earth_engine` and then the value is `the secret authorization token`. So if you use a different label then you will need to update the code snippet accordingly in order to reference your user secret properly.\n\nThe code snippet at the top of this thread was taken directly from the starter notebook. If you share a link to a public notebook it will be much easier to help you troubleshoot your code. Here is another quick demo to show how to use a Kaggle user secret:\n\n\n\n\nDon't forget to click on the \"save\" button when creating your user secret.\n", "votes": 1 }, { "id": 743132, "postDate": "2020-02-11T20:16:21.713Z", "content": "I was saving - it was saved when I closed and returned. I can print the token if desired.
\n\n\n\nHere's my kernel. Note that the kernel shows two error spots - I do not get the first error in ee.Authenticate() when I step thru the kernel. The 2nd error cell is the message I am getting.
\n\nAny ideas why its not working?
", "rawMarkdown": "I was saving - it was saved when I closed and returned. I can print the token if desired.\n\n[My Kernel](https://www.kaggle.com/pcjimmmy/getting-the-token-and-secret-to-work)\n\nHere's my kernel. Note that the kernel shows two error spots - I do not get the first error in ee.Authenticate() when I step thru the kernel. The 2nd error cell is the message I am getting.\n\nAny ideas why its not working?\n\n", "votes": 1 }, { "id": 743176, "postDate": "2020-02-11T21:26:53.910Z", "content": "\n\nThe example for Multi-class prediction with a DNN - loaded and tried to run it from google colab. It walks thru the process of getting a token - it's the same value I get in my kernel.
\n\nWhen attempting the authentication there I get a Permission Denied error.
", "rawMarkdown": "[Example](https://developers.google.com/earth-engine/tf_examples)\n\nThe example for Multi-class prediction with a DNN - loaded and tried to run it from google colab. It walks thru the process of getting a token - it's the same value I get in my kernel. \n\nWhen attempting the authentication there I get a Permission Denied error.", "votes": 1 }, { "id": 743184, "postDate": "2020-02-11T21:38:16.237Z", "content": "Thank you for sharing the public notebook that makes it a lot easier to troubleshoot. The previous version of the tutorial notebook was missing an important instruction. After running ee.Authenticate() and pasting in the value that is returned, your refresh token needs to be retrieved as follows:
\n\n\n!cat ~/.config/earthengine/credentials\n
The refresh token that is returned by !cat ~/.config/earthengine/credentials is the refresh_token that you will want to save as a Kaggle user secret.
Apologies for the confusion, thank you for pointing out that there was a missing step!
\n\nHow to connect a Kaggle notebook to the earthengine-api:\n* Step 0: Register your account using both of the following two links: Link #1, Link #2 \n* Step 1: Open an internet-enabled notebook and then retrieve your token by opening the ee.Authenticate() link in a new tab. You will also need to copy/paste that value into the relevant input box, and you will need to enable the internet (in the settings menu of the notebook editor) in order for the value to get created in the first place.\n* Step 2: After completing Step 1, retrieve your refresh-token by running !cat ~/.config/earthengine/credentials\n* Step 3: Save your refresh-token as a Kaggle user secret\n* Step 4: Run the code snippet that contains ee.Initialize()
Steps #0, #1, #2, and #3 only need to be performed once (for the initial setup). Step #4 is run every time that you run your code. Hopefully that helps!
", "rawMarkdown": "Thank you for sharing the public notebook that makes it a lot easier to troubleshoot. The previous version of the tutorial notebook was missing an important instruction. After running ee.Authenticate() and pasting in the value that is returned, your refresh token needs to be retrieved as follows:\n\n```\n!cat ~/.config/earthengine/credentials\n```\n\nThe refresh token that is returned by ```!cat ~/.config/earthengine/credentials``` is the refresh_token that you will want to save as a Kaggle user secret.\n\nApologies for the confusion, thank you for pointing out that there was a missing step!\n\nHow to connect a Kaggle notebook to the earthengine-api:\n* Step 0: Register your account using both of the following two links: [Link #1](https://earthengine.google.com/signup/), [Link #2](https://docs.google.com/forms/d/e/1FAIpQLScFk_pkrrDDF4O8imsEBMaryLDU-Ghf44eHbgujIAl_SXJTJQ/viewform) \n* Step 1: Open an **internet-enabled** notebook and then retrieve your token by opening the ee.Authenticate() link **in a new tab**. You will also need to copy/paste that value into the relevant **input box**, and you will need to enable the internet (in the settings menu of the notebook editor) in order for the value to get created in the first place.\n* Step 2: After completing Step 1, retrieve your **refresh-token** by running `!cat ~/.config/earthengine/credentials`\n* Step 3: Save your refresh-token as a **Kaggle user secret**\n* Step 4: Run the code snippet that contains ee.Initialize()\n\nSteps #0, #1, #2, and #3 only need to be performed once (for the initial setup). Step #4 is run every time that you run your code. Hopefully that helps!\n", "votes": 1 }, { "id": 743185, "postDate": "2020-02-11T21:39:17.487Z", "content": "I will edit a few of the previous comments to make sure that the final instructions are more visible to the rest of the Kaggle community.
", "rawMarkdown": "I will edit a few of the previous comments to make sure that the final instructions are more visible to the rest of the Kaggle community.", "votes": 1 }, { "id": 743214, "postDate": "2020-02-11T22:11:50.163Z", "content": "I guess there is progress - now instead of a bad request I get a \nPermission Denied error.
\n\nWill commit the revised kernel.
\n\nIn the last cell I print the value of ee.oauth.CLIENT_SECRET \nIt does not match either of the tokens. Is this now my issue?
", "rawMarkdown": "I guess there is progress - now instead of a bad request I get a \nPermission Denied error.\n\nWill commit the revised kernel.\n\nIn the last cell I print the value of ee.oauth.CLIENT_SECRET \nIt does not match either of the tokens. Is this now my issue?\n\n", "votes": 1 }, { "id": 743231, "postDate": "2020-02-11T22:45:49.783Z", "content": "Hmm, I will investigate the issue, but it looks like some accounts will need to get an additional authorization in order to get sufficient permissions to interact with the earthengine-api. See the [overview] page (https://developers.google.com/earth-engine/service_account#register-the-service-account-to-use-earth-engine) and fill out the application form.
\n\nIn the meantime, you should be able to get started with this competition without ever touching the Earth Engine API, as we have provided a dataset starter pack in the official Kaggle dataset.
", "rawMarkdown": "Hmm, I will investigate the issue, but it looks like some accounts will need to get an additional authorization in order to get sufficient permissions to interact with the earthengine-api. See the [overview] page (https://developers.google.com/earth-engine/service_account#register-the-service-account-to-use-earth-engine) and fill out the [application form](https://docs.google.com/forms/d/e/1FAIpQLScFk_pkrrDDF4O8imsEBMaryLDU-Ghf44eHbgujIAl_SXJTJQ/viewform).\n\nIn the meantime, you should be able to get started with this competition without ever touching the Earth Engine API, as we have provided a dataset starter pack in the official [Kaggle dataset](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/data).\n", "votes": 1 }, { "id": 743237, "postDate": "2020-02-11T23:02:32.873Z", "content": "I GOT IT WORKING :)
\n\nThere was one more step not included in your flow - I did not use the link you provided but instead went to Google Earth Engine and signed up. I would guess that the two paths get to the same authorization.
\n\n", "rawMarkdown": "I GOT IT WORKING :) \n\nThere was one more step not included in your flow - I did not use the link you provided but instead went to Google Earth Engine and signed up. I would guess that the two paths get to the same authorization.\n\n[Google Earth Engine](https://earthengine.google.com/)", "votes": 1 }, { "id": 743238, "postDate": "2020-02-11T23:03:22.733Z", "content": "OK - my link and sign up not as scary or complex looking as your link.
", "rawMarkdown": "OK - my link and sign up not as scary or complex looking as your link.", "votes": 1 }, { "id": 743243, "postDate": "2020-02-11T23:16:46.210Z", "content": "I am glad that you got it working! And thank you very much for replying and giving a status update. That is very helpful. In the future I will be sure to direct users to https://earthengine.google.com/signup/ before they attempt to interact with the earthengine-api (Step #0).
", "rawMarkdown": "I am glad that you got it working! And thank you very much for replying and giving a status update. That is very helpful. In the future I will be sure to direct users to https://earthengine.google.com/signup/ before they attempt to interact with the earthengine-api (Step #0). ", "votes": 1 }, { "id": 743258, "postDate": "2020-02-11T23:58:48.710Z", "content": "Also, I should encourage you to be careful with your refresh-token. Currently it is exposed in your public notebook @pcjimmy.
", "rawMarkdown": "Also, I should encourage you to be careful with your refresh-token. Currently it is exposed in your public notebook @pcjimmy." }, { "id": 743259, "postDate": "2020-02-11T23:58:49.430Z", "content": "I assume from your profile that you have lived in Boulder for a while. I grew up in Thornton and attended Mines in Golden (1964-1969). I drove to Mines from Thornton every day most of my school years - I still remember the winter sky. Don't tell anyone - but think you should be able to guess what one of my main factors for my model will be - thanks for the help.
", "rawMarkdown": "I assume from your profile that you have lived in Boulder for a while. I grew up in Thornton and attended Mines in Golden (1964-1969). I drove to Mines from Thornton every day most of my school years - I still remember the winter sky. Don't tell anyone - but think you should be able to guess what one of my main factors for my model will be - thanks for the help.", "votes": 1 }, { "id": 743279, "postDate": "2020-02-12T00:37:13.467Z", "content": "Thanks - will see if I can put some clothes on it. Looks like three forks that might have it - deleted and replaced with version that should not have anything buy the 'jimmy' secret word. Hope it still works :)
\n\nKernel showing how to do all of the above
", "rawMarkdown": "Thanks - will see if I can put some clothes on it. Looks like three forks that might have it - deleted and replaced with version that should not have anything buy the 'jimmy' secret word. Hope it still works :)\n\n[Kernel showing how to do all of the above](https://www.kaggle.com/pcjimmmy/how-to-get-the-secret-and-tokens-and-access)" }, { "id": 743302, "postDate": "2020-02-12T01:08:57.960Z", "content": "Great! Public notebooks and code snippets are helpful for the entire community. Given the \"commit\" workflow (where in addition to running the code in your interactive session you also run the notebook from top to bottom one time after you press \"commit\"), it might be easiest to comment out the ee.Authenticate() step after completing it for the first time because you only have to do that step one time. As per Colorado, that is awesome, I used to commute from Littleton to Boulder and I always liked stopping in Golden. I think that there is an outdoor exhibit at the National Center for Atmospheric Research (in Boulder) that describes the weather phenomenon that you are referring to!
", "rawMarkdown": "Great! Public notebooks and code snippets are helpful for the entire community. Given the \"commit\" workflow (where in addition to running the code in your interactive session you also run the notebook from top to bottom one time after you press \"commit\"), it might be easiest to comment out the ee.Authenticate() step after completing it for the first time because you only have to do that step one time. As per Colorado, that is awesome, I used to commute from Littleton to Boulder and I always liked stopping in Golden. I think that there is an outdoor exhibit at the National Center for Atmospheric Research (in Boulder) that describes the weather phenomenon that you are referring to! " }, { "id": 744712, "postDate": "2020-02-13T04:02:37.193Z", "content": "Paul - a non Kaggle question that you might know for the refresh-token.
\n\nI got your kernel running on my local PC in Ubuntu 19.10. I had to expose my refresh-token in the code to get it to run - so can't put it in a public place like a restroom or Github (not that I am a real github kind of guy).
\n\nOnly been using Linux for 5 months so no idea how to hide this but assume that I can. Do you happen to know how to hide the refresh-token on Linux in Python?
", "rawMarkdown": "Paul - a non Kaggle question that you might know for the refresh-token.\n\nI got your kernel running on my local PC in Ubuntu 19.10. I had to expose my refresh-token in the code to get it to run - so can't put it in a public place like a restroom or Github (not that I am a real github kind of guy).\n\nOnly been using Linux for 5 months so no idea how to hide this but assume that I can. Do you happen to know how to hide the refresh-token on Linux in Python?" }, { "id": 745109, "postDate": "2020-02-13T13:59:43.433Z", "content": "I am not sure what the best method is. I have a few tokens stored as environmental variables on my machine. Otherwise LastPass or KeePass might work.
\n\n\nexport PATH=\"/anaconda3/bin:$PATH\"\nexport TOKEN=\"asdfasdfasdf\"\n
Using Paul's kernel.
\n\nGot secret added but not doing something right because I get both an error message and congrats on being connected (which I am not).
\n\nAlso if I restart the session I get a whole new error that no secret exists - so the indication that doing it once is all that is needed is not totally correct.
", "votes": 0, "replies": [ { "id": 742947, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-11T16:54:51.543000", "content": "Tried again - after the \nee.Authenticate()\n When I click on the link I am taken to a blank kernel page - nothing happens
User secrets should remain attached and should not become detached if you restart your notebook. You can find more instructions on setting user secrets here. Can you create an arbitrary user secret to test whether or not it is really broken? It seems to be working fine for me.
\n\nAs per clicking on the link that pops up after you run import ee; ee.Authenticate(), you should not click on that link but rather you should open it in a new tab. It only works if you open it in a new tab.
Hopefully that helps, please let us know if you get it working!
", "votes": 1, "replies": [] }, { "id": 744484, "author_name": "Alexandra Andersson", "author_url": "", "post_date": "2020-02-12T21:50:23.003000", "content": "I had issues generating the token because my internet was switched off. There was issue both with ee.Authenticate() (after entering the code) and running !cat ~/.config/earthengine/credentials .
\n\nCould be good to add a comment about verifying internet in the final instructions.\nThanks for a good guidance!
", "votes": 1, "replies": [] }, { "id": 744495, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-12T22:14:27.847000", "content": "That is true, you need to enable the internet (in the settings menu of the notebook editor) in order to interact with the earthengine-api. So the updated steps would be:
\n\nHow to connect a Kaggle notebook to the earthengine-api:\n* Step 0: Register your account using both of the following two links: Link #1, Link #2 \n* Step 1: Open an internet-enabled notebook and then retrieve your token by opening the ee.Authenticate() link in a new tab. You will also need to copy/paste that value into the relevant input box, and you will need to enable the internet (in the settings menu of the notebook editor) in order for the value to get created in the first place.\n* Step 2: After completing Step 1, retrieve your refresh-token by running !cat ~/.config/earthengine/credentials\n* Step 3: Save your refresh-token as a Kaggle user secret\n* Step 4: Run the code snippet that contains ee.Initialize()
Steps #0, #1, #2, and #3 only need to be performed once (for the initial setup). Step #4 is run every time that you run your code. Hopefully that helps!
", "votes": 0, "replies": [] }, { "id": 764314, "author_name": "Chinmoy Sarangi", "author_url": "", "post_date": "2020-03-05T10:35:35.417000", "content": "@paultimothymooney Paul, THANKS A LOT!! Your steps saved me a lot of time. I believe your answer should be part of Kaggle knowledge base for troubleshooting issues with using Secrets addon.
\n\nI would add one step, maybe this is only in my case, but I was able to enable the Internet for the notebook by verifying my phone number. Thought I'll add this in case internet is not On for some users in their notebooks.
\n\nCheers!\nChinmoy.
", "votes": 1, "replies": [] } ] }, { "id": 742885, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-11T16:16:51.137000", "content": "I have attempted to answer this question at the bottom of the starter notebook. Saving the authorization token as a Kaggle user secret is the recommended method to securely connect to the Earth Engine API.
\n\n
Does it work if you use this code snippet in an internet-enabled notebook?
\n\n\nimport ee\nfrom kaggle_secrets import UserSecretsClient\nfrom google.oauth2.credentials import Credentials\nuser_secret = \"earth_engine\" # Your user secret, defined in the add-on menu of the notebook editor\nrefresh_token = UserSecretsClient().get_secret(user_secret)\ncredentials = Credentials(\n None,\n refresh_token=refresh_token,\n token_uri=ee.oauth.TOKEN_URI,\n client_id=ee.oauth.CLIENT_ID,\n client_secret=ee.oauth.CLIENT_SECRET,\n scopes=ee.oauth.SCOPES)\nee.Initialize(credentials=credentials)\n
I'm using your starter notebook, but I guess I'm confused on what exactly I enter for the value of the secret. I tried the verification code that I got from web page, but that didn't seem to work.
", "votes": 1, "replies": [] }, { "id": 742920, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-11T16:36:54.600000", "content": "If you run the following code snippet
\n\n\nimport ee;\nee.Authenticate()\n
it should give you a URL that you can copy and paste into a new tab in your browser. Next a pop-up window should appear where you sign into your Google account and then after you do that you should copy that secret token, paste it into the relevant input box, and then retrieve the actual refresh-token by running !cat ~/.config/earthengine/credentials. This is the value that can then be saved as a user secret in your Kaggle notebook. One place where it is easy to get stuck is if you open the URL in the same tab instead of opening the URL in a new tab. It only works if you open that URL in a new tab.
Hopefully that helps, please let us know if you get it working!
", "votes": 1, "replies": [] }, { "id": 742965, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-11T17:07:38.030000", "content": "Paul - any chance you can generate a clean flow of what code goes where.
\n\nI did manage to get my token - the need to go to a completly new brower page was the key to success - but now I crash on the next bit of code
\n\nimport ee\nfrom kaggle_secrets import UserSecretsClient\nfrom google.oauth2.credentials import Credentials\nuser_secret = \"earth_engine\" # Your user secret, defined in the add-on menu of the notebook editor\nrefresh_token = UserSecretsClient().get_secret(user_secret)\ncredentials = Credentials(\n None,\n refresh_token=refresh_token,\n token_uri=ee.oauth.TOKEN_URI,\n client_id=ee.oauth.CLIENT_ID,\n client_secret=ee.oauth.CLIENT_SECRET,\n scopes=ee.oauth.SCOPES)\nee.Initialize(credentials=credentials)
But not sure that's the right stuff
\n\nThe flow - give me the flow please
", "votes": 0, "replies": [] }, { "id": 742973, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-11T17:14:54.193000", "content": "Step 1: Retrieve your token (by opening the ee.Authenticate() link in a new tab)\nStep 2: Save your token as a Kaggle user secret\nStep 3: Run the code snippet that contains ee.Initialize()
Steps #1 and #2 only need to be performed once (for the initial setup). Step #3 is run every time that you run your code. Hopefully that helps!
\n\nStep 0: Register your account using both of the following two links: Link #1, Link #2 \nStep 1: Retrieve your first token by opening the ee.Authenticate() link in a new tab (and copying the value that is returned into the relevant input box)\nStep 2: Retrieve your refresh-token by running !cat ~/.config/earthengine/credentials\nStep 3: Save your refresh-token as a Kaggle user secret\nStep 4: Run the code snippet that contains ee.Initialize()
Steps #1, #2, and #3 only need to be performed once (for the initial setup). Step #4 is run every time that you run your code. Hopefully that helps!
", "votes": 1, "replies": [] }, { "id": 742996, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-11T17:31:46.167000", "content": "To clarify - after I get my token that value should be pasted into the \nVALUE box in the Secret dialog. I can give the secret any LABEL I want and the label is what goes in the code snippet.
\n\nRun the code snippet that you entered in this discussion topic rather than the code that is contained in your notebook?
\n\nEdit- must be something wrong - neither code snip from here or from your notebook works.\nIt fails at this step using either of the snips\nee.Initialize(credentials=credentials)
Yes, in my example the \"label\" is earth_engine and then the value is the secret authorization token. So if you use a different label then you will need to update the code snippet accordingly in order to reference your user secret properly.
The code snippet at the top of this thread was taken directly from the starter notebook. If you share a link to a public notebook it will be much easier to help you troubleshoot your code. Here is another quick demo to show how to use a Kaggle user secret:
\n\n\n
Don't forget to click on the \"save\" button when creating your user secret.
", "votes": 1, "replies": [] }, { "id": 743132, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-11T20:16:21.713000", "content": "I was saving - it was saved when I closed and returned. I can print the token if desired.
\n\n\n\nHere's my kernel. Note that the kernel shows two error spots - I do not get the first error in ee.Authenticate() when I step thru the kernel. The 2nd error cell is the message I am getting.
\n\nAny ideas why its not working?
", "votes": 1, "replies": [] }, { "id": 743176, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-11T21:26:53.910000", "content": "\n\nThe example for Multi-class prediction with a DNN - loaded and tried to run it from google colab. It walks thru the process of getting a token - it's the same value I get in my kernel.
\n\nWhen attempting the authentication there I get a Permission Denied error.
", "votes": 1, "replies": [] }, { "id": 743184, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-11T21:38:16.237000", "content": "Thank you for sharing the public notebook that makes it a lot easier to troubleshoot. The previous version of the tutorial notebook was missing an important instruction. After running ee.Authenticate() and pasting in the value that is returned, your refresh token needs to be retrieved as follows:
\n\n\n!cat ~/.config/earthengine/credentials\n
The refresh token that is returned by !cat ~/.config/earthengine/credentials is the refresh_token that you will want to save as a Kaggle user secret.
Apologies for the confusion, thank you for pointing out that there was a missing step!
\n\nHow to connect a Kaggle notebook to the earthengine-api:\n* Step 0: Register your account using both of the following two links: Link #1, Link #2 \n* Step 1: Open an internet-enabled notebook and then retrieve your token by opening the ee.Authenticate() link in a new tab. You will also need to copy/paste that value into the relevant input box, and you will need to enable the internet (in the settings menu of the notebook editor) in order for the value to get created in the first place.\n* Step 2: After completing Step 1, retrieve your refresh-token by running !cat ~/.config/earthengine/credentials\n* Step 3: Save your refresh-token as a Kaggle user secret\n* Step 4: Run the code snippet that contains ee.Initialize()
Steps #0, #1, #2, and #3 only need to be performed once (for the initial setup). Step #4 is run every time that you run your code. Hopefully that helps!
", "votes": 1, "replies": [] }, { "id": 743185, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-11T21:39:17.487000", "content": "I will edit a few of the previous comments to make sure that the final instructions are more visible to the rest of the Kaggle community.
", "votes": 1, "replies": [] }, { "id": 743214, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-11T22:11:50.163000", "content": "I guess there is progress - now instead of a bad request I get a \nPermission Denied error.
\n\nWill commit the revised kernel.
\n\nIn the last cell I print the value of ee.oauth.CLIENT_SECRET \nIt does not match either of the tokens. Is this now my issue?
", "votes": 1, "replies": [] }, { "id": 743231, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-11T22:45:49.783000", "content": "Hmm, I will investigate the issue, but it looks like some accounts will need to get an additional authorization in order to get sufficient permissions to interact with the earthengine-api. See the [overview] page (https://developers.google.com/earth-engine/service_account#register-the-service-account-to-use-earth-engine) and fill out the application form.
\n\nIn the meantime, you should be able to get started with this competition without ever touching the Earth Engine API, as we have provided a dataset starter pack in the official Kaggle dataset.
", "votes": 1, "replies": [] }, { "id": 743237, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-11T23:02:32.873000", "content": "I GOT IT WORKING :)
\n\nThere was one more step not included in your flow - I did not use the link you provided but instead went to Google Earth Engine and signed up. I would guess that the two paths get to the same authorization.
\n\n", "votes": 1, "replies": [] }, { "id": 743238, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-11T23:03:22.733000", "content": "OK - my link and sign up not as scary or complex looking as your link.
", "votes": 1, "replies": [] }, { "id": 743243, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-11T23:16:46.210000", "content": "I am glad that you got it working! And thank you very much for replying and giving a status update. That is very helpful. In the future I will be sure to direct users to https://earthengine.google.com/signup/ before they attempt to interact with the earthengine-api (Step #0).
", "votes": 1, "replies": [] }, { "id": 743258, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-11T23:58:48.710000", "content": "Also, I should encourage you to be careful with your refresh-token. Currently it is exposed in your public notebook @pcjimmy.
", "votes": 0, "replies": [] }, { "id": 743259, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-11T23:58:49.430000", "content": "I assume from your profile that you have lived in Boulder for a while. I grew up in Thornton and attended Mines in Golden (1964-1969). I drove to Mines from Thornton every day most of my school years - I still remember the winter sky. Don't tell anyone - but think you should be able to guess what one of my main factors for my model will be - thanks for the help.
", "votes": 1, "replies": [] }, { "id": 743279, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-12T00:37:13.467000", "content": "Thanks - will see if I can put some clothes on it. Looks like three forks that might have it - deleted and replaced with version that should not have anything buy the 'jimmy' secret word. Hope it still works :)
\n\nKernel showing how to do all of the above
", "votes": 0, "replies": [] }, { "id": 743302, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-12T01:08:57.960000", "content": "Great! Public notebooks and code snippets are helpful for the entire community. Given the \"commit\" workflow (where in addition to running the code in your interactive session you also run the notebook from top to bottom one time after you press \"commit\"), it might be easiest to comment out the ee.Authenticate() step after completing it for the first time because you only have to do that step one time. As per Colorado, that is awesome, I used to commute from Littleton to Boulder and I always liked stopping in Golden. I think that there is an outdoor exhibit at the National Center for Atmospheric Research (in Boulder) that describes the weather phenomenon that you are referring to!
", "votes": 0, "replies": [] }, { "id": 744712, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-02-13T04:02:37.193000", "content": "Paul - a non Kaggle question that you might know for the refresh-token.
\n\nI got your kernel running on my local PC in Ubuntu 19.10. I had to expose my refresh-token in the code to get it to run - so can't put it in a public place like a restroom or Github (not that I am a real github kind of guy).
\n\nOnly been using Linux for 5 months so no idea how to hide this but assume that I can. Do you happen to know how to hide the refresh-token on Linux in Python?
", "votes": 0, "replies": [] }, { "id": 745109, "author_name": "Paul Mooney", "author_url": "", "post_date": "2020-02-13T13:59:43.433000", "content": "I am not sure what the best method is. I have a few tokens stored as environmental variables on my machine. Otherwise LastPass or KeePass might work.
\n\n\nexport PATH=\"/anaconda3/bin:$PATH\"\nexport TOKEN=\"asdfasdfasdf\"\n