Anyone knows why he set 0.9 here ? Could we choose neg samples by other methods? and because of the random , it seems not stable when train this model. I want to find some better ways to choose neg samples , butI failed….
\nhttps://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training?scriptVersionId=96291806&cellId=17
\n`
\ndef generate_triplet(df, mode='train'):
\n triplets = []
\n ids = df.id.unique()
\n random_drop = np.random.random(size=10000)>0.9
\n count = 0
for id, df_tmp in tqdm(df.groupby('id')):
\n df_tmp_markdown = df_tmp[df_tmp['cell_type']=='markdown']
df_tmp_code = df_tmp[df_tmp['cell_type']=='code']\ndf_tmp_code_rank = df_tmp_code['rank'].values\ndf_tmp_code_cell_id = df_tmp_code['cell_id'].values\n\nfor cell_id, rank in df_tmp_markdown[['cell_id', 'rank']].values:\n labels = np.array([(r==(rank+1)) for r in df_tmp_code_rank]).astype('int')\n\n for cid, label in zip(df_tmp_code_cell_id, labels):\n count += 1\n if label==1:\n triplets.append( [cell_id, cid, label] )\n # triplets.append( [cid, cell_id, label] )\n elif mode == 'test':\n triplets.append( [cell_id, cid, label] )\n # triplets.append( [cid, cell_id, label] )\n elif random_drop[count%10000]:\n triplets.append( [cell_id, cid, label] )\n # triplets.append( [cid, cell_id, label] )\nreturn triplets
\ntriplets = generate_triplet(train_df)
\nval_triplets = generate_triplet(val_df, mode = 'test')`
Anyone knows why he set 0.9 here ? Could we choose neg samples by other methods? and because of the random , it seems not stable when train this model. I want to find some better ways to choose neg samples , butI failed….
\nhttps://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training?scriptVersionId=96291806&cellId=17
\n`
\ndef generate_triplet(df, mode='train'):
\n triplets = []
\n ids = df.id.unique()
\n random_drop = np.random.random(size=10000)>0.9
\n count = 0
for id, df_tmp in tqdm(df.groupby('id')):
\n df_tmp_markdown = df_tmp[df_tmp['cell_type']=='markdown']
df_tmp_code = df_tmp[df_tmp['cell_type']=='code']\ndf_tmp_code_rank = df_tmp_code['rank'].values\ndf_tmp_code_cell_id = df_tmp_code['cell_id'].values\n\nfor cell_id, rank in df_tmp_markdown[['cell_id', 'rank']].values:\n labels = np.array([(r==(rank+1)) for r in df_tmp_code_rank]).astype('int')\n\n for cid, label in zip(df_tmp_code_cell_id, labels):\n count += 1\n if label==1:\n triplets.append( [cell_id, cid, label] )\n # triplets.append( [cid, cell_id, label] )\n elif mode == 'test':\n triplets.append( [cell_id, cid, label] )\n # triplets.append( [cid, cell_id, label] )\n elif random_drop[count%10000]:\n triplets.append( [cell_id, cid, label] )\n # triplets.append( [cid, cell_id, label] )\nreturn triplets
\ntriplets = generate_triplet(train_df)
\nval_triplets = generate_triplet(val_df, mode = 'test')`
We ran this code with the same hyperparameters, but the results were worse. I would to make some experiments with that, but Pairwise is so slow and unstable (depending on seed choice).
", "rawMarkdown": "We ran this code with the same hyperparameters, but the results were worse. I would to make some experiments with that, but Pairwise is so slow and unstable (depending on seed choice).", "votes": null }, { "id": "1870039", "postDate": "07/25/2022 08:16:52", "content": "I remember him mentioning somewhere that this roughly translates to 5% sampling (of possible -ve pairs). He mentions that anything more or less deteriorates the score.
\nfor repeatability did u try numpy.random.seed(42)? There are some caveats though
\nI wanted to explore other triplet generating algos, but didnt get a chance so far. There are quite a few available
", "rawMarkdown": "I remember him mentioning somewhere that this roughly translates to 5% sampling (of possible -ve pairs). He mentions that anything more or less deteriorates the score. \n\nfor repeatability did u try numpy.random.seed(42)? There are some caveats though\n\nI wanted to explore other triplet generating algos, but didnt get a chance so far. There are quite a few available", "votes": null }, { "id": "1870251", "postDate": "07/25/2022 12:09:44", "content": "I'll try this numpy.random.seed(42), I tried some other ways to generate triplet, but the result is not good
", "rawMarkdown": "I'll try this numpy.random.seed(42), I tried some other ways to generate triplet, but the result is not good", "votes": null }, { "id": "1871442", "postDate": "07/26/2022 09:16:21", "content": "I think it's a way to balance the dataset..
\nBut yes it is very sensitive to the seed.. (which is not good)
We ran this code with the same hyperparameters, but the results were worse. I would to make some experiments with that, but Pairwise is so slow and unstable (depending on seed choice).
", "votes": null, "replies": [] }, { "id": 1870039, "author_name": "allohvk", "author_url": "", "post_date": "07/25/2022 08:16:52", "content": "I remember him mentioning somewhere that this roughly translates to 5% sampling (of possible -ve pairs). He mentions that anything more or less deteriorates the score.
\nfor repeatability did u try numpy.random.seed(42)? There are some caveats though
\nI wanted to explore other triplet generating algos, but didnt get a chance so far. There are quite a few available
", "votes": null, "replies": [ { "id": 1870251, "author_name": "zhzeng", "author_url": "", "post_date": "07/25/2022 12:09:44", "content": "I'll try this numpy.random.seed(42), I tried some other ways to generate triplet, but the result is not good
", "votes": null, "replies": [] } ] }, { "id": 1871442, "author_name": "thedevastator", "author_url": "", "post_date": "07/26/2022 09:16:21", "content": "I think it's a way to balance the dataset..
\nBut yes it is very sensitive to the seed.. (which is not good)