{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# ELO: Probey McProbeface\n", "\n", "Looking at the lower regions of the [leaderboard][1] for the [Elo Merchant Category Recommendation][2] competition, a cluster of users is very apparent. All with the same score: they all moved about 83 spots down, all made 25 submissions (0 after the deadline) and all have their last submission on the same day (16th Feb 2019).\n", "\n", "\n", "\n", "This looks very much like systematic leaderboard probing, with no attempt to hide this behaviour, for example by using a top kernel submission with a better score, they would blend into the regular crowd and be less conspicuous.\n", "\n", "Five are named \"Ella\" and there are further regularities hiding in the data...\n", "\n", "This notebook uses the [Meta Kaggle][3] dataset to pull on this thread and finds at least 56 teams **still** on the leaderboard that only made *probing* submissions.\n", "\n", "[1]: https://www.kaggle.com/c/elo-merchant-category-recommendation/leaderboard\n", "[2]: https://www.kaggle.com/c/elo-merchant-category-recommendation\n", "[3]: https://www.kaggle.com/kaggle/meta-kaggle\n"]}, {"cell_type": "code", "execution_count": 1, "metadata": {"_kg_hide-input": true}, "outputs": [], "source": ["%matplotlib inline\n", "import gc, os, sys, time\n", "import pandas as pd, numpy as np\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "from pathlib import Path\n", "from IPython.display import HTML, Image, display\n", "\n", "HOST = 'https://www.kaggle.com'\n", "SLUG = 'elo-merchant-category-recommendation'\n", "FIGSIZE = (9,9)\n", "\n", "MK = Path(f'../input/meta-kaggle')\n", "ELO = Path(f'../input/{SLUG}')\n", "\n", "def read_csv_filtered(csv, col, values, **kwargs):\n", " dfs = [df.loc[df[col].isin(values)]\n", " for df in pd.read_csv(csv, chunksize=50000, low_memory=False, **kwargs)]\n", " return pd.concat(dfs, axis=0)\n", "\n", "comp = read_csv_filtered(MK / 'Competitions.csv', 'Slug', {SLUG}).set_index('Id')"]}, {"cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": ["comp[[\n", " 'Title', 'Slug', 'Subtitle', 'DeadlineDate', 'EvaluationAlgorithmName',\n", " 'MaxDailySubmissions', 'LeaderboardPercentage', 'TotalTeams',\n", " 'TotalCompetitors', 'TotalSubmissions'\n", "]].T"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Leaderboard Probing\n", "\n", "Leaderboard probing was very prominent in the [Mercedes-Benz Greener Manufacturing competition][1]. See that thread for an introduction.\n", "\n", "Also:\n", "\n", "[How to get the exact y-values of all data points used for the computation of the public leaderboard score in the \u201dMercedes-Benz Greener Manufacturing\u201d competition on Kaggle][2].\n", "\n", "So, why probe?\n", "\n", "The ELO competition used the RMSE metric, with the important detail that about 1% of the training targets were severe outliers of -33.219281, very large compared to the overall standard deviation of just 3.850500.\n", "\n", "\n", "\n", "The winning solutions in this competition made heavy use of outlier detection, i.e. classification models to predict the probability a row is an outlier. Feature selection techniques will then select different feature sets based on classifier loss functions. The final predictions of winning teams were based on RMSE based regression models and classification models combined. Focussing on outlier loss and inlier loss separately lead to better predictions.\n", "\n", "Probing allows gathering extra target values from the public portion of the test set. Public LB labels are useless on their own for final standings. If they were indeed mined by one person/entity they would only be useful for **building a better outlier classifier**.\n", "\n", "\n", "[1]: https://www.kaggle.com/c/mercedes-benz-greener-manufacturing/discussion/35271\n", "[2]: https://crowdstats.eu/"]}, {"cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": ["train = pd.read_csv(ELO / 'train.csv')\n", "train.shape"]}, {"cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": ["test = pd.read_csv(ELO / 'test.csv')\n", "test.shape"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Clearly visible outliers:"]}, {"cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": ["train.target.plot.hist(bins=100)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The raw stats:"]}, {"cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": ["train.target.describe()"]}, {"cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": ["train.target.value_counts().head()"]}, {"cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": ["train.target.value_counts(normalize=True).head()"]}, {"cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": ["pOutlier = 0.010930"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Probing Demo\n", "\n", "For ELO, even though the public LB showed only 3 decimal places, that is enough to get feedback probing one line at a time. (Better schemes are possible but I will stick to one line per submission here.)\n", "\n", "Below I mock up a private LB and probing system and show the resulting submission scores. Remember only the public scores are available before the deadline. In my demo, a single submission predicts zeros, with **one row** predicting -33, the outlier value.\n"]}, {"cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": ["public_fraction = comp.LeaderboardPercentage.values[0] / 100\n", "public_fraction"]}, {"cell_type": "code", "execution_count": 11, "metadata": {"_kg_hide-input": true}, "outputs": [], "source": ["ntest = test.shape[0]\n", "npublic = int(ntest * public_fraction)\n", "# sample from training set - same size as test set\n", "fake_test = train.target.sample(n=ntest, random_state=42).to_frame('target')\n", "fake_test = fake_test.reset_index(drop=True)\n", "fake_test['isprivate'] = 1\n", "fake_test.loc[0:npublic, 'isprivate'] = 0\n", "# shuffle, so we can test just probing the first N rows\n", "fake_test = fake_test.sample(frac=1, random_state=42).reset_index(drop=True)\n", "\n", "def probe_sub(i):\n", " preds = np.zeros_like(fake_test.target)\n", " preds[i] = -33 # set one prediction to the outlier value\n", " # compute squared error per row\n", " tmp = fake_test.assign(err=(fake_test.target - preds) ** 2)\n", " return list(tmp.groupby('isprivate').err.mean() ** .5) + [\n", " tmp.target.values[i],\n", " tmp.isprivate.values[i]\n", " ]\n", "\n", "# do 1000 probing submissions - is this unrealistic? Spoiler alert: NO!\n", "cols = ['public', 'private', 'true_target', 'is_private']\n", "demo_probes = pd.DataFrame([probe_sub(i) for i in range(1000)], columns=cols)\n", "# save in case you want a closer look: remember only 3dp were shown\n", "demo_probes.to_csv('demo_probes.csv', index=False)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["This is how submissions scores look.\n", "\n", "For all probed lines that are in the private LB set, the public score received will be constant (red points). Those submissions are useless."]}, {"cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": ["demo_probes.plot.scatter('public',\n", " 'private',\n", " c=np.where(demo_probes.is_private, 'red', 'blue'),\n", " figsize=FIGSIZE,\n", " alpha=0.4)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["However, when a probe hits upon an outlier, the public score significantly drops. Now the outliers are in red, and it is clear you can detect them (on the public LB axis) with 3 dp scores: 3.858 versus around 3.866, with a large, reliable margin inbetween."]}, {"cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": ["demo_probes.plot.scatter('public',\n", " 'private',\n", " c=np.where(demo_probes.true_target<-20, 'red', 'blue'),\n", " figsize=FIGSIZE,\n", " alpha=0.4)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Here are the figures: note that 1000 naive probes get us only FOUR public outliers!!! (true_target==-33.219 and is_private=0).\n", "\n", "This is as expected. The public LB is only 30% of the data: we only get public LB feedback 30% of the time, and of those times an outlier is about 1% likely, so 3 in 1000 is average.\n", "\n", "However, if you use out of fold predictions of outliers to guide which rows you probe, the hit rate can probably be improved a lot."]}, {"cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": ["(1000 * public_fraction) * pOutlier"]}, {"cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": ["demo_probes.round(3).sort_values(['is_private', 'true_target']).query('true_target<-20')"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Meta Kaggle: Read ELO Teams and Submissions\n", "\n", "Now inspect all the teams and the submissions they made in the ELO competition.\n"]}, {"cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": ["COMP_ID = comp.index[0]\n", "COMP_ID"]}, {"cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": ["teams = read_csv_filtered(MK / 'Teams.csv', 'CompetitionId', {COMP_ID}).set_index('Id')\n", "teams.shape"]}, {"cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": ["tm = read_csv_filtered(MK / 'TeamMemberships.csv', 'TeamId', teams.index).set_index('Id')\n", "tm.shape"]}, {"cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": ["subs = read_csv_filtered(MK / 'Submissions.csv', 'TeamId', teams.index)\n", "subs.shape"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Post deadline submissions are obviously irrelevant! (Note this drops over 10k submissions - impressive! It's good that people go on working and improving their solutions after the competition has finished.)"]}, {"cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": ["subs = subs.query('not IsAfterDeadline').drop('IsAfterDeadline', axis=1)\n", "subs = subs.set_index('Id').sort_index().copy()\n", "subs.shape"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Quick plot for curiosity - public & private scores very correlated. The competition is all taking place in that narrow corridor. Only the private score matters at the end."]}, {"cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": ["subs.query('PublicScoreFullPrecision<5 and PrivateScoreFullPrecision<5').plot.scatter(\n", " 'PublicScoreFullPrecision',\n", " 'PrivateScoreFullPrecision',\n", " figsize=FIGSIZE,\n", " alpha=0.4)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Read all the users that took part."]}, {"cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": ["users = read_csv_filtered(MK / 'Users.csv', 'Id', tm.UserId).set_index('Id')\n", "users.shape"]}, {"cell_type": "markdown", "metadata": {}, "source": ["For convenience add user names and team names to all submissions."]}, {"cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": ["subs['UserName'] = subs.SubmittedUserId.map(users.UserName)\n", "subs['TeamName'] = subs.TeamId.map(teams.TeamName)"]}, {"cell_type": "code", "execution_count": 24, "metadata": {"_kg_hide-input": true}, "outputs": [], "source": ["SUBS_SHOW = [\n", " 'SubmittedUserId', 'TeamId', 'SubmissionDate', 'PublicScoreFullPrecision',\n", " 'PrivateScoreFullPrecision', 'UserName', 'TeamName'\n", "]"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Start off by inspecting **ellaneumann**... Five subs a day, all with uncompetitive probey-looking scores."]}, {"cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": ["subs.query('UserName==\"ellaneumann\"')[SUBS_SHOW]"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Better to use a plot - oh - that looks familiar!!!"]}, {"cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": ["subs.query('UserName==\"ellaneumann\"').plot.scatter('PublicScoreFullPrecision', 'PrivateScoreFullPrecision')"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Try another - BAM! HIT!"]}, {"cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": ["subs.query('UserName==\"bogodaellaneumann\"').plot.scatter('PublicScoreFullPrecision', 'PrivateScoreFullPrecision')"]}, {"cell_type": "markdown", "metadata": {}, "source": ["And another - BINGO!"]}, {"cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": ["subs.query('UserName==\"cubadoellaneumann\"').plot.scatter('PublicScoreFullPrecision', 'PrivateScoreFullPrecision')"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Someone is getting lazy with the username generation now - looks like all these subs by all these users are probes :/"]}, {"cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": ["subs.query('UserName==\"user1ellaneumann\"').plot.scatter('PublicScoreFullPrecision', 'PrivateScoreFullPrecision')"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Quick check - what are the most common public/private scores? These likely correspond to predicting zero for that subset. (If so, they are also the standard deviation of the target in public/private test.)"]}, {"cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": ["subs.PublicScoreFullPrecision.value_counts().head()"]}, {"cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": ["subs.PrivateScoreFullPrecision.value_counts().head()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Let's assume one bad actor is using multiple accounts and using ONE probing mechanism for all. What is the simplest thing that works here? Count the appearances of the FULL precision scores, amongst ALL submissions for the competition."]}, {"cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": ["subs['count_pub'] = subs.PublicScoreFullPrecision.map(subs.PublicScoreFullPrecision.value_counts())\n", "subs['count_prv'] = subs.PrivateScoreFullPrecision.map(subs.PrivateScoreFullPrecision.value_counts())"]}, {"cell_type": "code", "execution_count": 33, "metadata": {"_kg_hide-input": true}, "outputs": [], "source": ["SUBS_SHOW = [\n", " 'TeamId', 'SubmissionDate', 'PublicScoreFullPrecision',\n", " 'PrivateScoreFullPrecision', 'UserName', 'count_pub', 'count_prv'\n", "]"]}, {"cell_type": "markdown", "metadata": {}, "source": ["So the one thing true of all **ellaneumann**'s subs is they have a public score seen 1343 times in total, or a private score seen 900 times in total, **but never both**. This depends on whether the row probed is public or private. (We could use full precision float64 score values themselves but comparing floats directly is trickier. Counts work ok.)"]}, {"cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": ["subs.query('UserName==\"ellaneumann\"')[SUBS_SHOW]"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Mark all submissions that match the criteria - there are **1053 - WOW**"]}, {"cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [], "source": ["subs['IsProbe'] = (((subs.count_pub == 1343) & (subs.count_prv != 900)) |\n", " ((subs.count_pub != 1343) & (subs.count_prv == 900)))\n", "subs['IsProbe'].sum()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# IsProbe Stats\n", "\n", "Make stats on probing submission per team."]}, {"cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": ["stats = subs.groupby('TeamId').IsProbe.agg(['count','sum','mean'])\n", "stats.columns = ['count_subs', 'sum_probes', 'mean_probes']\n", "stats.insert(0, 'PrivateLeaderboardRank', stats.index.map(teams.PrivateLeaderboardRank))\n", "stats.insert(0, 'PublicLeaderboardRank', stats.index.map(teams.PublicLeaderboardRank))\n", "stats.insert(0, 'LeaderUserName', teams.loc[stats.index].TeamLeaderId.map(users.UserName))\n", "stats.insert(0, 'TeamName', stats.index.map(teams.TeamName))"]}, {"cell_type": "markdown", "metadata": {}, "source": ["It's usual that sum and mean of probing subs per team are 0. Good. Most teams don't probe."]}, {"cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": ["stats.describe().T"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Some random teams: sum and mean of probing submissions is 0 - none of these teams probed."]}, {"cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": ["stats.sample(n=15, random_state=42)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["So which teams made **only** probing submissions?"]}, {"cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": ["prober_stats = stats.query('mean_probes==1')\n", "prober_stats.shape"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Oooh err... 56 of them. The cluster that is visible in the screenshot at the top is all here, along with many more :-/\n", "\n", "Bonus \"evidence\": all but one are multiples of 5 submissions. The limit is 5 submissions per day.\n", "\n", "This table is indexed/sorted by TeamId → same as time. Regularities in the **TeamName** are obvious."]}, {"cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": ["prober_stats"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Mark subs by teams that ONLY probed."]}, {"cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": ["subs['PureProbeTeam'] = subs.TeamId.isin(prober_stats.index)\n", "subs['PureProbeTeam'].sum()"]}, {"cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [], "source": ["subs.to_csv('elo_submissions.csv')"]}, {"cell_type": "markdown", "metadata": {}, "source": ["What about teams that did SOME probe submissions matching this big block of pure probers?\n", "\n", "Two at the end \"Nika Zhidkonozhkina\" and \"Kim Pem\" have many probe submissions. But that does not prove they also created shell accounts.\n", "\n", "It's a fairly small set of teams. I don't want to accuse any of these; some could be coincidences. Meta Kaggle does not give us the evidence to say. Submission timing: something may be possible? (If you can think of some extra test to do here please comment!)\n", "\n", "One note: there is a [Shakeup incoming!][1] thread where [raddar][2] says:\n", "\n", " p.s. we haven't focused on overrides for a long time now\n", "\n", " There is couple overrides I was able to get during my testing phase early in the competition:\n", " C_ID_b789a1772c, C_ID_944c62886f\n", "\n", "But his team is not listed here - indicating he was using a different probing technique? Perhaps using the mean training value instead of zero. Or probably something smarter that gets more targets at once 8-)\n", "\n", "[1]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/81657\n", "[2]: https://www.kaggle.com/raddar"]}, {"cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": ["stats.query('mean_probes>0 and mean_probes<1').sort_values('mean_probes')"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Look at user IDs of members of pure-Probey-McProbeface teams."]}, {"cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [], "source": ["probe_members = tm.loc[tm.TeamId.isin(prober_stats.index)]\n", "probe_members.shape"]}, {"cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [], "source": ["probe_members.head()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Look them up in the users table - notice the **RegisterDate**.\n", "\n", "There are https link in the table - try some! Nearly all are **\"Joined a year ago \u00b7 last seen a year ago\"**.\n", "\n", "Note how \"neumann\", \"schulz\", \"hofmann\" are created early (lower user IDs), then repeated later, multiple times in streaks! Are these famous names? Is there some cultural link I am missing?\n", "\n", "Also: gaps in the index (Id = UserId) are quite low. We don't see signup time but could approximate it from Ids. Perhaps account creation is too easy?"]}, {"cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [], "source": ["df = users.loc[probe_members.UserId]\n", "df.assign(UserName=df.apply(lambda r: f'{r.UserName}', 1)).sort_index().style"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Six day span - they took 13th Feb off?"]}, {"cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [], "source": ["users.loc[probe_members.UserId].RegisterDate.value_counts().sort_index()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Plot all probe subs together... Looks like in their scheme - anything under 3.95 public indicated an outlier."]}, {"cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": ["subs.query('PureProbeTeam').plot.scatter('PublicScoreFullPrecision',\n", " 'PrivateScoreFullPrecision',\n", " title='All Probe Submissions',\n", " alpha=0.2,\n", " figsize=FIGSIZE)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["And so, it looks like they mined **54** public test set outliers with 794 probes. (54 of 794 is about 7% hit rate: way better than random guessing, so they were smart enough to target likely rows.)"]}, {"cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [], "source": ["subs.query('PureProbeTeam').shape"]}, {"cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [], "source": ["subs.query('PureProbeTeam and (PublicScoreFullPrecision<3.95)').shape"]}, {"cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [], "source": ["54 / 794"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Lump all these probe submission into one (worst case scenario is it is one entity), between them **90** outliers from **1053** probes for **8.5%** - better rate! Better initial outlier predictions for guiding which rows to probe?!"]}, {"cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [], "source": ["subs.query('IsProbe').shape"]}, {"cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [], "source": ["subs.query('IsProbe and (PublicScoreFullPrecision<3.95)').shape"]}, {"cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [], "source": ["90 / 1053"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Look at IDs of submissions made by the 56 McProbeface teams - submission ID appears to be a good proxy for time."]}, {"cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [], "source": ["subs.reset_index().groupby('PureProbeTeam').Id.agg(['min','max', np.ptp])"]}, {"cell_type": "markdown", "metadata": {}, "source": ["A really narrow band of time towards end of competition:"]}, {"cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [], "source": ["subs.PureProbeTeam.astype(int).plot()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Focus on just that time window."]}, {"cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [], "source": ["window = subs.reset_index().query('Id>=10060463 and Id<=10108434')\n", "window[['IsProbe', 'PureProbeTeam']] *= 1\n", "window.shape"]}, {"cell_type": "markdown", "metadata": {}, "source": ["All subs in that time with probe-only teams in red; probes are in short, concentrated streaks, suggesting one person working through all these accounts."]}, {"cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [], "source": ["window.plot.scatter('Id',\n", " 'IsProbe',\n", " c=np.where(window.PureProbeTeam, 'red', 'blue'),\n", " figsize=(15, 2),\n", " title='Subs Over Time - McProbeface teams in red',\n", " alpha=0.4)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Only probe subs - McProbeface teams in red. Nothing leaps out here - need a better plot to see if the semi-probing teams are connected in any way with the McProbeface teams."]}, {"cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [], "source": ["window = window.query('IsProbe>0')\n", "window.plot.scatter('Id',\n", " 'PureProbeTeam',\n", " c=np.where(window.PureProbeTeam, 'red', 'blue'),\n", " figsize=(15, 2),\n", " title='Probe Subs Over Time - McProbeface teams in red',\n", " alpha=0.4)"]}, {"cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [], "source": ["comp.DeadlineDate"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Here's a 2D grid - a new batch of McProbefaces was minted on 02/16/2019, did 5 subs each, along with the old ones, then the whole thing stops!"]}, {"cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [], "source": ["grid = subs.query('PureProbeTeam>0').groupby(['TeamId', 'SubmissionDate']).size().unstack().fillna('')\n", "grid.insert(0, 'TeamName', grid.index.map(teams.TeamName))\n", "grid"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Find the *range* of submission IDs each day for these users. Submission ID is GLOBAL - shared across ALL competitions. All the ranges are really small, fast bursts of submissions."]}, {"cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [], "source": ["gb = subs.query('PureProbeTeam>0').reset_index().groupby(['TeamId', 'SubmissionDate'])\n", "grid = (gb.Id.max() - gb.Id.min()).unstack().fillna('')\n", "grid.insert(0, 'TeamName', grid.index.map(teams.TeamName))\n", "grid"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Random sample of 20 users who made 5 subs for the last day shown above. Leaving TeamName out to keep them anon-ish. Gap in sub IDs usually bigger."]}, {"cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [], "source": ["gb = subs.query('PureProbeTeam==0 and SubmissionDate==\"02/16/2019\"').reset_index().groupby('TeamId')\n", "grid = gb.Id.agg(['count', 'min', 'max']).query('count==5').sample(n=20, random_state=42)\n", "grid['IdRange'] = grid['max'] - grid['min']\n", "grid"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Shell Account Stats\n", "\n", "What if this is all some kind of odd coincidence? What else can we tell about the accounts? Let's look at how many other tables the UserIds appear in. This summarizes all their activity on the site. Do these shell accounts vote for kernels? Vote for forum posts? Write kernels? *Follow* others?"]}, {"cell_type": "code", "execution_count": 64, "metadata": {"_kg_hide-input": true}, "outputs": [], "source": ["# Columns in other tables that represent user activity on the site\n", "USER_STATS = [\n", " ('Datasets', 'CreatorUserId'),\n", " ('DatasetVersions', 'CreatorUserId'),\n", " ('DatasetVotes', 'UserId'),\n", " ('ForumMessages', 'PostUserId'),\n", " ('ForumMessageVotes', 'FromUserId'),\n", " ('ForumMessageVotes', 'ToUserId'),\n", " ('Kernels', 'AuthorUserId'),\n", " ('KernelVersions', 'AuthorUserId'),\n", " ('KernelVotes', 'UserId'),\n", " ('UserFollowers', 'UserId'),\n", " ('UserFollowers', 'FollowingUserId')\n", "]\n", "\n", "def add_stats(dst, csv, col):\n", " df = read_csv_filtered(MK / (csv + '.csv'), col, users.index, usecols=[col])\n", " users[dst] = users.index.map(df[col].value_counts())\n", " print(dst, csv, col, users[dst].count())\n", " \n", "for i, (csv, col) in enumerate(USER_STATS, 1):\n", " add_stats(f'C{i}', csv, col)\n", "\n", "users['ActivitySums'] = (users[users.columns[users.columns.str.startswith('C')]]>0).sum(1)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Nope. Big fat ZERO. These are clearly not real users. They don't vote for anything, write any kernels, post any comments, create datasets, follow users and they are not followed by anyone either."]}, {"cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [], "source": ["users.loc[probe_members.UserId].describe().T"]}, {"cell_type": "markdown", "metadata": {}, "source": ["A random sample of the same number of users from the whole pool that took part in ELO shows it is hard to find a group with no activity..."]}, {"cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [], "source": ["users.sample(n=56, random_state=42).describe().T"]}, {"cell_type": "markdown", "metadata": {}, "source": ["For comparison - this is how activity counts look for users in the top 10 teams - heavy *Kagglers*...\n", "\n", "This is just to show - Kaggle would really miss these users. The shell accounts with 0 activity? Would not be missed at all ;)"]}, {"cell_type": "code", "execution_count": 67, "metadata": {}, "outputs": [], "source": ["top10 = teams.query('PrivateLeaderboardRank<=10')\n", "df = users.loc[tm[tm.TeamId.isin(top10.index)].UserId]\n", "df.fillna('').style.bar(subset=['ActivitySums'], vmin=0)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n", "# Conclusions\n", "\n", "Probing the leaderboard is not against the rules - enforcing it is unworkable.\n", "\n", "However, creating multiple accounts to submit from is most definitely against Kaggle's rules and is the most cited reason for removal from competitions.\n", "\n", "We only see half the story here - the real issue is who used these accounts and to what benefit? If the perpetrators were caught and these probe accounts are part of the evidence against them - why are they still there? It suggests Kaggle missed this.\n", "\n", "In other words: Kaggle missed this cluster of teams, or noticed it and decided to leave it. Either is bad.\n", "\n", "\n", "## Who?\n", "\n", "It is kind of fascinating that someone went to these lengths to get an unfair advantage. The hit rate of detected outliers suggests they knew enough to guide their probe submissions. But the fact it stopped abruptly 10 days before the deadline suggests to me an amateur/beginner who gave up.\n", "\n", "Who has the patience and spare time combination to create 56 shell accounts? It could be automated but I thought at least a phone number is required?\n", "\n", "\n", "\n", "*Go big or go home :)*\n", "\n", "[Rob Mulla][1] made a [similar notebook][2] and got a blizzard of downvotes in return. It *could* be the same user as featured there. Fake accounts are clearly already linked to him. It's not the same pattern though. Could that guy resist not adding new accounts as followers?\n", "\n", "Whoever it is, I am risking 56 downvotes on all my past/future discussion posts if the perp has nothing better to do :P\n", "\n", "## Solutions\n", "\n", "Seeing probing at this scale hints that it is not solved yet... Here are some ideas.\n", "\n", "1. As mentioned by [Triskelion][6] in this [Mercedes thread][4] : [The Ladder: A Reliable Leaderboard for Machine Learning Competitions][5]. (https://arxiv.org/abs/1502.04585) \n", "\n", "2. Even fewer decimals\n", "\n", "3. Alternative number bases. E.g. octal: 1 digit gives 1/8ths. 2 digits give 1/64ths. etc. If any crowd can learn non-decimal fractions it should be the Kaggle crowd :)\n", "\n", "Or perhaps none of these if Kaggle plan to move entirely to code competitions with hidden test sets.\n", "\n", "[1]: https://www.kaggle.com/robikscube\n", "[2]: https://www.kaggle.com/robikscube/identifying-potential-fake-accounts-on-kaggle\n", "[3]: https://www.kaggle.com/jtrotman/meta-kaggle-count-user-activities\n", "[4]: https://www.kaggle.com/c/mercedes-benz-greener-manufacturing/discussion/35566#200252\n", "[5]: https://arxiv.org/abs/1502.04585\n", "[6]: https://www.kaggle.com/triskelion\n"]}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1"}}, "nbformat": 4, "nbformat_minor": 2}