A spamming attack will be evaluated based on how much it can withstand our spam detectors to promote the revenue of the designated target products (yes, we know what products are manipulated, but the evaluation codes will not access that knowledge). We call this metric Practical Effect (PE).

More specifically, the pipeline looks like this:
1. Download the files parameters.csv, review_graph.txt, and target_spammer.pickle to create attacks using your algorithms.
2. An attack is in the form of reviews with time and ratings. There is no review content involved.
3. We add the reviews in each of your attacks to the existing review data and run our detectors.
4. We compute how much the remaining spams can promote the target products.
5. The final performance is averaged over the multiple attacks.

See the Notebooks for a full pipeline from data loading to attack generation and evaluation.

When your algorithm finishes the attacks specified in the file parameters.csv, you should generated 48 'submission_k.txt' files, where k=0,...,47. The format of the submission files are specified in the Notebooks. Zip and submit the submission files into a single file name <team_name>.zip and submit the zip file to the following Google Form:
[https://forms.gle/tF91i5FDGTaQmVEU9](https://forms.gle/tF91i5FDGTaQmVEU9)

Each team will be identified by a unique email address. You need to input the email address along with each submission. All submissions from a team will be linked to that email address. We will contact the winners using these addresses.

We will update the evaluation and the Leaderboard on the workshop website daily:
[https://truthdiscoverykdd2020.github.io](https://truthdiscoverykdd2020.github.io)