{ "id": 2670, "title": "indiv2012.csv contains a \\A0", "url": "/competitions/cir-prospect/discussion/2670", "author_name": "", "post_date": "2012-09-15T15:56:56.947Z", "votes": null, "comment_count": 8, "views": 4593, "content": "

While reading this in with python 3.2, either csv.reader or open, I got a \r\ncannot decode utf-8. No big deal, i've seen those before. But because the file is rather large I had to split this in linux using split -l 100000 indiv2012.csv indiv2012_split\r\nto figure out where the problem was. 

\r\n

nicarid 1386221, there appears this control character(?) \\xa0 inbetween  "SANSBERRY DICKMAN FREEMON & " in indiv2012.csv and a few more times in the rest of the file.

\r\n

My question is did anyone else have problems with this, reading this as ISO-8859-15 seems ok.

\r\n


\r\n

", "messages": [ { "id": "14351", "postDate": "09/15/2012 15:56:56", "content": "

While reading this in with python 3.2, either csv.reader or open, I got a \r\ncannot decode utf-8. No big deal, i've seen those before. But because the file is rather large I had to split this in linux using split -l 100000 indiv2012.csv indiv2012_split\r\nto figure out where the problem was. 

\r\n

nicarid 1386221, there appears this control character(?) \\xa0 inbetween  "SANSBERRY DICKMAN FREEMON & " in indiv2012.csv and a few more times in the rest of the file.

\r\n

My question is did anyone else have problems with this, reading this as ISO-8859-15 seems ok.

\r\n


\r\n

", "rawMarkdown": "", "votes": null }, { "id": "14599", "postDate": "09/19/2012 20:07:53", "content": "
 
\r\n
[code]
\r\n
import os
import csv
import json

filename = 'indiv2012.csv'

dict_states = {}

""" STEP 1, find sum donations per state """

if False:
    def open_csv_test(filename):
        csvfile = open(filename, 'r', encoding='ISO-8859-15', newline='')
        ofile = csv.reader(csvfile, delimiter=',')

        ofile.__next__()
        for row in ofile:
            try:
                current_val = dict_states.get(row[11], 0.0)
                dict_states[row[11]] = float(row[17]) + current_val
            except:
                print(row)

        with open('sum_per_state.json', 'w') as wfile:
            wfile.write(json.dumps(dict_states, sort_keys=True, indent=4))

    open_csv_test(filename)
\r\n
[/code]
\r\n
i'm wondering if this is a reasonable way to swathe through the content.
", "rawMarkdown": "", "votes": null }, { "id": "14752", "postDate": "09/24/2012 17:28:26", "content": "

A better formatted file would definitely be useful; I attempted to work with this in mysql but after receiving a multitude of data truncation errors I moved to python. Continue receiving extra columns where unintended (I'm assuming due to the control character\r\n you mentioned above) The majority of my errors were 27 columns instead of 26, with the occasional 28.

\r\n

May I ask how you rectified the extra control character?

", "rawMarkdown": "", "votes": null }, { "id": "14760", "postDate": "09/24/2012 20:55:46", "content": "

Hi guys -- I'll take a closer look at the file formatting and see if we can't get you something cleaner. More soon.

\r\n

Best,
\r\nChase

", "rawMarkdown": "", "votes": null }, { "id": "14868", "postDate": "09/27/2012 06:14:46", "content": "

Hi,

\r\n

Is it possible for you to upload a cleaned file or should we go ahead and try to clean it up ourselves?

\r\n

 

\r\n

Thanks,

", "rawMarkdown": "", "votes": null }, { "id": "14874", "postDate": "09/27/2012 15:30:03", "content": "

Best I can do for now is this MySQL dump that imported cleanly when I tested it about 10 minutes ago. I'll check with the actual data providers about the error in the CSVs.

", "rawMarkdown": "", "votes": null }, { "id": "14928", "postDate": "09/28/2012 17:53:40", "content": "

The data attached in the above comment is new ? and clean ?

", "rawMarkdown": "", "votes": null }, { "id": "14939", "postDate": "09/28/2012 21:06:30", "content": "

It's a SQL dump of basically the same data as the CSV. I sure think it's clean, because I've imported it myself with no troubles.

\r\n

The data is also slightly more up-to-date (to the tune of a few weeks), but that doesn't really matter for the purposes of this competition.

", "rawMarkdown": "", "votes": null }, { "id": "14994", "postDate": "10/01/2012 14:00:28", "content": "

My apologies for lack of response here, lack of spare time has been monstrous. The script posted is the last thing i did (and some d3js mapping).

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 14599, "author_name": "dealga", "author_url": "", "post_date": "09/19/2012 20:07:53", "content": "
 
\r\n
[code]
\r\n
import os
import csv
import json

filename = 'indiv2012.csv'

dict_states = {}

""" STEP 1, find sum donations per state """

if False:
    def open_csv_test(filename):
        csvfile = open(filename, 'r', encoding='ISO-8859-15', newline='')
        ofile = csv.reader(csvfile, delimiter=',')

        ofile.__next__()
        for row in ofile:
            try:
                current_val = dict_states.get(row[11], 0.0)
                dict_states[row[11]] = float(row[17]) + current_val
            except:
                print(row)

        with open('sum_per_state.json', 'w') as wfile:
            wfile.write(json.dumps(dict_states, sort_keys=True, indent=4))

    open_csv_test(filename)
\r\n
[/code]
\r\n
i'm wondering if this is a reasonable way to swathe through the content.
", "votes": null, "replies": [] }, { "id": 14752, "author_name": "joofeloof", "author_url": "", "post_date": "09/24/2012 17:28:26", "content": "

A better formatted file would definitely be useful; I attempted to work with this in mysql but after receiving a multitude of data truncation errors I moved to python. Continue receiving extra columns where unintended (I'm assuming due to the control character\r\n you mentioned above) The majority of my errors were 27 columns instead of 26, with the occasional 28.

\r\n

May I ask how you rectified the extra control character?

", "votes": null, "replies": [] }, { "id": 14760, "author_name": "cjdd3b", "author_url": "", "post_date": "09/24/2012 20:55:46", "content": "

Hi guys -- I'll take a closer look at the file formatting and see if we can't get you something cleaner. More soon.

\r\n

Best,
\r\nChase

", "votes": null, "replies": [] }, { "id": 14868, "author_name": "hutokshisethna", "author_url": "", "post_date": "09/27/2012 06:14:46", "content": "

Hi,

\r\n

Is it possible for you to upload a cleaned file or should we go ahead and try to clean it up ourselves?

\r\n

 

\r\n

Thanks,

", "votes": null, "replies": [] }, { "id": 14874, "author_name": "cjdd3b", "author_url": "", "post_date": "09/27/2012 15:30:03", "content": "

Best I can do for now is this MySQL dump that imported cleanly when I tested it about 10 minutes ago. I'll check with the actual data providers about the error in the CSVs.

", "votes": null, "replies": [] }, { "id": 14928, "author_name": "theory", "author_url": "", "post_date": "09/28/2012 17:53:40", "content": "

The data attached in the above comment is new ? and clean ?

", "votes": null, "replies": [] }, { "id": 14939, "author_name": "cjdd3b", "author_url": "", "post_date": "09/28/2012 21:06:30", "content": "

It's a SQL dump of basically the same data as the CSV. I sure think it's clean, because I've imported it myself with no troubles.

\r\n

The data is also slightly more up-to-date (to the tune of a few weeks), but that doesn't really matter for the purposes of this competition.

", "votes": null, "replies": [] }, { "id": 14994, "author_name": "dealga", "author_url": "", "post_date": "10/01/2012 14:00:28", "content": "

My apologies for lack of response here, lack of spare time has been monstrous. The script posted is the last thing i did (and some d3js mapping).

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "14351": "", "14599": "", "14752": "", "14760": "", "14868": "", "14874": "", "14928": "", "14939": "", "14994": "" }, "source": "meta" }