{
  "id": 1032,
  "title": "Importing the data into a database",
  "url": "/competitions/AlgorithmicTradingChallenge/discussion/1032",
  "author_name": "",
  "post_date": "2011-11-11T18:34:09.657Z",
  "votes": 6,
  "comment_count": 1,
  "views": 1948,
  "content": "<p>Since this dataset is quite large, I wanted to offer the database schema I used in case it helps other. I tried to use the smallest data type possible per column to be more efficient.</p>\r\n<p>I used SQL Server 2008, but see other related threads in the <a href=\"https://www.kaggle.com/c/wikichallenge/forums/t/668/importing-the-data-into-sql-server\">\r\nWikiChallenge</a> and <a href=\"https://www.kaggle.com/c/ClaimPredictionChallenge/forums/t/711/importing-to-sql-server-and-aggregate-statistics\">\r\nClaim Prediction Challenge</a> competitions for tips on how to do similar things in other databases.</p>\r\n<p>Here are my tables (note the use of smallmoney and time(3)):</p>\r\n<pre>CREATE TABLE [training](\r\n\t[row_id] [int] NOT NULL,\r\n\t[security_id] [tinyint] NOT NULL,\r\n\t[p_tcount] [int] NOT NULL,\r\n\t[p_value] [bigint] NOT NULL,\r\n\t[trade_vwap] [smallmoney] NOT NULL,\r\n\t[trade_volume] [int] NOT NULL,\r\n\t[initiator] [char](1) NOT NULL,\r\n\t[transtype1] [char](1) NOT NULL,\r\n\t[time1] [time](3) NOT NULL,\r\n\t[bid1] [smallmoney] NOT NULL,\r\n\t[ask1] [smallmoney] NOT NULL,\r\n\t[transtype2] [char](1) NOT NULL,\r\n\t[time2] [time](3) NOT NULL,\r\n\t[bid2] [smallmoney] NOT NULL,\r\n\t[ask2] [smallmoney] NOT NULL,\r\n\t[transtype3] [char](1) NOT NULL,\r\n\t[time3] [time](3) NOT NULL,\r\n\t[bid3] [smallmoney] NOT NULL,\r\n\t[ask3] [smallmoney] NOT NULL,\r\n\t[transtype4] [char](1) NOT NULL,\r\n\t[time4] [time](3) NOT NULL,\r\n\t[bid4] [smallmoney] NOT NULL,\r\n\t[ask4] [smallmoney] NOT NULL,\r\n\t[transtype5] [char](1) NOT NULL,\r\n\t[time5] [time](3) NOT NULL,\r\n\t[bid5] [smallmoney] NOT NULL,\r\n\t[ask5] [smallmoney] NOT NULL,\r\n\t[transtype6] [char](1) NOT NULL,\r\n\t[time6] [time](3) NOT NULL,\r\n\t[bid6] [smallmoney] NOT NULL,\r\n\t[ask6] [smallmoney] NOT NULL,\r\n\t[transtype7] [char](1) NOT NULL,\r\n\t[time7] [time](3) NOT NULL,\r\n\t[bid7] [smallmoney] NOT NULL,\r\n\t[ask7] [smallmoney] NOT NULL,\r\n\t[transtype8] [char](1) NOT NULL,\r\n\t[time8] [time](3) NOT NULL,\r\n\t[bid8] [smallmoney] NOT NULL,\r\n\t[ask8] [smallmoney] NOT NULL,\r\n\t[transtype9] [char](1) NOT NULL,\r\n\t[time9] [time](3) NOT NULL,\r\n\t[bid9] [smallmoney] NOT NULL,\r\n\t[ask9] [smallmoney] NOT NULL,\r\n\t[transtype10] [char](1) NOT NULL,\r\n\t[time10] [time](3) NOT NULL,\r\n\t[bid10] [smallmoney] NOT NULL,\r\n\t[ask10] [smallmoney] NOT NULL,\r\n\t[transtype11] [char](1) NOT NULL,\r\n\t[time11] [time](3) NOT NULL,\r\n\t[bid11] [smallmoney] NOT NULL,\r\n\t[ask11] [smallmoney] NOT NULL,\r\n\t[transtype12] [char](1) NOT NULL,\r\n\t[time12] [time](3) NOT NULL,\r\n\t[bid12] [smallmoney] NOT NULL,\r\n\t[ask12] [smallmoney] NOT NULL,\r\n\t[transtype13] [char](1) NOT NULL,\r\n\t[time13] [time](3) NOT NULL,\r\n\t[bid13] [smallmoney] NOT NULL,\r\n\t[ask13] [smallmoney] NOT NULL,\r\n\t[transtype14] [char](1) NOT NULL,\r\n\t[time14] [time](3) NOT NULL,\r\n\t[bid14] [smallmoney] NOT NULL,\r\n\t[ask14] [smallmoney] NOT NULL,\r\n\t[transtype15] [char](1) NOT NULL,\r\n\t[time15] [time](3) NOT NULL,\r\n\t[bid15] [smallmoney] NOT NULL,\r\n\t[ask15] [smallmoney] NOT NULL,\r\n\t[transtype16] [char](1) NOT NULL,\r\n\t[time16] [time](3) NOT NULL,\r\n\t[bid16] [smallmoney] NOT NULL,\r\n\t[ask16] [smallmoney] NOT NULL,\r\n\t[transtype17] [char](1) NOT NULL,\r\n\t[time17] [time](3) NOT NULL,\r\n\t[bid17] [smallmoney] NOT NULL,\r\n\t[ask17] [smallmoney] NOT NULL,\r\n\t[transtype18] [char](1) NOT NULL,\r\n\t[time18] [time](3) NOT NULL,\r\n\t[bid18] [smallmoney] NOT NULL,\r\n\t[ask18] [smallmoney] NOT NULL,\r\n\t[transtype19] [char](1) NOT NULL,\r\n\t[time19] [time](3) NOT NULL,\r\n\t[bid19] [smallmoney] NOT NULL,\r\n\t[ask19] [smallmoney] NOT NULL,\r\n\t[transtype20] [char](1) NOT NULL,\r\n\t[time20] [time](3) NOT NULL,\r\n\t[bid20] [smallmoney] NOT NULL,\r\n\t[ask20] [smallmoney] NOT NULL,\r\n\t[transtype21] [char](1) NOT NULL,\r\n\t[time21] [time](3) NOT NULL,\r\n\t[bid21] [smallmoney] NOT NULL,\r\n\t[ask21] [smallmoney] NOT NULL,\r\n\t[transtype22] [char](1) NOT NULL,\r\n\t[time22] [time](3) NOT NULL,\r\n\t[bid22] [smallmoney] NOT NULL,\r\n\t[ask22] [smallmoney] NOT NULL,\r\n\t[transtype23] [char](1) NOT NULL,\r\n\t[time23] [time](3) NOT NULL,\r\n\t[bid23] [smallmoney] NOT NULL,\r\n\t[ask23] [smallmoney] NOT NULL,\r\n\t[transtype24] [char](1) NOT NULL,\r\n\t[time24] [time](3) NOT NULL,\r\n\t[bid24] [smallmoney] NOT NULL,\r\n\t[ask24] [smallmoney] NOT NULL,\r\n\t[transtype25] [char](1) NOT NULL,\r\n\t[time25] [time](3) NOT NULL,\r\n\t[bid25] [smallmoney] NOT NULL,\r\n\t[ask25] [smallmoney] NOT NULL,\r\n\t[transtype26] [char](1) NOT NULL,\r\n\t[time26] [time](3) NOT NULL,\r\n\t[bid26] [smallmoney] NOT NULL,\r\n\t[ask26] [smallmoney] NOT NULL,\r\n\t[transtype27] [char](1) NOT NULL,\r\n\t[time27] [time](3) NOT NULL,\r\n\t[bid27] [smallmoney] NOT NULL,\r\n\t[ask27] [smallmoney] NOT NULL,\r\n\t[transtype28] [char](1) NOT NULL,\r\n\t[time28] [time](3) NOT NULL,\r\n\t[bid28] [smallmoney] NOT NULL,\r\n\t[ask28] [smallmoney] NOT NULL,\r\n\t[transtype29] [char](1) NOT NULL,\r\n\t[time29] [time](3) NOT NULL,\r\n\t[bid29] [smallmoney] NOT NULL,\r\n\t[ask29] [smallmoney] NOT NULL,\r\n\t[transtype30] [char](1) NOT NULL,\r\n\t[time30] [time](3) NOT NULL,\r\n\t[bid30] [smallmoney] NOT NULL,\r\n\t[ask30] [smallmoney] NOT NULL,\r\n\t[transtype31] [char](1) NOT NULL,\r\n\t[time31] [time](3) NOT NULL,\r\n\t[bid31] [smallmoney] NOT NULL,\r\n\t[ask31] [smallmoney] NOT NULL,\r\n\t[transtype32] [char](1) NOT NULL,\r\n\t[time32] [time](3) NOT NULL,\r\n\t[bid32] [smallmoney] NOT NULL,\r\n\t[ask32] [smallmoney] NOT NULL,\r\n\t[transtype33] [char](1) NOT NULL,\r\n\t[time33] [time](3) NOT NULL,\r\n\t[bid33] [smallmoney] NOT NULL,\r\n\t[ask33] [smallmoney] NOT NULL,\r\n\t[transtype34] [char](1) NOT NULL,\r\n\t[time34] [time](3) NOT NULL,\r\n\t[bid34] [smallmoney] NOT NULL,\r\n\t[ask34] [smallmoney] NOT NULL,\r\n\t[transtype35] [char](1) NOT NULL,\r\n\t[time35] [time](3) NOT NULL,\r\n\t[bid35] [smallmoney] NOT NULL,\r\n\t[ask35] [smallmoney] NOT NULL,\r\n\t[transtype36] [char](1) NOT NULL,\r\n\t[time36] [time](3) NOT NULL,\r\n\t[bid36] [smallmoney] NOT NULL,\r\n\t[ask36] [smallmoney] NOT NULL,\r\n\t[transtype37] [char](1) NOT NULL,\r\n\t[time37] [time](3) NOT NULL,\r\n\t[bid37] [smallmoney] NOT NULL,\r\n\t[ask37] [smallmoney] NOT NULL,\r\n\t[transtype38] [char](1) NOT NULL,\r\n\t[time38] [time](3) NOT NULL,\r\n\t[bid38] [smallmoney] NOT NULL,\r\n\t[ask38] [smallmoney] NOT NULL,\r\n\t[transtype39] [char](1) NOT NULL,\r\n\t[time39] [time](3) NOT NULL,\r\n\t[bid39] [smallmoney] NOT NULL,\r\n\t[ask39] [smallmoney] NOT NULL,\r\n\t[transtype40] [char](1) NOT NULL,\r\n\t[time40] [time](3) NOT NULL,\r\n\t[bid40] [smallmoney] NOT NULL,\r\n\t[ask40] [smallmoney] NOT NULL,\r\n\t[transtype41] [char](1) NOT NULL,\r\n\t[time41] [time](3) NOT NULL,\r\n\t[bid41] [smallmoney] NOT NULL,\r\n\t[ask41] [smallmoney] NOT NULL,\r\n\t[transtype42] [char](1) NOT NULL,\r\n\t[time42] [time](3) NOT NULL,\r\n\t[bid42] [smallmoney] NOT NULL,\r\n\t[ask42] [smallmoney] NOT NULL,\r\n\t[transtype43] [char](1) NOT NULL,\r\n\t[time43] [time](3) NOT NULL,\r\n\t[bid43] [smallmoney] NOT NULL,\r\n\t[ask43] [smallmoney] NOT NULL,\r\n\t[transtype44] [char](1) NOT NULL,\r\n\t[time44] [time](3) NOT NULL,\r\n\t[bid44] [smallmoney] NOT NULL,\r\n\t[ask44] [smallmoney] NOT NULL,\r\n\t[transtype45] [char](1) NOT NULL,\r\n\t[time45] [time](3) NOT NULL,\r\n\t[bid45] [smallmoney] NOT NULL,\r\n\t[ask45] [smallmoney] NOT NULL,\r\n\t[transtype46] [char](1) NOT NULL,\r\n\t[time46] [time](3) NOT NULL,\r\n\t[bid46] [smallmoney] NOT NULL,\r\n\t[ask46] [smallmoney] NOT NULL,\r\n\t[transtype47] [char](1) NOT NULL,\r\n\t[time47] [time](3) NOT NULL,\r\n\t[bid47] [smallmoney] NOT NULL,\r\n\t[ask47] [smallmoney] NOT NULL,\r\n\t[transtype48] [char](1) NOT NULL,\r\n\t[time48] [time](3) NOT NULL,\r\n\t[bid48] [smallmoney] NOT NULL,\r\n\t[ask48] [smallmoney] NOT NULL,\r\n\t[transtype49] [char](1) NOT NULL,\r\n\t[time49] [time](3) NOT NULL,\r\n\t[bid49] [smallmoney] NOT NULL,\r\n\t[ask49] [smallmoney] NOT NULL,\r\n\t[transtype50] [char](1) NOT NULL,\r\n\t[time50] [time](3) NOT NULL,\r\n\t[bid50] [smallmoney] NOT NULL,\r\n\t[ask50] [smallmoney] NOT NULL,\r\n\t[bid51] [smallmoney] NOT NULL,\r\n\t[ask51] [smallmoney] NOT NULL,\r\n\t[bid52] [smallmoney] NOT NULL,\r\n\t[ask52] [smallmoney] NOT NULL,\r\n\t[bid53] [smallmoney] NOT NULL,\r\n\t[ask53] [smallmoney] NOT NULL,\r\n\t[bid54] [smallmoney] NOT NULL,\r\n\t[ask54] [smallmoney] NOT NULL,\r\n\t[bid55] [smallmoney] NOT NULL,\r\n\t[ask55] [smallmoney] NOT NULL,\r\n\t[bid56] [smallmoney] NOT NULL,\r\n\t[ask56] [smallmoney] NOT NULL,\r\n\t[bid57] [smallmoney] NOT NULL,\r\n\t[ask57] [smallmoney] NOT NULL,\r\n\t[bid58] [smallmoney] NOT NULL,\r\n\t[ask58] [smallmoney] NOT NULL,\r\n\t[bid59] [smallmoney] NOT NULL,\r\n\t[ask59] [smallmoney] NOT NULL,\r\n\t[bid60] [smallmoney] NOT NULL,\r\n\t[ask60] [smallmoney] NOT NULL,\r\n\t[bid61] [smallmoney] NOT NULL,\r\n\t[ask61] [smallmoney] NOT NULL,\r\n\t[bid62] [smallmoney] NOT NULL,\r\n\t[ask62] [smallmoney] NOT NULL,\r\n\t[bid63] [smallmoney] NOT NULL,\r\n\t[ask63] [smallmoney] NOT NULL,\r\n\t[bid64] [smallmoney] NOT NULL,\r\n\t[ask64] [smallmoney] NOT NULL,\r\n\t[bid65] [smallmoney] NOT NULL,\r\n\t[ask65] [smallmoney] NOT NULL,\r\n\t[bid66] [smallmoney] NOT NULL,\r\n\t[ask66] [smallmoney] NOT NULL,\r\n\t[bid67] [smallmoney] NOT NULL,\r\n\t[ask67] [smallmoney] NOT NULL,\r\n\t[bid68] [smallmoney] NOT NULL,\r\n\t[ask68] [smallmoney] NOT NULL,\r\n\t[bid69] [smallmoney] NOT NULL,\r\n\t[ask69] [smallmoney] NOT NULL,\r\n\t[bid70] [smallmoney] NOT NULL,\r\n\t[ask70] [smallmoney] NOT NULL,\r\n\t[bid71] [smallmoney] NOT NULL,\r\n\t[ask71] [smallmoney] NOT NULL,\r\n\t[bid72] [smallmoney] NOT NULL,\r\n\t[ask72] [smallmoney] NOT NULL,\r\n\t[bid73] [smallmoney] NOT NULL,\r\n\t[ask73] [smallmoney] NOT NULL,\r\n\t[bid74] [smallmoney] NOT NULL,\r\n\t[ask74] [smallmoney] NOT NULL,\r\n\t[bid75] [smallmoney] NOT NULL,\r\n\t[ask75] [smallmoney] NOT NULL,\r\n\t[bid76] [smallmoney] NOT NULL,\r\n\t[ask76] [smallmoney] NOT NULL,\r\n\t[bid77] [smallmoney] NOT NULL,\r\n\t[ask77] [smallmoney] NOT NULL,\r\n\t[bid78] [smallmoney] NOT NULL,\r\n\t[ask78] [smallmoney] NOT NULL,\r\n\t[bid79] [smallmoney] NOT NULL,\r\n\t[ask79] [smallmoney] NOT NULL,\r\n\t[bid80] [smallmoney] NOT NULL,\r\n\t[ask80] [smallmoney] NOT NULL,\r\n\t[bid81] [smallmoney] NOT NULL,\r\n\t[ask81] [smallmoney] NOT NULL,\r\n\t[bid82] [smallmoney] NOT NULL,\r\n\t[ask82] [smallmoney] NOT NULL,\r\n\t[bid83] [smallmoney] NOT NULL,\r\n\t[ask83] [smallmoney] NOT NULL,\r\n\t[bid84] [smallmoney] NOT NULL,\r\n\t[ask84] [smallmoney] NOT NULL,\r\n\t[bid85] [smallmoney] NOT NULL,\r\n\t[ask85] [smallmoney] NOT NULL,\r\n\t[bid86] [smallmoney] NOT NULL,\r\n\t[ask86] [smallmoney] NOT NULL,\r\n\t[bid87] [smallmoney] NOT NULL,\r\n\t[ask87] [smallmoney] NOT NULL,\r\n\t[bid88] [smallmoney] NOT NULL,\r\n\t[ask88] [smallmoney] NOT NULL,\r\n\t[bid89] [smallmoney] NOT NULL,\r\n\t[ask89] [smallmoney] NOT NULL,\r\n\t[bid90] [smallmoney] NOT NULL,\r\n\t[ask90] [smallmoney] NOT NULL,\r\n\t[bid91] [smallmoney] NOT NULL,\r\n\t[ask91] [smallmoney] NOT NULL,\r\n\t[bid92] [smallmoney] NOT NULL,\r\n\t[ask92] [smallmoney] NOT NULL,\r\n\t[bid93] [smallmoney] NOT NULL,\r\n\t[ask93] [smallmoney] NOT NULL,\r\n\t[bid94] [smallmoney] NOT NULL,\r\n\t[ask94] [smallmoney] NOT NULL,\r\n\t[bid95] [smallmoney] NOT NULL,\r\n\t[ask95] [smallmoney] NOT NULL,\r\n\t[bid96] [smallmoney] NOT NULL,\r\n\t[ask96] [smallmoney] NOT NULL,\r\n\t[bid97] [smallmoney] NOT NULL,\r\n\t[ask97] [smallmoney] NOT NULL,\r\n\t[bid98] [smallmoney] NOT NULL,\r\n\t[ask98] [smallmoney] NOT NULL,\r\n\t[bid99] [smallmoney] NOT NULL,\r\n\t[ask99] [smallmoney] NOT NULL,\r\n\t[bid100] [smallmoney] NOT NULL,\r\n\t[ask100] [smallmoney] NOT NULL\r\n);\r\n\r\nCREATE TABLE [testing](\r\n\t[row_id] [int] NOT NULL,\r\n\t[security_id] [tinyint] NOT NULL,\r\n\t[p_tcount] [int] NOT NULL,\r\n\t[p_value] [bigint] NOT NULL,\r\n\t[trade_vwap] [smallmoney] NOT NULL,\r\n\t[trade_volume] [int] NOT NULL,\r\n\t[initiator] [char](1) NOT NULL,\r\n\t[transtype1] [char](1) NOT NULL,\r\n\t[time1] [time](3) NOT NULL,\r\n\t[bid1] [smallmoney] NOT NULL,\r\n\t[ask1] [smallmoney] NOT NULL,\r\n\t[transtype2] [char](1) NOT NULL,\r\n\t[time2] [time](3) NOT NULL,\r\n\t[bid2] [smallmoney] NOT NULL,\r\n\t[ask2] [smallmoney] NOT NULL,\r\n\t[transtype3] [char](1) NOT NULL,\r\n\t[time3] [time](3) NOT NULL,\r\n\t[bid3] [smallmoney] NOT NULL,\r\n\t[ask3] [smallmoney] NOT NULL,\r\n\t[transtype4] [char](1) NOT NULL,\r\n\t[time4] [time](3) NOT NULL,\r\n\t[bid4] [smallmoney] NOT NULL,\r\n\t[ask4] [smallmoney] NOT NULL,\r\n\t[transtype5] [char](1) NOT NULL,\r\n\t[time5] [time](3) NOT NULL,\r\n\t[bid5] [smallmoney] NOT NULL,\r\n\t[ask5] [smallmoney] NOT NULL,\r\n\t[transtype6] [char](1) NOT NULL,\r\n\t[time6] [time](3) NOT NULL,\r\n\t[bid6] [smallmoney] NOT NULL,\r\n\t[ask6] [smallmoney] NOT NULL,\r\n\t[transtype7] [char](1) NOT NULL,\r\n\t[time7] [time](3) NOT NULL,\r\n\t[bid7] [smallmoney] NOT NULL,\r\n\t[ask7] [smallmoney] NOT NULL,\r\n\t[transtype8] [char](1) NOT NULL,\r\n\t[time8] [time](3) NOT NULL,\r\n\t[bid8] [smallmoney] NOT NULL,\r\n\t[ask8] [smallmoney] NOT NULL,\r\n\t[transtype9] [char](1) NOT NULL,\r\n\t[time9] [time](3) NOT NULL,\r\n\t[bid9] [smallmoney] NOT NULL,\r\n\t[ask9] [smallmoney] NOT NULL,\r\n\t[transtype10] [char](1) NOT NULL,\r\n\t[time10] [time](3) NOT NULL,\r\n\t[bid10] [smallmoney] NOT NULL,\r\n\t[ask10] [smallmoney] NOT NULL,\r\n\t[transtype11] [char](1) NOT NULL,\r\n\t[time11] [time](3) NOT NULL,\r\n\t[bid11] [smallmoney] NOT NULL,\r\n\t[ask11] [smallmoney] NOT NULL,\r\n\t[transtype12] [char](1) NOT NULL,\r\n\t[time12] [time](3) NOT NULL,\r\n\t[bid12] [smallmoney] NOT NULL,\r\n\t[ask12] [smallmoney] NOT NULL,\r\n\t[transtype13] [char](1) NOT NULL,\r\n\t[time13] [time](3) NOT NULL,\r\n\t[bid13] [smallmoney] NOT NULL,\r\n\t[ask13] [smallmoney] NOT NULL,\r\n\t[transtype14] [char](1) NOT NULL,\r\n\t[time14] [time](3) NOT NULL,\r\n\t[bid14] [smallmoney] NOT NULL,\r\n\t[ask14] [smallmoney] NOT NULL,\r\n\t[transtype15] [char](1) NOT NULL,\r\n\t[time15] [time](3) NOT NULL,\r\n\t[bid15] [smallmoney] NOT NULL,\r\n\t[ask15] [smallmoney] NOT NULL,\r\n\t[transtype16] [char](1) NOT NULL,\r\n\t[time16] [time](3) NOT NULL,\r\n\t[bid16] [smallmoney] NOT NULL,\r\n\t[ask16] [smallmoney] NOT NULL,\r\n\t[transtype17] [char](1) NOT NULL,\r\n\t[time17] [time](3) NOT NULL,\r\n\t[bid17] [smallmoney] NOT NULL,\r\n\t[ask17] [smallmoney] NOT NULL,\r\n\t[transtype18] [char](1) NOT NULL,\r\n\t[time18] [time](3) NOT NULL,\r\n\t[bid18] [smallmoney] NOT NULL,\r\n\t[ask18] [smallmoney] NOT NULL,\r\n\t[transtype19] [char](1) NOT NULL,\r\n\t[time19] [time](3) NOT NULL,\r\n\t[bid19] [smallmoney] NOT NULL,\r\n\t[ask19] [smallmoney] NOT NULL,\r\n\t[transtype20] [char](1) NOT NULL,\r\n\t[time20] [time](3) NOT NULL,\r\n\t[bid20] [smallmoney] NOT NULL,\r\n\t[ask20] [smallmoney] NOT NULL,\r\n\t[transtype21] [char](1) NOT NULL,\r\n\t[time21] [time](3) NOT NULL,\r\n\t[bid21] [smallmoney] NOT NULL,\r\n\t[ask21] [smallmoney] NOT NULL,\r\n\t[transtype22] [char](1) NOT NULL,\r\n\t[time22] [time](3) NOT NULL,\r\n\t[bid22] [smallmoney] NOT NULL,\r\n\t[ask22] [smallmoney] NOT NULL,\r\n\t[transtype23] [char](1) NOT NULL,\r\n\t[time23] [time](3) NOT NULL,\r\n\t[bid23] [smallmoney] NOT NULL,\r\n\t[ask23] [smallmoney] NOT NULL,\r\n\t[transtype24] [char](1) NOT NULL,\r\n\t[time24] [time](3) NOT NULL,\r\n\t[bid24] [smallmoney] NOT NULL,\r\n\t[ask24] [smallmoney] NOT NULL,\r\n\t[transtype25] [char](1) NOT NULL,\r\n\t[time25] [time](3) NOT NULL,\r\n\t[bid25] [smallmoney] NOT NULL,\r\n\t[ask25] [smallmoney] NOT NULL,\r\n\t[transtype26] [char](1) NOT NULL,\r\n\t[time26] [time](3) NOT NULL,\r\n\t[bid26] [smallmoney] NOT NULL,\r\n\t[ask26] [smallmoney] NOT NULL,\r\n\t[transtype27] [char](1) NOT NULL,\r\n\t[time27] [time](3) NOT NULL,\r\n\t[bid27] [smallmoney] NOT NULL,\r\n\t[ask27] [smallmoney] NOT NULL,\r\n\t[transtype28] [char](1) NOT NULL,\r\n\t[time28] [time](3) NOT NULL,\r\n\t[bid28] [smallmoney] NOT NULL,\r\n\t[ask28] [smallmoney] NOT NULL,\r\n\t[transtype29] [char](1) NOT NULL,\r\n\t[time29] [time](3) NOT NULL,\r\n\t[bid29] [smallmoney] NOT NULL,\r\n\t[ask29] [smallmoney] NOT NULL,\r\n\t[transtype30] [char](1) NOT NULL,\r\n\t[time30] [time](3) NOT NULL,\r\n\t[bid30] [smallmoney] NOT NULL,\r\n\t[ask30] [smallmoney] NOT NULL,\r\n\t[transtype31] [char](1) NOT NULL,\r\n\t[time31] [time](3) NOT NULL,\r\n\t[bid31] [smallmoney] NOT NULL,\r\n\t[ask31] [smallmoney] NOT NULL,\r\n\t[transtype32] [char](1) NOT NULL,\r\n\t[time32] [time](3) NOT NULL,\r\n\t[bid32] [smallmoney] NOT NULL,\r\n\t[ask32] [smallmoney] NOT NULL,\r\n\t[transtype33] [char](1) NOT NULL,\r\n\t[time33] [time](3) NOT NULL,\r\n\t[bid33] [smallmoney] NOT NULL,\r\n\t[ask33] [smallmoney] NOT NULL,\r\n\t[transtype34] [char](1) NOT NULL,\r\n\t[time34] [time](3) NOT NULL,\r\n\t[bid34] [smallmoney] NOT NULL,\r\n\t[ask34] [smallmoney] NOT NULL,\r\n\t[transtype35] [char](1) NOT NULL,\r\n\t[time35] [time](3) NOT NULL,\r\n\t[bid35] [smallmoney] NOT NULL,\r\n\t[ask35] [smallmoney] NOT NULL,\r\n\t[transtype36] [char](1) NOT NULL,\r\n\t[time36] [time](3) NOT NULL,\r\n\t[bid36] [smallmoney] NOT NULL,\r\n\t[ask36] [smallmoney] NOT NULL,\r\n\t[transtype37] [char](1) NOT NULL,\r\n\t[time37] [time](3) NOT NULL,\r\n\t[bid37] [smallmoney] NOT NULL,\r\n\t[ask37] [smallmoney] NOT NULL,\r\n\t[transtype38] [char](1) NOT NULL,\r\n\t[time38] [time](3) NOT NULL,\r\n\t[bid38] [smallmoney] NOT NULL,\r\n\t[ask38] [smallmoney] NOT NULL,\r\n\t[transtype39] [char](1) NOT NULL,\r\n\t[time39] [time](3) NOT NULL,\r\n\t[bid39] [smallmoney] NOT NULL,\r\n\t[ask39] [smallmoney] NOT NULL,\r\n\t[transtype40] [char](1) NOT NULL,\r\n\t[time40] [time](3) NOT NULL,\r\n\t[bid40] [smallmoney] NOT NULL,\r\n\t[ask40] [smallmoney] NOT NULL,\r\n\t[transtype41] [char](1) NOT NULL,\r\n\t[time41] [time](3) NOT NULL,\r\n\t[bid41] [smallmoney] NOT NULL,\r\n\t[ask41] [smallmoney] NOT NULL,\r\n\t[transtype42] [char](1) NOT NULL,\r\n\t[time42] [time](3) NOT NULL,\r\n\t[bid42] [smallmoney] NOT NULL,\r\n\t[ask42] [smallmoney] NOT NULL,\r\n\t[transtype43] [char](1) NOT NULL,\r\n\t[time43] [time](3) NOT NULL,\r\n\t[bid43] [smallmoney] NOT NULL,\r\n\t[ask43] [smallmoney] NOT NULL,\r\n\t[transtype44] [char](1) NOT NULL,\r\n\t[time44] [time](3) NOT NULL,\r\n\t[bid44] [smallmoney] NOT NULL,\r\n\t[ask44] [smallmoney] NOT NULL,\r\n\t[transtype45] [char](1) NOT NULL,\r\n\t[time45] [time](3) NOT NULL,\r\n\t[bid45] [smallmoney] NOT NULL,\r\n\t[ask45] [smallmoney] NOT NULL,\r\n\t[transtype46] [char](1) NOT NULL,\r\n\t[time46] [time](3) NOT NULL,\r\n\t[bid46] [smallmoney] NOT NULL,\r\n\t[ask46] [smallmoney] NOT NULL,\r\n\t[transtype47] [char](1) NOT NULL,\r\n\t[time47] [time](3) NOT NULL,\r\n\t[bid47] [smallmoney] NOT NULL,\r\n\t[ask47] [smallmoney] NOT NULL,\r\n\t[transtype48] [char](1) NOT NULL,\r\n\t[time48] [time](3) NOT NULL,\r\n\t[bid48] [smallmoney] NOT NULL,\r\n\t[ask48] [smallmoney] NOT NULL,\r\n\t[transtype49] [char](1) NOT NULL,\r\n\t[time49] [time](3) NOT NULL,\r\n\t[bid49] [smallmoney] NOT NULL,\r\n\t[ask49] [smallmoney] NOT NULL,\r\n\t[transtype50] [char](1) NOT NULL,\r\n\t[time50] [time](3) NOT NULL,\r\n\t[bid50] [smallmoney] NOT NULL,\r\n\t[ask50] [smallmoney] NOT NULL\r\n);\r\n\r\n-- Optionally add primary keys\r\nALTER TABLE training ADD CONSTRAINT PK_training PRIMARY KEY CLUSTERED (row_id);\r\nALTER TABLE training ADD CONSTRAINT PK_training PRIMARY KEY CLUSTERED (row_id);\r\n\r\n</pre>\r\n<p>Finally, here's a quick sanity check:</p>\r\n<pre>SELECT COUNT(*) FROM training\r\n-- 754018\r\n\r\nSELECT SUM(p_value) FROM training\r\n-- 3542062365303403\r\n\r\nSELECT COUNT(*) FROM testing\r\n-- 50000\r\n\r\nSELECT SUM(p_value) FROM testing\r\n-- 155032668754821\r\n</pre>\r\n<p>Feel free to post in this thread any other tips you have while working with the dataset using a database.</p>",
  "messages": [
    {
      "id": "6472",
      "postDate": "11/11/2011 18:34:09",
      "content": "<p>Since this dataset is quite large, I wanted to offer the database schema I used in case it helps other. I tried to use the smallest data type possible per column to be more efficient.</p>\r\n<p>I used SQL Server 2008, but see other related threads in the <a href=\"https://www.kaggle.com/c/wikichallenge/forums/t/668/importing-the-data-into-sql-server\">\r\nWikiChallenge</a> and <a href=\"https://www.kaggle.com/c/ClaimPredictionChallenge/forums/t/711/importing-to-sql-server-and-aggregate-statistics\">\r\nClaim Prediction Challenge</a> competitions for tips on how to do similar things in other databases.</p>\r\n<p>Here are my tables (note the use of smallmoney and time(3)):</p>\r\n<pre>CREATE TABLE [training](\r\n\t[row_id] [int] NOT NULL,\r\n\t[security_id] [tinyint] NOT NULL,\r\n\t[p_tcount] [int] NOT NULL,\r\n\t[p_value] [bigint] NOT NULL,\r\n\t[trade_vwap] [smallmoney] NOT NULL,\r\n\t[trade_volume] [int] NOT NULL,\r\n\t[initiator] [char](1) NOT NULL,\r\n\t[transtype1] [char](1) NOT NULL,\r\n\t[time1] [time](3) NOT NULL,\r\n\t[bid1] [smallmoney] NOT NULL,\r\n\t[ask1] [smallmoney] NOT NULL,\r\n\t[transtype2] [char](1) NOT NULL,\r\n\t[time2] [time](3) NOT NULL,\r\n\t[bid2] [smallmoney] NOT NULL,\r\n\t[ask2] [smallmoney] NOT NULL,\r\n\t[transtype3] [char](1) NOT NULL,\r\n\t[time3] [time](3) NOT NULL,\r\n\t[bid3] [smallmoney] NOT NULL,\r\n\t[ask3] [smallmoney] NOT NULL,\r\n\t[transtype4] [char](1) NOT NULL,\r\n\t[time4] [time](3) NOT NULL,\r\n\t[bid4] [smallmoney] NOT NULL,\r\n\t[ask4] [smallmoney] NOT NULL,\r\n\t[transtype5] [char](1) NOT NULL,\r\n\t[time5] [time](3) NOT NULL,\r\n\t[bid5] [smallmoney] NOT NULL,\r\n\t[ask5] [smallmoney] NOT NULL,\r\n\t[transtype6] [char](1) NOT NULL,\r\n\t[time6] [time](3) NOT NULL,\r\n\t[bid6] [smallmoney] NOT NULL,\r\n\t[ask6] [smallmoney] NOT NULL,\r\n\t[transtype7] [char](1) NOT NULL,\r\n\t[time7] [time](3) NOT NULL,\r\n\t[bid7] [smallmoney] NOT NULL,\r\n\t[ask7] [smallmoney] NOT NULL,\r\n\t[transtype8] [char](1) NOT NULL,\r\n\t[time8] [time](3) NOT NULL,\r\n\t[bid8] [smallmoney] NOT NULL,\r\n\t[ask8] [smallmoney] NOT NULL,\r\n\t[transtype9] [char](1) NOT NULL,\r\n\t[time9] [time](3) NOT NULL,\r\n\t[bid9] [smallmoney] NOT NULL,\r\n\t[ask9] [smallmoney] NOT NULL,\r\n\t[transtype10] [char](1) NOT NULL,\r\n\t[time10] [time](3) NOT NULL,\r\n\t[bid10] [smallmoney] NOT NULL,\r\n\t[ask10] [smallmoney] NOT NULL,\r\n\t[transtype11] [char](1) NOT NULL,\r\n\t[time11] [time](3) NOT NULL,\r\n\t[bid11] [smallmoney] NOT NULL,\r\n\t[ask11] [smallmoney] NOT NULL,\r\n\t[transtype12] [char](1) NOT NULL,\r\n\t[time12] [time](3) NOT NULL,\r\n\t[bid12] [smallmoney] NOT NULL,\r\n\t[ask12] [smallmoney] NOT NULL,\r\n\t[transtype13] [char](1) NOT NULL,\r\n\t[time13] [time](3) NOT NULL,\r\n\t[bid13] [smallmoney] NOT NULL,\r\n\t[ask13] [smallmoney] NOT NULL,\r\n\t[transtype14] [char](1) NOT NULL,\r\n\t[time14] [time](3) NOT NULL,\r\n\t[bid14] [smallmoney] NOT NULL,\r\n\t[ask14] [smallmoney] NOT NULL,\r\n\t[transtype15] [char](1) NOT NULL,\r\n\t[time15] [time](3) NOT NULL,\r\n\t[bid15] [smallmoney] NOT NULL,\r\n\t[ask15] [smallmoney] NOT NULL,\r\n\t[transtype16] [char](1) NOT NULL,\r\n\t[time16] [time](3) NOT NULL,\r\n\t[bid16] [smallmoney] NOT NULL,\r\n\t[ask16] [smallmoney] NOT NULL,\r\n\t[transtype17] [char](1) NOT NULL,\r\n\t[time17] [time](3) NOT NULL,\r\n\t[bid17] [smallmoney] NOT NULL,\r\n\t[ask17] [smallmoney] NOT NULL,\r\n\t[transtype18] [char](1) NOT NULL,\r\n\t[time18] [time](3) NOT NULL,\r\n\t[bid18] [smallmoney] NOT NULL,\r\n\t[ask18] [smallmoney] NOT NULL,\r\n\t[transtype19] [char](1) NOT NULL,\r\n\t[time19] [time](3) NOT NULL,\r\n\t[bid19] [smallmoney] NOT NULL,\r\n\t[ask19] [smallmoney] NOT NULL,\r\n\t[transtype20] [char](1) NOT NULL,\r\n\t[time20] [time](3) NOT NULL,\r\n\t[bid20] [smallmoney] NOT NULL,\r\n\t[ask20] [smallmoney] NOT NULL,\r\n\t[transtype21] [char](1) NOT NULL,\r\n\t[time21] [time](3) NOT NULL,\r\n\t[bid21] [smallmoney] NOT NULL,\r\n\t[ask21] [smallmoney] NOT NULL,\r\n\t[transtype22] [char](1) NOT NULL,\r\n\t[time22] [time](3) NOT NULL,\r\n\t[bid22] [smallmoney] NOT NULL,\r\n\t[ask22] [smallmoney] NOT NULL,\r\n\t[transtype23] [char](1) NOT NULL,\r\n\t[time23] [time](3) NOT NULL,\r\n\t[bid23] [smallmoney] NOT NULL,\r\n\t[ask23] [smallmoney] NOT NULL,\r\n\t[transtype24] [char](1) NOT NULL,\r\n\t[time24] [time](3) NOT NULL,\r\n\t[bid24] [smallmoney] NOT NULL,\r\n\t[ask24] [smallmoney] NOT NULL,\r\n\t[transtype25] [char](1) NOT NULL,\r\n\t[time25] [time](3) NOT NULL,\r\n\t[bid25] [smallmoney] NOT NULL,\r\n\t[ask25] [smallmoney] NOT NULL,\r\n\t[transtype26] [char](1) NOT NULL,\r\n\t[time26] [time](3) NOT NULL,\r\n\t[bid26] [smallmoney] NOT NULL,\r\n\t[ask26] [smallmoney] NOT NULL,\r\n\t[transtype27] [char](1) NOT NULL,\r\n\t[time27] [time](3) NOT NULL,\r\n\t[bid27] [smallmoney] NOT NULL,\r\n\t[ask27] [smallmoney] NOT NULL,\r\n\t[transtype28] [char](1) NOT NULL,\r\n\t[time28] [time](3) NOT NULL,\r\n\t[bid28] [smallmoney] NOT NULL,\r\n\t[ask28] [smallmoney] NOT NULL,\r\n\t[transtype29] [char](1) NOT NULL,\r\n\t[time29] [time](3) NOT NULL,\r\n\t[bid29] [smallmoney] NOT NULL,\r\n\t[ask29] [smallmoney] NOT NULL,\r\n\t[transtype30] [char](1) NOT NULL,\r\n\t[time30] [time](3) NOT NULL,\r\n\t[bid30] [smallmoney] NOT NULL,\r\n\t[ask30] [smallmoney] NOT NULL,\r\n\t[transtype31] [char](1) NOT NULL,\r\n\t[time31] [time](3) NOT NULL,\r\n\t[bid31] [smallmoney] NOT NULL,\r\n\t[ask31] [smallmoney] NOT NULL,\r\n\t[transtype32] [char](1) NOT NULL,\r\n\t[time32] [time](3) NOT NULL,\r\n\t[bid32] [smallmoney] NOT NULL,\r\n\t[ask32] [smallmoney] NOT NULL,\r\n\t[transtype33] [char](1) NOT NULL,\r\n\t[time33] [time](3) NOT NULL,\r\n\t[bid33] [smallmoney] NOT NULL,\r\n\t[ask33] [smallmoney] NOT NULL,\r\n\t[transtype34] [char](1) NOT NULL,\r\n\t[time34] [time](3) NOT NULL,\r\n\t[bid34] [smallmoney] NOT NULL,\r\n\t[ask34] [smallmoney] NOT NULL,\r\n\t[transtype35] [char](1) NOT NULL,\r\n\t[time35] [time](3) NOT NULL,\r\n\t[bid35] [smallmoney] NOT NULL,\r\n\t[ask35] [smallmoney] NOT NULL,\r\n\t[transtype36] [char](1) NOT NULL,\r\n\t[time36] [time](3) NOT NULL,\r\n\t[bid36] [smallmoney] NOT NULL,\r\n\t[ask36] [smallmoney] NOT NULL,\r\n\t[transtype37] [char](1) NOT NULL,\r\n\t[time37] [time](3) NOT NULL,\r\n\t[bid37] [smallmoney] NOT NULL,\r\n\t[ask37] [smallmoney] NOT NULL,\r\n\t[transtype38] [char](1) NOT NULL,\r\n\t[time38] [time](3) NOT NULL,\r\n\t[bid38] [smallmoney] NOT NULL,\r\n\t[ask38] [smallmoney] NOT NULL,\r\n\t[transtype39] [char](1) NOT NULL,\r\n\t[time39] [time](3) NOT NULL,\r\n\t[bid39] [smallmoney] NOT NULL,\r\n\t[ask39] [smallmoney] NOT NULL,\r\n\t[transtype40] [char](1) NOT NULL,\r\n\t[time40] [time](3) NOT NULL,\r\n\t[bid40] [smallmoney] NOT NULL,\r\n\t[ask40] [smallmoney] NOT NULL,\r\n\t[transtype41] [char](1) NOT NULL,\r\n\t[time41] [time](3) NOT NULL,\r\n\t[bid41] [smallmoney] NOT NULL,\r\n\t[ask41] [smallmoney] NOT NULL,\r\n\t[transtype42] [char](1) NOT NULL,\r\n\t[time42] [time](3) NOT NULL,\r\n\t[bid42] [smallmoney] NOT NULL,\r\n\t[ask42] [smallmoney] NOT NULL,\r\n\t[transtype43] [char](1) NOT NULL,\r\n\t[time43] [time](3) NOT NULL,\r\n\t[bid43] [smallmoney] NOT NULL,\r\n\t[ask43] [smallmoney] NOT NULL,\r\n\t[transtype44] [char](1) NOT NULL,\r\n\t[time44] [time](3) NOT NULL,\r\n\t[bid44] [smallmoney] NOT NULL,\r\n\t[ask44] [smallmoney] NOT NULL,\r\n\t[transtype45] [char](1) NOT NULL,\r\n\t[time45] [time](3) NOT NULL,\r\n\t[bid45] [smallmoney] NOT NULL,\r\n\t[ask45] [smallmoney] NOT NULL,\r\n\t[transtype46] [char](1) NOT NULL,\r\n\t[time46] [time](3) NOT NULL,\r\n\t[bid46] [smallmoney] NOT NULL,\r\n\t[ask46] [smallmoney] NOT NULL,\r\n\t[transtype47] [char](1) NOT NULL,\r\n\t[time47] [time](3) NOT NULL,\r\n\t[bid47] [smallmoney] NOT NULL,\r\n\t[ask47] [smallmoney] NOT NULL,\r\n\t[transtype48] [char](1) NOT NULL,\r\n\t[time48] [time](3) NOT NULL,\r\n\t[bid48] [smallmoney] NOT NULL,\r\n\t[ask48] [smallmoney] NOT NULL,\r\n\t[transtype49] [char](1) NOT NULL,\r\n\t[time49] [time](3) NOT NULL,\r\n\t[bid49] [smallmoney] NOT NULL,\r\n\t[ask49] [smallmoney] NOT NULL,\r\n\t[transtype50] [char](1) NOT NULL,\r\n\t[time50] [time](3) NOT NULL,\r\n\t[bid50] [smallmoney] NOT NULL,\r\n\t[ask50] [smallmoney] NOT NULL,\r\n\t[bid51] [smallmoney] NOT NULL,\r\n\t[ask51] [smallmoney] NOT NULL,\r\n\t[bid52] [smallmoney] NOT NULL,\r\n\t[ask52] [smallmoney] NOT NULL,\r\n\t[bid53] [smallmoney] NOT NULL,\r\n\t[ask53] [smallmoney] NOT NULL,\r\n\t[bid54] [smallmoney] NOT NULL,\r\n\t[ask54] [smallmoney] NOT NULL,\r\n\t[bid55] [smallmoney] NOT NULL,\r\n\t[ask55] [smallmoney] NOT NULL,\r\n\t[bid56] [smallmoney] NOT NULL,\r\n\t[ask56] [smallmoney] NOT NULL,\r\n\t[bid57] [smallmoney] NOT NULL,\r\n\t[ask57] [smallmoney] NOT NULL,\r\n\t[bid58] [smallmoney] NOT NULL,\r\n\t[ask58] [smallmoney] NOT NULL,\r\n\t[bid59] [smallmoney] NOT NULL,\r\n\t[ask59] [smallmoney] NOT NULL,\r\n\t[bid60] [smallmoney] NOT NULL,\r\n\t[ask60] [smallmoney] NOT NULL,\r\n\t[bid61] [smallmoney] NOT NULL,\r\n\t[ask61] [smallmoney] NOT NULL,\r\n\t[bid62] [smallmoney] NOT NULL,\r\n\t[ask62] [smallmoney] NOT NULL,\r\n\t[bid63] [smallmoney] NOT NULL,\r\n\t[ask63] [smallmoney] NOT NULL,\r\n\t[bid64] [smallmoney] NOT NULL,\r\n\t[ask64] [smallmoney] NOT NULL,\r\n\t[bid65] [smallmoney] NOT NULL,\r\n\t[ask65] [smallmoney] NOT NULL,\r\n\t[bid66] [smallmoney] NOT NULL,\r\n\t[ask66] [smallmoney] NOT NULL,\r\n\t[bid67] [smallmoney] NOT NULL,\r\n\t[ask67] [smallmoney] NOT NULL,\r\n\t[bid68] [smallmoney] NOT NULL,\r\n\t[ask68] [smallmoney] NOT NULL,\r\n\t[bid69] [smallmoney] NOT NULL,\r\n\t[ask69] [smallmoney] NOT NULL,\r\n\t[bid70] [smallmoney] NOT NULL,\r\n\t[ask70] [smallmoney] NOT NULL,\r\n\t[bid71] [smallmoney] NOT NULL,\r\n\t[ask71] [smallmoney] NOT NULL,\r\n\t[bid72] [smallmoney] NOT NULL,\r\n\t[ask72] [smallmoney] NOT NULL,\r\n\t[bid73] [smallmoney] NOT NULL,\r\n\t[ask73] [smallmoney] NOT NULL,\r\n\t[bid74] [smallmoney] NOT NULL,\r\n\t[ask74] [smallmoney] NOT NULL,\r\n\t[bid75] [smallmoney] NOT NULL,\r\n\t[ask75] [smallmoney] NOT NULL,\r\n\t[bid76] [smallmoney] NOT NULL,\r\n\t[ask76] [smallmoney] NOT NULL,\r\n\t[bid77] [smallmoney] NOT NULL,\r\n\t[ask77] [smallmoney] NOT NULL,\r\n\t[bid78] [smallmoney] NOT NULL,\r\n\t[ask78] [smallmoney] NOT NULL,\r\n\t[bid79] [smallmoney] NOT NULL,\r\n\t[ask79] [smallmoney] NOT NULL,\r\n\t[bid80] [smallmoney] NOT NULL,\r\n\t[ask80] [smallmoney] NOT NULL,\r\n\t[bid81] [smallmoney] NOT NULL,\r\n\t[ask81] [smallmoney] NOT NULL,\r\n\t[bid82] [smallmoney] NOT NULL,\r\n\t[ask82] [smallmoney] NOT NULL,\r\n\t[bid83] [smallmoney] NOT NULL,\r\n\t[ask83] [smallmoney] NOT NULL,\r\n\t[bid84] [smallmoney] NOT NULL,\r\n\t[ask84] [smallmoney] NOT NULL,\r\n\t[bid85] [smallmoney] NOT NULL,\r\n\t[ask85] [smallmoney] NOT NULL,\r\n\t[bid86] [smallmoney] NOT NULL,\r\n\t[ask86] [smallmoney] NOT NULL,\r\n\t[bid87] [smallmoney] NOT NULL,\r\n\t[ask87] [smallmoney] NOT NULL,\r\n\t[bid88] [smallmoney] NOT NULL,\r\n\t[ask88] [smallmoney] NOT NULL,\r\n\t[bid89] [smallmoney] NOT NULL,\r\n\t[ask89] [smallmoney] NOT NULL,\r\n\t[bid90] [smallmoney] NOT NULL,\r\n\t[ask90] [smallmoney] NOT NULL,\r\n\t[bid91] [smallmoney] NOT NULL,\r\n\t[ask91] [smallmoney] NOT NULL,\r\n\t[bid92] [smallmoney] NOT NULL,\r\n\t[ask92] [smallmoney] NOT NULL,\r\n\t[bid93] [smallmoney] NOT NULL,\r\n\t[ask93] [smallmoney] NOT NULL,\r\n\t[bid94] [smallmoney] NOT NULL,\r\n\t[ask94] [smallmoney] NOT NULL,\r\n\t[bid95] [smallmoney] NOT NULL,\r\n\t[ask95] [smallmoney] NOT NULL,\r\n\t[bid96] [smallmoney] NOT NULL,\r\n\t[ask96] [smallmoney] NOT NULL,\r\n\t[bid97] [smallmoney] NOT NULL,\r\n\t[ask97] [smallmoney] NOT NULL,\r\n\t[bid98] [smallmoney] NOT NULL,\r\n\t[ask98] [smallmoney] NOT NULL,\r\n\t[bid99] [smallmoney] NOT NULL,\r\n\t[ask99] [smallmoney] NOT NULL,\r\n\t[bid100] [smallmoney] NOT NULL,\r\n\t[ask100] [smallmoney] NOT NULL\r\n);\r\n\r\nCREATE TABLE [testing](\r\n\t[row_id] [int] NOT NULL,\r\n\t[security_id] [tinyint] NOT NULL,\r\n\t[p_tcount] [int] NOT NULL,\r\n\t[p_value] [bigint] NOT NULL,\r\n\t[trade_vwap] [smallmoney] NOT NULL,\r\n\t[trade_volume] [int] NOT NULL,\r\n\t[initiator] [char](1) NOT NULL,\r\n\t[transtype1] [char](1) NOT NULL,\r\n\t[time1] [time](3) NOT NULL,\r\n\t[bid1] [smallmoney] NOT NULL,\r\n\t[ask1] [smallmoney] NOT NULL,\r\n\t[transtype2] [char](1) NOT NULL,\r\n\t[time2] [time](3) NOT NULL,\r\n\t[bid2] [smallmoney] NOT NULL,\r\n\t[ask2] [smallmoney] NOT NULL,\r\n\t[transtype3] [char](1) NOT NULL,\r\n\t[time3] [time](3) NOT NULL,\r\n\t[bid3] [smallmoney] NOT NULL,\r\n\t[ask3] [smallmoney] NOT NULL,\r\n\t[transtype4] [char](1) NOT NULL,\r\n\t[time4] [time](3) NOT NULL,\r\n\t[bid4] [smallmoney] NOT NULL,\r\n\t[ask4] [smallmoney] NOT NULL,\r\n\t[transtype5] [char](1) NOT NULL,\r\n\t[time5] [time](3) NOT NULL,\r\n\t[bid5] [smallmoney] NOT NULL,\r\n\t[ask5] [smallmoney] NOT NULL,\r\n\t[transtype6] [char](1) NOT NULL,\r\n\t[time6] [time](3) NOT NULL,\r\n\t[bid6] [smallmoney] NOT NULL,\r\n\t[ask6] [smallmoney] NOT NULL,\r\n\t[transtype7] [char](1) NOT NULL,\r\n\t[time7] [time](3) NOT NULL,\r\n\t[bid7] [smallmoney] NOT NULL,\r\n\t[ask7] [smallmoney] NOT NULL,\r\n\t[transtype8] [char](1) NOT NULL,\r\n\t[time8] [time](3) NOT NULL,\r\n\t[bid8] [smallmoney] NOT NULL,\r\n\t[ask8] [smallmoney] NOT NULL,\r\n\t[transtype9] [char](1) NOT NULL,\r\n\t[time9] [time](3) NOT NULL,\r\n\t[bid9] [smallmoney] NOT NULL,\r\n\t[ask9] [smallmoney] NOT NULL,\r\n\t[transtype10] [char](1) NOT NULL,\r\n\t[time10] [time](3) NOT NULL,\r\n\t[bid10] [smallmoney] NOT NULL,\r\n\t[ask10] [smallmoney] NOT NULL,\r\n\t[transtype11] [char](1) NOT NULL,\r\n\t[time11] [time](3) NOT NULL,\r\n\t[bid11] [smallmoney] NOT NULL,\r\n\t[ask11] [smallmoney] NOT NULL,\r\n\t[transtype12] [char](1) NOT NULL,\r\n\t[time12] [time](3) NOT NULL,\r\n\t[bid12] [smallmoney] NOT NULL,\r\n\t[ask12] [smallmoney] NOT NULL,\r\n\t[transtype13] [char](1) NOT NULL,\r\n\t[time13] [time](3) NOT NULL,\r\n\t[bid13] [smallmoney] NOT NULL,\r\n\t[ask13] [smallmoney] NOT NULL,\r\n\t[transtype14] [char](1) NOT NULL,\r\n\t[time14] [time](3) NOT NULL,\r\n\t[bid14] [smallmoney] NOT NULL,\r\n\t[ask14] [smallmoney] NOT NULL,\r\n\t[transtype15] [char](1) NOT NULL,\r\n\t[time15] [time](3) NOT NULL,\r\n\t[bid15] [smallmoney] NOT NULL,\r\n\t[ask15] [smallmoney] NOT NULL,\r\n\t[transtype16] [char](1) NOT NULL,\r\n\t[time16] [time](3) NOT NULL,\r\n\t[bid16] [smallmoney] NOT NULL,\r\n\t[ask16] [smallmoney] NOT NULL,\r\n\t[transtype17] [char](1) NOT NULL,\r\n\t[time17] [time](3) NOT NULL,\r\n\t[bid17] [smallmoney] NOT NULL,\r\n\t[ask17] [smallmoney] NOT NULL,\r\n\t[transtype18] [char](1) NOT NULL,\r\n\t[time18] [time](3) NOT NULL,\r\n\t[bid18] [smallmoney] NOT NULL,\r\n\t[ask18] [smallmoney] NOT NULL,\r\n\t[transtype19] [char](1) NOT NULL,\r\n\t[time19] [time](3) NOT NULL,\r\n\t[bid19] [smallmoney] NOT NULL,\r\n\t[ask19] [smallmoney] NOT NULL,\r\n\t[transtype20] [char](1) NOT NULL,\r\n\t[time20] [time](3) NOT NULL,\r\n\t[bid20] [smallmoney] NOT NULL,\r\n\t[ask20] [smallmoney] NOT NULL,\r\n\t[transtype21] [char](1) NOT NULL,\r\n\t[time21] [time](3) NOT NULL,\r\n\t[bid21] [smallmoney] NOT NULL,\r\n\t[ask21] [smallmoney] NOT NULL,\r\n\t[transtype22] [char](1) NOT NULL,\r\n\t[time22] [time](3) NOT NULL,\r\n\t[bid22] [smallmoney] NOT NULL,\r\n\t[ask22] [smallmoney] NOT NULL,\r\n\t[transtype23] [char](1) NOT NULL,\r\n\t[time23] [time](3) NOT NULL,\r\n\t[bid23] [smallmoney] NOT NULL,\r\n\t[ask23] [smallmoney] NOT NULL,\r\n\t[transtype24] [char](1) NOT NULL,\r\n\t[time24] [time](3) NOT NULL,\r\n\t[bid24] [smallmoney] NOT NULL,\r\n\t[ask24] [smallmoney] NOT NULL,\r\n\t[transtype25] [char](1) NOT NULL,\r\n\t[time25] [time](3) NOT NULL,\r\n\t[bid25] [smallmoney] NOT NULL,\r\n\t[ask25] [smallmoney] NOT NULL,\r\n\t[transtype26] [char](1) NOT NULL,\r\n\t[time26] [time](3) NOT NULL,\r\n\t[bid26] [smallmoney] NOT NULL,\r\n\t[ask26] [smallmoney] NOT NULL,\r\n\t[transtype27] [char](1) NOT NULL,\r\n\t[time27] [time](3) NOT NULL,\r\n\t[bid27] [smallmoney] NOT NULL,\r\n\t[ask27] [smallmoney] NOT NULL,\r\n\t[transtype28] [char](1) NOT NULL,\r\n\t[time28] [time](3) NOT NULL,\r\n\t[bid28] [smallmoney] NOT NULL,\r\n\t[ask28] [smallmoney] NOT NULL,\r\n\t[transtype29] [char](1) NOT NULL,\r\n\t[time29] [time](3) NOT NULL,\r\n\t[bid29] [smallmoney] NOT NULL,\r\n\t[ask29] [smallmoney] NOT NULL,\r\n\t[transtype30] [char](1) NOT NULL,\r\n\t[time30] [time](3) NOT NULL,\r\n\t[bid30] [smallmoney] NOT NULL,\r\n\t[ask30] [smallmoney] NOT NULL,\r\n\t[transtype31] [char](1) NOT NULL,\r\n\t[time31] [time](3) NOT NULL,\r\n\t[bid31] [smallmoney] NOT NULL,\r\n\t[ask31] [smallmoney] NOT NULL,\r\n\t[transtype32] [char](1) NOT NULL,\r\n\t[time32] [time](3) NOT NULL,\r\n\t[bid32] [smallmoney] NOT NULL,\r\n\t[ask32] [smallmoney] NOT NULL,\r\n\t[transtype33] [char](1) NOT NULL,\r\n\t[time33] [time](3) NOT NULL,\r\n\t[bid33] [smallmoney] NOT NULL,\r\n\t[ask33] [smallmoney] NOT NULL,\r\n\t[transtype34] [char](1) NOT NULL,\r\n\t[time34] [time](3) NOT NULL,\r\n\t[bid34] [smallmoney] NOT NULL,\r\n\t[ask34] [smallmoney] NOT NULL,\r\n\t[transtype35] [char](1) NOT NULL,\r\n\t[time35] [time](3) NOT NULL,\r\n\t[bid35] [smallmoney] NOT NULL,\r\n\t[ask35] [smallmoney] NOT NULL,\r\n\t[transtype36] [char](1) NOT NULL,\r\n\t[time36] [time](3) NOT NULL,\r\n\t[bid36] [smallmoney] NOT NULL,\r\n\t[ask36] [smallmoney] NOT NULL,\r\n\t[transtype37] [char](1) NOT NULL,\r\n\t[time37] [time](3) NOT NULL,\r\n\t[bid37] [smallmoney] NOT NULL,\r\n\t[ask37] [smallmoney] NOT NULL,\r\n\t[transtype38] [char](1) NOT NULL,\r\n\t[time38] [time](3) NOT NULL,\r\n\t[bid38] [smallmoney] NOT NULL,\r\n\t[ask38] [smallmoney] NOT NULL,\r\n\t[transtype39] [char](1) NOT NULL,\r\n\t[time39] [time](3) NOT NULL,\r\n\t[bid39] [smallmoney] NOT NULL,\r\n\t[ask39] [smallmoney] NOT NULL,\r\n\t[transtype40] [char](1) NOT NULL,\r\n\t[time40] [time](3) NOT NULL,\r\n\t[bid40] [smallmoney] NOT NULL,\r\n\t[ask40] [smallmoney] NOT NULL,\r\n\t[transtype41] [char](1) NOT NULL,\r\n\t[time41] [time](3) NOT NULL,\r\n\t[bid41] [smallmoney] NOT NULL,\r\n\t[ask41] [smallmoney] NOT NULL,\r\n\t[transtype42] [char](1) NOT NULL,\r\n\t[time42] [time](3) NOT NULL,\r\n\t[bid42] [smallmoney] NOT NULL,\r\n\t[ask42] [smallmoney] NOT NULL,\r\n\t[transtype43] [char](1) NOT NULL,\r\n\t[time43] [time](3) NOT NULL,\r\n\t[bid43] [smallmoney] NOT NULL,\r\n\t[ask43] [smallmoney] NOT NULL,\r\n\t[transtype44] [char](1) NOT NULL,\r\n\t[time44] [time](3) NOT NULL,\r\n\t[bid44] [smallmoney] NOT NULL,\r\n\t[ask44] [smallmoney] NOT NULL,\r\n\t[transtype45] [char](1) NOT NULL,\r\n\t[time45] [time](3) NOT NULL,\r\n\t[bid45] [smallmoney] NOT NULL,\r\n\t[ask45] [smallmoney] NOT NULL,\r\n\t[transtype46] [char](1) NOT NULL,\r\n\t[time46] [time](3) NOT NULL,\r\n\t[bid46] [smallmoney] NOT NULL,\r\n\t[ask46] [smallmoney] NOT NULL,\r\n\t[transtype47] [char](1) NOT NULL,\r\n\t[time47] [time](3) NOT NULL,\r\n\t[bid47] [smallmoney] NOT NULL,\r\n\t[ask47] [smallmoney] NOT NULL,\r\n\t[transtype48] [char](1) NOT NULL,\r\n\t[time48] [time](3) NOT NULL,\r\n\t[bid48] [smallmoney] NOT NULL,\r\n\t[ask48] [smallmoney] NOT NULL,\r\n\t[transtype49] [char](1) NOT NULL,\r\n\t[time49] [time](3) NOT NULL,\r\n\t[bid49] [smallmoney] NOT NULL,\r\n\t[ask49] [smallmoney] NOT NULL,\r\n\t[transtype50] [char](1) NOT NULL,\r\n\t[time50] [time](3) NOT NULL,\r\n\t[bid50] [smallmoney] NOT NULL,\r\n\t[ask50] [smallmoney] NOT NULL\r\n);\r\n\r\n-- Optionally add primary keys\r\nALTER TABLE training ADD CONSTRAINT PK_training PRIMARY KEY CLUSTERED (row_id);\r\nALTER TABLE training ADD CONSTRAINT PK_training PRIMARY KEY CLUSTERED (row_id);\r\n\r\n</pre>\r\n<p>Finally, here's a quick sanity check:</p>\r\n<pre>SELECT COUNT(*) FROM training\r\n-- 754018\r\n\r\nSELECT SUM(p_value) FROM training\r\n-- 3542062365303403\r\n\r\nSELECT COUNT(*) FROM testing\r\n-- 50000\r\n\r\nSELECT SUM(p_value) FROM testing\r\n-- 155032668754821\r\n</pre>\r\n<p>Feel free to post in this thread any other tips you have while working with the dataset using a database.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "6504",
      "postDate": "11/13/2011 11:23:40",
      "content": "<pre>This extra bit of SQL will load in the data for you as well...</pre>\r\n<pre>BULK INSERT training<br>FROM 'E:\\comps\\algorithmictradingchallenge\\data\\training\\training.csv'<br>WITH<br>(<br>MAXERRORS = 0,<br>FIRSTROW = 2,<br>FIELDTERMINATOR = ',',<br>ROWTERMINATOR = '\\n'<br>)</pre>\r\n<pre>&nbsp;</pre>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 6504,
      "author_name": "salimali",
      "author_url": "",
      "post_date": "11/13/2011 11:23:40",
      "content": "<pre>This extra bit of SQL will load in the data for you as well...</pre>\r\n<pre>BULK INSERT training<br>FROM 'E:\\comps\\algorithmictradingchallenge\\data\\training\\training.csv'<br>WITH<br>(<br>MAXERRORS = 0,<br>FIRSTROW = 2,<br>FIELDTERMINATOR = ',',<br>ROWTERMINATOR = '\\n'<br>)</pre>\r\n<pre>&nbsp;</pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "6472": "",
    "6504": ""
  },
  "source": "meta"
}