{ "id": 328979, "title": "Sharing some ideas: Feature Engineering!", "url": "/competitions/AI4Code/discussion/328979", "author_name": "", "post_date": "2022-06-03T23:16:22.766573200Z", "votes": 21, "comment_count": 6, "views": 0, "content": "

Sharing some ideas: feature engineering!

\n

In this competition, there are two main features: markdown and python code.
\nI think the most important thing is to find a way to extract useful information from them.

\n

Here are some ideas:

\n

Feature engineering for markdown cells 📚

\n\n
def count_escaped_characters(text):\n    return len(re.findall(r'\\\\', text))\n
\n\n
def count_words(text):\n    return len(text.split())\n
\n\n
def count_characters(text):\n    return len(text)\n
\n\n
def count_characters_per_word(text):\n    return len(text) / len(text.split())\n
\n\n
def count_words_per_sentence(text):\n    return len(text.split()) / len(text.split('.'))\n
\n\n
def count_sentences(text):\n    return len(text.split('.'))\n
\n\n
def count_links(text):\n    return len(re.findall(r'\\[.*\\]\\(.*\\)', text))\n
\n\n
def count_images(text):\n    return len(re.findall(r'!\\[.*\\]\\(.*\\)', text))\n
\n\n
def count_headings(text):\n    return len(re.findall(r'#+', text))\n
\n\n
def count_lists(text):\n    return len(re.findall(r'\\*', text))\n
\n\n
def count_quotes(text):\n    return len(re.findall(r'\\>', text))\n
\n\n
# Number of markdown cells\nlen(md_cells)\n
\n\n
# Number of code cells\nlen(code_cells)\n
\n\n
# Number of cells\nlen(cells)\n
\n\n
def dollar_signs(text):\n    return len(re.findall(r'\\$[^\\$]+\\$', text))\n
\n\n
def question_marks(text):\n    return len(re.findall(r'\\?', text))\n
\n\n
def exclamation_marks(text):\n    return len(re.findall(r'!', text))\n
\n\n
def periods(text):\n    return len(re.findall(r'\\.', text))\n
\n\n
def commas(text):\n    return len(re.findall(r',', text))\n
\n\n
def colons(text):\n    return len(re.findall(r':', text))\n
\n\n
def semicolons(text):\n
\n
for notebook in notebooks:\n    cell_features = {}\n    for cell in notebooks:\n        if cell['cell_type'] == 'markdown':\n            string = cell['source']\n           cell_features['n_chars'] = len(string)\n           cell_features['n_dollar_signs'] = string.count('$')\n           cell_features['n_slahes'] = string.count('\\\\')            \n
\n\n
for notebook in notebooks:\n    cell_features = {}\n    for cell in notebooks:\n        if cell['cell_type'] == 'markdown':\n            string = cell['source']\n            cell_features['n_urls'] = len(re.findall(r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', string))\n
\n\n
for notebook in notebooks:\n    cell_features = {}\n    for cell in notebooks:\n        if cell['cell_type'] == 'code':\n            cell_features['n_code_cells'] = 1\n
\n\n
for notebook in notebooks:\n    cell_features = {}\n    for cell in notebooks:\n        if cell['cell_type'] == 'markdown':\n            cell_features['n_mark\n
\n

Feature engineering for python code 🐍💻

\n\n
def comment_ratio(text):\n    return len(re.findall(r'#', text)) / len(text.split('\\n'))\n
\n\n
def lines_of_code(text):\n    return len(text.split('\\n'))\n
\n\n
def chars_per_line(text):\n    return len(text) / len(text.split('\\n'))\n
\n\n
def chars_per_cell(text):\n    return len(text)\n
\n\n
def imports(text):\n    return len(re.findall(r'import', text))\n
\n\n
def from_imports(text):\n    return len(re.findall(r'from', text))\n
\n\n
import_ratio = len(import_lines) / len(all_lines)\n
\n\n
comment_ratio = len(comment_lines) / len(all_lines)\n
\n\n
blank_ratio = len(blank_lines) / len(all_lines)\n
\n\n
code_ratio = len(code_lines) / len(all_lines)\n
\n\n
code_per_function = len(code_lines) / len(functions)\n
\n\n
code_per_class = len(code_lines) / len(classes)\n
\n\n
import_ratio = len(unique_imports) / len(all_imports)\n
\n\n
import_ratio = len(unique_imports) / len(all_lines)\n
\n\n
def get_code_length(code):\n    return len(code.split('\\n'))\n
\n\n
def get_num_imports(code):\n    return len(re.findall(r'import', code))\n
\n\n
def get_num_functions(code):\n    return len(re.findall(r'def', code))\n
\n\n
def get_num_classes(code):\n    return len(re.findall(r'class', code))\n
\n\n
def get_num_comments(code):\n    return len(re.findall(r'#', code))\n
\n\n
def get_num_prints(code):\n    return len(re.findall(r'print', code))\n
\n\n
def get_code_token_diversity(code):\n    tokens = re.findall(r'\\w+', code)\n    return len(set(tokens)) / len(tokens)\n
\n\n
def get_num_unique_imports(code):\n    imports = re.findall(r'import', code)\n    return len(set(imports))\n
\n\n
def get_num_unique_functions(code):\n    functions = re.findall(r'def', code)\n    return len(set(functions))\n
\n\n
def get_num_unique_classes(code):\n    classes = re.findall(r'class', code)\n    return len(set(classes))\n
\n\n
def get_num_unique_comments(code):\n    comments = re.findall(r'#', code)\n    return len(set(comments))\n
\n\n
# Number of sentences in a python code cell\ndef count_sentences(cell):\n    return len(re.findall(r'[^\\s](\\s*[^\\s]+)+[.!?]', cell))\n
\n\n
# Number of words in a python code cell\ndef count_words(cell):\n    return len(re.findall(r'\\w+', cell))\n
\n\n
# Number of characters in a python code cell\ndef count_characters(cell):\n    return len(cell)\n
\n\n
# Number of unique words in a python code cell\ndef count_unique_words(cell):\n    return len(set(re.findall(r'\\w+', cell)))\n
\n\n
# Number of unique characters in a python code cell\ndef count_unique_characters(cell):\n    return len\n
\n\n
from code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity()\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n
\n\n
from code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity(model_path='path/to/model')\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n
\n\n
from code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity(model_path='path/to/model', tokenizer_path='path/to/tokenizer')\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n
\n

.. … . .

", "messages": [ { "id": "1810808", "postDate": "06/03/2022 23:16:22", "content": "

Sharing some ideas: feature engineering!

\n

In this competition, there are two main features: markdown and python code.
\nI think the most important thing is to find a way to extract useful information from them.

\n

Here are some ideas:

\n

Feature engineering for markdown cells 📚

\n\n
def count_escaped_characters(text):\n    return len(re.findall(r'\\\\', text))\n
\n\n
def count_words(text):\n    return len(text.split())\n
\n\n
def count_characters(text):\n    return len(text)\n
\n\n
def count_characters_per_word(text):\n    return len(text) / len(text.split())\n
\n\n
def count_words_per_sentence(text):\n    return len(text.split()) / len(text.split('.'))\n
\n\n
def count_sentences(text):\n    return len(text.split('.'))\n
\n\n
def count_links(text):\n    return len(re.findall(r'\\[.*\\]\\(.*\\)', text))\n
\n\n
def count_images(text):\n    return len(re.findall(r'!\\[.*\\]\\(.*\\)', text))\n
\n\n
def count_headings(text):\n    return len(re.findall(r'#+', text))\n
\n\n
def count_lists(text):\n    return len(re.findall(r'\\*', text))\n
\n\n
def count_quotes(text):\n    return len(re.findall(r'\\>', text))\n
\n\n
# Number of markdown cells\nlen(md_cells)\n
\n\n
# Number of code cells\nlen(code_cells)\n
\n\n
# Number of cells\nlen(cells)\n
\n\n
def dollar_signs(text):\n    return len(re.findall(r'\\$[^\\$]+\\$', text))\n
\n\n
def question_marks(text):\n    return len(re.findall(r'\\?', text))\n
\n\n
def exclamation_marks(text):\n    return len(re.findall(r'!', text))\n
\n\n
def periods(text):\n    return len(re.findall(r'\\.', text))\n
\n\n
def commas(text):\n    return len(re.findall(r',', text))\n
\n\n
def colons(text):\n    return len(re.findall(r':', text))\n
\n\n
def semicolons(text):\n
\n
for notebook in notebooks:\n    cell_features = {}\n    for cell in notebooks:\n        if cell['cell_type'] == 'markdown':\n            string = cell['source']\n           cell_features['n_chars'] = len(string)\n           cell_features['n_dollar_signs'] = string.count('$')\n           cell_features['n_slahes'] = string.count('\\\\')            \n
\n\n
for notebook in notebooks:\n    cell_features = {}\n    for cell in notebooks:\n        if cell['cell_type'] == 'markdown':\n            string = cell['source']\n            cell_features['n_urls'] = len(re.findall(r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', string))\n
\n\n
for notebook in notebooks:\n    cell_features = {}\n    for cell in notebooks:\n        if cell['cell_type'] == 'code':\n            cell_features['n_code_cells'] = 1\n
\n\n
for notebook in notebooks:\n    cell_features = {}\n    for cell in notebooks:\n        if cell['cell_type'] == 'markdown':\n            cell_features['n_mark\n
\n

Feature engineering for python code 🐍💻

\n\n
def comment_ratio(text):\n    return len(re.findall(r'#', text)) / len(text.split('\\n'))\n
\n\n
def lines_of_code(text):\n    return len(text.split('\\n'))\n
\n\n
def chars_per_line(text):\n    return len(text) / len(text.split('\\n'))\n
\n\n
def chars_per_cell(text):\n    return len(text)\n
\n\n
def imports(text):\n    return len(re.findall(r'import', text))\n
\n\n
def from_imports(text):\n    return len(re.findall(r'from', text))\n
\n\n
import_ratio = len(import_lines) / len(all_lines)\n
\n\n
comment_ratio = len(comment_lines) / len(all_lines)\n
\n\n
blank_ratio = len(blank_lines) / len(all_lines)\n
\n\n
code_ratio = len(code_lines) / len(all_lines)\n
\n\n
code_per_function = len(code_lines) / len(functions)\n
\n\n
code_per_class = len(code_lines) / len(classes)\n
\n\n
import_ratio = len(unique_imports) / len(all_imports)\n
\n\n
import_ratio = len(unique_imports) / len(all_lines)\n
\n\n
def get_code_length(code):\n    return len(code.split('\\n'))\n
\n\n
def get_num_imports(code):\n    return len(re.findall(r'import', code))\n
\n\n
def get_num_functions(code):\n    return len(re.findall(r'def', code))\n
\n\n
def get_num_classes(code):\n    return len(re.findall(r'class', code))\n
\n\n
def get_num_comments(code):\n    return len(re.findall(r'#', code))\n
\n\n
def get_num_prints(code):\n    return len(re.findall(r'print', code))\n
\n\n
def get_code_token_diversity(code):\n    tokens = re.findall(r'\\w+', code)\n    return len(set(tokens)) / len(tokens)\n
\n\n
def get_num_unique_imports(code):\n    imports = re.findall(r'import', code)\n    return len(set(imports))\n
\n\n
def get_num_unique_functions(code):\n    functions = re.findall(r'def', code)\n    return len(set(functions))\n
\n\n
def get_num_unique_classes(code):\n    classes = re.findall(r'class', code)\n    return len(set(classes))\n
\n\n
def get_num_unique_comments(code):\n    comments = re.findall(r'#', code)\n    return len(set(comments))\n
\n\n
# Number of sentences in a python code cell\ndef count_sentences(cell):\n    return len(re.findall(r'[^\\s](\\s*[^\\s]+)+[.!?]', cell))\n
\n\n
# Number of words in a python code cell\ndef count_words(cell):\n    return len(re.findall(r'\\w+', cell))\n
\n\n
# Number of characters in a python code cell\ndef count_characters(cell):\n    return len(cell)\n
\n\n
# Number of unique words in a python code cell\ndef count_unique_words(cell):\n    return len(set(re.findall(r'\\w+', cell)))\n
\n\n
# Number of unique characters in a python code cell\ndef count_unique_characters(cell):\n    return len\n
\n\n
from code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity()\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n
\n\n
from code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity(model_path='path/to/model')\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n
\n\n
from code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity(model_path='path/to/model', tokenizer_path='path/to/tokenizer')\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n
\n

.. … . .

", "rawMarkdown": "### Sharing some ideas: feature engineering!\n\n\n\n\nIn this competition, there are two main features: markdown and python code. \nI think the most important thing is to find a way to extract useful information from them.\n\n\nHere are some ideas:\n\n\n#### Feature engineering for markdown cells 📚\n\n\n- Average number of escaped characters (such as \\* for a asterisk, \\** for bold text).\n\n```python\ndef count_escaped_characters(text):\n return len(re.findall(r'\\\\', text))\n```\n\n- Average number of words in a markdown cell.\n\n```python\ndef count_words(text):\n return len(text.split())\n```\n\n- Average number of characters in a markdown cell.\n\n```python\ndef count_characters(text):\n return len(text)\n```\n\n- Average number of characters in a word.\n\n```python\ndef count_characters_per_word(text):\n return len(text) / len(text.split())\n```\n\n- Average number of words in a sentence.\n\n```python\ndef count_words_per_sentence(text):\n return len(text.split()) / len(text.split('.'))\n```\n\n- Average number of sentences in a markdown cell.\n\n```python\ndef count_sentences(text):\n return len(text.split('.'))\n```\n\n- Average number of links in a markdown cell.\n\n```python\ndef count_links(text):\n return len(re.findall(r'\\[.*\\]\\(.*\\)', text))\n```\n\n- Average number of images in a markdown cell.\n\n```python\ndef count_images(text):\n return len(re.findall(r'!\\[.*\\]\\(.*\\)', text))\n```\n\n- Average number of headings in a markdown cell.\n\n```python\ndef count_headings(text):\n return len(re.findall(r'#+', text))\n```\n\n- Average number of lists in a markdown cell.\n\n```python\ndef count_lists(text):\n return len(re.findall(r'\\*', text))\n```\n\n- Average number of quotes in a markdown cell.\n\n```python\ndef count_quotes(text):\n return len(re.findall(r'\\>', text))\n```\n\n- Length of the markdown cells\n\n```python\n# Number of markdown cells\nlen(md_cells)\n```\n\n- Length of the code cells\n\n```python\n# Number of code cells\nlen(code_cells)\n```\n\n- Length of the cells\n\n```python\n# Number of cells\nlen(cells)\n```\n\n- Average number of dollar signs (such as math equations)\n\n```python\ndef dollar_signs(text):\n return len(re.findall(r'\\$[^\\$]+\\$', text))\n```\n\n- Average number of question marks\n\n```python\ndef question_marks(text):\n return len(re.findall(r'\\?', text))\n```\n\n- Average number of exclamation marks\n\n```python\ndef exclamation_marks(text):\n return len(re.findall(r'!', text))\n```\n\n- Average number of periods\n\n```python\ndef periods(text):\n return len(re.findall(r'\\.', text))\n```\n\n- Average number of commas\n\n```python\ndef commas(text):\n return len(re.findall(r',', text))\n```\n\n- Average number of colons\n\n```python\ndef colons(text):\n return len(re.findall(r':', text))\n```\n\n- Average number of semicolons\n\n```python\ndef semicolons(text):\n```\n\n```python\nfor notebook in notebooks:\n cell_features = {}\n for cell in notebooks:\n if cell['cell_type'] == 'markdown':\n string = cell['source']\n cell_features['n_chars'] = len(string)\n cell_features['n_dollar_signs'] = string.count('$')\n cell_features['n_slahes'] = string.count('\\\\') \n```\n\n\n- Average number of URLs\n\n```python\nfor notebook in notebooks:\n cell_features = {}\n for cell in notebooks:\n if cell['cell_type'] == 'markdown':\n string = cell['source']\n cell_features['n_urls'] = len(re.findall(r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', string))\n```\n\n- Average number of code cells\n\n```python\nfor notebook in notebooks:\n cell_features = {}\n for cell in notebooks:\n if cell['cell_type'] == 'code':\n cell_features['n_code_cells'] = 1\n```\n\n- Average number of markdown cells\n\n```python\nfor notebook in notebooks:\n cell_features = {}\n for cell in notebooks:\n if cell['cell_type'] == 'markdown':\n cell_features['n_mark\n```\n\n\n#### Feature engineering for python code 🐍💻\n\n- Ratio of number of commented lines relative to all lines of code\n\n```python\ndef comment_ratio(text):\n return len(re.findall(r'#', text)) / len(text.split('\\n'))\n```\n\n- Average number of lines of code per cell\n\n```python\ndef lines_of_code(text):\n return len(text.split('\\n'))\n```\n\n- Average number of characters per line of code\n\n```python\ndef chars_per_line(text):\n return len(text) / len(text.split('\\n'))\n```\n\n- Average number of characters per cell\n\n```python\ndef chars_per_cell(text):\n return len(text)\n```\n\n- Average number of imports\n\n```python\ndef imports(text):\n return len(re.findall(r'import', text))\n```\n\n- Average number of from imports\n\n```python\ndef from_imports(text):\n return len(re.findall(r'from', text))\n```\n\n- Ratio of number of import lines to all lines of code\n\n```python\nimport_ratio = len(import_lines) / len(all_lines)\n```\n\n- Ratio of number of comments to all lines of code\n\n```python\ncomment_ratio = len(comment_lines) / len(all_lines)\n```\n\n- Ratio of number of blank lines to all lines of code\n\n```python\nblank_ratio = len(blank_lines) / len(all_lines)\n```\n\n- Ratio of number of lines of code to all lines of code\n\n```python\ncode_ratio = len(code_lines) / len(all_lines)\n```\n\n- Ratio of number of lines of code to number of functions\n\n```python\ncode_per_function = len(code_lines) / len(functions)\n```\n\n- Ratio of number of lines of code to number of classes\n\n```python\ncode_per_class = len(code_lines) / len(classes)\n```\n\n- Ratio of unique imports to all import lines\n\n```python\nimport_ratio = len(unique_imports) / len(all_imports)\n```\n\n- Ratio of unique imports to all lines\n\n```python\nimport_ratio = len(unique_imports) / len(all_lines)\n```\n\n- Length of the python code cells\n\n```python\ndef get_code_length(code):\n return len(code.split('\\n'))\n```\n\n- Number of imports\n\n```python\ndef get_num_imports(code):\n return len(re.findall(r'import', code))\n```\n\n- Number of functions\n\n```python\ndef get_num_functions(code):\n return len(re.findall(r'def', code))\n```\n\n- Number of classes\n\n```python\ndef get_num_classes(code):\n return len(re.findall(r'class', code))\n```\n\n- Number of comments\n\n```python\ndef get_num_comments(code):\n return len(re.findall(r'#', code))\n```\n\n- Number of print statements\n\n```python\ndef get_num_prints(code):\n return len(re.findall(r'print', code))\n```\n\n- Code token diversity in the python code cells\n```python\ndef get_code_token_diversity(code):\n tokens = re.findall(r'\\w+', code)\n return len(set(tokens)) / len(tokens)\n```\n\n- Number of unique imports\n\n```python\ndef get_num_unique_imports(code):\n imports = re.findall(r'import', code)\n return len(set(imports))\n```\n\n- Number of unique functions\n\n```python\ndef get_num_unique_functions(code):\n functions = re.findall(r'def', code)\n return len(set(functions))\n```\n\n- Number of unique classes\n\n```python\ndef get_num_unique_classes(code):\n classes = re.findall(r'class', code)\n return len(set(classes))\n```\n\n- Number of unique comments\n\n```python\ndef get_num_unique_comments(code):\n comments = re.findall(r'#', code)\n return len(set(comments))\n```\n\n\n- Number of sentences in a python code cell\n\n```python\n# Number of sentences in a python code cell\ndef count_sentences(cell):\n return len(re.findall(r'[^\\s](\\s*[^\\s]+)+[.!?]', cell))\n```\n\n- Number of words in a python code cell\n\n```python\n# Number of words in a python code cell\ndef count_words(cell):\n return len(re.findall(r'\\w+', cell))\n```\n\n- Number of characters in a python code cell\n\n```python\n# Number of characters in a python code cell\ndef count_characters(cell):\n return len(cell)\n```\n\n- Number of unique words in a python code cell\n\n```python\n# Number of unique words in a python code cell\ndef count_unique_words(cell):\n return len(set(re.findall(r'\\w+', cell)))\n```\n\n- Number of unique characters in a python code cell\n\n```python\n# Number of unique characters in a python code cell\ndef count_unique_characters(cell):\n return len\n```\n\n- Textual similarity between markdowns and codes\n\n```python\nfrom code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity()\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n```\n\n- Textual similarity between markdowns and codes with pre-trained model\n\n```python\nfrom code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity(model_path='path/to/model')\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n```\n\n- Textual similarity between markdowns and codes with pre-trained model and custom tokenizer\n\n```python\nfrom code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity(model_path='path/to/model', tokenizer_path='path/to/tokenizer')\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n```\n\n.. ... . .", "votes": null }, { "id": "1811132", "postDate": "06/04/2022 11:03:46", "content": "

A good list. In order to be more efficient many of those features can be calculated in a single pass especially single tokens or characters.

\n
>>> from collections import Counter\n>>> some_text = \"abc def\"\n>>> tokens = some_text.split()\n>>> tokens\n['abc', 'def']\n>>> count_tokens = Counter(tokens)\n>>> count_tokens\nCounter({'abc': 1, 'def': 1})\n>>> count_chars = Counter(list(some_text))\n>>> count_chars\nCounter({'a': 1, 'b': 1, 'c': 1, ' ': 1, 'd': 1, 'e': 1, 'f': 1})\n
\n

So you can have 2 counters: characters and tokens.

\n

Also worth mentioning that you can just use CountVectorizer and CountVectorizer(analyzer=\"char\") to achieve similar thing.

", "rawMarkdown": "A good list. In order to be more efficient many of those features can be calculated in a single pass especially single tokens or characters.\n\n```\n>>> from collections import Counter\n>>> some_text = \"abc def\"\n>>> tokens = some_text.split()\n>>> tokens\n['abc', 'def']\n>>> count_tokens = Counter(tokens)\n>>> count_tokens\nCounter({'abc': 1, 'def': 1})\n>>> count_chars = Counter(list(some_text))\n>>> count_chars\nCounter({'a': 1, 'b': 1, 'c': 1, ' ': 1, 'd': 1, 'e': 1, 'f': 1})\n```\n\nSo you can have 2 counters: characters and tokens. \n\nAlso worth mentioning that you can just use CountVectorizer and CountVectorizer(analyzer=\"char\") to achieve similar thing.", "votes": null }, { "id": "1812820", "postDate": "06/06/2022 08:50:21", "content": "

You should put this in a notebook :-)

", "rawMarkdown": "You should put this in a notebook :-)", "votes": null }, { "id": "1815016", "postDate": "06/08/2022 15:05:32", "content": "

I will, my friend. Working on it as we speak.
\nbtw: You got some great examples of feature engineering on your notebooks :)

", "rawMarkdown": "I will, my friend. Working on it as we speak. \nbtw: You got some great examples of feature engineering on your notebooks :)", "votes": null }, { "id": "1859843", "postDate": "07/18/2022 02:20:31", "content": "

Definitely Bookmarking this!

", "rawMarkdown": "Definitely Bookmarking this!", "votes": null }, { "id": "1873023", "postDate": "07/27/2022 11:49:33", "content": "

Thanks for sharing good content!

", "rawMarkdown": "Thanks for sharing good content!", "votes": null }, { "id": "1873221", "postDate": "07/27/2022 13:51:53", "content": "

Thank you for sharing; interesting read! Upvoted & bookmarked.

", "rawMarkdown": "Thank you for sharing; interesting read! Upvoted & bookmarked.", "votes": null } ], "comments": [ { "id": 1811132, "author_name": "paweljankiewicz", "author_url": "", "post_date": "06/04/2022 11:03:46", "content": "

A good list. In order to be more efficient many of those features can be calculated in a single pass especially single tokens or characters.

\n
>>> from collections import Counter\n>>> some_text = \"abc def\"\n>>> tokens = some_text.split()\n>>> tokens\n['abc', 'def']\n>>> count_tokens = Counter(tokens)\n>>> count_tokens\nCounter({'abc': 1, 'def': 1})\n>>> count_chars = Counter(list(some_text))\n>>> count_chars\nCounter({'a': 1, 'b': 1, 'c': 1, ' ': 1, 'd': 1, 'e': 1, 'f': 1})\n
\n

So you can have 2 counters: characters and tokens.

\n

Also worth mentioning that you can just use CountVectorizer and CountVectorizer(analyzer=\"char\") to achieve similar thing.

", "votes": null, "replies": [] }, { "id": 1812820, "author_name": "lucasmorin", "author_url": "", "post_date": "06/06/2022 08:50:21", "content": "

You should put this in a notebook :-)

", "votes": null, "replies": [ { "id": 1815016, "author_name": "thedevastator", "author_url": "", "post_date": "06/08/2022 15:05:32", "content": "

I will, my friend. Working on it as we speak.
\nbtw: You got some great examples of feature engineering on your notebooks :)

", "votes": null, "replies": [] } ] }, { "id": 1859843, "author_name": "apurbapandey", "author_url": "", "post_date": "07/18/2022 02:20:31", "content": "

Definitely Bookmarking this!

", "votes": null, "replies": [] }, { "id": 1873023, "author_name": "yumyeom", "author_url": "", "post_date": "07/27/2022 11:49:33", "content": "

Thanks for sharing good content!

", "votes": null, "replies": [] }, { "id": 1873221, "author_name": "bellapisani", "author_url": "", "post_date": "07/27/2022 13:51:53", "content": "

Thank you for sharing; interesting read! Upvoted & bookmarked.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1810808": "### Sharing some ideas: feature engineering!\n\n\n\n\nIn this competition, there are two main features: markdown and python code. \nI think the most important thing is to find a way to extract useful information from them.\n\n\nHere are some ideas:\n\n\n#### Feature engineering for markdown cells 📚\n\n\n- Average number of escaped characters (such as \\* for a asterisk, \\** for bold text).\n\n```python\ndef count_escaped_characters(text):\n return len(re.findall(r'\\\\', text))\n```\n\n- Average number of words in a markdown cell.\n\n```python\ndef count_words(text):\n return len(text.split())\n```\n\n- Average number of characters in a markdown cell.\n\n```python\ndef count_characters(text):\n return len(text)\n```\n\n- Average number of characters in a word.\n\n```python\ndef count_characters_per_word(text):\n return len(text) / len(text.split())\n```\n\n- Average number of words in a sentence.\n\n```python\ndef count_words_per_sentence(text):\n return len(text.split()) / len(text.split('.'))\n```\n\n- Average number of sentences in a markdown cell.\n\n```python\ndef count_sentences(text):\n return len(text.split('.'))\n```\n\n- Average number of links in a markdown cell.\n\n```python\ndef count_links(text):\n return len(re.findall(r'\\[.*\\]\\(.*\\)', text))\n```\n\n- Average number of images in a markdown cell.\n\n```python\ndef count_images(text):\n return len(re.findall(r'!\\[.*\\]\\(.*\\)', text))\n```\n\n- Average number of headings in a markdown cell.\n\n```python\ndef count_headings(text):\n return len(re.findall(r'#+', text))\n```\n\n- Average number of lists in a markdown cell.\n\n```python\ndef count_lists(text):\n return len(re.findall(r'\\*', text))\n```\n\n- Average number of quotes in a markdown cell.\n\n```python\ndef count_quotes(text):\n return len(re.findall(r'\\>', text))\n```\n\n- Length of the markdown cells\n\n```python\n# Number of markdown cells\nlen(md_cells)\n```\n\n- Length of the code cells\n\n```python\n# Number of code cells\nlen(code_cells)\n```\n\n- Length of the cells\n\n```python\n# Number of cells\nlen(cells)\n```\n\n- Average number of dollar signs (such as math equations)\n\n```python\ndef dollar_signs(text):\n return len(re.findall(r'\\$[^\\$]+\\$', text))\n```\n\n- Average number of question marks\n\n```python\ndef question_marks(text):\n return len(re.findall(r'\\?', text))\n```\n\n- Average number of exclamation marks\n\n```python\ndef exclamation_marks(text):\n return len(re.findall(r'!', text))\n```\n\n- Average number of periods\n\n```python\ndef periods(text):\n return len(re.findall(r'\\.', text))\n```\n\n- Average number of commas\n\n```python\ndef commas(text):\n return len(re.findall(r',', text))\n```\n\n- Average number of colons\n\n```python\ndef colons(text):\n return len(re.findall(r':', text))\n```\n\n- Average number of semicolons\n\n```python\ndef semicolons(text):\n```\n\n```python\nfor notebook in notebooks:\n cell_features = {}\n for cell in notebooks:\n if cell['cell_type'] == 'markdown':\n string = cell['source']\n cell_features['n_chars'] = len(string)\n cell_features['n_dollar_signs'] = string.count('$')\n cell_features['n_slahes'] = string.count('\\\\') \n```\n\n\n- Average number of URLs\n\n```python\nfor notebook in notebooks:\n cell_features = {}\n for cell in notebooks:\n if cell['cell_type'] == 'markdown':\n string = cell['source']\n cell_features['n_urls'] = len(re.findall(r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', string))\n```\n\n- Average number of code cells\n\n```python\nfor notebook in notebooks:\n cell_features = {}\n for cell in notebooks:\n if cell['cell_type'] == 'code':\n cell_features['n_code_cells'] = 1\n```\n\n- Average number of markdown cells\n\n```python\nfor notebook in notebooks:\n cell_features = {}\n for cell in notebooks:\n if cell['cell_type'] == 'markdown':\n cell_features['n_mark\n```\n\n\n#### Feature engineering for python code 🐍💻\n\n- Ratio of number of commented lines relative to all lines of code\n\n```python\ndef comment_ratio(text):\n return len(re.findall(r'#', text)) / len(text.split('\\n'))\n```\n\n- Average number of lines of code per cell\n\n```python\ndef lines_of_code(text):\n return len(text.split('\\n'))\n```\n\n- Average number of characters per line of code\n\n```python\ndef chars_per_line(text):\n return len(text) / len(text.split('\\n'))\n```\n\n- Average number of characters per cell\n\n```python\ndef chars_per_cell(text):\n return len(text)\n```\n\n- Average number of imports\n\n```python\ndef imports(text):\n return len(re.findall(r'import', text))\n```\n\n- Average number of from imports\n\n```python\ndef from_imports(text):\n return len(re.findall(r'from', text))\n```\n\n- Ratio of number of import lines to all lines of code\n\n```python\nimport_ratio = len(import_lines) / len(all_lines)\n```\n\n- Ratio of number of comments to all lines of code\n\n```python\ncomment_ratio = len(comment_lines) / len(all_lines)\n```\n\n- Ratio of number of blank lines to all lines of code\n\n```python\nblank_ratio = len(blank_lines) / len(all_lines)\n```\n\n- Ratio of number of lines of code to all lines of code\n\n```python\ncode_ratio = len(code_lines) / len(all_lines)\n```\n\n- Ratio of number of lines of code to number of functions\n\n```python\ncode_per_function = len(code_lines) / len(functions)\n```\n\n- Ratio of number of lines of code to number of classes\n\n```python\ncode_per_class = len(code_lines) / len(classes)\n```\n\n- Ratio of unique imports to all import lines\n\n```python\nimport_ratio = len(unique_imports) / len(all_imports)\n```\n\n- Ratio of unique imports to all lines\n\n```python\nimport_ratio = len(unique_imports) / len(all_lines)\n```\n\n- Length of the python code cells\n\n```python\ndef get_code_length(code):\n return len(code.split('\\n'))\n```\n\n- Number of imports\n\n```python\ndef get_num_imports(code):\n return len(re.findall(r'import', code))\n```\n\n- Number of functions\n\n```python\ndef get_num_functions(code):\n return len(re.findall(r'def', code))\n```\n\n- Number of classes\n\n```python\ndef get_num_classes(code):\n return len(re.findall(r'class', code))\n```\n\n- Number of comments\n\n```python\ndef get_num_comments(code):\n return len(re.findall(r'#', code))\n```\n\n- Number of print statements\n\n```python\ndef get_num_prints(code):\n return len(re.findall(r'print', code))\n```\n\n- Code token diversity in the python code cells\n```python\ndef get_code_token_diversity(code):\n tokens = re.findall(r'\\w+', code)\n return len(set(tokens)) / len(tokens)\n```\n\n- Number of unique imports\n\n```python\ndef get_num_unique_imports(code):\n imports = re.findall(r'import', code)\n return len(set(imports))\n```\n\n- Number of unique functions\n\n```python\ndef get_num_unique_functions(code):\n functions = re.findall(r'def', code)\n return len(set(functions))\n```\n\n- Number of unique classes\n\n```python\ndef get_num_unique_classes(code):\n classes = re.findall(r'class', code)\n return len(set(classes))\n```\n\n- Number of unique comments\n\n```python\ndef get_num_unique_comments(code):\n comments = re.findall(r'#', code)\n return len(set(comments))\n```\n\n\n- Number of sentences in a python code cell\n\n```python\n# Number of sentences in a python code cell\ndef count_sentences(cell):\n return len(re.findall(r'[^\\s](\\s*[^\\s]+)+[.!?]', cell))\n```\n\n- Number of words in a python code cell\n\n```python\n# Number of words in a python code cell\ndef count_words(cell):\n return len(re.findall(r'\\w+', cell))\n```\n\n- Number of characters in a python code cell\n\n```python\n# Number of characters in a python code cell\ndef count_characters(cell):\n return len(cell)\n```\n\n- Number of unique words in a python code cell\n\n```python\n# Number of unique words in a python code cell\ndef count_unique_words(cell):\n return len(set(re.findall(r'\\w+', cell)))\n```\n\n- Number of unique characters in a python code cell\n\n```python\n# Number of unique characters in a python code cell\ndef count_unique_characters(cell):\n return len\n```\n\n- Textual similarity between markdowns and codes\n\n```python\nfrom code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity()\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n```\n\n- Textual similarity between markdowns and codes with pre-trained model\n\n```python\nfrom code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity(model_path='path/to/model')\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n```\n\n- Textual similarity between markdowns and codes with pre-trained model and custom tokenizer\n\n```python\nfrom code_markdown_similarity import CodeMarkdownSimilarity\n\n# Load the model\nmodel = CodeMarkdownSimilarity(model_path='path/to/model', tokenizer_path='path/to/tokenizer')\n\n# Get the similarity score\nmodel.get_similarity_score(code, markdown)\n```\n\n.. ... . .", "1811132": "A good list. In order to be more efficient many of those features can be calculated in a single pass especially single tokens or characters.\n\n```\n>>> from collections import Counter\n>>> some_text = \"abc def\"\n>>> tokens = some_text.split()\n>>> tokens\n['abc', 'def']\n>>> count_tokens = Counter(tokens)\n>>> count_tokens\nCounter({'abc': 1, 'def': 1})\n>>> count_chars = Counter(list(some_text))\n>>> count_chars\nCounter({'a': 1, 'b': 1, 'c': 1, ' ': 1, 'd': 1, 'e': 1, 'f': 1})\n```\n\nSo you can have 2 counters: characters and tokens. \n\nAlso worth mentioning that you can just use CountVectorizer and CountVectorizer(analyzer=\"char\") to achieve similar thing.", "1812820": "You should put this in a notebook :-)", "1815016": "I will, my friend. Working on it as we speak. \nbtw: You got some great examples of feature engineering on your notebooks :)", "1859843": "Definitely Bookmarking this!", "1873023": "Thanks for sharing good content!", "1873221": "Thank you for sharing; interesting read! Upvoted & bookmarked." }, "source": "meta" }