20240912 - TA - Lexical + Syntax Analysis

The meeting discussed text preprocessing, sentence normalization, transforming numbers into words, probability models for sentence structure, name entity recognition, and syntax analysis.

The meeting discussed the use of libraries such as NLTK, string, and regular expressions for text preprocessing.
The process of normalizing sentences was explained, including converting text to lowercase and removing numbers and punctuation.
Different techniques for transforming numbers into words were mentioned, but it was noted that specific libraries may be required for different languages.
The meeting discussed probability models for sentence structure and sequence prediction.
Name entity recognition can be achieved through dictionary-based or shallow parsing approaches.
Syntax analysis involves analyzing the structure of sentences and dealing with lexical and structural ambiguity.