NLP

Index:

1. Source : M.tech – Prof. Naveen Kumar / CP Magnani [recommended]
   1. Applications – 26-July-2026
   2. History – 2-Aug-2026
   3. Today’s lec 16-Aug-2026
2. Source : Deep learning
   1. NLP with Classification and Vector Spaces
   2. NLP with Probabilistic Models
   3. NLP with Sequence Models
   4. NLP with Attention Models

---

Introduction & Real-World Relevance

Everyday NLP Applications:

Everyday tools relying on NLP:

Ranging from messaging apps (WhatsApp) to autocomplete in Gmail/search engines , and modern Generative AI tools (ChatGPT, Claude, Gemini, Copilot, Cursor, etc.).

Evolution of AI/NLP:

The rapid advance in NLP/AI is driven by three main factors:

1. Massive availability of training data via the internet and social media.
2. Massive increase in computing power (e.g., NVIDIA GPUs).
3. Algorithmic breakthroughs.

Reference :

1. Speech and Language Processing by Daniel Jurafsky & James H. Martin (Stanford University)

---

Index

1. History
2. Fundamental Concepts
3. Why NLP is Hard?

Real-World Relevance & Modern AI Landscape

• Ubiquity of NLP: Everyday interactions rely heavily on NLP algorithms, including messaging auto-complete (WhatsApp), Gmail Smart Compose, and search engine suggestions.
• Modern Generative AI Tools: Discussion of mainstream AI products and developer assistants widely used in corporate and startup environments (e.g., ChatGPT, Claude, Gemini, Copilot, Cursor, Windsurf, Codex).
• Multi-modality: The shift from text-only models to multimodal architectures capable of seamlessly handling text, image, and audio inputs/outputs.

Historical Evolution of Natural Language Processing

the 80+ year historical trajectory of NLP research:

1. 1950s–1960s (Rule-Based Era): Early work by Alan Turing (Turing Test), rule-based systems, and formal syntactic grammars.
2. 1970s–1980s (Statistical Approaches): Shift toward statistical models and probabilistic methods.
3. Early 2000s (Corpus & Neural Foundations): Growth of large text corpora via the internet; introduction of the first Neural Language Model in 2003.
4. 2013 (Distributional Word Embeddings Breakthrough): Release of Google’s landmark paper on Word2Vec (Distributional Word Representations), introducing Skip-Gram and Continuous Bag-of-Words (CBOW) models, alongside Stanford’s GloVe (Global Vectors).
5. 2017 (Transformer Architectural Shift): Publication of Google’s seminal paper “Attention Is All You Need”, introducing the Transformer Architecture and shifting the paradigm away from Recurrent Neural Networks (RNNs/LSTMs).
6. 2022–Present (LLM & Agentic AI Era): Emergence of Large Language Models (LLMs), Small Language Models (SLMs), Agentic systems with memory mechanisms, and Multimodal models.

Foundations of Vector Semantics & Embeddings

• The Core Premise: Computer hardware and algorithms operate strictly on numbers (0s and 1s) and cannot directly process categorical text or raw words.
• Word Embeddings / Vector Semantics: The technique of converting words or categorical tokens into dense, continuous numerical vector space representations while preserving semantic meaning based on surrounding context.

Fundamental NLP Concepts & Mechanics

1. Numerical Representation / Word Embeddings: Computers operate strictly on numbers (0s and 1s). Word embeddings / vector semantics convert categorical text into numerical vectors so machines can process language.
2. Language Modeling: Predicting the probability of the next word given a preceding context. Real-world examples include Gmail smart compose and WhatsApp autocomplete predictions.
3. NLP Pipeline & Tasks: Standard pipeline processing steps, including text chunking, Parts-of-Speech (POS) tagging, entity recognition, and sentence boundary detection.

Why NLP is Hard: Linguistic Ambiguity

human language is inherently complex due to ambiguity at multiple levels:

1. Lexical Ambiguity (Homonymy & Polysemy): Words having multiple meanings based on context (e.g., “bank” as a financial institution vs. a river bank).
2. Grammatical / POS Tagging Ambiguity: Words changing their part of speech depending on usage (e.g., “book” as a verb in “I will book tickets” vs. a noun in “I have a book”).
3. Structural / Syntactic Ambiguity: Multiple structural interpretations of a single sentence (e.g., “I saw a man with a telescope on a hill”—who has the telescope, and who is on the hill?).
4. Co-reference Resolution: Resolving pronouns/articles across multi-turn conversations or sentences (e.g., tracking what “there” or “it” refers to in a chatbot conversation).
5. Sentence Truncation Ambiguity: Rules like splitting by periods (.) fail when encountering abbreviations (e.g., titles like “Dr. L. Naveen Kumar”).

---