Category: AI

Index

1. Introduction
   1. Types of ML
      1. Based on level of supervision
      2. Based on mode of training
      3. Based on learning mode.
   2. Challenges in Machine learning
      1. Data collection
      2. Insufficient data/labelled data
      3. Non representative data
         • Sampling Noise
         • Sampling Bias
      4. Poor quality data
      5. Irrelevant features
      6. Overfitting
      7. Underfitting
      8. Software Integration
      9. Offline learning & deployment
      10. Cost involved
   3. Application of ML
   4. MLDLC – Machine Learning Development lifecycle
   5. Job Roles
      1. Data Engineer
      2. Data Analyst
      3. Data Scientist
      4. MLE
2. ML Implementation
   1. Data Structure – Tensors (0-D to 5-D Tensors)
   2. Tools and Lib for ML
   3. E2E flow
   4. Step by Step flow
      1. framing the problem & planing the project
         • Netflix usecase

Introduction

Diagram – Hierarchy : AI , ML , DNN

1. Traditional programming – Crisp logic – You write an if/else statement defining anything >30 °C as Hot. Crisp logic is a Traditional programming logic . No data needed.
2. AI is Rule driven which involves fuzzy logic (a human expert must still sit down and write every single rule ).No data needed.
3. ML is data driven – then ML is helpful and data is requried.
4. when we don’t have exact features then comes powerful DNN.

When to use which:

Types Of ML

Based on level of supervision:

1. Supervised
   1. Regression – numerical
   2. Classification
2. Unsupervised
   1. Clustering
   2. Anomaly
   3. Association
3. Semi supervised
4. Reinforcement
   • reward/punishment mechanism

Based on mode of training:

1. Batch/offline
   • Offline learning of ML system
   • When to use? no concept drift e.g. Classification
     • cons – heavy size
2. Online
   • learning in small sizes – chuck of data – can be done on production environment.
   • When to use?
     • Concept drift e,g. stock market etc.
     • Faster solution

Based on learning

• Instance based
  • the system learns the training examples by heart. When a new data point arrives, it doesn’t look at a formula; it looks back at its memorized data and compares the new point to its closest neighbors.
    • example –
      • K-Nearest Neighbors (KNN). If you want to predict if a customer will default on a loan, KNN looks at the K closest customers in its database who have similar income/age and takes a majority vote.
      • SVM – Support vector machine
• Model based
  • the algorithm takes the training data, uncovers the underlying pattern, and condenses that pattern into a fixed mathematical formula (a model). Once the formula is created, the original data can be thrown away.
    • Example- Linear Regression (y = mx + c) or Logistic Regression. Once the model learns the exact values for m and c, it deletes the training data from its memory. To predict y for a new x, it just runs that basic equation.

Challenges in ML

1. Data collection
2. Insufficient data/labelled data
3. Non representative data
   • Sampling Noise
     • Sampling Noise (often called sampling error or variance) is the purely random fluctuation that occurs because you are looking at a smaller sample instead of the entire universe of data.
   • Sampling Bias
     • Sampling Bias occurs when your sample is collected in a way that some members of the intended population are less likely or more likely to be included than others. e.g from 4 countries only view of specific nationality person are taken on – which country will win cricket world-cup.
4. Poor quality data – 60% on time is consumed in correcting poor data – .
5. Irrelevant features – Saying GIGO
   • Feature engineering
6. Overfitting
7. Underfitting
8. Software Integration
   • Integration becomes difficult because not all libraries are natively supported in every other language which is used to run the software
9. Offline learning & deployment
10. Cost involved

Application of ML

1. Product recommendation
2. Chatbot
3. B2B
   • Retail (e-commerce)
     • Before Sale – Predicting which skus will be most in demand
       • Has huge impact financially if correctly predicted – not unnecessary stocking, if wrongly Out of stock issue
       • Usually Companies maintain user profile based on buying interest. used in targeting marketing
       • Positioning of products – Association rule (baby diapers and beer paper)
   • Finance
     • Loan – profile is compared against past defaulters how many changes of overlapping criteria.
     • Sharemarket
   • Transportation
     • Ola/Uber – Price surge. – “Prices are higher because of increased demand“
   • Manufacturing (Tesla car)
     • Predictive maintenance – based on input before product goes off , it is repaired
   • Banking
   • Social Media(Twitter)
     • Sentiment analysis
4. B2C

Machine Learning Development lifecycle

It has around 9 steps (Mind the sequence of 3->4->5 and onwards it comes as is):

1. Frame the problem
2. Data gathering
3. Preprocessing
   • Remove duplicates
   • Remove missing values etc.
4. EDA – Exploratory Data Analysis – EDA is how you find those patterns.
   1. univariate analysis
      • Categorical Data: You use Bar Charts or frequency counts to identify class imbalances (e.g., checking if $95\%$ of rows are “Approve” and only $5\%$ are “Reject”).
      • Numerical Data: You plot Histograms or Kernel Density Estimate (KDE) plots. You are checking for skewness (is the data normally distributed, or heavily shifted?).
   2. bivariate analysis
      • Scatter Plots: To check if two numerical variables have a linear or non-linear relationship.
   3. Outlier detection
5. Feature selection
   • Feature combination – because more the feature more is the cost.
   • Feature scaling
6. Model Training, Analysis and selection
   • ensemble learning – multiple algos combined to give better algo for a problem
7. Testing
8. Deployment
9. Optimisation

Job Roles in the field

1. Data Engineer – Core responsibility is fetching the data from different sources and making one warehouse for OLAP
2. Data Analyst – More like Business Analyst but involve a bit more technicality.
3. Data Scientist – Core responsibility is how to use the data for future business- where as DA gives the historical analysis, this role is more of a Full-stack kind.
4. Machine Learning Engineer – Core responsibility Model optimisation and training , integrating with software , Data Optimisation and deployment of Model etc.

Conclusion :