ML – Techglot

Index

Introduction

Diagram – Hierarchy : AI , ML , DNN

Traditional programming – Crisp logic – You write an if/else statement defining anything >30 °C as Hot. Crisp logic is a Traditional programming logic . No data needed.
AI is Rule driven which involves fuzzy logic (a human expert must still sit down and write every single rule ).No data needed.
ML is data driven – then ML is helpful and data is requried.
when we don’t have exact features then comes powerful DNN.

When to use which:

Types Of ML

Based on level of supervision:

Based on mode of training:

Batch/offline
- Offline learning of ML system
- When to use? no concept drift e.g. Classification
  - cons – heavy size
Online
- learning in small sizes – chuck of data – can be done on production environment.
- When to use?
  - Concept drift e,g. stock market etc.
  - Faster solution

Based on learning

Instance based
- the system learns the training examples by heart. When a new data point arrives, it doesn’t look at a formula; it looks back at its memorized data and compares the new point to its closest neighbors.
  - example –
    - K-Nearest Neighbors (KNN). If you want to predict if a customer will default on a loan, KNN looks at the K closest customers in its database who have similar income/age and takes a majority vote.
    - SVM – Support vector machine
Model based
- the algorithm takes the training data, uncovers the underlying pattern, and condenses that pattern into a fixed mathematical formula (a model). Once the formula is created, the original data can be thrown away.
  - Example- Linear Regression (y = mx + c) or Logistic Regression. Once the model learns the exact values for m and c, it deletes the training data from its memory. To predict y for a new x, it just runs that basic equation.

Challenges in ML

Data collection
Insufficient data/labelled data
Non representative data
- Sampling Noise
  - Sampling Noise (often called sampling error or variance) is the purely random fluctuation that occurs because you are looking at a smaller sample instead of the entire universe of data.
- Sampling Bias
  - Sampling Bias occurs when your sample is collected in a way that some members of the intended population are less likely or more likely to be included than others. e.g from 4 countries only view of specific nationality person are taken on – which country will win cricket world-cup.
Poor quality data – 60% on time is consumed in correcting poor data – .
Irrelevant features – Saying GIGO
- Feature engineering
Overfitting
Underfitting
Software Integration
- Integration becomes difficult because not all libraries are natively supported in every other language which is used to run the software
Offline learning & deployment
Cost involved

Application of ML

Machine Learning Development lifecycle

It has around 9 steps (Mind the sequence of 3->4->5 and onwards it comes as is):

Frame the problem
Data gathering
Preprocessing
- Remove duplicates
- Remove missing values etc.
EDA – Exploratory Data Analysis – EDA is how you find those patterns.
1. univariate analysis
  - Categorical Data: You use Bar Charts or frequency counts to identify class imbalances (e.g., checking if $95\%$ of rows are “Approve” and only $5\%$ are “Reject”).
  - Numerical Data: You plot Histograms or Kernel Density Estimate (KDE) plots. You are checking for skewness (is the data normally distributed, or heavily shifted?).
2. bivariate analysis
  - Scatter Plots: To check if two numerical variables have a linear or non-linear relationship.
3. Outlier detection
Feature selection
- Feature combination – because more the feature more is the cost.
- Feature scaling
Model Training, Analysis and selection
- ensemble learning – multiple algos combined to give better algo for a problem
Testing
Deployment
Optimisation

Job Roles in the field

Data Engineer – Core responsibility is fetching the data from different sources and making one warehouse for OLAP
Data Analyst – More like Business Analyst but involve a bit more technicality.
Data Scientist – Core responsibility is how to use the data for future business- where as DA gives the historical analysis, this role is more of a Full-stack kind.
Machine Learning Engineer – Core responsibility Model optimisation and training , integrating with software , Data Optimisation and deployment of Model etc.

Conclusion :

Pages: 12