Machine learning system design

area: US🇺🇸 date: 2022/10/19 → 2022/12/25

Requirements for ML systems

Reliability Scalability: resource scaling (up-scaling, down-scaling) Maintainability Adaptability

Data

Data is full of biases: from collecting, sampling and labeling

Data models

Relational model document model graph model

Batch processing vs Stream processing

Stream:

1. low latency, but nowadays, streaming technologies are highly scalable and fully distributed, can compute in parallel
2. the strength of streaming processing is in stateful computation

sampling

helps accomplish a task faster and cheaper with less data.

works or necessary when:

1. don’t have access to all real-world data, training on a subset
2. infeasible to process all data you have access to

Common sampling methods:

1. No-probability sampling
   1. convenience sampling, snowball sampling, judgement sampling, quota sampling.
   2. have selection bias
2. Simple random sampling:
   1. rare categories may disappear
3. Stratified sampling
   1. impossible to divide all samples into groups, e.g. multi-label
4. Weighted sampling
   1. allow you to leverage domain expertise
   2. correct data distribution to true data
5. Reservoir sampling
   1. deal with steaming data, maintain a reservoir with k elements
6. Importance sampling
   1. allow us to sample from a distribution when we only have access to another distribution

Labeling

Hand labels

1. expensive
2. poses threat to data privacy
3. slow
4. label multiplicity data lineage: keep track of the origin of each data samples and labels. helps you both flag potential biases in your data and debug your models.

Natural labels

feedback loop length may be long

Deal with lack of label

Weak supervision

• leverage heuristics to obtain noisy labels, simple but powerful paradigm.
• A good way to start to explore the effectiveness of ML without investing too much labeling effort Labeling function encodes features like: keyword heuristic, regular expressions, database lookup, outputs of other models

Semi-supervision

leverage structural assumptions to generate new labels bases on a small set of initial labels.

1. self-training
2. data samples share similar characteristics share the same labels
3. perturbation-based method: small perturbations shouldn’t change a sample’s label Use large evaluation set to select the best model, and continue training it on evaluation set.

Transfer learning

the larger the pre-trained base model, the better its performance ob the downstream tasks

Active learning

ML models can achieve greater accuracy with fewer labels if they can choose which samples to learn from.

1. label samples according to metrics or heuristics, e.g. uncertainty measurement
2. query-by-committee (ensemble method): use multi-models for voting

Class imbalance

• insufficient signal for your model to learn to detect minority classes.
• get stuck in non-optimal solution by exploring a simple heuristic instead of learning underlying data pattern
• leads to asymmetric costs of errors. (e.g. misclassification of cancerous cells is much more dangerous.)

Handle class imbalance

1. use right evaluation metrics
2. data-level methods: resampling (under_sampling: samples of majority class; oversampling: copy minority class)
3. algorithm-level methods: class-balanced loss, focal loss, etc.

Data augmentation

1. simple label-preserving transformations
2. perturbation
3. data synthesis

Data distribution shifts

covariate shift

1. sample selection bias problem
2. training data is artificially altered to make model easier to learn
3. caused by model’s learning process, especially active learning
4. major change in environment or the way your application is used

Detecting data distribution shifts

1. statical methods: metrics, two-sample hypothesis test

Addressing data distribution shifts

• design your system to make it more robust to shifts
• make it easier for it to adapt to shifts

Data leakage

The phenomenon when a form of the label (which is not available during inference) ‘leaks’ into the set of features used for making predictions.

Cautions for data leakage

1. splitting time-correlated data by time, instead of randomly.
2. split data before data analysis and processing, as not gaining infos about test split.
3. check duplicates before splitting / group leakage
4. use statics from only the train split, to scale your features and handle missing values.

Model development

Training

Tips for Model selection

1. Start with the simplest model, which you have a lot room to improve upon with.
2. potential for improvements in the future.
   1. Use learning curves, to know how easy/difficult to achieve further improvements.
3. evaluate trade-off: recall vs precision, latency vs performance, etc.

Ensemble of multi-models for prediction

1. Bootstrap aggregating (bagging): sample data with replacement for different models.
   1. use majority vote of all models, improves unstable methods like neural networks.
2. Boosting
   2. LightGBM
3. Stacking

Debugging ML models

Tips:

1. start with simple and gradually add more components
2. overfit a single batch
3. set a random seed

Distributed training

data & model parallelism

AutoML

1. soft AutoML: Hyper-parameter tuning
   1. as a part of standard pipelines
   2. Graduate student descent (GSD)
   3. popular methods: random search, grid search, Bayesian optimization
   4. sensitive hyper-parameters should be carefully tuned
2. hard AutoML: Architecture search and learned optimizer

Model offline validation

Baselines

random baseline, simple heuristic, zero rule baseline(always predict the most common class), human baseline, existing solutions

differentiate between ‘a good system’ and ‘a useful system’

Evaluation methods

Except performance metrics, we need our model to be robust, fair, calibrated and overall make sense.

1. perturbation tests: add noisy data for testing
2. invariance tests: Exclude sensitive info from features used for training
3. directional expectation tests
4. model calibration
5. slice-based evaluation: per-class, focus more on minority.
   1. focus on over-all performance may blind us to huge potential model improvements.
   2. some subsets of data may be more critical
   3. Simpson’s paradox: aggregation con conceal and contradict actual situation.
   4. help improve performance on both overall and critical data, and help detect potential biases
   5. methods to slice:
      1. heuristics-bases
      2. error analysis
      3. slice finder: beam search, clustering, decision

Monitoring

1. accuracy-related metrics
2. predictions (monitor predictions for data shifts)
3. features
   1. feature validation (table testing), expected schema
   2. two-sample test
4. raw input

Model compression

1. low-rank factorization: compact filters
2. knowledge distillation: sensitive to applications and model architectures
3. Pruning:
   1. remove entire nodes
   2. sparse: find least useful parameters and set them to 0
4. Quantization: straight forward to do, and generalizes over tasks and architectures.
   1. quantization aware training
   2. post-training
   3. fixed-point inference has become standard in the industry
   4. Tensorrt offer post-training quantization for free

Model optimization

1. vectorization, parallelization, loop tiling (change data accessing order in a loop to leverage hardware’s memory layout and cache), operator fusion
2. use ML to optimize: autoTVM