-
Notifications
You must be signed in to change notification settings - Fork 44
Expand file tree
/
Copy pathClass_Notes.txt
More file actions
3227 lines (1775 loc) · 70 KB
/
Class_Notes.txt
File metadata and controls
3227 lines (1775 loc) · 70 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
MLOps: Edu
Start Date: 29th Nov. 2025
8:30PM-11:30PM IST - SAT-SUN
####################
Day1: 29th Nov. 2025
####################
- MLOps Introduction :::
Github Account :
https://github.com/
- What is DevOps ???
SDLC : Software Development Life Cycle :
- Requirement Analysis
- Designg & Document
- Code & Development
- Testing
- Implement to Prod
- Monitor/Maintain
- Waterfall:
- It is Linear in fashion.
- It strictly follows Top-Down Approach
- Requirement Analysis
- Designg & Document
- Code & Development
- Testing
- Implement to Prod
- Monitor/Maintain
Software Applications :
- Desktop Applications
- Web Applications
- Mobile Applications
- Embedded Applications
- Desktop Applications :::
SuperMarket Billing System:
--> Functions/Modules
- User Interface Design
- Inventory Mgmt
- Billing Mgmt
- Supplier Mgmt
- Payment Mgmt
- Cash Payment
- Card Payment
- Online Payment
- UPI Payment
Core Project :
- Requirement Analysis --> 12 Months
- Designg & Document
- Code & Development --> 6th Month
- Testing
- Implement to Prod
- Monitor/Maintain
Enhancement Project :
- Requirement Analysis
- Designg & Document
- Code & Development
- Testing
- Implement to Prod
- Monitor/Maintain
AGILE Methodologies :::
- Desktop Applications :::
SuperMarket Billing System:
--> Functions/Modules --> iterations
- User Interface Design
- Inventory Mgmt
- Billing Mgmt
- Supplier Mgmt
- Payment Mgmt
- Cash Payment
- Card Payment
- Online Payment
- UPI Payment
Iteration 1 : User Interface Design
- Requirement Analysis
- Designg & Document
- Code & Development
- Testing
- Implement to Prod with proper manual approvals
- Monitor/Maintain
Iteration 2 : Inventory Mgmt
- Requirement Analysis
- Designg & Document
- Code & Development
- Testing
- Implement to Prod with proper manual approvals
- Monitor/Maintain
nth Iteration : Online Payment
- Requirement Analysis
- Designg & Document
- Code & Development
- Testing
- Implement to Prod with proper manual approvals
- Monitor/Maintain
Using AGILE Methodologies:
- We can Achieve :
- Continuous Development
- Continuous Integration
- Continuous Testing
- Continuous Delivery
- Used to perform Production release with manual approvals.
- We cannot Achieve :
- Continuous Deployment
- Used to perform Production release without any manual approvals.
- What is DevOps ???
- DevOps :::::
DevOps is a Software Development Strategy, which helps to promote the collaboration between the teams like Development Team and Operations Team to achieve Continuous Development, Continuous Integration, Continuous Testing, Continuous Delivery, Continuous Deployment and Continuous Monitoring in more Automated Fashion.
Continuous Delivery & Deployment:
- Both Continuous Delivery & Continuous Deployment are used to release the changes from Non-Prod to Prod.
Continuous Delivery:
- Continuous Delivery Exepects Manual Approvals
- We might experience the Application Down-time
Eg:
Banking Domain :
Online Banking / Credit Card Transaction
Create a Production Release Window:
4 - 6Hrs
Continuous Deployment:
- Never Exepects Manual Approvals
- Deploy the changes to production without any down-time.
Eg:
- Amazon/Netflix/Facebook/Google ::
- How to Implement DevOps :::
- Identify the Teams involved in the process of SDLC :
- DevOps Team
- Infra-Structure Management Team
- Application Development Team
- Testing Team
- Release Management Team
- Production Support Team
- Production Monitoring Team
- IT Security
- Environments :
Non-Prod Prod Environments
Dev Environment
Test Environment
QA
UAT ================> Production Servers - 1,2,3,4,5
- Application Architecture :
- Monolith Application Architecture:
- It Tightly-Coupled Application Architecture
- All the dependencies are tightly coupled
- With the we can achieve only Continuous Delivery.
- We cannot achieve Continuous Deployment
- Micro-Service Based Application Architecture:
- It is Loose-Coupled Application Architecture,
- Each and every function is considered as Micro-Service.
- With this we can achieve Continuous Deployment.
Devops :::
- People
- Strategy/Process
- Tools
####################
Day2: 30th Nov. 2025
####################
MLOps Lifecycle :
MLOps Usecase :
Versioning :
Version Control System using Git :
Git Demo :
ML Project and Packaging ML Model
CICD Pipeline:
- SCM_Checkout ==> VCS ?
- Application Build
- Create Application Artifacts
- Test
- Deploy
How to Implement MLOps :::
- Data Engineers -> Prepare the Source Datasets
- Data Scientists -> Explore the datasets, analyse, check the feasibility of datasets
Perform the training, tuning, and evaluate the Model
- ML Engineers -> Deploy the ML Models to prod -> MLOps Strageies CICD, define best practises
Should have MLOps Knowledge
- Business Stakeholders -> Bring business values using this ML Models
- Data Security/Governance Team --> To Ensure data security & Governance, data privacy
ML Model Use Case :
ML Model for Product Reviews :
E_Commerce Application Eg. www.amazon.com
Product Review:
CICD Pipeline:
- SCM_Checkout ==> VCS ?
- Application Build
- Create Application Artifacts
- Test
- Deploy
Process/Lifecycle of Machine Learning Ops:::
- Data Preparation
- Exploratory Data Analysis (EDA)
- Feature Engineering
- Model Training
- Model Validations
- Model Deployment
- Monitoring
MLOps Use cases:
- Healthcare
- Finance
- Retail Domain - E_Commerce
- Manufacturing
How to Implement MLOps :::
- Vesioning:
Version Control System :
- To Version Control the Changes
- To Track the Changes
Types of VCS :
- Centralized VCS:
- Distributed VCS:
GIT :
- Git is an Open-Source Distributed VCS
- To Version Control the Changes
- To Track the Changes
- To perform Parallel Development using Branching Techniques
Lab Requirements:
- Create GitHub Account
- https://github.com/
- Install Git Client on Local Machine
- Windows OS :
- Git Bash
- Git CMD
- Git GUI
- Linux/Mac OS :
- Terminal
Git - Client install on Local Machine
Github - Remote Repository Server
Next :::
- Create AWS Free Tier Account
- Create DockerHub Account
- Install Visual Studio Code on Local Machine
- Install Python on Local Machine
####################
Day2: 30th Nov. 2025
####################
MLOps Lifecycle :
MLOps Usecase :
Versioning :
Version Control System using Git :
Git Demo :
ML Project and Packaging ML Model
CICD Pipeline:
- SCM_Checkout ==> VCS ?
- Application Build
- Create Application Artifacts
- Test
- Deploy
How to Implement MLOps :::
- Data Engineers -> Prepare the Source Datasets
- Data Scientists -> Explore the datasets, analyse, check the feasibility of datasets
Perform the training, tuning, and evaluate the Model
- ML Engineers -> Deploy the ML Models to prod -> MLOps Strageies CICD, define best practises
Should have MLOps Knowledge
- Business Stakeholders -> Bring business values using this ML Models
- Data Security/Governance Team --> To Ensure data security & Governance, data privacy
ML Model Use Case :
ML Model for Product Reviews :
E_Commerce Application Eg. www.amazon.com
Product Review:
CICD Pipeline:
- SCM_Checkout ==> VCS ?
- Application Build
- Create Application Artifacts
- Test
- Deploy
Process/Lifecycle of Machine Learning Ops:::
- Data Preparation
- Exploratory Data Analysis (EDA)
- Feature Engineering
- Model Training
- Model Validations
- Model Deployment
- Monitoring
MLOps Use cases:
- Healthcare
- Finance
- Retail Domain - E_Commerce
- Manufacturing
How to Implement MLOps :::
- Vesioning:
Version Control System :
- To Version Control the Changes
- To Track the Changes
Types of VCS :
- Centralized VCS:
- Distributed VCS:
GIT :
- Git is an Open-Source Distributed VCS
- To Version Control the Changes
- To Track the Changes
- To perform Parallel Development using Branching Techniques
Lab Requirements:
- Create GitHub Account
- https://github.com/
- Install Git Client on Local Machine
- https://git-scm.com/install/windows
- https://git-scm.com/install/mac
- Windows OS :
- Git Bash
- Git CMD
- Git GUI
- Linux/Mac OS :
- Terminal
git --version
Git - Client install on Local Machine
Github - Remote Repository Server
Working with Git :::
Local Git Repository Remote Git Repository
Tracking/Versioning ?
*.py
asdfasdf
dfasdf
asdfasdfasd
----shion
save myMLApp1.py ===> myMLApp1.py_v1.0
save myMLApp1.py ===> myMLApp1.py_v1.1
save myMLApp1.py ===> myMLApp1.py_v1.2
save myMLApp1.py ===> myMLApp1.py_v1.3
save myMLApp1.py ===> myMLApp1.py_v1.4 Latest version (Tag/Version_Number/Commit_ID)
Git File Workflow :::
Local Machine Remote Server
Install git client Github/AzRepo/gitlab
Working_Directory Staging_Area Local_Repository Remote_Repsitory
myMLApp1.py --------------> myMLApp1.py --------------> myMLApp1.py_v1.0 ----------------> myMLApp1.py_v1.0
git add git commit git push
myMLApp1.py --------------> myMLApp1.py --------------> myMLApp1.py_v1.1 ----------------> myMLApp1.py_v1.1
git add git commit git push
Developer's Workload :
Types of Projects:
- Enhancement/Bugfix
- New Project
Git Cli Commands :
git --version
git clone : To Clone/Copy the entire Remote Repository to Local Machine
git add : To add the changes from Working Directory to Staging_Area
git commit : To commit the changes from Staging_Area to Local Repository
git push : To Push the Changes from Local Repository to Remote Repository
git pull/Fetch
- Both git pull & git fetch are used to handle the incremental changes from Remote Repository
- git Fetch :
To just check the incremental changes from remote repository. If any incremental changes exist, it will update the incremental changes only in the Local Repository. Not in the Working Directory.
- git pull :
Check the incremental changes from remote repository. If any incremental changes exist, it will update the incremental changes in the Local Repository as well in the Working Directory.
As a best practise, always it is recommended to use git pull to keep the local repo in-sync with remote repository
git pull = git fetch + git merge!
git init : To initialize a local git repository
It will create a .git directory in the project workspace
It will create a default branch called master/main
Fork : To copy/backup a entire remote repository to another remote repository
Work with Cli Commands :::
Organize the WorkSpace :::
cd d:
mkdir MLProjects
cd MLProjects
mkdir MLRepo1 # Workspace
mkdir MLRepo2
cd MLRepo1
git init
Local Machine
Working_Directory Staging_Area Local_Repositor
file1.txt --------------> file1.txt --------------> file1.txt
git add git commit
git status : To Get the status of repository
Git Configuration ::::
As a best practise, using git global config to setup the user name and email after initializing the first local repository.
Global Configuration: # Applicable to all the Repositories
git config --global user.name "LoksaiETA"
git config --global user.email "loksai@asdf.com"
git config --global --list
Local Configuration: # Applicable to a specific Repository
git config user.name "LoksaiETA"
git config user.email "loksai@asdf.com"
git config --list
git add ::::
git add file1.txt
git add file1.txt file2.txt
git add *.doc
git add . # . to select all files
git rm --cached 'file1.txt' # Unstage a file, the file changes will be back to working directory.
git rm -f 'file1.txt' # Permantly Unstage and a delete a file from working directory.
Next :::
- Git Reset / Revert
- Git Branches
- Git Branching Strategies
- Git rebase / Merge
- Git Remote Repositories Handling
- Create AWS Free Tier Account
- Create DockerHub Account
- Install Visual Studio Code on Local Machine
- Install Python on Local Machine
####################
Day3: 6th Dec. 2025
####################
git
github
docker
dockerhub
DVC
MLFLow
Kubernetes
KubeFLow
Prometheus
- git ignore :::
- Is used ignore the files from tracking.
- .gitignore is a hidden file, used to maintain the file names/patterns to ignore from tracking.
- As a best practise, .gitignore should be the very first commit in any repository.
Project_Workspace : ==> Push
- src
- *.py
- test
- config_file
- req.txt
- app.prop
- db_credentials.json
- Secret
git log ::::
git log
git log --oneline
Undo the Commited Changes :::
Undo/Revert the Committed Changes from Local Repository :::
git reset :::
git reset is used to undo the changes from the repository
git reset will reset the commit point to the previous commit point
git reset will not create any new commit point for track purpose
git reset is not recommended in the shared repository
Syntax :
- git reset <reset_option> <previous_Commit_ID>
Reset Options ::
git reset --soft <previous_Commit_ID> :::
- git reset will reset the HEAD pointer to the previous commit point.
- It will take the changes back to staging area
- The Changes will be available in staging area and working directory
git reset --mixed <previous_Commit_ID> ::: # Default
- git reset will reset the HEAD pointer to the previous commit point.
- It will take the changes back to working directory
- The Changes will be available only in working directory
git reset --hard <previous_Commit_ID> :::
- git reset will reset the HEAD pointer to the previous commit point.
- It will permanently delete the files from the system
ML Projects ::::
Dataset Source :
ETL Process :::
Extract ==> for 30 mins --> Raw Data ==> commited to a repository
Transform ==> Application Program to Process Data
Load ==> Saved to a Database
==> Reset --hard pointing to initial commit (.gitignore)
Git Revert :::
git revert :::
- Git Revert is same as git reset --hard option
- git revert is used to undo a specific commit
- git revert will create a new commit point for tracking purpose.
- git revert will maintain the commit history
- git revert is recommended in shared repositories
Syntax ::
git revert <specific_Commit_ID>
GIT Commit :::
- git commit # To Commit the changes from staging area to local repository
git commit -m "<Valid Commit_Message>"
As a best practise :
git commit -m "CR1001 - Message"
git commit -m "REL-0126 - Payment Module updated"
Git Branches ::::
-> Git Branches are the logical copy of the Repository/Default Branch
-> Used to perform parallel development
Git Branching Strategies :::
-> default branch - (master/main) ==> Used to maintain the production version of source.
-> Git Branching Strategeies are used to maintain the integrity of the main/master Branch(Prod version)
Repo1 :::
master : cm1,cm2,c3
feature1_branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3,...................
Scenario 1:
Repo1 :::
master : cm1,cm2,c3
cm1,cm2,c3,cm4 # 'cm4' is the changes merged from feature1_branch
feature1_branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3
Scenario 2:
Repo1 :::
master : cm1,cm2,c3
cm1,cm2,c3,cm4 # 'cm4' is the changes merged from Developer_Branch
Developer_Branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3,f2cm1,f2cm2,f2cm3
feature1_branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3
feature2_branch : cm1,cm2,c3
cm1,cm2,c3,f2cm1,f2cm2,f2cm3
Scenario 3:
Repo1 :::
master : cm1,cm2,c3
cm1,cm2,c3,cm4 # 'cm4' is the changes merged from Integration_Branch
Integration_Branch : cm1,cm2,c3
cm1,cm2,c3,Developer1_Changes,Developer2_Changes
Developer1_Branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3,f2cm1,f2cm2,f2cm3
feature1_branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3
feature2_branch : cm1,cm2,c3
cm1,cm2,c3,f2cm1,f2cm2,f2cm3
Developer2_Branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3,f2cm1,f2cm2,f2cm3
feature1_branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3
feature2_branch : cm1,cm2,c3
cm1,cm2,c3,f2cm1,f2cm2,f2cm3
Scenario 4:
Repo1 :::
master : cm1,cm2,c3
cm1,cm2,c3,cm4 # 'cm4' is the changes merged from Release_Branch
Release_Branch : cm1,cm2,c3
cm1,cm2,c3,Team1_Changes,Team2_Changes
Integration_Branch1 : cm1,cm2,c3 Team-1
cm1,cm2,c3,Developer1_Changes,Developer2_Changes
Developer1_Branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3,f2cm1,f2cm2,f2cm3
feature1_branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3
feature2_branch : cm1,cm2,c3
cm1,cm2,c3,f2cm1,f2cm2,f2cm3
Developer2_Branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3,f2cm1,f2cm2,f2cm3
feature1_branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3
feature2_branch : cm1,cm2,c3
cm1,cm2,c3,f2cm1,f2cm2,f2cm3
Integration_Branch2 : cm1,cm2,c3 Team-2
cm1,cm2,c3,Developer1_Changes,Developer2_Changes
Developer1_Branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3,f2cm1,f2cm2,f2cm3
feature1_branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3
feature2_branch : cm1,cm2,c3
cm1,cm2,c3,f2cm1,f2cm2,f2cm3
Developer2_Branch : cm1,cm2,c3
cm1,cm2,c3,f1cm1,f1cm2,f1cm3,f2cm1,f2cm2,f2cm3