Introduction to Machine Learning and Python

1. Overview of Machine Learning and its Applications.

Understanding AI:

• It can be defined as the simulation of human intelligence processes by machine learning, mainly by computers. It is a branch of computer science.
• AI systems can perform tasks associated with human cognitive functions, for example, elucidate speech, playing games, and recognize patterns. Usually, AI systems pick up this skill by sifting through vast volumes of data and searching for patterns to incorporate into their decision-making.
• AI is a grouping of several techniques, including machine learning.

Machine Learning:

Understanding Machine Learning:

• It is a branch of data science that applies statistical methods to improve performance based on previous experience and to detect new patterns where there is plenty of data. It is one of the ways of AI. It can learn from the data.
• Let’s take an example here it involves various steps, like starting with data, like numbers, photos, or text. 
• In order to use it as training data—the information the machine learning model will be trained on—this gathered data and prepared it. The program gets better the more data it has. After that, it will select a machine learning model to use, provide data, and allow the computer model to learn to identify patterns on its own.
• To test the machine learning model’s accuracy when presented with new data, a portion of the training data is withheld and used as evaluation data. This outcome is a model that can be applied to other data sets in the future. 

Let’s understand the above process in more detail

• Step 1:Collection of Data:
• Step 2:Data Preprocessing:
• Step 3:Choosing the appropriate model:
• Step 4:Training the model:
• Step 5:Evaluating Model:

Types of ML:

Application of ML:

Applications of ML are wide and in different domains. It is used in email filtering, Deep Learning – a subset of machine learning.

Fintech:

1)Algorithmic Trading:

• Algorithmic trading is a very popular strategy used by many companies to increase trade volume and automate their financial decisions.
• It involves executing trading orders based on pre-written trading directives that machine learning algorithms have made feasible.
• All major financial companies invest in algorithmic trading because it is impossible to replicate the frequency of trades executed by ML technology through manual means.

Technical skills needed: competence with natural language processing, machine learning, and real-time data processing technologies; additionally, knowledge of trading and financial markets is required.

Potential applications: trading firms, hedge funds, investment banks, and other financial institutions.

Common machine learning and deep learning models used today:

• Recurrent Neural Networks (RNN).
• Long Short Term Memory Network (LSTM)
• Ensemble algorithms
• Support vector Machines

2)Regulatory Compliance:

• Because machine learning algorithms are able to read and learn from a vast amount of regulatory documents, they are able to find correlations between guidelines.
• Consequently, cloud solutions with integrated machine learning algorithms for use in the finance industry can automatically track and monitor regulatory changes. Banking institutions can also monitor transaction data for anomalies.
• In this way, ML can ensure that customer transactions comply with regulatory requirements. 

3)Loan Automation:

• Beyond just processing income and FICO scores, machine learning algorithms can process multiple layers of data. These machine-learning applications in finance have given lenders access to new data sources.
• To create an accurate risk score, for instance, a wide range of indicators are now taken into account, including social media accounts, utility bills, rent payments, and even medical records.

4)Security and Fraud Detection:

• The rise in financial cybercrimes is expected to correspond with the worldwide implementation of digital transformation initiatives.
• The good news is that thanks to AI and ML, users and businesses can now secure their accounts and themselves. Financial cybersecurity and cryptocurrency are often associated with each other.
• Algorithms are not only able to recognize suspicious behavior; they can also notify users of it. These technologies make it possible to continuously monitor unusual patterns, so ongoing vigilance is not required.
• Users can monitor everything that goes on behind their backs and be certain that their assets are safe.  

Technical skills needed: Expertise in machine learning algorithms, acquaintance with real-time data processing technologies such as Apache Spark, and understanding of financial data and fraud detection methods.

Potential applications: online payment platforms, credit card companies, banks, and other financial institutions.

Example project: Build a machine learning model that can instantly detect fraudulent credit card transactions by using data from past transactions and customer profiles. After training on a labelled dataset of fraudulent and non-fraudulent transactions, the model’s performance is evaluated using various metrics such as precision, recall, and AUC (Area under the ROC Curve). Integrate the model into a real-time processing pipeline with Apache Spark so that it can identify dubious transactions as they occur.

 Some of the machine learning models that can be used are:

• Recurrent Neural Network (RNN)
• Long Short Term Memory Network (LSTM)
• Logistic Regression
• Ensemble Techniques

5)Customer Support:

• Bots are among the most well-known applications of AI and ML. Although they have been around for a while, their popularity has only lately started to grow thanks to ML algorithms. Robust chatbots that are able to interact with customers and quickly respond to a range of requests are becoming more and more popular.
• One important tool that FinTech companies use to handle customer issues is bots. Among the most popular machine learning solutions are automated customer service and robot advisors.
• Businesses using chatbots to reduce costs and increase customer satisfaction have seen notable benefits. 

6) Decision Making and Credit Scoring:

• One of the main benefits of machine learning technology for the financial sector is predictive analytics.
• Its advantages in the areas of credit scoring and decision-making are also increasing. Banks and other fintech businesses offer loans that are managed by machine learning (ML)-based credit scoring systems rather than just rule-based ones, which allows them to optimize the flow of money.

7) Customer’s Risk Profile:

• One of the machine learning projects in finance that is currently in high demand is customer profiling and needs understanding. However, identifying risky profiles involves more than just meeting consumer demands. 
• It’s more about prudent, planned decision-making and banking security. Based on the banking activities of the customers, risk profiles are also detected and segmented at this point.
• They make sure the borrower is reliable and has a successful loan history. In this manner, the bank can approve loan applications from any client with a “crystal-clear” banking record. With the aid of AI and ML, patterns in the services that consumers select or their online financial transactions can be tracked in real-time. 
• Furthermore, when concerns about trust arise, banks may even track usage patterns on social media, browsing history, geolocation pins on maps, and other metrics. In order to effectively identify creditworthy customers, the fintech industry uses AI/ML tools to automate customer risk evaluation processes.

Below is a link for datasets related to (Fintech):

HealthCare:

1)Fraud detection in healthcare

• Unsupervised learning techniques such as clustering and anomaly detection can be used to scan massive amounts of claim data, potentially revealing unexpected patterns that indicate fraudulent activity.
• These algorithms can operate without labeled data, which is frequently lacking in the context of fraud detection.
• They learn by examining the structure and relationships present in the data, as opposed to depending solely on conventional rule-based systems to identify anomalies and possible fraud scenarios.
• The effectiveness of fraud investigation teams can be enhanced by unsupervised learning algorithms by reducing the number of false positives produced by traditional rule-based systems.
• By identifying unexpected patterns and outliers, unsupervised learning can identify early warning signs of impending fraud schemes, enabling businesses to take preventative action to avoid losses.
• By identifying and getting rid of false claims, healthcare organizations can protect their bottom line and save money.

     2) Patient Segmentation:

• Because it enables medical professionals to recognize different patient groups and modify services and treatments accordingly, patient segmentation is a crucial component of healthcare management.
• Large amounts of patient data can be effectively analyzed by unsupervised learning algorithms, especially clustering techniques, which can also help identify hidden patterns that can help classify patients into meaningful groups. Better patient outcomes, targeted interventions, and more effective resource allocation are made possible by this process.
• By identifying distinct patient segments, medical professionals can develop tailored treatment plans and interventions that result in more effective and personalized patient care.
• Healthcare workers can better allocate their resources and ensure that every patient receives the right kind of care by being aware of the needs and traits of various patient segments.
• By addressing the unique needs of each patient group, healthcare providers can enhance patient outcomes, including lower hospital readmission rates, shorter hospital stays, and higher patient satisfaction.

3) Dermatology Assessments:

• One of the most common uses of machine learning in healthcare is medical imaging.
• From a technical standpoint, algorithms serve large biomedical datasets in the forms of facial images and electronic health records. 
• As more datasets are added to the systems, bots are trained to recognize symptoms of specific facial conditions, rate the severity of the condition, and frequently even suggest medications to treat it. 

4) Clinical Decision-Making for Heart Failure:

• Neural network and machine learning applications in healthcare have revolutionized disease prediction and, naturally, impacted prevention and care delivery.
• Pfizer is employing a novel prediction model, mainly for patients with heart failure, that performs well in symptom analysis domains. The study compared electronic health records with extant patient literature and demonstrated 87% accuracy in predicting heart failure diseases, according to Pfizer’s medical research team.   The goal of the study was to enable healthcare providers in diagnose
• procedures by finding digitally intuitive ways. The anticipated tool will help physicians identify at-risk heart patients more quickly and accurately. It is already being used by pharmaceutical companies with healthcare research models.

5) Drug Research and Identification:

• One significant feature that is driving machine learning’s application in healthcare is image recognition.  Using machine learning algorithms, important pharmaceutical and drug discoveries that require years of drawn-out laboratory experiments can be made, verified, and tested incredibly quickly.
• Deep learning is a kind of machine learning that goes beyond basic machine learning to teach computers how the eye interprets visual cues. 
• Scientists are experimenting with the effects of specific medications and substances on human cells using DL.
• More significantly, machine learning applications in drug discovery offer a never-before-seen capacity to evaluate the effectiveness of the current drug and medicine models and provide answers regarding their viability in the face of rising carbon emissions and climate change.

6) Automated Data Recording & Extraction via NLP:

• Automated medical data recording is a significant application of natural language processing (NLP). Algorithms assess spoken and written communications in the medical field to generate structured reports. 
• This facilitates the speedy retrieval and analysis of patient data while saving healthcare professionals time. Clinical notes that are not structured in a particular way can also provide valuable information to NLP. 
• These lengthy reports make it easy for doctors and nurses to identify illnesses, medications, and treatment plans. This enables the collection of data for research, quality improvement, and expert decision support tools.

Supply Chain Management (SCM):

1. Inventory management:

• One of the most common applications of machine learning in the supply chain is inventory management. The issue of under- or overstocking can be resolved with the aid of machine learning.
• With machine learning (ML), you can forecast the growth of demand based on data that can be sourced from various sources, including the marketplace environment, seasonal trends, promotions, sales, and historical analysis. You can prepare to fill your stores in advance as well as prevent excesses of goods or important parts for manufacturing. 

2. Warehouse Management:

• Machine learning is used in warehouses to automate manual tasks, anticipate potential problems, and minimize paperwork for warehouse employees. For instance, computer vision enables the management of conveyor belt operations and the anticipation of blockages. 
• Warehouse professionals can automatically identify the arrival of packages and modify their delivery statuses thanks to NLP and OCR. Barcodes and labels on the package are scanned by cameras, and the system immediately receives all the information that is needed. 

2. Introduction to Python for ML.

Why Python?

1. Python is a simpler language 

• Fewer alternatives (one way to do it)
• Better alternatives (easier to accomplish common tasks)
• This allows us to focus less on the language and more on problem-solving

2. Many of the best parts of other languages are included in Python
   • Data structures (lists, dictionaries)
   • Control (iteration, exceptions)
   • There are many packages for common tasks
   • Python is often described as “batteries included”.
3. Python is open source.
   • Freely available 
   • A large user base constantly contributing
   • New packages (libraries) are available to meet changing needs

Python is more generally useful for getting work done

Why Python for ML?

Wes McKinney, Section 1.2. in Python for Data Analysis, 2nd Edition

• Has a large and active scientific computing and data analysis community.
• In the last 10 years, it has become one of the most important languages for data science, machine learning, and general software development in academia and industry.
• Python is a suitable language not only for doing research and prototyping but also for building production systems. ➞ There is no need to maintain two development environments.

Other?

• Has an extensive set of libraries for data science..

       Python Applications

• Python is a powerful, multi-paradigm computer programming language. With Python, we can do many things. Below are some of the things that can be achieved using Python. 
• Systems Programming: Python’s built-in interfaces to operating-system services make it ideal for writing portable, maintainable system administration tools and utilities (sometimes called shell tools). Python programs can search files, directory trees, etc. 
• GUIs: Python’s simplicity and rapid turnaround make it a good match for graphical user interface programming on the desktop. Python comes with a standard object-oriented interface to the Tk GUI API called tkinter (Tkinterin 2.X) that allows Python programs to implement portable GUIs with a native look and feel.
• Internet Scripting: Python comes with standard Internet modules that allow Python programs to perform a wide variety of networking tasks in client and server modes.
• Database Programming: For traditional database demands, there are Python interfaces to all commonly used relational database systems like Sybase, Oracle, Informix, ODBC, MySQL, PostgreSQL, SQLite, and more.
• Rapid Prototyping: To Python programmers, the components written in Python and C look the same. Because of this, it’s possible to prototype systems in Python initially and then move selected components to a compiled language such as C or C++ for delivery. 
• Numeric and Scientific Programming: Python is also heavily used in numeric programming, a domain that would not traditionally have been considered to be in the scope of scripting languages but has grown to become one of Python’s most compelling use cases.

Pseudocode

• Abbreviated plain English version of actual computer code 
• Symbols used in flowcharts replaced by English-like statements 
• It allows the programmer to focus on steps required to solve problems
• Advantages
  • Can be easily converted into the Python language
  • It is compact (less likely to extend over many pages)
  • It looks like the code to be written is easy to understand

Hierarchy Chart

• Shows the overall program structure
• Depict organization of the program; omit specific processing logic
• Describe what each part, or module, of the program, does and how the parts relate to each other
• The chart is read from top to bottom and left to right
• Each module subdivided into a succession of submodules 
• Advantage:
  • The main benefit of the initial planning of a program is that breaking down its major parts tells us what must be done in general. Each part can then be refined using flowcharts or pseudocode.

Types of Errors: Syntax Errors

• Syntax errors are usually the easiest to spot; they occur when you make a typo. 
• Examples
  • Not ending an if statement with the colon 
  • Misspelling a Python keyword (e.g. using while instead of while). 
• Syntax errors usually appear at compile time and are reported by the interpreter, i.e., the program won’t run until the error is fixed.

Types of errors: Runtime Errors

• A program will be executed by the Python interpreter if it is syntactically correct, or devoid of syntax errors. However, if a runtime error occurs—a glitch that wasn’t discovered during program parsing and only becomes apparent when a specific line is executed—the program might end unexpectedly while it’s being executed. 
• When a program comes to a halt because of a runtime error, we say that it has crashed.
• Some examples of Python runtime errors:
  • division by zero
  • performing an operation on incompatible types
  • using an identifier which has not been defined
  • accessing list element, dictionary value or object attribute which doesn’t exist
  • trying to access a file which doesn’t exist
• If you don’t take into account every possible value that a variable could have, especially when processing user input, runtime errors frequently appear. Always strive to include checks in your code to ensure that it can handle error messages and edge cases with grace. 

Types of Errors: Logical Errors

• Also called semantic errors
• Logical errors cause the program to behave incorrectly, but they do not usually crash the program. 
• Unlike a program with syntax errors, a program with logic errors can be run, but it does not operate as intended.
• Examples
  • using the wrong variable name
  • indenting a block to the wrong level
  • using integer division instead of floating point division
  • getting operator precedence wrong
  • making a mistake in a boolean expression
  • off-by-one, and other numerical errors

3. Setting up Python environment, Jupyter notebooks, and basics of coding

A. Setting up Python environment and Jupyter Notebooks

Anaconda is a Leading Open data science platform powered by Python

For this, we will follow this document below for Windows

For Linux

Remote Conda Command

You can use any IDE like jupyter, spider,PyCharm, Canopy etc.-

But we are using a jupyter notebook here.

IPython Notebook

• IPython provides a rich architecture for interactive computing with: 
• A powerful interactive shell.
• A kernel for Jupyter.
• Easy to use, high-performance tools for parallel computing.
• Support for interactive data visualization and use of GUI toolkits.
• Flexible, embeddable interpreters to load into your projects.

The Four Most Helpful Commands 

? – Introduction and overview of IPython’s features.

%quickref – Quick reference.

Help- Python’s help system.

object? – Details about the object; use object?? for extra details.

IPython Notebook

• The %run magic command allows you to run any Python script and load all of its data directly into the interactive namespace. Since the file is re-read from disk each time, changes you make to it are reflected immediately (unlike imported modules, which have to be specifically reloaded). IPython also includes dreload, a recursive reload function. 
• %run has special flags for timing the execution of your scripts (-t), or for running them under the control of either Python’s pdb debugger (-d) or profiler (-p). 
• The %edit command gives a reasonable approximation of multiline editing, by invoking your favourite editor on the spot. IPython will execute the code you type in there as if it were typed interactively.

Magic Functions:

• IPython has a set of predefined magic functions that you can call with a command line style syntax. There are two kinds of magic, line-oriented and cell-oriented. 
• Line magics are prefixed with the % character and work much like OS command-line calls: they get as an argument the rest of the line, where arguments are passed without parentheses or quotes. 
• Cell magics are prefixed with a double %%, and they are functions that get as an argument not only the rest of the line but also the lines below it in a separate argument. 
• You can run script.py. You can toggle this behaviour by running the %automagic magic.
• A more detailed explanation of the magic system can be obtained by calling %magic. 
• To see all the available magic functions, call %lsmagic

B. Basics of Coding

We will divide it into 2 parts theory and the basic lab part.

       Program Steps or Program Flow

• Like a recipe or installation instructions, a program is a sequence of steps to be done in order.
• Some steps are conditional – they may be skipped.
• Sometimes a step or group of steps is to be repeated.
• Sometimes we store a set of steps to be used over and over as needed several places throughout the program

Constants

• Fixed values such as numbers, letters, and strings, are called “constants” because their value does not change
• Numeric constants are as you expect
• String constants use single quotes (‘)
  or double quotes (“)

Examples

>>> print(123)

• 123

>>> print(98.6)

• 98.6

>>> print(‘Hello world’)

• Hello world

Reserved Words

• You cannot use reserved words as variable names/identifiers as they have a special meaning for Python.

Variables

• A variable is a named place in the memory where a programmer can store data and later retrieve the data using the variable “name”
• Programmers get to choose the names of the variables
• You can change the contents of a variable in a later statement

x= 12.2

y=14

Python Variable Name Rules

• Must start with a letter or underscore ‘_’
• Must consist of letters, numbers, and underscores
• Case Sensitive

Standard Way

• The standard way to name most things in Python is lower case with an under, lower case with separate words joined by an underline:
  • this_is_a_var
  • my_list
  • square_root_function

Comments in Python

• Anything after a # is ignored by Python
• Why comment?

 –  Describe what is going to happen in a sequence of code

 –  Document who wrote the code or other ancillary information

 –  Turn off a line of code – perhaps temporarily

References Books:

For Machine Learning

1. Machine Learning for Absolute Beginners By Oliver Theobald
2. Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies.

For Python

1. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and Jupyter 3rd Edition by Wes McKinney.
2. Murach’s Python Programming (2nd Edition).