From Beginner to Expert
Table of Contents
- Introduction to Python
- Getting Started
- Python Basics
- Control Flow
- Functions
- Data Structures
- Object-Oriented Programming
- Error Handling
- File Operations
- Modules and Packages
- Advanced Topics
- Web Development
- Data Science and Analytics
- Testing and Debugging
- Best Practices
Introduction to Python
Python is a high-level, interpreted programming language known for its simplicity and readability. Created by Guido van Rossum in 1991, Python emphasizes code readability and allows programmers to express concepts in fewer lines of code.
Why Choose Python?
mindmap
root((Python Advantages))
Easy to Learn
Simple Syntax
Readable Code
Beginner Friendly
Versatile
Web Development
Data Science
AI/ML
Automation
Large Community
Extensive Libraries
Active Support
Open Source
Cross Platform
Windows
Mac
LinuxPython Application
graph TD
A[Python Applications] --> B[Web Development]
A --> C[Data Science]
A --> D[Machine Learning]
A --> E[Automation]
A --> F[Game Development]
A --> G[Desktop Applications]
B --> B1[Django]
B --> B2[Flask]
B --> B3[FastAPI]
C --> C1[Pandas]
C --> C2[NumPy]
C --> C3[Matplotlib]
D --> D1[TensorFlow]
D --> D2[PyTorch]
D --> D3[Scikit-learn]Getting Started
Installation
- Download Python: Visit python.org and download the latest version
- Install: Run the installer and ensure “Add Python to PATH” is checked
- Verify Installation: Open command prompt and type:
python --versionDevelopment Environment
graph LR
A[Choose IDE] --> B[VS Code]
A --> C[PyCharm]
A --> D[Jupyter Notebook]
A --> E[IDLE]
B --> F[Recommended for beginners]
C --> G[Professional development]
D --> H[Data science & research]
E --> I[Built-in with Python]Your First Python Program
# Hello World program
print("Hello, World!")
print("Welcome to Python programming!")
# Variables and basic operations
name = "Python"
version = 3.12
print(f"Learning {name} version {version}")Python Basics
Variables and Data Types
graph TD
A[Python Data Types] --> B[Numeric]
A --> C[Sequence]
A --> D[Mapping]
A --> E[Set]
A --> F[Boolean]
A --> G[None]
B --> B1[int]
B --> B2[float]
B --> B3[complex]
C --> C1[str]
C --> C2[list]
C --> C3[tuple]
D --> D1[dict]
E --> E1[set]
E --> E2[frozenset]# Numeric types
integer_num = 42
float_num = 3.14159
complex_num = 3 + 4j
# String
text = "Hello, Python!"
multiline = """This is a
multiline string"""
# Boolean
is_python_fun = True
is_difficult = False
# List (mutable)
fruits = ["apple", "banana", "orange"]
# Tuple (immutable)
coordinates = (10, 20)
# Dictionary
person = {
"name": "Alice",
"age": 30,
"city": "New York"
}
# Set
unique_numbers = {1, 2, 3, 4, 5}Operators
graph TD
A[Python Operators] --> B[Arithmetic]
A --> C[Comparison]
A --> D[Logical]
A --> E[Assignment]
A --> F[Membership]
A --> G[Identity]
B --> B1[+ - * / % ** //]
C --> C1[== != < > <= >=]
D --> D1[and or not]
E --> E1[= += -= *= /=]
F --> F1[in not in]
G --> G1[is is not]# Arithmetic operators
a, b = 10, 3
print(f"Addition: {a + b}") # 13
print(f"Subtraction: {a - b}") # 7
print(f"Multiplication: {a * b}") # 30
print(f"Division: {a / b}") # 3.333...
print(f"Floor Division: {a // b}") # 3
print(f"Modulus: {a % b}") # 1
print(f"Exponentiation: {a ** b}") # 1000
# Comparison operators
print(f"Equal: {a == b}") # False
print(f"Not equal: {a != b}") # True
print(f"Greater than: {a > b}") # True
# Logical operators
x, y = True, False
print(f"AND: {x and y}") # False
print(f"OR: {x or y}") # True
print(f"NOT: {not x}") # FalseInput and Output
# Getting user input
name = input("Enter your name: ")
age = int(input("Enter your age: "))
# Formatted output
print(f"Hello {name}, you are {age} years old!")
# Different ways to format strings
print("Hello %s, you are %d years old!" % (name, age))
print("Hello {}, you are {} years old!".format(name, age))
print("Hello {name}, you are {age} years old!".format(name=name, age=age))Control Flow
Conditional Statements
flowchart TD
A[Start] --> B{Condition}
B -->|True| C[Execute if block]
B -->|False| D{Elif condition?}
D -->|True| E[Execute elif block]
D -->|False| F[Execute else block]
C --> G[End]
E --> G
F --> G# If-elif-else statements
score = 85
if score >= 90:
grade = "A"
print("Excellent!")
elif score >= 80:
grade = "B"
print("Good job!")
elif score >= 70:
grade = "C"
print("Average")
elif score >= 60:
grade = "D"
print("Needs improvement")
else:
grade = "F"
print("Failed")
print(f"Your grade is: {grade}")
# Ternary operator
status = "Pass" if score >= 60 else "Fail"
print(f"Status: {status}")Loops
flowchart TD
A[For Loop] --> B[Iterate over sequence]
B --> C[Execute code block]
C --> D{More items?}
D -->|Yes| C
D -->|No| E[End]
F[While Loop] --> G{Condition True?}
G -->|Yes| H[Execute code block]
H --> G
G -->|No| I[End]# For loops
fruits = ["apple", "banana", "orange"]
# Iterate over list
for fruit in fruits:
print(f"I like {fruit}")
# Iterate with index
for index, fruit in enumerate(fruits):
print(f"{index + 1}. {fruit}")
# Range function
for i in range(1, 6): # 1 to 5
print(f"Number: {i}")
# While loops
count = 0
while count < 5:
print(f"Count: {count}")
count += 1
# Loop control statements
for i in range(10):
if i == 3:
continue # Skip iteration
if i == 7:
break # Exit loop
print(i)
# Nested loops
for i in range(3):
for j in range(3):
print(f"({i}, {j})")Loop Patterns
# List comprehension
squares = [x**2 for x in range(1, 6)]
print(squares) # [1, 4, 9, 16, 25]
# Conditional list comprehension
even_squares = [x**2 for x in range(1, 11) if x % 2 == 0]
print(even_squares) # [4, 16, 36, 64, 100]
# Dictionary comprehension
square_dict = {x: x**2 for x in range(1, 6)}
print(square_dict) # {1: 1, 2: 4, 3: 9, 4: 16, 5: 25}Functions
Function Basics
graph TD
A[Function Definition] --> B[def keyword]
A --> C[Function name]
A --> D[Parameters]
A --> E[Function body]
A --> F[Return statement]
G[Function Call] --> H[Function name]
G --> I[Arguments]
J[Function Benefits] --> K[Code reusability]
J --> L[Modularity]
J --> M[Easier debugging]
J --> N[Better organization]# Basic function
def greet(name):
"""Greet a person with their name."""
return f"Hello, {name}!"
# Function call
message = greet("Alice")
print(message)
# Function with multiple parameters
def add_numbers(a, b):
"""Add two numbers and return the result."""
result = a + b
return result
sum_result = add_numbers(5, 3)
print(f"Sum: {sum_result}")
# Function with default parameters
def introduce(name, age=25, city="Unknown"):
"""Introduce a person with optional age and city."""
return f"Hi, I'm {name}, {age} years old from {city}"
print(introduce("Bob"))
print(introduce("Carol", 30))
print(introduce("Dave", 35, "Boston"))Advanced Function Features
# Variable arguments
def sum_all(*numbers):
"""Sum all provided numbers."""
return sum(numbers)
print(sum_all(1, 2, 3, 4, 5)) # 15
# Keyword arguments
def create_profile(**info):
"""Create a profile with keyword arguments."""
profile = {}
for key, value in info.items():
profile[key] = value
return profile
profile = create_profile(name="Alice", age=30, profession="Engineer")
print(profile)
# Lambda functions
square = lambda x: x**2
print(square(5)) # 25
# Higher-order functions
numbers = [1, 2, 3, 4, 5]
squared = list(map(lambda x: x**2, numbers))
even_numbers = list(filter(lambda x: x % 2 == 0, numbers))
print(f"Squared: {squared}")
print(f"Even numbers: {even_numbers}")Decorators
# Basic decorator
def timing_decorator(func):
"""Decorator to measure function execution time."""
import time
def wrapper(*args, **kwargs):
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} took {end_time - start_time:.4f} seconds")
return result
return wrapper
@timing_decorator
def slow_function():
"""A function that takes some time."""
import time
time.sleep(1)
return "Done!"
result = slow_function()Data Structures
Lists
graph TD
A[List Operations] --> B[Creation]
A --> C[Access]
A --> D[Modification]
A --> E[Methods]
B --> B1["Empty list: []"]
B --> B2["With values: [1,2,3]"]
B --> B3["list() constructor"]
C --> C1["Indexing: list[0]"]
C --> C2["Slicing: list[1:3]"]
C --> C3["Negative indexing: list[-1]"]
D --> D1[Append]
D --> D2[Insert]
D --> D3[Remove]
D --> D4[Pop]
E --> E1["sort()"]
E --> E2["reverse()"]
E --> E3["count()"]
E --> E4["index()"]# List creation and operations
fruits = ["apple", "banana", "orange"]
# Adding elements
fruits.append("grape") # Add at end
fruits.insert(1, "kiwi") # Insert at index
fruits.extend(["mango", "peach"]) # Add multiple items
print(f"Fruits: {fruits}")
# Accessing elements
print(f"First fruit: {fruits[0]}")
print(f"Last fruit: {fruits[-1]}")
print(f"First three: {fruits[:3]}")
# Modifying elements
fruits[0] = "green apple"
print(f"Modified: {fruits}")
# List methods
fruits.sort() # Sort in place
print(f"Sorted: {fruits}")
fruits.reverse() # Reverse in place
print(f"Reversed: {fruits}")
# List comprehension with conditions
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
even_squares = [x**2 for x in numbers if x % 2 == 0]
print(f"Even squares: {even_squares}")Dictionaries
graph TD
A[Dictionary] --> B[Key-Value Pairs]
A --> C[Mutable]
A --> D[Unordered]
A --> E[Methods]
B --> B1[Keys must be immutable]
B --> B2[Values can be any type]
E --> E1["keys()"]
E --> E2["values()"]
E --> E3["items()"]
E --> E4["get()"]
E --> E5["pop()"]
E --> E6["update()"]# Dictionary creation and operations
student = {
"name": "Alice",
"age": 20,
"grades": [85, 90, 78, 92],
"is_enrolled": True
}
# Accessing values
print(f"Name: {student['name']}")
print(f"Age: {student.get('age', 'Unknown')}")
# Adding/updating values
student["major"] = "Computer Science"
student["age"] = 21
# Dictionary methods
print(f"Keys: {list(student.keys())}")
print(f"Values: {list(student.values())}")
print(f"Items: {list(student.items())}")
# Iterating through dictionary
for key, value in student.items():
print(f"{key}: {value}")
# Dictionary comprehension
squares_dict = {x: x**2 for x in range(1, 6)}
print(f"Squares: {squares_dict}")
# Nested dictionaries
class_roster = {
"student1": {"name": "Alice", "grade": 85},
"student2": {"name": "Bob", "grade": 90},
"student3": {"name": "Carol", "grade": 78}
}
for student_id, info in class_roster.items():
print(f"{student_id}: {info['name']} - Grade: {info['grade']}")Sets
# Set operations
fruits = {"apple", "banana", "orange"}
vegetables = {"carrot", "broccoli", "spinach"}
healthy_foods = {"apple", "banana", "carrot", "broccoli"}
# Set operations
print(f"Fruits: {fruits}")
print(f"Union: {fruits | healthy_foods}")
print(f"Intersection: {fruits & healthy_foods}")
print(f"Difference: {fruits - healthy_foods}")
# Set methods
fruits.add("grape")
fruits.discard("banana") # Won't raise error if not found
print(f"Modified fruits: {fruits}")Tuples
# Tuple operations
coordinates = (10, 20)
rgb_color = (255, 128, 0)
# Tuple unpacking
x, y = coordinates
r, g, b = rgb_color
print(f"Coordinates: x={x}, y={y}")
print(f"Color: R={r}, G={g}, B={b}")
# Named tuples
from collections import namedtuple
Point = namedtuple('Point', ['x', 'y'])
point = Point(10, 20)
print(f"Point: x={point.x}, y={point.y}")
Person = namedtuple('Person', ['name', 'age', 'city'])
person = Person("Alice", 30, "New York")
print(f"Person: {person.name}, {person.age}, {person.city}")Object-Oriented Programming
Classes and Objects
classDiagram
class Person {
-name: str
-age: int
-email: str
+__init__(name, age, email)
+introduce()
+have_birthday()
+get_age()
}
class Student {
-student_id: str
-grades: list
+__init__(name, age, email, student_id)
+add_grade(grade)
+get_average()
+is_passing()
}
class Teacher {
-subject: str
-salary: float
+__init__(name, age, email, subject, salary)
+teach()
+grade_student()
}
Person <|-- Student
Person <|-- Teacher# Basic class definition
class Person:
"""A class to represent a person."""
# Class variable
species = "Homo sapiens"
def __init__(self, name, age, email):
"""Initialize a Person object."""
# Instance variables
self.name = name
self.age = age
self.email = email
def introduce(self):
"""Return an introduction string."""
return f"Hi, I'm {self.name}, {self.age} years old."
def have_birthday(self):
"""Increment age by 1."""
self.age += 1
print(f"Happy birthday! {self.name} is now {self.age}")
def __str__(self):
"""String representation of the person."""
return f"Person(name='{self.name}', age={self.age})"
def __repr__(self):
"""Developer representation of the person."""
return f"Person('{self.name}', {self.age}, '{self.email}')"
# Creating objects
person1 = Person("Alice", 25, "alice@email.com")
person2 = Person("Bob", 30, "bob@email.com")
print(person1.introduce())
print(person2.introduce())
person1.have_birthday()
print(f"Alice's new age: {person1.age}")Inheritance
# Inheritance example
class Student(Person):
"""A class to represent a student, inheriting from Person."""
def __init__(self, name, age, email, student_id):
"""Initialize a Student object."""
super().__init__(name, age, email) # Call parent constructor
self.student_id = student_id
self.grades = []
def add_grade(self, grade):
"""Add a grade to the student's record."""
if 0 <= grade <= 100:
self.grades.append(grade)
else:
print("Grade must be between 0 and 100")
def get_average(self):
"""Calculate and return the average grade."""
if self.grades:
return sum(self.grades) / len(self.grades)
return 0
def is_passing(self, passing_grade=60):
"""Check if student is passing."""
return self.get_average() >= passing_grade
def introduce(self):
"""Override parent method."""
return f"Hi, I'm {self.name}, a student with ID {self.student_id}."
class Teacher(Person):
"""A class to represent a teacher."""
def __init__(self, name, age, email, subject, salary):
"""Initialize a Teacher object."""
super().__init__(name, age, email)
self.subject = subject
self.salary = salary
def teach(self):
"""Return a teaching message."""
return f"{self.name} is teaching {self.subject}"
def grade_student(self, student, grade):
"""Grade a student."""
student.add_grade(grade)
print(f"{self.name} gave {student.name} a grade of {grade}")
# Using inheritance
student = Student("Carol", 20, "carol@email.com", "S12345")
teacher = Teacher("Dr. Smith", 45, "smith@email.com", "Mathematics", 75000)
print(student.introduce())
print(teacher.introduce())
student.add_grade(85)
student.add_grade(92)
student.add_grade(78)
teacher.grade_student(student, 88)
print(f"Student average: {student.get_average():.2f}")
print(f"Is passing: {student.is_passing()}")Advanced OOP Concepts
# Abstract classes and methods
from abc import ABC, abstractmethod
class Shape(ABC):
"""Abstract base class for shapes."""
@abstractmethod
def area(self):
"""Calculate the area of the shape."""
pass
@abstractmethod
def perimeter(self):
"""Calculate the perimeter of the shape."""
pass
class Rectangle(Shape):
"""Rectangle class inheriting from Shape."""
def __init__(self, width, height):
self.width = width
self.height = height
def area(self):
return self.width * self.height
def perimeter(self):
return 2 * (self.width + self.height)
class Circle(Shape):
"""Circle class inheriting from Shape."""
def __init__(self, radius):
self.radius = radius
def area(self):
return 3.14159 * self.radius ** 2
def perimeter(self):
return 2 * 3.14159 * self.radius
# Property decorators
class BankAccount:
"""Bank account with property decorators."""
def __init__(self, initial_balance=0):
self._balance = initial_balance
@property
def balance(self):
"""Get the current balance."""
return self._balance
@balance.setter
def balance(self, amount):
"""Set the balance with validation."""
if amount < 0:
raise ValueError("Balance cannot be negative")
self._balance = amount
def deposit(self, amount):
"""Deposit money to the account."""
if amount > 0:
self._balance += amount
else:
raise ValueError("Deposit amount must be positive")
def withdraw(self, amount):
"""Withdraw money from the account."""
if amount > self._balance:
raise ValueError("Insufficient funds")
if amount <= 0:
raise ValueError("Withdrawal amount must be positive")
self._balance -= amount
# Using the classes
rectangle = Rectangle(5, 10)
circle = Circle(3)
print(f"Rectangle area: {rectangle.area()}")
print(f"Circle area: {circle.area():.2f}")
account = BankAccount(1000)
print(f"Initial balance: ${account.balance}")
account.deposit(500)
print(f"After deposit: ${account.balance}")
account.withdraw(200)
print(f"After withdrawal: ${account.balance}")Error Handling
Exception Handling Flow
flowchart TD
A[Start] --> B[Try Block]
B --> C{Exception Occurs?}
C -->|No| D[Continue Execution]
C -->|Yes| E{Matching Except Block?}
E -->|Yes| F[Execute Except Block]
E -->|No| G[Unhandled Exception]
F --> H[Finally Block]
D --> H
H --> I[End]
G --> J[Program Crashes]# Basic exception handling
def divide_numbers(a, b):
"""Divide two numbers with error handling."""
try:
result = a / b
return result
except ZeroDivisionError:
print("Error: Cannot divide by zero!")
return None
except TypeError:
print("Error: Please provide numeric values!")
return None
# Test the function
print(divide_numbers(10, 2)) # 5.0
print(divide_numbers(10, 0)) # Error message, returns None
print(divide_numbers(10, "2")) # Error message, returns None
# Multiple exception handling
def process_data(data):
"""Process data with comprehensive error handling."""
try:
# Try to convert to integer and perform operations
number = int(data)
result = 100 / number
index = [1, 2, 3][number] # This might raise IndexError
return result, index
except ValueError:
print(f"Error: '{data}' is not a valid integer")
except ZeroDivisionError:
print("Error: Cannot divide by zero")
except IndexError:
print("Error: Index out of range")
except Exception as e:
print(f"Unexpected error: {e}")
finally:
print("Processing completed")
# Test different scenarios
process_data("5") # Normal case
process_data("abc") # ValueError
process_data("0") # ZeroDivisionError
process_data("10") # IndexErrorCustom Exceptions
# Custom exception classes
class InsufficientFundsError(Exception):
"""Exception raised when account has insufficient funds."""
def __init__(self, balance, amount):
self.balance = balance
self.amount = amount
super().__init__(f"Insufficient funds: ${balance} available, ${amount} requested")
class InvalidAccountError(Exception):
"""Exception raised for invalid account operations."""
pass
class BankAccount:
"""Bank account with custom exception handling."""
def __init__(self, account_number, initial_balance=0):
if not account_number:
raise InvalidAccountError("Account number cannot be empty")
self.account_number = account_number
self.balance = initial_balance
def withdraw(self, amount):
"""Withdraw money with custom exception handling."""
if amount <= 0:
raise ValueError("Withdrawal amount must be positive")
if amount > self.balance:
raise InsufficientFundsError(self.balance, amount)
self.balance -= amount
return self.balance
def deposit(self, amount):
"""Deposit money to the account."""
if amount <= 0:
raise ValueError("Deposit amount must be positive")
self.balance += amount
return self.balance
# Using custom exceptions
try:
account = BankAccount("ACC123", 1000)
print(f"Initial balance: ${account.balance}")
account.withdraw(500)
print(f"After withdrawal: ${account.balance}")
account.withdraw(600) # This will raise InsufficientFundsError
except InsufficientFundsError as e:
print(f"Transaction failed: {e}")
except InvalidAccountError as e:
print(f"Account error: {e}")
except ValueError as e:
print(f"Value error: {e}")Context Managers
# Using context managers for resource management
class FileManager:
"""Custom context manager for file operations."""
def __init__(self, filename, mode):
self.filename = filename
self.mode = mode
self.file = None
def __enter__(self):
print(f"Opening file: {self.filename}")
self.file = open(self.filename, self.mode)
return self.file
def __exit__(self, exc_type, exc_val, exc_tb):
print(f"Closing file: {self.filename}")
if self.file:
self.file.close()
if exc_type:
print(f"Exception occurred: {exc_val}")
return False # Don't suppress exceptions
# Using the context manager
try:
with FileManager("test.txt", "w") as file:
file.write("Hello, World!")
file.write("\nThis is a test file.")
# File will be automatically closed even if an exception occurs
with FileManager("test.txt", "r") as file:
content = file.read()
print(f"File content:\n{content}")
except FileNotFoundError:
print("File not found!")
except PermissionError:
print("Permission denied!")File Operations
File Handling Flow
flowchart TD
A[File Operations] --> B[Open File]
B --> C{File Opened?}
C -->|Yes| D[Perform Operations]
C -->|No| E[Handle Error]
D --> F[Read/Write/Append]
F --> G[Close File]
G --> H[End]
E --> H
I[File Modes] --> J[r - Read]
I --> K[w - Write]
I --> L[a - Append]
I --> M[r+ - Read/Write]
I --> N[x - Exclusive Create]# Basic file operations
def write_to_file(filename, content):
"""Write content to a file."""
try:
with open(filename, 'w') as file:
file.write(content)
print(f"Successfully wrote to {filename}")
except Exception as e:
print(f"Error writing to file: {e}")
def read_from_file(filename):
"""Read content from a file."""
try:
with open(filename, 'r') as file:
content = file.read()
return content
except FileNotFoundError:
print(f"File {filename} not found")
return None
except Exception as e:
print(f"Error reading file: {e}")
return None
def append_to_file(filename, content):
"""Append content to a file."""
try:
with open(filename, 'a') as file:
file.write(content)
print(f"Successfully appended to {filename}")
except Exception as e:
print(f"Error appending to file: {e}")
# Example usage
sample_text = """Python File Handling
This is a sample text file.
It contains multiple lines.
Each line demonstrates file operations."""
# Write to file
write_to_file("sample.txt", sample_text)
# Read from file
content = read_from_file("sample.txt")
if content:
print("File content:")
print(content)
# Append to file
append_to_file("sample.txt", "\nThis line was appended.")
# Read updated content
updated_content = read_from_file("sample.txt")
if updated_content:
print("\nUpdated file content:")
print(updated_content)Advanced File Operations
import os
import json
import csv
from pathlib import Path
# Working with different file formats
# JSON files
def save_json(data, filename):
"""Save data to JSON file."""
try:
with open(filename, 'w') as file:
json.dump(data, file, indent=4)
print(f"Data saved to {filename}")
except Exception as e:
print(f"Error saving JSON: {e}")
def load_json(filename):
"""Load data from JSON file."""
try:
with open(filename, 'r') as file:
return json.load(file)
except FileNotFoundError:
print(f"JSON file {filename} not found")
return None
except json.JSONDecodeError:
print(f"Invalid JSON in {filename}")
return None
# CSV files
def save_csv(data, filename, headers):
"""Save data to CSV file."""
try:
with open(filename, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data)
print(f"CSV data saved to {filename}")
except Exception as e:
print(f"Error saving CSV: {e}")
def load_csv(filename):
"""Load data from CSV file."""
try:
with open(filename, 'r') as file:
reader = csv.reader(file)
data = list(reader)
return data
except FileNotFoundError:
print(f"CSV file {filename} not found")
return None
except Exception as e:
print(f"Error loading CSV: {e}")
return None
# Example data
student_data = {
"students": [
{"name": "Alice", "age": 20, "grade": 85},
{"name": "Bob", "age": 22, "grade": 90},
{"name": "Carol", "age": 19, "grade": 78}
],
"course": "Python Programming",
"semester": "Fall 2024"
}
# Save to JSON
save_json(student_data, "students.json")
# Load from JSON
loaded_data = load_json("students.json")
if loaded_data:
print("Loaded JSON data:")
print(f"Course: {loaded_data['course']}")
for student in loaded_data['students']:
print(f" {student['name']}: {student['grade']}")
# Save to CSV
csv_data = [[s['name'], s['age'], s['grade']] for s in student_data['students']]
csv_headers = ['Name', 'Age', 'Grade']
save_csv(csv_data, "students.csv", csv_headers)
# Load from CSV
loaded_csv = load_csv("students.csv")
if loaded_csv:
print("\nLoaded CSV data:")
for row in loaded_csv:
print(row)
# File and directory operations using pathlib
def explore_directory(path):
"""Explore directory contents."""
directory = Path(path)
if not directory.exists():
print(f"Directory {path} does not exist")
return
print(f"Contents of {path}:")
for item in directory.iterdir():
if item.is_file():
size = item.stat().st_size
print(f" 📄 {item.name} ({size} bytes)")
elif item.is_dir():
print(f" 📁 {item.name}/")
# Create directory if it doesn't exist
data_dir = Path("data")
data_dir.mkdir(exist_ok=True)
# Move files to the data directory
if Path("students.json").exists():
Path("students.json").rename(data_dir / "students.json")
if Path("students.csv").exists():
Path("students.csv").rename(data_dir / "students.csv")
explore_directory("data")Modules and Packages
Module Structure
graph TD
A[Python Module System] --> B[Built-in Modules]
A --> C[Standard Library]
A --> D[Third-party Packages]
A --> E[Custom Modules]
B --> B1[math, random, datetime]
C --> C1[os, sys, json, csv]
D --> D1[requests, pandas, numpy]
E --> E1[Your own .py files]
F[Import Methods] --> G[import module]
F --> H[from module import function]
F --> I[import module as alias]
F --> J[from module import *]Creating Custom Modules
# Create a file called math_operations.py
"""
Mathematical operations module.
Contains functions for common mathematical calculations.
"""
import math
def add(a, b):
"""Add two numbers."""
return a + b
def subtract(a, b):
"""Subtract two numbers."""
return a - b
def multiply(a, b):
"""Multiply two numbers."""
return a * b
def divide(a, b):
"""Divide two numbers."""
if b == 0:
raise ValueError("Cannot divide by zero")
return a / b
def power(base, exponent):
"""Calculate base raised to the power of exponent."""
return base ** exponent
def square_root(number):
"""Calculate square root of a number."""
if number < 0:
raise ValueError("Cannot calculate square root of negative number")
return math.sqrt(number)
def factorial(n):
"""Calculate factorial of a number."""
if n < 0:
raise ValueError("Factorial is not defined for negative numbers")
if n == 0 or n == 1:
return 1
return n * factorial(n - 1)
def is_prime(n):
"""Check if a number is prime."""
if n < 2:
return False
for i in range(2, int(math.sqrt(n)) + 1):
if n % i == 0:
return False
return True
def fibonacci(n):
"""Generate Fibonacci sequence up to n terms."""
if n <= 0:
return []
elif n == 1:
return [0]
elif n == 2:
return [0, 1]
sequence = [0, 1]
for i in range(2, n):
sequence.append(sequence[i-1] + sequence[i-2])
return sequence
# Module-level variables
PI = math.pi
E = math.e
# Module test code
if __name__ == "__main__":
# This code runs only when the module is executed directly
print("Testing math_operations module:")
print(f"Add: {add(5, 3)}")
print(f"Square root of 16: {square_root(16)}")
print(f"Factorial of 5: {factorial(5)}")
print(f"Is 17 prime? {is_prime(17)}")
print(f"First 10 Fibonacci numbers: {fibonacci(10)}")Using Modules
# Different ways to import modules
# Method 1: Import entire module
import math_operations
result = math_operations.add(10, 5)
print(f"10 + 5 = {result}")
# Method 2: Import specific functions
from math_operations import multiply, divide, is_prime
product = multiply(4, 7)
quotient = divide(20, 4)
prime_check = is_prime(23)
print(f"4 * 7 = {product}")
print(f"20 / 4 = {quotient}")
print(f"Is 23 prime? {prime_check}")
# Method 3: Import with alias
import math_operations as math_ops
fibonacci_seq = math_ops.fibonacci(8)
print(f"Fibonacci sequence: {fibonacci_seq}")
# Method 4: Import all (use with caution)
from math_operations import *
sqrt_result = square_root(25)
print(f"Square root of 25: {sqrt_result}")
# Using built-in modules
import random
import datetime
import os
# Random module
random_number = random.randint(1, 100)
random_choice = random.choice(['apple', 'banana', 'orange'])
print(f"Random number: {random_number}")
print(f"Random choice: {random_choice}")
# Datetime module
current_time = datetime.datetime.now()
formatted_time = current_time.strftime("%Y-%m-%d %H:%M:%S")
print(f"Current time: {formatted_time}")
# OS module
current_directory = os.getcwd()
files_in_directory = os.listdir(".")
print(f"Current directory: {current_directory}")
print(f"Files in directory: {files_in_directory[:5]}") # Show first 5 filesPackage Creation
# Create a package structure:
# my_package/
# __init__.py
# geometry/
# __init__.py
# shapes.py
# calculations.py
# utilities/
# __init__.py
# helpers.py
# validators.py
# geometry/shapes.py
"""Geometric shapes module."""
import math
class Circle:
"""Circle class."""
def __init__(self, radius):
self.radius = radius
def area(self):
return math.pi * self.radius ** 2
def circumference(self):
return 2 * math.pi * self.radius
class Rectangle:
"""Rectangle class."""
def __init__(self, width, height):
self.width = width
self.height = height
def area(self):
return self.width * self.height
def perimeter(self):
return 2 * (self.width + self.height)
# geometry/__init__.py
"""Geometry package."""
from .shapes import Circle, Rectangle
from .calculations import distance, midpoint
__version__ = "1.0.0"
__all__ = ["Circle", "Rectangle", "distance", "midpoint"]
# geometry/calculations.py
"""Geometric calculations."""
import math
def distance(point1, point2):
"""Calculate distance between two points."""
x1, y1 = point1
x2, y2 = point2
return math.sqrt((x2 - x1)**2 + (y2 - y1)**2)
def midpoint(point1, point2):
"""Calculate midpoint between two points."""
x1, y1 = point1
x2, y2 = point2
return ((x1 + x2) / 2, (y1 + y2) / 2)
# Using the package
from my_package.geometry import Circle, Rectangle, distance
# Create shapes
circle = Circle(5)
rectangle = Rectangle(4, 6)
print(f"Circle area: {circle.area():.2f}")
print(f"Rectangle area: {rectangle.area()}")
# Calculate distance
point_a = (0, 0)
point_b = (3, 4)
dist = distance(point_a, point_b)
print(f"Distance between {point_a} and {point_b}: {dist}")Advanced Topics
Generators and Iterators
graph TD
A[Generators] --> B[Memory Efficient]
A --> C[Lazy Evaluation]
A --> D[yield keyword]
A --> E[Generator Expressions]
F[Iterators] --> G[__iter__ method]
F --> H[__next__ method]
F --> I[StopIteration]
J[Benefits] --> K[Memory Conservation]
J --> L[Performance]
J --> M[Infinite Sequences]# Generator functions
def fibonacci_generator(n):
"""Generate Fibonacci sequence using generator."""
a, b = 0, 1
count = 0
while count < n:
yield a
a, b = b, a + b
count += 1
# Using generator
print("Fibonacci sequence using generator:")
for num in fibonacci_generator(10):
print(num, end=" ")
print()
# Generator expressions
squares_gen = (x**2 for x in range(1, 6))
print("Squares generator:", list(squares_gen))
# Infinite generator
def infinite_counter():
"""Generate infinite sequence of numbers."""
num = 0
while True:
yield num
num += 1
counter = infinite_counter()
print("First 5 numbers from infinite counter:")
for _ in range(5):
print(next(counter), end=" ")
print()
# Custom iterator
class CountDown:
"""Custom iterator for countdown."""
def __init__(self, start):
self.start = start
def __iter__(self):
return self
def __next__(self):
if self.start <= 0:
raise StopIteration
self.start -= 1
return self.start + 1
print("Countdown from 5:")
for num in CountDown(5):
print(num, end=" ")
print()
# Generator for file processing (memory efficient)
def read_large_file(file_path):
"""Generator to read large files line by line."""
try:
with open(file_path, 'r') as file:
for line in file:
yield line.strip()
except FileNotFoundError:
print(f"File {file_path} not found")
return
# Create a sample file for demonstration
with open("sample_data.txt", "w") as f:
for i in range(1000):
f.write(f"Line {i + 1}: Some data here\n")
# Process large file efficiently
line_count = 0
for line in read_large_file("sample_data.txt"):
line_count += 1
if line_count <= 5: # Show first 5 lines
print(line)
print(f"Total lines processed: {line_count}")Decorators
import time
import functools
# Basic decorator
def timer_decorator(func):
"""Decorator to measure function execution time."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} took {end_time - start_time:.4f} seconds")
return result
return wrapper
# Decorator with parameters
def repeat(times):
"""Decorator to repeat function execution."""
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
results = []
for _ in range(times):
result = func(*args, **kwargs)
results.append(result)
return results
return wrapper
return decorator
# Class-based decorator
class CallCounter:
"""Decorator to count function calls."""
def __init__(self, func):
self.func = func
self.call_count = 0
functools.update_wrapper(self, func)
def __call__(self, *args, **kwargs):
self.call_count += 1
print(f"{self.func.__name__} has been called {self.call_count} times")
return self.func(*args, **kwargs)
# Using decorators
@timer_decorator
@CallCounter
def slow_function():
"""A function that takes some time."""
time.sleep(0.1)
return "Task completed"
@repeat(3)
def greet(name):
"""Greet a person."""
return f"Hello, {name}!"
# Test decorated functions
result = slow_function()
print(f"Result: {result}")
slow_function() # Call again to see call count
greetings = greet("Alice")
print(f"Greetings: {greetings}")
# Property decorators for classes
class Temperature:
"""Temperature class with property decorators."""
def __init__(self, celsius=0):
self._celsius = celsius
@property
def celsius(self):
"""Get temperature in Celsius."""
return self._celsius
@celsius.setter
def celsius(self, value):
"""Set temperature in Celsius."""
if value < -273.15:
raise ValueError("Temperature cannot be below absolute zero")
self._celsius = value
@property
def fahrenheit(self):
"""Get temperature in Fahrenheit."""
return (self._celsius * 9/5) + 32
@fahrenheit.setter
def fahrenheit(self, value):
"""Set temperature in Fahrenheit."""
self.celsius = (value - 32) * 5/9
@property
def kelvin(self):
"""Get temperature in Kelvin."""
return self._celsius + 273.15
# Using property decorators
temp = Temperature(25)
print(f"Temperature: {temp.celsius}°C, {temp.fahrenheit}°F, {temp.kelvin}K")
temp.fahrenheit = 86
print(f"After setting to 86°F: {temp.celsius}°C")Context Managers
import sqlite3
import tempfile
import os
# Simple context manager using contextlib
from contextlib import contextmanager
@contextmanager
def timer_context():
"""Context manager to measure execution time."""
start_time = time.time()
print("Starting timer...")
try:
yield
finally:
end_time = time.time()
print(f"Execution took {end_time - start_time:.4f} seconds")
# Database context manager
class DatabaseConnection:
"""Context manager for database connections."""
def __init__(self, db_path):
self.db_path = db_path
self.connection = None
def __enter__(self):
print(f"Opening database connection to {self.db_path}")
self.connection = sqlite3.connect(self.db_path)
return self.connection
def __exit__(self, exc_type, exc_val, exc_tb):
if self.connection:
if exc_type is None:
print("Committing transaction...")
self.connection.commit()
else:
print("Rolling back transaction due to error...")
self.connection.rollback()
self.connection.close()
print("Database connection closed")
# File backup context manager
@contextmanager
def file_backup(file_path):
"""Context manager to backup and restore files."""
backup_path = f"{file_path}.backup"
# Create backup if original exists
if os.path.exists(file_path):
with open(file_path, 'r') as original:
with open(backup_path, 'w') as backup:
backup.write(original.read())
print(f"Backup created: {backup_path}")
try:
yield file_path
except Exception as e:
# Restore backup on error
if os.path.exists(backup_path):
with open(backup_path, 'r') as backup:
with open(file_path, 'w') as original:
original.write(backup.read())
print(f"File restored from backup due to error: {e}")
raise
finally:
# Clean up backup
if os.path.exists(backup_path):
os.remove(backup_path)
print("Backup file removed")
# Using context managers
print("Testing timer context manager:")
with timer_context():
time.sleep(0.5)
result = sum(range(1000000))
print(f"\nResult: {result}")
# Database operations with context manager
with DatabaseConnection(":memory:") as conn:
cursor = conn.cursor()
cursor.execute("""
CREATE TABLE users (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
email TEXT UNIQUE
)
""")
cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)",
("Alice", "alice@example.com"))
cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)",
("Bob", "bob@example.com"))
cursor.execute("SELECT * FROM users")
users = cursor.fetchall()
print("Users in database:")
for user in users:
print(f" {user}")
# File backup example
test_file = "test_file.txt"
with open(test_file, 'w') as f:
f.write("Original content")
print(f"\nTesting file backup context manager:")
try:
with file_backup(test_file) as file_path:
with open(file_path, 'w') as f:
f.write("Modified content")
# Simulate an error
# raise Exception("Something went wrong!")
print("File modified successfully")
except Exception as e:
print(f"Error occurred: {e}")
# Check final file content
with open(test_file, 'r') as f:
print(f"Final file content: {f.read()}")Web Development
Web Development Overview
graph TD
A[Web Development with Python] --> B[Frameworks]
A --> C[Components]
A --> D[Deployment]
B --> B1[Flask - Lightweight]
B --> B2[Django - Full-featured]
B --> B3[FastAPI - Modern & Fast]
C --> C1[Templates]
C --> C2[Forms]
C --> C3[Database]
C --> C4[Authentication]
C --> C5[APIs]
D --> D1[Heroku]
D --> D2[AWS]
D --> D3[Docker]Flask Web Application
# Install Flask: pip install flask
from flask import Flask, render_template, request, jsonify, redirect, url_for
import sqlite3
import os
app = Flask(__name__)
app.secret_key = 'your-secret-key-here'
# Database setup
def init_db():
"""Initialize the database."""
conn = sqlite3.connect('blog.db')
cursor = conn.cursor()
cursor.execute('''
CREATE TABLE IF NOT EXISTS posts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
title TEXT NOT NULL,
content TEXT NOT NULL,
author TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
''')
conn.commit()
conn.close()
def get_db_connection():
"""Get database connection."""
conn = sqlite3.connect('blog.db')
conn.row_factory = sqlite3.Row
try:
yield conn
finally:
conn.close()
def init_fastapi_db():
"""Initialize FastAPI database."""
conn = sqlite3.connect('fastapi_blog.db')
cursor = conn.cursor()
cursor.execute('''
CREATE TABLE IF NOT EXISTS posts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
title TEXT NOT NULL,
content TEXT NOT NULL,
author TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
''')
conn.commit()
conn.close()
# API Endpoints
@app.get("/", response_class=HTMLResponse)
async def read_root():
"""Serve the main page."""
html_content = """
<!DOCTYPE html>
<html>
<head>
<title>FastAPI Blog</title>
<style>
body { font-family: Arial, sans-serif; margin: 40px; }
.post { border: 1px solid #ddd; padding: 20px; margin: 10px 0; }
.nav { margin-bottom: 20px; }
.nav a { margin-right: 10px; color: #007bff; text-decoration: none; }
</style>
</head>
<body>
<h1>FastAPI Blog</h1>
<div class="nav">
<a href="/docs">API Documentation</a>
<a href="/api/posts">View Posts JSON</a>
</div>
<p>Use the API endpoints to manage blog posts. Check the documentation at <a href="/docs">/docs</a></p>
</body>
</html>
"""
return HTMLResponse(content=html_content)
@app.get("/api/posts", response_model=List[Post])
async def get_posts(db: sqlite3.Connection = Depends(get_db)):
"""Get all blog posts."""
cursor = db.cursor()
cursor.execute('SELECT * FROM posts ORDER BY created_at DESC')
posts = cursor.fetchall()
return [Post(
id=post[0],
title=post[1],
content=post[2],
author=post[3],
created_at=post[4]
) for post in posts]
@app.get("/api/posts/{post_id}", response_model=Post)
async def get_post(post_id: int, db: sqlite3.Connection = Depends(get_db)):
"""Get a specific post by ID."""
cursor = db.cursor()
cursor.execute('SELECT * FROM posts WHERE id = ?', (post_id,))
post = cursor.fetchone()
if not post:
raise HTTPException(status_code=404, detail="Post not found")
return Post(
id=post[0],
title=post[1],
content=post[2],
author=post[3],
created_at=post[4]
)
@app.post("/api/posts", response_model=dict)
async def create_post(post: PostCreate, db: sqlite3.Connection = Depends(get_db)):
"""Create a new blog post."""
cursor = db.cursor()
cursor.execute('INSERT INTO posts (title, content, author) VALUES (?, ?, ?)',
(post.title, post.content, post.author))
post_id = cursor.lastrowid
db.commit()
return {"message": "Post created successfully", "post_id": post_id}
@app.delete("/api/posts/{post_id}")
async def delete_post(post_id: int, db: sqlite3.Connection = Depends(get_db)):
"""Delete a blog post."""
cursor = db.cursor()
cursor.execute('DELETE FROM posts WHERE id = ?', (post_id,))
if cursor.rowcount == 0:
raise HTTPException(status_code=404, detail="Post not found")
db.commit()
return {"message": "Post deleted successfully"}
if __name__ == "__main__":
init_fastapi_db()
uvicorn.run(app, host="0.0.0.0", port=8000)REST API Client
# API client example using requests library
import requests
import json
class BlogAPIClient:
"""Client for interacting with the blog API."""
def __init__(self, base_url="http://localhost:8000"):
self.base_url = base_url
def get_posts(self):
"""Get all posts."""
response = requests.get(f"{self.base_url}/api/posts")
response.raise_for_status()
return response.json()
def get_post(self, post_id):
"""Get a specific post."""
response = requests.get(f"{self.base_url}/api/posts/{post_id}")
response.raise_for_status()
return response.json()
def create_post(self, title, content, author):
"""Create a new post."""
data = {
"title": title,
"content": content,
"author": author
}
response = requests.post(
f"{self.base_url}/api/posts",
json=data,
headers={"Content-Type": "application/json"}
)
response.raise_for_status()
return response.json()
def delete_post(self, post_id):
"""Delete a post."""
response = requests.delete(f"{self.base_url}/api/posts/{post_id}")
response.raise_for_status()
return response.json()
# Example usage
if __name__ == "__main__":
client = BlogAPIClient()
# Create a new post
new_post = client.create_post(
title="My First API Post",
content="This post was created using the API client!",
author="API User"
)
print(f"Created post: {new_post}")
# Get all posts
posts = client.get_posts()
print(f"Total posts: {len(posts)}")
# Get specific post
if posts:
first_post = client.get_post(posts[0]['id'])
print(f"First post: {first_post['title']}")Data Science and Analytics
Data Science Ecosystem
graph TD
A[Data Science with Python] --> B[Data Collection]
A --> C[Data Processing]
A --> D[Analysis & Visualization]
A --> E[Machine Learning]
B --> B1[Web Scraping]
B --> B2[APIs]
B --> B3[Databases]
B --> B4[Files - CSV/JSON/Excel]
C --> C1[Pandas]
C --> C2[NumPy]
C --> C3[Data Cleaning]
D --> D1[Matplotlib]
D --> D2[Seaborn]
D --> D3[Plotly]
D --> D4[Statistical Analysis]
E --> E1[Scikit-learn]
E --> E2[TensorFlow]
E --> E3[PyTorch]NumPy Fundamentals
# Install required packages: pip install numpy pandas matplotlib seaborn
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# NumPy basics
print("=== NumPy Fundamentals ===")
# Creating arrays
arr1 = np.array([1, 2, 3, 4, 5])
arr2 = np.array([[1, 2, 3], [4, 5, 6]])
arr3 = np.zeros((3, 4))
arr4 = np.ones((2, 3))
arr5 = np.arange(0, 10, 2) # Start, stop, step
arr6 = np.linspace(0, 1, 5) # Start, stop, number of points
print(f"1D Array: {arr1}")
print(f"2D Array:\n{arr2}")
print(f"Shape of arr2: {arr2.shape}")
print(f"Data type: {arr1.dtype}")
# Array operations
print("\n=== Array Operations ===")
a = np.array([1, 2, 3, 4])
b = np.array([5, 6, 7, 8])
print(f"Addition: {a + b}")
print(f"Multiplication: {a * b}")
print(f"Dot product: {np.dot(a, b)}")
print(f"Square root: {np.sqrt(a)}")
# Statistical operations
data = np.random.normal(100, 15, 1000) # Mean=100, Std=15, 1000 samples
print(f"\nStatistical Operations:")
print(f"Mean: {np.mean(data):.2f}")
print(f"Median: {np.median(data):.2f}")
print(f"Standard deviation: {np.std(data):.2f}")
print(f"Min: {np.min(data):.2f}, Max: {np.max(data):.2f}")
# Array indexing and slicing
matrix = np.array([[1, 2, 3, 4],
[5, 6, 7, 8],
[9, 10, 11, 12]])
print(f"\nMatrix:\n{matrix}")
print(f"Element at [1,2]: {matrix[1, 2]}")
print(f"First row: {matrix[0, :]}")
print(f"Second column: {matrix[:, 1]}")
print(f"Submatrix:\n{matrix[1:3, 1:3]}")
# Boolean indexing
arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
even_numbers = arr[arr % 2 == 0]
print(f"Even numbers: {even_numbers}")Pandas Data Manipulation
print("\n=== Pandas Data Manipulation ===")
# Creating DataFrames
data = {
'Name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
'Age': [25, 30, 35, 28, 32],
'City': ['New York', 'London', 'Tokyo', 'Paris', 'Sydney'],
'Salary': [70000, 80000, 90000, 75000, 85000],
'Department': ['IT', 'Finance', 'IT', 'HR', 'Finance']
}
df = pd.DataFrame(data)
print("Original DataFrame:")
print(df)
# Basic DataFrame operations
print(f"\nDataFrame Info:")
print(f"Shape: {df.shape}")
print(f"Columns: {list(df.columns)}")
print(f"Data types:\n{df.dtypes}")
# Data selection and filtering
print("\n=== Data Selection ===")
print(f"Names column:\n{df['Name']}")
print(f"First 3 rows:\n{df.head(3)}")
print(f"IT employees:\n{df[df['Department'] == 'IT']}")
print(f"High earners (>80000):\n{df[df['Salary'] > 80000]}")
# Adding new columns
df['Salary_USD'] = df['Salary']
df['Salary_EUR'] = df['Salary'] * 0.85 # Approximate conversion
df['Experience_Level'] = df['Age'].apply(lambda x: 'Senior' if x >= 30 else 'Junior')
print(f"\nDataFrame with new columns:\n{df}")
# Grouping and aggregation
print("\n=== Grouping and Aggregation ===")
dept_stats = df.groupby('Department').agg({
'Salary': ['mean', 'min', 'max'],
'Age': 'mean'
}).round(2)
print("Department Statistics:")
print(dept_stats)
# Data manipulation examples
print("\n=== Advanced Data Manipulation ===")
# Creating a larger dataset for demonstration
np.random.seed(42)
large_data = {
'Date': pd.date_range('2024-01-01', periods=100),
'Sales': np.random.normal(1000, 200, 100),
'Product': np.random.choice(['A', 'B', 'C'], 100),
'Region': np.random.choice(['North', 'South', 'East', 'West'], 100),
'Customer_Count': np.random.poisson(50, 100)
}
sales_df = pd.DataFrame(large_data)
sales_df['Month'] = sales_df['Date'].dt.month
sales_df['Quarter'] = sales_df['Date'].dt.quarter
print("Sales DataFrame (first 10 rows):")
print(sales_df.head(10))
# Pivot tables
print("\nPivot Table - Average Sales by Product and Region:")
pivot_table = sales_df.pivot_table(
values='Sales',
index='Product',
columns='Region',
aggfunc='mean'
).round(2)
print(pivot_table)
# Time series analysis
monthly_sales = sales_df.groupby('Month')['Sales'].sum()
print(f"\nMonthly Sales:\n{monthly_sales}")Data Visualization
print("\n=== Data Visualization ===")
# Set up the plotting style
plt.style.use('seaborn-v0_8')
fig, axes = plt.subplots(2, 2, figsize=(15, 12))
# 1. Line plot - Monthly sales trend
axes[0, 0].plot(monthly_sales.index, monthly_sales.values, marker='o', linewidth=2)
axes[0, 0].set_title('Monthly Sales Trend', fontsize=14)
axes[0, 0].set_xlabel('Month')
axes[0, 0].set_ylabel('Total Sales')
axes[0, 0].grid(True, alpha=0.3)
# 2. Bar plot - Sales by product
product_sales = sales_df.groupby('Product')['Sales'].mean()
axes[0, 1].bar(product_sales.index, product_sales.values, color=['#FF6B6B', '#4ECDC4', '#45B7D1'])
axes[0, 1].set_title('Average Sales by Product', fontsize=14)
axes[0, 1].set_xlabel('Product')
axes[0, 1].set_ylabel('Average Sales')
# 3. Histogram - Sales distribution
axes[1, 0].hist(sales_df['Sales'], bins=20, color='skyblue', alpha=0.7, edgecolor='black')
axes[1, 0].set_title('Sales Distribution', fontsize=14)
axes[1, 0].set_xlabel('Sales Amount')
axes[1, 0].set_ylabel('Frequency')
# 4. Scatter plot - Sales vs Customer Count
axes[1, 1].scatter(sales_df['Customer_Count'], sales_df['Sales'],
c=sales_df['Month'], cmap='viridis', alpha=0.6)
axes[1, 1].set_title('Sales vs Customer Count', fontsize=14)
axes[1, 1].set_xlabel('Customer Count')
axes[1, 1].set_ylabel('Sales')
colorbar = plt.colorbar(axes[1, 1].collections[0], ax=axes[1, 1])
colorbar.set_label('Month')
plt.tight_layout()
plt.savefig('sales_analysis.png', dpi=300, bbox_inches='tight')
plt.show()
# Advanced visualization with Seaborn
fig, axes = plt.subplots(2, 2, figsize=(15, 12))
# 1. Box plot - Sales by region
sns.boxplot(data=sales_df, x='Region', y='Sales', ax=axes[0, 0])
axes[0, 0].set_title('Sales Distribution by Region')
# 2. Heatmap - Correlation matrix
correlation_data = sales_df[['Sales', 'Customer_Count', 'Month', 'Quarter']].corr()
sns.heatmap(correlation_data, annot=True, cmap='coolwarm', center=0, ax=axes[0, 1])
axes[0, 1].set_title('Correlation Heatmap')
# 3. Violin plot - Sales by quarter
sns.violinplot(data=sales_df, x='Quarter', y='Sales', ax=axes[1, 0])
axes[1, 0].set_title('Sales Distribution by Quarter')
# 4. Pair plot data preparation and count plot
sns.countplot(data=sales_df, x='Product', hue='Region', ax=axes[1, 1])
axes[1, 1].set_title('Product Count by Region')
axes[1, 1].legend(title='Region', bbox_to_anchor=(1.05, 1), loc='upper left')
plt.tight_layout()
plt.savefig('advanced_sales_analysis.png', dpi=300, bbox_inches='tight')
plt.show()Basic Machine Learning
# Install scikit-learn: pip install scikit-learn
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import mean_squared_error, accuracy_score, classification_report
from sklearn.preprocessing import StandardScaler, LabelEncoder
print("\n=== Machine Learning Examples ===")
# 1. Linear Regression Example
print("1. Linear Regression - Predicting Sales")
# Prepare data for regression
X_reg = sales_df[['Customer_Count', 'Month', 'Quarter']].values
y_reg = sales_df['Sales'].values
# Split the data
X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(
X_reg, y_reg, test_size=0.2, random_state=42
)
# Train the model
reg_model = LinearRegression()
reg_model.fit(X_train_reg, y_train_reg)
# Make predictions
y_pred_reg = reg_model.predict(X_test_reg)
# Evaluate
mse = mean_squared_error(y_test_reg, y_pred_reg)
print(f"Mean Squared Error: {mse:.2f}")
print(f"R-squared Score: {reg_model.score(X_test_reg, y_test_reg):.3f}")
# Feature importance
feature_names = ['Customer_Count', 'Month', 'Quarter']
for name, coef in zip(feature_names, reg_model.coef_):
print(f"{name}: {coef:.2f}")
# 2. Classification Example
print("\n2. Classification - Predicting High/Low Sales")
# Create binary target variable
sales_median = sales_df['Sales'].median()
sales_df['High_Sales'] = (sales_df['Sales'] > sales_median).astype(int)
# Prepare data for classification
le_product = LabelEncoder()
le_region = LabelEncoder()
X_class = pd.DataFrame({
'Customer_Count': sales_df['Customer_Count'],
'Month': sales_df['Month'],
'Product': le_product.fit_transform(sales_df['Product']),
'Region': le_region.fit_transform(sales_df['Region'])
})
y_class = sales_df['High_Sales'].values
# Split and scale the data
X_train_class, X_test_class, y_train_class, y_test_class = train_test_split(
X_class, y_class, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train_class)
X_test_scaled = scaler.transform(X_test_class)
# Train models
log_model = LogisticRegression(random_state=42)
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
log_model.fit(X_train_scaled, y_train_class)
rf_model.fit(X_train_class, y_train_class) # Random Forest doesn't need scaling
# Make predictions
log_pred = log_model.predict(X_test_scaled)
rf_pred = rf_model.predict(X_test_class)
# Evaluate models
print("Logistic Regression Accuracy:", accuracy_score(y_test_class, log_pred))
print("Random Forest Accuracy:", accuracy_score(y_test_class, rf_pred))
print("\nRandom Forest Classification Report:")
print(classification_report(y_test_class, rf_pred))
# Feature importance for Random Forest
print("\nFeature Importance (Random Forest):")
for name, importance in zip(X_class.columns, rf_model.feature_importances_):
print(f"{name}: {importance:.3f}")
# 3. Data Analysis Summary
print("\n=== Data Analysis Summary ===")
summary_stats = sales_df.describe()
print("Summary Statistics:")
print(summary_stats)
# Create a comprehensive analysis report
analysis_report = f"""
DATA ANALYSIS REPORT
===================
Dataset Overview:
- Total Records: {len(sales_df)}
- Date Range: {sales_df['Date'].min()} to {sales_df['Date'].max()}
- Products: {', '.join(sales_df['Product'].unique())}
- Regions: {', '.join(sales_df['Region'].unique())}
Key Insights:
- Average Sales: ${sales_df['Sales'].mean():.2f}
- Total Sales: ${sales_df['Sales'].sum():.2f}
- Best Performing Product: {product_sales.idxmax()} (${product_sales.max():.2f} avg)
- Most Active Region: {sales_df['Region'].value_counts().index[0]}
Model Performance:
- Sales Prediction R²: {reg_model.score(X_test_reg, y_test_reg):.3f}
- High/Low Sales Classification: {accuracy_score(y_test_class, rf_pred):.3f}
"""
print(analysis_report)
# Save analysis results
with open('analysis_report.txt', 'w') as f:
f.write(analysis_report)
print("Analysis report saved to 'analysis_report.txt'")Testing and Debugging
Testing Framework
graph TD
A[Python Testing] --> B[Unit Testing]
A --> C[Integration Testing]
A --> D[Test-Driven Development]
B --> B1[unittest]
B --> B2[pytest]
B --> B3[doctest]
C --> C1[API Testing]
C --> C2[Database Testing]
C --> C3[End-to-End Testing]
D --> D1[Write Tests First]
D --> D2[Implement Code]
D --> D3[Refactor]Unit Testing with unittest
# calculator.py - Module to test
class Calculator:
"""A simple calculator class."""
def add(self, a, b):
"""Add two numbers."""
return a + b
def subtract(self, a, b):
"""Subtract two numbers."""
return a - b
def multiply(self, a, b):
"""Multiply two numbers."""
return a * b
def divide(self, a, b):
"""Divide two numbers."""
if b == 0:
raise ValueError("Cannot divide by zero")
return a / b
def power(self, base, exponent):
"""Calculate base raised to exponent."""
return base ** exponent
# test_calculator.py - Unit tests
import unittest
import sys
import os
class TestCalculator(unittest.TestCase):
"""Test cases for Calculator class."""
def setUp(self):
"""Set up test fixtures before each test method."""
self.calc = Calculator()
def test_add(self):
"""Test addition operation."""
self.assertEqual(self.calc.add(2, 3), 5)
self.assertEqual(self.calc.add(-1, 1), 0)
self.assertEqual(self.calc.add(0, 0), 0)
def test_subtract(self):
"""Test subtraction operation."""
self.assertEqual(self.calc.subtract(5, 3), 2)
self.assertEqual(self.calc.subtract(0, 5), -5)
self.assertEqual(self.calc.subtract(-2, -3), 1)
def test_multiply(self):
"""Test multiplication operation."""
self.assertEqual(self.calc.multiply(3, 4), 12)
self.assertEqual(self.calc.multiply(-2, 3), -6)
self.assertEqual(self.calc.multiply(0, 100), 0)
def test_divide(self):
"""Test division operation."""
self.assertEqual(self.calc.divide(8, 2), 4)
self.assertEqual(self.calc.divide(7, 2), 3.5)
self.assertAlmostEqual(self.calc.divide(1, 3), 0.33333, places=4)
def test_divide_by_zero(self):
"""Test division by zero raises ValueError."""
with self.assertRaises(ValueError):
self.calc.divide(5, 0)
# Test the error message
with self.assertRaisesRegex(ValueError, "Cannot divide by zero"):
self.calc.divide(10, 0)
def test_power(self):
"""Test power operation."""
self.assertEqual(self.calc.power(2, 3), 8)
self.assertEqual(self.calc.power(5, 0), 1)
self.assertEqual(self.calc.power(10, -1), 0.1)
def tearDown(self):
"""Clean up after each test method."""
pass # Nothing to clean up for this simple example
class TestCalculatorEdgeCases(unittest.TestCase):
"""Test edge cases for Calculator class."""
def setUp(self):
self.calc = Calculator()
def test_large_numbers(self):
"""Test with very large numbers."""
large_num = 10**10
self.assertEqual(self.calc.add(large_num, large_num), 2 * large_num)
def test_floating_point_precision(self):
"""Test floating point operations."""
result = self.calc.add(0.1, 0.2)
self.assertAlmostEqual(result, 0.3, places=10)
def test_negative_numbers(self):
"""Test with negative numbers."""
self.assertEqual(self.calc.multiply(-5, -4), 20)
self.assertEqual(self.calc.divide(-10, -2), 5)
# Run tests
if __name__ == '__main__':
# Create a test suite
suite = unittest.TestLoader().loadTestsFromTestCase(TestCalculator)
suite.addTests(unittest.TestLoader().loadTestsFromTestCase(TestCalculatorEdgeCases))
# Run tests with detailed output
runner = unittest.TextTestRunner(verbosity=2)
result = runner.run(suite)
# Print summary
print(f"\nTests run: {result.testsRun}")
print(f"Failures: {len(result.failures)}")
print(f"Errors: {len(result.errors)}")Testing with pytest
# Install pytest: pip install pytest
# test_calculator_pytest.py
import pytest
from calculator import Calculator
class TestCalculatorPytest:
"""Pytest test cases for Calculator."""
@pytest.fixture
def calc(self):
"""Fixture to provide Calculator instance."""
return Calculator()
def test_add(self, calc):
"""Test addition with pytest."""
assert calc.add(2, 3) == 5
assert calc.add(-1, 1) == 0
assert calc.add(0, 0) == 0
def test_subtract(self, calc):
"""Test subtraction with pytest."""
assert calc.subtract(5, 3) == 2
assert calc.subtract(0, 5) == -5
def test_multiply(self, calc):
"""Test multiplication with pytest."""
assert calc.multiply(3, 4) == 12
assert calc.multiply(-2, 3) == -6
def test_divide(self, calc):
"""Test division with pytest."""
assert calc.divide(8, 2) == 4
assert abs(calc.divide(1, 3) - 0.33333) < 0.00001
def test_divide_by_zero(self, calc):
"""Test division by zero with pytest."""
with pytest.raises(ValueError, match="Cannot divide by zero"):
calc.divide(5, 0)
@pytest.mark.parametrize("a,b,expected", [
(2, 3, 8),
(5, 0, 1),
(10, 1, 10),
(2, -1, 0.5),
])
def test_power_parametrized(self, calc, a, b, expected):
"""Test power operation with multiple parameters."""
assert calc.power(a, b) == expected
@pytest.mark.slow
def test_performance(self, calc):
"""Test performance - marked as slow."""
# This test would normally test performance
result = sum(calc.add(i, i) for i in range(10000))
assert result > 0
# Advanced pytest features
def test_temporary_file(tmp_path):
"""Test using temporary directory fixture."""
# tmp_path is a pytest fixture that provides a temporary directory
test_file = tmp_path / "test.txt"
test_file.write_text("Hello, World!")
assert test_file.read_text() == "Hello, World!"
assert test_file.exists()
@pytest.fixture(scope="session")
def database_connection():
"""Session-scoped fixture for database connection."""
# This would normally create a real database connection
connection = {"status": "connected", "type": "test_db"}
yield connection
# Cleanup code would go here
connection["status"] = "disconnected"
def test_database_operation(database_connection):
"""Test using session-scoped fixture."""
assert database_connection["status"] == "connected"
# Custom pytest markers (add to pytest.ini or pyproject.toml)
# [tool:pytest]
# markers =
# slow: marks tests as slow
# integration: marks tests as integration tests
# unit: marks tests as unit tests
@pytest.mark.integration
def test_api_integration():
"""Integration test example."""
# This would test actual API integration
assert True
# Conftest.py - shared fixtures
# This file would contain shared fixtures across multiple test filesDebugging Techniques
import pdb
import logging
import traceback
from functools import wraps
# 1. Logging for debugging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('app.log'),
logging.StreamHandler()
]
)
logger = logging.getLogger(__name__)
def debug_function_calls(func):
"""Decorator to log function calls and results."""
@wraps(func)
def wrapper(*args, **kwargs):
logger.debug(f"Calling {func.__name__} with args={args}, kwargs={kwargs}")
try:
result = func(*args, **kwargs)
logger.debug(f"{func.__name__} returned: {result}")
return result
except Exception as e:
logger.error(f"Error in {func.__name__}: {e}")
logger.error(traceback.format_exc())
raise
return wrapper
@debug_function_calls
def complex_calculation(x, y, debug_mode=False):
"""Example function with debugging features."""
logger.info(f"Starting complex calculation with x={x}, y={y}")
if debug_mode:
# Set breakpoint for debugging
pdb.set_trace()
try:
# Some complex calculations
step1 = x * 2
logger.debug(f"Step 1 result: {step1}")
step2 = step1 + y
logger.debug(f"Step 2 result: {step2}")
if step2 == 0:
raise ValueError("Result cannot be zero")
result = 100 / step2
logger.debug(f"Final result: {result}")
return result
except Exception as e:
logger.error(f"Error in calculation: {e}")
raise
# 2. Custom debugging context manager
class DebugContext:
"""Context manager for debugging code blocks."""
def __init__(self, name, log_level=logging.DEBUG):
self.name = name
self.log_level = log_level
self.logger = logging.getLogger(f"DebugContext.{name}")
def __enter__(self):
self.logger.log(self.log_level, f"Entering {self.name}")
return self
def __exit__(self, exc_type, exc_val, exc_tb):
if exc_type:
self.logger.error(f"Exception in {self.name}: {exc_val}")
else:
self.logger.log(self.log_level, f"Exiting {self.name} successfully")
# 3. Performance debugging
class PerformanceMonitor:
"""Monitor performance of code blocks."""
def __init__(self, name):
self.name = name
self.start_time = None
def __enter__(self):
self.start_time = time.time()
logger.info(f"Starting performance monitoring for {self.name}")
return self
def __exit__(self, exc_type, exc_val, exc_tb):
end_time = time.time()
duration = end_time - self.start_time
logger.info(f"Performance monitoring for {self.name}: {duration:.4f} seconds")
# Example usage of debugging tools
def demonstrate_debugging():
"""Demonstrate various debugging techniques."""
print("=== Debugging Demonstration ===")
# 1. Basic function with logging
try:
result1 = complex_calculation(5, 3)
print(f"Result 1: {result1}")
except Exception as e:
print(f"Error: {e}")
# 2. Using debug context
with DebugContext("data_processing"):
data = [1, 2, 3, 4, 5]
processed = [x * 2 for x in data]
logger.debug(f"Processed data: {processed}")
# 3. Performance monitoring
with PerformanceMonitor("large_calculation"):
# Simulate some time-consuming operation
total = sum(x**2 for x in range(100000))
logger.info(f"Calculation result: {total}")
# 4. Exception handling with detailed logging
try:
with DebugContext("error_prone_operation"):
risky_operation()
except Exception as e:
logger.error("Caught exception in demonstration")
print(f"Handled error: {e}")
def risky_operation():
"""Function that might raise an exception."""
import random
if random.random() < 0.5:
raise ValueError("Random error for demonstration")
return "Success!"
# Advanced debugging techniques
class Debugger:
"""Custom debugger with various utilities."""
@staticmethod
def print_variables(local_vars, filter_private=True):
"""Print all local variables."""
print("=== Local Variables ===")
for name, value in local_vars.items():
if filter_private and name.startswith('_'):
continue
print(f"{name}: {value} ({type(value).__name__})")
@staticmethod
def trace_calls(frame, event, arg):
"""Trace function calls."""
if event == 'call':
code = frame.f_code
print(f"Calling: {code.co_filename}:{code.co_firstlineno} {code.co_name}")
return Debugger.trace_calls
@staticmethod
def memory_usage():
"""Get current memory usage."""
import psutil
import os
process = psutil.Process(os.getpid())
memory_mb = process.memory_info().rss / 1024 / 1024
return f"Memory usage: {memory_mb:.2f} MB"
# Run debugging demonstration
if __name__ == "__main__":
demonstrate_debugging()
# Example of using the debugger utilities
debugger = Debugger()
print(debugger.memory_usage())
# Print local variables in current scope
local_var1 = "test"
local_var2 = 42
debugger.print_variables(locals())Best Practices
Code Quality and Style
graph TD
A[Python Best Practices] --> B[Code Style]
A --> C[Documentation]
A --> D[Error Handling]
A --> E[Performance]
A --> F[Security]
B --> B1[PEP 8]
B --> B2[Type Hints]
B --> B3[Code Formatting]
C --> C1[Docstrings]
C --> C2[Comments]
C --> C3[README Files]
D --> D1[Exception Handling]
D --> D2[Input Validation]
D --> D3[Logging]
E --> E1[Profiling]
E --> E2[Optimization]
E --> E3[Memory Management]Code Style and Standards
"""
Best Practices Example Module
This module demonstrates Python best practices including:
- PEP 8 compliance
- Type hints
- Proper documentation
- Error handling
- Design patterns
"""
from typing import List, Dict, Optional, Union, Callable, Any
from dataclasses import dataclass
from enum import Enum
import logging
from contextlib import contextmanager
# Configure logging
logging.basicConfig(level=logging.INFO)
# Constants (use UPPER_CASE)
MAX_RETRY_ATTEMPTS = 3
DEFAULT_TIMEOUT = 30
API_BASE_URL = "https://api.example.com"
# Enums for better code organization
class UserRole(Enum):
"""User roles enumeration."""
ADMIN = "admin"
USER = "user"
GUEST = "guest"
class Status(Enum):
"""Status enumeration."""
PENDING = "pending"
COMPLETED = "completed"
FAILED = "failed"
# Data classes for structured data
@dataclass
class User:
"""User data class with validation."""
name: str
email: str
age: int
role: UserRole = UserRole.USER
def __post_init__(self) -> None:
"""Validate user data after initialization."""
if not self.name.strip():
raise ValueError("Name cannot be empty")
if not self._is_valid_email(self.email):
raise ValueError("Invalid email format")
if self.age < 0 or self.age > 150:
raise ValueError("Age must be between 0 and 150")
@staticmethod
def _is_valid_email(email: str) -> bool:
"""Validate email format."""
return "@" in email and "." in email.split("@")[1]
def is_admin(self) -> bool:
"""Check if user has admin privileges."""
return self.role == UserRole.ADMIN
@dataclass
class Task:
"""Task data class."""
id: int
title: str
description: str
status: Status = Status.PENDING
assigned_to: Optional[User] = None
# Main classes with proper design patterns
class TaskManager:
"""
Task management class following best practices.
This class demonstrates:
- Single Responsibility Principle
- Proper error handling
- Type hints
- Documentation
- Design patterns
"""
def __init__(self) -> None:
"""Initialize TaskManager."""
self._tasks: Dict[int, Task] = {}
self._next_id: int = 1
logger.info("TaskManager initialized")
def create_task(
self,
title: str,
description: str,
assigned_to: Optional[User] = None
) -> Task:
"""
Create a new task.
Args:
title: Task title
description: Task description
assigned_to: User assigned to the task
Returns:
Created Task object
Raises:
ValueError: If title or description is empty
"""
if not title.strip():
raise ValueError("Task title cannot be empty")
if not description.strip():
raise ValueError("Task description cannot be empty")
task = Task(
id=self._next_id,
title=title.strip(),
description=description.strip(),
assigned_to=assigned_to
)
self._tasks[self._next_id] = task
self._next_id += 1
logger.info(f"Created task {task.id}: {task.title}")
return task
def get_task(self, task_id: int) -> Optional[Task]:
"""
Get task by ID.
Args:
task_id: Task ID
Returns:
Task object or None if not found
"""
return self._tasks.get(task_id)
def update_task_status(self, task_id: int, status: Status) -> bool:
"""
Update task status.
Args:
task_id: Task ID
status: New status
Returns:
True if updated successfully, False if task not found
"""
task = self._tasks.get(task_id)
if task is None:
logger.warning(f"Task {task_id} not found for status update")
return False
old_status = task.status
task.status = status
logger.info(f"Task {task_id} status changed from {old_status.value} to {status.value}")
return True
def get_tasks_by_status(self, status: Status) -> List[Task]:
"""
Get all tasks with given status.
Args:
status: Status to filter by
Returns:
List of tasks with the specified status
"""
return [task for task in self._tasks.values() if task.status == status]
def get_user_tasks(self, user: User) -> List[Task]:
"""
Get all tasks assigned to a specific user.
Args:
user: User object
Returns:
List of tasks assigned to the user
"""
return [
task for task in self._tasks.values()
if task.assigned_to == user
]
def delete_task(self, task_id: int) -> bool:
"""
Delete a task.
Args:
task_id: Task ID
Returns:
True if deleted successfully, False if task not found
"""
if task_id in self._tasks:
del self._tasks[task_id]
logger.info(f"Deleted task {task_id}")
return True
logger.warning(f"Task {task_id} not found for deletion")
return False
def get_task_summary(self) -> Dict[str, int]:
"""
Get summary of tasks by status.
Returns:
Dictionary with status counts
"""
summary = {status.value: 0 for status in Status}
for task in self._tasks.values():
summary[task.status.value] += 1
return summary
def __len__(self) -> int:
"""Return number of tasks."""
return len(self._tasks)
def __str__(self) -> str:
"""String representation of TaskManager."""
return f"TaskManager with {len(self._tasks)} tasks"
# Utility functions with proper typing and documentation
def validate_input(
value: Any,
validators: List[Callable[[Any], bool]],
error_message: str = "Validation failed"
) -> None:
"""
Validate input using multiple validators.
Args:
value: Value to validate
validators: List of validator functions
error_message: Error message if validation fails
Raises:
ValueError: If any validator fails
"""
for validator in validators:
if not validator(value):
raise ValueError(error_message)
def retry_operation(
operation: Callable[[], Any],
max_attempts: int = MAX_RETRY_ATTEMPTS,
delay: float = 1.0
) -> Any:
"""
Retry an operation with exponential backoff.
Args:
operation: Function to retry
max_attempts: Maximum number of attempts
delay: Initial delay between retries
Returns:
Result of the operation
Raises:
Exception: Last exception if all attempts fail
"""
import time
for attempt in range(max_attempts):
try:
return operation()
except Exception as e:
if attempt == max_attempts - 1:
logger.error(f"Operation failed after {max_attempts} attempts: {e}")
raise
logger.warning(f"Attempt {attempt + 1} failed: {e}. Retrying in {delay} seconds...")
time.sleep(delay)
delay *= 2 # Exponential backoff
@contextmanager
def performance_monitor(operation_name: str):
"""
Context manager for monitoring performance.
Args:
operation_name: Name of the operation being monitored
"""
import time
start_time = time.time()
logger.info(f"Starting {operation_name}")
try:
yield
finally:
end_time = time.time()
duration = end_time - start_time
logger.info(f"{operation_name} completed in {duration:.2f} seconds")
# Example usage demonstrating best practices
def demonstrate_best_practices() -> None:
"""Demonstrate best practices with comprehensive example."""
print("=== Python Best Practices Demonstration ===")
# Create users with validation
try:
admin_user = User(
name="Alice Admin",
email="alice@example.com",
age=30,
role=UserRole.ADMIN
)
regular_user = User(
name="Bob User",
email="bob@example.com",
age=25
)
print(f"Created users: {admin_user.name}, {regular_user.name}")
except ValueError as e:
print(f"User creation error: {e}")
return
# Create task manager and demonstrate operations
with performance_monitor("TaskManager operations"):
task_manager = TaskManager()
# Create tasks
task1 = task_manager.create_task(
"Implement login feature",
"Create user authentication system",
assigned_to=regular_user
)
task2 = task_manager.create_task(
"Review code",
"Review pull requests",
assigned_to=admin_user
)
task3 = task_manager.create_task(
"Update documentation",
"Update API documentation"
)
print(f"Created {len(task_manager)} tasks")
# Update task status
task_manager.update_task_status(task1.id, Status.COMPLETED)
task_manager.update_task_status(task2.id, Status.PENDING)
# Get tasks by various criteria
completed_tasks = task_manager.get_tasks_by_status(Status.COMPLETED)
user_tasks = task_manager.get_user_tasks(regular_user)
print(f"Completed tasks: {len(completed_tasks)}")
print(f"Tasks assigned to {regular_user.name}: {len(user_tasks)}")
# Get summary
summary = task_manager.get_task_summary()
print("Task Summary:")
for status, count in summary.items():
print(f" {status}: {count}")
# Demonstrate error handling and retry mechanism
def unreliable_operation():
"""Simulate an unreliable operation."""
import random
if random.random() < 0.7: # 70% chance of failure
raise ConnectionError("Network error")
return "Operation successful"
try:
with performance_monitor("Retry operation"):
result = retry_operation(unreliable_operation, max_attempts=5)
print(f"Retry result: {result}")
except Exception as e:
print(f"Operation failed permanently: {e}")
# Demonstrate input validation
validators = [
lambda x: isinstance(x, str),
lambda x: len(x) >= 3,
lambda x: x.strip() != ""
]
try:
validate_input("Valid input", validators)
print("Input validation passed")
except ValueError as e:
print(f"Input validation failed: {e}")
print("Best practices demonstration completed!")
# Security best practices
class SecureDataHandler:
"""Example of secure data handling practices."""
def __init__(self, encryption_key: Optional[str] = None):
"""Initialize with optional encryption key."""
self._encryption_key = encryption_key
self._sensitive_data: Dict[str, str] = {}
def store_sensitive_data(self, key: str, value: str) -> None:
"""
Store sensitive data with basic security measures.
Note: This is a simplified example. In production, use proper
encryption libraries like cryptography.
"""
if not key or not value:
raise ValueError("Key and value cannot be empty")
# In production, encrypt the value here
encrypted_value = self._simple_encrypt(value)
self._sensitive_data[key] = encrypted_value
logger.info(f"Stored sensitive data for key: {key}")
def retrieve_sensitive_data(self, key: str) -> Optional[str]:
"""Retrieve and decrypt sensitive data."""
encrypted_value = self._sensitive_data.get(key)
if encrypted_value is None:
return None
# In production, decrypt the value here
return self._simple_decrypt(encrypted_value)
def _simple_encrypt(self, value: str) -> str:
"""Simple encryption (NOT for production use)."""
# This is just for demonstration - use proper encryption in production
return "".join(chr(ord(c) + 1) for c in value)
def _simple_decrypt(self, encrypted_value: str) -> str:
"""Simple decryption (NOT for production use)."""
return "".join(chr(ord(c) - 1) for c in encrypted_value)
def clear_sensitive_data(self) -> None:
"""Clear all sensitive data from memory."""
self._sensitive_data.clear()
logger.info("Cleared all sensitive data")
if __name__ == "__main__":
# Run the demonstration
demonstrate_best_practices()
# Demonstrate secure data handling
print("\n=== Security Best Practices ===")
secure_handler = SecureDataHandler()
secure_handler.store_sensitive_data("api_key", "secret_key_123")
retrieved_value = secure_handler.retrieve_sensitive_data("api_key")
print(f"Retrieved value: {retrieved_value}")
secure_handler.clear_sensitive_data()
print("Security demonstration completed!")Performance Optimization
"""
Performance optimization techniques and best practices.
"""
import time
import cProfile
import pstats
from functools import lru_cache, wraps
from typing import Any, Callable
import sys
class PerformanceOptimizer:
"""Class demonstrating various performance optimization techniques."""
@staticmethod
def timing_decorator(func: Callable) -> Callable:
"""Decorator to measure function execution time."""
@wraps(func)
def wrapper(*args, **kwargs):
start_time = time.perf_counter()
result = func(*args, **kwargs)
end_time = time.perf_counter()
print(f"{func.__name__} took {end_time - start_time:.4f} seconds")
return result
return wrapper
# 1. Caching for expensive operations
@staticmethod
@lru_cache(maxsize=128)
def fibonacci_cached(n: int) -> int:
"""Cached Fibonacci calculation."""
if n <= 1:
return n
return PerformanceOptimizer.fibonacci_cached(n-1) + PerformanceOptimizer.fibonacci_cached(n-2)
@staticmethod
def fibonacci_uncached(n: int) -> int:
"""Uncached Fibonacci calculation."""
if n <= 1:
return n
return PerformanceOptimizer.fibonacci_uncached(n-1) + PerformanceOptimizer.fibonacci_uncached(n-2)
# 2. Generator vs List comprehension
@staticmethod
@timing_decorator
def process_with_list(data_size: int) -> list:
"""Process data using list comprehension."""
return [x**2 for x in range(data_size)]
@staticmethod
@timing_decorator
def process_with_generator(data_size: int) -> int:
"""Process data using generator."""
return sum(x**2 for x in range(data_size))
# 3. String operations optimization
@staticmethod
@timing_decorator
def string_concatenation_slow(words: list) -> str:
"""Slow string concatenation."""
result = ""
for word in words:
result += word + " "
return result.strip()
@staticmethod
@timing_decorator
def string_concatenation_fast(words: list) -> str:
"""Fast string concatenation."""
return " ".join(words)
# 4. Dictionary lookups vs multiple if-elif
@staticmethod
def process_with_if_elif(value: str) -> str:
"""Process using if-elif chain."""
if value == "a":
return "apple"
elif value == "b":
return "banana"
elif value == "c":
return "cherry"
elif value == "d":
return "date"
else:
return "unknown"
@staticmethod
def process_with_dict(value: str) -> str:
"""Process using dictionary lookup."""
mapping = {
"a": "apple",
"b": "banana",
"c": "cherry",
"d": "date"
}
return mapping.get(value, "unknown")
def profile_function(func: Callable) -> None:
"""Profile a function and display results."""
print(f"\nProfiling {func.__name__}:")
profiler = cProfile.Profile()
profiler.enable()
# Run the function
func()
profiler.disable()
# Display results
stats = pstats.Stats(profiler)
stats.sort_stats('cumulative')
stats.print_stats(10) # Show top 10 functions
def demonstrate_performance_optimization():
"""Demonstrate various performance optimization techniques."""
print("=== Performance Optimization Demonstration ===")
optimizer = PerformanceOptimizer()
# 1. Caching demonstration
print("\n1. Caching Benefits:")
@optimizer.timing_decorator
def test_fibonacci_cached():
return optimizer.fibonacci_cached(35)
@optimizer.timing_decorator
def test_fibonacci_uncached():
return optimizer.fibonacci_uncached(35)
print("Cached Fibonacci:")
result1 = test_fibonacci_cached()
print("Uncached Fibonacci:")
result2 = test_fibonacci_uncached()
print(f"Results match: {result1 == result2}")
# 2. Generator vs List
print("\n2. Generator vs List:")
data_size = 1000000
print("List comprehension (stores all in memory):")
optimizer.process_with_list(data_size)
print("Generator expression (processes on-demand):")
optimizer.process_with_generator(data_size)
# 3. String operations
print("\n3. String Operations:")
words = ["hello", "world", "python", "performance"] * 1000
print("Slow concatenation:")
result1 = optimizer.string_concatenation_slow(words)
print("Fast concatenation:")
result2 = optimizer.string_concatenation_fast(words)
print(f"Results match: {result1 == result2}")
# 4. Dictionary lookup vs if-elif
print("\n4. Dictionary Lookup vs If-Elif:")
test_values = ["a", "b", "c", "d", "x"] * 100000
@optimizer.timing_decorator
def test_if_elif():
return [optimizer.process_with_if_elif(v) for v in test_values]
@optimizer.timing_decorator
def test_dict_lookup():
return [optimizer.process_with_dict(v) for v in test_values]
print("If-elif chain:")
result1 = test_if_elif()
print("Dictionary lookup:")
result2 = test_dict_lookup()
print(f"Results match: {result1 == result2}")
# Memory optimization techniques
class MemoryOptimizer:
"""Techniques for memory optimization."""
def __init__(self):
self.data = []
def demonstrate_slots(self):
"""Demonstrate __slots__ for memory optimization."""
class RegularClass:
def __init__(self, x, y):
self.x = x
self.y = y
class SlottedClass:
__slots__ = ['x', 'y']
def __init__(self, x, y):
self.x = x
self.y = y
# Memory usage comparison
regular_objects = [RegularClass(i, i*2) for i in range(1000)]
slotted_objects = [SlottedClass(i, i*2) for i in range(1000)]
print("__slots__ reduces memory usage for instances")
print(f"Regular class instances: {len(regular_objects)}")
print(f"Slotted class instances: {len(slotted_objects)}")
def demonstrate_generators_memory(self):
"""Show memory efficiency of generators."""
def create_large_list():
return [x**2 for x in range(1000000)]
def create_large_generator():
return (x**2 for x in range(1000000))
print("\nMemory usage: List vs Generator")
# List stores all values in memory
large_list = create_large_list()
print(f"List created with {len(large_list)} items")
# Generator creates values on demand
large_gen = create_large_generator()
print("Generator created (no items stored in memory)")
# Process first 10 items from generator
first_ten = [next(large_gen) for _ in range(10)]
print(f"First 10 from generator: {first_ten}")
if __name__ == "__main__":
# Run the performance demonstrations
demonstrate_performance_optimization()
# Memory optimization
print("\n=== Memory Optimization ===")
memory_optimizer = MemoryOptimizer()
memory_optimizer.demonstrate_slots()
memory_optimizer.demonstrate_generators_memory()
print("\nPerformance optimization demonstration completed!")Conclusion
Congratulations! You’ve completed this comprehensive Python programming guide that takes you from beginner concepts to expert-level topics. This book has covered:
What You’ve Learned
mindmap
root((Python Mastery))
Fundamentals
Variables & Data Types
Control Flow
Functions
Error Handling
Intermediate
OOP Concepts
File Operations
Modules & Packages
Data Structures
Advanced
Generators & Iterators
Decorators
Context Managers
Metaclasses
Specializations
Web Development
Data Science
Testing & Debugging
Performance OptimizationNext Steps for Continued Learning
- Practice Projects: Build real-world applications using the concepts you’ve learned
- Open Source Contributions: Contribute to Python projects on GitHub
- Advanced Topics: Explore asyncio, multiprocessing, and advanced design patterns
- Frameworks: Dive deeper into Django, Flask, FastAPI, or data science libraries
- Community: Join Python communities, attend conferences, and keep learning
Key Takeaways
- Write Clean, Readable Code: Follow PEP 8 and use meaningful variable names
- Test Your Code: Write tests early and often
- Handle Errors Gracefully: Use proper exception handling
- Document Your Work: Write clear docstrings and comments
- Keep Learning: Python is constantly evolving with new features and libraries
Remember, becoming proficient in Python is a journey, not a destination. Keep practicing, building projects, and exploring new areas of Python development. The skills you’ve learned in this book provide a solid foundation for whatever direction your Python journey takes you.
Happy coding! 🐍
Discover more from Altgr Blog
Subscribe to get the latest posts sent to your email.
