A Complete Journey Through Python’s Functional Programming Paradigm with Interactive Examples and Real-World Applications
๐ Table of Contents
Part I: Foundations
- Introduction to Functional Programming
- Setting Up Your Environment
- Functions as First-Class Citizens
- Pure Functions and Immutability
Part II: Core Concepts
- Higher-Order Functions
- Lambda Functions and Closures
- Map, Filter, and Reduce
- Functional Error Handling
Part III: Advanced Topics
- Function Composition and Decorators
- Generators and Lazy Evaluation
- Partial Application and Currying
- Monads and Advanced Patterns
Part IV: Practical Applications
- Data Processing Pipelines
- Concurrent Functional Programming
- Testing Functional Code
- Performance and Optimization
- Real-World Projects and Best Practices
๐ฏ Learning Objectives
By the end of this book, you will:
- โ Master Python’s functional programming paradigm
- โ Write clean, maintainable, and bug-free code using FP principles
- โ Build efficient data processing pipelines
- โ Understand when to use functional vs object-oriented approaches
- โ Implement advanced functional patterns like monads and functors
- โ Create concurrent and parallel programs using functional techniques
๐ ๏ธ Quick Start Guide
Prerequisites
- Python 3.7+ installed
- Basic understanding of Python syntax
- Familiarity with functions and data structures
๐ฆ Required Libraries
pip install functools itertools operator collectionsBash๐ฏ How to Use This Book
- Read sequentially – Each section builds on previous concepts
- Run the code examples – All code is executable and tested
- Try the exercises – Practice reinforces learning
- Build projects – Apply concepts to real-world scenarios
1. Introduction to Functional Programming
๐ What is Functional Programming?
Functional Programming (FP) is a programming paradigm that treats computation as the evaluation of mathematical functions. It emphasizes:
- Immutability: Data doesn’t change after creation
- Pure Functions: Functions without side effects
- Function Composition: Building complex operations from simple functions
- Declarative Style: Focus on what to do, not how to do it
graph TD
A["๐งฎ Functional Programming"] --> B["๐ ImmutabilityData Never Changes"]
A --> C["โจ Pure FunctionsNo Side Effects"]
A --> D["๐ Function CompositionBuild Complex from Simple"]
A --> E["๐ Declarative StyleWhat, Not How"]
B --> B1["๐ก๏ธ Predictable Behavior"]
B --> B2["๐ Easy to Reason About"]
B --> B3["โก Thread-Safe"]
C --> C1["๐ฏ Same Input โ Same Output"]
C --> C2["๐ซ No Global State Changes"]
C --> C3["๐งช Easy to Test"]
D --> D1["๐งฉ Modular Design"]
D --> D2["โป๏ธ Reusable Components"]
D --> D3["๐ง Easy Debugging"]
E --> E1["๐ Readable Code"]
E --> E2["๐ค Less Bugs"]
E --> E3["๐ Maintainable"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px
style E fill:#fce4ec,stroke:#e91e63,stroke-width:2px๐ญ FP vs Other Paradigms
graph LR
A["๐ง Imperative ProgrammingHow to do it"] --> A1["Step-by-step instructions"]
A --> A2["Mutable state"]
A --> A3["Loops and conditions"]
B["๐๏ธ Object-Oriented ProgrammingObjects and methods"] --> B1["Encapsulation"]
B --> B2["Inheritance"]
B --> B3["Polymorphism"]
C["๐งฎ Functional ProgrammingWhat to compute"] --> C1["Pure functions"]
C --> C2["Immutable data"]
C --> C3["Function composition"]
style A fill:#ffcdd2
style B fill:#f0f4c3
style C fill:#c8e6c9๐ Why Choose Functional Programming?
๐ฏ Benefits:
- Fewer Bugs: Pure functions are predictable and easier to test
- Concurrent Safety: Immutable data eliminates race conditions
- Code Reusability: Functions are modular and composable
- Mathematical Reasoning: Code behavior is predictable
- Maintainability: Clear separation of concerns
โ ๏ธ Challenges:
- Learning Curve: Different mindset from imperative programming
- Performance: Some functional patterns can be slower
- Memory Usage: Immutability can increase memory consumption
- Debugging: Stack traces can be complex in composed functions
๐ Imperative vs Functional: A Comparison
Let’s see the difference with a simple example – calculating the sum of squares of even numbers:
๐ง Imperative Approach
"""
๐ง IMPERATIVE STYLE: Step-by-step instructions
Focus on HOW to do something
"""
def sum_squares_of_evens_imperative(numbers):
"""Calculate sum of squares of even numbers - imperative way"""
total = 0 # Mutable state
for num in numbers: # Step-by-step iteration
if num % 2 == 0: # Conditional check
square = num * num # Calculate square
total += square # Modify state
return total
# Example usage
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
result = sum_squares_of_evens_imperative(numbers)
print(f"Imperative result: {result}") # 220
# Step by step execution:
# 1. Initialize total = 0
# 2. Check 1: odd, skip
# 3. Check 2: even, total = 0 + 4 = 4
# 4. Check 3: odd, skip
# 5. Check 4: even, total = 4 + 16 = 20
# 6. ... and so onPython๐งฎ Functional Approach
"""
๐งฎ FUNCTIONAL STYLE: Express what you want
Focus on WHAT to compute
"""
from functools import reduce
def is_even(n):
"""Pure function: check if number is even"""
return n % 2 == 0
def square(n):
"""Pure function: calculate square"""
return n * n
def add(x, y):
"""Pure function: add two numbers"""
return x + y
def sum_squares_of_evens_functional(numbers):
"""Calculate sum of squares of even numbers - functional way"""
return reduce(add, # Sum all values
map(square, # Square each value
filter(is_even, # Keep only even numbers
numbers))) # From input numbers
# Alternative using built-in sum (more Pythonic)
def sum_squares_of_evens_pythonic(numbers):
"""Pythonic functional approach"""
return sum(n**2 for n in numbers if n % 2 == 0)
# Example usage
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
result1 = sum_squares_of_evens_functional(numbers)
result2 = sum_squares_of_evens_pythonic(numbers)
print(f"Functional result: {result1}") # 220
print(f"Pythonic result: {result2}") # 220
# Data flow visualization:
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# โ filter(is_even)
# [2, 4, 6, 8, 10]
# โ map(square)
# [4, 16, 36, 64, 100]
# โ reduce(add)
# 220Python๐ฏ Interactive Exploration: Function Transformation Pipeline
"""
๐ฌ INTERACTIVE DEMO: See functional programming in action
Try modifying the functions to see different transformations
"""
def demonstrate_fp_pipeline():
"""Interactive demonstration of functional programming concepts"""
# Sample data
data = [1, -2, 3, -4, 5, -6, 7, -8, 9, -10]
print("๐ FUNCTIONAL PROGRAMMING PIPELINE DEMO")
print("=" * 50)
print(f"Original data: {data}")
# Step 1: Filter (keep only positive numbers)
def is_positive(x):
return x > 0
step1 = list(filter(is_positive, data))
print(f"After filter(is_positive): {step1}")
# Step 2: Map (square each number)
def square(x):
return x * x
step2 = list(map(square, step1))
print(f"After map(square): {step2}")
# Step 3: Reduce (sum all numbers)
from functools import reduce
def add(x, y):
return x + y
step3 = reduce(add, step2)
print(f"After reduce(add): {step3}")
print("\n๐ CHAINED OPERATIONS:")
# All in one expression
result = reduce(add,
map(square,
filter(is_positive, data)))
print(f"Chained result: {result}")
# Using Python's built-in sum (more readable)
pythonic_result = sum(x**2 for x in data if x > 0)
print(f"Pythonic result: {pythonic_result}")
print("\n๐งช TRY DIFFERENT TRANSFORMATIONS:")
# Try 1: Negative numbers only, cubed
negative_cubes = [x**3 for x in data if x < 0]
print(f"Negative cubes: {negative_cubes}")
# Try 2: Absolute values, doubled
abs_doubled = [abs(x) * 2 for x in data]
print(f"Absolute doubled: {abs_doubled}")
# Try 3: Even numbers, halved
even_halved = [x / 2 for x in data if x % 2 == 0]
print(f"Even halved: {even_halved}")
# Run the demonstration
demonstrate_fp_pipeline()Python๐ When to Use Functional Programming
graph TD
A["๐ค When to Use FP?"] --> B["โ
Good Fit"]
A --> C["โ ๏ธ Consider Carefully"]
A --> D["โ Poor Fit"]
B --> B1["๐ Data Processing"]
B --> B2["๐งฎ Mathematical Computations"]
B --> B3["๐ Transformations"]
B --> B4["๐งช Testing & Validation"]
B --> B5["โก Concurrent Operations"]
C --> C1["๐ฎ Game Development"]
C --> C2["๐ฅ๏ธ UI Applications"]
C --> C3["๐ฑ Mobile Apps"]
C --> C4["๐ Web Backends"]
D --> D1["๐ฌ Real-time Systems"]
D --> D2["๐ฏ Performance-Critical Code"]
D --> D3["๐พ Memory-Constrained Systems"]
D --> D4["๐ Stateful Protocols"]
style B fill:#c8e6c9
style C fill:#fff3e0
style D fill:#ffcdd2Perfect for FP:
- Data analysis and ETL pipelines
- Mathematical modeling and simulations
- Configuration processing
- Validation and transformation logic
- Concurrent/parallel processing
Consider mixed approach:
- Web applications (functional business logic, OOP for structure)
- APIs (functional data processing, OOP for organization)
- CLIs (functional argument processing, OOP for commands)
Traditional OOP/Imperative better:
- GUI applications with complex state
- Game engines with performance requirements
- Real-time systems
- Hardware interfaces
2. Setting Up Your Environment
๐ Python Environment Setup
"""
๐ ๏ธ FUNCTIONAL PROGRAMMING ENVIRONMENT SETUP
Essential imports and utilities for functional programming in Python
"""
# Core functional programming imports
from functools import reduce, partial, singledispatch, lru_cache, wraps
from operator import add, mul, sub, truediv, mod, pow, and_, or_, not_
from itertools import (
chain, combinations, permutations, product,
groupby, accumulate, compress, cycle, repeat
)
from collections import namedtuple, defaultdict, Counter
from typing import Callable, Iterator, Iterable, Any, Optional, Union
# For advanced examples
import time
import threading
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
from dataclasses import dataclass
from abc import ABC, abstractmethod
print("โ
Functional Programming environment ready!")
print("๐ All imports successful - ready to explore FP concepts")Python๐งช Testing Your Setup
"""
๐งช ENVIRONMENT TEST SUITE
Verify your setup is working correctly
"""
def test_functional_features():
"""Test core functional programming features"""
print("๐ Testing functional programming setup...")
# Test 1: Higher-order functions
try:
numbers = [1, 2, 3, 4, 5]
squared = list(map(lambda x: x**2, numbers))
evens = list(filter(lambda x: x % 2 == 0, numbers))
total = reduce(add, numbers)
print("โ
map, filter, reduce working")
except Exception as e:
print(f"โ Basic functions failed: {e}")
return False
# Test 2: Operator module
try:
result = reduce(mul, [1, 2, 3, 4])
print("โ
operator module working")
except Exception as e:
print(f"โ Operator module failed: {e}")
return False
# Test 3: itertools
try:
combinations_result = list(combinations([1, 2, 3], 2))
chain_result = list(chain([1, 2], [3, 4]))
print("โ
itertools working")
except Exception as e:
print(f"โ itertools failed: {e}")
return False
# Test 4: Decorators and closures
try:
@lru_cache(maxsize=128)
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n-1) + fibonacci(n-2)
result = fibonacci(10)
print("โ
Decorators and memoization working")
except Exception as e:
print(f"โ Decorators failed: {e}")
return False
print("๐ All tests passed! Environment is ready for functional programming.")
return True
# Run the test
test_functional_features()Python๐ Performance Monitoring Utils
"""
๐ PERFORMANCE MONITORING UTILITIES
Tools for comparing functional vs imperative approaches
"""
import time
from functools import wraps
from contextlib import contextmanager
def timer(func):
"""Decorator to time function execution"""
@wraps(func)
def wrapper(*args, **kwargs):
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print(f"โฑ๏ธ {func.__name__} took {end - start:.4f} seconds")
return result
return wrapper
@contextmanager
def time_block(description):
"""Context manager for timing code blocks"""
start = time.time()
yield
end = time.time()
print(f"โฑ๏ธ {description} took {end - start:.4f} seconds")
def compare_approaches(name, functional_fn, imperative_fn, *args, **kwargs):
"""Compare performance of functional vs imperative approaches"""
print(f"\n๐ Comparing approaches: {name}")
print("-" * 40)
# Time functional approach
start = time.time()
func_result = functional_fn(*args, **kwargs)
func_time = time.time() - start
# Time imperative approach
start = time.time()
imp_result = imperative_fn(*args, **kwargs)
imp_time = time.time() - start
# Results
print(f"๐งฎ Functional: {func_time:.4f}s")
print(f"๐ง Imperative: {imp_time:.4f}s")
print(f"โก Speedup: {func_time / imp_time:.2f}x {'(functional faster)' if imp_time > func_time else '(imperative faster)'}")
print(f"โ
Results match: {func_result == imp_result}")
# Example usage
if __name__ == "__main__":
# Example comparison
def functional_sum_squares(numbers):
return sum(x**2 for x in numbers)
def imperative_sum_squares(numbers):
total = 0
for x in numbers:
total += x**2
return total
test_data = list(range(10000))
compare_approaches("Sum of squares", functional_sum_squares, imperative_sum_squares, test_data)Python๐จ Visualization Helpers
"""
๐จ VISUALIZATION HELPERS
Tools for visualizing functional programming concepts
"""
def visualize_pipeline(data, *operations):
"""Visualize data transformation pipeline"""
print("๐ DATA TRANSFORMATION PIPELINE")
print("=" * 50)
current_data = data
print(f"๐ฅ Input: {current_data}")
for i, operation in enumerate(operations, 1):
try:
if hasattr(operation, '__name__'):
op_name = operation.__name__
else:
op_name = str(operation)
current_data = operation(current_data)
print(f"Step {i} ({op_name}): {current_data}")
except Exception as e:
print(f"โ Step {i} failed: {e}")
break
print(f"๐ค Final result: {current_data}")
return current_data
def show_function_composition(functions, input_value):
"""Show step-by-step function composition"""
print("๐ FUNCTION COMPOSITION VISUALIZATION")
print("=" * 45)
result = input_value
print(f"Input: {result}")
for i, func in enumerate(functions):
result = func(result)
func_name = getattr(func, '__name__', 'lambda')
print(f" Step {i+1}: {func_name}({result})")
print(f"Final result: {result}")
return result
# Example usage
if __name__ == "__main__":
# Pipeline visualization
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
visualize_pipeline(
numbers,
lambda x: [n for n in x if n % 2 == 0], # filter evens
lambda x: [n**2 for n in x], # square
sum # sum all
)
# Function composition visualization
functions = [
lambda x: x + 1,
lambda x: x * 2,
lambda x: x**2
]
show_function_composition(functions, 3)Python3. Functions as First-Class Citizens
๐ญ Understanding First-Class Functions
In Python, functions are first-class citizens, meaning they can be:
- Assigned to variables
- Passed as arguments
- Returned from functions
- Stored in data structures
- Created at runtime
graph TD
A["๐ญ First-Class Functions"] --> B["๐ฆ Assign to Variables"]
A --> C["๐จ Pass as Arguments"]
A --> D["๐ค Return from Functions"]
A --> E["๐๏ธ Store in Collections"]
A --> F["๐๏ธ Create at Runtime"]
B --> B1["func = my_function"]
C --> C1["process(data, func)"]
D --> D1["return inner_function"]
E --> E1["funcs = [f1, f2, f3]"]
F --> F1["lambda x: x * 2"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px๐ฆ Assigning Functions to Variables
"""
๐ฆ FUNCTIONS AS VARIABLES
Functions can be assigned to variables just like any other value
"""
def greet(name):
"""A simple greeting function"""
return f"Hello, {name}!"
def farewell(name):
"""A farewell function"""
return f"Goodbye, {name}!"
# Assign functions to variables
greeting_func = greet
farewell_func = farewell
print(f"Function name: {greeting_func.__name__}") # greet
print(f"Function call: {greeting_func('Alice')}") # Hello, Alice!
# Functions have the same identity
print(f"Same function? {greet is greeting_func}") # True
# Can check if variable holds a function
print(f"Is callable? {callable(greeting_func)}") # True
# Store functions in data structures
function_library = {
'hello': greet,
'bye': farewell,
'welcome': lambda name: f"Welcome, {name}!"
}
# Use functions from the library
for func_name, func in function_library.items():
print(f"{func_name}: {func('Bob')}")
# Advanced: Function metadata
def analyze_function(func):
"""Analyze a function's properties"""
print(f"\n๐ ANALYZING FUNCTION: {func.__name__}")
print(f" Module: {func.__module__}")
print(f" Doc: {func.__doc__}")
if hasattr(func, '__code__'):
print(f" Arguments: {func.__code__.co_varnames}")
print(f" Line number: {func.__code__.co_firstlineno}")
analyze_function(greet)
analyze_function(lambda x: x * 2)Python๐จ Passing Functions as Arguments (Higher-Order Functions)
"""
๐จ HIGHER-ORDER FUNCTIONS
Functions that accept other functions as arguments
"""
def apply_operation(data, operation):
"""Apply an operation to each item in data"""
return [operation(item) for item in data]
def double(x):
"""Double a number"""
return x * 2
def square(x):
"""Square a number"""
return x ** 2
def cube(x):
"""Cube a number"""
return x ** 3
# Sample data
numbers = [1, 2, 3, 4, 5]
print("๐งฎ HIGHER-ORDER FUNCTION DEMO")
print("=" * 40)
# Pass different functions as arguments
print(f"Original: {numbers}")
print(f"Doubled: {apply_operation(numbers, double)}")
print(f"Squared: {apply_operation(numbers, square)}")
print(f"Cubed: {apply_operation(numbers, cube)}")
# Pass lambda functions
print(f"Plus 10: {apply_operation(numbers, lambda x: x + 10)}")
print(f"Negated: {apply_operation(numbers, lambda x: -x)}")
# More complex example: Multiple operations
def chain_operations(data, *operations):
"""Apply multiple operations in sequence"""
result = data
for operation in operations:
result = apply_operation(result, operation)
print(f"After {operation.__name__}: {result}")
return result
print(f"\n๐ CHAINING OPERATIONS:")
final_result = chain_operations(
[1, 2, 3],
double, # First double: [2, 4, 6]
square, # Then square: [4, 16, 36]
lambda x: x + 1 # Finally add 1: [5, 17, 37]
)
# Real-world example: Data processing pipeline
def process_data(data, filters=None, transformations=None, aggregations=None):
"""Generic data processing pipeline"""
result = data
# Apply filters
if filters:
for filter_func in filters:
result = [item for item in result if filter_func(item)]
print(f"After filter {filter_func.__name__}: {result}")
# Apply transformations
if transformations:
for transform_func in transformations:
result = [transform_func(item) for item in result]
print(f"After transform {transform_func.__name__}: {result}")
# Apply aggregations
if aggregations:
for agg_func in aggregations:
result = agg_func(result)
print(f"After aggregation {agg_func.__name__}: {result}")
return result
# Define some helper functions
def is_positive(x):
return x > 0
def is_even(x):
return x % 2 == 0
def add_one(x):
return x + 1
def multiply_by_ten(x):
return x * 10
# Example usage
sample_data = [-3, -2, -1, 0, 1, 2, 3, 4, 5, 6]
print(f"\n๐ DATA PROCESSING PIPELINE:")
print(f"Original data: {sample_data}")
result = process_data(
sample_data,
filters=[is_positive, is_even], # Keep positive even numbers
transformations=[add_one, multiply_by_ten], # Add 1, then multiply by 10
aggregations=[sum] # Sum all results
)Python๐ค Returning Functions from Functions (Function Factories)
"""
๐ค FUNCTION FACTORIES
Functions that create and return other functions
"""
def create_multiplier(factor):
"""Factory function that creates multiplier functions"""
def multiplier(x):
return x * factor
# Add metadata to the created function
multiplier.__name__ = f"multiply_by_{factor}"
multiplier.__doc__ = f"Multiply input by {factor}"
return multiplier
def create_validator(min_val, max_val):
"""Factory function that creates validator functions"""
def validator(value):
return min_val <= value <= max_val
validator.__name__ = f"validate_{min_val}_to_{max_val}"
return validator
def create_formatter(prefix, suffix=""):
"""Factory function that creates formatter functions"""
def formatter(text):
return f"{prefix}{text}{suffix}"
formatter.__name__ = f"format_with_{prefix}"
return formatter
# Create specific functions using factories
double = create_multiplier(2)
triple = create_multiplier(3)
multiply_by_10 = create_multiplier(10)
print("๐ญ FUNCTION FACTORIES DEMO")
print("=" * 40)
# Test multiplier functions
numbers = [1, 2, 3, 4, 5]
print(f"Original: {numbers}")
print(f"Doubled: {[double(x) for x in numbers]}")
print(f"Tripled: {[triple(x) for x in numbers]}")
print(f"ร 10: {[multiply_by_10(x) for x in numbers]}")
# Create validators
age_validator = create_validator(0, 120)
percentage_validator = create_validator(0, 100)
temperature_validator = create_validator(-273, 1000)
ages = [-5, 25, 150, 45]
print(f"\nโ
VALIDATION:")
for age in ages:
valid = age_validator(age)
print(f"Age {age}: {'โ
Valid' if valid else 'โ Invalid'}")
# Create formatters
html_bold = create_formatter("<b>", "</b>")
markdown_italic = create_formatter("*", "*")
quote_formatter = create_formatter('"', '"')
texts = ["Hello", "World", "Python"]
print(f"\n๐จ FORMATTING:")
for text in texts:
print(f"Original: {text}")
print(f"HTML Bold: {html_bold(text)}")
print(f"Markdown Italic: {markdown_italic(text)}")
print(f"Quoted: {quote_formatter(text)}")
print()
# Advanced: Configurable function factory
def create_operation(operation_name, operation_func, *args, **kwargs):
"""Create a customized operation function"""
def operation(data):
return operation_func(data, *args, **kwargs)
operation.__name__ = operation_name
operation.__doc__ = f"Apply {operation_name} operation"
return operation
# Create operations with different configurations
add_ten = create_operation("add_ten", lambda x, n: x + n, 10)
power_of = create_operation("square", lambda x, n: x ** n, 2)
modulo_check = create_operation("mod_3", lambda x, n: x % n, 3)
print(f"๐ง CONFIGURABLE OPERATIONS:")
test_val = 7
print(f"Original: {test_val}")
print(f"Add 10: {add_ten(test_val)}")
print(f"Square: {power_of(test_val)}")
print(f"Mod 3: {modulo_check(test_val)}")Python๐๏ธ Storing Functions in Data Structures
"""
๐๏ธ FUNCTIONS IN DATA STRUCTURES
Functions can be stored in lists, dictionaries, sets, etc.
"""
import math
# Calculator using function dictionary
calculator_operations = {
'add': lambda x, y: x + y,
'subtract': lambda x, y: x - y,
'multiply': lambda x, y: x * y,
'divide': lambda x, y: x / y if y != 0 else float('inf'),
'power': lambda x, y: x ** y,
'sqrt': lambda x, y=None: math.sqrt(x),
'log': lambda x, y=None: math.log(x),
'sin': lambda x, y=None: math.sin(x),
'cos': lambda x, y=None: math.cos(x)
}
def calculate(operation, x, y=None):
"""Generic calculator function"""
if operation in calculator_operations:
func = calculator_operations[operation]
try:
return func(x, y) if y is not None else func(x)
except Exception as e:
return f"Error: {e}"
else:
return f"Unknown operation: {operation}"
print("๐งฎ CALCULATOR DEMO")
print("=" * 30)
# Test calculator
test_cases = [
('add', 5, 3),
('multiply', 4, 7),
('sqrt', 16),
('power', 2, 8),
('divide', 10, 0),
('unknown', 1, 2)
]
for operation, x, *args in test_cases:
y = args[0] if args else None
result = calculate(operation, x, y)
if y is not None:
print(f"{operation}({x}, {y}) = {result}")
else:
print(f"{operation}({x}) = {result}")
# Event handling system using function lists
event_handlers = {
'user_login': [],
'user_logout': [],
'data_received': [],
'error_occurred': []
}
def on_user_login(username):
print(f"๐ง Sending welcome email to {username}")
def on_user_login_analytics(username):
print(f"๐ Recording login analytics for {username}")
def on_user_logout(username):
print(f"๐ User {username} logged out")
def on_data_received(data):
print(f"๐พ Processing data: {len(data)} items")
def on_error(error_msg):
print(f"๐จ Error logged: {error_msg}")
# Register event handlers
event_handlers['user_login'].extend([on_user_login, on_user_login_analytics])
event_handlers['user_logout'].append(on_user_logout)
event_handlers['data_received'].append(on_data_received)
event_handlers['error_occurred'].append(on_error)
def trigger_event(event_name, *args):
"""Trigger all handlers for an event"""
if event_name in event_handlers:
print(f"\n๐ฏ Triggering event: {event_name}")
for handler in event_handlers[event_name]:
try:
handler(*args)
except Exception as e:
print(f"โ Handler error: {e}")
print(f"\n๐ก EVENT SYSTEM DEMO")
print("=" * 30)
# Trigger events
trigger_event('user_login', 'alice@example.com')
trigger_event('user_logout', 'bob@example.com')
trigger_event('data_received', [1, 2, 3, 4, 5])
trigger_event('error_occurred', 'Database connection failed')
# Plugin system using function registry
plugin_registry = []
def register_plugin(plugin_func):
"""Decorator to register plugin functions"""
plugin_registry.append(plugin_func)
print(f"๐ Registered plugin: {plugin_func.__name__}")
return plugin_func
def execute_plugins(data):
"""Execute all registered plugins"""
result = data
for plugin in plugin_registry:
try:
result = plugin(result)
print(f"โ
Plugin {plugin.__name__} executed")
except Exception as e:
print(f"โ Plugin {plugin.__name__} failed: {e}")
return result
# Define and register plugins
@register_plugin
def uppercase_plugin(text):
"""Convert text to uppercase"""
return text.upper() if isinstance(text, str) else text
@register_plugin
def add_timestamp_plugin(data):
"""Add timestamp to data"""
import datetime
timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
return f"[{timestamp}] {data}"
@register_plugin
def add_prefix_plugin(data):
"""Add prefix to data"""
return f"PROCESSED: {data}"
print(f"\n๐ PLUGIN SYSTEM DEMO")
print("=" * 30)
# Execute plugin pipeline
original_data = "hello world"
final_result = execute_plugins(original_data)
print(f"Original: {original_data}")
print(f"Final: {final_result}")Python๐๏ธ Creating Functions at Runtime
"""
๐๏ธ RUNTIME FUNCTION CREATION
Functions can be created dynamically during program execution
"""
def create_dynamic_function(operation, operand):
"""Create a function dynamically based on operation type"""
operations = {
'add': lambda x: x + operand,
'subtract': lambda x: x - operand,
'multiply': lambda x: x * operand,
'divide': lambda x: x / operand if operand != 0 else float('inf'),
'power': lambda x: x ** operand,
'modulo': lambda x: x % operand if operand != 0 else 0
}
if operation not in operations:
raise ValueError(f"Unknown operation: {operation}")
func = operations[operation]
func.__name__ = f"{operation}_by_{operand}"
func.__doc__ = f"{operation.title()} input by {operand}"
return func
# Create functions at runtime based on user input
print("๐๏ธ DYNAMIC FUNCTION CREATION")
print("=" * 40)
# Simulate user configuration
user_configs = [
('add', 5),
('multiply', 3),
('power', 2),
('divide', 2)
]
# Create and test dynamic functions
test_values = [1, 2, 3, 4, 5]
for operation, operand in user_configs:
func = create_dynamic_function(operation, operand)
results = [func(x) for x in test_values]
print(f"{func.__name__}: {test_values} โ {results}")
# Advanced: Code generation with exec
def create_function_from_template(func_name, operation_code):
"""Create function from string template"""
template = f"""
def {func_name}(x):
'''Generated function: {func_name}'''
return {operation_code}
"""
# Execute the template to create the function
namespace = {}
exec(template, namespace)
return namespace[func_name]
print(f"\n๐ CODE GENERATION DEMO")
print("=" * 30)
# Generate functions from templates
templates = [
('double_plus_one', 'x * 2 + 1'),
('cube_minus_ten', 'x ** 3 - 10'),
('reciprocal_safe', '1 / x if x != 0 else 0')
]
generated_functions = []
for func_name, operation_code in templates:
func = create_function_from_template(func_name, operation_code)
generated_functions.append(func)
print(f"Generated: {func.__name__} - {func.__doc__}")
# Test generated functions
test_value = 4
for func in generated_functions:
result = func(test_value)
print(f"{func.__name__}({test_value}) = {result}")
# Meta-programming: Function factory with custom behavior
class FunctionBuilder:
"""Builder class for creating custom functions"""
def __init__(self, name):
self.name = name
self.operations = []
def add_operation(self, operation):
"""Add an operation to the function pipeline"""
self.operations.append(operation)
return self # Enable method chaining
def build(self):
"""Build the final function"""
def generated_function(x):
result = x
for operation in self.operations:
result = operation(result)
return result
generated_function.__name__ = self.name
generated_function.__doc__ = f"Generated function with {len(self.operations)} operations"
return generated_function
print(f"\n๐๏ธ FUNCTION BUILDER DEMO")
print("=" * 30)
# Build complex functions using builder pattern
complex_func = (FunctionBuilder("complex_transformation")
.add_operation(lambda x: x * 2) # Double
.add_operation(lambda x: x + 10) # Add 10
.add_operation(lambda x: x ** 2) # Square
.add_operation(lambda x: x % 100) # Modulo 100
.build())
# Test the built function
test_inputs = [1, 2, 3, 4, 5]
for inp in test_inputs:
result = complex_func(inp)
print(f"{complex_func.__name__}({inp}) = {result}")
# Show the transformation pipeline
def show_transformation_steps(func_builder, input_val):
"""Show each step of transformation"""
result = input_val
print(f"Input: {result}")
for i, operation in enumerate(func_builder.operations, 1):
result = operation(result)
print(f"Step {i}: {result}")
print(f"\n๐ TRANSFORMATION STEPS for input 3:")
builder = FunctionBuilder("step_by_step")
builder.add_operation(lambda x: x * 2) # 3 โ 6
builder.add_operation(lambda x: x + 10) # 6 โ 16
builder.add_operation(lambda x: x ** 2) # 16 โ 256
builder.add_operation(lambda x: x % 100) # 256 โ 56
show_transformation_steps(builder, 3)Python4. Pure Functions and Immutability
โจ Understanding Pure Functions
A pure function is a function that:
- Always returns the same output for the same input (deterministic)
- Has no side effects (doesn’t modify anything outside itself)
- Doesn’t depend on external state (only uses its parameters)
graph TD
A["โจ Pure Function"] --> B["๐ฏ DeterministicSame Input โ Same Output"]
A --> C["๐ซ No Side EffectsDoesn't Change Anything"]
A --> D["๐ Self-ContainedOnly Uses Parameters"]
B --> B1["โ
Predictable Behavior"]
B --> B2["๐งช Easy to Test"]
B --> B3["๐ Cacheable Results"]
C --> C1["๐ก๏ธ No Global Changes"]
C --> C2["๐ No File/DB Operations"]
C --> C3["๐ฅ๏ธ No Console Output"]
D --> D1["๐ซ No Global Variables"]
D --> D2["๐ซ No External Dependencies"]
D --> D3["๐ฆ Self-Sufficient"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐ Pure vs Impure Functions: Examples
"""
๐ PURE vs IMPURE FUNCTIONS
Understanding the difference through examples
"""
import random
import time
from datetime import datetime
# โ
PURE FUNCTIONS
def add_pure(x, y):
"""Pure: same input always gives same output, no side effects"""
return x + y
def calculate_area_pure(length, width):
"""Pure: mathematical calculation, deterministic"""
return length * width
def get_full_name_pure(first, last):
"""Pure: string manipulation, no external dependencies"""
return f"{first} {last}"
def factorial_pure(n):
"""Pure: mathematical calculation, recursive"""
if n <= 1:
return 1
return n * factorial_pure(n - 1)
def filter_evens_pure(numbers):
"""Pure: returns new list, doesn't modify input"""
return [n for n in numbers if n % 2 == 0]
print("โ
PURE FUNCTIONS DEMO")
print("=" * 30)
# Test pure functions - always same results
print(f"add_pure(2, 3) = {add_pure(2, 3)}")
print(f"add_pure(2, 3) = {add_pure(2, 3)}") # Same result
print(f"area(5, 4) = {calculate_area_pure(5, 4)}")
print(f"full_name = {get_full_name_pure('Alice', 'Smith')}")
print(f"factorial(5) = {factorial_pure(5)}")
numbers = [1, 2, 3, 4, 5, 6]
evens1 = filter_evens_pure(numbers)
evens2 = filter_evens_pure(numbers)
print(f"Original numbers: {numbers}")
print(f"Filtered evens (1st call): {evens1}")
print(f"Filtered evens (2nd call): {evens2}")
print(f"Results are equal: {evens1 == evens2}")
print("\nโ IMPURE FUNCTIONS DEMO")
print("=" * 30)
# โ IMPURE FUNCTIONS
# Global state dependency
counter = 0
def add_impure(x, y):
"""Impure: depends on global state"""
global counter
counter += 1 # Side effect: modifies global state
return x + y + counter # Result depends on external state
# Non-deterministic (depends on time/random)
def get_random_number():
"""Impure: non-deterministic output"""
return random.randint(1, 100)
def get_current_time():
"""Impure: depends on system state"""
return datetime.now().strftime("%Y-%m-%d %H:%M:%S")
# Modifies input (side effect)
def add_to_list_impure(lst, item):
"""Impure: modifies the input list"""
lst.append(item) # Side effect: modifies input
return lst
# I/O operations (side effects)
def log_message_impure(message):
"""Impure: has side effect (console output)"""
print(f"LOG: {message}") # Side effect: console output
return message
# Test impure functions - results vary or have side effects
print("Testing impure functions:")
# Global state dependency - results change
print(f"add_impure(2, 3) = {add_impure(2, 3)}") # 6 (2+3+1)
print(f"add_impure(2, 3) = {add_impure(2, 3)}") # 7 (2+3+2) - Different!
# Non-deterministic results
print(f"random_number = {get_random_number()}")
print(f"random_number = {get_random_number()}") # Different each time
# Time dependency
print(f"current_time = {get_current_time()}")
time.sleep(1) # Wait 1 second
print(f"current_time = {get_current_time()}") # Different time
# Modifying input
original_list = [1, 2, 3]
print(f"Original list: {original_list}")
result_list = add_to_list_impure(original_list, 4)
print(f"After impure function: {original_list}") # Modified!
print(f"Result list: {result_list}")
# Side effects
log_message_impure("This is a side effect") # Prints to consolePython๐ง Converting Impure to Pure Functions
"""
๐ง PURIFICATION: Converting impure functions to pure ones
Learn how to eliminate side effects and external dependencies
"""
# โ IMPURE VERSION: Global state dependency
total_processed = 0
def process_data_impure(data):
"""Impure: modifies global state"""
global total_processed
total_processed += len(data)
return [x * 2 for x in data]
# โ
PURE VERSION: No global state
def process_data_pure(data, current_total=0):
"""Pure: returns both result and new total"""
processed_data = [x * 2 for x in data]
new_total = current_total + len(data)
return processed_data, new_total
print("๐ง IMPURE โ PURE CONVERSION DEMO")
print("=" * 40)
# Test impure version
print("โ Impure version:")
data1 = [1, 2, 3]
data2 = [4, 5]
result1 = process_data_impure(data1)
result2 = process_data_impure(data2)
print(f"Data 1: {data1} โ {result1}, Total: {total_processed}")
print(f"Data 2: {data2} โ {result2}, Total: {total_processed}")
# Reset for comparison
total_processed = 0
# Test pure version
print("\nโ
Pure version:")
result1, total1 = process_data_pure(data1, 0)
result2, total2 = process_data_pure(data2, total1)
print(f"Data 1: {data1} โ {result1}, Total: {total1}")
print(f"Data 2: {data2} โ {result2}, Total: {total2}")
# โ IMPURE VERSION: Modifying input
def sort_list_impure(lst):
"""Impure: modifies the input list"""
lst.sort() # Side effect: modifies input
return lst
# โ
PURE VERSION: No input modification
def sort_list_pure(lst):
"""Pure: returns new sorted list"""
return sorted(lst) # Creates new list
print(f"\n๐ INPUT MODIFICATION DEMO")
print("=" * 30)
# Test input modification
original = [3, 1, 4, 1, 5, 9, 2, 6]
print(f"Original list: {original}")
# Impure version
impure_copy = original.copy()
sorted_impure = sort_list_impure(impure_copy)
print(f"After impure sort - Original: {original}")
print(f"After impure sort - Copy: {impure_copy}") # Modified!
print(f"Result: {sorted_impure}")
# Pure version
sorted_pure = sort_list_pure(original)
print(f"After pure sort - Original: {original}") # Unchanged!
print(f"Pure result: {sorted_pure}")
# โ IMPURE VERSION: I/O operations
def calculate_with_logging_impure(x, y, operation):
"""Impure: has logging side effect"""
if operation == "add":
result = x + y
print(f"LOG: Added {x} + {y} = {result}") # Side effect
elif operation == "multiply":
result = x * y
print(f"LOG: Multiplied {x} * {y} = {result}") # Side effect
else:
result = 0
print(f"LOG: Unknown operation") # Side effect
return result
# โ
PURE VERSION: Return both result and log message
def calculate_with_logging_pure(x, y, operation):
"""Pure: returns both result and log message"""
if operation == "add":
result = x + y
log_message = f"LOG: Added {x} + {y} = {result}"
elif operation == "multiply":
result = x * y
log_message = f"LOG: Multiplied {x} * {y} = {result}"
else:
result = 0
log_message = "LOG: Unknown operation"
return result, log_message
print(f"\n๐ LOGGING DEMO")
print("=" * 20)
# Test logging functions
print("โ Impure version (with side effects):")
result1 = calculate_with_logging_impure(5, 3, "add")
result2 = calculate_with_logging_impure(4, 7, "multiply")
print("\nโ
Pure version (no side effects):")
result3, log3 = calculate_with_logging_pure(5, 3, "add")
result4, log4 = calculate_with_logging_pure(4, 7, "multiply")
print(log3) # Explicitly handle logging
print(log4)
# Advanced: Pure function with dependency injection
def create_pure_calculator(logger_func=None):
"""Factory function that creates a pure calculator"""
def calculate(x, y, operation):
"""Pure calculation function"""
operations = {
'add': lambda a, b: a + b,
'subtract': lambda a, b: a - b,
'multiply': lambda a, b: a * b,
'divide': lambda a, b: a / b if b != 0 else float('inf')
}
if operation in operations:
result = operations[operation](x, y)
log_message = f"Calculated: {x} {operation} {y} = {result}"
else:
result = 0
log_message = f"Unknown operation: {operation}"
# Pure function returns all effects as data
return {
'result': result,
'log_message': log_message,
'success': operation in operations
}
return calculate
print(f"\n๐ญ DEPENDENCY INJECTION DEMO")
print("=" * 35)
# Create pure calculator
pure_calc = create_pure_calculator()
# Use the calculator
calc_result = pure_calc(10, 3, "divide")
print(f"Result: {calc_result['result']}")
print(f"Log: {calc_result['log_message']}")
print(f"Success: {calc_result['success']}")
# The logging effect is data that can be handled externally
if calc_result['log_message']:
print(f"External logger: {calc_result['log_message']}")Python๐ Immutability in Python
"""
๐ IMMUTABILITY: Working with unchangeable data
Understanding immutable vs mutable types and creating immutable patterns
"""
# Built-in immutable types
print("๐ BUILT-IN IMMUTABLE TYPES")
print("=" * 35)
# Strings are immutable
original_string = "Hello"
print(f"Original string: '{original_string}' (id: {id(original_string)})")
# "Modifying" creates new string
modified_string = original_string + " World"
print(f"Modified string: '{modified_string}' (id: {id(modified_string)})")
print(f"Original unchanged: '{original_string}'")
# Tuples are immutable
original_tuple = (1, 2, 3)
print(f"\nOriginal tuple: {original_tuple} (id: {id(original_tuple)})")
# Adding to tuple creates new tuple
extended_tuple = original_tuple + (4, 5)
print(f"Extended tuple: {extended_tuple} (id: {id(extended_tuple)})")
print(f"Original unchanged: {original_tuple}")
# Numbers are immutable
x = 42
original_id = id(x)
x += 1 # Creates new integer object
print(f"\nNumber changed: {x} (old id: {original_id}, new id: {id(x)})")
print("\n๐ MUTABLE TYPES BEHAVIOR")
print("=" * 30)
# Lists are mutable
original_list = [1, 2, 3]
list_id = id(original_list)
print(f"Original list: {original_list} (id: {list_id})")
# Modifying list changes same object
original_list.append(4)
print(f"After append: {original_list} (id: {id(original_list)})")
print(f"Same object? {id(original_list) == list_id}")
# Dictionaries are mutable
original_dict = {'a': 1, 'b': 2}
dict_id = id(original_dict)
original_dict['c'] = 3 # Modifies same object
print(f"Modified dict: {original_dict} (same id: {id(original_dict) == dict_id})")
# Creating immutable data patterns
from collections import namedtuple
from dataclasses import dataclass
from typing import List, Tuple
print("\n๐๏ธ CREATING IMMUTABLE PATTERNS")
print("=" * 35)
# 1. Using namedtuple
Person = namedtuple('Person', ['name', 'age', 'city'])
person1 = Person('Alice', 30, 'New York')
print(f"NamedTuple person: {person1}")
# Cannot modify namedtuple
try:
person1.age = 31 # This will fail
except AttributeError as e:
print(f"Cannot modify namedtuple: {e}")
# Create new namedtuple with changed values
person2 = person1._replace(age=31)
print(f"New person with updated age: {person2}")
print(f"Original unchanged: {person1}")
# 2. Using dataclass with frozen=True
@dataclass(frozen=True)
class ImmutablePoint:
"""Immutable point class using dataclass"""
x: float
y: float
def distance_from_origin(self):
"""Pure method: only uses instance data"""
return (self.x ** 2 + self.y ** 2) ** 0.5
def move(self, dx, dy):
"""Pure method: returns new point instead of modifying"""
return ImmutablePoint(self.x + dx, self.y + dy)
point1 = ImmutablePoint(3, 4)
print(f"Immutable point: {point1}")
print(f"Distance from origin: {point1.distance_from_origin()}")
# Cannot modify frozen dataclass
try:
point1.x = 5 # This will fail
except AttributeError as e:
print(f"Cannot modify frozen dataclass: {e}")
# Create new point with movement
point2 = point1.move(2, 1)
print(f"Moved point: {point2}")
print(f"Original unchanged: {point1}")
# 3. Immutable list operations
def immutable_list_operations():
"""Demonstrate immutable patterns with lists"""
def append_immutable(lst, item):
"""Pure function: returns new list with item appended"""
return lst + [item]
def remove_immutable(lst, item):
"""Pure function: returns new list with item removed"""
return [x for x in lst if x != item]
def update_immutable(lst, index, new_value):
"""Pure function: returns new list with value updated"""
if 0 <= index < len(lst):
new_list = lst.copy()
new_list[index] = new_value
return new_list
return lst
def sort_immutable(lst):
"""Pure function: returns new sorted list"""
return sorted(lst)
# Test immutable operations
original = [3, 1, 4, 1, 5]
print(f"\nImmutable list operations:")
print(f"Original: {original}")
appended = append_immutable(original, 9)
print(f"Appended 9: {appended}")
removed = remove_immutable(original, 1)
print(f"Removed 1: {removed}")
updated = update_immutable(original, 2, 99)
print(f"Updated index 2: {updated}")
sorted_list = sort_immutable(original)
print(f"Sorted: {sorted_list}")
print(f"Original still unchanged: {original}")
return {
'original': original,
'appended': appended,
'removed': removed,
'updated': updated,
'sorted': sorted_list
}
immutable_list_operations()
# 4. Immutable dictionary operations
def immutable_dict_operations():
"""Demonstrate immutable patterns with dictionaries"""
def set_value_immutable(d, key, value):
"""Pure function: returns new dict with key set"""
new_dict = d.copy()
new_dict[key] = value
return new_dict
def remove_key_immutable(d, key):
"""Pure function: returns new dict with key removed"""
new_dict = d.copy()
new_dict.pop(key, None)
return new_dict
def merge_immutable(d1, d2):
"""Pure function: returns new dict with merged values"""
merged = d1.copy()
merged.update(d2)
return merged
# Test immutable dictionary operations
original = {'a': 1, 'b': 2, 'c': 3}
print(f"\nImmutable dict operations:")
print(f"Original: {original}")
with_new_key = set_value_immutable(original, 'd', 4)
print(f"Added 'd': {with_new_key}")
without_b = remove_key_immutable(original, 'b')
print(f"Removed 'b': {without_b}")
additional = {'e': 5, 'f': 6}
merged = merge_immutable(original, additional)
print(f"Merged with {additional}: {merged}")
print(f"Original still unchanged: {original}")
immutable_dict_operations()
# 5. Working with nested immutable structures
def nested_immutable_update(data, path, new_value):
"""
Pure function: update nested data structure immutably
path is a list of keys/indices to navigate to the target
"""
if not path:
return new_value
if isinstance(data, dict):
key = path[0]
return {
**data,
key: nested_immutable_update(data.get(key), path[1:], new_value)
}
elif isinstance(data, list):
index = path[0]
if 0 <= index < len(data):
new_list = data.copy()
new_list[index] = nested_immutable_update(data[index], path[1:], new_value)
return new_list
return data
print(f"\n๐๏ธ NESTED IMMUTABLE UPDATES")
print("=" * 35)
# Test nested immutable updates
nested_data = {
'users': [
{'name': 'Alice', 'age': 30, 'preferences': {'theme': 'dark'}},
{'name': 'Bob', 'age': 25, 'preferences': {'theme': 'light'}}
],
'settings': {
'app_name': 'MyApp',
'version': '1.0'
}
}
print(f"Original nested data:")
import json
print(json.dumps(nested_data, indent=2))
# Update Alice's theme
updated_data = nested_immutable_update(
nested_data,
['users', 0, 'preferences', 'theme'],
'blue'
)
print(f"\nAfter updating Alice's theme to 'blue':")
print(json.dumps(updated_data, indent=2))
print(f"\nOriginal data unchanged:")
print(f"Alice's original theme: {nested_data['users'][0]['preferences']['theme']}")
print(f"Alice's new theme: {updated_data['users'][0]['preferences']['theme']}")Python๐งช Benefits of Pure Functions and Immutability
"""
๐งช BENEFITS DEMONSTRATION
See the practical advantages of pure functions and immutability
"""
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from functools import lru_cache
# 1. TESTABILITY
print("๐งช TESTABILITY DEMO")
print("=" * 25)
# Pure functions are easy to test
def calculate_tax_pure(income, tax_rate):
"""Pure function: easy to test"""
if income <= 0:
return 0
return income * tax_rate
def calculate_discount_pure(price, discount_percent):
"""Pure function: predictable behavior"""
if discount_percent < 0 or discount_percent > 100:
return price
return price * (1 - discount_percent / 100)
# Simple test cases
test_cases = [
(50000, 0.20, 10000), # income, tax_rate, expected_tax
(0, 0.20, 0),
(-1000, 0.15, 0),
]
print("Tax calculation tests:")
for income, rate, expected in test_cases:
result = calculate_tax_pure(income, rate)
status = "โ
PASS" if result == expected else "โ FAIL"
print(f" calculate_tax({income}, {rate}) = {result} [{status}]")
# 2. CACHEABILITY
print(f"\n๐พ CACHEABILITY DEMO")
print("=" * 25)
# Pure functions can be cached safely
@lru_cache(maxsize=128)
def expensive_calculation_pure(n):
"""Pure function: safe to cache"""
print(f" Computing expensive calculation for {n}...")
time.sleep(0.1) # Simulate expensive operation
return sum(i ** 2 for i in range(n))
print("First calls (computed):")
result1 = expensive_calculation_pure(100)
result2 = expensive_calculation_pure(200)
print("Second calls (cached):")
result3 = expensive_calculation_pure(100) # Cached - no computation
result4 = expensive_calculation_pure(200) # Cached - no computation
print(f"Results: {result1}, {result2}, {result3}, {result4}")
print(f"Cache info: {expensive_calculation_pure.cache_info()}")
# 3. THREAD SAFETY
print(f"\n๐ THREAD SAFETY DEMO")
print("=" * 25)
# Immutable data is thread-safe
immutable_config = {
'api_url': 'https://api.example.com',
'timeout': 30,
'retries': 3
}
def process_request_pure(request_data, config):
"""Pure function: thread-safe"""
url = config['api_url']
timeout = config['timeout']
# Simulate processing
return f"Processed {request_data} with {url} (timeout: {timeout}s)"
def worker_thread(thread_id, config):
"""Worker function for threading demo"""
results = []
for i in range(3):
request_data = f"request_{thread_id}_{i}"
result = process_request_pure(request_data, config)
results.append(result)
time.sleep(0.01) # Simulate work
return results
# Run multiple threads with shared immutable config
with ThreadPoolExecutor(max_workers=3) as executor:
futures = [executor.submit(worker_thread, i, immutable_config) for i in range(3)]
for i, future in enumerate(futures):
results = future.result()
print(f"Thread {i} results:")
for result in results:
print(f" {result}")
# 4. DEBUGGING AND REASONING
print(f"\n๐ DEBUGGING DEMO")
print("=" * 20)
# Pure functions are easier to debug
def data_processing_pipeline_pure(data):
"""Pure pipeline: easy to debug each step"""
def step1_filter_positive(numbers):
return [n for n in numbers if n > 0]
def step2_square_numbers(numbers):
return [n ** 2 for n in numbers]
def step3_sum_if_even(numbers):
return sum(n for n in numbers if n % 2 == 0)
# Each step is pure and testable independently
step1_result = step1_filter_positive(data)
print(f" Step 1 (filter positive): {step1_result}")
step2_result = step2_square_numbers(step1_result)
print(f" Step 2 (square): {step2_result}")
step3_result = step3_sum_if_even(step2_result)
print(f" Step 3 (sum evens): {step3_result}")
return step3_result
# Debug the pipeline
test_data = [-2, -1, 0, 1, 2, 3, 4, 5]
print(f"Input data: {test_data}")
final_result = data_processing_pipeline_pure(test_data)
print(f"Final result: {final_result}")
# 5. COMPOSITION AND REUSABILITY
print(f"\n๐งฉ COMPOSITION DEMO")
print("=" * 25)
# Pure functions compose naturally
def add_ten_pure(x):
return x + 10
def multiply_by_three_pure(x):
return x * 3
def square_pure(x):
return x ** 2
def compose_functions(*functions):
"""Compose multiple functions into a single function"""
def composed_function(x):
result = x
for func in functions:
result = func(result)
return result
return composed_function
# Create composed functions
pipeline1 = compose_functions(add_ten_pure, multiply_by_three_pure)
pipeline2 = compose_functions(square_pure, add_ten_pure, multiply_by_three_pure)
# Test compositions
test_values = [1, 2, 3, 4, 5]
print("Function composition results:")
for val in test_values:
result1 = pipeline1(val)
result2 = pipeline2(val)
print(f" {val} โ pipeline1: {result1}, pipeline2: {result2}")
# 6. MATHEMATICAL PROPERTIES
print(f"\n๐ข MATHEMATICAL PROPERTIES DEMO")
print("=" * 35)
# Pure functions follow mathematical laws
def add_pure(x, y):
return x + y
def multiply_pure(x, y):
return x * y
# Demonstrate mathematical properties
a, b, c = 2, 3, 4
# Associativity: (a + b) + c = a + (b + c)
associative_left = add_pure(add_pure(a, b), c) # (2 + 3) + 4 = 9
associative_right = add_pure(a, add_pure(b, c)) # 2 + (3 + 4) = 9
print(f"Associativity: ({a} + {b}) + {c} = {associative_left}")
print(f"Associativity: {a} + ({b} + {c}) = {associative_right}")
print(f"Equal: {associative_left == associative_right}")
# Commutativity: a + b = b + a
commutative_1 = add_pure(a, b) # 2 + 3 = 5
commutative_2 = add_pure(b, a) # 3 + 2 = 5
print(f"Commutativity: {a} + {b} = {commutative_1}")
print(f"Commutativity: {b} + {a} = {commutative_2}")
print(f"Equal: {commutative_1 == commutative_2}")
# Identity element: a + 0 = a
identity = add_pure(a, 0)
print(f"Identity: {a} + 0 = {identity} (should equal {a})")
print(f"Equal: {identity == a}")Python5. Higher-Order Functions
๐ Understanding Higher-Order Functions
A higher-order function is a function that:
- Takes one or more functions as arguments, or
- Returns a function as its result, or
- Both
They enable powerful functional programming patterns and are essential for function composition.
graph TD
A["๐ Higher-Order Functions"] --> B["๐จ Accept FunctionsAs Parameters"]
A --> C["๐ค Return FunctionsAs Results"]
A --> D["๐ Transform FunctionsModify Behavior"]
B --> B1["map(func, data)"]
B --> B2["filter(func, data)"]
B --> B3["reduce(func, data)"]
C --> C1["Decorators"]
C --> C2["Function Factories"]
C --> C3["Curried Functions"]
D --> D1["Add Logging"]
D --> D2["Add Caching"]
D --> D3["Add Validation"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐ฏ Built-in Higher-Order Functions
"""
๐ฏ PYTHON'S BUILT-IN HIGHER-ORDER FUNCTIONS
map, filter, reduce, sorted, max, min, and more
"""
from functools import reduce
import operator
# Sample data for demonstrations
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
words = ['python', 'java', 'javascript', 'go', 'rust', 'kotlin']
people = [
{'name': 'Alice', 'age': 30, 'salary': 70000},
{'name': 'Bob', 'age': 25, 'salary': 50000},
{'name': 'Charlie', 'age': 35, 'salary': 80000},
{'name': 'Diana', 'age': 28, 'salary': 65000}
]
print("๐ BUILT-IN HIGHER-ORDER FUNCTIONS DEMO")
print("=" * 50)
# 1. MAP - Apply function to every element
print("1๏ธโฃ MAP EXAMPLES:")
# Basic mapping
squared = list(map(lambda x: x ** 2, numbers))
print(f"Squared: {squared}")
# String operations
uppercase = list(map(str.upper, words))
print(f"Uppercase: {uppercase}")
# Working with dictionaries
names = list(map(lambda person: person['name'], people))
print(f"Names: {names}")
# Multiple iterables
numbers1 = [1, 2, 3, 4]
numbers2 = [10, 20, 30, 40]
sums = list(map(lambda x, y: x + y, numbers1, numbers2))
print(f"Sums: {sums}")
# Using operator module (more efficient)
products = list(map(operator.mul, numbers1, numbers2))
print(f"Products: {products}")
# 2. FILTER - Keep elements that match condition
print(f"\n2๏ธโฃ FILTER EXAMPLES:")
# Basic filtering
evens = list(filter(lambda x: x % 2 == 0, numbers))
print(f"Even numbers: {evens}")
# String filtering
short_words = list(filter(lambda word: len(word) <= 4, words))
print(f"Short words (โค4 chars): {short_words}")
# Complex filtering with dictionaries
high_earners = list(filter(lambda person: person['salary'] > 60000, people))
print(f"High earners: {[p['name'] for p in high_earners]}")
# Combining conditions
young_high_earners = list(filter(
lambda person: person['age'] < 30 and person['salary'] > 55000,
people
))
print(f"Young high earners: {[p['name'] for p in young_high_earners]}")
# 3. REDUCE - Combine all elements into single value
print(f"\n3๏ธโฃ REDUCE EXAMPLES:")
# Sum all numbers
total = reduce(lambda x, y: x + y, numbers)
print(f"Sum: {total}")
# Find maximum
maximum = reduce(lambda x, y: x if x > y else y, numbers)
print(f"Maximum: {maximum}")
# String concatenation
concatenated = reduce(lambda x, y: x + " " + y, words)
print(f"Concatenated: {concatenated}")
# Complex reduction - calculate total salary
total_salary = reduce(lambda acc, person: acc + person['salary'], people, 0)
print(f"Total salary: ${total_salary:,}")
# Find person with highest salary
richest = reduce(
lambda acc, person: person if person['salary'] > acc['salary'] else acc,
people
)
print(f"Richest person: {richest['name']} (${richest['salary']:,})")
# 4. SORTED - Sort with custom key functions
print(f"\n4๏ธโฃ SORTED EXAMPLES:")
# Sort by length
words_by_length = sorted(words, key=len)
print(f"Words by length: {words_by_length}")
# Sort by last character
words_by_last_char = sorted(words, key=lambda word: word[-1])
print(f"Words by last char: {words_by_last_char}")
# Sort people by age
people_by_age = sorted(people, key=lambda person: person['age'])
print(f"People by age: {[p['name'] for p in people_by_age]}")
# Sort by multiple criteria
people_by_salary_then_age = sorted(
people,
key=lambda person: (person['salary'], person['age']),
reverse=True
)
print(f"People by salary then age: {[f\"{p['name']} (${p['salary']}, {p['age']}y)\" for p in people_by_salary_then_age]}")
# 5. MAX/MIN with key functions
print(f"\n5๏ธโฃ MAX/MIN EXAMPLES:")
# Longest word
longest_word = max(words, key=len)
print(f"Longest word: {longest_word}")
# Oldest person
oldest = max(people, key=lambda person: person['age'])
print(f"Oldest person: {oldest['name']} ({oldest['age']} years)")
# Youngest high earner
high_earners_list = [p for p in people if p['salary'] > 60000]
youngest_high_earner = min(high_earners_list, key=lambda person: person['age'])
print(f"Youngest high earner: {youngest_high_earner['name']}")
# 6. ANY/ALL with conditions
print(f"\n6๏ธโฃ ANY/ALL EXAMPLES:")
# Check if any number is even
has_even = any(x % 2 == 0 for x in numbers)
print(f"Has even numbers: {has_even}")
# Check if all people earn more than 40k
all_well_paid = all(person['salary'] > 40000 for person in people)
print(f"All earn > $40k: {all_well_paid}")
# Check if any word starts with 'p'
starts_with_p = any(word.startswith('p') for word in words)
print(f"Any word starts with 'p': {starts_with_p}")Python๐๏ธ Creating Custom Higher-Order Functions
"""
๐๏ธ CUSTOM HIGHER-ORDER FUNCTIONS
Build your own functions that accept or return functions
"""
def apply_twice(func, value):
"""Apply a function twice to a value"""
return func(func(value))
def compose(f, g):
"""Compose two functions: compose(f, g)(x) = f(g(x))"""
return lambda x: f(g(x))
def flip(func):
"""Flip the arguments of a binary function"""
return lambda x, y: func(y, x)
def partial_application(func, *partial_args):
"""Create a new function with some arguments pre-filled"""
def inner(*remaining_args):
return func(*partial_args, *remaining_args)
return inner
def memoize(func):
"""Add memoization to a function"""
cache = {}
def memoized(*args):
if args in cache:
return cache[args]
result = func(*args)
cache[args] = result
return result
memoized.__name__ = f"memoized_{func.__name__}"
return memoized
def conditional(predicate, if_func, else_func):
"""Return a function that applies if_func or else_func based on predicate"""
def conditional_func(value):
if predicate(value):
return if_func(value)
else:
return else_func(value)
return conditional_func
print("๐๏ธ CUSTOM HIGHER-ORDER FUNCTIONS DEMO")
print("=" * 45)
# 1. Apply Twice
print("1๏ธโฃ APPLY TWICE:")
double = lambda x: x * 2
add_one = lambda x: x + 1
result1 = apply_twice(double, 5) # double(double(5)) = double(10) = 20
result2 = apply_twice(add_one, 5) # add_one(add_one(5)) = add_one(6) = 7
print(f"Apply double twice to 5: {result1}")
print(f"Apply add_one twice to 5: {result2}")
# 2. Function Composition
print(f"\n2๏ธโฃ FUNCTION COMPOSITION:")
multiply_by_3 = lambda x: x * 3
add_10 = lambda x: x + 10
# Compose: first multiply by 3, then add 10
composed = compose(add_10, multiply_by_3)
result = composed(5) # add_10(multiply_by_3(5)) = add_10(15) = 25
print(f"Compose (add_10 โ multiply_by_3)(5) = {result}")
# Chain multiple compositions
square = lambda x: x ** 2
chain = compose(add_10, compose(multiply_by_3, square))
result = chain(4) # add_10(multiply_by_3(square(4))) = add_10(multiply_by_3(16)) = add_10(48) = 58
print(f"Chain (add_10 โ multiply_by_3 โ square)(4) = {result}")
# 3. Flip Arguments
print(f"\n3๏ธโฃ FLIP ARGUMENTS:")
subtract = lambda x, y: x - y
divide = lambda x, y: x / y
flipped_subtract = flip(subtract)
flipped_divide = flip(divide)
print(f"Normal subtract(10, 3) = {subtract(10, 3)}") # 7
print(f"Flipped subtract(10, 3) = {flipped_subtract(10, 3)}") # -7
print(f"Normal divide(12, 3) = {divide(12, 3)}") # 4.0
print(f"Flipped divide(12, 3) = {flipped_divide(12, 3)}") # 0.25
# 4. Partial Application
print(f"\n4๏ธโฃ PARTIAL APPLICATION:")
def greet(greeting, name, punctuation="!"):
return f"{greeting} {name}{punctuation}"
# Pre-fill greeting
say_hello = partial_application(greet, "Hello")
say_goodbye = partial_application(greet, "Goodbye")
print(f"Say hello: {say_hello('Alice')}")
print(f"Say goodbye: {say_goodbye('Bob', '.')}")
# Mathematical example
multiply = lambda x, y: x * y
double_func = partial_application(multiply, 2)
triple_func = partial_application(multiply, 3)
print(f"Double 7: {double_func(7)}")
print(f"Triple 7: {triple_func(7)}")
# 5. Memoization
print(f"\n5๏ธโฃ MEMOIZATION:")
@memoize
def fibonacci(n):
"""Expensive recursive function - perfect for memoization"""
if n <= 1:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
@memoize
def factorial(n):
"""Another expensive recursive function"""
if n <= 1:
return 1
return n * factorial(n - 1)
# Test memoized functions
print(f"Fibonacci(10) = {fibonacci(10)}")
print(f"Fibonacci(15) = {fibonacci(15)}") # Uses cached results
print(f"Factorial(10) = {factorial(10)}")
# 6. Conditional Functions
print(f"\n6๏ธโฃ CONDITIONAL FUNCTIONS:")
# Create functions for different conditions
is_positive = lambda x: x > 0
make_positive = lambda x: abs(x)
make_negative = lambda x: -abs(x)
# Conditional function: make positive if negative, otherwise negate
conditional_func = conditional(is_positive, make_negative, make_positive)
test_values = [-5, 3, -2, 7, 0]
print("Conditional transformation (positiveโnegative, negativeโpositive):")
for val in test_values:
result = conditional_func(val)
print(f" {val} โ {result}")
# More complex example: process strings
is_short = lambda s: len(s) <= 5
make_uppercase = lambda s: s.upper()
add_exclamation = lambda s: s + "!!!"
string_processor = conditional(is_short, make_uppercase, add_exclamation)
test_strings = ["hi", "hello", "python", "code", "programming"]
print(f"\nString processing (shortโuppercase, longโadd exclamation):")
for s in test_strings:
result = string_processor(s)
print(f" '{s}' โ '{result}'")Python๐ Function Composition Patterns
"""
๐ ADVANCED FUNCTION COMPOSITION
Build complex operations from simple functions
"""
def pipe(*functions):
"""Pipe functions left to right: pipe(f, g, h)(x) = h(g(f(x)))"""
def piped_function(value):
result = value
for func in functions:
result = func(result)
return result
return piped_function
def compose_right(*functions):
"""Compose functions right to left: compose(f, g, h)(x) = f(g(h(x)))"""
def composed_function(value):
result = value
for func in reversed(functions):
result = func(result)
return result
return composed_function
def conditional_pipe(condition, true_pipe, false_pipe):
"""Apply different pipelines based on condition"""
def conditional_pipeline(value):
if condition(value):
return true_pipe(value)
else:
return false_pipe(value)
return conditional_pipeline
def parallel_map(func, *iterables):
"""Apply function to multiple iterables in parallel"""
return tuple(map(func, iterable) for iterable in iterables)
def chain_operations(*operations):
"""Chain operations that take and return iterables"""
def chained(data):
result = data
for operation in operations:
result = operation(result)
return result
return chained
print("๐ ADVANCED COMPOSITION PATTERNS")
print("=" * 40)
# 1. Pipe (Left to Right)
print("1๏ธโฃ PIPE (LEFT TO RIGHT):")
# Create simple functions
add_5 = lambda x: x + 5
multiply_2 = lambda x: x * 2
square_it = lambda x: x ** 2
# Pipe them together: ((x + 5) * 2) ^ 2
pipeline = pipe(add_5, multiply_2, square_it)
test_value = 3
result = pipeline(test_value)
print(f"Pipeline({test_value}): add_5 โ multiply_2 โ square")
print(f" Step 1: {test_value} + 5 = {add_5(test_value)}")
print(f" Step 2: {add_5(test_value)} * 2 = {multiply_2(add_5(test_value))}")
print(f" Step 3: {multiply_2(add_5(test_value))} ^ 2 = {result}")
# 2. String Processing Pipeline
print(f"\n2๏ธโฃ STRING PROCESSING PIPELINE:")
remove_spaces = lambda s: s.replace(" ", "")
to_lowercase = lambda s: s.lower()
reverse_string = lambda s: s[::-1]
add_prefix = lambda s: f"processed_{s}"
text_pipeline = pipe(remove_spaces, to_lowercase, reverse_string, add_prefix)
sample_text = "Hello World Python"
result = text_pipeline(sample_text)
print(f"Text pipeline: '{sample_text}' โ '{result}'")
# 3. Data Processing Pipeline
print(f"\n3๏ธโฃ DATA PROCESSING PIPELINE:")
# Process list of numbers
filter_positive = lambda lst: [x for x in lst if x > 0]
double_all = lambda lst: [x * 2 for x in lst]
sum_all = lambda lst: sum(lst)
data_pipeline = pipe(filter_positive, double_all, sum_all)
sample_data = [-3, -1, 2, 4, -2, 5, 3]
result = data_pipeline(sample_data)
print(f"Data pipeline: {sample_data}")
print(f" Filter positive: {filter_positive(sample_data)}")
print(f" Double all: {double_all(filter_positive(sample_data))}")
print(f" Sum: {result}")
# 4. Conditional Pipelines
print(f"\n4๏ธโฃ CONDITIONAL PIPELINES:")
is_even = lambda x: x % 2 == 0
# Different pipelines for even and odd numbers
even_pipeline = pipe(lambda x: x // 2, lambda x: x + 10) # Divide by 2, add 10
odd_pipeline = pipe(lambda x: x * 3, lambda x: x + 1) # Multiply by 3, add 1
conditional_processor = conditional_pipe(is_even, even_pipeline, odd_pipeline)
test_numbers = [2, 3, 8, 7, 12, 15]
print("Conditional processing (even: รท2+10, odd: ร3+1):")
for num in test_numbers:
result = conditional_processor(num)
pipeline_type = "even" if is_even(num) else "odd"
print(f" {num} ({pipeline_type}) โ {result}")
# 5. Complex Data Transformation
print(f"\n5๏ธโฃ COMPLEX DATA TRANSFORMATION:")
# Sample data: list of dictionaries
employees = [
{"name": "Alice", "department": "Engineering", "salary": 70000, "years": 3},
{"name": "Bob", "department": "Marketing", "salary": 50000, "years": 1},
{"name": "Charlie", "department": "Engineering", "salary": 80000, "years": 5},
{"name": "Diana", "department": "HR", "salary": 60000, "years": 2}
]
# Build transformation pipeline
filter_engineers = lambda emps: [e for e in emps if e["department"] == "Engineering"]
add_bonus = lambda emps: [{**e, "bonus": e["salary"] * 0.1 * e["years"]} for e in emps]
sort_by_total = lambda emps: sorted(emps, key=lambda e: e["salary"] + e["bonus"], reverse=True)
extract_names = lambda emps: [e["name"] for e in emps]
engineer_bonus_pipeline = pipe(
filter_engineers,
add_bonus,
sort_by_total,
extract_names
)
result = engineer_bonus_pipeline(employees)
print(f"Engineer bonus pipeline result: {result}")
# 6. Functional Error Handling
print(f"\n6๏ธโฃ FUNCTIONAL ERROR HANDLING:")
def safe_divide(x, y):
"""Safe division that returns None on error"""
try:
return x / y if y != 0 else None
except:
return None
def safe_sqrt(x):
"""Safe square root that returns None for negative numbers"""
return x ** 0.5 if x >= 0 else None
def safe_pipe(*functions):
"""Pipe that stops on None (error)"""
def safe_piped_function(value):
result = value
for func in functions:
if result is None:
return None
result = func(result)
return result
return safe_piped_function
# Create safe mathematical pipeline
safe_math_pipeline = safe_pipe(
lambda x: safe_divide(x, 2), # Divide by 2
lambda x: x - 1 if x else None, # Subtract 1
safe_sqrt # Square root
)
test_values = [16, -4, 0, 8]
print("Safe math pipeline (รท2, -1, โ):")
for val in test_values:
result = safe_math_pipeline(val)
status = "โ
" if result is not None else "โ"
print(f" {val} โ {result} {status}")Python6. Lambda Functions and Closures
๐น Understanding Lambda Functions
Lambda functions are anonymous functions defined with the lambda keyword. They’re perfect for short, one-line functions used in functional programming patterns.
graph TD
A["๐น Lambda Functions"] --> B["๐ Anonymous FunctionsNo def keyword"]
A --> C["๐ฏ Single ExpressionOne line only"]
A --> D["๐ฆ First-Class ObjectsCan be assigned"]
B --> B1["lambda x: x * 2"]
B --> B2["lambda x, y: x + y"]
B --> B3["lambda: 'Hello'"]
C --> C1["No statements allowed"]
C --> C2["No return keyword"]
C --> C3["Expression is returned"]
D --> D1["Assign to variables"]
D --> D2["Pass as arguments"]
D --> D3["Store in collections"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐ฏ Lambda Function Basics
"""
๐ฏ LAMBDA FUNCTION FUNDAMENTALS
Mastering anonymous functions in Python
"""
print("๐น LAMBDA FUNCTION BASICS")
print("=" * 35)
# 1. Basic lambda syntax
print("1๏ธโฃ BASIC SYNTAX:")
# Simple lambda
square = lambda x: x ** 2
print(f"Lambda square(5): {square(5)}")
# Multiple parameters
add = lambda x, y: x + y
print(f"Lambda add(3, 7): {add(3, 7)}")
# No parameters
greeting = lambda: "Hello, World!"
print(f"Lambda greeting(): {greeting()}")
# Default parameters
power = lambda x, n=2: x ** n
print(f"Lambda power(4): {power(4)}") # 4^2 = 16
print(f"Lambda power(4, 3): {power(4, 3)}") # 4^3 = 64
# 2. Lambda vs regular functions
print(f"\n2๏ธโฃ LAMBDA vs REGULAR FUNCTIONS:")
# Regular function
def multiply_regular(x, y):
return x * y
# Lambda equivalent
multiply_lambda = lambda x, y: x * y
print(f"Regular function: {multiply_regular(6, 7)}")
print(f"Lambda function: {multiply_lambda(6, 7)}")
# 3. Lambda with conditional expressions
print(f"\n3๏ธโฃ CONDITIONAL LAMBDAS:")
# Ternary operator in lambda
abs_value = lambda x: x if x >= 0 else -x
print(f"Abs value of -5: {abs_value(-5)}")
print(f"Abs value of 3: {abs_value(3)}")
# More complex conditions
classify_number = lambda x: "positive" if x > 0 else "negative" if x < 0 else "zero"
test_numbers = [-3, 0, 5, -1, 2]
for num in test_numbers:
print(f" {num} is {classify_number(num)}")
# 4. Lambda with built-in functions
print(f"\n4๏ธโฃ LAMBDA WITH BUILT-INS:")
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# With map
doubled = list(map(lambda x: x * 2, numbers))
print(f"Doubled: {doubled}")
# With filter
evens = list(filter(lambda x: x % 2 == 0, numbers))
print(f"Evens: {evens}")
# With sorted
words = ["python", "java", "c", "javascript", "go"]
by_length = sorted(words, key=lambda word: len(word))
print(f"Sorted by length: {by_length}")
# With max/min
people = [
{"name": "Alice", "age": 30},
{"name": "Bob", "age": 25},
{"name": "Charlie", "age": 35}
]
oldest = max(people, key=lambda person: person["age"])
print(f"Oldest person: {oldest['name']}")
# 5. Multiple expressions using tuples/lists
print(f"\n5๏ธโฃ MULTIPLE OPERATIONS:")
# Using tuple to return multiple values
coords = lambda x: (x, x**2, x**3) # x, x^2, x^3
result = coords(3)
print(f"Coordinates for 3: {result}")
# Using list comprehension for complex logic
process_list = lambda lst: [x*2 for x in lst if x > 0]
mixed_numbers = [-2, -1, 0, 1, 2, 3]
processed = process_list(mixed_numbers)
print(f"Processed list: {processed}")
# 6. Lambda limitations
print(f"\n6๏ธโฃ LAMBDA LIMITATIONS:")
# โ Can't use statements
try:
# This would fail: lambda x: print(x) # print is a statement
pass
except:
pass
print("โ Cannot use statements like print, if statements, loops")
print("โ Cannot have multiple statements")
print("โ Cannot have docstrings")
print("โ
Can only contain expressions")
# Workaround: use functions that return values
log_and_double = lambda x: (print(f"Processing {x}") or x * 2)
result = log_and_double(5) # This works because print() returns None
print(f"Result: {result}")Python๐โโ๏ธ Advanced Lambda Patterns
"""
๐โโ๏ธ ADVANCED LAMBDA PATTERNS
Complex use cases and techniques
"""
from operator import itemgetter, attrgetter
from functools import partial
print("๐โโ๏ธ ADVANCED LAMBDA PATTERNS")
print("=" * 35)
# 1. Lambda with complex data structures
print("1๏ธโฃ COMPLEX DATA STRUCTURES:")
# Working with nested dictionaries
users = [
{"name": "Alice", "profile": {"age": 30, "skills": ["Python", "SQL"]}},
{"name": "Bob", "profile": {"age": 25, "skills": ["Java", "React"]}},
{"name": "Charlie", "profile": {"age": 35, "skills": ["Python", "ML", "Docker"]}}
]
# Extract nested data
ages = list(map(lambda user: user["profile"]["age"], users))
print(f"Ages: {ages}")
# Filter by nested condition
python_users = list(filter(lambda user: "Python" in user["profile"]["skills"], users))
print(f"Python users: {[u['name'] for u in python_users]}")
# Sort by number of skills
by_skill_count = sorted(users, key=lambda user: len(user["profile"]["skills"]), reverse=True)
print(f"By skill count: {[(u['name'], len(u['profile']['skills'])) for u in by_skill_count]}")
# 2. Lambda factories (higher-order lambdas)
print(f"\n2๏ธโฃ LAMBDA FACTORIES:")
# Create specialized functions
make_multiplier = lambda factor: lambda x: x * factor
make_validator = lambda min_val, max_val: lambda x: min_val <= x <= max_val
make_formatter = lambda template: lambda value: template.format(value)
# Use the factories
double = make_multiplier(2)
triple = make_multiplier(3)
age_validator = make_validator(18, 65)
currency_formatter = make_formatter("${:.2f}")
print(f"Double 7: {double(7)}")
print(f"Triple 4: {triple(4)}")
print(f"Valid age 25: {age_validator(25)}")
print(f"Valid age 70: {age_validator(70)}")
print(f"Format currency: {currency_formatter(123.456)}")
# 3. Lambda with partial application
print(f"\n3๏ธโฃ PARTIAL APPLICATION WITH LAMBDAS:")
# Partially apply arguments
def greet(greeting, name, punctuation="!"):
return f"{greeting} {name}{punctuation}"
# Using partial
say_hello = partial(greet, "Hello")
say_goodbye = partial(greet, "Goodbye")
# Using lambda for partial application
create_greeter = lambda greeting: lambda name, punct="!": f"{greeting} {name}{punct}"
hello_greeter = create_greeter("Hello")
bye_greeter = create_greeter("Goodbye")
print(f"Partial hello: {say_hello('Alice')}")
print(f"Lambda hello: {hello_greeter('Bob')}")
# 4. Composition with lambdas
print(f"\n4๏ธโฃ FUNCTION COMPOSITION:")
# Simple composition
compose = lambda f, g: lambda x: f(g(x))
add_5 = lambda x: x + 5
multiply_2 = lambda x: x * 2
square = lambda x: x ** 2
# Compose functions
add_then_multiply = compose(multiply_2, add_5)
square_then_add = compose(add_5, square)
print(f"Add 5 then multiply by 2 (3): {add_then_multiply(3)}") # (3+5)*2 = 16
print(f"Square then add 5 (4): {square_then_add(4)}") # 4^2+5 = 21
# Chain multiple compositions
chain = lambda *funcs: lambda x: x if not funcs else chain(*funcs[1:])(funcs[0](x))
pipeline = chain(add_5, multiply_2, square)
result = pipeline(2) # ((2+5)*2)^2 = (14)^2 = 196
print(f"Pipeline result: {result}")
# 5. Lambda with recursion (Y combinator style)
print(f"\n5๏ธโฃ RECURSIVE LAMBDAS:")
# Factorial using lambda
factorial = (lambda f, n: 1 if n <= 1 else n * f(f, n-1))
result = factorial(factorial, 5)
print(f"Factorial 5: {result}")
# Fibonacci using lambda
fibonacci = (lambda f, n: n if n <= 1 else f(f, n-1) + f(f, n-2))
fib_sequence = [fibonacci(fibonacci, i) for i in range(10)]
print(f"Fibonacci sequence: {fib_sequence}")
# 6. Advanced filtering and transformation
print(f"\n6๏ธโฃ ADVANCED FILTERING:")
# Complex data processing
transactions = [
{"id": 1, "type": "credit", "amount": 1000, "date": "2024-01-15"},
{"id": 2, "type": "debit", "amount": 250, "date": "2024-01-16"},
{"id": 3, "type": "credit", "amount": 500, "date": "2024-01-17"},
{"id": 4, "type": "debit", "amount": 100, "date": "2024-01-18"},
]
# Multi-condition filtering
large_credits = list(filter(
lambda t: t["type"] == "credit" and t["amount"] > 500,
transactions
))
print(f"Large credits: {[t['id'] for t in large_credits]}")
# Complex transformation
summary = list(map(
lambda t: {
"id": t["id"],
"description": f"{t['type'].upper()}: ${t['amount']}",
"sign": 1 if t["type"] == "credit" else -1
},
transactions
))
for item in summary:
print(f" {item}")
# Calculate balance using lambda
balance = sum(map(
lambda t: t["amount"] if t["type"] == "credit" else -t["amount"],
transactions
))
print(f"Final balance: ${balance}")Python๐ Understanding Closures
Closures occur when a function defined inside another function references variables from the outer function. The inner function “closes over” these variables, keeping them alive even after the outer function returns.
graph TD
A["๐ Closures"] --> B["๐ Outer FunctionEnclosing Scope"]
A --> C["๐ Inner FunctionNested Function"]
A --> D["๐ Free VariablesNon-local Variables"]
B --> B1["def outer(x):"]
B --> B2[" local_var = x * 2"]
B --> B3[" return inner"]
C --> C1["def inner():"]
C --> C2[" return local_var + 1"]
C --> C3["# References outer scope"]
D --> D1["Captured variables"]
D --> D2["Persist after outer returns"]
D --> D3["Each closure is unique"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐ Closure Examples and Patterns
"""
๐ CLOSURES: Functions that capture their environment
Understanding lexical scoping and variable capture
"""
print("๐ CLOSURE FUNDAMENTALS")
print("=" * 30)
# 1. Basic closure
print("1๏ธโฃ BASIC CLOSURE:")
def make_multiplier(factor):
"""Outer function that returns a closure"""
def multiplier(number):
"""Inner function that closes over 'factor'"""
return number * factor
return multiplier
# Create closures with different factors
double = make_multiplier(2)
triple = make_multiplier(3)
ten_x = make_multiplier(10)
print(f"Double 5: {double(5)}")
print(f"Triple 7: {triple(7)}")
print(f"10x 4: {ten_x(4)}")
# Each closure maintains its own copy of the captured variable
print(f"Double's factor: {double.__closure__[0].cell_contents}")
print(f"Triple's factor: {triple.__closure__[0].cell_contents}")
# 2. Closure with mutable state
print(f"\n2๏ธโฃ CLOSURE WITH STATE:")
def make_counter(initial=0, step=1):
"""Create a counter function with persistent state"""
count = initial
def counter():
nonlocal count # Allows modification of captured variable
count += step
return count
def get_count():
return count
def reset():
nonlocal count
count = initial
# Return multiple functions that share state
counter.get = get_count
counter.reset = reset
return counter
# Create counters with different behaviors
basic_counter = make_counter()
even_counter = make_counter(0, 2)
countdown = make_counter(10, -1)
print("Basic counter:")
for i in range(3):
print(f" Count: {basic_counter()}")
print(f"Current count: {basic_counter.get()}")
print("Even counter:")
for i in range(3):
print(f" Count: {even_counter()}")
print("Countdown:")
for i in range(3):
print(f" Count: {countdown()}")
# 3. Closure for configuration
print(f"\n3๏ธโฃ CONFIGURATION CLOSURES:")
def create_validator(min_length=1, max_length=100, required_chars=None):
"""Create a specialized validator function"""
required_chars = required_chars or []
def validate(text):
"""Validation function with captured configuration"""
if len(text) < min_length:
return f"Too short (minimum {min_length} characters)"
if len(text) > max_length:
return f"Too long (maximum {max_length} characters)"
for char in required_chars:
if char not in text:
return f"Missing required character: '{char}'"
return "Valid"
return validate
# Create specialized validators
password_validator = create_validator(8, 50, ['@', '!'])
username_validator = create_validator(3, 20)
email_validator = create_validator(5, 100, ['@', '.'])
# Test validators
test_cases = [
("john", username_validator),
("john@example.com", email_validator),
("pass123", password_validator),
("MyPass@123!", password_validator)
]
print("Validation results:")
for text, validator in test_cases:
result = validator(text)
print(f" '{text}': {result}")
# 4. Closure with lambda
print(f"\n4๏ธโฃ LAMBDA CLOSURES:")
def make_operation(operation_name):
"""Factory that returns lambda closures"""
operations = {
'add': lambda x, y: x + y,
'multiply': lambda x, y: x * y,
'power': lambda x, y: x ** y,
'divide': lambda x, y: x / y if y != 0 else float('inf')
}
if operation_name in operations:
op_func = operations[operation_name]
# Lambda that closes over op_func and operation_name
return lambda x, y: {
'operation': operation_name,
'operands': (x, y),
'result': op_func(x, y)
}
return lambda x, y: {'error': f'Unknown operation: {operation_name}'}
# Create operation functions
add_op = make_operation('add')
mult_op = make_operation('multiply')
unknown_op = make_operation('unknown')
print("Lambda closure results:")
print(f"Add: {add_op(5, 3)}")
print(f"Multiply: {mult_op(4, 7)}")
print(f"Unknown: {unknown_op(1, 2)}")
# 5. Advanced closure patterns
print(f"\n5๏ธโฃ ADVANCED PATTERNS:")
def create_accumulator():
"""Create an accumulator with history"""
history = []
total = 0
def accumulate(value):
nonlocal total
total += value
history.append(value)
return total
def get_history():
return history.copy() # Return copy to prevent external modification
def get_average():
return sum(history) / len(history) if history else 0
def clear():
nonlocal total
total = 0
history.clear()
# Attach methods to function
accumulate.history = get_history
accumulate.average = get_average
accumulate.clear = clear
accumulate.total = lambda: total
return accumulate
# Use the accumulator
acc = create_accumulator()
values = [10, 20, 15, 5, 30]
print("Accumulator in action:")
for val in values:
result = acc(val)
print(f" Add {val}: total = {result}")
print(f"History: {acc.history()}")
print(f"Average: {acc.average():.2f}")
print(f"Total: {acc.total()}")
# 6. Closure gotchas and solutions
print(f"\n6๏ธโฃ CLOSURE GOTCHAS:")
# Problem: Late binding closure (common mistake)
print("โ Common mistake with loops:")
functions = []
for i in range(3):
functions.append(lambda x: x * i) # i is captured by reference!
# All functions use the final value of i (2)
print("Results (all use i=2):")
for func in functions:
print(f" func(5) = {func(5)}")
print("\nโ
Solution 1: Default parameter")
functions_fixed1 = []
for i in range(3):
functions_fixed1.append(lambda x, multiplier=i: x * multiplier)
print("Fixed with default parameter:")
for func in functions_fixed1:
print(f" func(5) = {func(5)}")
print("\nโ
Solution 2: Function factory")
def make_multiplier_func(multiplier):
return lambda x: x * multiplier
functions_fixed2 = [make_multiplier_func(i) for i in range(3)]
print("Fixed with function factory:")
for func in functions_fixed2:
print(f" func(5) = {func(5)}")
# Memory consideration
print(f"\n๐ง MEMORY CONSIDERATIONS:")
import sys
def show_closure_size():
x = list(range(1000)) # Large data
def small_closure():
return x[0] # Only uses first element
def references_all():
return sum(x) # Uses entire list
return small_closure, references_all
small, large = show_closure_size()
print(f"Small closure size: {sys.getsizeof(small.__closure__[0].cell_contents)} bytes")
print("Note: Both closures keep the entire list in memory!")Python๐ฏ Practical Applications of Lambdas and Closures
"""
๐ฏ PRACTICAL APPLICATIONS
Real-world uses of lambdas and closures
"""
from datetime import datetime
import json
from functools import wraps
print("๐ฏ PRACTICAL APPLICATIONS")
print("=" * 30)
# 1. Event System with Closures
print("1๏ธโฃ EVENT SYSTEM:")
def create_event_system():
"""Create an event system using closures"""
listeners = {}
def on(event_name, callback):
"""Register event listener"""
if event_name not in listeners:
listeners[event_name] = []
listeners[event_name].append(callback)
def emit(event_name, *args, **kwargs):
"""Emit event to all listeners"""
if event_name in listeners:
results = []
for callback in listeners[event_name]:
try:
result = callback(*args, **kwargs)
results.append(result)
except Exception as e:
results.append(f"Error: {e}")
return results
return []
def off(event_name, callback=None):
"""Remove event listener"""
if event_name in listeners:
if callback:
listeners[event_name] = [cb for cb in listeners[event_name] if cb != callback]
else:
listeners[event_name] = []
return on, emit, off
# Create event system
on, emit, off = create_event_system()
# Register event handlers using lambdas
on('user_login', lambda user: f"Welcome {user}!")
on('user_login', lambda user: f"Analytics: {user} logged in at {datetime.now()}")
on('data_update', lambda data: f"Processing {len(data)} items")
# Emit events
print("Login events:")
results = emit('user_login', 'Alice')
for result in results:
print(f" {result}")
print("Data update events:")
results = emit('data_update', [1, 2, 3, 4, 5])
for result in results:
print(f" {result}")
# 2. Memoization with Closures
print(f"\n2๏ธโฃ MEMOIZATION:")
def memoize_with_ttl(ttl_seconds=60):
"""Memoization decorator with time-to-live"""
def decorator(func):
cache = {}
@wraps(func)
def wrapper(*args, **kwargs):
# Create cache key
key = str(args) + str(sorted(kwargs.items()))
current_time = datetime.now().timestamp()
# Check if cached and not expired
if key in cache:
result, timestamp = cache[key]
if current_time - timestamp < ttl_seconds:
print(f"Cache hit for {func.__name__}{args}")
return result
# Compute and cache result
result = func(*args, **kwargs)
cache[key] = (result, current_time)
print(f"Cache miss for {func.__name__}{args}")
return result
wrapper.clear_cache = lambda: cache.clear()
wrapper.cache_info = lambda: {'size': len(cache), 'keys': list(cache.keys())}
return wrapper
return decorator
@memoize_with_ttl(5) # 5-second TTL
def expensive_computation(n):
"""Simulate expensive computation"""
import time
time.sleep(0.1) # Simulate work
return sum(i**2 for i in range(n))
# Test memoization
print("Testing memoized function:")
result1 = expensive_computation(100)
result2 = expensive_computation(100) # Should be cached
print(f"Cache info: {expensive_computation.cache_info()}")
# 3. Configuration Builder
print(f"\n3๏ธโฃ CONFIGURATION BUILDER:")
def create_config_builder():
"""Create a configuration builder using closures"""
config = {}
def set_value(key, value):
"""Set configuration value"""
config[key] = value
return config_builder # Enable chaining
def get_value(key, default=None):
"""Get configuration value"""
return config.get(key, default)
def build():
"""Build final configuration"""
return config.copy()
def reset():
"""Reset configuration"""
config.clear()
return config_builder
# Create the builder object with methods
config_builder = lambda: config.copy() # Default call returns current config
config_builder.set = set_value
config_builder.get = get_value
config_builder.build = build
config_builder.reset = reset
return config_builder
# Use configuration builder
config = create_config_builder()
final_config = (config
.set('database_url', 'postgresql://localhost:5432/mydb')
.set('debug', True)
.set('max_connections', 100)
.build())
print(f"Built configuration: {json.dumps(final_config, indent=2)}")
# 4. Data Processing Pipeline
print(f"\n4๏ธโฃ DATA PROCESSING PIPELINE:")
def create_pipeline():
"""Create a data processing pipeline"""
steps = []
def add_step(func):
"""Add processing step"""
steps.append(func)
return pipeline
def process(data):
"""Process data through all steps"""
result = data
for i, step in enumerate(steps):
try:
result = step(result)
print(f" Step {i+1}: {len(result) if hasattr(result, '__len__') else result}")
except Exception as e:
print(f" Step {i+1} failed: {e}")
break
return result
pipeline = lambda data: process(data)
pipeline.add = add_step
pipeline.clear = lambda: steps.clear()
return pipeline
# Create processing pipeline
pipeline = create_pipeline()
# Add processing steps using lambdas
pipeline.add(lambda data: [x for x in data if x > 0]) # Filter positive
pipeline.add(lambda data: [x * 2 for x in data]) # Double values
pipeline.add(lambda data: sum(data)) # Sum all
# Process data
sample_data = [-3, -1, 2, 4, -2, 5, 3]
print(f"Processing {sample_data}:")
result = pipeline(sample_data)
print(f"Final result: {result}")
# 5. State Machine with Closures
print(f"\n5๏ธโฃ STATE MACHINE:")
def create_state_machine(initial_state, transitions):
"""Create a finite state machine"""
current_state = initial_state
history = [initial_state]
def transition(event):
"""Transition to new state based on event"""
nonlocal current_state
if current_state in transitions and event in transitions[current_state]:
new_state = transitions[current_state][event]
print(f"Transition: {current_state} --{event}--> {new_state}")
current_state = new_state
history.append(new_state)
return True
else:
print(f"Invalid transition: {current_state} --{event}--> ?")
return False
def get_state():
return current_state
def get_history():
return history.copy()
def can_transition(event):
return current_state in transitions and event in transitions[current_state]
# Create state machine object
sm = transition
sm.state = get_state
sm.history = get_history
sm.can = can_transition
return sm
# Define state machine for a simple lock
lock_transitions = {
'locked': {'insert_key': 'key_inserted'},
'key_inserted': {'turn_key': 'unlocked', 'remove_key': 'locked'},
'unlocked': {'turn_key': 'locked'}
}
# Create and use state machine
lock = create_state_machine('locked', lock_transitions)
print("State machine demo:")
events = ['insert_key', 'turn_key', 'turn_key', 'invalid_event']
for event in events:
print(f"Current state: {lock.state()}")
print(f"Can '{event}'? {lock.can(event)}")
lock(event)
print()
print(f"Final state: {lock.state()}")
print(f"History: {' -> '.join(lock.history())}")Python7. Map, Filter, and Reduce
๐บ๏ธ The Holy Trinity of Functional Programming
Map, Filter, and Reduce are the three fundamental operations that form the backbone of functional programming. They allow you to transform, filter, and aggregate data in a declarative, composable way.
graph TD
A["๐บ๏ธ Map, Filter, Reduce"] --> B["๐ MapTransform Each Element"]
A --> C["๐ FilterSelect Elements"]
A --> D["๐ ReduceCombine Elements"]
B --> B1["Apply function to each"]
B --> B2["One-to-one mapping"]
B --> B3["Same length output"]
C --> C1["Test each element"]
C --> C2["Keep matching elements"]
C --> C3["Smaller/equal output"]
D --> D1["Accumulate values"]
D --> D2["Combine into single result"]
D --> D3["Single output value"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐ Map: Transform Every Element
"""
๐ MAP: Apply a function to every element in a sequence
Transform data from one form to another
"""
print("๐ MAP FUNCTION MASTERY")
print("=" * 30)
# 1. Basic mapping examples
print("1๏ธโฃ BASIC MAPPING:")
numbers = [1, 2, 3, 4, 5]
# Square all numbers
squared = list(map(lambda x: x ** 2, numbers))
print(f"Original: {numbers}")
print(f"Squared: {squared}")
# Convert to strings
strings = list(map(str, numbers))
print(f"As strings: {strings}")
# Mathematical operations
doubled = list(map(lambda x: x * 2, numbers))
print(f"Doubled: {doubled}")
# 2. String transformations
print(f"\n2๏ธโฃ STRING TRANSFORMATIONS:")
words = ['python', 'java', 'javascript', 'golang']
# Uppercase all words
upper_words = list(map(str.upper, words))
print(f"Uppercase: {upper_words}")
# Get word lengths
lengths = list(map(len, words))
print(f"Lengths: {lengths}")
# Add prefix to each word
prefixed = list(map(lambda word: f"lang_{word}", words))
print(f"Prefixed: {prefixed}")
# Capitalize first letter
capitalized = list(map(lambda word: word.capitalize(), words))
print(f"Capitalized: {capitalized}")
# 3. Working with multiple iterables
print(f"\n3๏ธโฃ MULTIPLE ITERABLES:")
numbers1 = [1, 2, 3, 4, 5]
numbers2 = [10, 20, 30, 40, 50]
numbers3 = [100, 200, 300, 400, 500]
# Add corresponding elements
sums = list(map(lambda x, y: x + y, numbers1, numbers2))
print(f"Sums: {sums}")
# Multiply three sequences
products = list(map(lambda x, y, z: x * y * z, numbers1, numbers2, numbers3))
print(f"Products: {products}")
# Different length sequences (stops at shortest)
short_seq = [1, 2]
long_seq = [10, 20, 30, 40]
result = list(map(lambda x, y: x + y, short_seq, long_seq))
print(f"Different lengths: {result}") # Only [11, 22]
# 4. Complex data transformations
print(f"\n4๏ธโฃ COMPLEX DATA TRANSFORMATIONS:")
people = [
{'name': 'Alice', 'age': 30, 'city': 'New York'},
{'name': 'Bob', 'age': 25, 'city': 'London'},
{'name': 'Charlie', 'age': 35, 'city': 'Tokyo'}
]
# Extract names
names = list(map(lambda person: person['name'], people))
print(f"Names: {names}")
# Create full descriptions
descriptions = list(map(
lambda person: f"{person['name']}, {person['age']} years old, lives in {person['city']}",
people
))
for desc in descriptions:
print(f" {desc}")
# Transform to new structure
simplified = list(map(
lambda person: {
'id': person['name'].lower(),
'display_name': f"{person['name']} ({person['city']})",
'is_senior': person['age'] >= 30
},
people
))
for item in simplified:
print(f" {item}")
# 5. Nested data transformation
print(f"\n5๏ธโฃ NESTED DATA TRANSFORMATION:")
# Nested lists
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
# Square each element in each row
squared_matrix = list(map(lambda row: list(map(lambda x: x ** 2, row)), matrix))
print(f"Original matrix: {matrix}")
print(f"Squared matrix: {squared_matrix}")
# Sum each row
row_sums = list(map(sum, matrix))
print(f"Row sums: {row_sums}")
# Get first element of each row
first_elements = list(map(lambda row: row[0], matrix))
print(f"First elements: {first_elements}")
# 6. Map with custom functions
print(f"\n6๏ธโฃ CUSTOM FUNCTIONS:")
def process_text(text):
"""Complex text processing function"""
# Remove spaces, lowercase, reverse, then capitalize
processed = text.replace(' ', '').lower()[::-1].capitalize()
return processed
def calculate_grade(score):
"""Convert numeric score to letter grade"""
if score >= 90:
return 'A'
elif score >= 80:
return 'B'
elif score >= 70:
return 'C'
elif score >= 60:
return 'D'
else:
return 'F'
def format_currency(amount, currency='USD'):
"""Format number as currency"""
symbols = {'USD': '$', 'EUR': 'โฌ', 'GBP': 'ยฃ', 'JPY': 'ยฅ'}
symbol = symbols.get(currency, '$')
return f"{symbol}{amount:,.2f}"
# Test custom functions with map
texts = ['Hello World', 'Python Programming', 'Functional Code']
processed_texts = list(map(process_text, texts))
print(f"Processed texts: {processed_texts}")
scores = [95, 87, 72, 65, 45]
grades = list(map(calculate_grade, scores))
print(f"Scores to grades: {list(zip(scores, grades))}")
amounts = [1234.56, 9876.54, 100.00]
formatted = list(map(lambda x: format_currency(x, 'EUR'), amounts))
print(f"Formatted currency: {formatted}")
# 7. Performance considerations and alternatives
print(f"\n7๏ธโฃ PERFORMANCE & ALTERNATIVES:")
# Map vs list comprehension performance comparison
import time
large_numbers = list(range(100000))
# Using map
start_time = time.time()
mapped_result = list(map(lambda x: x ** 2, large_numbers))
map_time = time.time() - start_time
# Using list comprehension
start_time = time.time()
list_comp_result = [x ** 2 for x in large_numbers]
list_comp_time = time.time() - start_time
print(f"Map time: {map_time:.4f} seconds")
print(f"List comprehension time: {list_comp_time:.4f} seconds")
print(f"Results equal: {mapped_result == list_comp_result}")
# Generator version (memory efficient)
mapped_generator = map(lambda x: x ** 2, large_numbers)
print(f"Map returns iterator: {type(mapped_generator)}")
print(f"Memory efficient - processes on demand")
# Using map with built-in functions (faster)
print(f"String to int conversion: {converted}")
### ๐ Filter: Select Elements That Match
```python
"""
๐ FILTER: Keep only elements that satisfy a condition
Remove unwanted data based on criteria
"""
print("๐ FILTER FUNCTION MASTERY")
print("=" * 30)
# 1. Basic filtering examples
print("1๏ธโฃ BASIC FILTERING:")
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Filter even numbers
evens = list(filter(lambda x: x % 2 == 0, numbers))
print(f"Original: {numbers}")
print(f"Even numbers: {evens}")
# Filter positive numbers
mixed_numbers = [-3, -1, 0, 1, 2, 3, 4]
positives = list(filter(lambda x: x > 0, mixed_numbers))
print(f"Mixed: {mixed_numbers}")
print(f"Positives: {positives}")
# Filter numbers greater than 5
large_numbers = list(filter(lambda x: x > 5, numbers))
print(f"Numbers > 5: {large_numbers}")
# 2. String filtering
print(f"\n2๏ธโฃ STRING FILTERING:")
words = ['python', 'java', 'c', 'javascript', 'go', 'rust', 'kotlin', 'swift']
# Filter short words
short_words = list(filter(lambda word: len(word) <= 4, words))
print(f"Short words (โค4 chars): {short_words}")
# Filter words starting with specific letter
j_words = list(filter(lambda word: word.startswith('j'), words))
print(f"Words starting with 'j': {j_words}")
# Filter words containing specific substring
script_words = list(filter(lambda word: 'script' in word, words))
print(f"Words containing 'script': {script_words}")
# Filter by multiple conditions
short_and_vowel = list(filter(
lambda word: len(word) <= 4 and word[0].lower() in 'aeiou',
words
))
print(f"Short words starting with vowel: {short_and_vowel}")
# 3. Complex object filtering
print(f"\n3๏ธโฃ COMPLEX OBJECT FILTERING:")
employees = [
{'name': 'Alice', 'department': 'Engineering', 'salary': 75000, 'years': 3},
{'name': 'Bob', 'department': 'Marketing', 'salary': 45000, 'years': 1},
{'name': 'Charlie', 'department': 'Engineering', 'salary': 85000, 'years': 5},
{'name': 'Diana', 'department': 'HR', 'salary': 55000, 'years': 2},
{'name': 'Eve', 'department': 'Engineering', 'salary': 65000, 'years': 1}
]
# Filter high earners
high_earners = list(filter(lambda emp: emp['salary'] > 60000, employees))
print(f"High earners (>$60k): {[emp['name'] for emp in high_earners]}")
# Filter by department
engineers = list(filter(lambda emp: emp['department'] == 'Engineering', employees))
print(f"Engineers: {[emp['name'] for emp in engineers]}")
# Complex multi-condition filtering
senior_engineers = list(filter(
lambda emp: emp['department'] == 'Engineering' and emp['years'] >= 3,
employees
))
print(f"Senior engineers: {[emp['name'] for emp in senior_engineers]}")
# Filter using multiple criteria with ranges
mid_level_employees = list(filter(
lambda emp: 50000 <= emp['salary'] <= 70000 and emp['years'] >= 2,
employees
))
print(f"Mid-level employees: {[emp['name'] for emp in mid_level_employees]}")
# 4. Filtering with None values
print(f"\n4๏ธโฃ FILTERING NULL VALUES:")
data_with_nones = [1, 2, None, 4, None, 6, 7, None, 9]
# Remove None values
clean_data = list(filter(None, data_with_nones)) # Built-in None filter
print(f"With Nones: {data_with_nones}")
print(f"Clean data: {clean_data}")
# Explicit None filtering
explicit_clean = list(filter(lambda x: x is not None, data_with_nones))
print(f"Explicit clean: {explicit_clean}")
# Filter empty strings and None
mixed_data = ['hello', '', 'world', None, 'python', '', None]
filtered_data = list(filter(lambda x: x and x.strip(), mixed_data))
print(f"Mixed data: {mixed_data}")
print(f"Filtered (no empty/None): {filtered_data}")
# 5. Advanced filtering patterns
print(f"\n5๏ธโฃ ADVANCED PATTERNS:")
# Filter with enumerate for index-based conditions
numbers = [10, 15, 20, 25, 30, 35, 40]
even_indices = list(filter(
lambda item: item[0] % 2 == 0, # Even index
enumerate(numbers)
))
print(f"Even index items: {even_indices}")
even_index_values = [value for index, value in even_indices]
print(f"Values at even indices: {even_index_values}")
# Filter duplicates (keep first occurrence)
numbers_with_dups = [1, 2, 3, 2, 4, 1, 5, 3, 6]
seen = set()
unique_numbers = list(filter(
lambda x: not (x in seen or seen.add(x)),
numbers_with_dups
))
print(f"With duplicates: {numbers_with_dups}")
print(f"Unique (first occurrence): {unique_numbers}")
# Filter by comparing with other sequences
list1 = [1, 2, 3, 4, 5]
list2 = [2, 4, 6, 8, 10]
common_elements = list(filter(lambda x: x in list2, list1))
print(f"Common elements: {common_elements}")
# 6. Custom filter functions
print(f"\n6๏ธโฃ CUSTOM FILTER FUNCTIONS:")
def is_prime(n):
"""Check if a number is prime"""
if n < 2:
return False
for i in range(2, int(n ** 0.5) + 1):
if n % i == 0:
return False
return True
def is_palindrome(s):
"""Check if a string is a palindrome"""
s = s.lower().replace(' ', '')
return s == s[::-1]
def is_valid_email(email):
"""Simple email validation"""
return '@' in email and '.' in email.split('@')[1]
# Test custom filters
numbers_range = list(range(1, 21))
primes = list(filter(is_prime, numbers_range))
print(f"Prime numbers 1-20: {primes}")
test_words = ['radar', 'hello', 'level', 'python', 'madam', 'world']
palindromes = list(filter(is_palindrome, test_words))
print(f"Palindromes: {palindromes}")
email_list = ['user@example.com', 'invalid-email', 'test@domain.org', 'bad@', '@bad.com']
valid_emails = list(filter(is_valid_email, email_list))
print(f"Valid emails: {valid_emails}")
# 7. Performance considerations
print(f"\n7๏ธโฃ PERFORMANCE & ALTERNATIVES:")
# Filter vs list comprehension
large_numbers = list(range(100000))
# Using filter
import time
start_time = time.time()
filtered_result = list(filter(lambda x: x % 2 == 0, large_numbers))
filter_time = time.time() - start_time
# Using list comprehension
start_time = time.time()
list_comp_result = [x for x in large_numbers if x % 2 == 0]
list_comp_time = time.time() - start_time
print(f"Filter time: {filter_time:.4f} seconds")
print(f"List comprehension time: {list_comp_time:.4f} seconds")
print(f"Results equal: {filtered_result == list_comp_result}")
# Generator version (memory efficient)
filtered_generator = filter(lambda x: x % 2 == 0, large_numbers)
print(f"Filter returns iterator: {type(filtered_generator)}")
print("Memory efficient - processes on demand")Python๐ Reduce: Combine All Elements
"""
๐ REDUCE: Combine all elements into a single value
Aggregate data through accumulation
"""
from functools import reduce
import operator
print("๐ REDUCE FUNCTION MASTERY")
print("=" * 30)
# 1. Basic reduction examples
print("1๏ธโฃ BASIC REDUCTIONS:")
numbers = [1, 2, 3, 4, 5]
# Sum all numbers
total = reduce(lambda acc, x: acc + x, numbers)
print(f"Numbers: {numbers}")
print(f"Sum: {total}")
# Product of all numbers
product = reduce(lambda acc, x: acc * x, numbers)
print(f"Product: {product}")
# Find maximum
maximum = reduce(lambda acc, x: acc if acc > x else x, numbers)
print(f"Maximum: {maximum}")
# Find minimum
minimum = reduce(lambda acc, x: acc if acc < x else x, numbers)
print(f"Minimum: {minimum}")
# 2. String reductions
print(f"\n2๏ธโฃ STRING REDUCTIONS:")
words = ['Functional', 'Programming', 'Is', 'Awesome']
# Concatenate all words
sentence = reduce(lambda acc, word: acc + ' ' + word, words)
print(f"Words: {words}")
print(f"Sentence: {sentence}")
# Concatenate with initial value
sentence_with_prefix = reduce(lambda acc, word: acc + ' ' + word, words, 'Learning:')
print(f"With prefix: {sentence_with_prefix}")
# Find longest word
longest = reduce(lambda acc, word: acc if len(acc) > len(word) else word, words)
print(f"Longest word: {longest}")
# Count total characters
total_chars = reduce(lambda acc, word: acc + len(word), words, 0)
print(f"Total characters: {total_chars}")
# 3. Using operator module for efficiency
print(f"\n3๏ธโฃ OPERATOR MODULE:")
# Sum using operator.add
sum_op = reduce(operator.add, numbers)
print(f"Sum with operator.add: {sum_op}")
# Product using operator.mul
product_op = reduce(operator.mul, numbers)
print(f"Product with operator.mul: {product_op}")
# String concatenation with operator.add
concat_op = reduce(operator.add, words)
print(f"Concatenation with operator.add: {concat_op}")
# Comparison operations
max_op = reduce(lambda a, b: a if a > b else b, numbers) # max
print(f"Max comparison: {max_op}")
# 4. Complex data reductions
print(f"\n4๏ธโฃ COMPLEX DATA REDUCTIONS:")
transactions = [
{'type': 'credit', 'amount': 100},
{'type': 'debit', 'amount': 50},
{'type': 'credit', 'amount': 200},
{'type': 'debit', 'amount': 30},
{'type': 'credit', 'amount': 150}
]
# Calculate total balance
balance = reduce(
lambda acc, trans: acc + trans['amount'] if trans['type'] == 'credit'
else acc - trans['amount'],
transactions,
0
)
print(f"Total balance: ${balance}")
# Group transactions by type
grouped = reduce(
lambda acc, trans: {
**acc,
trans['type']: acc.get(trans['type'], []) + [trans['amount']]
},
transactions,
{}
)
print(f"Grouped transactions: {grouped}")
# Find transaction with maximum amount
max_transaction = reduce(
lambda acc, trans: acc if acc['amount'] > trans['amount'] else trans,
transactions
)
print(f"Largest transaction: {max_transaction}")
# 5. Advanced reduction patterns
print(f"\n5๏ธโฃ ADVANCED PATTERNS:")
# Factorial using reduce
def factorial_reduce(n):
return reduce(operator.mul, range(1, n + 1), 1)
print(f"Factorial of 5: {factorial_reduce(5)}")
print(f"Factorial of 0: {factorial_reduce(0)}")
# Flatten nested lists
nested_lists = [[1, 2], [3, 4, 5], [6], [7, 8, 9]]
flattened = reduce(lambda acc, lst: acc + lst, nested_lists, [])
print(f"Nested: {nested_lists}")
print(f"Flattened: {flattened}")
# Build frequency counter
text = "hello world"
char_count = reduce(
lambda acc, char: {**acc, char: acc.get(char, 0) + 1},
text,
{}
)
print(f"Character frequencies: {char_count}")
# Compose functions using reduce
functions = [
lambda x: x + 1,
lambda x: x * 2,
lambda x: x ** 2
]
def compose_functions(funcs):
return lambda x: reduce(lambda acc, func: func(acc), funcs, x)
composed = compose_functions(functions)
result = composed(3) # ((3 + 1) * 2) ** 2 = (8) ** 2 = 64
print(f"Function composition result: {result}")
# 6. Reduction with early termination
print(f"\n6๏ธโฃ EARLY TERMINATION PATTERNS:")
# Find first element matching condition using reduce
def find_first(predicate, iterable, default=None):
"""Find first element matching predicate using reduce"""
try:
return reduce(
lambda acc, x: x if predicate(x) and acc is None else acc,
iterable,
None
)
except:
return default
numbers = [1, 3, 5, 8, 9, 11, 12, 15]
first_even = find_first(lambda x: x % 2 == 0, numbers)
print(f"First even number: {first_even}")
# Reduce until condition met
def reduce_until(func, condition, iterable, initial=None):
"""Reduce until condition is met"""
result = initial
for item in iterable:
if condition(result):
break
result = func(result, item)
return result
# Sum until we exceed 10
partial_sum = reduce_until(
lambda acc, x: acc + x,
lambda acc: acc > 10,
[1, 2, 3, 4, 5, 6, 7, 8],
0
)
print(f"Sum until > 10: {partial_sum}")
# 7. Performance and alternatives
print(f"\n7๏ธโฃ PERFORMANCE & ALTERNATIVES:")
# Reduce vs built-in functions
large_numbers = list(range(10000))
# Sum using reduce
start_time = time.time()
reduce_sum = reduce(operator.add, large_numbers)
reduce_time = time.time() - start_time
# Sum using built-in
start_time = time.time()
builtin_sum = sum(large_numbers)
builtin_time = time.time() - start_time
print(f"Reduce sum time: {reduce_time:.4f} seconds")
print(f"Built-in sum time: {builtin_time:.4f} seconds")
print(f"Results equal: {reduce_sum == builtin_sum}")
# When to use reduce vs alternatives
print("\n๐ก WHEN TO USE REDUCE:")
print("โ
Use reduce when:")
print(" - No built-in alternative exists")
print(" - Complex accumulation logic")
print(" - Custom reduction operations")
print("โ Avoid reduce for:")
print(" - Simple sum (use sum())")
print(" - Simple max/min (use max()/min())")
print(" - String joining (use ''.join())")Python๐ Combining Map, Filter, and Reduce
"""
๐ COMBINING MAP, FILTER, AND REDUCE
The power of functional composition
"""
from functools import reduce
import json
print("๐ MAP, FILTER, REDUCE COMPOSITION")
print("=" * 40)
# 1. Classic data processing pipeline
print("1๏ธโฃ DATA PROCESSING PIPELINE:")
# Sample data: sales transactions
sales_data = [
{'product': 'laptop', 'price': 1200, 'quantity': 2, 'category': 'electronics'},
{'product': 'mouse', 'price': 25, 'quantity': 5, 'category': 'electronics'},
{'product': 'book', 'price': 15, 'quantity': 3, 'category': 'education'},
{'product': 'phone', 'price': 800, 'quantity': 1, 'category': 'electronics'},
{'product': 'pen', 'price': 2, 'quantity': 10, 'category': 'education'},
{'product': 'tablet', 'price': 400, 'quantity': 1, 'category': 'electronics'}
]
print("Sample sales data:")
for item in sales_data[:3]: # Show first 3 items
print(f" {item}")
# Step 1: MAP - Calculate total value for each transaction
with_totals = list(map(
lambda item: {**item, 'total_value': item['price'] * item['quantity']},
sales_data
))
# Step 2: FILTER - Keep only electronics with value > 50
electronics_valuable = list(filter(
lambda item: item['category'] == 'electronics' and item['total_value'] > 50,
with_totals
))
# Step 3: REDUCE - Calculate total revenue
total_revenue = reduce(
lambda acc, item: acc + item['total_value'],
electronics_valuable,
0
)
print(f"\nElectronics items > $50:")
for item in electronics_valuable:
print(f" {item['product']}: ${item['total_value']}")
print(f"Total electronics revenue (>$50): ${total_revenue}")
# 2. Text processing pipeline
print(f"\n2๏ธโฃ TEXT PROCESSING PIPELINE:")
text_data = [
"Python is awesome for functional programming",
"Map, filter, and reduce are powerful tools",
"Functional programming promotes clean code",
"Lambda functions make code concise",
"Immutability prevents many bugs"
]
# Pipeline: Extract words -> Filter long words -> Count total length
def process_text_data(texts):
# Step 1: MAP - Split into words and flatten
all_words = reduce(
lambda acc, sentence: acc + sentence.lower().split(),
texts,
[]
)
# Step 2: FILTER - Keep words longer than 4 characters
long_words = list(filter(lambda word: len(word) > 4, all_words))
# Step 3: MAP - Get word lengths
word_lengths = list(map(len, long_words))
# Step 4: REDUCE - Calculate average length
if word_lengths:
avg_length = reduce(lambda acc, length: acc + length, word_lengths) / len(word_lengths)
else:
avg_length = 0
return {
'total_words': len(all_words),
'long_words': len(long_words),
'long_word_list': long_words[:5], # First 5 long words
'average_long_word_length': round(avg_length, 2)
}
text_stats = process_text_data(text_data)
print("Text processing results:")
print(json.dumps(text_stats, indent=2))
# 3. Complex data analysis pipeline
print(f"\n3๏ธโฃ COMPLEX ANALYSIS PIPELINE:")
# Student grade data
students = [
{'name': 'Alice', 'grades': [85, 92, 78, 95], 'major': 'CS'},
{'name': 'Bob', 'grades': [72, 85, 90, 88], 'major': 'Math'},
{'name': 'Charlie', 'grades': [95, 98, 92, 96], 'major': 'CS'},
{'name': 'Diana', 'grades': [68, 75, 82, 79], 'major': 'Physics'},
{'name': 'Eve', 'grades': [88, 91, 85, 93], 'major': 'CS'},
]
def analyze_student_data(students):
# Step 1: MAP - Calculate average grade for each student
with_averages = list(map(
lambda student: {
**student,
'average': sum(student['grades']) / len(student['grades'])
},
students
))
# Step 2: FILTER - Keep CS majors with average >= 85
top_cs_students = list(filter(
lambda student: student['major'] == 'CS' and student['average'] >= 85,
with_averages
))
# Step 3: MAP - Extract names and averages
cs_performance = list(map(
lambda student: (student['name'], round(student['average'], 2)),
top_cs_students
))
# Step 4: REDUCE - Find highest performing CS student
if cs_performance:
top_performer = reduce(
lambda best, current: current if current[1] > best[1] else best,
cs_performance
)
else:
top_performer = None
return {
'top_cs_students': cs_performance,
'top_performer': top_performer,
'count': len(cs_performance)
}
analysis_results = analyze_student_data(students)
print("Student analysis results:")
print(json.dumps(analysis_results, indent=2))
# 4. One-liner functional pipelines
print(f"\n4๏ธโฃ ONE-LINER PIPELINES:")
# Calculate sum of squares of even numbers
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
sum_of_even_squares = reduce(
lambda acc, x: acc + x,
map(lambda x: x ** 2, filter(lambda x: x % 2 == 0, numbers)),
0
)
print(f"Sum of squares of even numbers: {sum_of_even_squares}")
# Count vowels in words longer than 5 characters
words = ['python', 'programming', 'functional', 'code', 'awesome', 'map', 'filter', 'reduce']
vowel_count = reduce(
lambda acc, count: acc + count,
map(
lambda word: sum(1 for char in word.lower() if char in 'aeiou'),
filter(lambda word: len(word) > 5, words)
),
0
)
print(f"Vowels in long words: {vowel_count}")
# Find the length of the longest word that starts with 'p'
p_words_max_length = reduce(
lambda acc, word: max(acc, len(word)),
filter(lambda word: word.startswith('p'), words),
0
)
print(f"Longest 'p' word length: {p_words_max_length}")
# 5. Building a functional data processor
print(f"\n5๏ธโฃ FUNCTIONAL DATA PROCESSOR:")
def create_data_processor():
"""Factory function that returns a configurable data processor"""
def process(data, transforms=None, filters=None, aggregations=None):
"""Process data through functional pipeline"""
result = data
# Apply transformations (map operations)
if transforms:
for transform in transforms:
result = list(map(transform, result))
print(f" After transform: {len(result)} items")
# Apply filters
if filters:
for filter_func in filters:
result = list(filter(filter_func, result))
print(f" After filter: {len(result)} items")
# Apply aggregations (reduce operations)
if aggregations:
for agg_func in aggregations:
result = agg_func(result)
print(f" After aggregation: {result}")
break # Only one aggregation for final result
return result
return process
# Create processor and use it
processor = create_data_processor()
# Process sales data
print("Processing sales data:")
final_result = processor(
sales_data,
transforms=[
lambda item: {**item, 'revenue': item['price'] * item['quantity']},
lambda item: {**item, 'high_value': item['revenue'] > 100}
],
filters=[
lambda item: item['category'] == 'electronics',
lambda item: item['high_value']
],
aggregations=[
lambda items: {
'total_revenue': sum(item['revenue'] for item in items),
'average_price': sum(item['price'] for item in items) / len(items) if items else 0,
'product_count': len(items)
}
]
)
print(f"Final result: {json.dumps(final_result, indent=2)}")Python๐ฏ Real-World Applications
"""
๐ฏ REAL-WORLD APPLICATIONS
Practical examples using map, filter, reduce
"""
import csv
from io import StringIO
from datetime import datetime, timedelta
import re
print("๐ฏ REAL-WORLD APPLICATIONS")
print("=" * 35)
# 1. Log file analysis
print("1๏ธโฃ LOG FILE ANALYSIS:")
# Simulated log entries
log_data = [
"2024-01-15 10:30:15 INFO User login successful for user123",
"2024-01-15 10:31:22 ERROR Database connection failed",
"2024-01-15 10:32:10 INFO User logout for user123",
"2024-01-15 10:33:05 WARN High memory usage detected",
"2024-01-15 10:34:18 ERROR Invalid API key provided",
"2024-01-15 10:35:30 INFO User login successful for user456",
"2024-01-15 10:36:45 ERROR File not found: /tmp/data.txt"
]
def analyze_logs(logs):
# Step 1: MAP - Parse log entries
parsed_logs = list(map(
lambda log: {
'timestamp': log[:19],
'level': log[20:].split()[0],
'message': ' '.join(log[20:].split()[1:])
},
logs
))
# Step 2: FILTER - Get only ERROR logs
error_logs = list(filter(
lambda log: log['level'] == 'ERROR',
parsed_logs
))
# Step 3: REDUCE - Count errors by type
error_summary = reduce(
lambda acc, log: {
**acc,
'total_errors': acc.get('total_errors', 0) + 1,
'error_messages': acc.get('error_messages', []) + [log['message']]
},
error_logs,
{}
)
return error_summary
log_analysis = analyze_logs(log_data)
print(f"Error analysis:")
print(f" Total errors: {log_analysis['total_errors']}")
print(" Error messages:")
for msg in log_analysis['error_messages']:
print(f" - {msg}")
# 2. E-commerce data processing
print(f"\n2๏ธโฃ E-COMMERCE DATA PROCESSING:")
# Simulated order data
orders = [
{'id': 1, 'customer': 'Alice', 'items': [{'product': 'laptop', 'price': 1000, 'qty': 1}], 'status': 'completed'},
{'id': 2, 'customer': 'Bob', 'items': [{'product': 'mouse', 'price': 25, 'qty': 2}, {'product': 'keyboard', 'price': 75, 'qty': 1}], 'status': 'completed'},
{'id': 3, 'customer': 'Charlie', 'items': [{'product': 'phone', 'price': 800, 'qty': 1}], 'status': 'pending'},
{'id': 4, 'customer': 'Alice', 'items': [{'product': 'tablet', 'price': 400, 'qty': 1}, {'product': 'case', 'price': 30, 'qty': 1}], 'status': 'completed'}
]
def analyze_orders(orders):
# MAP: Calculate order totals
orders_with_totals = list(map(
lambda order: {
**order,
'total': sum(item['price'] * item['qty'] for item in order['items'])
},
orders
))
# FILTER: Get completed orders only
completed_orders = list(filter(
lambda order: order['status'] == 'completed',
orders_with_totals
))
# REDUCE: Calculate customer statistics
customer_stats = reduce(
lambda acc, order: {
**acc,
order['customer']: {
'order_count': acc.get(order['customer'], {}).get('order_count', 0) + 1,
'total_spent': acc.get(order['customer'], {}).get('total_spent', 0) + order['total']
}
},
completed_orders,
{}
)
# Find top customer
top_customer = reduce(
lambda best, customer_data: customer_data if customer_data[1]['total_spent'] > best[1]['total_spent'] else best,
customer_stats.items(),
('', {'total_spent': 0})
)
return {
'total_revenue': sum(order['total'] for order in completed_orders),
'customer_stats': customer_stats,
'top_customer': top_customer[0],
'top_customer_spent': top_customer[1]['total_spent']
}
ecommerce_analysis = analyze_orders(orders)
print("E-commerce analysis:")
print(json.dumps(ecommerce_analysis, indent=2))
# 3. Data cleaning pipeline
print(f"\n3๏ธโฃ DATA CLEANING PIPELINE:")
# Messy data that needs cleaning
raw_data = [
" ALICE SMITH , alice@email.com, 25 ",
"bob jones,BOB@DOMAIN.COM,30",
" Charlie Brown , charlie.brown@test.com , invalid_age ",
"DIANA PRINCE,diana@company.org,28",
" ,missing@email.com,35", # Missing name
"Eve Wilson,INVALID-EMAIL,29", # Invalid email
]
def clean_data(raw_entries):
"""Complete data cleaning pipeline using functional approach"""
# Step 1: MAP - Parse and clean each entry
parsed_entries = list(map(
lambda entry: {
'name': entry.split(',')[0].strip().title() if entry.split(',')[0].strip() else None,
'email': entry.split(',')[1].strip().lower() if len(entry.split(',')) > 1 else None,
'age_str': entry.split(',')[2].strip() if len(entry.split(',')) > 2 else None
},
raw_data
))
# Step 2: MAP - Convert age to integer
with_ages = list(map(
lambda entry: {
**entry,
'age': int(entry['age_str']) if entry['age_str'] and entry['age_str'].isdigit() else None
},
parsed_entries
))
# Step 3: FILTER - Remove entries with missing required fields
valid_entries = list(filter(
lambda entry: (
entry['name'] and
entry['email'] and
'@' in entry['email'] and
'.' in entry['email'].split('@')[1] and
entry['age'] is not None
),
with_ages
))
# Step 4: REDUCE - Create summary statistics
summary = reduce(
lambda acc, entry: {
'count': acc['count'] + 1,
'total_age': acc['total_age'] + entry['age'],
'domains': acc['domains'] | {entry['email'].split('@')[1]}
},
valid_entries,
{'count': 0, 'total_age': 0, 'domains': set()}
)
return {
'cleaned_data': valid_entries,
'summary': {
**summary,
'average_age': round(summary['total_age'] / summary['count'], 1) if summary['count'] > 0 else 0,
'domains': list(summary['domains'])
}
}
cleaning_result = clean_data(raw_data)
print("Data cleaning results:")
print(f"Original entries: {len(raw_data)}")
print(f"Valid entries: {len(cleaning_result['cleaned_data'])}")
print(f"Average age: {cleaning_result['summary']['average_age']}")
print(f"Email domains: {cleaning_result['summary']['domains']}")
print("\nCleaned data:")
for entry in cleaning_result['cleaned_data']:
print(f" {entry['name']} ({entry['age']}) - {entry['email']}")PythonThis completes the comprehensive Map, Filter, and Reduce section. Would you like me to continue with the next section on Function Composition and Decorators, or would you prefer me to focus on any specific part of the functional programming content?
8. Functional Error Handling
๐ก๏ธ Functional Approach to Error Handling
Traditional error handling relies on exceptions and try-catch blocks, which can break the flow of functional composition. Functional programming offers elegant alternatives that treat errors as data, allowing for composable and predictable error handling.
graph TD
A["๐ก๏ธ Functional Error Handling"] --> B["๐ฆ Maybe/Option TypesHandle Missing Values"]
A --> C["โ ๏ธ Either/Result TypesSuccess or Error"]
A --> D["๐ Railway ProgrammingHappy/Error Paths"]
B --> B1["Some(value)"]
B --> B2["None/Nothing"]
B --> B3["Eliminates None checks"]
C --> C1["Success(result)"]
C --> C2["Error(message)"]
C --> C3["Composable operations"]
D --> D1["Chain operations safely"]
D --> D2["Automatic error propagation"]
D --> D3["Clean success path"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐ฆ Maybe/Option Pattern
"""
๐ฆ MAYBE/OPTION PATTERN
Handle missing values without None checks
"""
from typing import TypeVar, Generic, Optional, Callable, Union
from abc import ABC, abstractmethod
T = TypeVar('T')
U = TypeVar('U')
class Maybe(Generic[T], ABC):
"""Abstract base class for Maybe type"""
@abstractmethod
def map(self, func: Callable[[T], U]) -> 'Maybe[U]':
"""Apply function if value exists"""
pass
@abstractmethod
def flat_map(self, func: Callable[[T], 'Maybe[U]']) -> 'Maybe[U]':
"""Apply function that returns Maybe"""
pass
@abstractmethod
def filter(self, predicate: Callable[[T], bool]) -> 'Maybe[T]':
"""Filter value based on predicate"""
pass
@abstractmethod
def get_or_else(self, default: T) -> T:
"""Get value or return default"""
pass
@abstractmethod
def is_some(self) -> bool:
"""Check if value exists"""
pass
class Some(Maybe[T]):
"""Contains a value"""
def __init__(self, value: T):
if value is None:
raise ValueError("Some cannot contain None")
self._value = value
def map(self, func: Callable[[T], U]) -> Maybe[U]:
"""Apply function to contained value"""
try:
result = func(self._value)
return Some(result) if result is not None else Nothing()
except Exception as e:
return Nothing()
def flat_map(self, func: Callable[[T], Maybe[U]]) -> Maybe[U]:
"""Apply function that returns Maybe"""
try:
return func(self._value)
except Exception as e:
return Nothing()
def filter(self, predicate: Callable[[T], bool]) -> Maybe[T]:
"""Filter based on predicate"""
try:
return self if predicate(self._value) else Nothing()
except Exception as e:
return Nothing()
def get_or_else(self, default: T) -> T:
"""Return contained value"""
return self._value
def is_some(self) -> bool:
"""Always True for Some"""
return True
def __str__(self):
return f"Some({self._value})"
def __repr__(self):
return self.__str__()
class Nothing(Maybe[T]):
"""Contains no value"""
def map(self, func: Callable[[T], U]) -> Maybe[U]:
"""No-op for Nothing"""
return Nothing()
def flat_map(self, func: Callable[[T], Maybe[U]]) -> Maybe[U]:
"""No-op for Nothing"""
return Nothing()
def filter(self, predicate: Callable[[T], bool]) -> Maybe[T]:
"""No-op for Nothing"""
return Nothing()
def get_or_else(self, default: T) -> T:
"""Return default value"""
return default
def is_some(self) -> bool:
"""Always False for Nothing"""
return False
def __str__(self):
return "Nothing"
def __repr__(self):
return self.__str__()
def maybe(value: Optional[T]) -> Maybe[T]:
"""Create Maybe from optional value"""
return Some(value) if value is not None else Nothing()
print("๐ฆ MAYBE PATTERN EXAMPLES")
print("=" * 30)
# 1. Basic Maybe operations
print("1๏ธโฃ BASIC OPERATIONS:")
# Safe division
def safe_divide(x: float, y: float) -> Maybe[float]:
"""Division that returns Maybe instead of throwing exception"""
if y == 0:
return Nothing()
return Some(x / y)
# Chain operations safely
result1 = safe_divide(10, 2).map(lambda x: x * 2).map(lambda x: x + 1)
result2 = safe_divide(10, 0).map(lambda x: x * 2).map(lambda x: x + 1)
print(f"10 รท 2 * 2 + 1 = {result1}") # Some(11.0)
print(f"10 รท 0 * 2 + 1 = {result2}") # Nothing
# Using get_or_else for defaults
value1 = result1.get_or_else(0)
value2 = result2.get_or_else(0)
print(f"With defaults: {value1}, {value2}")
# 2. Dictionary lookup with Maybe
print(f"\n2๏ธโฃ SAFE DICTIONARY OPERATIONS:")
def safe_get(dictionary: dict, key: str) -> Maybe[any]:
"""Safe dictionary lookup"""
return maybe(dictionary.get(key))
user_data = {
'name': 'Alice',
'age': 30,
'email': 'alice@example.com'
}
# Chain dictionary lookups
user_info = (safe_get(user_data, 'name')
.map(lambda name: f"User: {name}")
.map(str.upper))
missing_info = (safe_get(user_data, 'phone')
.map(lambda phone: f"Phone: {phone}"))
print(f"User info: {user_info}")
print(f"Phone info: {missing_info}")
print(f"Phone with default: {missing_info.get_or_else('No phone provided')}")
# 3. Filtering with Maybe
print(f"\n3๏ธโฃ FILTERING OPERATIONS:")
def is_adult(age: int) -> bool:
return age >= 18
def is_senior(age: int) -> bool:
return age >= 65
# Process age with validation
ages = [25, 16, 70, None, 45]
for age in ages:
age_maybe = maybe(age)
adult_status = age_maybe.filter(is_adult).map(lambda x: "Adult")
senior_status = age_maybe.filter(is_senior).map(lambda x: "Senior")
print(f"Age {age}: Adult={adult_status.get_or_else('Not adult')}, Senior={senior_status.get_or_else('Not senior')}")
# 4. Complex data processing with Maybe
print(f"\n4๏ธโฃ COMPLEX DATA PROCESSING:")
def process_user_data(data: dict) -> Maybe[str]:
"""Process user data safely"""
return (safe_get(data, 'user')
.flat_map(lambda user: safe_get(user, 'profile'))
.flat_map(lambda profile: safe_get(profile, 'email'))
.filter(lambda email: '@' in email)
.map(lambda email: f"Valid email: {email}"))
# Test data with different completeness
test_data = [
{
'user': {
'profile': {
'email': 'valid@example.com'
}
}
},
{
'user': {
'profile': {
'email': 'invalid-email'
}
}
},
{
'user': {
'profile': {}
}
},
{}
]
for i, data in enumerate(test_data):
result = process_user_data(data)
print(f"Data {i+1}: {result.get_or_else('Invalid or missing data')}")Pythonโ ๏ธ Either/Result Pattern
"""
โ ๏ธ EITHER/RESULT PATTERN
Represent success or failure with detailed error information
"""
from typing import TypeVar, Generic, Callable, Union
T = TypeVar('T')
U = TypeVar('U')
E = TypeVar('E')
class Either(Generic[E, T], ABC):
"""Abstract base class for Either type"""
@abstractmethod
def map(self, func: Callable[[T], U]) -> 'Either[E, U]':
"""Transform success value"""
pass
@abstractmethod
def map_error(self, func: Callable[[E], U]) -> 'Either[U, T]':
"""Transform error value"""
pass
@abstractmethod
def flat_map(self, func: Callable[[T], 'Either[E, U]']) -> 'Either[E, U]':
"""Chain operations that return Either"""
pass
@abstractmethod
def is_success(self) -> bool:
"""Check if this is a success"""
pass
@abstractmethod
def get_or_else(self, default: T) -> T:
"""Get success value or default"""
pass
class Success(Either[E, T]):
"""Represents a successful result"""
def __init__(self, value: T):
self._value = value
def map(self, func: Callable[[T], U]) -> Either[E, U]:
"""Apply function to success value"""
try:
return Success(func(self._value))
except Exception as e:
return Error(str(e))
def map_error(self, func: Callable[[E], U]) -> Either[U, T]:
"""No-op for success"""
return Success(self._value)
def flat_map(self, func: Callable[[T], Either[E, U]]) -> Either[E, U]:
"""Chain operations"""
try:
return func(self._value)
except Exception as e:
return Error(str(e))
def is_success(self) -> bool:
return True
def get_or_else(self, default: T) -> T:
return self._value
def __str__(self):
return f"Success({self._value})"
def __repr__(self):
return self.__str__()
class Error(Either[E, T]):
"""Represents an error result"""
def __init__(self, error: E):
self._error = error
def map(self, func: Callable[[T], U]) -> Either[E, U]:
"""No-op for error"""
return Error(self._error)
def map_error(self, func: Callable[[E], U]) -> Either[U, T]:
"""Transform error value"""
try:
return Error(func(self._error))
except Exception as e:
return Error(str(e))
def flat_map(self, func: Callable[[T], Either[E, U]]) -> Either[E, U]:
"""No-op for error"""
return Error(self._error)
def is_success(self) -> bool:
return False
def get_or_else(self, default: T) -> T:
return default
def get_error(self) -> E:
"""Get error value"""
return self._error
def __str__(self):
return f"Error({self._error})"
def __repr__(self):
return self.__str__()
print("\nโ ๏ธ EITHER/RESULT PATTERN EXAMPLES")
print("=" * 40)
# 1. Safe arithmetic operations
print("1๏ธโฃ SAFE ARITHMETIC:")
def safe_divide_either(x: float, y: float) -> Either[str, float]:
"""Division that returns Either"""
if y == 0:
return Error("Division by zero")
return Success(x / y)
def safe_sqrt(x: float) -> Either[str, float]:
"""Square root that returns Either"""
if x < 0:
return Error("Cannot take square root of negative number")
return Success(x ** 0.5)
def safe_log(x: float) -> Either[str, float]:
"""Logarithm that returns Either"""
if x <= 0:
return Error("Cannot take logarithm of non-positive number")
import math
return Success(math.log(x))
# Chain operations with error handling
def complex_calculation(x: float, y: float) -> Either[str, float]:
"""Complex calculation: sqrt(x/y) then log"""
return (safe_divide_either(x, y)
.flat_map(safe_sqrt)
.flat_map(safe_log))
# Test calculations
test_cases = [
(16, 4), # Should work: sqrt(16/4) = sqrt(4) = 2, log(2) โ 0.693
(16, 0), # Division by zero
(-16, 4), # Negative square root
(0, 4), # Log of zero
]
for x, y in test_cases:
result = complex_calculation(x, y)
if result.is_success():
print(f"โ
{x}/{y} โ sqrt โ log = {result.get_or_else(0):.3f}")
else:
print(f"โ {x}/{y}: {result.get_error()}")
# 2. File processing with Either
print(f"\n2๏ธโฃ FILE PROCESSING:")
def read_file_lines(filename: str) -> Either[str, list]:
"""Simulate reading file lines"""
# Simulate different file scenarios
if filename == "missing.txt":
return Error("File not found")
elif filename == "empty.txt":
return Success([])
elif filename == "data.txt":
return Success(["line1", "line2", "line3", ""])
else:
return Error("Permission denied")
def filter_empty_lines(lines: list) -> Either[str, list]:
"""Filter out empty lines"""
filtered = [line for line in lines if line.strip()]
if not filtered:
return Error("No content after filtering empty lines")
return Success(filtered)
def process_lines(lines: list) -> Either[str, dict]:
"""Process lines into statistics"""
if not lines:
return Error("No lines to process")
stats = {
'total_lines': len(lines),
'total_chars': sum(len(line) for line in lines),
'avg_length': sum(len(line) for line in lines) / len(lines)
}
return Success(stats)
# File processing pipeline
def process_file(filename: str) -> Either[str, dict]:
"""Complete file processing pipeline"""
return (read_file_lines(filename)
.flat_map(filter_empty_lines)
.flat_map(process_lines))
# Test file processing
test_files = ["data.txt", "missing.txt", "empty.txt", "protected.txt"]
for filename in test_files:
result = process_file(filename)
if result.is_success():
stats = result.get_or_else({})
print(f"โ
{filename}: {stats}")
else:
print(f"โ {filename}: {result.get_error()}")
# 3. User validation pipeline
print(f"\n3๏ธโฃ USER VALIDATION:")
def validate_email(email: str) -> Either[str, str]:
"""Validate email format"""
if '@' not in email:
return Error("Email must contain @")
if '.' not in email.split('@')[1]:
return Error("Email domain must contain .")
return Success(email)
def validate_age(age: int) -> Either[str, int]:
"""Validate age range"""
if age < 0:
return Error("Age cannot be negative")
if age > 150:
return Error("Age cannot exceed 150")
return Success(age)
def validate_name(name: str) -> Either[str, str]:
"""Validate name format"""
if not name.strip():
return Error("Name cannot be empty")
if len(name) < 2:
return Error("Name must be at least 2 characters")
return Success(name.strip())
def create_user(name: str, email: str, age: int) -> Either[str, dict]:
"""Create user with validation"""
# Validate each field
name_result = validate_name(name)
email_result = validate_email(email)
age_result = validate_age(age)
# Combine results (all must succeed)
if not name_result.is_success():
return name_result.map_error(lambda err: f"Name error: {err}")
if not email_result.is_success():
return email_result.map_error(lambda err: f"Email error: {err}")
if not age_result.is_success():
return age_result.map_error(lambda err: f"Age error: {err}")
# All validations passed
return Success({
'name': name_result.get_or_else(""),
'email': email_result.get_or_else(""),
'age': age_result.get_or_else(0),
'created_at': '2024-01-15 10:30:00'
})
# Test user creation
test_users = [
("Alice Smith", "alice@example.com", 25),
("", "bob@test.com", 30),
("Charlie", "invalid-email", 35),
("Diana", "diana@company.org", -5),
("Eve", "eve@domain.com", 200),
]
for name, email, age in test_users:
result = create_user(name, email, age)
if result.is_success():
user = result.get_or_else({})
print(f"โ
Created user: {user['name']} ({user['email']})")
else:
print(f"โ Failed to create user: {result.get_error()}")Python๐ Railway Programming Pattern
"""
๐ RAILWAY PROGRAMMING PATTERN
Chain operations with automatic error handling
"""
from typing import Callable, Any
from functools import wraps
class Railway:
"""Railway programming implementation"""
def __init__(self, value: Any, is_success: bool = True, error: str = ""):
self.value = value
self.is_success = is_success
self.error = error
def bind(self, func: Callable) -> 'Railway':
"""Chain operations (railway bind)"""
if not self.is_success:
return self # Stay on error track
try:
result = func(self.value)
if isinstance(result, Railway):
return result
else:
return Railway(result, True)
except Exception as e:
return Railway(None, False, str(e))
def map(self, func: Callable) -> 'Railway':
"""Transform value if on success track"""
return self.bind(lambda x: func(x))
def tee(self, func: Callable) -> 'Railway':
"""Execute side effect without changing value"""
if self.is_success:
try:
func(self.value)
except Exception as e:
return Railway(None, False, str(e))
return self
@classmethod
def success(cls, value: Any) -> 'Railway':
"""Create successful railway"""
return cls(value, True)
@classmethod
def failure(cls, error: str) -> 'Railway':
"""Create failed railway"""
return cls(None, False, error)
def __str__(self):
if self.is_success:
return f"Success({self.value})"
else:
return f"Failure({self.error})"
def railway_function(func: Callable) -> Callable:
"""Decorator to convert regular functions to railway functions"""
@wraps(func)
def wrapper(*args, **kwargs):
try:
result = func(*args, **kwargs)
return Railway.success(result)
except Exception as e:
return Railway.failure(str(e))
return wrapper
print("\n๐ RAILWAY PROGRAMMING EXAMPLES")
print("=" * 35)
# 1. Basic railway operations
print("1๏ธโฃ BASIC RAILWAY OPERATIONS:")
@railway_function
def add_ten(x):
return x + 10
@railway_function
def multiply_by_two(x):
return x * 2
@railway_function
def divide_by_three(x):
if x == 0:
raise ValueError("Cannot divide zero by three meaningfully in this context")
return x / 3
# Chain operations
result1 = (Railway.success(5)
.bind(add_ten) # 5 + 10 = 15
.bind(multiply_by_two) # 15 * 2 = 30
.bind(divide_by_three)) # 30 / 3 = 10
result2 = (Railway.success(0)
.bind(add_ten) # 0 + 10 = 10
.bind(multiply_by_two) # 10 * 2 = 20
.bind(divide_by_three)) # 20 / 3 = 6.67
print(f"Chain 1 (start with 5): {result1}")
print(f"Chain 2 (start with 0): {result2}")
# 2. User registration pipeline
print(f"\n2๏ธโฃ USER REGISTRATION PIPELINE:")
@railway_function
def validate_username(username):
if not username or len(username) < 3:
raise ValueError("Username must be at least 3 characters")
if not username.isalnum():
raise ValueError("Username must be alphanumeric")
return username
@railway_function
def validate_password(password):
if len(password) < 8:
raise ValueError("Password must be at least 8 characters")
if not any(c.isdigit() for c in password):
raise ValueError("Password must contain at least one digit")
return password
@railway_function
def hash_password(password):
# Simulate password hashing
return f"hashed_{password}"
@railway_function
def create_user_record(data):
# Simulate database insertion
if data['username'] == 'admin':
raise ValueError("Username 'admin' is reserved")
return {
'id': 12345,
'username': data['username'],
'password_hash': data['password'],
'created_at': '2024-01-15'
}
def register_user(username, password):
"""Complete user registration pipeline"""
user_data = {'username': username, 'password': password}
return (Railway.success(user_data)
.bind(lambda data: validate_username(data['username'])
.map(lambda u: {**data, 'username': u}))
.bind(lambda data: validate_password(data['password'])
.map(lambda p: {**data, 'password': p}))
.bind(lambda data: hash_password(data['password'])
.map(lambda h: {**data, 'password': h}))
.bind(create_user_record)
.tee(lambda user: print(f"๐ง Sending welcome email to user {user['id']}")))
# Test user registrations
test_registrations = [
("alice123", "password123"),
("ab", "password123"), # Username too short
("charlie", "pass"), # Password too short
("diana456", "nodigits"), # Password no digits
("admin", "validpass123"), # Reserved username
]
for username, password in test_registrations:
result = register_user(username, password)
if result.is_success:
print(f"โ
User registered: {result.value['username']}")
else:
print(f"โ Registration failed: {result.error}")
# 3. Data processing pipeline with logging
print(f"\n3๏ธโฃ DATA PROCESSING WITH LOGGING:")
@railway_function
def load_data(source):
"""Simulate loading data"""
data_sources = {
'api': [{'id': 1, 'value': 100}, {'id': 2, 'value': 200}],
'file': [{'id': 3, 'value': 300}, {'id': 4, 'value': 400}],
'db': [] # Empty data
}
if source not in data_sources:
raise ValueError(f"Unknown data source: {source}")
data = data_sources[source]
if not data:
raise ValueError(f"No data available from {source}")
return data
@railway_function
def transform_data(data):
"""Transform data format"""
return [{'id': item['id'], 'processed_value': item['value'] * 1.1} for item in data]
@railway_function
def validate_data(data):
"""Validate transformed data"""
if not all(item['processed_value'] > 0 for item in data):
raise ValueError("All processed values must be positive")
return data
@railway_function
def save_data(data):
"""Simulate saving data"""
# Simulate saving to database
return f"Saved {len(data)} records to database"
def process_data_pipeline(source):
"""Complete data processing pipeline with logging"""
return (Railway.success(source)
.tee(lambda s: print(f"๐ Starting processing from {s}"))
.bind(load_data)
.tee(lambda d: print(f"๐ฅ Loaded {len(d)} records"))
.bind(transform_data)
.tee(lambda d: print(f"๐ง Transformed {len(d)} records"))
.bind(validate_data)
.tee(lambda d: print(f"โ
Validated {len(d)} records"))
.bind(save_data)
.tee(lambda r: print(f"๐พ {r}")))
# Test data processing
data_sources = ['api', 'file', 'db', 'unknown']
for source in data_sources:
print(f"\n--- Processing from {source} ---")
result = process_data_pipeline(source)
if result.is_success:
print(f"๐ Pipeline completed: {result.value}")
else:
print(f"๐จ Pipeline failed: {result.error}")
# 4. Configuration loading with fallbacks
print(f"\n4๏ธโฃ CONFIGURATION WITH FALLBACKS:")
def try_load_config(source):
"""Try to load configuration from a source"""
configs = {
'environment': {'db_host': 'prod.db.com', 'debug': False},
'file': {'db_host': 'dev.db.com', 'debug': True},
'defaults': {'db_host': 'localhost', 'debug': True, 'port': 5432}
}
if source in configs:
return Railway.success(configs[source])
else:
return Railway.failure(f"Config source {source} not available")
def load_config_with_fallbacks():
"""Load configuration with multiple fallback sources"""
# Try sources in order of preference
config_sources = ['environment', 'file', 'defaults']
for source in config_sources:
result = try_load_config(source)
if result.is_success:
return result.tee(lambda cfg: print(f"๐ Loaded config from {source}: {cfg}"))
return Railway.failure("All configuration sources failed")
# Test configuration loading
config_result = load_config_with_fallbacks()
if config_result.is_success:
print(f"โ
Final configuration: {config_result.value}")
else:
print(f"โ Configuration loading failed: {config_result.error}")Python๐ ๏ธ Practical Error Handling Patterns
"""
๐ ๏ธ PRACTICAL ERROR HANDLING PATTERNS
Real-world applications of functional error handling
"""
import json
from typing import List, Dict, Any
from datetime import datetime
import re
print("\n๐ ๏ธ PRACTICAL ERROR HANDLING")
print("=" * 35)
# 1. API Response Processing
print("1๏ธโฃ API RESPONSE PROCESSING:")
def parse_json_response(response_text: str) -> Either[str, dict]:
"""Parse JSON response safely"""
try:
return Success(json.loads(response_text))
except json.JSONDecodeError as e:
return Error(f"Invalid JSON: {str(e)}")
def validate_api_response(data: dict) -> Either[str, dict]:
"""Validate API response structure"""
if 'status' not in data:
return Error("Missing 'status' field")
if data['status'] != 'success':
return Error(f"API error: {data.get('message', 'Unknown error')}")
if 'data' not in data:
return Error("Missing 'data' field")
return Success(data['data'])
def extract_user_info(data: dict) -> Either[str, dict]:
"""Extract user information from API response"""
required_fields = ['id', 'name', 'email']
for field in required_fields:
if field not in data:
return Error(f"Missing required field: {field}")
return Success({
'user_id': data['id'],
'full_name': data['name'],
'email_address': data['email'],
'extracted_at': datetime.now().isoformat()
})
# Test API response processing
api_responses = [
'{"status": "success", "data": {"id": 123, "name": "Alice", "email": "alice@test.com"}}',
'{"status": "error", "message": "User not found"}',
'{"status": "success", "data": {"id": 456, "name": "Bob"}}', # Missing email
'invalid json{',
]
for i, response in enumerate(api_responses, 1):
result = (parse_json_response(response)
.flat_map(validate_api_response)
.flat_map(extract_user_info))
if result.is_success():
user = result.get_or_else({})
print(f"โ
Response {i}: User {user.get('full_name')} extracted")
else:
print(f"โ Response {i}: {result.get_error()}")
# 2. Form Validation Pipeline
print(f"\n2๏ธโฃ FORM VALIDATION PIPELINE:")
class ValidationResult:
"""Accumulate multiple validation errors"""
def __init__(self, value: Any = None, errors: List[str] = None):
self.value = value
self.errors = errors or []
self.is_valid = len(self.errors) == 0
def add_error(self, error: str):
self.errors.append(error)
self.is_valid = False
return self
def combine(self, other: 'ValidationResult'):
"""Combine validation results"""
combined_errors = self.errors + other.errors
combined_value = self.value if self.is_valid else other.value
return ValidationResult(combined_value, combined_errors)
def validate_field(value: Any, validators: List[Callable]) -> ValidationResult:
"""Validate a field using multiple validators"""
result = ValidationResult(value)
for validator in validators:
try:
validator(value)
except ValueError as e:
result.add_error(str(e))
return result
def validate_required(value):
"""Validator: field is required"""
if not value or (isinstance(value, str) and not value.strip()):
raise ValueError("Field is required")
def validate_email(value):
"""Validator: valid email format"""
if not isinstance(value, str) or '@' not in value:
raise ValueError("Invalid email format")
email_pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
if not re.match(email_pattern, value):
raise ValueError("Invalid email format")
def validate_min_length(min_len):
"""Validator factory: minimum length"""
def validator(value):
if len(str(value)) < min_len:
raise ValueError(f"Must be at least {min_len} characters")
return validator
def validate_age_range(value):
"""Validator: age within valid range"""
try:
age = int(value)
if age < 0 or age > 120:
raise ValueError("Age must be between 0 and 120")
except (ValueError, TypeError):
raise ValueError("Age must be a valid number")
def validate_form(form_data: dict) -> ValidationResult:
"""Validate entire form"""
# Validate each field
name_result = validate_field(
form_data.get('name'),
[validate_required, validate_min_length(2)]
)
email_result = validate_field(
form_data.get('email'),
[validate_required, validate_email]
)
age_result = validate_field(
form_data.get('age'),
[validate_required, validate_age_range]
)
# Combine all validation results
combined_result = name_result.combine(email_result).combine(age_result)
if combined_result.is_valid:
combined_result.value = {
'name': form_data['name'].strip(),
'email': form_data['email'].lower(),
'age': int(form_data['age']),
'validated_at': datetime.now().isoformat()
}
return combined_result
# Test form validation
test_forms = [
{'name': 'Alice Smith', 'email': 'alice@example.com', 'age': '25'},
{'name': '', 'email': 'bob@test.com', 'age': '30'},
{'name': 'Charlie', 'email': 'invalid-email', 'age': '35'},
{'name': 'Diana', 'email': 'diana@company.org', 'age': 'not-a-number'},
{'name': 'A', 'email': 'eve@domain.com', 'age': '-5'},
]
for i, form in enumerate(test_forms, 1):
result = validate_form(form)
if result.is_valid:
print(f"โ
Form {i}: Valid - {result.value['name']}")
else:
print(f"โ Form {i}: Errors - {', '.join(result.errors)}")
# 3. Database operation with retry logic
print(f"\n3๏ธโฃ DATABASE OPERATIONS WITH RETRY:")
class DatabaseError(Exception):
"""Custom database exception"""
pass
class ConnectionError(DatabaseError):
"""Connection specific error"""
pass
def simulate_database_operation(operation_id: int) -> Either[str, dict]:
"""Simulate database operation that might fail"""
import random
# Simulate different failure scenarios
outcomes = {
1: Success({'id': 1, 'data': 'success'}),
2: Error('Connection timeout'),
3: Error('Record not found'),
4: Success({'id': 4, 'data': 'retrieved'}),
5: Error('Permission denied'),
}
return outcomes.get(operation_id, Error('Unknown operation'))
def retry_operation(operation: Callable, max_retries: int = 3) -> Either[str, Any]:
"""Retry operation with exponential backoff"""
import time
for attempt in range(max_retries):
result = operation()
if result.is_success():
return result
# Check if error is retryable
error_msg = result.get_error()
if 'timeout' in error_msg.lower() or 'connection' in error_msg.lower():
if attempt < max_retries - 1:
wait_time = 2 ** attempt # Exponential backoff
print(f" ๐ Retry {attempt + 1}/{max_retries} after {wait_time}s")
time.sleep(wait_time)
continue
# Non-retryable error or max retries reached
return result
return Error(f"Operation failed after {max_retries} retries")
# Test database operations with retry
operations_to_test = [1, 2, 3, 4, 5]
for op_id in operations_to_test:
print(f"Testing operation {op_id}:")
result = retry_operation(lambda: simulate_database_operation(op_id))
if result.is_success():
data = result.get_or_else({})
print(f" โ
Success: {data}")
else:
print(f" โ Failed: {result.get_error()}")
print()
print("\n๐ฏ KEY TAKEAWAYS:")
print("โ
Functional error handling treats errors as data")
print("โ
Maybe/Option types eliminate null pointer exceptions")
print("โ
Either/Result types provide detailed error information")
print("โ
Railway programming chains operations safely")
print("โ
Validation can accumulate multiple errors")
print("โ
Retry logic can be implemented functionally")PythonThis completes the comprehensive section on Functional Error Handling. The content covers:
- Maybe/Option Pattern – Handling missing values safely
- Either/Result Pattern – Representing success or failure with detailed errors
- Railway Programming – Chaining operations with automatic error propagation
- Practical Applications – Real-world examples including API processing, form validation, and database operations with retry logic
The functional approach to error handling eliminates many common bugs and makes error handling more composable and predictable than traditional exception-based approaches.
9. Function Composition and Decorators
๐ Understanding Function Composition
Function Composition is the process of combining simple functions to build more complex ones. It’s one of the core principles of functional programming, allowing you to create sophisticated behavior by connecting smaller, focused functions.
graph TD
A["๐ Function Composition"] --> B["๐งฉ Simple FunctionsSingle Responsibility"]
A --> C["โก Combine FunctionsChain Operations"]
A --> D["๐๏ธ Complex BehaviorFrom Simple Parts"]
B --> B1["f(x) = x + 1"]
B --> B2["g(x) = x * 2"]
B --> B3["h(x) = x ** 2"]
C --> C1["(f โ g)(x) = f(g(x))"]
C --> C2["Left-to-right: pipe"]
C --> C3["Right-to-left: compose"]
D --> D1["Readable pipelines"]
D --> D2["Reusable components"]
D --> D3["Testable units"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐งฉ Basic Function Composition
"""
๐งฉ BASIC FUNCTION COMPOSITION
Building complex functions from simple ones
"""
from functools import reduce
from typing import Callable, Any
print("๐ FUNCTION COMPOSITION FUNDAMENTALS")
print("=" * 40)
# 1. Simple function composition
print("1๏ธโฃ SIMPLE COMPOSITION:")
def add_one(x):
"""Add 1 to input"""
return x + 1
def double(x):
"""Double the input"""
return x * 2
def square(x):
"""Square the input"""
return x ** 2
# Manual composition
def add_double_square(x):
"""Manually composed function"""
return square(double(add_one(x)))
# Test manual composition
test_value = 3
result = add_double_square(test_value)
print(f"Manual composition f(g(h({test_value}))): {result}")
print(f"Step by step: {test_value} โ {add_one(test_value)} โ {double(add_one(test_value))} โ {result}")
# 2. Compose function (right-to-left)
print(f"\n2๏ธโฃ COMPOSE FUNCTION (RIGHT-TO-LEFT):")
def compose(*functions):
"""Compose functions right-to-left: compose(f, g, h)(x) = f(g(h(x)))"""
if not functions:
return lambda x: x
def composed_function(x):
return reduce(lambda acc, func: func(acc), reversed(functions), x)
# Add metadata
func_names = [f.__name__ for f in reversed(functions)]
composed_function.__name__ = f"compose({' โ '.join(func_names)})"
return composed_function
# Create composed function
composed = compose(square, double, add_one)
result = composed(3)
print(f"Composed function: {composed.__name__}")
print(f"Result: {result}")
# Multiple compositions
math_operations = compose(
lambda x: x - 5, # Subtract 5
lambda x: x / 2, # Divide by 2
lambda x: x + 10, # Add 10
lambda x: x * 3 # Multiply by 3
)
test_values = [1, 2, 3, 4, 5]
print(f"Complex composition results:")
for val in test_values:
result = math_operations(val)
print(f" {val} โ {result}")
# 3. Pipe function (left-to-right)
print(f"\n3๏ธโฃ PIPE FUNCTION (LEFT-TO-RIGHT):")
def pipe(*functions):
"""Pipe functions left-to-right: pipe(f, g, h)(x) = h(g(f(x)))"""
if not functions:
return lambda x: x
def piped_function(x):
return reduce(lambda acc, func: func(acc), functions, x)
# Add metadata
func_names = [f.__name__ for f in functions]
piped_function.__name__ = f"pipe({' | '.join(func_names)})"
return piped_function
# Create piped function (more intuitive order)
piped = pipe(add_one, double, square)
result = piped(3)
print(f"Piped function: {piped.__name__}")
print(f"Result: {result}")
# String processing pipeline
def remove_spaces(text):
return text.replace(" ", "")
def to_lowercase(text):
return text.lower()
def reverse_text(text):
return text[::-1]
def add_exclamation(text):
return text + "!"
text_processor = pipe(
remove_spaces,
to_lowercase,
reverse_text,
add_exclamation
)
sample_text = "Hello World Python"
processed = text_processor(sample_text)
print(f"Text processing: '{sample_text}' โ '{processed}'")
# 4. Conditional composition
print(f"\n4๏ธโฃ CONDITIONAL COMPOSITION:")
def conditional_pipe(condition, true_functions, false_functions):
"""Apply different function pipelines based on condition"""
def conditional_function(x):
if condition(x):
return pipe(*true_functions)(x)
else:
return pipe(*false_functions)(x)
return conditional_function
# Different processing for even/odd numbers
is_even = lambda x: x % 2 == 0
even_odd_processor = conditional_pipe(
is_even,
[lambda x: x // 2, lambda x: x + 100], # Even: divide by 2, add 100
[lambda x: x * 3, lambda x: x - 10] # Odd: multiply by 3, subtract 10
)
numbers = [2, 3, 8, 7, 12, 15]
print("Conditional processing:")
for num in numbers:
result = even_odd_processor(num)
number_type = "even" if is_even(num) else "odd"
print(f" {num} ({number_type}) โ {result}")
# 5. Partial composition
print(f"\n5๏ธโฃ PARTIAL COMPOSITION:")
def partial_compose(func, *args, **kwargs):
"""Create a partially applied composition"""
def partial_function(x):
return func(x, *args, **kwargs)
partial_function.__name__ = f"partial_{func.__name__}"
return partial_function
def multiply_by(x, factor):
"""Multiply x by factor"""
return x * factor
def add_number(x, number):
"""Add number to x"""
return x + number
def power_of(x, exponent):
"""Raise x to the power of exponent"""
return x ** exponent
# Create partially applied functions
multiply_by_5 = partial_compose(multiply_by, factor=5)
add_100 = partial_compose(add_number, number=100)
cube = partial_compose(power_of, exponent=3)
# Compose with partial functions
math_pipeline = pipe(multiply_by_5, add_100, cube)
test_numbers = [1, 2, 3, 4]
print("Partial composition pipeline:")
for num in test_numbers:
result = math_pipeline(num)
print(f" {num} โ ร5 โ +100 โ ^3 = {result}")Python๐จ Decorators: Composition in Action
"""
๐จ DECORATORS: Function composition as syntax
Python's decorator syntax makes function composition elegant
"""
from functools import wraps
import time
from datetime import datetime
import logging
print("\n๐จ DECORATORS: COMPOSITION SYNTAX")
print("=" * 40)
# 1. Basic decorator concepts
print("1๏ธโฃ BASIC DECORATOR CONCEPTS:")
def simple_decorator(func):
"""A simple decorator that adds behavior"""
@wraps(func)
def wrapper(*args, **kwargs):
print(f"๐ง Before calling {func.__name__}")
result = func(*args, **kwargs)
print(f"โ
After calling {func.__name__}")
return result
return wrapper
@simple_decorator
def greet(name):
"""A simple greeting function"""
return f"Hello, {name}!"
# Test decorated function
result = greet("Alice")
print(f"Result: {result}")
# 2. Timing decorator
print(f"\n2๏ธโฃ TIMING DECORATOR:")
def timer(func):
"""Decorator to measure function execution time"""
@wraps(func)
def wrapper(*args, **kwargs):
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
execution_time = end_time - start_time
print(f"โฑ๏ธ {func.__name__} executed in {execution_time:.4f} seconds")
return result
return wrapper
@timer
def slow_function(n):
"""Simulate a slow function"""
total = 0
for i in range(n):
total += i ** 2
return total
# Test timing
result = slow_function(100000)
print(f"Result: {result}")
# 3. Caching decorator
print(f"\n3๏ธโฃ CACHING DECORATOR:")
def memoize(func):
"""Simple memoization decorator"""
cache = {}
@wraps(func)
def wrapper(*args, **kwargs):
# Create cache key
key = str(args) + str(sorted(kwargs.items()))
if key in cache:
print(f"๐พ Cache hit for {func.__name__}{args}")
return cache[key]
print(f"๐ Computing {func.__name__}{args}")
result = func(*args, **kwargs)
cache[key] = result
return result
# Add cache management methods
wrapper.cache = cache
wrapper.clear_cache = lambda: cache.clear()
wrapper.cache_size = lambda: len(cache)
return wrapper
@memoize
def fibonacci(n):
"""Recursive Fibonacci with memoization"""
if n <= 1:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
# Test memoization
print("Fibonacci with memoization:")
for i in [5, 8, 5, 10, 8]: # Repeated values to show caching
result = fibonacci(i)
print(f" fib({i}) = {result}")
print(f"Cache size: {fibonacci.cache_size()}")
# 4. Validation decorator
print(f"\n4๏ธโฃ VALIDATION DECORATOR:")
def validate_types(**type_checks):
"""Decorator to validate argument types"""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
# Get function signature
import inspect
sig = inspect.signature(func)
bound_args = sig.bind(*args, **kwargs)
bound_args.apply_defaults()
# Validate types
for param_name, expected_type in type_checks.items():
if param_name in bound_args.arguments:
value = bound_args.arguments[param_name]
if not isinstance(value, expected_type):
raise TypeError(
f"{func.__name__}(): {param_name} must be {expected_type.__name__}, "
f"got {type(value).__name__}"
)
return func(*args, **kwargs)
return wrapper
return decorator
@validate_types(name=str, age=int, salary=float)
def create_employee(name, age, salary=0.0):
"""Create employee record with type validation"""
return {
'name': name,
'age': age,
'salary': salary,
'created_at': datetime.now().isoformat()
}
# Test validation
try:
employee1 = create_employee("Alice", 30, 75000.0)
print(f"โ
Valid employee: {employee1['name']}")
employee2 = create_employee("Bob", "thirty", 50000.0) # Invalid age type
except TypeError as e:
print(f"โ Validation error: {e}")
# 5. Retry decorator
print(f"\n5๏ธโฃ RETRY DECORATOR:")
def retry(max_attempts=3, delay=1.0, exceptions=(Exception,)):
"""Decorator to retry function on failure"""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
last_exception = None
for attempt in range(max_attempts):
try:
result = func(*args, **kwargs)
if attempt > 0:
print(f"โ
{func.__name__} succeeded on attempt {attempt + 1}")
return result
except exceptions as e:
last_exception = e
if attempt < max_attempts - 1:
print(f"๐ {func.__name__} failed (attempt {attempt + 1}), retrying in {delay}s...")
time.sleep(delay)
else:
print(f"โ {func.__name__} failed after {max_attempts} attempts")
raise last_exception
return wrapper
return decorator
# Simulate unreliable function
import random
@retry(max_attempts=3, delay=0.5)
def unreliable_api_call(data):
"""Simulate an API call that sometimes fails"""
if random.random() < 0.7: # 70% chance of failure
raise ConnectionError("Network timeout")
return f"Success: processed {data}"
# Test retry mechanism
try:
result = unreliable_api_call("test_data")
print(f"API result: {result}")
except Exception as e:
print(f"Final failure: {e}")
# 6. Decorator composition
print(f"\n6๏ธโฃ DECORATOR COMPOSITION:")
def log_calls(func):
"""Decorator to log function calls"""
@wraps(func)
def wrapper(*args, **kwargs):
print(f"๐ Calling {func.__name__} with args={args}, kwargs={kwargs}")
result = func(*args, **kwargs)
print(f"๐ {func.__name__} returned: {result}")
return result
return wrapper
def convert_result(conversion_func):
"""Decorator to convert function result"""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
result = func(*args, **kwargs)
converted = conversion_func(result)
print(f"๐ Converted {result} โ {converted}")
return converted
return wrapper
return decorator
# Compose multiple decorators
@log_calls
@timer
@convert_result(lambda x: x.upper())
def process_text(text, prefix="Processed: "):
"""Function with multiple decorators"""
return prefix + text
# Test composed decorators
result = process_text("hello world", prefix="Result: ")
print(f"Final result: {result}")
# 7. Class-based decorators
print(f"\n7๏ธโฃ CLASS-BASED DECORATORS:")
class CountCalls:
"""Class-based decorator to count function calls"""
def __init__(self, func):
self.func = func
self.count = 0
# Preserve function metadata
self.__name__ = func.__name__
self.__doc__ = func.__doc__
def __call__(self, *args, **kwargs):
self.count += 1
print(f"๐ Call #{self.count} to {self.func.__name__}")
return self.func(*args, **kwargs)
def reset_count(self):
"""Reset the call counter"""
self.count = 0
@CountCalls
def calculate_area(length, width):
"""Calculate rectangle area"""
return length * width
# Test class decorator
area1 = calculate_area(5, 3)
area2 = calculate_area(10, 2)
area3 = calculate_area(7, 4)
print(f"Total calls: {calculate_area.count}")
calculate_area.reset_count()
print(f"After reset: {calculate_area.count}")Python๐ญ Advanced Composition Patterns
"""
๐ญ ADVANCED COMPOSITION PATTERNS
Sophisticated ways to combine functions
"""
from typing import Callable, Any, List
import asyncio
from concurrent.futures import ThreadPoolExecutor
import operator
print("\n๐ญ ADVANCED COMPOSITION PATTERNS")
print("=" * 40)
# 1. Monadic composition (Maybe/Option pattern integration)
print("1๏ธโฃ MONADIC COMPOSITION:")
class Maybe:
"""Simple Maybe monad for safe composition"""
def __init__(self, value):
self.value = value
self.is_nothing = value is None
def map(self, func):
"""Apply function if value exists"""
if self.is_nothing:
return Maybe(None)
try:
return Maybe(func(self.value))
except Exception:
return Maybe(None)
def flat_map(self, func):
"""Apply function that returns Maybe"""
if self.is_nothing:
return Maybe(None)
try:
result = func(self.value)
return result if isinstance(result, Maybe) else Maybe(result)
except Exception:
return Maybe(None)
def get_or_else(self, default):
"""Get value or return default"""
return default if self.is_nothing else self.value
def __str__(self):
return f"Nothing" if self.is_nothing else f"Some({self.value})"
def safe_divide(x, y):
"""Safe division returning Maybe"""
return Maybe(x / y if y != 0 else None)
def safe_sqrt(x):
"""Safe square root returning Maybe"""
return Maybe(x ** 0.5 if x >= 0 else None)
def safe_log(x):
"""Safe logarithm returning Maybe"""
import math
return Maybe(math.log(x) if x > 0 else None)
# Compose Maybe operations
def safe_computation(x, y):
"""Chain safe operations using Maybe"""
return (Maybe(x)
.flat_map(lambda val: safe_divide(val, y))
.flat_map(lambda val: safe_sqrt(val.value))
.flat_map(lambda val: safe_log(val.value)))
# Test safe composition
test_cases = [(16, 4), (16, 0), (-16, 4), (0, 4)]
print("Safe computation chain (x/y โ โ โ log):")
for x, y in test_cases:
result = safe_computation(x, y)
print(f" ({x}, {y}) โ {result} = {result.get_or_else('Error')}")
# 2. Parallel composition
print(f"\n2๏ธโฃ PARALLEL COMPOSITION:")
def parallel_compose(*functions):
"""Compose functions to run in parallel"""
def parallel_function(inputs):
if len(inputs) != len(functions):
raise ValueError("Number of inputs must match number of functions")
with ThreadPoolExecutor() as executor:
futures = [
executor.submit(func, inp)
for func, inp in zip(functions, inputs)
]
results = [future.result() for future in futures]
return results
return parallel_function
# Create parallel operations
def slow_square(x):
"""Slow square operation"""
time.sleep(0.1) # Simulate work
return x ** 2
def slow_cube(x):
"""Slow cube operation"""
time.sleep(0.1) # Simulate work
return x ** 3
def slow_factorial(x):
"""Slow factorial operation"""
time.sleep(0.1) # Simulate work
import math
return math.factorial(min(x, 10)) # Limit to prevent overflow
parallel_ops = parallel_compose(slow_square, slow_cube, slow_factorial)
# Test parallel composition
start_time = time.time()
results = parallel_ops([5, 4, 6])
end_time = time.time()
print(f"Parallel results: {results}")
print(f"Execution time: {end_time - start_time:.3f}s (would be ~0.3s sequential)")
# 3. Async composition
print(f"\n3๏ธโฃ ASYNC COMPOSITION:")
async def async_pipe(*functions):
"""Async version of pipe composition"""
async def async_piped_function(x):
result = x
for func in functions:
if asyncio.iscoroutinefunction(func):
result = await func(result)
else:
result = func(result)
return result
return async_piped_function
async def async_fetch_data(x):
"""Simulate async data fetching"""
await asyncio.sleep(0.1) # Simulate network delay
return x * 2
async def async_process_data(x):
"""Simulate async data processing"""
await asyncio.sleep(0.1) # Simulate processing delay
return x + 10
def sync_finalize(x):
"""Synchronous finalization"""
return f"Final result: {x}"
# Create async pipeline
async def test_async_composition():
async_pipeline = await async_pipe(
async_fetch_data,
async_process_data,
sync_finalize
)
result = await async_pipeline(5)
print(f"Async pipeline result: {result}")
# Run async test (if in async context)
# asyncio.run(test_async_composition())
print("Async composition defined (run with asyncio.run() in async context)")
# 4. Conditional composition chain
print(f"\n4๏ธโฃ CONDITIONAL COMPOSITION CHAIN:")
class CompositionChain:
"""Fluent interface for conditional composition"""
def __init__(self, initial_value):
self.value = initial_value
self.functions = []
def then(self, func):
"""Add function to chain"""
self.functions.append(('always', func))
return self
def when(self, condition, func):
"""Add conditional function to chain"""
self.functions.append(('when', condition, func))
return self
def otherwise(self, func):
"""Add function for when previous condition was false"""
self.functions.append(('otherwise', func))
return self
def execute(self):
"""Execute the composition chain"""
result = self.value
last_condition_result = True
for operation in self.functions:
if operation[0] == 'always':
_, func = operation
result = func(result)
print(f" Applied {func.__name__}: {result}")
elif operation[0] == 'when':
_, condition, func = operation
last_condition_result = condition(result)
if last_condition_result:
result = func(result)
print(f" Applied {func.__name__} (condition met): {result}")
else:
print(f" Skipped {func.__name__} (condition not met)")
elif operation[0] == 'otherwise':
_, func = operation
if not last_condition_result:
result = func(result)
print(f" Applied {func.__name__} (otherwise): {result}")
else:
print(f" Skipped {func.__name__} (condition was met)")
return result
# Test conditional composition
def is_even(x):
return x % 2 == 0
def is_large(x):
return x > 50
print("Conditional composition chain:")
result1 = (CompositionChain(10)
.then(lambda x: x * 2)
.when(is_even, lambda x: x + 100)
.otherwise(lambda x: x - 50)
.when(is_large, lambda x: x // 10)
.execute())
print(f"Final result 1: {result1}")
result2 = (CompositionChain(7)
.then(lambda x: x * 3)
.when(is_even, lambda x: x + 100)
.otherwise(lambda x: x - 5)
.when(is_large, lambda x: x // 2)
.execute())
print(f"Final result 2: {result2}")
# 5. Compositional operators
print(f"\n5๏ธโฃ COMPOSITIONAL OPERATORS:")
class ComposableFunction:
"""Wrapper to make functions composable with operators"""
def __init__(self, func):
self.func = func
self.__name__ = getattr(func, '__name__', 'anonymous')
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
def __rshift__(self, other):
""">> operator for left-to-right composition (pipe)"""
return ComposableFunction(lambda x: other(self(x)))
def __lshift__(self, other):
"""<< operator for right-to-left composition"""
return ComposableFunction(lambda x: self(other(x)))
def __or__(self, other):
"""| operator for parallel composition"""
return ComposableFunction(lambda x: (self(x), other(x)))
def __and__(self, other):
"""& operator for conditional composition"""
return ComposableFunction(lambda x: other(x) if self(x) else x)
# Create composable functions
cf_add_5 = ComposableFunction(lambda x: x + 5)
cf_multiply_2 = ComposableFunction(lambda x: x * 2)
cf_square = ComposableFunction(lambda x: x ** 2)
cf_is_positive = ComposableFunction(lambda x: x > 0)
# Use compositional operators
pipe_result = (cf_add_5 >> cf_multiply_2 >> cf_square)(3)
print(f"Pipe composition (3 + 5) * 2 ^ 2: {pipe_result}")
compose_result = (cf_square << cf_multiply_2 << cf_add_5)(3)
print(f"Regular composition: {compose_result}")
parallel_result = (cf_add_5 | cf_multiply_2)(10)
print(f"Parallel composition: {parallel_result}")
# 6. Function composition with error handling
print(f"\n6๏ธโฃ ERROR-SAFE COMPOSITION:")
def safe_compose(*functions):
"""Compose functions with error handling"""
def safe_composed_function(x):
result = x
for i, func in enumerate(functions):
try:
result = func(result)
print(f" Step {i+1}: {func.__name__}({result})")
except Exception as e:
print(f" โ Step {i+1} failed: {func.__name__} - {e}")
return None
return result
return safe_composed_function
def risky_divide(x):
"""Function that might divide by zero"""
if x == 0:
raise ZeroDivisionError("Cannot process zero")
return 100 / x
def risky_sqrt(x):
"""Function that might fail with negative numbers"""
if x < 0:
raise ValueError("Cannot take square root of negative")
return x ** 0.5
# Test safe composition
safe_pipeline = safe_compose(
lambda x: x - 5,
risky_divide,
risky_sqrt,
lambda x: round(x, 2)
)
test_values = [10, 5, 0, -5]
print("Safe composition with error handling:")
for val in test_values:
print(f"Input: {val}")
result = safe_pipeline(val)
print(f"Output: {result}\n")
print("\n๐ฏ COMPOSITION BEST PRACTICES:")
print("โ
Keep functions small and focused")
print("โ
Use descriptive names for composed functions")
print("โ
Handle errors gracefully in compositions")
print("โ
Consider performance implications of composition")
print("โ
Use appropriate composition direction (pipe vs compose)")
print("โ
Document complex compositions")Python๐ง Practical Decorator Patterns
"""
๐ง PRACTICAL DECORATOR PATTERNS
Real-world decorator applications and patterns
"""
from functools import wraps, lru_cache
import json
from datetime import datetime, timedelta
from threading import Lock
import weakref
print("\n๐ง PRACTICAL DECORATOR PATTERNS")
print("=" * 40)
# 1. Rate limiting decorator
print("1๏ธโฃ RATE LIMITING DECORATOR:")
class RateLimiter:
"""Rate limiting decorator using token bucket algorithm"""
def __init__(self, max_calls, time_window):
self.max_calls = max_calls
self.time_window = time_window
self.calls = []
self.lock = Lock()
def __call__(self, func):
@wraps(func)
def wrapper(*args, **kwargs):
with self.lock:
now = datetime.now()
# Remove old calls outside time window
self.calls = [call_time for call_time in self.calls
if now - call_time < timedelta(seconds=self.time_window)]
if len(self.calls) >= self.max_calls:
raise Exception(f"Rate limit exceeded: {self.max_calls} calls per {self.time_window}s")
self.calls.append(now)
return func(*args, **kwargs)
wrapper.reset_rate_limit = lambda: setattr(self, 'calls', [])
return wrapper
@RateLimiter(max_calls=3, time_window=2) # 3 calls per 2 seconds
def api_call(endpoint):
"""Simulate API call with rate limiting"""
return f"Called {endpoint} at {datetime.now().strftime('%H:%M:%S')}"
# Test rate limiting
print("Testing rate limiting (3 calls per 2 seconds):")
try:
for i in range(5):
result = api_call(f"/endpoint{i}")
print(f" {result}")
time.sleep(0.3) # Small delay between calls
except Exception as e:
print(f" โ {e}")
# 2. Circuit breaker decorator
print(f"\n2๏ธโฃ CIRCUIT BREAKER DECORATOR:")
class CircuitBreaker:
"""Circuit breaker pattern implementation"""
def __init__(self, failure_threshold=3, timeout=5.0):
self.failure_threshold = failure_threshold
self.timeout = timeout
self.failure_count = 0
self.last_failure_time = None
self.state = 'CLOSED' # CLOSED, OPEN, HALF_OPEN
def __call__(self, func):
@wraps(func)
def wrapper(*args, **kwargs):
if self.state == 'OPEN':
if datetime.now().timestamp() - self.last_failure_time < self.timeout:
raise Exception("Circuit breaker OPEN - service unavailable")
else:
self.state = 'HALF_OPEN'
print("๐ Circuit breaker: Trying HALF_OPEN")
try:
result = func(*args, **kwargs)
if self.state == 'HALF_OPEN':
self.state = 'CLOSED'
self.failure_count = 0
print("โ
Circuit breaker: Reset to CLOSED")
return result
except Exception as e:
self.failure_count += 1
self.last_failure_time = datetime.now().timestamp()
if self.failure_count >= self.failure_threshold:
self.state = 'OPEN'
print(f"โ Circuit breaker: Opened after {self.failure_count} failures")
raise e
wrapper.get_state = lambda: self.state
wrapper.reset = lambda: setattr(self, 'failure_count', 0) or setattr(self, 'state', 'CLOSED')
return wrapper
@CircuitBreaker(failure_threshold=2, timeout=3.0)
def unreliable_service(success_rate=0.3):
"""Simulate unreliable service"""
import random
if random.random() > success_rate:
raise ConnectionError("Service failed")
return "Service response: OK"
# Test circuit breaker
print("Testing circuit breaker (fails 70% of time):")
for i in range(8):
try:
result = unreliable_service()
print(f" Call {i+1}: {result}")
except Exception as e:
print(f" Call {i+1}: Failed - {e}")
print(f" State: {unreliable_service.get_state()}")
time.sleep(0.5)
# 3. Database connection decorator
print(f"\n3๏ธโฃ DATABASE TRANSACTION DECORATOR:")
class DatabaseConnection:
"""Mock database connection for demonstration"""
def __init__(self):
self.in_transaction = False
self.committed = False
def begin_transaction(self):
self.in_transaction = True
self.committed = False
print("๐ Transaction started")
def commit(self):
if self.in_transaction:
self.committed = True
self.in_transaction = False
print("โ
Transaction committed")
def rollback(self):
if self.in_transaction:
self.in_transaction = False
self.committed = False
print("โฉ๏ธ Transaction rolled back")
# Global connection for demo
db_connection = DatabaseConnection()
def transactional(func):
"""Decorator to wrap function in database transaction"""
@wraps(func)
def wrapper(*args, **kwargs):
db_connection.begin_transaction()
try:
result = func(*args, **kwargs)
db_connection.commit()
return result
except Exception as e:
db_connection.rollback()
print(f"โ Transaction failed: {e}")
raise
return wrapper
@transactional
def create_user_record(name, email):
"""Create user with transaction safety"""
if not email or '@' not in email:
raise ValueError("Invalid email address")
print(f"๐ค Creating user: {name} ({email})")
return {'id': 123, 'name': name, 'email': email}
@transactional
def update_user_balance(user_id, amount):
"""Update user balance with transaction safety"""
if amount < 0:
raise ValueError("Cannot set negative balance")
print(f"๐ฐ Updating balance for user {user_id}: ${amount}")
return {'user_id': user_id, 'new_balance': amount}
# Test transactional decorator
print("Testing transactional operations:")
try:
user = create_user_record("Alice", "alice@example.com")
print(f" Created: {user}")
except Exception as e:
print(f" Failed: {e}")
try:
balance = update_user_balance(123, -100) # This will fail
print(f" Updated: {balance}")
except Exception as e:
print(f" Failed: {e}")
# 4. Audit logging decorator
print(f"\n4๏ธโฃ AUDIT LOGGING DECORATOR:")
def audit_log(operation_type, sensitive_params=None):
"""Decorator for audit logging"""
sensitive_params = sensitive_params or []
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
# Get function signature for parameter names
import inspect
sig = inspect.signature(func)
bound_args = sig.bind(*args, **kwargs)
bound_args.apply_defaults()
# Sanitize sensitive parameters
sanitized_args = {}
for param_name, value in bound_args.arguments.items():
if param_name in sensitive_params:
sanitized_args[param_name] = "***REDACTED***"
else:
sanitized_args[param_name] = value
# Log before execution
log_entry = {
'timestamp': datetime.now().isoformat(),
'operation': operation_type,
'function': func.__name__,
'parameters': sanitized_args,
'user': 'system' # In real app, get from context
}
print(f"๐ AUDIT: {json.dumps(log_entry, indent=2)}")
try:
result = func(*args, **kwargs)
# Log success
log_entry['status'] = 'SUCCESS'
log_entry['result_type'] = type(result).__name__
print(f"๐ AUDIT SUCCESS: {func.__name__}")
return result
except Exception as e:
# Log failure
log_entry['status'] = 'FAILED'
log_entry['error'] = str(e)
print(f"๐ AUDIT FAILED: {func.__name__} - {e}")
raise
return wrapper
return decorator
@audit_log("USER_MANAGEMENT", sensitive_params=['password'])
def change_password(username, old_password, new_password):
"""Change user password with audit logging"""
if len(new_password) < 8:
raise ValueError("Password too short")
print(f"๐ Password changed for {username}")
return True
@audit_log("DATA_ACCESS")
def get_user_data(user_id):
"""Get user data with audit logging"""
print(f"๐ Retrieved data for user {user_id}")
return {'user_id': user_id, 'name': 'Alice', 'role': 'admin'}
# Test audit logging
print("Testing audit logging:")
try:
change_password("alice", "old_secret_123", "new_secret_456")
except Exception as e:
print(f"Password change failed: {e}")
user_data = get_user_data(123)
print(f"Retrieved: {user_data}")
# 5. Performance profiling decorator
print(f"\n5๏ธโฃ PERFORMANCE PROFILING DECORATOR:")
class PerformanceProfiler:
"""Performance profiling decorator"""
def __init__(self, include_memory=False):
self.include_memory = include_memory
self.call_stats = {}
def __call__(self, func):
@wraps(func)
def wrapper(*args, **kwargs):
func_name = func.__name__
# Initialize stats if first call
if func_name not in self.call_stats:
self.call_stats[func_name] = {
'call_count': 0,
'total_time': 0,
'min_time': float('inf'),
'max_time': 0,
'avg_time': 0
}
# Measure performance
start_time = time.time()
if self.include_memory:
import tracemalloc
tracemalloc.start()
try:
result = func(*args, **kwargs)
execution_time = time.time() - start_time
# Update stats
stats = self.call_stats[func_name]
stats['call_count'] += 1
stats['total_time'] += execution_time
stats['min_time'] = min(stats['min_time'], execution_time)
stats['max_time'] = max(stats['max_time'], execution_time)
stats['avg_time'] = stats['total_time'] / stats['call_count']
if self.include_memory:
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
print(f"๐ง Memory usage: {current / 1024:.1f} KB (peak: {peak / 1024:.1f} KB)")
print(f"โก {func_name}: {execution_time:.4f}s (avg: {stats['avg_time']:.4f}s)")
return result
except Exception as e:
if self.include_memory and tracemalloc.is_tracing():
tracemalloc.stop()
raise
wrapper.get_stats = lambda: self.call_stats
wrapper.reset_stats = lambda: self.call_stats.clear()
return wrapper
profiler = PerformanceProfiler(include_memory=True)
@profiler
def data_processing_task(size):
"""Simulate data processing task"""
data = list(range(size))
processed = [x ** 2 + x for x in data if x % 2 == 0]
return sum(processed)
@profiler
def string_operations(text, repeat=1000):
"""Simulate string operations"""
result = text
for i in range(repeat):
result = result.upper().lower().strip()
return result
# Test performance profiling
print("Testing performance profiling:")
for i in [1000, 5000, 10000]:
result = data_processing_task(i)
print(f" Data task ({i} items): {result}")
for i in [500, 1000, 1500]:
result = string_operations("hello world", i)
print(f" String task ({i} ops): {len(result)} chars")
# Show accumulated stats
print("\nPerformance Statistics:")
stats = profiler.get_stats()
for func_name, data in stats.items():
print(f"๐ {func_name}:")
print(f" Calls: {data['call_count']}")
print(f" Total time: {data['total_time']:.4f}s")
print(f" Average: {data['avg_time']:.4f}s")
print(f" Range: {data['min_time']:.4f}s - {data['max_time']:.4f}s")
print("\n๐ฏ PRACTICAL DECORATOR BEST PRACTICES:")
print("โ
Use decorators for cross-cutting concerns")
print("โ
Keep decorators focused on single responsibilities")
print("โ
Preserve function metadata with @wraps")
print("โ
Make decorators configurable when needed")
print("โ
Consider performance implications")
print("โ
Test decorated and undecorated versions")
print("โ
Document decorator behavior clearly")Python๐ Real-World Composition Examples
"""
๐ REAL-WORLD COMPOSITION EXAMPLES
Practical applications combining composition and decorators
"""
from dataclasses import dataclass
from typing import Dict, List, Optional
import json
import hashlib
print("\n๐ REAL-WORLD COMPOSITION EXAMPLES")
print("=" * 40)
# 1. Web API Request Pipeline
print("1๏ธโฃ WEB API REQUEST PIPELINE:")
@dataclass
class APIRequest:
"""API request data structure"""
method: str
url: str
headers: Dict[str, str]
body: Optional[str] = None
params: Optional[Dict[str, str]] = None
@dataclass
class APIResponse:
"""API response data structure"""
status_code: int
headers: Dict[str, str]
body: str
request_time: float
# Request processing pipeline
def add_authentication(request: APIRequest) -> APIRequest:
"""Add authentication headers"""
request.headers['Authorization'] = 'Bearer token_12345'
print(f"๐ Added authentication to {request.url}")
return request
def add_content_type(request: APIRequest) -> APIRequest:
"""Add content type header"""
if request.body:
request.headers['Content-Type'] = 'application/json'
print(f"๐ Added content type to {request.url}")
return request
def validate_request(request: APIRequest) -> APIRequest:
"""Validate request structure"""
if not request.url.startswith('http'):
raise ValueError("Invalid URL format")
if request.method not in ['GET', 'POST', 'PUT', 'DELETE']:
raise ValueError("Invalid HTTP method")
print(f"โ
Validated request to {request.url}")
return request
def log_request(request: APIRequest) -> APIRequest:
"""Log request details"""
print(f"๐ก {request.method} {request.url}")
print(f" Headers: {len(request.headers)} items")
if request.body:
print(f" Body: {len(request.body)} chars")
return request
# Create request processing pipeline
process_request = pipe(
validate_request,
add_authentication,
add_content_type,
log_request
)
# Test the pipeline
sample_request = APIRequest(
method='POST',
url='https://api.example.com/users',
headers={'User-Agent': 'MyApp/1.0'},
body='{"name": "Alice", "email": "alice@example.com"}'
)
print("Processing API request:")
try:
processed_request = process_request(sample_request)
print(f"โ
Request processed successfully")
except Exception as e:
print(f"โ Request processing failed: {e}")
# 2. Data Validation and Transformation Pipeline
print(f"\n2๏ธโฃ DATA VALIDATION PIPELINE:")
def validate_email_format(data: dict) -> dict:
"""Validate email format"""
if 'email' in data:
email = data['email']
if '@' not in email or '.' not in email.split('@')[1]:
raise ValueError(f"Invalid email format: {email}")
print(f"โ
Email format validated")
return data
def validate_required_fields(required_fields: List[str]):
"""Factory for required field validation"""
def validator(data: dict) -> dict:
missing = [field for field in required_fields if field not in data or not data[field]]
if missing:
raise ValueError(f"Missing required fields: {missing}")
print(f"โ
Required fields validated: {required_fields}")
return data
return validator
def normalize_data(data: dict) -> dict:
"""Normalize data formats"""
normalized = data.copy()
# Normalize email to lowercase
if 'email' in normalized:
normalized['email'] = normalized['email'].lower().strip()
# Normalize name to title case
if 'name' in normalized:
normalized['name'] = normalized['name'].strip().title()
# Add timestamp
normalized['processed_at'] = datetime.now().isoformat()
print(f"๐ Data normalized")
return normalized
def enrich_data(data: dict) -> dict:
"""Enrich data with additional information"""
enriched = data.copy()
# Add user ID hash
if 'email' in data:
enriched['user_id'] = hashlib.md5(data['email'].encode()).hexdigest()[:8]
# Add domain from email
if 'email' in data:
enriched['domain'] = data['email'].split('@')[1]
print(f"๐ Data enriched")
return enriched
# Create data processing pipeline
process_user_data = pipe(
validate_required_fields(['name', 'email']),
validate_email_format,
normalize_data,
enrich_data
)
# Test data processing
test_users = [
{'name': 'alice smith', 'email': 'ALICE@Example.COM', 'age': 30},
{'name': 'bob jones', 'email': 'bob@invalid', 'age': 25}, # Invalid email
{'name': '', 'email': 'charlie@test.com'}, # Missing name
]
print("Processing user data:")
for i, user_data in enumerate(test_users, 1):
print(f"\nUser {i}: {user_data}")
try:
processed_user = process_user_data(user_data)
print(f"โ
Processed: {processed_user}")
except Exception as e:
print(f"โ Processing failed: {e}")
# 3. ETL Pipeline with Error Recovery
print(f"\n3๏ธโฃ ETL PIPELINE WITH ERROR RECOVERY:")
class ETLResult:
"""Result wrapper for ETL operations"""
def __init__(self, data=None, success=True, error=None, metadata=None):
self.data = data
self.success = success
self.error = error
self.metadata = metadata or {}
def __str__(self):
status = "SUCCESS" if self.success else "FAILED"
return f"ETLResult({status}, {len(self.data) if self.data else 0} records)"
def safe_etl_step(step_name: str):
"""Decorator to make ETL steps safe with error recovery"""
def decorator(func):
@wraps(func)
def wrapper(etl_result: ETLResult) -> ETLResult:
if not etl_result.success:
print(f"โญ๏ธ Skipping {step_name} due to previous error")
return etl_result
try:
print(f"๐ Executing {step_name}...")
new_data = func(etl_result.data)
new_metadata = etl_result.metadata.copy()
new_metadata[step_name] = {
'status': 'success',
'timestamp': datetime.now().isoformat(),
'records_processed': len(new_data) if new_data else 0
}
return ETLResult(
data=new_data,
success=True,
metadata=new_metadata
)
except Exception as e:
print(f"โ {step_name} failed: {e}")
new_metadata = etl_result.metadata.copy()
new_metadata[step_name] = {
'status': 'failed',
'timestamp': datetime.now().isoformat(),
'error': str(e)
}
return ETLResult(
data=etl_result.data, # Keep original data
success=False,
error=str(e),
metadata=new_metadata
)
return wrapper
return decorator
@safe_etl_step("EXTRACT")
def extract_data(data_source: str) -> List[dict]:
"""Extract data from source"""
if data_source == "database":
return [
{'id': 1, 'name': 'Alice', 'score': 95},
{'id': 2, 'name': 'Bob', 'score': 87},
{'id': 3, 'name': 'Charlie', 'score': 92}
]
elif data_source == "api":
return [
{'id': 4, 'name': 'Diana', 'score': 88},
{'id': 5, 'name': 'Eve', 'score': 94}
]
else:
raise ValueError(f"Unknown data source: {data_source}")
@safe_etl_step("TRANSFORM")
def transform_data(raw_data: List[dict]) -> List[dict]:
"""Transform extracted data"""
transformed = []
for record in raw_data:
if record['score'] < 60: # This could cause issues with some data
raise ValueError(f"Invalid score for {record['name']}: {record['score']}")
transformed_record = {
'user_id': f"USER_{record['id']:04d}",
'full_name': record['name'].upper(),
'grade': 'A' if record['score'] >= 90 else 'B' if record['score'] >= 80 else 'C',
'score': record['score'],
'processed_date': datetime.now().date().isoformat()
}
transformed.append(transformed_record)
return transformed
@safe_etl_step("LOAD")
def load_data(transformed_data: List[dict]) -> List[dict]:
"""Load transformed data to destination"""
print(f"๐พ Loading {len(transformed_data)} records to data warehouse")
# Simulate loading to database/warehouse
for record in transformed_data:
print(f" Loaded: {record['user_id']} - {record['full_name']}")
return transformed_data
# Create ETL pipeline
def run_etl_pipeline(data_source: str) -> ETLResult:
"""Run complete ETL pipeline"""
initial_result = ETLResult(data=data_source, success=True)
# Chain ETL operations
result = (initial_result
|> extract_data
|> transform_data
|> load_data)
return result
# Define custom pipe operator for ETL
class ETLResult:
# ... (previous ETLResult code)
def __or__(self, func):
"""Custom pipe operator for ETL steps"""
return func(self)
# Test ETL pipeline
print("Running ETL pipeline:")
for source in ['database', 'api', 'invalid_source']:
print(f"\n--- Processing from {source} ---")
result = run_etl_pipeline(source)
print(f"Final result: {result}")
if result.metadata:
print("Pipeline metadata:")
for step, info in result.metadata.items():
print(f" {step}: {info['status']} at {info['timestamp']}")
if 'error' in info:
print(f" Error: {info['error']}")
print("\n๐ฏ REAL-WORLD COMPOSITION TAKEAWAYS:")
print("โ
Compose small, focused functions for complex workflows")
print("โ
Use data classes for structured pipeline communication")
print("โ
Implement error recovery in pipeline steps")
print("โ
Add comprehensive logging and metadata tracking")
print("โ
Make pipelines testable by keeping steps pure")
print("โ
Use decorators for cross-cutting concerns (logging, timing, etc.)")
print("โ
Consider performance and memory usage in long pipelines")PythonThis completes the comprehensive section on Function Composition and Decorators. The content covers:
- Basic Function Composition – Building complex functions from simple ones using compose and pipe
- Decorators – Python’s elegant syntax for function composition with practical examples
- Advanced Composition Patterns – Monadic, parallel, async, and conditional composition
- Practical Decorator Patterns – Real-world decorator applications like rate limiting, circuit breakers, transactions
- Real-World Examples – Complete pipelines for API processing, data validation, and ETL operations
The functional composition and decorator patterns shown here provide powerful tools for building maintainable, reusable, and testable code by combining simple, focused functions into sophisticated workflows.
10. Generators and Lazy Evaluation
๐ Understanding Generators and Lazy Evaluation
Generators are functions that can pause and resume their execution, yielding values one at a time rather than computing all values at once. Lazy Evaluation means computations are delayed until their results are actually needed, enabling efficient memory usage and infinite sequences.
graph TD
A["๐ Generators & Lazy Evaluation"] --> B["โธ๏ธ Suspend & ResumePause Execution"]
A --> C["๐ฆ Yield ValuesOne at a Time"]
A --> D["๐พ Memory EfficientNo Storage of All Values"]
B --> B1["Generator Functions"]
B --> B2["Generator Expressions"]
B --> B3["Coroutines"]
C --> C1["yield keyword"]
C --> C2["Iterator Protocol"]
C --> C3["next() function"]
D --> D1["Constant Memory"]
D --> D2["Infinite Sequences"]
D --> D3["Stream Processing"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐ฏ Generator Basics
"""
๐ฏ GENERATOR FUNDAMENTALS
Understanding yield, generator functions, and lazy evaluation
"""
import sys
import time
from itertools import count, cycle, repeat, takewhile, dropwhile, chain
print("๐ GENERATOR FUNDAMENTALS")
print("=" * 30)
# 1. Basic generator function
print("1๏ธโฃ BASIC GENERATOR FUNCTION:")
def simple_generator():
"""A simple generator that yields three values"""
print(" Starting generator")
yield 1
print(" After yielding 1")
yield 2
print(" After yielding 2")
yield 3
print(" After yielding 3")
print(" Generator finished")
# Create generator object
gen = simple_generator()
print(f"Generator object: {gen}")
print(f"Type: {type(gen)}")
# Iterate through generator
print("\nIterating through generator:")
for value in gen:
print(f"Received: {value}")
# 2. Generator vs List memory comparison
print(f"\n2๏ธโฃ MEMORY EFFICIENCY:")
def number_generator(n):
"""Generator that yields numbers from 0 to n-1"""
for i in range(n):
yield i
def number_list(n):
"""Function that returns a list of numbers from 0 to n-1"""
return list(range(n))
# Compare memory usage
n = 10000
gen_obj = number_generator(n)
list_obj = number_list(n)
print(f"Generator memory: {sys.getsizeof(gen_obj)} bytes")
print(f"List memory: {sys.getsizeof(list_obj)} bytes")
print(f"Memory ratio: {sys.getsizeof(list_obj) / sys.getsizeof(gen_obj):.1f}x")
# 3. Generator expressions
print(f"\n3๏ธโฃ GENERATOR EXPRESSIONS:")
# List comprehension vs generator expression
numbers = range(10)
# List comprehension (eager evaluation)
squares_list = [x**2 for x in numbers]
print(f"List comprehension: {squares_list}")
print(f"List memory: {sys.getsizeof(squares_list)} bytes")
# Generator expression (lazy evaluation)
squares_gen = (x**2 for x in numbers)
print(f"Generator expression: {squares_gen}")
print(f"Generator memory: {sys.getsizeof(squares_gen)} bytes")
# Convert generator to list to see values
print(f"Generator values: {list(squares_gen)}")
# 4. Generator state preservation
print(f"\n4๏ธโฃ STATE PRESERVATION:")
def fibonacci_generator():
"""Generator that produces Fibonacci sequence"""
a, b = 0, 1
while True:
yield a
a, b = b, a + b
# Create Fibonacci generator
fib = fibonacci_generator()
# Get first 10 Fibonacci numbers
print("First 10 Fibonacci numbers:")
for i, num in enumerate(fib):
print(f"F({i}): {num}")
if i >= 9: # Stop after 10 numbers
break
# Generator remembers state - continue from where we left off
print("\nNext 5 Fibonacci numbers:")
for i, num in enumerate(fib):
print(f"F({i+10}): {num}")
if i >= 4: # Stop after 5 more numbers
break
# 5. Generator with send() method
print(f"\n5๏ธโฃ GENERATOR COMMUNICATION:")
def accumulator_generator():
"""Generator that accumulates sent values"""
total = 0
while True:
value = yield total # Yield current total, receive new value
if value is not None:
total += value
print(f" Added {value}, total now: {total}")
# Create accumulator
acc = accumulator_generator()
next(acc) # Prime the generator
print("Using generator as accumulator:")
print(f"Send 5: {acc.send(5)}")
print(f"Send 10: {acc.send(10)}")
print(f"Send -3: {acc.send(-3)}")
print(f"Current total: {next(acc)}")
# 6. Generator cleanup with try/finally
print(f"\n6๏ธโฃ GENERATOR CLEANUP:")
def resource_generator():
"""Generator that manages resources"""
print(" ๐ Opening resource")
try:
for i in range(5):
print(f" ๐ค Yielding {i}")
yield i
finally:
print(" ๐๏ธ Cleaning up resource")
print("Using resource generator:")
gen = resource_generator()
for value in gen:
print(f"Received: {value}")
if value == 2: # Early termination
print("Breaking early")
break
print("Resource cleaned up automatically")Pythonโก Advanced Generator Patterns
"""
โก ADVANCED GENERATOR PATTERNS
Complex use cases and functional patterns with generators
"""
from functools import wraps
import asyncio
from typing import Iterator, Callable, Any, TypeVar
T = TypeVar('T')
print("\nโก ADVANCED GENERATOR PATTERNS")
print("=" * 35)
# 1. Generator decorators
print("1๏ธโฃ GENERATOR DECORATORS:")
def generator_timer(func):
"""Decorator to time each yield in a generator"""
@wraps(func)
def wrapper(*args, **kwargs):
gen = func(*args, **kwargs)
start_time = time.time()
try:
while True:
value = next(gen)
current_time = time.time()
elapsed = current_time - start_time
print(f" โฑ๏ธ Yielded {value} after {elapsed:.3f}s")
yield value
start_time = current_time
except StopIteration:
print(" ๐ Generator finished")
return wrapper
@generator_timer
def slow_counter(n, delay=0.1):
"""Generator with artificial delay"""
for i in range(n):
time.sleep(delay)
yield i
print("Timed generator:")
for value in slow_counter(3, 0.2):
print(f"Got: {value}")
# 2. Generator pipelines
print(f"\n2๏ธโฃ GENERATOR PIPELINES:")
def numbers_from(start):
"""Infinite generator starting from a number"""
while True:
yield start
start += 1
def filter_generator(predicate, iterable):
"""Generator that filters based on predicate"""
for item in iterable:
if predicate(item):
yield item
def map_generator(func, iterable):
"""Generator that maps function over items"""
for item in iterable:
yield func(item)
def take_generator(n, iterable):
"""Generator that takes first n items"""
for i, item in enumerate(iterable):
if i >= n:
break
yield item
# Create processing pipeline
def create_pipeline():
"""Create a processing pipeline using generators"""
# Start with infinite numbers
numbers = numbers_from(1)
# Filter even numbers
evens = filter_generator(lambda x: x % 2 == 0, numbers)
# Square them
squares = map_generator(lambda x: x ** 2, evens)
# Take first 5
result = take_generator(5, squares)
return result
# Test pipeline
print("Generator pipeline (first 5 squares of even numbers):")
pipeline = create_pipeline()
for value in pipeline:
print(f" {value}")
# 3. Coroutine-style generators
print(f"\n3๏ธโฃ COROUTINE-STYLE GENERATORS:")
def data_processor():
"""Coroutine that processes incoming data"""
print(" ๐ Data processor ready")
processed_count = 0
try:
while True:
data = yield
if data is not None:
processed_count += 1
result = data.upper() if isinstance(data, str) else data * 2
print(f" ๐ Processed '{data}' โ '{result}' (count: {processed_count})")
except GeneratorExit:
print(f" ๐ Processor shutting down. Processed {processed_count} items.")
def data_sender(processor, data_list):
"""Send data to processor coroutine"""
next(processor) # Prime the coroutine
for data in data_list:
print(f"๐ค Sending: {data}")
processor.send(data)
processor.close() # Trigger GeneratorExit
print("Coroutine data processing:")
processor = data_processor()
data_sender(processor, ["hello", "world", 42, "python"])
# 4. Generator composition
print(f"\n4๏ธโฃ GENERATOR COMPOSITION:")
def chain_generators(*generators):
"""Chain multiple generators together"""
for gen in generators:
for item in gen:
yield item
def repeat_generator(value, times):
"""Generator that repeats a value"""
for _ in range(times):
yield value
def countdown_generator(start):
"""Generator that counts down"""
while start > 0:
yield start
start -= 1
# Compose generators
print("Composed generators:")
gen1 = repeat_generator("A", 3)
gen2 = countdown_generator(3)
gen3 = repeat_generator("B", 2)
composed = chain_generators(gen1, gen2, gen3)
for item in composed:
print(f" {item}")
# 5. Generator-based state machines
print(f"\n5๏ธโฃ GENERATOR STATE MACHINE:")
def traffic_light():
"""Traffic light state machine using generator"""
states = ['RED', 'GREEN', 'YELLOW']
durations = [3, 4, 2] # seconds
while True:
for state, duration in zip(states, durations):
print(f" ๐ฆ Light is {state}")
for second in range(duration):
yield f"{state} - {duration - second}s remaining"
time.sleep(0.1) # Simulate time passing (shortened for demo)
# Run traffic light for limited time
print("Traffic light simulation:")
light = traffic_light()
for i, status in enumerate(light):
print(f" {status}")
if i >= 15: # Run for limited iterations
break
# 6. Generator-based streaming processing
print(f"\n6๏ธโฃ STREAMING DATA PROCESSING:")
def data_stream():
"""Simulate streaming data"""
import random
while True:
yield {
'timestamp': time.time(),
'value': random.randint(1, 100),
'sensor': random.choice(['temp', 'humidity', 'pressure'])
}
def filter_stream(stream, condition):
"""Filter streaming data"""
for item in stream:
if condition(item):
yield item
def window_aggregate(stream, window_size):
"""Aggregate data in sliding windows"""
window = []
for item in stream:
window.append(item)
if len(window) >= window_size:
# Calculate average of values in window
avg_value = sum(d['value'] for d in window) / len(window)
yield {
'window_avg': avg_value,
'window_size': len(window),
'sensors': list(set(d['sensor'] for d in window))
}
window = window[1:] # Slide the window
# Create streaming pipeline
print("Streaming data processing:")
stream = data_stream()
# Filter high-value readings
high_values = filter_stream(stream, lambda d: d['value'] > 50)
# Aggregate in windows of 3
windowed = window_aggregate(high_values, 3)
# Process first few windows
for i, result in enumerate(windowed):
print(f" Window {i+1}: avg={result['window_avg']:.1f}, sensors={result['sensors']}")
if i >= 4: # Limit output
breakPython๐ Lazy Evaluation Patterns
"""
๐ LAZY EVALUATION PATTERNS
Advanced lazy evaluation techniques and functional patterns
"""
from itertools import islice, tee, accumulate, compress, groupby
from functools import partial
print("\n๐ LAZY EVALUATION PATTERNS")
print("=" * 35)
# 1. Lazy sequences
print("1๏ธโฃ LAZY SEQUENCES:")
class LazyRange:
"""Lazy range implementation"""
def __init__(self, start, stop=None, step=1):
if stop is None:
start, stop = 0, start
self.start = start
self.stop = stop
self.step = step
def __iter__(self):
current = self.start
while (self.step > 0 and current < self.stop) or (self.step < 0 and current > self.stop):
yield current
current += self.step
def __getitem__(self, key):
if isinstance(key, slice):
start, stop, step = key.indices(len(self))
return LazyRange(
self.start + start * self.step,
self.start + stop * self.step,
self.step * (step or 1)
)
elif isinstance(key, int):
if key < 0:
key += len(self)
if 0 <= key < len(self):
return self.start + key * self.step
raise IndexError("Index out of range")
def __len__(self):
return max(0, (self.stop - self.start + self.step - 1) // self.step)
# Test lazy range
print("Lazy range operations:")
lazy_range = LazyRange(1, 20, 2) # 1, 3, 5, 7, ..., 19
print(f"Length: {len(lazy_range)}")
print(f"First 5 items: {list(islice(lazy_range, 5))}")
print(f"Item at index 3: {lazy_range[3]}")
print(f"Slice [2:5]: {list(lazy_range[2:5])}")
# 2. Lazy functional operations
print(f"\n2๏ธโฃ LAZY FUNCTIONAL OPERATIONS:")
def lazy_map(func, iterable):
"""Lazy map implementation"""
return (func(item) for item in iterable)
def lazy_filter(predicate, iterable):
"""Lazy filter implementation"""
return (item for item in iterable if predicate(item))
def lazy_reduce(func, iterable, initial=None):
"""Lazy reduce that yields intermediate results"""
iterator = iter(iterable)
if initial is None:
try:
accumulator = next(iterator)
except StopIteration:
return
else:
accumulator = initial
yield accumulator
for item in iterator:
accumulator = func(accumulator, item)
yield accumulator
# Test lazy operations
print("Lazy functional operations:")
numbers = range(1, 6) # [1, 2, 3, 4, 5]
# Chain lazy operations
doubled = lazy_map(lambda x: x * 2, numbers)
evens = lazy_filter(lambda x: x % 2 == 0, doubled)
running_sum = lazy_reduce(lambda a, b: a + b, evens)
print(f"Original: {list(numbers)}")
print(f"Doubled: {list(lazy_map(lambda x: x * 2, numbers))}")
print(f"Even doubles: {list(lazy_filter(lambda x: x % 2 == 0, lazy_map(lambda x: x * 2, numbers)))}")
print(f"Running sums: {list(running_sum)}")
# 3. Memoized generators
print(f"\n3๏ธโฃ MEMOIZED GENERATORS:")
def memoized_generator(func):
"""Decorator to memoize generator results"""
cache = {}
def wrapper(*args):
key = args
if key not in cache:
cache[key] = []
gen = func(*args)
def memoized_gen():
# First, yield cached results
for item in cache[key]:
yield item
# Then generate and cache new results
try:
while True:
item = next(gen)
cache[key].append(item)
yield item
except StopIteration:
pass
return memoized_gen()
else:
return iter(cache[key])
return wrapper
@memoized_generator
def expensive_sequence(n):
"""Expensive computation sequence"""
print(f" ๐ฐ Computing expensive sequence for {n}")
for i in range(n):
time.sleep(0.1) # Simulate expensive computation
yield i ** 2
print("Memoized generator:")
print("First call:")
seq1 = expensive_sequence(3)
print(f"Results: {list(seq1)}")
print("Second call (cached):")
seq2 = expensive_sequence(3)
print(f"Results: {list(seq2)}")
# 4. Infinite lazy sequences
print(f"\n4๏ธโฃ INFINITE SEQUENCES:")
def primes():
"""Generator for prime numbers using Sieve of Eratosthenes"""
yield 2
sieve = {} # composite -> first prime factor
candidate = 3
while True:
if candidate not in sieve:
yield candidate
sieve[candidate * candidate] = candidate
else:
prime = sieve.pop(candidate)
next_multiple = candidate + prime
while next_multiple in sieve:
next_multiple += prime
sieve[next_multiple] = prime
candidate += 2
def collatz_sequence(n):
"""Infinite Collatz sequence starting from n"""
while True:
yield n
if n == 1:
break
n = n // 2 if n % 2 == 0 else 3 * n + 1
# Test infinite sequences
print("First 10 prime numbers:")
prime_gen = primes()
for i, prime in enumerate(prime_gen):
print(f" Prime {i+1}: {prime}")
if i >= 9:
break
print("\nCollatz sequence for 7:")
collatz_gen = collatz_sequence(7)
sequence = list(collatz_gen)
print(f" Sequence: {sequence}")
# 5. Lazy data structures
print(f"\n5๏ธโฃ LAZY DATA STRUCTURES:")
class LazyList:
"""Lazy list implementation"""
def __init__(self, generator):
self._generator = generator
self._cache = []
self._exhausted = False
def _ensure_index(self, index):
"""Ensure cache contains item at index"""
while len(self._cache) <= index and not self._exhausted:
try:
item = next(self._generator)
self._cache.append(item)
except StopIteration:
self._exhausted = True
break
def __getitem__(self, index):
if isinstance(index, slice):
# Handle slice
start, stop, step = index.indices(len(self))
return [self[i] for i in range(start, stop, step)]
else:
self._ensure_index(index)
if index < len(self._cache):
return self._cache[index]
raise IndexError("Index out of range")
def __len__(self):
if not self._exhausted:
# Force evaluation of entire generator
list(self._generator)
self._exhausted = True
return len(self._cache)
def __iter__(self):
# First yield cached items
for item in self._cache:
yield item
# Then yield remaining items
if not self._exhausted:
for item in self._generator:
self._cache.append(item)
yield item
self._exhausted = True
# Test lazy list
def fibonacci_gen():
"""Fibonacci generator"""
a, b = 0, 1
while True:
yield a
a, b = b, a + b
print("Lazy list with Fibonacci sequence:")
lazy_fib = LazyList(fibonacci_gen())
print(f"Item at index 10: {lazy_fib[10]}")
print(f"Items 5-8: {lazy_fib[5:9]}")
print(f"First 12 items: {list(islice(lazy_fib, 12))}")
# 6. Streaming transformations
print(f"\n6๏ธโฃ STREAMING TRANSFORMATIONS:")
def stream_transform(transformation):
"""Decorator for streaming transformations"""
def decorator(func):
@wraps(func)
def wrapper(stream):
return transformation(func, stream)
return wrapper
return decorator
def batch_transform(func, stream, batch_size=3):
"""Apply function to batches of stream"""
batch = []
for item in stream:
batch.append(item)
if len(batch) >= batch_size:
yield func(batch)
batch = []
if batch: # Handle remaining items
yield func(batch)
def sliding_window_transform(func, stream, window_size=3):
"""Apply function to sliding windows"""
window = []
for item in stream:
window.append(item)
if len(window) >= window_size:
yield func(window)
window = window[1:]
@stream_transform(partial(batch_transform, batch_size=3))
def sum_batches(batch):
"""Sum items in batch"""
return sum(batch)
@stream_transform(partial(sliding_window_transform, window_size=3))
def average_window(window):
"""Calculate average of window"""
return sum(window) / len(window)
# Test streaming transformations
print("Streaming transformations:")
number_stream = range(1, 11) # 1 through 10
batch_sums = sum_batches(number_stream)
print(f"Batch sums (size 3): {list(batch_sums)}")
window_averages = average_window(range(1, 11))
print(f"Sliding averages (window 3): {[round(avg, 2) for avg in window_averages]}")Python๐ Performance and Memory Benefits
"""
๐ PERFORMANCE AND MEMORY BENEFITS
Demonstrating efficiency gains from lazy evaluation
"""
import psutil
import os
from memory_profiler import profile
print("\n๐ PERFORMANCE AND MEMORY BENEFITS")
print("=" * 40)
# 1. Memory usage comparison
print("1๏ธโฃ MEMORY USAGE COMPARISON:")
def memory_usage():
"""Get current memory usage"""
process = psutil.Process(os.getpid())
return process.memory_info().rss / 1024 / 1024 # MB
def eager_processing(n):
"""Eager evaluation: create all data in memory"""
# Create large list
numbers = list(range(n))
# Transform all at once
squared = [x**2 for x in numbers]
filtered = [x for x in squared if x % 2 == 0]
return sum(filtered)
def lazy_processing(n):
"""Lazy evaluation: process on-demand"""
# Generator expressions (no memory allocation)
numbers = range(n)
squared = (x**2 for x in numbers)
filtered = (x for x in squared if x % 2 == 0)
return sum(filtered)
# Compare memory usage
n = 100000
print(f"Processing {n} numbers:")
initial_memory = memory_usage()
# Eager processing
eager_start = memory_usage()
eager_result = eager_processing(n)
eager_memory = memory_usage() - eager_start
print(f"Eager processing:")
print(f" Result: {eager_result}")
print(f" Memory used: {eager_memory:.2f} MB")
# Lazy processing
lazy_start = memory_usage()
lazy_result = lazy_processing(n)
lazy_memory = memory_usage() - lazy_start
print(f"Lazy processing:")
print(f" Result: {lazy_result}")
print(f" Memory used: {lazy_memory:.2f} MB")
print(f"Memory savings: {((eager_memory - lazy_memory) / eager_memory * 100):.1f}%")
# 2. Early termination benefits
print(f"\n2๏ธโฃ EARLY TERMINATION BENEFITS:")
def find_first_large_prime_eager(limit):
"""Eagerly generate all primes then find first large one"""
def is_prime(n):
if n < 2:
return False
for i in range(2, int(n**0.5) + 1):
if n % i == 0:
return False
return True
# Generate ALL primes up to limit (wasteful if we find one early)
all_primes = [n for n in range(2, limit) if is_prime(n)]
# Find first prime > 1000
for prime in all_primes:
if prime > 1000:
return prime
return None
def find_first_large_prime_lazy(limit):
"""Lazily generate primes and stop at first large one"""
def is_prime(n):
if n < 2:
return False
for i in range(2, int(n**0.5) + 1):
if n % i == 0:
return False
return True
# Generate primes lazily
primes = (n for n in range(2, limit) if is_prime(n))
# Stop at first prime > 1000
for prime in primes:
if prime > 1000:
return prime
return None
# Compare performance
limit = 10000
print("Finding first prime > 1000:")
# Time eager approach
start_time = time.time()
eager_prime = find_first_large_prime_eager(limit)
eager_time = time.time() - start_time
# Time lazy approach
start_time = time.time()
lazy_prime = find_first_large_prime_lazy(limit)
lazy_time = time.time() - start_time
print(f"Eager approach:")
print(f" Prime found: {eager_prime}")
print(f" Time taken: {eager_time:.4f} seconds")
print(f"Lazy approach:")
print(f" Prime found: {lazy_prime}")
print(f" Time taken: {lazy_time:.4f} seconds")
print(f"Speedup: {eager_time / lazy_time:.1f}x")
# 3. Infinite sequences without memory issues
print(f"\n3๏ธโฃ INFINITE SEQUENCES:")
def fibonacci_infinite():
"""Infinite Fibonacci sequence"""
a, b = 0, 1
while True:
yield a
a, b = b, a + b
def prime_factors(n):
"""Generator for prime factors of n"""
d = 2
while d * d <= n:
while n % d == 0:
yield d
n //= d
d += 1
if n > 1:
yield n
# Work with infinite sequences safely
print("Working with infinite sequences:")
# Get 100th Fibonacci number without storing first 99
fib_gen = fibonacci_infinite()
for i, fib in enumerate(fib_gen):
if i == 20: # Get 21st Fibonacci number
print(f"21st Fibonacci number: {fib}")
break
# Factor large numbers efficiently
large_number = 123456789
factors = list(prime_factors(large_number))
print(f"Prime factors of {large_number}: {factors}")
# 4. Stream processing example
print(f"\n4๏ธโฃ STREAM PROCESSING:")
def log_entries():
"""Simulate infinite log entries"""
import random
log_levels = ['INFO', 'WARNING', 'ERROR', 'DEBUG']
counter = 0
while True:
counter += 1
yield {
'id': counter,
'level': random.choice(log_levels),
'message': f'Log message {counter}',
'timestamp': time.time()
}
def process_logs_lazy(log_stream, max_errors=3):
"""Process logs lazily, stopping after max errors"""
error_count = 0
for entry in log_stream:
if entry['level'] == 'ERROR':
error_count += 1
print(f" โ ๏ธ Error #{error_count}: {entry['message']}")
if error_count >= max_errors:
print(f" ๐ Stopping after {max_errors} errors")
break
elif entry['level'] == 'WARNING':
print(f" โ ๏ธ Warning: {entry['message']}")
# Simulate some processing delay
time.sleep(0.01)
# Process log stream
print("Processing log stream (stopping after 3 errors):")
logs = log_entries()
process_logs_lazy(logs, max_errors=3)
print("\n๐ฏ LAZY EVALUATION BENEFITS:")
print("โ
Constant memory usage regardless of data size")
print("โ
Early termination saves computation")
print("โ
Infinite sequences are possible")
print("โ
Composable and chainable operations")
print("โ
Better performance for large datasets")
print("โ
Natural streaming data processing")Python๐ง Practical Applications
"""
๐ง PRACTICAL APPLICATIONS
Real-world uses of generators and lazy evaluation
"""
import csv
import json
from pathlib import Path
import requests
print("\n๐ง PRACTICAL APPLICATIONS")
print("=" * 30)
# 1. File processing with generators
print("1๏ธโฃ FILE PROCESSING:")
def read_large_csv_lazy(filename, chunk_size=1000):
"""Read large CSV file lazily in chunks"""
def csv_chunks():
# Simulate large CSV file
sample_data = [
['name', 'age', 'city', 'salary'],
['Alice', '30', 'New York', '75000'],
['Bob', '25', 'London', '65000'],
['Charlie', '35', 'Tokyo', '80000'],
['Diana', '28', 'Paris', '70000'],
['Eve', '32', 'Sydney', '72000']
] * 200 # Simulate 1000+ rows
chunk = []
for i, row in enumerate(sample_data):
chunk.append(row)
if len(chunk) >= chunk_size:
yield chunk
chunk = []
if chunk: # Yield remaining rows
yield chunk
return csv_chunks()
def process_csv_data(csv_chunks):
"""Process CSV data incrementally"""
total_salary = 0
employee_count = 0
for chunk in csv_chunks:
headers = chunk[0] if employee_count == 0 else None
data_rows = chunk[1:] if headers else chunk
for row in data_rows:
if len(row) >= 4 and row[3].isdigit(): # Has salary column
total_salary += int(row[3])
employee_count += 1
if employee_count % 100 == 0: # Progress indicator
print(f" ๐ Processed {employee_count} employees")
return {
'total_employees': employee_count,
'average_salary': total_salary / employee_count if employee_count > 0 else 0
}
print("Processing large CSV file:")
csv_data = read_large_csv_lazy('employees.csv', chunk_size=50)
stats = process_csv_data(csv_data)
print(f"Results: {stats}")
# 2. API pagination with generators
print(f"\n2๏ธโฃ API PAGINATION:")
def fetch_paginated_data(base_url, page_size=10):
"""Generator for paginated API data"""
page = 1
while True:
# Simulate API call
print(f" ๐ก Fetching page {page}")
# Mock API response
mock_response = {
'data': [
{'id': i + (page-1) * page_size, 'name': f'Item {i + (page-1) * page_size}'}
for i in range(1, page_size + 1)
],
'has_more': page < 3, # Only 3 pages of mock data
'page': page,
'total_pages': 3
}
# Yield each item
for item in mock_response['data']:
yield item
# Check if more pages available
if not mock_response['has_more']:
break
page += 1
def process_api_data(data_stream, condition):
"""Process API data with early termination"""
processed = 0
for item in data_stream:
processed += 1
print(f" Processing: {item['name']}")
if condition(item):
print(f" ๐ฏ Found matching item: {item['name']}")
return item, processed
if processed >= 15: # Limit for demo
print(" โธ๏ธ Stopping early for demo")
break
return None, processed
print("Fetching paginated API data:")
api_data = fetch_paginated_data('https://api.example.com/items')
# Stop at first item with ID > 12
result, count = process_api_data(
api_data,
lambda item: item['id'] > 12
)
print(f"Found: {result}, Processed: {count} items")
# 3. Data transformation pipeline
print(f"\n3๏ธโฃ DATA TRANSFORMATION PIPELINE:")
def data_source():
"""Simulate streaming data source"""
import random
for i in range(100):
yield {
'id': i,
'value': random.randint(1, 100),
'category': random.choice(['A', 'B', 'C']),
'timestamp': time.time() + i
}
def validate_data(stream):
"""Validate data in stream"""
for item in stream:
if 'id' in item and 'value' in item and 'category' in item:
yield item
else:
print(f" โ ๏ธ Invalid data: {item}")
def enrich_data(stream):
"""Enrich data with additional fields"""
for item in stream:
enriched = item.copy()
enriched['processed_at'] = time.time()
enriched['value_category'] = 'high' if item['value'] > 70 else 'medium' if item['value'] > 30 else 'low'
yield enriched
def filter_data(stream, condition):
"""Filter data based on condition"""
for item in stream:
if condition(item):
yield item
def aggregate_data(stream, key_func, agg_func, window_size=5):
"""Aggregate data in windows"""
window = []
for item in stream:
window.append(item)
if len(window) >= window_size:
# Group by key and aggregate
groups = {}
for w_item in window:
key = key_func(w_item)
if key not in groups:
groups[key] = []
groups[key].append(w_item)
# Apply aggregation function
for key, group in groups.items():
yield {
'key': key,
'count': len(group),
'aggregated_value': agg_func([item['value'] for item in group])
}
window = []
# Build and execute pipeline
print("Data transformation pipeline:")
pipeline = data_source()
pipeline = validate_data(pipeline)
pipeline = enrich_data(pipeline)
pipeline = filter_data(pipeline, lambda x: x['value'] > 20) # Only values > 20
pipeline = aggregate_data(
pipeline,
key_func=lambda x: x['category'], # Group by category
agg_func=sum, # Sum values
window_size=10
)
# Process first few results
for i, result in enumerate(pipeline):
print(f" Aggregation {i+1}: Category {result['key']}, Count: {result['count']}, Sum: {result['aggregated_value']}")
if i >= 4: # Limit output
break
# 4. Real-time monitoring with generators
print(f"\n4๏ธโฃ REAL-TIME MONITORING:")
def system_metrics():
"""Generate system metrics"""
import random
while True:
yield {
'timestamp': time.time(),
'cpu_percent': random.uniform(0, 100),
'memory_percent': random.uniform(20, 90),
'disk_io': random.randint(0, 1000),
'network_io': random.randint(0, 5000)
}
def alert_on_threshold(metrics_stream, thresholds):
"""Generate alerts based on thresholds"""
for metrics in metrics_stream:
alerts = []
for metric, threshold in thresholds.items():
if metric in metrics and metrics[metric] > threshold:
alerts.append(f"{metric.upper()} high: {metrics[metric]:.1f}")
if alerts:
yield {
'timestamp': metrics['timestamp'],
'alerts': alerts,
'metrics': metrics
}
def sliding_average(stream, window_size=5):
"""Calculate sliding average of metrics"""
window = []
for item in stream:
window.append(item)
if len(window) > window_size:
window.pop(0)
# Calculate averages
averages = {}
for key in item['metrics']:
if isinstance(item['metrics'][key], (int, float)):
averages[f"avg_{key}"] = sum(w['metrics'][key] for w in window) / len(window)
yield {
'timestamp': item['timestamp'],
'alerts': item['alerts'],
'window_averages': averages
}
# Create monitoring pipeline
print("System monitoring pipeline:")
metrics = system_metrics()
# Set thresholds
thresholds = {
'cpu_percent': 80,
'memory_percent': 85,
'disk_io': 800
}
# Create alert pipeline
alerts = alert_on_threshold(metrics, thresholds)
averaged_alerts = sliding_average(alerts)
# Monitor for a short time
start_time = time.time()
for alert in averaged_alerts:
print(f" ๐จ ALERT at {alert['timestamp']:.0f}:")
for alert_msg in alert['alerts']:
print(f" - {alert_msg}")
# Show some averages
cpu_avg = alert['window_averages'].get('avg_cpu_percent', 0)
mem_avg = alert['window_averages'].get('avg_memory_percent', 0)
print(f" ๐ 5-min averages: CPU {cpu_avg:.1f}%, Memory {mem_avg:.1f}%")
time.sleep(0.1) # Short delay for demo
if time.time() - start_time > 2: # Run for 2 seconds
break
print("\n๐ฏ PRACTICAL BENEFITS:")
print("โ
Process large files without loading entirely into memory")
print("โ
Handle infinite data streams efficiently")
print("โ
Early termination saves time and resources")
print("โ
Composable processing pipelines")
print("โ
Real-time data processing")
print("โ
Memory-efficient data transformations")PythonThis completes the comprehensive section on Generators and Lazy Evaluation. The content covers:
- Generator Basics – Understanding yield, generator functions, and state preservation
- Advanced Generator Patterns – Decorators, pipelines, coroutines, and composition
- Lazy Evaluation Patterns – Lazy sequences, memoization, infinite sequences, and data structures
- Performance Benefits – Memory efficiency, early termination, and streaming processing
- Practical Applications – File processing, API pagination, data pipelines, and real-time monitoring
The generator and lazy evaluation patterns shown here enable efficient processing of large datasets, infinite sequences, and real-time data streams while maintaining low memory usage and high composability.
11. Partial Application and Currying
๐งฉ Understanding Partial Application and Currying
Partial Application is the process of fixing some arguments of a function, producing another function with fewer parameters. Currying transforms a function that takes multiple arguments into a sequence of functions that each take a single argument. Both techniques enable powerful functional programming patterns for creating specialized, reusable functions.
graph TD
A["๐งฉ Partial Application & Currying"] --> B["๐ฆ Partial ApplicationFix Some Arguments"]
A --> C["๐ CurryingChain Single Arguments"]
A --> D["๐ฏ Function SpecializationCreate Focused Functions"]
B --> B1["functools.partial"]
B --> B2["Custom partial functions"]
B --> B3["Argument pre-filling"]
C --> C1["f(a, b, c) โ f(a)(b)(c)"]
C --> C2["Sequential application"]
C --> C3["Intermediate functions"]
D --> D1["Configuration functions"]
D --> D2["Event handlers"]
D --> D3["API wrappers"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐ฆ Partial Application Fundamentals
"""
๐ฆ PARTIAL APPLICATION FUNDAMENTALS
Understanding and implementing partial application
"""
from functools import partial
import operator
from typing import Callable, Any
print("๐ฆ PARTIAL APPLICATION FUNDAMENTALS")
print("=" * 40)
# 1. Basic partial application with functools.partial
print("1๏ธโฃ BASIC PARTIAL APPLICATION:")
def multiply(x, y, z):
"""Multiply three numbers"""
return x * y * z
def greet(greeting, name, punctuation="!"):
"""Create a greeting message"""
return f"{greeting} {name}{punctuation}"
# Create specialized functions using partial
double = partial(multiply, 2) # Fix first argument to 2
print(f"Double function: double(3, 4) = {double(3, 4)}") # 2 * 3 * 4 = 24
triple = partial(multiply, 3) # Fix first argument to 3
print(f"Triple function: triple(2, 5) = {triple(2, 5)}") # 3 * 2 * 5 = 30
# Partial with multiple arguments
multiply_by_6 = partial(multiply, 2, 3) # Fix first two arguments
print(f"Multiply by 6: multiply_by_6(4) = {multiply_by_6(4)}") # 2 * 3 * 4 = 24
# Partial with keyword arguments
say_hello = partial(greet, "Hello")
say_goodbye = partial(greet, "Goodbye", punctuation=".")
print(f"Say hello: {say_hello('Alice')}")
print(f"Say goodbye: {say_goodbye('Bob')}")
# 2. Custom partial implementation
print(f"\n2๏ธโฃ CUSTOM PARTIAL IMPLEMENTATION:")
def my_partial(func, *partial_args, **partial_kwargs):
"""Custom partial function implementation"""
def wrapper(*args, **kwargs):
# Combine partial and new arguments
combined_args = partial_args + args
combined_kwargs = {**partial_kwargs, **kwargs}
return func(*combined_args, **combined_kwargs)
# Preserve function metadata
wrapper.__name__ = f"partial_{func.__name__}"
wrapper.__doc__ = f"Partial application of {func.__name__}"
return wrapper
def power(base, exponent, modulo=None):
"""Calculate base^exponent, optionally with modulo"""
result = base ** exponent
return result % modulo if modulo else result
# Test custom partial
square = my_partial(power, exponent=2)
cube = my_partial(power, exponent=3)
mod_square = my_partial(power, exponent=2, modulo=10)
print(f"Square of 5: {square(5)}")
print(f"Cube of 4: {cube(4)}")
print(f"5ยฒ mod 10: {mod_square(5)}")
# 3. Partial application with operators
print(f"\n3๏ธโฃ PARTIAL WITH OPERATORS:")
# Create specialized arithmetic operations
add_10 = partial(operator.add, 10)
multiply_by_5 = partial(operator.mul, 5)
divide_by_2 = partial(operator.truediv, 1) # Note: 1 / x for reciprocal
subtract_from_100 = partial(operator.sub, 100)
numbers = [2, 4, 6, 8, 10]
print(f"Original numbers: {numbers}")
print(f"Add 10: {list(map(add_10, numbers))}")
print(f"Multiply by 5: {list(map(multiply_by_5, numbers))}")
print(f"Subtract from 100: {list(map(subtract_from_100, numbers))}")
# String operations
starts_with_py = partial(str.startswith, prefix='py')
ends_with_ing = partial(str.endswith, suffix='ing')
words = ['python', 'java', 'programming', 'coding', 'typing']
print(f"\nWords: {words}")
print(f"Starts with 'py': {list(filter(starts_with_py, words))}")
print(f"Ends with 'ing': {list(filter(ends_with_ing, words))}")
# 4. Partial application for configuration
print(f"\n4๏ธโฃ CONFIGURATION WITH PARTIAL:")
def create_logger(level, format_string, name):
"""Create a logger with specified configuration"""
def log(message):
formatted_message = format_string.format(
level=level,
name=name,
message=message
)
print(formatted_message)
return log
# Create specialized loggers
debug_logger = partial(create_logger, "DEBUG", "[{level}] {name}: {message}")
error_logger = partial(create_logger, "ERROR", "โ [{level}] {name}: {message}")
info_logger = partial(create_logger, "INFO", "โน๏ธ [{level}] {name}: {message}")
# Use specialized loggers
app_debug = debug_logger("MyApp")
db_error = error_logger("Database")
api_info = info_logger("API")
print("Logging examples:")
app_debug("Application started")
db_error("Connection failed")
api_info("Request processed")
# 5. Partial application with higher-order functions
print(f"\n5๏ธโฃ PARTIAL WITH HIGHER-ORDER FUNCTIONS:")
def apply_discount(price, discount_rate, tax_rate=0.08):
"""Apply discount and tax to a price"""
discounted = price * (1 - discount_rate)
final_price = discounted * (1 + tax_rate)
return round(final_price, 2)
def bulk_process(processor, items):
"""Apply a processor function to multiple items"""
return [processor(item) for item in items]
# Create specialized discount functions
student_discount = partial(apply_discount, discount_rate=0.15) # 15% student discount
senior_discount = partial(apply_discount, discount_rate=0.20) # 20% senior discount
employee_discount = partial(apply_discount, discount_rate=0.25, tax_rate=0.05) # 25% discount, 5% tax
prices = [100, 250, 75, 500]
print(f"Original prices: {prices}")
# Apply discounts to all prices
student_prices = bulk_process(student_discount, prices)
senior_prices = bulk_process(senior_discount, prices)
employee_prices = bulk_process(employee_discount, prices)
print(f"Student prices: {student_prices}")
print(f"Senior prices: {senior_prices}")
print(f"Employee prices: {employee_prices}")
# 6. Partial application for event handling
print(f"\n6๏ธโฃ EVENT HANDLING WITH PARTIAL:")
def handle_button_click(button_id, action, event_data):
"""Generic button click handler"""
print(f"Button '{button_id}' clicked: {action}")
print(f"Event data: {event_data}")
if action == "save":
return "Data saved successfully"
elif action == "delete":
return "Item deleted"
elif action == "cancel":
return "Operation cancelled"
else:
return "Unknown action"
# Create specialized event handlers
save_handler = partial(handle_button_click, button_id="save_btn", action="save")
delete_handler = partial(handle_button_click, button_id="delete_btn", action="delete")
cancel_handler = partial(handle_button_click, button_id="cancel_btn", action="cancel")
# Simulate event handling
print("Event handling simulation:")
result1 = save_handler({"user_id": 123, "form_data": {"name": "Alice"}})
print(f"Result: {result1}\n")
result2 = delete_handler({"item_id": 456})
print(f"Result: {result2}\n")
result3 = cancel_handler({"reason": "user_request"})
print(f"Result: {result3}")Python๐ Currying Deep Dive
"""
๐ CURRYING DEEP DIVE
Understanding and implementing currying
"""
from functools import reduce
import inspect
print("\n๐ CURRYING DEEP DIVE")
print("=" * 25)
# 1. Manual currying examples
print("1๏ธโฃ MANUAL CURRYING:")
def add_three_numbers(x):
"""Manually curried addition of three numbers"""
def add_second(y):
def add_third(z):
return x + y + z
return add_third
return add_second
# Use the curried function
result1 = add_three_numbers(1)(2)(3) # 1 + 2 + 3 = 6
print(f"add_three_numbers(1)(2)(3) = {result1}")
# Partial application of curried function
add_5 = add_three_numbers(5)
add_5_and_3 = add_5(3)
result2 = add_5_and_3(7) # 5 + 3 + 7 = 15
print(f"Step by step: add_5_and_3(7) = {result2}")
# More examples
def multiply_curry(x):
"""Curried multiplication"""
return lambda y: lambda z: x * y * z
def power_curry(base):
"""Curried exponentiation"""
return lambda exponent: base ** exponent
multiply_result = multiply_curry(2)(3)(4) # 2 * 3 * 4 = 24
power_result = power_curry(2)(8) # 2^8 = 256
print(f"multiply_curry(2)(3)(4) = {multiply_result}")
print(f"power_curry(2)(8) = {power_result}")
# 2. Automatic currying implementation
print(f"\n2๏ธโฃ AUTOMATIC CURRYING:")
def curry(func):
"""Automatic currying decorator"""
# Get function signature
sig = inspect.signature(func)
param_count = len(sig.parameters)
def curried(*args, **kwargs):
# If we have enough arguments, call the function
if len(args) + len(kwargs) >= param_count:
return func(*args, **kwargs)
# Otherwise, return a new curried function
def partial_curry(*new_args, **new_kwargs):
combined_args = args + new_args
combined_kwargs = {**kwargs, **new_kwargs}
return curried(*combined_args, **combined_kwargs)
return partial_curry
return curried
@curry
def add_four(a, b, c, d):
"""Add four numbers"""
return a + b + c + d
@curry
def format_message(template, name, age, city):
"""Format a message with template and values"""
return template.format(name=name, age=age, city=city)
# Test automatic currying
print("Automatic currying examples:")
# All at once
result1 = add_four(1, 2, 3, 4)
print(f"add_four(1, 2, 3, 4) = {result1}")
# Step by step
result2 = add_four(1)(2)(3)(4)
print(f"add_four(1)(2)(3)(4) = {result2}")
# Mixed application
add_1_2 = add_four(1, 2)
result3 = add_1_2(3)(4)
print(f"add_1_2(3)(4) = {result3}")
# String formatting
template = "Hello {name}, you are {age} years old and live in {city}."
format_greeting = format_message(template)
format_alice = format_greeting("Alice")
result4 = format_alice(30, "New York")
print(f"Formatted message: {result4}")
# 3. Currying with keyword arguments
print(f"\n3๏ธโฃ CURRYING WITH KEYWORD ARGUMENTS:")
def advanced_curry(func):
"""Advanced currying that handles keyword arguments properly"""
sig = inspect.signature(func)
param_names = list(sig.parameters.keys())
def curried(*args, **kwargs):
# Bind arguments to parameter names
bound_args = {}
# Add positional arguments
for i, arg in enumerate(args):
if i < len(param_names):
bound_args[param_names[i]] = arg
# Add keyword arguments
bound_args.update(kwargs)
# Check if we have all required parameters
missing_params = set(param_names) - set(bound_args.keys())
if not missing_params:
return func(**bound_args)
# Return partial function for missing parameters
def partial_curry(*new_args, **new_kwargs):
combined_args = args + new_args
combined_kwargs = {**kwargs, **new_kwargs}
return curried(*combined_args, **combined_kwargs)
return partial_curry
return curried
@advanced_curry
def create_user(name, email, age, city="Unknown", active=True):
"""Create a user with various parameters"""
return {
'name': name,
'email': email,
'age': age,
'city': city,
'active': active
}
# Test advanced currying
print("Advanced currying with keywords:")
# Traditional call
user1 = create_user("Alice", "alice@email.com", 30, city="New York")
print(f"User 1: {user1}")
# Curried calls
create_bob = create_user("Bob")
create_bob_with_email = create_bob("bob@email.com")
user2 = create_bob_with_email(25, active=False)
print(f"User 2: {user2}")
# Mixed positional and keyword
create_charlie = create_user("Charlie", age=35)
user3 = create_charlie("charlie@email.com", city="London")
print(f"User 3: {user3}")
# 4. Practical currying applications
print(f"\n4๏ธโฃ PRACTICAL CURRYING APPLICATIONS:")
# Database query builder
@curry
def build_query(table, columns, where_clause, limit=None):
"""Build a SQL query"""
query = f"SELECT {', '.join(columns)} FROM {table}"
if where_clause:
query += f" WHERE {where_clause}"
if limit:
query += f" LIMIT {limit}"
return query
# Create specialized query builders
users_query = build_query("users")
active_users_query = users_query(["id", "name", "email"])
limited_active_users = active_users_query("active = 1", 10)
print("SQL Query Builder:")
print(f"Query: {limited_active_users}")
# API endpoint creator
@curry
def create_api_endpoint(base_url, version, resource, action):
"""Create API endpoint URL"""
return f"{base_url}/v{version}/{resource}/{action}"
# Specialized endpoint creators
api_v1 = create_api_endpoint("https://api.example.com", 1)
user_api = api_v1("users")
print(f"\nAPI Endpoints:")
print(f"Get users: {user_api('list')}")
print(f"Create user: {user_api('create')}")
print(f"Update user: {user_api('update')}")
# Configuration builder
@curry
def create_config(env, database_url, redis_url, debug_mode):
"""Create application configuration"""
return {
'environment': env,
'database': database_url,
'cache': redis_url,
'debug': debug_mode
}
# Environment-specific configurations
dev_config = create_config("development")
prod_config = create_config("production")
dev_with_db = dev_config("sqlite:///dev.db")
prod_with_db = prod_config("postgresql://prod-server/db")
dev_complete = dev_with_db("redis://localhost:6379", True)
prod_complete = prod_with_db("redis://prod-cache:6379", False)
print(f"\nConfigurations:")
print(f"Dev config: {dev_complete}")
print(f"Prod config: {prod_complete}")
# 5. Currying for functional composition
print(f"\n5๏ธโฃ CURRYING FOR COMPOSITION:")
@curry
def map_curry(func, iterable):
"""Curried map function"""
return list(map(func, iterable))
@curry
def filter_curry(predicate, iterable):
"""Curried filter function"""
return list(filter(predicate, iterable))
@curry
def reduce_curry(func, iterable, initial=None):
"""Curried reduce function"""
if initial is not None:
return reduce(func, iterable, initial)
return reduce(func, iterable)
# Create specialized functions
double_all = map_curry(lambda x: x * 2)
keep_evens = filter_curry(lambda x: x % 2 == 0)
sum_all = reduce_curry(lambda a, b: a + b, initial=0)
# Compose operations
def compose_operations(data):
"""Compose multiple operations"""
doubled = double_all(data)
evens = keep_evens(doubled)
total = sum_all(evens)
return total
# Test composition
test_data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print(f"Original data: {test_data}")
result = compose_operations(test_data)
print(f"After doubleโfilter_evensโsum: {result}")
# Step by step to see intermediate results
doubled = double_all(test_data)
print(f"After doubling: {doubled}")
evens = keep_evens(doubled)
print(f"After filtering evens: {evens}")
total = sum_all(evens)
print(f"Final sum: {total}")Python๐ฏ Advanced Patterns and Techniques
"""
๐ฏ ADVANCED PATTERNS AND TECHNIQUES
Sophisticated uses of partial application and currying
"""
import time
from datetime import datetime
from functools import wraps
print("\n๐ฏ ADVANCED PATTERNS AND TECHNIQUES")
print("=" * 40)
# 1. Partial application with decorators
print("1๏ธโฃ PARTIAL APPLICATION WITH DECORATORS:")
def with_timing(func):
"""Decorator that adds timing to function calls"""
@wraps(func)
def wrapper(*args, **kwargs):
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"โฑ๏ธ {func.__name__} took {end_time - start_time:.4f} seconds")
return result
return wrapper
def create_validator(validation_func, error_message):
"""Create a validation decorator"""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
if not validation_func(*args, **kwargs):
raise ValueError(error_message)
return func(*args, **kwargs)
return wrapper
return decorator
# Specialized validators using partial application
positive_validator = partial(
create_validator,
lambda x: x > 0,
"Value must be positive"
)
non_empty_string_validator = partial(
create_validator,
lambda s: isinstance(s, str) and len(s.strip()) > 0,
"String must not be empty"
)
@with_timing
@positive_validator
def calculate_square_root(x):
"""Calculate square root with validation and timing"""
return x ** 0.5
@with_timing
@non_empty_string_validator
def process_name(name):
"""Process a name with validation and timing"""
return name.strip().title()
# Test decorated functions
print("Testing decorated functions:")
try:
result1 = calculate_square_root(16)
print(f"sqrt(16) = {result1}")
result2 = process_name(" alice smith ")
print(f"Processed name: '{result2}'")
# This will raise an error
result3 = calculate_square_root(-4)
except ValueError as e:
print(f"โ Validation error: {e}")
# 2. Curried function composition
print(f"\n2๏ธโฃ CURRIED FUNCTION COMPOSITION:")
def pipe(*functions):
"""Compose functions left-to-right using curried functions"""
def compose_two(f, g):
return lambda x: g(f(x))
return reduce(compose_two, functions, lambda x: x)
# Curried string processing functions
@curry
def replace_chars(old_char, new_char, text):
"""Replace characters in text"""
return text.replace(old_char, new_char)
@curry
def add_prefix(prefix, text):
"""Add prefix to text"""
return prefix + text
@curry
def add_suffix(suffix, text):
"""Add suffix to text"""
return text + suffix
# Create specialized processors
replace_spaces = replace_chars(" ", "_")
add_processed_prefix = add_prefix("processed_")
add_txt_suffix = add_suffix(".txt")
# Create processing pipeline
text_pipeline = pipe(
str.lower,
replace_spaces,
add_processed_prefix,
add_txt_suffix
)
# Test pipeline
test_text = "My Important Document"
result = text_pipeline(test_text)
print(f"Text processing pipeline:")
print(f"'{test_text}' โ '{result}'")
# 3. Memoization with partial application
print(f"\n3๏ธโฃ MEMOIZATION WITH PARTIAL:")
def memoize_with_key(key_func):
"""Memoization decorator with custom key function"""
def decorator(func):
cache = {}
@wraps(func)
def wrapper(*args, **kwargs):
key = key_func(*args, **kwargs)
if key not in cache:
print(f"๐ Computing {func.__name__} for key: {key}")
cache[key] = func(*args, **kwargs)
else:
print(f"๐พ Cache hit for {func.__name__} with key: {key}")
return cache[key]
wrapper.cache = cache
wrapper.clear_cache = lambda: cache.clear()
return wrapper
return decorator
# Specialized memoization decorators
simple_memo = memoize_with_key(lambda *args, **kwargs: args)
rounded_memo = memoize_with_key(lambda x: round(x, 2)) # Round to 2 decimal places
@simple_memo
def factorial(n):
"""Calculate factorial"""
if n <= 1:
return 1
return n * factorial(n - 1)
@rounded_memo
def expensive_calculation(x):
"""Expensive calculation that we want to memoize by rounded value"""
time.sleep(0.1) # Simulate expensive computation
return x ** 3 + x ** 2 + x
# Test memoization
print("Memoization examples:")
print(f"factorial(5) = {factorial(5)}")
print(f"factorial(5) = {factorial(5)}") # Cache hit
print(f"expensive_calculation(2.1) = {expensive_calculation(2.1)}")
print(f"expensive_calculation(2.12) = {expensive_calculation(2.12)}") # Cache hit due to rounding
# 4. Event system with curried handlers
print(f"\n4๏ธโฃ EVENT SYSTEM WITH CURRIED HANDLERS:")
class EventSystem:
"""Simple event system using curried handlers"""
def __init__(self):
self.handlers = {}
def register(self, event_type, handler):
"""Register an event handler"""
if event_type not in self.handlers:
self.handlers[event_type] = []
self.handlers[event_type].append(handler)
def emit(self, event_type, event_data):
"""Emit an event"""
if event_type in self.handlers:
for handler in self.handlers[event_type]:
handler(event_data)
@curry
def log_event(log_level, event_type, event_data):
"""Curried event logger"""
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
print(f"[{timestamp}] {log_level}: {event_type} - {event_data}")
@curry
def save_to_file(filename, event_type, event_data):
"""Curried file saver (simulated)"""
print(f"๐ Saving to {filename}: {event_type} = {event_data}")
@curry
def send_notification(notification_type, event_type, event_data):
"""Curried notification sender"""
print(f"๐ฑ {notification_type} notification: {event_type} occurred with data: {event_data}")
# Create event system and specialized handlers
event_system = EventSystem()
# Create specialized handlers
debug_logger = log_event("DEBUG")
error_logger = log_event("ERROR")
info_logger = log_event("INFO")
error_file_saver = save_to_file("errors.log")
user_file_saver = save_to_file("users.log")
email_notifier = send_notification("EMAIL")
sms_notifier = send_notification("SMS")
# Register handlers for different events
event_system.register("user_login", debug_logger)
event_system.register("user_login", user_file_saver)
event_system.register("system_error", error_logger)
event_system.register("system_error", error_file_saver)
event_system.register("system_error", email_notifier)
event_system.register("critical_error", error_logger)
event_system.register("critical_error", error_file_saver)
event_system.register("critical_error", email_notifier)
event_system.register("critical_error", sms_notifier)
# Emit some events
print("Event system in action:")
event_system.emit("user_login", {"user_id": 123, "username": "alice"})
print()
event_system.emit("system_error", {"error_code": 500, "message": "Database connection failed"})
print()
event_system.emit("critical_error", {"error_code": 503, "message": "Service unavailable"})
# 5. Configuration management with currying
print(f"\n5๏ธโฃ CONFIGURATION MANAGEMENT:")
@curry
def create_database_connection(host, port, database, username, password):
"""Create database connection string"""
return f"postgresql://{username}:{password}@{host}:{port}/{database}"
@curry
def create_cache_config(host, port, db_number, ttl):
"""Create cache configuration"""
return {
'host': host,
'port': port,
'db': db_number,
'ttl': ttl
}
@curry
def create_app_config(env, debug, database_config, cache_config):
"""Create complete application configuration"""
return {
'environment': env,
'debug': debug,
'database': database_config,
'cache': cache_config
}
# Build configurations for different environments
# Production database
prod_db = create_database_connection("prod-db.example.com", 5432, "myapp_prod")
prod_db_with_user = prod_db("prod_user", "secure_password")
# Development database
dev_db = create_database_connection("localhost", 5432, "myapp_dev", "dev_user", "dev_password")
# Cache configurations
prod_cache = create_cache_config("prod-redis.example.com", 6379, 0, 3600)
dev_cache = create_cache_config("localhost", 6379, 1, 300)
# Complete application configurations
create_prod_config = create_app_config("production", False)
create_dev_config = create_app_config("development", True)
prod_config = create_prod_config(prod_db_with_user, prod_cache)
dev_config = create_dev_config(dev_db, dev_cache)
print("Configuration management:")
print(f"Production config: {prod_config}")
print(f"Development config: {dev_config}")
# 6. API client with curried methods
print(f"\n6๏ธโฃ API CLIENT WITH CURRIED METHODS:")
@curry
def make_request(method, base_url, endpoint, headers, params=None):
"""Make an HTTP request (simulated)"""
url = f"{base_url.rstrip('/')}/{endpoint.lstrip('/')}"
request_info = {
'method': method,
'url': url,
'headers': headers,
'params': params or {}
}
# Simulate API call
print(f"๐ Making {method} request to {url}")
if params:
print(f" Parameters: {params}")
print(f" Headers: {headers}")
return f"Response from {method} {url}"
# Create specialized API methods
api_base = "https://api.example.com"
default_headers = {"Authorization": "Bearer token123", "Content-Type": "application/json"}
# Partially apply method and base configuration
make_get = make_request("GET", api_base)
make_post = make_request("POST", api_base)
make_put = make_request("PUT", api_base)
# Further specialize with headers
api_get = make_get(headers=default_headers)
api_post = make_post(headers=default_headers)
# Create specific endpoint methods
get_users = api_get("users")
get_user_by_id = api_get("users") # Will need params for specific user
create_user = api_post("users")
# Use the API methods
print("API client in action:")
response1 = get_users()
print(f"Response: {response1}\n")
response2 = get_user_by_id(params={"id": 123})
print(f"Response: {response2}\n")
response3 = create_user(params={"name": "Alice", "email": "alice@example.com"})
print(f"Response: {response3}")
print("\n๐ฏ ADVANCED PATTERNS SUMMARY:")
print("โ
Partial application creates specialized, reusable functions")
print("โ
Currying enables elegant function composition")
print("โ
Both techniques improve code modularity and readability")
print("โ
Excellent for configuration, event handling, and API design")
print("โ
Powerful when combined with decorators and higher-order functions")
print("โ
Essential tools for functional programming in Python")Python๐ ๏ธ Real-World Applications
"""
๐ ๏ธ REAL-WORLD APPLICATIONS
Practical uses of partial application and currying
"""
import json
import re
from datetime import datetime, timedelta
from typing import Dict, List, Any
print("\n๐ ๏ธ REAL-WORLD APPLICATIONS")
print("=" * 35)
# 1. Form validation system
print("1๏ธโฃ FORM VALIDATION SYSTEM:")
@curry
def validate_field(validator_func, error_message, field_name, value):
"""Generic field validator"""
if not validator_func(value):
return {'field': field_name, 'error': error_message}
return None
@curry
def validate_length(min_length, max_length, value):
"""Validate string length"""
if not isinstance(value, str):
return False
return min_length <= len(value.strip()) <= max_length
@curry
def validate_pattern(pattern, value):
"""Validate using regex pattern"""
if not isinstance(value, str):
return False
return bool(re.match(pattern, value))
@curry
def validate_range(min_val, max_val, value):
"""Validate numeric range"""
try:
num = float(value)
return min_val <= num <= max_val
except (ValueError, TypeError):
return False
# Create specialized validators
validate_name = validate_field(
validate_length(2, 50),
"Name must be between 2 and 50 characters"
)
validate_email = validate_field(
validate_pattern(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'),
"Invalid email format"
)
validate_age = validate_field(
validate_range(0, 120),
"Age must be between 0 and 120"
)
validate_phone = validate_field(
validate_pattern(r'^\+?1?-?\d{3}-?\d{3}-?\d{4}$'),
"Invalid phone number format"
)
def validate_form(form_data):
"""Validate entire form"""
validators = [
validate_name("name"),
validate_email("email"),
validate_age("age"),
validate_phone("phone")
]
errors = []
for validator in validators:
field_name = validator.__name__.split('_')[-1]
if field_name in form_data:
error = validator(form_data[field_name])
if error:
errors.append(error)
else:
errors.append({'field': field_name, 'error': f'{field_name.title()} is required'})
return errors
# Test form validation
test_forms = [
{
'name': 'Alice Smith',
'email': 'alice@example.com',
'age': '30',
'phone': '555-123-4567'
},
{
'name': 'A', # Too short
'email': 'invalid-email', # Invalid format
'age': '150', # Too old
'phone': '123' # Invalid format
},
{
'name': 'Bob Johnson',
'email': 'bob@test.com',
# Missing age and phone
}
]
print("Form validation results:")
for i, form in enumerate(test_forms, 1):
print(f"\nForm {i}: {form}")
errors = validate_form(form)
if errors:
print("โ Validation errors:")
for error in errors:
print(f" - {error['field']}: {error['error']}")
else:
print("โ
Form is valid")
# 2. Database query builder
print(f"\n2๏ธโฃ DATABASE QUERY BUILDER:")
@curry
def select_query(columns, table, where_conditions=None, order_by=None, limit=None):
"""Build SELECT query"""
query = f"SELECT {', '.join(columns)} FROM {table}"
if where_conditions:
conditions = [f"{k} = '{v}'" for k, v in where_conditions.items()]
query += f" WHERE {' AND '.join(conditions)}"
if order_by:
query += f" ORDER BY {order_by}"
if limit:
query += f" LIMIT {limit}"
return query
@curry
def insert_query(table, data):
"""Build INSERT query"""
columns = ', '.join(data.keys())
values = ', '.join([f"'{v}'" for v in data.values()])
return f"INSERT INTO {table} ({columns}) VALUES ({values})"
@curry
def update_query(table, data, where_conditions):
"""Build UPDATE query"""
set_clauses = [f"{k} = '{v}'" for k, v in data.items()]
where_clauses = [f"{k} = '{v}'" for k, v in where_conditions.items()]
query = f"UPDATE {table} SET {', '.join(set_clauses)}"
query += f" WHERE {' AND '.join(where_clauses)}"
return query
# Create table-specific query builders
users_select = select_query(columns=["id", "name", "email"])
products_select = select_query(columns=["id", "name", "price", "category"])
users_insert = insert_query("users")
products_insert = insert_query("products")
users_update = update_query("users")
# Build specific queries
queries = [
users_select("users", where_conditions={"active": "1"}, order_by="name", limit=10),
products_select("products", where_conditions={"category": "electronics"}, order_by="price DESC"),
users_insert({"name": "Alice", "email": "alice@example.com", "active": "1"}),
users_update({"email": "newalice@example.com"}, {"id": "123"})
]
print("Generated SQL queries:")
for i, query in enumerate(queries, 1):
print(f"{i}. {query}")
# 3. HTTP client with method specialization
print(f"\n3๏ธโฃ HTTP CLIENT:")
@curry
def http_request(method, base_url, headers, endpoint, params=None, data=None):
"""Generic HTTP request function"""
url = f"{base_url.rstrip('/')}/{endpoint.lstrip('/')}"
request_details = {
'method': method,
'url': url,
'headers': headers
}
if params:
request_details['params'] = params
if data:
request_details['data'] = data
# Simulate making the request
print(f"๐ {method} {url}")
if params:
print(f" Query params: {params}")
if data:
print(f" Request data: {data}")
return f"Mock response from {method} {url}"
# Create API client for specific service
api_base = "https://jsonplaceholder.typicode.com"
default_headers = {
"Content-Type": "application/json",
"User-Agent": "MyApp/1.0"
}
# Create method-specific functions
make_get = http_request("GET", api_base, default_headers)
make_post = http_request("POST", api_base, default_headers)
make_put = http_request("PUT", api_base, default_headers)
make_delete = http_request("DELETE", api_base, default_headers)
# Create resource-specific functions
get_posts = make_get("posts")
get_users = make_get("users")
create_post = make_post("posts")
update_post = make_put("posts/1") # Specific post
# Use the specialized functions
print("HTTP client in action:")
response1 = get_posts(params={"userId": 1})
print(f"Response: {response1[:50]}...\n")
response2 = create_post(data={"title": "New Post", "body": "Content", "userId": 1})
print(f"Response: {response2[:50]}...\n")
# 4. Configuration management system
print(f"\n4๏ธโฃ CONFIGURATION MANAGEMENT:")
@curry
def create_config_section(section_name, defaults, overrides=None):
"""Create configuration section with defaults and overrides"""
config = defaults.copy()
if overrides:
config.update(overrides)
return {section_name: config}
@curry
def merge_configs(*config_dicts):
"""Merge multiple configuration dictionaries"""
merged = {}
for config in config_dicts:
for section, values in config.items():
if section not in merged:
merged[section] = {}
merged[section].update(values)
return merged
# Default configurations
database_defaults = {
'host': 'localhost',
'port': 5432,
'ssl': False,
'pool_size': 10
}
cache_defaults = {
'host': 'localhost',
'port': 6379,
'db': 0,
'ttl': 300
}
app_defaults = {
'debug': False,
'log_level': 'INFO',
'secret_key': 'default-secret'
}
# Create configuration builders
create_db_config = create_config_section('database', database_defaults)
create_cache_config = create_config_section('cache', cache_defaults)
create_app_config = create_config_section('application', app_defaults)
# Environment-specific configurations
dev_overrides = {
'database': {'host': 'dev-db.local', 'ssl': False, 'debug': True},
'cache': {'db': 1},
'application': {'debug': True, 'log_level': 'DEBUG'}
}
prod_overrides = {
'database': {'host': 'prod-db.example.com', 'ssl': True, 'pool_size': 50},
'cache': {'host': 'prod-cache.example.com', 'ttl': 3600},
'application': {'secret_key': 'production-secret-key'}
}
# Build configurations
base_config = merge_configs(
create_db_config(),
create_cache_config(),
create_app_config()
)
dev_config = merge_configs(
base_config,
create_db_config(dev_overrides.get('database', {})),
create_cache_config(dev_overrides.get('cache', {})),
create_app_config(dev_overrides.get('application', {}))
)
prod_config = merge_configs(
base_config,
create_db_config(prod_overrides.get('database', {})),
create_cache_config(prod_overrides.get('cache', {})),
create_app_config(prod_overrides.get('application', {}))
)
print("Configuration management:")
print("Base config:", json.dumps(base_config, indent=2))
print("\nDev config:", json.dumps(dev_config, indent=2))
print("\nProd config:", json.dumps(prod_config, indent=2))
# 5. Event processing pipeline
print(f"\n5๏ธโฃ EVENT PROCESSING PIPELINE:")
@curry
def filter_events(condition, events):
"""Filter events based on condition"""
return [event for event in events if condition(event)]
@curry
def transform_events(transformer, events):
"""Transform events using transformer function"""
return [transformer(event) for event in events]
@curry
def aggregate_events(key_func, agg_func, events):
"""Aggregate events by key"""
groups = {}
for event in events:
key = key_func(event)
if key not in groups:
groups[key] = []
groups[key].append(event)
return {key: agg_func(group) for key, group in groups.items()}
# Sample events
events = [
{'type': 'user_login', 'user_id': 1, 'timestamp': '2024-01-01T10:00:00', 'success': True},
{'type': 'user_login', 'user_id': 2, 'timestamp': '2024-01-01T10:01:00', 'success': False},
{'type': 'purchase', 'user_id': 1, 'timestamp': '2024-01-01T10:05:00', 'amount': 99.99},
{'type': 'user_login', 'user_id': 1, 'timestamp': '2024-01-01T10:10:00', 'success': True},
{'type': 'purchase', 'user_id': 2, 'timestamp': '2024-01-01T10:15:00', 'amount': 149.99},
]
# Create specialized processors
filter_successful = filter_events(lambda e: e.get('success', True))
filter_purchases = filter_events(lambda e: e['type'] == 'purchase')
filter_logins = filter_events(lambda e: e['type'] == 'user_login')
add_hour = transform_events(lambda e: {
**e,
'hour': datetime.fromisoformat(e['timestamp']).hour
})
aggregate_by_user = aggregate_events(
lambda e: e['user_id'],
lambda group: {'count': len(group), 'types': list(set(e['type'] for e in group))}
)
aggregate_purchase_totals = aggregate_events(
lambda e: e['user_id'],
lambda group: {'total_spent': sum(e['amount'] for e in group)}
)
# Process events
print("Event processing pipeline:")
print(f"Total events: {len(events)}")
successful_events = filter_successful(events)
print(f"Successful events: {len(successful_events)}")
purchases = filter_purchases(events)
print(f"Purchase events: {len(purchases)}")
purchase_totals = aggregate_purchase_totals(purchases)
print(f"Purchase totals by user: {purchase_totals}")
events_with_hour = add_hour(events)
user_activity = aggregate_by_user(events_with_hour)
print(f"User activity summary: {user_activity}")
print("\n๐ฏ REAL-WORLD APPLICATIONS SUMMARY:")
print("โ
Form validation with reusable field validators")
print("โ
Database query builders with method specialization")
print("โ
HTTP clients with endpoint-specific functions")
print("โ
Configuration management with environment overrides")
print("โ
Event processing pipelines with composable operations")
print("โ
All examples demonstrate code reuse and modularity")PythonThis completes the comprehensive section on Partial Application and Currying. The content covers:
- Partial Application Fundamentals – Using functools.partial and custom implementations
- Currying Deep Dive – Manual and automatic currying with practical examples
- Advanced Patterns – Combining with decorators, composition, and memoization
- Real-World Applications – Form validation, query builders, HTTP clients, configuration management, and event processing
The partial application and currying techniques shown here enable the creation of specialized, reusable functions that improve code modularity, readability, and maintainability while following functional programming principles.
12. Monads and Advanced Patterns
๐ฎ Understanding Monads
Monads are one of the most powerful and abstract concepts in functional programming. They provide a structured way to handle computations with context, such as handling null values, errors, asynchronous operations, or stateful computations. A monad must satisfy three laws and provide two essential operations: bind (flatMap) and return (unit).
graph TD
A["๐ฎ Monads"] --> B["๐ฆ Container Pattern<br/><i>Wrap Values with Context</i>"]
A --> C["๐ Bind Operation<br/><i>Chain Computations</i>"]
A --> D["โฉ๏ธ Return Operation<br/><i>Wrap Pure Values</i>"]
B --> B1["Maybe<T>"]
B --> B2["Either<E, T>"]
B --> B3["IO<T>"]
C --> C1["flatMap / >>= operator"]
C --> C2["Composable operations"]
C --> C3["Context preservation"]
D --> D1["Pure value wrapper"]
D --> D2["Minimal context"]
D3["Identity function"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐งฎ Monad Laws and Implementation
"""
๐งฎ MONAD LAWS AND IMPLEMENTATION
Understanding the fundamental structure of monads
"""
from abc import ABC, abstractmethod
from typing import TypeVar, Generic, Callable, Any, Union
import inspect
T = TypeVar('T')
U = TypeVar('U')
E = TypeVar('E')
print("๐ฎ MONAD FUNDAMENTALS")
print("=" * 25)
# 1. Abstract Monad Base Class
print("1๏ธโฃ ABSTRACT MONAD:")
class Monad(Generic[T], ABC):
"""Abstract base class for all monads"""
@abstractmethod
def bind(self, func: Callable[[T], 'Monad[U]']) -> 'Monad[U]':
"""Bind operation (flatMap) - the heart of monads"""
pass
@classmethod
@abstractmethod
def unit(cls, value: T) -> 'Monad[T]':
"""Return operation - wrap a pure value in the monad"""
pass
def map(self, func: Callable[[T], U]) -> 'Monad[U]':
"""Map operation - can be derived from bind and unit"""
return self.bind(lambda x: self.__class__.unit(func(x)))
def __rshift__(self, func):
""">> operator for bind (Haskell-style)"""
return self.bind(func)
# 2. Identity Monad (simplest monad)
print("2๏ธโฃ IDENTITY MONAD:")
class Identity(Monad[T]):
"""The Identity monad - simplest possible monad"""
def __init__(self, value: T):
self._value = value
def bind(self, func: Callable[[T], 'Identity[U]']) -> 'Identity[U]':
"""Apply function that returns an Identity"""
return func(self._value)
@classmethod
def unit(cls, value: T) -> 'Identity[T]':
"""Wrap value in Identity monad"""
return cls(value)
@property
def value(self) -> T:
"""Get the wrapped value"""
return self._value
def __str__(self):
return f"Identity({self._value})"
def __repr__(self):
return self.__str__()
# Test Identity monad
print("Testing Identity monad:")
id1 = Identity.unit(5)
print(f"Identity(5): {id1}")
# Chain operations
id2 = id1.bind(lambda x: Identity.unit(x * 2))
print(f"After doubling: {id2}")
id3 = id2.bind(lambda x: Identity.unit(x + 10))
print(f"After adding 10: {id3}")
# Using map
id4 = Identity.unit(3).map(lambda x: x ** 2)
print(f"Squared: {id4}")
# 3. Monad Laws Verification
print(f"\n3๏ธโฃ MONAD LAWS:")
def verify_monad_laws():
"""Verify the three monad laws with Identity monad"""
print("Verifying monad laws with Identity:")
# Law 1: Left Identity
# unit(a).bind(f) == f(a)
a = 42
f = lambda x: Identity.unit(x * 2)
left_side = Identity.unit(a).bind(f)
right_side = f(a)
print(f"Left Identity: {left_side.value == right_side.value} "
f"({left_side} == {right_side})")
# Law 2: Right Identity
# m.bind(unit) == m
m = Identity.unit(42)
left_side = m.bind(Identity.unit)
right_side = m
print(f"Right Identity: {left_side.value == right_side.value} "
f"({left_side} == {right_side})")
# Law 3: Associativity
# m.bind(f).bind(g) == m.bind(lambda x: f(x).bind(g))
g = lambda x: Identity.unit(x + 10)
left_side = m.bind(f).bind(g)
right_side = m.bind(lambda x: f(x).bind(g))
print(f"Associativity: {left_side.value == right_side.value} "
f"({left_side} == {right_side})")
verify_monad_laws()
# 4. Maybe Monad Implementation
print(f"\n4๏ธโฃ MAYBE MONAD:")
class Maybe(Monad[T]):
"""Maybe monad for handling null values"""
def __init__(self, value: T = None):
self._value = value
self._is_nothing = value is None
def bind(self, func: Callable[[T], 'Maybe[U]']) -> 'Maybe[U]':
"""Bind operation for Maybe"""
if self._is_nothing:
return Nothing()
try:
result = func(self._value)
return result if isinstance(result, Maybe) else Maybe.unit(result)
except Exception:
return Nothing()
@classmethod
def unit(cls, value: T) -> 'Maybe[T]':
"""Create Some or Nothing based on value"""
return Some(value) if value is not None else Nothing()
def is_some(self) -> bool:
"""Check if this contains a value"""
return not self._is_nothing
def is_nothing(self) -> bool:
"""Check if this is Nothing"""
return self._is_nothing
def get_or_else(self, default: T) -> T:
"""Get value or return default"""
return self._value if not self._is_nothing else default
class Some(Maybe[T]):
"""Some variant of Maybe - contains a value"""
def __init__(self, value: T):
if value is None:
raise ValueError("Some cannot contain None")
super().__init__(value)
def __str__(self):
return f"Some({self._value})"
class Nothing(Maybe[T]):
"""Nothing variant of Maybe - contains no value"""
def __init__(self):
super().__init__(None)
def __str__(self):
return "Nothing"
# Test Maybe monad
print("Testing Maybe monad:")
# Safe division chain
def safe_divide(x: float, y: float) -> Maybe[float]:
return Maybe.unit(x / y) if y != 0 else Nothing()
def safe_sqrt(x: float) -> Maybe[float]:
return Maybe.unit(x ** 0.5) if x >= 0 else Nothing()
# Chain safe operations
result1 = (Maybe.unit(16)
.bind(lambda x: safe_divide(x, 4)) # 16/4 = 4
.bind(lambda x: safe_sqrt(x)) # sqrt(4) = 2
.bind(lambda x: safe_divide(x, 2))) # 2/2 = 1
print(f"Safe computation (16/4 โ sqrt โ /2): {result1}")
print(f"Final value: {result1.get_or_else('Error')}")
# Chain with error
result2 = (Maybe.unit(16)
.bind(lambda x: safe_divide(x, 0)) # Division by zero
.bind(lambda x: safe_sqrt(x))) # Won't execute
print(f"With division by zero: {result2}")
print(f"Final value: {result2.get_or_else('Error')}")
# 5. Either Monad Implementation
print(f"\n5๏ธโฃ EITHER MONAD:")
class Either(Monad[T]):
"""Either monad for error handling with detailed error information"""
def __init__(self, value: T = None, error: Any = None, is_right: bool = True):
self._value = value
self._error = error
self._is_right = is_right
def bind(self, func: Callable[[T], 'Either[U]']) -> 'Either[U]':
"""Bind operation for Either"""
if not self._is_right:
return Left(self._error)
try:
result = func(self._value)
return result if isinstance(result, Either) else Right(result)
except Exception as e:
return Left(str(e))
@classmethod
def unit(cls, value: T) -> 'Either[T]':
"""Create Right with value"""
return Right(value)
def is_right(self) -> bool:
"""Check if this is Right (success)"""
return self._is_right
def is_left(self) -> bool:
"""Check if this is Left (error)"""
return not self._is_right
def get_or_else(self, default: T) -> T:
"""Get value or return default"""
return self._value if self._is_right else default
def get_error(self) -> Any:
"""Get error value"""
return self._error if not self._is_right else None
class Right(Either[T]):
"""Right variant of Either - contains success value"""
def __init__(self, value: T):
super().__init__(value=value, is_right=True)
def __str__(self):
return f"Right({self._value})"
class Left(Either[T]):
"""Left variant of Either - contains error"""
def __init__(self, error: Any):
super().__init__(error=error, is_right=False)
def __str__(self):
return f"Left({self._error})"
# Test Either monad
print("Testing Either monad:")
def safe_divide_either(x: float, y: float) -> Either[float]:
if y == 0:
return Left("Division by zero")
return Right(x / y)
def safe_sqrt_either(x: float) -> Either[float]:
if x < 0:
return Left("Square root of negative number")
return Right(x ** 0.5)
def safe_log_either(x: float) -> Either[float]:
if x <= 0:
return Left("Logarithm of non-positive number")
import math
return Right(math.log(x))
# Chain operations with detailed error handling
computation = (Either.unit(16)
.bind(lambda x: safe_divide_either(x, 4))
.bind(safe_sqrt_either)
.bind(safe_log_either))
print(f"Safe computation with Either: {computation}")
if computation.is_right():
print(f"Result: {computation.get_or_else(0):.4f}")
else:
print(f"Error: {computation.get_error()}")
# Test error propagation
error_computation = (Either.unit(16)
.bind(lambda x: safe_divide_either(x, 0)) # This will fail
.bind(safe_sqrt_either) # Won't execute
.bind(safe_log_either)) # Won't execute
print(f"Error computation: {error_computation}")
print(f"Error details: {error_computation.get_error()}")Python๐ State Monad and Stateful Computations
"""
๐ STATE MONAD
Managing stateful computations functionally
"""
from typing import Tuple, Callable
from dataclasses import dataclass
print("\n๐ STATE MONAD")
print("=" * 20)
# 1. State Monad Implementation
print("1๏ธโฃ STATE MONAD IMPLEMENTATION:")
S = TypeVar('S') # State type
class State(Monad[T]):
"""State monad for stateful computations"""
def __init__(self, run_state: Callable[[S], Tuple[T, S]]):
self._run_state = run_state
def bind(self, func: Callable[[T], 'State[S, U]']) -> 'State[S, U]':
"""Bind operation for State monad"""
def run_state(initial_state: S) -> Tuple[U, S]:
# Run this computation
value, new_state = self._run_state(initial_state)
# Run the next computation with the new state
next_computation = func(value)
return next_computation._run_state(new_state)
return State(run_state)
@classmethod
def unit(cls, value: T) -> 'State[S, T]':
"""Return a value without changing state"""
return cls(lambda state: (value, state))
def run(self, initial_state: S) -> Tuple[T, S]:
"""Execute the stateful computation"""
return self._run_state(initial_state)
def eval(self, initial_state: S) -> T:
"""Execute and return only the value"""
value, _ = self._run_state(initial_state)
return value
def exec(self, initial_state: S) -> S:
"""Execute and return only the final state"""
_, final_state = self._run_state(initial_state)
return final_state
# State manipulation functions
def get_state() -> State[S, S]:
"""Get the current state"""
return State(lambda state: (state, state))
def put_state(new_state: S) -> State[S, None]:
"""Set the state"""
return State(lambda _: (None, new_state))
def modify_state(func: Callable[[S], S]) -> State[S, None]:
"""Modify the state with a function"""
return State(lambda state: (None, func(state)))
# 2. Counter Example
print("2๏ธโฃ COUNTER EXAMPLE:")
@dataclass
class Counter:
"""Simple counter state"""
value: int = 0
def increment(self):
return Counter(self.value + 1)
def add(self, n):
return Counter(self.value + n)
def increment() -> State[Counter, int]:
"""Increment counter and return new value"""
return (get_state()
.bind(lambda counter: put_state(counter.increment()))
.bind(lambda _: get_state())
.map(lambda counter: counter.value))
def add_to_counter(n: int) -> State[Counter, int]:
"""Add to counter and return new value"""
return (get_state()
.bind(lambda counter: put_state(counter.add(n)))
.bind(lambda _: get_state())
.map(lambda counter: counter.value))
def get_counter_value() -> State[Counter, int]:
"""Get current counter value"""
return get_state().map(lambda counter: counter.value)
# Test counter operations
print("Testing counter operations:")
# Chain counter operations
counter_computation = (increment()
.bind(lambda _: add_to_counter(5))
.bind(lambda _: increment())
.bind(lambda _: get_counter_value()))
initial_counter = Counter(0)
final_value, final_state = counter_computation.run(initial_counter)
print(f"Initial state: {initial_counter}")
print(f"Final value: {final_value}")
print(f"Final state: {final_state}")
# 3. Random Number Generator Example
print(f"\n3๏ธโฃ RANDOM NUMBER GENERATOR:")
import random
from typing import List
@dataclass
class RandomState:
"""Random number generator state"""
seed: int
def next_random(self) -> Tuple[int, 'RandomState']:
"""Generate next random number and new state"""
# Simple linear congruential generator
next_seed = (self.seed * 1103515245 + 12345) & 0x7fffffff
return next_seed % 100, RandomState(next_seed)
def next_random() -> State[RandomState, int]:
"""Get next random number"""
def run_state(state: RandomState) -> Tuple[int, RandomState]:
return state.next_random()
return State(run_state)
def random_list(n: int) -> State[RandomState, List[int]]:
"""Generate list of n random numbers"""
if n <= 0:
return State.unit([])
return (next_random()
.bind(lambda first: random_list(n - 1)
.map(lambda rest: [first] + rest)))
# Test random number generation
print("Testing random number generation:")
random_computation = random_list(5)
initial_random_state = RandomState(42) # Fixed seed for reproducibility
numbers, final_random_state = random_computation.run(initial_random_state)
print(f"Generated numbers: {numbers}")
print(f"Final random state seed: {final_random_state.seed}")
# Generate another batch with same initial seed for consistency
numbers2, _ = random_list(5).run(RandomState(42))
print(f"Same seed, same numbers: {numbers2}")
print(f"Deterministic: {numbers == numbers2}")
# 4. Bank Account Example
print(f"\n4๏ธโฃ BANK ACCOUNT EXAMPLE:")
@dataclass
class BankAccount:
"""Bank account state"""
balance: float
transaction_log: List[str]
def deposit(self, amount: float) -> 'BankAccount':
new_balance = self.balance + amount
new_log = self.transaction_log + [f"Deposit: +${amount:.2f}"]
return BankAccount(new_balance, new_log)
def withdraw(self, amount: float) -> 'BankAccount':
if amount > self.balance:
new_log = self.transaction_log + [f"Withdrawal failed: insufficient funds (${amount:.2f})"]
return BankAccount(self.balance, new_log)
new_balance = self.balance - amount
new_log = self.transaction_log + [f"Withdrawal: -${amount:.2f}"]
return BankAccount(new_balance, new_log)
def deposit(amount: float) -> State[BankAccount, bool]:
"""Deposit money to account"""
return (get_state()
.bind(lambda account: put_state(account.deposit(amount)))
.map(lambda _: True))
def withdraw(amount: float) -> State[BankAccount, bool]:
"""Withdraw money from account"""
def run_state(account: BankAccount) -> Tuple[bool, BankAccount]:
new_account = account.withdraw(amount)
success = new_account.balance < account.balance or amount == 0
return success, new_account
return State(run_state)
def get_balance() -> State[BankAccount, float]:
"""Get current balance"""
return get_state().map(lambda account: account.balance)
def get_transaction_log() -> State[BankAccount, List[str]]:
"""Get transaction log"""
return get_state().map(lambda account: account.transaction_log)
# Test bank account operations
print("Testing bank account operations:")
# Chain banking operations
banking_computation = (deposit(1000.0)
.bind(lambda _: withdraw(200.0))
.bind(lambda _: deposit(50.0))
.bind(lambda _: withdraw(1500.0)) # Should fail
.bind(lambda _: get_balance())
.bind(lambda balance: get_transaction_log()
.map(lambda log: (balance, log))))
initial_account = BankAccount(0.0, [])
(final_balance, transactions), final_account = banking_computation.run(initial_account)
print(f"Final balance: ${final_balance:.2f}")
print("Transaction log:")
for transaction in transactions:
print(f" - {transaction}")
print(f"Account state: {final_account}")Python๐ IO Monad and Effect Management
"""
๐ IO MONAD
Managing side effects and I/O operations functionally
"""
import sys
from typing import Iterator
from datetime import datetime
print("\n๐ IO MONAD")
print("=" * 15)
# 1. IO Monad Implementation
print("1๏ธโฃ IO MONAD IMPLEMENTATION:")
class IO(Monad[T]):
"""IO monad for managing side effects"""
def __init__(self, action: Callable[[], T]):
self._action = action
def bind(self, func: Callable[[T], 'IO[U]']) -> 'IO[U]':
"""Bind operation for IO monad"""
def action():
# Perform this IO action
value = self._action()
# Get next IO action and perform it
next_io = func(value)
return next_io._action()
return IO(action)
@classmethod
def unit(cls, value: T) -> 'IO[T]':
"""Wrap pure value in IO monad"""
return cls(lambda: value)
def run(self) -> T:
"""Execute the IO action"""
return self._action()
def __call__(self) -> T:
"""Execute the IO action (callable interface)"""
return self.run()
# IO utility functions
def io_print(message: str) -> IO[None]:
"""Print a message (side effect wrapped in IO)"""
return IO(lambda: print(message))
def io_input(prompt: str) -> IO[str]:
"""Get input from user (side effect wrapped in IO)"""
return IO(lambda: input(prompt))
def io_get_time() -> IO[datetime]:
"""Get current time (side effect wrapped in IO)"""
return IO(lambda: datetime.now())
def io_read_file(filename: str) -> IO[str]:
"""Read file content (simulated)"""
def read_action():
# Simulate file reading
if filename == "config.txt":
return "debug=true\nport=8080\ndatabase=postgresql"
elif filename == "data.txt":
return "line1\nline2\nline3"
else:
raise FileNotFoundError(f"File {filename} not found")
return IO(read_action)
def io_write_file(filename: str, content: str) -> IO[None]:
"""Write to file (simulated)"""
def write_action():
print(f"๐ Writing to {filename}:")
print(f" Content: {content}")
return None
return IO(write_action)
# 2. Simple IO Examples
print("2๏ธโฃ SIMPLE IO EXAMPLES:")
# Chain IO operations
greeting_program = (io_print("Welcome to the IO Monad demo!")
.bind(lambda _: io_get_time())
.bind(lambda time: io_print(f"Current time: {time}"))
.bind(lambda _: IO.unit("Program completed")))
print("Running greeting program:")
result = greeting_program.run()
print(f"Result: {result}")
# 3. File Processing Pipeline
print(f"\n3๏ธโฃ FILE PROCESSING PIPELINE:")
def parse_config(content: str) -> dict:
"""Parse configuration content"""
config = {}
for line in content.split('\n'):
if '=' in line:
key, value = line.split('=', 1)
config[key.strip()] = value.strip()
return config
def process_config_file() -> IO[dict]:
"""Read and process configuration file"""
return (io_read_file("config.txt")
.map(parse_config)
.bind(lambda config: io_print(f"๐ Loaded config: {config}")
.map(lambda _: config)))
def create_log_entry(config: dict) -> str:
"""Create log entry based on config"""
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
return f"[{timestamp}] Server started with config: {config}"
def config_and_log_program() -> IO[str]:
"""Complete program that processes config and creates log"""
return (process_config_file()
.bind(lambda config:
IO.unit(create_log_entry(config))
.bind(lambda log_entry:
io_write_file("server.log", log_entry)
.map(lambda _: log_entry))))
print("Running config and log program:")
log_entry = config_and_log_program().run()
print(f"Generated log entry: {log_entry}")
# 4. Interactive Program
print(f"\n4๏ธโฃ INTERACTIVE PROGRAM (simulated):")
def simulate_user_input(prompt: str) -> str:
"""Simulate user input for demo purposes"""
inputs = {
"Enter your name: ": "Alice",
"Enter your age: ": "25",
"Enter your city: ": "New York"
}
return inputs.get(prompt, "Unknown")
def io_simulated_input(prompt: str) -> IO[str]:
"""Simulated input for demo"""
def input_action():
result = simulate_user_input(prompt)
print(f"{prompt}{result}") # Show the simulated input
return result
return IO(input_action)
def validate_age(age_str: str) -> IO[int]:
"""Validate and parse age"""
def validate_action():
try:
age = int(age_str)
if 0 <= age <= 120:
return age
else:
raise ValueError("Age must be between 0 and 120")
except ValueError as e:
raise ValueError(f"Invalid age: {age_str}")
return IO(validate_action)
def create_user_profile() -> IO[dict]:
"""Interactive user profile creation"""
return (io_print("Creating user profile...")
.bind(lambda _: io_simulated_input("Enter your name: "))
.bind(lambda name:
io_simulated_input("Enter your age: ")
.bind(lambda age_str: validate_age(age_str)
.bind(lambda age:
io_simulated_input("Enter your city: ")
.map(lambda city: {
'name': name,
'age': age,
'city': city,
'created_at': datetime.now().isoformat()
})))))
def save_user_profile(profile: dict) -> IO[None]:
"""Save user profile to file"""
profile_json = f"User Profile:\n{profile}"
return io_write_file("user_profile.json", profile_json)
def complete_user_program() -> IO[str]:
"""Complete user registration program"""
return (create_user_profile()
.bind(lambda profile:
save_user_profile(profile)
.bind(lambda _:
io_print(f"โ
Profile created for {profile['name']}")
.map(lambda _: f"User {profile['name']} registered successfully"))))
print("Running user registration program:")
try:
registration_result = complete_user_program().run()
print(f"Program result: {registration_result}")
except Exception as e:
print(f"โ Program failed: {e}")
# 5. IO with Error Handling
print(f"\n5๏ธโฃ IO WITH ERROR HANDLING:")
class IOEither(IO[Either[str, T]]):
"""IO monad combined with Either for error handling"""
def __init__(self, action: Callable[[], Either[str, T]]):
super().__init__(action)
@classmethod
def from_io(cls, io_action: IO[T]) -> 'IOEither[T]':
"""Convert IO to IOEither"""
def safe_action():
try:
result = io_action.run()
return Right(result)
except Exception as e:
return Left(str(e))
return cls(safe_action)
@classmethod
def unit(cls, value: T) -> 'IOEither[T]':
"""Wrap pure value in IOEither"""
return cls(lambda: Right(value))
def bind_either(self, func: Callable[[T], 'IOEither[U]']) -> 'IOEither[U]':
"""Bind operation that handles Either results"""
def action():
either_result = self.run()
if either_result.is_left():
return Left(either_result.get_error())
try:
next_io_either = func(either_result.get_or_else(None))
return next_io_either.run()
except Exception as e:
return Left(str(e))
return IOEither(action)
def safe_read_file(filename: str) -> IOEither[str]:
"""Safe file reading with error handling"""
return IOEither.from_io(io_read_file(filename))
def safe_write_file(filename: str, content: str) -> IOEither[None]:
"""Safe file writing with error handling"""
return IOEither.from_io(io_write_file(filename, content))
def safe_file_program() -> IOEither[str]:
"""File processing program with error handling"""
return (safe_read_file("data.txt")
.bind_either(lambda content:
safe_write_file("backup.txt", content)
.bind_either(lambda _:
IOEither.unit(f"Successfully backed up {len(content)} characters"))))
print("Running safe file program:")
safe_result = safe_file_program().run()
if safe_result.is_right():
print(f"โ
Success: {safe_result.get_or_else('No result')}")
else:
print(f"โ Error: {safe_result.get_error()}")
# Test with non-existent file
error_program = safe_read_file("nonexistent.txt")
error_result = error_program.run()
if error_result.is_left():
print(f"Expected error: {error_result.get_error()}")Python๐งฌ Advanced Functional Patterns
"""
๐งฌ ADVANCED FUNCTIONAL PATTERNS
Sophisticated patterns using monads and functional techniques
"""
from typing import Protocol, runtime_checkable
import json
from enum import Enum
print("\n๐งฌ ADVANCED FUNCTIONAL PATTERNS")
print("=" * 35)
# 1. Functor Protocol
print("1๏ธโฃ FUNCTOR PROTOCOL:")
@runtime_checkable
class Functor(Protocol[T]):
"""Protocol for Functor - things that can be mapped over"""
def map(self, func: Callable[[T], U]) -> 'Functor[U]':
"""Map a function over the functor"""
...
# List as Functor
class FunctorList:
"""List that implements Functor protocol"""
def __init__(self, items: list):
self._items = items
def map(self, func: Callable[[T], U]) -> 'FunctorList':
"""Map function over all items"""
return FunctorList([func(item) for item in self._items])
@property
def items(self):
return self._items
def __str__(self):
return f"FunctorList({self._items})"
# Tree as Functor
class Tree:
"""Binary tree that implements Functor protocol"""
def __init__(self, value: T, left: 'Tree' = None, right: 'Tree' = None):
self.value = value
self.left = left
self.right = right
def map(self, func: Callable[[T], U]) -> 'Tree[U]':
"""Map function over all nodes in the tree"""
mapped_left = self.left.map(func) if self.left else None
mapped_right = self.right.map(func) if self.right else None
return Tree(func(self.value), mapped_left, mapped_right)
def to_list(self) -> list:
"""Convert tree to list (in-order traversal)"""
result = []
if self.left:
result.extend(self.left.to_list())
result.append(self.value)
if self.right:
result.extend(self.right.to_list())
return result
def __str__(self):
return f"Tree({self.value}, {self.left}, {self.right})"
# Test functors
print("Testing Functors:")
# List functor
func_list = FunctorList([1, 2, 3, 4, 5])
doubled_list = func_list.map(lambda x: x * 2)
print(f"Original list: {func_list}")
print(f"Doubled list: {doubled_list}")
# Tree functor
tree = Tree(1, Tree(2, Tree(4), Tree(5)), Tree(3, Tree(6), Tree(7)))
doubled_tree = tree.map(lambda x: x * 2)
print(f"Original tree values: {tree.to_list()}")
print(f"Doubled tree values: {doubled_tree.to_list()}")
# 2. Applicative Functors
print(f"\n2๏ธโฃ APPLICATIVE FUNCTORS:")
class Applicative(Monad[T]):
"""Applicative functor - can apply functions wrapped in context"""
@abstractmethod
def apply(self, func_in_context: 'Applicative[Callable[[T], U]]') -> 'Applicative[U]':
"""Apply a function that's also in the applicative context"""
pass
def lift_a2(self, func: Callable[[T, U], V], other: 'Applicative[U]') -> 'Applicative[V]':
"""Lift a binary function to work with applicatives"""
# lift_a2 f x y = f <$> x <*> y
return other.apply(self.map(func))
# Maybe as Applicative
class ApplicativeMaybe(Maybe[T]):
"""Maybe with applicative interface"""
def apply(self, func_maybe: 'ApplicativeMaybe[Callable[[T], U]]') -> 'ApplicativeMaybe[U]':
"""Apply function in Maybe context"""
if isinstance(func_maybe, Nothing) or isinstance(self, Nothing):
return Nothing()
try:
if hasattr(func_maybe, '_value') and hasattr(self, '_value'):
result = func_maybe._value(self._value)
return ApplicativeSome(result) if result is not None else Nothing()
except Exception:
pass
return Nothing()
@classmethod
def unit(cls, value: T) -> 'ApplicativeMaybe[T]':
return ApplicativeSome(value) if value is not None else Nothing()
class ApplicativeSome(ApplicativeMaybe[T]):
def __init__(self, value: T):
super().__init__(value)
def __str__(self):
return f"Some({self._value})"
# Test applicative functors
print("Testing Applicative Functors:")
# Apply a function to multiple Maybe values
add_func = ApplicativeSome(lambda x: lambda y: lambda z: x + y + z)
maybe_5 = ApplicativeSome(5)
maybe_3 = ApplicativeSome(3)
maybe_2 = ApplicativeSome(2)
# Chain applications
result = maybe_2.apply(maybe_3.apply(maybe_5.apply(add_func)))
print(f"Applicative addition: {result}")
print(f"Value: {result.get_or_else('Error')}")
# With one Nothing value
maybe_none = Nothing()
result_with_none = maybe_none.apply(maybe_3.apply(maybe_5.apply(add_func)))
print(f"With Nothing: {result_with_none}")
# 3. Free Monads
print(f"\n3๏ธโฃ FREE MONADS:")
class FreeMonad:
"""Free monad implementation for DSL creation"""
pass
class Pure(FreeMonad):
"""Pure value in free monad"""
def __init__(self, value: T):
self.value = value
def __str__(self):
return f"Pure({self.value})"
class Free(FreeMonad):
"""Free computation in free monad"""
def __init__(self, operation: Any):
self.operation = operation
def __str__(self):
return f"Free({self.operation})"
# Console DSL using Free Monad
class ConsoleOp:
"""Base class for console operations"""
pass
class PrintOp(ConsoleOp):
"""Print operation"""
def __init__(self, message: str, next_computation: FreeMonad):
self.message = message
self.next = next_computation
def __str__(self):
return f"Print('{self.message}', {self.next})"
class ReadOp(ConsoleOp):
"""Read operation"""
def __init__(self, prompt: str, continuation: Callable[[str], FreeMonad]):
self.prompt = prompt
self.continuation = continuation
def __str__(self):
return f"Read('{self.prompt}', <continuation>)"
# DSL functions
def print_line(message: str) -> FreeMonad:
"""Print a line (free monad)"""
return Free(PrintOp(message, Pure(None)))
def read_line(prompt: str) -> FreeMonad:
"""Read a line (free monad)"""
return Free(ReadOp(prompt, lambda x: Pure(x)))
def console_bind(free_monad: FreeMonad, func: Callable[[T], FreeMonad]) -> FreeMonad:
"""Bind operation for console free monad"""
if isinstance(free_monad, Pure):
return func(free_monad.value)
elif isinstance(free_monad, Free):
op = free_monad.operation
if isinstance(op, PrintOp):
return Free(PrintOp(op.message, console_bind(op.next, func)))
elif isinstance(op, ReadOp):
return Free(ReadOp(op.prompt,
lambda x: console_bind(op.continuation(x), func)))
return free_monad
# Build console program
def console_program() -> FreeMonad:
"""Build a console program using free monad"""
# Simulate binding operations
print_hello = print_line("Hello! What's your name?")
# In a real implementation, we'd use proper monadic bind
return print_hello
# Interpreter for console DSL
def interpret_console(program: FreeMonad, inputs: list = None) -> Any:
"""Interpret console program"""
if inputs is None:
inputs = ["Alice", "25"]
input_index = 0
current = program
results = []
while not isinstance(current, Pure):
if isinstance(current, Free):
op = current.operation
if isinstance(op, PrintOp):
print(f"Console: {op.message}")
results.append(f"Printed: {op.message}")
current = op.next
elif isinstance(op, ReadOp):
if input_index < len(inputs):
user_input = inputs[input_index]
input_index += 1
print(f"Console: {op.prompt}{user_input}")
results.append(f"Read: {user_input}")
current = op.continuation(user_input)
else:
break
else:
break
return results
print("Testing Free Monads:")
program = console_program()
results = interpret_console(program)
print(f"Interpretation results: {results}")
# 4. Lens Pattern (Functional Updates)
print(f"\n4๏ธโฃ LENS PATTERN:")
class Lens:
"""Lens for functional updates of nested data structures"""
def __init__(self, getter: Callable[[Any], T], setter: Callable[[Any, T], Any]):
self._getter = getter
self._setter = setter
def get(self, obj: Any) -> T:
"""Get value using lens"""
return self._getter(obj)
def set(self, obj: Any, value: T) -> Any:
"""Set value using lens (returns new object)"""
return self._setter(obj, value)
def modify(self, obj: Any, func: Callable[[T], T]) -> Any:
"""Modify value using lens"""
current_value = self.get(obj)
new_value = func(current_value)
return self.set(obj, new_value)
def compose(self, other: 'Lens') -> 'Lens':
"""Compose two lenses"""
def composed_getter(obj):
return other.get(self.get(obj))
def composed_setter(obj, value):
inner_obj = self.get(obj)
new_inner_obj = other.set(inner_obj, value)
return self.set(obj, new_inner_obj)
return Lens(composed_getter, composed_setter)
# Example data structures
@dataclass
class Address:
street: str
city: str
zipcode: str
@dataclass
class Person:
name: str
age: int
address: Address
# Create lenses
person_name_lens = Lens(
lambda person: person.name,
lambda person, name: Person(name, person.age, person.address)
)
person_address_lens = Lens(
lambda person: person.address,
lambda person, address: Person(person.name, person.age, address)
)
address_city_lens = Lens(
lambda address: address.city,
lambda address, city: Address(address.street, city, address.zipcode)
)
# Compose lenses
person_city_lens = person_address_lens.compose(address_city_lens)
# Test lenses
print("Testing Lenses:")
original_person = Person(
"Alice",
30,
Address("123 Main St", "New York", "10001")
)
print(f"Original: {original_person}")
# Update name
updated_name = person_name_lens.set(original_person, "Alice Smith")
print(f"Updated name: {updated_name}")
# Update city using composed lens
updated_city = person_city_lens.set(original_person, "Boston")
print(f"Updated city: {updated_city}")
# Modify age (add 1)
updated_age = person_name_lens.modify(original_person, lambda name: name.upper())
print(f"Modified name (uppercase): {updated_age}")
# Original remains unchanged
print(f"Original unchanged: {original_person}")
print("\n๐ฏ ADVANCED PATTERNS SUMMARY:")
print("โ
Monads provide structured computation with context")
print("โ
State monad manages stateful computations functionally")
print("โ
IO monad isolates side effects")
print("โ
Free monads enable DSL creation")
print("โ
Lenses provide functional data structure updates")
print("โ
All patterns compose naturally")Python๐๏ธ Building a Monad-Based Application
"""
๐๏ธ MONAD-BASED APPLICATION
Complete application using monadic patterns
"""
from typing import Dict, List, Optional
import json
from datetime import datetime, timedelta
print("\n๐๏ธ MONAD-BASED APPLICATION")
print("=" * 35)
# 1. Application Configuration
print("1๏ธโฃ APPLICATION SETUP:")
@dataclass
class User:
id: int
name: str
email: str
age: int
active: bool
@dataclass
class DatabaseConfig:
host: str
port: int
database: str
@dataclass
class AppConfig:
database: DatabaseConfig
log_level: str
debug: bool
# Mock database
class Database:
def __init__(self):
self._users = {
1: User(1, "Alice Johnson", "alice@example.com", 30, True),
2: User(2, "Bob Smith", "bob@example.com", 25, True),
3: User(3, "Charlie Brown", "charlie@example.com", 35, False),
}
def find_user(self, user_id: int) -> Optional[User]:
return self._users.get(user_id)
def update_user(self, user: User) -> bool:
if user.id in self._users:
self._users[user.id] = user
return True
return False
def list_active_users(self) -> List[User]:
return [user for user in self._users.values() if user.active]
# 2. Application Monad
print("2๏ธโฃ APPLICATION MONAD:")
@dataclass
class AppState:
"""Application state containing database and config"""
database: Database
config: AppConfig
log_entries: List[str]
def log(self, message: str) -> 'AppState':
"""Add log entry"""
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
log_entry = f"[{timestamp}] {message}"
new_log_entries = self.log_entries + [log_entry]
return AppState(self.database, self.config, new_log_entries)
class App(State[AppState, T]):
"""Application monad combining State and IO with error handling"""
def __init__(self, run_state: Callable[[AppState], Tuple[Either[str, T], AppState]]):
# Wrap the state function to return Either results
def wrapped_run_state(state: AppState) -> Tuple[Either[str, T], AppState]:
return run_state(state)
super().__init__(wrapped_run_state)
def bind(self, func: Callable[[T], 'App[U]']) -> 'App[U]':
"""Bind operation for App monad"""
def run_state(initial_state: AppState) -> Tuple[Either[str, U], AppState]:
# Run this computation
either_value, new_state = self._run_state(initial_state)
if either_value.is_left():
# Propagate error
return either_value, new_state
# Run the next computation with the new state
value = either_value.get_or_else(None)
next_computation = func(value)
return next_computation._run_state(new_state)
return App(run_state)
@classmethod
def unit(cls, value: T) -> 'App[T]':
"""Return a value without changing state"""
return cls(lambda state: (Right(value), state))
@classmethod
def error(cls, message: str) -> 'App[T]':
"""Return an error"""
return cls(lambda state: (Left(message), state))
def run_app(self, initial_state: AppState) -> Tuple[Either[str, T], AppState]:
"""Run the application"""
return self._run_state(initial_state)
# Application operations
def app_log(message: str) -> App[None]:
"""Log a message in the application"""
def run_state(state: AppState) -> Tuple[Either[str, None], AppState]:
new_state = state.log(message)
return Right(None), new_state
return App(run_state)
def get_database() -> App[Database]:
"""Get database from application state"""
def run_state(state: AppState) -> Tuple[Either[str, Database], AppState]:
return Right(state.database), state
return App(run_state)
def find_user(user_id: int) -> App[User]:
"""Find user by ID"""
return (app_log(f"Looking up user {user_id}")
.bind(lambda _: get_database())
.bind(lambda db:
App.unit(db.find_user(user_id))
.bind(lambda user:
App.unit(user) if user else App.error(f"User {user_id} not found"))))
def update_user_age(user_id: int, new_age: int) -> App[User]:
"""Update user's age"""
return (find_user(user_id)
.bind(lambda user:
app_log(f"Updating user {user.name} age from {user.age} to {new_age}")
.bind(lambda _:
get_database()
.bind(lambda db:
App.unit(User(user.id, user.name, user.email, new_age, user.active))
.bind(lambda updated_user:
App.unit(updated_user) if db.update_user(updated_user)
else App.error(f"Failed to update user {user_id}"))))))
def validate_user_age(age: int) -> App[int]:
"""Validate user age"""
if 0 <= age <= 120:
return App.unit(age)
else:
return App.error(f"Invalid age: {age}. Must be between 0 and 120.")
def get_active_users() -> App[List[User]]:
"""Get all active users"""
return (app_log("Retrieving active users")
.bind(lambda _: get_database())
.bind(lambda db: App.unit(db.list_active_users())))
def user_summary(user: User) -> str:
"""Create user summary"""
status = "Active" if user.active else "Inactive"
return f"{user.name} ({user.email}) - Age {user.age} - {status}"
# 3. Application Workflows
print("3๏ธโฃ APPLICATION WORKFLOWS:")
def user_management_workflow() -> App[str]:
"""Complete user management workflow"""
return (app_log("Starting user management workflow")
.bind(lambda _: find_user(1))
.bind(lambda user:
app_log(f"Found user: {user_summary(user)}")
.bind(lambda _: validate_user_age(31))
.bind(lambda valid_age: update_user_age(user.id, valid_age))
.bind(lambda updated_user:
app_log(f"Updated user: {user_summary(updated_user)}")
.bind(lambda _: get_active_users())
.bind(lambda users:
app_log(f"Found {len(users)} active users")
.bind(lambda _:
App.unit(f"Workflow completed. Updated {updated_user.name} and found {len(users)} active users."))))))
def error_handling_workflow() -> App[str]:
"""Workflow demonstrating error handling"""
return (app_log("Starting error handling workflow")
.bind(lambda _: find_user(999)) # Non-existent user
.bind(lambda user: App.unit(f"Found user: {user.name}"))
.bind(lambda _: App.unit("This won't execute due to error")))
def validation_workflow() -> App[str]:
"""Workflow demonstrating validation"""
return (app_log("Starting validation workflow")
.bind(lambda _: validate_user_age(150)) # Invalid age
.bind(lambda valid_age:
update_user_age(1, valid_age)
.bind(lambda user: App.unit(f"Updated user with age {valid_age}"))))
# 4. Application Execution
print("4๏ธโฃ APPLICATION EXECUTION:")
# Initialize application state
initial_config = AppConfig(
database=DatabaseConfig("localhost", 5432, "app_db"),
log_level="INFO",
debug=True
)
initial_state = AppState(
database=Database(),
config=initial_config,
log_entries=[]
)
# Run successful workflow
print("Running successful workflow:")
result, final_state = user_management_workflow().run_app(initial_state)
if result.is_right():
print(f"โ
Success: {result.get_or_else('No result')}")
else:
print(f"โ Error: {result.get_error()}")
print("\nApplication logs:")
for log_entry in final_state.log_entries:
print(f" {log_entry}")
# Run error workflow
print("\nRunning error workflow:")
error_result, error_state = error_handling_workflow().run_app(initial_state)
if error_result.is_right():
print(f"โ
Success: {error_result.get_or_else('No result')}")
else:
print(f"โ Expected error: {error_result.get_error()}")
print("\nError workflow logs:")
for log_entry in error_state.log_entries:
print(f" {log_entry}")
# Run validation workflow
print("\nRunning validation workflow:")
validation_result, validation_state = validation_workflow().run_app(initial_state)
if validation_result.is_right():
print(f"โ
Success: {validation_result.get_or_else('No result')}")
else:
print(f"โ Expected validation error: {validation_result.get_error()}")
print("\nValidation workflow logs:")
for log_entry in validation_state.log_entries:
print(f" {log_entry}")
# 5. Monad Composition
print(f"\n5๏ธโฃ MONAD COMPOSITION:")
def compose_multiple_operations() -> App[dict]:
"""Compose multiple operations into a complex workflow"""
return (app_log("Starting complex composition")
.bind(lambda _: get_active_users())
.bind(lambda users:
# Process each user
App.unit([user_summary(user) for user in users])
.bind(lambda summaries:
find_user(1)
.bind(lambda user:
update_user_age(user.id, user.age + 1)
.bind(lambda updated_user:
App.unit({
'total_active_users': len(users),
'user_summaries': summaries,
'updated_user': user_summary(updated_user),
'operation_timestamp': datetime.now().isoformat()
}))))))
print("Running complex composition:")
composition_result, composition_state = compose_multiple_operations().run_app(initial_state)
if composition_result.is_right():
result_data = composition_result.get_or_else({})
print("โ
Complex composition succeeded:")
print(json.dumps(result_data, indent=2))
else:
print(f"โ Composition failed: {composition_result.get_error()}")
print("\nComposition logs:")
for log_entry in composition_state.log_entries[-5:]: # Show last 5 entries
print(f" {log_entry}")
print("\n๐ฏ MONADIC APPLICATION BENEFITS:")
print("โ
Composable error handling")
print("โ
Stateful computations with pure functions")
print("โ
Logging and side effects managed")
print("โ
Type-safe operation chaining")
print("โ
Separation of business logic and effects")
print("โ
Easy testing and reasoning about code")PythonThis completes the comprehensive section on Monads and Advanced Patterns. The content covers:
- Monad Laws and Implementation – Abstract monad interface, Identity monad, and law verification
- State Monad – Managing stateful computations with counter, random number, and bank account examples
- IO Monad – Managing side effects and I/O operations functionally
- Advanced Functional Patterns – Functors, Applicative functors, Free monads, and Lenses
- Monad-Based Application – Complete application demonstrating practical monad usage
The monadic patterns shown here provide powerful abstractions for handling complex computational contexts like state, errors, and side effects while maintaining the benefits of functional programming such as composability, type safety, and pure functions.
13. Data Processing Pipelines
๐ญ Building Functional Data Processing Pipelines
Data Processing Pipelines are sequences of data transformations where the output of one stage becomes the input of the next. Functional programming excels at building these pipelines through composition, immutability, and lazy evaluation, creating maintainable, testable, and efficient data workflows.
graph TD
A["๐ญ Data Processing Pipelines"] --> B["๐ฅ ExtractData Ingestion"]
A --> C["๐ง TransformData Processing"]
A --> D["๐ค LoadData Output"]
B --> B1["File Systems"]
B --> B2["APIs"]
B --> B3["Databases"]
B --> B4["Streams"]
C --> C1["Filter"]
C --> C2["Map"]
C --> C3["Reduce"]
C --> C4["Aggregate"]
D --> D1["Files"]
D --> D2["Databases"]
D --> D3["APIs"]
D --> D4["Reports"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px๐ ETL Pipeline Fundamentals
"""
๐ ETL PIPELINE FUNDAMENTALS
Extract, Transform, Load patterns with functional programming
"""
from typing import List, Dict, Any, Iterator, Callable, Optional
from dataclasses import dataclass, asdict
from datetime import datetime, timedelta
import json
import csv
from io import StringIO
from functools import reduce, partial
import re
print("๐ญ ETL PIPELINE FUNDAMENTALS")
print("=" * 35)
# 1. Data structures for pipeline processing
print("1๏ธโฃ PIPELINE DATA STRUCTURES:")
@dataclass
class DataRecord:
"""Immutable data record for pipeline processing"""
id: str
timestamp: datetime
data: Dict[str, Any]
metadata: Dict[str, Any] = None
def __post_init__(self):
if self.metadata is None:
self.metadata = {}
def with_data(self, **updates) -> 'DataRecord':
"""Create new record with updated data"""
new_data = {**self.data, **updates}
return DataRecord(self.id, self.timestamp, new_data, self.metadata)
def with_metadata(self, **updates) -> 'DataRecord':
"""Create new record with updated metadata"""
new_metadata = {**self.metadata, **updates}
return DataRecord(self.id, self.timestamp, self.data, new_metadata)
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary"""
return {
'id': self.id,
'timestamp': self.timestamp.isoformat(),
'data': self.data,
'metadata': self.metadata
}
@dataclass
class PipelineResult:
"""Result of pipeline processing"""
records: List[DataRecord]
success_count: int
error_count: int
errors: List[str]
processing_time: float
def add_error(self, error: str) -> 'PipelineResult':
"""Add error to result"""
return PipelineResult(
self.records,
self.success_count,
self.error_count + 1,
self.errors + [error],
self.processing_time
)
# 2. Basic pipeline operations
print("\n2๏ธโฃ BASIC PIPELINE OPERATIONS:")
def extract_from_dicts(data_list: List[Dict[str, Any]]) -> List[DataRecord]:
"""Extract data from list of dictionaries"""
records = []
for i, item in enumerate(data_list):
record = DataRecord(
id=item.get('id', f"record_{i}"),
timestamp=datetime.fromisoformat(item.get('timestamp', datetime.now().isoformat())),
data=item,
metadata={'source': 'dict_list', 'index': i}
)
records.append(record)
return records
def filter_records(predicate: Callable[[DataRecord], bool]) -> Callable[[List[DataRecord]], List[DataRecord]]:
"""Higher-order function to filter records"""
def filter_fn(records: List[DataRecord]) -> List[DataRecord]:
return [record for record in records if predicate(record)]
return filter_fn
def transform_records(transform_fn: Callable[[DataRecord], DataRecord]) -> Callable[[List[DataRecord]], List[DataRecord]]:
"""Higher-order function to transform records"""
def transform_fn_list(records: List[DataRecord]) -> List[DataRecord]:
return [transform_fn(record) for record in records]
return transform_fn_list
def aggregate_records(group_by: Callable[[DataRecord], str],
agg_fn: Callable[[List[DataRecord]], DataRecord]) -> Callable[[List[DataRecord]], List[DataRecord]]:
"""Higher-order function to aggregate records"""
def aggregate_fn(records: List[DataRecord]) -> List[DataRecord]:
groups = {}
for record in records:
key = group_by(record)
if key not in groups:
groups[key] = []
groups[key].append(record)
return [agg_fn(group_records) for group_records in groups.values()]
return aggregate_fn
# Test basic operations
sample_data = [
{
'id': 'user_001',
'timestamp': '2024-01-15T10:00:00',
'name': 'Alice Johnson',
'email': 'alice@example.com',
'age': 30,
'department': 'Engineering',
'salary': 75000
},
{
'id': 'user_002',
'timestamp': '2024-01-15T10:01:00',
'name': 'Bob Smith',
'email': 'bob@example.com',
'age': 25,
'department': 'Marketing',
'salary': 50000
},
{
'id': 'user_003',
'timestamp': '2024-01-15T10:02:00',
'name': 'Charlie Brown',
'email': 'charlie@example.com',
'age': 35,
'department': 'Engineering',
'salary': 80000
}
]
print("Testing basic pipeline operations:")
records = extract_from_dicts(sample_data)
print(f"Extracted {len(records)} records")
# Filter high earners
high_earners = filter_records(lambda r: r.data.get('salary', 0) > 60000)(records)
print(f"High earners: {len(high_earners)}")
# Transform to add computed fields
def add_seniority_level(record: DataRecord) -> DataRecord:
"""Add seniority level based on age"""
age = record.data.get('age', 0)
if age >= 30:
level = 'Senior'
elif age >= 25:
level = 'Mid'
else:
level = 'Junior'
return record.with_data(seniority_level=level)
with_seniority = transform_records(add_seniority_level)(records)
for record in with_seniority:
print(f" {record.data['name']}: {record.data['seniority_level']}")
# Aggregate by department
def group_by_department(record: DataRecord) -> str:
return record.data.get('department', 'Unknown')
def create_department_summary(records: List[DataRecord]) -> DataRecord:
"""Create summary record for department"""
if not records:
return None
dept = records[0].data.get('department', 'Unknown')
total_salary = sum(r.data.get('salary', 0) for r in records)
avg_salary = total_salary / len(records)
avg_age = sum(r.data.get('age', 0) for r in records) / len(records)
return DataRecord(
id=f"dept_summary_{dept.lower()}",
timestamp=datetime.now(),
data={
'department': dept,
'employee_count': len(records),
'total_salary': total_salary,
'average_salary': round(avg_salary, 2),
'average_age': round(avg_age, 1)
},
metadata={'type': 'department_summary'}
)
dept_summaries = aggregate_records(group_by_department, create_department_summary)(records)
print(f"\nDepartment summaries:")
for summary in dept_summaries:
data = summary.data
print(f" {data['department']}: {data['employee_count']} employees, avg salary: ${data['average_salary']:,}")
# 3. Composable pipeline functions
print(f"\n3๏ธโฃ COMPOSABLE PIPELINE:")
def compose_pipeline(*operations):
"""Compose multiple pipeline operations"""
def pipeline(data):
result = data
for operation in operations:
result = operation(result)
return result
return pipeline
def safe_operation(operation, error_handler=None):
"""Wrap operation with error handling"""
def safe_op(data):
try:
return operation(data)
except Exception as e:
error_msg = f"Operation failed: {str(e)}"
if error_handler:
return error_handler(data, error_msg)
else:
print(f"โ {error_msg}")
return data # Return original data on error
return safe_op
# Create a comprehensive pipeline
comprehensive_pipeline = compose_pipeline(
safe_operation(filter_records(lambda r: r.data.get('email', '').endswith('@example.com'))),
safe_operation(transform_records(add_seniority_level)),
safe_operation(transform_records(lambda r: r.with_metadata(processed=True, processed_at=datetime.now().isoformat()))),
safe_operation(filter_records(lambda r: r.data.get('age', 0) >= 25))
)
print("Running comprehensive pipeline:")
pipeline_result = comprehensive_pipeline(records)
print(f"Pipeline processed {len(pipeline_result)} records")
for record in pipeline_result:
name = record.data['name']
level = record.data['seniority_level']
processed = record.metadata.get('processed', False)
print(f" {name} ({level}) - Processed: {processed}")Python๐ Stream Processing Patterns
"""
๐ STREAM PROCESSING PATTERNS
Real-time data processing with functional streams
"""
import time
import threading
from queue import Queue, Empty
from typing import Generator, Callable
from contextlib import contextmanager
import random
print("\n๐ STREAM PROCESSING PATTERNS")
print("=" * 35)
# 1. Stream data structures
print("1๏ธโฃ STREAM DATA STRUCTURES:")
class DataStream:
"""Functional data stream for processing"""
def __init__(self, source: Iterator[DataRecord]):
self._source = source
def map(self, transform_fn: Callable[[DataRecord], DataRecord]) -> 'DataStream':
"""Transform each record in the stream"""
def mapped_source():
for record in self._source:
yield transform_fn(record)
return DataStream(mapped_source())
def filter(self, predicate: Callable[[DataRecord], bool]) -> 'DataStream':
"""Filter records in the stream"""
def filtered_source():
for record in self._source:
if predicate(record):
yield record
return DataStream(filtered_source())
def window(self, size: int) -> 'DataStream':
"""Create windowed batches of records"""
def windowed_source():
window = []
for record in self._source:
window.append(record)
if len(window) >= size:
# Create a batch record
batch_record = DataRecord(
id=f"batch_{len(window)}_{datetime.now().timestamp()}",
timestamp=datetime.now(),
data={'batch': [r.to_dict() for r in window], 'size': len(window)},
metadata={'type': 'batch', 'window_size': size}
)
yield batch_record
window = []
# Handle remaining records
if window:
batch_record = DataRecord(
id=f"batch_{len(window)}_{datetime.now().timestamp()}",
timestamp=datetime.now(),
data={'batch': [r.to_dict() for r in window], 'size': len(window)},
metadata={'type': 'batch', 'window_size': size, 'partial': True}
)
yield batch_record
return DataStream(windowed_source())
def take(self, n: int) -> 'DataStream':
"""Take first n records from stream"""
def limited_source():
count = 0
for record in self._source:
if count >= n:
break
yield record
count += 1
return DataStream(limited_source())
def collect(self) -> List[DataRecord]:
"""Collect all records from stream"""
return list(self._source)
def for_each(self, action: Callable[[DataRecord], None]) -> None:
"""Execute action for each record"""
for record in self._source:
action(record)
# 2. Stream sources
print("\n2๏ธโฃ STREAM SOURCES:")
def generate_sensor_data(duration_seconds: int = 10, interval: float = 0.1) -> Generator[DataRecord, None, None]:
"""Generate simulated sensor data stream"""
start_time = time.time()
sensor_id = 0
while time.time() - start_time < duration_seconds:
sensor_id += 1
record = DataRecord(
id=f"sensor_{sensor_id:06d}",
timestamp=datetime.now(),
data={
'temperature': round(random.uniform(18.0, 25.0), 2),
'humidity': round(random.uniform(40.0, 70.0), 2),
'pressure': round(random.uniform(1010.0, 1030.0), 2),
'sensor_type': random.choice(['indoor', 'outdoor', 'warehouse'])
},
metadata={'source': 'sensor_network'}
)
yield record
time.sleep(interval)
def file_line_stream(content: str) -> Generator[DataRecord, None, None]:
"""Convert file content to record stream"""
for line_num, line in enumerate(content.split('\n'), 1):
if line.strip():
record = DataRecord(
id=f"line_{line_num:04d}",
timestamp=datetime.now(),
data={'line': line.strip(), 'line_number': line_num},
metadata={'source': 'file'}
)
yield record
def api_response_stream(api_responses: List[Dict]) -> Generator[DataRecord, None, None]:
"""Convert API responses to record stream"""
for response_num, response in enumerate(api_responses, 1):
record = DataRecord(
id=f"api_response_{response_num:04d}",
timestamp=datetime.now(),
data=response,
metadata={'source': 'api', 'response_number': response_num}
)
yield record
# Test stream processing
print("Testing stream processing:")
# Create a simple data stream
sample_api_responses = [
{'user_id': 1, 'action': 'login', 'timestamp': '2024-01-15T10:00:00'},
{'user_id': 2, 'action': 'view_page', 'page': 'dashboard', 'timestamp': '2024-01-15T10:01:00'},
{'user_id': 1, 'action': 'logout', 'timestamp': '2024-01-15T10:05:00'},
{'user_id': 3, 'action': 'login', 'timestamp': '2024-01-15T10:06:00'},
{'user_id': 2, 'action': 'click_button', 'button': 'submit', 'timestamp': '2024-01-15T10:07:00'},
]
api_stream = DataStream(api_response_stream(sample_api_responses))
# Process the stream
processed_records = (api_stream
.filter(lambda r: r.data.get('action') in ['login', 'logout'])
.map(lambda r: r.with_data(action_type='authentication'))
.map(lambda r: r.with_metadata(processed=True))
.collect())
print(f"Processed {len(processed_records)} authentication records:")
for record in processed_records:
user_id = record.data['user_id']
action = record.data['action']
print(f" User {user_id}: {action}")
# 3. Real-time stream processing
print(f"\n3๏ธโฃ REAL-TIME PROCESSING:")
class StreamProcessor:
"""Real-time stream processor with functional operations"""
def __init__(self):
self.operations = []
self.sinks = []
def add_operation(self, operation: Callable[[DataRecord], DataRecord]):
"""Add processing operation"""
self.operations.append(operation)
return self
def add_sink(self, sink: Callable[[DataRecord], None]):
"""Add data sink"""
self.sinks.append(sink)
return self
def process_record(self, record: DataRecord) -> DataRecord:
"""Process a single record through all operations"""
result = record
for operation in self.operations:
try:
result = operation(result)
except Exception as e:
print(f"โ Processing error: {e}")
return result
return result
def send_to_sinks(self, record: DataRecord):
"""Send record to all sinks"""
for sink in self.sinks:
try:
sink(record)
except Exception as e:
print(f"โ Sink error: {e}")
def run(self, stream_source: Generator[DataRecord, None, None]):
"""Run processor on stream"""
processed_count = 0
for record in stream_source:
processed_record = self.process_record(record)
self.send_to_sinks(processed_record)
processed_count += 1
print(f"โ
Processed {processed_count} records")
# Create stream processor
def add_processing_timestamp(record: DataRecord) -> DataRecord:
"""Add processing timestamp"""
return record.with_metadata(processing_timestamp=datetime.now().isoformat())
def enrich_sensor_data(record: DataRecord) -> DataRecord:
"""Enrich sensor data with additional fields"""
temp = record.data.get('temperature', 0)
alert_level = 'high' if temp > 23 else 'normal' if temp > 20 else 'low'
return record.with_data(temperature_alert=alert_level)
def console_sink(record: DataRecord):
"""Print record to console"""
data = record.data
if 'temperature' in data:
temp = data['temperature']
alert = data.get('temperature_alert', 'unknown')
print(f" ๐ก๏ธ Sensor {record.id}: {temp}ยฐC ({alert})")
def file_sink(filename: str):
"""Create file sink function"""
def sink(record: DataRecord):
with open(filename, 'a') as f:
f.write(json.dumps(record.to_dict()) + '\n')
return sink
# Configure and run stream processor
processor = (StreamProcessor()
.add_operation(add_processing_timestamp)
.add_operation(enrich_sensor_data)
.add_sink(console_sink))
print("Running real-time stream processor (5 records):")
# Limit to 5 records for demo
limited_stream = DataStream(generate_sensor_data(duration_seconds=1, interval=0.2)).take(5)
processor.run(limited_stream.collect())Python๐ Batch Processing Pipelines
"""
๐ BATCH PROCESSING PIPELINES
Large-scale data processing with functional patterns
"""
import csv
from pathlib import Path
from typing import Union, TextIO
from functools import partial
import hashlib
print("\n๐ BATCH PROCESSING PIPELINES")
print("=" * 35)
# 1. File processing utilities
print("1๏ธโฃ FILE PROCESSING UTILITIES:")
def read_csv_records(content: str, delimiter: str = ',') -> List[DataRecord]:
"""Read CSV content into DataRecord objects"""
reader = csv.DictReader(StringIO(content), delimiter=delimiter)
records = []
for row_num, row in enumerate(reader, 1):
record = DataRecord(
id=f"csv_row_{row_num:06d}",
timestamp=datetime.now(),
data=dict(row), # Convert OrderedDict to regular dict
metadata={'source': 'csv', 'row_number': row_num}
)
records.append(record)
return records
def write_csv_records(records: List[DataRecord], include_metadata: bool = False) -> str:
"""Write DataRecord objects to CSV format"""
if not records:
return ""
# Get all possible field names
data_fields = set()
metadata_fields = set()
for record in records:
data_fields.update(record.data.keys())
if include_metadata:
metadata_fields.update(record.metadata.keys())
# Create field names list
fieldnames = ['id', 'timestamp'] + sorted(data_fields)
if include_metadata:
fieldnames.extend([f'meta_{field}' for field in sorted(metadata_fields)])
output = StringIO()
writer = csv.DictWriter(output, fieldnames=fieldnames)
writer.writeheader()
for record in records:
row = {
'id': record.id,
'timestamp': record.timestamp.isoformat(),
**record.data
}
if include_metadata:
for field in metadata_fields:
row[f'meta_{field}'] = record.metadata.get(field, '')
writer.writerow(row)
return output.getvalue()
def read_json_records(content: str) -> List[DataRecord]:
"""Read JSON Lines format into DataRecord objects"""
records = []
for line_num, line in enumerate(content.strip().split('\n'), 1):
if line.strip():
try:
data = json.loads(line)
record = DataRecord(
id=data.get('id', f'json_line_{line_num:06d}'),
timestamp=datetime.fromisoformat(data.get('timestamp', datetime.now().isoformat())),
data=data,
metadata={'source': 'json', 'line_number': line_num}
)
records.append(record)
except json.JSONDecodeError as e:
print(f"โ JSON decode error on line {line_num}: {e}")
return records
# Test file processing
print("Testing file processing:")
sample_csv = """id,name,department,salary,hire_date
emp_001,Alice Johnson,Engineering,75000,2022-01-15
emp_002,Bob Smith,Marketing,55000,2021-08-22
emp_003,Carol Davis,Engineering,82000,2020-03-10
emp_004,David Wilson,Sales,48000,2023-02-01"""
csv_records = read_csv_records(sample_csv)
print(f"Read {len(csv_records)} records from CSV")
# Convert back to CSV
output_csv = write_csv_records(csv_records, include_metadata=True)
print(f"Generated CSV output ({len(output_csv)} characters)")
# 2. Data validation and cleaning
print(f"\n2๏ธโฃ DATA VALIDATION AND CLEANING:")
class ValidationError(Exception):
"""Custom validation error"""
pass
def validate_required_fields(*fields):
"""Create validator for required fields"""
def validator(record: DataRecord) -> DataRecord:
missing_fields = [field for field in fields if field not in record.data or not record.data[field]]
if missing_fields:
raise ValidationError(f"Missing required fields: {missing_fields}")
return record
return validator
def validate_data_types(**field_types):
"""Create validator for data types"""
def validator(record: DataRecord) -> DataRecord:
for field, expected_type in field_types.items():
if field in record.data:
value = record.data[field]
if expected_type == int:
try:
record = record.with_data(**{field: int(value)})
except ValueError:
raise ValidationError(f"Field {field} must be an integer, got: {value}")
elif expected_type == float:
try:
record = record.with_data(**{field: float(value)})
except ValueError:
raise ValidationError(f"Field {field} must be a float, got: {value}")
elif expected_type == str:
record = record.with_data(**{field: str(value)})
return record
return validator
def validate_email_format(field_name: str):
"""Create email format validator"""
email_pattern = re.compile(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$')
def validator(record: DataRecord) -> DataRecord:
if field_name in record.data:
email = record.data[field_name]
if not email_pattern.match(str(email)):
raise ValidationError(f"Invalid email format: {email}")
return record
return validator
def clean_whitespace(*fields):
"""Create whitespace cleaner for specified fields"""
def cleaner(record: DataRecord) -> DataRecord:
updates = {}
for field in fields:
if field in record.data and isinstance(record.data[field], str):
updates[field] = record.data[field].strip()
return record.with_data(**updates) if updates else record
return cleaner
def standardize_case(field_name: str, case_type: str = 'lower'):
"""Create case standardizer"""
def standardizer(record: DataRecord) -> DataRecord:
if field_name in record.data:
value = str(record.data[field_name])
if case_type == 'lower':
standardized = value.lower()
elif case_type == 'upper':
standardized = value.upper()
elif case_type == 'title':
standardized = value.title()
else:
standardized = value
return record.with_data(**{field_name: standardized})
return record
return standardizer
# Create validation and cleaning pipeline
def safe_transform(transform_fn, error_handler=None):
"""Safely apply transformation with error handling"""
def safe_transformer(record: DataRecord) -> DataRecord:
try:
return transform_fn(record)
except Exception as e:
error_msg = f"Transform error on record {record.id}: {str(e)}"
if error_handler:
return error_handler(record, error_msg)
else:
# Add error to metadata
return record.with_metadata(error=error_msg, validation_failed=True)
return safe_transformer
# Build comprehensive cleaning pipeline
cleaning_pipeline = compose_pipeline(
transform_records(safe_transform(clean_whitespace('name', 'department'))),
transform_records(safe_transform(standardize_case('department', 'title'))),
transform_records(safe_transform(validate_required_fields('name', 'department'))),
transform_records(safe_transform(validate_data_types(salary=int))),
filter_records(lambda r: not r.metadata.get('validation_failed', False))
)
print("Testing data cleaning pipeline:")
test_records = [
DataRecord('test_1', datetime.now(), {'name': ' alice johnson ', 'department': 'engineering', 'salary': '75000'}),
DataRecord('test_2', datetime.now(), {'name': 'Bob Smith', 'department': '', 'salary': '55000'}), # Missing department
DataRecord('test_3', datetime.now(), {'name': 'Carol Davis', 'department': 'MARKETING', 'salary': 'invalid'}), # Invalid salary
]
cleaned_records = cleaning_pipeline(test_records)
print(f"Cleaned {len(cleaned_records)} out of {len(test_records)} records")
for record in cleaned_records:
print(f" โ
{record.data['name']} - {record.data['department']} - ${record.data['salary']:,}")
# Show failed records
failed_records = [r for r in test_records if r.metadata.get('validation_failed', False)]
for record in failed_records:
print(f" โ {record.id}: {record.metadata.get('error', 'Unknown error')}")
# 3. Complex batch processing workflow
print(f"\n3๏ธโฃ COMPLEX BATCH WORKFLOW:")
def calculate_statistics(records: List[DataRecord]) -> DataRecord:
"""Calculate statistics from batch of records"""
if not records:
return None
salaries = [r.data.get('salary', 0) for r in records if isinstance(r.data.get('salary'), (int, float))]
departments = [r.data.get('department', 'Unknown') for r in records]
stats = {
'total_records': len(records),
'total_salary': sum(salaries),
'average_salary': sum(salaries) / len(salaries) if salaries else 0,
'min_salary': min(salaries) if salaries else 0,
'max_salary': max(salaries) if salaries else 0,
'departments': len(set(departments)),
'department_breakdown': {dept: departments.count(dept) for dept in set(departments)}
}
return DataRecord(
id='batch_statistics',
timestamp=datetime.now(),
data=stats,
metadata={'type': 'statistics', 'source_count': len(records)}
)
def partition_by_department(records: List[DataRecord]) -> Dict[str, List[DataRecord]]:
"""Partition records by department"""
partitions = {}
for record in records:
dept = record.data.get('department', 'Unknown')
if dept not in partitions:
partitions[dept] = []
partitions[dept].append(record)
return partitions
def enrich_with_hash(record: DataRecord) -> DataRecord:
"""Add hash of record data"""
data_str = json.dumps(record.data, sort_keys=True)
data_hash = hashlib.md5(data_str.encode()).hexdigest()[:8]
return record.with_metadata(data_hash=data_hash)
def batch_processing_workflow(input_records: List[DataRecord]) -> Dict[str, Any]:
"""Complete batch processing workflow"""
print(f" ๐ฅ Input: {len(input_records)} records")
# Step 1: Clean and validate
cleaned = cleaning_pipeline(input_records)
print(f" ๐งน Cleaned: {len(cleaned)} records")
# Step 2: Enrich with metadata
enriched = transform_records(enrich_with_hash)(cleaned)
print(f" ๐ Enriched: {len(enriched)} records")
# Step 3: Calculate statistics
stats = calculate_statistics(enriched)
if stats:
print(f" ๐ Statistics calculated")
print(f" Total salary: ${stats.data['total_salary']:,}")
print(f" Average salary: ${stats.data['average_salary']:,.2f}")
print(f" Departments: {stats.data['departments']}")
# Step 4: Partition by department
partitions = partition_by_department(enriched)
print(f" ๐ Partitioned into {len(partitions)} departments")
return {
'processed_records': enriched,
'statistics': stats,
'partitions': partitions,
'summary': {
'input_count': len(input_records),
'output_count': len(enriched),
'success_rate': len(enriched) / len(input_records) if input_records else 0
}
}
# Create sample batch data
print("Running complex batch processing workflow:")
batch_data = [
DataRecord(f'emp_{i:03d}', datetime.now(), {
'name': random.choice(['Alice Johnson', 'Bob Smith', 'Carol Davis', 'David Wilson', 'Eve Brown']),
'department': random.choice(['Engineering', 'Marketing', 'Sales', 'HR']),
'salary': random.randint(45000, 95000),
'hire_date': f'2{random.randint(20, 23)}-{random.randint(1, 12):02d}-{random.randint(1, 28):02d}'
})
for i in range(20)
]
# Run the workflow
workflow_result = batch_processing_workflow(batch_data)
print(f" โ
Workflow completed:")
print(f" Success rate: {workflow_result['summary']['success_rate']*100:.1f}%")
print(f" Department partitions: {list(workflow_result['partitions'].keys())}")Python๐ Advanced Pipeline Patterns
"""
๐ ADVANCED PIPELINE PATTERNS
Sophisticated data processing patterns and optimizations
"""
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, as_completed
from typing import AsyncIterator
import asyncio
from dataclasses import replace
import pickle
print("\n๐ ADVANCED PIPELINE PATTERNS")
print("=" * 35)
# 1. Parallel processing patterns
print("1๏ธโฃ PARALLEL PROCESSING:")
def parallel_map(transform_fn: Callable[[DataRecord], DataRecord],
records: List[DataRecord],
max_workers: int = 4) -> List[DataRecord]:
"""Apply transformation in parallel using threads"""
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = [executor.submit(transform_fn, record) for record in records]
results = []
for future in as_completed(futures):
try:
result = future.result()
results.append(result)
except Exception as e:
print(f"โ Parallel processing error: {e}")
return results
def process_intensive_transform(record: DataRecord) -> DataRecord:
"""CPU intensive transformation (simulation)"""
# Simulate expensive computation
time.sleep(0.1)
# Add computed field based on complex calculation
name_hash = sum(ord(c) for c in record.data.get('name', ''))
complexity_score = (name_hash * 17) % 1000
return record.with_data(complexity_score=complexity_score)
# Test parallel processing
print("Testing parallel processing:")
test_records = [
DataRecord(f'parallel_test_{i}', datetime.now(), {'name': f'User {i}'})
for i in range(8)
]
# Sequential processing
start_time = time.time()
sequential_results = [process_intensive_transform(r) for r in test_records]
sequential_time = time.time() - start_time
# Parallel processing
start_time = time.time()
parallel_results = parallel_map(process_intensive_transform, test_records, max_workers=4)
parallel_time = time.time() - start_time
print(f"Sequential time: {sequential_time:.3f}s")
print(f"Parallel time: {parallel_time:.3f}s")
print(f"Speedup: {sequential_time / parallel_time:.2f}x")
# 2. Pipeline with checkpoints
print(f"\n2๏ธโฃ PIPELINE CHECKPOINTS:")
class CheckpointPipeline:
"""Pipeline with checkpoint/resume capability"""
def __init__(self, checkpoint_dir: str = '/tmp/pipeline_checkpoints'):
self.checkpoint_dir = checkpoint_dir
self.operations = []
self.checkpoints = {}
def add_operation(self, name: str, operation: Callable, checkpoint_after: bool = False):
"""Add operation to pipeline"""
self.operations.append({
'name': name,
'operation': operation,
'checkpoint_after': checkpoint_after
})
return self
def save_checkpoint(self, name: str, data: List[DataRecord]):
"""Save checkpoint data"""
checkpoint_path = f"{self.checkpoint_dir}/{name}.pkl"
with open(checkpoint_path, 'wb') as f:
pickle.dump(data, f)
print(f" ๐พ Checkpoint saved: {name}")
def load_checkpoint(self, name: str) -> Optional[List[DataRecord]]:
"""Load checkpoint data"""
checkpoint_path = f"{self.checkpoint_dir}/{name}.pkl"
try:
with open(checkpoint_path, 'rb') as f:
data = pickle.load(f)
print(f" ๐ Checkpoint loaded: {name}")
return data
except FileNotFoundError:
return None
def run(self, initial_data: List[DataRecord], resume_from: str = None) -> List[DataRecord]:
"""Run pipeline with checkpoint support"""
data = initial_data
start_from = 0
# Resume from checkpoint if specified
if resume_from:
checkpoint_data = self.load_checkpoint(resume_from)
if checkpoint_data:
data = checkpoint_data
# Find start position
for i, op in enumerate(self.operations):
if op['name'] == resume_from:
start_from = i + 1
break
# Execute operations
for i, op_config in enumerate(self.operations[start_from:], start_from):
name = op_config['name']
operation = op_config['operation']
checkpoint_after = op_config['checkpoint_after']
print(f" ๐ Executing: {name}")
try:
data = operation(data)
print(f" Result: {len(data)} records")
if checkpoint_after:
self.save_checkpoint(name, data)
except Exception as e:
print(f" โ Operation {name} failed: {e}")
# Save emergency checkpoint
self.save_checkpoint(f"{name}_error", data)
raise
return data
# Test checkpoint pipeline
print("Testing checkpoint pipeline:")
# Create checkpoint directory (simulated)
import tempfile
import os
temp_dir = tempfile.mkdtemp()
checkpoint_pipeline = CheckpointPipeline(temp_dir)
checkpoint_pipeline.add_operation(
'clean_data',
lambda records: [r.with_data(name=r.data['name'].strip()) for r in records],
checkpoint_after=True
).add_operation(
'add_id_hash',
lambda records: [r.with_metadata(id_hash=hash(r.id)) for r in records],
checkpoint_after=True
).add_operation(
'filter_valid',
lambda records: [r for r in records if r.data.get('name')],
checkpoint_after=False
)
# Create test data
checkpoint_test_data = [
DataRecord('cp_1', datetime.now(), {'name': ' Alice '}),
DataRecord('cp_2', datetime.now(), {'name': 'Bob'}),
DataRecord('cp_3', datetime.now(), {'name': ''}), # Will be filtered out
]
# Run pipeline
print("First run:")
result1 = checkpoint_pipeline.run(checkpoint_test_data)
print(f"Final result: {len(result1)} records")
# Simulate resuming from checkpoint
print("\nResuming from checkpoint:")
result2 = checkpoint_pipeline.run([], resume_from='clean_data')
print(f"Resumed result: {len(result2)} records")
# 3. Adaptive pipeline
print(f"\n3๏ธโฃ ADAPTIVE PIPELINE:")
class AdaptivePipeline:
"""Pipeline that adapts based on data characteristics"""
def __init__(self):
self.base_operations = []
self.conditional_operations = []
self.statistics = {}
def add_base_operation(self, operation: Callable):
"""Add operation that always runs"""
self.base_operations.append(operation)
return self
def add_conditional_operation(self, condition: Callable[[List[DataRecord]], bool],
operation: Callable):
"""Add operation that runs based on data condition"""
self.conditional_operations.append({
'condition': condition,
'operation': operation
})
return self
def analyze_data(self, data: List[DataRecord]) -> Dict[str, Any]:
"""Analyze data characteristics"""
if not data:
return {}
analysis = {
'record_count': len(data),
'avg_data_fields': sum(len(r.data) for r in data) / len(data),
'has_timestamps': all('timestamp' in r.data for r in data),
'has_numeric_fields': any(
isinstance(v, (int, float))
for r in data
for v in r.data.values()
),
'departments': len(set(r.data.get('department', 'Unknown') for r in data)),
'date_range': self._get_date_range(data)
}
return analysis
def _get_date_range(self, data: List[DataRecord]) -> Optional[Dict[str, str]]:
"""Get date range from timestamps"""
timestamps = [r.timestamp for r in data if r.timestamp]
if timestamps:
return {
'earliest': min(timestamps).isoformat(),
'latest': max(timestamps).isoformat(),
'span_days': (max(timestamps) - min(timestamps)).days
}
return None
def run(self, data: List[DataRecord]) -> List[DataRecord]:
"""Run adaptive pipeline"""
print(" ๐ Analyzing data...")
analysis = self.analyze_data(data)
self.statistics = analysis
print(f" ๐ Analysis results:")
print(f" Records: {analysis.get('record_count', 0)}")
print(f" Avg fields per record: {analysis.get('avg_data_fields', 0):.1f}")
print(f" Departments: {analysis.get('departments', 0)}")
result = data
# Run base operations
print(" ๐ง Running base operations...")
for operation in self.base_operations:
result = operation(result)
# Run conditional operations
print(" ๐ฏ Evaluating conditional operations...")
for config in self.conditional_operations:
condition = config['condition']
operation = config['operation']
if condition(result):
print(f" โ
Condition met, running operation")
result = operation(result)
else:
print(f" โ Condition not met, skipping operation")
return result
# Create adaptive pipeline
adaptive_pipeline = AdaptivePipeline()
# Add base operations
adaptive_pipeline.add_base_operation(
lambda records: [r.with_metadata(processed=True) for r in records]
)
# Add conditional operations
adaptive_pipeline.add_conditional_operation(
# Condition: if more than 5 records, add batch ID
lambda records: len(records) > 5,
lambda records: [r.with_metadata(batch_id='large_batch') for r in records]
).add_conditional_operation(
# Condition: if multiple departments, add department stats
lambda records: len(set(r.data.get('department', 'Unknown') for r in records)) > 1,
lambda records: [
r.with_data(
dept_diversity=len(set(rec.data.get('department', 'Unknown') for rec in records))
)
for r in records
]
)
# Test adaptive pipeline
print("Testing adaptive pipeline:")
# Small dataset
small_data = [
DataRecord('small_1', datetime.now(), {'name': 'Alice', 'department': 'Engineering'}),
DataRecord('small_2', datetime.now(), {'name': 'Bob', 'department': 'Engineering'}),
]
print(" Small dataset:")
small_result = adaptive_pipeline.run(small_data)
print(f" Result: {len(small_result)} records")
# Large diverse dataset
large_data = [
DataRecord(f'large_{i}', datetime.now(), {
'name': f'Person {i}',
'department': random.choice(['Engineering', 'Marketing', 'Sales'])
})
for i in range(8)
]
print("\n Large diverse dataset:")
large_result = adaptive_pipeline.run(large_data)
print(f" Result: {len(large_result)} records")
# Check if conditional operations ran
has_batch_id = any('batch_id' in r.metadata for r in large_result)
has_dept_diversity = any('dept_diversity' in r.data for r in large_result)
print(f" Batch ID added: {has_batch_id}")
print(f" Department diversity added: {has_dept_diversity}")
print("\n๐ฏ ADVANCED PIPELINE BENEFITS:")
print("โ
Parallel processing improves performance")
print("โ
Checkpoints enable fault tolerance and resumability")
print("โ
Adaptive pipelines optimize based on data characteristics")
print("โ
Functional composition maintains pipeline modularity")
print("โ
Immutable data structures prevent side effects")
print("โ
Stream processing enables real-time data handling")Python๐ฏ Real-World Pipeline Examples
"""
๐ฏ REAL-WORLD PIPELINE EXAMPLES
Complete data processing pipelines for common scenarios
"""
import xml.etree.ElementTree as ET
from urllib.parse import urlparse
import base64
print("\n๐ฏ REAL-WORLD PIPELINE EXAMPLES")
print("=" * 35)
# 1. Log Analysis Pipeline
print("1๏ธโฃ LOG ANALYSIS PIPELINE:")
def parse_log_line(line: str) -> Dict[str, Any]:
"""Parse common log format line"""
# Common Log Format: IP - - [timestamp] "method path protocol" status size
log_pattern = re.compile(
r'(\S+) \S+ \S+ \[(.*?)\] "(.*?)" (\d+) (\d+)'
)
match = log_pattern.match(line)
if match:
ip, timestamp, request, status, size = match.groups()
method, path, protocol = request.split(' ', 2) if ' ' in request else (request, '/', 'HTTP/1.1')
return {
'ip': ip,
'timestamp': timestamp,
'method': method,
'path': path,
'protocol': protocol,
'status_code': int(status),
'response_size': int(size)
}
return {'raw_line': line, 'parse_error': True}
def create_log_analysis_pipeline():
"""Create comprehensive log analysis pipeline"""
def extract_from_log_lines(log_content: str) -> List[DataRecord]:
"""Extract records from log content"""
records = []
for line_num, line in enumerate(log_content.strip().split('\n'), 1):
if line.strip():
parsed_data = parse_log_line(line.strip())
record = DataRecord(
id=f'log_entry_{line_num:06d}',
timestamp=datetime.now(),
data=parsed_data,
metadata={'source': 'access_log', 'line_number': line_num}
)
records.append(record)
return records
def filter_valid_requests(records: List[DataRecord]) -> List[DataRecord]:
"""Filter out parsing errors and invalid requests"""
return [r for r in records if not r.data.get('parse_error', False)]
def enrich_with_request_info(record: DataRecord) -> DataRecord:
"""Enrich with request categorization"""
path = record.data.get('path', '')
status = record.data.get('status_code', 0)
# Categorize request type
if path.endswith(('.css', '.js', '.png', '.jpg', '.gif')):
request_type = 'static'
elif path.startswith('/api/'):
request_type = 'api'
elif path.startswith('/admin/'):
request_type = 'admin'
else:
request_type = 'page'
# Categorize response
if status < 300:
response_category = 'success'
elif status < 400:
response_category = 'redirect'
elif status < 500:
response_category = 'client_error'
else:
response_category = 'server_error'
return record.with_data(
request_type=request_type,
response_category=response_category,
is_error=status >= 400
)
def aggregate_by_ip(records: List[DataRecord]) -> List[DataRecord]:
"""Aggregate requests by IP address"""
ip_stats = {}
for record in records:
ip = record.data.get('ip', 'unknown')
if ip not in ip_stats:
ip_stats[ip] = {
'ip': ip,
'request_count': 0,
'error_count': 0,
'total_bytes': 0,
'request_types': set(),
'unique_paths': set()
}
stats = ip_stats[ip]
stats['request_count'] += 1
stats['total_bytes'] += record.data.get('response_size', 0)
stats['unique_paths'].add(record.data.get('path', ''))
stats['request_types'].add(record.data.get('request_type', 'unknown'))
if record.data.get('is_error', False):
stats['error_count'] += 1
# Convert to records
aggregated_records = []
for ip, stats in ip_stats.items():
# Convert sets to lists for JSON serialization
stats['request_types'] = list(stats['request_types'])
stats['unique_paths'] = len(stats['unique_paths']) # Count only
record = DataRecord(
id=f'ip_summary_{ip.replace(".", "_")}',
timestamp=datetime.now(),
data=stats,
metadata={'type': 'ip_aggregation'}
)
aggregated_records.append(record)
return aggregated_records
# Build pipeline
return compose_pipeline(
filter_valid_requests,
transform_records(enrich_with_request_info),
aggregate_by_ip
)
# Test log analysis pipeline
sample_log_data = """192.168.1.100 - - [15/Jan/2024:10:00:00 +0000] "GET /index.html HTTP/1.1" 200 1234
192.168.1.101 - - [15/Jan/2024:10:00:01 +0000] "POST /api/users HTTP/1.1" 201 567
192.168.1.100 - - [15/Jan/2024:10:00:02 +0000] "GET /styles.css HTTP/1.1" 200 890
192.168.1.102 - - [15/Jan/2024:10:00:03 +0000] "GET /admin/dashboard HTTP/1.1" 403 123
192.168.1.101 - - [15/Jan/2024:10:00:04 +0000] "GET /nonexistent HTTP/1.1" 404 456"""
print("Running log analysis pipeline:")
# Extract records
def extract_from_log_lines(log_content: str) -> List[DataRecord]:
"""Extract records from log content"""
records = []
for line_num, line in enumerate(log_content.strip().split('\n'), 1):
if line.strip():
parsed_data = parse_log_line(line.strip())
record = DataRecord(
id=f'log_entry_{line_num:06d}',
timestamp=datetime.now(),
data=parsed_data,
metadata={'source': 'access_log', 'line_number': line_num}
)
records.append(record)
return records
log_records = extract_from_log_lines(sample_log_data)
log_pipeline = create_log_analysis_pipeline()
analyzed_logs = log_pipeline(log_records)
print(f"Analyzed {len(analyzed_logs)} IP addresses:")
for record in analyzed_logs:
data = record.data
print(f" IP {data['ip']}: {data['request_count']} requests, {data['error_count']} errors")
# 2. E-commerce Order Processing Pipeline
print(f"\n2๏ธโฃ E-COMMERCE ORDER PIPELINE:")
@dataclass
class OrderItem:
product_id: str
quantity: int
unit_price: float
@property
def total_price(self) -> float:
return self.quantity * self.unit_price
def create_order_processing_pipeline():
"""Create order processing pipeline"""
def validate_order_data(record: DataRecord) -> DataRecord:
"""Validate order data completeness"""
required_fields = ['order_id', 'customer_id', 'items', 'order_date']
missing_fields = [field for field in required_fields if field not in record.data]
if missing_fields:
return record.with_metadata(validation_error=f"Missing fields: {missing_fields}")
# Validate items structure
items = record.data.get('items', [])
if not isinstance(items, list) or not items:
return record.with_metadata(validation_error="Invalid or empty items")
return record.with_metadata(validated=True)
def calculate_order_totals(record: DataRecord) -> DataRecord:
"""Calculate order totals and statistics"""
if record.metadata.get('validation_error'):
return record
items = record.data.get('items', [])
# Create OrderItem objects and calculate totals
order_items = []
subtotal = 0
total_quantity = 0
for item_data in items:
item = OrderItem(
product_id=item_data.get('product_id', ''),
quantity=item_data.get('quantity', 0),
unit_price=item_data.get('unit_price', 0.0)
)
order_items.append(item)
subtotal += item.total_price
total_quantity += item.quantity
# Calculate taxes and shipping
tax_rate = 0.08 # 8% tax
tax_amount = subtotal * tax_rate
# Free shipping over $100, otherwise $10
shipping_cost = 0 if subtotal > 100 else 10
total_amount = subtotal + tax_amount + shipping_cost
return record.with_data(
subtotal=round(subtotal, 2),
tax_amount=round(tax_amount, 2),
shipping_cost=shipping_cost,
total_amount=round(total_amount, 2),
total_quantity=total_quantity,
item_count=len(order_items)
)
def categorize_order(record: DataRecord) -> DataRecord:
"""Categorize order by size and value"""
if record.metadata.get('validation_error'):
return record
total_amount = record.data.get('total_amount', 0)
item_count = record.data.get('item_count', 0)
# Value category
if total_amount >= 500:
value_category = 'high_value'
elif total_amount >= 100:
value_category = 'medium_value'
else:
value_category = 'low_value'
# Size category
if item_count >= 10:
size_category = 'bulk'
elif item_count >= 3:
size_category = 'medium'
else:
size_category = 'small'
# Priority (high value or bulk orders get priority)
priority = 'high' if value_category == 'high_value' or size_category == 'bulk' else 'normal'
return record.with_data(
value_category=value_category,
size_category=size_category,
priority=priority
)
def determine_fulfillment_center(record: DataRecord) -> DataRecord:
"""Determine fulfillment center based on order characteristics"""
if record.metadata.get('validation_error'):
return record
priority = record.data.get('priority', 'normal')
size_category = record.data.get('size_category', 'small')
# Fulfillment logic
if priority == 'high':
fulfillment_center = 'premium_center'
estimated_shipping_days = 1
elif size_category == 'bulk':
fulfillment_center = 'bulk_center'
estimated_shipping_days = 3
else:
fulfillment_center = 'standard_center'
estimated_shipping_days = 2
return record.with_data(
fulfillment_center=fulfillment_center,
estimated_shipping_days=estimated_shipping_days
)
# Build pipeline
return compose_pipeline(
transform_records(validate_order_data),
filter_records(lambda r: not r.metadata.get('validation_error')),
transform_records(calculate_order_totals),
transform_records(categorize_order),
transform_records(determine_fulfillment_center)
)
# Test order processing pipeline
sample_orders = [
{
'order_id': 'ORD-001',
'customer_id': 'CUST-123',
'order_date': '2024-01-15T10:30:00',
'items': [
{'product_id': 'PROD-A', 'quantity': 2, 'unit_price': 29.99},
{'product_id': 'PROD-B', 'quantity': 1, 'unit_price': 149.99}
]
},
{
'order_id': 'ORD-002',
'customer_id': 'CUST-456',
'order_date': '2024-01-15T11:00:00',
'items': [
{'product_id': 'PROD-C', 'quantity': 15, 'unit_price': 12.50}
]
},
{
'order_id': 'ORD-003',
'customer_id': 'CUST-789',
'order_date': '2024-01-15T11:30:00',
'items': [] # Invalid - empty items
}
]
print("Running order processing pipeline:")
order_records = extract_from_dicts(sample_orders)
order_pipeline = create_order_processing_pipeline()
processed_orders = order_pipeline(order_records)
print(f"Processed {len(processed_orders)} valid orders:")
for record in processed_orders:
data = record.data
print(f" Order {data['order_id']}:")
print(f" Total: ${data['total_amount']} ({data['value_category']})")
print(f" Items: {data['item_count']} ({data['size_category']})")
print(f" Priority: {data['priority']}")
print(f" Fulfillment: {data['fulfillment_center']} ({data['estimated_shipping_days']} days)")
print("\n๐ฏ REAL-WORLD PIPELINE SUMMARY:")
print("โ
Log analysis pipeline processes web server logs")
print("โ
Order processing pipeline handles e-commerce workflows")
print("โ
Functional composition enables modular pipeline design")
print("โ
Immutable records prevent data corruption")
print("โ
Error handling maintains pipeline robustness")
print("โ
Validation ensures data quality throughout processing")PythonThis completes the comprehensive section on Data Processing Pipelines. The content covers:
- ETL Pipeline Fundamentals – Extract, Transform, Load patterns with functional programming
- Stream Processing Patterns – Real-time data processing with functional streams
- Batch Processing Pipelines – Large-scale data processing with validation and cleaning
- Advanced Pipeline Patterns – Parallel processing, checkpoints, and adaptive pipelines
- Real-World Pipeline Examples – Complete pipelines for log analysis and e-commerce order processing
The data processing pipeline patterns shown here demonstrate how functional programming principles create maintainable, testable, and efficient data workflows that can handle both batch and real-time processing scenarios.
14. Concurrent Functional Programming
โก Functional Programming in Concurrent Systems
Concurrent Functional Programming combines the safety of immutable data with the power of parallel execution. Since functional programming emphasizes immutability and pure functions, it naturally avoids many common concurrency issues like race conditions, deadlocks, and data corruption.
graph TD
A["โก Concurrent Functional Programming"] --> B["๐ Immutable SafetyNo Race Conditions"]
A --> C["๐ Pure FunctionsThread-Safe by Default"]
A --> D["๐ Parallel ProcessingCPU-Intensive Tasks"]
A --> E["๐ Async StreamsI/O-Bound Operations"]
B --> B1["Shared data is read-only"]
B --> B2["No mutex/locks needed"]
B --> B3["Memory consistency guaranteed"]
C --> C1["No side effects"]
C --> C2["Deterministic results"]
C --> C3["Easy to parallelize"]
D --> D1["CPU-bound computations"]
D --> D2["Data processing pipelines"]
D --> D3["Mathematical operations"]
E --> E1["Async I/O operations"]
E --> E2["Network requests"]
E --> E3["Database queries"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px
style E fill:#fce4ec,stroke:#e91e63,stroke-width:2px๐ Parallel Processing with Pure Functions
"""
๐ PARALLEL PROCESSING WITH PURE FUNCTIONS
Using concurrent.futures for CPU-intensive functional operations
"""
import concurrent.futures
import multiprocessing
import time
import math
from typing import List, Callable, Any, Tuple
from functools import reduce, partial
import random
print("โก CONCURRENT FUNCTIONAL PROGRAMMING")
print("=" * 40)
# 1. Basic parallel map operations
print("1๏ธโฃ PARALLEL MAP OPERATIONS:")
def cpu_intensive_function(n: int) -> int:
"""Pure function for CPU-intensive computation"""
# Simulate expensive computation (finding prime factors)
factors = []
d = 2
while d * d <= n:
while (n % d) == 0:
factors.append(d)
n //= d
d += 1
if n > 1:
factors.append(n)
return len(factors) # Return count of prime factors
def parallel_map(func: Callable, items: List[Any], max_workers: int = None) -> List[Any]:
"""Parallel map using ProcessPoolExecutor for CPU-bound tasks"""
if max_workers is None:
max_workers = multiprocessing.cpu_count()
with concurrent.futures.ProcessPoolExecutor(max_workers=max_workers) as executor:
results = list(executor.map(func, items))
return results
def thread_parallel_map(func: Callable, items: List[Any], max_workers: int = None) -> List[Any]:
"""Parallel map using ThreadPoolExecutor for I/O-bound tasks"""
if max_workers is None:
max_workers = min(32, (len(items) or 4) + 4)
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
results = list(executor.map(func, items))
return results
# Test parallel processing
test_numbers = [982451653, 982451654, 982451655, 982451656, 982451657, 982451658, 982451659, 982451660]
print("Testing CPU-intensive operations:")
print(f"Processing {len(test_numbers)} numbers")
# Sequential processing
start_time = time.time()
sequential_results = [cpu_intensive_function(n) for n in test_numbers]
sequential_time = time.time() - start_time
# Parallel processing
start_time = time.time()
parallel_results = parallel_map(cpu_intensive_function, test_numbers, max_workers=4)
parallel_time = time.time() - start_time
print(f"Sequential time: {sequential_time:.3f} seconds")
print(f"Parallel time: {parallel_time:.3f} seconds")
print(f"Speedup: {sequential_time / parallel_time:.2f}x")
print(f"Results match: {sequential_results == parallel_results}")
# 2. Parallel reduction operations
print(f"\n2๏ธโฃ PARALLEL REDUCTION:")
def parallel_reduce(func: Callable, items: List[Any], initial=None, chunk_size: int = None) -> Any:
"""Parallel reduce using divide-and-conquer approach"""
if not items:
return initial
if len(items) == 1:
return items[0] if initial is None else func(initial, items[0])
# Determine chunk size
if chunk_size is None:
chunk_size = max(1, len(items) // multiprocessing.cpu_count())
# Split into chunks
chunks = [items[i:i + chunk_size] for i in range(0, len(items), chunk_size)]
# Reduce each chunk in parallel
def reduce_chunk(chunk):
if initial is not None and chunk:
return reduce(func, chunk, initial)
elif chunk:
return reduce(func, chunk)
else:
return initial
with concurrent.futures.ProcessPoolExecutor() as executor:
chunk_results = list(executor.map(reduce_chunk, chunks))
# Final reduction of chunk results
if chunk_results:
return reduce(func, chunk_results)
else:
return initial
# Test parallel reduction
large_numbers = list(range(1, 1000001)) # 1 million numbers
print("Testing parallel reduction (sum of 1 million numbers):")
# Sequential sum
start_time = time.time()
sequential_sum = sum(large_numbers)
sequential_time = time.time() - start_time
# Parallel sum
start_time = time.time()
parallel_sum = parallel_reduce(lambda x, y: x + y, large_numbers, 0)
parallel_time = time.time() - start_time
print(f"Sequential sum: {sequential_sum} ({sequential_time:.3f}s)")
print(f"Parallel sum: {parallel_sum} ({parallel_time:.3f}s)")
print(f"Results match: {sequential_sum == parallel_sum}")
print(f"Speedup: {sequential_time / parallel_time:.2f}x")
# 3. Parallel filter operations
print(f"\n3๏ธโฃ PARALLEL FILTER:")
def is_prime(n: int) -> bool:
"""Pure function to check if number is prime"""
if n < 2:
return False
if n == 2:
return True
if n % 2 == 0:
return False
for i in range(3, int(math.sqrt(n)) + 1, 2):
if n % i == 0:
return False
return True
def parallel_filter(predicate: Callable, items: List[Any], max_workers: int = None) -> List[Any]:
"""Parallel filter using ProcessPoolExecutor"""
if max_workers is None:
max_workers = multiprocessing.cpu_count()
with concurrent.futures.ProcessPoolExecutor(max_workers=max_workers) as executor:
# Map predicate over all items
results = list(executor.map(predicate, items))
# Filter based on results
return [item for item, keep in zip(items, results) if keep]
# Test parallel filter
test_range = list(range(10000, 11000)) # Test 1000 numbers
print("Testing parallel filter (finding primes in range 10000-11000):")
# Sequential filter
start_time = time.time()
sequential_primes = [n for n in test_range if is_prime(n)]
sequential_time = time.time() - start_time
# Parallel filter
start_time = time.time()
parallel_primes = parallel_filter(is_prime, test_range)
parallel_time = time.time() - start_time
print(f"Found {len(sequential_primes)} primes")
print(f"Sequential time: {sequential_time:.3f} seconds")
print(f"Parallel time: {parallel_time:.3f} seconds")
print(f"Results match: {sequential_primes == parallel_primes}")
print(f"Speedup: {sequential_time / parallel_time:.2f}x")
# 4. Functional pipeline with parallelism
print(f"\n4๏ธโฃ PARALLEL FUNCTIONAL PIPELINE:")
def parallel_pipeline(*operations):
"""Create a pipeline that can execute operations in parallel where possible"""
def pipeline(data):
result = data
for operation in operations:
if hasattr(operation, '_parallel') and operation._parallel:
# This operation can be parallelized
result = operation(result)
else:
# Sequential operation
result = operation(result)
return result
return pipeline
def make_parallel(func, chunk_size=1000):
"""Decorator to make a function parallel"""
def parallel_func(items):
if len(items) <= chunk_size:
return [func(item) for item in items]
# Split into chunks and process in parallel
chunks = [items[i:i + chunk_size] for i in range(0, len(items), chunk_size)]
def process_chunk(chunk):
return [func(item) for item in chunk]
with concurrent.futures.ProcessPoolExecutor() as executor:
chunk_results = list(executor.map(process_chunk, chunks))
# Flatten results
result = []
for chunk_result in chunk_results:
result.extend(chunk_result)
return result
parallel_func._parallel = True
return parallel_func
# Create parallel operations
@make_parallel
def expensive_transform(x):
"""Expensive transformation"""
# Simulate complex computation
result = x
for _ in range(1000):
result = math.sin(result) * 1.1
return abs(result)
@make_parallel
def expensive_computation(x):
"""Another expensive computation"""
return sum(i**2 for i in range(min(100, x + 1)))
# Create pipeline
expensive_pipeline = parallel_pipeline(
expensive_transform,
expensive_computation,
lambda items: [x for x in items if x > 0.5] # Sequential filter
)
# Test pipeline
test_data = list(range(1, 5001)) # 5000 items
print("Testing parallel pipeline:")
start_time = time.time()
pipeline_result = expensive_pipeline(test_data)
pipeline_time = time.time() - start_time
print(f"Processed {len(test_data)} items โ {len(pipeline_result)} results")
print(f"Pipeline time: {pipeline_time:.3f} seconds")Python๐ Asynchronous Functional Programming
"""
๐ ASYNCHRONOUS FUNCTIONAL PROGRAMMING
Combining async/await with functional patterns
"""
import asyncio
import aiohttp
import json
from typing import AsyncIterator, Awaitable
from dataclasses import dataclass
from datetime import datetime
print("\n๐ ASYNCHRONOUS FUNCTIONAL PROGRAMMING")
print("=" * 40)
# 1. Async functional utilities
print("1๏ธโฃ ASYNC FUNCTIONAL UTILITIES:")
async def async_map(func: Callable, items: List[Any], concurrency: int = 10) -> List[Any]:
"""Asynchronous map with concurrency control"""
semaphore = asyncio.Semaphore(concurrency)
async def bounded_func(item):
async with semaphore:
if asyncio.iscoroutinefunction(func):
return await func(item)
else:
return func(item)
return await asyncio.gather(*[bounded_func(item) for item in items])
async def async_filter(predicate: Callable, items: List[Any], concurrency: int = 10) -> List[Any]:
"""Asynchronous filter with concurrency control"""
semaphore = asyncio.Semaphore(concurrency)
async def bounded_predicate(item):
async with semaphore:
if asyncio.iscoroutinefunction(predicate):
return await predicate(item)
else:
return predicate(item)
results = await asyncio.gather(*[bounded_predicate(item) for item in items])
return [item for item, keep in zip(items, results) if keep]
async def async_reduce(func: Callable, items: List[Any], initial=None) -> Any:
"""Asynchronous reduce operation"""
if not items:
return initial
accumulator = initial if initial is not None else items[0]
start_index = 0 if initial is not None else 1
for item in items[start_index:]:
if asyncio.iscoroutinefunction(func):
accumulator = await func(accumulator, item)
else:
accumulator = func(accumulator, item)
return accumulator
# Test async utilities
async def async_square(x: int) -> int:
"""Async function that simulates I/O delay"""
await asyncio.sleep(0.01) # Simulate network delay
return x ** 2
async def async_is_even(x: int) -> bool:
"""Async predicate"""
await asyncio.sleep(0.005)
return x % 2 == 0
async def test_async_utilities():
print("Testing async functional utilities:")
test_numbers = list(range(1, 21)) # 1 to 20
start_time = time.time()
# Async map
squared = await async_map(async_square, test_numbers, concurrency=5)
print(f"Async map completed: {len(squared)} results")
# Async filter
evens = await async_filter(async_is_even, squared, concurrency=5)
print(f"Async filter completed: {len(evens)} even squares")
# Async reduce
total = await async_reduce(lambda acc, x: acc + x, evens, 0)
print(f"Async reduce completed: sum = {total}")
elapsed = time.time() - start_time
print(f"Total time: {elapsed:.3f} seconds")
# Run async test
asyncio.run(test_async_utilities())
# 2. Async data processing pipeline
print(f"\n2๏ธโฃ ASYNC DATA PROCESSING PIPELINE:")
@dataclass
class APIResponse:
"""Data class for API responses"""
url: str
status_code: int
data: dict
response_time: float
timestamp: datetime
async def fetch_data(session: aiohttp.ClientSession, url: str) -> APIResponse:
"""Fetch data from URL asynchronously"""
start_time = time.time()
try:
async with session.get(url, timeout=aiohttp.ClientTimeout(total=5)) as response:
data = await response.json() if response.content_type == 'application/json' else {}
response_time = time.time() - start_time
return APIResponse(
url=url,
status_code=response.status,
data=data,
response_time=response_time,
timestamp=datetime.now()
)
except Exception as e:
return APIResponse(
url=url,
status_code=0,
data={'error': str(e)},
response_time=time.time() - start_time,
timestamp=datetime.now()
)
async def process_api_response(response: APIResponse) -> dict:
"""Process API response"""
await asyncio.sleep(0.01) # Simulate processing time
processed = {
'url': response.url,
'success': response.status_code == 200,
'response_time_ms': round(response.response_time * 1000, 2),
'data_size': len(str(response.data)),
'timestamp': response.timestamp.isoformat()
}
if 'error' in response.data:
processed['error'] = response.data['error']
return processed
async def async_data_pipeline(urls: List[str], max_concurrent: int = 10) -> List[dict]:
"""Async pipeline for fetching and processing multiple URLs"""
async with aiohttp.ClientSession() as session:
# Fetch all URLs concurrently
responses = await async_map(
lambda url: fetch_data(session, url),
urls,
concurrency=max_concurrent
)
# Process all responses
processed_responses = await async_map(
process_api_response,
responses,
concurrency=max_concurrent
)
return processed_responses
# Test async pipeline (using httpbin for testing)
async def test_async_pipeline():
print("Testing async data processing pipeline:")
# Test URLs (using httpbin.org for testing)
test_urls = [
'https://httpbin.org/json',
'https://httpbin.org/delay/1',
'https://httpbin.org/status/404',
'https://httpbin.org/uuid',
'https://httpbin.org/headers'
]
start_time = time.time()
results = await async_data_pipeline(test_urls, max_concurrent=3)
elapsed_time = time.time() - start_time
print(f"Processed {len(results)} URLs in {elapsed_time:.3f} seconds")
for result in results:
status = "โ
" if result['success'] else "โ"
print(f" {status} {result['url']}: {result['response_time_ms']}ms")
# Note: Uncomment to run (requires internet connection)
# asyncio.run(test_async_pipeline())
print("Async pipeline test defined (requires internet connection to run)")
# 3. Async stream processing
print(f"\n3๏ธโฃ ASYNC STREAM PROCESSING:")
async def async_range(start: int, stop: int, delay: float = 0.1) -> AsyncIterator[int]:
"""Async generator for numbers with delay"""
for i in range(start, stop):
await asyncio.sleep(delay)
yield i
async def async_transform_stream(stream: AsyncIterator, transform_func: Callable) -> AsyncIterator:
"""Transform async stream"""
async for item in stream:
if asyncio.iscoroutinefunction(transform_func):
yield await transform_func(item)
else:
yield transform_func(item)
async def async_filter_stream(stream: AsyncIterator, predicate: Callable) -> AsyncIterator:
"""Filter async stream"""
async for item in stream:
if asyncio.iscoroutinefunction(predicate):
keep = await predicate(item)
else:
keep = predicate(item)
if keep:
yield item
async def async_take(stream: AsyncIterator, n: int) -> AsyncIterator:
"""Take first n items from async stream"""
count = 0
async for item in stream:
if count >= n:
break
yield item
count += 1
async def async_collect(stream: AsyncIterator) -> List:
"""Collect all items from async stream"""
return [item async for item in stream]
# Test async stream processing
async def test_async_streams():
print("Testing async stream processing:")
# Create async stream
number_stream = async_range(1, 21, delay=0.05) # Numbers 1-20 with 50ms delay
# Transform stream
squared_stream = async_transform_stream(number_stream, lambda x: x ** 2)
# Filter stream
even_squared_stream = async_filter_stream(squared_stream, lambda x: x % 2 == 0)
# Take first 5
limited_stream = async_take(even_squared_stream, 5)
# Collect results
start_time = time.time()
results = await async_collect(limited_stream)
elapsed_time = time.time() - start_time
print(f"Stream processing results: {results}")
print(f"Processing time: {elapsed_time:.3f} seconds")
# Run async stream test
asyncio.run(test_async_streams())
# 4. Async pipeline with error handling
print(f"\n4๏ธโฃ ASYNC ERROR HANDLING:")
class AsyncEither:
"""Async version of Either monad"""
def __init__(self, value=None, error=None, is_success=True):
self.value = value
self.error = error
self.is_success = is_success
@classmethod
def success(cls, value):
return cls(value=value, is_success=True)
@classmethod
def failure(cls, error):
return cls(error=error, is_success=False)
async def bind(self, func):
"""Async bind operation"""
if not self.is_success:
return self
try:
if asyncio.iscoroutinefunction(func):
result = await func(self.value)
else:
result = func(self.value)
if isinstance(result, AsyncEither):
return result
else:
return AsyncEither.success(result)
except Exception as e:
return AsyncEither.failure(str(e))
def map(self, func):
"""Map operation"""
return self.bind(lambda x: func(x))
async def safe_async_divide(x: float, y: float) -> AsyncEither:
"""Safe async division"""
await asyncio.sleep(0.01) # Simulate async operation
if y == 0:
return AsyncEither.failure("Division by zero")
return AsyncEither.success(x / y)
async def safe_async_sqrt(x: float) -> AsyncEither:
"""Safe async square root"""
await asyncio.sleep(0.01)
if x < 0:
return AsyncEither.failure("Cannot take square root of negative number")
return AsyncEither.success(math.sqrt(x))
async def test_async_error_handling():
print("Testing async error handling:")
# Successful computation chain
result1 = await (AsyncEither.success(16)
.bind(lambda x: safe_async_divide(x, 4)) # 16/4 = 4
.bind(lambda x: safe_async_sqrt(x))) # sqrt(4) = 2
if result1.is_success:
print(f"โ
Successful computation: {result1.value}")
else:
print(f"โ Error: {result1.error}")
# Error computation chain
result2 = await (AsyncEither.success(16)
.bind(lambda x: safe_async_divide(x, 0)) # Division by zero
.bind(lambda x: safe_async_sqrt(x))) # Won't execute
if result2.is_success:
print(f"โ
Successful computation: {result2.value}")
else:
print(f"โ Expected error: {result2.error}")
# Run async error handling test
asyncio.run(test_async_error_handling())Python๐ Actor Model with Functional Programming
"""
๐ ACTOR MODEL WITH FUNCTIONAL PROGRAMMING
Implementing actor-like patterns using async and functional principles
"""
import asyncio
from dataclasses import dataclass
from typing import Any, Dict, Callable, Optional
from abc import ABC, abstractmethod
import uuid
print("\n๐ ACTOR MODEL IMPLEMENTATION")
print("=" * 35)
# 1. Message-based actor system
print("1๏ธโฃ MESSAGE-BASED ACTORS:")
@dataclass
class Message:
"""Immutable message for actor communication"""
type: str
data: Any
sender: Optional[str] = None
message_id: str = None
def __post_init__(self):
if self.message_id is None:
self.message_id = str(uuid.uuid4())
class Actor:
"""Functional actor implementation"""
def __init__(self, name: str, initial_state: Any = None):
self.name = name
self.state = initial_state
self.mailbox = asyncio.Queue()
self.message_handlers = {}
self.running = False
self.statistics = {'messages_processed': 0, 'errors': 0}
def register_handler(self, message_type: str, handler: Callable):
"""Register message handler (pure function)"""
self.message_handlers[message_type] = handler
return self
async def send_message(self, message: Message):
"""Send message to actor"""
await self.mailbox.put(message)
async def start(self):
"""Start the actor's message processing loop"""
self.running = True
print(f"๐ญ Actor {self.name} started")
while self.running:
try:
# Wait for message with timeout
message = await asyncio.wait_for(self.mailbox.get(), timeout=1.0)
await self._process_message(message)
except asyncio.TimeoutError:
continue # Check if still running
except Exception as e:
self.statistics['errors'] += 1
print(f"โ Actor {self.name} error: {e}")
async def _process_message(self, message: Message):
"""Process incoming message"""
handler = self.message_handlers.get(message.type)
if handler:
try:
# Handler should be a pure function that returns new state
new_state = handler(self.state, message.data)
# If handler is async
if asyncio.iscoroutinefunction(handler):
new_state = await handler(self.state, message.data)
self.state = new_state
self.statistics['messages_processed'] += 1
except Exception as e:
self.statistics['errors'] += 1
print(f"โ Handler error in {self.name}: {e}")
else:
print(f"โ ๏ธ No handler for message type '{message.type}' in {self.name}")
async def stop(self):
"""Stop the actor"""
self.running = False
print(f"๐ Actor {self.name} stopped")
print(f" Statistics: {self.statistics}")
# Create example actors
def counter_add_handler(state: int, data: int) -> int:
"""Pure handler function for adding to counter"""
return state + data
def counter_multiply_handler(state: int, data: int) -> int:
"""Pure handler function for multiplying counter"""
return state * data
def counter_reset_handler(state: int, data: Any) -> int:
"""Pure handler function for resetting counter"""
return 0
# Test actor system
async def test_actor_system():
print("Testing actor system:")
# Create counter actor
counter_actor = Actor("Counter", initial_state=0)
counter_actor.register_handler("add", counter_add_handler)
counter_actor.register_handler("multiply", counter_multiply_handler)
counter_actor.register_handler("reset", counter_reset_handler)
# Start actor
actor_task = asyncio.create_task(counter_actor.start())
# Send messages
await counter_actor.send_message(Message("add", 5))
await counter_actor.send_message(Message("add", 3))
await counter_actor.send_message(Message("multiply", 2))
await counter_actor.send_message(Message("add", 10))
# Give actor time to process
await asyncio.sleep(0.5)
print(f"Counter state: {counter_actor.state}")
print(f"Messages processed: {counter_actor.statistics['messages_processed']}")
# Stop actor
await counter_actor.stop()
actor_task.cancel()
# Run actor test
asyncio.run(test_actor_system())
# 2. Functional actor with state transformations
print(f"\n2๏ธโฃ FUNCTIONAL STATE TRANSFORMATIONS:")
@dataclass
class BankAccount:
"""Immutable bank account state"""
balance: float
transactions: List[dict]
def deposit(self, amount: float, description: str = "Deposit") -> 'BankAccount':
"""Pure function to create new state with deposit"""
new_transaction = {
'type': 'deposit',
'amount': amount,
'description': description,
'timestamp': datetime.now().isoformat(),
'balance_after': self.balance + amount
}
return BankAccount(
balance=self.balance + amount,
transactions=self.transactions + [new_transaction]
)
def withdraw(self, amount: float, description: str = "Withdrawal") -> 'BankAccount':
"""Pure function to create new state with withdrawal"""
if amount > self.balance:
# Return same state if insufficient funds
return self
new_transaction = {
'type': 'withdrawal',
'amount': amount,
'description': description,
'timestamp': datetime.now().isoformat(),
'balance_after': self.balance - amount
}
return BankAccount(
balance=self.balance - amount,
transactions=self.transactions + [new_transaction]
)
# Bank account actor handlers
def deposit_handler(state: BankAccount, data: dict) -> BankAccount:
"""Pure handler for deposits"""
amount = data.get('amount', 0)
description = data.get('description', 'Deposit')
return state.deposit(amount, description)
def withdraw_handler(state: BankAccount, data: dict) -> BankAccount:
"""Pure handler for withdrawals"""
amount = data.get('amount', 0)
description = data.get('description', 'Withdrawal')
return state.withdraw(amount, description)
def get_balance_handler(state: BankAccount, data: Any) -> BankAccount:
"""Pure handler for balance inquiry (side effect through print)"""
print(f"๐ฐ Account balance: ${state.balance:.2f}")
print(f"๐ Transaction count: {len(state.transactions)}")
return state
# Test bank account actor
async def test_bank_actor():
print("Testing bank account actor:")
# Create account with initial balance
initial_account = BankAccount(balance=100.0, transactions=[])
bank_actor = Actor("BankAccount", initial_state=initial_account)
bank_actor.register_handler("deposit", deposit_handler)
bank_actor.register_handler("withdraw", withdraw_handler)
bank_actor.register_handler("balance", get_balance_handler)
# Start actor
actor_task = asyncio.create_task(bank_actor.start())
# Perform banking operations
await bank_actor.send_message(Message("deposit", {"amount": 500, "description": "Salary"}))
await bank_actor.send_message(Message("withdraw", {"amount": 200, "description": "Rent"}))
await bank_actor.send_message(Message("deposit", {"amount": 150, "description": "Bonus"}))
await bank_actor.send_message(Message("withdraw", {"amount": 1000, "description": "Large withdrawal"})) # Should fail
await bank_actor.send_message(Message("balance", {}))
# Wait for processing
await asyncio.sleep(0.5)
final_state = bank_actor.state
print(f"Final balance: ${final_state.balance:.2f}")
print(f"Transaction history:")
for i, transaction in enumerate(final_state.transactions[-3:], 1): # Last 3 transactions
print(f" {i}. {transaction['type'].title()}: ${transaction['amount']:.2f} - {transaction['description']}")
await bank_actor.stop()
actor_task.cancel()
# Run bank actor test
asyncio.run(test_bank_actor())
# 3. Actor supervision and fault tolerance
print(f"\n3๏ธโฃ ACTOR SUPERVISION:")
class SupervisorActor(Actor):
"""Supervisor actor that manages child actors"""
def __init__(self, name: str):
super().__init__(name, initial_state={'children': {}, 'restart_count': 0})
async def spawn_child(self, child_name: str, child_class, *args, **kwargs):
"""Spawn a child actor"""
child = child_class(child_name, *args, **kwargs)
child_task = asyncio.create_task(child.start())
self.state['children'][child_name] = {
'actor': child,
'task': child_task,
'restart_count': 0
}
print(f"๐ถ Supervisor {self.name} spawned child {child_name}")
return child
async def restart_child(self, child_name: str):
"""Restart a failed child actor"""
if child_name in self.state['children']:
child_info = self.state['children'][child_name]
old_actor = child_info['actor']
# Cancel old task
child_info['task'].cancel()
# Create new actor instance
new_actor = Actor(child_name, initial_state=old_actor.state)
new_actor.message_handlers = old_actor.message_handlers.copy()
# Start new task
new_task = asyncio.create_task(new_actor.start())
# Update supervisor state
self.state['children'][child_name] = {
'actor': new_actor,
'task': new_task,
'restart_count': child_info['restart_count'] + 1
}
print(f"๐ Supervisor restarted child {child_name} (restart #{child_info['restart_count'] + 1})")
# Test supervisor
async def test_supervisor():
print("Testing supervisor pattern:")
supervisor = SupervisorActor("Supervisor")
supervisor_task = asyncio.create_task(supervisor.start())
# Spawn child actors
child1 = await supervisor.spawn_child("Child1", Actor, 0)
child1.register_handler("increment", lambda state, data: state + 1)
# Send messages to child
await child1.send_message(Message("increment", 1))
await child1.send_message(Message("increment", 1))
await asyncio.sleep(0.2)
print(f"Child1 state: {child1.state}")
# Simulate restart
await supervisor.restart_child("Child1")
await asyncio.sleep(0.1)
# Get restarted child
restarted_child = supervisor.state['children']['Child1']['actor']
await restarted_child.send_message(Message("increment", 1))
await asyncio.sleep(0.1)
print(f"Restarted child state: {restarted_child.state}")
# Cleanup
await supervisor.stop()
supervisor_task.cancel()
# Run supervisor test
asyncio.run(test_supervisor())
print("\n๐ฏ CONCURRENT FUNCTIONAL PROGRAMMING BENEFITS:")
print("โ
Immutable data eliminates race conditions")
print("โ
Pure functions are inherently thread-safe")
print("โ
Parallel processing leverages multiple CPU cores")
print("โ
Async programming handles I/O-bound operations efficiently")
print("โ
Actor model provides isolated, concurrent computation")
print("โ
Functional error handling composes well with concurrency")
print("โ
No shared mutable state reduces complexity")Python15. Testing Functional Code
๐งช Testing Pure Functions and Functional Code
Testing functional code is often easier and more reliable than testing imperative code because pure functions are predictable, isolated, and have no side effects. This makes them perfect candidates for comprehensive unit testing, property-based testing, and compositional testing strategies.
graph TD
A["๐งช Testing Functional Code"] --> B["โ
Unit TestingTest Pure Functions"]
A --> C["๐ฒ Property-Based TestingTest Function Properties"]
A --> D["๐ Compositional TestingTest Function Combinations"]
A --> E["๐ญ Mock-Free TestingNo Side Effects to Mock"]
B --> B1["Deterministic results"]
B --> B2["Easy to test"]
B --> B3["No setup/teardown"]
C --> C1["Generate test cases"]
C --> C2["Test invariants"]
C --> C3["Find edge cases"]
D --> D1["Test pipelines"]
D --> D2["Test compositions"]
D --> D3["Integration testing"]
E --> E1["No external dependencies"]
E --> E2["Isolated testing"]
E --> E3["Fast execution"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px
style E fill:#fce4ec,stroke:#e91e63,stroke-width:2pxโ Unit Testing Pure Functions
"""
โ
UNIT TESTING PURE FUNCTIONS
Testing functional code with standard unit testing frameworks
"""
import unittest
from typing import List, Callable, Any
import math
from functools import reduce
from dataclasses import dataclass
print("๐งช TESTING FUNCTIONAL CODE")
print("=" * 30)
# 1. Testing pure functions - they're deterministic and easy to test
print("1๏ธโฃ UNIT TESTING PURE FUNCTIONS:")
# Pure functions to test
def add(x: int, y: int) -> int:
"""Pure function: add two numbers"""
return x + y
def multiply(x: int, y: int) -> int:
"""Pure function: multiply two numbers"""
return x * y
def factorial(n: int) -> int:
"""Pure function: calculate factorial"""
if n <= 1:
return 1
return n * factorial(n - 1)
def is_prime(n: int) -> bool:
"""Pure function: check if number is prime"""
if n < 2:
return False
for i in range(2, int(math.sqrt(n)) + 1):
if n % i == 0:
return False
return True
def filter_evens(numbers: List[int]) -> List[int]:
"""Pure function: filter even numbers"""
return [n for n in numbers if n % 2 == 0]
def compose(f: Callable, g: Callable) -> Callable:
"""Pure function: compose two functions"""
return lambda x: f(g(x))
class TestPureFunctions(unittest.TestCase):
"""Test cases for pure functions"""
def test_add(self):
"""Test addition function"""
# Basic cases
self.assertEqual(add(2, 3), 5)
self.assertEqual(add(-1, 1), 0)
self.assertEqual(add(0, 0), 0)
# Edge cases
self.assertEqual(add(-5, -3), -8)
self.assertEqual(add(1000000, 2000000), 3000000)
def test_multiply(self):
"""Test multiplication function"""
self.assertEqual(multiply(3, 4), 12)
self.assertEqual(multiply(-2, 5), -10)
self.assertEqual(multiply(0, 100), 0)
self.assertEqual(multiply(-3, -4), 12)
def test_factorial(self):
"""Test factorial function"""
# Known values
self.assertEqual(factorial(0), 1)
self.assertEqual(factorial(1), 1)
self.assertEqual(factorial(5), 120)
self.assertEqual(factorial(10), 3628800)
# Property: n! = n * (n-1)! for n > 0
for n in range(1, 10):
self.assertEqual(factorial(n), n * factorial(n - 1))
def test_is_prime(self):
"""Test prime checking function"""
# Known primes
primes = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
for p in primes:
self.assertTrue(is_prime(p), f"{p} should be prime")
# Known non-primes
non_primes = [0, 1, 4, 6, 8, 9, 10, 12, 15, 16]
for n in non_primes:
self.assertFalse(is_prime(n), f"{n} should not be prime")
def test_filter_evens(self):
"""Test even number filtering"""
# Basic test
self.assertEqual(filter_evens([1, 2, 3, 4, 5, 6]), [2, 4, 6])
# Edge cases
self.assertEqual(filter_evens([]), [])
self.assertEqual(filter_evens([1, 3, 5]), [])
self.assertEqual(filter_evens([2, 4, 6]), [2, 4, 6])
self.assertEqual(filter_evens([-2, -1, 0, 1, 2]), [-2, 0, 2])
def test_compose(self):
"""Test function composition"""
# Compose add_one and double
add_one = lambda x: x + 1
double = lambda x: x * 2
composed = compose(double, add_one)
# Test: double(add_one(x)) = 2*(x+1) = 2x + 2
self.assertEqual(composed(5), 12) # double(5+1) = double(6) = 12
self.assertEqual(composed(0), 2) # double(0+1) = double(1) = 2
self.assertEqual(composed(-1), 0) # double(-1+1) = double(0) = 0
def test_function_properties(self):
"""Test mathematical properties of functions"""
# Commutativity of addition
self.assertEqual(add(3, 5), add(5, 3))
self.assertEqual(add(-2, 7), add(7, -2))
# Associativity: (a + b) + c = a + (b + c)
a, b, c = 2, 3, 4
self.assertEqual(add(add(a, b), c), add(a, add(b, c)))
# Identity element for multiplication
for x in [-5, 0, 1, 10]:
self.assertEqual(multiply(x, 1), x)
self.assertEqual(multiply(1, x), x)
# Run unit tests
def run_unit_tests():
"""Run the unit tests and display results"""
print("Running unit tests for pure functions:")
# Create test suite
suite = unittest.TestLoader().loadTestsFromTestCase(TestPureFunctions)
# Run tests with custom result handler
class CustomTestResult(unittest.TextTestResult):
def startTest(self, test):
super().startTest(test)
def addSuccess(self, test):
super().addSuccess(test)
print(f" โ
{test._testMethodName}")
def addFailure(self, test, err):
super().addFailure(test, err)
print(f" โ {test._testMethodName}: FAILED")
def addError(self, test, err):
super().addError(test, err)
print(f" ๐ฅ {test._testMethodName}: ERROR")
runner = unittest.TextTestRunner(resultclass=CustomTestResult, verbosity=0)
result = runner.run(suite)
print(f"\nTest Results:")
print(f" Tests run: {result.testsRun}")
print(f" Failures: {len(result.failures)}")
print(f" Errors: {len(result.errors)}")
print(f" Success rate: {((result.testsRun - len(result.failures) - len(result.errors)) / result.testsRun * 100):.1f}%")
run_unit_tests()
# 2. Testing higher-order functions
print(f"\n2๏ธโฃ TESTING HIGHER-ORDER FUNCTIONS:")
def map_func(func: Callable, items: List[Any]) -> List[Any]:
"""Our own map implementation"""
return [func(item) for item in items]
def filter_func(predicate: Callable, items: List[Any]) -> List[Any]:
"""Our own filter implementation"""
return [item for item in items if predicate(item)]
def reduce_func(func: Callable, items: List[Any], initial=None):
"""Our own reduce implementation"""
if initial is not None:
return reduce(func, items, initial)
return reduce(func, items)
class TestHigherOrderFunctions(unittest.TestCase):
"""Test higher-order functions"""
def test_map_func(self):
"""Test map implementation"""
# Test with square function
square = lambda x: x ** 2
self.assertEqual(map_func(square, [1, 2, 3, 4]), [1, 4, 9, 16])
# Test with string function
upper = str.upper
self.assertEqual(map_func(upper, ['hello', 'world']), ['HELLO', 'WORLD'])
# Test empty list
self.assertEqual(map_func(square, []), [])
def test_filter_func(self):
"""Test filter implementation"""
# Test with even predicate
is_even = lambda x: x % 2 == 0
self.assertEqual(filter_func(is_even, [1, 2, 3, 4, 5, 6]), [2, 4, 6])
# Test with string predicate
long_words = lambda s: len(s) > 4
self.assertEqual(
filter_func(long_words, ['cat', 'elephant', 'dog', 'hippopotamus']),
['elephant', 'hippopotamus']
)
# Test empty result
self.assertEqual(filter_func(lambda x: x > 10, [1, 2, 3]), [])
def test_reduce_func(self):
"""Test reduce implementation"""
# Test sum
add_func = lambda x, y: x + y
self.assertEqual(reduce_func(add_func, [1, 2, 3, 4, 5]), 15)
# Test product
multiply_func = lambda x, y: x * y
self.assertEqual(reduce_func(multiply_func, [1, 2, 3, 4]), 24)
# Test with initial value
self.assertEqual(reduce_func(add_func, [1, 2, 3], 10), 16)
# Test max
max_func = lambda x, y: x if x > y else y
self.assertEqual(reduce_func(max_func, [3, 7, 2, 9, 1]), 9)
# Run higher-order function tests
print("Running tests for higher-order functions:")
suite = unittest.TestLoader().loadTestsFromTestCase(TestHigherOrderFunctions)
runner = unittest.TextTestRunner(verbosity=0)
result = runner.run(suite)
print(f"Higher-order function tests: {result.testsRun - len(result.failures) - len(result.errors)}/{result.testsRun} passed")Python๐ฒ Property-Based Testing
"""
๐ฒ PROPERTY-BASED TESTING
Testing functional properties rather than specific examples
"""
import random
from typing import Generator
import itertools
print("\n๐ฒ PROPERTY-BASED TESTING")
print("=" * 30)
# 1. Simple property-based testing framework
print("1๏ธโฃ SIMPLE PROPERTY-BASED TESTING:")
class PropertyTest:
"""Simple property-based testing framework"""
def __init__(self, num_tests: int = 100):
self.num_tests = num_tests
self.passed = 0
self.failed = 0
self.failures = []
def test_property(self, name: str, property_func: Callable, generator: Generator) -> bool:
"""Test a property with generated inputs"""
print(f" Testing property: {name}")
for i in range(self.num_tests):
try:
test_input = next(generator)
if not property_func(test_input):
self.failed += 1
self.failures.append((name, test_input))
print(f" โ Failed on input: {test_input}")
return False
else:
self.passed += 1
except StopIteration:
print(f" โ ๏ธ Generator exhausted after {i} tests")
break
except Exception as e:
self.failed += 1
self.failures.append((name, test_input, str(e)))
print(f" ๐ฅ Exception on input {test_input}: {e}")
return False
print(f" โ
Passed all {min(self.num_tests, i+1)} tests")
return True
def report(self):
"""Report test results"""
total = self.passed + self.failed
success_rate = (self.passed / total * 100) if total > 0 else 0
print(f"\nProperty-based test results:")
print(f" Total tests: {total}")
print(f" Passed: {self.passed}")
print(f" Failed: {self.failed}")
print(f" Success rate: {success_rate:.1f}%")
# 2. Generators for test data
print("\n2๏ธโฃ TEST DATA GENERATORS:")
def int_generator(min_val: int = -1000, max_val: int = 1000) -> Generator[int, None, None]:
"""Generate random integers"""
while True:
yield random.randint(min_val, max_val)
def positive_int_generator(max_val: int = 1000) -> Generator[int, None, None]:
"""Generate positive integers"""
while True:
yield random.randint(1, max_val)
def list_generator(item_generator: Generator, min_length: int = 0, max_length: int = 20) -> Generator[List, None, None]:
"""Generate lists using an item generator"""
while True:
length = random.randint(min_length, max_length)
yield [next(item_generator) for _ in range(length)]
def pair_generator(gen1: Generator, gen2: Generator) -> Generator[tuple, None, None]:
"""Generate pairs from two generators"""
while True:
yield (next(gen1), next(gen2))
# Test our pure functions with properties
def test_arithmetic_properties():
"""Test arithmetic function properties"""
prop_test = PropertyTest(100)
# Property: Addition is commutative
def addition_commutative(pair):
a, b = pair
return add(a, b) == add(b, a)
prop_test.test_property(
"Addition is commutative",
addition_commutative,
pair_generator(int_generator(), int_generator())
)
# Property: Addition is associative
def addition_associative(triple):
a, b, c = triple
return add(add(a, b), c) == add(a, add(b, c))
triple_gen = (lambda: (next(int_generator()), next(int_generator()), next(int_generator())))
prop_test.test_property(
"Addition is associative",
addition_associative,
(triple_gen() for _ in itertools.count())
)
# Property: Multiplication by zero
def multiplication_by_zero(n):
return multiply(n, 0) == 0
prop_test.test_property(
"Multiplication by zero",
multiplication_by_zero,
int_generator()
)
# Property: Factorial is always positive for non-negative inputs
def factorial_positive(n):
if n < 0:
return True # Skip negative inputs
return factorial(n) > 0
prop_test.test_property(
"Factorial is positive",
factorial_positive,
int_generator(0, 10) # Limit to avoid large factorials
)
prop_test.report()
test_arithmetic_properties()
# 3. Testing list operations properties
print(f"\n3๏ธโฃ TESTING LIST OPERATION PROPERTIES:")
def test_list_properties():
"""Test properties of list operations"""
prop_test = PropertyTest(50)
# Property: Filter preserves order
def filter_preserves_order(numbers):
evens = filter_evens(numbers)
# Check if evens appear in the same relative order as in original
even_positions = [i for i, n in enumerate(numbers) if n % 2 == 0]
for i in range(len(evens)):
if i < len(even_positions) - 1:
# Current even should appear before next even in original list
if even_positions[i] >= even_positions[i + 1]:
return False
return True
prop_test.test_property(
"Filter preserves order",
filter_preserves_order,
list_generator(int_generator(), 0, 10)
)
# Property: Filter result is subset
def filter_is_subset(numbers):
evens = filter_evens(numbers)
return all(n in numbers for n in evens)
prop_test.test_property(
"Filter result is subset",
filter_is_subset,
list_generator(int_generator(), 0, 15)
)
# Property: Filter with identity predicate returns original list
def filter_identity(numbers):
result = [n for n in numbers if True] # Always true predicate
return result == numbers
prop_test.test_property(
"Filter with identity",
filter_identity,
list_generator(int_generator(), 0, 10)
)
# Property: Map preserves length
def map_preserves_length(numbers):
squared = [n ** 2 for n in numbers]
return len(squared) == len(numbers)
prop_test.test_property(
"Map preserves length",
map_preserves_length,
list_generator(int_generator(), 0, 20)
)
prop_test.report()
test_list_properties()
# 4. Testing function composition properties
print(f"\n4๏ธโฃ TESTING COMPOSITION PROPERTIES:")
def test_composition_properties():
"""Test function composition properties"""
prop_test = PropertyTest(50)
# Helper functions
add_one = lambda x: x + 1
multiply_two = lambda x: x * 2
square = lambda x: x ** 2
# Property: Composition associativity
# (f โ g) โ h = f โ (g โ h)
def composition_associative(x):
# (square โ multiply_two) โ add_one
left = compose(compose(square, multiply_two), add_one)
# square โ (multiply_two โ add_one)
right = compose(square, compose(multiply_two, add_one))
return left(x) == right(x)
prop_test.test_property(
"Composition is associative",
composition_associative,
int_generator()
)
# Property: Identity composition
# f โ id = id โ f = f
def identity_composition(x):
identity = lambda y: y
f = add_one
# f โ identity = f
left_identity = compose(f, identity)(x) == f(x)
# identity โ f = f
right_identity = compose(identity, f)(x) == f(x)
return left_identity and right_identity
prop_test.test_property(
"Identity composition",
identity_composition,
int_generator()
)
prop_test.report()
test_composition_properties()
# 5. Advanced property testing with invariants
print(f"\n5๏ธโฃ TESTING INVARIANTS:")
@dataclass
class Stack:
"""Immutable stack for testing"""
items: List[Any]
def push(self, item: Any) -> 'Stack':
return Stack(self.items + [item])
def pop(self) -> tuple['Stack', Any]:
if not self.items:
raise ValueError("Cannot pop from empty stack")
return Stack(self.items[:-1]), self.items[-1]
def peek(self) -> Any:
if not self.items:
raise ValueError("Cannot peek empty stack")
return self.items[-1]
def is_empty(self) -> bool:
return len(self.items) == 0
def size(self) -> int:
return len(self.items)
def test_stack_properties():
"""Test stack data structure properties"""
prop_test = PropertyTest(30)
# Property: Push then pop returns original stack and pushed item
def push_pop_identity(pair):
stack_items, item = pair
stack = Stack(stack_items)
# Push item then pop
pushed_stack = stack.push(item)
popped_stack, popped_item = pushed_stack.pop()
return popped_stack.items == stack.items and popped_item == item
prop_test.test_property(
"Push-pop identity",
push_pop_identity,
pair_generator(
list_generator(int_generator(), 0, 10),
int_generator()
)
)
# Property: Push increases size by 1
def push_increases_size(pair):
stack_items, item = pair
stack = Stack(stack_items)
pushed_stack = stack.push(item)
return pushed_stack.size() == stack.size() + 1
prop_test.test_property(
"Push increases size",
push_increases_size,
pair_generator(
list_generator(int_generator(), 0, 10),
int_generator()
)
)
# Property: Peek doesn't change the stack
def peek_no_change(stack_items):
if not stack_items: # Skip empty stacks
return True
stack = Stack(stack_items)
peeked_item = stack.peek()
# Stack should be unchanged and peeked item should be the last item
return (stack.items == stack_items and
peeked_item == stack_items[-1])
prop_test.test_property(
"Peek doesn't change stack",
peek_no_change,
list_generator(int_generator(), 1, 10) # At least 1 item
)
prop_test.report()
test_stack_properties()Python๐ Testing Function Composition and Pipelines
"""
๐ TESTING FUNCTION COMPOSITION AND PIPELINES
Testing composed functions and data processing pipelines
"""
from typing import Dict, Any
import json
print("\n๐ TESTING COMPOSITION AND PIPELINES")
print("=" * 40)
# 1. Testing function pipelines
print("1๏ธโฃ TESTING FUNCTION PIPELINES:")
def pipe(*functions):
"""Compose functions left-to-right"""
def pipeline(x):
result = x
for func in functions:
result = func(result)
return result
return pipeline
# Data processing functions to test
def clean_text(text: str) -> str:
"""Clean text by stripping whitespace and converting to lowercase"""
return text.strip().lower()
def remove_punctuation(text: str) -> str:
"""Remove punctuation from text"""
import string
return text.translate(str.maketrans('', '', string.punctuation))
def split_words(text: str) -> List[str]:
"""Split text into words"""
return text.split()
def count_words(words: List[str]) -> Dict[str, int]:
"""Count word frequencies"""
word_count = {}
for word in words:
word_count[word] = word_count.get(word, 0) + 1
return word_count
def filter_common_words(word_count: Dict[str, int], min_count: int = 2) -> Dict[str, int]:
"""Filter words that appear at least min_count times"""
return {word: count for word, count in word_count.items() if count >= min_count}
class TestPipelines(unittest.TestCase):
"""Test function pipelines"""
def test_text_processing_pipeline(self):
"""Test complete text processing pipeline"""
# Create pipeline
process_text = pipe(
clean_text,
remove_punctuation,
split_words,
count_words
)
# Test with sample text
sample_text = " Hello, World! Hello Python World. "
result = process_text(sample_text)
expected = {'hello': 2, 'world': 2, 'python': 1}
self.assertEqual(result, expected)
def test_pipeline_components(self):
"""Test individual pipeline components"""
# Test clean_text
self.assertEqual(clean_text(" HELLO "), "hello")
# Test remove_punctuation
self.assertEqual(remove_punctuation("hello, world!"), "hello world")
# Test split_words
self.assertEqual(split_words("hello world python"), ["hello", "world", "python"])
# Test count_words
self.assertEqual(
count_words(["hello", "world", "hello"]),
{"hello": 2, "world": 1}
)
def test_pipeline_properties(self):
"""Test pipeline properties"""
# Pipeline should be deterministic
text = "Test text for pipeline"
pipeline = pipe(clean_text, remove_punctuation, split_words, count_words)
result1 = pipeline(text)
result2 = pipeline(text)
self.assertEqual(result1, result2)
# Pipeline should handle empty input
empty_result = pipeline("")
self.assertEqual(empty_result, {})
def test_partial_pipelines(self):
"""Test partial pipeline execution"""
text = "Hello, World!"
# Test first three steps
partial_pipeline = pipe(clean_text, remove_punctuation, split_words)
result = partial_pipeline(text)
expected = ["hello", "world"]
self.assertEqual(result, expected)
# Test that we can continue the pipeline
final_result = count_words(result)
expected_final = {"hello": 1, "world": 1}
self.assertEqual(final_result, expected_final)
# Run pipeline tests
print("Running pipeline tests:")
suite = unittest.TestLoader().loadTestsFromTestCase(TestPipelines)
runner = unittest.TextTestRunner(verbosity=0)
result = runner.run(suite)
print(f"Pipeline tests: {result.testsRun - len(result.failures) - len(result.errors)}/{result.testsRun} passed")
# 2. Testing data transformation pipelines
print(f"\n2๏ธโฃ TESTING DATA TRANSFORMATION PIPELINES:")
@dataclass
class Person:
"""Person data class for testing"""
name: str
age: int
email: str
salary: float
def to_dict(self) -> Dict[str, Any]:
return {
'name': self.name,
'age': self.age,
'email': self.email,
'salary': self.salary
}
def validate_person(person: Person) -> bool:
"""Validate person data"""
return (person.age > 0 and
'@' in person.email and
person.salary >= 0 and
len(person.name.strip()) > 0)
def normalize_email(person: Person) -> Person:
"""Normalize email to lowercase"""
return Person(person.name, person.age, person.email.lower(), person.salary)
def calculate_tax_bracket(person: Person) -> Person:
"""Add tax bracket based on salary"""
if person.salary >= 100000:
bracket = 'high'
elif person.salary >= 50000:
bracket = 'medium'
else:
bracket = 'low'
# Return new person with tax bracket (if we had that field)
# For this example, we'll just return the person unchanged
return person
def person_processing_pipeline(people: List[Person]) -> List[Dict[str, Any]]:
"""Complete person data processing pipeline"""
return [
person.to_dict()
for person in [
calculate_tax_bracket(normalize_email(person))
for person in people
if validate_person(person)
]
]
class TestDataPipeline(unittest.TestCase):
"""Test data processing pipeline"""
def setUp(self):
"""Set up test data"""
self.valid_people = [
Person("Alice Smith", 30, "ALICE@EXAMPLE.COM", 75000.0),
Person("Bob Johnson", 25, "bob@test.com", 45000.0),
Person("Carol Davis", 35, "carol@company.org", 120000.0)
]
self.invalid_people = [
Person("", 30, "invalid@test.com", 50000.0), # Empty name
Person("David Wilson", -5, "david@test.com", 60000.0), # Negative age
Person("Eve Brown", 28, "invalid-email", 40000.0), # Invalid email
Person("Frank Miller", 40, "frank@test.com", -1000.0) # Negative salary
]
def test_validation_function(self):
"""Test person validation"""
for person in self.valid_people:
self.assertTrue(validate_person(person), f"Valid person failed: {person}")
for person in self.invalid_people:
self.assertFalse(validate_person(person), f"Invalid person passed: {person}")
def test_normalization_function(self):
"""Test email normalization"""
person = Person("Test User", 25, "TEST@EXAMPLE.COM", 50000.0)
normalized = normalize_email(person)
self.assertEqual(normalized.email, "test@example.com")
self.assertEqual(normalized.name, person.name)
self.assertEqual(normalized.age, person.age)
self.assertEqual(normalized.salary, person.salary)
def test_pipeline_filtering(self):
"""Test that pipeline filters invalid data"""
mixed_data = self.valid_people + self.invalid_people
result = person_processing_pipeline(mixed_data)
# Should only have valid people
self.assertEqual(len(result), len(self.valid_people))
# All results should have normalized emails
for person_data in result:
self.assertEqual(person_data['email'], person_data['email'].lower())
def test_pipeline_transformation(self):
"""Test pipeline transformations"""
result = person_processing_pipeline(self.valid_people)
# Check that all expected fields are present
for person_data in result:
required_fields = ['name', 'age', 'email', 'salary']
for field in required_fields:
self.assertIn(field, person_data)
# Check email normalization
alice_result = next(p for p in result if p['name'] == 'Alice Smith')
self.assertEqual(alice_result['email'], 'alice@example.com')
def test_pipeline_properties(self):
"""Test pipeline properties"""
# Idempotency: running pipeline twice should give same result
result1 = person_processing_pipeline(self.valid_people)
result2 = person_processing_pipeline(self.valid_people)
self.assertEqual(result1, result2)
# Empty input should return empty output
empty_result = person_processing_pipeline([])
self.assertEqual(empty_result, [])
# Pipeline should not modify original data
original_person = self.valid_people[0]
original_email = original_person.email
person_processing_pipeline([original_person])
self.assertEqual(original_person.email, original_email)
# Run data pipeline tests
print("Running data pipeline tests:")
suite = unittest.TestLoader().loadTestsFromTestCase(TestDataPipeline)
runner = unittest.TextTestRunner(verbosity=0)
result = runner.run(suite)
print(f"Data pipeline tests: {result.testsRun - len(result.failures) - len(result.errors)}/{result.testsRun} passed")Python๐ญ Testing Side Effects and IO
"""
๐ญ TESTING SIDE EFFECTS AND IO
Testing functional code that interacts with external systems
"""
from unittest.mock import Mock, patch, MagicMock
from io import StringIO
import tempfile
import os
print("\n๐ญ TESTING SIDE EFFECTS AND IO")
print("=" * 35)
# 1. Testing IO operations functionally
print("1๏ธโฃ TESTING IO OPERATIONS:")
def read_numbers_from_file(filename: str) -> List[int]:
"""Read numbers from file (one per line)"""
try:
with open(filename, 'r') as f:
return [int(line.strip()) for line in f if line.strip().isdigit()]
except FileNotFoundError:
return []
except ValueError:
return []
def write_numbers_to_file(numbers: List[int], filename: str) -> bool:
"""Write numbers to file"""
try:
with open(filename, 'w') as f:
for number in numbers:
f.write(f"{number}\n")
return True
except Exception:
return False
def process_number_file(input_file: str, output_file: str, processor: Callable[[int], int]) -> bool:
"""Process numbers from input file and write to output file"""
numbers = read_numbers_from_file(input_file)
if not numbers:
return False
processed_numbers = [processor(n) for n in numbers]
return write_numbers_to_file(processed_numbers, output_file)
class TestIOOperations(unittest.TestCase):
"""Test IO operations with temporary files"""
def test_read_numbers_from_file(self):
"""Test reading numbers from file"""
# Create temporary file
with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
f.write("1\n2\n3\n4\n5\n")
temp_filename = f.name
try:
# Test reading
numbers = read_numbers_from_file(temp_filename)
self.assertEqual(numbers, [1, 2, 3, 4, 5])
finally:
# Cleanup
os.unlink(temp_filename)
def test_read_from_nonexistent_file(self):
"""Test reading from nonexistent file"""
numbers = read_numbers_from_file("nonexistent_file.txt")
self.assertEqual(numbers, [])
def test_write_numbers_to_file(self):
"""Test writing numbers to file"""
numbers = [10, 20, 30, 40]
with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
temp_filename = f.name
try:
# Test writing
success = write_numbers_to_file(numbers, temp_filename)
self.assertTrue(success)
# Verify contents
with open(temp_filename, 'r') as f:
content = f.read()
expected_content = "10\n20\n30\n40\n"
self.assertEqual(content, expected_content)
finally:
os.unlink(temp_filename)
def test_process_number_file(self):
"""Test complete file processing pipeline"""
# Create input file
with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
f.write("1\n2\n3\n4\n")
input_filename = f.name
with tempfile.NamedTemporaryFile(delete=False) as f:
output_filename = f.name
try:
# Process file (square each number)
success = process_number_file(input_filename, output_filename, lambda x: x ** 2)
self.assertTrue(success)
# Verify output
result_numbers = read_numbers_from_file(output_filename)
self.assertEqual(result_numbers, [1, 4, 9, 16])
finally:
os.unlink(input_filename)
os.unlink(output_filename)
# Run IO tests
print("Running IO operation tests:")
suite = unittest.TestLoader().loadTestsFromTestCase(TestIOOperations)
runner = unittest.TextTestRunner(verbosity=0)
result = runner.run(suite)
print(f"IO tests: {result.testsRun - len(result.failures) - len(result.errors)}/{result.testsRun} passed")
# 2. Testing with dependency injection
print(f"\n2๏ธโฃ TESTING WITH DEPENDENCY INJECTION:")
# Functional approach: inject dependencies as parameters
def fetch_user_data(user_id: int, http_client: Callable[[str], Dict]) -> Dict[str, Any]:
"""Fetch user data using injected HTTP client"""
url = f"https://api.example.com/users/{user_id}"
response = http_client(url)
if response.get('status_code') == 200:
return response.get('data', {})
else:
return {}
def process_user_data(user_data: Dict[str, Any], validator: Callable[[Dict], bool]) -> Dict[str, Any]:
"""Process user data with injected validator"""
if not validator(user_data):
return {'error': 'Invalid user data'}
return {
'id': user_data.get('id'),
'name': user_data.get('name', '').title(),
'email': user_data.get('email', '').lower(),
'status': 'processed'
}
def user_pipeline(user_id: int,
http_client: Callable[[str], Dict],
validator: Callable[[Dict], bool]) -> Dict[str, Any]:
"""Complete user processing pipeline with injected dependencies"""
user_data = fetch_user_data(user_id, http_client)
return process_user_data(user_data, validator)
class TestDependencyInjection(unittest.TestCase):
"""Test functions with dependency injection"""
def test_fetch_user_data_success(self):
"""Test successful user data fetching"""
# Mock HTTP client
mock_client = Mock(return_value={
'status_code': 200,
'data': {'id': 123, 'name': 'Alice', 'email': 'alice@example.com'}
})
result = fetch_user_data(123, mock_client)
# Verify client was called correctly
mock_client.assert_called_once_with("https://api.example.com/users/123")
# Verify result
expected = {'id': 123, 'name': 'Alice', 'email': 'alice@example.com'}
self.assertEqual(result, expected)
def test_fetch_user_data_failure(self):
"""Test failed user data fetching"""
# Mock HTTP client that returns error
mock_client = Mock(return_value={'status_code': 404})
result = fetch_user_data(999, mock_client)
# Should return empty dict on failure
self.assertEqual(result, {})
def test_process_user_data_valid(self):
"""Test processing valid user data"""
user_data = {'id': 123, 'name': 'alice smith', 'email': 'ALICE@EXAMPLE.COM'}
# Mock validator that returns True
mock_validator = Mock(return_value=True)
result = process_user_data(user_data, mock_validator)
# Verify validator was called
mock_validator.assert_called_once_with(user_data)
# Verify processing
expected = {
'id': 123,
'name': 'Alice Smith',
'email': 'alice@example.com',
'status': 'processed'
}
self.assertEqual(result, expected)
def test_process_user_data_invalid(self):
"""Test processing invalid user data"""
user_data = {'invalid': 'data'}
# Mock validator that returns False
mock_validator = Mock(return_value=False)
result = process_user_data(user_data, mock_validator)
# Should return error for invalid data
expected = {'error': 'Invalid user data'}
self.assertEqual(result, expected)
def test_complete_user_pipeline(self):
"""Test complete user processing pipeline"""
# Mock HTTP client
mock_client = Mock(return_value={
'status_code': 200,
'data': {'id': 456, 'name': 'bob johnson', 'email': 'BOB@TEST.COM'}
})
# Mock validator
mock_validator = Mock(return_value=True)
result = user_pipeline(456, mock_client, mock_validator)
# Verify both dependencies were used
mock_client.assert_called_once()
mock_validator.assert_called_once()
# Verify end-to-end processing
expected = {
'id': 456,
'name': 'Bob Johnson',
'email': 'bob@test.com',
'status': 'processed'
}
self.assertEqual(result, expected)
# Run dependency injection tests
print("Running dependency injection tests:")
suite = unittest.TestLoader().loadTestsFromTestCase(TestDependencyInjection)
runner = unittest.TextTestRunner(verbosity=0)
result = runner.run(suite)
print(f"Dependency injection tests: {result.testsRun - len(result.failures) - len(result.errors)}/{result.testsRun} passed")
# 3. Performance testing for functional code
print(f"\n3๏ธโฃ PERFORMANCE TESTING:")
import time
from functools import wraps
def performance_test(func):
"""Decorator to measure function performance"""
@wraps(func)
def wrapper(*args, **kwargs):
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
execution_time = end_time - start_time
print(f" {func.__name__}: {execution_time:.4f} seconds")
return result, execution_time
return wrapper
class TestPerformance(unittest.TestCase):
"""Test performance characteristics of functional code"""
def test_pure_function_performance(self):
"""Test that pure functions have predictable performance"""
@performance_test
def factorial_iterative(n):
result = 1
for i in range(1, n + 1):
result *= i
return result
@performance_test
def factorial_functional(n):
return reduce(lambda x, y: x * y, range(1, n + 1), 1)
# Test both implementations
n = 1000
result1, time1 = factorial_iterative(n)
result2, time2 = factorial_functional(n)
# Results should be the same
self.assertEqual(result1, result2)
# Both should complete in reasonable time (less than 1 second)
self.assertLess(time1, 1.0)
self.assertLess(time2, 1.0)
def test_list_processing_performance(self):
"""Test performance of functional list operations"""
large_list = list(range(100000))
@performance_test
def imperative_sum_squares(numbers):
total = 0
for n in numbers:
if n % 2 == 0:
total += n * n
return total
@performance_test
def functional_sum_squares(numbers):
return sum(n * n for n in numbers if n % 2 == 0)
# Test both approaches
result1, time1 = imperative_sum_squares(large_list)
result2, time2 = functional_sum_squares(large_list)
# Results should be the same
self.assertEqual(result1, result2)
# Performance should be comparable (within 2x of each other)
ratio = max(time1, time2) / min(time1, time2)
self.assertLess(ratio, 2.0, "Performance difference too large")
# Run performance tests
print("Running performance tests:")
suite = unittest.TestLoader().loadTestsFromTestCase(TestPerformance)
runner = unittest.TextTestRunner(verbosity=0)
result = runner.run(suite)
print(f"Performance tests: {result.testsRun - len(result.failures) - len(result.errors)}/{result.testsRun} passed")
print("\n๐ฏ FUNCTIONAL TESTING SUMMARY:")
print("โ
Pure functions are easy to test - no setup/teardown needed")
print("โ
Property-based testing finds edge cases automatically")
print("โ
Function composition can be tested at multiple levels")
print("โ
Dependency injection makes side effects testable")
print("โ
Performance testing ensures functional code is efficient")
print("โ
Immutable data eliminates test pollution")
print("โ
Deterministic functions produce reliable tests")PythonThis completes the comprehensive section on Testing Functional Code. The content covers:
- Unit Testing Pure Functions – Testing deterministic, side-effect-free functions with traditional unit tests
- Property-Based Testing – Testing function properties and invariants with generated test cases
- Testing Function Composition and Pipelines – Testing composed functions and data processing pipelines
- Testing Side Effects and IO – Strategies for testing functions that interact with external systems using dependency injection and mocking
The functional programming approach to testing emphasizes the benefits of pure functions, immutable data, and composable operations, making tests more reliable, maintainable, and comprehensive than traditional imperative testing approaches.
Would you like me to continue with the next section on “Performance and Optimization” or focus on any specific
16. Performance and Optimization
โก Optimizing Functional Code for Performance
Performance optimization in functional programming requires understanding the trade-offs between immutability, composition, and efficiency. While functional code offers many benefits like safety and maintainability, certain patterns can impact performance if not carefully implemented.
graph TD
A["โก Performance & Optimization"] --> B["๐พ Memory OptimizationEfficient Data Structures"]
A --> C["๐ CPU OptimizationAlgorithm Efficiency"]
A --> D["๐ Lazy EvaluationOn-Demand Computation"]
A --> E["๐ฆ Caching StrategiesMemoization & Storage"]
B --> B1["Persistent data structures"]
B --> B2["Memory pooling"]
B --> B3["Generator expressions"]
C --> C1["Tail call optimization"]
C --> C2["Vectorization"]
C --> C3["Parallel processing"]
D --> D1["Lazy sequences"]
D --> D2["Stream processing"]
D --> D3["Short-circuit evaluation"]
E --> E1["Function memoization"]
E --> E2["Result caching"]
E --> E3["Computed properties"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px
style E fill:#fce4ec,stroke:#e91e63,stroke-width:2px๐ Performance Profiling and Measurement
"""
๐ PERFORMANCE PROFILING AND MEASUREMENT
Tools and techniques for measuring functional code performance
"""
import time
import sys
import tracemalloc
from functools import wraps, lru_cache
from typing import Callable, Any, List, Dict
import cProfile
import pstats
from io import StringIO
import gc
from dataclasses import dataclass
import psutil
import os
print("โก PERFORMANCE AND OPTIMIZATION")
print("=" * 35)
# 1. Performance measurement decorators
print("1๏ธโฃ PERFORMANCE MEASUREMENT:")
def performance_timer(func: Callable) -> Callable:
"""Decorator to measure function execution time"""
@wraps(func)
def wrapper(*args, **kwargs):
start_time = time.perf_counter()
result = func(*args, **kwargs)
end_time = time.perf_counter()
execution_time = end_time - start_time
print(f"โฑ๏ธ {func.__name__}: {execution_time:.6f} seconds")
return result
return wrapper
def memory_profiler(func: Callable) -> Callable:
"""Decorator to measure memory usage"""
@wraps(func)
def wrapper(*args, **kwargs):
# Start tracing
tracemalloc.start()
# Get initial memory usage
process = psutil.Process(os.getpid())
initial_memory = process.memory_info().rss / 1024 / 1024 # MB
# Execute function
result = func(*args, **kwargs)
# Get peak memory usage during execution
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
# Get final memory usage
final_memory = process.memory_info().rss / 1024 / 1024 # MB
print(f"๐ง {func.__name__}:")
print(f" Peak memory during execution: {peak / 1024 / 1024:.2f} MB")
print(f" Memory change: {final_memory - initial_memory:.2f} MB")
return result
return wrapper
def comprehensive_profiler(func: Callable) -> Callable:
"""Decorator combining time and memory profiling"""
@wraps(func)
def wrapper(*args, **kwargs):
# Start profiling
tracemalloc.start()
start_time = time.perf_counter()
# Execute function
result = func(*args, **kwargs)
# Stop profiling
end_time = time.perf_counter()
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
execution_time = end_time - start_time
print(f"๐ {func.__name__} Performance Report:")
print(f" Execution time: {execution_time:.6f} seconds")
print(f" Peak memory: {peak / 1024 / 1024:.2f} MB")
print(f" Current memory: {current / 1024 / 1024:.2f} MB")
return result
return wrapper
# Test functions to profile
@performance_timer
def functional_sum_squares(numbers: List[int]) -> int:
"""Functional approach using map and sum"""
return sum(map(lambda x: x ** 2, numbers))
@performance_timer
def imperative_sum_squares(numbers: List[int]) -> int:
"""Imperative approach using loop"""
total = 0
for x in numbers:
total += x ** 2
return total
@performance_timer
def comprehension_sum_squares(numbers: List[int]) -> int:
"""List comprehension approach"""
return sum([x ** 2 for x in numbers])
@performance_timer
def generator_sum_squares(numbers: List[int]) -> int:
"""Generator expression approach"""
return sum(x ** 2 for x in numbers)
# Test with different data sizes
test_sizes = [1000, 10000, 100000]
print("Comparing different approaches to sum of squares:")
for size in test_sizes:
print(f"\nTesting with {size:,} numbers:")
test_data = list(range(size))
result1 = functional_sum_squares(test_data)
result2 = imperative_sum_squares(test_data)
result3 = comprehension_sum_squares(test_data)
result4 = generator_sum_squares(test_data)
# Verify all results are the same
assert result1 == result2 == result3 == result4, "Results don't match!"
# 2. Advanced profiling with cProfile
print(f"\n2๏ธโฃ ADVANCED PROFILING:")
def detailed_profile(func: Callable, *args, **kwargs):
"""Profile function execution in detail"""
profiler = cProfile.Profile()
profiler.enable()
result = func(*args, **kwargs)
profiler.disable()
# Capture profiling output
string_io = StringIO()
ps = pstats.Stats(profiler, stream=string_io).sort_stats('cumulative')
ps.print_stats(10) # Top 10 functions
profile_output = string_io.getvalue()
print(f"๐ Detailed profiling for {func.__name__}:")
print(profile_output)
return result
# Complex functional operations to profile
def fibonacci_recursive(n: int) -> int:
"""Recursive fibonacci (inefficient)"""
if n <= 1:
return n
return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)
@lru_cache(maxsize=None)
def fibonacci_memoized(n: int) -> int:
"""Memoized fibonacci (efficient)"""
if n <= 1:
return n
return fibonacci_memoized(n - 1) + fibonacci_memoized(n - 2)
def fibonacci_iterative(n: int) -> int:
"""Iterative fibonacci (most efficient)"""
if n <= 1:
return n
a, b = 0, 1
for _ in range(2, n + 1):
a, b = b, a + b
return b
print("Comparing Fibonacci implementations:")
# Profile different Fibonacci implementations
test_n = 25
print(f"\nCalculating Fibonacci({test_n}):")
print("Recursive approach:")
result1 = detailed_profile(fibonacci_recursive, test_n)
print(f"Result: {result1}")
print("\nMemoized approach:")
fibonacci_memoized.cache_clear() # Clear cache for fair comparison
result2 = detailed_profile(fibonacci_memoized, test_n)
print(f"Result: {result2}")
print("\nIterative approach:")
result3 = detailed_profile(fibonacci_iterative, test_n)
print(f"Result: {result3}")
# 3. Memory usage analysis
print(f"\n3๏ธโฃ MEMORY USAGE ANALYSIS:")
@memory_profiler
def create_large_list_imperative(n: int) -> List[int]:
"""Create large list imperatively"""
result = []
for i in range(n):
result.append(i ** 2)
return result
@memory_profiler
def create_large_list_functional(n: int) -> List[int]:
"""Create large list functionally"""
return list(map(lambda x: x ** 2, range(n)))
@memory_profiler
def create_large_list_comprehension(n: int) -> List[int]:
"""Create large list with comprehension"""
return [x ** 2 for x in range(n)]
@memory_profiler
def create_large_generator(n: int):
"""Create generator (lazy evaluation)"""
return (x ** 2 for x in range(n))
print("Memory usage comparison for creating large sequences:")
test_size = 1000000
print(f"\nCreating sequence of {test_size:,} squared numbers:")
# Test memory usage
list1 = create_large_list_imperative(test_size)
list2 = create_large_list_functional(test_size)
list3 = create_large_list_comprehension(test_size)
gen1 = create_large_generator(test_size)
print(f"List size: {sys.getsizeof(list1) / 1024 / 1024:.2f} MB")
print(f"Generator size: {sys.getsizeof(gen1) / 1024:.2f} KB")
# Clean up
del list1, list2, list3
gc.collect()Python๐ Optimization Techniques
"""
๐ OPTIMIZATION TECHNIQUES
Practical strategies for optimizing functional code
"""
import operator
from itertools import accumulate, chain, islice
from collections import defaultdict, Counter
import numpy as np
print("\n๐ OPTIMIZATION TECHNIQUES")
print("=" * 30)
# 1. Efficient data structures and operations
print("1๏ธโฃ EFFICIENT DATA STRUCTURES:")
@comprehensive_profiler
def inefficient_word_count(text: str) -> Dict[str, int]:
"""Inefficient word counting using functional style"""
words = text.lower().split()
return {
word: len(list(filter(lambda w: w == word, words)))
for word in set(words)
}
@comprehensive_profiler
def efficient_word_count(text: str) -> Dict[str, int]:
"""Efficient word counting using Counter"""
return Counter(text.lower().split())
@comprehensive_profiler
def manual_word_count(text: str) -> Dict[str, int]:
"""Manual word counting with defaultdict"""
word_count = defaultdict(int)
for word in text.lower().split():
word_count[word] += 1
return dict(word_count)
# Test word counting efficiency
sample_text = """
Functional programming is a programming paradigm that treats computation as the evaluation
of mathematical functions and avoids changing state and mutable data. It is a declarative
programming paradigm in that programming is done with expressions or declarations instead
of statements. Functional programming emphasizes the application of functions, in contrast
to the imperative programming style, which emphasizes changes in state.
""" * 1000 # Repeat to make it larger
print("Word counting comparison:")
result1 = inefficient_word_count(sample_text)
result2 = efficient_word_count(sample_text)
result3 = manual_word_count(sample_text)
print(f"Results match: {dict(result2) == result3}")
# 2. Lazy evaluation optimization
print(f"\n2๏ธโฃ LAZY EVALUATION OPTIMIZATION:")
@comprehensive_profiler
def eager_pipeline(data: List[int]) -> List[int]:
"""Eager evaluation pipeline"""
step1 = [x * 2 for x in data]
step2 = [x + 1 for x in step1]
step3 = [x for x in step2 if x % 3 == 0]
return step3[:100] # Only need first 100
@comprehensive_profiler
def lazy_pipeline(data: List[int]) -> List[int]:
"""Lazy evaluation pipeline"""
pipeline = (x * 2 + 1 for x in data)
filtered = (x for x in pipeline if x % 3 == 0)
return list(islice(filtered, 100)) # Only compute what we need
@comprehensive_profiler
def optimized_pipeline(data: List[int]) -> List[int]:
"""Optimized pipeline with early termination"""
result = []
for x in data:
transformed = x * 2 + 1
if transformed % 3 == 0:
result.append(transformed)
if len(result) >= 100:
break
return result
print("Pipeline evaluation comparison:")
large_data = list(range(1000000))
result1 = eager_pipeline(large_data)
result2 = lazy_pipeline(large_data)
result3 = optimized_pipeline(large_data)
print(f"Results match: {result1 == result2 == result3}")
# 3. Function composition optimization
print(f"\n3๏ธโฃ FUNCTION COMPOSITION OPTIMIZATION:")
# Inefficient composition
def compose_many(*functions):
"""Compose many functions (can be inefficient for deep nesting)"""
def composed(x):
result = x
for func in functions:
result = func(result)
return result
return composed
# Optimized composition with memoization
class MemoizedComposition:
"""Memoized function composition"""
def __init__(self, *functions):
self.functions = functions
self.cache = {}
def __call__(self, x):
if x not in self.cache:
result = x
for func in self.functions:
result = func(result)
self.cache[x] = result
return self.cache[x]
def clear_cache(self):
self.cache.clear()
@comprehensive_profiler
def test_regular_composition(data: List[int]) -> List[int]:
"""Test regular function composition"""
composed = compose_many(
lambda x: x + 1,
lambda x: x * 2,
lambda x: x - 3,
lambda x: x ** 2,
lambda x: x % 1000
)
return [composed(x) for x in data]
@comprehensive_profiler
def test_memoized_composition(data: List[int]) -> List[int]:
"""Test memoized function composition"""
composed = MemoizedComposition(
lambda x: x + 1,
lambda x: x * 2,
lambda x: x - 3,
lambda x: x ** 2,
lambda x: x % 1000
)
return [composed(x) for x in data]
print("Function composition optimization:")
# Test with repeated values to see memoization benefits
test_data = [i % 100 for i in range(10000)] # Lots of repeated values
result1 = test_regular_composition(test_data)
result2 = test_memoized_composition(test_data)
print(f"Results match: {result1 == result2}")
# 4. Vectorization with NumPy
print(f"\n4๏ธโฃ VECTORIZATION:")
@comprehensive_profiler
def pure_python_operations(data: List[float]) -> float:
"""Pure Python mathematical operations"""
# Apply complex mathematical transformation
step1 = [x ** 2 + 2 * x + 1 for x in data]
step2 = [x / (x + 1) for x in step1]
step3 = [x * 0.5 + 0.25 for x in step2]
return sum(step3)
@comprehensive_profiler
def numpy_vectorized_operations(data: np.ndarray) -> float:
"""Vectorized operations with NumPy"""
# Same operations but vectorized
step1 = data ** 2 + 2 * data + 1
step2 = step1 / (step1 + 1)
step3 = step1 * 0.5 + 0.25
return float(np.sum(step3))
print("Vectorization comparison:")
python_data = [float(i) for i in range(1000000)]
numpy_data = np.array(python_data)
result1 = pure_python_operations(python_data)
result2 = numpy_vectorized_operations(numpy_data)
print(f"Results close: {abs(result1 - result2) < 1e-10}")
# 5. Memory-efficient functional patterns
print(f"\n5๏ธโฃ MEMORY-EFFICIENT PATTERNS:")
@comprehensive_profiler
def memory_inefficient_processing(data: List[int]) -> int:
"""Memory-inefficient processing creating many intermediate lists"""
squared = [x ** 2 for x in data]
filtered = [x for x in squared if x % 2 == 0]
doubled = [x * 2 for x in filtered]
return sum(doubled)
@comprehensive_profiler
def memory_efficient_processing(data: List[int]) -> int:
"""Memory-efficient processing using generators"""
return sum(
x * 2
for x in (x ** 2 for x in data)
if x % 2 == 0
)
@comprehensive_profiler
def stream_processing(data: List[int]) -> int:
"""Stream processing with itertools"""
squared_stream = map(lambda x: x ** 2, data)
filtered_stream = filter(lambda x: x % 2 == 0, squared_stream)
doubled_stream = map(lambda x: x * 2, filtered_stream)
return sum(doubled_stream)
print("Memory efficiency comparison:")
large_dataset = list(range(1000000))
result1 = memory_inefficient_processing(large_dataset)
result2 = memory_efficient_processing(large_dataset)
result3 = stream_processing(large_dataset)
print(f"Results match: {result1 == result2 == result3}")
# 6. Caching and memoization strategies
print(f"\n6๏ธโฃ CACHING STRATEGIES:")
from functools import lru_cache
import time
# Different caching approaches
@lru_cache(maxsize=128)
def expensive_computation_lru(n: int) -> int:
"""Expensive computation with LRU cache"""
time.sleep(0.001) # Simulate expensive operation
return sum(i ** 2 for i in range(n))
class CustomCache:
"""Custom cache with statistics"""
def __init__(self, maxsize=128):
self.cache = {}
self.maxsize = maxsize
self.hits = 0
self.misses = 0
def get(self, key):
if key in self.cache:
self.hits += 1
return self.cache[key]
else:
self.misses += 1
return None
def put(self, key, value):
if len(self.cache) >= self.maxsize:
# Remove oldest entry (simple FIFO)
oldest_key = next(iter(self.cache))
del self.cache[oldest_key]
self.cache[key] = value
def stats(self):
total = self.hits + self.misses
hit_rate = self.hits / total if total > 0 else 0
return {
'hits': self.hits,
'misses': self.misses,
'hit_rate': hit_rate,
'cache_size': len(self.cache)
}
custom_cache = CustomCache(maxsize=64)
def expensive_computation_custom(n: int) -> int:
"""Expensive computation with custom cache"""
cached_result = custom_cache.get(n)
if cached_result is not None:
return cached_result
time.sleep(0.001) # Simulate expensive operation
result = sum(i ** 2 for i in range(n))
custom_cache.put(n, result)
return result
@comprehensive_profiler
def test_lru_cache(iterations: int):
"""Test LRU cache performance"""
return [expensive_computation_lru(i % 50) for i in range(iterations)]
@comprehensive_profiler
def test_custom_cache(iterations: int):
"""Test custom cache performance"""
return [expensive_computation_custom(i % 50) for i in range(iterations)]
print("Caching strategy comparison:")
iterations = 1000
result1 = test_lru_cache(iterations)
result2 = test_custom_cache(iterations)
print(f"LRU cache info: {expensive_computation_lru.cache_info()}")
print(f"Custom cache stats: {custom_cache.stats()}")
print(f"Results match: {result1 == result2}")
print("\n๐ฏ OPTIMIZATION SUMMARY:")
print("โ
Use appropriate data structures (Counter, defaultdict)")
print("โ
Leverage lazy evaluation and generators")
print("โ
Implement memoization for expensive computations")
print("โ
Consider vectorization with NumPy for math operations")
print("โ
Avoid creating unnecessary intermediate collections")
print("โ
Profile code to identify bottlenecks")
print("โ
Balance readability with performance optimizations")Python๐ง Memory Management and Immutability
"""
๐ง MEMORY MANAGEMENT AND IMMUTABILITY
Strategies for efficient memory usage in functional programming
"""
import sys
import weakref
from collections import namedtuple
import copy
print("\n๐ง MEMORY MANAGEMENT STRATEGIES")
print("=" * 35)
# 1. Efficient immutable data structures
print("1๏ธโฃ EFFICIENT IMMUTABLE DATA STRUCTURES:")
# Naive approach - creates new objects frequently
class InefficientImmutableList:
"""Inefficient immutable list implementation"""
def __init__(self, items=None):
self._items = tuple(items or [])
def append(self, item):
"""Add item (creates new list)"""
return InefficientImmutableList(self._items + (item,))
def prepend(self, item):
"""Add item at beginning (creates new list)"""
return InefficientImmutableList((item,) + self._items)
def __len__(self):
return len(self._items)
def __getitem__(self, index):
return self._items[index]
def __str__(self):
return f"ImmutableList({list(self._items)})"
# More efficient approach using structural sharing
class EfficientImmutableList:
"""More efficient immutable list with structural sharing"""
def __init__(self, items=None, shared_prefix=None, new_items=None):
if shared_prefix is not None and new_items is not None:
self._shared_prefix = shared_prefix
self._new_items = tuple(new_items)
else:
self._shared_prefix = None
self._new_items = tuple(items or [])
def append(self, item):
"""Add item efficiently"""
return EfficientImmutableList(
shared_prefix=self,
new_items=[item]
)
def _flatten(self):
"""Flatten the structure when needed"""
if self._shared_prefix is None:
return self._new_items
else:
return self._shared_prefix._flatten() + self._new_items
def __len__(self):
if self._shared_prefix is None:
return len(self._new_items)
else:
return len(self._shared_prefix) + len(self._new_items)
def __getitem__(self, index):
flattened = self._flatten()
return flattened[index]
def __str__(self):
return f"EfficientList({list(self._flatten())})"
@memory_profiler
def test_inefficient_list():
"""Test memory usage of inefficient immutable list"""
lst = InefficientImmutableList()
for i in range(1000):
lst = lst.append(i)
return lst
@memory_profiler
def test_efficient_list():
"""Test memory usage of efficient immutable list"""
lst = EfficientImmutableList()
for i in range(1000):
lst = lst.append(i)
return lst
print("Comparing immutable list implementations:")
inefficient_list = test_inefficient_list()
efficient_list = test_efficient_list()
print(f"Inefficient list size: {sys.getsizeof(inefficient_list)} bytes")
print(f"Efficient list size: {sys.getsizeof(efficient_list)} bytes")
# 2. Memory-conscious functional patterns
print(f"\n2๏ธโฃ MEMORY-CONSCIOUS PATTERNS:")
@memory_profiler
def memory_heavy_functional_ops(data: List[int]) -> int:
"""Memory-heavy functional operations"""
# Creates many intermediate lists
doubled = [x * 2 for x in data]
squared = [x ** 2 for x in doubled]
filtered = [x for x in squared if x > 100]
summed = sum(filtered)
return summed
@memory_profiler
def memory_light_functional_ops(data: List[int]) -> int:
"""Memory-light functional operations"""
# Uses generators to avoid intermediate storage
return sum(
x ** 2
for x in (x * 2 for x in data)
if x ** 2 > 100
)
@memory_profiler
def iterator_based_ops(data: List[int]) -> int:
"""Iterator-based operations"""
doubled = map(lambda x: x * 2, data)
squared = map(lambda x: x ** 2, doubled)
filtered = filter(lambda x: x > 100, squared)
return sum(filtered)
print("Memory usage comparison for functional operations:")
test_data = list(range(100000))
result1 = memory_heavy_functional_ops(test_data)
result2 = memory_light_functional_ops(test_data)
result3 = iterator_based_ops(test_data)
print(f"Results match: {result1 == result2 == result3}")
# 3. Efficient copying strategies
print(f"\n3๏ธโฃ COPYING STRATEGIES:")
@dataclass
class DataRecord:
"""Sample data record for copying tests"""
id: int
name: str
values: List[int]
metadata: Dict[str, Any]
def shallow_update(record: DataRecord, **updates) -> DataRecord:
"""Shallow update using copy and update"""
new_record = copy.copy(record)
for key, value in updates.items():
setattr(new_record, key, value)
return new_record
def deep_update(record: DataRecord, **updates) -> DataRecord:
"""Deep update using deep copy"""
new_record = copy.deepcopy(record)
for key, value in updates.items():
setattr(new_record, key, value)
return new_record
def efficient_update(record: DataRecord, **updates) -> DataRecord:
"""Efficient update using dataclass replace"""
return record.__class__(
id=updates.get('id', record.id),
name=updates.get('name', record.name),
values=updates.get('values', record.values),
metadata=updates.get('metadata', record.metadata)
)
@comprehensive_profiler
def test_shallow_updates():
"""Test shallow update performance"""
record = DataRecord(1, "test", [1, 2, 3], {"key": "value"})
results = []
for i in range(10000):
updated = shallow_update(record, id=i)
results.append(updated)
return results
@comprehensive_profiler
def test_deep_updates():
"""Test deep update performance"""
record = DataRecord(1, "test", [1, 2, 3], {"key": "value"})
results = []
for i in range(10000):
updated = deep_update(record, id=i)
results.append(updated)
return results
@comprehensive_profiler
def test_efficient_updates():
"""Test efficient update performance"""
record = DataRecord(1, "test", [1, 2, 3], {"key": "value"})
results = []
for i in range(10000):
updated = efficient_update(record, id=i)
results.append(updated)
return results
print("Copying strategy comparison:")
results1 = test_shallow_updates()
results2 = test_deep_updates()
results3 = test_efficient_updates()
print(f"All methods produced same number of results: {len(results1) == len(results2) == len(results3)}")
# 4. Memory pooling and object reuse
print(f"\n4๏ธโฃ MEMORY POOLING:")
class ObjectPool:
"""Simple object pool for reusing objects"""
def __init__(self, create_func, reset_func, max_size=100):
self.create_func = create_func
self.reset_func = reset_func
self.max_size = max_size
self.pool = []
self.created_count = 0
self.reused_count = 0
def acquire(self):
"""Acquire object from pool or create new one"""
if self.pool:
obj = self.pool.pop()
self.reset_func(obj)
self.reused_count += 1
return obj
else:
self.created_count += 1
return self.create_func()
def release(self, obj):
"""Release object back to pool"""
if len(self.pool) < self.max_size:
self.pool.append(obj)
def stats(self):
return {
'created': self.created_count,
'reused': self.reused_count,
'pool_size': len(self.pool)
}
# Example: Pool for list objects
def create_list():
return []
def reset_list(lst):
lst.clear()
list_pool = ObjectPool(create_list, reset_list, max_size=50)
@comprehensive_profiler
def test_with_pool():
"""Test operations using object pool"""
results = []
for i in range(1000):
# Acquire list from pool
work_list = list_pool.acquire()
# Do some work
for j in range(100):
work_list.append(j * i)
# Store result
results.append(sum(work_list))
# Release back to pool
list_pool.release(work_list)
return results
@comprehensive_profiler
def test_without_pool():
"""Test operations without object pool"""
results = []
for i in range(1000):
# Create new list each time
work_list = []
# Do some work
for j in range(100):
work_list.append(j * i)
# Store result
results.append(sum(work_list))
return results
print("Object pooling comparison:")
results1 = test_with_pool()
results2 = test_without_pool()
print(f"Pool statistics: {list_pool.stats()}")
print(f"Results match: {results1 == results2}")
# 5. Weak references for memory management
print(f"\n5๏ธโฃ WEAK REFERENCES:")
class CachedComputation:
"""Computation cache using weak references"""
def __init__(self):
# Use weak references so cached results can be garbage collected
self._cache = weakref.WeakValueDictionary()
self.hits = 0
self.misses = 0
def compute(self, key: str, computation_func: Callable):
"""Compute and cache result"""
if key in self._cache:
self.hits += 1
return self._cache[key]
self.misses += 1
result = computation_func()
# Store in weak reference cache
# Result will be garbage collected when no longer referenced elsewhere
try:
self._cache[key] = result
except TypeError:
# Some objects can't be weakly referenced
pass
return result
def stats(self):
return {
'hits': self.hits,
'misses': self.misses,
'cache_size': len(self._cache)
}
class ExpensiveResult:
"""Sample class for expensive computation results"""
def __init__(self, data):
self.data = data
self.size = len(data) if hasattr(data, '__len__') else 1
# Test weak reference cache
cached_computation = CachedComputation()
@performance_timer
def test_weak_reference_cache():
"""Test weak reference caching"""
results = []
# Compute and cache some results
for i in range(10):
result = cached_computation.compute(
f"computation_{i}",
lambda: ExpensiveResult([j ** 2 for j in range(1000)])
)
results.append(result)
# Access some cached results
for i in range(5):
result = cached_computation.compute(
f"computation_{i}",
lambda: ExpensiveResult([j ** 2 for j in range(1000)])
)
print(f"Cache stats: {cached_computation.stats()}")
# Clear strong references to allow garbage collection
results.clear()
# Force garbage collection
import gc
gc.collect()
print(f"Cache stats after GC: {cached_computation.stats()}")
print("Testing weak reference cache:")
test_weak_reference_cache()
print("\n๐ฏ MEMORY MANAGEMENT SUMMARY:")
print("โ
Use structural sharing for immutable data structures")
print("โ
Prefer generators and iterators over lists when possible")
print("โ
Implement efficient copying strategies")
print("โ
Consider object pooling for frequently created/destroyed objects")
print("โ
Use weak references for caches to allow garbage collection")
print("โ
Monitor memory usage and identify memory leaks")
print("โ
Balance memory efficiency with code readability")PythonThis completes the comprehensive section on Performance and Optimization. The content covers:
- Performance Profiling and Measurement – Tools and decorators for measuring execution time and memory usage
- Optimization Techniques – Practical strategies including efficient data structures, lazy evaluation, vectorization, and caching
- Memory Management and Immutability – Strategies for efficient memory usage including structural sharing, memory pooling, and weak references
The optimization patterns shown here demonstrate how to maintain the benefits of functional programming while achieving good performance through careful implementation choices and profiling-guided optimizations.
17. Real-World Projects and Best Practices
๐๏ธ Building Production-Ready Functional Applications
This final section demonstrates how to apply functional programming principles in real-world projects, combining all the concepts we’ve learned into complete, production-ready applications with proper architecture, testing, and deployment considerations.
graph TD
A["๐๏ธ Real-World FP Projects"] --> B["๐ Architecture PatternsClean Architecture"]
A --> C["๐ง Development PracticesTesting & CI/CD"]
A --> D["๐ Deployment StrategiesContainerization & Scaling"]
A --> E["๐ Monitoring & ObservabilityLogging & Metrics"]
B --> B1["Layered architecture"]
B --> B2["Dependency injection"]
B --> B3["Pure business logic"]
C --> C1["Property-based testing"]
C --> C2["Integration testing"]
C --> C3["Continuous integration"]
D --> D1["Docker containers"]
D --> D2["Microservices"]
D --> D3["Auto-scaling"]
E --> E1["Structured logging"]
E --> E2["Metrics collection"]
E3["Error tracking"]
style A fill:#e8f5e8,stroke:#4caf50,stroke-width:3px
style B fill:#e3f2fd,stroke:#2196f3,stroke-width:2px
style C fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style D fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px
style E fill:#fce4ec,stroke:#e91e63,stroke-width:2px๐ฏ Project 1: Functional Data Pipeline Service
"""
๐ฏ PROJECT 1: FUNCTIONAL DATA PIPELINE SERVICE
A complete microservice for processing data using functional programming patterns
"""
import asyncio
import json
import logging
from datetime import datetime, timedelta
from typing import Dict, List, Any, Optional, Callable, AsyncIterator
from dataclasses import dataclass, asdict
from pathlib import Path
import aiohttp
from aiohttp import web
import aiofiles
import structlog
from functools import reduce, partial
import uvloop
print("๐๏ธ REAL-WORLD PROJECTS AND BEST PRACTICES")
print("=" * 45)
# 1. Domain Models (Immutable Data Structures)
print("1๏ธโฃ DOMAIN MODELS:")
@dataclass(frozen=True)
class DataRecord:
"""Immutable data record"""
id: str
timestamp: datetime
source: str
data: Dict[str, Any]
metadata: Optional[Dict[str, Any]] = None
def with_data(self, **updates) -> 'DataRecord':
"""Create new record with updated data"""
new_data = {**self.data, **updates}
return DataRecord(
self.id, self.timestamp, self.source,
new_data, self.metadata
)
def with_metadata(self, **updates) -> 'DataRecord':
"""Create new record with updated metadata"""
current_meta = self.metadata or {}
new_metadata = {**current_meta, **updates}
return DataRecord(
self.id, self.timestamp, self.source,
self.data, new_metadata
)
@dataclass(frozen=True)
class ProcessingResult:
"""Result of processing pipeline"""
processed_records: List[DataRecord]
errors: List[str]
metrics: Dict[str, Any]
@property
def success_count(self) -> int:
return len(self.processed_records)
@property
def error_count(self) -> int:
return len(self.errors)
@property
def success_rate(self) -> float:
total = self.success_count + self.error_count
return self.success_count / total if total > 0 else 0.0
@dataclass(frozen=True)
class PipelineConfig:
"""Configuration for data pipeline"""
name: str
input_source: str
output_destination: str
transformations: List[str]
batch_size: int = 100
timeout_seconds: int = 30
retry_attempts: int = 3
# 2. Pure Business Logic Functions
print("\n2๏ธโฃ PURE BUSINESS LOGIC:")
def validate_record(record: DataRecord) -> DataRecord:
"""Pure function to validate data record"""
errors = []
# Check required fields
required_fields = ['id', 'type', 'value']
for field in required_fields:
if field not in record.data:
errors.append(f"Missing required field: {field}")
# Validate data types
if 'value' in record.data:
try:
float(record.data['value'])
except (ValueError, TypeError):
errors.append("Value must be numeric")
if errors:
return record.with_metadata(validation_errors=errors)
return record.with_metadata(validated=True)
def enrich_record(record: DataRecord) -> DataRecord:
"""Pure function to enrich record with computed fields"""
if record.metadata and record.metadata.get('validation_errors'):
return record # Don't enrich invalid records
enrichments = {}
# Add computed timestamp fields
enrichments['processing_date'] = datetime.now().isoformat()
enrichments['day_of_week'] = record.timestamp.strftime('%A')
# Add value category
if 'value' in record.data:
value = float(record.data['value'])
if value >= 100:
enrichments['value_category'] = 'high'
elif value >= 50:
enrichments['value_category'] = 'medium'
else:
enrichments['value_category'] = 'low'
return record.with_data(**enrichments)
def aggregate_records(records: List[DataRecord]) -> Dict[str, Any]:
"""Pure function to aggregate processed records"""
if not records:
return {'total_records': 0}
# Aggregate metrics
total_value = sum(
float(r.data.get('value', 0))
for r in records
if 'value' in r.data
)
categories = [
r.data.get('value_category', 'unknown')
for r in records
]
category_counts = {}
for category in categories:
category_counts[category] = category_counts.get(category, 0) + 1
return {
'total_records': len(records),
'total_value': total_value,
'average_value': total_value / len(records) if records else 0,
'category_distribution': category_counts,
'processing_timestamp': datetime.now().isoformat()
}
# 3. Functional Pipeline Composition
print("\n3๏ธโฃ FUNCTIONAL PIPELINE:")
class DataPipeline:
"""Functional data processing pipeline"""
def __init__(self, config: PipelineConfig, logger=None):
self.config = config
self.logger = logger or structlog.get_logger()
async def process_batch(self, records: List[DataRecord]) -> ProcessingResult:
"""Process a batch of records through the pipeline"""
self.logger.info("Processing batch", batch_size=len(records))
# Apply transformations in sequence
processed_records = []
errors = []
for record in records:
try:
# Apply pure transformations
validated = validate_record(record)
# Skip invalid records
if validated.metadata and validated.metadata.get('validation_errors'):
errors.extend(validated.metadata['validation_errors'])
continue
enriched = enrich_record(validated)
processed_records.append(enriched)
except Exception as e:
errors.append(f"Processing error for record {record.id}: {str(e)}")
self.logger.error("Record processing failed",
record_id=record.id, error=str(e))
# Generate metrics
metrics = aggregate_records(processed_records)
result = ProcessingResult(processed_records, errors, metrics)
self.logger.info("Batch processed",
success_count=result.success_count,
error_count=result.error_count,
success_rate=result.success_rate)
return result
# 4. I/O Layer (Separated from Business Logic)
print("\n4๏ธโฃ I/O LAYER:")
class DataSource:
"""Abstract data source"""
async def read_records(self, limit: Optional[int] = None) -> AsyncIterator[DataRecord]:
"""Read records from source"""
raise NotImplementedError
class FileDataSource(DataSource):
"""File-based data source"""
def __init__(self, file_path: str):
self.file_path = Path(file_path)
async def read_records(self, limit: Optional[int] = None) -> AsyncIterator[DataRecord]:
"""Read records from JSON lines file"""
count = 0
if not self.file_path.exists():
return
async with aiofiles.open(self.file_path, 'r') as f:
async for line in f:
if limit and count >= limit:
break
try:
data = json.loads(line.strip())
record = DataRecord(
id=data.get('id', f'record_{count}'),
timestamp=datetime.fromisoformat(data.get('timestamp', datetime.now().isoformat())),
source='file',
data=data
)
yield record
count += 1
except json.JSONDecodeError as e:
continue
class DataSink:
"""Abstract data sink"""
async def write_result(self, result: ProcessingResult) -> bool:
"""Write processing result"""
raise NotImplementedError
class FileDataSink(DataSink):
"""File-based data sink"""
def __init__(self, output_dir: str):
self.output_dir = Path(output_dir)
self.output_dir.mkdir(exist_ok=True)
async def write_result(self, result: ProcessingResult) -> bool:
"""Write result to files"""
try:
# Write processed records
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
records_file = self.output_dir / f'processed_records_{timestamp}.jsonl'
async with aiofiles.open(records_file, 'w') as f:
for record in result.processed_records:
record_dict = asdict(record)
record_dict['timestamp'] = record.timestamp.isoformat()
await f.write(json.dumps(record_dict) + '\n')
# Write metrics
metrics_file = self.output_dir / f'metrics_{timestamp}.json'
async with aiofiles.open(metrics_file, 'w') as f:
await f.write(json.dumps(result.metrics, indent=2))
return True
except Exception as e:
structlog.get_logger().error("Failed to write result", error=str(e))
return False
# 5. Service Layer with Dependency Injection
print("\n5๏ธโฃ SERVICE LAYER:")
class DataProcessingService:
"""Main service coordinating data processing"""
def __init__(self,
pipeline: DataPipeline,
data_source: DataSource,
data_sink: DataSink,
logger=None):
self.pipeline = pipeline
self.data_source = data_source
self.data_sink = data_sink
self.logger = logger or structlog.get_logger()
async def process_stream(self, batch_size: int = 100) -> Dict[str, Any]:
"""Process data stream in batches"""
self.logger.info("Starting data processing", batch_size=batch_size)
total_processed = 0
total_errors = 0
batch_count = 0
try:
# Process in batches
batch = []
async for record in self.data_source.read_records():
batch.append(record)
if len(batch) >= batch_size:
# Process batch
result = await self.pipeline.process_batch(batch)
# Save results
await self.data_sink.write_result(result)
# Update counters
total_processed += result.success_count
total_errors += result.error_count
batch_count += 1
# Clear batch
batch = []
self.logger.info("Batch completed",
batch_number=batch_count,
processed=result.success_count,
errors=result.error_count)
# Process remaining records
if batch:
result = await self.pipeline.process_batch(batch)
await self.data_sink.write_result(result)
total_processed += result.success_count
total_errors += result.error_count
batch_count += 1
summary = {
'total_processed': total_processed,
'total_errors': total_errors,
'batches_processed': batch_count,
'success_rate': total_processed / (total_processed + total_errors) if (total_processed + total_errors) > 0 else 0
}
self.logger.info("Processing completed", **summary)
return summary
except Exception as e:
self.logger.error("Processing failed", error=str(e))
raise
# 6. Web API Layer
print("\n6๏ธโฃ WEB API LAYER:")
class DataProcessingAPI:
"""REST API for data processing service"""
def __init__(self, service: DataProcessingService):
self.service = service
self.app = web.Application()
self.setup_routes()
def setup_routes(self):
"""Setup API routes"""
self.app.router.add_post('/api/v1/process', self.process_data)
self.app.router.add_get('/api/v1/health', self.health_check)
self.app.router.add_get('/api/v1/metrics', self.get_metrics)
async def process_data(self, request: web.Request) -> web.Response:
"""Process data endpoint"""
try:
# Parse request
data = await request.json()
batch_size = data.get('batch_size', 100)
# Process data
result = await self.service.process_stream(batch_size)
return web.json_response({
'status': 'success',
'result': result
})
except Exception as e:
return web.json_response({
'status': 'error',
'message': str(e)
}, status=500)
async def health_check(self, request: web.Request) -> web.Response:
"""Health check endpoint"""
return web.json_response({
'status': 'healthy',
'timestamp': datetime.now().isoformat()
})
async def get_metrics(self, request: web.Request) -> web.Response:
"""Get service metrics"""
return web.json_response({
'service': 'data-processing',
'uptime': '1h 23m', # Would be calculated from start time
'memory_usage': '256MB', # Would be actual metrics
'processed_records_today': 1500
})
# 7. Application Factory and Dependency Injection
print("\n7๏ธโฃ APPLICATION FACTORY:")
def create_application(config_path: str = None) -> web.Application:
"""Create and configure the application"""
# Setup structured logging
structlog.configure(
processors=[
structlog.stdlib.filter_by_level,
structlog.stdlib.add_logger_name,
structlog.stdlib.add_log_level,
structlog.stdlib.PositionalArgumentsFormatter(),
structlog.processors.TimeStamper(fmt="ISO"),
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
structlog.processors.UnicodeDecoder(),
structlog.processors.JSONRenderer()
],
context_class=dict,
logger_factory=structlog.stdlib.LoggerFactory(),
wrapper_class=structlog.stdlib.BoundLogger,
cache_logger_on_first_use=True,
)
logger = structlog.get_logger()
# Load configuration
config = PipelineConfig(
name="data-processing-pipeline",
input_source="./data/input",
output_destination="./data/output",
transformations=["validate", "enrich"],
batch_size=100
)
# Create components
data_source = FileDataSource("./data/sample_data.jsonl")
data_sink = FileDataSink("./data/output")
pipeline = DataPipeline(config, logger)
service = DataProcessingService(pipeline, data_source, data_sink, logger)
api = DataProcessingAPI(service)
logger.info("Application created", config=config.name)
return api.app
# 8. Sample data generation for testing
print("\n8๏ธโฃ SAMPLE DATA GENERATION:")
async def generate_sample_data():
"""Generate sample data file for testing"""
sample_records = [
{
'id': f'record_{i:04d}',
'timestamp': (datetime.now() - timedelta(minutes=i)).isoformat(),
'type': 'sensor_reading',
'value': 50 + (i % 100),
'sensor_id': f'sensor_{i % 10}'
}
for i in range(1000)
]
# Create data directory
Path('./data').mkdir(exist_ok=True)
# Write sample data
async with aiofiles.open('./data/sample_data.jsonl', 'w') as f:
for record in sample_records:
await f.write(json.dumps(record) + '\n')
print(f"Generated {len(sample_records)} sample records")
# 9. Main application runner
print("\n9๏ธโฃ APPLICATION RUNNER:")
async def run_service():
"""Run the data processing service"""
# Generate sample data
await generate_sample_data()
# Create application
app = create_application()
# Run web server
runner = web.AppRunner(app)
await runner.setup()
site = web.TCPSite(runner, 'localhost', 8080)
await site.start()
print("๐ Data Processing Service started at http://localhost:8080")
print("Available endpoints:")
print(" POST /api/v1/process - Process data")
print(" GET /api/v1/health - Health check")
print(" GET /api/v1/metrics - Service metrics")
# Keep running
try:
await asyncio.sleep(3600) # Run for 1 hour
except KeyboardInterrupt:
print("Shutting down...")
finally:
await runner.cleanup()
# Note: To run the service, uncomment the following line
# asyncio.run(run_service())
print("Data processing service defined (uncomment last line to run)")Python๐งช Testing Strategy for Functional Applications
"""
๐งช COMPREHENSIVE TESTING STRATEGY
Testing approaches for functional applications
"""
import unittest
from unittest.mock import Mock, AsyncMock, patch
import pytest
import asyncio
from hypothesis import given, strategies as st
from typing import List
import tempfile
import json
print("\n๐งช TESTING STRATEGY")
print("=" * 20)
# 1. Unit Tests for Pure Functions
print("1๏ธโฃ UNIT TESTS FOR PURE FUNCTIONS:")
class TestPureFunctions(unittest.TestCase):
"""Test pure business logic functions"""
def test_validate_record_valid_data(self):
"""Test validation with valid record"""
record = DataRecord(
id='test_001',
timestamp=datetime.now(),
source='test',
data={'id': 'test', 'type': 'sensor', 'value': 42.5}
)
result = validate_record(record)
self.assertTrue(result.metadata.get('validated'))
self.assertNotIn('validation_errors', result.metadata or {})
def test_validate_record_missing_fields(self):
"""Test validation with missing required fields"""
record = DataRecord(
id='test_002',
timestamp=datetime.now(),
source='test',
data={'id': 'test'} # Missing 'type' and 'value'
)
result = validate_record(record)
self.assertIn('validation_errors', result.metadata)
errors = result.metadata['validation_errors']
self.assertIn('Missing required field: type', errors)
self.assertIn('Missing required field: value', errors)
def test_enrich_record_adds_computed_fields(self):
"""Test that enrichment adds computed fields"""
record = DataRecord(
id='test_003',
timestamp=datetime.now(),
source='test',
data={'id': 'test', 'type': 'sensor', 'value': 75},
metadata={'validated': True}
)
result = enrich_record(record)
self.assertIn('value_category', result.data)
self.assertEqual(result.data['value_category'], 'medium')
self.assertIn('processing_date', result.data)
def test_aggregate_records_empty_list(self):
"""Test aggregation with empty list"""
result = aggregate_records([])
self.assertEqual(result['total_records'], 0)
def test_aggregate_records_with_data(self):
"""Test aggregation with sample records"""
records = [
DataRecord('1', datetime.now(), 'test', {'value': 100, 'value_category': 'high'}),
DataRecord('2', datetime.now(), 'test', {'value': 50, 'value_category': 'medium'}),
DataRecord('3', datetime.now(), 'test', {'value': 25, 'value_category': 'low'}),
]
result = aggregate_records(records)
self.assertEqual(result['total_records'], 3)
self.assertEqual(result['total_value'], 175)
self.assertAlmostEqual(result['average_value'], 58.33, places=2)
self.assertEqual(result['category_distribution']['high'], 1)
# Run unit tests
print("Running unit tests for pure functions:")
suite = unittest.TestLoader().loadTestsFromTestCase(TestPureFunctions)
runner = unittest.TextTestRunner(verbosity=0)
result = runner.run(suite)
print(f"Unit tests: {result.testsRun - len(result.failures) - len(result.errors)}/{result.testsRun} passed")
# 2. Property-Based Testing
print(f"\n2๏ธโฃ PROPERTY-BASED TESTING:")
def test_validate_record_properties():
"""Property-based tests for record validation"""
@given(st.text(min_size=1), st.floats(allow_nan=False, allow_infinity=False))
def test_valid_record_stays_valid(record_id, value):
"""Valid records should remain valid after validation"""
record = DataRecord(
id=record_id,
timestamp=datetime.now(),
source='test',
data={'id': record_id, 'type': 'test', 'value': value}
)
result = validate_record(record)
# Should be marked as validated
assert result.metadata.get('validated') is True
# Should not have validation errors
assert 'validation_errors' not in (result.metadata or {})
# Run property test
try:
test_valid_record_stays_valid()
print("โ
Property test: Valid records stay valid")
except Exception as e:
print(f"โ Property test failed: {e}")
def test_enrichment_properties():
"""Property-based tests for record enrichment"""
@given(st.floats(min_value=0, max_value=1000, allow_nan=False))
def test_value_category_assignment(value):
"""Test that value categories are assigned correctly"""
record = DataRecord(
id='test',
timestamp=datetime.now(),
source='test',
data={'id': 'test', 'type': 'test', 'value': value},
metadata={'validated': True}
)
result = enrich_record(record)
expected_category = 'high' if value >= 100 else 'medium' if value >= 50 else 'low'
assert result.data['value_category'] == expected_category
# Run property test
try:
test_value_category_assignment()
print("โ
Property test: Value categories assigned correctly")
except Exception as e:
print(f"โ Property test failed: {e}")
test_validate_record_properties()
test_enrichment_properties()
# 3. Integration Tests
print(f"\n3๏ธโฃ INTEGRATION TESTS:")
class TestDataPipelineIntegration(unittest.TestCase):
"""Integration tests for data pipeline"""
def setUp(self):
"""Setup test fixtures"""
self.config = PipelineConfig(
name="test_pipeline",
input_source="test_input",
output_destination="test_output",
transformations=["validate", "enrich"]
)
self.pipeline = DataPipeline"""
๐งช COMPREHENSIVE TESTING STRATEGY
Testing approaches for functional applications
"""
import unittest
from unittest.mock import Mock, AsyncMock, patch
import pytest
import asyncio
from hypothesis import given, strategies as st
from typing import List
import tempfile
import json
print("\n๐งช TESTING STRATEGY")
print("=" * 20)
# 1. Unit Tests for Pure Functions
print("1๏ธโฃ UNIT TESTS FOR PURE FUNCTIONS:")
class TestPureFunctions(unittest.TestCase):
"""Test pure business logic functions"""
def test_validate_record_valid_data(self):
"""Test validation with valid record"""
record = DataRecord(
id='test_001',
timestamp=datetime.now(),
source='test',
data={'id': 'test', 'type': 'sensor', 'value': 42.5}
)
result = validate_record(record)
self.assertTrue(result.metadata.get('validated'))
self.assertNotIn('validation_errors', result.metadata or {})
def test_validate_record_missing_fields(self):
"""Test validation with missing required fields"""
record = DataRecord(
id='test_002',
timestamp=datetime.now(),
source='test',
data={'id': 'test'} # Missing 'type' and 'value'
)
result = validate_record(record)
self.assertIn('validation_errors', result.metadata)
errors = result.metadata['validation_errors']
self.assertIn('Missing required field: type', errors)
self.assertIn('Missing required field: value', errors)
def test_enrich_record_adds_computed_fields(self):
"""Test that enrichment adds computed fields"""
record = DataRecord(
id='test_003',
timestamp=datetime.now(),
source='test',
data={'id': 'test', 'type': 'sensor', 'value': 75},
metadata={'validated': True}
)
result = enrich_record(record)
self.assertIn('value_category', result.data)
self.assertEqual(result.data['value_category'], 'medium')
self.assertIn('processing_date', result.data)
def test_aggregate_records_empty_list(self):
"""Test aggregation with empty list"""
result = aggregate_records([])
self.assertEqual(result['total_records'], 0)
def test_aggregate_records_with_data(self):
"""Test aggregation with sample records"""
records = [
DataRecord('1', datetime.now(), 'test', {'value': 100, 'value_category': 'high'}),
DataRecord('2', datetime.now(), 'test', {'value': 50, 'value_category': 'medium'}),
DataRecord('3', datetime.now(), 'test', {'value': 25, 'value_category': 'low'}),
]
result = aggregate_records(records)
self.assertEqual(result['total_records'], 3)
self.assertEqual(result['total_value'], 175)
self.assertAlmostEqual(result['average_value'], 58.33, places=2)
self.assertEqual(result['category_distribution']['high'], 1)
# Run unit tests
print("Running unit tests for pure functions:")
suite = unittest.TestLoader().loadTestsFromTestCase(TestPureFunctions)
runner = unittest.TextTestRunner(verbosity=0)
result = runner.run(suite)
print(f"Unit tests: {result.testsRun - len(result.failures) - len(result.errors)}/{result.testsRun} passed")
# 2. Property-Based Testing
print(f"\n2๏ธโฃ PROPERTY-BASED TESTING:")
def test_validate_record_properties():
"""Property-based tests for record validation"""
@given(st.text(min_size=1), st.floats(allow_nan=False, allow_infinity=False))
def test_valid_record_stays_valid(record_id, value):
"""Valid records should remain valid after validation"""
record = DataRecord(
id=record_id,
timestamp=datetime.now(),
source='test',
data={'id': record_id, 'type': 'test', 'value': value}
)
result = validate_record(record)
# Should be marked as validated
assert result.metadata.get('validated') is True
# Should not have validation errors
assert 'validation_errors' not in (result.metadata or {})
# Run property test
try:
test_valid_record_stays_valid()
print("โ
Property test: Valid records stay valid")
except Exception as e:
print(f"โ Property test failed: {e}")
def test_enrichment_properties():
"""Property-based tests for record enrichment"""
@given(st.floats(min_value=0, max_value=1000, allow_nan=False))
def test_value_category_assignment(value):
"""Test that value categories are assigned correctly"""
record = DataRecord(
id='test',
timestamp=datetime.now(),
source='test',
data={'id': 'test', 'type': 'test', 'value': value},
metadata={'validated': True}
)
result = enrich_record(record)
expected_category = 'high' if value >= 100 else 'medium' if value >= 50 else 'low'
assert result.data['value_category'] == expected_category
# Run property test
try:
test_value_category_assignment()
print("โ
Property test: Value categories assigned correctly")
except Exception as e:
print(f"โ Property test failed: {e}")
test_validate_record_properties()
test_enrichment_properties()
# 3. Integration Tests
print(f"\n3๏ธโฃ INTEGRATION TESTS:")
class TestDataPipelineIntegration(unittest.TestCase):
"""Integration tests for data pipeline"""
def setUp(self):
"""Setup test fixtures"""
self.config = PipelineConfig(
name="test-pipeline",
input_source="test",
output_destination="test",
transformations=["validate", "enrich"],
batch_size=10
)
self.pipeline = DataPipeline(self.config)
async def test_end_to_end_processing(self):
"""Test complete pipeline processing"""
test_records = [
DataRecord(
id=f"test_{i}",
timestamp=datetime.now(),
source="test",
data={"id": f"test_{i}", "type": "measurement", "value": i * 10.0}
)
for i in range(5)
]
result = await self.pipeline.process_batch(test_records)
self.assertEqual(result.success_count, 5)
self.assertEqual(result.error_count, 0)
self.assertGreater(result.success_rate, 0.9)
async def test_error_handling_in_pipeline(self):
"""Test pipeline handles errors gracefully"""
test_records = [
DataRecord(
id="valid",
timestamp=datetime.now(),
source="test",
data={"id": "valid", "type": "measurement", "value": 100.0}
),
DataRecord(
id="invalid",
timestamp=datetime.now(),
source="test",
data={"id": "invalid"} # Missing required fields
)
]
result = await self.pipeline.process_batch(test_records)
self.assertEqual(result.success_count, 1)
self.assertEqual(result.error_count, 1)
# Run async integration tests
async def run_integration_tests():
"""Run integration tests"""
print("Running integration tests:")
test_case = TestDataPipelineIntegration()
test_case.setUp()
await test_case.test_end_to_end_processing()
print("โ
End-to-end processing test passed")
await test_case.test_error_handling_in_pipeline()
print("โ
Error handling test passed")
asyncio.run(run_integration_tests())
# 4. API Tests
print(f"\n4๏ธโฃ API INTEGRATION TESTS:")
class TestAPI(unittest.TestCase):
"""Test API endpoints"""
async def test_health_endpoint(self):
"""Test health check endpoint"""
app = create_application()
async with aiohttp.test.TestClient(app) as client:
resp = await client.get('/api/v1/health')
self.assertEqual(resp.status, 200)
data = await resp.json()
self.assertEqual(data['status'], 'healthy')
async def test_process_endpoint(self):
"""Test data processing endpoint"""
app = create_application()
async with aiohttp.test.TestClient(app) as client:
resp = await client.post('/api/v1/process')
self.assertEqual(resp.status, 200)
# 5. Performance Tests
print(f"\n5๏ธโฃ PERFORMANCE TESTS:")
class TestPerformance(unittest.TestCase):
"""Performance tests for functional components"""
def test_pipeline_performance(self):
"""Test pipeline can handle expected load"""
import time
# Create large batch
large_batch = [
DataRecord(
id=f"perf_test_{i}",
timestamp=datetime.now(),
source="performance_test",
data={"id": f"perf_test_{i}", "type": "measurement", "value": float(i)}
)
for i in range(1000)
]
start_time = time.time()
# Run pipeline synchronously for performance test
pipeline = DataPipeline(PipelineConfig("perf-test", "test", "test", []))
# Simulate processing (since we can't run async easily here)
processed_count = 0
for record in large_batch:
validated = validate_record(record)
enriched = enrich_record(validated)
processed_count += 1
end_time = time.time()
processing_time = end_time - start_time
throughput = processed_count / processing_time
print(f"Processed {processed_count} records in {processing_time:.3f} seconds")
print(f"Throughput: {throughput:.1f} records/second")
# Assert minimum performance threshold
self.assertGreater(throughput, 100, "Pipeline should process at least 100 records/second")
# Run performance tests
print("Running performance tests:")
suite = unittest.TestLoader().loadTestsFromTestCase(TestPerformance)
runner = unittest.TextTestRunner(verbosity=0)
result = runner.run(suite)
print(f"Performance tests: {result.testsRun - len(result.failures) - len(result.errors)}/{result.testsRun} passed")
print("\nโ
Testing strategy complete!")Python๐ฆ Project 2: Functional Configuration Management System
"""
๐ฆ PROJECT 2: FUNCTIONAL CONFIGURATION MANAGEMENT SYSTEM
A configuration system using functional programming principles
"""
from typing import Dict, Any, Union, Optional, Callable
from dataclasses import dataclass, field
from pathlib import Path
import yaml
import json
from functools import reduce, partial
import os
print("\n๐ฆ FUNCTIONAL CONFIGURATION SYSTEM")
print("=" * 40)
# 1. Immutable Configuration Model
print("1๏ธโฃ CONFIGURATION MODEL:")
@dataclass(frozen=True)
class ConfigValue:
"""Immutable configuration value with metadata"""
value: Any
source: str
priority: int = 0
description: Optional[str] = None
def with_value(self, new_value: Any) -> 'ConfigValue':
"""Create new ConfigValue with different value"""
return ConfigValue(new_value, self.source, self.priority, self.description)
def with_source(self, new_source: str) -> 'ConfigValue':
"""Create new ConfigValue with different source"""
return ConfigValue(self.value, new_source, self.priority, self.description)
@dataclass(frozen=True)
class Configuration:
"""Immutable configuration container"""
values: Dict[str, ConfigValue] = field(default_factory=dict)
def get(self, key: str, default: Any = None) -> Any:
"""Get configuration value"""
return self.values.get(key, ConfigValue(default, "default")).value
def set(self, key: str, config_value: ConfigValue) -> 'Configuration':
"""Create new configuration with updated value"""
new_values = {**self.values, key: config_value}
return Configuration(new_values)
def merge(self, other: 'Configuration') -> 'Configuration':
"""Merge two configurations (other takes precedence)"""
merged_values = {}
# Start with current values
for key, value in self.values.items():
merged_values[key] = value
# Override with other values (higher priority wins)
for key, other_value in other.values.items():
if key in merged_values:
current_value = merged_values[key]
if other_value.priority >= current_value.priority:
merged_values[key] = other_value
else:
merged_values[key] = other_value
return Configuration(merged_values)
def filter_by_prefix(self, prefix: str) -> 'Configuration':
"""Get configuration subset by key prefix"""
filtered_values = {
key: value for key, value in self.values.items()
if key.startswith(prefix)
}
return Configuration(filtered_values)
def to_dict(self) -> Dict[str, Any]:
"""Convert to plain dictionary"""
return {key: value.value for key, value in self.values.items()}
# 2. Configuration Sources (Pure Functions)
print("\n2๏ธโฃ CONFIGURATION SOURCES:")
def load_from_dict(data: Dict[str, Any], source: str, priority: int = 0) -> Configuration:
"""Load configuration from dictionary"""
values = {}
def _process_nested(obj: Any, prefix: str = "") -> None:
if isinstance(obj, dict):
for key, value in obj.items():
full_key = f"{prefix}.{key}" if prefix else key
_process_nested(value, full_key)
else:
values[prefix] = ConfigValue(obj, source, priority)
_process_nested(data)
return Configuration(values)
def load_from_file(file_path: str, priority: int = 0) -> Configuration:
"""Load configuration from file"""
path = Path(file_path)
if not path.exists():
return Configuration()
try:
with open(path, 'r') as f:
if path.suffix.lower() in ['.yaml', '.yml']:
data = yaml.safe_load(f)
elif path.suffix.lower() == '.json':
data = json.load(f)
else:
raise ValueError(f"Unsupported file format: {path.suffix}")
return load_from_dict(data, f"file:{file_path}", priority)
except Exception as e:
print(f"Warning: Could not load config file {file_path}: {e}")
return Configuration()
def load_from_env(prefix: str = "", priority: int = 100) -> Configuration:
"""Load configuration from environment variables"""
values = {}
for key, value in os.environ.items():
if not prefix or key.startswith(prefix):
# Remove prefix and convert to lowercase
config_key = key[len(prefix):].lower() if prefix else key.lower()
# Replace underscores with dots for nested keys
config_key = config_key.replace('_', '.')
# Try to parse as JSON for complex values
try:
parsed_value = json.loads(value)
except (json.JSONDecodeError, TypeError):
parsed_value = value
values[config_key] = ConfigValue(parsed_value, f"env:{key}", priority)
return Configuration(values)
def load_from_cli_args(args: Dict[str, Any], priority: int = 200) -> Configuration:
"""Load configuration from CLI arguments"""
return load_from_dict(args, "cli", priority)
# 3. Configuration Transformations (Pure Functions)
print("\n3๏ธโฃ CONFIGURATION TRANSFORMATIONS:")
def validate_config(config: Configuration, validators: Dict[str, Callable]) -> Configuration:
"""Validate configuration values"""
validated_values = {}
for key, config_value in config.values.items():
if key in validators:
validator = validators[key]
try:
# Validator should return validated value or raise exception
validated_value = validator(config_value.value)
validated_values[key] = config_value.with_value(validated_value)
except Exception as e:
print(f"Validation error for {key}: {e}")
# Keep original value on validation error
validated_values[key] = config_value
else:
validated_values[key] = config_value
return Configuration(validated_values)
def transform_config(config: Configuration, transformers: Dict[str, Callable]) -> Configuration:
"""Transform configuration values"""
transformed_values = {}
for key, config_value in config.values.items():
if key in transformers:
transformer = transformers[key]
try:
transformed_value = transformer(config_value.value)
transformed_values[key] = config_value.with_value(transformed_value)
except Exception as e:
print(f"Transformation error for {key}: {e}")
transformed_values[key] = config_value
else:
transformed_values[key] = config_value
return Configuration(transformed_values)
def expand_config(config: Configuration) -> Configuration:
"""Expand configuration with variable substitution"""
expanded_values = {}
# First pass: collect all values
all_values = {key: str(value.value) for key, value in config.values.items()}
# Second pass: expand variables
for key, config_value in config.values.items():
if isinstance(config_value.value, str):
expanded_value = config_value.value
# Simple variable expansion: ${key} -> value
import re
for match in re.finditer(r'\$\{([^}]+)\}', expanded_value):
var_name = match.group(1)
if var_name in all_values:
expanded_value = expanded_value.replace(
match.group(0),
all_values[var_name]
)
expanded_values[key] = config_value.with_value(expanded_value)
else:
expanded_values[key] = config_value
return Configuration(expanded_values)
# 4. Configuration Builder (Functional)
print("\n4๏ธโฃ CONFIGURATION BUILDER:")
class ConfigurationBuilder:
"""Builder for constructing configuration functionally"""
def __init__(self):
self._sources = []
self._validators = {}
self._transformers = {}
def add_source(self, source_func: Callable[[], Configuration]) -> 'ConfigurationBuilder':
"""Add configuration source"""
self._sources.append(source_func)
return self
def add_file(self, file_path: str, priority: int = 0) -> 'ConfigurationBuilder':
"""Add file source"""
return self.add_source(lambda: load_from_file(file_path, priority))
def add_env(self, prefix: str = "", priority: int = 100) -> 'ConfigurationBuilder':
"""Add environment variable source"""
return self.add_source(lambda: load_from_env(prefix, priority))
def add_dict(self, data: Dict[str, Any], source: str, priority: int = 0) -> 'ConfigurationBuilder':
"""Add dictionary source"""
return self.add_source(lambda: load_from_dict(data, source, priority))
def add_validator(self, key: str, validator: Callable) -> 'ConfigurationBuilder':
"""Add value validator"""
self._validators[key] = validator
return self
def add_transformer(self, key: str, transformer: Callable) -> 'ConfigurationBuilder':
"""Add value transformer"""
self._transformers[key] = transformer
return self
def build(self) -> Configuration:
"""Build final configuration"""
# Load all sources
configs = [source() for source in self._sources]
# Merge all configurations
final_config = reduce(lambda acc, cfg: acc.merge(cfg), configs, Configuration())
# Apply transformations
if self._transformers:
final_config = transform_config(final_config, self._transformers)
# Apply validation
if self._validators:
final_config = validate_config(final_config, self._validators)
# Expand variables
final_config = expand_config(final_config)
return final_config
# 5. Example Usage and Testing
print("\n5๏ธโฃ EXAMPLE USAGE:")
def create_app_config() -> Configuration:
"""Create application configuration"""
# Define validators
def validate_port(port):
port_int = int(port)
if not (1 <= port_int <= 65535):
raise ValueError(f"Port must be between 1 and 65535, got {port_int}")
return port_int
def validate_log_level(level):
valid_levels = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
level_upper = str(level).upper()
if level_upper not in valid_levels:
raise ValueError(f"Log level must be one of {valid_levels}, got {level}")
return level_upper
# Define transformers
def to_boolean(value):
if isinstance(value, bool):
return value
if isinstance(value, str):
return value.lower() in ('true', '1', 'yes', 'on')
return bool(value)
# Create configuration
config = (ConfigurationBuilder()
.add_dict({
'app': {
'name': 'my-functional-app',
'version': '1.0.0',
'debug': False
},
'server': {
'host': '0.0.0.0',
'port': 8080
},
'logging': {
'level': 'INFO',
'format': '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
}
}, "defaults", priority=0)
.add_file('config.yaml', priority=50) # Would load if file exists
.add_env('MYAPP_', priority=100)
.add_validator('server.port', validate_port)
.add_validator('logging.level', validate_log_level)
.add_transformer('app.debug', to_boolean)
.build())
return config
# Test the configuration system
print("Testing configuration system:")
app_config = create_app_config()
print("Configuration values:")
print(f" App name: {app_config.get('app.name')}")
print(f" Server port: {app_config.get('server.port')} (type: {type(app_config.get('server.port'))})")
print(f" Debug mode: {app_config.get('app.debug')} (type: {type(app_config.get('app.debug'))})")
print(f" Log level: {app_config.get('logging.level')}")
# Test configuration merging
override_config = load_from_dict({
'server': {'port': 9000},
'app': {'debug': True}
}, "override", priority=150)
merged_config = app_config.merge(override_config)
print(f"\nAfter override:")
print(f" Server port: {merged_config.get('server.port')}")
print(f" Debug mode: {merged_config.get('app.debug')}")
# Test configuration filtering
server_config = merged_config.filter_by_prefix('server')
print(f"\nServer configuration: {server_config.to_dict()}")
print("\n๐ฆ Configuration system complete!")Python๐ Deployment and Production Best Practices
"""
๐ DEPLOYMENT AND PRODUCTION BEST PRACTICES
Production-ready patterns for functional applications
"""
import logging
import structlog
from typing import Dict, Any
import os
import signal
import asyncio
from datetime import datetime
import json
print("\n๐ PRODUCTION BEST PRACTICES")
print("=" * 30)
# 1. Structured Logging Setup
print("1๏ธโฃ STRUCTURED LOGGING:")
def setup_logging(config: Configuration) -> structlog.BoundLogger:
"""Setup structured logging for production"""
log_level = config.get('logging.level', 'INFO')
# Configure structlog
structlog.configure(
processors=[
structlog.stdlib.filter_by_level,
structlog.stdlib.add_logger_name,
structlog.stdlib.add_log_level,
structlog.stdlib.PositionalArgumentsFormatter(),
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
structlog.processors.UnicodeDecoder(),
structlog.processors.JSONRenderer()
],
context_class=dict,
logger_factory=structlog.stdlib.LoggerFactory(),
wrapper_class=structlog.stdlib.BoundLogger,
cache_logger_on_first_use=True,
)
# Setup standard library logging
logging.basicConfig(
format="%(message)s",
level=getattr(logging, log_level.upper()),
)
logger = structlog.get_logger()
logger.info("Logging configured", log_level=log_level)
return logger
# 2. Health Check System
print("\n2๏ธโฃ HEALTH CHECK SYSTEM:")
@dataclass(frozen=True)
class HealthStatus:
"""Health check status"""
name: str
healthy: bool
message: str
details: Dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> Dict[str, Any]:
return {
'name': self.name,
'healthy': self.healthy,
'message': self.message,
'details': self.details
}
class HealthChecker:
"""Health check coordinator"""
def __init__(self):
self._checks = {}
def add_check(self, name: str, check_func: Callable[[], HealthStatus]) -> 'HealthChecker':
"""Add health check"""
self._checks[name] = check_func
return self
async def check_all(self) -> Dict[str, Any]:
"""Run all health checks"""
results = {}
overall_healthy = True
for name, check_func in self._checks.items():
try:
if asyncio.iscoroutinefunction(check_func):
status = await check_func()
else:
status = check_func()
results[name] = status.to_dict()
if not status.healthy:
overall_healthy = False
except Exception as e:
results[name] = HealthStatus(
name=name,
healthy=False,
message=f"Health check failed: {str(e)}"
).to_dict()
overall_healthy = False
return {
'overall_healthy': overall_healthy,
'timestamp': datetime.now().isoformat(),
'checks': results
}
# Example health checks
def database_health_check() -> HealthStatus:
"""Check database connectivity"""
try:
# Simulate database check
# In real implementation, would check actual database
return HealthStatus(
name="database",
healthy=True,
message="Database connection OK",
details={'connection_pool_size': 10, 'active_connections': 3}
)
except Exception as e:
return HealthStatus(
name="database",
healthy=False,
message=f"Database connection failed: {e}"
)
def external_api_health_check() -> HealthStatus:
"""Check external API connectivity"""
return HealthStatus(
name="external_api",
healthy=True,
message="External API accessible",
details={'response_time_ms': 150}
)
# 3. Metrics Collection
print("\n3๏ธโฃ METRICS COLLECTION:")
class MetricsCollector:
"""Simple metrics collection for functional apps"""
def __init__(self):
self._counters = {}
self._gauges = {}
self._histograms = {}
def increment_counter(self, name: str, value: int = 1, tags: Dict[str, str] = None):
"""Increment counter metric"""
key = self._make_key(name, tags)
self._counters[key] = self._counters.get(key, 0) + value
def set_gauge(self, name: str, value: float, tags: Dict[str, str] = None):
"""Set gauge metric"""
key = self._make_key(name, tags)
self._gauges[key] = value
def record_histogram(self, name: str, value: float, tags: Dict[str, str] = None):
"""Record histogram value"""
key = self._make_key(name, tags)
if key not in self._histograms:
self._histograms[key] = []
self._histograms[key].append(value)
def _make_key(self, name: str, tags: Dict[str, str] = None) -> str:
"""Create metric key with tags"""
if not tags:
return name
tag_string = ','.join(f"{k}={v}" for k, v in sorted(tags.items()))
return f"{name}[{tag_string}]"
def get_metrics(self) -> Dict[str, Any]:
"""Get all metrics"""
metrics = {
'counters': self._counters.copy(),
'gauges': self._gauges.copy(),
'histograms': {}
}
# Calculate histogram statistics
for key, values in self._histograms.items():
if values:
metrics['histograms'][key] = {
'count': len(values),
'sum': sum(values),
'min': min(values),
'max': max(values),
'avg': sum(values) / len(values)
}
return metrics
def reset(self):
"""Reset all metrics"""
self._counters.clear()
self._gauges.clear()
self._histograms.clear()
# 4. Graceful Shutdown
print("\n4๏ธโฃ GRACEFUL SHUTDOWN:")
class GracefulShutdown:
"""Handle graceful application shutdown"""
def __init__(self, logger):
self.logger = logger
self._shutdown_handlers = []
self._shutdown_event = asyncio.Event()
def add_shutdown_handler(self, handler: Callable):
"""Add shutdown handler"""
self._shutdown_handlers.append(handler)
def setup_signal_handlers(self):
"""Setup signal handlers for graceful shutdown"""
for sig in (signal.SIGTERM, signal.SIGINT):
signal.signal(sig, self._signal_handler)
def _signal_handler(self, signum, frame):
"""Handle shutdown signals"""
self.logger.info("Shutdown signal received", signal=signum)
asyncio.create_task(self._shutdown())
async def _shutdown(self):
"""Execute shutdown sequence"""
self.logger.info("Starting graceful shutdown")
for handler in reversed(self._shutdown_handlers): # LIFO order
try:
if asyncio.iscoroutinefunction(handler):
await handler()
else:
handler()
self.logger.info("Shutdown handler completed", handler=handler.__name__)
except Exception as e:
self.logger.error("Shutdown handler failed",
handler=handler.__name__, error=str(e))
self.logger.info("Graceful shutdown completed")
self._shutdown_event.set()
async def wait_for_shutdown(self):
"""Wait for shutdown signal"""
await self._shutdown_event.wait()
# 5. Production Application Factory
print("\n5๏ธโฃ PRODUCTION APPLICATION:")
async def create_production_application():
"""Create production-ready application"""
# Load configuration
config = create_app_config()
# Setup logging
logger = setup_logging(config)
logger.info("Application starting",
app_name=config.get('app.name'),
version=config.get('app.version'))
# Setup metrics
metrics = MetricsCollector()
# Setup health checks
health_checker = (HealthChecker()
.add_check('database', database_health_check)
.add_check('external_api', external_api_health_check))
# Setup graceful shutdown
shutdown = GracefulShutdown(logger)
# Create main application components
pipeline_config = PipelineConfig(
name=config.get('app.name'),
input_source=config.get('pipeline.input_source', 'default'),
output_destination=config.get('pipeline.output_destination', 'default'),
transformations=config.get('pipeline.transformations', ['validate', 'enrich'])
)
pipeline = DataPipeline(pipeline_config, logger)
data_source = FileDataSource('./data/input')
data_sink = FileDataSink('./data/output')
service = DataProcessingService(pipeline, data_source, data_sink, logger)
# Setup web API
app = web.Application()
# Add middleware for metrics and logging
async def metrics_middleware(request, handler):
start_time = asyncio.get_event_loop().time()
try:
response = await handler(request)
metrics.increment_counter('http_requests_total',
tags={'method': request.method, 'status': str(response.status)})
return response
except Exception as e:
metrics.increment_counter('http_requests_total',
tags={'method': request.method, 'status': 'error'})
raise
finally:
duration = asyncio.get_event_loop().time() - start_time
metrics.record_histogram('http_request_duration_seconds', duration)
app.middlewares.append(metrics_middleware)
# Add routes
async def health_endpoint(request):
health_status = await health_checker.check_all()
status = 200 if health_status['overall_healthy'] else 503
return web.json_response(health_status, status=status)
async def metrics_endpoint(request):
return web.json_response(metrics.get_metrics())
async def process_endpoint(request):
try:
result = await service.process_stream()
metrics.increment_counter('data_processing_jobs_total', tags={'status': 'success'})
return web.json_response({
'status': 'completed',
'processed_records': result.get('processed_count', 0),
'processing_time_seconds': result.get('processing_time', 0)
})
except Exception as e:
logger.error("Processing failed", error=str(e))
metrics.increment_counter('data_processing_jobs_total', tags={'status': 'error'})
return web.json_response({
'status': 'error',
'error': str(e)
}, status=500)
app.router.add_get('/health', health_endpoint)
app.router.add_get('/metrics', metrics_endpoint)
app.router.add_post('/api/v1/process', process_endpoint)
# Setup shutdown handlers
async def shutdown_web_server():
logger.info("Shutting down web server")
async def shutdown_service():
logger.info("Shutting down processing service")
shutdown.add_shutdown_handler(shutdown_web_server)
shutdown.add_shutdown_handler(shutdown_service)
shutdown.setup_signal_handlers()
# Start server
runner = web.AppRunner(app)
await runner.setup()
host = config.get('server.host', '0.0.0.0')
port = config.get('server.port', 8080)
site = web.TCPSite(runner, host, port)
await site.start()
logger.info("Application started", host=host, port=port)
# Wait for shutdown
await shutdown.wait_for_shutdown()
# Cleanup
await runner.cleanup()
logger.info("Application stopped")
# Note: To run the production application
# asyncio.run(create_production_application())
print("\n๐ฏ PRODUCTION BEST PRACTICES SUMMARY:")
print("โ
Structured logging with context")
print("โ
Comprehensive health checks")
print("โ
Metrics collection and monitoring")
print("โ
Graceful shutdown handling")
print("โ
Configuration management")
print("โ
Error handling and recovery")
print("โ
Performance monitoring")
print("โ
Production-ready architecture")
print("\n๐ FUNCTIONAL PROGRAMMING JOURNEY COMPLETE!")
print("=" * 50)
print("You've learned:")
print("โ
Functional programming fundamentals")
print("โ
Pure functions and immutability")
print("โ
Higher-order functions and composition")
print("โ
Monads and advanced patterns")
print("โ
Concurrent and parallel programming")
print("โ
Testing strategies")
print("โ
Performance optimization")
print("โ
Real-world application development")
print("\n๐ Ready to build functional applications!")PythonThis completes the comprehensive Python Functional Programming: From Beginner to Expert guide. The final section demonstrates how to combine all the functional programming concepts into production-ready applications with proper architecture, testing, monitoring, and deployment practices.
Key Takeaways from This Complete Guide:
- Functional Programming Foundations – Pure functions, immutability, and composition create maintainable code
- Advanced Patterns – Monads, functors, and other patterns provide powerful abstractions
- Practical Applications – Real-world projects show how FP principles apply to production systems
- Testing Excellence – Functional code is inherently more testable and reliable
- Performance Optimization – FP doesn’t sacrifice performance when implemented thoughtfully
- Production Readiness – Functional applications can be robust, scalable, and maintainable
The guide provides a complete journey from basic concepts to advanced patterns, with practical examples and real-world applications that demonstrate the power and elegance of functional programming in Python.
Discover more from Altgr Blog
Subscribe to get the latest posts sent to your email.
