Python Memory Model

      Ankush More

      A Comprehensive Guide to Mutable and Immutable Data Types

      Table of Contents

      1. Introduction
      2. Python Object Model Fundamentals
      3. Immutable Data Types
      4. Mutable Data Types
      5. Heap vs Stack Memory Model
      6. Python Internals and CPython Implementation
      7. Practical Examples and Best Practices
      8. Advanced Topics and Edge Cases
      9. Performance Implications and Optimization
      10. Conclusion

      Introduction

      Understanding Python’s memory model is crucial for writing efficient and bug-free code. This book provides a pragmatic explanation of how Python handles mutable and immutable data types, their relationship with heap and stack memory, and how Python works internally under the hood.

      Why This Matters

      Python’s approach to memory management is unique among programming languages. Unlike languages like C++ where you manually manage memory, Python handles most memory operations automatically. However, understanding the underlying mechanisms helps you:

      • Write more efficient code
      • Avoid common pitfalls and bugs
      • Debug memory-related issues
      • Optimize performance-critical applications
      • Make informed design decisions

      What You’ll Learn

      By the end of this book, you’ll have a deep understanding of:

      • How Python objects are stored in memory
      • The difference between mutable and immutable types
      • When objects are created, modified, or destroyed
      • How Python’s reference counting and garbage collection work
      • Memory optimization techniques and best practices

      Prerequisites

      This book assumes basic Python knowledge but explains concepts from first principles. Code examples are provided throughout, and diagrams help visualize complex concepts.

      Python Object Model Fundamentals

      Everything is an Object

      In Python, everything is an object. This fundamental principle affects how memory is managed and how data types behave.

      # Even simple values are objects
x = 42
print(type(x))          # <class 'int'>
print(id(x))            # Memory address
print(x.__class__)      # <class 'int'>
      Python

      Object Identity, Type, and Value

      Every Python object has three characteristics:

      1. Identity: Unique identifier (memory address)
      2. Type: Determines operations and possible values
      3. Value: The actual data stored
      graph TD
    A[Python Object] --> B["Identityid()"]
    A --> C["Typetype()"]
    A --> D[ValueContent]

    B --> E[Memory AddressNever Changes]
    C --> F[Object ClassNever Changes]
    D --> G[Actual DataMay Change]

    style A fill:#e1f5fe
    style B fill:#fff3e0
    style C fill:#f3e5f5
    style D fill:#e8f5e8

      References vs Objects

      Understanding the distinction between references and objects is crucial:

      # Creating an object and a reference
a = [1, 2, 3]  # Creates a list object, 'a' is a reference to it
b = a          # 'b' is another reference to the same object
c = [1, 2, 3]  # Creates a new, separate list object

print(id(a))   # Same as id(b)
print(id(b))   # Same as id(a)
print(id(c))   # Different from id(a) and id(b)
      Python
      graph LR
    A[Reference 'a'] --> D["List Object[1, 2, 3]<br/>id: 0x1234"]
    B[Reference 'b'] --> D
    C[Reference 'c'] --> E["List Object[1, 2, 3]<br/>id: 0x5678"]

    style D fill:#ffebee
    style E fill:#e8f5e8

      Reference Counting Basics

      Python uses reference counting to track how many references point to an object:

      import sys

# Create an object
data = [1, 2, 3]
print(sys.getrefcount(data))  # At least 2 (data + function parameter)

# Create another reference
backup = data
print(sys.getrefcount(data))  # Increased by 1

# Remove a reference
del backup
print(sys.getrefcount(data))  # Decreased by 1
      Python
      graph TD
    A["List Object[1, 2, 3]"] --> B[Reference Count: 2]

    C[Reference: data] --> A
    D[Reference: backup] --> A

    E[When count reaches 0] --> F[Object is deallocated]

    style A fill:#e3f2fd
    style B fill:#fff3e0
    style F fill:#ffebee

      Mutability Defined

      The key distinction in Python data types:

      • Immutable: Object’s value cannot be changed after creation
      • Mutable: Object’s value can be changed after creation
      # Immutable example
x = "hello"
y = x.upper()  # Creates new string object
print(id(x) == id(y))  # False - different objects

# Mutable example
lst = [1, 2, 3]
original_id = id(lst)
lst.append(4)  # Modifies existing object
print(id(lst) == original_id)  # True - same object
      Python
      graph TD
    subgraph "Immutable Operation"
        A["String: 'hello'<br/>id: 0x1000"] 
        B["x reference"] --> A
        C["String: 'HELLO'<br/>id: 0x2000"]
        D["y reference"] --> C
        E["x.upper() creates<br/>new object"] --> C
    end

    subgraph "Mutable Operation"
        F["List: [1,2,3]<br/>id: 0x3000"]
        G["lst reference"] --> F
        H["List: [1,2,3,4]<br/>id: 0x3000"]
        I["lst reference"] --> H
        J["append() modifies<br/>same object"] --> H
        F -.-> H
    end

    style A fill:#e8f5e8
    style C fill:#e8f5e8
    style F fill:#ffebee
    style H fill:#ffebee

      Assignment vs Mutation

      Understanding the difference between assignment and mutation is crucial:

      # Assignment creates new binding
a = [1, 2, 3]
original_id = id(a)
a = [4, 5, 6]  # Assignment - new object
print(id(a) == original_id)  # False

# Mutation modifies existing object
b = [1, 2, 3]
original_id = id(b)
b[0] = 99  # Mutation - same object
print(id(b) == original_id)  # True
      Python
      sequenceDiagram
    participant Ref as Reference
    participant Obj1 as Object [1,2,3]
    participant Obj2 as Object [4,5,6]

    Note over Ref,Obj1: Assignment Example
    Ref->>Obj1: Points to initially
    Note over Ref: a = [4,5,6]
    Ref->>Obj1: Stops pointing
    Ref->>Obj2: Now points to new object

    Note over Obj1: Original object may be garbage collected

      Immutable Data Types

      Immutable objects cannot be changed after creation. Any operation that appears to modify an immutable object actually creates a new object.

      Integers (int)

      Python integers have special behavior due to optimization:

      # Small integers are cached (-5 to 256)
a = 100
b = 100
print(a is b)  # True - same object

# Large integers are not cached
x = 1000
y = 1000
print(x is y)  # May be False - different objects
      Python
      graph TD
    subgraph "Small Integer Cache (-5 to 256)"
        A[Integer Pool] --> B["-5"]
        A --> C["..."]
        A --> D["100"]
        A --> E["..."]
        A --> F["256"]

        G[Variable 'a'] --> D
        H[Variable 'b'] --> D
    end

    subgraph "Large Integers"
        I[Variable 'x'] --> J["1000id: 0x1234"]
        K[Variable 'y'] --> L["1000id: 0x5678"]
    end

    style A fill:#e3f2fd
    style D fill:#fff3e0

      Integer Arithmetic Memory Behavior

      # Immutable arithmetic creates new objects
x = 42
original_id = id(x)
x += 1  # Same as: x = x + 1
print(id(x) == original_id)  # False

# Demonstrating immutability
def modify_int(num):
    print(f"Before: {id(num)}")
    num += 10
    print(f"After: {id(num)}")
    return num

original = 5
result = modify_int(original)
print(f"Original: {original}, id: {id(original)}")
print(f"Result: {result}, id: {id(result)}")
      Python

      Strings (str)

      Strings are immutable sequences of Unicode characters:

      # String operations create new objects
s = "hello"
original_id = id(s)
s = s.upper()  # Creates new string
print(id(s) == original_id)  # False

# String concatenation
a = "Hello"
b = " World"
c = a + b  # Creates new string object
print(f"a id: {id(a)}")
print(f"b id: {id(b)}")
print(f"c id: {id(c)}")  # Different from both a and b
      Python

      String Interning

      Python interns some strings for optimization:

      # String literals are often interned
x = "hello"
y = "hello"
print(x is y)  # True

# Strings with special characters may not be interned
a = "hello world!"
b = "hello world!"
print(a is b)  # May be False

# Force interning
import sys
c = sys.intern("hello world!")
d = sys.intern("hello world!")
print(c is d)  # True
      Python
      graph TD
    subgraph "String Interning"
        A[String Intern Pool] --> B["'hello'"]
        A --> C["'python'"]
        A --> D["'__main__'"]

        E[Variable x] --> B
        F[Variable y] --> B
    end

    subgraph "Non-Interned Strings"
        G[Variable a] --> H["'hello world!'<br/>id: 0x1000"]
        I[Variable b] --> J["'hello world!'<br/>id: 0x2000"]
    end

    style A fill:#e8f5e8
    style B fill:#fff3e0

      Tuples (tuple)

      Tuples are immutable sequences that can contain any objects:

      # Tuple creation
t1 = (1, 2, 3)
t2 = (1, 2, 3)
print(t1 is t2)  # May be False - different objects

# Tuple with mutable elements
t = ([1, 2], [3, 4])
print(id(t))  # Tuple id

# Can't change tuple structure
# t[0] = [5, 6]  # TypeError

# But can modify mutable elements
t[0].append(3)
print(t)  # ([1, 2, 3], [3, 4])
print(id(t))  # Same tuple id
      Python
      graph TD
    subgraph "Tuple Structure"
        A["Tuple Objectid: 0x1000"] --> B[Index 0]
        A --> C[Index 1]

        B --> D["List [1,2,3]id: 0x2000"]
        C --> E["List [3,4]id: 0x3000"]
    end

    F[Tuple reference 't'] --> A

    G["t[0].append(3)"] --> D
    H[Modifies list contentTuple structure unchanged]

    style A fill:#e8f5e8
    style D fill:#ffebee
    style E fill:#ffebee

      Tuple Packing and Unpacking

      # Tuple packing
coordinates = 10, 20  # Creates tuple (10, 20)
print(type(coordinates))  # <class 'tuple'>

# Tuple unpacking
x, y = coordinates
print(f"x: {x}, y: {y}")

# Memory efficiency of tuples
import sys
tuple_data = (1, 2, 3, 4, 5)
list_data = [1, 2, 3, 4, 5]
print(f"Tuple size: {sys.getsizeof(tuple_data)}")
print(f"List size: {sys.getsizeof(list_data)}")
      Python

      Floating Point Numbers (float)

      Floats are immutable but have unique characteristics:

      # Float precision and identity
a = 0.1 + 0.2
b = 0.3
print(a == b)  # False due to floating point precision
print(a is b)  # False - different objects

# Float operations create new objects
x = 3.14
original_id = id(x)
x *= 2
print(id(x) == original_id)  # False
      Python

      Frozensets (frozenset)

      Immutable version of sets:

      # Frozenset creation
fs1 = frozenset([1, 2, 3, 4])
fs2 = frozenset([1, 2, 3, 4])
print(fs1 == fs2)  # True - same content
print(fs1 is fs2)  # May be True due to optimization

# Frozensets can be dictionary keys or set elements
data = {fs1: "first", frozenset([5, 6]): "second"}
nested_set = {fs1, frozenset([7, 8])}

# Operations create new frozensets
fs3 = fs1 | frozenset([5, 6])  # Union
print(id(fs1) == id(fs3))  # False
      Python
      graph TD
    subgraph "Immutable Operations"
        A[Original Object] --> B[Operation]
        B --> C[New Object]

        D[Original References] --> A
        E[New References] --> C
    end

    F["Examples:"] --> G["String: 'hello' → 'HELLO'"]
    F --> H["Int: 5 → 6"]
    F --> I["Tuple: (1,2) → (1,2,3)"]
    F --> J["Frozenset: {1,2} → {1,2,3}"]

    style A fill:#e8f5e8
    style C fill:#e8f5e8

      Mutable Data Types

      Mutable objects can be changed after creation. Operations on mutable objects often modify the object in-place rather than creating new objects.

      Lists (list)

      Lists are mutable sequences that can hold any objects:

      # List modification in-place
lst = [1, 2, 3]
original_id = id(lst)

# Various modification operations
lst.append(4)        # Adds element
lst[0] = 99          # Item assignment
lst.extend([5, 6])   # Extends list
lst.insert(1, 100)   # Inserts element

print(id(lst) == original_id)  # True - same object
print(lst)  # [99, 100, 2, 3, 4, 5, 6]
      Python

      List Memory Allocation

      Lists allocate extra capacity for efficient appending:

      import sys

# Observe list capacity growth
lst = []
for i in range(10):
    before_size = sys.getsizeof(lst)
    lst.append(i)
    after_size = sys.getsizeof(lst)
    print(f"Length: {len(lst)}, Size: {after_size}")
      Python
      graph TD
    subgraph "List Growth Strategy"
        A[Empty List<br/>Capacity: 0] --> B[First Append<br/>Capacity: 4]
        B --> C[Length ≤ 4<br/>No reallocation]
        C --> D[Length > 4<br/>Reallocate with<br/>~1.5x capacity]
        D --> E[New capacity: 8]
        E --> F[Continue pattern...]
    end

    subgraph "Memory Layout"
        G[List Object] --> H[Size: 3]
        G --> I[Capacity: 8]
        G --> J[Pointer Array]

        J --> K[Ref to obj1]
        J --> L[Ref to obj2]
        J --> M[Ref to obj3]
        J --> N[Empty slot]
        J --> O[Empty slot]
    end

    style G fill:#ffebee

      List Operations and Memory

      # Different operations, same object
lst = [1, 2, 3, 4, 5]
original_id = id(lst)

# Slice assignment - modifies existing list
lst[1:3] = [20, 30, 40]
print(id(lst) == original_id)  # True
print(lst)  # [1, 20, 30, 40, 4, 5]

# List comprehension - creates new list
new_lst = [x * 2 for x in lst]
print(id(new_lst) == original_id)  # False

# In-place sort vs sorted()
lst.sort()  # Modifies existing list
print(id(lst) == original_id)  # True

sorted_lst = sorted(lst)  # Creates new list
print(id(sorted_lst) == original_id)  # False
      Python

      Dictionaries (dict)

      Dictionaries are mutable mappings from keys to values:

      # Dictionary modification
d = {'a': 1, 'b': 2}
original_id = id(d)

# Various modification operations
d['c'] = 3           # Add new key-value
d['a'] = 99          # Modify existing value
d.update({'d': 4})   # Update with another dict
del d['b']           # Delete key-value pair

print(id(d) == original_id)  # True - same object
print(d)  # {'a': 99, 'c': 3, 'd': 4}
      Python

      Dictionary Internal Structure

      # Dictionary implementation details
d = {'name': 'Alice', 'age': 30, 'city': 'NYC'}

# Dictionaries maintain insertion order (Python 3.7+)
for key in d:
    print(key)  # name, age, city (in insertion order)

# Hash table structure
print(hash('name'))  # Hash value for string key
      Python
      graph TD
    subgraph "Dictionary Hash Table"
        A[Dict Object] --> B[Hash Table]
        B --> C[Bucket 0]
        B --> D[Bucket 1]
        B --> E[Bucket 2]
        B --> F[...]

        C --> G[Key: 'name'<br/>Value: 'Alice']
        D --> H[Empty]
        E --> I[Key: 'age'<br/>Value: 30]
    end

    subgraph "Key Hashing Process"
        J[Key: 'name'] --> K["hash('name')"]
        K --> L["Hash Value: -1234567"]
        L --> M["Bucket Index: 0"]
        M --> G
    end

    style A fill:#ffebee
    style B fill:#fff3e0

      Dictionary Views and References

      # Dictionary views share references to the original dict
d = {'a': 1, 'b': 2, 'c': 3}

keys_view = d.keys()
values_view = d.values()
items_view = d.items()

# Views reflect changes to original dict
d['d'] = 4
print(list(keys_view))   # ['a', 'b', 'c', 'd']
print(list(values_view)) # [1, 2, 3, 4]

# Views are live references, not copies
print(id(d))  # Original dict
# Views don't have independent identity from dict
      Python

      Sets (set)

      Sets are mutable collections of unique elements:

      # Set modification
s = {1, 2, 3}
original_id = id(s)

# Various modification operations
s.add(4)           # Add element
s.remove(2)        # Remove element (KeyError if not found)
s.discard(5)       # Remove element (no error if not found)
s.update([6, 7])   # Add multiple elements

print(id(s) == original_id)  # True - same object
print(s)  # {1, 3, 4, 6, 7}
      Python

      Set Operations

      # Set operations create new sets or modify existing ones
s1 = {1, 2, 3}
s2 = {3, 4, 5}

# Creates new sets
union_set = s1 | s2          # {1, 2, 3, 4, 5}
intersection_set = s1 & s2   # {3}
difference_set = s1 - s2     # {1, 2}

# Modifies existing set
s1 |= s2  # Equivalent to s1.update(s2)
print(id(s1))  # Same as original s1
      Python
      graph LR
    subgraph "Set Operations"
        A["Set s1: {1,2,3}"] --> B["Operation: |="]
        C["Set s2: {3,4,5}"] --> B
        B --> D["Modified s1: {1,2,3,4,5}"]

        E["Set s1: {1,2,3}"] --> F["Operation: |"]
        G["Set s2: {3,4,5}"] --> F
        F --> H["New Set: {1,2,3,4,5}"]
    end

    style A fill:#ffebee
    style D fill:#ffebee
    style H fill:#e8f5e8

      Custom Objects and Classes

      User-defined classes create mutable objects by default:

      class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __repr__(self):
        return f"Person('{self.name}', {self.age})"

# Object modification
person = Person("Alice", 30)
original_id = id(person)

# Modify attributes
person.age = 31
person.city = "NYC"  # Add new attribute

print(id(person) == original_id)  # True - same object
print(person)  # Person('Alice', 31)
print(person.city)  # NYC
      Python

      Making Custom Objects Immutable

      class ImmutablePerson:
    def __init__(self, name, age):
        self._name = name
        self._age = age

    @property
    def name(self):
        return self._name

    @property
    def age(self):
        return self._age

    def __setattr__(self, name, value):
        if hasattr(self, '_initialized'):
            raise AttributeError("Cannot modify immutable object")
        super().__setattr__(name, value)
        if name == '_age':  # Last attribute set
            super().__setattr__('_initialized', True)

# Usage
person = ImmutablePerson("Bob", 25)
# person.age = 26  # Would raise AttributeError
      Python

      Memory Sharing in Mutable Objects

      # Shallow vs deep references
matrix = [[1, 2], [3, 4]]
row1, row2 = matrix[0], matrix[1]

# Modifying through reference affects original
row1[0] = 99
print(matrix)  # [[99, 2], [3, 4]]

# List of same mutable object
default_list = []
data = [default_list, default_list, default_list]
data[0].append(1)
print(data)  # [[1], [1], [1]] - all refer to same list!
      Python
      graph TD
    subgraph "Shared Mutable References"
        A[Matrix] --> B["Row 0: [99, 2]"]
        A --> C["Row 1: [3, 4]"]

        D[Variable row1] --> B
        E[Variable row2] --> C

        F[Modification through row1] --> B
        G["Affects matrix[0]"] --> B
    end

    subgraph "Dangerous Pattern"
        H[data list] --> I[Index 0]
        H --> J[Index 1] 
        H --> K[Index 2]

        I --> L["Same List Object: [1]"]
        J --> L
        K --> L

        M["Append to data[0]"] --> L
        N[Affects all indices!] --> L
    end

    style B fill:#ffebee
    style L fill:#ffebee

      Heap vs Stack Memory Model

      Understanding how Python manages memory between the heap and stack is crucial for grasping object behavior and performance characteristics.

      Stack Memory in Python

      The stack stores function call frames, local variable references, and execution context:

      def outer_function(x):
    local_var = x * 2

    def inner_function(y):
        inner_var = y + local_var
        return inner_var

    result = inner_function(5)
    return result

# Call stack demonstration
final_result = outer_function(10)
      Python
      graph TD
    subgraph "Call Stack (Stack Memory)"
        A[Frame: main] --> B[Variables:final_result]

        C[Frame: outer_function] --> D[Parameters: x=10<br/>Variables: local_var=20, result=25]

        E[Frame: inner_function] --> F[Parameters: y=5<br/>Variables: inner_var=25]

        G[Stack Growth Direction] --> H[← Newest Frame]
        H --> E
        E --> C
        C --> A
    end

    style A fill:#e3f2fd
    style C fill:#e3f2fd
    style E fill:#e3f2fd

      What Lives on the Stack

      def demonstrate_stack():
    # These are references stored on the stack
    number_ref = 42           # Reference to integer object
    string_ref = "hello"      # Reference to string object
    list_ref = [1, 2, 3]      # Reference to list object

    # The actual objects live on the heap
    print(f"Number object id: {id(number_ref)}")
    print(f"String object id: {id(string_ref)}")
    print(f"List object id: {id(list_ref)}")

    return list_ref  # Returns reference, not the object

# Stack frame is destroyed when function returns
# But heap objects may persist if referenced elsewhere
      Python

      Heap Memory in Python

      The heap stores all Python objects and their data:

      # All these objects are created on the heap
integer_obj = 42
string_obj = "Python"
list_obj = [1, 2, 3, 4, 5]
dict_obj = {"key": "value", "count": 10}

class CustomObject:
    def __init__(self, data):
        self.data = data

custom_obj = CustomObject("example")
      Python
      graph TD
    subgraph "Heap Memory Layout"
        A[Integer ObjectValue: 42<br/>RefCount: 1<br/>Type: int]

        B[String ObjectValue: 'Python'<br/>RefCount: 1<br/>Type: str<br/>Length: 6]

        C[List ObjectSize: 5<br/>RefCount: 1<br/>Type: list] --> D["Element Array[ref1, ref2, ref3, ref4, ref5]"]

        E[Dict ObjectSize: 2<br/>RefCount: 1<br/>Type: dict] --> F[Hash TableBuckets...]

        G[Custom ObjectRefCount: 1<br/>Type: CustomObject] --> H[Attribute Dictdata: 'example']
    end

    subgraph "Stack References"
        I[integer_ref] -.-> A
        J[string_ref] -.-> B
        K[list_ref] -.-> C
        L[dict_ref] -.-> E
        M[custom_ref] -.-> G
    end

    style A fill:#fff3e0
    style B fill:#fff3e0
    style C fill:#fff3e0
    style E fill:#fff3e0
    style G fill:#fff3e0

      Memory Allocation Process

      # Step-by-step memory allocation
def create_objects():
    # 1. Stack frame created for function

    # 2. Heap objects allocated
    my_list = [1, 2, 3]  # List object on heap
    my_dict = {"a": 1}   # Dict object on heap

    # 3. Stack holds references to heap objects
    print(f"List at: {hex(id(my_list))}")
    print(f"Dict at: {hex(id(my_dict))}")

    # 4. Local references stored on stack
    another_ref = my_list  # Another stack reference to same heap object

    return my_list  # Stack reference copied to caller

# Memory after function call
returned_list = create_objects()
# Function stack frame destroyed, but heap object persists
      Python
      sequenceDiagram
    participant S as Stack
    participant H as Heap
    participant GC as Garbage Collector

    Note over S,H: Function Call Begins
    S->>S: Create stack frame
    S->>H: Allocate list object
    H-->>S: Return object address
    S->>S: Store reference in local variable

    Note over S,H: Function Returns
    S->>S: Copy reference to caller
    S->>S: Destroy local stack frame

    Note over H: Object remains on heap
    Note over GC: Reference count > 0, keep object

    Note over S,H: When all references removed
    S->>GC: Reference count reaches 0
    GC->>H: Deallocate object memory

      Reference Counting and Memory

      import sys

def memory_management_demo():
    # Create object
    data = [1, 2, 3, 4, 5]
    print(f"Initial ref count: {sys.getrefcount(data)}")

    # Add references
    backup = data
    another = data
    print(f"After adding refs: {sys.getrefcount(data)}")

    # Remove references
    del backup
    print(f"After del backup: {sys.getrefcount(data)}")

    del another
    print(f"After del another: {sys.getrefcount(data)}")

    # Function parameter adds temporary reference
    return data

result = memory_management_demo()
print(f"Final ref count: {sys.getrefcount(result)}")
      Python

      Memory Layout Visualization

      graph TB
    subgraph "Python Memory Model"
        subgraph "Stack (Function Frames)"
            SF1["main() frame<br/>result_ref → 0x1000"]
            SF2["demo() frame<br/>data_ref → 0x1000<br/>backup_ref → 0x1000"]
        end

        subgraph "Heap (Objects)"
            O1["List Object @ 0x1000<br/>Type: PyListObject<br/>Size: 5<br/>RefCount: 3<br/>Items: [ptr1, ptr2, ptr3, ptr4, ptr5]"]

            O2[Int Object @ 0x2000<br/>Type: PyLongObject<br/>Value: 1<br/>RefCount: many]

            O3[Int Object @ 0x3000<br/>Type: PyLongObject<br/>Value: 2<br/>RefCount: many]

            O1 --> O2
            O1 --> O3
            O1 --> O4[Int Object @ 0x4000]
            O1 --> O5[Int Object @ 0x5000]
            O1 --> O6[Int Object @ 0x6000]
        end

        subgraph "Small Int Cache"
            SC[Cached Integers-5 to 256<br/>Shared objects]
            O2 -.-> SC
            O3 -.-> SC
        end
    end

    style SF1 fill:#e3f2fd
    style SF2 fill:#e3f2fd
    style O1 fill:#fff3e0
    style SC fill:#e8f5e8

      Stack Overflow vs Heap Exhaustion

      # Stack overflow example (deep recursion)
def recursive_function(n):
    if n <= 0:
        return 0
    return n + recursive_function(n - 1)

# This will eventually cause RecursionError (stack overflow)
try:
    result = recursive_function(5000)
except RecursionError:
    print("Stack overflow occurred!")

# Heap exhaustion example
def heap_exhaustion():
    data = []
    try:
        while True:
            # Keep allocating memory on heap
            data.append([0] * 1000000)  # 1 million integers per iteration
    except MemoryError:
        print("Heap exhausted!")
        return len(data)

# This will eventually cause MemoryError
      Python

      Memory Efficiency Considerations

      import sys

# Memory-efficient vs inefficient patterns
def memory_comparison():
    # Inefficient: Creating many intermediate objects
    result1 = ""
    for i in range(1000):
        result1 += str(i)  # Creates new string each time

    # Efficient: Using list and join
    parts = []
    for i in range(1000):
        parts.append(str(i))  # Appends to existing list
    result2 = "".join(parts)  # Single string creation

    # Memory usage comparison
    print(f"String concat result size: {sys.getsizeof(result1)}")
    print(f"List join result size: {sys.getsizeof(result2)}")

    return result1, result2

# Generator for memory efficiency
def memory_efficient_processing():
    # Instead of creating large list in memory
    def process_large_data():
        return [x * x for x in range(1000000)]  # 1M integers in memory

    # Use generator for lazy evaluation
    def process_large_data_generator():
        return (x * x for x in range(1000000))  # Computed on demand

    list_data = process_large_data()
    gen_data = process_large_data_generator()

    print(f"List size: {sys.getsizeof(list_data)}")
    print(f"Generator size: {sys.getsizeof(gen_data)}")
      Python

      Python Internals and CPython Implementation

      Understanding how CPython implements objects and memory management provides insight into Python’s behavior and performance characteristics.

      PyObject Structure

      Every Python object is built on the fundamental PyObject structure:

      // Simplified CPython PyObject structure
typedef struct _object {
    Py_ssize_t ob_refcnt;    // Reference count
    PyTypeObject *ob_type;    // Type pointer
} PyObject;

// Variable-size objects extend this
typedef struct {
    PyObject ob_base;
    Py_ssize_t ob_size;      // Number of items
} PyVarObject;
      Python
      graph TD
    subgraph "PyObject Layout"
        A[PyObject Header] --> B[Reference Count<br/>8 bytes]
        A --> C[Type Pointer8 bytes]

        D[Object-Specific Data] --> E[Value/Content<br/>Variable size]
    end

    subgraph "Example: Integer Object"
        F[PyLongObject] --> G[ob_refcnt: 3]
        F --> H[ob_type: &PyLong_Type]
        F --> I[ob_size: 1]
        F --> J["ob_digit[0]: 42"]
    end

    subgraph "Example: List Object"
        K[PyListObject] --> L[ob_refcnt: 1]
        K --> M[ob_type: &PyList_Type]
        K --> N[ob_size: 3]
        K --> O[ob_item: ptr to array]
        O --> P[Array of PyObject*]
    end

    style A fill:#e3f2fd
    style F fill:#fff3e0
    style K fill:#ffebee

      Reference Counting Implementation

      import sys
import ctypes

def explore_reference_counting():
    # Create an object
    data = [1, 2, 3]
    print(f"Initial ref count: {sys.getrefcount(data)}")

    # Get memory address
    addr = id(data)
    print(f"Object address: {hex(addr)}")

    # Add references
    ref1 = data
    ref2 = data
    print(f"After adding refs: {sys.getrefcount(data)}")

    # Reference count is first 8 bytes of object
    # (This is for educational purposes - don't do this in production!)
    if sys.platform == 'win32':
        # Windows-specific low-level access
        pass  # Simplified for safety

    return data

# Demonstrate reference counting behavior
obj = explore_reference_counting()
      Python
      sequenceDiagram
    participant Code as Python Code
    participant RC as Reference Counter
    participant GC as Garbage Collector
    participant Heap as Heap Memory

    Code->>Heap: Create object
    Heap-->>RC: Initialize ref_count = 1

    Code->>RC: Add reference (assignment)
    RC->>RC: Increment ref_count

    Code->>RC: Remove reference (del/scope exit)
    RC->>RC: Decrement ref_count

    alt ref_count > 0
        RC-->>Code: Keep object alive
    else ref_count == 0
        RC->>GC: Object ready for collection
        GC->>Heap: Deallocate memory
    end

      Object Creation Process

      # Understanding object creation
def object_creation_analysis():
    # Integer creation
    x = 42
    # CPython checks small integer cache first
    # If cached, returns existing object
    # Otherwise, allocates new PyLongObject

    # String creation
    s = "hello"
    # CPython may intern string literals
    # Interned strings share memory

    # List creation
    lst = [1, 2, 3]
    # Allocates PyListObject
    # Allocates array for object pointers
    # Each element points to existing integer objects

    return x, s, lst

# Demonstrate object sharing
def object_sharing():
    # Small integers are shared
    a = 100
    b = 100
    print(f"Small int sharing: {a is b}")  # True

    # Large integers may not be shared
    x = 1000
    y = 1000
    print(f"Large int sharing: {x is y}")  # May be False

    # String interning
    s1 = "hello"
    s2 = "hello"
    print(f"String interning: {s1 is s2}")  # True
      Python

      Garbage Collection Details

      Python uses multiple garbage collection strategies:

      1. Reference Counting (primary)
      2. Cyclic Garbage Collector (for cycles)
      3. Generational Garbage Collection
      import gc
import weakref

def garbage_collection_demo():
    # Enable garbage collection debugging
    gc.set_debug(gc.DEBUG_STATS)

    # Create circular reference
    class Node:
        def __init__(self, value):
            self.value = value
            self.children = []
            self.parent = None

    # Create cycle: parent → child → parent
    parent = Node("parent")
    child = Node("child")
    parent.children.append(child)
    child.parent = parent

    # Weak reference to track object
    weak_parent = weakref.ref(parent)
    weak_child = weakref.ref(child)

    # Remove strong references
    del parent, child

    # Objects still exist due to cycle
    print(f"Parent alive: {weak_parent() is not None}")
    print(f"Child alive: {weak_child() is not None}")

    # Force garbage collection
    collected = gc.collect()
    print(f"Objects collected: {collected}")

    # Check if objects were collected
    print(f"Parent alive after GC: {weak_parent() is not None}")
    print(f"Child alive after GC: {weak_child() is not None}")
      Python
      graph TD
    subgraph "Garbage Collection Process"
        A[Reference Counting] --> B{RefCount = 0?}
        B -->|Yes| C[Immediate Deallocation]
        B -->|No| D[Check for Cycles]

        D --> E[Mark & Sweep GC]
        E --> F[Mark Reachable Objects]
        F --> G[Sweep Unreachable Objects]
        G --> H[Deallocate Cyclic Garbage]
    end

    subgraph "Generational GC"
        I[Generation 0Young Objects] --> J[Survive Collection]
        J --> K[Generation 1Middle-aged Objects]
        K --> L[Survive Collection]
        L --> M[Generation 2Old Objects]

        N[Collection Frequency] --> O[Gen 0: Most Frequent]
        N --> P[Gen 1: Medium]
        N --> Q[Gen 2: Least Frequent]
    end

    style C fill:#e8f5e8
    style H fill:#ffebee

      Memory Pools and Allocation

      CPython uses a sophisticated memory allocation system:

      import sys

def memory_allocation_details():
    # Python uses different allocators for different sizes

    # Small objects (< 512 bytes) use pymalloc
    small_objects = []
    for i in range(100):
        obj = [i]  # Small list object
        small_objects.append(obj)
        if i % 10 == 0:
            print(f"Small object {i}: {hex(id(obj))}")

    # Large objects use system malloc
    large_objects = []
    for i in range(10):
        obj = list(range(10000))  # Large list
        large_objects.append(obj)
        print(f"Large object {i}: {hex(id(obj))}")

    # Memory statistics
    print(f"Small objects size: {len(small_objects)}")
    print(f"Large objects size: {len(large_objects)}")

    return small_objects, large_objects

# Demonstrate memory pools
def memory_pool_behavior():
    # Create and destroy many small objects
    for _ in range(1000):
        temp_list = [1, 2, 3]
        temp_dict = {"key": "value"}
        # Objects destroyed at end of iteration
        # Memory may be reused from pools

    # Memory usage may not decrease immediately
    # due to pool retention for efficiency
      Python
      graph TD
    subgraph "CPython Memory Architecture"
        A[Object Allocation Request] --> B{Size < 512 bytes?}

        B -->|Yes| C[PyMalloc Arena System]
        B -->|No| D["System malloc()"]

        C --> E[Arena Pool]
        E --> F[Size Class Pools8, 16, 24, ... 512 bytes]
        F --> G[Memory Blocks]

        D --> H[Large Object Heap]

        I[Memory Reuse] --> C
        J[Pool Management] --> E
        K[Block Alignment] --> G
    end

    subgraph "Arena Structure"
        L[Arena256KB] --> M[Pool 14KB]
        L --> N[Pool 24KB]
        L --> O[Pool N4KB]

        M --> P[Blocks of same size]
        N --> Q[Blocks of same size]
    end

    style C fill:#e3f2fd
    style D fill:#fff3e0
    style L fill:#e8f5e8

      Type Objects and Method Resolution

      # Understanding Python's type system
class BaseClass:
    def method(self):
        return "base"

class DerivedClass(BaseClass):
    def method(self):
        return "derived"

    def derived_only(self):
        return "derived only"

def type_system_exploration():
    obj = DerivedClass()

    # Method resolution order
    print("MRO:", DerivedClass.__mro__)

    # Type object examination
    print("Type:", type(obj))
    print("Type of type:", type(type(obj)))

    # Attribute lookup process
    print("Method lookup:")
    print("  obj.method:", obj.method)
    print("  Type method:", DerivedClass.method)
    print("  Base method:", BaseClass.method)

    # Dynamic attribute access
    method_name = "method"
    dynamic_method = getattr(obj, method_name)
    print("Dynamic access:", dynamic_method())

# Demonstrate method caching
def method_caching():
    class Example:
        def __init__(self):
            self.call_count = 0

        def expensive_method(self):
            self.call_count += 1
            return f"Called {self.call_count} times"

    obj = Example()

    # First call
    result1 = obj.expensive_method()
    print(result1)

    # Method object is cached in instance __dict__
    # for subsequent faster lookups
    result2 = obj.expensive_method()
    print(result2)
      Python

      CPython Implementation Insights

      def cpython_insights():
    # Object layout inspection
    class InspectableClass:
        def __init__(self, value):
            self.value = value
            self.data = [1, 2, 3]

    obj = InspectableClass(42)

    # Object memory layout
    print(f"Object size: {sys.getsizeof(obj)}")
    print(f"Object dict: {obj.__dict__}")
    print(f"Dict size: {sys.getsizeof(obj.__dict__)}")

    # Slot-based classes for memory efficiency
    class SlottedClass:
        __slots__ = ['value', 'data']

        def __init__(self, value):
            self.value = value
            self.data = [1, 2, 3]

    slotted_obj = SlottedClass(42)
    print(f"Slotted object size: {sys.getsizeof(slotted_obj)}")

    # No __dict__ for slotted objects
    try:
        print(slotted_obj.__dict__)
    except AttributeError:
        print("Slotted objects have no __dict__")

# Performance implications
def performance_implications():
    import time

    # Dict-based attribute access
    class RegularClass:
        def __init__(self):
            self.x = 1
            self.y = 2

    # Slot-based attribute access
    class SlottedClass:
        __slots__ = ['x', 'y']

        def __init__(self):
            self.x = 1
            self.y = 2

    # Timing comparison
    regular_objs = [RegularClass() for _ in range(100000)]
    slotted_objs = [SlottedClass() for _ in range(100000)]

    # Access timing
    start = time.time()
    for obj in regular_objs:
        _ = obj.x + obj.y
    regular_time = time.time() - start

    start = time.time()
    for obj in slotted_objs:
        _ = obj.x + obj.y
    slotted_time = time.time() - start

    print(f"Regular class access time: {regular_time:.4f}s")
    print(f"Slotted class access time: {slotted_time:.4f}s")
    print(f"Speedup: {regular_time/slotted_time:.2f}x")
      Python

      Practical Examples and Best Practices

      Real-world scenarios where understanding mutability and memory management makes a significant difference in code quality and performance.

      Common Pitfalls and How to Avoid Them

      Mutable Default Arguments

      # WRONG: Mutable default argument
def add_item_wrong(item, target_list=[]):
    target_list.append(item)
    return target_list

# Problem: Same list object reused across calls
list1 = add_item_wrong("first")
list2 = add_item_wrong("second")
print(list1)  # ['first', 'second'] - Unexpected!
print(list2)  # ['first', 'second'] - Same object!
print(list1 is list2)  # True - Same object!

# CORRECT: Use None as default
def add_item_correct(item, target_list=None):
    if target_list is None:
        target_list = []
    target_list.append(item)
    return target_list

# Each call gets a new list
list3 = add_item_correct("first")
list4 = add_item_correct("second")
print(list3)  # ['first']
print(list4)  # ['second']
print(list3 is list4)  # False - Different objects
      Python
      graph TD
    subgraph "Wrong Pattern"
        A[Function Definition] --> B[Default List ObjectCreated Once]
        C[Call 1] --> B
        D[Call 2] --> B
        E[Call 3] --> B

        B --> F[Shared Mutable StateAccumulates Changes]
    end

    subgraph "Correct Pattern"
        G[Function Definition] --> H[None Default]
        I[Call 1] --> J[New List Object]
        K[Call 2] --> L[New List Object]
        M[Call 3] --> N[New List Object]

        O[Each Call Independent]
    end

    style F fill:#ffebee
    style O fill:#e8f5e8

      Shallow vs Deep Copy Issues

      import copy

# Shallow copy problems with nested mutable objects
original = [[1, 2], [3, 4], [5, 6]]

# Shallow copy
shallow = copy.copy(original)
shallow[0][0] = 99  # Modifies nested list

print(original)  # [[99, 2], [3, 4], [5, 6]] - Original affected!
print(shallow)   # [[99, 2], [3, 4], [5, 6]]

# Deep copy solution
original = [[1, 2], [3, 4], [5, 6]]
deep = copy.deepcopy(original)
deep[0][0] = 99  # Only affects copy

print(original)  # [[1, 2], [3, 4], [5, 6]] - Original unchanged
print(deep)      # [[99, 2], [3, 4], [5, 6]]

# Memory-efficient alternative for simple cases
def safe_list_copy(lst):
    return [row[:] for row in lst]  # List comprehension with slice copy

original = [[1, 2], [3, 4], [5, 6]]
safe_copy = safe_list_copy(original)
safe_copy[0][0] = 99

print(original)   # [[1, 2], [3, 4], [5, 6]] - Original unchanged
print(safe_copy)  # [[99, 2], [3, 4], [5, 6]]
      Python

      Loop Variable References

      # Closure and loop variable problem
functions = []

# WRONG: Late binding closure
for i in range(3):
    functions.append(lambda: i)  # All lambdas refer to same 'i'

# All functions return the final value of i
for func in functions:
    print(func())  # 2, 2, 2 - Not what we wanted!

# CORRECT: Early binding with default argument
functions = []
for i in range(3):
    functions.append(lambda x=i: x)  # Capture current value of i

for func in functions:
    print(func())  # 0, 1, 2 - Correct!

# ALTERNATIVE: Using functools.partial
from functools import partial

def return_value(x):
    return x

functions = []
for i in range(3):
    functions.append(partial(return_value, i))

for func in functions:
    print(func())  # 0, 1, 2 - Correct!
      Python

      Performance Optimization Techniques

      String Building Optimization

      import time

def string_concatenation_benchmark():
    n = 10000

    # SLOW: String concatenation creates many objects
    start_time = time.time()
    result = ""
    for i in range(n):
        result += str(i)  # Creates new string each time
    slow_time = time.time() - start_time

    # FAST: List join approach
    start_time = time.time()
    parts = []
    for i in range(n):
        parts.append(str(i))  # Appends to existing list
    result = "".join(parts)  # Single string creation
    fast_time = time.time() - start_time

    print(f"String concatenation: {slow_time:.4f}s")
    print(f"List join: {fast_time:.4f}s")
    print(f"Speedup: {slow_time/fast_time:.1f}x")

# Memory-efficient string formatting
def efficient_string_formatting():
    data = [("Alice", 30), ("Bob", 25), ("Charlie", 35)]

    # Memory-efficient with generator expression
    result = ", ".join(f"{name}({age})" for name, age in data)
    print(result)

    # For very large datasets, consider itertools
    from itertools import chain

    def format_person(name, age):
        return f"{name}({age})"

    # Process in chunks to manage memory
    def process_large_dataset(data, chunk_size=1000):
        for i in range(0, len(data), chunk_size):
            chunk = data[i:i + chunk_size]
            yield ", ".join(format_person(name, age) for name, age in chunk)

    # Usage for large datasets
    large_data = [("Person" + str(i), 20 + i % 50) for i in range(10000)]
    for chunk_result in process_large_dataset(large_data):
        # Process each chunk...
        pass
      Python

      List vs Collections Performance

      import time
from collections import deque, defaultdict, Counter
import sys

def collection_performance_comparison():
    # List vs deque for queue operations
    n = 100000

    # List as queue (inefficient)
    start_time = time.time()
    lst = []
    for i in range(n):
        lst.append(i)
    for i in range(n):
        lst.pop(0)  # O(n) operation!
    list_time = time.time() - start_time

    # Deque as queue (efficient)
    start_time = time.time()
    dq = deque()
    for i in range(n):
        dq.append(i)
    for i in range(n):
        dq.popleft()  # O(1) operation
    deque_time = time.time() - start_time

    print(f"List as queue: {list_time:.4f}s")
    print(f"Deque as queue: {deque_time:.4f}s")
    print(f"Speedup: {list_time/deque_time:.1f}x")

def memory_efficient_counting():
    # Manual counting vs Counter
    data = ["apple", "banana", "apple", "cherry", "banana", "apple"] * 1000

    # Manual dictionary counting
    start_time = time.time()
    manual_count = {}
    for item in data:
        if item in manual_count:
            manual_count[item] += 1
        else:
            manual_count[item] = 1
    manual_time = time.time() - start_time

    # Counter (optimized)
    start_time = time.time()
    counter_count = Counter(data)
    counter_time = time.time() - start_time

    # defaultdict approach
    start_time = time.time()
    default_count = defaultdict(int)
    for item in data:
        default_count[item] += 1
    default_time = time.time() - start_time

    print(f"Manual counting: {manual_time:.4f}s")
    print(f"Counter: {counter_time:.4f}s")
    print(f"defaultdict: {default_time:.4f}s")
      Python
      graph TD
    subgraph "Performance Optimization Strategies"
        A[Identify Bottleneck] --> B{Data Structure Choice}

        B --> C[Queue OperationsUse deque]
        B --> D[CountingUse Counter]
        B --> E[Default ValuesUse defaultdict]
        B --> F["String BuildingUse join()"]

        G[Memory Optimization] --> H[Use __slots__]
        G --> I[Use generators]
        G --> J[Use appropriate data types]

        K[Algorithm Choice] --> L[Time Complexity]
        K --> M[Space Complexity]
        K --> N[Cache Locality]
    end

    style A fill:#e3f2fd
    style G fill:#fff3e0
    style K fill:#e8f5e8

      Real-World Design Patterns

      Immutable Data Structures

      from typing import NamedTuple, List
from dataclasses import dataclass
from functools import lru_cache

# Using NamedTuple for immutable records
class Person(NamedTuple):
    name: str
    age: int
    email: str

    def with_age(self, new_age: int) -> 'Person':
        """Return new Person with updated age"""
        return self._replace(age=new_age)

    def is_adult(self) -> bool:
        return self.age >= 18

# Usage
person = Person("Alice", 30, "alice@example.com")
older_person = person.with_age(31)  # Creates new object

print(person)        # Original unchanged
print(older_person)  # New object with updated age

# Frozen dataclass alternative
@dataclass(frozen=True)
class ImmutableConfig:
    host: str
    port: int
    debug: bool = False

    def with_debug(self, debug: bool) -> 'ImmutableConfig':
        return ImmutableConfig(self.host, self.port, debug)

config = ImmutableConfig("localhost", 8080)
debug_config = config.with_debug(True)
      Python

      Memory-Efficient Caching

      from functools import lru_cache
import weakref

class MemoizedClass:
    """Class that caches expensive computations"""

    def __init__(self):
        self._cache = {}

    @lru_cache(maxsize=128)
    def expensive_computation(self, x: int) -> int:
        """Cached method using lru_cache"""
        print(f"Computing for {x}")  # Shows when actually computed
        return x ** 2 + x * 3 + 1

    def manual_cache_method(self, x: int) -> int:
        """Manual caching with size limit"""
        if x in self._cache:
            return self._cache[x]

        result = x ** 3 + x ** 2 + x + 1

        # Prevent cache from growing too large
        if len(self._cache) > 100:
            # Remove oldest items (simplified LRU)
            oldest_key = next(iter(self._cache))
            del self._cache[oldest_key]

        self._cache[x] = result
        return result

# Weak reference caching for object management
class ObjectManager:
    """Manages objects with weak references to prevent memory leaks"""

    def __init__(self):
        self._objects = weakref.WeakValueDictionary()

    def get_object(self, key: str, factory_func):
        """Get cached object or create new one"""
        obj = self._objects.get(key)
        if obj is None:
            obj = factory_func()
            self._objects[key] = obj
        return obj

# Usage example
def create_expensive_object():
    return [i ** 2 for i in range(10000)]

manager = ObjectManager()
obj1 = manager.get_object("data1", create_expensive_object)
obj2 = manager.get_object("data1", create_expensive_object)  # Returns cached object
print(obj1 is obj2)  # True - same object returned
      Python

      Context Managers for Resource Management

      import contextlib
from typing import Any, Iterator

class ManagedResource:
    """Example resource that needs proper cleanup"""

    def __init__(self, name: str):
        self.name = name
        self.data = []
        print(f"Acquiring resource: {name}")

    def use_resource(self, value: Any):
        self.data.append(value)

    def cleanup(self):
        print(f"Cleaning up resource: {self.name}")
        self.data.clear()

# Traditional context manager
class ResourceManager:
    def __init__(self, name: str):
        self.name = name
        self.resource = None

    def __enter__(self):
        self.resource = ManagedResource(self.name)
        return self.resource

    def __exit__(self, exc_type, exc_val, exc_tb):
        if self.resource:
            self.resource.cleanup()
        return False

# Using contextlib for cleaner code
@contextlib.contextmanager
def managed_resource(name: str) -> Iterator[ManagedResource]:
    resource = ManagedResource(name)
    try:
        yield resource
    finally:
        resource.cleanup()

# Usage examples
with ResourceManager("database") as db:
    db.use_resource("data1")
    db.use_resource("data2")
# Automatic cleanup when exiting context

with managed_resource("file_handler") as handler:
    handler.use_resource("line1")
    handler.use_resource("line2")
# Automatic cleanup guaranteed
      Python

      Memory Profiling and Debugging

      import tracemalloc
import sys
import gc
from memory_profiler import profile  # pip install memory-profiler

def memory_debugging_tools():
    # Enable memory tracing
    tracemalloc.start()

    # Create some objects
    data = []
    for i in range(1000):
        data.append([i] * 100)

    # Get memory statistics
    current, peak = tracemalloc.get_traced_memory()
    print(f"Current memory usage: {current / 1024 / 1024:.1f} MB")
    print(f"Peak memory usage: {peak / 1024 / 1024:.1f} MB")

    # Get top memory consumers
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')

    print("\nTop 3 memory consumers:")
    for stat in top_stats[:3]:
        print(stat)

    tracemalloc.stop()

@profile  # Decorator for line-by-line memory profiling
def memory_intensive_function():
    # This function will be profiled line by line
    large_list = []
    for i in range(100000):
        large_list.append(i ** 2)

    # Transform data
    processed = [x * 2 for x in large_list]

    # Clean up
    del large_list
    return processed

def garbage_collection_analysis():
    # Analyze garbage collection behavior
    gc.set_debug(gc.DEBUG_STATS)

    # Create objects with circular references
    objects = []
    for i in range(1000):
        obj = {"id": i, "refs": []}
        obj["refs"].append(obj)  # Self-reference
        objects.append(obj)

    # Check generation counts before collection
    print("Before GC:", gc.get_count())

    # Force garbage collection
    collected = gc.collect()
    print(f"Objects collected: {collected}")
    print("After GC:", gc.get_count())

    # Clean up
    objects.clear()
      Python

      Advanced Topics and Edge Cases

      Exploring sophisticated aspects of Python’s memory model, including optimization techniques and unusual behaviors.

      String Interning Deep Dive

      import sys

def string_interning_exploration():
    # Automatic interning for identifiers
    a = "hello_world"
    b = "hello_world"
    print(f"Identifier-like strings: {a is b}")  # True

    # Non-identifier strings may not be interned
    x = "hello world!"
    y = "hello world!"
    print(f"Non-identifier strings: {x is y}")  # May be False

    # Manual interning
    x_interned = sys.intern(x)
    y_interned = sys.intern(y)
    print(f"Manually interned: {x_interned is y_interned}")  # True

    # Compile-time vs runtime string creation
    compile_time_1 = "constant"
    compile_time_2 = "constant"
    print(f"Compile-time constants: {compile_time_1 is compile_time_2}")  # True

    # Runtime string creation
    prefix = "run"
    suffix = "time"
    runtime_1 = prefix + suffix
    runtime_2 = prefix + suffix
    print(f"Runtime strings: {runtime_1 is runtime_2}")  # May be False

    # String interning criteria
    def test_interning(s):
        s1 = s
        s2 = s
        return s1 is s2

    test_cases = [
        "simple",           # Likely interned
        "hello_world",      # Identifier-like, likely interned
        "hello world",      # Contains space, may not be interned
        "a" * 20,          # Long string, likely not interned
        "π",               # Unicode, may not be interned
        "",                # Empty string, interned
    ]

    for case in test_cases:
        print(f"'{case}': {test_interning(case)}")

# Practical implications of string interning
def interning_performance():
    import time

    # Large number of repeated strings
    strings = ["identifier_" + str(i % 100) for i in range(100000)]

    # Without interning - many duplicate string objects
    start = time.time()
    unique_count = len(set(strings))
    no_intern_time = time.time() - start

    # With interning - shared string objects
    start = time.time()
    interned_strings = [sys.intern(s) for s in strings]
    unique_interned = len(set(interned_strings))
    intern_time = time.time() - start

    print(f"Without interning: {no_intern_time:.4f}s")
    print(f"With interning: {intern_time:.4f}s")
    print(f"Memory savings: {unique_count == unique_interned}")
      Python
      graph TD
    subgraph "String Interning Process"
        A[String Literal] --> B{Is Identifier-like?}
        B -->|Yes| C[Check Intern Table]
        B -->|No| D[Create New String]

        C --> E{Already Interned?}
        E -->|Yes| F[Return Existing Object]
        E -->|No| G[Add to Intern Table]
        G --> H[Return New Interned Object]

        I["sys.intern() Call"] --> J[Force Interning]
        J --> C
    end

    subgraph "Intern Table Structure"
        K[Intern Dictionary] --> L[String Value: Object Reference]
        L --> M["'hello': 0x1000"]
        L --> N["'world': 0x2000"]
        L --> O["'python': 0x3000"]
    end

    style F fill:#e8f5e8
    style H fill:#e8f5e8
    style D fill:#fff3e0

      Copy vs Deepcopy Advanced Scenarios

      import copy
import weakref

class ComplexObject:
    def __init__(self, name, data=None):
        self.name = name
        self.data = data or []
        self.references = []
        self.metadata = {"created": True}

    def add_reference(self, obj):
        self.references.append(weakref.ref(obj))

    def __deepcopy__(self, memo):
        # Custom deepcopy implementation
        print(f"Deep copying {self.name}")

        # Create new instance
        new_obj = type(self)(
            copy.deepcopy(self.name, memo),
            copy.deepcopy(self.data, memo)
        )

        # Handle references specially
        new_obj.references = []  # Don't copy weak references
        new_obj.metadata = copy.deepcopy(self.metadata, memo)

        return new_obj

def copy_scenarios():
    # Scenario 1: Circular references
    obj1 = ComplexObject("Object1")
    obj2 = ComplexObject("Object2")

    obj1.data = [obj2]
    obj2.data = [obj1]  # Circular reference

    # Shallow copy handles circular references
    shallow = copy.copy(obj1)
    print(f"Shallow copy shares data: {shallow.data is obj1.data}")

    # Deep copy handles circular references correctly
    deep = copy.deepcopy(obj1)
    print(f"Deep copy creates new data: {deep.data is not obj1.data}")
    print(f"Circular reference preserved: {deep.data[0].data[0] is deep}")

    # Scenario 2: Custom copy behavior
    obj3 = ComplexObject("Object3", [1, 2, 3])
    obj4 = ComplexObject("Object4")
    obj3.add_reference(obj4)

    deep_custom = copy.deepcopy(obj3)
    print(f"Custom deepcopy preserves type: {type(deep_custom) == type(obj3)}")
    print(f"References not copied: {len(deep_custom.references) == 0}")

# Memory-efficient copying strategies
def efficient_copying():
    # Copy-on-write simulation
    class CowList:
        def __init__(self, data=None):
            self._data = data or []
            self._copied = False

        def _ensure_copied(self):
            if not self._copied:
                self._data = self._data[:]  # Shallow copy
                self._copied = True

        def append(self, item):
            self._ensure_copied()
            self._data.append(item)

        def __getitem__(self, index):
            return self._data[index]

        def __len__(self):
            return len(self._data)

        def copy(self):
            # Cheap copy - shares data until modification
            new_cow = CowList(self._data)
            return new_cow

    # Demonstration
    original = CowList([1, 2, 3, 4, 5])
    copy1 = original.copy()  # Cheap operation
    copy2 = original.copy()  # Cheap operation

    print(f"Shared data initially: {copy1._data is original._data}")

    copy1.append(6)  # Triggers copy
    print(f"After modification: {copy1._data is original._data}")  # False
    print(f"Copy2 still shares: {copy2._data is original._data}")   # True

# Immutable data structure patterns
def immutable_patterns():
    from collections import namedtuple
    from typing import Tuple, Any

    # Persistent data structure simulation
    class ImmutableList:
        def __init__(self, items: Tuple[Any, ...] = ()):
            self._items = items

        def append(self, item) -> 'ImmutableList':
            return ImmutableList(self._items + (item,))

        def prepend(self, item) -> 'ImmutableList':
            return ImmutableList((item,) + self._items)

        def __getitem__(self, index):
            return self._items[index]

        def __len__(self):
            return len(self._items)

        def __iter__(self):
            return iter(self._items)

    # Usage - each operation returns new object
    list1 = ImmutableList((1, 2, 3))
    list2 = list1.append(4)
    list3 = list1.prepend(0)

    print(f"Original: {tuple(list1)}")  # (1, 2, 3)
    print(f"Appended: {tuple(list2)}")  # (1, 2, 3, 4)
    print(f"Prepended: {tuple(list3)}") # (0, 1, 2, 3)
    print(f"Original unchanged: {tuple(list1)}")  # (1, 2, 3)
      Python

      Memory Optimization Techniques

      import sys
import array
from collections import defaultdict

def memory_optimization_strategies():
    # Strategy 1: Use __slots__ for classes
    class RegularClass:
        def __init__(self, x, y, z):
            self.x = x
            self.y = y
            self.z = z

    class SlottedClass:
        __slots__ = ['x', 'y', 'z']

        def __init__(self, x, y, z):
            self.x = x
            self.y = y
            self.z = z

    # Memory comparison
    regular_obj = RegularClass(1, 2, 3)
    slotted_obj = SlottedClass(1, 2, 3)

    print(f"Regular object: {sys.getsizeof(regular_obj)} bytes")
    print(f"Slotted object: {sys.getsizeof(slotted_obj)} bytes")

    # Strategy 2: Use array for numeric data
    python_list = [i for i in range(1000)]
    int_array = array.array('i', range(1000))  # 'i' for signed int

    print(f"Python list: {sys.getsizeof(python_list)} bytes")
    print(f"Array: {sys.getsizeof(int_array)} bytes")

    # Strategy 3: Use generators for large datasets
    def large_dataset_list():
        return [x ** 2 for x in range(1000000)]

    def large_dataset_generator():
        return (x ** 2 for x in range(1000000))

    list_data = large_dataset_list()
    gen_data = large_dataset_generator()

    print(f"List memory: {sys.getsizeof(list_data)} bytes")
    print(f"Generator memory: {sys.getsizeof(gen_data)} bytes")

def memory_pooling_patterns():
    # Object pooling for expensive objects
    class ObjectPool:
        def __init__(self, factory_func, max_size=10):
            self._factory = factory_func
            self._pool = []
            self._max_size = max_size

        def acquire(self):
            if self._pool:
                return self._pool.pop()
            return self._factory()

        def release(self, obj):
            if len(self._pool) < self._max_size:
                # Reset object state if needed
                if hasattr(obj, 'reset'):
                    obj.reset()
                self._pool.append(obj)

    class ExpensiveObject:
        def __init__(self):
            self.data = [0] * 1000  # Simulate expensive initialization
            self.state = "ready"

        def reset(self):
            self.data = [0] * 1000
            self.state = "ready"

        def use(self, value):
            self.state = "in_use"
            self.data[0] = value

    # Usage
    pool = ObjectPool(ExpensiveObject, max_size=5)

    # Acquire objects
    obj1 = pool.acquire()
    obj2 = pool.acquire()

    # Use objects
    obj1.use(42)
    obj2.use(99)

    # Release back to pool
    pool.release(obj1)
    pool.release(obj2)

    # Next acquisition reuses pooled objects
    obj3 = pool.acquire()  # Gets obj2 or obj1
    print(f"Reused object state: {obj3.state}")

# Memory leak detection and prevention
def memory_leak_patterns():
    import weakref
    import gc

    class LeakyClass:
        instances = []  # Class variable holding references

        def __init__(self, name):
            self.name = name
            LeakyClass.instances.append(self)  # Creates memory leak!

    class NonLeakyClass:
        _instances = weakref.WeakSet()  # Weak references

        def __init__(self, name):
            self.name = name
            NonLeakyClass._instances.add(self)

        @classmethod
        def get_instance_count(cls):
            return len(cls._instances)

    # Demonstrate leak
    for i in range(100):
        obj = LeakyClass(f"leaky_{i}")
        # obj goes out of scope, but reference remains in class list

    for i in range(100):
        obj = NonLeakyClass(f"non_leaky_{i}")
        # obj can be garbage collected when out of scope

    print(f"Leaky instances: {len(LeakyClass.instances)}")
    print(f"Non-leaky instances: {NonLeakyClass.get_instance_count()}")

    # Force garbage collection
    gc.collect()
    print(f"After GC - Non-leaky instances: {NonLeakyClass.get_instance_count()}")
      Python
      graph TD
    subgraph "Memory Optimization Techniques"
        A[Identify Memory Hotspots] --> B[Choose Optimization Strategy]

        B --> C[Use __slots__<br/>For classes with fixed attributes]
        B --> D[Use array module<br/>For numeric data]
        B --> E[Use generators<br/>For large sequences]
        B --> F[Implement object pooling<br/>For expensive objects]
        B --> G[Use weak references<br/>To prevent leaks]

        H[Memory Profiling] --> I[tracemalloc]
        H --> J[memory_profiler]
        H --> K[pympler]

        L[Best Practices] --> M[Avoid circular references]
        L --> N[Use context managers]
        L --> O[Profile before optimizing]
    end

    style A fill:#e3f2fd
    style H fill:#fff3e0
    style L fill:#e8f5e8

      Performance Implications and Optimization

      Understanding when and how to optimize based on Python’s memory model.

      Benchmarking and Profiling

      import time
import cProfile
import pstats
from functools import wraps
from memory_profiler import profile

def timing_decorator(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        end = time.perf_counter()
        print(f"{func.__name__} took {end - start:.4f} seconds")
        return result
    return wrapper

@timing_decorator
def compare_data_structures():
    n = 100000

    # List operations
    lst = []
    for i in range(n):
        lst.append(i)

    # Deque operations
    from collections import deque
    dq = deque()
    for i in range(n):
        dq.append(i)

    # Set operations
    s = set()
    for i in range(n):
        s.add(i)

def profile_memory_usage():
    """Comprehensive memory profiling example"""

    @profile
    def memory_intensive_operation():
        # Create large data structures
        data = []
        for i in range(10000):
            row = [j for j in range(100)]
            data.append(row)

        # Transform data
        processed = []
        for row in data:
            processed.append([x * 2 for x in row])

        # Clean up intermediate data
        del data

        # Final processing
        result = sum(sum(row) for row in processed)
        return result

    return memory_intensive_operation()

def cpu_profiling():
    """CPU profiling with cProfile"""

    def fibonacci(n):
        if n <= 1:
            return n
        return fibonacci(n-1) + fibonacci(n-2)

    def optimized_fibonacci(n, memo={}):
        if n in memo:
            return memo[n]
        if n <= 1:
            return n
        memo[n] = optimized_fibonacci(n-1, memo) + optimized_fibonacci(n-2, memo)
        return memo[n]

    # Profile both versions
    pr = cProfile.Profile()

    print("Profiling naive fibonacci...")
    pr.enable()
    result1 = fibonacci(30)
    pr.disable()

    stats = pstats.Stats(pr)
    stats.sort_stats('cumulative')
    stats.print_stats(10)

    print("\nProfiling optimized fibonacci...")
    pr = cProfile.Profile()
    pr.enable()
    result2 = optimized_fibonacci(30)
    pr.disable()

    stats = pstats.Stats(pr)
    stats.sort_stats('cumulative')
    stats.print_stats(10)
      Python

      Conclusion

      Understanding Python’s memory model and the distinction between mutable and immutable data types is fundamental to writing efficient, maintainable Python code. This knowledge impacts:

      Key Takeaways

      1. Everything is an Object: Python’s object model means all data has identity, type, and value
      2. References vs Objects: Variables are references to objects, not the objects themselves
      3. Mutability Matters: Understanding when objects can change affects program correctness
      4. Memory Management: Python’s automatic memory management has performance implications
      5. Optimization Opportunities: Knowing internals reveals optimization strategies

      Best Practices Summary

      • Use immutable data types when possible for safer code
      • Be aware of mutable default arguments and shared references
      • Choose appropriate data structures for your use case
      • Profile before optimizing
      • Understand the performance characteristics of your code

      Further Learning

      • Study CPython source code for deeper understanding
      • Experiment with memory profiling tools
      • Practice with different data structure scenarios
      • Learn about concurrent programming and the GIL
      • Explore other Python implementations (PyPy, Jython, etc.)

      The journey to mastering Python’s memory model is ongoing, but the concepts in this book provide a solid foundation for writing better Python code.

      Discover more from Altgr Blog

      Subscribe to get the latest posts sent to your email.

      Related Posts

      Leave a Reply

      Your email address will not be published. Required fields are marked *