Series: Stacks · Queues and deques · Linked lists

COL 1101 Programming with Data Structures — Lectures 14 and 15, rewritten as one guide

Linked lists

An array keeps its elements side by side, which makes indexing instant and inserting in the middle slow. A linked list makes the opposite trade: each element lives in its own node that points to the next, so you can splice anywhere you are standing in constant time, but you can only get there by walking. This page builds singly, circularly and doubly linked lists, then the positional list that lets users hold their place, and two algorithms that run on it.

Four nodes. Each holds an element and a link to the next node; the last links to nothing. The list itself only remembers where the chain starts and ends.

How to use this page

Parts 1–3 are Lecture 14; Parts 4–7 are Lecture 15 (which begins with a recap of the end of Lecture 14). Try every Check yourself before opening the answers. Dashed boxes marked Beyond the lecture are optional. Part 8 is the topic on one page. Stacks, queues and amortized cost are explained on their own pages; this one uses them freely.

Reference (pointer): a value that says where an object is in memory. A Python variable holding an object holds a reference to it. None is the reference to nothing.

Node: a small object holding one element (really, a reference to it) plus one or more references to other nodes.

Traverse (walk): follow next references from node to node. Reaching the k-th node takes k steps; there is no shortcut.

Part 1

Why not just use an array?

1.1What arrays are bad at

A Python list is a dynamic array: references stored in one contiguous block, with O(1) access to any index and O(1) amortized append. The lecture lists three drawbacks:

  1. Wasted space. The block's capacity is usually larger than the number of elements, by up to a constant factor, so that appends stay cheap.
  2. Insertions and deletions in the interior cost O(n). An array has no gaps, so inserting at index i shifts every element from i onward one place right; deleting shifts them left.
  3. Amortized bounds can be unacceptable in real-time systems. An append is usually O(1) but occasionally O(n) when the array resizes. A system that must respond within a deadline every time cannot accept the occasional long pause.
Inserting X after B. Top: in an array, C, D and E each move one slot right (O(n) in general). Bottom: in a linked list, nothing moves; a new node is created and two links are set (O(1), once you are at B).
Linked lists trade fast indexing for fast splicing. The question to ask of any operation is: do I already have a reference to the node where it happens, or do I have to walk to it?

Part 2

Singly linked lists

2.1Structure

A singly linked list is a sequence of nodes. Each node stores a reference to its element and a reference next to the following node; the last node's next is None. The list object stores a reference to the first node, the head, and optionally a reference to the last node, the tail, and the number of nodes, size.

The lecture calls this a “collaboration of many objects”: the element objects (which the list does not own and never copies), the nodes (one per element), and the list object itself (head, tail, size).

Two consequences follow immediately:

  • There is no spare capacity: a node is created for each element and discarded when it leaves, so memory is proportional to the current size at all times.
  • There is no index arithmetic: to reach the 5th element you start at the head and follow next four times. Access by index is O(n).

2.2Inserting at the head: O(1)

  1. Allocate a new node.
  2. Store the new element in it.
  3. Make the new node's next point to the old head.
  4. Update head to point to the new node.
Inserting LAX at the head. Step 3 must come before step 4: if head moved first, nothing would point to MSP any more and the rest of the list would be lost.

In Python, steps 1–4 collapse into one line, self.head = Node(e, self.head), because the right-hand side is evaluated (using the old head) before the assignment. If the list was empty, the new node is also the tail.

2.3Inserting at the tail: O(1), if you keep a tail reference

  1. Allocate a new node and store the element.
  2. Make the new node's next point to None (it will be last).
  3. Make the old last node's next point to the new node.
  4. Update tail to point to the new node.
Inserting MIA at the tail. The lecture's warning: set the old tail's next before moving tail. Once tail points to MIA you have no reference to BOS left (except by walking from the head).

Without a stored tail, you would have to walk from the head to find the last node: O(n). Keeping one extra reference makes it O(1). If the list was empty, the new node is also the head.

2.4Removing from the head: O(1)

  1. Update head to point to the next node in the list.
  2. Let Python's garbage collector reclaim the former first node, which nothing references any more.

If that empties the list, tail must be set to None as well; otherwise it still points to the removed node.

2.5Removing from the tail: O(n), and why

After removing the last node, tail must point to the second-to-last node, and that node's next must become None. But links only point forward: the last node does not know who points to it. The only way to find the node before the tail is to start at the head and walk until walk.next is tail. That walk is n − 2 steps.

There is no constant-time way to remove the tail of a singly linked list, even with a tail reference. A tail reference tells you where the last node is, not where the one before it is. Part 4 fixes this with backward links.

Search for e: walk from the head comparing each element with e; stop at the first match or at None. Worst case n comparisons.

Delete the first node containing e: search for it, but also remember the node before it (call it prev), because deleting means making prev.next skip over the found node. Three cases: the match is the head (move head instead, there is no prev); the match is the tail (also move tail to prev); otherwise just link around it.

Python
class Empty(Exception):
    pass

class SinglyLinkedList:

    class _Node:
        __slots__ = "element", "next"            # two fields, no per-object dict
        def __init__(self, element, nxt):
            self.element = element
            self.next = nxt

    def __init__(self):
        self.head = None
        self.tail = None
        self.size = 0

    def add_head(self, e):                       # O(1)
        self.head = self._Node(e, self.head)     # new node points to old head
        if self.size == 0:
            self.tail = self.head                # first node is also the last
        self.size += 1

    def add_tail(self, e):                       # O(1), thanks to the tail reference
        newest = self._Node(e, None)
        if self.size == 0:
            self.head = newest
        else:
            self.tail.next = newest              # link the old tail FIRST...
        self.tail = newest                       # ...then move tail
        self.size += 1

    def remove_head(self):                       # O(1)
        if self.size == 0:
            raise Empty("list is empty")
        e = self.head.element
        self.head = self.head.next               # old first node is now unreachable
        self.size -= 1
        if self.size == 0:
            self.tail = None
        return e

    def remove_tail(self):                       # O(n): must find the node before tail
        if self.size == 0:
            raise Empty("list is empty")
        if self.size == 1:
            return self.remove_head()
        walk = self.head
        while walk.next is not self.tail:
            walk = walk.next
        e = self.tail.element
        walk.next = None
        self.tail = walk
        self.size -= 1
        return e

    def find(self, e):                           # O(n)
        walk = self.head
        while walk is not None and walk.element != e:
            walk = walk.next
        return walk                              # the node, or None

    def delete(self, e):                         # O(n): delete first node holding e
        prev, walk = None, self.head
        while walk is not None and walk.element != e:
            prev, walk = walk, walk.next         # remember the node BEFORE walk
        if walk is None:
            return False
        if prev is None:
            self.head = walk.next                # deleting the head
        else:
            prev.next = walk.next                # link around walk
        if walk is self.tail:
            self.tail = prev
        self.size -= 1
        return True

    def __iter__(self):
        walk = self.head
        while walk is not None:
            yield walk.element
            walk = walk.next

L = SinglyLinkedList()
for code in ["MSP", "ATL", "BOS"]:
    L.add_tail(code)
L.add_head("LAX"); print(list(L))
L.add_tail("MIA"); print(list(L))
print(L.remove_head(), L.remove_tail(), list(L), "tail =", L.tail.element)
print(L.find("ATL").element, L.find("SFO"), L.delete("ATL"), list(L))
['LAX', 'MSP', 'ATL', 'BOS']
['LAX', 'MSP', 'ATL', 'BOS', 'MIA']
LAX MIA ['MSP', 'ATL', 'BOS'] tail = BOS
ATL None True ['MSP', 'BOS']

Singly linked list playground

Every operation reports how many nodes it had to walk past and which links it changed. Try remove_tail on a long list, then remove_head, and compare.

Singly linked list operationCostWhy
add_head, remove_headO(1)head is stored
add_tailO(1)tail is stored (O(n) without it)
remove_tailO(n)must walk to the node before the tail
access by index, search, delete by valueO(n)must walk from the head
lenO(1)size is stored

Check yourself: singly linked lists

  1. Write the two lines of add_tail that change links, in the right order. What goes wrong if you swap them?
  2. A list holds 1,000 nodes. How many next references must be followed to remove the tail?
  3. In delete(e), why do we need prev? Why not just make the found node “disappear”?
  4. After remove_head empties the list, what must happen to tail?
Show answers
  1. self.tail.next = newest, then self.tail = newest. Swapped, the first line makes tail the new node, and the second sets the new node's next to itself: the old last node never links to it, so the new node is unreachable from the head, and walking off the end loops forever.
  2. 998: from the head (node 1) to node 999, the one before the tail.
  3. A node cannot remove itself: something points to it, and that pointer must be changed to skip it. In a singly linked list the only thing pointing to a node is the previous node's next, so we must have the previous node in hand.
  4. It must become None; otherwise it refers to a node that is no longer in the list, and the next add_tail would link onto that dead node.

Part 3

Stacks, queues and circular lists

3.1Linked stack and linked queue

Choose the ends so that every operation avoids remove_tail:

ADTPlacementOperations usedCost
Stacktop = headpush = add_head; pop = remove_head; top = head.elementall O(1)
Queuefront = head, rear = tailenqueue = add_tail; dequeue = remove_head; first = head.elementall O(1)

Unlike the array versions, these are O(1) in the worst case, not amortized: nothing ever resizes. Space is O(n). Full code for both is on the Stacks and Queues pages.

3.2Circularly linked lists

Make the tail's next point back to the head instead of None. There is no longer a natural start or end, so instead of head and tail the list keeps one reference, usually called current. If current is treated as the “last” node, then current.next is the “first”, so both ends are reachable from one reference.

A circularly linked list. Following next from any node eventually returns to it. rotate() moves current one step, which makes the old first node the new last one.

Circular lists fit data that is naturally cyclic: a circular bus or train route, a playlist on repeat-all, the turn order in a multiplayer game.

3.3Example: round-robin scheduling

A round-robin scheduler iterates through a collection in a circle and “services” each element in turn, for example giving each running application a slice of CPU time.

  • With a queue: dequeue the next element, service it, enqueue it again. Every turn removes a node and creates a new one.
  • With a circular list: service front(), then rotate(). One reference moves; nothing is removed or created.
Python
class CircularList:
    """Circularly linked list, kept by one reference: current (the 'last' node)."""

    class _Node:
        __slots__ = "element", "next"
        def __init__(self, element, nxt):
            self.element, self.next = element, nxt

    def __init__(self):
        self.current = None
        self.size = 0

    def add(self, e):                       # insert after current, become current
        newest = self._Node(e, None)
        if self.current is None:
            newest.next = newest            # a one-node circle points at itself
        else:
            newest.next = self.current.next
            self.current.next = newest
        self.current = newest
        self.size += 1

    def front(self):
        return self.current.next.element    # the node after 'last' is the 'first'

    def rotate(self):                       # O(1): one reference moves
        self.current = self.current.next

C = CircularList()
for app in ["editor", "browser", "music"]:
    C.add(app)
schedule = []
for turn in range(7):
    schedule.append(C.front())              # give this app its time slice
    C.rotate()                              # move on: no removal, no insertion
print(schedule)
['editor', 'browser', 'music', 'editor', 'browser', 'music', 'editor']

Check yourself: circular lists

  1. A circular list holds A → B → C → (back to A) with current at C. What does front() return? After rotate()?
  2. How do you know when a traversal of a circular list has visited every node once?
Show answers
  1. front() = C.next = A. After rotate, current = A, so front() = B.
  2. There is no None to stop at. Either count size steps, or remember the starting node and stop when you return to it.

Part 4

Doubly linked lists

4.1Links in both directions

The singly linked list could not remove its tail quickly because a node does not know its predecessor. So give it one. In a doubly linked list every node stores three references: its element, next and prev. Now from any node you can step either way in O(1).

4.2Sentinels: the header and trailer

The list also keeps two extra nodes that hold no element: a header before the first real node and a trailer after the last. They are called sentinels (or guards), are created with the list, and are never removed. An empty list is just header ⇄ trailer.

A doubly linked list holding A, B, C. Upper arrows are next links; lower arrows are prev links. The header and trailer are permanent and store no element.

Why spend two nodes on nothing? Because every real node is now guaranteed to have a real node on both sides. Inserting at the front is just “insert between header and its successor”; inserting into an empty list is “insert between header and trailer”. There is no special case for an empty list, the first node or the last node, and the same two short procedures below handle every insertion and every deletion. Fewer special cases means fewer bugs.

4.3Insertion: four links

To insert a new node q holding X between node p and its successor s (= p.next):

Insert X between p and its successor
s = p.next
q = new node(X)
q.prev = p              # new node's own two links
q.next = s
s.prev = q              # then the neighbours' links
p.next = q

Order matters in one way only: read p.next (into s) before overwriting it. After that, the four assignments can happen in any order.

4.4Deletion: two links

To remove node p, make its neighbours point at each other:

Delete node p
p.prev.next = p.next
p.next.prev = p.prev
# nothing in the list refers to p any more; return p.element

Because of the sentinels, p.prev and p.next always exist, even when p is the only real node.

Doubly linked list, one link at a time

Pick an operation and step through it. The link being set on each step is drawn heavier; dashed links are about to be replaced.

4.5Costs, and the deque

Doubly linked list operationCost
insert or delete at either endO(1)
insert or delete next to a node you already have a reference toO(1)
find a node by index or by valueO(n)
spaceO(n)

That makes it the natural home for a deque: add_first and delete_first work just after the header, add_last and delete_last just before the trailer, all in O(1) worst case. The array-based deque achieves O(1) only amortized, because of occasional resizing.

Python
class _DoublyLinkedBase:
    """Doubly linked list with header and trailer sentinels."""

    class _Node:
        __slots__ = "element", "prev", "next"
        def __init__(self, element, prev, nxt):
            self.element, self.prev, self.next = element, prev, nxt

    def __init__(self):
        self._header = self._Node(None, None, None)    # sentinels hold no element
        self._trailer = self._Node(None, None, None)
        self._header.next = self._trailer
        self._trailer.prev = self._header
        self._size = 0

    def __len__(self):
        return self._size

    def is_empty(self):
        return self._size == 0

    def _insert_between(self, e, predecessor, successor):     # O(1)
        newest = self._Node(e, predecessor, successor)       # new node's two links
        predecessor.next = newest                            # neighbours' two links
        successor.prev = newest
        self._size += 1
        return newest

    def _delete_node(self, node):                            # O(1)
        predecessor, successor = node.prev, node.next
        predecessor.next = successor                         # link around node
        successor.prev = predecessor
        self._size -= 1
        element = node.element
        node.prev = node.next = node.element = None          # mark node as dead
        return element

class LinkedDeque(_DoublyLinkedBase):
    def first(self):
        if self.is_empty(): raise IndexError("deque is empty")
        return self._header.next.element
    def last(self):
        if self.is_empty(): raise IndexError("deque is empty")
        return self._trailer.prev.element
    def add_first(self, e):
        self._insert_between(e, self._header, self._header.next)
    def add_last(self, e):
        self._insert_between(e, self._trailer.prev, self._trailer)
    def delete_first(self):
        if self.is_empty(): raise IndexError("deque is empty")
        return self._delete_node(self._header.next)
    def delete_last(self):
        if self.is_empty(): raise IndexError("deque is empty")
        return self._delete_node(self._trailer.prev)

D = LinkedDeque()
D.add_last(5); D.add_first(3); D.add_first(7)
print(D.first(), D.last(), len(D))
print(D.delete_last(), D.delete_first(), D.first(), len(D))
7 5 3
5 7 3 1

_delete_node sets the dead node's fields to None. That helps the garbage collector, and Part 5 uses it to detect positions that have been deleted.

Check yourself: doubly linked lists

  1. How many link assignments does inserting into an empty doubly linked list take? Which nodes are involved?
  2. In the insertion code, what goes wrong if you write p.next = q as the very first line?
  3. Why is delete_last O(1) here but O(n) in a singly linked list?
Show answers
  1. Four, exactly as always: q.prev = header, q.next = trailer, trailer.prev = q, header.next = q. That is the point of sentinels.
  2. You lose the successor: q.next = p.next would then make q point to itself, and the old successor s is no longer reachable from p. Save s = p.next first.
  3. The last node is trailer.prev and its predecessor is trailer.prev.prev: both one step away. A singly linked list has no prev links, so it must walk from the head.

Part 5

The positional list

5.1The problem: insert and leave anywhere

The clinic's waiting room, modified once more: a person may be inserted at any position, and anyone may leave from any position. A doubly linked list can do both in O(1), but only if the user can say where. How?

  • By index? Indexes shift every time someone ahead of you arrives or leaves, so an index recorded earlier may now name a different person. And finding index i in a linked list is an O(n) walk anyway.
  • By handing out nodes? Fast, but dangerous: see 5.3.

The answer is a third option, a position: a marker that stays attached to one element. Think of the cursor in a text editor. It sits between two particular characters, and when you type elsewhere it does not jump; it stays with “its” characters.

5.2The Positional List ADT

A position is a simple object supporting exactly one method:

p.element(): return the element stored at position p.

A position is unaffected by changes elsewhere in the list. The only way it becomes invalid is an explicit command to delete it.

Accessors of a positional list L:

MethodReturns
L.first(), L.last()position of the first / last element, or None if L is empty
L.before(p), L.after(p)position just before / after p, or None if p is first / last
L.is_empty(), len(L)as usual
iter(L)a forward iterator over the elements

Updates:

MethodEffectReturns
L.add_first(e), L.add_last(e)insert at the front / backnew position
L.add_before(p, e), L.add_after(p, e)insert just before / after pnew position
L.replace(p, e)put e at pthe old element
L.delete(p)remove the element at p; p becomes invalidthe removed element

Slide correction. The slide says replace(p, e) returns “the position of the new element”. It returns the element formerly at p, as the slide's own example table shows (L.replace(p, 7) returns 8).

The lecture's example. Position objects are named p, q, r, s; list contents show which position holds each value.

OperationReturnsL
L.add_last(8)p8p
L.first()p8p
L.add_after(p, 5)q8p, 5q
L.before(q)p8p, 5q
L.add_before(q, 3)r8p, 3r, 5q
r.element()38p, 3r, 5q
L.after(p)r8p, 3r, 5q
L.before(p)None8p, 3r, 5q
L.add_first(9)s9s, 8p, 3r, 5q
L.delete(L.last())59s, 8p, 3r
L.replace(p, 7)89s, 7p, 3r

Notice that after add_first(9), p still refers to 8 even though 8 is now second. That is what “unaffected by changes elsewhere” means.

5.3Why not hand out node references?

  • Simplicity for users. They should not have to manipulate prev/next links or know that sentinels exist.
  • Robustness. A user holding a raw node could change its links and corrupt the whole list.
  • Safety checks. A position records which list it belongs to, so the list can reject a position from a different list, or one that has been deleted.
  • Freedom to redesign. Because users only see positions, the implementation could later switch to an array-based sequence without changing any user code.

5.4Implementation

A Position wraps a node and remembers its container. Every public method validates the position it is given (correct type, belongs to this list, not deleted) and converts it to a node; every method that returns a location converts a node back into a position, returning None instead of ever exposing a sentinel. It builds on _DoublyLinkedBase from Part 4.

Python
class PositionalList(_DoublyLinkedBase):

    class Position:
        """A marker for one element's location. Users never see nodes."""
        __slots__ = "_container", "_node"
        def __init__(self, container, node):
            self._container, self._node = container, node
        def element(self):
            return self._node.element
        def __eq__(self, other):                  # same node = same position
            return type(other) is type(self) and other._node is self._node
        def __ne__(self, other):
            return not (self == other)

    def _validate(self, p):                       # position -> node, with checks
        if not isinstance(p, self.Position):
            raise TypeError("p must be a proper Position type")
        if p._container is not self:
            raise ValueError("p does not belong to this container")
        if p._node.next is None:                  # _delete_node cleared it
            raise ValueError("p is no longer valid")
        return p._node

    def _make_position(self, node):               # node -> position (never a sentinel)
        if node is self._header or node is self._trailer:
            return None
        return self.Position(self, node)

    # accessors
    def first(self):     return self._make_position(self._header.next)
    def last(self):      return self._make_position(self._trailer.prev)
    def before(self, p): return self._make_position(self._validate(p).prev)
    def after(self, p):  return self._make_position(self._validate(p).next)
    def __iter__(self):
        cursor = self.first()
        while cursor is not None:
            yield cursor.element()
            cursor = self.after(cursor)

    # updates (all O(1))
    def _insert_between(self, e, predecessor, successor):
        return self._make_position(super()._insert_between(e, predecessor, successor))
    def add_first(self, e):     return self._insert_between(e, self._header, self._header.next)
    def add_last(self, e):      return self._insert_between(e, self._trailer.prev, self._trailer)
    def add_before(self, p, e):
        node = self._validate(p)
        return self._insert_between(e, node.prev, node)
    def add_after(self, p, e):
        node = self._validate(p)
        return self._insert_between(e, node, node.next)
    def delete(self, p):
        return self._delete_node(self._validate(p))
    def replace(self, p, e):
        node = self._validate(p)
        old = node.element
        node.element = e
        return old

L = PositionalList()
p = L.add_last(8);         print("add_last(8)      ", list(L))
print("first() == p     ", L.first() == p)
q = L.add_after(p, 5);     print("add_after(p, 5)  ", list(L))
print("before(q) == p   ", L.before(q) == p)
r = L.add_before(q, 3);    print("add_before(q, 3) ", list(L))
print("r.element()      ", r.element())
print("after(p) == r    ", L.after(p) == r)
print("before(p)        ", L.before(p))
s = L.add_first(9);        print("add_first(9)     ", list(L))
print("delete(last())   ", L.delete(L.last()), list(L))
print("replace(p, 7)    ", L.replace(p, 7), list(L))
try:
    L.after(q)                                    # q was deleted above
except ValueError as err:
    print("after(q)          ValueError:", err)
add_last(8)       [8]
first() == p      True
add_after(p, 5)   [8, 5]
before(q) == p    True
add_before(q, 3)  [8, 3, 5]
r.element()       3
after(p) == r     True
before(p)         None
add_first(9)      [9, 8, 3, 5]
delete(last())    5 [9, 8, 3]
replace(p, 7)     8 [9, 7, 3]
after(q)          ValueError: p is no longer valid

Every operation touches only p and its immediate neighbours, so all of them are O(1). Two positions compare equal when they wrap the same node, which is why L.first() == p is True even though first() builds a fresh Position object.

A text editor on a positional list

Each character is an element of a positional list, and the cursor is a position (the character just before it, or none at the start). Typing is add_after(cursor, ch); backspace is delete(cursor). The counter compares the links touched with the elements an array would have to shift.

Check yourself: positions

  1. L = [4a, 6b]. Run: c = L.add_after(a, 5); L.delete(b); d = L.add_last(7); L.replace(a, 1). Give the list and the return value of replace.
  2. After L.delete(b), what should L.after(b) do?
  3. Why does before(p) return None for the first position instead of the header?
Show answers
  1. [4a, 5c, 6b] → [4a, 5c] → [4a, 5c, 7d] → [1a, 5c, 7d]. replace returns 4.
  2. Raise an error: b is no longer a valid position. The implementation detects this because deleting cleared the node's links.
  3. Sentinels are an implementation detail. Exposing them would let users read or replace a “non-element”, and would break the promise that a position always holds an element.

Part 6

Insertion sort on a positional list

6.1The idea

Insertion sort is how most people sort a hand of cards: keep the left part sorted, take the next card, and slide it left until it fits. On a positional list, three positions drive it:

marker: the rightmost position of the currently sorted portion.

pivot: the position just after the marker, holding the next element to place.

walk: moves leftward from the marker until it finds where the pivot's element belongs.

M = marker, P = pivot, W = walk. The lecture's snapshot. The sorted portion is 15 … 36 (marker at 36). The pivot holds 23. Walk moves left from 36 past 29 and 25, stopping at 25 because 22 < 23; the 23 is then deleted and re-inserted before 25.

6.2The algorithm

  1. If the pivot's value is greater than the marker's, it is already in place: advance the marker to the pivot.
  2. Otherwise, set walk to the marker and move it left while the element before walk is greater than the pivot's value.
  3. Delete the pivot and insert its value before walk. The marker stays where it is (it is still the end of the sorted part).
  4. Repeat until the marker is the last position.
Python
def insertion_sort(L):
    """Sort a PositionalList of comparable elements into nondecreasing order."""
    if len(L) > 1:
        marker = L.first()                   # end of the sorted prefix
        while marker != L.last():
            pivot = L.after(marker)          # next element to place
            value = pivot.element()
            if value > marker.element():     # already in the right place
                marker = pivot               # the sorted prefix grows by one
            else:
                walk = marker                # walk left to find the spot
                while walk != L.first() and L.before(walk).element() > value:
                    walk = L.before(walk)
                L.delete(pivot)
                L.add_before(walk, value)    # O(1) re-link, no shifting

L = PositionalList()
for x in [15, 22, 25, 29, 36, 23, 53, 11, 42]:
    L.add_last(x)
insertion_sort(L)
print(list(L))
[11, 15, 22, 23, 25, 29, 36, 42, 53]

Cost. Moving the pivot is O(1): delete and add_before relink nodes, with no shifting. But walking left can pass the whole sorted portion, so the worst case (a reverse-sorted list) is 1 + 2 + … + (n − 1) comparisons: O(n²). On an already sorted list, step 1 always applies and it runs in O(n).

Insertion sort, step by step

Each step is one comparison or one move. The sorted portion is underlined; M, P and W mark marker, pivot and walk.

Check yourself: insertion sort

  1. How many comparisons does insertion sort make on [4, 3, 2, 1]?
  2. An array-based insertion sort also makes O(n²) comparisons. What does the positional version save?
Show answers
  1. 6, counting as the code and the demo do. Each of the three pivots (3, 2, 1) is first compared with the marker: 3 comparisons. Then walk moves left comparing with its left neighbour, stopping without a comparison when it reaches the first position: 0 for the 3, 1 for the 2, 2 for the 1. Total 3 + 3 = 6. On a reversed list of length n this grows like n²/2.
  2. Element moves: the array version shifts every element between walk and pivot one slot right (O(n) writes per placement); the positional version relinks a single node in O(1). The walk itself is still O(n).

Part 7

The favorites list

7.1The problem

Maintain a collection of elements while counting how many times each is accessed, so you can report the most popular ones: a browser's most-visited sites, a music app's most-played songs.

MethodWhat it does
access(e)access e: add 1 to its count, adding it to the list if new
remove(e)remove e if present
top(k)iterate over the k most-accessed elements
len(), is_empty()as usual

7.2Design 1: keep it sorted by count

Store the elements in a positional list in non-increasing order of access count.

  • access(e): search for e (O(n)), add 1 to its count, then move it backward (towards the front) past any elements that now have a smaller count.
  • remove(e): search, then delete the position.
  • top(k): the first k entries. O(k).

The lecture's bad case: element 1 is accessed n times, then element 2 is accessed n times, …, then element n is accessed n times. When element k is being accessed, elements 1 … k − 1 are all ahead of it with count n, at least as large as its own, so it never moves forward. Every one of its n accesses searches past those k − 1 elements: about kn steps. Summing over k:

n + 2n + 3n + … + n·n = n · n(n + 1)2 = O(n³)

7.3Design 2: the move-to-front heuristic

Many access sequences have locality of reference: something used recently is likely to be used again soon. Exploit it: order the list by recency, and on every access move the element to the front.

  • access(e): search, add 1, move to the front. The move is O(1) (delete + add_first).

Same bad case: the first access to element k searches the whole list, O(n). After that, element k is at the front, so its remaining n − 1 accesses are O(1) each. Per element: O(n) + (n − 1)·O(1) = O(n). Over n elements: O(n²), a factor of n better.

A heuristic is a rule that usually helps but carries no guarantee. Move-to-front can do badly too: cycling through all n elements in the same order over and over always finds the needed element at the back, O(n) per access.

7.4The price: top(k)

With move-to-front the list is no longer sorted by count, so top(k) has to work:

  1. Copy all entries into a temporary list. O(n).
  2. Find the entry with the largest count and remove it from the temporary list. O(n). Repeat k times.
  3. Report the k entries removed.

Total O(kn). For a constant k, such as a top-10 list, that is O(n). For k proportional to n, such as the top 25%, it is O(n²). (Sorting the copy instead would give O(n log n), a later topic.)

Python
class FavoritesList:
    """Elements kept in a PositionalList sorted by access count, highest first."""

    class _Item:
        __slots__ = "_value", "_count"
        def __init__(self, e):
            self._value, self._count = e, 0

    def __init__(self):
        self._data = PositionalList()

    def _find_position(self, e):                  # O(n) linear search
        walk = self._data.first()
        while walk is not None and walk.element()._value != e:
            walk = self._data.after(walk)
        return walk

    def _move_up(self, p):                        # restore the sorted order
        if p != self._data.first():
            cnt = p.element()._count
            walk = self._data.before(p)
            if cnt > walk.element()._count:       # out of order: walk backwards
                while (walk != self._data.first() and
                       cnt > self._data.before(walk).element()._count):
                    walk = self._data.before(walk)
                self._data.add_before(walk, self._data.delete(p))

    def access(self, e):
        p = self._find_position(e)
        if p is None:
            p = self._data.add_last(self._Item(e))   # new elements start at the end
        p.element()._count += 1
        self._move_up(p)

    def remove(self, e):
        p = self._find_position(e)
        if p is not None:
            self._data.delete(p)

    def top(self, k):                             # O(k): the first k are the answer
        walk = self._data.first()
        for _ in range(k):
            item = walk.element()
            yield item._value, item._count
            walk = self._data.after(walk)

class FavoritesListMTF(FavoritesList):
    """Same ADT, ordered by recency: every access moves the element to the front."""

    def _move_up(self, p):                        # O(1)
        if p != self._data.first():
            self._data.add_first(self._data.delete(p))

    def top(self, k):                             # O(kn): order no longer helps
        temp = PositionalList()
        for item in self._data:                   # copy every item: O(n)
            temp.add_last(item)
        for _ in range(k):                        # k times: find and remove the max
            high = temp.first()
            walk = temp.after(high)
            while walk is not None:
                if walk.element()._count > high.element()._count:
                    high = walk
                walk = temp.after(walk)
            yield high.element()._value, high.element()._count
            temp.delete(high)

Measured on the bad case, counting nodes walked past:

Python
# FavoritesList     keeps items sorted by access count (highest first).
# FavoritesListMTF  moves each accessed item to the front instead.
# Both count .steps = how many nodes their searches walk past.

for n in [10, 20, 40, 80]:
    A, B = FavoritesList(), FavoritesListMTF()
    for e in range(1, n + 1):        # element 1 accessed n times, then element 2 ...
        for _ in range(n):
            A.access(e)
            B.access(e)
    print(f"n = {n:2}   sorted list: {A.steps:7,} steps   move-to-front: {B.steps:5,} steps")

F = FavoritesListMTF()
for site in "ACABACAAB":
    F.access(site)
F.access("D")
print("list order (most recent first):", [item._value for item in F._data])
print("top(2):", list(F.top(2)))
n = 10   sorted list:     450 steps   move-to-front:    45 steps
n = 20   sorted list:   3,800 steps   move-to-front:   190 steps
n = 40   sorted list:  31,200 steps   move-to-front:   780 steps
n = 80   sorted list: 252,800 steps   move-to-front: 3,160 steps
list order (most recent first): ['D', 'B', 'A', 'C']
top(2): [('A', 5), ('B', 2)]

Each doubling of n multiplies the sorted list's steps by about 8 (cubic) and move-to-front's by about 4 (quadratic). The last two lines show recency order versus count order: D was accessed last so it is first in the list, yet top(2) still reports A and B by count.

Sorted versus move-to-front

Choose an access pattern and the number of distinct elements. The chart shows cumulative search-and-move steps for both designs over the whole sequence.

Check yourself: favorites

  1. Access sequence A, B, A, C, A, B on a move-to-front list. Give the list order after each access, and the counts at the end.
  2. Give an access pattern where move-to-front is no better than the sorted list.
  3. A music app shows a top-10 chart and has n = 100,000 songs. Is O(kn) for top(k) acceptable?
Show answers
  1. [A] → [B, A] → [A, B] → [C, A, B] → [A, C, B] → [B, A, C]. Counts: A 3, B 2, C 1.
  2. Cycling through all elements repeatedly in the same order (A, B, C, …, A, B, C, …): each access finds the element at the very back, O(n) every time.
  3. Yes: k = 10 is a constant, so it is O(n), about a million steps, fast. It would not be for “top 25%”.

Part 8

Linked lists on one page

8.1Arrays versus linked lists

FeatureArray-based sequenceLink-based sequence
access by indexO(1)O(n): walk from an end
insert / deleteO(n): shiftingO(1) given the location (e.g. a text-editor cursor)
memory overheadless per element, but capacity exceeds sizeextra references in every node
cache performancegood: contiguous memorypoor: nodes scattered across memory
worst case of end operationsamortized O(1) (occasional resize)O(1) every time

Cache, briefly: the processor keeps a small, very fast copy of recently used memory, and loads it in chunks of neighbouring bytes. Walking an array uses every byte of each chunk; walking a linked list jumps to a new, unrelated address at every node, so it waits on slow main memory far more often. Two algorithms with the same O(·) can differ several-fold in practice because of this.

Beyond the lecture: what “extra memory” means in Python

Measured on this machine, for 1,000 elements (the element objects themselves are not counted in any line):

Python
import sys

class Node:
    __slots__ = "element", "next"
    def __init__(self, element, nxt):
        self.element, self.next = element, nxt

class NodeNoSlots:
    def __init__(self, element, nxt):
        self.element, self.next = element, nxt

n = 1000
array_bytes = sys.getsizeof([None] * n)
slot_nodes = sum(sys.getsizeof(Node(None, None)) for _ in range(n))
plain = NodeNoSlots(None, None)
dict_nodes = n * (sys.getsizeof(plain) + sys.getsizeof(plain.__dict__))
print(f"list of {n} references : {array_bytes:6,} bytes")
print(f"{n} nodes with __slots__: {slot_nodes:6,} bytes")
print(f"{n} nodes without       : {dict_nodes:6,} bytes")
list of 1000 references :  8,056 bytes
1000 nodes with __slots__: 48,000 bytes
1000 nodes without       : 344,000 bytes

A list stores one 8-byte reference per element. A node with __slots__ is a whole Python object, 48 bytes; without __slots__ each node also carries its own attribute dictionary. That is why every node class on this page declares __slots__.

8.2Summary of costs

StructureEndsMiddle (given a position)Find
singly linkedadd head/tail O(1); remove head O(1); remove tail O(n)insert after O(1); delete needs predecessorO(n)
circularly linkedrotate O(1); front O(1)as singly linkedO(n)
doubly linked (sentinels)all O(1)insert / delete O(1)O(n)
positional listall O(1)all O(1)O(n)
favorites, sortedaccess O(n); top(k) O(k)bad case O(n³) total
favorites, move-to-frontaccess O(n), O(1) for repeats; top(k) O(kn)bad case O(n²) total

Ideas to carry forward

  1. Order of link updates matters: save any reference you are about to overwrite.
  2. Sentinels remove special cases. Two dummy nodes buy you code with no empty-list branches.
  3. O(1) insertion is “given the location”. Finding the location is a separate O(n) cost.
  4. Abstraction protects structure. Positions let users hold a place without touching nodes.
  5. How you order data changes total cost: sorted by count vs by recency differ by a factor of n on the same input.

Final mixed check

  1. You need a queue whose every operation must finish in bounded time. Array or linked? Why?
  2. Which single operation makes a singly linked list a poor choice for a deque?
  3. A doubly linked list has 10 elements. Inserting after the 7th element takes how many link assignments, and how many steps to reach the 7th from the header?
  4. Why is replace(p, e) O(1) on a positional list but replace by value O(n)?
Show answers
  1. Linked: every operation is O(1) worst case. The array version is O(1) only amortized; a resize can take O(n).
  2. delete_last (remove at tail), which is O(n).
  3. Four link assignments. Reaching the 7th node takes 7 steps from the header (or 4 from the trailer, since you can walk backwards). The splice is O(1); the walk is not.
  4. With a position you already hold the node. Replacing by value first has to search for it.