std/sys/sync/rwlock/
queue.rs

1//! Efficient read-write locking without `pthread_rwlock_t`.
2//!
3//! The readers-writer lock provided by the `pthread` library has a number of problems which make it
4//! a suboptimal choice for `std`:
5//!
6//! * It is non-movable, so it needs to be allocated (lazily, to make the constructor `const`).
7//! * `pthread` is an external library, meaning the fast path of acquiring an uncontended lock
8//! cannot be inlined.
9//! * Some platforms (at least glibc before version 2.25) have buggy implementations that can easily
10//! lead to undefined behaviour in safe Rust code when not properly guarded against.
11//! * On some platforms (e.g. macOS), the lock is very slow.
12//!
13//! Therefore, we implement our own [`RwLock`]! Naively, one might reach for a spinlock, but those
14//! can be quite [problematic] when the lock is contended.
15//!
16//! Instead, this [`RwLock`] copies its implementation strategy from the Windows [SRWLOCK] and the
17//! [usync] library implementations.
18//!
19//! Spinning is still used for the fast path, but it is bounded: after spinning fails, threads will
20//! locklessly add an information structure ([`Node`]) containing a [`Thread`] handle into a queue
21//! of waiters associated with the lock. The lock owner, upon releasing the lock, will scan through
22//! the queue and wake up threads as appropriate, and the newly-awoken threads will then try to
23//! acquire the lock themselves.
24//!
25//! The resulting [`RwLock`] is:
26//!
27//! * adaptive, since it spins before doing any heavyweight parking operations
28//! * allocation-free, modulo the per-thread [`Thread`] handle, which is allocated anyways when
29//! using threads created by `std`
30//! * writer-preferring, even if some readers may still slip through
31//! * unfair, which reduces context-switching and thus drastically improves performance
32//!
33//! and also quite fast in most cases.
34//!
35//! [problematic]: https://matklad.github.io/2020/01/02/spinlocks-considered-harmful.html
36//! [SRWLOCK]: https://learn.microsoft.com/en-us/windows/win32/sync/slim-reader-writer--srw--locks
37//! [usync]: https://crates.io/crates/usync
38//!
39//! # Implementation
40//!
41//! ## State
42//!
43//! A single [`AtomicPtr`] is used as state variable. The lowest four bits are used to indicate the
44//! meaning of the remaining bits:
45//!
46//! | [`LOCKED`] | [`QUEUED`] | [`QUEUE_LOCKED`] | [`DOWNGRADED`] | Remaining    |                                                                                                                             |
47//! |------------|:-----------|:-----------------|:---------------|:-------------|:----------------------------------------------------------------------------------------------------------------------------|
48//! | 0          | 0          | 0                | 0              | 0            | The lock is unlocked, no threads are waiting                                                                                |
49//! | 1          | 0          | 0                | 0              | 0            | The lock is write-locked, no threads waiting                                                                                |
50//! | 1          | 0          | 0                | 0              | n > 0        | The lock is read-locked with n readers                                                                                      |
51//! | 0          | 1          | *                | 0              | `*mut Node`  | The lock is unlocked, but some threads are waiting. Only writers may lock the lock                                          |
52//! | 1          | 1          | *                | *              | `*mut Node`  | The lock is locked, but some threads are waiting. If the lock is read-locked, the last queue node contains the reader count |
53//!
54//! ## Waiter Queue
55//!
56//! When threads are waiting on the lock (the `QUEUE` bit is set), the lock state points to a queue
57//! of waiters, which is implemented as a linked list of nodes stored on the stack to avoid memory
58//! allocation.
59//!
60//! To enable lock-free enqueuing of new nodes to the queue, the linked list is singly-linked upon
61//! creation.
62//!
63//! When the lock is read-locked, the lock count (number of readers) is stored in the last link of
64//! the queue. Threads have to traverse the queue to find the last element upon releasing the lock.
65//! To avoid having to traverse the entire list every time we want to access the reader count, a
66//! pointer to the found tail is cached in the (current) first element of the queue.
67//!
68//! Also, while the lock is unfair for performance reasons, it is still best to wake the tail node
69//! first (FIFO ordering). Since we always pop nodes off the tail of the queue, we must store
70//! backlinks to previous nodes so that we can update the `tail` field of the (current) first
71//! element of the queue. Adding backlinks is done at the same time as finding the tail (via the
72//! function [`find_tail_and_add_backlinks`]), and thus encountering a set tail field on a node
73//! indicates that all following nodes in the queue are initialized.
74//!
75//! TLDR: Here's a diagram of what the queue looks like:
76//!
77//! ```text
78//! state
79//!   │
80//!   ▼
81//! ╭───────╮ next ╭───────╮ next ╭───────╮ next ╭───────╮
82//! │       ├─────►│       ├─────►│       ├─────►│ count │
83//! │       │      │       │      │       │      │       │
84//! │       │      │       │◄─────┤       │◄─────┤       │
85//! ╰───────╯      ╰───────╯ prev ╰───────╯ prev ╰───────╯
86//!                      │                           ▲
87//!                      └───────────────────────────┘
88//!                                  tail
89//! ```
90//!
91//! Invariants:
92//! 1. At least one node must contain a non-null, current `tail` field.
93//! 2. The first non-null `tail` field must be valid and current.
94//! 3. All nodes preceding this node must have a correct, non-null `next` field.
95//! 4. All nodes following this node must have a correct, non-null `prev` field.
96//!
97//! Access to the queue is controlled by the `QUEUE_LOCKED` bit. Threads will try to set this bit
98//! in two cases: one is when a thread enqueues itself and eagerly adds backlinks to the queue
99//! (which drastically improves performance), and the other is after a thread unlocks the lock to
100//! wake up the next waiter(s).
101//!
102//! `QUEUE_LOCKED` is set atomically at the same time as the enqueuing/unlocking operations. The
103//! thread releasing the `QUEUE_LOCKED` bit will check the state of the lock (in particular, whether
104//! a downgrade was requested using the [`DOWNGRADED`] bit) and wake up waiters as appropriate. This
105//! guarantees forward progress even if the unlocking thread could not acquire the queue lock.
106//!
107//! ## Memory Orderings
108//!
109//! To properly synchronize changes to the data protected by the lock, the lock is acquired and
110//! released with [`Acquire`] and [`Release`] ordering, respectively. To propagate the
111//! initialization of nodes, changes to the queue lock are also performed using these orderings.
112
113#![forbid(unsafe_op_in_unsafe_fn)]
114
115use crate::cell::OnceCell;
116use crate::hint::spin_loop;
117use crate::mem;
118use crate::ptr::{self, NonNull, null_mut, without_provenance_mut};
119use crate::sync::atomic::Ordering::{AcqRel, Acquire, Relaxed, Release};
120use crate::sync::atomic::{Atomic, AtomicBool, AtomicPtr};
121use crate::thread::{self, Thread};
122
123/// The atomic lock state.
124type AtomicState = Atomic<State>;
125/// The inner lock state.
126type State = *mut ();
127
128const UNLOCKED: State = without_provenance_mut(0);
129const LOCKED: usize = 1 << 0;
130const QUEUED: usize = 1 << 1;
131const QUEUE_LOCKED: usize = 1 << 2;
132const DOWNGRADED: usize = 1 << 3;
133const SINGLE: usize = 1 << 4;
134const STATE: usize = DOWNGRADED | QUEUE_LOCKED | QUEUED | LOCKED;
135const NODE_MASK: usize = !STATE;
136
137/// Locking uses exponential backoff. `SPIN_COUNT` indicates how many times the locking operation
138/// will be retried.
139///
140/// In other words, `spin_loop` will be called `2.pow(SPIN_COUNT) - 1` times.
141const SPIN_COUNT: usize = 7;
142
143/// Marks the state as write-locked, if possible.
144#[inline]
145fn write_lock(state: State) -> Option<State> {
146    if state.addr() & LOCKED == 0 { Some(state.map_addr(|addr| addr | LOCKED)) } else { None }
147}
148
149/// Marks the state as read-locked, if possible.
150#[inline]
151fn read_lock(state: State) -> Option<State> {
152    if state.addr() & QUEUED == 0 && state.addr() != LOCKED {
153        Some(without_provenance_mut(state.addr().checked_add(SINGLE)? | LOCKED))
154    } else {
155        None
156    }
157}
158
159/// Converts a `State` into a `Node` by masking out the bottom bits of the state, assuming that the
160/// state points to a queue node.
161///
162/// # Safety
163///
164/// The state must contain a valid pointer to a queue node.
165#[inline]
166unsafe fn to_node(state: State) -> NonNull<Node> {
167    unsafe { NonNull::new_unchecked(state.mask(NODE_MASK)).cast() }
168}
169
170/// The representation of a thread waiting on the lock queue.
171///
172/// We initialize these `Node`s on thread execution stacks to avoid allocation.
173///
174/// Note that we need an alignment of 16 to ensure that the last 4 bits of any
175/// pointers to `Node`s are always zeroed (for the bit flags described in the
176/// module-level documentation).
177#[repr(align(16))]
178struct Node {
179    next: AtomicLink,
180    prev: AtomicLink,
181    tail: AtomicLink,
182    write: bool,
183    thread: OnceCell<Thread>,
184    completed: Atomic<bool>,
185}
186
187/// An atomic node pointer with relaxed operations.
188struct AtomicLink(Atomic<*mut Node>);
189
190impl AtomicLink {
191    fn new(v: Option<NonNull<Node>>) -> AtomicLink {
192        AtomicLink(AtomicPtr::new(v.map_or(null_mut(), NonNull::as_ptr)))
193    }
194
195    fn get(&self) -> Option<NonNull<Node>> {
196        NonNull::new(self.0.load(Relaxed))
197    }
198
199    fn set(&self, v: Option<NonNull<Node>>) {
200        self.0.store(v.map_or(null_mut(), NonNull::as_ptr), Relaxed);
201    }
202}
203
204impl Node {
205    /// Creates a new queue node.
206    fn new(write: bool) -> Node {
207        Node {
208            next: AtomicLink::new(None),
209            prev: AtomicLink::new(None),
210            tail: AtomicLink::new(None),
211            write,
212            thread: OnceCell::new(),
213            completed: AtomicBool::new(false),
214        }
215    }
216
217    /// Prepare this node for waiting.
218    fn prepare(&mut self) {
219        // Fall back to creating an unnamed `Thread` handle to allow locking in TLS destructors.
220        self.thread.get_or_init(thread::current_or_unnamed);
221        self.completed = AtomicBool::new(false);
222    }
223
224    /// Wait until this node is marked as [`complete`](Node::complete)d by another thread.
225    ///
226    /// # Safety
227    ///
228    /// May only be called from the thread that created the node.
229    unsafe fn wait(&self) {
230        while !self.completed.load(Acquire) {
231            unsafe {
232                self.thread.get().unwrap().park();
233            }
234        }
235    }
236
237    /// Atomically mark this node as completed.
238    ///
239    /// # Safety
240    ///
241    /// `node` must point to a valid `Node`, and the node may not outlive this call.
242    unsafe fn complete(node: NonNull<Node>) {
243        // Since the node may be destroyed immediately after the completed flag is set, clone the
244        // thread handle before that.
245        let thread = unsafe { node.as_ref().thread.get().unwrap().clone() };
246        unsafe {
247            node.as_ref().completed.store(true, Release);
248        }
249        thread.unpark();
250    }
251}
252
253/// Traverse the queue and find the tail, adding backlinks to the queue while traversing.
254///
255/// This may be called from multiple threads at the same time as long as the queue is not being
256/// modified (this happens when unlocking multiple readers).
257///
258/// # Safety
259///
260/// * `head` must point to a node in a valid queue.
261/// * `head` must be in front of the previous head node that was used to perform the last removal.
262/// * The part of the queue starting with `head` must not be modified during this call.
263unsafe fn find_tail_and_add_backlinks(head: NonNull<Node>) -> NonNull<Node> {
264    let mut current = head;
265
266    // Traverse the queue until we find a node that has a set `tail`.
267    let tail = loop {
268        let c = unsafe { current.as_ref() };
269        if let Some(tail) = c.tail.get() {
270            break tail;
271        }
272
273        // SAFETY: All `next` fields before the first node with a set `tail` are non-null and valid
274        // (by Invariant 3).
275        unsafe {
276            let next = c.next.get().unwrap_unchecked();
277            next.as_ref().prev.set(Some(current));
278            current = next;
279        }
280    };
281
282    unsafe {
283        head.as_ref().tail.set(Some(tail));
284        tail
285    }
286}
287
288/// [`complete`](Node::complete)s all threads in the queue ending with `tail`.
289///
290/// # Safety
291///
292/// * `tail` must be a valid tail of a fully linked queue.
293/// * The current thread must have exclusive access to that queue.
294unsafe fn complete_all(tail: NonNull<Node>) {
295    let mut current = tail;
296
297    // Traverse backwards through the queue (FIFO) and `complete` all of the nodes.
298    loop {
299        let prev = unsafe { current.as_ref().prev.get() };
300        unsafe {
301            Node::complete(current);
302        }
303        match prev {
304            Some(prev) => current = prev,
305            None => return,
306        }
307    }
308}
309
310/// A type to guard against the unwinds of stacks that nodes are located on due to panics.
311struct PanicGuard;
312
313impl Drop for PanicGuard {
314    fn drop(&mut self) {
315        rtabort!("tried to drop node in intrusive list.");
316    }
317}
318
319/// The public inner `RwLock` type.
320pub struct RwLock {
321    state: AtomicState,
322}
323
324impl RwLock {
325    #[inline]
326    pub const fn new() -> RwLock {
327        RwLock { state: AtomicPtr::new(UNLOCKED) }
328    }
329
330    #[inline]
331    pub fn try_read(&self) -> bool {
332        self.state.fetch_update(Acquire, Relaxed, read_lock).is_ok()
333    }
334
335    #[inline]
336    pub fn read(&self) {
337        if !self.try_read() {
338            self.lock_contended(false)
339        }
340    }
341
342    #[inline]
343    pub fn try_write(&self) -> bool {
344        // Atomically set the `LOCKED` bit. This is lowered to a single atomic instruction on most
345        // modern processors (e.g. "lock bts" on x86 and "ldseta" on modern AArch64), and therefore
346        // is more efficient than `fetch_update(lock(true))`, which can spuriously fail if a new
347        // node is appended to the queue.
348        self.state.fetch_or(LOCKED, Acquire).addr() & LOCKED == 0
349    }
350
351    #[inline]
352    pub fn write(&self) {
353        if !self.try_write() {
354            self.lock_contended(true)
355        }
356    }
357
358    #[cold]
359    fn lock_contended(&self, write: bool) {
360        let mut node = Node::new(write);
361        let mut state = self.state.load(Relaxed);
362        let mut count = 0;
363        let update_fn = if write { write_lock } else { read_lock };
364
365        loop {
366            // Optimistically update the state.
367            if let Some(next) = update_fn(state) {
368                // The lock is available, try locking it.
369                match self.state.compare_exchange_weak(state, next, Acquire, Relaxed) {
370                    Ok(_) => return,
371                    Err(new) => state = new,
372                }
373                continue;
374            } else if state.addr() & QUEUED == 0 && count < SPIN_COUNT {
375                // If the lock is not available and no threads are queued, optimistically spin for a
376                // while, using exponential backoff to decrease cache contention.
377                for _ in 0..(1 << count) {
378                    spin_loop();
379                }
380                state = self.state.load(Relaxed);
381                count += 1;
382                continue;
383            }
384            // The optimistic paths did not succeed, so fall back to parking the thread.
385
386            // First, prepare the node.
387            node.prepare();
388
389            // If there are threads queued, this will set the `next` field to be a pointer to the
390            // first node in the queue.
391            // If the state is read-locked, this will set `next` to the lock count.
392            // If it is write-locked, it will set `next` to zero.
393            node.next.0 = AtomicPtr::new(state.mask(NODE_MASK).cast());
394            node.prev = AtomicLink::new(None);
395
396            // Set the `QUEUED` bit and preserve the `LOCKED` and `DOWNGRADED` bit.
397            let mut next = ptr::from_ref(&node)
398                .map_addr(|addr| addr | QUEUED | (state.addr() & (DOWNGRADED | LOCKED)))
399                as State;
400
401            let mut is_queue_locked = false;
402            if state.addr() & QUEUED == 0 {
403                // If this is the first node in the queue, set the `tail` field to the node itself
404                // to ensure there is a valid `tail` field in the queue (Invariants 1 & 2).
405                // This needs to use `set` to avoid invalidating the new pointer.
406                node.tail.set(Some(NonNull::from(&node)));
407            } else {
408                // Otherwise, the tail of the queue is not known.
409                node.tail.set(None);
410
411                // Try locking the queue to eagerly add backlinks.
412                next = next.map_addr(|addr| addr | QUEUE_LOCKED);
413
414                // Track if we changed the `QUEUE_LOCKED` bit from off to on.
415                is_queue_locked = state.addr() & QUEUE_LOCKED == 0;
416            }
417
418            // Register the node, using release ordering to propagate our changes to the waking
419            // thread.
420            if let Err(new) = self.state.compare_exchange_weak(state, next, AcqRel, Relaxed) {
421                // The state has changed, just try again.
422                state = new;
423                continue;
424            }
425            // The node has been registered, so the structure must not be mutably accessed or
426            // destroyed while other threads may be accessing it.
427
428            // Guard against unwinds using a `PanicGuard` that aborts when dropped.
429            let guard = PanicGuard;
430
431            // If the current thread locked the queue, unlock it to eagerly adding backlinks.
432            if is_queue_locked {
433                // SAFETY: This thread set the `QUEUE_LOCKED` bit above.
434                unsafe {
435                    self.unlock_queue(next);
436                }
437            }
438
439            // Wait until the node is removed from the queue.
440            // SAFETY: the node was created by the current thread.
441            unsafe {
442                node.wait();
443            }
444
445            // The node was removed from the queue, disarm the guard.
446            mem::forget(guard);
447
448            // Reload the state and try again.
449            state = self.state.load(Relaxed);
450            count = 0;
451        }
452    }
453
454    #[inline]
455    pub unsafe fn read_unlock(&self) {
456        match self.state.fetch_update(Release, Acquire, |state| {
457            if state.addr() & QUEUED == 0 {
458                // If there are no threads queued, simply decrement the reader count.
459                let count = state.addr() - (SINGLE | LOCKED);
460                Some(if count > 0 { without_provenance_mut(count | LOCKED) } else { UNLOCKED })
461            } else if state.addr() & DOWNGRADED != 0 {
462                // This thread used to have exclusive access, but requested a downgrade. This has
463                // not been completed yet, so we still have exclusive access.
464                // Retract the downgrade request and unlock, but leave waking up new threads to the
465                // thread that already holds the queue lock.
466                Some(state.mask(!(DOWNGRADED | LOCKED)))
467            } else {
468                None
469            }
470        }) {
471            Ok(_) => {}
472            // There are waiters queued and the lock count was moved to the tail of the queue.
473            Err(state) => unsafe { self.read_unlock_contended(state) },
474        }
475    }
476
477    /// # Safety
478    ///
479    /// * There must be threads queued on the lock.
480    /// * `state` must be a pointer to a node in a valid queue.
481    /// * There cannot be a `downgrade` in progress.
482    #[cold]
483    unsafe fn read_unlock_contended(&self, state: State) {
484        // SAFETY:
485        // The state was observed with acquire ordering above, so the current thread will have
486        // observed all node initializations.
487        // We also know that no threads can be modifying the queue starting at `state`: because new
488        // read-locks cannot be acquired while there are any threads queued on the lock, all
489        // queue-lock owners will observe a set `LOCKED` bit in `self.state` and will not modify
490        // the queue. The other case that a thread could modify the queue is if a downgrade is in
491        // progress (removal of the entire queue), but since that is part of this function's safety
492        // contract, we can guarantee that no other threads can modify the queue.
493        let tail = unsafe { find_tail_and_add_backlinks(to_node(state)).as_ref() };
494
495        // The lock count is stored in the `next` field of `tail`.
496        // Decrement it, making sure to observe all changes made to the queue by the other lock
497        // owners by using acquire-release ordering.
498        let was_last = tail.next.0.fetch_byte_sub(SINGLE, AcqRel).addr() - SINGLE == 0;
499        if was_last {
500            // SAFETY: Other threads cannot read-lock while threads are queued. Also, the `LOCKED`
501            // bit is still set, so there are no writers. Thus the current thread exclusively owns
502            // this lock, even though it is a reader.
503            unsafe { self.unlock_contended(state) }
504        }
505    }
506
507    #[inline]
508    pub unsafe fn write_unlock(&self) {
509        if let Err(state) =
510            self.state.compare_exchange(without_provenance_mut(LOCKED), UNLOCKED, Release, Relaxed)
511        {
512            // SAFETY: Since other threads cannot acquire the lock, the state can only have changed
513            // because there are threads queued on the lock.
514            unsafe { self.unlock_contended(state) }
515        }
516    }
517
518    /// # Safety
519    ///
520    /// * The lock must be exclusively owned by this thread.
521    /// * There must be threads queued on the lock.
522    /// * `state` must be a pointer to a node in a valid queue.
523    /// * There cannot be a `downgrade` in progress.
524    #[cold]
525    unsafe fn unlock_contended(&self, state: State) {
526        debug_assert_eq!(state.addr() & (DOWNGRADED | QUEUED | LOCKED), QUEUED | LOCKED);
527
528        let mut current = state;
529
530        // We want to atomically release the lock and try to acquire the queue lock.
531        loop {
532            // First check if the queue lock is already held.
533            if current.addr() & QUEUE_LOCKED != 0 {
534                // Another thread holds the queue lock, so let them wake up waiters for us.
535                let next = current.mask(!LOCKED);
536                match self.state.compare_exchange_weak(current, next, Release, Relaxed) {
537                    Ok(_) => return,
538                    Err(new) => {
539                        current = new;
540                        continue;
541                    }
542                }
543            }
544
545            // Atomically release the lock and try to acquire the queue lock.
546            let next = current.map_addr(|addr| (addr & !LOCKED) | QUEUE_LOCKED);
547            match self.state.compare_exchange_weak(current, next, AcqRel, Relaxed) {
548                // Now that we have the queue lock, we can wake up the next waiter.
549                Ok(_) => {
550                    // SAFETY: This thread just acquired the queue lock, and this function's safety
551                    // contract requires that there are threads already queued on the lock.
552                    unsafe { self.unlock_queue(next) };
553                    return;
554                }
555                Err(new) => current = new,
556            }
557        }
558    }
559
560    /// # Safety
561    ///
562    /// * The lock must be write-locked by this thread.
563    #[inline]
564    pub unsafe fn downgrade(&self) {
565        // Optimistically change the state from write-locked with a single writer and no waiters to
566        // read-locked with a single reader and no waiters.
567        if let Err(state) = self.state.compare_exchange(
568            without_provenance_mut(LOCKED),
569            without_provenance_mut(SINGLE | LOCKED),
570            Release,
571            Relaxed,
572        ) {
573            // SAFETY: The only way the state can have changed is if there are threads queued.
574            // Wake all of them up.
575            unsafe { self.downgrade_slow(state) }
576        }
577    }
578
579    /// Downgrades the lock from write-locked to read-locked in the case that there are threads
580    /// waiting on the wait queue.
581    ///
582    /// This function will either wake up all of the waiters on the wait queue or designate the
583    /// current holder of the queue lock to wake up all of the waiters instead. Once the waiters
584    /// wake up, they will continue in the execution loop of `lock_contended`.
585    ///
586    /// # Safety
587    ///
588    /// * The lock must be write-locked by this thread.
589    /// * `state` must be a pointer to a node in a valid queue.
590    /// * There must be threads queued on the lock.
591    #[cold]
592    unsafe fn downgrade_slow(&self, mut state: State) {
593        debug_assert_eq!(state.addr() & (DOWNGRADED | QUEUED | LOCKED), QUEUED | LOCKED);
594
595        // Attempt to wake up all waiters by taking ownership of the entire waiter queue.
596        loop {
597            if state.addr() & QUEUE_LOCKED != 0 {
598                // Another thread already holds the queue lock. Tell it to wake up all waiters.
599                // If the other thread succeeds in waking up waiters before we release our lock, the
600                // effect will be just the same as if we had changed the state below.
601                // Otherwise, the `DOWNGRADED` bit will still be set, meaning that when this thread
602                // calls `read_unlock` later (because it holds a read lock and must unlock
603                // eventually), it will realize that the lock is still exclusively locked and act
604                // accordingly.
605                let next = state.map_addr(|addr| addr | DOWNGRADED);
606                match self.state.compare_exchange_weak(state, next, Release, Relaxed) {
607                    Ok(_) => return,
608                    Err(new) => state = new,
609                }
610            } else {
611                // Grab the entire queue by swapping the `state` with a single reader.
612                let next = ptr::without_provenance_mut(SINGLE | LOCKED);
613                if let Err(new) = self.state.compare_exchange_weak(state, next, AcqRel, Relaxed) {
614                    state = new;
615                    continue;
616                }
617
618                // SAFETY: We have full ownership of this queue now, so nobody else can modify it.
619                let tail = unsafe { find_tail_and_add_backlinks(to_node(state)) };
620
621                // Wake up all waiters.
622                // SAFETY: `tail` was just computed, meaning the whole queue is linked, and we have
623                // full ownership of the queue, so we have exclusive access.
624                unsafe { complete_all(tail) };
625
626                return;
627            }
628        }
629    }
630
631    /// Unlocks the queue. Wakes up all threads if a downgrade was requested, otherwise wakes up the
632    /// next eligible thread(s) if the lock is unlocked.
633    ///
634    /// # Safety
635    ///
636    /// * The queue lock must be held by the current thread.
637    /// * `state` must be a pointer to a node in a valid queue.
638    /// * There must be threads queued on the lock.
639    unsafe fn unlock_queue(&self, mut state: State) {
640        debug_assert_eq!(state.addr() & (QUEUED | QUEUE_LOCKED), QUEUED | QUEUE_LOCKED);
641
642        loop {
643            // SAFETY: Since we have the queue lock, nobody else can be modifying the queue.
644            let tail = unsafe { find_tail_and_add_backlinks(to_node(state)) };
645
646            if state.addr() & (DOWNGRADED | LOCKED) == LOCKED {
647                // Another thread has locked the lock and no downgrade was requested.
648                // Leave waking up waiters to them by releasing the queue lock.
649                match self.state.compare_exchange_weak(
650                    state,
651                    state.mask(!QUEUE_LOCKED),
652                    Release,
653                    Acquire,
654                ) {
655                    Ok(_) => return,
656                    Err(new) => {
657                        state = new;
658                        continue;
659                    }
660                }
661            }
662
663            // Since we hold the queue lock and downgrades cannot be requested if the lock is
664            // already read-locked, we have exclusive control over the queue here and can make
665            // modifications.
666
667            let downgrade = state.addr() & DOWNGRADED != 0;
668            let is_writer = unsafe { tail.as_ref().write };
669            if !downgrade
670                && is_writer
671                && let Some(prev) = unsafe { tail.as_ref().prev.get() }
672            {
673                // If we are not downgrading and the next thread is a writer, only wake up that
674                // writing thread.
675
676                // Split off `tail`.
677                // There are no set `tail` links before the node pointed to by `state`, so the first
678                // non-null tail field will be current (Invariant 2).
679                // We also fulfill Invariant 4 since `find_tail` was called on this node, which
680                // ensures all backlinks are set.
681                unsafe {
682                    to_node(state).as_ref().tail.set(Some(prev));
683                }
684
685                // Try to release the queue lock. We need to check the state again since another
686                // thread might have acquired the lock and requested a downgrade.
687                let next = state.mask(!QUEUE_LOCKED);
688                if let Err(new) = self.state.compare_exchange_weak(state, next, Release, Acquire) {
689                    // Undo the tail modification above, so that we can find the tail again above.
690                    // As mentioned above, we have exclusive control over the queue, so no other
691                    // thread could have noticed the change.
692                    unsafe {
693                        to_node(state).as_ref().tail.set(Some(tail));
694                    }
695                    state = new;
696                    continue;
697                }
698
699                // The tail was split off and the lock was released. Mark the node as completed.
700                unsafe {
701                    return Node::complete(tail);
702                }
703            } else {
704                // We are either downgrading, the next waiter is a reader, or the queue only
705                // consists of one waiter. In any case, just wake all threads.
706
707                // Clear the queue.
708                let next =
709                    if downgrade { ptr::without_provenance_mut(SINGLE | LOCKED) } else { UNLOCKED };
710                if let Err(new) = self.state.compare_exchange_weak(state, next, Release, Acquire) {
711                    state = new;
712                    continue;
713                }
714
715                // SAFETY: we computed `tail` above, and no new nodes can have been added since
716                // (otherwise the CAS above would have failed).
717                // Thus we have complete control over the whole queue.
718                unsafe {
719                    return complete_all(tail);
720                }
721            }
722        }
723    }
724}