std/sys/sync/rwlock/queue.rs
1//! Efficient read-write locking without `pthread_rwlock_t`.
2//!
3//! The readers-writer lock provided by the `pthread` library has a number of problems which make it
4//! a suboptimal choice for `std`:
5//!
6//! * It is non-movable, so it needs to be allocated (lazily, to make the constructor `const`).
7//! * `pthread` is an external library, meaning the fast path of acquiring an uncontended lock
8//! cannot be inlined.
9//! * Some platforms (at least glibc before version 2.25) have buggy implementations that can easily
10//! lead to undefined behaviour in safe Rust code when not properly guarded against.
11//! * On some platforms (e.g. macOS), the lock is very slow.
12//!
13//! Therefore, we implement our own [`RwLock`]! Naively, one might reach for a spinlock, but those
14//! can be quite [problematic] when the lock is contended.
15//!
16//! Instead, this [`RwLock`] copies its implementation strategy from the Windows [SRWLOCK] and the
17//! [usync] library implementations.
18//!
19//! Spinning is still used for the fast path, but it is bounded: after spinning fails, threads will
20//! locklessly add an information structure ([`Node`]) containing a [`Thread`] handle into a queue
21//! of waiters associated with the lock. The lock owner, upon releasing the lock, will scan through
22//! the queue and wake up threads as appropriate, and the newly-awoken threads will then try to
23//! acquire the lock themselves.
24//!
25//! The resulting [`RwLock`] is:
26//!
27//! * adaptive, since it spins before doing any heavyweight parking operations
28//! * allocation-free, modulo the per-thread [`Thread`] handle, which is allocated anyways when
29//! using threads created by `std`
30//! * writer-preferring, even if some readers may still slip through
31//! * unfair, which reduces context-switching and thus drastically improves performance
32//!
33//! and also quite fast in most cases.
34//!
35//! [problematic]: https://matklad.github.io/2020/01/02/spinlocks-considered-harmful.html
36//! [SRWLOCK]: https://learn.microsoft.com/en-us/windows/win32/sync/slim-reader-writer--srw--locks
37//! [usync]: https://crates.io/crates/usync
38//!
39//! # Implementation
40//!
41//! ## State
42//!
43//! A single [`AtomicPtr`] is used as state variable. The lowest four bits are used to indicate the
44//! meaning of the remaining bits:
45//!
46//! | [`LOCKED`] | [`QUEUED`] | [`QUEUE_LOCKED`] | [`DOWNGRADED`] | Remaining | |
47//! |------------|:-----------|:-----------------|:---------------|:-------------|:----------------------------------------------------------------------------------------------------------------------------|
48//! | 0 | 0 | 0 | 0 | 0 | The lock is unlocked, no threads are waiting |
49//! | 1 | 0 | 0 | 0 | 0 | The lock is write-locked, no threads waiting |
50//! | 1 | 0 | 0 | 0 | n > 0 | The lock is read-locked with n readers |
51//! | 0 | 1 | * | 0 | `*mut Node` | The lock is unlocked, but some threads are waiting. Only writers may lock the lock |
52//! | 1 | 1 | * | * | `*mut Node` | The lock is locked, but some threads are waiting. If the lock is read-locked, the last queue node contains the reader count |
53//!
54//! ## Waiter Queue
55//!
56//! When threads are waiting on the lock (the `QUEUE` bit is set), the lock state points to a queue
57//! of waiters, which is implemented as a linked list of nodes stored on the stack to avoid memory
58//! allocation.
59//!
60//! To enable lock-free enqueuing of new nodes to the queue, the linked list is singly-linked upon
61//! creation.
62//!
63//! When the lock is read-locked, the lock count (number of readers) is stored in the last link of
64//! the queue. Threads have to traverse the queue to find the last element upon releasing the lock.
65//! To avoid having to traverse the entire list every time we want to access the reader count, a
66//! pointer to the found tail is cached in the (current) first element of the queue.
67//!
68//! Also, while the lock is unfair for performance reasons, it is still best to wake the tail node
69//! first (FIFO ordering). Since we always pop nodes off the tail of the queue, we must store
70//! backlinks to previous nodes so that we can update the `tail` field of the (current) first
71//! element of the queue. Adding backlinks is done at the same time as finding the tail (via the
72//! function [`find_tail_and_add_backlinks`]), and thus encountering a set tail field on a node
73//! indicates that all following nodes in the queue are initialized.
74//!
75//! TLDR: Here's a diagram of what the queue looks like:
76//!
77//! ```text
78//! state
79//! │
80//! ▼
81//! ╭───────╮ next ╭───────╮ next ╭───────╮ next ╭───────╮
82//! │ ├─────►│ ├─────►│ ├─────►│ count │
83//! │ │ │ │ │ │ │ │
84//! │ │ │ │◄─────┤ │◄─────┤ │
85//! ╰───────╯ ╰───────╯ prev ╰───────╯ prev ╰───────╯
86//! │ ▲
87//! └───────────────────────────┘
88//! tail
89//! ```
90//!
91//! Invariants:
92//! 1. At least one node must contain a non-null, current `tail` field.
93//! 2. The first non-null `tail` field must be valid and current.
94//! 3. All nodes preceding this node must have a correct, non-null `next` field.
95//! 4. All nodes following this node must have a correct, non-null `prev` field.
96//!
97//! Access to the queue is controlled by the `QUEUE_LOCKED` bit. Threads will try to set this bit
98//! in two cases: one is when a thread enqueues itself and eagerly adds backlinks to the queue
99//! (which drastically improves performance), and the other is after a thread unlocks the lock to
100//! wake up the next waiter(s).
101//!
102//! `QUEUE_LOCKED` is set atomically at the same time as the enqueuing/unlocking operations. The
103//! thread releasing the `QUEUE_LOCKED` bit will check the state of the lock (in particular, whether
104//! a downgrade was requested using the [`DOWNGRADED`] bit) and wake up waiters as appropriate. This
105//! guarantees forward progress even if the unlocking thread could not acquire the queue lock.
106//!
107//! ## Memory Orderings
108//!
109//! To properly synchronize changes to the data protected by the lock, the lock is acquired and
110//! released with [`Acquire`] and [`Release`] ordering, respectively. To propagate the
111//! initialization of nodes, changes to the queue lock are also performed using these orderings.
112
113#![forbid(unsafe_op_in_unsafe_fn)]
114
115use crate::cell::OnceCell;
116use crate::hint::spin_loop;
117use crate::mem;
118use crate::ptr::{self, NonNull, null_mut, without_provenance_mut};
119use crate::sync::atomic::Ordering::{AcqRel, Acquire, Relaxed, Release};
120use crate::sync::atomic::{AtomicBool, AtomicPtr};
121use crate::thread::{self, Thread};
122
123/// The atomic lock state.
124type AtomicState = AtomicPtr<()>;
125/// The inner lock state.
126type State = *mut ();
127
128const UNLOCKED: State = without_provenance_mut(0);
129const LOCKED: usize = 1 << 0;
130const QUEUED: usize = 1 << 1;
131const QUEUE_LOCKED: usize = 1 << 2;
132const DOWNGRADED: usize = 1 << 3;
133const SINGLE: usize = 1 << 4;
134const STATE: usize = DOWNGRADED | QUEUE_LOCKED | QUEUED | LOCKED;
135const NODE_MASK: usize = !STATE;
136
137/// Locking uses exponential backoff. `SPIN_COUNT` indicates how many times the locking operation
138/// will be retried.
139///
140/// In other words, `spin_loop` will be called `2.pow(SPIN_COUNT) - 1` times.
141const SPIN_COUNT: usize = 7;
142
143/// Marks the state as write-locked, if possible.
144#[inline]
145fn write_lock(state: State) -> Option<State> {
146 if state.addr() & LOCKED == 0 { Some(state.map_addr(|addr| addr | LOCKED)) } else { None }
147}
148
149/// Marks the state as read-locked, if possible.
150#[inline]
151fn read_lock(state: State) -> Option<State> {
152 if state.addr() & QUEUED == 0 && state.addr() != LOCKED {
153 Some(without_provenance_mut(state.addr().checked_add(SINGLE)? | LOCKED))
154 } else {
155 None
156 }
157}
158
159/// Converts a `State` into a `Node` by masking out the bottom bits of the state, assuming that the
160/// state points to a queue node.
161///
162/// # Safety
163///
164/// The state must contain a valid pointer to a queue node.
165#[inline]
166unsafe fn to_node(state: State) -> NonNull<Node> {
167 unsafe { NonNull::new_unchecked(state.mask(NODE_MASK)).cast() }
168}
169
170/// The representation of a thread waiting on the lock queue.
171///
172/// We initialize these `Node`s on thread execution stacks to avoid allocation.
173///
174/// Note that we need an alignment of 16 to ensure that the last 4 bits of any
175/// pointers to `Node`s are always zeroed (for the bit flags described in the
176/// module-level documentation).
177#[repr(align(16))]
178struct Node {
179 next: AtomicLink,
180 prev: AtomicLink,
181 tail: AtomicLink,
182 write: bool,
183 thread: OnceCell<Thread>,
184 completed: AtomicBool,
185}
186
187/// An atomic node pointer with relaxed operations.
188struct AtomicLink(AtomicPtr<Node>);
189
190impl AtomicLink {
191 fn new(v: Option<NonNull<Node>>) -> AtomicLink {
192 AtomicLink(AtomicPtr::new(v.map_or(null_mut(), NonNull::as_ptr)))
193 }
194
195 fn get(&self) -> Option<NonNull<Node>> {
196 NonNull::new(self.0.load(Relaxed))
197 }
198
199 fn set(&self, v: Option<NonNull<Node>>) {
200 self.0.store(v.map_or(null_mut(), NonNull::as_ptr), Relaxed);
201 }
202}
203
204impl Node {
205 /// Creates a new queue node.
206 fn new(write: bool) -> Node {
207 Node {
208 next: AtomicLink::new(None),
209 prev: AtomicLink::new(None),
210 tail: AtomicLink::new(None),
211 write,
212 thread: OnceCell::new(),
213 completed: AtomicBool::new(false),
214 }
215 }
216
217 /// Prepare this node for waiting.
218 fn prepare(&mut self) {
219 // Fall back to creating an unnamed `Thread` handle to allow locking in TLS destructors.
220 self.thread.get_or_init(thread::current_or_unnamed);
221 self.completed = AtomicBool::new(false);
222 }
223
224 /// Wait until this node is marked as [`complete`](Node::complete)d by another thread.
225 ///
226 /// # Safety
227 ///
228 /// May only be called from the thread that created the node.
229 unsafe fn wait(&self) {
230 while !self.completed.load(Acquire) {
231 unsafe {
232 self.thread.get().unwrap().park();
233 }
234 }
235 }
236
237 /// Atomically mark this node as completed.
238 ///
239 /// # Safety
240 ///
241 /// `node` must point to a valid `Node`, and the node may not outlive this call.
242 unsafe fn complete(node: NonNull<Node>) {
243 // Since the node may be destroyed immediately after the completed flag is set, clone the
244 // thread handle before that.
245 let thread = unsafe { node.as_ref().thread.get().unwrap().clone() };
246 unsafe {
247 node.as_ref().completed.store(true, Release);
248 }
249 thread.unpark();
250 }
251}
252
253/// Traverse the queue and find the tail, adding backlinks to the queue while traversing.
254///
255/// This may be called from multiple threads at the same time as long as the queue is not being
256/// modified (this happens when unlocking multiple readers).
257///
258/// # Safety
259///
260/// * `head` must point to a node in a valid queue.
261/// * `head` must be in front of the previous head node that was used to perform the last removal.
262/// * The part of the queue starting with `head` must not be modified during this call.
263unsafe fn find_tail_and_add_backlinks(head: NonNull<Node>) -> NonNull<Node> {
264 let mut current = head;
265
266 // Traverse the queue until we find a node that has a set `tail`.
267 let tail = loop {
268 let c = unsafe { current.as_ref() };
269 if let Some(tail) = c.tail.get() {
270 break tail;
271 }
272
273 // SAFETY: All `next` fields before the first node with a set `tail` are non-null and valid
274 // (by Invariant 3).
275 unsafe {
276 let next = c.next.get().unwrap_unchecked();
277 next.as_ref().prev.set(Some(current));
278 current = next;
279 }
280 };
281
282 unsafe {
283 head.as_ref().tail.set(Some(tail));
284 tail
285 }
286}
287
288/// [`complete`](Node::complete)s all threads in the queue ending with `tail`.
289///
290/// # Safety
291///
292/// * `tail` must be a valid tail of a fully linked queue.
293/// * The current thread must have exclusive access to that queue.
294unsafe fn complete_all(tail: NonNull<Node>) {
295 let mut current = tail;
296
297 // Traverse backwards through the queue (FIFO) and `complete` all of the nodes.
298 loop {
299 let prev = unsafe { current.as_ref().prev.get() };
300 unsafe {
301 Node::complete(current);
302 }
303 match prev {
304 Some(prev) => current = prev,
305 None => return,
306 }
307 }
308}
309
310/// A type to guard against the unwinds of stacks that nodes are located on due to panics.
311struct PanicGuard;
312
313impl Drop for PanicGuard {
314 fn drop(&mut self) {
315 rtabort!("tried to drop node in intrusive list.");
316 }
317}
318
319/// The public inner `RwLock` type.
320pub struct RwLock {
321 state: AtomicState,
322}
323
324impl RwLock {
325 #[inline]
326 pub const fn new() -> RwLock {
327 RwLock { state: AtomicPtr::new(UNLOCKED) }
328 }
329
330 #[inline]
331 pub fn try_read(&self) -> bool {
332 self.state.fetch_update(Acquire, Relaxed, read_lock).is_ok()
333 }
334
335 #[inline]
336 pub fn read(&self) {
337 if !self.try_read() {
338 self.lock_contended(false)
339 }
340 }
341
342 #[inline]
343 pub fn try_write(&self) -> bool {
344 // Atomically set the `LOCKED` bit. This is lowered to a single atomic instruction on most
345 // modern processors (e.g. "lock bts" on x86 and "ldseta" on modern AArch64), and therefore
346 // is more efficient than `fetch_update(lock(true))`, which can spuriously fail if a new
347 // node is appended to the queue.
348 self.state.fetch_or(LOCKED, Acquire).addr() & LOCKED == 0
349 }
350
351 #[inline]
352 pub fn write(&self) {
353 if !self.try_write() {
354 self.lock_contended(true)
355 }
356 }
357
358 #[cold]
359 fn lock_contended(&self, write: bool) {
360 let mut node = Node::new(write);
361 let mut state = self.state.load(Relaxed);
362 let mut count = 0;
363 let update_fn = if write { write_lock } else { read_lock };
364
365 loop {
366 // Optimistically update the state.
367 if let Some(next) = update_fn(state) {
368 // The lock is available, try locking it.
369 match self.state.compare_exchange_weak(state, next, Acquire, Relaxed) {
370 Ok(_) => return,
371 Err(new) => state = new,
372 }
373 continue;
374 } else if state.addr() & QUEUED == 0 && count < SPIN_COUNT {
375 // If the lock is not available and no threads are queued, optimistically spin for a
376 // while, using exponential backoff to decrease cache contention.
377 for _ in 0..(1 << count) {
378 spin_loop();
379 }
380 state = self.state.load(Relaxed);
381 count += 1;
382 continue;
383 }
384 // The optimistic paths did not succeed, so fall back to parking the thread.
385
386 // First, prepare the node.
387 node.prepare();
388
389 // If there are threads queued, this will set the `next` field to be a pointer to the
390 // first node in the queue.
391 // If the state is read-locked, this will set `next` to the lock count.
392 // If it is write-locked, it will set `next` to zero.
393 node.next.0 = AtomicPtr::new(state.mask(NODE_MASK).cast());
394 node.prev = AtomicLink::new(None);
395
396 // Set the `QUEUED` bit and preserve the `LOCKED` and `DOWNGRADED` bit.
397 let mut next = ptr::from_ref(&node)
398 .map_addr(|addr| addr | QUEUED | (state.addr() & (DOWNGRADED | LOCKED)))
399 as State;
400
401 let mut is_queue_locked = false;
402 if state.addr() & QUEUED == 0 {
403 // If this is the first node in the queue, set the `tail` field to the node itself
404 // to ensure there is a valid `tail` field in the queue (Invariants 1 & 2).
405 // This needs to use `set` to avoid invalidating the new pointer.
406 node.tail.set(Some(NonNull::from(&node)));
407 } else {
408 // Otherwise, the tail of the queue is not known.
409 node.tail.set(None);
410
411 // Try locking the queue to eagerly add backlinks.
412 next = next.map_addr(|addr| addr | QUEUE_LOCKED);
413
414 // Track if we changed the `QUEUE_LOCKED` bit from off to on.
415 is_queue_locked = state.addr() & QUEUE_LOCKED == 0;
416 }
417
418 // Register the node, using release ordering to propagate our changes to the waking
419 // thread.
420 if let Err(new) = self.state.compare_exchange_weak(state, next, AcqRel, Relaxed) {
421 // The state has changed, just try again.
422 state = new;
423 continue;
424 }
425 // The node has been registered, so the structure must not be mutably accessed or
426 // destroyed while other threads may be accessing it.
427
428 // Guard against unwinds using a `PanicGuard` that aborts when dropped.
429 let guard = PanicGuard;
430
431 // If the current thread locked the queue, unlock it to eagerly adding backlinks.
432 if is_queue_locked {
433 // SAFETY: This thread set the `QUEUE_LOCKED` bit above.
434 unsafe {
435 self.unlock_queue(next);
436 }
437 }
438
439 // Wait until the node is removed from the queue.
440 // SAFETY: the node was created by the current thread.
441 unsafe {
442 node.wait();
443 }
444
445 // The node was removed from the queue, disarm the guard.
446 mem::forget(guard);
447
448 // Reload the state and try again.
449 state = self.state.load(Relaxed);
450 count = 0;
451 }
452 }
453
454 #[inline]
455 pub unsafe fn read_unlock(&self) {
456 match self.state.fetch_update(Release, Acquire, |state| {
457 if state.addr() & QUEUED == 0 {
458 // If there are no threads queued, simply decrement the reader count.
459 let count = state.addr() - (SINGLE | LOCKED);
460 Some(if count > 0 { without_provenance_mut(count | LOCKED) } else { UNLOCKED })
461 } else if state.addr() & DOWNGRADED != 0 {
462 // This thread used to have exclusive access, but requested a downgrade. This has
463 // not been completed yet, so we still have exclusive access.
464 // Retract the downgrade request and unlock, but leave waking up new threads to the
465 // thread that already holds the queue lock.
466 Some(state.mask(!(DOWNGRADED | LOCKED)))
467 } else {
468 None
469 }
470 }) {
471 Ok(_) => {}
472 // There are waiters queued and the lock count was moved to the tail of the queue.
473 Err(state) => unsafe { self.read_unlock_contended(state) },
474 }
475 }
476
477 /// # Safety
478 ///
479 /// * There must be threads queued on the lock.
480 /// * `state` must be a pointer to a node in a valid queue.
481 /// * There cannot be a `downgrade` in progress.
482 #[cold]
483 unsafe fn read_unlock_contended(&self, state: State) {
484 // SAFETY:
485 // The state was observed with acquire ordering above, so the current thread will have
486 // observed all node initializations.
487 // We also know that no threads can be modifying the queue starting at `state`: because new
488 // read-locks cannot be acquired while there are any threads queued on the lock, all
489 // queue-lock owners will observe a set `LOCKED` bit in `self.state` and will not modify
490 // the queue. The other case that a thread could modify the queue is if a downgrade is in
491 // progress (removal of the entire queue), but since that is part of this function's safety
492 // contract, we can guarantee that no other threads can modify the queue.
493 let tail = unsafe { find_tail_and_add_backlinks(to_node(state)).as_ref() };
494
495 // The lock count is stored in the `next` field of `tail`.
496 // Decrement it, making sure to observe all changes made to the queue by the other lock
497 // owners by using acquire-release ordering.
498 let was_last = tail.next.0.fetch_byte_sub(SINGLE, AcqRel).addr() - SINGLE == 0;
499 if was_last {
500 // SAFETY: Other threads cannot read-lock while threads are queued. Also, the `LOCKED`
501 // bit is still set, so there are no writers. Thus the current thread exclusively owns
502 // this lock, even though it is a reader.
503 unsafe { self.unlock_contended(state) }
504 }
505 }
506
507 #[inline]
508 pub unsafe fn write_unlock(&self) {
509 if let Err(state) =
510 self.state.compare_exchange(without_provenance_mut(LOCKED), UNLOCKED, Release, Relaxed)
511 {
512 // SAFETY: Since other threads cannot acquire the lock, the state can only have changed
513 // because there are threads queued on the lock.
514 unsafe { self.unlock_contended(state) }
515 }
516 }
517
518 /// # Safety
519 ///
520 /// * The lock must be exclusively owned by this thread.
521 /// * There must be threads queued on the lock.
522 /// * `state` must be a pointer to a node in a valid queue.
523 /// * There cannot be a `downgrade` in progress.
524 #[cold]
525 unsafe fn unlock_contended(&self, state: State) {
526 debug_assert_eq!(state.addr() & (DOWNGRADED | QUEUED | LOCKED), QUEUED | LOCKED);
527
528 let mut current = state;
529
530 // We want to atomically release the lock and try to acquire the queue lock.
531 loop {
532 // First check if the queue lock is already held.
533 if current.addr() & QUEUE_LOCKED != 0 {
534 // Another thread holds the queue lock, so let them wake up waiters for us.
535 let next = current.mask(!LOCKED);
536 match self.state.compare_exchange_weak(current, next, Release, Relaxed) {
537 Ok(_) => return,
538 Err(new) => {
539 current = new;
540 continue;
541 }
542 }
543 }
544
545 // Atomically release the lock and try to acquire the queue lock.
546 let next = current.map_addr(|addr| (addr & !LOCKED) | QUEUE_LOCKED);
547 match self.state.compare_exchange_weak(current, next, AcqRel, Relaxed) {
548 // Now that we have the queue lock, we can wake up the next waiter.
549 Ok(_) => {
550 // SAFETY: This thread just acquired the queue lock, and this function's safety
551 // contract requires that there are threads already queued on the lock.
552 unsafe { self.unlock_queue(next) };
553 return;
554 }
555 Err(new) => current = new,
556 }
557 }
558 }
559
560 /// # Safety
561 ///
562 /// * The lock must be write-locked by this thread.
563 #[inline]
564 pub unsafe fn downgrade(&self) {
565 // Optimistically change the state from write-locked with a single writer and no waiters to
566 // read-locked with a single reader and no waiters.
567 if let Err(state) = self.state.compare_exchange(
568 without_provenance_mut(LOCKED),
569 without_provenance_mut(SINGLE | LOCKED),
570 Release,
571 Relaxed,
572 ) {
573 // SAFETY: The only way the state can have changed is if there are threads queued.
574 // Wake all of them up.
575 unsafe { self.downgrade_slow(state) }
576 }
577 }
578
579 /// Downgrades the lock from write-locked to read-locked in the case that there are threads
580 /// waiting on the wait queue.
581 ///
582 /// This function will either wake up all of the waiters on the wait queue or designate the
583 /// current holder of the queue lock to wake up all of the waiters instead. Once the waiters
584 /// wake up, they will continue in the execution loop of `lock_contended`.
585 ///
586 /// # Safety
587 ///
588 /// * The lock must be write-locked by this thread.
589 /// * `state` must be a pointer to a node in a valid queue.
590 /// * There must be threads queued on the lock.
591 #[cold]
592 unsafe fn downgrade_slow(&self, mut state: State) {
593 debug_assert_eq!(state.addr() & (DOWNGRADED | QUEUED | LOCKED), QUEUED | LOCKED);
594
595 // Attempt to wake up all waiters by taking ownership of the entire waiter queue.
596 loop {
597 if state.addr() & QUEUE_LOCKED != 0 {
598 // Another thread already holds the queue lock. Tell it to wake up all waiters.
599 // If the other thread succeeds in waking up waiters before we release our lock, the
600 // effect will be just the same as if we had changed the state below.
601 // Otherwise, the `DOWNGRADED` bit will still be set, meaning that when this thread
602 // calls `read_unlock` later (because it holds a read lock and must unlock
603 // eventually), it will realize that the lock is still exclusively locked and act
604 // accordingly.
605 let next = state.map_addr(|addr| addr | DOWNGRADED);
606 match self.state.compare_exchange_weak(state, next, Release, Relaxed) {
607 Ok(_) => return,
608 Err(new) => state = new,
609 }
610 } else {
611 // Grab the entire queue by swapping the `state` with a single reader.
612 let next = ptr::without_provenance_mut(SINGLE | LOCKED);
613 if let Err(new) = self.state.compare_exchange_weak(state, next, AcqRel, Relaxed) {
614 state = new;
615 continue;
616 }
617
618 // SAFETY: We have full ownership of this queue now, so nobody else can modify it.
619 let tail = unsafe { find_tail_and_add_backlinks(to_node(state)) };
620
621 // Wake up all waiters.
622 // SAFETY: `tail` was just computed, meaning the whole queue is linked, and we have
623 // full ownership of the queue, so we have exclusive access.
624 unsafe { complete_all(tail) };
625
626 return;
627 }
628 }
629 }
630
631 /// Unlocks the queue. Wakes up all threads if a downgrade was requested, otherwise wakes up the
632 /// next eligible thread(s) if the lock is unlocked.
633 ///
634 /// # Safety
635 ///
636 /// * The queue lock must be held by the current thread.
637 /// * `state` must be a pointer to a node in a valid queue.
638 /// * There must be threads queued on the lock.
639 unsafe fn unlock_queue(&self, mut state: State) {
640 debug_assert_eq!(state.addr() & (QUEUED | QUEUE_LOCKED), QUEUED | QUEUE_LOCKED);
641
642 loop {
643 // SAFETY: Since we have the queue lock, nobody else can be modifying the queue.
644 let tail = unsafe { find_tail_and_add_backlinks(to_node(state)) };
645
646 if state.addr() & (DOWNGRADED | LOCKED) == LOCKED {
647 // Another thread has locked the lock and no downgrade was requested.
648 // Leave waking up waiters to them by releasing the queue lock.
649 match self.state.compare_exchange_weak(
650 state,
651 state.mask(!QUEUE_LOCKED),
652 Release,
653 Acquire,
654 ) {
655 Ok(_) => return,
656 Err(new) => {
657 state = new;
658 continue;
659 }
660 }
661 }
662
663 // Since we hold the queue lock and downgrades cannot be requested if the lock is
664 // already read-locked, we have exclusive control over the queue here and can make
665 // modifications.
666
667 let downgrade = state.addr() & DOWNGRADED != 0;
668 let is_writer = unsafe { tail.as_ref().write };
669 if !downgrade
670 && is_writer
671 && let Some(prev) = unsafe { tail.as_ref().prev.get() }
672 {
673 // If we are not downgrading and the next thread is a writer, only wake up that
674 // writing thread.
675
676 // Split off `tail`.
677 // There are no set `tail` links before the node pointed to by `state`, so the first
678 // non-null tail field will be current (Invariant 2).
679 // We also fulfill Invariant 4 since `find_tail` was called on this node, which
680 // ensures all backlinks are set.
681 unsafe {
682 to_node(state).as_ref().tail.set(Some(prev));
683 }
684
685 // Try to release the queue lock. We need to check the state again since another
686 // thread might have acquired the lock and requested a downgrade.
687 let next = state.mask(!QUEUE_LOCKED);
688 if let Err(new) = self.state.compare_exchange_weak(state, next, Release, Acquire) {
689 // Undo the tail modification above, so that we can find the tail again above.
690 // As mentioned above, we have exclusive control over the queue, so no other
691 // thread could have noticed the change.
692 unsafe {
693 to_node(state).as_ref().tail.set(Some(tail));
694 }
695 state = new;
696 continue;
697 }
698
699 // The tail was split off and the lock was released. Mark the node as completed.
700 unsafe {
701 return Node::complete(tail);
702 }
703 } else {
704 // We are either downgrading, the next waiter is a reader, or the queue only
705 // consists of one waiter. In any case, just wake all threads.
706
707 // Clear the queue.
708 let next =
709 if downgrade { ptr::without_provenance_mut(SINGLE | LOCKED) } else { UNLOCKED };
710 if let Err(new) = self.state.compare_exchange_weak(state, next, Release, Acquire) {
711 state = new;
712 continue;
713 }
714
715 // SAFETY: we computed `tail` above, and no new nodes can have been added since
716 // (otherwise the CAS above would have failed).
717 // Thus we have complete control over the whole queue.
718 unsafe {
719 return complete_all(tail);
720 }
721 }
722 }
723 }
724}