atomic.rs - source
Description: Source of the Rust file `library/core/src/sync/atomic.rs`.

#
core/sync/
atomic.rs

    1//! Atomic types
    2//!
    3//! Atomic types provide primitive shared-memory communication between
    4//! threads, and are the building blocks of other concurrent
    5//! types.
    6//!
    7//! This module defines atomic versions of a select number of primitive
    8//! types, including [`AtomicBool`], [`AtomicIsize`], [`AtomicUsize`],
    9//! [`AtomicI8`], [`AtomicU16`], etc.
    10//! Atomic types present operations that, when used correctly, synchronize
    11//! updates between threads.
    12//!
    13//! Atomic variables are safe to share between threads (they implement [`Sync`])
    14//! but they do not themselves provide the mechanism for sharing and follow the
    15//! [threading model](../../../std/thread/index.html#the-threading-model) of Rust.
    16//! The most common way to share an atomic variable is to put it into an [`Arc`][arc] (an
    17//! atomically-reference-counted shared pointer).
    18//!
    19//! [arc]: ../../../std/sync/struct.Arc.html
    20//!
    21//! Atomic types may be stored in static variables, initialized using
    22//! the constant initializers like [`AtomicBool::new`]. Atomic statics
    23//! are often used for lazy global initialization.
    24//!
    25//! ## Memory model for atomic accesses
    26//!
    27//! Rust atomics currently follow the same rules as [C++20 atomics][cpp], specifically the rules
    28//! from the [`intro.races`][cpp-intro.races] section, without the "consume" memory ordering. Since
    29//! C++ uses an object-based memory model whereas Rust is access-based, a bit of translation work
    30//! has to be done to apply the C++ rules to Rust: whenever C++ talks about "the value of an
    31//! object", we understand that to mean the resulting bytes obtained when doing a read. When the C++
    32//! standard talks about "the value of an atomic object", this refers to the result of doing an
    33//! atomic load (via the operations provided in this module). A "modification of an atomic object"
    34//! refers to an atomic store.
    35//!
    36//! The end result is *almost* equivalent to saying that creating a *shared reference* to one of the
    37//! Rust atomic types corresponds to creating an `atomic_ref` in C++, with the `atomic_ref` being
    38//! destroyed when the lifetime of the shared reference ends. The main difference is that Rust
    39//! permits concurrent atomic and non-atomic reads to the same memory as those cause no issue in the
    40//! C++ memory model, they are just forbidden in C++ because memory is partitioned into "atomic
    41//! objects" and "non-atomic objects" (with `atomic_ref` temporarily converting a non-atomic object
    42//! into an atomic object).
    43//!
    44//! The most important aspect of this model is that *data races* are undefined behavior. A data race
    45//! is defined as conflicting non-synchronized accesses where at least one of the accesses is
    46//! non-atomic. Here, accesses are *conflicting* if they affect overlapping regions of memory and at
    47//! least one of them is a write. (A `compare_exchange` or `compare_exchange_weak` that does not
    48//! succeed is not considered a write.) They are *non-synchronized* if neither of them
    49//! *happens-before* the other, according to the happens-before order of the memory model.
    50//!
    51//! The other possible cause of undefined behavior in the memory model are mixed-size accesses: Rust
    52//! inherits the C++ limitation that non-synchronized conflicting atomic accesses may not partially
    53//! overlap. In other words, every pair of non-synchronized atomic accesses must be either disjoint,
    54//! access the exact same memory (including using the same access size), or both be reads.
    55//!
    56//! Each atomic access takes an [`Ordering`] which defines how the operation interacts with the
    57//! happens-before order. These orderings behave the same as the corresponding [C++20 atomic
    58//! orderings][cpp_memory_order]. For more information, see the [nomicon].
    59//!
    60//! [cpp]: https://en.cppreference.com/w/cpp/atomic
    61//! [cpp-intro.races]: https://timsong-cpp.github.io/cppwp/n4868/intro.multithread#intro.races
    62//! [cpp_memory_order]: https://en.cppreference.com/w/cpp/atomic/memory_order
    63//! [nomicon]: ../../../nomicon/atomics.html
    64//!
    65//! ```rust,no_run undefined_behavior
    66//! use std::sync::atomic::{AtomicU16, AtomicU8, Ordering};
    67//! use std::mem::transmute;
    68//! use std::thread;
    69//!
    70//! let atomic = AtomicU16::new(0);
    71//!
    72//! thread::scope(|s| {
    73//!     // This is UB: conflicting non-synchronized accesses, at least one of which is non-atomic.
    74//!     s.spawn(|| atomic.store(1, Ordering::Relaxed)); // atomic store
    75//!     s.spawn(|| unsafe { atomic.as_ptr().write(2) }); // non-atomic write
    76//! });
    77//!
    78//! thread::scope(|s| {
    79//!     // This is fine: the accesses do not conflict (as none of them performs any modification).
    80//!     // In C++ this would be disallowed since creating an `atomic_ref` precludes
    81//!     // further non-atomic accesses, but Rust does not have that limitation.
    82//!     s.spawn(|| atomic.load(Ordering::Relaxed)); // atomic load
    83//!     s.spawn(|| unsafe { atomic.as_ptr().read() }); // non-atomic read
    84//! });
    85//!
    86//! thread::scope(|s| {
    87//!     // This is fine: `join` synchronizes the code in a way such that the atomic
    88//!     // store happens-before the non-atomic write.
    89//!     let handle = s.spawn(|| atomic.store(1, Ordering::Relaxed)); // atomic store
    90//!     handle.join().expect("thread won't panic"); // synchronize
    91//!     s.spawn(|| unsafe { atomic.as_ptr().write(2) }); // non-atomic write
    92//! });
    93//!
    94//! thread::scope(|s| {
    95//!     // This is UB: non-synchronized conflicting differently-sized atomic accesses.
    96//!     s.spawn(|| atomic.store(1, Ordering::Relaxed));
    97//!     s.spawn(|| unsafe {
    98//!         let differently_sized = transmute::<&AtomicU16, &AtomicU8>(&atomic);
    99//!         differently_sized.store(2, Ordering::Relaxed);
    100//!     });
    101//! });
    102//!
    103//! thread::scope(|s| {
    104//!     // This is fine: `join` synchronizes the code in a way such that
    105//!     // the 1-byte store happens-before the 2-byte store.
    106//!     let handle = s.spawn(|| atomic.store(1, Ordering::Relaxed));
    107//!     handle.join().expect("thread won't panic");
    108//!     s.spawn(|| unsafe {
    109//!         let differently_sized = transmute::<&AtomicU16, &AtomicU8>(&atomic);
    110//!         differently_sized.store(2, Ordering::Relaxed);
    111//!     });
    112//! });
    113//! ```
    114//!
    115//! # Portability
    116//!
    117//! All atomic types in this module are guaranteed to be [lock-free] if they're
    118//! available. This means they don't internally acquire a global mutex. Atomic
    119//! types and operations are not guaranteed to be wait-free. This means that
    120//! operations like `fetch_or` may be implemented with a compare-and-swap loop.
    121//!
    122//! Atomic operations may be implemented at the instruction layer with
    123//! larger-size atomics. For example some platforms use 4-byte atomic
    124//! instructions to implement `AtomicI8`. Note that this emulation should not
    125//! have an impact on correctness of code, it's just something to be aware of.
    126//!
    127//! The atomic types in this module might not be available on all platforms. The
    128//! atomic types here are all widely available, however, and can generally be
    129//! relied upon existing. Some notable exceptions are:
    130//!
    131//! * PowerPC and MIPS platforms with 32-bit pointers do not have `AtomicU64` or
    132//!   `AtomicI64` types.
    133//! * Legacy ARM platforms like ARMv4T and ARMv5TE have very limited hardware
    134//!   support for atomics. The bare-metal targets disable this module
    135//!   entirely, but the Linux targets [use the kernel] to assist (which comes
    136//!   with a performance penalty). It's not until ARMv6K onwards that ARM CPUs
    137//!   have support for load/store and Compare and Swap (CAS) atomics in hardware.
    138//! * ARMv6-M and ARMv8-M baseline targets (`thumbv6m-*` and
    139//!   `thumbv8m.base-*`) only provide `load` and `store` operations, and do
    140//!   not support Compare and Swap (CAS) operations, such as `swap`,
    141//!   `fetch_add`, etc. Full CAS support is available on ARMv7-M and ARMv8-M
    142//!   Mainline (`thumbv7m-*`, `thumbv7em*` and `thumbv8m.main-*`).
    143//!
    144//! [use the kernel]: https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt
    145//!
    146//! Note that future platforms may be added that also do not have support for
    147//! some atomic operations. Maximally portable code will want to be careful
    148//! about which atomic types are used. `AtomicUsize` and `AtomicIsize` are
    149//! generally the most portable, but even then they're not available everywhere.
    150//! For reference, the `std` library requires `AtomicBool`s and pointer-sized atomics, although
    151//! `core` does not.
    152//!
    153//! The `#[cfg(target_has_atomic)]` attribute can be used to conditionally
    154//! compile based on the target's supported bit widths. It is a key-value
    155//! option set for each supported size, with values "8", "16", "32", "64",
    156//! "128", and "ptr" for pointer-sized atomics.
    157//!
    158//! [lock-free]: https://en.wikipedia.org/wiki/Non-blocking_algorithm
    159//!
    160//! # Atomic accesses to read-only memory
    161//!
    162//! In general, *all* atomic accesses on read-only memory are undefined behavior. For instance, attempting
    163//! to do a `compare_exchange` that will definitely fail (making it conceptually a read-only
    164//! operation) can still cause a segmentation fault if the underlying memory page is mapped read-only. Since
    165//! atomic `load`s might be implemented using compare-exchange operations, even a `load` can fault
    166//! on read-only memory.
    167//!
    168//! For the purpose of this section, "read-only memory" is defined as memory that is read-only in
    169//! the underlying target, i.e., the pages are mapped with a read-only flag and any attempt to write
    170//! will cause a page fault. In particular, an `&u128` reference that points to memory that is
    171//! read-write mapped is *not* considered to point to "read-only memory". In Rust, almost all memory
    172//! is read-write; the only exceptions are memory created by `const` items or `static` items without
    173//! interior mutability, and memory that was specifically marked as read-only by the operating
    174//! system via platform-specific APIs.
    175//!
    176//! As an exception from the general rule stated above, "sufficiently small" atomic loads with
    177//! `Ordering::Relaxed` are implemented in a way that works on read-only memory, and are hence not
    178//! undefined behavior. The exact size limit for what makes a load "sufficiently small" varies
    179//! depending on the target:
    180//!
    181//! | `target_arch` | Size limit |
    182//! |---------------|---------|
    183//! | `x86`, `arm`, `loongarch32`, `mips`, `mips32r6`, `powerpc`, `riscv32`, `sparc`, `hexagon` | 4 bytes |
    184//! | `x86_64`, `aarch64`, `loongarch64`, `mips64`, `mips64r6`, `powerpc64`, `riscv64`, `sparc64`, `s390x` | 8 bytes |
    185//!
    186//! Atomics loads that are larger than this limit as well as atomic loads with ordering other
    187//! than `Relaxed`, as well as *all* atomic loads on targets not listed in the table, might still be
    188//! read-only under certain conditions, but that is not a stable guarantee and should not be relied
    189//! upon.
    190//!
    191//! If you need to do an acquire load on read-only memory, you can do a relaxed load followed by an
    192//! acquire fence instead.
    193//!
    194//! # Examples
    195//!
    196//! A simple spinlock:
    197//!
    198//! ```ignore-wasm
    199//! use std::sync::Arc;
    200//! use std::sync::atomic::{AtomicUsize, Ordering};
    201//! use std::{hint, thread};
    202//!
    203//! fn main() {
    204//!     let spinlock = Arc::new(AtomicUsize::new(1));
    205//!
    206//!     let spinlock_clone = Arc::clone(&spinlock);
    207//!
    208//!     let thread = thread::spawn(move || {
    209//!         spinlock_clone.store(0, Ordering::Release);
    210//!     });
    211//!
    212//!     // Wait for the other thread to release the lock
    213//!     while spinlock.load(Ordering::Acquire) != 0 {
    214//!         hint::spin_loop();
    215//!     }
    216//!
    217//!     if let Err(panic) = thread.join() {
    218//!         println!("Thread had an error: {panic:?}");
    219//!     }
    220//! }
    221//! ```
    222//!
    223//! Keep a global count of live threads:
    224//!
    225//! ```
    226//! use std::sync::atomic::{AtomicUsize, Ordering};
    227//!
    228//! static GLOBAL_THREAD_COUNT: AtomicUsize = AtomicUsize::new(0);
    229//!
    230//! // Note that Relaxed ordering doesn't synchronize anything
    231//! // except the global thread counter itself.
    232//! let old_thread_count = GLOBAL_THREAD_COUNT.fetch_add(1, Ordering::Relaxed);
    233//! // Note that this number may not be true at the moment of printing
    234//! // because some other thread may have changed static value already.
    235//! println!("live threads: {}", old_thread_count + 1);
    236//! ```
    237
    238#![stable(feature = "rust1", since = "1.0.0")]
    239#![cfg_attr(not(target_has_atomic_load_store = "8"), allow(dead_code))]
    240#![cfg_attr(not(target_has_atomic_load_store = "8"), allow(unused_imports))]
    241#![rustc_diagnostic_item = "atomic_mod"]
    242// Clippy complains about the pattern of "safe function calling unsafe function taking pointers".
    243// This happens with AtomicPtr intrinsics but is fine, as the pointers clippy is concerned about
    244// are just normal values that get loaded/stored, but not dereferenced.
    245#![allow(clippy::not_unsafe_ptr_arg_deref)]
    246
    247use self::Ordering::*;
    248use crate::cell::UnsafeCell;
    249use crate::hint::spin_loop;
    250use crate::intrinsics::AtomicOrdering as AO;
    251use crate::{fmt, intrinsics};
    252
    253trait Sealed {}
    254
    255/// A marker trait for primitive types which can be modified atomically.
    256///
    257/// This is an implementation detail for <code>[Atomic]\<T></code> which may disappear or be replaced at any time.
    258///
    259/// # Safety
    260///
    261/// Types implementing this trait must be primitives that can be modified atomically.
    262///
    263/// The associated `Self::AtomicInner` type must have the same size and bit validity as `Self`,
    264/// but may have a higher alignment requirement, so the following `transmute`s are sound:
    265///
    266/// - `&mut Self::AtomicInner` as `&mut Self`
    267/// - `Self` as `Self::AtomicInner` or the reverse
    268#[unstable(
    269    feature = "atomic_internals",
    270    reason = "implementation detail which may disappear or be replaced at any time",
    271    issue = "none"
    272)]
    273#[expect(private_bounds)]
    274pub unsafe trait AtomicPrimitive: Sized + Copy + Sealed {
    275    /// Temporary implementation detail.
    276    type AtomicInner: Sized;
    277}
    278
    279macro impl_atomic_primitive(
    280    $Atom:ident $(<$T:ident>)? ($Primitive:ty),
    281    size($size:literal),
    282    align($align:literal) $(,)?
    283) {
    284    impl $(<$T>)? Sealed for $Primitive {}
    285
    286    #[unstable(
    287        feature = "atomic_internals",
    288        reason = "implementation detail which may disappear or be replaced at any time",
    289        issue = "none"
    290    )]
    291    #[cfg(target_has_atomic_load_store = $size)]
    292    unsafe impl $(<$T>)? AtomicPrimitive for $Primitive {
    293        type AtomicInner = $Atom $(<$T>)?;
    294    }
    295}
    296
    297impl_atomic_primitive!(AtomicBool(bool), size("8"), align(1));
    298impl_atomic_primitive!(AtomicI8(i8), size("8"), align(1));
    299impl_atomic_primitive!(AtomicU8(u8), size("8"), align(1));
    300impl_atomic_primitive!(AtomicI16(i16), size("16"), align(2));
    301impl_atomic_primitive!(AtomicU16(u16), size("16"), align(2));
    302impl_atomic_primitive!(AtomicI32(i32), size("32"), align(4));
    303impl_atomic_primitive!(AtomicU32(u32), size("32"), align(4));
    304impl_atomic_primitive!(AtomicI64(i64), size("64"), align(8));
    305impl_atomic_primitive!(AtomicU64(u64), size("64"), align(8));
    306impl_atomic_primitive!(AtomicI128(i128), size("128"), align(16));
    307impl_atomic_primitive!(AtomicU128(u128), size("128"), align(16));
    308
    309#[cfg(target_pointer_width = "16")]
    310impl_atomic_primitive!(AtomicIsize(isize), size("ptr"), align(2));
    311#[cfg(target_pointer_width = "32")]
    312impl_atomic_primitive!(AtomicIsize(isize), size("ptr"), align(4));
    313#[cfg(target_pointer_width = "64")]
    314impl_atomic_primitive!(AtomicIsize(isize), size("ptr"), align(8));
    315
    316#[cfg(target_pointer_width = "16")]
    317impl_atomic_primitive!(AtomicUsize(usize), size("ptr"), align(2));
    318#[cfg(target_pointer_width = "32")]
    319impl_atomic_primitive!(AtomicUsize(usize), size("ptr"), align(4));
    320#[cfg(target_pointer_width = "64")]
    321impl_atomic_primitive!(AtomicUsize(usize), size("ptr"), align(8));
    322
    323#[cfg(target_pointer_width = "16")]
    324impl_atomic_primitive!(AtomicPtr<T>(*mut T), size("ptr"), align(2));
    325#[cfg(target_pointer_width = "32")]
    326impl_atomic_primitive!(AtomicPtr<T>(*mut T), size("ptr"), align(4));
    327#[cfg(target_pointer_width = "64")]
    328impl_atomic_primitive!(AtomicPtr<T>(*mut T), size("ptr"), align(8));
    329
    330/// A memory location which can be safely modified from multiple threads.
    331///
    332/// This has the same size and bit validity as the underlying type `T`. However,
    333/// the alignment of this type is always equal to its size, even on targets where
    334/// `T` has alignment less than its size.
    335///
    336/// For more about the differences between atomic types and non-atomic types as
    337/// well as information about the portability of this type, please see the
    338/// [module-level documentation].
    339///
    340/// **Note:** This type is only available on platforms that support atomic loads
    341/// and stores of `T`.
    342///
    343/// [module-level documentation]: crate::sync::atomic
    344#[unstable(feature = "generic_atomic", issue = "130539")]
    345pub type Atomic<T> = <T as AtomicPrimitive>::AtomicInner;
    346
    347// Some architectures don't have byte-sized atomics, which results in LLVM
    348// emulating them using a LL/SC loop. However for AtomicBool we can take
    349// advantage of the fact that it only ever contains 0 or 1 and use atomic OR/AND
    350// instead, which LLVM can emulate using a larger atomic OR/AND operation.
    351//
    352// This list should only contain architectures which have word-sized atomic-or/
    353// atomic-and instructions but don't natively support byte-sized atomics.
    354#[cfg(target_has_atomic = "8")]
    355const EMULATE_ATOMIC_BOOL: bool = cfg!(any(
    356    target_arch = "riscv32",
    357    target_arch = "riscv64",
    358    target_arch = "loongarch32",
    359    target_arch = "loongarch64"
    360));
    361
    362/// A boolean type which can be safely shared between threads.
    363///
    364/// This type has the same size, alignment, and bit validity as a [`bool`].
    365///
    366/// **Note**: This type is only available on platforms that support atomic
    367/// loads and stores of `u8`.
    368#[cfg(target_has_atomic_load_store = "8")]
    369#[stable(feature = "rust1", since = "1.0.0")]
    370#[rustc_diagnostic_item = "AtomicBool"]
    371#[repr(C, align(1))]
    372pub struct AtomicBool {
    373    v: UnsafeCell<u8>,
    374}
    375
    376#[cfg(target_has_atomic_load_store = "8")]
    377#[stable(feature = "rust1", since = "1.0.0")]
    378impl Default for AtomicBool {
    379    /// Creates an `AtomicBool` initialized to `false`.
    380    #[inline]
    381    fn default() -> Self {
    382        Self::new(false)
    383    }
    384}
    385
    386// Send is implicitly implemented for AtomicBool.
    387#[cfg(target_has_atomic_load_store = "8")]
    388#[stable(feature = "rust1", since = "1.0.0")]
    389unsafe impl Sync for AtomicBool {}
    390
    391/// A raw pointer type which can be safely shared between threads.
    392///
    393/// This type has the same size and bit validity as a `*mut T`.
    394///
    395/// **Note**: This type is only available on platforms that support atomic
    396/// loads and stores of pointers. Its size depends on the target pointer's size.
    397#[cfg(target_has_atomic_load_store = "ptr")]
    398#[stable(feature = "rust1", since = "1.0.0")]
    399#[rustc_diagnostic_item = "AtomicPtr"]
    400#[cfg_attr(target_pointer_width = "16", repr(C, align(2)))]
    401#[cfg_attr(target_pointer_width = "32", repr(C, align(4)))]
    402#[cfg_attr(target_pointer_width = "64", repr(C, align(8)))]
    403pub struct AtomicPtr<T> {
    404    p: UnsafeCell<*mut T>,
    405}
    406
    407#[cfg(target_has_atomic_load_store = "ptr")]
    408#[stable(feature = "rust1", since = "1.0.0")]
    409impl<T> Default for AtomicPtr<T> {
    410    /// Creates a null `AtomicPtr<T>`.
    411    fn default() -> AtomicPtr<T> {
    412        AtomicPtr::new(crate::ptr::null_mut())
    413    }
    414}
    415
    416#[cfg(target_has_atomic_load_store = "ptr")]
    417#[stable(feature = "rust1", since = "1.0.0")]
    418unsafe impl<T> Send for AtomicPtr<T> {}
    419#[cfg(target_has_atomic_load_store = "ptr")]
    420#[stable(feature = "rust1", since = "1.0.0")]
    421unsafe impl<T> Sync for AtomicPtr<T> {}
    422
    423/// Atomic memory orderings
    424///
    425/// Memory orderings specify the way atomic operations synchronize memory.
    426/// In its weakest [`Ordering::Relaxed`], only the memory directly touched by the
    427/// operation is synchronized. On the other hand, a store-load pair of [`Ordering::SeqCst`]
    428/// operations synchronize other memory while additionally preserving a total order of such
    429/// operations across all threads.
    430///
    431/// Rust's memory orderings are [the same as those of
    432/// C++20](https://en.cppreference.com/w/cpp/atomic/memory_order).
    433///
    434/// For more information see the [nomicon].
    435///
    436/// [nomicon]: ../../../nomicon/atomics.html
    437#[stable(feature = "rust1", since = "1.0.0")]
    438#[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)]
    439#[non_exhaustive]
    440#[rustc_diagnostic_item = "Ordering"]
    441pub enum Ordering {
    442    /// No ordering constraints, only atomic operations.
    443    ///
    444    /// Corresponds to [`memory_order_relaxed`] in C++20.
    445    ///
    446    /// [`memory_order_relaxed`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Relaxed_ordering
    447    #[stable(feature = "rust1", since = "1.0.0")]
    448    Relaxed,
    449    /// When coupled with a store, all previous operations become ordered
    450    /// before any load of this value with [`Acquire`] (or stronger) ordering.
    451    /// In particular, all previous writes become visible to all threads
    452    /// that perform an [`Acquire`] (or stronger) load of this value.
    453    ///
    454    /// Notice that using this ordering for an operation that combines loads
    455    /// and stores leads to a [`Relaxed`] load operation!
    456    ///
    457    /// This ordering is only applicable for operations that can perform a store.
    458    ///
    459    /// Corresponds to [`memory_order_release`] in C++20.
    460    ///
    461    /// [`memory_order_release`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Release-Acquire_ordering
    462    #[stable(feature = "rust1", since = "1.0.0")]
    463    Release,
    464    /// When coupled with a load, if the loaded value was written by a store operation with
    465    /// [`Release`] (or stronger) ordering, then all subsequent operations
    466    /// become ordered after that store. In particular, all subsequent loads will see data
    467    /// written before the store.
    468    ///
    469    /// Notice that using this ordering for an operation that combines loads
    470    /// and stores leads to a [`Relaxed`] store operation!
    471    ///
    472    /// This ordering is only applicable for operations that can perform a load.
    473    ///
    474    /// Corresponds to [`memory_order_acquire`] in C++20.
    475    ///
    476    /// [`memory_order_acquire`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Release-Acquire_ordering
    477    #[stable(feature = "rust1", since = "1.0.0")]
    478    Acquire,
    479    /// Has the effects of both [`Acquire`] and [`Release`] together:
    480    /// For loads it uses [`Acquire`] ordering. For stores it uses the [`Release`] ordering.
    481    ///
    482    /// Notice that in the case of `compare_and_swap`, it is possible that the operation ends up
    483    /// not performing any store and hence it has just [`Acquire`] ordering. However,
    484    /// `AcqRel` will never perform [`Relaxed`] accesses.
    485    ///
    486    /// This ordering is only applicable for operations that combine both loads and stores.
    487    ///
    488    /// Corresponds to [`memory_order_acq_rel`] in C++20.
    489    ///
    490    /// [`memory_order_acq_rel`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Release-Acquire_ordering
    491    #[stable(feature = "rust1", since = "1.0.0")]
    492    AcqRel,
    493    /// Like [`Acquire`]/[`Release`]/[`AcqRel`] (for load, store, and load-with-store
    494    /// operations, respectively) with the additional guarantee that all threads see all
    495    /// sequentially consistent operations in the same order.
    496    ///
    497    /// Corresponds to [`memory_order_seq_cst`] in C++20.
    498    ///
    499    /// [`memory_order_seq_cst`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Sequentially-consistent_ordering
    500    #[stable(feature = "rust1", since = "1.0.0")]
    501    SeqCst,
    502}
    503
    504/// An [`AtomicBool`] initialized to `false`.
    505#[cfg(target_has_atomic_load_store = "8")]
    506#[stable(feature = "rust1", since = "1.0.0")]
    507#[deprecated(
    508    since = "1.34.0",
    509    note = "the `new` function is now preferred",
    510    suggestion = "AtomicBool::new(false)"
    511)]
    512pub const ATOMIC_BOOL_INIT: AtomicBool = AtomicBool::new(false);
    513
    514#[cfg(target_has_atomic_load_store = "8")]
    515impl AtomicBool {
    516    /// Creates a new `AtomicBool`.
    517    ///
    518    /// # Examples
    519    ///
    520    /// ```
    521    /// use std::sync::atomic::AtomicBool;
    522    ///
    523    /// let atomic_true = AtomicBool::new(true);
    524    /// let atomic_false = AtomicBool::new(false);
    525    /// ```
    526    #[inline]
    527    #[stable(feature = "rust1", since = "1.0.0")]
    528    #[rustc_const_stable(feature = "const_atomic_new", since = "1.24.0")]
    529    #[must_use]
    530    pub const fn new(v: bool) -> AtomicBool {
    531        AtomicBool { v: UnsafeCell::new(v as u8) }
    532    }
    533
    534    /// Creates a new `AtomicBool` from a pointer.
    535    ///
    536    /// # Examples
    537    ///
    538    /// ```
    539    /// use std::sync::atomic::{self, AtomicBool};
    540    ///
    541    /// // Get a pointer to an allocated value
    542    /// let ptr: *mut bool = Box::into_raw(Box::new(false));
    543    ///
    544    /// assert!(ptr.cast::<AtomicBool>().is_aligned());
    545    ///
    546    /// {
    547    ///     // Create an atomic view of the allocated value
    548    ///     let atomic = unsafe { AtomicBool::from_ptr(ptr) };
    549    ///
    550    ///     // Use `atomic` for atomic operations, possibly share it with other threads
    551    ///     atomic.store(true, atomic::Ordering::Relaxed);
    552    /// }
    553    ///
    554    /// // It's ok to non-atomically access the value behind `ptr`,
    555    /// // since the reference to the atomic ended its lifetime in the block above
    556    /// assert_eq!(unsafe { *ptr }, true);
    557    ///
    558    /// // Deallocate the value
    559    /// unsafe { drop(Box::from_raw(ptr)) }
    560    /// ```
    561    ///
    562    /// # Safety
    563    ///
    564    /// * `ptr` must be aligned to `align_of::<AtomicBool>()` (note that this is always true, since
    565    ///   `align_of::<AtomicBool>() == 1`).
    566    /// * `ptr` must be [valid] for both reads and writes for the whole lifetime `'a`.
    567    /// * You must adhere to the [Memory model for atomic accesses]. In particular, it is not
    568    ///   allowed to mix conflicting atomic and non-atomic accesses, or atomic accesses of different
    569    ///   sizes, without synchronization.
    570    ///
    571    /// [valid]: crate::ptr#safety
    572    /// [Memory model for atomic accesses]: self#memory-model-for-atomic-accesses
    573    #[inline]
    574    #[stable(feature = "atomic_from_ptr", since = "1.75.0")]
    575    #[rustc_const_stable(feature = "const_atomic_from_ptr", since = "1.84.0")]
    576    pub const unsafe fn from_ptr<'a>(ptr: *mut bool) -> &'a AtomicBool {
    577        // SAFETY: guaranteed by the caller
    578        unsafe { &*ptr.cast() }
    579    }
    580
    581    /// Returns a mutable reference to the underlying [`bool`].
    582    ///
    583    /// This is safe because the mutable reference guarantees that no other threads are
    584    /// concurrently accessing the atomic data.
    585    ///
    586    /// # Examples
    587    ///
    588    /// ```
    589    /// use std::sync::atomic::{AtomicBool, Ordering};
    590    ///
    591    /// let mut some_bool = AtomicBool::new(true);
    592    /// assert_eq!(*some_bool.get_mut(), true);
    593    /// *some_bool.get_mut() = false;
    594    /// assert_eq!(some_bool.load(Ordering::SeqCst), false);
    595    /// ```
    596    #[inline]
    597    #[stable(feature = "atomic_access", since = "1.15.0")]
    598    pub fn get_mut(&mut self) -> &mut bool {
    599        // SAFETY: the mutable reference guarantees unique ownership.
    600        unsafe { &mut *(self.v.get() as *mut bool) }
    601    }
    602
    603    /// Gets atomic access to a `&mut bool`.
    604    ///
    605    /// # Examples
    606    ///
    607    /// ```
    608    /// #![feature(atomic_from_mut)]
    609    /// use std::sync::atomic::{AtomicBool, Ordering};
    610    ///
    611    /// let mut some_bool = true;
    612    /// let a = AtomicBool::from_mut(&mut some_bool);
    613    /// a.store(false, Ordering::Relaxed);
    614    /// assert_eq!(some_bool, false);
    615    /// ```
    616    #[inline]
    617    #[cfg(target_has_atomic_equal_alignment = "8")]
    618    #[unstable(feature = "atomic_from_mut", issue = "76314")]
    619    pub fn from_mut(v: &mut bool) -> &mut Self {
    620        // SAFETY: the mutable reference guarantees unique ownership, and
    621        // alignment of both `bool` and `Self` is 1.
    622        unsafe { &mut *(v as *mut bool as *mut Self) }
    623    }
    624
    625    /// Gets non-atomic access to a `&mut [AtomicBool]` slice.
    626    ///
    627    /// This is safe because the mutable reference guarantees that no other threads are
    628    /// concurrently accessing the atomic data.
    629    ///
    630    /// # Examples
    631    ///
    632    /// ```ignore-wasm
    633    /// #![feature(atomic_from_mut)]
    634    /// use std::sync::atomic::{AtomicBool, Ordering};
    635    ///
    636    /// let mut some_bools = [const { AtomicBool::new(false) }; 10];
    637    ///
    638    /// let view: &mut [bool] = AtomicBool::get_mut_slice(&mut some_bools);
    639    /// assert_eq!(view, [false; 10]);
    640    /// view[..5].copy_from_slice(&[true; 5]);
    641    ///
    642    /// std::thread::scope(|s| {
    643    ///     for t in &some_bools[..5] {
    644    ///         s.spawn(move || assert_eq!(t.load(Ordering::Relaxed), true));
    645    ///     }
    646    ///
    647    ///     for f in &some_bools[5..] {
    648    ///         s.spawn(move || assert_eq!(f.load(Ordering::Relaxed), false));
    649    ///     }
    650    /// });
    651    /// ```
    652    #[inline]
    653    #[unstable(feature = "atomic_from_mut", issue = "76314")]
    654    pub fn get_mut_slice(this: &mut [Self]) -> &mut [bool] {
    655        // SAFETY: the mutable reference guarantees unique ownership.
    656        unsafe { &mut *(this as *mut [Self] as *mut [bool]) }
    657    }
    658
    659    /// Gets atomic access to a `&mut [bool]` slice.
    660    ///
    661    /// # Examples
    662    ///
    663    /// ```rust,ignore-wasm
    664    /// #![feature(atomic_from_mut)]
    665    /// use std::sync::atomic::{AtomicBool, Ordering};
    666    ///
    667    /// let mut some_bools = [false; 10];
    668    /// let a = &*AtomicBool::from_mut_slice(&mut some_bools);
    669    /// std::thread::scope(|s| {
    670    ///     for i in 0..a.len() {
    671    ///         s.spawn(move || a[i].store(true, Ordering::Relaxed));
    672    ///     }
    673    /// });
    674    /// assert_eq!(some_bools, [true; 10]);
    675    /// ```
    676    #[inline]
    677    #[cfg(target_has_atomic_equal_alignment = "8")]
    678    #[unstable(feature = "atomic_from_mut", issue = "76314")]
    679    pub fn from_mut_slice(v: &mut [bool]) -> &mut [Self] {
    680        // SAFETY: the mutable reference guarantees unique ownership, and
    681        // alignment of both `bool` and `Self` is 1.
    682        unsafe { &mut *(v as *mut [bool] as *mut [Self]) }
    683    }
    684
    685    /// Consumes the atomic and returns the contained value.
    686    ///
    687    /// This is safe because passing `self` by value guarantees that no other threads are
    688    /// concurrently accessing the atomic data.
    689    ///
    690    /// # Examples
    691    ///
    692    /// ```
    693    /// use std::sync::atomic::AtomicBool;
    694    ///
    695    /// let some_bool = AtomicBool::new(true);
    696    /// assert_eq!(some_bool.into_inner(), true);
    697    /// ```
    698    #[inline]
    699    #[stable(feature = "atomic_access", since = "1.15.0")]
    700    #[rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0")]
    701    pub const fn into_inner(self) -> bool {
    702        self.v.into_inner() != 0
    703    }
    704
    705    /// Loads a value from the bool.
    706    ///
    707    /// `load` takes an [`Ordering`] argument which describes the memory ordering
    708    /// of this operation. Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
    709    ///
    710    /// # Panics
    711    ///
    712    /// Panics if `order` is [`Release`] or [`AcqRel`].
    713    ///
    714    /// # Examples
    715    ///
    716    /// ```
    717    /// use std::sync::atomic::{AtomicBool, Ordering};
    718    ///
    719    /// let some_bool = AtomicBool::new(true);
    720    ///
    721    /// assert_eq!(some_bool.load(Ordering::Relaxed), true);
    722    /// ```
    723    #[inline]
    724    #[stable(feature = "rust1", since = "1.0.0")]
    725    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    726    pub fn load(&self, order: Ordering) -> bool {
    727        // SAFETY: any data races are prevented by atomic intrinsics and the raw
    728        // pointer passed in is valid because we got it from a reference.
    729        unsafe { atomic_load(self.v.get(), order) != 0 }
    730    }
    731
    732    /// Stores a value into the bool.
    733    ///
    734    /// `store` takes an [`Ordering`] argument which describes the memory ordering
    735    /// of this operation. Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
    736    ///
    737    /// # Panics
    738    ///
    739    /// Panics if `order` is [`Acquire`] or [`AcqRel`].
    740    ///
    741    /// # Examples
    742    ///
    743    /// ```
    744    /// use std::sync::atomic::{AtomicBool, Ordering};
    745    ///
    746    /// let some_bool = AtomicBool::new(true);
    747    ///
    748    /// some_bool.store(false, Ordering::Relaxed);
    749    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
    750    /// ```
    751    #[inline]
    752    #[stable(feature = "rust1", since = "1.0.0")]
    753    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    754    #[rustc_should_not_be_called_on_const_items]
    755    pub fn store(&self, val: bool, order: Ordering) {
    756        // SAFETY: any data races are prevented by atomic intrinsics and the raw
    757        // pointer passed in is valid because we got it from a reference.
    758        unsafe {
    759            atomic_store(self.v.get(), val as u8, order);
    760        }
    761    }
    762
    763    /// Stores a value into the bool, returning the previous value.
    764    ///
    765    /// `swap` takes an [`Ordering`] argument which describes the memory ordering
    766    /// of this operation. All ordering modes are possible. Note that using
    767    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    768    /// using [`Release`] makes the load part [`Relaxed`].
    769    ///
    770    /// **Note:** This method is only available on platforms that support atomic
    771    /// operations on `u8`.
    772    ///
    773    /// # Examples
    774    ///
    775    /// ```
    776    /// use std::sync::atomic::{AtomicBool, Ordering};
    777    ///
    778    /// let some_bool = AtomicBool::new(true);
    779    ///
    780    /// assert_eq!(some_bool.swap(false, Ordering::Relaxed), true);
    781    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
    782    /// ```
    783    #[inline]
    784    #[stable(feature = "rust1", since = "1.0.0")]
    785    #[cfg(target_has_atomic = "8")]
    786    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    787    #[rustc_should_not_be_called_on_const_items]
    788    pub fn swap(&self, val: bool, order: Ordering) -> bool {
    789        if EMULATE_ATOMIC_BOOL {
    790            if val { self.fetch_or(true, order) } else { self.fetch_and(false, order) }
    791        } else {
    792            // SAFETY: data races are prevented by atomic intrinsics.
    793            unsafe { atomic_swap(self.v.get(), val as u8, order) != 0 }
    794        }
    795    }
    796
    797    /// Stores a value into the [`bool`] if the current value is the same as the `current` value.
    798    ///
    799    /// The return value is always the previous value. If it is equal to `current`, then the value
    800    /// was updated.
    801    ///
    802    /// `compare_and_swap` also takes an [`Ordering`] argument which describes the memory
    803    /// ordering of this operation. Notice that even when using [`AcqRel`], the operation
    804    /// might fail and hence just perform an `Acquire` load, but not have `Release` semantics.
    805    /// Using [`Acquire`] makes the store part of this operation [`Relaxed`] if it
    806    /// happens, and using [`Release`] makes the load part [`Relaxed`].
    807    ///
    808    /// **Note:** This method is only available on platforms that support atomic
    809    /// operations on `u8`.
    810    ///
    811    /// # Migrating to `compare_exchange` and `compare_exchange_weak`
    812    ///
    813    /// `compare_and_swap` is equivalent to `compare_exchange` with the following mapping for
    814    /// memory orderings:
    815    ///
    816    /// Original | Success | Failure
    817    /// -------- | ------- | -------
    818    /// Relaxed  | Relaxed | Relaxed
    819    /// Acquire  | Acquire | Acquire
    820    /// Release  | Release | Relaxed
    821    /// AcqRel   | AcqRel  | Acquire
    822    /// SeqCst   | SeqCst  | SeqCst
    823    ///
    824    /// `compare_and_swap` and `compare_exchange` also differ in their return type. You can use
    825    /// `compare_exchange(...).unwrap_or_else(|x| x)` to recover the behavior of `compare_and_swap`,
    826    /// but in most cases it is more idiomatic to check whether the return value is `Ok` or `Err`
    827    /// rather than to infer success vs failure based on the value that was read.
    828    ///
    829    /// During migration, consider whether it makes sense to use `compare_exchange_weak` instead.
    830    /// `compare_exchange_weak` is allowed to fail spuriously even when the comparison succeeds,
    831    /// which allows the compiler to generate better assembly code when the compare and swap
    832    /// is used in a loop.
    833    ///
    834    /// # Examples
    835    ///
    836    /// ```
    837    /// use std::sync::atomic::{AtomicBool, Ordering};
    838    ///
    839    /// let some_bool = AtomicBool::new(true);
    840    ///
    841    /// assert_eq!(some_bool.compare_and_swap(true, false, Ordering::Relaxed), true);
    842    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
    843    ///
    844    /// assert_eq!(some_bool.compare_and_swap(true, true, Ordering::Relaxed), false);
    845    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
    846    /// ```
    847    #[inline]
    848    #[stable(feature = "rust1", since = "1.0.0")]
    849    #[deprecated(
    850        since = "1.50.0",
    851        note = "Use `compare_exchange` or `compare_exchange_weak` instead"
    852    )]
    853    #[cfg(target_has_atomic = "8")]
    854    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    855    #[rustc_should_not_be_called_on_const_items]
    856    pub fn compare_and_swap(&self, current: bool, new: bool, order: Ordering) -> bool {
    857        match self.compare_exchange(current, new, order, strongest_failure_ordering(order)) {
    858            Ok(x) => x,
    859            Err(x) => x,
    860        }
    861    }
    862
    863    /// Stores a value into the [`bool`] if the current value is the same as the `current` value.
    864    ///
    865    /// The return value is a result indicating whether the new value was written and containing
    866    /// the previous value. On success this value is guaranteed to be equal to `current`.
    867    ///
    868    /// `compare_exchange` takes two [`Ordering`] arguments to describe the memory
    869    /// ordering of this operation. `success` describes the required ordering for the
    870    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
    871    /// `failure` describes the required ordering for the load operation that takes place when
    872    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
    873    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
    874    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    875    ///
    876    /// **Note:** This method is only available on platforms that support atomic
    877    /// operations on `u8`.
    878    ///
    879    /// # Examples
    880    ///
    881    /// ```
    882    /// use std::sync::atomic::{AtomicBool, Ordering};
    883    ///
    884    /// let some_bool = AtomicBool::new(true);
    885    ///
    886    /// assert_eq!(some_bool.compare_exchange(true,
    887    ///                                       false,
    888    ///                                       Ordering::Acquire,
    889    ///                                       Ordering::Relaxed),
    890    ///            Ok(true));
    891    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
    892    ///
    893    /// assert_eq!(some_bool.compare_exchange(true, true,
    894    ///                                       Ordering::SeqCst,
    895    ///                                       Ordering::Acquire),
    896    ///            Err(false));
    897    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
    898    /// ```
    899    ///
    900    /// # Considerations
    901    ///
    902    /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
    903    /// of CAS operations. In particular, a load of the value followed by a successful
    904    /// `compare_exchange` with the previous load *does not ensure* that other threads have not
    905    /// changed the value in the interim. This is usually important when the *equality* check in
    906    /// the `compare_exchange` is being used to check the *identity* of a value, but equality
    907    /// does not necessarily imply identity. In this case, `compare_exchange` can lead to the
    908    /// [ABA problem].
    909    ///
    910    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    911    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    912    #[inline]
    913    #[stable(feature = "extended_compare_and_swap", since = "1.10.0")]
    914    #[doc(alias = "compare_and_swap")]
    915    #[cfg(target_has_atomic = "8")]
    916    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    917    #[rustc_should_not_be_called_on_const_items]
    918    pub fn compare_exchange(
    919        &self,
    920        current: bool,
    921        new: bool,
    922        success: Ordering,
    923        failure: Ordering,
    924    ) -> Result<bool, bool> {
    925        if EMULATE_ATOMIC_BOOL {
    926            // Pick the strongest ordering from success and failure.
    927            let order = match (success, failure) {
    928                (SeqCst, _) => SeqCst,
    929                (_, SeqCst) => SeqCst,
    930                (AcqRel, _) => AcqRel,
    931                (_, AcqRel) => {
    932                    panic!("there is no such thing as an acquire-release failure ordering")
    933                }
    934                (Release, Acquire) => AcqRel,
    935                (Acquire, _) => Acquire,
    936                (_, Acquire) => Acquire,
    937                (Release, Relaxed) => Release,
    938                (_, Release) => panic!("there is no such thing as a release failure ordering"),
    939                (Relaxed, Relaxed) => Relaxed,
    940            };
    941            let old = if current == new {
    942                // This is a no-op, but we still need to perform the operation
    943                // for memory ordering reasons.
    944                self.fetch_or(false, order)
    945            } else {
    946                // This sets the value to the new one and returns the old one.
    947                self.swap(new, order)
    948            };
    949            if old == current { Ok(old) } else { Err(old) }
    950        } else {
    951            // SAFETY: data races are prevented by atomic intrinsics.
    952            match unsafe {
    953                atomic_compare_exchange(self.v.get(), current as u8, new as u8, success, failure)
    954            } {
    955                Ok(x) => Ok(x != 0),
    956                Err(x) => Err(x != 0),
    957            }
    958        }
    959    }
    960
    961    /// Stores a value into the [`bool`] if the current value is the same as the `current` value.
    962    ///
    963    /// Unlike [`AtomicBool::compare_exchange`], this function is allowed to spuriously fail even when the
    964    /// comparison succeeds, which can result in more efficient code on some platforms. The
    965    /// return value is a result indicating whether the new value was written and containing the
    966    /// previous value.
    967    ///
    968    /// `compare_exchange_weak` takes two [`Ordering`] arguments to describe the memory
    969    /// ordering of this operation. `success` describes the required ordering for the
    970    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
    971    /// `failure` describes the required ordering for the load operation that takes place when
    972    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
    973    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
    974    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    975    ///
    976    /// **Note:** This method is only available on platforms that support atomic
    977    /// operations on `u8`.
    978    ///
    979    /// # Examples
    980    ///
    981    /// ```
    982    /// use std::sync::atomic::{AtomicBool, Ordering};
    983    ///
    984    /// let val = AtomicBool::new(false);
    985    ///
    986    /// let new = true;
    987    /// let mut old = val.load(Ordering::Relaxed);
    988    /// loop {
    989    ///     match val.compare_exchange_weak(old, new, Ordering::SeqCst, Ordering::Relaxed) {
    990    ///         Ok(_) => break,
    991    ///         Err(x) => old = x,
    992    ///     }
    993    /// }
    994    /// ```
    995    ///
    996    /// # Considerations
    997    ///
    998    /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
    999    /// of CAS operations. In particular, a load of the value followed by a successful
    1000    /// `compare_exchange` with the previous load *does not ensure* that other threads have not
    1001    /// changed the value in the interim. This is usually important when the *equality* check in
    1002    /// the `compare_exchange` is being used to check the *identity* of a value, but equality
    1003    /// does not necessarily imply identity. In this case, `compare_exchange` can lead to the
    1004    /// [ABA problem].
    1005    ///
    1006    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    1007    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    1008    #[inline]
    1009    #[stable(feature = "extended_compare_and_swap", since = "1.10.0")]
    1010    #[doc(alias = "compare_and_swap")]
    1011    #[cfg(target_has_atomic = "8")]
    1012    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1013    #[rustc_should_not_be_called_on_const_items]
    1014    pub fn compare_exchange_weak(
    1015        &self,
    1016        current: bool,
    1017        new: bool,
    1018        success: Ordering,
    1019        failure: Ordering,
    1020    ) -> Result<bool, bool> {
    1021        if EMULATE_ATOMIC_BOOL {
    1022            return self.compare_exchange(current, new, success, failure);
    1023        }
    1024
    1025        // SAFETY: data races are prevented by atomic intrinsics.
    1026        match unsafe {
    1027            atomic_compare_exchange_weak(self.v.get(), current as u8, new as u8, success, failure)
    1028        } {
    1029            Ok(x) => Ok(x != 0),
    1030            Err(x) => Err(x != 0),
    1031        }
    1032    }
    1033
    1034    /// Logical "and" with a boolean value.
    1035    ///
    1036    /// Performs a logical "and" operation on the current value and the argument `val`, and sets
    1037    /// the new value to the result.
    1038    ///
    1039    /// Returns the previous value.
    1040    ///
    1041    /// `fetch_and` takes an [`Ordering`] argument which describes the memory ordering
    1042    /// of this operation. All ordering modes are possible. Note that using
    1043    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    1044    /// using [`Release`] makes the load part [`Relaxed`].
    1045    ///
    1046    /// **Note:** This method is only available on platforms that support atomic
    1047    /// operations on `u8`.
    1048    ///
    1049    /// # Examples
    1050    ///
    1051    /// ```
    1052    /// use std::sync::atomic::{AtomicBool, Ordering};
    1053    ///
    1054    /// let foo = AtomicBool::new(true);
    1055    /// assert_eq!(foo.fetch_and(false, Ordering::SeqCst), true);
    1056    /// assert_eq!(foo.load(Ordering::SeqCst), false);
    1057    ///
    1058    /// let foo = AtomicBool::new(true);
    1059    /// assert_eq!(foo.fetch_and(true, Ordering::SeqCst), true);
    1060    /// assert_eq!(foo.load(Ordering::SeqCst), true);
    1061    ///
    1062    /// let foo = AtomicBool::new(false);
    1063    /// assert_eq!(foo.fetch_and(false, Ordering::SeqCst), false);
    1064    /// assert_eq!(foo.load(Ordering::SeqCst), false);
    1065    /// ```
    1066    #[inline]
    1067    #[stable(feature = "rust1", since = "1.0.0")]
    1068    #[cfg(target_has_atomic = "8")]
    1069    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1070    #[rustc_should_not_be_called_on_const_items]
    1071    pub fn fetch_and(&self, val: bool, order: Ordering) -> bool {
    1072        // SAFETY: data races are prevented by atomic intrinsics.
    1073        unsafe { atomic_and(self.v.get(), val as u8, order) != 0 }
    1074    }
    1075
    1076    /// Logical "nand" with a boolean value.
    1077    ///
    1078    /// Performs a logical "nand" operation on the current value and the argument `val`, and sets
    1079    /// the new value to the result.
    1080    ///
    1081    /// Returns the previous value.
    1082    ///
    1083    /// `fetch_nand` takes an [`Ordering`] argument which describes the memory ordering
    1084    /// of this operation. All ordering modes are possible. Note that using
    1085    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    1086    /// using [`Release`] makes the load part [`Relaxed`].
    1087    ///
    1088    /// **Note:** This method is only available on platforms that support atomic
    1089    /// operations on `u8`.
    1090    ///
    1091    /// # Examples
    1092    ///
    1093    /// ```
    1094    /// use std::sync::atomic::{AtomicBool, Ordering};
    1095    ///
    1096    /// let foo = AtomicBool::new(true);
    1097    /// assert_eq!(foo.fetch_nand(false, Ordering::SeqCst), true);
    1098    /// assert_eq!(foo.load(Ordering::SeqCst), true);
    1099    ///
    1100    /// let foo = AtomicBool::new(true);
    1101    /// assert_eq!(foo.fetch_nand(true, Ordering::SeqCst), true);
    1102    /// assert_eq!(foo.load(Ordering::SeqCst) as usize, 0);
    1103    /// assert_eq!(foo.load(Ordering::SeqCst), false);
    1104    ///
    1105    /// let foo = AtomicBool::new(false);
    1106    /// assert_eq!(foo.fetch_nand(false, Ordering::SeqCst), false);
    1107    /// assert_eq!(foo.load(Ordering::SeqCst), true);
    1108    /// ```
    1109    #[inline]
    1110    #[stable(feature = "rust1", since = "1.0.0")]
    1111    #[cfg(target_has_atomic = "8")]
    1112    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1113    #[rustc_should_not_be_called_on_const_items]
    1114    pub fn fetch_nand(&self, val: bool, order: Ordering) -> bool {
    1115        // We can't use atomic_nand here because it can result in a bool with
    1116        // an invalid value. This happens because the atomic operation is done
    1117        // with an 8-bit integer internally, which would set the upper 7 bits.
    1118        // So we just use fetch_xor or swap instead.
    1119        if val {
    1120            // !(x & true) == !x
    1121            // We must invert the bool.
    1122            self.fetch_xor(true, order)
    1123        } else {
    1124            // !(x & false) == true
    1125            // We must set the bool to true.
    1126            self.swap(true, order)
    1127        }
    1128    }
    1129
    1130    /// Logical "or" with a boolean value.
    1131    ///
    1132    /// Performs a logical "or" operation on the current value and the argument `val`, and sets the
    1133    /// new value to the result.
    1134    ///
    1135    /// Returns the previous value.
    1136    ///
    1137    /// `fetch_or` takes an [`Ordering`] argument which describes the memory ordering
    1138    /// of this operation. All ordering modes are possible. Note that using
    1139    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    1140    /// using [`Release`] makes the load part [`Relaxed`].
    1141    ///
    1142    /// **Note:** This method is only available on platforms that support atomic
    1143    /// operations on `u8`.
    1144    ///
    1145    /// # Examples
    1146    ///
    1147    /// ```
    1148    /// use std::sync::atomic::{AtomicBool, Ordering};
    1149    ///
    1150    /// let foo = AtomicBool::new(true);
    1151    /// assert_eq!(foo.fetch_or(false, Ordering::SeqCst), true);
    1152    /// assert_eq!(foo.load(Ordering::SeqCst), true);
    1153    ///
    1154    /// let foo = AtomicBool::new(false);
    1155    /// assert_eq!(foo.fetch_or(true, Ordering::SeqCst), false);
    1156    /// assert_eq!(foo.load(Ordering::SeqCst), true);
    1157    ///
    1158    /// let foo = AtomicBool::new(false);
    1159    /// assert_eq!(foo.fetch_or(false, Ordering::SeqCst), false);
    1160    /// assert_eq!(foo.load(Ordering::SeqCst), false);
    1161    /// ```
    1162    #[inline]
    1163    #[stable(feature = "rust1", since = "1.0.0")]
    1164    #[cfg(target_has_atomic = "8")]
    1165    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1166    #[rustc_should_not_be_called_on_const_items]
    1167    pub fn fetch_or(&self, val: bool, order: Ordering) -> bool {
    1168        // SAFETY: data races are prevented by atomic intrinsics.
    1169        unsafe { atomic_or(self.v.get(), val as u8, order) != 0 }
    1170    }
    1171
    1172    /// Logical "xor" with a boolean value.
    1173    ///
    1174    /// Performs a logical "xor" operation on the current value and the argument `val`, and sets
    1175    /// the new value to the result.
    1176    ///
    1177    /// Returns the previous value.
    1178    ///
    1179    /// `fetch_xor` takes an [`Ordering`] argument which describes the memory ordering
    1180    /// of this operation. All ordering modes are possible. Note that using
    1181    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    1182    /// using [`Release`] makes the load part [`Relaxed`].
    1183    ///
    1184    /// **Note:** This method is only available on platforms that support atomic
    1185    /// operations on `u8`.
    1186    ///
    1187    /// # Examples
    1188    ///
    1189    /// ```
    1190    /// use std::sync::atomic::{AtomicBool, Ordering};
    1191    ///
    1192    /// let foo = AtomicBool::new(true);
    1193    /// assert_eq!(foo.fetch_xor(false, Ordering::SeqCst), true);
    1194    /// assert_eq!(foo.load(Ordering::SeqCst), true);
    1195    ///
    1196    /// let foo = AtomicBool::new(true);
    1197    /// assert_eq!(foo.fetch_xor(true, Ordering::SeqCst), true);
    1198    /// assert_eq!(foo.load(Ordering::SeqCst), false);
    1199    ///
    1200    /// let foo = AtomicBool::new(false);
    1201    /// assert_eq!(foo.fetch_xor(false, Ordering::SeqCst), false);
    1202    /// assert_eq!(foo.load(Ordering::SeqCst), false);
    1203    /// ```
    1204    #[inline]
    1205    #[stable(feature = "rust1", since = "1.0.0")]
    1206    #[cfg(target_has_atomic = "8")]
    1207    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1208    #[rustc_should_not_be_called_on_const_items]
    1209    pub fn fetch_xor(&self, val: bool, order: Ordering) -> bool {
    1210        // SAFETY: data races are prevented by atomic intrinsics.
    1211        unsafe { atomic_xor(self.v.get(), val as u8, order) != 0 }
    1212    }
    1213
    1214    /// Logical "not" with a boolean value.
    1215    ///
    1216    /// Performs a logical "not" operation on the current value, and sets
    1217    /// the new value to the result.
    1218    ///
    1219    /// Returns the previous value.
    1220    ///
    1221    /// `fetch_not` takes an [`Ordering`] argument which describes the memory ordering
    1222    /// of this operation. All ordering modes are possible. Note that using
    1223    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    1224    /// using [`Release`] makes the load part [`Relaxed`].
    1225    ///
    1226    /// **Note:** This method is only available on platforms that support atomic
    1227    /// operations on `u8`.
    1228    ///
    1229    /// # Examples
    1230    ///
    1231    /// ```
    1232    /// use std::sync::atomic::{AtomicBool, Ordering};
    1233    ///
    1234    /// let foo = AtomicBool::new(true);
    1235    /// assert_eq!(foo.fetch_not(Ordering::SeqCst), true);
    1236    /// assert_eq!(foo.load(Ordering::SeqCst), false);
    1237    ///
    1238    /// let foo = AtomicBool::new(false);
    1239    /// assert_eq!(foo.fetch_not(Ordering::SeqCst), false);
    1240    /// assert_eq!(foo.load(Ordering::SeqCst), true);
    1241    /// ```
    1242    #[inline]
    1243    #[stable(feature = "atomic_bool_fetch_not", since = "1.81.0")]
    1244    #[cfg(target_has_atomic = "8")]
    1245    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1246    #[rustc_should_not_be_called_on_const_items]
    1247    pub fn fetch_not(&self, order: Ordering) -> bool {
    1248        self.fetch_xor(true, order)
    1249    }
    1250
    1251    /// Returns a mutable pointer to the underlying [`bool`].
    1252    ///
    1253    /// Doing non-atomic reads and writes on the resulting boolean can be a data race.
    1254    /// This method is mostly useful for FFI, where the function signature may use
    1255    /// `*mut bool` instead of `&AtomicBool`.
    1256    ///
    1257    /// Returning an `*mut` pointer from a shared reference to this atomic is safe because the
    1258    /// atomic types work with interior mutability. All modifications of an atomic change the value
    1259    /// through a shared reference, and can do so safely as long as they use atomic operations. Any
    1260    /// use of the returned raw pointer requires an `unsafe` block and still has to uphold the
    1261    /// requirements of the [memory model].
    1262    ///
    1263    /// # Examples
    1264    ///
    1265    /// ```ignore (extern-declaration)
    1266    /// # fn main() {
    1267    /// use std::sync::atomic::AtomicBool;
    1268    ///
    1269    /// extern "C" {
    1270    ///     fn my_atomic_op(arg: *mut bool);
    1271    /// }
    1272    ///
    1273    /// let mut atomic = AtomicBool::new(true);
    1274    /// unsafe {
    1275    ///     my_atomic_op(atomic.as_ptr());
    1276    /// }
    1277    /// # }
    1278    /// ```
    1279    ///
    1280    /// [memory model]: self#memory-model-for-atomic-accesses
    1281    #[inline]
    1282    #[stable(feature = "atomic_as_ptr", since = "1.70.0")]
    1283    #[rustc_const_stable(feature = "atomic_as_ptr", since = "1.70.0")]
    1284    #[rustc_never_returns_null_ptr]
    1285    #[rustc_should_not_be_called_on_const_items]
    1286    pub const fn as_ptr(&self) -> *mut bool {
    1287        self.v.get().cast()
    1288    }
    1289
    1290    /// Fetches the value, and applies a function to it that returns an optional
    1291    /// new value. Returns a `Result` of `Ok(previous_value)` if the function
    1292    /// returned `Some(_)`, else `Err(previous_value)`.
    1293    ///
    1294    /// Note: This may call the function multiple times if the value has been
    1295    /// changed from other threads in the meantime, as long as the function
    1296    /// returns `Some(_)`, but the function will have been applied only once to
    1297    /// the stored value.
    1298    ///
    1299    /// `fetch_update` takes two [`Ordering`] arguments to describe the memory
    1300    /// ordering of this operation. The first describes the required ordering for
    1301    /// when the operation finally succeeds while the second describes the
    1302    /// required ordering for loads. These correspond to the success and failure
    1303    /// orderings of [`AtomicBool::compare_exchange`] respectively.
    1304    ///
    1305    /// Using [`Acquire`] as success ordering makes the store part of this
    1306    /// operation [`Relaxed`], and using [`Release`] makes the final successful
    1307    /// load [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`],
    1308    /// [`Acquire`] or [`Relaxed`].
    1309    ///
    1310    /// **Note:** This method is only available on platforms that support atomic
    1311    /// operations on `u8`.
    1312    ///
    1313    /// # Considerations
    1314    ///
    1315    /// This method is not magic; it is not provided by the hardware, and does not act like a
    1316    /// critical section or mutex.
    1317    ///
    1318    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    1319    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem].
    1320    ///
    1321    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    1322    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    1323    ///
    1324    /// # Examples
    1325    ///
    1326    /// ```rust
    1327    /// use std::sync::atomic::{AtomicBool, Ordering};
    1328    ///
    1329    /// let x = AtomicBool::new(false);
    1330    /// assert_eq!(x.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |_| None), Err(false));
    1331    /// assert_eq!(x.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(!x)), Ok(false));
    1332    /// assert_eq!(x.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(!x)), Ok(true));
    1333    /// assert_eq!(x.load(Ordering::SeqCst), false);
    1334    /// ```
    1335    #[inline]
    1336    #[stable(feature = "atomic_fetch_update", since = "1.53.0")]
    1337    #[cfg(target_has_atomic = "8")]
    1338    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1339    #[rustc_should_not_be_called_on_const_items]
    1340    pub fn fetch_update<F>(
    1341        &self,
    1342        set_order: Ordering,
    1343        fetch_order: Ordering,
    1344        mut f: F,
    1345    ) -> Result<bool, bool>
    1346    where
    1347        F: FnMut(bool) -> Option<bool>,
    1348    {
    1349        let mut prev = self.load(fetch_order);
    1350        while let Some(next) = f(prev) {
    1351            match self.compare_exchange_weak(prev, next, set_order, fetch_order) {
    1352                x @ Ok(_) => return x,
    1353                Err(next_prev) => prev = next_prev,
    1354            }
    1355        }
    1356        Err(prev)
    1357    }
    1358
    1359    /// Fetches the value, and applies a function to it that returns an optional
    1360    /// new value. Returns a `Result` of `Ok(previous_value)` if the function
    1361    /// returned `Some(_)`, else `Err(previous_value)`.
    1362    ///
    1363    /// See also: [`update`](`AtomicBool::update`).
    1364    ///
    1365    /// Note: This may call the function multiple times if the value has been
    1366    /// changed from other threads in the meantime, as long as the function
    1367    /// returns `Some(_)`, but the function will have been applied only once to
    1368    /// the stored value.
    1369    ///
    1370    /// `try_update` takes two [`Ordering`] arguments to describe the memory
    1371    /// ordering of this operation. The first describes the required ordering for
    1372    /// when the operation finally succeeds while the second describes the
    1373    /// required ordering for loads. These correspond to the success and failure
    1374    /// orderings of [`AtomicBool::compare_exchange`] respectively.
    1375    ///
    1376    /// Using [`Acquire`] as success ordering makes the store part of this
    1377    /// operation [`Relaxed`], and using [`Release`] makes the final successful
    1378    /// load [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`],
    1379    /// [`Acquire`] or [`Relaxed`].
    1380    ///
    1381    /// **Note:** This method is only available on platforms that support atomic
    1382    /// operations on `u8`.
    1383    ///
    1384    /// # Considerations
    1385    ///
    1386    /// This method is not magic; it is not provided by the hardware, and does not act like a
    1387    /// critical section or mutex.
    1388    ///
    1389    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    1390    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem].
    1391    ///
    1392    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    1393    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    1394    ///
    1395    /// # Examples
    1396    ///
    1397    /// ```rust
    1398    /// #![feature(atomic_try_update)]
    1399    /// use std::sync::atomic::{AtomicBool, Ordering};
    1400    ///
    1401    /// let x = AtomicBool::new(false);
    1402    /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |_| None), Err(false));
    1403    /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(!x)), Ok(false));
    1404    /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(!x)), Ok(true));
    1405    /// assert_eq!(x.load(Ordering::SeqCst), false);
    1406    /// ```
    1407    #[inline]
    1408    #[unstable(feature = "atomic_try_update", issue = "135894")]
    1409    #[cfg(target_has_atomic = "8")]
    1410    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1411    #[rustc_should_not_be_called_on_const_items]
    1412    pub fn try_update(
    1413        &self,
    1414        set_order: Ordering,
    1415        fetch_order: Ordering,
    1416        f: impl FnMut(bool) -> Option<bool>,
    1417    ) -> Result<bool, bool> {
    1418        // FIXME(atomic_try_update): this is currently an unstable alias to `fetch_update`;
    1419        //      when stabilizing, turn `fetch_update` into a deprecated alias to `try_update`.
    1420        self.fetch_update(set_order, fetch_order, f)
    1421    }
    1422
    1423    /// Fetches the value, applies a function to it that it return a new value.
    1424    /// The new value is stored and the old value is returned.
    1425    ///
    1426    /// See also: [`try_update`](`AtomicBool::try_update`).
    1427    ///
    1428    /// Note: This may call the function multiple times if the value has been changed from other threads in
    1429    /// the meantime, but the function will have been applied only once to the stored value.
    1430    ///
    1431    /// `update` takes two [`Ordering`] arguments to describe the memory
    1432    /// ordering of this operation. The first describes the required ordering for
    1433    /// when the operation finally succeeds while the second describes the
    1434    /// required ordering for loads. These correspond to the success and failure
    1435    /// orderings of [`AtomicBool::compare_exchange`] respectively.
    1436    ///
    1437    /// Using [`Acquire`] as success ordering makes the store part
    1438    /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
    1439    /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    1440    ///
    1441    /// **Note:** This method is only available on platforms that support atomic operations on `u8`.
    1442    ///
    1443    /// # Considerations
    1444    ///
    1445    /// This method is not magic; it is not provided by the hardware, and does not act like a
    1446    /// critical section or mutex.
    1447    ///
    1448    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    1449    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem].
    1450    ///
    1451    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    1452    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    1453    ///
    1454    /// # Examples
    1455    ///
    1456    /// ```rust
    1457    /// #![feature(atomic_try_update)]
    1458    ///
    1459    /// use std::sync::atomic::{AtomicBool, Ordering};
    1460    ///
    1461    /// let x = AtomicBool::new(false);
    1462    /// assert_eq!(x.update(Ordering::SeqCst, Ordering::SeqCst, |x| !x), false);
    1463    /// assert_eq!(x.update(Ordering::SeqCst, Ordering::SeqCst, |x| !x), true);
    1464    /// assert_eq!(x.load(Ordering::SeqCst), false);
    1465    /// ```
    1466    #[inline]
    1467    #[unstable(feature = "atomic_try_update", issue = "135894")]
    1468    #[cfg(target_has_atomic = "8")]
    1469    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1470    #[rustc_should_not_be_called_on_const_items]
    1471    pub fn update(
    1472        &self,
    1473        set_order: Ordering,
    1474        fetch_order: Ordering,
    1475        mut f: impl FnMut(bool) -> bool,
    1476    ) -> bool {
    1477        let mut prev = self.load(fetch_order);
    1478        loop {
    1479            match self.compare_exchange_weak(prev, f(prev), set_order, fetch_order) {
    1480                Ok(x) => break x,
    1481                Err(next_prev) => prev = next_prev,
    1482            }
    1483        }
    1484    }
    1485}
    1486
    1487#[cfg(target_has_atomic_load_store = "ptr")]
    1488impl<T> AtomicPtr<T> {
    1489    /// Creates a new `AtomicPtr`.
    1490    ///
    1491    /// # Examples
    1492    ///
    1493    /// ```
    1494    /// use std::sync::atomic::AtomicPtr;
    1495    ///
    1496    /// let ptr = &mut 5;
    1497    /// let atomic_ptr = AtomicPtr::new(ptr);
    1498    /// ```
    1499    #[inline]
    1500    #[stable(feature = "rust1", since = "1.0.0")]
    1501    #[rustc_const_stable(feature = "const_atomic_new", since = "1.24.0")]
    1502    pub const fn new(p: *mut T) -> AtomicPtr<T> {
    1503        AtomicPtr { p: UnsafeCell::new(p) }
    1504    }
    1505
    1506    /// Creates a new `AtomicPtr` from a pointer.
    1507    ///
    1508    /// # Examples
    1509    ///
    1510    /// ```
    1511    /// use std::sync::atomic::{self, AtomicPtr};
    1512    ///
    1513    /// // Get a pointer to an allocated value
    1514    /// let ptr: *mut *mut u8 = Box::into_raw(Box::new(std::ptr::null_mut()));
    1515    ///
    1516    /// assert!(ptr.cast::<AtomicPtr<u8>>().is_aligned());
    1517    ///
    1518    /// {
    1519    ///     // Create an atomic view of the allocated value
    1520    ///     let atomic = unsafe { AtomicPtr::from_ptr(ptr) };
    1521    ///
    1522    ///     // Use `atomic` for atomic operations, possibly share it with other threads
    1523    ///     atomic.store(std::ptr::NonNull::dangling().as_ptr(), atomic::Ordering::Relaxed);
    1524    /// }
    1525    ///
    1526    /// // It's ok to non-atomically access the value behind `ptr`,
    1527    /// // since the reference to the atomic ended its lifetime in the block above
    1528    /// assert!(!unsafe { *ptr }.is_null());
    1529    ///
    1530    /// // Deallocate the value
    1531    /// unsafe { drop(Box::from_raw(ptr)) }
    1532    /// ```
    1533    ///
    1534    /// # Safety
    1535    ///
    1536    /// * `ptr` must be aligned to `align_of::<AtomicPtr<T>>()` (note that on some platforms this
    1537    ///   can be bigger than `align_of::<*mut T>()`).
    1538    /// * `ptr` must be [valid] for both reads and writes for the whole lifetime `'a`.
    1539    /// * You must adhere to the [Memory model for atomic accesses]. In particular, it is not
    1540    ///   allowed to mix conflicting atomic and non-atomic accesses, or atomic accesses of different
    1541    ///   sizes, without synchronization.
    1542    ///
    1543    /// [valid]: crate::ptr#safety
    1544    /// [Memory model for atomic accesses]: self#memory-model-for-atomic-accesses
    1545    #[inline]
    1546    #[stable(feature = "atomic_from_ptr", since = "1.75.0")]
    1547    #[rustc_const_stable(feature = "const_atomic_from_ptr", since = "1.84.0")]
    1548    pub const unsafe fn from_ptr<'a>(ptr: *mut *mut T) -> &'a AtomicPtr<T> {
    1549        // SAFETY: guaranteed by the caller
    1550        unsafe { &*ptr.cast() }
    1551    }
    1552
    1553    /// Creates a new `AtomicPtr` initialized with a null pointer.
    1554    ///
    1555    /// # Examples
    1556    ///
    1557    /// ```
    1558    /// #![feature(atomic_ptr_null)]
    1559    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1560    ///
    1561    /// let atomic_ptr = AtomicPtr::<()>::null();
    1562    /// assert!(atomic_ptr.load(Ordering::Relaxed).is_null());
    1563    /// ```
    1564    #[inline]
    1565    #[must_use]
    1566    #[unstable(feature = "atomic_ptr_null", issue = "150733")]
    1567    pub const fn null() -> AtomicPtr<T> {
    1568        AtomicPtr::new(crate::ptr::null_mut())
    1569    }
    1570
    1571    /// Returns a mutable reference to the underlying pointer.
    1572    ///
    1573    /// This is safe because the mutable reference guarantees that no other threads are
    1574    /// concurrently accessing the atomic data.
    1575    ///
    1576    /// # Examples
    1577    ///
    1578    /// ```
    1579    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1580    ///
    1581    /// let mut data = 10;
    1582    /// let mut atomic_ptr = AtomicPtr::new(&mut data);
    1583    /// let mut other_data = 5;
    1584    /// *atomic_ptr.get_mut() = &mut other_data;
    1585    /// assert_eq!(unsafe { *atomic_ptr.load(Ordering::SeqCst) }, 5);
    1586    /// ```
    1587    #[inline]
    1588    #[stable(feature = "atomic_access", since = "1.15.0")]
    1589    pub fn get_mut(&mut self) -> &mut *mut T {
    1590        self.p.get_mut()
    1591    }
    1592
    1593    /// Gets atomic access to a pointer.
    1594    ///
    1595    /// **Note:** This function is only available on targets where `AtomicPtr<T>` has the same alignment as `*const T`
    1596    ///
    1597    /// # Examples
    1598    ///
    1599    /// ```
    1600    /// #![feature(atomic_from_mut)]
    1601    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1602    ///
    1603    /// let mut data = 123;
    1604    /// let mut some_ptr = &mut data as *mut i32;
    1605    /// let a = AtomicPtr::from_mut(&mut some_ptr);
    1606    /// let mut other_data = 456;
    1607    /// a.store(&mut other_data, Ordering::Relaxed);
    1608    /// assert_eq!(unsafe { *some_ptr }, 456);
    1609    /// ```
    1610    #[inline]
    1611    #[cfg(target_has_atomic_equal_alignment = "ptr")]
    1612    #[unstable(feature = "atomic_from_mut", issue = "76314")]
    1613    pub fn from_mut(v: &mut *mut T) -> &mut Self {
    1614        let [] = [(); align_of::<AtomicPtr<()>>() - align_of::<*mut ()>()];
    1615        // SAFETY:
    1616        //  - the mutable reference guarantees unique ownership.
    1617        //  - the alignment of `*mut T` and `Self` is the same on all platforms
    1618        //    supported by rust, as verified above.
    1619        unsafe { &mut *(v as *mut *mut T as *mut Self) }
    1620    }
    1621
    1622    /// Gets non-atomic access to a `&mut [AtomicPtr]` slice.
    1623    ///
    1624    /// This is safe because the mutable reference guarantees that no other threads are
    1625    /// concurrently accessing the atomic data.
    1626    ///
    1627    /// # Examples
    1628    ///
    1629    /// ```ignore-wasm
    1630    /// #![feature(atomic_from_mut)]
    1631    /// use std::ptr::null_mut;
    1632    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1633    ///
    1634    /// let mut some_ptrs = [const { AtomicPtr::new(null_mut::<String>()) }; 10];
    1635    ///
    1636    /// let view: &mut [*mut String] = AtomicPtr::get_mut_slice(&mut some_ptrs);
    1637    /// assert_eq!(view, [null_mut::<String>(); 10]);
    1638    /// view
    1639    ///     .iter_mut()
    1640    ///     .enumerate()
    1641    ///     .for_each(|(i, ptr)| *ptr = Box::into_raw(Box::new(format!("iteration#{i}"))));
    1642    ///
    1643    /// std::thread::scope(|s| {
    1644    ///     for ptr in &some_ptrs {
    1645    ///         s.spawn(move || {
    1646    ///             let ptr = ptr.load(Ordering::Relaxed);
    1647    ///             assert!(!ptr.is_null());
    1648    ///
    1649    ///             let name = unsafe { Box::from_raw(ptr) };
    1650    ///             println!("Hello, {name}!");
    1651    ///         });
    1652    ///     }
    1653    /// });
    1654    /// ```
    1655    #[inline]
    1656    #[unstable(feature = "atomic_from_mut", issue = "76314")]
    1657    pub fn get_mut_slice(this: &mut [Self]) -> &mut [*mut T] {
    1658        // SAFETY: the mutable reference guarantees unique ownership.
    1659        unsafe { &mut *(this as *mut [Self] as *mut [*mut T]) }
    1660    }
    1661
    1662    /// Gets atomic access to a slice of pointers.
    1663    ///
    1664    /// **Note:** This function is only available on targets where `AtomicPtr<T>` has the same alignment as `*const T`
    1665    ///
    1666    /// # Examples
    1667    ///
    1668    /// ```ignore-wasm
    1669    /// #![feature(atomic_from_mut)]
    1670    /// use std::ptr::null_mut;
    1671    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1672    ///
    1673    /// let mut some_ptrs = [null_mut::<String>(); 10];
    1674    /// let a = &*AtomicPtr::from_mut_slice(&mut some_ptrs);
    1675    /// std::thread::scope(|s| {
    1676    ///     for i in 0..a.len() {
    1677    ///         s.spawn(move || {
    1678    ///             let name = Box::new(format!("thread{i}"));
    1679    ///             a[i].store(Box::into_raw(name), Ordering::Relaxed);
    1680    ///         });
    1681    ///     }
    1682    /// });
    1683    /// for p in some_ptrs {
    1684    ///     assert!(!p.is_null());
    1685    ///     let name = unsafe { Box::from_raw(p) };
    1686    ///     println!("Hello, {name}!");
    1687    /// }
    1688    /// ```
    1689    #[inline]
    1690    #[cfg(target_has_atomic_equal_alignment = "ptr")]
    1691    #[unstable(feature = "atomic_from_mut", issue = "76314")]
    1692    pub fn from_mut_slice(v: &mut [*mut T]) -> &mut [Self] {
    1693        // SAFETY:
    1694        //  - the mutable reference guarantees unique ownership.
    1695        //  - the alignment of `*mut T` and `Self` is the same on all platforms
    1696        //    supported by rust, as verified above.
    1697        unsafe { &mut *(v as *mut [*mut T] as *mut [Self]) }
    1698    }
    1699
    1700    /// Consumes the atomic and returns the contained value.
    1701    ///
    1702    /// This is safe because passing `self` by value guarantees that no other threads are
    1703    /// concurrently accessing the atomic data.
    1704    ///
    1705    /// # Examples
    1706    ///
    1707    /// ```
    1708    /// use std::sync::atomic::AtomicPtr;
    1709    ///
    1710    /// let mut data = 5;
    1711    /// let atomic_ptr = AtomicPtr::new(&mut data);
    1712    /// assert_eq!(unsafe { *atomic_ptr.into_inner() }, 5);
    1713    /// ```
    1714    #[inline]
    1715    #[stable(feature = "atomic_access", since = "1.15.0")]
    1716    #[rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0")]
    1717    pub const fn into_inner(self) -> *mut T {
    1718        self.p.into_inner()
    1719    }
    1720
    1721    /// Loads a value from the pointer.
    1722    ///
    1723    /// `load` takes an [`Ordering`] argument which describes the memory ordering
    1724    /// of this operation. Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
    1725    ///
    1726    /// # Panics
    1727    ///
    1728    /// Panics if `order` is [`Release`] or [`AcqRel`].
    1729    ///
    1730    /// # Examples
    1731    ///
    1732    /// ```
    1733    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1734    ///
    1735    /// let ptr = &mut 5;
    1736    /// let some_ptr = AtomicPtr::new(ptr);
    1737    ///
    1738    /// let value = some_ptr.load(Ordering::Relaxed);
    1739    /// ```
    1740    #[inline]
    1741    #[stable(feature = "rust1", since = "1.0.0")]
    1742    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1743    pub fn load(&self, order: Ordering) -> *mut T {
    1744        // SAFETY: data races are prevented by atomic intrinsics.
    1745        unsafe { atomic_load(self.p.get(), order) }
    1746    }
    1747
    1748    /// Stores a value into the pointer.
    1749    ///
    1750    /// `store` takes an [`Ordering`] argument which describes the memory ordering
    1751    /// of this operation. Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
    1752    ///
    1753    /// # Panics
    1754    ///
    1755    /// Panics if `order` is [`Acquire`] or [`AcqRel`].
    1756    ///
    1757    /// # Examples
    1758    ///
    1759    /// ```
    1760    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1761    ///
    1762    /// let ptr = &mut 5;
    1763    /// let some_ptr = AtomicPtr::new(ptr);
    1764    ///
    1765    /// let other_ptr = &mut 10;
    1766    ///
    1767    /// some_ptr.store(other_ptr, Ordering::Relaxed);
    1768    /// ```
    1769    #[inline]
    1770    #[stable(feature = "rust1", since = "1.0.0")]
    1771    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1772    #[rustc_should_not_be_called_on_const_items]
    1773    pub fn store(&self, ptr: *mut T, order: Ordering) {
    1774        // SAFETY: data races are prevented by atomic intrinsics.
    1775        unsafe {
    1776            atomic_store(self.p.get(), ptr, order);
    1777        }
    1778    }
    1779
    1780    /// Stores a value into the pointer, returning the previous value.
    1781    ///
    1782    /// `swap` takes an [`Ordering`] argument which describes the memory ordering
    1783    /// of this operation. All ordering modes are possible. Note that using
    1784    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    1785    /// using [`Release`] makes the load part [`Relaxed`].
    1786    ///
    1787    /// **Note:** This method is only available on platforms that support atomic
    1788    /// operations on pointers.
    1789    ///
    1790    /// # Examples
    1791    ///
    1792    /// ```
    1793    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1794    ///
    1795    /// let ptr = &mut 5;
    1796    /// let some_ptr = AtomicPtr::new(ptr);
    1797    ///
    1798    /// let other_ptr = &mut 10;
    1799    ///
    1800    /// let value = some_ptr.swap(other_ptr, Ordering::Relaxed);
    1801    /// ```
    1802    #[inline]
    1803    #[stable(feature = "rust1", since = "1.0.0")]
    1804    #[cfg(target_has_atomic = "ptr")]
    1805    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1806    #[rustc_should_not_be_called_on_const_items]
    1807    pub fn swap(&self, ptr: *mut T, order: Ordering) -> *mut T {
    1808        // SAFETY: data races are prevented by atomic intrinsics.
    1809        unsafe { atomic_swap(self.p.get(), ptr, order) }
    1810    }
    1811
    1812    /// Stores a value into the pointer if the current value is the same as the `current` value.
    1813    ///
    1814    /// The return value is always the previous value. If it is equal to `current`, then the value
    1815    /// was updated.
    1816    ///
    1817    /// `compare_and_swap` also takes an [`Ordering`] argument which describes the memory
    1818    /// ordering of this operation. Notice that even when using [`AcqRel`], the operation
    1819    /// might fail and hence just perform an `Acquire` load, but not have `Release` semantics.
    1820    /// Using [`Acquire`] makes the store part of this operation [`Relaxed`] if it
    1821    /// happens, and using [`Release`] makes the load part [`Relaxed`].
    1822    ///
    1823    /// **Note:** This method is only available on platforms that support atomic
    1824    /// operations on pointers.
    1825    ///
    1826    /// # Migrating to `compare_exchange` and `compare_exchange_weak`
    1827    ///
    1828    /// `compare_and_swap` is equivalent to `compare_exchange` with the following mapping for
    1829    /// memory orderings:
    1830    ///
    1831    /// Original | Success | Failure
    1832    /// -------- | ------- | -------
    1833    /// Relaxed  | Relaxed | Relaxed
    1834    /// Acquire  | Acquire | Acquire
    1835    /// Release  | Release | Relaxed
    1836    /// AcqRel   | AcqRel  | Acquire
    1837    /// SeqCst   | SeqCst  | SeqCst
    1838    ///
    1839    /// `compare_and_swap` and `compare_exchange` also differ in their return type. You can use
    1840    /// `compare_exchange(...).unwrap_or_else(|x| x)` to recover the behavior of `compare_and_swap`,
    1841    /// but in most cases it is more idiomatic to check whether the return value is `Ok` or `Err`
    1842    /// rather than to infer success vs failure based on the value that was read.
    1843    ///
    1844    /// During migration, consider whether it makes sense to use `compare_exchange_weak` instead.
    1845    /// `compare_exchange_weak` is allowed to fail spuriously even when the comparison succeeds,
    1846    /// which allows the compiler to generate better assembly code when the compare and swap
    1847    /// is used in a loop.
    1848    ///
    1849    /// # Examples
    1850    ///
    1851    /// ```
    1852    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1853    ///
    1854    /// let ptr = &mut 5;
    1855    /// let some_ptr = AtomicPtr::new(ptr);
    1856    ///
    1857    /// let other_ptr = &mut 10;
    1858    ///
    1859    /// let value = some_ptr.compare_and_swap(ptr, other_ptr, Ordering::Relaxed);
    1860    /// ```
    1861    #[inline]
    1862    #[stable(feature = "rust1", since = "1.0.0")]
    1863    #[deprecated(
    1864        since = "1.50.0",
    1865        note = "Use `compare_exchange` or `compare_exchange_weak` instead"
    1866    )]
    1867    #[cfg(target_has_atomic = "ptr")]
    1868    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1869    #[rustc_should_not_be_called_on_const_items]
    1870    pub fn compare_and_swap(&self, current: *mut T, new: *mut T, order: Ordering) -> *mut T {
    1871        match self.compare_exchange(current, new, order, strongest_failure_ordering(order)) {
    1872            Ok(x) => x,
    1873            Err(x) => x,
    1874        }
    1875    }
    1876
    1877    /// Stores a value into the pointer if the current value is the same as the `current` value.
    1878    ///
    1879    /// The return value is a result indicating whether the new value was written and containing
    1880    /// the previous value. On success this value is guaranteed to be equal to `current`.
    1881    ///
    1882    /// `compare_exchange` takes two [`Ordering`] arguments to describe the memory
    1883    /// ordering of this operation. `success` describes the required ordering for the
    1884    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
    1885    /// `failure` describes the required ordering for the load operation that takes place when
    1886    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
    1887    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
    1888    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    1889    ///
    1890    /// **Note:** This method is only available on platforms that support atomic
    1891    /// operations on pointers.
    1892    ///
    1893    /// # Examples
    1894    ///
    1895    /// ```
    1896    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1897    ///
    1898    /// let ptr = &mut 5;
    1899    /// let some_ptr = AtomicPtr::new(ptr);
    1900    ///
    1901    /// let other_ptr = &mut 10;
    1902    ///
    1903    /// let value = some_ptr.compare_exchange(ptr, other_ptr,
    1904    ///                                       Ordering::SeqCst, Ordering::Relaxed);
    1905    /// ```
    1906    ///
    1907    /// # Considerations
    1908    ///
    1909    /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
    1910    /// of CAS operations. In particular, a load of the value followed by a successful
    1911    /// `compare_exchange` with the previous load *does not ensure* that other threads have not
    1912    /// changed the value in the interim. This is usually important when the *equality* check in
    1913    /// the `compare_exchange` is being used to check the *identity* of a value, but equality
    1914    /// does not necessarily imply identity. This is a particularly common case for pointers, as
    1915    /// a pointer holding the same address does not imply that the same object exists at that
    1916    /// address! In this case, `compare_exchange` can lead to the [ABA problem].
    1917    ///
    1918    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    1919    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    1920    #[inline]
    1921    #[stable(feature = "extended_compare_and_swap", since = "1.10.0")]
    1922    #[cfg(target_has_atomic = "ptr")]
    1923    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1924    #[rustc_should_not_be_called_on_const_items]
    1925    pub fn compare_exchange(
    1926        &self,
    1927        current: *mut T,
    1928        new: *mut T,
    1929        success: Ordering,
    1930        failure: Ordering,
    1931    ) -> Result<*mut T, *mut T> {
    1932        // SAFETY: data races are prevented by atomic intrinsics.
    1933        unsafe { atomic_compare_exchange(self.p.get(), current, new, success, failure) }
    1934    }
    1935
    1936    /// Stores a value into the pointer if the current value is the same as the `current` value.
    1937    ///
    1938    /// Unlike [`AtomicPtr::compare_exchange`], this function is allowed to spuriously fail even when the
    1939    /// comparison succeeds, which can result in more efficient code on some platforms. The
    1940    /// return value is a result indicating whether the new value was written and containing the
    1941    /// previous value.
    1942    ///
    1943    /// `compare_exchange_weak` takes two [`Ordering`] arguments to describe the memory
    1944    /// ordering of this operation. `success` describes the required ordering for the
    1945    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
    1946    /// `failure` describes the required ordering for the load operation that takes place when
    1947    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
    1948    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
    1949    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    1950    ///
    1951    /// **Note:** This method is only available on platforms that support atomic
    1952    /// operations on pointers.
    1953    ///
    1954    /// # Examples
    1955    ///
    1956    /// ```
    1957    /// use std::sync::atomic::{AtomicPtr, Ordering};
    1958    ///
    1959    /// let some_ptr = AtomicPtr::new(&mut 5);
    1960    ///
    1961    /// let new = &mut 10;
    1962    /// let mut old = some_ptr.load(Ordering::Relaxed);
    1963    /// loop {
    1964    ///     match some_ptr.compare_exchange_weak(old, new, Ordering::SeqCst, Ordering::Relaxed) {
    1965    ///         Ok(_) => break,
    1966    ///         Err(x) => old = x,
    1967    ///     }
    1968    /// }
    1969    /// ```
    1970    ///
    1971    /// # Considerations
    1972    ///
    1973    /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
    1974    /// of CAS operations. In particular, a load of the value followed by a successful
    1975    /// `compare_exchange` with the previous load *does not ensure* that other threads have not
    1976    /// changed the value in the interim. This is usually important when the *equality* check in
    1977    /// the `compare_exchange` is being used to check the *identity* of a value, but equality
    1978    /// does not necessarily imply identity. This is a particularly common case for pointers, as
    1979    /// a pointer holding the same address does not imply that the same object exists at that
    1980    /// address! In this case, `compare_exchange` can lead to the [ABA problem].
    1981    ///
    1982    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    1983    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    1984    #[inline]
    1985    #[stable(feature = "extended_compare_and_swap", since = "1.10.0")]
    1986    #[cfg(target_has_atomic = "ptr")]
    1987    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    1988    #[rustc_should_not_be_called_on_const_items]
    1989    pub fn compare_exchange_weak(
    1990        &self,
    1991        current: *mut T,
    1992        new: *mut T,
    1993        success: Ordering,
    1994        failure: Ordering,
    1995    ) -> Result<*mut T, *mut T> {
    1996        // SAFETY: This intrinsic is unsafe because it operates on a raw pointer
    1997        // but we know for sure that the pointer is valid (we just got it from
    1998        // an `UnsafeCell` that we have by reference) and the atomic operation
    1999        // itself allows us to safely mutate the `UnsafeCell` contents.
    2000        unsafe { atomic_compare_exchange_weak(self.p.get(), current, new, success, failure) }
    2001    }
    2002
    2003    /// Fetches the value, and applies a function to it that returns an optional
    2004    /// new value. Returns a `Result` of `Ok(previous_value)` if the function
    2005    /// returned `Some(_)`, else `Err(previous_value)`.
    2006    ///
    2007    /// Note: This may call the function multiple times if the value has been
    2008    /// changed from other threads in the meantime, as long as the function
    2009    /// returns `Some(_)`, but the function will have been applied only once to
    2010    /// the stored value.
    2011    ///
    2012    /// `fetch_update` takes two [`Ordering`] arguments to describe the memory
    2013    /// ordering of this operation. The first describes the required ordering for
    2014    /// when the operation finally succeeds while the second describes the
    2015    /// required ordering for loads. These correspond to the success and failure
    2016    /// orderings of [`AtomicPtr::compare_exchange`] respectively.
    2017    ///
    2018    /// Using [`Acquire`] as success ordering makes the store part of this
    2019    /// operation [`Relaxed`], and using [`Release`] makes the final successful
    2020    /// load [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`],
    2021    /// [`Acquire`] or [`Relaxed`].
    2022    ///
    2023    /// **Note:** This method is only available on platforms that support atomic
    2024    /// operations on pointers.
    2025    ///
    2026    /// # Considerations
    2027    ///
    2028    /// This method is not magic; it is not provided by the hardware, and does not act like a
    2029    /// critical section or mutex.
    2030    ///
    2031    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    2032    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem],
    2033    /// which is a particularly common pitfall for pointers!
    2034    ///
    2035    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    2036    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    2037    ///
    2038    /// # Examples
    2039    ///
    2040    /// ```rust
    2041    /// use std::sync::atomic::{AtomicPtr, Ordering};
    2042    ///
    2043    /// let ptr: *mut _ = &mut 5;
    2044    /// let some_ptr = AtomicPtr::new(ptr);
    2045    ///
    2046    /// let new: *mut _ = &mut 10;
    2047    /// assert_eq!(some_ptr.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |_| None), Err(ptr));
    2048    /// let result = some_ptr.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |x| {
    2049    ///     if x == ptr {
    2050    ///         Some(new)
    2051    ///     } else {
    2052    ///         None
    2053    ///     }
    2054    /// });
    2055    /// assert_eq!(result, Ok(ptr));
    2056    /// assert_eq!(some_ptr.load(Ordering::SeqCst), new);
    2057    /// ```
    2058    #[inline]
    2059    #[stable(feature = "atomic_fetch_update", since = "1.53.0")]
    2060    #[cfg(target_has_atomic = "ptr")]
    2061    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2062    #[rustc_should_not_be_called_on_const_items]
    2063    pub fn fetch_update<F>(
    2064        &self,
    2065        set_order: Ordering,
    2066        fetch_order: Ordering,
    2067        mut f: F,
    2068    ) -> Result<*mut T, *mut T>
    2069    where
    2070        F: FnMut(*mut T) -> Option<*mut T>,
    2071    {
    2072        let mut prev = self.load(fetch_order);
    2073        while let Some(next) = f(prev) {
    2074            match self.compare_exchange_weak(prev, next, set_order, fetch_order) {
    2075                x @ Ok(_) => return x,
    2076                Err(next_prev) => prev = next_prev,
    2077            }
    2078        }
    2079        Err(prev)
    2080    }
    2081    /// Fetches the value, and applies a function to it that returns an optional
    2082    /// new value. Returns a `Result` of `Ok(previous_value)` if the function
    2083    /// returned `Some(_)`, else `Err(previous_value)`.
    2084    ///
    2085    /// See also: [`update`](`AtomicPtr::update`).
    2086    ///
    2087    /// Note: This may call the function multiple times if the value has been
    2088    /// changed from other threads in the meantime, as long as the function
    2089    /// returns `Some(_)`, but the function will have been applied only once to
    2090    /// the stored value.
    2091    ///
    2092    /// `try_update` takes two [`Ordering`] arguments to describe the memory
    2093    /// ordering of this operation. The first describes the required ordering for
    2094    /// when the operation finally succeeds while the second describes the
    2095    /// required ordering for loads. These correspond to the success and failure
    2096    /// orderings of [`AtomicPtr::compare_exchange`] respectively.
    2097    ///
    2098    /// Using [`Acquire`] as success ordering makes the store part of this
    2099    /// operation [`Relaxed`], and using [`Release`] makes the final successful
    2100    /// load [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`],
    2101    /// [`Acquire`] or [`Relaxed`].
    2102    ///
    2103    /// **Note:** This method is only available on platforms that support atomic
    2104    /// operations on pointers.
    2105    ///
    2106    /// # Considerations
    2107    ///
    2108    /// This method is not magic; it is not provided by the hardware, and does not act like a
    2109    /// critical section or mutex.
    2110    ///
    2111    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    2112    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem],
    2113    /// which is a particularly common pitfall for pointers!
    2114    ///
    2115    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    2116    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    2117    ///
    2118    /// # Examples
    2119    ///
    2120    /// ```rust
    2121    /// #![feature(atomic_try_update)]
    2122    /// use std::sync::atomic::{AtomicPtr, Ordering};
    2123    ///
    2124    /// let ptr: *mut _ = &mut 5;
    2125    /// let some_ptr = AtomicPtr::new(ptr);
    2126    ///
    2127    /// let new: *mut _ = &mut 10;
    2128    /// assert_eq!(some_ptr.try_update(Ordering::SeqCst, Ordering::SeqCst, |_| None), Err(ptr));
    2129    /// let result = some_ptr.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| {
    2130    ///     if x == ptr {
    2131    ///         Some(new)
    2132    ///     } else {
    2133    ///         None
    2134    ///     }
    2135    /// });
    2136    /// assert_eq!(result, Ok(ptr));
    2137    /// assert_eq!(some_ptr.load(Ordering::SeqCst), new);
    2138    /// ```
    2139    #[inline]
    2140    #[unstable(feature = "atomic_try_update", issue = "135894")]
    2141    #[cfg(target_has_atomic = "ptr")]
    2142    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2143    #[rustc_should_not_be_called_on_const_items]
    2144    pub fn try_update(
    2145        &self,
    2146        set_order: Ordering,
    2147        fetch_order: Ordering,
    2148        f: impl FnMut(*mut T) -> Option<*mut T>,
    2149    ) -> Result<*mut T, *mut T> {
    2150        // FIXME(atomic_try_update): this is currently an unstable alias to `fetch_update`;
    2151        //      when stabilizing, turn `fetch_update` into a deprecated alias to `try_update`.
    2152        self.fetch_update(set_order, fetch_order, f)
    2153    }
    2154
    2155    /// Fetches the value, applies a function to it that it return a new value.
    2156    /// The new value is stored and the old value is returned.
    2157    ///
    2158    /// See also: [`try_update`](`AtomicPtr::try_update`).
    2159    ///
    2160    /// Note: This may call the function multiple times if the value has been changed from other threads in
    2161    /// the meantime, but the function will have been applied only once to the stored value.
    2162    ///
    2163    /// `update` takes two [`Ordering`] arguments to describe the memory
    2164    /// ordering of this operation. The first describes the required ordering for
    2165    /// when the operation finally succeeds while the second describes the
    2166    /// required ordering for loads. These correspond to the success and failure
    2167    /// orderings of [`AtomicPtr::compare_exchange`] respectively.
    2168    ///
    2169    /// Using [`Acquire`] as success ordering makes the store part
    2170    /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
    2171    /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    2172    ///
    2173    /// **Note:** This method is only available on platforms that support atomic
    2174    /// operations on pointers.
    2175    ///
    2176    /// # Considerations
    2177    ///
    2178    /// This method is not magic; it is not provided by the hardware, and does not act like a
    2179    /// critical section or mutex.
    2180    ///
    2181    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    2182    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem],
    2183    /// which is a particularly common pitfall for pointers!
    2184    ///
    2185    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    2186    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    2187    ///
    2188    /// # Examples
    2189    ///
    2190    /// ```rust
    2191    /// #![feature(atomic_try_update)]
    2192    ///
    2193    /// use std::sync::atomic::{AtomicPtr, Ordering};
    2194    ///
    2195    /// let ptr: *mut _ = &mut 5;
    2196    /// let some_ptr = AtomicPtr::new(ptr);
    2197    ///
    2198    /// let new: *mut _ = &mut 10;
    2199    /// let result = some_ptr.update(Ordering::SeqCst, Ordering::SeqCst, |_| new);
    2200    /// assert_eq!(result, ptr);
    2201    /// assert_eq!(some_ptr.load(Ordering::SeqCst), new);
    2202    /// ```
    2203    #[inline]
    2204    #[unstable(feature = "atomic_try_update", issue = "135894")]
    2205    #[cfg(target_has_atomic = "8")]
    2206    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2207    #[rustc_should_not_be_called_on_const_items]
    2208    pub fn update(
    2209        &self,
    2210        set_order: Ordering,
    2211        fetch_order: Ordering,
    2212        mut f: impl FnMut(*mut T) -> *mut T,
    2213    ) -> *mut T {
    2214        let mut prev = self.load(fetch_order);
    2215        loop {
    2216            match self.compare_exchange_weak(prev, f(prev), set_order, fetch_order) {
    2217                Ok(x) => break x,
    2218                Err(next_prev) => prev = next_prev,
    2219            }
    2220        }
    2221    }
    2222
    2223    /// Offsets the pointer's address by adding `val` (in units of `T`),
    2224    /// returning the previous pointer.
    2225    ///
    2226    /// This is equivalent to using [`wrapping_add`] to atomically perform the
    2227    /// equivalent of `ptr = ptr.wrapping_add(val);`.
    2228    ///
    2229    /// This method operates in units of `T`, which means that it cannot be used
    2230    /// to offset the pointer by an amount which is not a multiple of
    2231    /// `size_of::<T>()`. This can sometimes be inconvenient, as you may want to
    2232    /// work with a deliberately misaligned pointer. In such cases, you may use
    2233    /// the [`fetch_byte_add`](Self::fetch_byte_add) method instead.
    2234    ///
    2235    /// `fetch_ptr_add` takes an [`Ordering`] argument which describes the
    2236    /// memory ordering of this operation. All ordering modes are possible. Note
    2237    /// that using [`Acquire`] makes the store part of this operation
    2238    /// [`Relaxed`], and using [`Release`] makes the load part [`Relaxed`].
    2239    ///
    2240    /// **Note**: This method is only available on platforms that support atomic
    2241    /// operations on [`AtomicPtr`].
    2242    ///
    2243    /// [`wrapping_add`]: pointer::wrapping_add
    2244    ///
    2245    /// # Examples
    2246    ///
    2247    /// ```
    2248    /// use core::sync::atomic::{AtomicPtr, Ordering};
    2249    ///
    2250    /// let atom = AtomicPtr::<i64>::new(core::ptr::null_mut());
    2251    /// assert_eq!(atom.fetch_ptr_add(1, Ordering::Relaxed).addr(), 0);
    2252    /// // Note: units of `size_of::<i64>()`.
    2253    /// assert_eq!(atom.load(Ordering::Relaxed).addr(), 8);
    2254    /// ```
    2255    #[inline]
    2256    #[cfg(target_has_atomic = "ptr")]
    2257    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
    2258    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2259    #[rustc_should_not_be_called_on_const_items]
    2260    pub fn fetch_ptr_add(&self, val: usize, order: Ordering) -> *mut T {
    2261        self.fetch_byte_add(val.wrapping_mul(size_of::<T>()), order)
    2262    }
    2263
    2264    /// Offsets the pointer's address by subtracting `val` (in units of `T`),
    2265    /// returning the previous pointer.
    2266    ///
    2267    /// This is equivalent to using [`wrapping_sub`] to atomically perform the
    2268    /// equivalent of `ptr = ptr.wrapping_sub(val);`.
    2269    ///
    2270    /// This method operates in units of `T`, which means that it cannot be used
    2271    /// to offset the pointer by an amount which is not a multiple of
    2272    /// `size_of::<T>()`. This can sometimes be inconvenient, as you may want to
    2273    /// work with a deliberately misaligned pointer. In such cases, you may use
    2274    /// the [`fetch_byte_sub`](Self::fetch_byte_sub) method instead.
    2275    ///
    2276    /// `fetch_ptr_sub` takes an [`Ordering`] argument which describes the memory
    2277    /// ordering of this operation. All ordering modes are possible. Note that
    2278    /// using [`Acquire`] makes the store part of this operation [`Relaxed`],
    2279    /// and using [`Release`] makes the load part [`Relaxed`].
    2280    ///
    2281    /// **Note**: This method is only available on platforms that support atomic
    2282    /// operations on [`AtomicPtr`].
    2283    ///
    2284    /// [`wrapping_sub`]: pointer::wrapping_sub
    2285    ///
    2286    /// # Examples
    2287    ///
    2288    /// ```
    2289    /// use core::sync::atomic::{AtomicPtr, Ordering};
    2290    ///
    2291    /// let array = [1i32, 2i32];
    2292    /// let atom = AtomicPtr::new(array.as_ptr().wrapping_add(1) as *mut _);
    2293    ///
    2294    /// assert!(core::ptr::eq(
    2295    ///     atom.fetch_ptr_sub(1, Ordering::Relaxed),
    2296    ///     &array[1],
    2297    /// ));
    2298    /// assert!(core::ptr::eq(atom.load(Ordering::Relaxed), &array[0]));
    2299    /// ```
    2300    #[inline]
    2301    #[cfg(target_has_atomic = "ptr")]
    2302    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
    2303    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2304    #[rustc_should_not_be_called_on_const_items]
    2305    pub fn fetch_ptr_sub(&self, val: usize, order: Ordering) -> *mut T {
    2306        self.fetch_byte_sub(val.wrapping_mul(size_of::<T>()), order)
    2307    }
    2308
    2309    /// Offsets the pointer's address by adding `val` *bytes*, returning the
    2310    /// previous pointer.
    2311    ///
    2312    /// This is equivalent to using [`wrapping_byte_add`] to atomically
    2313    /// perform `ptr = ptr.wrapping_byte_add(val)`.
    2314    ///
    2315    /// `fetch_byte_add` takes an [`Ordering`] argument which describes the
    2316    /// memory ordering of this operation. All ordering modes are possible. Note
    2317    /// that using [`Acquire`] makes the store part of this operation
    2318    /// [`Relaxed`], and using [`Release`] makes the load part [`Relaxed`].
    2319    ///
    2320    /// **Note**: This method is only available on platforms that support atomic
    2321    /// operations on [`AtomicPtr`].
    2322    ///
    2323    /// [`wrapping_byte_add`]: pointer::wrapping_byte_add
    2324    ///
    2325    /// # Examples
    2326    ///
    2327    /// ```
    2328    /// use core::sync::atomic::{AtomicPtr, Ordering};
    2329    ///
    2330    /// let atom = AtomicPtr::<i64>::new(core::ptr::null_mut());
    2331    /// assert_eq!(atom.fetch_byte_add(1, Ordering::Relaxed).addr(), 0);
    2332    /// // Note: in units of bytes, not `size_of::<i64>()`.
    2333    /// assert_eq!(atom.load(Ordering::Relaxed).addr(), 1);
    2334    /// ```
    2335    #[inline]
    2336    #[cfg(target_has_atomic = "ptr")]
    2337    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
    2338    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2339    #[rustc_should_not_be_called_on_const_items]
    2340    pub fn fetch_byte_add(&self, val: usize, order: Ordering) -> *mut T {
    2341        // SAFETY: data races are prevented by atomic intrinsics.
    2342        unsafe { atomic_add(self.p.get(), val, order).cast() }
    2343    }
    2344
    2345    /// Offsets the pointer's address by subtracting `val` *bytes*, returning the
    2346    /// previous pointer.
    2347    ///
    2348    /// This is equivalent to using [`wrapping_byte_sub`] to atomically
    2349    /// perform `ptr = ptr.wrapping_byte_sub(val)`.
    2350    ///
    2351    /// `fetch_byte_sub` takes an [`Ordering`] argument which describes the
    2352    /// memory ordering of this operation. All ordering modes are possible. Note
    2353    /// that using [`Acquire`] makes the store part of this operation
    2354    /// [`Relaxed`], and using [`Release`] makes the load part [`Relaxed`].
    2355    ///
    2356    /// **Note**: This method is only available on platforms that support atomic
    2357    /// operations on [`AtomicPtr`].
    2358    ///
    2359    /// [`wrapping_byte_sub`]: pointer::wrapping_byte_sub
    2360    ///
    2361    /// # Examples
    2362    ///
    2363    /// ```
    2364    /// use core::sync::atomic::{AtomicPtr, Ordering};
    2365    ///
    2366    /// let mut arr = [0i64, 1];
    2367    /// let atom = AtomicPtr::<i64>::new(&raw mut arr[1]);
    2368    /// assert_eq!(atom.fetch_byte_sub(8, Ordering::Relaxed).addr(), (&raw const arr[1]).addr());
    2369    /// assert_eq!(atom.load(Ordering::Relaxed).addr(), (&raw const arr[0]).addr());
    2370    /// ```
    2371    #[inline]
    2372    #[cfg(target_has_atomic = "ptr")]
    2373    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
    2374    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2375    #[rustc_should_not_be_called_on_const_items]
    2376    pub fn fetch_byte_sub(&self, val: usize, order: Ordering) -> *mut T {
    2377        // SAFETY: data races are prevented by atomic intrinsics.
    2378        unsafe { atomic_sub(self.p.get(), val, order).cast() }
    2379    }
    2380
    2381    /// Performs a bitwise "or" operation on the address of the current pointer,
    2382    /// and the argument `val`, and stores a pointer with provenance of the
    2383    /// current pointer and the resulting address.
    2384    ///
    2385    /// This is equivalent to using [`map_addr`] to atomically perform
    2386    /// `ptr = ptr.map_addr(|a| a | val)`. This can be used in tagged
    2387    /// pointer schemes to atomically set tag bits.
    2388    ///
    2389    /// **Caveat**: This operation returns the previous value. To compute the
    2390    /// stored value without losing provenance, you may use [`map_addr`]. For
    2391    /// example: `a.fetch_or(val).map_addr(|a| a | val)`.
    2392    ///
    2393    /// `fetch_or` takes an [`Ordering`] argument which describes the memory
    2394    /// ordering of this operation. All ordering modes are possible. Note that
    2395    /// using [`Acquire`] makes the store part of this operation [`Relaxed`],
    2396    /// and using [`Release`] makes the load part [`Relaxed`].
    2397    ///
    2398    /// **Note**: This method is only available on platforms that support atomic
    2399    /// operations on [`AtomicPtr`].
    2400    ///
    2401    /// This API and its claimed semantics are part of the Strict Provenance
    2402    /// experiment, see the [module documentation for `ptr`][crate::ptr] for
    2403    /// details.
    2404    ///
    2405    /// [`map_addr`]: pointer::map_addr
    2406    ///
    2407    /// # Examples
    2408    ///
    2409    /// ```
    2410    /// use core::sync::atomic::{AtomicPtr, Ordering};
    2411    ///
    2412    /// let pointer = &mut 3i64 as *mut i64;
    2413    ///
    2414    /// let atom = AtomicPtr::<i64>::new(pointer);
    2415    /// // Tag the bottom bit of the pointer.
    2416    /// assert_eq!(atom.fetch_or(1, Ordering::Relaxed).addr() & 1, 0);
    2417    /// // Extract and untag.
    2418    /// let tagged = atom.load(Ordering::Relaxed);
    2419    /// assert_eq!(tagged.addr() & 1, 1);
    2420    /// assert_eq!(tagged.map_addr(|p| p & !1), pointer);
    2421    /// ```
    2422    #[inline]
    2423    #[cfg(target_has_atomic = "ptr")]
    2424    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
    2425    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2426    #[rustc_should_not_be_called_on_const_items]
    2427    pub fn fetch_or(&self, val: usize, order: Ordering) -> *mut T {
    2428        // SAFETY: data races are prevented by atomic intrinsics.
    2429        unsafe { atomic_or(self.p.get(), val, order).cast() }
    2430    }
    2431
    2432    /// Performs a bitwise "and" operation on the address of the current
    2433    /// pointer, and the argument `val`, and stores a pointer with provenance of
    2434    /// the current pointer and the resulting address.
    2435    ///
    2436    /// This is equivalent to using [`map_addr`] to atomically perform
    2437    /// `ptr = ptr.map_addr(|a| a & val)`. This can be used in tagged
    2438    /// pointer schemes to atomically unset tag bits.
    2439    ///
    2440    /// **Caveat**: This operation returns the previous value. To compute the
    2441    /// stored value without losing provenance, you may use [`map_addr`]. For
    2442    /// example: `a.fetch_and(val).map_addr(|a| a & val)`.
    2443    ///
    2444    /// `fetch_and` takes an [`Ordering`] argument which describes the memory
    2445    /// ordering of this operation. All ordering modes are possible. Note that
    2446    /// using [`Acquire`] makes the store part of this operation [`Relaxed`],
    2447    /// and using [`Release`] makes the load part [`Relaxed`].
    2448    ///
    2449    /// **Note**: This method is only available on platforms that support atomic
    2450    /// operations on [`AtomicPtr`].
    2451    ///
    2452    /// This API and its claimed semantics are part of the Strict Provenance
    2453    /// experiment, see the [module documentation for `ptr`][crate::ptr] for
    2454    /// details.
    2455    ///
    2456    /// [`map_addr`]: pointer::map_addr
    2457    ///
    2458    /// # Examples
    2459    ///
    2460    /// ```
    2461    /// use core::sync::atomic::{AtomicPtr, Ordering};
    2462    ///
    2463    /// let pointer = &mut 3i64 as *mut i64;
    2464    /// // A tagged pointer
    2465    /// let atom = AtomicPtr::<i64>::new(pointer.map_addr(|a| a | 1));
    2466    /// assert_eq!(atom.fetch_or(1, Ordering::Relaxed).addr() & 1, 1);
    2467    /// // Untag, and extract the previously tagged pointer.
    2468    /// let untagged = atom.fetch_and(!1, Ordering::Relaxed)
    2469    ///     .map_addr(|a| a & !1);
    2470    /// assert_eq!(untagged, pointer);
    2471    /// ```
    2472    #[inline]
    2473    #[cfg(target_has_atomic = "ptr")]
    2474    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
    2475    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2476    #[rustc_should_not_be_called_on_const_items]
    2477    pub fn fetch_and(&self, val: usize, order: Ordering) -> *mut T {
    2478        // SAFETY: data races are prevented by atomic intrinsics.
    2479        unsafe { atomic_and(self.p.get(), val, order).cast() }
    2480    }
    2481
    2482    /// Performs a bitwise "xor" operation on the address of the current
    2483    /// pointer, and the argument `val`, and stores a pointer with provenance of
    2484    /// the current pointer and the resulting address.
    2485    ///
    2486    /// This is equivalent to using [`map_addr`] to atomically perform
    2487    /// `ptr = ptr.map_addr(|a| a ^ val)`. This can be used in tagged
    2488    /// pointer schemes to atomically toggle tag bits.
    2489    ///
    2490    /// **Caveat**: This operation returns the previous value. To compute the
    2491    /// stored value without losing provenance, you may use [`map_addr`]. For
    2492    /// example: `a.fetch_xor(val).map_addr(|a| a ^ val)`.
    2493    ///
    2494    /// `fetch_xor` takes an [`Ordering`] argument which describes the memory
    2495    /// ordering of this operation. All ordering modes are possible. Note that
    2496    /// using [`Acquire`] makes the store part of this operation [`Relaxed`],
    2497    /// and using [`Release`] makes the load part [`Relaxed`].
    2498    ///
    2499    /// **Note**: This method is only available on platforms that support atomic
    2500    /// operations on [`AtomicPtr`].
    2501    ///
    2502    /// This API and its claimed semantics are part of the Strict Provenance
    2503    /// experiment, see the [module documentation for `ptr`][crate::ptr] for
    2504    /// details.
    2505    ///
    2506    /// [`map_addr`]: pointer::map_addr
    2507    ///
    2508    /// # Examples
    2509    ///
    2510    /// ```
    2511    /// use core::sync::atomic::{AtomicPtr, Ordering};
    2512    ///
    2513    /// let pointer = &mut 3i64 as *mut i64;
    2514    /// let atom = AtomicPtr::<i64>::new(pointer);
    2515    ///
    2516    /// // Toggle a tag bit on the pointer.
    2517    /// atom.fetch_xor(1, Ordering::Relaxed);
    2518    /// assert_eq!(atom.load(Ordering::Relaxed).addr() & 1, 1);
    2519    /// ```
    2520    #[inline]
    2521    #[cfg(target_has_atomic = "ptr")]
    2522    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
    2523    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2524    #[rustc_should_not_be_called_on_const_items]
    2525    pub fn fetch_xor(&self, val: usize, order: Ordering) -> *mut T {
    2526        // SAFETY: data races are prevented by atomic intrinsics.
    2527        unsafe { atomic_xor(self.p.get(), val, order).cast() }
    2528    }
    2529
    2530    /// Returns a mutable pointer to the underlying pointer.
    2531    ///
    2532    /// Doing non-atomic reads and writes on the resulting pointer can be a data race.
    2533    /// This method is mostly useful for FFI, where the function signature may use
    2534    /// `*mut *mut T` instead of `&AtomicPtr<T>`.
    2535    ///
    2536    /// Returning an `*mut` pointer from a shared reference to this atomic is safe because the
    2537    /// atomic types work with interior mutability. All modifications of an atomic change the value
    2538    /// through a shared reference, and can do so safely as long as they use atomic operations. Any
    2539    /// use of the returned raw pointer requires an `unsafe` block and still has to uphold the
    2540    /// requirements of the [memory model].
    2541    ///
    2542    /// # Examples
    2543    ///
    2544    /// ```ignore (extern-declaration)
    2545    /// use std::sync::atomic::AtomicPtr;
    2546    ///
    2547    /// extern "C" {
    2548    ///     fn my_atomic_op(arg: *mut *mut u32);
    2549    /// }
    2550    ///
    2551    /// let mut value = 17;
    2552    /// let atomic = AtomicPtr::new(&mut value);
    2553    ///
    2554    /// // SAFETY: Safe as long as `my_atomic_op` is atomic.
    2555    /// unsafe {
    2556    ///     my_atomic_op(atomic.as_ptr());
    2557    /// }
    2558    /// ```
    2559    ///
    2560    /// [memory model]: self#memory-model-for-atomic-accesses
    2561    #[inline]
    2562    #[stable(feature = "atomic_as_ptr", since = "1.70.0")]
    2563    #[rustc_const_stable(feature = "atomic_as_ptr", since = "1.70.0")]
    2564    #[rustc_never_returns_null_ptr]
    2565    pub const fn as_ptr(&self) -> *mut *mut T {
    2566        self.p.get()
    2567    }
    2568}
    2569
    2570#[cfg(target_has_atomic_load_store = "8")]
    2571#[stable(feature = "atomic_bool_from", since = "1.24.0")]
    2572#[rustc_const_unstable(feature = "const_convert", issue = "143773")]
    2573impl const From<bool> for AtomicBool {
    2574    /// Converts a `bool` into an `AtomicBool`.
    2575    ///
    2576    /// # Examples
    2577    ///
    2578    /// ```
    2579    /// use std::sync::atomic::AtomicBool;
    2580    /// let atomic_bool = AtomicBool::from(true);
    2581    /// assert_eq!(format!("{atomic_bool:?}"), "true")
    2582    /// ```
    2583    #[inline]
    2584    fn from(b: bool) -> Self {
    2585        Self::new(b)
    2586    }
    2587}
    2588
    2589#[cfg(target_has_atomic_load_store = "ptr")]
    2590#[stable(feature = "atomic_from", since = "1.23.0")]
    2591#[rustc_const_unstable(feature = "const_convert", issue = "143773")]
    2592impl<T> const From<*mut T> for AtomicPtr<T> {
    2593    /// Converts a `*mut T` into an `AtomicPtr<T>`.
    2594    #[inline]
    2595    fn from(p: *mut T) -> Self {
    2596        Self::new(p)
    2597    }
    2598}
    2599
    2600#[allow(unused_macros)] // This macro ends up being unused on some architectures.
    2601macro_rules! if_8_bit {
    2602    (u8, $( yes = [$($yes:tt)*], )? $( no = [$($no:tt)*], )? ) => { concat!("", $($($yes)*)?) };
    2603    (i8, $( yes = [$($yes:tt)*], )? $( no = [$($no:tt)*], )? ) => { concat!("", $($($yes)*)?) };
    2604    ($_:ident, $( yes = [$($yes:tt)*], )? $( no = [$($no:tt)*], )? ) => { concat!("", $($($no)*)?) };
    2605}
    2606
    2607#[cfg(target_has_atomic_load_store)]
    2608macro_rules! atomic_int {
    2609    ($cfg_cas:meta,
    2610     $cfg_align:meta,
    2611     $stable:meta,
    2612     $stable_cxchg:meta,
    2613     $stable_debug:meta,
    2614     $stable_access:meta,
    2615     $stable_from:meta,
    2616     $stable_nand:meta,
    2617     $const_stable_new:meta,
    2618     $const_stable_into_inner:meta,
    2619     $diagnostic_item:meta,
    2620     $s_int_type:literal,
    2621     $extra_feature:expr,
    2622     $min_fn:ident, $max_fn:ident,
    2623     $align:expr,
    2624     $int_type:ident $atomic_type:ident) => {
    2625        /// An integer type which can be safely shared between threads.
    2626        ///
    2627        /// This type has the same
    2628        #[doc = if_8_bit!(
    2629            $int_type,
    2630            yes = ["size, alignment, and bit validity"],
    2631            no = ["size and bit validity"],
    2632        )]
    2633        /// as the underlying integer type, [`
    2634        #[doc = $s_int_type]
    2635        /// `].
    2636        #[doc = if_8_bit! {
    2637            $int_type,
    2638            no = [
    2639                "However, the alignment of this type is always equal to its ",
    2640                "size, even on targets where [`", $s_int_type, "`] has a ",
    2641                "lesser alignment."
    2642            ],
    2643        }]
    2644        ///
    2645        /// For more about the differences between atomic types and
    2646        /// non-atomic types as well as information about the portability of
    2647        /// this type, please see the [module-level documentation].
    2648        ///
    2649        /// **Note:** This type is only available on platforms that support
    2650        /// atomic loads and stores of [`
    2651        #[doc = $s_int_type]
    2652        /// `].
    2653        ///
    2654        /// [module-level documentation]: crate::sync::atomic
    2655        #[$stable]
    2656        #[$diagnostic_item]
    2657        #[repr(C, align($align))]
    2658        pub struct $atomic_type {
    2659            v: UnsafeCell<$int_type>,
    2660        }
    2661
    2662        #[$stable]
    2663        impl Default for $atomic_type {
    2664            #[inline]
    2665            fn default() -> Self {
    2666                Self::new(Default::default())
    2667            }
    2668        }
    2669
    2670        #[$stable_from]
    2671        #[rustc_const_unstable(feature = "const_convert", issue = "143773")]
    2672        impl const From<$int_type> for $atomic_type {
    2673            #[doc = concat!("Converts an `", stringify!($int_type), "` into an `", stringify!($atomic_type), "`.")]
    2674            #[inline]
    2675            fn from(v: $int_type) -> Self { Self::new(v) }
    2676        }
    2677
    2678        #[$stable_debug]
    2679        impl fmt::Debug for $atomic_type {
    2680            fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
    2681                fmt::Debug::fmt(&self.load(Ordering::Relaxed), f)
    2682            }
    2683        }
    2684
    2685        // Send is implicitly implemented.
    2686        #[$stable]
    2687        unsafe impl Sync for $atomic_type {}
    2688
    2689        impl $atomic_type {
    2690            /// Creates a new atomic integer.
    2691            ///
    2692            /// # Examples
    2693            ///
    2694            /// ```
    2695            #[doc = concat!($extra_feature, "use std::sync::atomic::", stringify!($atomic_type), ";")]
    2696            ///
    2697            #[doc = concat!("let atomic_forty_two = ", stringify!($atomic_type), "::new(42);")]
    2698            /// ```
    2699            #[inline]
    2700            #[$stable]
    2701            #[$const_stable_new]
    2702            #[must_use]
    2703            pub const fn new(v: $int_type) -> Self {
    2704                Self {v: UnsafeCell::new(v)}
    2705            }
    2706
    2707            /// Creates a new reference to an atomic integer from a pointer.
    2708            ///
    2709            /// # Examples
    2710            ///
    2711            /// ```
    2712            #[doc = concat!($extra_feature, "use std::sync::atomic::{self, ", stringify!($atomic_type), "};")]
    2713            ///
    2714            /// // Get a pointer to an allocated value
    2715            #[doc = concat!("let ptr: *mut ", stringify!($int_type), " = Box::into_raw(Box::new(0));")]
    2716            ///
    2717            #[doc = concat!("assert!(ptr.cast::<", stringify!($atomic_type), ">().is_aligned());")]
    2718            ///
    2719            /// {
    2720            ///     // Create an atomic view of the allocated value
    2721            // SAFETY: this is a doc comment, tidy, it can't hurt you (also guaranteed by the construction of `ptr` and the assert above)
    2722            #[doc = concat!("    let atomic = unsafe {", stringify!($atomic_type), "::from_ptr(ptr) };")]
    2723            ///
    2724            ///     // Use `atomic` for atomic operations, possibly share it with other threads
    2725            ///     atomic.store(1, atomic::Ordering::Relaxed);
    2726            /// }
    2727            ///
    2728            /// // It's ok to non-atomically access the value behind `ptr`,
    2729            /// // since the reference to the atomic ended its lifetime in the block above
    2730            /// assert_eq!(unsafe { *ptr }, 1);
    2731            ///
    2732            /// // Deallocate the value
    2733            /// unsafe { drop(Box::from_raw(ptr)) }
    2734            /// ```
    2735            ///
    2736            /// # Safety
    2737            ///
    2738            /// * `ptr` must be aligned to
    2739            #[doc = concat!("  `align_of::<", stringify!($atomic_type), ">()`")]
    2740            #[doc = if_8_bit!{
    2741                $int_type,
    2742                yes = [
    2743                    "  (note that this is always true, since `align_of::<",
    2744                    stringify!($atomic_type), ">() == 1`)."
    2745                ],
    2746                no = [
    2747                    "  (note that on some platforms this can be bigger than `align_of::<",
    2748                    stringify!($int_type), ">()`)."
    2749                ],
    2750            }]
    2751            /// * `ptr` must be [valid] for both reads and writes for the whole lifetime `'a`.
    2752            /// * You must adhere to the [Memory model for atomic accesses]. In particular, it is not
    2753            ///   allowed to mix conflicting atomic and non-atomic accesses, or atomic accesses of different
    2754            ///   sizes, without synchronization.
    2755            ///
    2756            /// [valid]: crate::ptr#safety
    2757            /// [Memory model for atomic accesses]: self#memory-model-for-atomic-accesses
    2758            #[inline]
    2759            #[stable(feature = "atomic_from_ptr", since = "1.75.0")]
    2760            #[rustc_const_stable(feature = "const_atomic_from_ptr", since = "1.84.0")]
    2761            pub const unsafe fn from_ptr<'a>(ptr: *mut $int_type) -> &'a $atomic_type {
    2762                // SAFETY: guaranteed by the caller
    2763                unsafe { &*ptr.cast() }
    2764            }
    2765
    2766
    2767            /// Returns a mutable reference to the underlying integer.
    2768            ///
    2769            /// This is safe because the mutable reference guarantees that no other threads are
    2770            /// concurrently accessing the atomic data.
    2771            ///
    2772            /// # Examples
    2773            ///
    2774            /// ```
    2775            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    2776            ///
    2777            #[doc = concat!("let mut some_var = ", stringify!($atomic_type), "::new(10);")]
    2778            /// assert_eq!(*some_var.get_mut(), 10);
    2779            /// *some_var.get_mut() = 5;
    2780            /// assert_eq!(some_var.load(Ordering::SeqCst), 5);
    2781            /// ```
    2782            #[inline]
    2783            #[$stable_access]
    2784            pub fn get_mut(&mut self) -> &mut $int_type {
    2785                self.v.get_mut()
    2786            }
    2787
    2788            #[doc = concat!("Get atomic access to a `&mut ", stringify!($int_type), "`.")]
    2789            ///
    2790            #[doc = if_8_bit! {
    2791                $int_type,
    2792                no = [
    2793                    "**Note:** This function is only available on targets where `",
    2794                    stringify!($atomic_type), "` has the same alignment as `", stringify!($int_type), "`."
    2795                ],
    2796            }]
    2797            ///
    2798            /// # Examples
    2799            ///
    2800            /// ```
    2801            /// #![feature(atomic_from_mut)]
    2802            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    2803            ///
    2804            /// let mut some_int = 123;
    2805            #[doc = concat!("let a = ", stringify!($atomic_type), "::from_mut(&mut some_int);")]
    2806            /// a.store(100, Ordering::Relaxed);
    2807            /// assert_eq!(some_int, 100);
    2808            /// ```
    2809            ///
    2810            #[inline]
    2811            #[$cfg_align]
    2812            #[unstable(feature = "atomic_from_mut", issue = "76314")]
    2813            pub fn from_mut(v: &mut $int_type) -> &mut Self {
    2814                let [] = [(); align_of::<Self>() - align_of::<$int_type>()];
    2815                // SAFETY:
    2816                //  - the mutable reference guarantees unique ownership.
    2817                //  - the alignment of `$int_type` and `Self` is the
    2818                //    same, as promised by $cfg_align and verified above.
    2819                unsafe { &mut *(v as *mut $int_type as *mut Self) }
    2820            }
    2821
    2822            #[doc = concat!("Get non-atomic access to a `&mut [", stringify!($atomic_type), "]` slice")]
    2823            ///
    2824            /// This is safe because the mutable reference guarantees that no other threads are
    2825            /// concurrently accessing the atomic data.
    2826            ///
    2827            /// # Examples
    2828            ///
    2829            /// ```ignore-wasm
    2830            /// #![feature(atomic_from_mut)]
    2831            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    2832            ///
    2833            #[doc = concat!("let mut some_ints = [const { ", stringify!($atomic_type), "::new(0) }; 10];")]
    2834            ///
    2835            #[doc = concat!("let view: &mut [", stringify!($int_type), "] = ", stringify!($atomic_type), "::get_mut_slice(&mut some_ints);")]
    2836            /// assert_eq!(view, [0; 10]);
    2837            /// view
    2838            ///     .iter_mut()
    2839            ///     .enumerate()
    2840            ///     .for_each(|(idx, int)| *int = idx as _);
    2841            ///
    2842            /// std::thread::scope(|s| {
    2843            ///     some_ints
    2844            ///         .iter()
    2845            ///         .enumerate()
    2846            ///         .for_each(|(idx, int)| {
    2847            ///             s.spawn(move || assert_eq!(int.load(Ordering::Relaxed), idx as _));
    2848            ///         })
    2849            /// });
    2850            /// ```
    2851            #[inline]
    2852            #[unstable(feature = "atomic_from_mut", issue = "76314")]
    2853            pub fn get_mut_slice(this: &mut [Self]) -> &mut [$int_type] {
    2854                // SAFETY: the mutable reference guarantees unique ownership.
    2855                unsafe { &mut *(this as *mut [Self] as *mut [$int_type]) }
    2856            }
    2857
    2858            #[doc = concat!("Get atomic access to a `&mut [", stringify!($int_type), "]` slice.")]
    2859            ///
    2860            #[doc = if_8_bit! {
    2861                $int_type,
    2862                no = [
    2863                    "**Note:** This function is only available on targets where `",
    2864                    stringify!($atomic_type), "` has the same alignment as `", stringify!($int_type), "`."
    2865                ],
    2866            }]
    2867            ///
    2868            /// # Examples
    2869            ///
    2870            /// ```ignore-wasm
    2871            /// #![feature(atomic_from_mut)]
    2872            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    2873            ///
    2874            /// let mut some_ints = [0; 10];
    2875            #[doc = concat!("let a = &*", stringify!($atomic_type), "::from_mut_slice(&mut some_ints);")]
    2876            /// std::thread::scope(|s| {
    2877            ///     for i in 0..a.len() {
    2878            ///         s.spawn(move || a[i].store(i as _, Ordering::Relaxed));
    2879            ///     }
    2880            /// });
    2881            /// for (i, n) in some_ints.into_iter().enumerate() {
    2882            ///     assert_eq!(i, n as usize);
    2883            /// }
    2884            /// ```
    2885            #[inline]
    2886            #[$cfg_align]
    2887            #[unstable(feature = "atomic_from_mut", issue = "76314")]
    2888            pub fn from_mut_slice(v: &mut [$int_type]) -> &mut [Self] {
    2889                let [] = [(); align_of::<Self>() - align_of::<$int_type>()];
    2890                // SAFETY:
    2891                //  - the mutable reference guarantees unique ownership.
    2892                //  - the alignment of `$int_type` and `Self` is the
    2893                //    same, as promised by $cfg_align and verified above.
    2894                unsafe { &mut *(v as *mut [$int_type] as *mut [Self]) }
    2895            }
    2896
    2897            /// Consumes the atomic and returns the contained value.
    2898            ///
    2899            /// This is safe because passing `self` by value guarantees that no other threads are
    2900            /// concurrently accessing the atomic data.
    2901            ///
    2902            /// # Examples
    2903            ///
    2904            /// ```
    2905            #[doc = concat!($extra_feature, "use std::sync::atomic::", stringify!($atomic_type), ";")]
    2906            ///
    2907            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
    2908            /// assert_eq!(some_var.into_inner(), 5);
    2909            /// ```
    2910            #[inline]
    2911            #[$stable_access]
    2912            #[$const_stable_into_inner]
    2913            pub const fn into_inner(self) -> $int_type {
    2914                self.v.into_inner()
    2915            }
    2916
    2917            /// Loads a value from the atomic integer.
    2918            ///
    2919            /// `load` takes an [`Ordering`] argument which describes the memory ordering of this operation.
    2920            /// Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
    2921            ///
    2922            /// # Panics
    2923            ///
    2924            /// Panics if `order` is [`Release`] or [`AcqRel`].
    2925            ///
    2926            /// # Examples
    2927            ///
    2928            /// ```
    2929            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    2930            ///
    2931            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
    2932            ///
    2933            /// assert_eq!(some_var.load(Ordering::Relaxed), 5);
    2934            /// ```
    2935            #[inline]
    2936            #[$stable]
    2937            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2938            pub fn load(&self, order: Ordering) -> $int_type {
    2939                // SAFETY: data races are prevented by atomic intrinsics.
    2940                unsafe { atomic_load(self.v.get(), order) }
    2941            }
    2942
    2943            /// Stores a value into the atomic integer.
    2944            ///
    2945            /// `store` takes an [`Ordering`] argument which describes the memory ordering of this operation.
    2946            ///  Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
    2947            ///
    2948            /// # Panics
    2949            ///
    2950            /// Panics if `order` is [`Acquire`] or [`AcqRel`].
    2951            ///
    2952            /// # Examples
    2953            ///
    2954            /// ```
    2955            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    2956            ///
    2957            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
    2958            ///
    2959            /// some_var.store(10, Ordering::Relaxed);
    2960            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
    2961            /// ```
    2962            #[inline]
    2963            #[$stable]
    2964            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2965            #[rustc_should_not_be_called_on_const_items]
    2966            pub fn store(&self, val: $int_type, order: Ordering) {
    2967                // SAFETY: data races are prevented by atomic intrinsics.
    2968                unsafe { atomic_store(self.v.get(), val, order); }
    2969            }
    2970
    2971            /// Stores a value into the atomic integer, returning the previous value.
    2972            ///
    2973            /// `swap` takes an [`Ordering`] argument which describes the memory ordering
    2974            /// of this operation. All ordering modes are possible. Note that using
    2975            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    2976            /// using [`Release`] makes the load part [`Relaxed`].
    2977            ///
    2978            /// **Note**: This method is only available on platforms that support atomic operations on
    2979            #[doc = concat!("[`", $s_int_type, "`].")]
    2980            ///
    2981            /// # Examples
    2982            ///
    2983            /// ```
    2984            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    2985            ///
    2986            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
    2987            ///
    2988            /// assert_eq!(some_var.swap(10, Ordering::Relaxed), 5);
    2989            /// ```
    2990            #[inline]
    2991            #[$stable]
    2992            #[$cfg_cas]
    2993            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    2994            #[rustc_should_not_be_called_on_const_items]
    2995            pub fn swap(&self, val: $int_type, order: Ordering) -> $int_type {
    2996                // SAFETY: data races are prevented by atomic intrinsics.
    2997                unsafe { atomic_swap(self.v.get(), val, order) }
    2998            }
    2999
    3000            /// Stores a value into the atomic integer if the current value is the same as
    3001            /// the `current` value.
    3002            ///
    3003            /// The return value is always the previous value. If it is equal to `current`, then the
    3004            /// value was updated.
    3005            ///
    3006            /// `compare_and_swap` also takes an [`Ordering`] argument which describes the memory
    3007            /// ordering of this operation. Notice that even when using [`AcqRel`], the operation
    3008            /// might fail and hence just perform an `Acquire` load, but not have `Release` semantics.
    3009            /// Using [`Acquire`] makes the store part of this operation [`Relaxed`] if it
    3010            /// happens, and using [`Release`] makes the load part [`Relaxed`].
    3011            ///
    3012            /// **Note**: This method is only available on platforms that support atomic operations on
    3013            #[doc = concat!("[`", $s_int_type, "`].")]
    3014            ///
    3015            /// # Migrating to `compare_exchange` and `compare_exchange_weak`
    3016            ///
    3017            /// `compare_and_swap` is equivalent to `compare_exchange` with the following mapping for
    3018            /// memory orderings:
    3019            ///
    3020            /// Original | Success | Failure
    3021            /// -------- | ------- | -------
    3022            /// Relaxed  | Relaxed | Relaxed
    3023            /// Acquire  | Acquire | Acquire
    3024            /// Release  | Release | Relaxed
    3025            /// AcqRel   | AcqRel  | Acquire
    3026            /// SeqCst   | SeqCst  | SeqCst
    3027            ///
    3028            /// `compare_and_swap` and `compare_exchange` also differ in their return type. You can use
    3029            /// `compare_exchange(...).unwrap_or_else(|x| x)` to recover the behavior of `compare_and_swap`,
    3030            /// but in most cases it is more idiomatic to check whether the return value is `Ok` or `Err`
    3031            /// rather than to infer success vs failure based on the value that was read.
    3032            ///
    3033            /// During migration, consider whether it makes sense to use `compare_exchange_weak` instead.
    3034            /// `compare_exchange_weak` is allowed to fail spuriously even when the comparison succeeds,
    3035            /// which allows the compiler to generate better assembly code when the compare and swap
    3036            /// is used in a loop.
    3037            ///
    3038            /// # Examples
    3039            ///
    3040            /// ```
    3041            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3042            ///
    3043            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
    3044            ///
    3045            /// assert_eq!(some_var.compare_and_swap(5, 10, Ordering::Relaxed), 5);
    3046            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
    3047            ///
    3048            /// assert_eq!(some_var.compare_and_swap(6, 12, Ordering::Relaxed), 10);
    3049            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
    3050            /// ```
    3051            #[inline]
    3052            #[$stable]
    3053            #[deprecated(
    3054                since = "1.50.0",
    3055                note = "Use `compare_exchange` or `compare_exchange_weak` instead")
    3056            ]
    3057            #[$cfg_cas]
    3058            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3059            #[rustc_should_not_be_called_on_const_items]
    3060            pub fn compare_and_swap(&self,
    3061                                    current: $int_type,
    3062                                    new: $int_type,
    3063                                    order: Ordering) -> $int_type {
    3064                match self.compare_exchange(current,
    3065                                            new,
    3066                                            order,
    3067                                            strongest_failure_ordering(order)) {
    3068                    Ok(x) => x,
    3069                    Err(x) => x,
    3070                }
    3071            }
    3072
    3073            /// Stores a value into the atomic integer if the current value is the same as
    3074            /// the `current` value.
    3075            ///
    3076            /// The return value is a result indicating whether the new value was written and
    3077            /// containing the previous value. On success this value is guaranteed to be equal to
    3078            /// `current`.
    3079            ///
    3080            /// `compare_exchange` takes two [`Ordering`] arguments to describe the memory
    3081            /// ordering of this operation. `success` describes the required ordering for the
    3082            /// read-modify-write operation that takes place if the comparison with `current` succeeds.
    3083            /// `failure` describes the required ordering for the load operation that takes place when
    3084            /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
    3085            /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
    3086            /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    3087            ///
    3088            /// **Note**: This method is only available on platforms that support atomic operations on
    3089            #[doc = concat!("[`", $s_int_type, "`].")]
    3090            ///
    3091            /// # Examples
    3092            ///
    3093            /// ```
    3094            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3095            ///
    3096            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
    3097            ///
    3098            /// assert_eq!(some_var.compare_exchange(5, 10,
    3099            ///                                      Ordering::Acquire,
    3100            ///                                      Ordering::Relaxed),
    3101            ///            Ok(5));
    3102            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
    3103            ///
    3104            /// assert_eq!(some_var.compare_exchange(6, 12,
    3105            ///                                      Ordering::SeqCst,
    3106            ///                                      Ordering::Acquire),
    3107            ///            Err(10));
    3108            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
    3109            /// ```
    3110            ///
    3111            /// # Considerations
    3112            ///
    3113            /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
    3114            /// of CAS operations. In particular, a load of the value followed by a successful
    3115            /// `compare_exchange` with the previous load *does not ensure* that other threads have not
    3116            /// changed the value in the interim! This is usually important when the *equality* check in
    3117            /// the `compare_exchange` is being used to check the *identity* of a value, but equality
    3118            /// does not necessarily imply identity. This is a particularly common case for pointers, as
    3119            /// a pointer holding the same address does not imply that the same object exists at that
    3120            /// address! In this case, `compare_exchange` can lead to the [ABA problem].
    3121            ///
    3122            /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    3123            /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    3124            #[inline]
    3125            #[$stable_cxchg]
    3126            #[$cfg_cas]
    3127            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3128            #[rustc_should_not_be_called_on_const_items]
    3129            pub fn compare_exchange(&self,
    3130                                    current: $int_type,
    3131                                    new: $int_type,
    3132                                    success: Ordering,
    3133                                    failure: Ordering) -> Result<$int_type, $int_type> {
    3134                // SAFETY: data races are prevented by atomic intrinsics.
    3135                unsafe { atomic_compare_exchange(self.v.get(), current, new, success, failure) }
    3136            }
    3137
    3138            /// Stores a value into the atomic integer if the current value is the same as
    3139            /// the `current` value.
    3140            ///
    3141            #[doc = concat!("Unlike [`", stringify!($atomic_type), "::compare_exchange`],")]
    3142            /// this function is allowed to spuriously fail even
    3143            /// when the comparison succeeds, which can result in more efficient code on some
    3144            /// platforms. The return value is a result indicating whether the new value was
    3145            /// written and containing the previous value.
    3146            ///
    3147            /// `compare_exchange_weak` takes two [`Ordering`] arguments to describe the memory
    3148            /// ordering of this operation. `success` describes the required ordering for the
    3149            /// read-modify-write operation that takes place if the comparison with `current` succeeds.
    3150            /// `failure` describes the required ordering for the load operation that takes place when
    3151            /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
    3152            /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
    3153            /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    3154            ///
    3155            /// **Note**: This method is only available on platforms that support atomic operations on
    3156            #[doc = concat!("[`", $s_int_type, "`].")]
    3157            ///
    3158            /// # Examples
    3159            ///
    3160            /// ```
    3161            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3162            ///
    3163            #[doc = concat!("let val = ", stringify!($atomic_type), "::new(4);")]
    3164            ///
    3165            /// let mut old = val.load(Ordering::Relaxed);
    3166            /// loop {
    3167            ///     let new = old * 2;
    3168            ///     match val.compare_exchange_weak(old, new, Ordering::SeqCst, Ordering::Relaxed) {
    3169            ///         Ok(_) => break,
    3170            ///         Err(x) => old = x,
    3171            ///     }
    3172            /// }
    3173            /// ```
    3174            ///
    3175            /// # Considerations
    3176            ///
    3177            /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
    3178            /// of CAS operations. In particular, a load of the value followed by a successful
    3179            /// `compare_exchange` with the previous load *does not ensure* that other threads have not
    3180            /// changed the value in the interim. This is usually important when the *equality* check in
    3181            /// the `compare_exchange` is being used to check the *identity* of a value, but equality
    3182            /// does not necessarily imply identity. This is a particularly common case for pointers, as
    3183            /// a pointer holding the same address does not imply that the same object exists at that
    3184            /// address! In this case, `compare_exchange` can lead to the [ABA problem].
    3185            ///
    3186            /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    3187            /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    3188            #[inline]
    3189            #[$stable_cxchg]
    3190            #[$cfg_cas]
    3191            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3192            #[rustc_should_not_be_called_on_const_items]
    3193            pub fn compare_exchange_weak(&self,
    3194                                         current: $int_type,
    3195                                         new: $int_type,
    3196                                         success: Ordering,
    3197                                         failure: Ordering) -> Result<$int_type, $int_type> {
    3198                // SAFETY: data races are prevented by atomic intrinsics.
    3199                unsafe {
    3200                    atomic_compare_exchange_weak(self.v.get(), current, new, success, failure)
    3201                }
    3202            }
    3203
    3204            /// Adds to the current value, returning the previous value.
    3205            ///
    3206            /// This operation wraps around on overflow.
    3207            ///
    3208            /// `fetch_add` takes an [`Ordering`] argument which describes the memory ordering
    3209            /// of this operation. All ordering modes are possible. Note that using
    3210            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    3211            /// using [`Release`] makes the load part [`Relaxed`].
    3212            ///
    3213            /// **Note**: This method is only available on platforms that support atomic operations on
    3214            #[doc = concat!("[`", $s_int_type, "`].")]
    3215            ///
    3216            /// # Examples
    3217            ///
    3218            /// ```
    3219            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3220            ///
    3221            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0);")]
    3222            /// assert_eq!(foo.fetch_add(10, Ordering::SeqCst), 0);
    3223            /// assert_eq!(foo.load(Ordering::SeqCst), 10);
    3224            /// ```
    3225            #[inline]
    3226            #[$stable]
    3227            #[$cfg_cas]
    3228            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3229            #[rustc_should_not_be_called_on_const_items]
    3230            pub fn fetch_add(&self, val: $int_type, order: Ordering) -> $int_type {
    3231                // SAFETY: data races are prevented by atomic intrinsics.
    3232                unsafe { atomic_add(self.v.get(), val, order) }
    3233            }
    3234
    3235            /// Subtracts from the current value, returning the previous value.
    3236            ///
    3237            /// This operation wraps around on overflow.
    3238            ///
    3239            /// `fetch_sub` takes an [`Ordering`] argument which describes the memory ordering
    3240            /// of this operation. All ordering modes are possible. Note that using
    3241            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    3242            /// using [`Release`] makes the load part [`Relaxed`].
    3243            ///
    3244            /// **Note**: This method is only available on platforms that support atomic operations on
    3245            #[doc = concat!("[`", $s_int_type, "`].")]
    3246            ///
    3247            /// # Examples
    3248            ///
    3249            /// ```
    3250            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3251            ///
    3252            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(20);")]
    3253            /// assert_eq!(foo.fetch_sub(10, Ordering::SeqCst), 20);
    3254            /// assert_eq!(foo.load(Ordering::SeqCst), 10);
    3255            /// ```
    3256            #[inline]
    3257            #[$stable]
    3258            #[$cfg_cas]
    3259            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3260            #[rustc_should_not_be_called_on_const_items]
    3261            pub fn fetch_sub(&self, val: $int_type, order: Ordering) -> $int_type {
    3262                // SAFETY: data races are prevented by atomic intrinsics.
    3263                unsafe { atomic_sub(self.v.get(), val, order) }
    3264            }
    3265
    3266            /// Bitwise "and" with the current value.
    3267            ///
    3268            /// Performs a bitwise "and" operation on the current value and the argument `val`, and
    3269            /// sets the new value to the result.
    3270            ///
    3271            /// Returns the previous value.
    3272            ///
    3273            /// `fetch_and` takes an [`Ordering`] argument which describes the memory ordering
    3274            /// of this operation. All ordering modes are possible. Note that using
    3275            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    3276            /// using [`Release`] makes the load part [`Relaxed`].
    3277            ///
    3278            /// **Note**: This method is only available on platforms that support atomic operations on
    3279            #[doc = concat!("[`", $s_int_type, "`].")]
    3280            ///
    3281            /// # Examples
    3282            ///
    3283            /// ```
    3284            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3285            ///
    3286            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0b101101);")]
    3287            /// assert_eq!(foo.fetch_and(0b110011, Ordering::SeqCst), 0b101101);
    3288            /// assert_eq!(foo.load(Ordering::SeqCst), 0b100001);
    3289            /// ```
    3290            #[inline]
    3291            #[$stable]
    3292            #[$cfg_cas]
    3293            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3294            #[rustc_should_not_be_called_on_const_items]
    3295            pub fn fetch_and(&self, val: $int_type, order: Ordering) -> $int_type {
    3296                // SAFETY: data races are prevented by atomic intrinsics.
    3297                unsafe { atomic_and(self.v.get(), val, order) }
    3298            }
    3299
    3300            /// Bitwise "nand" with the current value.
    3301            ///
    3302            /// Performs a bitwise "nand" operation on the current value and the argument `val`, and
    3303            /// sets the new value to the result.
    3304            ///
    3305            /// Returns the previous value.
    3306            ///
    3307            /// `fetch_nand` takes an [`Ordering`] argument which describes the memory ordering
    3308            /// of this operation. All ordering modes are possible. Note that using
    3309            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    3310            /// using [`Release`] makes the load part [`Relaxed`].
    3311            ///
    3312            /// **Note**: This method is only available on platforms that support atomic operations on
    3313            #[doc = concat!("[`", $s_int_type, "`].")]
    3314            ///
    3315            /// # Examples
    3316            ///
    3317            /// ```
    3318            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3319            ///
    3320            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0x13);")]
    3321            /// assert_eq!(foo.fetch_nand(0x31, Ordering::SeqCst), 0x13);
    3322            /// assert_eq!(foo.load(Ordering::SeqCst), !(0x13 & 0x31));
    3323            /// ```
    3324            #[inline]
    3325            #[$stable_nand]
    3326            #[$cfg_cas]
    3327            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3328            #[rustc_should_not_be_called_on_const_items]
    3329            pub fn fetch_nand(&self, val: $int_type, order: Ordering) -> $int_type {
    3330                // SAFETY: data races are prevented by atomic intrinsics.
    3331                unsafe { atomic_nand(self.v.get(), val, order) }
    3332            }
    3333
    3334            /// Bitwise "or" with the current value.
    3335            ///
    3336            /// Performs a bitwise "or" operation on the current value and the argument `val`, and
    3337            /// sets the new value to the result.
    3338            ///
    3339            /// Returns the previous value.
    3340            ///
    3341            /// `fetch_or` takes an [`Ordering`] argument which describes the memory ordering
    3342            /// of this operation. All ordering modes are possible. Note that using
    3343            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    3344            /// using [`Release`] makes the load part [`Relaxed`].
    3345            ///
    3346            /// **Note**: This method is only available on platforms that support atomic operations on
    3347            #[doc = concat!("[`", $s_int_type, "`].")]
    3348            ///
    3349            /// # Examples
    3350            ///
    3351            /// ```
    3352            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3353            ///
    3354            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0b101101);")]
    3355            /// assert_eq!(foo.fetch_or(0b110011, Ordering::SeqCst), 0b101101);
    3356            /// assert_eq!(foo.load(Ordering::SeqCst), 0b111111);
    3357            /// ```
    3358            #[inline]
    3359            #[$stable]
    3360            #[$cfg_cas]
    3361            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3362            #[rustc_should_not_be_called_on_const_items]
    3363            pub fn fetch_or(&self, val: $int_type, order: Ordering) -> $int_type {
    3364                // SAFETY: data races are prevented by atomic intrinsics.
    3365                unsafe { atomic_or(self.v.get(), val, order) }
    3366            }
    3367
    3368            /// Bitwise "xor" with the current value.
    3369            ///
    3370            /// Performs a bitwise "xor" operation on the current value and the argument `val`, and
    3371            /// sets the new value to the result.
    3372            ///
    3373            /// Returns the previous value.
    3374            ///
    3375            /// `fetch_xor` takes an [`Ordering`] argument which describes the memory ordering
    3376            /// of this operation. All ordering modes are possible. Note that using
    3377            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    3378            /// using [`Release`] makes the load part [`Relaxed`].
    3379            ///
    3380            /// **Note**: This method is only available on platforms that support atomic operations on
    3381            #[doc = concat!("[`", $s_int_type, "`].")]
    3382            ///
    3383            /// # Examples
    3384            ///
    3385            /// ```
    3386            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3387            ///
    3388            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0b101101);")]
    3389            /// assert_eq!(foo.fetch_xor(0b110011, Ordering::SeqCst), 0b101101);
    3390            /// assert_eq!(foo.load(Ordering::SeqCst), 0b011110);
    3391            /// ```
    3392            #[inline]
    3393            #[$stable]
    3394            #[$cfg_cas]
    3395            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3396            #[rustc_should_not_be_called_on_const_items]
    3397            pub fn fetch_xor(&self, val: $int_type, order: Ordering) -> $int_type {
    3398                // SAFETY: data races are prevented by atomic intrinsics.
    3399                unsafe { atomic_xor(self.v.get(), val, order) }
    3400            }
    3401
    3402            /// Fetches the value, and applies a function to it that returns an optional
    3403            /// new value. Returns a `Result` of `Ok(previous_value)` if the function returned `Some(_)`, else
    3404            /// `Err(previous_value)`.
    3405            ///
    3406            /// Note: This may call the function multiple times if the value has been changed from other threads in
    3407            /// the meantime, as long as the function returns `Some(_)`, but the function will have been applied
    3408            /// only once to the stored value.
    3409            ///
    3410            /// `fetch_update` takes two [`Ordering`] arguments to describe the memory ordering of this operation.
    3411            /// The first describes the required ordering for when the operation finally succeeds while the second
    3412            /// describes the required ordering for loads. These correspond to the success and failure orderings of
    3413            #[doc = concat!("[`", stringify!($atomic_type), "::compare_exchange`]")]
    3414            /// respectively.
    3415            ///
    3416            /// Using [`Acquire`] as success ordering makes the store part
    3417            /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
    3418            /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    3419            ///
    3420            /// **Note**: This method is only available on platforms that support atomic operations on
    3421            #[doc = concat!("[`", $s_int_type, "`].")]
    3422            ///
    3423            /// # Considerations
    3424            ///
    3425            /// This method is not magic; it is not provided by the hardware, and does not act like a
    3426            /// critical section or mutex.
    3427            ///
    3428            /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    3429            /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem]
    3430            /// if this atomic integer is an index or more generally if knowledge of only the *bitwise value*
    3431            /// of the atomic is not in and of itself sufficient to ensure any required preconditions.
    3432            ///
    3433            /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    3434            /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    3435            ///
    3436            /// # Examples
    3437            ///
    3438            /// ```rust
    3439            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3440            ///
    3441            #[doc = concat!("let x = ", stringify!($atomic_type), "::new(7);")]
    3442            /// assert_eq!(x.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |_| None), Err(7));
    3443            /// assert_eq!(x.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(x + 1)), Ok(7));
    3444            /// assert_eq!(x.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(x + 1)), Ok(8));
    3445            /// assert_eq!(x.load(Ordering::SeqCst), 9);
    3446            /// ```
    3447            #[inline]
    3448            #[stable(feature = "no_more_cas", since = "1.45.0")]
    3449            #[$cfg_cas]
    3450            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3451            #[rustc_should_not_be_called_on_const_items]
    3452            pub fn fetch_update<F>(&self,
    3453                                   set_order: Ordering,
    3454                                   fetch_order: Ordering,
    3455                                   mut f: F) -> Result<$int_type, $int_type>
    3456            where F: FnMut($int_type) -> Option<$int_type> {
    3457                let mut prev = self.load(fetch_order);
    3458                while let Some(next) = f(prev) {
    3459                    match self.compare_exchange_weak(prev, next, set_order, fetch_order) {
    3460                        x @ Ok(_) => return x,
    3461                        Err(next_prev) => prev = next_prev
    3462                    }
    3463                }
    3464                Err(prev)
    3465            }
    3466
    3467            /// Fetches the value, and applies a function to it that returns an optional
    3468            /// new value. Returns a `Result` of `Ok(previous_value)` if the function returned `Some(_)`, else
    3469            /// `Err(previous_value)`.
    3470            ///
    3471            #[doc = concat!("See also: [`update`](`", stringify!($atomic_type), "::update`).")]
    3472            ///
    3473            /// Note: This may call the function multiple times if the value has been changed from other threads in
    3474            /// the meantime, as long as the function returns `Some(_)`, but the function will have been applied
    3475            /// only once to the stored value.
    3476            ///
    3477            /// `try_update` takes two [`Ordering`] arguments to describe the memory ordering of this operation.
    3478            /// The first describes the required ordering for when the operation finally succeeds while the second
    3479            /// describes the required ordering for loads. These correspond to the success and failure orderings of
    3480            #[doc = concat!("[`", stringify!($atomic_type), "::compare_exchange`]")]
    3481            /// respectively.
    3482            ///
    3483            /// Using [`Acquire`] as success ordering makes the store part
    3484            /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
    3485            /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    3486            ///
    3487            /// **Note**: This method is only available on platforms that support atomic operations on
    3488            #[doc = concat!("[`", $s_int_type, "`].")]
    3489            ///
    3490            /// # Considerations
    3491            ///
    3492            /// This method is not magic; it is not provided by the hardware, and does not act like a
    3493            /// critical section or mutex.
    3494            ///
    3495            /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    3496            /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem]
    3497            /// if this atomic integer is an index or more generally if knowledge of only the *bitwise value*
    3498            /// of the atomic is not in and of itself sufficient to ensure any required preconditions.
    3499            ///
    3500            /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    3501            /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    3502            ///
    3503            /// # Examples
    3504            ///
    3505            /// ```rust
    3506            /// #![feature(atomic_try_update)]
    3507            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3508            ///
    3509            #[doc = concat!("let x = ", stringify!($atomic_type), "::new(7);")]
    3510            /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |_| None), Err(7));
    3511            /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(x + 1)), Ok(7));
    3512            /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(x + 1)), Ok(8));
    3513            /// assert_eq!(x.load(Ordering::SeqCst), 9);
    3514            /// ```
    3515            #[inline]
    3516            #[unstable(feature = "atomic_try_update", issue = "135894")]
    3517            #[$cfg_cas]
    3518            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3519            #[rustc_should_not_be_called_on_const_items]
    3520            pub fn try_update(
    3521                &self,
    3522                set_order: Ordering,
    3523                fetch_order: Ordering,
    3524                f: impl FnMut($int_type) -> Option<$int_type>,
    3525            ) -> Result<$int_type, $int_type> {
    3526                // FIXME(atomic_try_update): this is currently an unstable alias to `fetch_update`;
    3527                //      when stabilizing, turn `fetch_update` into a deprecated alias to `try_update`.
    3528                self.fetch_update(set_order, fetch_order, f)
    3529            }
    3530
    3531            /// Fetches the value, applies a function to it that it return a new value.
    3532            /// The new value is stored and the old value is returned.
    3533            ///
    3534            #[doc = concat!("See also: [`try_update`](`", stringify!($atomic_type), "::try_update`).")]
    3535            ///
    3536            /// Note: This may call the function multiple times if the value has been changed from other threads in
    3537            /// the meantime, but the function will have been applied only once to the stored value.
    3538            ///
    3539            /// `update` takes two [`Ordering`] arguments to describe the memory ordering of this operation.
    3540            /// The first describes the required ordering for when the operation finally succeeds while the second
    3541            /// describes the required ordering for loads. These correspond to the success and failure orderings of
    3542            #[doc = concat!("[`", stringify!($atomic_type), "::compare_exchange`]")]
    3543            /// respectively.
    3544            ///
    3545            /// Using [`Acquire`] as success ordering makes the store part
    3546            /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
    3547            /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    3548            ///
    3549            /// **Note**: This method is only available on platforms that support atomic operations on
    3550            #[doc = concat!("[`", $s_int_type, "`].")]
    3551            ///
    3552            /// # Considerations
    3553            ///
    3554            /// [CAS operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    3555            /// This method is not magic; it is not provided by the hardware, and does not act like a
    3556            /// critical section or mutex.
    3557            ///
    3558            /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    3559            /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem]
    3560            /// if this atomic integer is an index or more generally if knowledge of only the *bitwise value*
    3561            /// of the atomic is not in and of itself sufficient to ensure any required preconditions.
    3562            ///
    3563            /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    3564            /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    3565            ///
    3566            /// # Examples
    3567            ///
    3568            /// ```rust
    3569            /// #![feature(atomic_try_update)]
    3570            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3571            ///
    3572            #[doc = concat!("let x = ", stringify!($atomic_type), "::new(7);")]
    3573            /// assert_eq!(x.update(Ordering::SeqCst, Ordering::SeqCst, |x| x + 1), 7);
    3574            /// assert_eq!(x.update(Ordering::SeqCst, Ordering::SeqCst, |x| x + 1), 8);
    3575            /// assert_eq!(x.load(Ordering::SeqCst), 9);
    3576            /// ```
    3577            #[inline]
    3578            #[unstable(feature = "atomic_try_update", issue = "135894")]
    3579            #[$cfg_cas]
    3580            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3581            #[rustc_should_not_be_called_on_const_items]
    3582            pub fn update(
    3583                &self,
    3584                set_order: Ordering,
    3585                fetch_order: Ordering,
    3586                mut f: impl FnMut($int_type) -> $int_type,
    3587            ) -> $int_type {
    3588                let mut prev = self.load(fetch_order);
    3589                loop {
    3590                    match self.compare_exchange_weak(prev, f(prev), set_order, fetch_order) {
    3591                        Ok(x) => break x,
    3592                        Err(next_prev) => prev = next_prev,
    3593                    }
    3594                }
    3595            }
    3596
    3597            /// Maximum with the current value.
    3598            ///
    3599            /// Finds the maximum of the current value and the argument `val`, and
    3600            /// sets the new value to the result.
    3601            ///
    3602            /// Returns the previous value.
    3603            ///
    3604            /// `fetch_max` takes an [`Ordering`] argument which describes the memory ordering
    3605            /// of this operation. All ordering modes are possible. Note that using
    3606            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    3607            /// using [`Release`] makes the load part [`Relaxed`].
    3608            ///
    3609            /// **Note**: This method is only available on platforms that support atomic operations on
    3610            #[doc = concat!("[`", $s_int_type, "`].")]
    3611            ///
    3612            /// # Examples
    3613            ///
    3614            /// ```
    3615            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3616            ///
    3617            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(23);")]
    3618            /// assert_eq!(foo.fetch_max(42, Ordering::SeqCst), 23);
    3619            /// assert_eq!(foo.load(Ordering::SeqCst), 42);
    3620            /// ```
    3621            ///
    3622            /// If you want to obtain the maximum value in one step, you can use the following:
    3623            ///
    3624            /// ```
    3625            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3626            ///
    3627            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(23);")]
    3628            /// let bar = 42;
    3629            /// let max_foo = foo.fetch_max(bar, Ordering::SeqCst).max(bar);
    3630            /// assert!(max_foo == 42);
    3631            /// ```
    3632            #[inline]
    3633            #[stable(feature = "atomic_min_max", since = "1.45.0")]
    3634            #[$cfg_cas]
    3635            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3636            #[rustc_should_not_be_called_on_const_items]
    3637            pub fn fetch_max(&self, val: $int_type, order: Ordering) -> $int_type {
    3638                // SAFETY: data races are prevented by atomic intrinsics.
    3639                unsafe { $max_fn(self.v.get(), val, order) }
    3640            }
    3641
    3642            /// Minimum with the current value.
    3643            ///
    3644            /// Finds the minimum of the current value and the argument `val`, and
    3645            /// sets the new value to the result.
    3646            ///
    3647            /// Returns the previous value.
    3648            ///
    3649            /// `fetch_min` takes an [`Ordering`] argument which describes the memory ordering
    3650            /// of this operation. All ordering modes are possible. Note that using
    3651            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    3652            /// using [`Release`] makes the load part [`Relaxed`].
    3653            ///
    3654            /// **Note**: This method is only available on platforms that support atomic operations on
    3655            #[doc = concat!("[`", $s_int_type, "`].")]
    3656            ///
    3657            /// # Examples
    3658            ///
    3659            /// ```
    3660            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3661            ///
    3662            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(23);")]
    3663            /// assert_eq!(foo.fetch_min(42, Ordering::Relaxed), 23);
    3664            /// assert_eq!(foo.load(Ordering::Relaxed), 23);
    3665            /// assert_eq!(foo.fetch_min(22, Ordering::Relaxed), 23);
    3666            /// assert_eq!(foo.load(Ordering::Relaxed), 22);
    3667            /// ```
    3668            ///
    3669            /// If you want to obtain the minimum value in one step, you can use the following:
    3670            ///
    3671            /// ```
    3672            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
    3673            ///
    3674            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(23);")]
    3675            /// let bar = 12;
    3676            /// let min_foo = foo.fetch_min(bar, Ordering::SeqCst).min(bar);
    3677            /// assert_eq!(min_foo, 12);
    3678            /// ```
    3679            #[inline]
    3680            #[stable(feature = "atomic_min_max", since = "1.45.0")]
    3681            #[$cfg_cas]
    3682            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    3683            #[rustc_should_not_be_called_on_const_items]
    3684            pub fn fetch_min(&self, val: $int_type, order: Ordering) -> $int_type {
    3685                // SAFETY: data races are prevented by atomic intrinsics.
    3686                unsafe { $min_fn(self.v.get(), val, order) }
    3687            }
    3688
    3689            /// Returns a mutable pointer to the underlying integer.
    3690            ///
    3691            /// Doing non-atomic reads and writes on the resulting integer can be a data race.
    3692            /// This method is mostly useful for FFI, where the function signature may use
    3693            #[doc = concat!("`*mut ", stringify!($int_type), "` instead of `&", stringify!($atomic_type), "`.")]
    3694            ///
    3695            /// Returning an `*mut` pointer from a shared reference to this atomic is safe because the
    3696            /// atomic types work with interior mutability. All modifications of an atomic change the value
    3697            /// through a shared reference, and can do so safely as long as they use atomic operations. Any
    3698            /// use of the returned raw pointer requires an `unsafe` block and still has to uphold the
    3699            /// requirements of the [memory model].
    3700            ///
    3701            /// # Examples
    3702            ///
    3703            /// ```ignore (extern-declaration)
    3704            /// # fn main() {
    3705            #[doc = concat!($extra_feature, "use std::sync::atomic::", stringify!($atomic_type), ";")]
    3706            ///
    3707            /// extern "C" {
    3708            #[doc = concat!("    fn my_atomic_op(arg: *mut ", stringify!($int_type), ");")]
    3709            /// }
    3710            ///
    3711            #[doc = concat!("let atomic = ", stringify!($atomic_type), "::new(1);")]
    3712            ///
    3713            /// // SAFETY: Safe as long as `my_atomic_op` is atomic.
    3714            /// unsafe {
    3715            ///     my_atomic_op(atomic.as_ptr());
    3716            /// }
    3717            /// # }
    3718            /// ```
    3719            ///
    3720            /// [memory model]: self#memory-model-for-atomic-accesses
    3721            #[inline]
    3722            #[stable(feature = "atomic_as_ptr", since = "1.70.0")]
    3723            #[rustc_const_stable(feature = "atomic_as_ptr", since = "1.70.0")]
    3724            #[rustc_never_returns_null_ptr]
    3725            pub const fn as_ptr(&self) -> *mut $int_type {
    3726                self.v.get()
    3727            }
    3728        }
    3729    }
    3730}
    3731
    3732#[cfg(target_has_atomic_load_store = "8")]
    3733atomic_int! {
    3734    cfg(target_has_atomic = "8"),
    3735    cfg(target_has_atomic_equal_alignment = "8"),
    3736    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3737    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3738    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3739    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3740    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3741    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3742    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
    3743    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3744    rustc_diagnostic_item = "AtomicI8",
    3745    "i8",
    3746    "",
    3747    atomic_min, atomic_max,
    3748    1,
    3749    i8 AtomicI8
    3750}
    3751#[cfg(target_has_atomic_load_store = "8")]
    3752atomic_int! {
    3753    cfg(target_has_atomic = "8"),
    3754    cfg(target_has_atomic_equal_alignment = "8"),
    3755    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3756    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3757    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3758    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3759    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3760    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3761    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
    3762    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3763    rustc_diagnostic_item = "AtomicU8",
    3764    "u8",
    3765    "",
    3766    atomic_umin, atomic_umax,
    3767    1,
    3768    u8 AtomicU8
    3769}
    3770#[cfg(target_has_atomic_load_store = "16")]
    3771atomic_int! {
    3772    cfg(target_has_atomic = "16"),
    3773    cfg(target_has_atomic_equal_alignment = "16"),
    3774    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3775    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3776    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3777    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3778    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3779    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3780    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
    3781    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3782    rustc_diagnostic_item = "AtomicI16",
    3783    "i16",
    3784    "",
    3785    atomic_min, atomic_max,
    3786    2,
    3787    i16 AtomicI16
    3788}
    3789#[cfg(target_has_atomic_load_store = "16")]
    3790atomic_int! {
    3791    cfg(target_has_atomic = "16"),
    3792    cfg(target_has_atomic_equal_alignment = "16"),
    3793    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3794    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3795    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3796    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3797    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3798    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3799    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
    3800    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3801    rustc_diagnostic_item = "AtomicU16",
    3802    "u16",
    3803    "",
    3804    atomic_umin, atomic_umax,
    3805    2,
    3806    u16 AtomicU16
    3807}
    3808#[cfg(target_has_atomic_load_store = "32")]
    3809atomic_int! {
    3810    cfg(target_has_atomic = "32"),
    3811    cfg(target_has_atomic_equal_alignment = "32"),
    3812    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3813    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3814    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3815    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3816    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3817    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3818    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
    3819    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3820    rustc_diagnostic_item = "AtomicI32",
    3821    "i32",
    3822    "",
    3823    atomic_min, atomic_max,
    3824    4,
    3825    i32 AtomicI32
    3826}
    3827#[cfg(target_has_atomic_load_store = "32")]
    3828atomic_int! {
    3829    cfg(target_has_atomic = "32"),
    3830    cfg(target_has_atomic_equal_alignment = "32"),
    3831    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3832    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3833    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3834    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3835    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3836    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3837    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
    3838    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3839    rustc_diagnostic_item = "AtomicU32",
    3840    "u32",
    3841    "",
    3842    atomic_umin, atomic_umax,
    3843    4,
    3844    u32 AtomicU32
    3845}
    3846#[cfg(target_has_atomic_load_store = "64")]
    3847atomic_int! {
    3848    cfg(target_has_atomic = "64"),
    3849    cfg(target_has_atomic_equal_alignment = "64"),
    3850    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3851    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3852    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3853    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3854    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3855    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3856    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
    3857    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3858    rustc_diagnostic_item = "AtomicI64",
    3859    "i64",
    3860    "",
    3861    atomic_min, atomic_max,
    3862    8,
    3863    i64 AtomicI64
    3864}
    3865#[cfg(target_has_atomic_load_store = "64")]
    3866atomic_int! {
    3867    cfg(target_has_atomic = "64"),
    3868    cfg(target_has_atomic_equal_alignment = "64"),
    3869    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3870    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3871    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3872    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3873    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3874    stable(feature = "integer_atomics_stable", since = "1.34.0"),
    3875    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
    3876    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3877    rustc_diagnostic_item = "AtomicU64",
    3878    "u64",
    3879    "",
    3880    atomic_umin, atomic_umax,
    3881    8,
    3882    u64 AtomicU64
    3883}
    3884#[cfg(target_has_atomic_load_store = "128")]
    3885atomic_int! {
    3886    cfg(target_has_atomic = "128"),
    3887    cfg(target_has_atomic_equal_alignment = "128"),
    3888    unstable(feature = "integer_atomics", issue = "99069"),
    3889    unstable(feature = "integer_atomics", issue = "99069"),
    3890    unstable(feature = "integer_atomics", issue = "99069"),
    3891    unstable(feature = "integer_atomics", issue = "99069"),
    3892    unstable(feature = "integer_atomics", issue = "99069"),
    3893    unstable(feature = "integer_atomics", issue = "99069"),
    3894    rustc_const_unstable(feature = "integer_atomics", issue = "99069"),
    3895    rustc_const_unstable(feature = "integer_atomics", issue = "99069"),
    3896    rustc_diagnostic_item = "AtomicI128",
    3897    "i128",
    3898    "#![feature(integer_atomics)]\n\n",
    3899    atomic_min, atomic_max,
    3900    16,
    3901    i128 AtomicI128
    3902}
    3903#[cfg(target_has_atomic_load_store = "128")]
    3904atomic_int! {
    3905    cfg(target_has_atomic = "128"),
    3906    cfg(target_has_atomic_equal_alignment = "128"),
    3907    unstable(feature = "integer_atomics", issue = "99069"),
    3908    unstable(feature = "integer_atomics", issue = "99069"),
    3909    unstable(feature = "integer_atomics", issue = "99069"),
    3910    unstable(feature = "integer_atomics", issue = "99069"),
    3911    unstable(feature = "integer_atomics", issue = "99069"),
    3912    unstable(feature = "integer_atomics", issue = "99069"),
    3913    rustc_const_unstable(feature = "integer_atomics", issue = "99069"),
    3914    rustc_const_unstable(feature = "integer_atomics", issue = "99069"),
    3915    rustc_diagnostic_item = "AtomicU128",
    3916    "u128",
    3917    "#![feature(integer_atomics)]\n\n",
    3918    atomic_umin, atomic_umax,
    3919    16,
    3920    u128 AtomicU128
    3921}
    3922
    3923#[cfg(target_has_atomic_load_store = "ptr")]
    3924macro_rules! atomic_int_ptr_sized {
    3925    ( $($target_pointer_width:literal $align:literal)* ) => { $(
    3926        #[cfg(target_pointer_width = $target_pointer_width)]
    3927        atomic_int! {
    3928            cfg(target_has_atomic = "ptr"),
    3929            cfg(target_has_atomic_equal_alignment = "ptr"),
    3930            stable(feature = "rust1", since = "1.0.0"),
    3931            stable(feature = "extended_compare_and_swap", since = "1.10.0"),
    3932            stable(feature = "atomic_debug", since = "1.3.0"),
    3933            stable(feature = "atomic_access", since = "1.15.0"),
    3934            stable(feature = "atomic_from", since = "1.23.0"),
    3935            stable(feature = "atomic_nand", since = "1.27.0"),
    3936            rustc_const_stable(feature = "const_ptr_sized_atomics", since = "1.24.0"),
    3937            rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3938            rustc_diagnostic_item = "AtomicIsize",
    3939            "isize",
    3940            "",
    3941            atomic_min, atomic_max,
    3942            $align,
    3943            isize AtomicIsize
    3944        }
    3945        #[cfg(target_pointer_width = $target_pointer_width)]
    3946        atomic_int! {
    3947            cfg(target_has_atomic = "ptr"),
    3948            cfg(target_has_atomic_equal_alignment = "ptr"),
    3949            stable(feature = "rust1", since = "1.0.0"),
    3950            stable(feature = "extended_compare_and_swap", since = "1.10.0"),
    3951            stable(feature = "atomic_debug", since = "1.3.0"),
    3952            stable(feature = "atomic_access", since = "1.15.0"),
    3953            stable(feature = "atomic_from", since = "1.23.0"),
    3954            stable(feature = "atomic_nand", since = "1.27.0"),
    3955            rustc_const_stable(feature = "const_ptr_sized_atomics", since = "1.24.0"),
    3956            rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
    3957            rustc_diagnostic_item = "AtomicUsize",
    3958            "usize",
    3959            "",
    3960            atomic_umin, atomic_umax,
    3961            $align,
    3962            usize AtomicUsize
    3963        }
    3964
    3965        /// An [`AtomicIsize`] initialized to `0`.
    3966        #[cfg(target_pointer_width = $target_pointer_width)]
    3967        #[stable(feature = "rust1", since = "1.0.0")]
    3968        #[deprecated(
    3969            since = "1.34.0",
    3970            note = "the `new` function is now preferred",
    3971            suggestion = "AtomicIsize::new(0)",
    3972        )]
    3973        pub const ATOMIC_ISIZE_INIT: AtomicIsize = AtomicIsize::new(0);
    3974
    3975        /// An [`AtomicUsize`] initialized to `0`.
    3976        #[cfg(target_pointer_width = $target_pointer_width)]
    3977        #[stable(feature = "rust1", since = "1.0.0")]
    3978        #[deprecated(
    3979            since = "1.34.0",
    3980            note = "the `new` function is now preferred",
    3981            suggestion = "AtomicUsize::new(0)",
    3982        )]
    3983        pub const ATOMIC_USIZE_INIT: AtomicUsize = AtomicUsize::new(0);
    3984    )* };
    3985}
    3986
    3987#[cfg(target_has_atomic_load_store = "ptr")]
    3988atomic_int_ptr_sized! {
    3989    "16" 2
    3990    "32" 4
    3991    "64" 8
    3992}
    3993
    3994#[inline]
    3995#[cfg(target_has_atomic)]
    3996fn strongest_failure_ordering(order: Ordering) -> Ordering {
    3997    match order {
    3998        Release => Relaxed,
    3999        Relaxed => Relaxed,
    4000        SeqCst => SeqCst,
    4001        Acquire => Acquire,
    4002        AcqRel => Acquire,
    4003    }
    4004}
    4005
    4006#[inline]
    4007#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4008unsafe fn atomic_store<T: Copy>(dst: *mut T, val: T, order: Ordering) {
    4009    // SAFETY: the caller must uphold the safety contract for `atomic_store`.
    4010    unsafe {
    4011        match order {
    4012            Relaxed => intrinsics::atomic_store::<T, { AO::Relaxed }>(dst, val),
    4013            Release => intrinsics::atomic_store::<T, { AO::Release }>(dst, val),
    4014            SeqCst => intrinsics::atomic_store::<T, { AO::SeqCst }>(dst, val),
    4015            Acquire => panic!("there is no such thing as an acquire store"),
    4016            AcqRel => panic!("there is no such thing as an acquire-release store"),
    4017        }
    4018    }
    4019}
    4020
    4021#[inline]
    4022#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4023unsafe fn atomic_load<T: Copy>(dst: *const T, order: Ordering) -> T {
    4024    // SAFETY: the caller must uphold the safety contract for `atomic_load`.
    4025    unsafe {
    4026        match order {
    4027            Relaxed => intrinsics::atomic_load::<T, { AO::Relaxed }>(dst),
    4028            Acquire => intrinsics::atomic_load::<T, { AO::Acquire }>(dst),
    4029            SeqCst => intrinsics::atomic_load::<T, { AO::SeqCst }>(dst),
    4030            Release => panic!("there is no such thing as a release load"),
    4031            AcqRel => panic!("there is no such thing as an acquire-release load"),
    4032        }
    4033    }
    4034}
    4035
    4036#[inline]
    4037#[cfg(target_has_atomic)]
    4038#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4039unsafe fn atomic_swap<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
    4040    // SAFETY: the caller must uphold the safety contract for `atomic_swap`.
    4041    unsafe {
    4042        match order {
    4043            Relaxed => intrinsics::atomic_xchg::<T, { AO::Relaxed }>(dst, val),
    4044            Acquire => intrinsics::atomic_xchg::<T, { AO::Acquire }>(dst, val),
    4045            Release => intrinsics::atomic_xchg::<T, { AO::Release }>(dst, val),
    4046            AcqRel => intrinsics::atomic_xchg::<T, { AO::AcqRel }>(dst, val),
    4047            SeqCst => intrinsics::atomic_xchg::<T, { AO::SeqCst }>(dst, val),
    4048        }
    4049    }
    4050}
    4051
    4052/// Returns the previous value (like __sync_fetch_and_add).
    4053#[inline]
    4054#[cfg(target_has_atomic)]
    4055#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4056unsafe fn atomic_add<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
    4057    // SAFETY: the caller must uphold the safety contract for `atomic_add`.
    4058    unsafe {
    4059        match order {
    4060            Relaxed => intrinsics::atomic_xadd::<T, U, { AO::Relaxed }>(dst, val),
    4061            Acquire => intrinsics::atomic_xadd::<T, U, { AO::Acquire }>(dst, val),
    4062            Release => intrinsics::atomic_xadd::<T, U, { AO::Release }>(dst, val),
    4063            AcqRel => intrinsics::atomic_xadd::<T, U, { AO::AcqRel }>(dst, val),
    4064            SeqCst => intrinsics::atomic_xadd::<T, U, { AO::SeqCst }>(dst, val),
    4065        }
    4066    }
    4067}
    4068
    4069/// Returns the previous value (like __sync_fetch_and_sub).
    4070#[inline]
    4071#[cfg(target_has_atomic)]
    4072#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4073unsafe fn atomic_sub<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
    4074    // SAFETY: the caller must uphold the safety contract for `atomic_sub`.
    4075    unsafe {
    4076        match order {
    4077            Relaxed => intrinsics::atomic_xsub::<T, U, { AO::Relaxed }>(dst, val),
    4078            Acquire => intrinsics::atomic_xsub::<T, U, { AO::Acquire }>(dst, val),
    4079            Release => intrinsics::atomic_xsub::<T, U, { AO::Release }>(dst, val),
    4080            AcqRel => intrinsics::atomic_xsub::<T, U, { AO::AcqRel }>(dst, val),
    4081            SeqCst => intrinsics::atomic_xsub::<T, U, { AO::SeqCst }>(dst, val),
    4082        }
    4083    }
    4084}
    4085
    4086/// Publicly exposed for stdarch; nobody else should use this.
    4087#[inline]
    4088#[cfg(target_has_atomic)]
    4089#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4090#[unstable(feature = "core_intrinsics", issue = "none")]
    4091#[doc(hidden)]
    4092pub unsafe fn atomic_compare_exchange<T: Copy>(
    4093    dst: *mut T,
    4094    old: T,
    4095    new: T,
    4096    success: Ordering,
    4097    failure: Ordering,
    4098) -> Result<T, T> {
    4099    // SAFETY: the caller must uphold the safety contract for `atomic_compare_exchange`.
    4100    let (val, ok) = unsafe {
    4101        match (success, failure) {
    4102            (Relaxed, Relaxed) => {
    4103                intrinsics::atomic_cxchg::<T, { AO::Relaxed }, { AO::Relaxed }>(dst, old, new)
    4104            }
    4105            (Relaxed, Acquire) => {
    4106                intrinsics::atomic_cxchg::<T, { AO::Relaxed }, { AO::Acquire }>(dst, old, new)
    4107            }
    4108            (Relaxed, SeqCst) => {
    4109                intrinsics::atomic_cxchg::<T, { AO::Relaxed }, { AO::SeqCst }>(dst, old, new)
    4110            }
    4111            (Acquire, Relaxed) => {
    4112                intrinsics::atomic_cxchg::<T, { AO::Acquire }, { AO::Relaxed }>(dst, old, new)
    4113            }
    4114            (Acquire, Acquire) => {
    4115                intrinsics::atomic_cxchg::<T, { AO::Acquire }, { AO::Acquire }>(dst, old, new)
    4116            }
    4117            (Acquire, SeqCst) => {
    4118                intrinsics::atomic_cxchg::<T, { AO::Acquire }, { AO::SeqCst }>(dst, old, new)
    4119            }
    4120            (Release, Relaxed) => {
    4121                intrinsics::atomic_cxchg::<T, { AO::Release }, { AO::Relaxed }>(dst, old, new)
    4122            }
    4123            (Release, Acquire) => {
    4124                intrinsics::atomic_cxchg::<T, { AO::Release }, { AO::Acquire }>(dst, old, new)
    4125            }
    4126            (Release, SeqCst) => {
    4127                intrinsics::atomic_cxchg::<T, { AO::Release }, { AO::SeqCst }>(dst, old, new)
    4128            }
    4129            (AcqRel, Relaxed) => {
    4130                intrinsics::atomic_cxchg::<T, { AO::AcqRel }, { AO::Relaxed }>(dst, old, new)
    4131            }
    4132            (AcqRel, Acquire) => {
    4133                intrinsics::atomic_cxchg::<T, { AO::AcqRel }, { AO::Acquire }>(dst, old, new)
    4134            }
    4135            (AcqRel, SeqCst) => {
    4136                intrinsics::atomic_cxchg::<T, { AO::AcqRel }, { AO::SeqCst }>(dst, old, new)
    4137            }
    4138            (SeqCst, Relaxed) => {
    4139                intrinsics::atomic_cxchg::<T, { AO::SeqCst }, { AO::Relaxed }>(dst, old, new)
    4140            }
    4141            (SeqCst, Acquire) => {
    4142                intrinsics::atomic_cxchg::<T, { AO::SeqCst }, { AO::Acquire }>(dst, old, new)
    4143            }
    4144            (SeqCst, SeqCst) => {
    4145                intrinsics::atomic_cxchg::<T, { AO::SeqCst }, { AO::SeqCst }>(dst, old, new)
    4146            }
    4147            (_, AcqRel) => panic!("there is no such thing as an acquire-release failure ordering"),
    4148            (_, Release) => panic!("there is no such thing as a release failure ordering"),
    4149        }
    4150    };
    4151    if ok { Ok(val) } else { Err(val) }
    4152}
    4153
    4154#[inline]
    4155#[cfg(target_has_atomic)]
    4156#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4157unsafe fn atomic_compare_exchange_weak<T: Copy>(
    4158    dst: *mut T,
    4159    old: T,
    4160    new: T,
    4161    success: Ordering,
    4162    failure: Ordering,
    4163) -> Result<T, T> {
    4164    // SAFETY: the caller must uphold the safety contract for `atomic_compare_exchange_weak`.
    4165    let (val, ok) = unsafe {
    4166        match (success, failure) {
    4167            (Relaxed, Relaxed) => {
    4168                intrinsics::atomic_cxchgweak::<T, { AO::Relaxed }, { AO::Relaxed }>(dst, old, new)
    4169            }
    4170            (Relaxed, Acquire) => {
    4171                intrinsics::atomic_cxchgweak::<T, { AO::Relaxed }, { AO::Acquire }>(dst, old, new)
    4172            }
    4173            (Relaxed, SeqCst) => {
    4174                intrinsics::atomic_cxchgweak::<T, { AO::Relaxed }, { AO::SeqCst }>(dst, old, new)
    4175            }
    4176            (Acquire, Relaxed) => {
    4177                intrinsics::atomic_cxchgweak::<T, { AO::Acquire }, { AO::Relaxed }>(dst, old, new)
    4178            }
    4179            (Acquire, Acquire) => {
    4180                intrinsics::atomic_cxchgweak::<T, { AO::Acquire }, { AO::Acquire }>(dst, old, new)
    4181            }
    4182            (Acquire, SeqCst) => {
    4183                intrinsics::atomic_cxchgweak::<T, { AO::Acquire }, { AO::SeqCst }>(dst, old, new)
    4184            }
    4185            (Release, Relaxed) => {
    4186                intrinsics::atomic_cxchgweak::<T, { AO::Release }, { AO::Relaxed }>(dst, old, new)
    4187            }
    4188            (Release, Acquire) => {
    4189                intrinsics::atomic_cxchgweak::<T, { AO::Release }, { AO::Acquire }>(dst, old, new)
    4190            }
    4191            (Release, SeqCst) => {
    4192                intrinsics::atomic_cxchgweak::<T, { AO::Release }, { AO::SeqCst }>(dst, old, new)
    4193            }
    4194            (AcqRel, Relaxed) => {
    4195                intrinsics::atomic_cxchgweak::<T, { AO::AcqRel }, { AO::Relaxed }>(dst, old, new)
    4196            }
    4197            (AcqRel, Acquire) => {
    4198                intrinsics::atomic_cxchgweak::<T, { AO::AcqRel }, { AO::Acquire }>(dst, old, new)
    4199            }
    4200            (AcqRel, SeqCst) => {
    4201                intrinsics::atomic_cxchgweak::<T, { AO::AcqRel }, { AO::SeqCst }>(dst, old, new)
    4202            }
    4203            (SeqCst, Relaxed) => {
    4204                intrinsics::atomic_cxchgweak::<T, { AO::SeqCst }, { AO::Relaxed }>(dst, old, new)
    4205            }
    4206            (SeqCst, Acquire) => {
    4207                intrinsics::atomic_cxchgweak::<T, { AO::SeqCst }, { AO::Acquire }>(dst, old, new)
    4208            }
    4209            (SeqCst, SeqCst) => {
    4210                intrinsics::atomic_cxchgweak::<T, { AO::SeqCst }, { AO::SeqCst }>(dst, old, new)
    4211            }
    4212            (_, AcqRel) => panic!("there is no such thing as an acquire-release failure ordering"),
    4213            (_, Release) => panic!("there is no such thing as a release failure ordering"),
    4214        }
    4215    };
    4216    if ok { Ok(val) } else { Err(val) }
    4217}
    4218
    4219#[inline]
    4220#[cfg(target_has_atomic)]
    4221#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4222unsafe fn atomic_and<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
    4223    // SAFETY: the caller must uphold the safety contract for `atomic_and`
    4224    unsafe {
    4225        match order {
    4226            Relaxed => intrinsics::atomic_and::<T, U, { AO::Relaxed }>(dst, val),
    4227            Acquire => intrinsics::atomic_and::<T, U, { AO::Acquire }>(dst, val),
    4228            Release => intrinsics::atomic_and::<T, U, { AO::Release }>(dst, val),
    4229            AcqRel => intrinsics::atomic_and::<T, U, { AO::AcqRel }>(dst, val),
    4230            SeqCst => intrinsics::atomic_and::<T, U, { AO::SeqCst }>(dst, val),
    4231        }
    4232    }
    4233}
    4234
    4235#[inline]
    4236#[cfg(target_has_atomic)]
    4237#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4238unsafe fn atomic_nand<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
    4239    // SAFETY: the caller must uphold the safety contract for `atomic_nand`
    4240    unsafe {
    4241        match order {
    4242            Relaxed => intrinsics::atomic_nand::<T, U, { AO::Relaxed }>(dst, val),
    4243            Acquire => intrinsics::atomic_nand::<T, U, { AO::Acquire }>(dst, val),
    4244            Release => intrinsics::atomic_nand::<T, U, { AO::Release }>(dst, val),
    4245            AcqRel => intrinsics::atomic_nand::<T, U, { AO::AcqRel }>(dst, val),
    4246            SeqCst => intrinsics::atomic_nand::<T, U, { AO::SeqCst }>(dst, val),
    4247        }
    4248    }
    4249}
    4250
    4251#[inline]
    4252#[cfg(target_has_atomic)]
    4253#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4254unsafe fn atomic_or<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
    4255    // SAFETY: the caller must uphold the safety contract for `atomic_or`
    4256    unsafe {
    4257        match order {
    4258            SeqCst => intrinsics::atomic_or::<T, U, { AO::SeqCst }>(dst, val),
    4259            Acquire => intrinsics::atomic_or::<T, U, { AO::Acquire }>(dst, val),
    4260            Release => intrinsics::atomic_or::<T, U, { AO::Release }>(dst, val),
    4261            AcqRel => intrinsics::atomic_or::<T, U, { AO::AcqRel }>(dst, val),
    4262            Relaxed => intrinsics::atomic_or::<T, U, { AO::Relaxed }>(dst, val),
    4263        }
    4264    }
    4265}
    4266
    4267#[inline]
    4268#[cfg(target_has_atomic)]
    4269#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4270unsafe fn atomic_xor<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
    4271    // SAFETY: the caller must uphold the safety contract for `atomic_xor`
    4272    unsafe {
    4273        match order {
    4274            SeqCst => intrinsics::atomic_xor::<T, U, { AO::SeqCst }>(dst, val),
    4275            Acquire => intrinsics::atomic_xor::<T, U, { AO::Acquire }>(dst, val),
    4276            Release => intrinsics::atomic_xor::<T, U, { AO::Release }>(dst, val),
    4277            AcqRel => intrinsics::atomic_xor::<T, U, { AO::AcqRel }>(dst, val),
    4278            Relaxed => intrinsics::atomic_xor::<T, U, { AO::Relaxed }>(dst, val),
    4279        }
    4280    }
    4281}
    4282
    4283/// Updates `*dst` to the max value of `val` and the old value (signed comparison)
    4284#[inline]
    4285#[cfg(target_has_atomic)]
    4286#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4287unsafe fn atomic_max<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
    4288    // SAFETY: the caller must uphold the safety contract for `atomic_max`
    4289    unsafe {
    4290        match order {
    4291            Relaxed => intrinsics::atomic_max::<T, { AO::Relaxed }>(dst, val),
    4292            Acquire => intrinsics::atomic_max::<T, { AO::Acquire }>(dst, val),
    4293            Release => intrinsics::atomic_max::<T, { AO::Release }>(dst, val),
    4294            AcqRel => intrinsics::atomic_max::<T, { AO::AcqRel }>(dst, val),
    4295            SeqCst => intrinsics::atomic_max::<T, { AO::SeqCst }>(dst, val),
    4296        }
    4297    }
    4298}
    4299
    4300/// Updates `*dst` to the min value of `val` and the old value (signed comparison)
    4301#[inline]
    4302#[cfg(target_has_atomic)]
    4303#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4304unsafe fn atomic_min<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
    4305    // SAFETY: the caller must uphold the safety contract for `atomic_min`
    4306    unsafe {
    4307        match order {
    4308            Relaxed => intrinsics::atomic_min::<T, { AO::Relaxed }>(dst, val),
    4309            Acquire => intrinsics::atomic_min::<T, { AO::Acquire }>(dst, val),
    4310            Release => intrinsics::atomic_min::<T, { AO::Release }>(dst, val),
    4311            AcqRel => intrinsics::atomic_min::<T, { AO::AcqRel }>(dst, val),
    4312            SeqCst => intrinsics::atomic_min::<T, { AO::SeqCst }>(dst, val),
    4313        }
    4314    }
    4315}
    4316
    4317/// Updates `*dst` to the max value of `val` and the old value (unsigned comparison)
    4318#[inline]
    4319#[cfg(target_has_atomic)]
    4320#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4321unsafe fn atomic_umax<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
    4322    // SAFETY: the caller must uphold the safety contract for `atomic_umax`
    4323    unsafe {
    4324        match order {
    4325            Relaxed => intrinsics::atomic_umax::<T, { AO::Relaxed }>(dst, val),
    4326            Acquire => intrinsics::atomic_umax::<T, { AO::Acquire }>(dst, val),
    4327            Release => intrinsics::atomic_umax::<T, { AO::Release }>(dst, val),
    4328            AcqRel => intrinsics::atomic_umax::<T, { AO::AcqRel }>(dst, val),
    4329            SeqCst => intrinsics::atomic_umax::<T, { AO::SeqCst }>(dst, val),
    4330        }
    4331    }
    4332}
    4333
    4334/// Updates `*dst` to the min value of `val` and the old value (unsigned comparison)
    4335#[inline]
    4336#[cfg(target_has_atomic)]
    4337#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4338unsafe fn atomic_umin<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
    4339    // SAFETY: the caller must uphold the safety contract for `atomic_umin`
    4340    unsafe {
    4341        match order {
    4342            Relaxed => intrinsics::atomic_umin::<T, { AO::Relaxed }>(dst, val),
    4343            Acquire => intrinsics::atomic_umin::<T, { AO::Acquire }>(dst, val),
    4344            Release => intrinsics::atomic_umin::<T, { AO::Release }>(dst, val),
    4345            AcqRel => intrinsics::atomic_umin::<T, { AO::AcqRel }>(dst, val),
    4346            SeqCst => intrinsics::atomic_umin::<T, { AO::SeqCst }>(dst, val),
    4347        }
    4348    }
    4349}
    4350
    4351/// An atomic fence.
    4352///
    4353/// Fences create synchronization between themselves and atomic operations or fences in other
    4354/// threads. To achieve this, a fence prevents the compiler and CPU from reordering certain types of
    4355/// memory operations around it.
    4356///
    4357/// There are 3 different ways to use an atomic fence:
    4358///
    4359/// - atomic - fence synchronization: an atomic operation with (at least) [`Release`] ordering
    4360///   semantics synchronizes with a fence with (at least) [`Acquire`] ordering semantics.
    4361/// - fence - atomic synchronization: a fence with (at least) [`Release`] ordering semantics
    4362///   synchronizes with an atomic operation with (at least) [`Acquire`] ordering semantics.
    4363/// - fence - fence synchronization: a fence with (at least) [`Release`] ordering semantics
    4364///   synchronizes with a fence with (at least) [`Acquire`] ordering semantics.
    4365///
    4366/// These 3 ways complement the regular, fence-less, atomic - atomic synchronization.
    4367///
    4368/// ## Atomic - Fence
    4369///
    4370/// An atomic operation on one thread will synchronize with a fence on another thread when:
    4371///
    4372/// -   on thread 1:
    4373///     -   an atomic operation 'X' with (at least) [`Release`] ordering semantics on some atomic
    4374///         object 'm',
    4375///
    4376/// -   is paired on thread 2 with:
    4377///     -   an atomic read 'Y' with any order on 'm',
    4378///     -   followed by a fence 'B' with (at least) [`Acquire`] ordering semantics.
    4379///
    4380/// This provides a happens-before dependence between X and B.
    4381///
    4382/// ```text
    4383///     Thread 1                                          Thread 2
    4384///
    4385/// m.store(3, Release); X ---------
    4386///                                |
    4387///                                |
    4388///                                -------------> Y  if m.load(Relaxed) == 3 {
    4389///                                               B      fence(Acquire);
    4390///                                                      ...
    4391///                                                  }
    4392/// ```
    4393///
    4394/// ## Fence - Atomic
    4395///
    4396/// A fence on one thread will synchronize with an atomic operation on another thread when:
    4397///
    4398/// -   on thread:
    4399///     -   a fence 'A' with (at least) [`Release`] ordering semantics,
    4400///     -   followed by an atomic write 'X' with any ordering on some atomic object 'm',
    4401///
    4402/// -   is paired on thread 2 with:
    4403///     -   an atomic operation 'Y' with (at least) [`Acquire`] ordering semantics.
    4404///
    4405/// This provides a happens-before dependence between A and Y.
    4406///
    4407/// ```text
    4408///     Thread 1                                          Thread 2
    4409///
    4410/// fence(Release);      A
    4411/// m.store(3, Relaxed); X ---------
    4412///                                |
    4413///                                |
    4414///                                -------------> Y  if m.load(Acquire) == 3 {
    4415///                                                      ...
    4416///                                                  }
    4417/// ```
    4418///
    4419/// ## Fence - Fence
    4420///
    4421/// A fence on one thread will synchronize with a fence on another thread when:
    4422///
    4423/// -   on thread 1:
    4424///     -   a fence 'A' which has (at least) [`Release`] ordering semantics,
    4425///     -   followed by an atomic write 'X' with any ordering on some atomic object 'm',
    4426///
    4427/// -   is paired on thread 2 with:
    4428///     -   an atomic read 'Y' with any ordering on 'm',
    4429///     -   followed by a fence 'B' with (at least) [`Acquire`] ordering semantics.
    4430///
    4431/// This provides a happens-before dependence between A and B.
    4432///
    4433/// ```text
    4434///     Thread 1                                          Thread 2
    4435///
    4436/// fence(Release);      A --------------
    4437/// m.store(3, Relaxed); X ---------    |
    4438///                                |    |
    4439///                                |    |
    4440///                                -------------> Y  if m.load(Relaxed) == 3 {
    4441///                                     |-------> B      fence(Acquire);
    4442///                                                      ...
    4443///                                                  }
    4444/// ```
    4445///
    4446/// ## Mandatory Atomic
    4447///
    4448/// Note that in the examples above, it is crucial that the access to `m` are atomic. Fences cannot
    4449/// be used to establish synchronization between non-atomic accesses in different threads. However,
    4450/// thanks to the happens-before relationship, any non-atomic access that happen-before the atomic
    4451/// operation or fence with (at least) [`Release`] ordering semantics are now also properly
    4452/// synchronized with any non-atomic accesses that happen-after the atomic operation or fence with
    4453/// (at least) [`Acquire`] ordering semantics.
    4454///
    4455/// ## Memory Ordering
    4456///
    4457/// A fence which has [`SeqCst`] ordering, in addition to having both [`Acquire`] and [`Release`]
    4458/// semantics, participates in the global program order of the other [`SeqCst`] operations and/or
    4459/// fences.
    4460///
    4461/// Accepts [`Acquire`], [`Release`], [`AcqRel`] and [`SeqCst`] orderings.
    4462///
    4463/// # Panics
    4464///
    4465/// Panics if `order` is [`Relaxed`].
    4466///
    4467/// # Examples
    4468///
    4469/// ```
    4470/// use std::sync::atomic::AtomicBool;
    4471/// use std::sync::atomic::fence;
    4472/// use std::sync::atomic::Ordering;
    4473///
    4474/// // A mutual exclusion primitive based on spinlock.
    4475/// pub struct Mutex {
    4476///     flag: AtomicBool,
    4477/// }
    4478///
    4479/// impl Mutex {
    4480///     pub fn new() -> Mutex {
    4481///         Mutex {
    4482///             flag: AtomicBool::new(false),
    4483///         }
    4484///     }
    4485///
    4486///     pub fn lock(&self) {
    4487///         // Wait until the old value is `false`.
    4488///         while self
    4489///             .flag
    4490///             .compare_exchange_weak(false, true, Ordering::Relaxed, Ordering::Relaxed)
    4491///             .is_err()
    4492///         {}
    4493///         // This fence synchronizes-with store in `unlock`.
    4494///         fence(Ordering::Acquire);
    4495///     }
    4496///
    4497///     pub fn unlock(&self) {
    4498///         self.flag.store(false, Ordering::Release);
    4499///     }
    4500/// }
    4501/// ```
    4502#[inline]
    4503#[stable(feature = "rust1", since = "1.0.0")]
    4504#[rustc_diagnostic_item = "fence"]
    4505#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4506pub fn fence(order: Ordering) {
    4507    // SAFETY: using an atomic fence is safe.
    4508    unsafe {
    4509        match order {
    4510            Acquire => intrinsics::atomic_fence::<{ AO::Acquire }>(),
    4511            Release => intrinsics::atomic_fence::<{ AO::Release }>(),
    4512            AcqRel => intrinsics::atomic_fence::<{ AO::AcqRel }>(),
    4513            SeqCst => intrinsics::atomic_fence::<{ AO::SeqCst }>(),
    4514            Relaxed => panic!("there is no such thing as a relaxed fence"),
    4515        }
    4516    }
    4517}
    4518
    4519/// A "compiler-only" atomic fence.
    4520///
    4521/// Like [`fence`], this function establishes synchronization with other atomic operations and
    4522/// fences. However, unlike [`fence`], `compiler_fence` only establishes synchronization with
    4523/// operations *in the same thread*. This may at first sound rather useless, since code within a
    4524/// thread is typically already totally ordered and does not need any further synchronization.
    4525/// However, there are cases where code can run on the same thread without being ordered:
    4526/// - The most common case is that of a *signal handler*: a signal handler runs in the same thread
    4527///   as the code it interrupted, but it is not ordered with respect to that code. `compiler_fence`
    4528///   can be used to establish synchronization between a thread and its signal handler, the same way
    4529///   that `fence` can be used to establish synchronization across threads.
    4530/// - Similar situations can arise in embedded programming with interrupt handlers, or in custom
    4531///   implementations of preemptive green threads. In general, `compiler_fence` can establish
    4532///   synchronization with code that is guaranteed to run on the same hardware CPU.
    4533///
    4534/// See [`fence`] for how a fence can be used to achieve synchronization. Note that just like
    4535/// [`fence`], synchronization still requires atomic operations to be used in both threads -- it is
    4536/// not possible to perform synchronization entirely with fences and non-atomic operations.
    4537///
    4538/// `compiler_fence` does not emit any machine code, but restricts the kinds of memory re-ordering
    4539/// the compiler is allowed to do. `compiler_fence` corresponds to [`atomic_signal_fence`] in C and
    4540/// C++.
    4541///
    4542/// [`atomic_signal_fence`]: https://en.cppreference.com/w/cpp/atomic/atomic_signal_fence
    4543///
    4544/// # Panics
    4545///
    4546/// Panics if `order` is [`Relaxed`].
    4547///
    4548/// # Examples
    4549///
    4550/// Without the two `compiler_fence` calls, the read of `IMPORTANT_VARIABLE` in `signal_handler`
    4551/// is *undefined behavior* due to a data race, despite everything happening in a single thread.
    4552/// This is because the signal handler is considered to run concurrently with its associated
    4553/// thread, and explicit synchronization is required to pass data between a thread and its
    4554/// signal handler. The code below uses two `compiler_fence` calls to establish the usual
    4555/// release-acquire synchronization pattern (see [`fence`] for an image).
    4556///
    4557/// ```
    4558/// use std::sync::atomic::AtomicBool;
    4559/// use std::sync::atomic::Ordering;
    4560/// use std::sync::atomic::compiler_fence;
    4561///
    4562/// static mut IMPORTANT_VARIABLE: usize = 0;
    4563/// static IS_READY: AtomicBool = AtomicBool::new(false);
    4564///
    4565/// fn main() {
    4566///     unsafe { IMPORTANT_VARIABLE = 42 };
    4567///     // Marks earlier writes as being released with future relaxed stores.
    4568///     compiler_fence(Ordering::Release);
    4569///     IS_READY.store(true, Ordering::Relaxed);
    4570/// }
    4571///
    4572/// fn signal_handler() {
    4573///     if IS_READY.load(Ordering::Relaxed) {
    4574///         // Acquires writes that were released with relaxed stores that we read from.
    4575///         compiler_fence(Ordering::Acquire);
    4576///         assert_eq!(unsafe { IMPORTANT_VARIABLE }, 42);
    4577///     }
    4578/// }
    4579/// ```
    4580#[inline]
    4581#[stable(feature = "compiler_fences", since = "1.21.0")]
    4582#[rustc_diagnostic_item = "compiler_fence"]
    4583#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
    4584pub fn compiler_fence(order: Ordering) {
    4585    // SAFETY: using an atomic fence is safe.
    4586    unsafe {
    4587        match order {
    4588            Acquire => intrinsics::atomic_singlethreadfence::<{ AO::Acquire }>(),
    4589            Release => intrinsics::atomic_singlethreadfence::<{ AO::Release }>(),
    4590            AcqRel => intrinsics::atomic_singlethreadfence::<{ AO::AcqRel }>(),
    4591            SeqCst => intrinsics::atomic_singlethreadfence::<{ AO::SeqCst }>(),
    4592            Relaxed => panic!("there is no such thing as a relaxed fence"),
    4593        }
    4594    }
    4595}
    4596
    4597#[cfg(target_has_atomic_load_store = "8")]
    4598#[stable(feature = "atomic_debug", since = "1.3.0")]
    4599impl fmt::Debug for AtomicBool {
    4600    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
    4601        fmt::Debug::fmt(&self.load(Ordering::Relaxed), f)
    4602    }
    4603}
    4604
    4605#[cfg(target_has_atomic_load_store = "ptr")]
    4606#[stable(feature = "atomic_debug", since = "1.3.0")]
    4607impl<T> fmt::Debug for AtomicPtr<T> {
    4608    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
    4609        fmt::Debug::fmt(&self.load(Ordering::Relaxed), f)
    4610    }
    4611}
    4612
    4613#[cfg(target_has_atomic_load_store = "ptr")]
    4614#[stable(feature = "atomic_pointer", since = "1.24.0")]
    4615impl<T> fmt::Pointer for AtomicPtr<T> {
    4616    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
    4617        fmt::Pointer::fmt(&self.load(Ordering::Relaxed), f)
    4618    }
    4619}
    4620
    4621/// Signals the processor that it is inside a busy-wait spin-loop ("spin lock").
    4622///
    4623/// This function is deprecated in favor of [`hint::spin_loop`].
    4624///
    4625/// [`hint::spin_loop`]: crate::hint::spin_loop
    4626#[inline]
    4627#[stable(feature = "spin_loop_hint", since = "1.24.0")]
    4628#[deprecated(since = "1.51.0", note = "use hint::spin_loop instead")]
    4629pub fn spin_loop_hint() {
    4630    spin_loop()
    4631}