No public description PiperOrigin-RevId: 1015716258 Change-Id: I7c9f85878de373109302018fed4a9a2bee835f9c
diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a6ef15..a9a572b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md
@@ -3,14 +3,20 @@ ## [Unreleased] New features: +- `AutoDiffScalar` overloads of `atan`, `log1p`, `expm1`, `log2`, `log10`, `cbrt`, `asinh`, `acosh` and `atanh` [#234, !3448] - ComplexQZ implementation [!1962] - Generic clang vector extension backend [!2051] - `ArrayBase::logicalShiftRight<N>()`, a zero-filling right shift, alongside `arithmeticShiftRight<N>()` and `logicalShiftLeft<N>()` as the spelled-out names for the existing shift operators [!2816] - Support building Eigen's in-tree BLAS library with 64-bit (`ILP64`) integers (`eigen_blas_ilp64` and `eigen_blas_ilp64_static`, controlled by `EIGEN_BUILD_BLAS_ILP64`) - `PreconditionSquareMatrix` option, which applies JacobiSVD's QR preconditioners to square matrices. BDCSVD forwards this option to JacobiSVD for matrices smaller than the switch size [!3212] - `DenseBase::replicate` and `VectorwiseOp::replicate` accept `Eigen::fix<N>` factors [!3262, !3282] +- `SelfAdjointView::applyPermutation()` and `SelfAdjointView::applyTransposition()`, which replace the viewed matrix `S` by `P S P^T` in place, reading and writing only the stored triangle [#3174, !3395] +- `DiscreteAlgebraicRiccatiSolver` in the new contrib module `Riccati`, for the stabilizing solution of the (generalized) discrete-time algebraic Riccati equation with real or complex scalars [!3403] +- The new contrib module `GPU` (`#define EIGEN_USE_GPU`, then `#include <contrib/Eigen/GPU>`), which runs dense and sparse linear algebra on NVIDIA GPUs from host code through cuBLAS, cuSOLVER, cuSPARSE, cuFFT and cuDSS, and runs `ConjugateGradient` on device-resident vectors. Requires the CUDA Toolkit; see the manual page "Offloading linear algebra to a GPU with the GPU module" [!2412, !2413, !2414, !2415, !2476, !2681, !3130, !3135] +- A matrix-matrix product whose operand is a cast of a real matrix with direct access, such as `A.cast<float>() * B.cast<float>()` or `A * B.cast<float>()`, converts that operand while packing it instead of materializing the cast, unless `EIGEN_USE_BLAS` is defined or SME is enabled [!3454] Bug fixes: +- A matrix-matrix product of two `half` or two `bfloat16` matrices on the general matrix-matrix kernel rounded every multiply-add, or every depth block, to the operand type; it now sums in `float` over the whole depth and rounds each result coefficient once, which makes `half` products about 2x slower on hardware with native `half` arithmetic [#3158, !3454] - Row-major triangular matrix-vector products could return incorrect results or double-destroy temporaries for scalars requiring initialization, and could leak some large right-hand-side buffers [!2818] - `MatrixBase::isLowerTriangular()` skipped the last strictly-upper coefficient of every column past the diagonal block, reporting some wide matrices as lower triangular [!2814] - `ArrayBase::shiftRight<N>()` and `ArrayBase::shiftLeft<N>()` did not compile in any configuration, their functors having dropped the shift count [!2814] @@ -20,12 +26,21 @@ - `BDCSVD` with QR-only `Options`, such as `DisableQRDecomposition`, dropped runtime unitary requests for inputs below the switch size, so `matrixU()` asserted, or returned an empty `U` and `V` under `NDEBUG` [!3212] - `JacobiSVD` with `FullPivHouseholderQRPreconditioner` accepted `ComputeThinV` and returned a non-orthogonal `V` for wide inputs; it now fails to compile, or asserts when requested at runtime [!3212] - `HouseholderQR` (and the decompositions and `SplineFitting` built on it) and `JacobiSVD` did not compile with `AutoDiffScalar` [#3192, !3265] +- Constructing or assigning an `Affine` or `Isometry` `Transform` from a (Dim+1)x(Dim+1) matrix copied its last row verbatim; the last row is now set to `[0 ... 0 1]`, as for Dim x Dim and Dim x (Dim+1) inputs [#2715, !3412] +- `LDLT` pivoted on the diagonal of its input rather than of the remaining Schur complement, so zero pivots of a semidefinite matrix could come before non-zero ones and report `NumericalIssue`, as for `[1 1 0; 1 1 0; 0 0 1]`; a NaN or infinite pivot reported `Success`. `LDLT` now also factors dynamic-size matrices larger than 96 in blocks, with a panel width that grows as sqrt(3 n): 1.7x faster at n = 128, 3.1x at n = 1024 and 6.2x at n = 3000 [!3423] +- `IOFormat` could insert unwanted spaces between matrix rows when using custom row separators or prefixes. Breaking changes: +- `PaStiXSupport` now requires PaStiX 6.x; version 5 is no longer supported. `FindPASTIX.cmake` uses pkg-config metadata + instead of dependency-selection components. Removed the unused `FindBLASEXT`, `FindHWLOC`, `FindSCOTCH` and + `FindPTSCOTCH` modules. Renamed `FindMetis.cmake` to `FindMETIS.cmake` to match `find_package(METIS)` [#2256, !3467] +- `LDLT` picks a different pivot order for most inputs, which changes `transpositionsP()` and the order of `vectorD()`. A NaN or infinite pivot reports `NumericalIssue`. A preallocated `compute()` of a dynamic-size matrix larger than 96 can allocate the packing buffers of its matrix products, as those of `LLT`, `PartialPivLU` and `BunchKaufman` already do [!3423] - The default `epsfcn` for central mode differences was changed to `cbrt(NumTraits<Scalar>::epsilon())` (used to be `sqrt(NumTraits<Scalar>::epsilon())`). The new default should reduce numerical error in most cases. +- Removed `Eigen/QtAlignedMalloc`, which replaced Qt 4's `qMalloc`, `qFree` and `qRealloc` with aligned versions on platforms whose `malloc` is not already sufficiently aligned. Remove the include. [!3413] Deprecations: - `ArrayBase::shiftRight<N>()` and `ArrayBase::shiftLeft<N>()`, in favour of `arithmeticShiftRight<N>()` and `logicalShiftLeft<N>()` [!2816] +- The `Transform` conversions from and to Qt's `QMatrix` and `QTransform`, which `EIGEN_QT_SUPPORT` enables, in favor of copying the coefficients explicitly [!3413] Other changes: - The top-level `unsupported/` directory was renamed to `contrib/`. Existing @@ -33,6 +48,7 @@ unchanged via forwarding header shims. The CMake subproject label and build-group target were renamed accordingly: `Unsupported` → `Contrib`, `BuildUnsupported` → `BuildContrib`. +- Removed the `demos/` directory, including its two Qt GUI demos, and the `EIGEN_BUILD_DEMOS` and `EIGEN_TEST_NOQT` CMake options. The sparse tutorial example no longer needs Qt; it writes a PGM image. [!3413] ## [5.0.1] - 2025-11-11
diff --git a/CMakeLists.txt b/CMakeLists.txt index 69ba5e7..8635e0a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt
@@ -139,8 +139,6 @@ endif() option(EIGEN_BUILD_DOC "Enable creation of Eigen documentation" ${EIGEN_BUILD_DOC_DEFAULT}) -option(EIGEN_BUILD_DEMOS "Toggles the building of the Eigen demos" ${PROJECT_IS_TOP_LEVEL}) - # Disable pkgconfig only for native Windows builds if(NOT WIN32 OR NOT CMAKE_HOST_SYSTEM_NAME MATCHES Windows) option(EIGEN_BUILD_PKGCONFIG "Build pkg-config .pc file for Eigen" ${PROJECT_IS_TOP_LEVEL}) @@ -151,7 +149,7 @@ # headers, eigen3.pc, the CMake package files, and the BLAS/LAPACK libraries. option(EIGEN_INSTALL "Enable Eigen's install rules" ON) -if (EIGEN_BUILD_TESTING OR EIGEN_BUILD_BLAS OR EIGEN_BUILD_LAPACK OR EIGEN_BUILD_DOC OR EIGEN_BUILD_DEMOS) +if (EIGEN_BUILD_TESTING OR EIGEN_BUILD_BLAS OR EIGEN_BUILD_LAPACK OR EIGEN_BUILD_DOC) set(EIGEN_IS_BUILDING_ ON) endif() @@ -826,10 +824,6 @@ add_subdirectory(doc EXCLUDE_FROM_ALL) endif() -if (EIGEN_BUILD_DEMOS) - add_subdirectory(demos EXCLUDE_FROM_ALL) -endif() - if (PROJECT_IS_TOP_LEVEL) # must be after test and contrib, for configuring buildtests.in add_subdirectory(scripts EXCLUDE_FROM_ALL)
diff --git a/COPYING.README b/COPYING.README index 93ec692..504052b 100644 --- a/COPYING.README +++ b/COPYING.README
@@ -5,8 +5,16 @@ Some files contain third-party code under permissive or otherwise MPL2-compatible licenses, hence the other COPYING.* files here. These -include Apache-2.0, BSD-style notices, the MINPACK license, and the MORSE -CMake module BSD-style notice in cmake/MORSE-Copyright.txt. +include Apache-2.0, BSD-style notices, the MINPACK license, the MORSE +CMake module BSD-style notice in cmake/MORSE-Copyright.txt, and the LAPACK +license (BSD-3-Clause-Open-MPI) of the netlib BLAS test programs in +blas/testing/ and the LAPACK timer stubs and test script in lapack/. + +The Riccati module (contrib/Eigen/Riccati) incorporates code adapted from +SciPy (BSD-3-Clause) and LAPACK (BSD-3-Clause-Open-MPI). Binary +redistributions of software that uses it must reproduce the copyright +notices, conditions and disclaimers in +contrib/Eigen/src/Riccati/DiscreteAlgebraicRiccatiSolver.h. Note that some optional external dependencies (e.g. FFTW, MPFR C++) are distributed under different licenses, including the GPL. Refer to
diff --git a/Eigen/Core b/Eigen/Core index 2330b76..bfd7c82 100644 --- a/Eigen/Core +++ b/Eigen/Core
@@ -192,8 +192,8 @@ #include "src/Core/util/Assert.h" #include "src/Core/util/ForwardDeclarations.h" #include "src/Core/util/StaticAssert.h" -#include "src/Core/util/XprHelper.h" #include "src/Core/util/ShapeChecks.h" +#include "src/Core/util/XprHelper.h" #include "src/Core/util/Memory.h" #include "src/Core/util/IntegralConstant.h" #include "src/Core/util/Serializer.h" @@ -275,12 +275,14 @@ #include "src/Core/arch/NEON/PacketMath.h" #include "src/Core/arch/NEON/TypeCasting.h" #include "src/Core/arch/NEON/MathFunctions.h" +#include "src/Core/arch/NEON/PacketMathBF16.h" #include "src/Core/arch/NEON/Complex.h" #elif defined EIGEN_VECTORIZE_SME // Keep NEON packet types outside the specialized streaming kernels. #include "src/Core/arch/NEON/PacketMath.h" #include "src/Core/arch/NEON/TypeCasting.h" #include "src/Core/arch/NEON/MathFunctions.h" +#include "src/Core/arch/NEON/PacketMathBF16.h" #include "src/Core/arch/NEON/Complex.h" #include "src/Core/arch/SME/PacketMath.h" #elif defined EIGEN_VECTORIZE_LSX
diff --git a/Eigen/PaStiXSupport b/Eigen/PaStiXSupport index 0f3aa66..76b41c3 100644 --- a/Eigen/PaStiXSupport +++ b/Eigen/PaStiXSupport
@@ -15,10 +15,13 @@ #include "src/Core/util/DisableStupidWarnings.h" extern "C" { -#include <pastix_nompi.h> #include <pastix.h> } +#if !defined(PASTIX_VERSION_MAJOR) || PASTIX_VERSION_MAJOR != 6 +#error Eigen PaStiXSupport requires PaStiX 6.x +#endif + #ifdef complex #undef complex #endif @@ -26,11 +29,11 @@ /** \ingroup Support_modules * \defgroup PaStiXSupport_Module PaStiXSupport module * - * This module provides an interface to the <a href="http://pastix.gforge.inria.fr/">PaSTiX</a> library. + * This module provides an interface to the <a href="https://solverstack.gitlabpages.inria.fr/pastix/">PaSTiX</a> library. * PaSTiX is a general \b supernodal, \b parallel and \b opensource sparse solver. - * It provides the two following main factorization classes: - * - class PastixLLT : a supernodal, parallel LLt Cholesky factorization. - * - class PastixLDLT: a supernodal, parallel LDLt Cholesky factorization. + * It provides the following factorization classes: + * - class PastixLLT : a supernodal, parallel LLT (real) or LL* (complex) Cholesky factorization. + * - class PastixLDLT: a supernodal, parallel LDLT (real) or LDL* (complex) factorization. * - class PastixLU : a supernodal, parallel LU factorization (optimized for a symmetric pattern). * * \code @@ -38,9 +41,9 @@ * \endcode * * In order to use this module, the PaSTiX headers must be accessible from the include paths, and your binary must be - * linked to the PaSTiX library and its dependencies. This wrapper requires PaStiX version 5.x compiled without MPI - * support. The dependencies depend on how PaSTiX has been compiled. For a cmake based project, you can use our - * FindPaSTiX.cmake module to help you in this task. + * linked to the PaSTiX library and its dependencies. This wrapper supports PaStiX 6.x compiled + * without MPI support. The dependencies depend on how PaSTiX has been compiled. For a cmake based project, you can use our + * FindPASTIX.cmake module to help you in this task. * */
diff --git a/Eigen/QtAlignedMalloc b/Eigen/QtAlignedMalloc deleted file mode 100644 index cb34bfa..0000000 --- a/Eigen/QtAlignedMalloc +++ /dev/null
@@ -1,34 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-FileCopyrightText: The Eigen Authors -// SPDX-License-Identifier: MPL-2.0 - -#ifndef EIGEN_QTMALLOC_MODULE_H -#define EIGEN_QTMALLOC_MODULE_H - -#include "Core" - -#if (!EIGEN_MALLOC_ALREADY_ALIGNED) - -#include "src/Core/util/DisableStupidWarnings.h" - -inline void *qMalloc(std::size_t size) { return Eigen::internal::aligned_malloc(size); } - -inline void qFree(void *ptr) { Eigen::internal::aligned_free(ptr); } - -inline void *qRealloc(void *ptr, std::size_t size) { - void *newPtr = Eigen::internal::aligned_malloc(size); - std::memcpy(newPtr, ptr, size); - Eigen::internal::aligned_free(ptr); - return newPtr; -} - -#include "src/Core/util/ReenableStupidWarnings.h" - -#endif - -#endif // EIGEN_QTMALLOC_MODULE_H
diff --git a/Eigen/src/Cholesky/BunchKaufman.h b/Eigen/src/Cholesky/BunchKaufman.h index d35554a..ecfaa5d 100644 --- a/Eigen/src/Cholesky/BunchKaufman.h +++ b/Eigen/src/Cholesky/BunchKaufman.h
@@ -394,8 +394,7 @@ * (Bunch & Kaufman, 1977). */ template <typename RealScalar> EIGEN_DEVICE_FUNC inline RealScalar bunch_kaufman_alpha() { - using std::sqrt; - return (RealScalar(1) + sqrt(RealScalar(17))) / RealScalar(8); + return (RealScalar(1) + numext::sqrt(RealScalar(17))) / RealScalar(8); } /** \internal Panel width for the blocked (level-3) Bunch-Kaufman factorization. Can be overridden (mainly @@ -414,31 +413,6 @@ template <> struct bunch_kaufman<Lower> { - // Interchange rows and columns kk and kp (kp >= kk) in the lower triangle of the Hermitian matrix - // `mat`, including the already-computed factor columns to the left of column `kfirst` (so that the - // stored unit triangular factor stays consistent with a single up-front permutation P). `kfirst` is - // the first column of the pivot block (== kk for a 1x1 pivot, == kk-1 for a 2x2 pivot). - template <typename MatrixType> - static void apply_symmetric_pivot(MatrixType& mat, Index kfirst, Index kk, Index kp, Index kstep) { - using Scalar = typename MatrixType::Scalar; - const Index n = mat.rows(); - const Index s = n - kp - 1; - if (s > 0) mat.col(kk).tail(s).swap(mat.col(kp).tail(s)); - for (Index i = kk + 1; i < kp; ++i) { - Scalar tmp = mat.coeff(i, kk); - mat.coeffRef(i, kk) = numext::conj(mat.coeff(kp, i)); - mat.coeffRef(kp, i) = numext::conj(tmp); - } - numext::swap(mat.coeffRef(kk, kk), mat.coeffRef(kp, kp)); - EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::IsComplex) { - mat.coeffRef(kp, kk) = numext::conj(mat.coeff(kp, kk)); - } - if (kfirst > 0) mat.row(kk).head(kfirst).swap(mat.row(kp).head(kfirst)); - if (kstep == 2) { - numext::swap(mat.coeffRef(kfirst + 1, kfirst), mat.coeffRef(kp, kfirst)); - } - } - // Unblocked (level-2 BLAS) Bunch-Kaufman factorization of the lower triangle of `mat`, in place. // Columns [k0, n) are factorized. On output: // - the strictly-lower triangle holds the unit lower factor L (with explicit zeros at the @@ -449,7 +423,6 @@ // Returns 0 on success, or the (1-based) index of the first exactly-zero pivot encountered. template <typename MatrixType, typename TranspositionType, typename SubDiagType> static Index unblocked(MatrixType& mat, TranspositionType& transpositions, SubDiagType& subdiag, Index k0 = 0) { - using numext::abs; using Scalar = typename MatrixType::Scalar; using RealScalar = typename MatrixType::RealScalar; using StorageIndex = typename TranspositionType::StorageIndex; @@ -461,7 +434,7 @@ while (k < n) { Index kstep = 1; Index kp = k; - const RealScalar absakk = abs(numext::real(mat.coeff(k, k))); + const RealScalar absakk = numext::abs(numext::real(mat.coeff(k, k))); // colmax = max_{i>k} |mat(i,k)|, attained at row imax. Index imax = k; @@ -490,7 +463,7 @@ if (absakk >= alpha * colmax * (colmax / rowmax)) { // 1x1 pivot at k. kp = k; - } else if (abs(numext::real(mat.coeff(imax, imax))) >= alpha * rowmax) { + } else if (numext::abs(numext::real(mat.coeff(imax, imax))) >= alpha * rowmax) { // 1x1 pivot at imax: interchange rows/columns k and imax. kp = imax; } else { @@ -501,7 +474,7 @@ } const Index kk = k + kstep - 1; // column of the pivot block to interchange with kp - if (kp != kk) apply_symmetric_pivot(mat, k, kk, kp, kstep); + if (kp != kk) selfadjoint_swap_lower(mat, kk, kp); if (kstep == 1) { transpositions.coeffRef(k) = StorageIndex(kp); @@ -601,7 +574,6 @@ template <typename MatrixType, typename WorkspaceType, typename TranspositionType, typename SubDiagType> static Index partial_factor(MatrixType& mat, Index k0, Index nb, WorkspaceType& W, TranspositionType& transpositions, SubDiagType& subdiag, Index& info) { - using numext::abs; using Scalar = typename MatrixType::Scalar; using RealScalar = typename MatrixType::RealScalar; using StorageIndex = typename TranspositionType::StorageIndex; @@ -625,7 +597,7 @@ Index kstep = 1; Index kp = jc; - const RealScalar absakk = abs(numext::real(W.coeff(jc, j))); + const RealScalar absakk = numext::abs(numext::real(W.coeff(jc, j))); Index imax = jc; RealScalar colmax(0); if (jc + 1 < n) { @@ -659,7 +631,7 @@ } if (absakk >= alpha * colmax * (colmax / rowmax)) { kp = jc; - } else if (abs(numext::real(W.coeff(imax, j + 1))) >= alpha * rowmax) { + } else if (numext::abs(numext::real(W.coeff(imax, j + 1))) >= alpha * rowmax) { kp = imax; // imax becomes the 1x1 pivot column: its updated column (in W(:,j+1)) replaces W(:,j). W.col(j).segment(jc, n - jc) = W.col(j + 1).segment(jc, n - jc); @@ -674,7 +646,7 @@ const Index kk = jc + kstep - 1; if (kp != kk) { - apply_symmetric_pivot(mat, jc, kk, kp, kstep); + selfadjoint_swap_lower(mat, kk, kp); const Index nwc = j + kstep; // number of populated W columns W.row(kk).head(nwc).swap(W.row(kp).head(nwc)); } @@ -832,7 +804,6 @@ template <typename MatrixType, int UpLo_> template <bool Conjugate, typename Derived> void BunchKaufman<MatrixType, UpLo_>::solveInPlaceD(MatrixBase<Derived>& x) const { - using numext::abs; const Index n = m_matrix.rows(); // Use the pseudo-inverse of singular 1x1 blocks (see Eigen bug 241 for LDLT). The 2x2 blocks // produced by Bunch-Kaufman pivoting are non-singular by construction. @@ -861,7 +832,7 @@ k += 2; } else { const RealScalar dk = numext::real(m_matrix.coeff(k, k)); - if (abs(dk) > tol) + if (numext::abs(dk) > tol) x.row(k) /= dk; else x.row(k).setZero(); @@ -1086,15 +1057,13 @@ template <typename MatrixType, int UpLo_> MatrixType BunchKaufman<MatrixType, UpLo_>::reconstructedMatrix() const { eigen_assert(m_isInitialized && "BunchKaufman is not initialized."); - const Index size = m_matrix.rows(); - MatrixType res(size, size); - - res.setIdentity(); - res = transpositionsP() * res; // P - res = matrixU() * res; // U P = L^* P - applyD(res); // D L^* P - res = matrixL() * res; // L D L^* P - res = transpositionsP().transpose() * res; // P^T L D L^* P + MatrixType res = matrixU(); + applyD(res); + res = matrixL() * res; + // P^T (L D L^*) P; the stored pivots cannot alias the new result. + auto lower_triangle = res.template selfadjointView<Lower>(); + for (Index k = m_matrix.rows() - 1; k >= 0; --k) lower_triangle.applyTransposition(k, m_transpositions.coeff(k)); + res.template triangularView<StrictlyUpper>() = res.adjoint(); return res; }
diff --git a/Eigen/src/Cholesky/LDLT.h b/Eigen/src/Cholesky/LDLT.h index deca15e..5342b9f 100644 --- a/Eigen/src/Cholesky/LDLT.h +++ b/Eigen/src/Cholesky/LDLT.h
@@ -33,6 +33,10 @@ // PositiveSemiDef means positive semi-definite and non-zero; same for NegativeSemiDef enum SignMatrix { PositiveSemiDef, NegativeSemiDef, ZeroSign, Indefinite }; + +// Panel width of the blocked factorization (defined below); declared here for the size constructor. +template <typename MatrixType> +inline Index ldlt_blocksize(Index size); } // namespace internal /** \ingroup Cholesky_Module @@ -49,17 +53,21 @@ * matrix \f$ A \f$ such that \f$ A = P^TLDL^*P \f$, where P is a permutation matrix, L * is lower triangular with a unit diagonal and D is a diagonal matrix. * - * The decomposition uses pivoting to ensure stability, so that D will have - * zeros in the bottom right rank(A) - n submatrix. Avoiding the square root - * on D also stabilizes the computation. + * Each step pivots on the diagonal entry of largest magnitude of the remaining Schur complement + * (complete diagonal pivoting, as in LAPACK's xPSTRF), so that for a semidefinite matrix of rank r + * the n - r entries of D that vanish in exact arithmetic come last, at the level of rounding errors + * and of either sign. Avoiding the square root on D also stabilizes the computation. * - * Remember that Cholesky decompositions are not rank-revealing. Also, do not use a Cholesky - * decomposition to determine whether a system of equations has a solution. + * LDLT does not determine a numerical rank, though. Also, do not use a Cholesky decomposition to + * determine whether a system of equations has a solution. * * This class supports the \link InplaceDecomposition inplace decomposition \endlink mechanism. * - * D is purely diagonal, so this class cannot factor an indefinite matrix. For a self-adjoint matrix - * that is indefinite, use BunchKaufman, which produces a block-diagonal D. + * D is purely diagonal, so this class is not backward stable on indefinite matrices: it can report + * \c Success for a factorization with a large backward error. If isPositive() and isNegative() both + * return false, D has pivots of both signs: unless that comes from the rounding-level pivots of a + * singular semidefinite matrix, the matrix is indefinite, and BunchKaufman, which produces a + * block-diagonal D, should be used instead. * * \sa MatrixBase::ldlt(), SelfAdjointView::ldlt(), class LLT, class BunchKaufman */ @@ -100,6 +108,10 @@ * * Like the default constructor but with preallocation of the internal data * according to the specified problem \a size. + * + * Above 96 rows, compute() factors a dynamic-size matrix in blocks, and the matrix products of the blocked + * factorization can allocate their packing buffers, as those of LLT and PartialPivLU do. Fixed-size and bounded + * matrices are factored unblocked and allocate nothing. * \sa LDLT() */ explicit LDLT(Index size) @@ -107,6 +119,8 @@ m_l1_norm(0), m_transpositions(size), m_temporary(size), + // The shape compute() gives the panel workspace, so that it does not reallocate it for this size. + m_workspace(size, internal::ldlt_blocksize<MatrixType>(size)), m_sign(internal::ZeroSign), m_isInitialized(false), m_info(InvalidInput) {} @@ -311,7 +325,9 @@ /** \brief Reports whether previous computation was successful. * * \returns \c Success if computation was successful, - * \c NumericalIssue if the factorization failed because of a zero pivot. + * \c NumericalIssue if the factorization failed: a pivot is not finite, or is zero above a + * non-zero column. In exact arithmetic only an indefinite matrix has the latter, but rounding + * errors can give it to a singular semidefinite matrix too. */ ComputationInfo info() const { eigen_assert(m_isInitialized && "LDLT is not initialized."); @@ -339,6 +355,8 @@ RealScalar m_l1_norm; TranspositionType m_transpositions; TmpMatrixType m_temporary; + // L21 D for the trailing update of the blocked factorization; it stays empty below the blocking threshold. + Matrix<Scalar, Dynamic, Dynamic> m_workspace; internal::SignMatrix m_sign; bool m_isInitialized; ComputationInfo m_info; @@ -346,21 +364,45 @@ namespace internal { +/** \internal Panel width of the blocked factorization of a \a size x \a size matrix, or 0 where a single + * unblocked panel covers the whole matrix: up to three panel widths, below which the level-3 update does not pay + * off, and for fixed-size or bounded matrices, which must not allocate the panel workspace. + * + * The panel's level-2 updates run at a rate R, and the trailing update at P / (1 + kappa / nb), kappa being the + * fixed cost of a pass in steps of its depth. Then T(nb) = n^3 / (3 P) (1 + kappa / nb) + n^2 nb / (2 R) is least at + * nb = sqrt(c n), c = 2 kappa R / (3 P). Measured c ranges from about 0.7 to 4 with AVX2, AVX-512, NEON and SME + * products, and grows once the trailing matrix outgrows the last-level cache. c = 3, with nb rounded to a multiple + * of 16 and clamped to [32, 128], the range the measured optimum spans, loses about 2% to the best width on average. + * Can be overridden (mainly for testing the panel logic at small sizes) by defining EIGEN_LDLT_BLOCKSIZE. */ +template <typename MatrixType> +inline Index ldlt_blocksize(Index size) { + if (int(MatrixType::MaxRowsAtCompileTime) != Dynamic || int(MatrixType::MaxColsAtCompileTime) != Dynamic) return 0; +#ifdef EIGEN_LDLT_BLOCKSIZE + const Index nb = Index(EIGEN_LDLT_BLOCKSIZE); +#else + constexpr Index kMinBlockSize = 32; + // Three of the narrowest panels are not blocked either, so small matrices skip the square root. + if (size <= 3 * kMinBlockSize) return 0; + const Index unclamped = 16 * Index(numext::round(numext::sqrt(3.0 * double(size)) / 16)); + const Index nb = numext::mini(numext::maxi(unclamped, kMinBlockSize), Index(128)); +#endif + return nb > 0 && size > 3 * nb ? nb : 0; +} + template <int UpLo> struct ldlt_inplace; template <> struct ldlt_inplace<Lower> { + // Settles the cases that leave nothing to factor, and returns whether it did: an empty or 1x1 matrix, and a + // diagonal that is entirely zero, which offers no pivot and leaves the matrix either zero or indefinite. Otherwise + // it copies the diagonal into temp, where factor_panel() expects it. template <typename MatrixType, typename TranspositionType, typename Workspace> - static bool unblocked(MatrixType& mat, TranspositionType& transpositions, Workspace& temp, SignMatrix& sign) { - using std::abs; + static bool factor_trivial(const MatrixType& mat, TranspositionType& transpositions, Workspace& temp, + SignMatrix& sign, bool& ret) { using Scalar = typename MatrixType::Scalar; using RealScalar = typename MatrixType::RealScalar; - using IndexType = typename TranspositionType::StorageIndex; - EIGEN_INTERNAL_ASSERT_SQUARE(mat); const Index size = mat.rows(); - bool found_zero_pivot = false; - bool ret = true; if (size <= 1) { transpositions.setIdentity(); @@ -372,74 +414,81 @@ sign = NegativeSemiDef; else sign = ZeroSign; + ret = size == 0 || (numext::isfinite)(numext::real(mat.coeff(0, 0))); return true; } - for (Index k = 0; k < size; ++k) { + temp = mat.diagonal().real().template cast<Scalar>(); + if ((temp.array() == Scalar(0)).all()) { + transpositions.setIdentity(); + sign = ZeroSign; + ret = true; + for (Index j = 0; ret && j < size; ++j) ret = (mat.col(j).tail(size - j - 1).array() == Scalar(0)).all(); + return true; + } + return false; + } + + // Factors columns [k0, k0 + kb) left-looking, column k pivoting on max |diag(S_k)|, S_k = A22 - L21 D L21^* + // (complete diagonal pivoting, as in LAPACK's xPSTF2). Invariant: mat(k0:n, k0:n) holds S_k0 off its diagonal and + // temp(k:n) holds diag(S_k). The caller interchanges the rows of the columns before k0. + template <typename MatrixType, typename TranspositionType, typename Workspace> + static bool factor_panel(MatrixType& mat, Index k0, Index kb, TranspositionType& transpositions, Workspace& temp, + SignMatrix& sign) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename MatrixType::RealScalar; + using IndexType = typename TranspositionType::StorageIndex; + const Index size = mat.rows(); + bool ret = true; + + for (Index k = k0; k < k0 + kb; ++k) { // Find largest diagonal element Index index_of_biggest_in_corner; - mat.diagonal().tail(size - k).cwiseAbs().maxCoeff(&index_of_biggest_in_corner); + temp.tail(size - k).real().cwiseAbs().maxCoeff(&index_of_biggest_in_corner); index_of_biggest_in_corner += k; transpositions.coeffRef(k) = IndexType(index_of_biggest_in_corner); if (k != index_of_biggest_in_corner) { - // apply the transposition while taking care to consider only - // the lower triangular part - Index s = size - index_of_biggest_in_corner - 1; // trailing size after the biggest element - mat.row(k).head(k).swap(mat.row(index_of_biggest_in_corner).head(k)); - mat.col(k).tail(s).swap(mat.col(index_of_biggest_in_corner).tail(s)); - std::swap(mat.coeffRef(k, k), mat.coeffRef(index_of_biggest_in_corner, index_of_biggest_in_corner)); - for (Index i = k + 1; i < index_of_biggest_in_corner; ++i) { - Scalar tmp = mat.coeffRef(i, k); - mat.coeffRef(i, k) = numext::conj(mat.coeffRef(index_of_biggest_in_corner, i)); - mat.coeffRef(index_of_biggest_in_corner, i) = numext::conj(tmp); - } - EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::IsComplex) - mat.coeffRef(index_of_biggest_in_corner, k) = numext::conj(mat.coeff(index_of_biggest_in_corner, k)); + auto trailing = mat.bottomRightCorner(size - k0, size - k0); + selfadjoint_swap_lower(trailing, k - k0, index_of_biggest_in_corner - k0); + numext::swap(temp.coeffRef(k), temp.coeffRef(index_of_biggest_in_corner)); } // partition the matrix: // A00 | - | - // lu = A10 | A11 | - // A20 | A21 | A22 + // where A10 and A20 span the panel columns [k0, k); those before k0 are already applied to A21. The + // product's operand D A10^* takes temp(k0:k), whose diagonal entries of S have all been used as pivots. Index rs = size - k - 1; - Block<MatrixType, Dynamic, 1> A21(mat, k + 1, k, rs, 1); - Block<MatrixType, 1, Dynamic> A10(mat, k, 0, 1, k); - Block<MatrixType, Dynamic, Dynamic> A20(mat, k + 1, 0, rs, k); + Index done = k - k0; + auto A21 = mat.block(k + 1, k, rs, fix<1>); - if (k > 0) { - temp.head(k) = mat.diagonal().real().head(k).asDiagonal() * A10.adjoint(); - mat.coeffRef(k, k) -= (A10 * temp.head(k)).value(); - if (rs > 0) A21.noalias() -= A20 * temp.head(k); + if (done > 0 && rs > 0) { + auto A10 = mat.block(k, k0, fix<1>, done); + auto A20 = mat.block(k + 1, k0, rs, done); + temp.segment(k0, done) = mat.diagonal().real().segment(k0, done).asDiagonal() * A10.adjoint(); + A21.noalias() -= A20 * temp.segment(k0, done); } // In some previous versions of Eigen (e.g., 3.2.1), the scaling was omitted if the pivot // was smaller than the cutoff value. However, since LDLT is not rank-revealing // we should only make sure that we do not introduce INF or NaN values. // Remark that LAPACK also uses 0 as the cutoff value. - RealScalar realAkk = numext::real(mat.coeffRef(k, k)); - bool pivot_is_valid = (abs(realAkk) > RealScalar(0)); + RealScalar realAkk = numext::real(temp.coeff(k)); + mat.coeffRef(k, k) = Scalar(realAkk); + bool pivot_is_valid = (numext::abs(realAkk) > RealScalar(0)); + if (!(numext::isfinite)(realAkk)) ret = false; - if (k == 0 && !pivot_is_valid) { - // The entire diagonal is zero, there is nothing more to do - // except filling the transpositions, and checking whether the matrix is zero. - sign = ZeroSign; - for (Index j = 0; j < size; ++j) { - transpositions.coeffRef(j) = IndexType(j); - ret = ret && (mat.col(j).tail(size - j - 1).array() == Scalar(0)).all(); - } - return ret; - } - - if ((rs > 0) && pivot_is_valid) + // A zero pivot is the largest remaining, so the rest of D is zero too, and its column must be zero as well. + if ((rs > 0) && pivot_is_valid) { A21 /= realAkk; - else if (rs > 0) + // diag(S_{k+1}) = diag(S_k) - conj(L21) .* (D_k L21), where the next pivot is searched for. Squaring L21 + // first would underflow for |L21| < sqrt(min) on a graded matrix, where D_k |L21|^2 is representable. + temp.tail(rs).real() -= A21.conjugate().cwiseProduct(realAkk * A21).real(); + } else if (rs > 0) { ret = ret && (A21.array() == Scalar(0)).all(); - - if (found_zero_pivot && pivot_is_valid) - ret = false; // factorization failed - else if (!pivot_is_valid) - found_zero_pivot = true; + } if (sign == PositiveSemiDef) { if (realAkk < static_cast<RealScalar>(0)) sign = Indefinite; @@ -456,6 +505,59 @@ return ret; } + template <typename MatrixType, typename TranspositionType, typename Workspace> + static bool unblocked(MatrixType& mat, TranspositionType& transpositions, Workspace& temp, SignMatrix& sign) { + EIGEN_INTERNAL_ASSERT_SQUARE(mat); + bool ret = true; + if (factor_trivial(mat, transpositions, temp, sign, ret)) return ret; + return factor_panel(mat, 0, mat.rows(), transpositions, temp, sign); + } + + // Blocked (level-3) variant, with the panel/trailing-update structure of LAPACK's xPSTRF: each panel is factored + // by factor_panel(), then a single product applies it to the trailing Schur complement, A22 -= L21 D L21^*. + template <typename MatrixType, typename TranspositionType, typename Workspace, typename BlockWorkspace> + static bool blocked(MatrixType& mat, TranspositionType& transpositions, Workspace& temp, BlockWorkspace& workspace, + SignMatrix& sign) { + EIGEN_INTERNAL_ASSERT_SQUARE(mat); + const Index size = mat.rows(); + const Index nb = ldlt_blocksize<MatrixType>(size); + if (nb == 0) return unblocked(mat, transpositions, temp, sign); + bool ret = true; + if (factor_trivial(mat, transpositions, temp, sign, ret)) return ret; + + workspace.resize(size, nb); + for (Index k0 = 0; k0 < size; k0 += nb) { + const Index kb = numext::mini(nb, size - k0); + const Index k1 = k0 + kb; + if (!factor_panel(mat, k0, kb, transpositions, temp, sign)) ret = false; + + // The panel's interchanges, applied to the rows of the columns before it. Column-major storage takes them a + // column at a time, where a row swap would stride through memory. + EIGEN_IF_CONSTEXPR (MatrixType::IsRowMajor) { + for (Index k = k0; k < k1; ++k) { + const Index p = transpositions.coeff(k); + if (p != k) mat.row(k).head(k0).swap(mat.row(p).head(k0)); + } + } else { + for (Index c = 0; c < k0; ++c) { + for (Index k = k0; k < k1; ++k) { + const Index p = transpositions.coeff(k); + if (p != k) numext::swap(mat.coeffRef(k, c), mat.coeffRef(p, c)); + } + } + } + + const Index rs = size - k1; + if (rs > 0) { + auto L21 = mat.block(k1, k0, rs, kb); + auto W = workspace.topLeftCorner(rs, kb); + W.noalias() = L21 * mat.diagonal().real().segment(k0, kb).asDiagonal(); + mat.block(k1, k1, rs, rs).template triangularView<Lower>() -= L21 * W.adjoint(); + } + } + return ret; + } + // Reference for the algorithm: Davis and Hager, "Multiple Rank // Modifications of a Sparse Cholesky Factorization" (Algorithm 1) // Trivial rearrangements of their computations (Timothy E. Holy) @@ -466,7 +568,6 @@ template <typename MatrixType, typename WDerived> static bool updateInPlace(MatrixType& mat, MatrixBase<WDerived>& w, const typename MatrixType::RealScalar& sigma = 1) { - using numext::isfinite; using Scalar = typename MatrixType::Scalar; using RealScalar = typename MatrixType::RealScalar; @@ -479,7 +580,7 @@ // Apply the update for (Index j = 0; j < size; j++) { // Check for termination due to an original decomposition of low-rank - if (!(isfinite)(alpha)) break; + if (!(numext::isfinite)(alpha)) break; // Update the diagonal terms RealScalar dj = numext::real(mat.coeff(j, j)); @@ -524,6 +625,13 @@ return ldlt_inplace<Lower>::unblocked(matt, transpositions, temp, sign); } + template <typename MatrixType, typename TranspositionType, typename Workspace, typename BlockWorkspace> + static EIGEN_STRONG_INLINE bool blocked(MatrixType& mat, TranspositionType& transpositions, Workspace& temp, + BlockWorkspace& workspace, SignMatrix& sign) { + Transpose<MatrixType> matt(mat); + return ldlt_inplace<Lower>::blocked(matt, transpositions, temp, workspace, sign); + } + template <typename MatrixType, typename TranspositionType, typename Workspace, typename WType> static EIGEN_STRONG_INLINE bool update(MatrixType& mat, TranspositionType& transpositions, Workspace& tmp, WType& w, const typename MatrixType::RealScalar& sigma = 1) { @@ -568,8 +676,9 @@ m_temporary.resize(size); m_sign = internal::ZeroSign; - m_info = internal::ldlt_inplace<UpLo>::unblocked(m_matrix, m_transpositions, m_temporary, m_sign) ? Success - : NumericalIssue; + m_info = internal::ldlt_inplace<UpLo>::blocked(m_matrix, m_transpositions, m_temporary, m_workspace, m_sign) + ? Success + : NumericalIssue; m_isInitialized = true; return *this; @@ -669,7 +778,6 @@ // dst = D^-* (L^-1 P b) // dst = D^-1 (L^-*T P b) // more precisely, use pseudo-inverse of D (see bug 241) - using std::abs; const typename Diagonal<const MatrixType>::RealReturnType vecD(vectorD()); // In some previous versions, tolerance was set to the max of 1/highest (or rather numeric_limits::min()) // and the maximal diagonal entry * epsilon as motivated by LAPACK's xGELSS: @@ -679,7 +787,7 @@ // xSYTRS routines use 0 for the tolerance. Using numeric_limits::min() gives us more robustness to denormals. RealScalar tolerance = (std::numeric_limits<RealScalar>::min)(); for (Index i = 0; i < vecD.size(); ++i) { - if (abs(vecD(i)) > tolerance) + if (numext::abs(vecD(i)) > tolerance) dst.row(i) /= vecD(i); else dst.row(i).setZero(); @@ -725,20 +833,13 @@ template <typename MatrixType, int UpLo_> MatrixType LDLT<MatrixType, UpLo_>::reconstructedMatrix() const { eigen_assert(m_isInitialized && "LDLT is not initialized."); - const Index size = m_matrix.rows(); - MatrixType res(size, size); - - // P - res.setIdentity(); - res = transpositionsP() * res; - // L^* P - res = matrixU() * res; - // D(L^*P) + MatrixType res = matrixU(); res = vectorD().real().asDiagonal() * res; - // L(DL^*P) res = matrixL() * res; - // P^T (LDL^*P) - res = transpositionsP().transpose() * res; + // P^T (L D L^*) P; the stored pivots cannot alias the new result. + auto lower_triangle = res.template selfadjointView<Lower>(); + for (Index k = m_matrix.rows() - 1; k >= 0; --k) lower_triangle.applyTransposition(k, m_transpositions.coeff(k)); + res.template triangularView<StrictlyUpper>() = res.adjoint(); return res; }
diff --git a/Eigen/src/Cholesky/LLT.h b/Eigen/src/Cholesky/LLT.h index 716f9d5..5412aee 100644 --- a/Eigen/src/Cholesky/LLT.h +++ b/Eigen/src/Cholesky/LLT.h
@@ -321,7 +321,6 @@ template <typename MatrixType, typename VectorType> static Index llt_rank_update_lower(MatrixType& mat, const VectorType& vec, const typename MatrixType::RealScalar& sigma) { - using std::sqrt; using Scalar = typename MatrixType::Scalar; using RealScalar = typename MatrixType::RealScalar; using ColXpr = typename MatrixType::ColXpr; @@ -340,7 +339,7 @@ // This version is based on Givens rotations. // It is faster than the other one below, but only works for updates, // i.e., for sigma > 0 - temp = sqrt(sigma) * vec; + temp = numext::sqrt(sigma) * vec; for (Index i = 0; i < n; ++i) { JacobiRotation<Scalar> g; @@ -365,7 +364,7 @@ RealScalar x = dj + swj2 / beta; if (x <= RealScalar(0)) return j; - RealScalar nLjj = sqrt(x); + RealScalar nLjj = numext::sqrt(x); mat.coeffRef(j, j) = nLjj; beta += swj2 / dj; @@ -387,8 +386,6 @@ using RealScalar = typename NumTraits<Scalar>::Real; template <typename MatrixType> static Index unblocked(MatrixType& mat) { - using std::sqrt; - EIGEN_INTERNAL_ASSERT_SQUARE(mat); const Index size = mat.rows(); for (Index k = 0; k < size; ++k) { @@ -401,7 +398,7 @@ RealScalar x = numext::real(mat.coeff(k, k)); if (k > 0) x -= A10.squaredNorm(); if (x <= RealScalar(0)) return k; - mat.coeffRef(k, k) = x = sqrt(x); + mat.coeffRef(k, k) = x = numext::sqrt(x); if (k > 0 && rs > 0) A21.noalias() -= A20 * A10.adjoint(); if (rs > 0) A21 /= x; }
diff --git a/Eigen/src/CholmodSupport/CholmodSupport.h b/Eigen/src/CholmodSupport/CholmodSupport.h index eb0877d..351a7ea 100644 --- a/Eigen/src/CholmodSupport/CholmodSupport.h +++ b/Eigen/src/CholmodSupport/CholmodSupport.h
@@ -410,15 +410,10 @@ } /** \returns the determinant of the underlying matrix from the current factorization */ - Scalar determinant() const { - using std::exp; - return exp(logDeterminant()); - } + Scalar determinant() const { return numext::exp(logDeterminant()); } /** \returns the log determinant of the underlying matrix from the current factorization */ Scalar logDeterminant() const { - using numext::real; - using std::log; eigen_assert(m_factorizationIsOk && "The decomposition is not in a valid state for solving, you must first call either compute() or " "symbolic()/numeric()"); @@ -448,7 +443,7 @@ // Simplicial factorization stored as standard CSC matrix. StorageIndex* p = static_cast<StorageIndex*>(m_cholmodFactor->p); Index size = m_cholmodFactor->n; - for (Index k = 0; k < size; ++k) logDet += log(real(x[p[k]])); + for (Index k = 0; k < size; ++k) logDet += numext::log(numext::real(x[p[k]])); } if (m_cholmodFactor->is_ll) logDet *= 2.0; return logDet;
diff --git a/Eigen/src/Core/AssignEvaluator.h b/Eigen/src/Core/AssignEvaluator.h index 419f090..2728899 100644 --- a/Eigen/src/Core/AssignEvaluator.h +++ b/Eigen/src/Core/AssignEvaluator.h
@@ -535,8 +535,21 @@ head_loop::run(kernel, 0, alignedStart); - for (Index index = alignedStart; index <= size - PacketSize; index += PacketSize) - kernel.template assignPacket<Alignment, SrcAlignment, PacketType>(index); +#if EIGEN_GNUC_STRICT_AT_LEAST(8, 0, 0) && EIGEN_ARCH_i386_OR_x86_64 && !defined(EIGEN_GPUCC) + // GCC leaves cheap unary packet operations rolled. Copies, multi-source expressions and expensive functors + // showed no consistent benefit, so use the coefficient cost as a conservative unrolling heuristic. + using SrcEvaluator = typename Kernel::SrcEvaluatorType; + EIGEN_IF_CONSTEXPR (int(SrcEvaluator::CoeffReadCost) > NumTraits<Scalar>::ReadCost && + int(SrcEvaluator::CoeffReadCost) <= 2 * NumTraits<Scalar>::ReadCost) { +#pragma GCC unroll 2 + for (Index index = alignedStart; index <= size - PacketSize; index += PacketSize) + kernel.template assignPacket<Alignment, SrcAlignment, PacketType>(index); + } else +#endif + { + for (Index index = alignedStart; index <= size - PacketSize; index += PacketSize) + kernel.template assignPacket<Alignment, SrcAlignment, PacketType>(index); + } // Derive the tail bound directly; GCC can lose its range when it comes from the packet-loop induction variable. const Index alignedEnd = size - (size - alignedStart) % PacketSize; @@ -930,11 +943,19 @@ } } +// The evaluator a dense assignment reads its source through. The assignment loops index the source by the +// destination's storage order and use only the source evaluator's flags, so a source may be evaluated in another +// storage order than its expression reports; see the coefficient-based product in ProductEvaluators.h. +template <typename DstEvaluatorType, typename SrcXprType, typename Enable = void> +struct dense_assignment_source_evaluator { + using type = evaluator<SrcXprType>; +}; + template <typename DstXprType, typename SrcXprType, typename Functor> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE constexpr void call_dense_assignment_loop(DstXprType& dst, const SrcXprType& src, const Functor& func) { using DstEvaluatorType = evaluator<DstXprType>; - using SrcEvaluatorType = evaluator<SrcXprType>; + using SrcEvaluatorType = typename dense_assignment_source_evaluator<DstEvaluatorType, SrcXprType>::type; SrcEvaluatorType srcEvaluator(src); @@ -1044,7 +1065,7 @@ EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE void call_restricted_packet_assignment_no_alias(Dst& dst, const Src& src, const Func& func) { using DstEvaluatorType = evaluator<Dst>; - using SrcEvaluatorType = evaluator<Src>; + using SrcEvaluatorType = typename dense_assignment_source_evaluator<DstEvaluatorType, Src>::type; using Kernel = restricted_packet_dense_assignment_kernel<DstEvaluatorType, SrcEvaluatorType, Func>; EIGEN_STATIC_ASSERT_LVALUE(Dst)
diff --git a/Eigen/src/Core/Block.h b/Eigen/src/Core/Block.h index c995430..6c337e3 100644 --- a/Eigen/src/Core/Block.h +++ b/Eigen/src/Core/Block.h
@@ -187,11 +187,22 @@ namespace internal { -/** \internal Internal implementation of dense Blocks in the general case. */ +/** \internal Start of the row or column taken by the Block(xpr, i) constructor: a row if and only if BlockRows == 1 + * and BlockCols == XprType::ColsAtCompileTime, a column if and only if BlockRows == XprType::RowsAtCompileTime and + * BlockCols == 1. A 1x1 xpr is both, with i == 0. */ +template <typename XprType, int BlockRows, int BlockCols> +struct block_vector_start { + static constexpr Index row(Index i) { return (BlockRows == 1) && (BlockCols == XprType::ColsAtCompileTime) ? i : 0; } + static constexpr Index col(Index i) { return (BlockRows == XprType::RowsAtCompileTime) && (BlockCols == 1) ? i : 0; } +}; + +/** \internal Internal implementation of dense Blocks in the general case. Coefficient and packet access go through + * DenseCoeffsBase and evaluator<Block>. */ template <typename XprType, int BlockRows, int BlockCols, bool InnerPanel, bool HasDirectAccess> class BlockImpl_dense : public internal::dense_xpr_base<Block<XprType, BlockRows, BlockCols, InnerPanel>>::type { using BlockType = Block<XprType, BlockRows, BlockCols, InnerPanel>; using XprTypeNested = typename internal::ref_selector<XprType>::non_const_type; + using VectorStart = block_vector_start<XprType, BlockRows, BlockCols>; public: using Base = typename internal::dense_xpr_base<BlockType>::type; @@ -202,12 +213,8 @@ */ EIGEN_DEVICE_FUNC constexpr BlockImpl_dense(XprType& xpr, Index i) : m_xpr(xpr), - // It is a row if and only if BlockRows==1 and BlockCols==XprType::ColsAtCompileTime, - // and it is a column if and only if BlockRows==XprType::RowsAtCompileTime and BlockCols==1, - // all other cases are invalid. - // The case a 1x1 matrix seems ambiguous, but the result is the same anyway. - m_startRow((BlockRows == 1) && (BlockCols == XprType::ColsAtCompileTime) ? i : 0), - m_startCol((BlockRows == XprType::RowsAtCompileTime) && (BlockCols == 1) ? i : 0), + m_startRow(VectorStart::row(i)), + m_startCol(VectorStart::col(i)), m_blockRows(BlockRows == 1 ? 1 : xpr.rows()), m_blockCols(BlockCols == 1 ? 1 : xpr.cols()) {} @@ -225,57 +232,6 @@ EIGEN_DEVICE_FUNC constexpr Index rows() const { return m_blockRows.value(); } EIGEN_DEVICE_FUNC constexpr Index cols() const { return m_blockCols.value(); } - EIGEN_DEVICE_FUNC inline Scalar& coeffRef(Index rowId, Index colId) { - EIGEN_STATIC_ASSERT_LVALUE(XprType) - return m_xpr.coeffRef(rowId + m_startRow.value(), colId + m_startCol.value()); - } - - EIGEN_DEVICE_FUNC inline const Scalar& coeffRef(Index rowId, Index colId) const { - return m_xpr.derived().coeffRef(rowId + m_startRow.value(), colId + m_startCol.value()); - } - - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE const CoeffReturnType coeff(Index rowId, Index colId) const { - return m_xpr.coeff(rowId + m_startRow.value(), colId + m_startCol.value()); - } - - EIGEN_DEVICE_FUNC inline Scalar& coeffRef(Index index) { - EIGEN_STATIC_ASSERT_LVALUE(XprType) - return m_xpr.coeffRef(m_startRow.value() + (RowsAtCompileTime == 1 ? 0 : index), - m_startCol.value() + (RowsAtCompileTime == 1 ? index : 0)); - } - - EIGEN_DEVICE_FUNC inline const Scalar& coeffRef(Index index) const { - return m_xpr.coeffRef(m_startRow.value() + (RowsAtCompileTime == 1 ? 0 : index), - m_startCol.value() + (RowsAtCompileTime == 1 ? index : 0)); - } - - EIGEN_DEVICE_FUNC inline const CoeffReturnType coeff(Index index) const { - return m_xpr.coeff(m_startRow.value() + (RowsAtCompileTime == 1 ? 0 : index), - m_startCol.value() + (RowsAtCompileTime == 1 ? index : 0)); - } - - template <int LoadMode> - EIGEN_DEVICE_FUNC inline PacketScalar packet(Index rowId, Index colId) const { - return m_xpr.template packet<Unaligned>(rowId + m_startRow.value(), colId + m_startCol.value()); - } - - template <int LoadMode> - EIGEN_DEVICE_FUNC inline void writePacket(Index rowId, Index colId, const PacketScalar& val) { - m_xpr.template writePacket<Unaligned>(rowId + m_startRow.value(), colId + m_startCol.value(), val); - } - - template <int LoadMode> - EIGEN_DEVICE_FUNC inline PacketScalar packet(Index index) const { - return m_xpr.template packet<Unaligned>(m_startRow.value() + (RowsAtCompileTime == 1 ? 0 : index), - m_startCol.value() + (RowsAtCompileTime == 1 ? index : 0)); - } - - template <int LoadMode> - EIGEN_DEVICE_FUNC inline void writePacket(Index index, const PacketScalar& val) { - m_xpr.template writePacket<Unaligned>(m_startRow.value() + (RowsAtCompileTime == 1 ? 0 : index), - m_startCol.value() + (RowsAtCompileTime == 1 ? index : 0), val); - } - #ifdef EIGEN_PARSED_BY_DOXYGEN /** \sa MapBase::data() */ EIGEN_DEVICE_FUNC constexpr const Scalar* data() const; @@ -309,14 +265,16 @@ : public MapBase<Block<XprType, BlockRows, BlockCols, InnerPanel>> { using BlockType = Block<XprType, BlockRows, BlockCols, InnerPanel>; using XprTypeNested = typename internal::ref_selector<XprType>::non_const_type; - enum { XprTypeIsRowMajor = (int(traits<XprType>::Flags) & RowMajorBit) != 0 }; + using VectorStart = block_vector_start<XprType, BlockRows, BlockCols>; - /** \internal Returns base+offset (unless base is null, in which case returns null). - * Adding an offset to nullptr is undefined behavior, so we must avoid it. - */ - template <typename Scalar> - EIGEN_DEVICE_FUNC constexpr EIGEN_ALWAYS_INLINE static Scalar* add_to_nullable_pointer(Scalar* base, Index offset) { - return base != nullptr ? base + offset : nullptr; + /** \internal An empty block gets a null pointer: its start can lie past the end of xpr's storage. So does a block of + * a null xpr, since offsetting nullptr is undefined behavior. */ + EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE static auto block_data(XprType& xpr, Index startRow, Index startCol, + Index blockRows, Index blockCols) + -> decltype(xpr.data()) { + return (blockRows == 0 || blockCols == 0 || xpr.data() == nullptr) + ? nullptr + : xpr.data() + startRow * xpr.rowStride() + startCol * xpr.colStride(); } public: @@ -327,51 +285,32 @@ /** Column or Row constructor */ EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE BlockImpl_dense(XprType& xpr, Index i) - : Base((BlockRows == 0 || BlockCols == 0) - ? nullptr - : add_to_nullable_pointer( - xpr.data(), - i * (((BlockRows == 1) && (BlockCols == XprType::ColsAtCompileTime) && (!XprTypeIsRowMajor)) || - ((BlockRows == XprType::RowsAtCompileTime) && (BlockCols == 1) && - (XprTypeIsRowMajor)) - ? xpr.innerStride() - : xpr.outerStride())), + : Base(block_data(xpr, VectorStart::row(i), VectorStart::col(i), BlockRows == 1 ? 1 : xpr.rows(), + BlockCols == 1 ? 1 : xpr.cols()), BlockRows == 1 ? 1 : xpr.rows(), BlockCols == 1 ? 1 : xpr.cols()), m_xpr(xpr), - m_startRow((BlockRows == 1) && (BlockCols == XprType::ColsAtCompileTime) ? i : 0), - m_startCol((BlockRows == XprType::RowsAtCompileTime) && (BlockCols == 1) ? i : 0) { - init(); - } + m_startRow(VectorStart::row(i)), + m_startCol(VectorStart::col(i)), + m_outerStride(outer_stride(xpr)) {} /** Fixed-size constructor */ EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE BlockImpl_dense(XprType& xpr, Index startRow, Index startCol) - : Base((BlockRows == 0 || BlockCols == 0) - ? nullptr - : add_to_nullable_pointer(xpr.data(), - xpr.innerStride() * (XprTypeIsRowMajor ? startCol : startRow) + - xpr.outerStride() * (XprTypeIsRowMajor ? startRow : startCol))), + : Base(block_data(xpr, startRow, startCol, BlockRows, BlockCols)), m_xpr(xpr), m_startRow(startRow), - m_startCol(startCol) { - init(); - } + m_startCol(startCol), + m_outerStride(outer_stride(xpr)) {} /** Dynamic-size constructor */ EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE BlockImpl_dense(XprType& xpr, Index startRow, Index startCol, Index blockRows, Index blockCols) - : Base((blockRows == 0 || blockCols == 0) - ? nullptr - : add_to_nullable_pointer(xpr.data(), - xpr.innerStride() * (XprTypeIsRowMajor ? startCol : startRow) + - xpr.outerStride() * (XprTypeIsRowMajor ? startRow : startCol)), - blockRows, blockCols), + : Base(block_data(xpr, startRow, startCol, blockRows, blockCols), blockRows, blockCols), m_xpr(xpr), m_startRow(startRow), - m_startCol(startCol) { - init(); - } + m_startCol(startCol), + m_outerStride(outer_stride(xpr)) {} EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE const internal::remove_all_t<XprTypeNested>& nestedExpression() const noexcept { return m_xpr; @@ -392,9 +331,8 @@ EIGEN_DEVICE_FUNC constexpr StorageIndex startCol() const noexcept { return m_startCol.value(); } protected: - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void init() { - m_outerStride = - internal::traits<BlockType>::HasSameStorageOrderAsXprType ? m_xpr.outerStride() : m_xpr.innerStride(); + EIGEN_DEVICE_FUNC static constexpr Index outer_stride(const XprType& xpr) { + return internal::traits<BlockType>::HasSameStorageOrderAsXprType ? xpr.outerStride() : xpr.innerStride(); } XprTypeNested m_xpr;
diff --git a/Eigen/src/Core/CoreEvaluators.h b/Eigen/src/Core/CoreEvaluators.h index 67fe5df..975b307 100644 --- a/Eigen/src/Core/CoreEvaluators.h +++ b/Eigen/src/Core/CoreEvaluators.h
@@ -1450,22 +1450,27 @@ : int(outer_stride_at_compile_time<ArgType>::value), OuterStrideAtCompileTime = HasSameStorageOrderAsArgType ? int(outer_stride_at_compile_time<ArgType>::value) : int(inner_stride_at_compile_time<ArgType>::value), - // Direct-access blocks use mapbase_evaluator's contiguous loads, not the parent's packet method. - MaskPacketAccessBit = (InnerStrideAtCompileTime == 1 || - (!(evaluator<ArgType>::Flags & DirectAccessBit) && HasSameStorageOrderAsArgType)) - ? PacketAccessBit - : 0, + // Matches block_evaluator's dispatch. Direct-access blocks use mapbase_evaluator's contiguous loads; the others + // forward to the parent's packet method, which only loads along the parent's inner direction. + ArgHasDirectAccess = bool(has_direct_access<ArgType>::value) && bool(evaluator<ArgType>::Flags & DirectAccessBit), + MaskPacketAccessBit = + (ArgHasDirectAccess ? InnerStrideAtCompileTime == 1 : bool(HasSameStorageOrderAsArgType)) ? PacketAccessBit : 0, + ArgInnerSize = ArgTypeIsRowMajor ? int(ArgType::ColsAtCompileTime) : int(ArgType::RowsAtCompileTime), + // InnerPanel is deduced from the parent's traits, so it only holds in the parent evaluator's storage order. A block + // spanning its parent's fixed inner size is an inner panel too, e.g. block<6, 4>(0, j) of a 6x6 matrix. + IsInnerPanel = HasSameStorageOrderAsArgType && (InnerPanel || (InnerSize != Dynamic && InnerSize == ArgInnerSize)), FlagsLinearAccessBit = (RowsAtCompileTime == 1 || ColsAtCompileTime == 1 || - (InnerPanel && (evaluator<ArgType>::Flags & LinearAccessBit))) + (IsInnerPanel && (evaluator<ArgType>::Flags & LinearAccessBit))) ? LinearAccessBit : 0, FlagsRowMajorBit = XprType::Flags & RowMajorBit, - Flags0 = evaluator<ArgType>::Flags & ((HereditaryBits & ~RowMajorBit) | DirectAccessBit | MaskPacketAccessBit), + Flags0 = evaluator<ArgType>::Flags & + ((HereditaryBits & ~RowMajorBit) | (ArgHasDirectAccess ? DirectAccessBit : 0) | MaskPacketAccessBit), Flags = Flags0 | FlagsLinearAccessBit | FlagsRowMajorBit, PacketAlignment = unpacket_traits<PacketScalar>::alignment, - Alignment0 = (InnerPanel && (OuterStrideAtCompileTime != Dynamic) && (OuterStrideAtCompileTime != 0) && + Alignment0 = (IsInnerPanel && (OuterStrideAtCompileTime != Dynamic) && (OuterStrideAtCompileTime != 0) && (((OuterStrideAtCompileTime * int(sizeof(Scalar))) % int(PacketAlignment)) == 0)) ? int(PacketAlignment) : 0,
diff --git a/Eigen/src/Core/CwiseBinaryOp.h b/Eigen/src/Core/CwiseBinaryOp.h index 8064d27..149bcce 100644 --- a/Eigen/src/Core/CwiseBinaryOp.h +++ b/Eigen/src/Core/CwiseBinaryOp.h
@@ -18,6 +18,17 @@ namespace Eigen { namespace internal { +template <typename BinaryOp> +struct binary_op_compatibility { + using type = BinaryOp; +}; + +// The optimized multiply has the same scalar contract as scalar_product_op. +template <typename LhsScalar, typename RhsScalar> +struct binary_op_compatibility<fast_mult_op<LhsScalar, RhsScalar>> { + using type = scalar_product_op<LhsScalar, RhsScalar>; +}; + template <typename BinaryOp, typename Lhs, typename Rhs> struct traits<CwiseBinaryOp<BinaryOp, Lhs, Rhs>> { // we must not inherit from traits<Lhs> since it has @@ -89,7 +100,8 @@ typename internal::traits<Rhs>::StorageKind, BinaryOp>::ret>::Base; EIGEN_GENERIC_PUBLIC_INTERFACE(CwiseBinaryOp) - EIGEN_CHECK_BINARY_COMPATIBILITY(BinaryOp, typename Lhs::Scalar, typename Rhs::Scalar) + EIGEN_CHECK_BINARY_COMPATIBILITY(typename internal::binary_op_compatibility<BinaryOp>::type, typename Lhs::Scalar, + typename Rhs::Scalar) EIGEN_STATIC_ASSERT_SAME_MATRIX_SIZE(Lhs, Rhs) using LhsNested = typename internal::ref_selector<LhsType>::type;
diff --git a/Eigen/src/Core/Dot.h b/Eigen/src/Core/Dot.h index 3efa06c..391dd64 100644 --- a/Eigen/src/Core/Dot.h +++ b/Eigen/src/Core/Dot.h
@@ -72,7 +72,7 @@ // Check before dividing: an overflowing reciprocal can raise FE_OVERFLOW even if the two-step fallback is used. if (sqrtNorm < Accumulator(1) && invScale > Accumulator(NumTraits<Accumulator>::highest()) * sqrtNorm) return false; Accumulator localSqrtNorm = sqrtNorm; - EIGEN_OPTIMIZATION_BARRIER(localSqrtNorm) + pbarrier(localSqrtNorm); factor = invScale / localSqrtNorm; return factor >= stable_normalization_normal_min<Accumulator, Accumulator>::run(); } @@ -478,8 +478,7 @@ struct lpNorm_selector { using RealScalar = typename NumTraits<typename traits<Derived>::Scalar>::Real; EIGEN_DEVICE_FUNC static inline RealScalar run(const MatrixBase<Derived>& m) { - EIGEN_USING_STD(pow) - return pow(m.cwiseAbs().array().pow(p).sum(), RealScalar(1) / p); + return numext::pow(m.cwiseAbs().array().pow(p).sum(), RealScalar(1) / p); } };
diff --git a/Eigen/src/Core/GeneralProduct.h b/Eigen/src/Core/GeneralProduct.h index 1ff75fd..8477313 100644 --- a/Eigen/src/Core/GeneralProduct.h +++ b/Eigen/src/Core/GeneralProduct.h
@@ -396,16 +396,72 @@ } } -// The vector is on the left => transposition +// Transposing a left-vector GEMV is valid only for commutative coefficient multiplication. template <int StorageOrder, bool BlasCompatible> struct gemv_dense_selector<OnTheLeft, StorageOrder, BlasCompatible> { template <typename Lhs, typename Rhs, typename Dest> static void run(const Lhs& lhs, const Rhs& rhs, Dest& dest, const typename Dest::Scalar& alpha) { + run_impl(lhs, rhs, dest, alpha, + functor_is_commutative<scalar_product_op<typename Lhs::Scalar, typename Rhs::Scalar>>()); + } + + private: + template <typename Lhs, typename Rhs, typename Dest> + static void run_impl(const Lhs& lhs, const Rhs& rhs, Dest& dest, const typename Dest::Scalar& alpha, true_type) { Transpose<Dest> destT(dest); enum { OtherStorageOrder = StorageOrder == RowMajor ? ColMajor : RowMajor }; gemv_dense_selector<OnTheRight, OtherStorageOrder, BlasCompatible>::run(rhs.transpose(), lhs.transpose(), destT, alpha); } + + template <typename Lhs, typename Rhs, typename Dest> + static void run_impl(const Lhs& lhs, const Rhs& rhs, Dest& dest, const typename Dest::Scalar& alpha, false_type) { + if (numext::is_exactly_zero(alpha)) return; + evaluator<Lhs> lhsEval(lhs); + evaluator<Rhs> rhsEval(rhs); + for (Index col = 0; col < rhs.cols(); ++col) { + typename Product<Lhs, Rhs>::Scalar sum(0); + for (Index k = 0; k < rhs.rows(); ++k) sum += lhsEval.coeff(k) * rhsEval.coeff(k, col); + dest.coeffRef(col) += alpha * sum; + } + } +}; + +// Keep the shortcut's expression out of the general GEMV caller's inlining budget. +template <bool ConjugateLhs, typename LhsScalar, typename RhsScalar, typename Dest> +EIGEN_DONT_INLINE void gemv_single_column_update(const LhsScalar* data, Index rows, Index stride, RhsScalar rhsValue, + typename Dest::Scalar alpha, Dest& dest) { + using Column = Matrix<LhsScalar, Dynamic, 1>; + // A runtime stride of one must be visible to the assignment evaluator to enable packets. + // Preserve alpha * (lhs * rhs): folding alpha into rhs can overflow before the product. + if (stride == 1) { + Map<const Column> column(data, rows); + dest.col(0).matrix() += alpha * (column.template conjugateIf<ConjugateLhs>() * rhsValue); + } else { + Map<const Column, Unaligned, InnerStride<>> column(data, rows, InnerStride<>(stride)); + dest.col(0).matrix() += alpha * (column.template conjugateIf<ConjugateLhs>() * rhsValue); + } +} + +template <bool Enable, bool ConjugateLhs, bool ConjugateRhs> +struct gemv_single_column { + template <typename Lhs, typename Rhs, typename Scalar, typename Dest> + static EIGEN_STRONG_INLINE bool run(const Lhs&, const Rhs&, const Scalar&, Dest&, bool) { + return false; + } +}; + +template <bool ConjugateLhs, bool ConjugateRhs> +struct gemv_single_column<true, ConjugateLhs, ConjugateRhs> { + template <typename Lhs, typename Rhs, typename Scalar, typename Dest> + static EIGEN_STRONG_INLINE bool run(const Lhs& lhs, const Rhs& rhs, const Scalar& alpha, Dest& dest, bool eligible) { + if (!eligible || lhs.cols() != 1) return false; + if (!numext::is_exactly_zero(alpha)) { + gemv_single_column_update<ConjugateLhs>(lhs.data(), lhs.rows(), lhs.col(0).innerStride(), + conj_if<ConjugateRhs>()(rhs.coeff(0, 0)), alpha, dest); + } + return true; + } }; template <> @@ -426,6 +482,16 @@ ResScalar actualAlpha = combine_scalar_factors(alpha, lhs, rhs); + constexpr bool DoubleScalars = std::is_same<LhsScalar, RhsScalar>::value && + std::is_same<LhsScalar, ResScalar>::value && std::is_same<ResScalar, double>::value; + // Float updates can be slower than the buffered packet kernel; bypass it only for matching double scalars. + constexpr bool EnableSingleColumn = + Lhs::ColsAtCompileTime == Dynamic && DoubleScalars && Dest::InnerStrideAtCompileTime != 1; + if (gemv_single_column<EnableSingleColumn, LhsBlasTraits::NeedToConjugate, RhsBlasTraits::NeedToConjugate>::run( + actualLhs, actualRhs, actualAlpha, dest, + Dest::InnerStrideAtCompileTime != Dynamic || dest.innerStride() != 1)) + return; + // make sure Dest is a compile-time vector type (bug 1166) using ActualDest = std::conditional_t<Dest::IsVectorAtCompileTime, Dest, typename Dest::ColXpr>; @@ -493,6 +559,15 @@ ResScalar actualAlpha = combine_scalar_factors(alpha, lhs, rhs); + // Cost heuristic: broadcasts may allocate for owning scalars. Complex shortcuts can worsen GCC's kernel spilling. + constexpr bool EnableSingleColumn = Lhs::ColsAtCompileTime == Dynamic && !NumTraits<ResScalar>::IsComplex && + std::is_trivially_copyable<LhsScalar>::value && + std::is_trivially_copyable<RhsScalar>::value && + std::is_trivially_copyable<ResScalar>::value; + if (gemv_single_column<EnableSingleColumn, LhsBlasTraits::NeedToConjugate, RhsBlasTraits::NeedToConjugate>::run( + actualLhs, actualRhs, actualAlpha, dest, true)) + return; + enum { // FIXME: find a way to allow an inner stride on the result if packet_traits<Scalar>::size==1 // on the other hand it is good for the cache to pack the vector anyways...
diff --git a/Eigen/src/Core/GenericPacketMath.h b/Eigen/src/Core/GenericPacketMath.h index a913986..53a8e4c 100644 --- a/Eigen/src/Core/GenericPacketMath.h +++ b/Eigen/src/Core/GenericPacketMath.h
@@ -321,6 +321,18 @@ } }; +// Make x opaque to the compiler without changing its value, where supported. +template <typename Packet> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void pbarrier(Packet& x) { + EIGEN_UNUSED_VARIABLE(x); + EIGEN_OPTIMIZATION_BARRIER(x) +} +// The wrapper is a class, which EIGEN_OPTIMIZATION_BARRIER passes through memory; its vector can stay in a register. +template <typename T, int unique_id> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void pbarrier(eigen_packet_wrapper<T, unique_id>& x) { + pbarrier(x.m_val); +} + /** \internal \returns a + b (coeff-wise) */ template <typename Packet> EIGEN_DEVICE_FUNC inline Packet padd(const Packet& a, const Packet& b) { @@ -744,8 +756,7 @@ /** \internal \returns the phase angle of \a a */ template <typename Packet> EIGEN_DEVICE_FUNC inline Packet parg(const Packet& a) { - using numext::arg; - return arg(a); + return numext::arg(a); } /** \internal \returns \a a arithmetically shifted by N bits to the right */ @@ -783,8 +794,7 @@ */ template <typename Packet> EIGEN_DEVICE_FUNC inline Packet pldexp(const Packet& a, const Packet& exponent) { - EIGEN_USING_STD(ldexp) - return static_cast<Packet>(ldexp(a, static_cast<int>(exponent))); + return static_cast<Packet>(numext::ldexp(a, static_cast<int>(exponent))); } /** \internal \returns the absolute difference of \a a and \a b (coeff-wise) */ @@ -876,7 +886,7 @@ template <typename Scalar> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE Scalar pscalar_from_float_bits(numext::uint32_t bits) { #if EIGEN_COMP_GNUC_STRICT && defined(__FINITE_MATH_ONLY__) && __FINITE_MATH_ONLY__ && \ - !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(SYCL_DEVICE_ONLY) + !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) // GCC also needs the integer pattern hidden before bit_cast and widening. Use a memory operand so this works on // targets where EIGEN_OPTIMIZATION_BARRIER is intentionally unavailable. __asm__("" : "+m"(bits)); @@ -1296,85 +1306,73 @@ /** \internal \returns the sine of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet psin(const Packet& a) { - EIGEN_USING_STD(sin); - return sin(a); + return numext::sin(a); } /** \internal \returns the cosine of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet pcos(const Packet& a) { - EIGEN_USING_STD(cos); - return cos(a); + return numext::cos(a); } /** \internal \returns the tan of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet ptan(const Packet& a) { - EIGEN_USING_STD(tan); - return tan(a); + return numext::tan(a); } /** \internal \returns the arc sine of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet pasin(const Packet& a) { - EIGEN_USING_STD(asin); - return asin(a); + return numext::asin(a); } /** \internal \returns the arc cosine of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet pacos(const Packet& a) { - EIGEN_USING_STD(acos); - return acos(a); + return numext::acos(a); } /** \internal \returns the hyperbolic sine of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet psinh(const Packet& a) { - EIGEN_USING_STD(sinh); - return sinh(a); + return numext::sinh(a); } /** \internal \returns the hyperbolic cosine of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet pcosh(const Packet& a) { - EIGEN_USING_STD(cosh); - return cosh(a); + return numext::cosh(a); } /** \internal \returns the arc tangent of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet patan(const Packet& a) { - EIGEN_USING_STD(atan); - return atan(a); + return numext::atan(a); } /** \internal \returns the hyperbolic tan of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet ptanh(const Packet& a) { - EIGEN_USING_STD(tanh); - return tanh(a); + return numext::tanh(a); } /** \internal \returns the arc tangent of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet patanh(const Packet& a) { - EIGEN_USING_STD(atanh); - return atanh(a); + return numext::atanh(a); } /** \internal \returns the inverse hyperbolic sine of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet pasinh(const Packet& a) { - EIGEN_USING_STD(asinh); - return asinh(a); + return numext::asinh(a); } /** \internal \returns the inverse hyperbolic cosine of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet pacosh(const Packet& a) { - EIGEN_USING_STD(acosh); - return acosh(a); + return numext::acosh(a); } /** \internal \returns the exp of \a a (coeff-wise) */ @@ -1398,8 +1396,7 @@ /** \internal \returns the log of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet plog(const Packet& a) { - EIGEN_USING_STD(log); - return log(a); + return numext::log(a); } /** \internal \returns the log1p of \a a (coeff-wise) */ @@ -1411,8 +1408,7 @@ /** \internal \returns the log10 of \a a (coeff-wise) */ template <typename Packet> EIGEN_DECLARE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet plog10(const Packet& a) { - EIGEN_USING_STD(log10); - return log10(a); + return numext::log10(a); } template <typename Packet, bool IsComplexPacket = NumTraits<typename unpacket_traits<Packet>::type>::IsComplex &&
diff --git a/Eigen/src/Core/GlobalFunctions.h b/Eigen/src/Core/GlobalFunctions.h index f464c2b..aa6e1a7 100644 --- a/Eigen/src/Core/GlobalFunctions.h +++ b/Eigen/src/Core/GlobalFunctions.h
@@ -176,7 +176,10 @@ inline const CwiseBinaryOp<internal::scalar_pow_op<Scalar, Derived::Scalar>, Constant<Scalar>, Derived> pow( const Scalar& x, const Eigen::ArrayBase<Derived>& exponents); #else -template <typename Scalar, typename Derived> +template < + typename Scalar, typename Derived, + std::enable_if_t< + !internal::has_competing_expression_binary_op<Derived, Scalar, internal::scalar_pow_op, true>::value, int> = 0> EIGEN_DEVICE_FUNC inline const EIGEN_SCALAR_BINARYOP_EXPR_RETURN_TYPE( typename internal::promote_scalar_arg<typename Derived::Scalar EIGEN_COMMA Scalar EIGEN_COMMA EIGEN_SCALAR_BINARY_SUPPORTED(internal::scalar_pow_op, Scalar, @@ -235,6 +238,29 @@ EIGEN_ARRAY_DECLARE_GLOBAL_EIGEN_UNARY(imag, scalar_imag_op) EIGEN_ARRAY_DECLARE_GLOBAL_EIGEN_UNARY(abs2, scalar_abs2_op) } // namespace internal + +#ifndef EIGEN_PARSED_BY_DOXYGEN +// A namespace-scope overload avoids one ADL candidate per associated expression class. +#define EIGEN_MAKE_EXPRESSION_BINARY_OP(METHOD, FUNCTOR) \ + template <typename ActualLhs, typename ActualRhs, \ + std::enable_if_t<internal::has_ambiguous_expression_binary_op< \ + ActualLhs, ActualRhs, FUNCTOR, internal::expression_binary_base_t<ActualRhs>>::value, \ + int> = 0> \ + EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE auto METHOD(const ActualLhs& lhs, const ActualRhs& rhs) { \ + using Dispatch = internal::expression_binary_op< \ + FUNCTOR, internal::has_ambiguous_expression_binary_op< \ + ActualLhs, ActualRhs, FUNCTOR, internal::expression_binary_base_t<ActualRhs>>::DenseSparse>; \ + return Dispatch::run(static_cast<const internal::expression_binary_base_t<ActualLhs>&>(lhs), \ + static_cast<const internal::expression_binary_base_t<ActualRhs>&>(rhs)); \ + } + +EIGEN_MAKE_EXPRESSION_BINARY_OP(operator*, internal::scalar_product_op) +EIGEN_MAKE_EXPRESSION_BINARY_OP(operator+, internal::scalar_sum_op) +EIGEN_MAKE_EXPRESSION_BINARY_OP(operator-, internal::scalar_difference_op) +EIGEN_MAKE_EXPRESSION_BINARY_OP(operator/, internal::scalar_quotient_op) +EIGEN_MAKE_EXPRESSION_BINARY_OP(operator%, internal::scalar_modulus_op) +#undef EIGEN_MAKE_EXPRESSION_BINARY_OP +#endif } // namespace Eigen // TODO: cleanly disable those functions that are not supported on Array (numext::real_ref, internal::random,
diff --git a/Eigen/src/Core/IO.h b/Eigen/src/Core/IO.h index 3f1223a..b9f3d9b 100644 --- a/Eigen/src/Core/IO.h +++ b/Eigen/src/Core/IO.h
@@ -66,14 +66,18 @@ fill(_fill), precision(_precision), flags(_flags) { - // TODO: check if rowPrefix, rowSuffix or rowSeparator contains a newline - // don't add rowSpacer if columns are not to be aligned if ((flags & DontAlignCols)) return; - int i = int(matPrefix.length()) - 1; - while (i >= 0 && matPrefix[i] != '\n') { - rowSpacer += ' '; - i--; - } + + // Find last newline between consecutive rows. + const std::string between{rowSuffix + rowSeparator}; + const std::size_t nlLastPos{between.rfind('\n')}; + if (nlLastPos == std::string::npos) return; + + // Align subsequent rows, accounting for characters after last newline. + const std::size_t indent{matPrefix.length() - (matPrefix.rfind('\n') + 1)}; + const std::size_t carried{between.length() - (nlLastPos + 1)}; + + if (indent > carried) rowSpacer.assign(indent - carried, ' '); } std::string matPrefix, matSuffix; std::string rowPrefix, rowSuffix, rowSeparator, rowSpacer;
diff --git a/Eigen/src/Core/MathFunctions.h b/Eigen/src/Core/MathFunctions.h index bbd4e46..9b85ed6 100644 --- a/Eigen/src/Core/MathFunctions.h +++ b/Eigen/src/Core/MathFunctions.h
@@ -952,8 +952,31 @@ EIGEN_DEVICE_FUNC static inline bool run(const bool& a) { return a; } }; +// Extra namespace to prevent leaking std::copysign into Eigen::internal. +namespace has_copysign_detail { + +template <typename T, typename EnableIf = void> +struct has_copysign_impl : public std::false_type {}; + +EIGEN_USING_STD(copysign) + +template <typename T> +struct has_copysign_impl<T, std::enable_if_t<std::is_constructible< + T, decltype(copysign(std::declval<const T&>(), std::declval<const T&>()))>::value>> + : public std::true_type {}; + +} // namespace has_copysign_detail + +// Detects whether copysign(T, T) is available via std:: or ADL. When unavailable, +// copysign_impl falls back to `< Scalar(0)` comparisons, which cannot inspect the +// sign bit of negative zero or NaN (e.g., copysign(-0, 1) returns -0, copysign(1, -0) +// returns +1, and complex_sqrt(-1 - 0i) returns +i rather than -i). +template <typename T> +struct has_copysign : public has_copysign_detail::has_copysign_impl<T> {}; + template <typename Scalar, bool IsComplex = (NumTraits<Scalar>::IsComplex != 0), - bool IsInteger = (NumTraits<Scalar>::IsInteger != 0)> + bool IsInteger = (NumTraits<Scalar>::IsInteger != 0), + bool HasCopysign = (!IsComplex && !IsInteger && has_copysign<Scalar>::value)> struct copysign_impl { EIGEN_DEVICE_FUNC static inline Scalar run(const Scalar& a, const Scalar& b) { EIGEN_USING_STD(copysign); @@ -962,24 +985,25 @@ }; template <typename Scalar, bool IsInteger> -struct copysign_impl<Scalar, true, IsInteger> { +struct copysign_impl<Scalar, true, IsInteger, false> { + using RealScalar = typename NumTraits<Scalar>::Real; EIGEN_DEVICE_FUNC static inline Scalar run(const Scalar& a, const Scalar& b) { - EIGEN_USING_STD(copysign); - return Scalar(copysign(numext::real(a), numext::real(b)), copysign(numext::imag(a), numext::imag(b))); + return Scalar(copysign_impl<RealScalar>::run(numext::real(a), numext::real(b)), + copysign_impl<RealScalar>::run(numext::imag(a), numext::imag(b))); } }; -template <typename Scalar> -struct copysign_impl<Scalar, false, true> { +template <typename Scalar, bool IsInteger> +struct copysign_impl<Scalar, false, IsInteger, false> { EIGEN_DEVICE_FUNC static inline Scalar run(const Scalar& a, const Scalar& b) { - EIGEN_IF_CONSTEXPR (!NumTraits<Scalar>::IsSigned) return a; + EIGEN_IF_CONSTEXPR (IsInteger && !NumTraits<Scalar>::IsSigned) return a; const Scalar abs_a = a < Scalar(0) ? -a : a; return b < Scalar(0) ? -abs_a : abs_a; } }; template <> -struct copysign_impl<bool, false, true> { +struct copysign_impl<bool, false, true, false> { EIGEN_DEVICE_FUNC static inline bool run(const bool& a, const bool&) { return a; } }; @@ -1279,7 +1303,7 @@ #endif #endif -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) #define SYCL_SPECIALIZE_SIGNED_INTEGER_TYPES_BINARY(NAME, FUNC) \ SYCL_SPECIALIZE_BINARY_FUNC(NAME, FUNC, cl::sycl::cl_char) \ @@ -1375,6 +1399,10 @@ return EIGEN_MATHFUNC_IMPL(copysign, Scalar)::run(x, y); } +#if defined(EIGEN_SYCL_DEVICE_ONLY) +SYCL_SPECIALIZE_FLOATING_TYPES_BINARY(copysign, copysign) +#endif + template <typename Scalar> EIGEN_DEVICE_FUNC inline Scalar negate(const Scalar& x) { return EIGEN_MATHFUNC_IMPL(negate, Scalar)::run(x); @@ -1418,7 +1446,7 @@ return EIGEN_MATHFUNC_IMPL(hypot, Scalar)::run(x, y); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_BINARY(hypot, hypot) #endif @@ -1427,7 +1455,7 @@ return EIGEN_MATHFUNC_IMPL(log1p, Scalar)::run(x); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(log1p, log1p) #endif @@ -1449,7 +1477,7 @@ return internal::pow_impl<ScalarX, ScalarY>::run(x, y); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_BINARY(pow, pow) #endif @@ -1466,7 +1494,7 @@ return internal::isfinite_impl(x); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY_FUNC_RET_TYPE(isnan, isnan, bool) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY_FUNC_RET_TYPE(isinf, isinf, bool) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY_FUNC_RET_TYPE(isfinite, isfinite, bool) @@ -1497,7 +1525,8 @@ return internal::nearest_integer_impl<Scalar>::run_trunc(x); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) +SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(rint, rint) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(round, round) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(floor, floor) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(ceil, ceil) @@ -1559,11 +1588,36 @@ } template <typename T> +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE T log10(T x) { + EIGEN_USING_STD(log10); + return log10(x); +} + +template <typename T> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE T log2(T x) { EIGEN_USING_STD(log2); return log2(x); } +#if defined(EIGEN_SYCL_DEVICE_ONLY) +template <> +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE float log10(float x) { + return cl::sycl::log10(x); +} +template <> +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE double log10(double x) { + return cl::sycl::log10(x); +} +template <> +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE float log2(float x) { + return cl::sycl::log2(x); +} +template <> +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE double log2(double x) { + return cl::sycl::log2(x); +} +#endif + /** Log base 2 for 32 bits positive integers. * Conveniently returns 0 for x==0. */ constexpr int log2(int x) { @@ -1598,10 +1652,18 @@ return x; } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(sqrt, sqrt) #endif +#if defined(EIGEN_VECTORIZE_SSE) +// Half/BFloat16 scalar wrappers precede the SSE definitions and must not instantiate the primary template. +template <> +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE float sqrt(const float& x); +template <> +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE double sqrt(const double& x); +#endif + /** \returns the cube root of \a x. **/ template <typename T> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE std::enable_if_t<!NumTraits<T>::IsComplex, T> cbrt(const T& x) { @@ -1615,6 +1677,10 @@ return pow(x, typename NumTraits<T>::Real(1.0 / 3.0)); } +#if defined(EIGEN_SYCL_DEVICE_ONLY) +SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(cbrt, cbrt) +#endif + /** \returns the reciprocal square root of \a x. **/ template <typename T> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE T rsqrt(const T& x) { @@ -1626,7 +1692,7 @@ return internal::log_impl<T>::run(x); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(log, log) #endif @@ -1643,9 +1709,9 @@ #endif template <typename T> -EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE - std::enable_if_t<NumTraits<T>::IsSigned && !NumTraits<T>::IsComplex, typename NumTraits<T>::Real> - abs(const T& x) { +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE std::enable_if_t< + (NumTraits<T>::IsSigned || !NumTraits<T>::IsInteger) && !NumTraits<T>::IsComplex, typename NumTraits<T>::Real> +abs(const T& x) { EIGEN_USING_STD(abs); return abs(x); } @@ -1657,13 +1723,13 @@ } template <typename T> -EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE - std::enable_if_t<!NumTraits<T>::IsSigned && !NumTraits<T>::IsComplex, typename NumTraits<T>::Real> - abs(const T& x) { +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE std::enable_if_t< + !NumTraits<T>::IsSigned && NumTraits<T>::IsInteger && !NumTraits<T>::IsComplex, typename NumTraits<T>::Real> +abs(const T& x) { return x; } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_INTEGER_TYPES_UNARY(abs, abs) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(abs, fabs) #endif @@ -1748,7 +1814,7 @@ } #endif -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(exp, exp) #endif @@ -1794,7 +1860,7 @@ return internal::complex_exp2_impl<RealScalar>::run(x); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(exp2, exp2) #endif @@ -1818,7 +1884,7 @@ return static_cast<T>(ldexp(x, exponent)); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) template <> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE cl::sycl::cl_float ldexp(const cl::sycl::cl_float& x, int exponent) { return cl::sycl::ldexp(x, exponent); @@ -1846,7 +1912,7 @@ return EIGEN_MATHFUNC_IMPL(expm1, Scalar)::run(x); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(expm1, expm1) #endif @@ -1868,7 +1934,7 @@ return cos(x); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(cos, cos) #endif @@ -1890,7 +1956,7 @@ return sin(x); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(sin, sin) #endif @@ -1912,7 +1978,7 @@ return tan(x); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(tan, tan) #endif @@ -1940,7 +2006,7 @@ return static_cast<T>(acosh(x)); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(acos, acos) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(acosh, acosh) #endif @@ -1969,7 +2035,7 @@ return static_cast<T>(asinh(x)); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(asin, asin) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(asinh, asinh) #endif @@ -2004,7 +2070,8 @@ return static_cast<T>(atanh(x)); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) +SYCL_SPECIALIZE_FLOATING_TYPES_BINARY(atan2, atan2) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(atan, atan) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(atanh, atanh) #endif @@ -2027,7 +2094,7 @@ return static_cast<T>(cosh(x)); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(cosh, cosh) #endif @@ -2049,7 +2116,7 @@ return static_cast<T>(sinh(x)); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(sinh, sinh) #endif @@ -2071,11 +2138,11 @@ return tanh(x); } -#if (!defined(EIGEN_GPUCC)) && EIGEN_FAST_MATH && !defined(SYCL_DEVICE_ONLY) +#if (!defined(EIGEN_GPUCC)) && EIGEN_FAST_MATH && !defined(EIGEN_SYCL_DEVICE_ONLY) EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE float tanh(float x) { return internal::ptanh_float(x); } #endif -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_UNARY(tanh, tanh) #endif @@ -2097,7 +2164,7 @@ return fmod(a, b); } -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_BINARY(fmod, fmod) #endif @@ -2154,7 +2221,7 @@ } #endif -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) SYCL_SPECIALIZE_FLOATING_TYPES_BINARY(nextafter, nextafter) #endif @@ -2170,7 +2237,7 @@ } #endif -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) #undef SYCL_SPECIALIZE_SIGNED_INTEGER_TYPES_BINARY #undef SYCL_SPECIALIZE_SIGNED_INTEGER_TYPES_UNARY #undef SYCL_SPECIALIZE_UNSIGNED_INTEGER_TYPES_BINARY @@ -2229,6 +2296,12 @@ namespace internal { +// Two-product residual using FMA, including software FMA when hardware FMA is unavailable. +template <typename Scalar> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Scalar twoprod_low_fma(const Scalar& x, const Scalar& y, const Scalar& xy) { + return numext::fma(x, y, Scalar(-xy)); +} + template <typename T> EIGEN_DEVICE_FUNC bool isfinite_impl(const std::complex<T>& x) { EIGEN_IF_CONSTEXPR ((std::is_floating_point<T>::value)) { @@ -2456,7 +2529,7 @@ const bool negative = b < RealScalar(0); const RealScalar abs_b = numext::abs(b); const RealScalar t_hi = abs_b * ln2_hi; - const RealScalar t_lo = numext::fma(abs_b, ln2_hi, -t_hi) + abs_b * ln2_lo; + const RealScalar t_lo = twoprod_low_fma(abs_b, ln2_hi, t_hi) + abs_b * ln2_lo; // |t_lo| <= ulp(t_hi) / 2, so the Taylor terms below are exact to rounding unless |t| exceeds about 2 / sqrt(eps). // The phase is within an ulp of b ln(2) while |t| < 1 / eps. RealScalar sin_lo = t_lo;
diff --git a/Eigen/src/Core/MathFunctionsImpl.h b/Eigen/src/Core/MathFunctionsImpl.h index b97a069..1c54592 100644 --- a/Eigen/src/Core/MathFunctionsImpl.h +++ b/Eigen/src/Core/MathFunctionsImpl.h
@@ -157,11 +157,10 @@ if ((numext::isinf)(x) || (numext::isinf)(y)) return NumTraits<RealScalar>::infinity(); if ((numext::isnan)(x) || (numext::isnan)(y)) return NumTraits<RealScalar>::quiet_NaN(); - EIGEN_USING_STD(sqrt); RealScalar p = numext::maxi(x, y); if (numext::is_exactly_zero(p)) return RealScalar(0); RealScalar qp = numext::mini(y, x) / p; - return p * sqrt(RealScalar(1) + qp * qp); + return p * numext::sqrt<RealScalar>(RealScalar(1) + qp * qp); } template <typename Scalar> @@ -180,7 +179,7 @@ const T zero = T(0); const T inf = NumTraits<T>::infinity(); EIGEN_IF_CONSTEXPR (Reciprocal) { - if ((numext::isinf)(x) || (numext::isinf)(y)) return ComplexT(zero, numext::copysign(zero, -y)); + if ((numext::isinf)(x) || (numext::isinf)(y)) return ComplexT(zero, numext::copysign<T>(zero, -y)); } else { if ((numext::isinf)(y)) return ComplexT(inf, y); if ((numext::isinf)(x)) { @@ -272,8 +271,7 @@ // Computes complex log. using T = typename NumTraits<ComplexT>::Real; T a = numext::abs(z); - EIGEN_USING_STD(atan2); - T b = atan2(z.imag(), z.real()); + T b = numext::atan2(z.imag(), z.real()); // log|z| is finite where |z| itself overflows or rounds to a subnormal: scale z exactly by a power of two first. if (EIGEN_PREDICT_FALSE((numext::isinf)(a) && (numext::isfinite)(z.real()) && (numext::isfinite)(z.imag()))) { const T half = T(0.5); @@ -292,8 +290,7 @@ EIGEN_DEVICE_FUNC ComplexT complex_log2(const ComplexT& z) { using T = typename NumTraits<ComplexT>::Real; const T a = numext::abs(z); - EIGEN_USING_STD(atan2); - const T b = T(EIGEN_LOG2E) * atan2(z.imag(), z.real()); + const T b = T(EIGEN_LOG2E) * numext::atan2(z.imag(), z.real()); if (EIGEN_PREDICT_FALSE((numext::isinf)(a) && (numext::isfinite)(z.real()) && (numext::isfinite)(z.imag()))) { const T half = T(0.5); return ComplexT(numext::log2(numext::abs(ComplexT(z.real() * half, z.imag() * half))) + T(1), b);
diff --git a/Eigen/src/Core/MatrixBase.h b/Eigen/src/Core/MatrixBase.h index ae17e0d..53c9df6 100644 --- a/Eigen/src/Core/MatrixBase.h +++ b/Eigen/src/Core/MatrixBase.h
@@ -455,9 +455,10 @@ ///////// SparseCore module ///////// template <typename OtherDerived> - EIGEN_STRONG_INLINE const typename SparseMatrixBase<OtherDerived>::template CwiseProductDenseReturnType<Derived>::Type + EIGEN_STRONG_INLINE const CwiseBinaryOp<internal::scalar_product_op<Scalar, typename OtherDerived::Scalar>, + const Derived, const OtherDerived> cwiseProduct(const SparseMatrixBase<OtherDerived>& other) const { - return other.cwiseProduct(derived()); + return {derived(), other.derived()}; } ///////// MatrixFunctions module /////////
diff --git a/Eigen/src/Core/NumTraits.h b/Eigen/src/Core/NumTraits.h index 417a71c..90ef95d 100644 --- a/Eigen/src/Core/NumTraits.h +++ b/Eigen/src/Core/NumTraits.h
@@ -43,6 +43,31 @@ EIGEN_DEVICE_FUNC constexpr static int run() { return 0; } }; +namespace precision_detail { +using std::ceil; +using std::log2; + +template <typename T, typename = void> +struct has_epsilon_digits : std::false_type {}; + +// Detect the complete calculation: ADL can return another custom expression type. +template <typename T> +struct has_epsilon_digits<T, void_t<decltype(int(ceil(-log2(NumTraits<typename NumTraits<T>::Real>::epsilon()))) + 1)>> + : std::true_type {}; +} // namespace precision_detail + +// Only formatting tolerates unspecified precision; numerical users of digits() still require it. +template <typename T, bool Known = std::numeric_limits<T>::is_specialized || NumTraits<T>::IsInteger || + precision_detail::has_epsilon_digits<T>::value> +struct decimal_digits_source { + EIGEN_DEVICE_FUNC constexpr static int run() { return default_digits_impl<T>::run(); } +}; + +template <typename T> +struct decimal_digits_source<T, false> { + EIGEN_DEVICE_FUNC constexpr static int run() { return 0; } +}; + // default implementation of digits10(), based on numeric_limits if specialized, // 0 for integer types, and floor((digits()-1)*log10(2)) otherwise. template <typename T, bool use_numeric_limits = std::numeric_limits<T>::is_specialized, @@ -58,7 +83,8 @@ using std::floor; using std::log10; using Real = typename NumTraits<T>::Real; - return int(floor((internal::default_digits_impl<Real>::run() - 1) * log10(2))); + const int digits = decimal_digits_source<Real>::run(); + return digits > 0 ? int(floor((digits - 1) * log10(2))) : 0; } }; @@ -83,7 +109,8 @@ using std::ceil; using std::log10; using Real = typename NumTraits<T>::Real; - return int(ceil(internal::default_digits_impl<Real>::run() * log10(2) + 1)); + const int digits = decimal_digits_source<Real>::run(); + return digits > 0 ? int(ceil(digits * log10(2) + 1)) : 0; } }; @@ -98,7 +125,9 @@ namespace numext { /** \internal bit-wise cast without changing the underlying bit representation. */ -#if defined(__cpp_lib_bit_cast) && __cpp_lib_bit_cast >= 201806L +// std::bit_cast has no __device__ qualifier; in GPU device passes it requires relaxed constexpr +// and, for non-literal return types such as Eigen::half, stays host-only and miscompiles under NVCC. +#if defined(__cpp_lib_bit_cast) && __cpp_lib_bit_cast >= 201806L && !defined(EIGEN_GPU_COMPILE_PHASE) template <typename Tgt, typename Src> EIGEN_DEVICE_FUNC constexpr Tgt bit_cast(const Src& src) { return std::bit_cast<Tgt>(src); @@ -139,6 +168,11 @@ * \tparam T the numeric type at hand * * This class stores enums, typedefs and static methods giving information about a numeric type. + * The default precision queries use std::numeric_limits when specialized. Otherwise binary precision uses an + * epsilon-based calculation when available, and decimal precision is derived from the real type's binary precision. + * Decimal queries without usable precision information return zero (unspecified precision). + * The binary digits() query still requires usable precision information. + * Custom scalar types can specialize these methods when finite precision information is available. * * The provided data consists of: * \li A typedef \c Real, giving the "real part" type of \a T. If \a T is already real,
diff --git a/Eigen/src/Core/PartialReduxEvaluator.h b/Eigen/src/Core/PartialReduxEvaluator.h index 0650218..1b0ac9d 100644 --- a/Eigen/src/Core/PartialReduxEvaluator.h +++ b/Eigen/src/Core/PartialReduxEvaluator.h
@@ -48,7 +48,9 @@ static constexpr int Cost = OuterSize == Dynamic ? HugeCost : OuterSize * Evaluator::CoeffReadCost + (OuterSize - 1) * functor_traits<Func>::Cost; - static constexpr int Unrolling = Cost <= EIGEN_UNROLLING_LIMIT ? CompleteUnrolling : NoUnrolling; + // A user-set EIGEN_UNROLLING_LIMIT can reach HugeCost, which therefore does not rule out a runtime size. + static constexpr int Unrolling = + (OuterSize != Dynamic && Cost <= EIGEN_UNROLLING_LIMIT) ? CompleteUnrolling : NoUnrolling; }; /* Value to be returned when size==0 , by default let's return 0 */
diff --git a/Eigen/src/Core/PermutationMatrix.h b/Eigen/src/Core/PermutationMatrix.h index 212f8f6..9e95178 100644 --- a/Eigen/src/Core/PermutationMatrix.h +++ b/Eigen/src/Core/PermutationMatrix.h
@@ -226,25 +226,25 @@ * * \note \blank \note_try_to_help_rvo */ - inline InverseReturnType inverse() const { return InverseReturnType(derived()); } + EIGEN_DEVICE_FUNC inline InverseReturnType inverse() const { return InverseReturnType(derived()); } /** \returns the transpose permutation matrix. * * \note \blank \note_try_to_help_rvo */ - inline InverseReturnType transpose() const { return InverseReturnType(derived()); } + EIGEN_DEVICE_FUNC inline InverseReturnType transpose() const { return InverseReturnType(derived()); } /** \returns the adjoint of the permutation matrix. Its entries are real and it is orthogonal, so this equals * transpose() and inverse(). * * \note \blank \note_try_to_help_rvo */ - InverseReturnType adjoint() const { return InverseReturnType(derived()); } + EIGEN_DEVICE_FUNC InverseReturnType adjoint() const { return InverseReturnType(derived()); } /**** multiplication helpers to hopefully get RVO ****/ #ifndef EIGEN_PARSED_BY_DOXYGEN protected: template <typename OtherDerived> - void assignTranspose(const PermutationBase<OtherDerived>& other) { + EIGEN_DEVICE_FUNC void assignTranspose(const PermutationBase<OtherDerived>& other) { for (Index i = 0; i < rows(); ++i) indices().coeffRef(Index(other.indices().coeff(i))) = StorageIndex(i); } template <typename Lhs, typename Rhs> @@ -405,7 +405,7 @@ #ifndef EIGEN_PARSED_BY_DOXYGEN template <typename Other> - PermutationMatrix(const InverseImpl<Other, PermutationStorage>& other) + EIGEN_DEVICE_FUNC PermutationMatrix(const InverseImpl<Other, PermutationStorage>& other) : m_indices(other.derived().nestedExpression().size()) { internal::convert_index<StorageIndex>(m_indices.size()); // asserts that size fits StorageIndex Base::assignTranspose(other.derived().nestedExpression());
diff --git a/Eigen/src/Core/PlainObjectBase.h b/Eigen/src/Core/PlainObjectBase.h index bebc44f..1896bea 100644 --- a/Eigen/src/Core/PlainObjectBase.h +++ b/Eigen/src/Core/PlainObjectBase.h
@@ -934,7 +934,9 @@ template <typename Derived, typename OtherDerived, bool IsVector> struct conservative_resize_like_impl { - static constexpr bool IsRelocatable = std::is_trivially_copyable<typename Derived::Scalar>::value; + // Fixed-capacity storage only changes its dimensions; its coefficients never relocate. + static constexpr bool IsRelocatable = + Derived::MaxSizeAtCompileTime != Dynamic || std::is_trivially_copyable<typename Derived::Scalar>::value; #ifdef EIGEN_INITIALIZE_COEFFS // In-place reallocation leaves appended coefficients uninitialized, so under the initialization macros a resize that // appends coefficients without assigning them copies into a new, initialized object instead. @@ -1003,17 +1005,25 @@ } }; -// Here, the specialization for vectors inherits from the general matrix case -// to allow calling .conservativeResize(rows,cols) on vectors. +// Vectors preserve their linear layout when their length changes. template <typename Derived, typename OtherDerived> struct conservative_resize_like_impl<Derived, OtherDerived, true> : conservative_resize_like_impl<Derived, OtherDerived, false> { using Base = conservative_resize_like_impl<Derived, OtherDerived, false>; using Base::IsRelocatable; using Base::MustInitializeAppended; - using Base::run; + + static void run(DenseBase<Derived>& _this, Index rows, Index cols) { + EIGEN_STATIC_ASSERT_DYNAMIC_SIZE(Derived) + const Index size = Derived::RowsAtCompileTime == 1 ? cols : rows; + eigen_assert((Derived::RowsAtCompileTime == 1 ? rows == 1 : cols == 1) && + "Invalid sizes when resizing a vector."); + run(_this, size); + } static void run(DenseBase<Derived>& _this, Index size) { + eigen_assert(size >= 0 && "Invalid sizes when resizing a vector."); + if (_this.size() == size) return; const Index new_rows = Derived::RowsAtCompileTime == 1 ? 1 : size; const Index new_cols = Derived::RowsAtCompileTime == 1 ? size : 1; EIGEN_IF_CONSTEXPR (IsRelocatable) {
diff --git a/Eigen/src/Core/Product.h b/Eigen/src/Core/Product.h index 68ca72c..caa1f24 100644 --- a/Eigen/src/Core/Product.h +++ b/Eigen/src/Core/Product.h
@@ -46,11 +46,19 @@ // FIXME: only needed by GeneralMatrixMatrixTriangular InnerSize = min_size_prefer_fixed(LhsTraits::ColsAtCompileTime, RhsTraits::RowsAtCompileTime), - // The storage order is somewhat arbitrary here. The correct one will be determined through the evaluator. + // Products served by the coefficient-based evaluator are evaluated in the storage order reported below, which + // Reverse, Redux, Block, FindCoeff and others use to index their packets and coefficients. Row-major * row-major + // is row-major there, so that the rhs rows can be vectorized. A dense product of inner size 1 is served by it in + // LazyCoeffBasedProductMode, and materialized otherwise; unlike product_type, this test agrees between host and + // device code. + CoeffBasedEval = Option == LazyProduct || (std::is_same<StorageKind, Dense>::value && InnerSize == 1), + + // For other products the storage order is somewhat arbitrary: their evaluators materialize a PlainObject of it. Flags = (MaxRowsAtCompileTime == 1 && MaxColsAtCompileTime != 1) ? RowMajorBit : (MaxColsAtCompileTime == 1 && MaxRowsAtCompileTime != 1) ? 0 : (((LhsTraits::Flags & NoPreferredStorageOrderBit) && (RhsTraits::Flags & RowMajorBit)) || - ((RhsTraits::Flags & NoPreferredStorageOrderBit) && (LhsTraits::Flags & RowMajorBit))) + ((RhsTraits::Flags & NoPreferredStorageOrderBit) && (LhsTraits::Flags & RowMajorBit)) || + (CoeffBasedEval && (LhsTraits::Flags & RowMajorBit) && (RhsTraits::Flags & RowMajorBit))) ? RowMajorBit : NoPreferredStorageOrderBit };
diff --git a/Eigen/src/Core/ProductEvaluators.h b/Eigen/src/Core/ProductEvaluators.h index bd898a2..daf162d 100644 --- a/Eigen/src/Core/ProductEvaluators.h +++ b/Eigen/src/Core/ProductEvaluators.h
@@ -590,16 +590,15 @@ static constexpr bool DstRowMajor = bool(Dst::IsRowMajor); static constexpr bool DstContiguous = (int(DstEval::Flags) & DirectAccessBit) != 0 && int(Dst::InnerStrideAtCompileTime) == 1; - // Orientation of the packets the product evaluator natively produces. When a - // product can vectorize *both* the lhs (columns) and the rhs (rows), its - // packet() returns column packets (EvalToRowMajor == 0). The cascade axis must - // match this orientation -- and the destination's storage order must match it - // too, so the contiguous packet store lands on contiguous coefficients. + // The product's packets run along its storage order. The cascade axis must match + // it -- and so must the destination's storage order, so the contiguous packet + // store lands on contiguous coefficients. ProdEval::Vectorize is false for + // mixed-scalar products, which keep the generic (scalar) path. static constexpr bool ProdRowMajor = (int(ProdEval::Flags) & RowMajorBit) != 0; static constexpr bool VecLhs = // column-major product & dst: vectorize down each column - bool(ProdEval::CanVectorizeLhs) && !ProdRowMajor && !DstRowMajor && Rows != Dynamic; + bool(ProdEval::Vectorize) && !ProdRowMajor && !DstRowMajor && Rows != Dynamic; static constexpr bool VecRhs = // row-major product & dst: vectorize along each row - bool(ProdEval::CanVectorizeRhs) && ProdRowMajor && DstRowMajor && Cols != Dynamic; + bool(ProdEval::Vectorize) && ProdRowMajor && DstRowMajor && Cols != Dynamic; static constexpr int Extent = VecLhs ? Rows : VecRhs ? Cols : 1; using Packet = typename find_largest_packet<Scalar, Extent>::type; static constexpr int LargestSize = unpacket_traits<Packet>::size; @@ -608,12 +607,8 @@ bool(EIGEN_UNALIGNED_VECTORIZE) && (int(DstEval::Flags) & int(ProdEval::Flags) & int(ActualPacketAccessBit)) != 0 && bool(functor_traits<Func>::PacketAccess); - // SameType mirrors the product's own PacketAccessBit (SameType && (CanVectorizeLhs || - // CanVectorizeRhs)): CanVectorizeLhs/Rhs alone test only the operand's packet access, so a - // mixed-scalar product (e.g. double * complex) would otherwise enable the cascade and try to - // read the wrong-width packet out of an operand. Mixed types keep the generic (scalar) path. - static constexpr bool Enable = bool(ProdEval::SameType) && (VecLhs || VecRhs) && DstContiguous && MightVectorize && - (LargestSize <= Extent) && (LargestSize > GenericSize); + static constexpr bool Enable = + (VecLhs || VecRhs) && DstContiguous && MightVectorize && (LargestSize <= Extent) && (LargestSize > GenericSize); }; // Run the cascade over every outer index of dst. Traits::VecLhs is a compile-time @@ -635,10 +630,17 @@ // a plain `if`) so the cascade is only *instantiated* for the vectorizable // fixed-size case -- a runtime branch would instantiate it for every product, // including scalar types with no packet support. +template <typename Lhs, typename Rhs, bool EvalToRowMajor> +struct coeff_based_product_evaluator; + +// The coefficient-based evaluator of lhs * rhs with packets along dst's storage order. +template <typename Lhs, typename Rhs, typename Dst> +using dst_ordered_product_evaluator = coeff_based_product_evaluator<Lhs, Rhs, bool(Dst::IsRowMajor)>; + template <typename Func, typename Dst, typename Lhs, typename Rhs> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void product_packet_assign(std::true_type, const Func& func, Dst& dst, const Lhs& lhs, const Rhs& rhs) { - using ProdEval = product_evaluator<Product<Lhs, Rhs, LazyProduct>, CoeffBasedProductMode, DenseShape, DenseShape>; + using ProdEval = dst_ordered_product_evaluator<Lhs, Rhs, Dst>; using Traits = product_packet_cascade_traits<ProdEval, Dst, Func>; const ProdEval prodEval(lhs, rhs); product_run_packet_cascade<Traits>(func, dst, prodEval); @@ -669,7 +671,7 @@ template <typename Dst> static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void addTo(Dst& dst, const Lhs& lhs, const Rhs& rhs) { // dst.noalias() += lhs.lazyProduct(rhs); - using ProdEval = product_evaluator<Product<Lhs, Rhs, LazyProduct>, CoeffBasedProductMode, DenseShape, DenseShape>; + using ProdEval = dst_ordered_product_evaluator<Lhs, Rhs, Dst>; using Func = internal::add_assign_op<typename Dst::Scalar, Scalar>; using Traits = product_packet_cascade_traits<ProdEval, Dst, Func>; product_packet_assign(bool_constant<Traits::Enable>(), Func(), dst, lhs, rhs); @@ -678,7 +680,7 @@ template <typename Dst> static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void subTo(Dst& dst, const Lhs& lhs, const Rhs& rhs) { // dst.noalias() -= lhs.lazyProduct(rhs); - using ProdEval = product_evaluator<Product<Lhs, Rhs, LazyProduct>, CoeffBasedProductMode, DenseShape, DenseShape>; + using ProdEval = dst_ordered_product_evaluator<Lhs, Rhs, Dst>; using Func = internal::sub_assign_op<typename Dst::Scalar, Scalar>; using Traits = product_packet_cascade_traits<ProdEval, Dst, Func>; product_packet_assign(bool_constant<Traits::Enable>(), Func(), dst, lhs, rhs); @@ -746,19 +748,19 @@ EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE explicit product_empty_packet_evaluator(const Xpr&) {} }; -template <typename Lhs, typename Rhs, int ProductTag> -struct product_evaluator<Product<Lhs, Rhs, LazyProduct>, ProductTag, DenseShape, DenseShape> - : evaluator_base<Product<Lhs, Rhs, LazyProduct>> { +// Coefficient-based evaluator of lhs * rhs whose packets run along rows if EvalToRowMajor_, else along columns. +template <typename Lhs, typename Rhs, bool EvalToRowMajor_> +struct coeff_based_product_evaluator : evaluator_base<Product<Lhs, Rhs, LazyProduct>> { using XprType = Product<Lhs, Rhs, LazyProduct>; using Scalar = typename XprType::Scalar; using CoeffReturnType = typename XprType::CoeffReturnType; - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE explicit product_evaluator(const XprType& xpr) - : product_evaluator(xpr.lhs(), xpr.rhs()) {} + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE explicit coeff_based_product_evaluator(const XprType& xpr) + : coeff_based_product_evaluator(xpr.lhs(), xpr.rhs()) {} // Also construct directly from the product operands so packet-cascade callers // do not have to materialize a temporary Product expression first. - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE product_evaluator(const Lhs& lhs, const Rhs& rhs) + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE coeff_based_product_evaluator(const Lhs& lhs, const Rhs& rhs) : m_lhs(lhs), m_rhs(rhs), m_lhsImpl(m_lhs), // Real evaluator objects for packet products; empty placeholders for scalar-only products. @@ -826,15 +828,13 @@ static constexpr bool CanVectorizeRhs = bool(RhsRowMajor) && (RhsFlags & PacketAccessBit) && (ColsAtCompileTime != 1); static constexpr bool CanVectorizeLhs = (!LhsRowMajor) && (LhsFlags & PacketAccessBit) && (RowsAtCompileTime != 1); - static constexpr int EvalToRowMajor = (MaxRowsAtCompileTime == 1 && MaxColsAtCompileTime != 1) ? 1 - : (MaxColsAtCompileTime == 1 && MaxRowsAtCompileTime != 1) - ? 0 - : (bool(RhsRowMajor) && !CanVectorizeLhs); + // Only the operand that is contiguous along the packets' axis can vectorize the product. + static constexpr bool EvalToRowMajor = EvalToRowMajor_; + // TODO: enable vectorization for mixed types + static constexpr bool Vectorize = SameType && (EvalToRowMajor ? CanVectorizeRhs : CanVectorizeLhs); static constexpr int Flags = ((int(LhsFlags) | int(RhsFlags)) & HereditaryBits & ~RowMajorBit) | - (EvalToRowMajor ? RowMajorBit : 0) - // TODO: enable vectorization for mixed types - | (SameType && (CanVectorizeLhs || CanVectorizeRhs) ? PacketAccessBit : 0) | + (EvalToRowMajor ? RowMajorBit : 0) | (Vectorize ? PacketAccessBit : 0) | (XprType::IsVectorAtCompileTime ? LinearAccessBit : 0); using LhsPacketEtorType = std::conditional_t<bool(int(Flags) & PacketAccessBit), LhsEtorType, product_empty_packet_evaluator>; @@ -849,15 +849,12 @@ static constexpr int RhsOuterStrideBytes = int(RhsNestedCleaned::OuterStrideAtCompileTime) * int(sizeof(typename RhsNestedCleaned::Scalar)); + // Packets come from the operand that vectorizes the product, at its outer index k. + static constexpr int VecOuterStrideBytes = EvalToRowMajor ? RhsOuterStrideBytes : LhsOuterStrideBytes; + static constexpr int VecAlignment = EvalToRowMajor ? RhsAlignment : LhsAlignment; static constexpr int Alignment = - bool(CanVectorizeLhs) - ? (LhsOuterStrideBytes <= 0 || (int(LhsOuterStrideBytes) % plain_enum_max(1, LhsAlignment)) != 0 - ? 0 - : LhsAlignment) - : bool(CanVectorizeRhs) - ? (RhsOuterStrideBytes <= 0 || (int(RhsOuterStrideBytes) % plain_enum_max(1, RhsAlignment)) != 0 - ? 0 - : RhsAlignment) + (Vectorize && VecOuterStrideBytes > 0 && VecOuterStrideBytes % plain_enum_max(1, VecAlignment) == 0) + ? VecAlignment : 0; EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE const CoeffReturnType coeff(Index row, Index col) const { @@ -932,12 +929,39 @@ variable_if_dynamic<Index, InnerSize> m_innerDim; }; +// Evaluate in the storage order the expression reports, which consumers such as Reverse, Redux, Block and FindCoeff +// use to index packets and size their loops. +template <typename Lhs, typename Rhs, int ProductTag> +struct product_evaluator<Product<Lhs, Rhs, LazyProduct>, ProductTag, DenseShape, DenseShape> + : coeff_based_product_evaluator<Lhs, Rhs, (int(traits<Product<Lhs, Rhs, LazyProduct>>::Flags) & RowMajorBit) != 0> { + using XprType = Product<Lhs, Rhs, LazyProduct>; + using Base = + coeff_based_product_evaluator<Lhs, Rhs, (int(traits<Product<Lhs, Rhs, LazyProduct>>::Flags) & RowMajorBit) != 0>; + + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE explicit product_evaluator(const XprType& xpr) : Base(xpr) {} + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE product_evaluator(const Lhs& lhs, const Rhs& rhs) : Base(lhs, rhs) {} +}; + +// A dense assignment reads its source only through the evaluator, so it may evaluate a coefficient-based product +// along the destination's storage order, which the destination's packets need. It does so only where that order +// vectorizes; otherwise the assignment is scalar in either order, and the expression's evaluator is kept. +template <typename DstEvaluatorType, typename Lhs, typename Rhs> +struct dense_assignment_source_evaluator< + DstEvaluatorType, Product<Lhs, Rhs, LazyProduct>, + std::enable_if_t<std::is_same<typename evaluator_traits<Lhs>::Shape, DenseShape>::value && + std::is_same<typename evaluator_traits<Rhs>::Shape, DenseShape>::value>> { + using DstOrdered = coeff_based_product_evaluator<Lhs, Rhs, (int(DstEvaluatorType::Flags) & RowMajorBit) != 0>; + using type = std::conditional_t<DstOrdered::Vectorize, DstOrdered, evaluator<Product<Lhs, Rhs, LazyProduct>>>; +}; + template <typename Lhs, typename Rhs> struct product_evaluator<Product<Lhs, Rhs, DefaultProduct>, LazyCoeffBasedProductMode, DenseShape, DenseShape> : product_evaluator<Product<Lhs, Rhs, LazyProduct>, CoeffBasedProductMode, DenseShape, DenseShape> { using XprType = Product<Lhs, Rhs, DefaultProduct>; using BaseProduct = Product<Lhs, Rhs, LazyProduct>; using Base = product_evaluator<BaseProduct, CoeffBasedProductMode, DenseShape, DenseShape>; + static_assert((int(traits<XprType>::Flags) & RowMajorBit) == (int(Base::Flags) & RowMajorBit), + "the evaluator must use the storage order its expression reports"); enum { Flags = Base::Flags | EvalBeforeNestingBit }; EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE explicit product_evaluator(const XprType& xpr) : Base(BaseProduct(xpr.lhs(), xpr.rhs())) {} @@ -1561,7 +1585,7 @@ template <typename MatrixType, typename DiagonalType, typename Derived, int ProductOrder> struct diagonal_product_evaluator_base : evaluator_base<Derived> { - using Scalar = typename ScalarBinaryOpTraits<typename MatrixType::Scalar, typename DiagonalType::Scalar>::ReturnType; + using Scalar = typename Derived::Scalar; public: enum { @@ -1606,13 +1630,27 @@ } EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE const Scalar coeff(Index idx) const { + using DiagonalFirst = bool_constant< + ProductOrder == OnTheLeft || + functor_is_commutative<scalar_product_op<typename MatrixType::Scalar, typename DiagonalType::Scalar>>::value>; + return coeff_impl(idx, DiagonalFirst()); + } + + protected: + EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE Scalar coeff_impl(Index idx, std::true_type) const { EIGEN_IF_CONSTEXPR (AsScalarProduct) return m_diagImpl.coeff(0) * m_matImpl.coeff(idx); else return m_diagImpl.coeff(idx) * m_matImpl.coeff(idx); } - protected: + EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE Scalar coeff_impl(Index idx, std::false_type) const { + EIGEN_IF_CONSTEXPR (AsScalarProduct) + return m_matImpl.coeff(idx) * m_diagImpl.coeff(0); + else + return m_matImpl.coeff(idx) * m_diagImpl.coeff(idx); + } + template <int LoadMode, typename PacketType> EIGEN_STRONG_INLINE PacketType packet_impl(Index row, Index col, Index id, std::true_type) const { return internal::pmul(m_matImpl.template packet<LoadMode, PacketType>(row, col), @@ -1746,6 +1784,20 @@ #endif }; +template <typename MatrixScalar, typename DiagonalScalar> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE auto diagonal_product_coeff(const MatrixScalar& matrix, + const DiagonalScalar& diagonal, true_type) + -> decltype(diagonal * matrix) { + return diagonal * matrix; +} + +template <typename MatrixScalar, typename DiagonalScalar> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE auto diagonal_product_coeff(const MatrixScalar& matrix, + const DiagonalScalar& diagonal, false_type) + -> decltype(matrix * diagonal) { + return matrix * diagonal; +} + // Lazy evaluators for triangular/selfadjoint × diagonal products. // // The triangular-assignment dispatcher in TriangularMatrix.h routes @@ -1757,9 +1809,10 @@ // triangle one coefficient at a time, so a coeff()-only evaluator avoids the // temporary at no loss in functionality. -template <int Mode, int ProductOrder, typename MatrixType, typename DiagonalType, typename Derived> -struct triangular_diagonal_product_lazy_evaluator_base : evaluator_base<Derived> { - using Scalar = typename ScalarBinaryOpTraits<typename MatrixType::Scalar, typename DiagonalType::Scalar>::ReturnType; +template <int Mode, int ProductOrder, typename MatrixType, typename DiagonalType, typename ProductXpr> +struct triangular_diagonal_product_lazy_evaluator_base : evaluator_base<ProductXpr> { + using Scalar = typename ProductXpr::Scalar; + using DiagonalOnLeft = bool_constant<ProductOrder == OnTheLeft>; using MatrixScalar = typename MatrixType::Scalar; enum { @@ -1777,13 +1830,12 @@ if (!inActive) return Scalar(0); if (row == col) { EIGEN_IF_CONSTEXPR ((Mode & UnitDiag) == UnitDiag) { - return ProductOrder == OnTheLeft ? Scalar(m_diagImpl.coeff(row) * MatrixScalar(1)) - : Scalar(MatrixScalar(1) * m_diagImpl.coeff(col)); + return Scalar(diagonal_product_coeff(MatrixScalar(1), m_diagImpl.coeff(row), DiagonalOnLeft())); } EIGEN_IF_CONSTEXPR ((Mode & ZeroDiag) == ZeroDiag) return Scalar(0); } - return ProductOrder == OnTheLeft ? Scalar(m_diagImpl.coeff(row) * m_matImpl.coeff(row, col)) - : Scalar(m_matImpl.coeff(row, col) * m_diagImpl.coeff(col)); + return Scalar(diagonal_product_coeff(m_matImpl.coeff(row, col), + m_diagImpl.coeff(ProductOrder == OnTheLeft ? row : col), DiagonalOnLeft())); } protected: @@ -1794,13 +1846,12 @@ // Triangular × Diagonal template <typename Lhs, typename Rhs, int ProductKind, int ProductTag> struct product_evaluator<Product<Lhs, Rhs, ProductKind>, ProductTag, TriangularShape, DiagonalShape> - : triangular_diagonal_product_lazy_evaluator_base< - Lhs::Mode, OnTheRight, typename Lhs::MatrixType, typename Rhs::DiagonalVectorType, - product_evaluator<Product<Lhs, Rhs, ProductKind>, ProductTag, TriangularShape, DiagonalShape>> { + : triangular_diagonal_product_lazy_evaluator_base<Lhs::Mode, OnTheRight, typename Lhs::MatrixType, + typename Rhs::DiagonalVectorType, + Product<Lhs, Rhs, ProductKind>> { using XprType = Product<Lhs, Rhs, ProductKind>; - using Base = triangular_diagonal_product_lazy_evaluator_base< - Lhs::Mode, OnTheRight, typename Lhs::MatrixType, typename Rhs::DiagonalVectorType, - product_evaluator<XprType, ProductTag, TriangularShape, DiagonalShape>>; + using Base = triangular_diagonal_product_lazy_evaluator_base<Lhs::Mode, OnTheRight, typename Lhs::MatrixType, + typename Rhs::DiagonalVectorType, XprType>; EIGEN_DEVICE_FUNC explicit product_evaluator(const XprType& xpr) : Base(xpr.lhs().nestedExpression(), xpr.rhs().diagonal()) {} @@ -1809,13 +1860,12 @@ // Diagonal × Triangular template <typename Lhs, typename Rhs, int ProductKind, int ProductTag> struct product_evaluator<Product<Lhs, Rhs, ProductKind>, ProductTag, DiagonalShape, TriangularShape> - : triangular_diagonal_product_lazy_evaluator_base< - Rhs::Mode, OnTheLeft, typename Rhs::MatrixType, typename Lhs::DiagonalVectorType, - product_evaluator<Product<Lhs, Rhs, ProductKind>, ProductTag, DiagonalShape, TriangularShape>> { + : triangular_diagonal_product_lazy_evaluator_base<Rhs::Mode, OnTheLeft, typename Rhs::MatrixType, + typename Lhs::DiagonalVectorType, + Product<Lhs, Rhs, ProductKind>> { using XprType = Product<Lhs, Rhs, ProductKind>; - using Base = triangular_diagonal_product_lazy_evaluator_base< - Rhs::Mode, OnTheLeft, typename Rhs::MatrixType, typename Lhs::DiagonalVectorType, - product_evaluator<XprType, ProductTag, DiagonalShape, TriangularShape>>; + using Base = triangular_diagonal_product_lazy_evaluator_base<Rhs::Mode, OnTheLeft, typename Rhs::MatrixType, + typename Lhs::DiagonalVectorType, XprType>; EIGEN_DEVICE_FUNC explicit product_evaluator(const XprType& xpr) : Base(xpr.rhs().nestedExpression(), xpr.lhs().diagonal()) {} @@ -1827,7 +1877,9 @@ bool Materialize = NumTraits<typename MatrixType::Scalar>::IsComplex && blas_traits<MatrixType>::HasScalarFactor> struct selfadjoint_diagonal_product_lazy_evaluator_base : evaluator_base<ProductXpr> { - using Scalar = typename ScalarBinaryOpTraits<typename MatrixType::Scalar, typename DiagonalType::Scalar>::ReturnType; + using Scalar = typename ProductXpr::Scalar; + using MatrixScalar = typename MatrixType::Scalar; + using DiagonalOnLeft = bool_constant<ProductOrder == OnTheLeft>; enum { CoeffReadCost = int(NumTraits<Scalar>::MulCost) + int(evaluator<MatrixType>::CoeffReadCost) + @@ -1842,10 +1894,10 @@ EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Scalar coeff(Index row, Index col) const { const bool storedHere = ((Mode & Upper) == Upper) ? (row <= col) : (row >= col); - const Scalar matCoeff = - storedHere ? Scalar(m_matImpl.coeff(row, col)) : Scalar(numext::conj(m_matImpl.coeff(col, row))); - return ProductOrder == OnTheLeft ? Scalar(m_diagImpl.coeff(row) * matCoeff) - : Scalar(matCoeff * m_diagImpl.coeff(col)); + const MatrixScalar matCoeff = + storedHere ? MatrixScalar(m_matImpl.coeff(row, col)) : MatrixScalar(numext::conj(m_matImpl.coeff(col, row))); + return Scalar( + diagonal_product_coeff(matCoeff, m_diagImpl.coeff(ProductOrder == OnTheLeft ? row : col), DiagonalOnLeft())); } protected:
diff --git a/Eigen/src/Core/RankRevealingBase.h b/Eigen/src/Core/RankRevealingBase.h index 45d1393..4e4e128 100644 --- a/Eigen/src/Core/RankRevealingBase.h +++ b/Eigen/src/Core/RankRevealingBase.h
@@ -93,9 +93,8 @@ * setThreshold(const RealScalar&). */ inline Index rank() const { - using std::abs; eigen_assert(self().m_isInitialized && "Decomposition is not initialized."); - RealScalar premultiplied_threshold = abs(m_maxpivot) * threshold(); + RealScalar premultiplied_threshold = numext::abs(m_maxpivot) * threshold(); Index result = 0; for (Index i = 0; i < m_nonzero_pivots; ++i) result += (self().pivotCoeff(i) > premultiplied_threshold); return result;
diff --git a/Eigen/src/Core/Redux.h b/Eigen/src/Core/Redux.h index 524dd92..b440125 100644 --- a/Eigen/src/Core/Redux.h +++ b/Eigen/src/Core/Redux.h
@@ -105,7 +105,10 @@ }; public: - enum { Unrolling = Cost <= UnrollingLimit ? CompleteUnrolling : NoUnrolling }; + // UnrollingLimit grows with the packet size and by default reaches HugeCost from 91 scalars per packet on, so the + // Cost of a runtime size alone does not rule out unrolling it. + static constexpr int Unrolling = + (Evaluator::SizeAtCompileTime != Dynamic && Cost <= UnrollingLimit) ? CompleteUnrolling : NoUnrolling; #ifdef EIGEN_DEBUG_ASSIGN static void debug() { @@ -451,7 +454,7 @@ ? int(packetAlignment) : int(Unaligned); constexpr int alignment = plain_enum_max(alignment0, Evaluator::Alignment); - const Index alignedStart = internal::first_default_aligned(xpr); + const Index alignedStart = internal::first_aligned<alignment>(xpr); const Index alignedSize4 = numext::round_down(size - alignedStart, 4 * packetSize); const Index alignedSize = numext::round_down(size - alignedStart, packetSize); const Index alignedEnd4 = alignedStart + alignedSize4; @@ -741,7 +744,7 @@ }; // Convert identity to Scalar only for empty inputs; otherwise reduce without inserting it. -// Tag dispatch avoids instantiating redux for statically empty inputs in C++14. +// Tag dispatch avoids instantiating redux for statically empty inputs or converting an unused identity in C++14. // TODO(C++17): replace tag dispatch with if constexpr. template <typename Derived, typename Func, typename Identity> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE typename Derived::Scalar redux_with_identity_impl(const DenseBase<Derived>&, @@ -751,14 +754,23 @@ return typename Derived::Scalar(std::forward<Identity>(identity)); } -template <typename Derived, typename Func, typename Identity> +// Compare against 0 rather than Dynamic: non-inline constexpr Dynamic has internal linkage in C++14, +// which keeps Clang header modules from merging template redeclarations across textual inclusions. +template <typename Derived, typename Func, typename Identity, + std::enable_if_t<(Derived::SizeAtCompileTime < 0), int> = 0> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE typename Derived::Scalar redux_with_identity_impl(const DenseBase<Derived>& xpr, const Func& func, Identity&& identity, false_type) { - EIGEN_IF_CONSTEXPR (Derived::SizeAtCompileTime == Dynamic) { - if (xpr.size() == 0) return typename Derived::Scalar(std::forward<Identity>(identity)); - } + if (xpr.size() == 0) return typename Derived::Scalar(std::forward<Identity>(identity)); + return xpr.derived().redux(func); +} + +template <typename Derived, typename Func, typename Identity, + std::enable_if_t<(Derived::SizeAtCompileTime >= 0), int> = 0> +EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE typename Derived::Scalar redux_with_identity_impl(const DenseBase<Derived>& xpr, + const Func& func, Identity&&, + false_type) { return xpr.derived().redux(func); }
diff --git a/Eigen/src/Core/SafeScaling.h b/Eigen/src/Core/SafeScaling.h index ddc3b83..7155cd5 100644 --- a/Eigen/src/Core/SafeScaling.h +++ b/Eigen/src/Core/SafeScaling.h
@@ -692,8 +692,7 @@ if (exponent < kMinScaleExponent) exponent = kMinScaleExponent; if (exponent > kMaxScaleExponent) exponent = kMaxScaleExponent; - EIGEN_USING_STD(ldexp); - const long double scale = ldexp(1.0L, exponent); + const long double scale = numext::ldexp(1.0L, exponent); return {scale, 1.0L / scale}; } };
diff --git a/Eigen/src/Core/SelfAdjointView.h b/Eigen/src/Core/SelfAdjointView.h index 18850c3..b1bcf57 100644 --- a/Eigen/src/Core/SelfAdjointView.h +++ b/Eigen/src/Core/SelfAdjointView.h
@@ -34,6 +34,42 @@ namespace internal { +// Requires 0 <= i < j < matrix.rows(). The row prefixes may contain already-computed factors. +template <typename MatrixType> +EIGEN_DEVICE_FUNC void selfadjoint_swap_lower(MatrixType& matrix, Index i, Index j) { + using Scalar = typename MatrixType::Scalar; + const Index size = matrix.rows(); + eigen_internal_assert(matrix.cols() == size && i >= 0 && i < j && j < size); + matrix.template block<1, Dynamic>(i, 0, 1, i).swap(matrix.template block<1, Dynamic>(j, 0, 1, i)); + for (Index k = i + 1; k < j; ++k) { + const Scalar value = matrix.coeff(k, i); + matrix.coeffRef(k, i) = numext::conj(matrix.coeff(j, k)); + matrix.coeffRef(j, k) = numext::conj(value); + } + matrix.template block<Dynamic, 1>(j + 1, i, size - j - 1, 1) + .swap(matrix.template block<Dynamic, 1>(j + 1, j, size - j - 1, 1)); + EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::IsComplex) { + matrix.coeffRef(j, i) = numext::conj(matrix.coeff(j, i)); + } + numext::swap(matrix.coeffRef(i, i), matrix.coeffRef(j, j)); +} + +// Consumes an owning copy of the permutation, disjoint from matrix. +template <typename MatrixType, typename PermutationType> +EIGEN_DEVICE_FUNC void selfadjoint_apply_permutation_lower(MatrixType& matrix, PermutationType& remaining) { + const Index size = matrix.rows(); + for (Index i = 0; i < size; ++i) { + while (Index(remaining.indices().coeff(i)) != i) { + const Index j = convert_index<Index>(remaining.indices().coeff(i)); + // Earlier indices are fixed, so j > i. Each swap removes one edge from the permutation cycle. + // If P(j) == j as well, index j repeats and this loop would not terminate. + eigen_assert(i < j && j < size && Index(remaining.indices().coeff(j)) != j && "indices are not a permutation"); + selfadjoint_swap_lower(matrix, i, j); + numext::swap(remaining.indices().coeffRef(i), remaining.indices().coeffRef(j)); + } + } +} + template <typename MatrixType, unsigned int UpLo> struct traits<SelfAdjointView<MatrixType, UpLo>> : traits<MatrixType> { using MatrixTypeNested = typename ref_selector<MatrixType>::non_const_type; @@ -130,6 +166,69 @@ return *this; } + /** Replaces this view's matrix \f$ S \f$ by \f$ P S P^T \f$ in place, where \a permutation represents \f$ P \f$. + * Only the referenced triangle is read or written; the opposite triangle is unchanged. Complex coefficients + * crossing the diagonal are conjugated. The matrix must be square and the permutation must have matching size. + * The operation takes \f$ O(n^2) \f$ time and \f$ O(n) \f$ auxiliary storage for a copy of the permutation. + * + * \returns a reference to \c *this, retaining its self-adjoint structure. + * + * \code + * A.selfadjointView<Lower>().applyPermutation(P); + * \endcode + */ + template <typename PermutationDerived> + EIGEN_DEVICE_FUNC SelfAdjointView& applyPermutation(const PermutationBase<PermutationDerived>& permutation) { + typename PermutationDerived::PlainPermutationType remaining(permutation); + return applyPermutationInPlace(remaining); + } + + /** Applies an inverse (or transposed) permutation in place. \sa applyPermutation() */ + template <typename PermutationDerived> + EIGEN_DEVICE_FUNC SelfAdjointView& applyPermutation( + const InverseImpl<PermutationDerived, PermutationStorage>& permutation) { + typename PermutationDerived::PlainPermutationType remaining(permutation); + return applyPermutationInPlace(remaining); + } + + /** Applies the permutation represented by a sequence of transpositions in place. + * The indices are copied before any matrix writes, including when they alias the matrix. + * The operation takes \f$ O(n^2) \f$ time and \f$ O(n) \f$ auxiliary storage. + * \sa applyPermutation(), applyTransposition() + */ + template <typename TranspositionsDerived> + EIGEN_DEVICE_FUNC SelfAdjointView& applyPermutation(const TranspositionsBase<TranspositionsDerived>& transpositions) { + return applyTranspositions<false>(transpositions); + } + + /** Applies an inverse (or transposed) sequence of transpositions in place. \sa applyPermutation() */ + template <typename TranspositionsDerived> + EIGEN_DEVICE_FUNC SelfAdjointView& applyPermutation( + const Transpose<TranspositionsBase<TranspositionsDerived>>& transpositions) { + return applyTranspositions<true>(transpositions.nestedExpression()); + } + + /** Interchanges rows and columns \a i and \a j of this self-adjoint view in place. + * Only the referenced triangle is read or written; complex coefficients crossing the diagonal are conjugated. + * The matrix must be square and both indices must lie in [0, rows()). Equal indices leave the view unchanged. + * The operation takes \f$ O(n) \f$ time and \f$ O(1) \f$ auxiliary storage, without heap allocation. + * \returns a reference to \c *this. + * \sa applyPermutation() + */ + EIGEN_DEVICE_FUNC SelfAdjointView& applyTransposition(Index i, Index j) { + EIGEN_STATIC_ASSERT_LVALUE(MatrixType); + eigen_assert(this->rows() == this->cols() && i >= 0 && j >= 0 && i < this->rows() && j < this->rows()); + if (i == j) return *this; + if (i > j) numext::swap(i, j); + EIGEN_IF_CONSTEXPR (UpLo == Lower) { + internal::selfadjoint_swap_lower(m_matrix, i, j); + } else { + auto transposed = m_matrix.transpose(); + internal::selfadjoint_swap_lower(transposed, i, j); + } + return *this; + } + /** \internal */ EIGEN_DEVICE_FUNC constexpr const MatrixTypeNestedCleaned& _expression() const noexcept { return m_matrix; } @@ -234,6 +333,34 @@ EIGEN_DEVICE_FUNC EigenvaluesReturnType eigenvalues() const; EIGEN_DEVICE_FUNC RealScalar operatorNorm() const; + private: + template <typename PermutationType> + EIGEN_DEVICE_FUNC SelfAdjointView& applyPermutationInPlace(PermutationType& remaining) { + EIGEN_STATIC_ASSERT_LVALUE(MatrixType); + eigen_assert(this->rows() == this->cols() && this->rows() == remaining.size()); + EIGEN_IF_CONSTEXPR (UpLo == Lower) { + internal::selfadjoint_apply_permutation_lower(m_matrix, remaining); + } else { + auto transposed = m_matrix.transpose(); + internal::selfadjoint_apply_permutation_lower(transposed, remaining); + } + return *this; + } + + template <bool Reverse, typename TranspositionsDerived> + EIGEN_DEVICE_FUNC SelfAdjointView& applyTranspositions( + const TranspositionsBase<TranspositionsDerived>& transpositions) { + EIGEN_STATIC_ASSERT_LVALUE(MatrixType); + eigen_assert(this->rows() == this->cols() && this->rows() == transpositions.size()); + const typename TranspositionsDerived::IndicesType::PlainObject indices(transpositions.indices()); + const Index size = this->rows(); + for (Index k = 0; k < size; ++k) { + const Index i = Reverse ? size - 1 - k : k; + applyTransposition(i, internal::convert_index<Index>(indices.coeff(i))); + } + return *this; + } + protected: MatrixTypeNested m_matrix; }; @@ -252,6 +379,23 @@ using Shape = SelfAdjointShape; }; +// GCC PR target/124138 (https://gcc.gnu.org/PR124138): since GCC 4.9, avx_vpermilp_parallel accepted selectors 4-7 +// for elements 2 and 3 of a V8DF permutation, emitting the lane-crossing {0, 1, 2, 6, 4, 5, 6, 7} as vpermilpd, which +// yields {0, 1, 2, 2, 4, 5, 6, 7}. SLP forms that permutation from the unrolled mirrored stores of a 4x4 double copy, +// at -O2 since GCC 12. Hiding the copied value from the vectorizer avoids it; the loop used for larger sizes is +// unaffected. Fixed in GCC 14.4, 15.3 and 16, and on the GCC 13 branch after 13.4. +template <bool Unrolled, typename Scalar> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void selfadjoint_mirror_barrier(Scalar&) {} +#if EIGEN_COMP_GNUC_STRICT && defined(__AVX512F__) && !defined(EIGEN_GPU_COMPILE_PHASE) && \ + (EIGEN_GNUC_STRICT_LESS_THAN(13, 5, 0) || \ + (EIGEN_GNUC_STRICT_AT_LEAST(14, 0, 0) && EIGEN_GNUC_STRICT_LESS_THAN(14, 4, 0)) || \ + (EIGEN_GNUC_STRICT_AT_LEAST(15, 0, 0) && EIGEN_GNUC_STRICT_LESS_THAN(15, 3, 0))) +template <> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void selfadjoint_mirror_barrier<true, double>(double& value) { + EIGEN_OPTIMIZATION_BARRIER(value); +} +#endif + template <int UpLo, int SetOpposite, typename DstEvaluatorTypeT, typename SrcEvaluatorTypeT, typename Functor, int Version> class triangular_dense_assignment_kernel<UpLo, SelfAdjoint, SetOpposite, DstEvaluatorTypeT, SrcEvaluatorTypeT, Functor, @@ -278,6 +422,7 @@ EIGEN_DEVICE_FUNC void assignCoeff(Index row, Index col) { eigen_internal_assert(row != col); Scalar tmp = m_src.coeff(row, col); + selfadjoint_mirror_barrier<triangular_assignment_unrolls<DstEvaluatorType, SrcEvaluatorType>::value>(tmp); m_functor.assignCoeff(m_dst.coeffRef(row, col), tmp); m_functor.assignCoeff(m_dst.coeffRef(col, row), numext::conj(tmp)); }
diff --git a/Eigen/src/Core/StableNorm.h b/Eigen/src/Core/StableNorm.h index 654d78c..dea0ab2 100644 --- a/Eigen/src/Core/StableNorm.h +++ b/Eigen/src/Core/StableNorm.h
@@ -157,8 +157,6 @@ template <typename VectorType, std::enable_if_t<VectorType::IsVectorAtCompileTime && VectorType::MaxSizeAtCompileTime != 0, int> = 0> typename VectorType::RealScalar stable_norm_impl(const VectorType& vec) { - using std::sqrt; - Index n = vec.size(); if (EIGEN_PREDICT_FALSE(n == 1)) return numext::abs(vec.coeff(0)); @@ -170,14 +168,12 @@ stable_norm_vector_dispatch<VectorType, Accumulator>::run(vec, ssq, scale, invScale); - return RealScalar(scale * sqrt(ssq)); + return RealScalar(scale * numext::sqrt(ssq)); } template <typename MatrixType, std::enable_if_t<!MatrixType::IsVectorAtCompileTime && MatrixType::MaxSizeAtCompileTime != 0, int> = 0> typename MatrixType::RealScalar stable_norm_impl(const MatrixType& mat) { - using std::sqrt; - using RealScalar = typename MatrixType::RealScalar; using Accumulator = typename stable_norm_accumulator<RealScalar>::type; Accumulator scale(0); @@ -185,7 +181,7 @@ Accumulator ssq(0); // sum of squares stable_norm_matrix_dispatch<MatrixType, Accumulator>::run(mat, ssq, scale, invScale); - return RealScalar(scale * sqrt(ssq)); + return RealScalar(scale * numext::sqrt(ssq)); } inline int stable_norm_floor_div2(int value) { return value / 2 - ((value < 0 && value % 2 != 0) ? 1 : 0); } @@ -234,7 +230,6 @@ using Accumulator = typename stable_norm_accumulator<RealScalar>::type; using Scalar = typename traits<Derived>::Scalar; using std::pow; - using std::sqrt; const Derived& vec(_vec.derived()); if (vec.size() == 0) return RealScalar(0); @@ -276,8 +271,8 @@ sumsq = abig; } else if (asml > Accumulator(0)) { if (amed > Accumulator(0) || amed > NumTraits<Accumulator>::highest() || amed != amed) { - amed = sqrt(amed); - asml = sqrt(asml) / ssml; + amed = numext::sqrt(amed); + asml = numext::sqrt(asml) / ssml; // Spell this as in xNRM2 rather than with min/max: when amed is NaN, // it must become ymax so that the final result remains NaN. const bool smallIsLarger = asml > amed; @@ -291,7 +286,7 @@ } else { sumsq = amed; } - return RealScalar(scale * sqrt(sumsq)); + return RealScalar(scale * numext::sqrt(sumsq)); } } // end namespace internal
diff --git a/Eigen/src/Core/StlIterators.h b/Eigen/src/Core/StlIterators.h index e8179a0..621ad98 100644 --- a/Eigen/src/Core/StlIterators.h +++ b/Eigen/src/Core/StlIterators.h
@@ -315,7 +315,8 @@ template <typename XprType> class pointer_based_stl_iterator { - enum { is_lvalue = internal::is_lvalue<XprType>::value }; + static constexpr bool is_lvalue = internal::is_lvalue<XprType>::value; + static constexpr bool is_contiguous = XprType::InnerStrideAtCompileTime == 1; using non_const_iterator = pointer_based_stl_iterator<std::remove_const_t<XprType>>; using const_iterator = pointer_based_stl_iterator<std::add_const_t<XprType>>; using other_iterator = std::conditional_t<std::is_const<XprType>::value, non_const_iterator, const_iterator>; @@ -327,8 +328,8 @@ using difference_type = Index; using value_type = typename XprType::Scalar; #if EIGEN_COMP_CXXVER >= 20 && defined(__cpp_lib_concepts) && __cpp_lib_concepts >= 202002L - using iterator_category = std::conditional_t<XprType::InnerStrideAtCompileTime == 1, std::contiguous_iterator_tag, - std::random_access_iterator_tag>; + using iterator_category = + std::conditional_t<is_contiguous, std::contiguous_iterator_tag, std::random_access_iterator_tag>; #else using iterator_category = std::random_access_iterator_tag; #endif @@ -337,28 +338,26 @@ pointer_based_stl_iterator() noexcept = default; pointer_based_stl_iterator(XprType& xpr, Index index) noexcept - : m_ptr(xpr.data() + index * xpr.innerStride()), m_incr(xpr.innerStride()) {} + : m_ptr(is_contiguous && index != 0 ? xpr.data() + index : xpr.data()), + m_index(is_contiguous ? 0 : index), + m_incr(xpr.innerStride()) {} - pointer_based_stl_iterator(const non_const_iterator& other) noexcept : m_ptr(other.m_ptr), m_incr(other.m_incr) {} + pointer_based_stl_iterator(const non_const_iterator& other) noexcept + : m_ptr(other.m_ptr), m_index(other.m_index), m_incr(other.m_incr) {} pointer_based_stl_iterator& operator=(const non_const_iterator& other) noexcept { m_ptr = other.m_ptr; + m_index = other.m_index; m_incr.setValue(other.m_incr); return *this; } - reference operator*() const { return *m_ptr; } - reference operator[](Index i) const { return *(m_ptr + i * m_incr.value()); } - pointer operator->() const { return m_ptr; } + reference operator*() const { return m_ptr[m_index.value() * m_incr.value()]; } + reference operator[](Index i) const { return m_ptr[(m_index.value() + i) * m_incr.value()]; } + pointer operator->() const { return m_index.value() == 0 ? m_ptr : m_ptr + m_index.value() * m_incr.value(); } - pointer_based_stl_iterator& operator++() { - m_ptr += m_incr.value(); - return *this; - } - pointer_based_stl_iterator& operator--() { - m_ptr -= m_incr.value(); - return *this; - } + pointer_based_stl_iterator& operator++() { return *this += 1; } + pointer_based_stl_iterator& operator--() { return *this -= 1; } pointer_based_stl_iterator operator++(int) { pointer_based_stl_iterator prev(*this); @@ -393,41 +392,64 @@ } pointer_based_stl_iterator& operator+=(Index b) { - m_ptr += b * m_incr.value(); + EIGEN_IF_CONSTEXPR (is_contiguous) { + if (b != 0) m_ptr += b; + } else { + m_index.setValue(m_index.value() + b); + } return *this; } pointer_based_stl_iterator& operator-=(Index b) { - m_ptr -= b * m_incr.value(); + EIGEN_IF_CONSTEXPR (is_contiguous) { + if (b != 0) m_ptr -= b; + } else { + m_index.setValue(m_index.value() - b); + } return *this; } difference_type operator-(const pointer_based_stl_iterator& other) const { - return (m_ptr - other.m_ptr) / m_incr.value(); + EIGEN_IF_CONSTEXPR (is_contiguous) return m_ptr == other.m_ptr ? 0 : m_ptr - other.m_ptr; + eigen_assert(m_ptr == other.m_ptr); + return m_index.value() - other.m_index.value(); } - difference_type operator-(const other_iterator& other) const { return (m_ptr - other.m_ptr) / m_incr.value(); } + difference_type operator-(const other_iterator& other) const { + EIGEN_IF_CONSTEXPR (is_contiguous) return m_ptr == other.m_ptr ? 0 : m_ptr - other.m_ptr; + eigen_assert(m_ptr == other.m_ptr); + return m_index.value() - other.m_index.value(); + } - bool operator==(const pointer_based_stl_iterator& other) const { return m_ptr == other.m_ptr; } - bool operator!=(const pointer_based_stl_iterator& other) const { return m_ptr != other.m_ptr; } + bool operator==(const pointer_based_stl_iterator& other) const { + return m_ptr == other.m_ptr && m_index.value() == other.m_index.value(); + } + bool operator!=(const pointer_based_stl_iterator& other) const { return !(*this == other); } bool operator<(const pointer_based_stl_iterator& other) const { - // Iterator order follows the stride, not necessarily increasing addresses. - return m_incr.value() < 0 ? m_ptr > other.m_ptr : m_ptr < other.m_ptr; + EIGEN_IF_CONSTEXPR (is_contiguous) return m_ptr < other.m_ptr; + eigen_assert(m_ptr == other.m_ptr); + return m_index.value() < other.m_index.value(); } bool operator<=(const pointer_based_stl_iterator& other) const { return !(other < *this); } bool operator>(const pointer_based_stl_iterator& other) const { return other < *this; } bool operator>=(const pointer_based_stl_iterator& other) const { return !(*this < other); } - bool operator==(const other_iterator& other) const { return m_ptr == other.m_ptr; } - bool operator!=(const other_iterator& other) const { return m_ptr != other.m_ptr; } + bool operator==(const other_iterator& other) const { + return m_ptr == other.m_ptr && m_index.value() == other.m_index.value(); + } + bool operator!=(const other_iterator& other) const { return !(*this == other); } bool operator<(const other_iterator& other) const { - return m_incr.value() < 0 ? m_ptr > other.m_ptr : m_ptr < other.m_ptr; + EIGEN_IF_CONSTEXPR (is_contiguous) return m_ptr < other.m_ptr; + eigen_assert(m_ptr == other.m_ptr); + return m_index.value() < other.m_index.value(); } bool operator<=(const other_iterator& other) const { return !(other < *this); } bool operator>(const other_iterator& other) const { return other < *this; } bool operator>=(const other_iterator& other) const { return !(*this < other); } protected: + // Only unit stride guarantees a valid end pointer. Otherwise keep the base pointer and a logical index. pointer m_ptr = nullptr; + internal::variable_if_dynamic<Index, is_contiguous ? 0 : Dynamic> m_index{0}; internal::variable_if_dynamic<Index, XprType::InnerStrideAtCompileTime> m_incr{XprType::InnerStrideAtCompileTime}; };
diff --git a/Eigen/src/Core/TriangularMatrix.h b/Eigen/src/Core/TriangularMatrix.h index 0d083aa..ee0e14f 100644 --- a/Eigen/src/Core/TriangularMatrix.h +++ b/Eigen/src/Core/TriangularMatrix.h
@@ -846,6 +846,15 @@ template <typename Kernel, unsigned int Mode, int UnrollCount, bool SetOpposite> struct triangular_assignment_loop; +// Whether call_triangular_assignment_loop fully unrolls the assignment. +template <typename DstEvaluatorType, typename SrcEvaluatorType> +struct triangular_assignment_unrolls { + static constexpr int Size = DstEvaluatorType::XprType::SizeAtCompileTime; + static constexpr bool value = + Size != Dynamic && SrcEvaluatorType::CoeffReadCost < HugeCost && + Size * (int(DstEvaluatorType::CoeffReadCost) + int(SrcEvaluatorType::CoeffReadCost)) / 2 <= EIGEN_UNROLLING_LIMIT; +}; + /** \internal Specialization of the dense assignment kernel for triangular matrices. * The main difference is that the triangular, diagonal, and opposite parts are processed through three different * functions. \tparam UpLo must be either Lower or Upper \tparam Mode must be either 0, UnitDiag, ZeroDiag, or @@ -917,13 +926,7 @@ SetOpposite, DstEvaluatorType, SrcEvaluatorType, Functor>; Kernel kernel(dstEvaluator, srcEvaluator, func, dst.const_cast_derived()); - enum { - unroll = DstXprType::SizeAtCompileTime != Dynamic && SrcEvaluatorType::CoeffReadCost < HugeCost && - DstXprType::SizeAtCompileTime * - (int(DstEvaluatorType::CoeffReadCost) + int(SrcEvaluatorType::CoeffReadCost)) / 2 <= - EIGEN_UNROLLING_LIMIT - }; - + constexpr bool unroll = triangular_assignment_unrolls<DstEvaluatorType, SrcEvaluatorType>::value; triangular_assignment_loop<Kernel, Mode, unroll ? int(DstXprType::SizeAtCompileTime) : Dynamic, SetOpposite>::run( kernel); }
diff --git a/Eigen/src/Core/arch/AVX/Complex.h b/Eigen/src/Core/arch/AVX/Complex.h index ce6b0ce..1eb81cf 100644 --- a/Eigen/src/Core/arch/AVX/Complex.h +++ b/Eigen/src/Core/arch/AVX/Complex.h
@@ -543,7 +543,13 @@ template <> inline Packet2cf ploaduSegment<Packet2cf>(const std::complex<float>* from, Index begin, Index count) { - return (Packet2cf)_mm_maskload_ps(&numext::real_ref(*from), segment_mask_2x64(begin, count)); + eigen_assert(begin >= 0 && begin <= 2 && count >= 0 && count <= 2 - begin); + if (count == 2) return ploadu<Packet2cf>(from); + if (count == 0) return Packet2cf(_mm_setzero_ps()); + // A single complex<float> needs only a 64-bit load, not a masked 128-bit load. + const void* address = from + begin; + const __m128 value = _mm_castsi128_ps(_mm_loadl_epi64(static_cast<const __m128i*>(address))); + return Packet2cf(begin == 0 ? value : _mm_movelh_ps(_mm_setzero_ps(), value)); } template <>
diff --git a/Eigen/src/Core/arch/AVX/MathFunctions.h b/Eigen/src/Core/arch/AVX/MathFunctions.h index f09e144..930cdff 100644 --- a/Eigen/src/Core/arch/AVX/MathFunctions.h +++ b/Eigen/src/Core/arch/AVX/MathFunctions.h
@@ -138,6 +138,7 @@ } EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_BF16(Packet8f, Packet8bf) +EIGEN_INSTANTIATE_GENERIC_SQRT_FUNCS_BF16(Packet8f, Packet8bf) #ifndef EIGEN_VECTORIZE_AVX512FP16 EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_F16(Packet8f, Packet8h)
diff --git a/Eigen/src/Core/arch/AVX/PacketMath.h b/Eigen/src/Core/arch/AVX/PacketMath.h index a5399b7..5130016 100644 --- a/Eigen/src/Core/arch/AVX/PacketMath.h +++ b/Eigen/src/Core/arch/AVX/PacketMath.h
@@ -2892,7 +2892,7 @@ // returns a mask of 8-bit elements (at most 4) that are all 1's in the range [begin, begin + count) and 0 elsewhere. inline __m128i segment_mask_4x8(Index begin, Index count) { - eigen_assert(begin >= 0 && begin + count <= 4); + eigen_assert(begin >= 0 && count >= 0 && begin + count <= 4); long long mask = 1; mask <<= CHAR_BIT * count; mask--; @@ -2906,7 +2906,7 @@ // returns a mask of 8-bit elements (at most 8) that are all 1's in the range [begin, begin + count) and 0 elsewhere. inline __m128i segment_mask_8x8(Index begin, Index count) { - eigen_assert(begin >= 0 && begin + count <= 8); + eigen_assert(begin >= 0 && count >= 0 && begin + count <= 8); unsigned long long mask = 1; // avoid UB when count == 8 mask <<= (CHAR_BIT / 2) * count; @@ -2923,13 +2923,13 @@ // returns a mask of 32-bit elements (at most 4) that are all 1's in the range [begin, begin + count) and 0 elsewhere. inline __m128i segment_mask_4x32(Index begin, Index count) { - eigen_assert(begin >= 0 && begin + count <= 4); + eigen_assert(begin >= 0 && count >= 0 && begin + count <= 4); return _mm_cvtepi8_epi32(segment_mask_4x8(begin, count)); } // returns a mask of 64-bit elements (at most 2) that are all 1's in the range [begin, begin + count) and 0 elsewhere. inline __m128i segment_mask_2x64(Index begin, Index count) { - eigen_assert(begin >= 0 && begin + count <= 2); + eigen_assert(begin >= 0 && count >= 0 && begin + count <= 2); return _mm_cvtepi8_epi64(segment_mask_4x8(begin, count)); }
diff --git a/Eigen/src/Core/arch/AVX512/Complex.h b/Eigen/src/Core/arch/AVX512/Complex.h index b3d90e2..ee1d9f5 100644 --- a/Eigen/src/Core/arch/AVX512/Complex.h +++ b/Eigen/src/Core/arch/AVX512/Complex.h
@@ -460,14 +460,15 @@ template <> inline Packet8cf ploaduSegment<Packet8cf>(const std::complex<float>* from, Index begin, Index count) { - return Packet8cf( - _mm512_maskz_loadu_ps(static_cast<__mmask16>(segment_kmask(2 * begin, 2 * count)), &numext::real_ref(*from))); + return Packet8cf(_mm512_maskz_loadu_ps(static_cast<__mmask16>(segment_kmask<Packet16f>(2 * begin, 2 * count)), + &numext::real_ref(*from))); } template <> inline void pstoreuSegment<std::complex<float>, Packet8cf>(std::complex<float>* to, const Packet8cf& from, Index begin, Index count) { - _mm512_mask_storeu_ps(&numext::real_ref(*to), static_cast<__mmask16>(segment_kmask(2 * begin, 2 * count)), from.v); + _mm512_mask_storeu_ps(&numext::real_ref(*to), static_cast<__mmask16>(segment_kmask<Packet16f>(2 * begin, 2 * count)), + from.v); } /*---------------- std::complex<double> ----------------*/ @@ -477,14 +478,15 @@ template <> inline Packet4cd ploaduSegment<Packet4cd>(const std::complex<double>* from, Index begin, Index count) { - return Packet4cd( - _mm512_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask(2 * begin, 2 * count)), &numext::real_ref(*from))); + return Packet4cd(_mm512_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask<Packet8d>(2 * begin, 2 * count)), + &numext::real_ref(*from))); } template <> inline void pstoreuSegment<std::complex<double>, Packet4cd>(std::complex<double>* to, const Packet4cd& from, Index begin, Index count) { - _mm512_mask_storeu_pd(&numext::real_ref(*to), static_cast<__mmask8>(segment_kmask(2 * begin, 2 * count)), from.v); + _mm512_mask_storeu_pd(&numext::real_ref(*to), static_cast<__mmask8>(segment_kmask<Packet8d>(2 * begin, 2 * count)), + from.v); } #ifdef EIGEN_VECTORIZE_AVX512VL @@ -493,38 +495,41 @@ template <> inline Packet4cf ploaduSegment<Packet4cf>(const std::complex<float>* from, Index begin, Index count) { - return Packet4cf( - _mm256_maskz_loadu_ps(static_cast<__mmask8>(segment_kmask(2 * begin, 2 * count)), &numext::real_ref(*from))); + return Packet4cf(_mm256_maskz_loadu_ps(static_cast<__mmask8>(segment_kmask<Packet8f>(2 * begin, 2 * count)), + &numext::real_ref(*from))); } template <> inline void pstoreuSegment<std::complex<float>, Packet4cf>(std::complex<float>* to, const Packet4cf& from, Index begin, Index count) { - _mm256_mask_storeu_ps(&numext::real_ref(*to), static_cast<__mmask8>(segment_kmask(2 * begin, 2 * count)), from.v); + _mm256_mask_storeu_ps(&numext::real_ref(*to), static_cast<__mmask8>(segment_kmask<Packet8f>(2 * begin, 2 * count)), + from.v); } template <> inline Packet2cf ploaduSegment<Packet2cf>(const std::complex<float>* from, Index begin, Index count) { - return Packet2cf( - _mm_maskz_loadu_ps(static_cast<__mmask8>(segment_kmask(2 * begin, 2 * count)), &numext::real_ref(*from))); + return Packet2cf(_mm_maskz_loadu_ps(static_cast<__mmask8>(segment_kmask<Packet4f>(2 * begin, 2 * count)), + &numext::real_ref(*from))); } template <> inline void pstoreuSegment<std::complex<float>, Packet2cf>(std::complex<float>* to, const Packet2cf& from, Index begin, Index count) { - _mm_mask_storeu_ps(&numext::real_ref(*to), static_cast<__mmask8>(segment_kmask(2 * begin, 2 * count)), from.v); + _mm_mask_storeu_ps(&numext::real_ref(*to), static_cast<__mmask8>(segment_kmask<Packet4f>(2 * begin, 2 * count)), + from.v); } template <> inline Packet2cd ploaduSegment<Packet2cd>(const std::complex<double>* from, Index begin, Index count) { - return Packet2cd( - _mm256_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask(2 * begin, 2 * count)), &numext::real_ref(*from))); + return Packet2cd(_mm256_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask<Packet4d>(2 * begin, 2 * count)), + &numext::real_ref(*from))); } template <> inline void pstoreuSegment<std::complex<double>, Packet2cd>(std::complex<double>* to, const Packet2cd& from, Index begin, Index count) { - _mm256_mask_storeu_pd(&numext::real_ref(*to), static_cast<__mmask8>(segment_kmask(2 * begin, 2 * count)), from.v); + _mm256_mask_storeu_pd(&numext::real_ref(*to), static_cast<__mmask8>(segment_kmask<Packet4d>(2 * begin, 2 * count)), + from.v); } #endif // EIGEN_VECTORIZE_AVX512VL
diff --git a/Eigen/src/Core/arch/AVX512/MathFunctions.h b/Eigen/src/Core/arch/AVX512/MathFunctions.h index c38a0b1..757d542 100644 --- a/Eigen/src/Core/arch/AVX512/MathFunctions.h +++ b/Eigen/src/Core/arch/AVX512/MathFunctions.h
@@ -117,6 +117,7 @@ #endif EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_BF16(Packet16f, Packet16bf) +EIGEN_INSTANTIATE_GENERIC_SQRT_FUNCS_BF16(Packet16f, Packet16bf) #ifndef EIGEN_VECTORIZE_AVX512FP16 EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_F16(Packet16f, Packet16h)
diff --git a/Eigen/src/Core/arch/AVX512/PacketMath.h b/Eigen/src/Core/arch/AVX512/PacketMath.h index 77a4489..c5d051f 100644 --- a/Eigen/src/Core/arch/AVX512/PacketMath.h +++ b/Eigen/src/Core/arch/AVX512/PacketMath.h
@@ -3212,10 +3212,11 @@ /*---------------- load/store segment support ----------------*/ -// Returns a k-mask with bits [begin, begin + count) set. 64-bit arithmetic lets count reach the lane -// count of the widest AVX-512 packet (32) without shifting a 1 past the top bit. +// Returns a k-mask with bits [begin, begin + count) set over the lanes of Packet. 64-bit arithmetic lets count +// reach the lane count of the widest AVX-512 packet (32) without shifting a 1 past the top bit. +template <typename Packet> EIGEN_STRONG_INLINE uint64_t segment_kmask(Index begin, Index count) { - eigen_assert(begin >= 0 && count >= 0 && begin + count <= 32); + eigen_assert(begin >= 0 && count >= 0 && begin + count <= unpacket_traits<Packet>::size); return ((uint64_t(1) << count) - 1) << begin; } @@ -3226,12 +3227,12 @@ template <> inline Packet16f ploaduSegment<Packet16f>(const float* from, Index begin, Index count) { - return _mm512_maskz_loadu_ps(static_cast<__mmask16>(segment_kmask(begin, count)), from); + return _mm512_maskz_loadu_ps(static_cast<__mmask16>(segment_kmask<Packet16f>(begin, count)), from); } template <> inline void pstoreuSegment<float, Packet16f>(float* to, const Packet16f& from, Index begin, Index count) { - _mm512_mask_storeu_ps(to, static_cast<__mmask16>(segment_kmask(begin, count)), from); + _mm512_mask_storeu_ps(to, static_cast<__mmask16>(segment_kmask<Packet16f>(begin, count)), from); } /*---------------- double ----------------*/ @@ -3241,12 +3242,12 @@ template <> inline Packet8d ploaduSegment<Packet8d>(const double* from, Index begin, Index count) { - return _mm512_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm512_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask<Packet8d>(begin, count)), from); } template <> inline void pstoreuSegment<double, Packet8d>(double* to, const Packet8d& from, Index begin, Index count) { - _mm512_mask_storeu_pd(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm512_mask_storeu_pd(to, static_cast<__mmask8>(segment_kmask<Packet8d>(begin, count)), from); } /*---------------- int32 ----------------*/ @@ -3256,12 +3257,12 @@ template <> inline Packet16i ploaduSegment<Packet16i>(const int* from, Index begin, Index count) { - return _mm512_maskz_loadu_epi32(static_cast<__mmask16>(segment_kmask(begin, count)), from); + return _mm512_maskz_loadu_epi32(static_cast<__mmask16>(segment_kmask<Packet16i>(begin, count)), from); } template <> inline void pstoreuSegment<int, Packet16i>(int* to, const Packet16i& from, Index begin, Index count) { - _mm512_mask_storeu_epi32(to, static_cast<__mmask16>(segment_kmask(begin, count)), from); + _mm512_mask_storeu_epi32(to, static_cast<__mmask16>(segment_kmask<Packet16i>(begin, count)), from); } /*---------------- int64_t ----------------*/ @@ -3271,12 +3272,12 @@ template <> inline Packet8l ploaduSegment<Packet8l>(const int64_t* from, Index begin, Index count) { - return _mm512_maskz_loadu_epi64(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm512_maskz_loadu_epi64(static_cast<__mmask8>(segment_kmask<Packet8l>(begin, count)), from); } template <> inline void pstoreuSegment<int64_t, Packet8l>(int64_t* to, const Packet8l& from, Index begin, Index count) { - _mm512_mask_storeu_epi64(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm512_mask_storeu_epi64(to, static_cast<__mmask8>(segment_kmask<Packet8l>(begin, count)), from); } #ifdef EIGEN_VECTORIZE_AVX512VL @@ -3288,66 +3289,66 @@ template <> inline Packet8f ploaduSegment<Packet8f>(const float* from, Index begin, Index count) { - return _mm256_maskz_loadu_ps(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm256_maskz_loadu_ps(static_cast<__mmask8>(segment_kmask<Packet8f>(begin, count)), from); } template <> inline void pstoreuSegment<float, Packet8f>(float* to, const Packet8f& from, Index begin, Index count) { - _mm256_mask_storeu_ps(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm256_mask_storeu_ps(to, static_cast<__mmask8>(segment_kmask<Packet8f>(begin, count)), from); } template <> inline Packet4f ploaduSegment<Packet4f>(const float* from, Index begin, Index count) { - return _mm_maskz_loadu_ps(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm_maskz_loadu_ps(static_cast<__mmask8>(segment_kmask<Packet4f>(begin, count)), from); } template <> inline void pstoreuSegment<float, Packet4f>(float* to, const Packet4f& from, Index begin, Index count) { - _mm_mask_storeu_ps(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm_mask_storeu_ps(to, static_cast<__mmask8>(segment_kmask<Packet4f>(begin, count)), from); } /*---------------- double ----------------*/ template <> inline Packet4d ploaduSegment<Packet4d>(const double* from, Index begin, Index count) { - return _mm256_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm256_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask<Packet4d>(begin, count)), from); } template <> inline void pstoreuSegment<double, Packet4d>(double* to, const Packet4d& from, Index begin, Index count) { - _mm256_mask_storeu_pd(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm256_mask_storeu_pd(to, static_cast<__mmask8>(segment_kmask<Packet4d>(begin, count)), from); } template <> inline Packet2d ploaduSegment<Packet2d>(const double* from, Index begin, Index count) { - return _mm_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm_maskz_loadu_pd(static_cast<__mmask8>(segment_kmask<Packet2d>(begin, count)), from); } template <> inline void pstoreuSegment<double, Packet2d>(double* to, const Packet2d& from, Index begin, Index count) { - _mm_mask_storeu_pd(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm_mask_storeu_pd(to, static_cast<__mmask8>(segment_kmask<Packet2d>(begin, count)), from); } /*---------------- int32 ----------------*/ template <> inline Packet8i ploaduSegment<Packet8i>(const int* from, Index begin, Index count) { - return _mm256_maskz_loadu_epi32(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm256_maskz_loadu_epi32(static_cast<__mmask8>(segment_kmask<Packet8i>(begin, count)), from); } template <> inline void pstoreuSegment<int, Packet8i>(int* to, const Packet8i& from, Index begin, Index count) { - _mm256_mask_storeu_epi32(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm256_mask_storeu_epi32(to, static_cast<__mmask8>(segment_kmask<Packet8i>(begin, count)), from); } template <> inline Packet4i ploaduSegment<Packet4i>(const int* from, Index begin, Index count) { - return _mm_maskz_loadu_epi32(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm_maskz_loadu_epi32(static_cast<__mmask8>(segment_kmask<Packet4i>(begin, count)), from); } template <> inline void pstoreuSegment<int, Packet4i>(int* to, const Packet4i& from, Index begin, Index count) { - _mm_mask_storeu_epi32(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm_mask_storeu_epi32(to, static_cast<__mmask8>(segment_kmask<Packet4i>(begin, count)), from); } /*---------------- uint32 ----------------*/ @@ -3376,22 +3377,22 @@ template <> inline Packet4l ploaduSegment<Packet4l>(const int64_t* from, Index begin, Index count) { - return _mm256_maskz_loadu_epi64(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm256_maskz_loadu_epi64(static_cast<__mmask8>(segment_kmask<Packet4l>(begin, count)), from); } template <> inline void pstoreuSegment<int64_t, Packet4l>(int64_t* to, const Packet4l& from, Index begin, Index count) { - _mm256_mask_storeu_epi64(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm256_mask_storeu_epi64(to, static_cast<__mmask8>(segment_kmask<Packet4l>(begin, count)), from); } template <> inline Packet2l ploaduSegment<Packet2l>(const int64_t* from, Index begin, Index count) { - return _mm_maskz_loadu_epi64(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm_maskz_loadu_epi64(static_cast<__mmask8>(segment_kmask<Packet2l>(begin, count)), from); } template <> inline void pstoreuSegment<int64_t, Packet2l>(int64_t* to, const Packet2l& from, Index begin, Index count) { - _mm_mask_storeu_epi64(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm_mask_storeu_epi64(to, static_cast<__mmask8>(segment_kmask<Packet2l>(begin, count)), from); } /*---------------- uint64_t ----------------*/ @@ -3420,22 +3421,22 @@ template <> inline Packet16bf ploaduSegment<Packet16bf>(const bfloat16* from, Index begin, Index count) { - return _mm256_maskz_loadu_epi16(static_cast<__mmask16>(segment_kmask(begin, count)), from); + return _mm256_maskz_loadu_epi16(static_cast<__mmask16>(segment_kmask<Packet16bf>(begin, count)), from); } template <> inline void pstoreuSegment<bfloat16, Packet16bf>(bfloat16* to, const Packet16bf& from, Index begin, Index count) { - _mm256_mask_storeu_epi16(to, static_cast<__mmask16>(segment_kmask(begin, count)), from); + _mm256_mask_storeu_epi16(to, static_cast<__mmask16>(segment_kmask<Packet16bf>(begin, count)), from); } template <> inline Packet8bf ploaduSegment<Packet8bf>(const bfloat16* from, Index begin, Index count) { - return _mm_maskz_loadu_epi16(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm_maskz_loadu_epi16(static_cast<__mmask8>(segment_kmask<Packet8bf>(begin, count)), from); } template <> inline void pstoreuSegment<bfloat16, Packet8bf>(bfloat16* to, const Packet8bf& from, Index begin, Index count) { - _mm_mask_storeu_epi16(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm_mask_storeu_epi16(to, static_cast<__mmask8>(segment_kmask<Packet8bf>(begin, count)), from); } #ifndef EIGEN_VECTORIZE_AVX512FP16 @@ -3450,22 +3451,22 @@ template <> inline Packet16h ploaduSegment<Packet16h>(const Eigen::half* from, Index begin, Index count) { - return _mm256_maskz_loadu_epi16(static_cast<__mmask16>(segment_kmask(begin, count)), from); + return _mm256_maskz_loadu_epi16(static_cast<__mmask16>(segment_kmask<Packet16h>(begin, count)), from); } template <> inline void pstoreuSegment<Eigen::half, Packet16h>(Eigen::half* to, const Packet16h& from, Index begin, Index count) { - _mm256_mask_storeu_epi16(to, static_cast<__mmask16>(segment_kmask(begin, count)), from); + _mm256_mask_storeu_epi16(to, static_cast<__mmask16>(segment_kmask<Packet16h>(begin, count)), from); } template <> inline Packet8h ploaduSegment<Packet8h>(const Eigen::half* from, Index begin, Index count) { - return _mm_maskz_loadu_epi16(static_cast<__mmask8>(segment_kmask(begin, count)), from); + return _mm_maskz_loadu_epi16(static_cast<__mmask8>(segment_kmask<Packet8h>(begin, count)), from); } template <> inline void pstoreuSegment<Eigen::half, Packet8h>(Eigen::half* to, const Packet8h& from, Index begin, Index count) { - _mm_mask_storeu_epi16(to, static_cast<__mmask8>(segment_kmask(begin, count)), from); + _mm_mask_storeu_epi16(to, static_cast<__mmask8>(segment_kmask<Packet8h>(begin, count)), from); } #endif // EIGEN_VECTORIZE_AVX512FP16
diff --git a/Eigen/src/Core/arch/AVX512/PacketMathFP16.h b/Eigen/src/Core/arch/AVX512/PacketMathFP16.h index d08941e..e7a2b8d 100644 --- a/Eigen/src/Core/arch/AVX512/PacketMathFP16.h +++ b/Eigen/src/Core/arch/AVX512/PacketMathFP16.h
@@ -1451,32 +1451,36 @@ template <> inline Packet32h ploaduSegment<Packet32h>(const Eigen::half* from, Index begin, Index count) { - return _mm512_castsi512_ph(_mm512_maskz_loadu_epi16(static_cast<__mmask32>(segment_kmask(begin, count)), from)); + return _mm512_castsi512_ph( + _mm512_maskz_loadu_epi16(static_cast<__mmask32>(segment_kmask<Packet32h>(begin, count)), from)); } template <> inline void pstoreuSegment<Eigen::half, Packet32h>(Eigen::half* to, const Packet32h& from, Index begin, Index count) { - _mm512_mask_storeu_epi16(to, static_cast<__mmask32>(segment_kmask(begin, count)), _mm512_castph_si512(from)); + _mm512_mask_storeu_epi16(to, static_cast<__mmask32>(segment_kmask<Packet32h>(begin, count)), + _mm512_castph_si512(from)); } template <> inline Packet16h ploaduSegment<Packet16h>(const Eigen::half* from, Index begin, Index count) { - return _mm256_castsi256_ph(_mm256_maskz_loadu_epi16(static_cast<__mmask16>(segment_kmask(begin, count)), from)); + return _mm256_castsi256_ph( + _mm256_maskz_loadu_epi16(static_cast<__mmask16>(segment_kmask<Packet16h>(begin, count)), from)); } template <> inline void pstoreuSegment<Eigen::half, Packet16h>(Eigen::half* to, const Packet16h& from, Index begin, Index count) { - _mm256_mask_storeu_epi16(to, static_cast<__mmask16>(segment_kmask(begin, count)), _mm256_castph_si256(from)); + _mm256_mask_storeu_epi16(to, static_cast<__mmask16>(segment_kmask<Packet16h>(begin, count)), + _mm256_castph_si256(from)); } template <> inline Packet8h ploaduSegment<Packet8h>(const Eigen::half* from, Index begin, Index count) { - return _mm_castsi128_ph(_mm_maskz_loadu_epi16(static_cast<__mmask8>(segment_kmask(begin, count)), from)); + return _mm_castsi128_ph(_mm_maskz_loadu_epi16(static_cast<__mmask8>(segment_kmask<Packet8h>(begin, count)), from)); } template <> inline void pstoreuSegment<Eigen::half, Packet8h>(Eigen::half* to, const Packet8h& from, Index begin, Index count) { - _mm_mask_storeu_epi16(to, static_cast<__mmask8>(segment_kmask(begin, count)), _mm_castph_si128(from)); + _mm_mask_storeu_epi16(to, static_cast<__mmask8>(segment_kmask<Packet8h>(begin, count)), _mm_castph_si128(from)); } /*---------------- end load/store segment support ----------------*/
diff --git a/Eigen/src/Core/arch/Default/BFloat16.h b/Eigen/src/Core/arch/Default/BFloat16.h index 5fb7659..6312932 100644 --- a/Eigen/src/Core/arch/Default/BFloat16.h +++ b/Eigen/src/Core/arch/Default/BFloat16.h
@@ -39,44 +39,70 @@ return F32ToBf16(METHOD<PACKET_F>(Bf16ToF32(_x))); \ } +// BF16_PACKET_FUNCTION for a bfloat16 packet that spans two PACKET_F, through the backend's +// Bf16ToF32Low, Bf16ToF32High and two-argument F32ToBf16. +// METHOD is a template name, which cannot be parenthesized. +// NOLINTBEGIN(bugprone-macro-parentheses) +#define BF16_PACKET_FUNCTION_HALVES(PACKET_F, PACKET_BF16, METHOD) \ + template <> \ + EIGEN_DEFINE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS EIGEN_UNUSED PACKET_BF16 METHOD<PACKET_BF16>( \ + const PACKET_BF16& _x) { \ + return F32ToBf16(METHOD<PACKET_F>(Bf16ToF32Low(_x)), METHOD<PACKET_F>(Bf16ToF32High(_x))); \ + } +// NOLINTEND(bugprone-macro-parentheses) + +// The _WITH forms take the macro that defines each function, BF16_PACKET_FUNCTION or +// BF16_PACKET_FUNCTION_HALVES, so that every backend instantiates the same list. +#define EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_BF16_WITH(FUNCTION, PACKET_F, PACKET_BF16) \ + FUNCTION(PACKET_F, PACKET_BF16, pcos) \ + FUNCTION(PACKET_F, PACKET_BF16, psin) \ + FUNCTION(PACKET_F, PACKET_BF16, psinh) \ + FUNCTION(PACKET_F, PACKET_BF16, pcosh) \ + FUNCTION(PACKET_F, PACKET_BF16, pasinh) \ + FUNCTION(PACKET_F, PACKET_BF16, pacosh) \ + FUNCTION(PACKET_F, PACKET_BF16, pexp) \ + FUNCTION(PACKET_F, PACKET_BF16, pexp2) \ + FUNCTION(PACKET_F, PACKET_BF16, pexpm1) \ + FUNCTION(PACKET_F, PACKET_BF16, plog) \ + FUNCTION(PACKET_F, PACKET_BF16, plog1p) \ + FUNCTION(PACKET_F, PACKET_BF16, plog2) \ + FUNCTION(PACKET_F, PACKET_BF16, plog10) \ + FUNCTION(PACKET_F, PACKET_BF16, preciprocal) \ + FUNCTION(PACKET_F, PACKET_BF16, pcbrt) \ + FUNCTION(PACKET_F, PACKET_BF16, ptanh) + #define EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_BF16(PACKET_F, PACKET_BF16) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pcos) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, psin) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, psinh) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pcosh) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pasinh) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pacosh) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pexp) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pexp2) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pexpm1) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, plog) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, plog1p) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, plog2) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, plog10) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, preciprocal) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, prsqrt) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pcbrt) \ + EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_BF16_WITH(BF16_PACKET_FUNCTION, PACKET_F, PACKET_BF16) + +// psqrt and prsqrt through the float round trip, for backends without their own bfloat16 kernels. +#define EIGEN_INSTANTIATE_GENERIC_SQRT_FUNCS_BF16(PACKET_F, PACKET_BF16) \ BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, psqrt) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, ptanh) + BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, prsqrt) // BF16 wrappers for contrib/SpecialFunctions. +#define EIGEN_INSTANTIATE_SPECIAL_FUNCS_BF16_WITH(FUNCTION, PACKET_F, PACKET_BF16) \ + FUNCTION(PACKET_F, PACKET_BF16, perf) \ + FUNCTION(PACKET_F, PACKET_BF16, pndtri) + #define EIGEN_INSTANTIATE_SPECIAL_FUNCS_BF16(PACKET_F, PACKET_BF16) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, perf) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pndtri) + EIGEN_INSTANTIATE_SPECIAL_FUNCS_BF16_WITH(BF16_PACKET_FUNCTION, PACKET_F, PACKET_BF16) + +#define EIGEN_INSTANTIATE_BESSEL_FUNCS_BF16_WITH(FUNCTION, PACKET_F, PACKET_BF16) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_i0) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_i0e) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_i1) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_i1e) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_j0) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_j1) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_k0) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_k0e) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_k1) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_k1e) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_y0) \ + FUNCTION(PACKET_F, PACKET_BF16, pbessel_y1) #define EIGEN_INSTANTIATE_BESSEL_FUNCS_BF16(PACKET_F, PACKET_BF16) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_i0) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_i0e) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_i1) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_i1e) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_j0) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_j1) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_k0) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_k0e) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_k1) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_k1e) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_y0) \ - BF16_PACKET_FUNCTION(PACKET_F, PACKET_BF16, pbessel_y1) + EIGEN_INSTANTIATE_BESSEL_FUNCS_BF16_WITH(BF16_PACKET_FUNCTION, PACKET_F, PACKET_BF16) // Only use HIP GPU bf16 in kernels #if defined(EIGEN_HAS_HIP_BF16) && defined(EIGEN_GPU_COMPILE_PHASE) @@ -697,60 +723,61 @@ numext::uint16_t x = numext::bit_cast<numext::uint16_t>(a) & 0x7FFF; return numext::bit_cast<bfloat16>(x); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 exp(const bfloat16& a) { return bfloat16(::expf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 exp2(const bfloat16& a) { return bfloat16(::exp2f(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 exp(const bfloat16& a) { return bfloat16(numext::exp(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 exp2(const bfloat16& a) { return bfloat16(numext::exp2(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 expm1(const bfloat16& a) { return bfloat16(numext::expm1(float(a))); } // float covers the bfloat16 range, so the single conversion back rounds correctly. EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 ldexp(const bfloat16& a, int exponent) { return bfloat16(numext::ldexp(float(a), exponent)); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 log(const bfloat16& a) { return bfloat16(::logf(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 log(const bfloat16& a) { return bfloat16(numext::log(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 log1p(const bfloat16& a) { return bfloat16(numext::log1p(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 log10(const bfloat16& a) { return bfloat16(::log10f(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 log2(const bfloat16& a) { - return bfloat16(static_cast<float>(EIGEN_LOG2E) * ::logf(float(a))); -} -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 sqrt(const bfloat16& a) { return bfloat16(::sqrtf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 cbrt(const bfloat16& a) { return bfloat16(::cbrtf(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 log10(const bfloat16& a) { return bfloat16(numext::log10(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 log2(const bfloat16& a) { return bfloat16(numext::log2(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 sqrt(const bfloat16& a) { return bfloat16(numext::sqrt(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 cbrt(const bfloat16& a) { return bfloat16(numext::cbrt(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 pow(const bfloat16& a, const bfloat16& b) { - return bfloat16(::powf(float(a), float(b))); + return bfloat16(numext::pow(float(a), float(b))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 atan2(const bfloat16& a, const bfloat16& b) { - return bfloat16(::atan2f(float(a), float(b))); + return bfloat16(numext::atan2(float(a), float(b))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 sin(const bfloat16& a) { return bfloat16(::sinf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 cos(const bfloat16& a) { return bfloat16(::cosf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 tan(const bfloat16& a) { return bfloat16(::tanf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 asin(const bfloat16& a) { return bfloat16(::asinf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 acos(const bfloat16& a) { return bfloat16(::acosf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 atan(const bfloat16& a) { return bfloat16(::atanf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 sinh(const bfloat16& a) { return bfloat16(::sinhf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 cosh(const bfloat16& a) { return bfloat16(::coshf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 tanh(const bfloat16& a) { return bfloat16(::tanhf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 asinh(const bfloat16& a) { return bfloat16(::asinhf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 acosh(const bfloat16& a) { return bfloat16(::acoshf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 atanh(const bfloat16& a) { return bfloat16(::atanhf(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 sin(const bfloat16& a) { return bfloat16(numext::sin(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 cos(const bfloat16& a) { return bfloat16(numext::cos(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 tan(const bfloat16& a) { return bfloat16(numext::tan(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 asin(const bfloat16& a) { return bfloat16(numext::asin(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 acos(const bfloat16& a) { return bfloat16(numext::acos(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 atan(const bfloat16& a) { return bfloat16(numext::atan(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 sinh(const bfloat16& a) { return bfloat16(numext::sinh(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 cosh(const bfloat16& a) { return bfloat16(numext::cosh(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 tanh(const bfloat16& a) { + // Select the scalar template, not the host fast-math overload, to preserve bfloat16 rounding. + return bfloat16(numext::tanh<float>(float(a))); +} +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 asinh(const bfloat16& a) { return bfloat16(numext::asinh(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 acosh(const bfloat16& a) { return bfloat16(numext::acosh(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 atanh(const bfloat16& a) { return bfloat16(numext::atanh(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 exact_float_to_bfloat16(float f) { return raw_uint16_to_bfloat16(static_cast<numext::uint16_t>(numext::bit_cast<numext::uint32_t>(f) >> 16)); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 floor(const bfloat16& a) { - return exact_float_to_bfloat16(::floorf(float(a))); + return exact_float_to_bfloat16(numext::floor(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 ceil(const bfloat16& a) { - return exact_float_to_bfloat16(::ceilf(float(a))); + return exact_float_to_bfloat16(numext::ceil(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 rint(const bfloat16& a) { - return exact_float_to_bfloat16(::rintf(float(a))); + return exact_float_to_bfloat16(numext::rint(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 round(const bfloat16& a) { - return exact_float_to_bfloat16(::roundf(float(a))); + return exact_float_to_bfloat16(numext::round(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 trunc(const bfloat16& a) { - return exact_float_to_bfloat16(::truncf(float(a))); + return exact_float_to_bfloat16(numext::trunc(float(a))); } // fmod is exact: a - n*b is either a itself or a multiple of the ulp of b that is smaller than |b|. EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 fmod(const bfloat16& a, const bfloat16& b) { - return exact_float_to_bfloat16(::fmodf(float(a), float(b))); + return exact_float_to_bfloat16(numext::fmod(float(a), float(b))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16(min)(const bfloat16& a, const bfloat16& b) { @@ -768,17 +795,19 @@ EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 fmin(const bfloat16& a, const bfloat16& b) { const float f1 = static_cast<float>(a); const float f2 = static_cast<float>(b); - return exact_float_to_bfloat16(::fminf(f1, f2)); + EIGEN_USING_STD(fmin); + return exact_float_to_bfloat16(fmin(f1, f2)); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bfloat16 fmax(const bfloat16& a, const bfloat16& b) { const float f1 = static_cast<float>(a); const float f2 = static_cast<float>(b); - return exact_float_to_bfloat16(::fmaxf(f1, f2)); + EIGEN_USING_STD(fmax); + return exact_float_to_bfloat16(fmax(f1, f2)); } EIGEN_DEVICE_FUNC inline bfloat16 fma(const bfloat16& a, const bfloat16& b, const bfloat16& c) { -#if defined(EIGEN_GPU_COMPILE_PHASE) || defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_GPU_COMPILE_PHASE) || defined(__SYCL_DEVICE_ONLY__) // Rounds twice; see half_impl::fma_round_to_odd. return bfloat16(numext::fma(static_cast<float>(a), static_cast<float>(b), static_cast<float>(c))); #else
diff --git a/Eigen/src/Core/arch/Default/GenericPacketMathComplex.h b/Eigen/src/Core/arch/Default/GenericPacketMathComplex.h index 1f1147d..afc13de 100644 --- a/Eigen/src/Core/arch/Default/GenericPacketMathComplex.h +++ b/Eigen/src/Core/arch/Default/GenericPacketMathComplex.h
@@ -87,47 +87,77 @@ return padd(tmp_im, tmp_re); // a*c - b*d, a*d + b*c } -template <typename Packet, bool Base2> -EIGEN_DEFINE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet plog_complex(const Packet& x) { +// \internal plog_complex. Only when `Scaled` does it scale the elements outside [lo, hi] and handle infinities, which +// plog_complex guarantees are absent otherwise. +template <typename Packet, bool Base2, bool Scaled> +EIGEN_DEFINE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet plog_complex_impl(const Packet& x) { using RealPacket = typename unpacket_traits<Packet>::as_real; using RealScalar = typename unpacket_traits<RealPacket>::type; - // Real part. phypot_complex squares the components, which neither overflows nor loses accuracy to underflow for - // M = max(|a|, |b|) in [lo, hi] = [sqrt(min / eps), sqrt(highest) / 2]. Elements with M > hi are scaled by 2^-k with - // k = max_exponent / 2 + 2, and those with M < lo by 2^-k with k = -(3 digits - min_exponent - 1) / 2; both are exact - // and land M in [lo, hi]. Then log|x| = log|x * 2^-k| + k log(2). RealPacket x_flip = pcplxflip(x).v; // b, a - RealPacket x_abs = pabs(x.v); - const RealPacket x_max = pmax(x_abs, pcplxflip(Packet(x_abs)).v); - const RealScalar lo = numext::sqrt((numext::numeric_limits<RealScalar>::min)() / NumTraits<RealScalar>::epsilon()); - const RealScalar hi = numext::sqrt(NumTraits<RealScalar>::highest()) / RealScalar(2); - const RealPacket is_large = pcmp_lt(pset1<RealPacket>(hi), x_max); - const RealPacket is_small = pcmp_lt(x_max, pset1<RealPacket>(lo)); - // Both divisions round down: for double, k_small = -589 maps denorm_min = 2^-1074 exactly onto lo = 2^-485. - constexpr int kLarge = NumTraits<RealScalar>::max_exponent() / 2 + 2; - constexpr int kSmall = -((3 * NumTraits<RealScalar>::digits() - NumTraits<RealScalar>::min_exponent() - 1) / 2); - const RealPacket k_large = pset1<RealPacket>(RealScalar(kLarge)); - const RealPacket k_small = pset1<RealPacket>(RealScalar(kSmall)); - const RealPacket cst_one = pset1<RealPacket>(RealScalar(1)); - const RealPacket k = pselect(is_large, k_large, pand(is_small, k_small)); - const RealPacket scale = pselect(is_large, pldexp_fast(cst_one, pnegate(k_large)), - pselect(is_small, pldexp_fast(cst_one, pnegate(k_small)), cst_one)); - // log(sqrt(a^2 + b^2)), log(sqrt(a^2 + b^2)); in base 2 the real part comes from plog2, which is exact where |x| is a - // power of two, rather than from log2(e) log|x|. - const RealPacket r = phypot_complex(Packet(pmul(x.v, scale))).v; - const RealPacket xlogr = Base2 ? padd(k, plog2(r)) : pmadd(k, pset1<RealPacket>(RealScalar(EIGEN_LN2)), plog(r)); + RealPacket xlogr; + if (Scaled) { + // Real part. phypot_complex squares the components, which neither overflows nor loses accuracy to underflow for + // M = max(|a|, |b|) in [lo, hi] = [sqrt(min / eps), sqrt(highest) / 2]. Elements with M > hi are scaled by 2^-k + // with k = max_exponent / 2 + 2, and those with M < lo by 2^-k with k = -(3 digits - min_exponent - 1) / 2; both + // are exact and land M in [lo, hi]. Then log|x| = log|x * 2^-k| + k log(2). + RealPacket x_abs = pabs(x.v); + const RealPacket x_max = pmax(x_abs, pcplxflip(Packet(x_abs)).v); + const RealScalar lo = numext::sqrt((numext::numeric_limits<RealScalar>::min)() / NumTraits<RealScalar>::epsilon()); + const RealScalar hi = numext::sqrt(NumTraits<RealScalar>::highest()) / RealScalar(2); + const RealPacket is_large = pcmp_lt(pset1<RealPacket>(hi), x_max); + const RealPacket is_small = pcmp_lt(x_max, pset1<RealPacket>(lo)); + // Both divisions round down: for double, k_small = -589 maps denorm_min = 2^-1074 exactly onto lo = 2^-485. + constexpr int kLarge = NumTraits<RealScalar>::max_exponent() / 2 + 2; + constexpr int kSmall = -((3 * NumTraits<RealScalar>::digits() - NumTraits<RealScalar>::min_exponent() - 1) / 2); + const RealPacket k_large = pset1<RealPacket>(RealScalar(kLarge)); + const RealPacket k_small = pset1<RealPacket>(RealScalar(kSmall)); + const RealPacket cst_one = pset1<RealPacket>(RealScalar(1)); + const RealPacket k = pselect(is_large, k_large, pand(is_small, k_small)); + const RealPacket scale = pselect(is_large, pldexp_fast(cst_one, pnegate(k_large)), + pselect(is_small, pldexp_fast(cst_one, pnegate(k_small)), cst_one)); + // log(sqrt(a^2 + b^2)), log(sqrt(a^2 + b^2)); in base 2 the real part comes from plog2, which is exact where |x| is + // a power of two, rather than from log2(e) log|x|. + const RealPacket r = phypot_complex(Packet(pmul(x.v, scale))).v; + xlogr = Base2 ? padd(k, plog2(r)) : pmadd(k, pset1<RealPacket>(RealScalar(EIGEN_LN2)), plog(r)); + + const RealPacket cst_pos_inf = pinf<RealPacket>(); + RealPacket is_x_pos_inf = pcmp_eq(x_abs, cst_pos_inf); + RealPacket is_y_pos_inf = pcplxflip(Packet(is_x_pos_inf)).v; + xlogr = pselect(por(is_x_pos_inf, is_y_pos_inf), cst_pos_inf, xlogr); + } else { + const RealPacket r = phypot_complex(x).v; + xlogr = Base2 ? plog2(r) : plog(r); + } // Imag part RealPacket ximg = patan2(x.v, x_flip); // atan2(a, b), atan2(b, a) if (Base2) ximg = pmul(pset1<RealPacket>(RealScalar(EIGEN_LOG2E)), ximg); - const RealPacket cst_pos_inf = pinf<RealPacket>(); - RealPacket is_x_pos_inf = pcmp_eq(x_abs, cst_pos_inf); - RealPacket is_y_pos_inf = pcplxflip(Packet(is_x_pos_inf)).v; - RealPacket is_any_inf = por(is_x_pos_inf, is_y_pos_inf); - RealPacket xreal = pselect(is_any_inf, cst_pos_inf, xlogr); + return Packet(pselect(peven_mask(xlogr), xlogr, ximg)); // log(sqrt(a^2 + b^2)), atan2(b, a) +} - return Packet(pselect(peven_mask(xreal), xreal, ximg)); // log(sqrt(a^2 + b^2)), atan2(b, a) +// Out of line, so that the common path stays small enough to inline into the caller. +template <typename Packet, bool Base2> +EIGEN_DEVICE_FUNC EIGEN_DONT_INLINE Packet plog_complex_scaled(const Packet& x) { + return plog_complex_impl<Packet, Base2, true>(x); +} + +template <typename Packet, bool Base2> +EIGEN_DEFINE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet plog_complex(const Packet& x) { + using RealPacket = typename unpacket_traits<Packet>::as_real; + using RealScalar = typename unpacket_traits<RealPacket>::type; + const RealPacket x_abs = pabs(x.v); + const RealPacket x_max = pmax(x_abs, pcplxflip(Packet(x_abs)).v); + const RealScalar lo = numext::sqrt((numext::numeric_limits<RealScalar>::min)() / NumTraits<RealScalar>::epsilon()); + const RealScalar hi = numext::sqrt(NumTraits<RealScalar>::highest()) / RealScalar(2); + // Infinite components exceed hi. |a| and |b| are compared separately, because pmax may drop the inf of (NaN, inf). + // Zero, which is common, needs no scaling: log|0| = -inf either way. + const RealPacket is_small = pand(pcmp_lt(pzero(x_max), x_max), pcmp_lt(x_max, pset1<RealPacket>(lo))); + if (predux_any(por(pcmp_lt(pset1<RealPacket>(hi), x_abs), is_small))) { + return plog_complex_scaled<Packet, Base2>(x); + } + return plog_complex_impl<Packet, Base2, false>(x); } template <typename Packet> @@ -374,14 +404,21 @@ if (predux_any(por(pcmp_lt(hi, z_abs), is_small))) { Packet z_scaled = z; // Keeps compilers from executing this branch speculatively on the common path. - EIGEN_OPTIMIZATION_BARRIER(z_scaled.v) + pbarrier(z_scaled.v); return psqrt_complex_impl(z_scaled, true); } return psqrt_complex_impl(z, false); } -// \internal \returns the norm of a complex number z = x + i*y, defined as sqrt(x^2 + y^2). -// Implemented using the hypot(a,b) algorithm from https://doi.org/10.48550/arXiv.1904.09481 +// \internal \returns the norm of a complex number z = x + i*y, defined as sqrt(x^2 + y^2), in both lanes of each +// element. This is the corrected hypot of Borges (2019), "An improved algorithm for hypot(a, b)", +// https://doi.org/10.48550/arXiv.1904.09481: h = fl(sqrt(fl(a^2 + b^2))) is refined by one Newton step with the +// residual x = h^2 - a^2 - b^2 formed from the exact squares h^2 = h_sq + h_lo, a^2 = a_sq + a_lo and b^2 = b_sq + +// b_lo. With a the larger component, a_sq <= h_sq <= 2 a_sq and h_sq - a_sq is exact; the remaining terms are of order +// u h^2, so their rounding errors are of order u^2 h^2 and the result is within (1/2 + O(u)) ulp of sqrt(a^2 + b^2). +// It is correctly rounded unless the exact value lies within O(u) ulp of a rounding midpoint: for a = b^2 with b odd, +// sqrt(a^2 + a) = a + 1/2 - 1/(8a) + ..., and the result rounds up to a + 1. Both the order and the exact residuals +// matter: with the components unordered, or with residuals from an unfused pmadd, the error exceeds 0.8 and 1.3 ulps. template <typename Packet> EIGEN_DEFINE_FUNCTION_ALLOWING_MULTIPLE_DEFINITIONS Packet phypot_complex(const Packet& a) { using Scalar = typename unpacket_traits<Packet>::type; @@ -389,24 +426,24 @@ using RealPacket = typename unpacket_traits<Packet>::as_real; const RealPacket cst_zero_rp = pset1<RealPacket>(static_cast<RealScalar>(0.0)); - const RealPacket cst_minus_one_rp = pset1<RealPacket>(static_cast<RealScalar>(-1.0)); - const RealPacket cst_two_rp = pset1<RealPacket>(static_cast<RealScalar>(2.0)); - const RealPacket evenmask = peven_mask(a.v); + const RealPacket cst_half_rp = pset1<RealPacket>(static_cast<RealScalar>(0.5)); RealPacket a_abs = pabs(a.v); - RealPacket a_flip = pcplxflip(Packet(a_abs)).v; // |b|, |a| - RealPacket a_all = pselect(evenmask, a_abs, a_flip); // |a|, |a| - RealPacket b_all = pselect(evenmask, a_flip, a_abs); // |b|, |b| + RealPacket a_flip = pcplxflip(Packet(a_abs)).v; // |b|, |a| + RealPacket a_all = pmax(a_abs, a_flip); // the larger component, in both lanes + RealPacket b_all = pmin(a_abs, a_flip); // the smaller component, in both lanes - RealPacket a2 = pmul(a.v, a.v); // |a^2, b^2| - RealPacket a2_flip = pcplxflip(Packet(a2)).v; // |b^2, a^2| - RealPacket h = psqrt(padd(a2, a2_flip)); // |sqrt(a^2 + b^2), sqrt(a^2 + b^2)| - RealPacket h_sq = pmul(h, h); // |a^2 + b^2, a^2 + b^2| - RealPacket a_sq = pselect(evenmask, a2, a2_flip); // |a^2, a^2| - RealPacket m_h_sq = pmul(h_sq, cst_minus_one_rp); - RealPacket m_a_sq = pmul(a_sq, cst_minus_one_rp); - RealPacket x = psub(psub(pmadd(h, h, m_h_sq), pmadd(b_all, b_all, psub(a_sq, h_sq))), pmadd(a_all, a_all, m_a_sq)); - h = psub(h, pdiv(x, pmul(cst_two_rp, h))); // |h - x/(2*h), h - x/(2*h)| + RealPacket a2 = pmul(a.v, a.v); // |a^2, b^2| + RealPacket a2_flip = pcplxflip(Packet(a2)).v; // |b^2, a^2| + RealPacket h = psqrt(padd(a2, a2_flip)); // |sqrt(a^2 + b^2), sqrt(a^2 + b^2)| + RealPacket h_sq = pmul(h, h); + RealPacket a_sq = pmax(a2, a2_flip); + RealPacket b_sq = pmin(a2, a2_flip); + RealPacket h_lo = twoprod_low(h, h, h_sq); + RealPacket a_lo = twoprod_low(a_all, a_all, a_sq); + RealPacket b_lo = twoprod_low(b_all, b_all, b_sq); + RealPacket x = padd(psub(psub(h_sq, a_sq), b_sq), psub(h_lo, padd(a_lo, b_lo))); + h = pnmadd(pdiv(x, h), cst_half_rp, h); // |h - x/(2*h), h - x/(2*h)| // handle zero-case RealPacket iszero = pcmp_eq(por(a_abs, a_flip), cst_zero_rp);
diff --git a/Eigen/src/Core/arch/Default/GenericPacketMathDoubleWord.h b/Eigen/src/Core/arch/Default/GenericPacketMathDoubleWord.h index 9e3acc2..ab08126 100644 --- a/Eigen/src/Core/arch/Default/GenericPacketMathDoubleWord.h +++ b/Eigen/src/Core/arch/Default/GenericPacketMathDoubleWord.h
@@ -25,6 +25,22 @@ r = psub(x, n); } +// Knuth two-sum: x + y = s_hi + s_lo exactly, with no magnitude ordering required. +template <typename Packet> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void twosum(const Packet& x, const Packet& y, Packet& s_hi, Packet& s_lo) { + s_hi = padd(x, y); + pbarrier(s_hi); + Packet y_part = psub(s_hi, x); + pbarrier(y_part); + Packet x_part = psub(s_hi, y_part); + pbarrier(x_part); + Packet x_err = psub(x, x_part); + pbarrier(x_err); + Packet y_err = psub(y, y_part); + pbarrier(y_err); + s_lo = padd(x_err, y_err); +} + // This function computes the sum {s_hi, s_lo}, such that x + y = s_hi + s_lo // holds exactly, and s_hi = fl(x+y), if |x| >= |y|. template <typename Packet> @@ -44,14 +60,14 @@ template <typename Scalar, std::enable_if_t<is_scalar<Scalar>::value, int> = 0> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Scalar twoprod_low(const Scalar& x, const Scalar& y, const Scalar& xy) { // Error-free products require FMA even when EIGEN_SCALAR_MADD_USE_FMA disables fusion in scalar madd. - return numext::fma(x, y, Scalar(-xy)); + return twoprod_low_fma(x, y, xy); } // This function implements the extended precision product of // a pair of floating point numbers. Given {x, y}, it computes the pair // {p_hi, p_lo} such that x * y = p_hi + p_lo holds exactly and -// p_hi = fl(x * y). -template <typename Packet> +// p_hi = fl(x * y). PreventReassociation preserves the separately rounded non-FMA operations. +template <typename Packet, bool PreventReassociation = false> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void twoprod(const Packet& x, const Packet& y, Packet& p_hi, Packet& p_lo) { p_hi = pmul(x, y); p_lo = twoprod_low(x, y, p_hi); @@ -79,10 +95,10 @@ template <typename Scalar, std::enable_if_t<has_fast_fma<Scalar>::value, int> = 0> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Scalar twoprod_low(const Scalar& x, const Scalar& y, const Scalar& xy) { - return numext::fma(x, y, Scalar(-xy)); + return twoprod_low_fma(x, y, xy); } -template <typename Scalar, std::enable_if_t<has_fast_fma<Scalar>::value, int> = 0> +template <typename Scalar, bool PreventReassociation = false, std::enable_if_t<has_fast_fma<Scalar>::value, int> = 0> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void twoprod(const Scalar& x, const Scalar& y, Scalar& p_hi, Scalar& p_lo) { p_hi = x * y; p_lo = twoprod_low(x, y, p_hi); @@ -93,49 +109,50 @@ // exactly and that half of the significand of x fits in x_hi. // This is Algorithm 3 from Jean-Michel Muller, "Elementary Functions", // 3rd edition, Birkh\"auser, 2016. -template <typename Packet> +template <typename Packet, bool PreventReassociation = false> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void veltkamp_splitting(const Packet& x, Packet& x_hi, Packet& x_lo) { using Scalar = typename unpacket_traits<Packet>::type; constexpr int shift = (NumTraits<Scalar>::digits() + 1) / 2; const Scalar shift_scale = Scalar(uint64_t(1) << shift); // Scalar constructor not necessarily constexpr. - const Packet gamma = pmul(pset1<Packet>(shift_scale + Scalar(1)), x); + Packet gamma = pmul(pset1<Packet>(shift_scale + Scalar(1)), x); + EIGEN_IF_CONSTEXPR (PreventReassociation) pbarrier(gamma); Packet rho = psub(x, gamma); + EIGEN_IF_CONSTEXPR (PreventReassociation) pbarrier(rho); x_hi = padd(rho, gamma); + EIGEN_IF_CONSTEXPR (PreventReassociation) pbarrier(x_hi); x_lo = psub(x, x_hi); } -// This function implements Dekker's algorithm for products x * y. -// Given floating point numbers {x, y} computes the pair -// {p_hi, p_lo} such that x * y = p_hi + p_lo holds exactly and -// p_hi = fl(x * y). -template <typename Packet, std::enable_if_t<!has_fast_fma<Packet>::value, int> = 0> -EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void twoprod(const Packet& x, const Packet& y, Packet& p_hi, Packet& p_lo) { - Packet x_hi, x_lo, y_hi, y_lo; - veltkamp_splitting(x, x_hi, x_lo); - veltkamp_splitting(y, y_hi, y_lo); - - p_hi = pmul(x, y); - p_lo = pmadd(x_hi, y_hi, pnegate(p_hi)); - p_lo = pmadd(x_hi, y_lo, p_lo); - p_lo = pmadd(x_lo, y_hi, p_lo); - p_lo = pmadd(x_lo, y_lo, p_lo); -} - // A version of twoprod that takes x, y, and fl(x*y) as input and returns the p_lo such that -// x * y = xy + p_lo holds exactly. -template <typename Packet, std::enable_if_t<!has_fast_fma<Packet>::value, int> = 0> +// x * y = xy + p_lo holds exactly. PreventReassociation preserves the separately rounded operations. +template <typename Packet, bool PreventReassociation = false, std::enable_if_t<!has_fast_fma<Packet>::value, int> = 0> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Packet twoprod_low(const Packet& x, const Packet& y, const Packet& xy) { Packet x_hi, x_lo, y_hi, y_lo; - veltkamp_splitting(x, x_hi, x_lo); - veltkamp_splitting(y, y_hi, y_lo); + veltkamp_splitting<Packet, PreventReassociation>(x, x_hi, x_lo); + veltkamp_splitting<Packet, PreventReassociation>(y, y_hi, y_lo); - Packet p_lo = pmadd(x_hi, y_hi, pnegate(xy)); + Packet p_lo = pmsub(x_hi, y_hi, xy); + EIGEN_IF_CONSTEXPR (PreventReassociation) pbarrier(p_lo); p_lo = pmadd(x_hi, y_lo, p_lo); + EIGEN_IF_CONSTEXPR (PreventReassociation) pbarrier(p_lo); p_lo = pmadd(x_lo, y_hi, p_lo); + EIGEN_IF_CONSTEXPR (PreventReassociation) pbarrier(p_lo); p_lo = pmadd(x_lo, y_lo, p_lo); return p_lo; } +// This function implements Dekker's algorithm for products x * y. +// Given floating point numbers {x, y}, it computes {p_hi, p_lo} such that +// x * y = p_hi + p_lo exactly and p_hi = fl(x * y). +template <typename Packet, bool PreventReassociation = false, std::enable_if_t<!has_fast_fma<Packet>::value, int> = 0> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void twoprod(const Packet& x, const Packet& y, Packet& p_hi, Packet& p_lo) { + Packet hi = pmul(x, y); + EIGEN_IF_CONSTEXPR (PreventReassociation) pbarrier(hi); + const Packet lo = twoprod_low<Packet, PreventReassociation>(x, y, hi); + p_hi = hi; + p_lo = lo; +} + #endif // EIGEN_VECTORIZE_FMA // This function implements the addition of two double word numbers
diff --git a/Eigen/src/Core/arch/Default/GenericPacketMathFrexpLdexp.h b/Eigen/src/Core/arch/Default/GenericPacketMathFrexpLdexp.h index 4516b05..a1a7311 100644 --- a/Eigen/src/Core/arch/Default/GenericPacketMathFrexpLdexp.h +++ b/Eigen/src/Core/arch/Default/GenericPacketMathFrexpLdexp.h
@@ -93,9 +93,9 @@ template <typename Packet> EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC Packet pldexp_apply_factors(const Packet& a, const Packet& c1, const Packet& c2) { Packet out = pmul(a, c1); - EIGEN_OPTIMIZATION_BARRIER(out) + pbarrier(out); out = pmul(out, c1); - EIGEN_OPTIMIZATION_BARRIER(out) + pbarrier(out); return pmul(out, c2); }
diff --git a/Eigen/src/Core/arch/Default/GenericPacketMathFunctions.h b/Eigen/src/Core/arch/Default/GenericPacketMathFunctions.h index 53970a0..a08a1d4 100644 --- a/Eigen/src/Core/arch/Default/GenericPacketMathFunctions.h +++ b/Eigen/src/Core/arch/Default/GenericPacketMathFunctions.h
@@ -654,7 +654,7 @@ if (!predux_any(pcmp_lt(pset1<Packet>(Scalar(max_exponent - 2)), pabs(n)))) return pldexp_fast(y, n); const Packet h = print(pmul(n, pset1<Packet>(Scalar(0.5)))); Packet out = pldexp_fast(y, h); - EIGEN_OPTIMIZATION_BARRIER(out) + pbarrier(out); return pldexp_fast(out, psub(n, h)); } @@ -828,7 +828,7 @@ Packet sign_a = pandnot(a, abs_a); Packet rint_a = padd(abs_a, cst_limit); // Don't compile-away addition and subtraction. - EIGEN_OPTIMIZATION_BARRIER(rint_a); + pbarrier(rint_a); rint_a = psub(rint_a, cst_limit); rint_a = por(rint_a, sign_a); // If greater than limit (or NaN), simply return a.
diff --git a/Eigen/src/Core/arch/Default/GenericPacketMathTrig.h b/Eigen/src/Core/arch/Default/GenericPacketMathTrig.h index 0884542..a98b7c7 100644 --- a/Eigen/src/Core/arch/Default/GenericPacketMathTrig.h +++ b/Eigen/src/Core/arch/Default/GenericPacketMathTrig.h
@@ -100,7 +100,7 @@ // Rounding trick to find nearest integer: Packet y_round = padd(y, cst_rounding_magic); - EIGEN_OPTIMIZATION_BARRIER(y_round) + pbarrier(y_round); PacketI y_int = preinterpret<PacketI>(y_round); // last 23 digits represent integer (if abs(x)<2^24) y = psub(y_round, cst_rounding_magic); // nearest integer to x * (2/pi) @@ -122,11 +122,11 @@ // and 2 ULP up to: constexpr float huge_th = (Func == TrigFunction::Sin) ? 25966.f : 18838.f; x = pmadd(y, pset1<Packet>(-1.5703125), x); // = 0xbfc90000 - EIGEN_OPTIMIZATION_BARRIER(x) + pbarrier(x); x = pmadd(y, pset1<Packet>(-0.000483989715576171875), x); // = 0xb9fdc000 - EIGEN_OPTIMIZATION_BARRIER(x) + pbarrier(x); x = pmadd(y, pset1<Packet>(1.62865035235881805419921875e-07), x); // = 0x342ee000 - EIGEN_OPTIMIZATION_BARRIER(x) + pbarrier(x); x = pmadd(y, pset1<Packet>(5.5644315544167710640977020375430583953857421875e-11), x); // = 0x2e74b9ee // For the record, the following set of coefficients maintain 2ULP up @@ -263,24 +263,21 @@ // Compute q * pio2_1 exactly as a double-word using Dekker's algorithm. Packet qp_hi, qp_lo; - twoprod(cst_pio2_1<Packet>(), q, qp_hi, qp_lo); + twoprod<Packet, true>(cst_pio2_1<Packet>(), q, qp_hi, qp_lo); // Error-free addition of x and qp_hi using Knuth's 2sum. // Returns t_hi + t_lo = x + qp_hi exactly, with t_hi = fl(x + qp_hi). - Packet t_hi = padd(x, qp_hi); - Packet v = psub(t_hi, x); - Packet t_lo = padd(psub(x, psub(t_hi, v)), psub(qp_hi, v)); + Packet t_hi, t_lo; + twosum(x, qp_hi, t_hi, t_lo); // Near a multiple of pi/2, t_hi + qp_lo cancels against q * pio2_2 too, so that product is formed exactly as well // and the two large terms are added by 2sum: only terms of order u * |q * pio2_2| are rounded. Packet qp2_hi, qp2_lo; - twoprod(cst_pio2_2<Packet>(), q, qp2_hi, qp2_lo); - Packet a_hi = padd(t_hi, qp_lo); - v = psub(a_hi, t_hi); - t_lo = padd(t_lo, padd(psub(t_hi, psub(a_hi, v)), psub(qp_lo, v))); - t_hi = padd(a_hi, qp2_hi); - v = psub(t_hi, a_hi); - t_lo = padd(t_lo, padd(psub(a_hi, psub(t_hi, v)), psub(qp2_hi, v))); + twoprod<Packet, true>(cst_pio2_2<Packet>(), q, qp2_hi, qp2_lo); + Packet a_hi, a_lo, b_lo; + twosum(t_hi, qp_lo, a_hi, a_lo); + twosum(a_hi, qp2_hi, t_hi, b_lo); + t_lo = padd(t_lo, padd(a_lo, b_lo)); // Accumulate the low part of the product and the remaining pi/2 term. t_lo = padd(t_lo, qp2_lo); @@ -319,7 +316,7 @@ if (EIGEN_PREDICT_FALSE(predux_any(pcmp_le(pset1<Packet>(small_th), x_abs)))) { // Medium path: use double-word product x * (2/pi) for precise quadrant computation. Packet prod_hi, prod_lo; - twoprod(x_abs, cst_2oPI_hi, prod_hi, prod_lo); + twoprod<Packet, true>(x_abs, cst_2oPI_hi, prod_hi, prod_lo); // Correction for 2/pi truncation: add x * lo(2/pi) prod_lo = pmadd(x_abs, cst_2oPI_lo, prod_lo); @@ -336,9 +333,26 @@ } else { // Small path: simple reduction with triple-double pi/2 split. Packet qval_noround = pmul(x_abs, cst_2oPI_hi); +#ifdef EIGEN_HAS_SINGLE_INSTRUCTION_MADD q_int = pcast<Packet, PacketI>(padd(qval_noround, pset1<Packet>(0.5))); Packet q = pcast<PacketI, Packet>(q_int); s = trig_reduce_small_double(x_abs, q); +#else + // Only q_int & 3 is consumed below. In round-to-nearest mode, the bias leaves q in its low bits. + const Packet bias = pset1<Packet>(6755399441055744.0); // 1.5 * 2^52. + Packet biased_q = padd(qval_noround, bias); + pbarrier(biased_q); + q_int = preinterpret<PacketI>(biased_q); + const Packet q = psub(biased_q, bias); + // |x| < 15 implies q <= 10. The first two 49-bit parts give exact q*c products; + // the third leaves < 2^-155 error in pi/2. Generated with MPFR at 256 bits: + // c1 = trunc49(pi/2), c2 = trunc49(pi/2-c1), c3 = RN53(pi/2-c1-c2). + s = pmadd(q, pset1<Packet>(-1.5707963267948948), x_abs); + pbarrier(s); + s = pmadd(q, pset1<Packet>(-1.837589179357618e-15), s); + pbarrier(s); + s = pmadd(q, pset1<Packet>(-8.4784276603688996e-32), s); +#endif } Packet ss = pmul(s, s);
diff --git a/Eigen/src/Core/arch/Default/Half.h b/Eigen/src/Core/arch/Default/Half.h index 6983b9c..509c84d 100644 --- a/Eigen/src/Core/arch/Default/Half.h +++ b/Eigen/src/Core/arch/Default/Half.h
@@ -160,7 +160,7 @@ // CUDA GPU compile phase. -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) using __half_raw = cl::sycl::half; #endif @@ -447,7 +447,7 @@ namespace half_impl { -#if defined(EIGEN_GPU_COMPILE_PHASE) +#if (defined(EIGEN_CUDA_ARCH) && EIGEN_CUDA_ARCH >= 530) || defined(EIGEN_HIP_DEVICE_COMPILE) // Note: We deliberately do *not* define this to 1 even if we have Arm's native // fp16 type since GPU half types are rather different from native CPU half types. #define EIGEN_HAS_NATIVE_GPU_FP16 @@ -755,7 +755,7 @@ return numext::bit_cast<numext::uint16_t>(h.x); #elif defined(EIGEN_HAS_BUILTIN_FLOAT16) return numext::bit_cast<numext::uint16_t>(h.x); -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) return numext::bit_cast<numext::uint16_t>(h); #else return h.x; @@ -907,7 +907,7 @@ EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bool(isinf)(const half& a) { return (raw_half_as_uint16(a) & 0x7fff) == 0x7c00; } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC bool(isnan)(const half& a) { -#if defined(EIGEN_GPU_COMPILE_PHASE) +#if defined(EIGEN_HAS_NATIVE_GPU_FP16) return __hisnan(a); #else return (raw_half_as_uint16(a) & 0x7fff) > 0x7c00; @@ -925,17 +925,17 @@ #endif } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half exp(const half& a) { -#if defined(EIGEN_CUDA_ARCH) || defined(EIGEN_HIP_DEVICE_COMPILE) +#if defined(EIGEN_HAS_NATIVE_GPU_FP16) return half(hexp(::__half(a))); #else - return half(::expf(float(a))); + return half(numext::exp(float(a))); #endif } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half exp2(const half& a) { -#if defined(EIGEN_CUDA_ARCH) || defined(EIGEN_HIP_DEVICE_COMPILE) +#if defined(EIGEN_HAS_NATIVE_GPU_FP16) return half(hexp2(::__half(a))); #else - return half(::exp2f(float(a))); + return half(numext::exp2(float(a))); #endif } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half expm1(const half& a) { return half(numext::expm1(float(a))); } @@ -944,58 +944,59 @@ return half(numext::ldexp(float(a), exponent)); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half log(const half& a) { -#if defined(EIGEN_GPU_COMPILE_PHASE) +#if defined(EIGEN_HAS_NATIVE_GPU_FP16) return half(hlog(::__half(a))); #else - return half(::logf(float(a))); + return half(numext::log(float(a))); #endif } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half log1p(const half& a) { return half(numext::log1p(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half log10(const half& a) { return half(::log10f(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half log2(const half& a) { - return half(static_cast<float>(EIGEN_LOG2E) * ::logf(float(a))); -} +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half log10(const half& a) { return half(numext::log10(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half log2(const half& a) { return half(numext::log2(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half sqrt(const half& a) { -#if defined(EIGEN_CUDA_ARCH) || defined(EIGEN_HIP_DEVICE_COMPILE) +#if defined(EIGEN_HAS_NATIVE_GPU_FP16) return half(hsqrt(::__half(a))); #else - return half(::sqrtf(float(a))); + return half(numext::sqrt(float(a))); #endif } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half pow(const half& a, const half& b) { - return half(::powf(float(a), float(b))); + return half(numext::pow(float(a), float(b))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half atan2(const half& a, const half& b) { - return half(::atan2f(float(a), float(b))); + return half(numext::atan2(float(a), float(b))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half sin(const half& a) { return half(::sinf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half cos(const half& a) { return half(::cosf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half tan(const half& a) { return half(::tanf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half tanh(const half& a) { return half(::tanhf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half asin(const half& a) { return half(::asinf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half acos(const half& a) { return half(::acosf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half atan(const half& a) { return half(::atanf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half atanh(const half& a) { return half(::atanhf(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half sin(const half& a) { return half(numext::sin(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half cos(const half& a) { return half(numext::cos(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half tan(const half& a) { return half(numext::tan(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half tanh(const half& a) { + // Select the scalar template, not the host fast-math overload, to preserve half rounding. + return half(numext::tanh<float>(float(a))); +} +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half asin(const half& a) { return half(numext::asin(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half acos(const half& a) { return half(numext::acos(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half atan(const half& a) { return half(numext::atan(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half atanh(const half& a) { return half(numext::atanh(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half floor(const half& a) { -#if (defined(EIGEN_CUDA_ARCH)) || defined(EIGEN_HIP_DEVICE_COMPILE) +#if defined(EIGEN_HAS_NATIVE_GPU_FP16) return half(hfloor(::__half(a))); #else - return half(::floorf(float(a))); + return half(numext::floor(float(a))); #endif } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half ceil(const half& a) { -#if (defined(EIGEN_CUDA_ARCH)) || defined(EIGEN_HIP_DEVICE_COMPILE) +#if defined(EIGEN_HAS_NATIVE_GPU_FP16) return half(hceil(::__half(a))); #else - return half(::ceilf(float(a))); + return half(numext::ceil(float(a))); #endif } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half rint(const half& a) { return half(::rintf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half round(const half& a) { return half(::roundf(float(a))); } -EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half trunc(const half& a) { return half(::truncf(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half rint(const half& a) { return half(numext::rint(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half round(const half& a) { return half(numext::round(float(a))); } +EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half trunc(const half& a) { return half(numext::trunc(float(a))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half fmod(const half& a, const half& b) { - return half(::fmodf(float(a), float(b))); + return half(numext::fmod(float(a), float(b))); } EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC half(min)(const half& a, const half& b) { return b < a ? b : a; } @@ -1025,7 +1026,7 @@ #if defined(EIGEN_HAS_NATIVE_GPU_FP16) // Rounded once in hardware. return __hfma(::__half(a), ::__half(b), ::__half(c)); -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_GPU_COMPILE_PHASE) || defined(__SYCL_DEVICE_ONLY__) // Rounds twice; see fma_round_to_odd. return half(numext::fma(static_cast<float>(a), static_cast<float>(b), static_cast<float>(c))); #elif defined(EIGEN_HAS_ARM64_FP16_SCALAR_ARITHMETIC)
diff --git a/Eigen/src/Core/arch/GPU/PacketMath.h b/Eigen/src/Core/arch/GPU/PacketMath.h index 12dfd21..058bbbb 100644 --- a/Eigen/src/Core/arch/GPU/PacketMath.h +++ b/Eigen/src/Core/arch/GPU/PacketMath.h
@@ -647,6 +647,10 @@ // --------------------------------------------------------------------------------------------------------------- // half2, the native two-lane packet. +#if (defined(EIGEN_CUDA_ARCH) && EIGEN_CUDA_ARCH >= 530) || defined(EIGEN_HIP_DEVICE_COMPILE) +#define EIGEN_GPU_HAS_FP16_ARITHMETIC 1 +#endif + template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 pset1<half2>(const Eigen::half& from) { return __half2half2(from); @@ -754,7 +758,11 @@ template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 plset<half2>(const Eigen::half& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return __halves2half2(a, __hadd(a, __float2half(1.0f))); +#else + return __halves2half2(a, __float2half(__half2float(a) + 1.0f)); +#endif } template <> @@ -802,32 +810,57 @@ template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 padd<half2>(const half2& a, const half2& b) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return __hadd2(a, b); +#else + return __floats2half2_rn(__low2float(a) + __low2float(b), __high2float(a) + __high2float(b)); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 psub<half2>(const half2& a, const half2& b) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return __hsub2(a, b); +#else + return __floats2half2_rn(__low2float(a) - __low2float(b), __high2float(a) - __high2float(b)); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 pnegate(const half2& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return __hneg2(a); +#else + return __floats2half2_rn(-__low2float(a), -__high2float(a)); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 pmul<half2>(const half2& a, const half2& b) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return __hmul2(a, b); +#else + return __floats2half2_rn(__low2float(a) * __low2float(b), __high2float(a) * __high2float(b)); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 pmadd<half2>(const half2& a, const half2& b, const half2& c) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return __hfma2(a, b, c); +#else + return __floats2half2_rn(fmaf(__low2float(a), __low2float(b), __low2float(c)), + fmaf(__high2float(a), __high2float(b), __high2float(c))); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 pdiv<half2>(const half2& a, const half2& b) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return __h2div(a, b); +#else + return __floats2half2_rn(__low2float(a) / __low2float(b), __high2float(a) / __high2float(b)); +#endif } // Compared through float, and every comparison against a NaN is false, so the result is b. This differs from @@ -849,7 +882,11 @@ template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Eigen::half predux<half2>(const half2& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return __hadd(__low2half(a), __high2half(a)); +#else + return __float2half(__low2float(a) + __high2float(a)); +#endif } template <> @@ -859,41 +896,69 @@ template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Eigen::half predux_max<half2>(const half2& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) const __half first = __low2half(a); const __half second = __high2half(a); return __hgt(first, second) ? first : second; +#else + return __low2float(a) > __high2float(a) ? __low2half(a) : __high2half(a); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Eigen::half predux_min<half2>(const half2& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) const __half first = __low2half(a); const __half second = __high2half(a); return __hlt(first, second) ? first : second; +#else + return __low2float(a) < __high2float(a) ? __low2half(a) : __high2half(a); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Eigen::half predux_mul<half2>(const half2& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return __hmul(__low2half(a), __high2half(a)); +#else + return __float2half(__low2float(a) * __high2float(a)); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 plog<half2>(const half2& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return h2log(a); +#else + return __floats2half2_rn(logf(__low2float(a)), logf(__high2float(a))); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 pexp<half2>(const half2& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return h2exp(a); +#else + return __floats2half2_rn(expf(__low2float(a)), expf(__high2float(a))); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 psqrt<half2>(const half2& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return h2sqrt(a); +#else + return __floats2half2_rn(sqrtf(__low2float(a)), sqrtf(__high2float(a))); +#endif } template <> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE half2 prsqrt<half2>(const half2& a) { +#if defined(EIGEN_GPU_HAS_FP16_ARITHMETIC) return h2rsqrt(a); +#else + return __floats2half2_rn(rsqrtf(__low2float(a)), rsqrtf(__high2float(a))); +#endif } // No native h2log1p/h2expm1; both go through float. @@ -1050,8 +1115,9 @@ EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Packet4h2 plset<Packet4h2>(const Eigen::half& a) { // Add each offset once: (a + 2*k) + 1 can round twice. Preserve a's sign in lane zero. const half2 base = pset1<half2>(a); - return make_packet4h2(plset<half2>(a), __hadd2(base, __floats2half2_rn(2.0f, 3.0f)), - __hadd2(base, __floats2half2_rn(4.0f, 5.0f)), __hadd2(base, __floats2half2_rn(6.0f, 7.0f))); + return make_packet4h2(plset<half2>(a), padd<half2>(base, __floats2half2_rn(2.0f, 3.0f)), + padd<half2>(base, __floats2half2_rn(4.0f, 5.0f)), + padd<half2>(base, __floats2half2_rn(6.0f, 7.0f))); } template <> @@ -1131,6 +1197,8 @@ return predux_mul<half2>(product); } +#undef EIGEN_GPU_HAS_FP16_ARITHMETIC + #endif // defined(EIGEN_GPU_COMPILE_PHASE) } // end namespace internal
diff --git a/Eigen/src/Core/arch/NEON/GeneralBlockPanelKernel.h b/Eigen/src/Core/arch/NEON/GeneralBlockPanelKernel.h index 11b3c24..3efa2a3 100644 --- a/Eigen/src/Core/arch/NEON/GeneralBlockPanelKernel.h +++ b/Eigen/src/Core/arch/NEON/GeneralBlockPanelKernel.h
@@ -6,6 +6,17 @@ namespace Eigen { namespace internal { +#if EIGEN_ARCH_ARM +template <> +struct gebp_traits<double, double, false, false, Architecture::NEON, GEBPPacketFull> + : gebp_traits<double, double, false, false, Architecture::Generic, GEBPPacketFull> { + // AArch32 has 32 scalar double registers, enough for a 3x4 tile despite having only 16 NEON registers. + static constexpr int mr = 3; + // Keep scalar-aligned loads: pbroadcast4's unaligned memcpy loads spill through the stack on ARM32 GCC. + using RhsPacketx4 = RhsPacket; +}; +#endif + #if EIGEN_ARCH_ARM && EIGEN_COMP_CLANG // Clang seems to excessively spill registers in the GEBP kernel on 32-bit arm.
diff --git a/Eigen/src/Core/arch/NEON/MathFunctions.h b/Eigen/src/Core/arch/NEON/MathFunctions.h index 9c07f4f..63b2571 100644 --- a/Eigen/src/Core/arch/NEON/MathFunctions.h +++ b/Eigen/src/Core/arch/NEON/MathFunctions.h
@@ -35,23 +35,6 @@ } #endif // EIGEN_ARCH_ARM64 && EIGEN_HAS_ARM64_FP16 -EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_BF16(Packet4f, Packet4bf) - -template <> -EIGEN_STRONG_INLINE Packet4bf pfrexp(const Packet4bf& a, Packet4bf& exponent) { - // Both results are exact: the mantissa keeps the input's significand, and the exponent is a - // small integer. - Packet4f fexponent; - const Packet4bf out = F32ToBf16Truncate(pfrexp<Packet4f>(Bf16ToF32(a), fexponent)); - exponent = F32ToBf16Truncate(fexponent); - return out; -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pldexp(const Packet4bf& a, const Packet4bf& exponent) { - return F32ToBf16(pldexp<Packet4f>(Bf16ToF32(a), Bf16ToF32(exponent))); -} - //---------- double ---------- #if EIGEN_ARCH_ARM64
diff --git a/Eigen/src/Core/arch/NEON/PacketMath.h b/Eigen/src/Core/arch/NEON/PacketMath.h index ee0a78b..2159449 100644 --- a/Eigen/src/Core/arch/NEON/PacketMath.h +++ b/Eigen/src/Core/arch/NEON/PacketMath.h
@@ -4744,379 +4744,6 @@ } #endif -//---------- bfloat16 ---------- -// TODO: Add support for native armv8.6-a bfloat16_t - -// TODO: Guard if we have native bfloat16 support -typedef eigen_packet_wrapper<uint16x4_t, 19> Packet4bf; - -template <> -struct is_arithmetic<Packet4bf> : std::true_type {}; - -template <> -struct packet_traits<bfloat16> : default_packet_traits { - typedef Packet4bf type; - typedef Packet4bf half; - enum { - Vectorizable = 1, - AlignedOnScalar = 1, - size = 4, - - HasCmp = 1, - HasAdd = 1, - HasSub = 1, - HasShift = 1, - HasMul = 1, - HasNegate = 1, - HasAbs = 1, - HasArg = 0, - HasAbsDiff = 1, - HasMin = 1, - HasMax = 1, - HasConj = 1, - HasSetLinear = 1, - HasDiv = 1, - HasSin = EIGEN_FAST_MATH, - HasCos = EIGEN_FAST_MATH, - HasLog = 1, - HasExp = 1, - HasSqrt = 0, - HasTanh = EIGEN_FAST_MATH, - HasErf = EIGEN_FAST_MATH, - HasBessel = 0, // Issues with accuracy. - HasNdtri = 0 - }; -}; - -template <> -struct unpacket_traits<Packet4bf> : neon_unpacket_default<Packet4bf, bfloat16> {}; - -namespace detail { -template <> -EIGEN_ALWAYS_INLINE void zip_in_place<Packet4bf>(Packet4bf& p1, Packet4bf& p2) { - const uint16x4x2_t tmp = vzip_u16(p1, p2); - p1 = tmp.val[0]; - p2 = tmp.val[1]; -} -} // namespace detail - -EIGEN_STRONG_INLINE Packet4bf F32ToBf16(const Packet4f& p) { - // See the scalar implementation in BFloat16.h for a comprehensible explanation - // of this fast rounding algorithm - Packet4ui input = Packet4ui(vreinterpretq_u32_f32(p)); - - // lsb = (input >> 16) & 1 - Packet4ui lsb = vandq_u32(vshrq_n_u32(input, 16), vdupq_n_u32(1)); - - // rounding_bias = 0x7fff + lsb - Packet4ui rounding_bias = vaddq_u32(lsb, vdupq_n_u32(0x7fff)); - - // input += rounding_bias - input = vaddq_u32(input, rounding_bias); - - // input = input >> 16 - input = vshrq_n_u32(input, 16); - - // Replace float-nans by bfloat16-nans, that is 0x7fc0 - const Packet4ui bf16_nan = vdupq_n_u32(0x7fc0); - const Packet4ui mask = vceqq_f32(p, p); - input = vbslq_u32(mask, input, bf16_nan); - - // output = static_cast<uint16_t>(input) - return vmovn_u32(input); -} - -// Discard the low 16 bits of each float. Only valid when every lane already holds an exact -// bfloat16 value, in which case this agrees with F32ToBf16 but skips its rounding and NaN -// canonicalization. pmin/pmax qualify: they return one of their operands bit for bit. -EIGEN_STRONG_INLINE Packet4bf F32ToBf16Truncate(const Packet4f& p) { return vshrn_n_u32(vreinterpretq_u32_f32(p), 16); } - -EIGEN_STRONG_INLINE Packet4f Bf16ToF32(const Packet4bf& p) { - return Packet4f(vreinterpretq_f32_u32(vshlq_n_u32(vmovl_u16(p), 16))); -} - -EIGEN_STRONG_INLINE Packet4bf F32MaskToBf16Mask(const Packet4f& p) { return vmovn_u32(vreinterpretq_u32_f32(p)); } - -template <> -EIGEN_STRONG_INLINE Packet4bf pset1<Packet4bf>(const bfloat16& from) { - return Packet4bf(pset1<Packet4us>(from.value)); -} - -template <> -EIGEN_STRONG_INLINE bfloat16 pfirst<Packet4bf>(const Packet4bf& from) { - return bfloat16_impl::raw_uint16_to_bfloat16(static_cast<uint16_t>(pfirst<Packet4us>(Packet4us(from)))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pload<Packet4bf>(const bfloat16* from) { - return Packet4bf( - pload<Packet4us>(reinterpret_cast<const uint16_t*>(assume_aligned<unpacket_traits<Packet4bf>::alignment>(from)))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf ploadu<Packet4bf>(const bfloat16* from) { - return Packet4bf(ploadu<Packet4us>(reinterpret_cast<const uint16_t*>(from))); -} - -template <> -EIGEN_STRONG_INLINE void pstore<bfloat16>(bfloat16* to, const Packet4bf& from) { - EIGEN_DEBUG_ALIGNED_STORE vst1_u16( - reinterpret_cast<uint16_t*>(assume_aligned<unpacket_traits<Packet4bf>::alignment>(to)), from); -} - -template <> -EIGEN_STRONG_INLINE void pstoreu<bfloat16>(bfloat16* to, const Packet4bf& from) { - EIGEN_DEBUG_UNALIGNED_STORE vst1_u16(reinterpret_cast<uint16_t*>(to), from); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf ploaddup<Packet4bf>(const bfloat16* from) { - return Packet4bf(ploaddup<Packet4us>(reinterpret_cast<const uint16_t*>(from))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pabs(const Packet4bf& a) { - return Packet4bf(vand_u16(a, vdup_n_u16(0x7fff))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pmin<PropagateNumbers, Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16Truncate(pmin<PropagateNumbers, Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} -template <> -EIGEN_STRONG_INLINE Packet4bf pmin<PropagateNaN, Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16Truncate(pmin<PropagateNaN, Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pmin<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16Truncate(pmin<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pmax<PropagateNumbers, Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16Truncate(pmax<PropagateNumbers, Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} -template <> -EIGEN_STRONG_INLINE Packet4bf pmax<PropagateNaN, Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16Truncate(pmax<PropagateNaN, Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pmax<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16Truncate(pmax<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf plset<Packet4bf>(const bfloat16& a) { - return F32ToBf16(plset<Packet4f>(static_cast<float>(a))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf por(const Packet4bf& a, const Packet4bf& b) { - return Packet4bf(por<Packet4us>(Packet4us(a), Packet4us(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pxor(const Packet4bf& a, const Packet4bf& b) { - return Packet4bf(pxor<Packet4us>(Packet4us(a), Packet4us(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pand(const Packet4bf& a, const Packet4bf& b) { - return Packet4bf(pand<Packet4us>(Packet4us(a), Packet4us(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pandnot(const Packet4bf& a, const Packet4bf& b) { - return Packet4bf(pandnot<Packet4us>(Packet4us(a), Packet4us(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pnot(const Packet4bf& a) { - return Packet4bf(pnot<Packet4us>(Packet4us(a))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf ptrue(const Packet4bf& /*a*/) { - return Packet4bf(vdup_n_u16(0xffff)); -} - -template <> -EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Packet4bf pselect(const Packet4bf& mask, const Packet4bf& a, const Packet4bf& b) { - return Packet4bf(pselect<Packet4us>(Packet4us(mask), Packet4us(a), Packet4us(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf print<Packet4bf>(const Packet4bf& a) { - return F32ToBf16Truncate(print<Packet4f>(Bf16ToF32(a))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pfloor<Packet4bf>(const Packet4bf& a) { - return F32ToBf16Truncate(pfloor<Packet4f>(Bf16ToF32(a))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pceil<Packet4bf>(const Packet4bf& a) { - return F32ToBf16Truncate(pceil<Packet4f>(Bf16ToF32(a))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pround<Packet4bf>(const Packet4bf& a) { - return F32ToBf16Truncate(pround<Packet4f>(Bf16ToF32(a))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf ptrunc<Packet4bf>(const Packet4bf& a) { - return F32ToBf16Truncate(ptrunc<Packet4f>(Bf16ToF32(a))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf padd<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16(padd<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf psub<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16(psub<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pmul<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16(pmul<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pmadd<Packet4bf>(const Packet4bf& a, const Packet4bf& b, const Packet4bf& c) { - return F32ToBf16(pmadd<Packet4f>(Bf16ToF32(a), Bf16ToF32(b), Bf16ToF32(c))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pmsub<Packet4bf>(const Packet4bf& a, const Packet4bf& b, const Packet4bf& c) { - return F32ToBf16(pmsub<Packet4f>(Bf16ToF32(a), Bf16ToF32(b), Bf16ToF32(c))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pnmadd<Packet4bf>(const Packet4bf& a, const Packet4bf& b, const Packet4bf& c) { - return F32ToBf16(pnmadd<Packet4f>(Bf16ToF32(a), Bf16ToF32(b), Bf16ToF32(c))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pnmsub<Packet4bf>(const Packet4bf& a, const Packet4bf& b, const Packet4bf& c) { - return F32ToBf16(pnmsub<Packet4f>(Bf16ToF32(a), Bf16ToF32(b), Bf16ToF32(c))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pdiv<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16(pdiv<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pgather<bfloat16, Packet4bf>(const bfloat16* from, Index stride) { - return Packet4bf(pgather<uint16_t, Packet4us>(reinterpret_cast<const uint16_t*>(from), stride)); -} - -template <> -EIGEN_STRONG_INLINE void pscatter<bfloat16, Packet4bf>(bfloat16* to, const Packet4bf& from, Index stride) { - pscatter<uint16_t, Packet4us>(reinterpret_cast<uint16_t*>(to), Packet4us(from), stride); -} - -template <> -EIGEN_STRONG_INLINE bfloat16 predux<Packet4bf>(const Packet4bf& a) { - return static_cast<bfloat16>(predux<Packet4f>(Bf16ToF32(a))); -} - -template <> -EIGEN_STRONG_INLINE bfloat16 predux_max<Packet4bf>(const Packet4bf& a) { - return bfloat16_impl::exact_float_to_bfloat16(predux_max<Packet4f>(Bf16ToF32(a))); -} - -template <> -EIGEN_STRONG_INLINE bfloat16 predux_min<Packet4bf>(const Packet4bf& a) { - return bfloat16_impl::exact_float_to_bfloat16(predux_min<Packet4f>(Bf16ToF32(a))); -} - -template <> -EIGEN_STRONG_INLINE bfloat16 predux_mul<Packet4bf>(const Packet4bf& a) { - return static_cast<bfloat16>(predux_mul<Packet4f>(Bf16ToF32(a))); -} - -template <> -EIGEN_STRONG_INLINE bool predux_any(const Packet4bf& a) { - return vget_lane_u64(vreinterpret_u64_u16(Packet4us(a)), 0) != 0; -} - -template <> -EIGEN_STRONG_INLINE Packet4bf preverse<Packet4bf>(const Packet4bf& a) { - return Packet4bf(preverse<Packet4us>(Packet4us(a))); -} - -EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void ptranspose(PacketBlock<Packet4bf, 4>& kernel) { - detail::ptranspose_impl(kernel); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pabsdiff<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32ToBf16(pabsdiff<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pcmp_eq<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32MaskToBf16Mask(pcmp_eq<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pcmp_lt<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32MaskToBf16Mask(pcmp_lt<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pcmp_lt_or_nan<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32MaskToBf16Mask(pcmp_lt_or_nan<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -// Classify on the raw bits, as the scalar isinf/isnan/isfinite do: |a| ==, >, < 0x7f80. -template <> -EIGEN_STRONG_INLINE Packet4bf pisinf<Packet4bf>(const Packet4bf& a) { - constexpr uint16_t kInf = ((1 << 8) - 1) << 7; - constexpr uint16_t kAbsMask = (1 << 15) - 1; - return vceq_u16(vand_u16(a, vdup_n_u16(kAbsMask)), vdup_n_u16(kInf)); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pisnan<Packet4bf>(const Packet4bf& a) { - constexpr uint16_t kInf = ((1 << 8) - 1) << 7; - constexpr uint16_t kAbsMask = (1 << 15) - 1; - return vcgt_u16(vand_u16(a, vdup_n_u16(kAbsMask)), vdup_n_u16(kInf)); -} - -// Compare encoded lanes: widening bf16 subnormals to float loses them under DAZ/FZ. -template <> -EIGEN_STRONG_INLINE Packet4bf psign<Packet4bf>(const Packet4bf& a) { - const uint16x4_t abs_mask = vdup_n_u16(0x7fff); - const uint16x4_t is_nan = vcgt_u16(vand_u16(a, abs_mask), vdup_n_u16(0x7f80)); - const uint16x4_t keep = vorr_u16(is_nan, vdup_n_u16(0x8000)); - return vand_u16(vbsl_u16(keep, a, vdup_n_u16(0x3f80)), vtst_u16(a, abs_mask)); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pisfinite<Packet4bf>(const Packet4bf& a) { - constexpr uint16_t kInf = ((1 << 8) - 1) << 7; - constexpr uint16_t kAbsMask = (1 << 15) - 1; - return vclt_u16(vand_u16(a, vdup_n_u16(kAbsMask)), vdup_n_u16(kInf)); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pcmp_le<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { - return F32MaskToBf16Mask(pcmp_le<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); -} - -template <> -EIGEN_STRONG_INLINE Packet4bf pnegate<Packet4bf>(const Packet4bf& a) { - return Packet4bf(pxor<Packet4us>(Packet4us(a), pset1<Packet4us>(static_cast<uint16_t>(0x8000)))); -} - //---------- double ---------- #if EIGEN_ARCH_ARM64
diff --git a/Eigen/src/Core/arch/NEON/PacketMathBF16.h b/Eigen/src/Core/arch/NEON/PacketMathBF16.h new file mode 100644 index 0000000..3f64f87 --- /dev/null +++ b/Eigen/src/Core/arch/NEON/PacketMathBF16.h
@@ -0,0 +1,935 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// This Source Code Form is subject to the terms of the Mozilla +// Public License v. 2.0. If a copy of the MPL was not distributed +// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_PACKET_MATH_BF16_NEON_H +#define EIGEN_PACKET_MATH_BF16_NEON_H + +// IWYU pragma: private +#include "../../InternalHeaderCheck.h" + +namespace Eigen { + +namespace internal { + +// TODO: Add support for native armv8.6-a bfloat16_t + +// TODO: Guard if we have native bfloat16 support +typedef eigen_packet_wrapper<uint16x4_t, 19> Packet4bf; +using Packet8bf = eigen_packet_wrapper<uint16x8_t, 20>; + +template <> +struct is_arithmetic<Packet4bf> : std::true_type {}; +template <> +struct is_arithmetic<Packet8bf> : std::true_type {}; + +// Packet8bf is the full packet: every bfloat16 op widens to float, and the float work of two +// Packet4f halves overlaps where one Packet4bf's chain would leave the pipes idle. +template <> +struct packet_traits<bfloat16> : default_packet_traits { + using type = Packet8bf; + using half = Packet4bf; + enum { + Vectorizable = 1, + AlignedOnScalar = 1, + size = 8, + + HasCmp = 1, + HasAdd = 1, + HasSub = 1, + HasShift = 1, + HasMul = 1, + HasNegate = 1, + HasAbs = 1, + HasArg = 0, + HasAbsDiff = 1, + HasMin = 1, + HasMax = 1, + HasConj = 1, + HasSetLinear = 1, + HasDiv = 1, + HasSin = EIGEN_FAST_MATH, + HasCos = EIGEN_FAST_MATH, + HasLog = 1, + HasExp = 1, + HasSqrt = 1, + HasRsqrt = 1, + HasTanh = EIGEN_FAST_MATH, + HasErf = EIGEN_FAST_MATH, + HasBessel = 0, // Issues with accuracy. + HasNdtri = 0 + }; +}; + +template <> +struct unpacket_traits<Packet4bf> : neon_unpacket_default<Packet4bf, bfloat16> {}; +template <> +struct unpacket_traits<Packet8bf> : neon_unpacket_default<Packet8bf, bfloat16> { + using half = Packet4bf; +}; + +namespace detail { +template <> +EIGEN_ALWAYS_INLINE void zip_in_place<Packet4bf>(Packet4bf& p1, Packet4bf& p2) { + const uint16x4x2_t tmp = vzip_u16(p1, p2); + p1 = tmp.val[0]; + p2 = tmp.val[1]; +} +template <> +EIGEN_ALWAYS_INLINE void zip_in_place<Packet8bf>(Packet8bf& p1, Packet8bf& p2) { + const uint16x8x2_t tmp = vzipq_u16(p1, p2); + p1 = tmp.val[0]; + p2 = tmp.val[1]; +} +} // namespace detail + +// Round to nearest even on the bfloat16 boundary. With lsb the lowest bit kept, the scalar +// formula is (p + 0x7fff + lsb) >> 16, and RADDHN computes (p + d + 0x8000) >> 16, so d = lsb - 1, +// that is -1 where lsb is clear. Written as an add and RSHRN, the add stays a separate op in +// loops with GCC and clang on both targets. The caller guarantees that every NaN lane has a zero low +// half, which the FPU's NaN propagation does for bfloat16 operands: the rounding then leaves the +// NaN's high half alone. A NaN with a nonzero low half can instead round to +-inf or carry into a +// zero. +EIGEN_STRONG_INLINE Packet4bf F32ToBf16KeepNaN(const Packet4f& p) { + Packet4ui bits = vreinterpretq_u32_f32(p); + Packet4ui round_down = vceqq_u32(vandq_u32(bits, vdupq_n_u32(0x10000)), vdupq_n_u32(0)); + return vraddhn_u32(bits, round_down); +} + +// F32ToBf16KeepNaN for any float: NaN lanes become the bfloat16 NaN 0x7fc0, as on AVX. The scalar +// conversion keeps the NaN's sign; doing that here costs an op on the critical path of every round +// trip, 25% of a cast loop and 6% of exp on a Cortex-A72, for a bit that carries no value. +// |p| <= inf is false only for NaN. The NaN lanes are selected after narrowing, so the rounding +// reads the raw bits. Selecting first costs clang an extra AND and MOV. +EIGEN_STRONG_INLINE Packet4bf F32ToBf16(const Packet4f& p) { + uint16x4_t not_nan = vmovn_u32(vcageq_f32(pinf<Packet4f>(), p)); + return vbsl_u16(not_nan, F32ToBf16KeepNaN(p), vdup_n_u16(0x7fc0)); +} + +// Discard the low 16 bits of each float. Only valid when every lane already holds an exact +// bfloat16 value, in which case this agrees with F32ToBf16 but skips its rounding and NaN +// canonicalization. pmin/pmax qualify: they return one of their operands bit for bit. +EIGEN_STRONG_INLINE Packet4bf F32ToBf16Truncate(const Packet4f& p) { return vshrn_n_u32(vreinterpretq_u32_f32(p), 16); } + +EIGEN_STRONG_INLINE Packet4f Bf16ToF32(const Packet4bf& p) { + return Packet4f(vreinterpretq_f32_u32(vshlq_n_u32(vmovl_u16(p), 16))); +} + +EIGEN_STRONG_INLINE Packet4bf F32MaskToBf16Mask(const Packet4f& p) { return vmovn_u32(vreinterpretq_u32_f32(p)); } + +// Packet8bf conversions work on two Packet4f halves, lanes 0-3 and 4-7, with the same rounding as +// the Packet4bf ones. +EIGEN_STRONG_INLINE Packet4f Bf16ToF32Low(const Packet8bf& p) { + return vreinterpretq_f32_u32(vshll_n_u16(vget_low_u16(p), 16)); +} + +EIGEN_STRONG_INLINE Packet4f Bf16ToF32High(const Packet8bf& p) { +#if EIGEN_ARCH_ARM64 + return vreinterpretq_f32_u32(vshll_high_n_u16(p, 16)); +#else + return vreinterpretq_f32_u32(vshll_n_u16(vget_high_u16(p), 16)); +#endif +} + +// (v + 0x8000) >> 16 on each lane of lo and hi. +EIGEN_STRONG_INLINE Packet8bf RoundingNarrowBf16(const Packet4ui& lo, const Packet4ui& hi) { +#if EIGEN_ARCH_ARM64 + return vrshrn_high_n_u32(vrshrn_n_u32(lo, 16), hi, 16); +#else + return vcombine_u16(vrshrn_n_u32(lo, 16), vrshrn_n_u32(hi, 16)); +#endif +} + +EIGEN_STRONG_INLINE Packet8bf F32ToBf16KeepNaN(const Packet4f& lo, const Packet4f& hi) { + Packet4ui lo_bits = vreinterpretq_u32_f32(lo); + Packet4ui hi_bits = vreinterpretq_u32_f32(hi); + Packet4ui lo_round_down = vceqq_u32(vandq_u32(lo_bits, vdupq_n_u32(0x10000)), vdupq_n_u32(0)); + Packet4ui hi_round_down = vceqq_u32(vandq_u32(hi_bits, vdupq_n_u32(0x10000)), vdupq_n_u32(0)); +#if EIGEN_ARCH_ARM64 + return vraddhn_high_u32(vraddhn_u32(lo_bits, lo_round_down), hi_bits, hi_round_down); +#else + return vcombine_u16(vraddhn_u32(lo_bits, lo_round_down), vraddhn_u32(hi_bits, hi_round_down)); +#endif +} + +// As the Packet4bf F32ToBf16, with one UZP1 narrowing both NaN masks. +EIGEN_STRONG_INLINE Packet8bf F32ToBf16(const Packet4f& lo, const Packet4f& hi) { + Packet4f inf = pinf<Packet4f>(); + uint16x8_t not_nan = + vuzpq_u16(vreinterpretq_u16_u32(vcageq_f32(inf, lo)), vreinterpretq_u16_u32(vcageq_f32(inf, hi))).val[0]; + return vbslq_u16(not_nan, F32ToBf16KeepNaN(lo, hi), vdupq_n_u16(0x7fc0)); +} + +// The high and low 16 bits of every float: one UZP2 or UZP1 on AArch64, one VUZP on ARMv7. +EIGEN_STRONG_INLINE Packet8bf F32ToBf16Truncate(const Packet4f& lo, const Packet4f& hi) { + return vuzpq_u16(vreinterpretq_u16_f32(lo), vreinterpretq_u16_f32(hi)).val[1]; +} + +EIGEN_STRONG_INLINE Packet8bf F32MaskToBf16Mask(const Packet4f& lo, const Packet4f& hi) { + return vuzpq_u16(vreinterpretq_u16_f32(lo), vreinterpretq_u16_f32(hi)).val[0]; +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pset1<Packet4bf>(const bfloat16& from) { + return Packet4bf(pset1<Packet4us>(from.value)); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 pfirst<Packet4bf>(const Packet4bf& from) { + return bfloat16_impl::raw_uint16_to_bfloat16(static_cast<uint16_t>(pfirst<Packet4us>(Packet4us(from)))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pload<Packet4bf>(const bfloat16* from) { + return Packet4bf( + pload<Packet4us>(reinterpret_cast<const uint16_t*>(assume_aligned<unpacket_traits<Packet4bf>::alignment>(from)))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf ploadu<Packet4bf>(const bfloat16* from) { + return Packet4bf(ploadu<Packet4us>(reinterpret_cast<const uint16_t*>(from))); +} + +template <> +EIGEN_STRONG_INLINE void pstore<bfloat16>(bfloat16* to, const Packet4bf& from) { + EIGEN_DEBUG_ALIGNED_STORE vst1_u16( + reinterpret_cast<uint16_t*>(assume_aligned<unpacket_traits<Packet4bf>::alignment>(to)), from); +} + +template <> +EIGEN_STRONG_INLINE void pstoreu<bfloat16>(bfloat16* to, const Packet4bf& from) { + EIGEN_DEBUG_UNALIGNED_STORE vst1_u16(reinterpret_cast<uint16_t*>(to), from); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf ploaddup<Packet4bf>(const bfloat16* from) { + return Packet4bf(ploaddup<Packet4us>(reinterpret_cast<const uint16_t*>(from))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pabs(const Packet4bf& a) { + return Packet4bf(vand_u16(a, vdup_n_u16(0x7fff))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pmin<PropagateNumbers, Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16Truncate(pmin<PropagateNumbers, Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} +template <> +EIGEN_STRONG_INLINE Packet4bf pmin<PropagateNaN, Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16Truncate(pmin<PropagateNaN, Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pmin<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16Truncate(pmin<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pmax<PropagateNumbers, Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16Truncate(pmax<PropagateNumbers, Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} +template <> +EIGEN_STRONG_INLINE Packet4bf pmax<PropagateNaN, Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16Truncate(pmax<PropagateNaN, Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pmax<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16Truncate(pmax<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf plset<Packet4bf>(const bfloat16& a) { + return F32ToBf16(plset<Packet4f>(static_cast<float>(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf por(const Packet4bf& a, const Packet4bf& b) { + return Packet4bf(por<Packet4us>(Packet4us(a), Packet4us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pxor(const Packet4bf& a, const Packet4bf& b) { + return Packet4bf(pxor<Packet4us>(Packet4us(a), Packet4us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pand(const Packet4bf& a, const Packet4bf& b) { + return Packet4bf(pand<Packet4us>(Packet4us(a), Packet4us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pandnot(const Packet4bf& a, const Packet4bf& b) { + return Packet4bf(pandnot<Packet4us>(Packet4us(a), Packet4us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pnot(const Packet4bf& a) { + return Packet4bf(pnot<Packet4us>(Packet4us(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf ptrue(const Packet4bf& /*a*/) { + return Packet4bf(vdup_n_u16(0xffff)); +} + +template <> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Packet4bf pselect(const Packet4bf& mask, const Packet4bf& a, const Packet4bf& b) { + return Packet4bf(pselect<Packet4us>(Packet4us(mask), Packet4us(a), Packet4us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf print<Packet4bf>(const Packet4bf& a) { + return F32ToBf16Truncate(print<Packet4f>(Bf16ToF32(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pfloor<Packet4bf>(const Packet4bf& a) { + return F32ToBf16Truncate(pfloor<Packet4f>(Bf16ToF32(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pceil<Packet4bf>(const Packet4bf& a) { + return F32ToBf16Truncate(pceil<Packet4f>(Bf16ToF32(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pround<Packet4bf>(const Packet4bf& a) { + return F32ToBf16Truncate(pround<Packet4f>(Bf16ToF32(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf ptrunc<Packet4bf>(const Packet4bf& a) { + return F32ToBf16Truncate(ptrunc<Packet4f>(Bf16ToF32(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf padd<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16KeepNaN(padd<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf psub<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16KeepNaN(psub<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pmul<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16KeepNaN(pmul<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pmadd<Packet4bf>(const Packet4bf& a, const Packet4bf& b, const Packet4bf& c) { + return F32ToBf16KeepNaN(pmadd<Packet4f>(Bf16ToF32(a), Bf16ToF32(b), Bf16ToF32(c))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pmsub<Packet4bf>(const Packet4bf& a, const Packet4bf& b, const Packet4bf& c) { + return F32ToBf16KeepNaN(pmsub<Packet4f>(Bf16ToF32(a), Bf16ToF32(b), Bf16ToF32(c))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pnmadd<Packet4bf>(const Packet4bf& a, const Packet4bf& b, const Packet4bf& c) { + return F32ToBf16KeepNaN(pnmadd<Packet4f>(Bf16ToF32(a), Bf16ToF32(b), Bf16ToF32(c))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pnmsub<Packet4bf>(const Packet4bf& a, const Packet4bf& b, const Packet4bf& c) { + return F32ToBf16KeepNaN(pnmsub<Packet4f>(Bf16ToF32(a), Bf16ToF32(b), Bf16ToF32(c))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pdiv<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16KeepNaN(pdiv<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pgather<bfloat16, Packet4bf>(const bfloat16* from, Index stride) { + return Packet4bf(pgather<uint16_t, Packet4us>(reinterpret_cast<const uint16_t*>(from), stride)); +} + +template <> +EIGEN_STRONG_INLINE void pscatter<bfloat16, Packet4bf>(bfloat16* to, const Packet4bf& from, Index stride) { + pscatter<uint16_t, Packet4us>(reinterpret_cast<uint16_t*>(to), Packet4us(from), stride); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 predux<Packet4bf>(const Packet4bf& a) { + return static_cast<bfloat16>(predux<Packet4f>(Bf16ToF32(a))); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 predux_max<Packet4bf>(const Packet4bf& a) { + return bfloat16_impl::exact_float_to_bfloat16(predux_max<Packet4f>(Bf16ToF32(a))); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 predux_min<Packet4bf>(const Packet4bf& a) { + return bfloat16_impl::exact_float_to_bfloat16(predux_min<Packet4f>(Bf16ToF32(a))); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 predux_mul<Packet4bf>(const Packet4bf& a) { + return static_cast<bfloat16>(predux_mul<Packet4f>(Bf16ToF32(a))); +} + +template <> +EIGEN_STRONG_INLINE bool predux_any(const Packet4bf& a) { + return vget_lane_u64(vreinterpret_u64_u16(Packet4us(a)), 0) != 0; +} + +template <> +EIGEN_STRONG_INLINE Packet4bf preverse<Packet4bf>(const Packet4bf& a) { + return Packet4bf(preverse<Packet4us>(Packet4us(a))); +} + +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void ptranspose(PacketBlock<Packet4bf, 4>& kernel) { + detail::ptranspose_impl(kernel); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pabsdiff<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32ToBf16KeepNaN(pabsdiff<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pcmp_eq<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32MaskToBf16Mask(pcmp_eq<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pcmp_lt<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32MaskToBf16Mask(pcmp_lt<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pcmp_lt_or_nan<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32MaskToBf16Mask(pcmp_lt_or_nan<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +// Classify on the raw bits, as the scalar isinf/isnan/isfinite do: |a| ==, >, < 0x7f80. +template <> +EIGEN_STRONG_INLINE Packet4bf pisinf<Packet4bf>(const Packet4bf& a) { + constexpr uint16_t kInf = ((1 << 8) - 1) << 7; + constexpr uint16_t kAbsMask = (1 << 15) - 1; + return vceq_u16(vand_u16(a, vdup_n_u16(kAbsMask)), vdup_n_u16(kInf)); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pisnan<Packet4bf>(const Packet4bf& a) { + constexpr uint16_t kInf = ((1 << 8) - 1) << 7; + constexpr uint16_t kAbsMask = (1 << 15) - 1; + return vcgt_u16(vand_u16(a, vdup_n_u16(kAbsMask)), vdup_n_u16(kInf)); +} + +// Compare encoded lanes: widening bf16 subnormals to float loses them under DAZ/FZ. +template <> +EIGEN_STRONG_INLINE Packet4bf psign<Packet4bf>(const Packet4bf& a) { + const uint16x4_t abs_mask = vdup_n_u16(0x7fff); + const uint16x4_t is_nan = vcgt_u16(vand_u16(a, abs_mask), vdup_n_u16(0x7f80)); + const uint16x4_t keep = vorr_u16(is_nan, vdup_n_u16(0x8000)); + return vand_u16(vbsl_u16(keep, a, vdup_n_u16(0x3f80)), vtst_u16(a, abs_mask)); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pisfinite<Packet4bf>(const Packet4bf& a) { + constexpr uint16_t kInf = ((1 << 8) - 1) << 7; + constexpr uint16_t kAbsMask = (1 << 15) - 1; + return vclt_u16(vand_u16(a, vdup_n_u16(kAbsMask)), vdup_n_u16(kInf)); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pcmp_le<Packet4bf>(const Packet4bf& a, const Packet4bf& b) { + return F32MaskToBf16Mask(pcmp_le<Packet4f>(Bf16ToF32(a), Bf16ToF32(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pnegate<Packet4bf>(const Packet4bf& a) { + return Packet4bf(pxor<Packet4us>(Packet4us(a), pset1<Packet4us>(static_cast<uint16_t>(0x8000)))); +} + +//---------- Packet8bf ---------- + +template <> +EIGEN_STRONG_INLINE Packet8bf pset1<Packet8bf>(const bfloat16& from) { + return Packet8bf(pset1<Packet8us>(from.value)); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 pfirst<Packet8bf>(const Packet8bf& from) { + return bfloat16_impl::raw_uint16_to_bfloat16(static_cast<uint16_t>(pfirst<Packet8us>(Packet8us(from)))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pload<Packet8bf>(const bfloat16* from) { + return Packet8bf( + pload<Packet8us>(reinterpret_cast<const uint16_t*>(assume_aligned<unpacket_traits<Packet8bf>::alignment>(from)))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf ploadu<Packet8bf>(const bfloat16* from) { + return Packet8bf(ploadu<Packet8us>(reinterpret_cast<const uint16_t*>(from))); +} + +template <> +EIGEN_STRONG_INLINE void pstore<bfloat16>(bfloat16* to, const Packet8bf& from) { + EIGEN_DEBUG_ALIGNED_STORE vst1q_u16( + reinterpret_cast<uint16_t*>(assume_aligned<unpacket_traits<Packet8bf>::alignment>(to)), from); +} + +template <> +EIGEN_STRONG_INLINE void pstoreu<bfloat16>(bfloat16* to, const Packet8bf& from) { + EIGEN_DEBUG_UNALIGNED_STORE vst1q_u16(reinterpret_cast<uint16_t*>(to), from); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf ploaddup<Packet8bf>(const bfloat16* from) { + return Packet8bf(ploaddup<Packet8us>(reinterpret_cast<const uint16_t*>(from))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf ploadquad<Packet8bf>(const bfloat16* from) { + return Packet8bf(ploadquad<Packet8us>(reinterpret_cast<const uint16_t*>(from))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pabs(const Packet8bf& a) { + return Packet8bf(vandq_u16(a, vdupq_n_u16(0x7fff))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pmin<PropagateNumbers, Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16Truncate(pmin<PropagateNumbers, Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pmin<PropagateNumbers, Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} +template <> +EIGEN_STRONG_INLINE Packet8bf pmin<PropagateNaN, Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16Truncate(pmin<PropagateNaN, Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pmin<PropagateNaN, Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pmin<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16Truncate(pmin<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pmin<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pmax<PropagateNumbers, Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16Truncate(pmax<PropagateNumbers, Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pmax<PropagateNumbers, Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} +template <> +EIGEN_STRONG_INLINE Packet8bf pmax<PropagateNaN, Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16Truncate(pmax<PropagateNaN, Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pmax<PropagateNaN, Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pmax<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16Truncate(pmax<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pmax<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf plset<Packet8bf>(const bfloat16& a) { + Packet4f lo = plset<Packet4f>(static_cast<float>(a)); + return F32ToBf16(lo, padd(lo, pset1<Packet4f>(4.0f))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf por(const Packet8bf& a, const Packet8bf& b) { + return Packet8bf(por<Packet8us>(Packet8us(a), Packet8us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pxor(const Packet8bf& a, const Packet8bf& b) { + return Packet8bf(pxor<Packet8us>(Packet8us(a), Packet8us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pand(const Packet8bf& a, const Packet8bf& b) { + return Packet8bf(pand<Packet8us>(Packet8us(a), Packet8us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pandnot(const Packet8bf& a, const Packet8bf& b) { + return Packet8bf(pandnot<Packet8us>(Packet8us(a), Packet8us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pnot(const Packet8bf& a) { + return Packet8bf(pnot<Packet8us>(Packet8us(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf ptrue(const Packet8bf& /*a*/) { + return Packet8bf(vdupq_n_u16(0xffff)); +} + +template <> +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Packet8bf pselect(const Packet8bf& mask, const Packet8bf& a, const Packet8bf& b) { + return Packet8bf(pselect<Packet8us>(Packet8us(mask), Packet8us(a), Packet8us(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf print<Packet8bf>(const Packet8bf& a) { + return F32ToBf16Truncate(print<Packet4f>(Bf16ToF32Low(a)), print<Packet4f>(Bf16ToF32High(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pfloor<Packet8bf>(const Packet8bf& a) { + return F32ToBf16Truncate(pfloor<Packet4f>(Bf16ToF32Low(a)), pfloor<Packet4f>(Bf16ToF32High(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pceil<Packet8bf>(const Packet8bf& a) { + return F32ToBf16Truncate(pceil<Packet4f>(Bf16ToF32Low(a)), pceil<Packet4f>(Bf16ToF32High(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pround<Packet8bf>(const Packet8bf& a) { + return F32ToBf16Truncate(pround<Packet4f>(Bf16ToF32Low(a)), pround<Packet4f>(Bf16ToF32High(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf ptrunc<Packet8bf>(const Packet8bf& a) { + return F32ToBf16Truncate(ptrunc<Packet4f>(Bf16ToF32Low(a)), ptrunc<Packet4f>(Bf16ToF32High(a))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf padd<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16KeepNaN(padd<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + padd<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf psub<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16KeepNaN(psub<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + psub<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pmul<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16KeepNaN(pmul<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pmul<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pmadd<Packet8bf>(const Packet8bf& a, const Packet8bf& b, const Packet8bf& c) { + return F32ToBf16KeepNaN(pmadd<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b), Bf16ToF32Low(c)), + pmadd<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b), Bf16ToF32High(c))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pmsub<Packet8bf>(const Packet8bf& a, const Packet8bf& b, const Packet8bf& c) { + return F32ToBf16KeepNaN(pmsub<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b), Bf16ToF32Low(c)), + pmsub<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b), Bf16ToF32High(c))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pnmadd<Packet8bf>(const Packet8bf& a, const Packet8bf& b, const Packet8bf& c) { + return F32ToBf16KeepNaN(pnmadd<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b), Bf16ToF32Low(c)), + pnmadd<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b), Bf16ToF32High(c))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pnmsub<Packet8bf>(const Packet8bf& a, const Packet8bf& b, const Packet8bf& c) { + return F32ToBf16KeepNaN(pnmsub<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b), Bf16ToF32Low(c)), + pnmsub<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b), Bf16ToF32High(c))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pdiv<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16KeepNaN(pdiv<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pdiv<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pgather<bfloat16, Packet8bf>(const bfloat16* from, Index stride) { + return Packet8bf(pgather<uint16_t, Packet8us>(reinterpret_cast<const uint16_t*>(from), stride)); +} + +template <> +EIGEN_STRONG_INLINE void pscatter<bfloat16, Packet8bf>(bfloat16* to, const Packet8bf& from, Index stride) { + pscatter<uint16_t, Packet8us>(reinterpret_cast<uint16_t*>(to), Packet8us(from), stride); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 predux<Packet8bf>(const Packet8bf& a) { + return static_cast<bfloat16>(predux<Packet4f>(padd<Packet4f>(Bf16ToF32Low(a), Bf16ToF32High(a)))); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 predux_max<Packet8bf>(const Packet8bf& a) { + return bfloat16_impl::exact_float_to_bfloat16( + predux_max<Packet4f>(pmax<Packet4f>(Bf16ToF32Low(a), Bf16ToF32High(a)))); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 predux_min<Packet8bf>(const Packet8bf& a) { + return bfloat16_impl::exact_float_to_bfloat16( + predux_min<Packet4f>(pmin<Packet4f>(Bf16ToF32Low(a), Bf16ToF32High(a)))); +} + +template <> +EIGEN_STRONG_INLINE bfloat16 predux_mul<Packet8bf>(const Packet8bf& a) { + return static_cast<bfloat16>(predux_mul<Packet4f>(pmul<Packet4f>(Bf16ToF32Low(a), Bf16ToF32High(a)))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf predux_half<Packet8bf>(const Packet8bf& a) { + return F32ToBf16KeepNaN(padd<Packet4f>(Bf16ToF32Low(a), Bf16ToF32High(a))); +} + +template <> +EIGEN_STRONG_INLINE bool predux_any(const Packet8bf& a) { + return predux_any(Packet8us(a)); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf preverse<Packet8bf>(const Packet8bf& a) { + return Packet8bf(preverse<Packet8us>(Packet8us(a))); +} + +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void ptranspose(PacketBlock<Packet8bf, 4>& kernel) { + detail::ptranspose_impl(kernel); +} + +EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void ptranspose(PacketBlock<Packet8bf, 8>& kernel) { + detail::ptranspose_impl(kernel); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pabsdiff<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32ToBf16KeepNaN(pabsdiff<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pabsdiff<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pcmp_eq<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32MaskToBf16Mask(pcmp_eq<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pcmp_eq<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pcmp_lt<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32MaskToBf16Mask(pcmp_lt<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pcmp_lt<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pcmp_lt_or_nan<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + // Negate a >= b once on the narrowed mask rather than on each float half. + return vmvnq_u16(F32MaskToBf16Mask(pcmp_le<Packet4f>(Bf16ToF32Low(b), Bf16ToF32Low(a)), + pcmp_le<Packet4f>(Bf16ToF32High(b), Bf16ToF32High(a)))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pcmp_le<Packet8bf>(const Packet8bf& a, const Packet8bf& b) { + return F32MaskToBf16Mask(pcmp_le<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(b)), + pcmp_le<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(b))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pisinf<Packet8bf>(const Packet8bf& a) { + constexpr uint16_t kInf = ((1 << 8) - 1) << 7; + constexpr uint16_t kAbsMask = (1 << 15) - 1; + return vceqq_u16(vandq_u16(a, vdupq_n_u16(kAbsMask)), vdupq_n_u16(kInf)); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pisnan<Packet8bf>(const Packet8bf& a) { + constexpr uint16_t kInf = ((1 << 8) - 1) << 7; + constexpr uint16_t kAbsMask = (1 << 15) - 1; + return vcgtq_u16(vandq_u16(a, vdupq_n_u16(kAbsMask)), vdupq_n_u16(kInf)); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pisfinite<Packet8bf>(const Packet8bf& a) { + constexpr uint16_t kInf = ((1 << 8) - 1) << 7; + constexpr uint16_t kAbsMask = (1 << 15) - 1; + return vcltq_u16(vandq_u16(a, vdupq_n_u16(kAbsMask)), vdupq_n_u16(kInf)); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf psign<Packet8bf>(const Packet8bf& a) { + uint16x8_t abs_mask = vdupq_n_u16(0x7fff); + uint16x8_t is_nan = vcgtq_u16(vandq_u16(a, abs_mask), vdupq_n_u16(0x7f80)); + uint16x8_t keep = vorrq_u16(is_nan, vdupq_n_u16(0x8000)); + return vandq_u16(vbslq_u16(keep, a, vdupq_n_u16(0x3f80)), vtstq_u16(a, abs_mask)); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pnegate<Packet8bf>(const Packet8bf& a) { + return Packet8bf(pxor<Packet8us>(Packet8us(a), pset1<Packet8us>(static_cast<uint16_t>(0x8000)))); +} + +// psqrt and prsqrt have their own kernels below. +EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_BF16(Packet4f, Packet4bf) +EIGEN_INSTANTIATE_GENERIC_MATH_FUNCS_BF16_WITH(BF16_PACKET_FUNCTION_HALVES, Packet4f, Packet8bf) + +template <> +EIGEN_STRONG_INLINE Packet4bf psqrt<Packet4bf>(const Packet4bf& a) { + // y = x r with r = FRSQRTE(x), then one FRSQRTS step y <- y (3 - y r) / 2. FRSQRTE is specified + // bit for bit, and rounding y to bfloat16 equals bfloat16(sqrtf(x)) for all 65536 inputs + // (packetmath checks them all); the raw estimate alone misrounds 7366 of them. RSHRN adds half a + // bfloat16 ulp and truncates. That is round half up, which differs from F32ToBf16 only on an + // exact float midpoint, and no input produces one; NaNs stay NaN since their low half is zero. + Packet4f x = Bf16ToF32(a); + Packet4f r = vrsqrteq_f32(x); + Packet4f y = pmul(x, r); + y = pmul(y, vrsqrtsq_f32(y, r)); + Packet4bf ybf = vrshrn_n_u32(vreinterpretq_u32_f32(y), 16); + // y is NaN where r is +-inf (x is +-0, or a subnormal the FPU flushes) or +0 (x is +inf), and + // the answer there is the sign and exponent of a. Reading r rather than a keeps that right under + // FPCR.FZ and on ARMv7. The high half of r plus 0x80 is 0x80, 0 or 0x8000 on exactly those + // lanes; every other r, a positive normal or a NaN, has a bit set under 0x7f7f. + uint16x4_t u = vadd_u16(vshrn_n_u32(vreinterpretq_u32_f32(r), 16), vdup_n_u16(0x80)); + uint16x4_t step_lane = vtst_u16(u, vdup_n_u16(0x7f7f)); + return vbsl_u16(step_lane, ybf, vand_u16(a, vdup_n_u16(0xff80))); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf prsqrt<Packet4bf>(const Packet4bf& a) { + // r = FRSQRTE(x) refined by one FRSQRTS step, r1 = r (3 - x r r) / 2, then the same RSHRN + // narrowing as psqrt. r1 rounds to the correctly rounded bfloat16 rsqrt for all 65536 inputs + // (packetmath checks them all). FRSQRTE has 8 fraction bits, so the raw estimate always lands on + // a bfloat16 value or midpoint and misrounds 1270 of them. + Packet4f x = Bf16ToF32(a); + Packet4f r = vrsqrteq_f32(x); +#if EIGEN_ARCH_ARM64 + // FMULX gives 2 for 0 * inf where FMUL gives NaN, so no lane needs a select. Where the FPU reads + // x as +-0 (a subnormal too under FPCR.FZ), r is +-inf, y = 2 and the step is -+inf, and |step| + // restores r1 = +-inf. Where x is +inf, r is 0, y = 2 and r1 = 0 * 1.5. Negative and NaN x stay + // NaN. On every other lane FMULX is FMUL and the step is positive, so r1 is the plain step. + Packet4f y = vmulxq_f32(x, r); + Packet4f r1 = pmul(r, vabsq_f32(vrsqrtsq_f32(y, r))); + return vrshrn_n_u32(vreinterpretq_u32_f32(r1), 16); +#else + // ARMv7 has no FMULX. r1 is NaN where x is +-0, +inf or a subnormal, which ARMv7 NEON flushes, + // and there r is already the answer (+-inf, 0, +-inf) with a zero low half, so those lanes take + // r truncated. They are the bit patterns with a - 0x80 >= 0x7f00 unsigned: the exponent-0 range + // wraps around, and +inf, NaN and every negative value sit above. + Packet4f r1 = pmul(r, vrsqrtsq_f32(pmul(x, r), r)); + Packet4bf refined = vrshrn_n_u32(vreinterpretq_u32_f32(r1), 16); + Packet4bf raw = vshrn_n_u32(vreinterpretq_u32_f32(r), 16); + uint16x4_t special = vcge_u16(vsub_u16(a, vdup_n_u16(0x80)), vdup_n_u16(0x7f00)); + return vbsl_u16(special, raw, refined); +#endif +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pfrexp(const Packet4bf& a, Packet4bf& exponent) { + // Both results are exact: the mantissa keeps the input's significand, and the exponent is a + // small integer. + Packet4f fexponent; + const Packet4bf out = F32ToBf16Truncate(pfrexp<Packet4f>(Bf16ToF32(a), fexponent)); + exponent = F32ToBf16Truncate(fexponent); + return out; +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pldexp(const Packet4bf& a, const Packet4bf& exponent) { + return F32ToBf16(pldexp<Packet4f>(Bf16ToF32(a), Bf16ToF32(exponent))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf psqrt<Packet8bf>(const Packet8bf& a) { + // psqrt<Packet4bf> on each float half, narrowed and selected once over all 8 lanes. + Packet4f x_lo = Bf16ToF32Low(a); + Packet4f x_hi = Bf16ToF32High(a); + Packet4f r_lo = vrsqrteq_f32(x_lo); + Packet4f r_hi = vrsqrteq_f32(x_hi); + Packet4f y_lo = pmul(x_lo, r_lo); + Packet4f y_hi = pmul(x_hi, r_hi); + y_lo = pmul(y_lo, vrsqrtsq_f32(y_lo, r_lo)); + y_hi = pmul(y_hi, vrsqrtsq_f32(y_hi, r_hi)); + Packet8bf ybf = RoundingNarrowBf16(vreinterpretq_u32_f32(y_lo), vreinterpretq_u32_f32(y_hi)); + uint16x8_t u = vaddq_u16(F32ToBf16Truncate(r_lo, r_hi), vdupq_n_u16(0x80)); + uint16x8_t step_lane = vtstq_u16(u, vdupq_n_u16(0x7f7f)); + return vbslq_u16(step_lane, ybf, vandq_u16(a, vdupq_n_u16(0xff80))); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf prsqrt<Packet8bf>(const Packet8bf& a) { + // prsqrt<Packet4bf> on each float half. + Packet4f x_lo = Bf16ToF32Low(a); + Packet4f x_hi = Bf16ToF32High(a); + Packet4f r_lo = vrsqrteq_f32(x_lo); + Packet4f r_hi = vrsqrteq_f32(x_hi); +#if EIGEN_ARCH_ARM64 + Packet4f r1_lo = pmul(r_lo, vabsq_f32(vrsqrtsq_f32(vmulxq_f32(x_lo, r_lo), r_lo))); + Packet4f r1_hi = pmul(r_hi, vabsq_f32(vrsqrtsq_f32(vmulxq_f32(x_hi, r_hi), r_hi))); + return RoundingNarrowBf16(vreinterpretq_u32_f32(r1_lo), vreinterpretq_u32_f32(r1_hi)); +#else + Packet4f r1_lo = pmul(r_lo, vrsqrtsq_f32(pmul(x_lo, r_lo), r_lo)); + Packet4f r1_hi = pmul(r_hi, vrsqrtsq_f32(pmul(x_hi, r_hi), r_hi)); + Packet8bf refined = RoundingNarrowBf16(vreinterpretq_u32_f32(r1_lo), vreinterpretq_u32_f32(r1_hi)); + Packet8bf raw = F32ToBf16Truncate(r_lo, r_hi); + uint16x8_t special = vcgeq_u16(vsubq_u16(a, vdupq_n_u16(0x80)), vdupq_n_u16(0x7f00)); + return vbslq_u16(special, raw, refined); +#endif +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pfrexp(const Packet8bf& a, Packet8bf& exponent) { + Packet4f fexponent_lo; + Packet4f fexponent_hi; + Packet4f lo = pfrexp<Packet4f>(Bf16ToF32Low(a), fexponent_lo); + Packet4f hi = pfrexp<Packet4f>(Bf16ToF32High(a), fexponent_hi); + exponent = F32ToBf16Truncate(fexponent_lo, fexponent_hi); + return F32ToBf16Truncate(lo, hi); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pldexp(const Packet8bf& a, const Packet8bf& exponent) { + return F32ToBf16(pldexp<Packet4f>(Bf16ToF32Low(a), Bf16ToF32Low(exponent)), + pldexp<Packet4f>(Bf16ToF32High(a), Bf16ToF32High(exponent))); +} + +template <> +struct type_casting_traits<bfloat16, float> : vectorized_type_casting_traits<bfloat16, float> {}; +template <> +struct type_casting_traits<float, bfloat16> : vectorized_type_casting_traits<float, bfloat16> {}; + +template <> +EIGEN_STRONG_INLINE Packet4f pcast<Packet4bf, Packet4f>(const Packet4bf& a) { + return Bf16ToF32(a); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pcast<Packet4f, Packet4bf>(const Packet4f& a) { + return F32ToBf16(a); +} + +template <> +EIGEN_STRONG_INLINE Packet4f pcast<Packet8bf, Packet4f>(const Packet8bf& a) { + return Bf16ToF32Low(a); +} + +template <> +EIGEN_STRONG_INLINE Packet8bf pcast<Packet4f, Packet8bf>(const Packet4f& a, const Packet4f& b) { + return F32ToBf16(a, b); +} + +template <> +EIGEN_STRONG_INLINE Packet2f pcast<Packet4bf, Packet2f>(const Packet4bf& a) { + return vget_low_f32(Bf16ToF32(a)); +} + +template <> +EIGEN_STRONG_INLINE Packet4bf pcast<Packet2f, Packet4bf>(const Packet2f& a, const Packet2f& b) { + return F32ToBf16(vcombine_f32(a, b)); +} +} // end namespace internal + +} // end namespace Eigen + +#endif // EIGEN_PACKET_MATH_BF16_NEON_H
diff --git a/Eigen/src/Core/arch/SME/GeneralBlockPanelKernel.h b/Eigen/src/Core/arch/SME/GeneralBlockPanelKernel.h index d56c6f8..59f345b 100644 --- a/Eigen/src/Core/arch/SME/GeneralBlockPanelKernel.h +++ b/Eigen/src/Core/arch/SME/GeneralBlockPanelKernel.h
@@ -112,12 +112,37 @@ // do reach them, from the SYMM above-diagonal transposed pack. // --------------------------------------------------------------------------- +// Appends the leading w <= svl lanes of v to the `fill` < svl lanes of cur that `tail` selects, and stores cur at out +// whenever it fills. Lanes of v left over after a store stay in place, selected by `tail`; the next splice moves them +// to the front. A packed panel's lines are contiguous, so a panel narrower than whole vectors is stored in whole +// vectors this way: a predicated store of each partial line costs about 10 ns on Apple M4 in streaming mode. The +// counts are int64_t, the type whilelt takes, so the loop-carried `fill` is not sign-extended before each whilelt. +template <typename Scalar> +EIGEN_ALWAYS_INLINE void sme_append_lanes(typename sme_packet_traits<Scalar>::type& cur, svbool_t& tail, int64_t& fill, + Scalar*& out, typename sme_packet_traits<Scalar>::type v, int64_t w, + int64_t svl) __arm_streaming EIGEN_SME_ZA_AGNOSTIC { + using Traits = sme_packet_traits<Scalar>; + const auto joined = psplice(tail, cur, v); + if (fill + w < svl) { + cur = joined; + fill += w; + tail = Traits::whilelt(0, fill); + return; + } + pstoreu(Traits::ptrue(), out, joined); + out += svl; + tail = svnot_b_z(Traits::whilelt(0, w), Traits::whilelt(fill, svl)); // v[svl - fill, w) + cur = v; + fill += w - svl; +} + // Copy `width` contiguous source columns per depth step into a depth-major // packed panel of width `width`, for the depth sub-range [k0, k1). Both dst and // src are indexed by the absolute depth index k (dst[k*width+off], // src[k*src_stride+off]); the caller offsets `src` to the region's column base -// and `dst` to the panel base. A panel with svl < width <= 2*svl stores each packed -// line with one two-vector store; others store svl-wide predicated chunks, each +// and `dst` to the panel base. A full 2*svl panel stores each packed line with one +// two-vector store, a narrower one other than svl appends its lines into whole vectors +// (sme_append_lanes); others store svl-wide predicated chunks, each // streamed over the depth sub-range. The chunk loop is outermost so each chunk's // predicate is computed once instead of per depth step (the runtime chunk count // keeps the compiler from hoisting it on its own). The symm packers reuse this @@ -146,21 +171,20 @@ for (; k < k1; ++k) pstoreu_x2(pn, &dst[k * width], ploadu_x2(pn, &src[k * src_stride])); return; } - if (width > svl && width < 2 * svl) { - // Tail panel: each packed line in one counter-predicated two-vector store, four steps in flight. - const svcount_t pn = Traits::whilelt_c2(0, width); - Index k = k0; - for (; k + 4 <= k1; k += 4) { - const auto v0 = ploadu_x2(pn, &src[k * src_stride]); - const auto v1 = ploadu_x2(pn, &src[(k + 1) * src_stride]); - const auto v2 = ploadu_x2(pn, &src[(k + 2) * src_stride]); - const auto v3 = ploadu_x2(pn, &src[(k + 3) * src_stride]); - pstoreu_x2(pn, &dst[k * width], v0); - pstoreu_x2(pn, &dst[(k + 1) * width], v1); - pstoreu_x2(pn, &dst[(k + 2) * width], v2); - pstoreu_x2(pn, &dst[(k + 3) * width], v3); + if (width < 2 * svl && width != svl) { + // Tail panel: whole-vector stores of the appended lines (see sme_append_lanes). + const int w_lo = sme_min(width, svl), w_hi = width - w_lo; + const svbool_t pg_lo = Traits::whilelt(0, w_lo), pg_hi = Traits::whilelt(0, w_hi); + typename Traits::type cur = pset1<typename Traits::type>(Scalar(0)); + svbool_t tail = svpfalse_b(); + int64_t fill = 0; + Scalar* out = dst + k0 * width; + for (Index k = k0; k < k1; ++k) { + const Scalar* s = &src[k * src_stride]; + sme_append_lanes(cur, tail, fill, out, ploadu(pg_lo, s), w_lo, svl); + if (w_hi > 0) sme_append_lanes(cur, tail, fill, out, ploadu(pg_hi, sme_offset(s, svl)), w_hi, svl); } - for (; k < k1; ++k) pstoreu_x2(pn, &dst[k * width], ploadu_x2(pn, &src[k * src_stride])); + if (fill > 0) pstoreu(Traits::whilelt(0, fill), out, psplice(tail, cur, cur)); return; } for (int off = 0; off < width; off += svl) { @@ -232,6 +256,42 @@ const RealScalar* EIGEN_RESTRICT rsrc = reinterpret_cast<const RealScalar*>(src); const int svl = Traits::size(); const Index step = Index(2 * width); + if (width % svl != 0 && width < 2 * svl) { + // A depth step is width real parts, then width imaginary parts, of at most two chunks each: appended into + // whole vectors (see sme_append_lanes). + const int w0 = sme_min(width, svl), w1 = width - w0; + const svbool_t pg_w0 = Traits::whilelt(0, w0), pg_w1 = Traits::whilelt(0, w1); + const svbool_t pg_lo0 = Traits::whilelt(0, 2 * w0), pg_hi0 = Traits::whilelt(svl, 2 * w0); + const svbool_t pg_lo1 = Traits::whilelt(0, 2 * w1), pg_hi1 = Traits::whilelt(svl, 2 * w1); + Vec cur = pset1<Vec>(RealScalar(0)); + svbool_t tail = svpfalse_b(); + int64_t fill = 0; + RealScalar* out = rdst + k0 * step; + for (Index k = k0; k < k1; ++k) { + const RealScalar* p = rsrc + Index(2) * k * src_stride; + const Vec lo0 = ploadu(pg_lo0, p), hi0 = ploadu(pg_hi0, sme_offset(p, svl)); + Vec im0 = puzp2(lo0, hi0); + EIGEN_IF_CONSTEXPR (Conjugate) { + im0 = pnegate(pg_w0, im0); + } + sme_append_lanes(cur, tail, fill, out, puzp1(lo0, hi0), w0, svl); + if (w1 > 0) { + const RealScalar* p1 = sme_offset(p, 2 * svl); + const Vec lo1 = ploadu(pg_lo1, p1), hi1 = ploadu(pg_hi1, sme_offset(p1, svl)); + Vec im1 = puzp2(lo1, hi1); + EIGEN_IF_CONSTEXPR (Conjugate) { + im1 = pnegate(pg_w1, im1); + } + sme_append_lanes(cur, tail, fill, out, puzp1(lo1, hi1), w1, svl); + sme_append_lanes(cur, tail, fill, out, im0, w0, svl); + sme_append_lanes(cur, tail, fill, out, im1, w1, svl); + } else { + sme_append_lanes(cur, tail, fill, out, im0, w0, svl); + } + } + if (fill > 0) pstoreu(Traits::whilelt(0, fill), out, psplice(tail, cur, cur)); + return; + } for (int off = 0; off < width; off += svl) { const int w = width - off; const int lanes = 2 * w; @@ -1416,6 +1476,45 @@ #undef EIGEN_SME_DECLARE_GEMM_PACKERS +// Overwriting stores (gebp_overwrite_t) still load C, then clear the loaded bits and accumulate onto that +0: the +// result is what accumulating into a zeroed C gives, whatever C held, and the memory accesses stay those of +// accumulation. On Apple M4, stores into lines the SME unit has not just loaded measure slower than the load, FMA +// and store they replace, so dropping the loads loses at many sizes. The clearing predicate is opaque to the +// compiler, which would otherwise fold the predicated zero and the loads away. +template <bool Overwrite> +EIGEN_ALWAYS_INLINE svbool_t sme_c_mask() __arm_streaming EIGEN_SME_ZA_AGNOSTIC { + // b32 sets the predicate bit of every 32-bit and every 64-bit lane. + svbool_t m = Overwrite ? svptrue_b32() : svpfalse_b(); + EIGEN_IF_CONSTEXPR (Overwrite) __asm__("" : "+Upa"(m)); + return m; +} +template <bool Overwrite> +EIGEN_ALWAYS_INLINE svfloat32_t sme_c_value(svbool_t m, svfloat32_t v) __arm_streaming EIGEN_SME_ZA_AGNOSTIC { + return Overwrite ? svsel_f32(m, svdup_f32(0.0f), v) : v; +} +template <bool Overwrite> +EIGEN_ALWAYS_INLINE svfloat64_t sme_c_value(svbool_t m, svfloat64_t v) __arm_streaming EIGEN_SME_ZA_AGNOSTIC { + return Overwrite ? svsel_f64(m, svdup_f64(0.0), v) : v; +} + +// Zeroes a rows x cols block of C with streaming stores, so that the SME unit, which accumulates into it next, is the +// first to write its lines. +template <typename Scalar, typename Index> +EIGEN_DONT_INLINE __arm_locally_streaming void sme_zero_c(Scalar* C, Index rs, Index cs, Index rows, Index cols) { + if (rs == 1 || cs == 1) { + const Index inner = rs == 1 ? rows : cols, outer = rs == 1 ? cols : rows, stride = rs == 1 ? cs : rs; + const int64_t bytes = int64_t(inner) * int64_t(sizeof(Scalar)), vl = int64_t(svcntb()); + const svuint8_t z = svdup_u8(0); + for (Index o = 0; o < outer; ++o) { + uint8_t* p = reinterpret_cast<uint8_t*>(C + o * stride); + for (int64_t b = 0; b < bytes; b += vl) svst1_u8(svwhilelt_b8_s64(b, bytes), p + b, z); + } + } else { + for (Index c = 0; c < cols; ++c) + for (Index r = 0; r < rows; ++r) C[r * rs + c * cs] = Scalar(0); + } +} + /***************************************************************************** * sme_store_za_tile -- Store one ZA tile back to C with alpha scaling. * @@ -1423,12 +1522,13 @@ * (both <= the runtime svl). *****************************************************************************/ -template <typename Scalar, int TileId, typename Index> +template <typename Scalar, int TileId, bool Overwrite = false, typename Index> EIGEN_ALWAYS_INLINE void sme_store_za_tile(Scalar* EIGEN_RESTRICT C, Index C_stride_row, Index C_stride_col, Scalar alpha, Index row_start, int pw, Index col_start, int cw) __arm_streaming __arm_inout("za") { using Traits = sme_packet_traits<Scalar>; using Vec = typename Traits::type; + const svbool_t cmask = sme_c_mask<Overwrite>(); const svbool_t pg_m = Traits::whilelt(0, pw); const svbool_t pg_n = Traits::whilelt(0, cw); // FMLA and FADD have equal latency/throughput on ARMv9 cores, and @@ -1440,25 +1540,23 @@ const Vec vzero = pset1<Vec>(Scalar(0)); const Vec valpha = pset1<Vec>(alpha); - // Two C slices are loaded before either is stored: a C line the caller wrote - // from non-streaming code just before the kernel does not forward across the - // mode switch on Apple M4, and a serial load/store pays that latency per slice. - // C = A*B meets the condition on every call, since evalTo zeroes the - // destination first. SVE vectors are sizeless, hence the spelled-out pair. + // Two C slices are loaded before either is stored: on Apple M4 a C line the SME unit does not hold (as after a + // non-streaming access) costs 10-30 ns, which loads issued together overlap. Four measured up to 1.7x slower on + // double blocks. SVE vectors are sizeless, hence the spelled-out pair. if (C_stride_row == 1) { // Column-major C: extract vertical slices (columns of the ZA tile) int ci = 0; for (; ci + 2 <= cw; ci += 2) { Scalar* p0 = C + row_start + (col_start + ci) * C_stride_col; Scalar* p1 = p0 + C_stride_col; - Vec c0 = ploadu(pg_m, p0); - Vec c1 = ploadu(pg_m, p1); + Vec c0 = sme_c_value<Overwrite>(cmask, ploadu(pg_m, p0)); + Vec c1 = sme_c_value<Overwrite>(cmask, ploadu(pg_m, p1)); pstoreu(pg_m, p0, pmadd(pg_m, sme_read_ver_za<TileId>(vzero, pg_m, (uint32_t)ci), valpha, c0)); pstoreu(pg_m, p1, pmadd(pg_m, sme_read_ver_za<TileId>(vzero, pg_m, (uint32_t)(ci + 1)), valpha, c1)); } if (ci < cw) { Scalar* pC = C + row_start + (col_start + ci) * C_stride_col; - Vec vc = ploadu(pg_m, pC); + Vec vc = sme_c_value<Overwrite>(cmask, ploadu(pg_m, pC)); pstoreu(pg_m, pC, pmadd(pg_m, sme_read_ver_za<TileId>(vzero, pg_m, (uint32_t)ci), valpha, vc)); } } else if (C_stride_col == 1) { @@ -1467,14 +1565,14 @@ for (; ri + 2 <= pw; ri += 2) { Scalar* p0 = C + (row_start + ri) * C_stride_row + col_start; Scalar* p1 = p0 + C_stride_row; - Vec c0 = ploadu(pg_n, p0); - Vec c1 = ploadu(pg_n, p1); + Vec c0 = sme_c_value<Overwrite>(cmask, ploadu(pg_n, p0)); + Vec c1 = sme_c_value<Overwrite>(cmask, ploadu(pg_n, p1)); pstoreu(pg_n, p0, pmadd(pg_n, sme_read_hor_za<TileId>(vzero, pg_n, (uint32_t)ri), valpha, c0)); pstoreu(pg_n, p1, pmadd(pg_n, sme_read_hor_za<TileId>(vzero, pg_n, (uint32_t)(ri + 1)), valpha, c1)); } if (ri < pw) { Scalar* pC = C + (row_start + ri) * C_stride_row + col_start; - Vec vc = ploadu(pg_n, pC); + Vec vc = sme_c_value<Overwrite>(cmask, ploadu(pg_n, pC)); pstoreu(pg_n, pC, pmadd(pg_n, sme_read_hor_za<TileId>(vzero, pg_n, (uint32_t)ri), valpha, vc)); } } else { @@ -1488,7 +1586,8 @@ vres = pmul(pg_n, vres, valpha); pstoreu(pg_n, scratch, vres); for (int ci = 0; ci < cw; ++ci) { - C[(row_start + ri) * C_stride_row + (col_start + ci) * C_stride_col] += scratch[ci]; + Scalar& c = C[(row_start + ri) * C_stride_row + (col_start + ci) * C_stride_col]; + c = (Overwrite ? Scalar(0) : c) + scratch[ci]; } } } @@ -1508,49 +1607,62 @@ // Full 2*svl x 2*svl block into column-major C: four columns per vertical four-slice read of the two tiles that // share them, each column loaded and stored whole with two-vector accesses, all four loaded before any is stored. -template <int TileLo, int TileHi, typename Scalar, typename Index> +template <int TileLo, int TileHi, bool Overwrite = false, typename Scalar, typename Index> EIGEN_ALWAYS_INLINE void sme_store_tile_pair_colmajor(Scalar* EIGEN_RESTRICT C, Index ldc, Scalar alpha, int svl) __arm_streaming __arm_inout("za") { using Traits = sme_packet_traits<Scalar>; const svcount_t pn = Traits::ptrue_c(); const svbool_t pg = Traits::ptrue(); const typename Traits::type valpha = pset1<typename Traits::type>(alpha); + const svbool_t m = sme_c_mask<Overwrite>(); for (int c = 0; c < svl; c += 4) { + // GCC 14 can overwrite live ZA tuple components with C loads. +#if !EIGEN_COMP_GNUC_STRICT const auto lo = sme_read_ver_za_vg4<TileLo, Scalar>(uint32_t(c)); const auto hi = sme_read_ver_za_vg4<TileHi, Scalar>(uint32_t(c)); +#endif Scalar* p = C + Index(c) * ldc; const auto c0 = ploadu_x2(pn, p), c1 = ploadu_x2(pn, p + ldc), c2 = ploadu_x2(pn, p + 2 * ldc), c3 = ploadu_x2(pn, p + 3 * ldc); +#if EIGEN_COMP_GNUC_STRICT + const auto lo = sme_read_ver_za_vg4<TileLo, Scalar>(uint32_t(c)); + const auto hi = sme_read_ver_za_vg4<TileHi, Scalar>(uint32_t(c)); +#endif pstoreu_x2(pn, p, - pcreate(pmadd(pg, pget<0>(lo), valpha, pget<0>(c0)), pmadd(pg, pget<0>(hi), valpha, pget<1>(c0)))); + pcreate(pmadd(pg, pget<0>(lo), valpha, sme_c_value<Overwrite>(m, pget<0>(c0))), + pmadd(pg, pget<0>(hi), valpha, sme_c_value<Overwrite>(m, pget<1>(c0))))); pstoreu_x2(pn, p + ldc, - pcreate(pmadd(pg, pget<1>(lo), valpha, pget<0>(c1)), pmadd(pg, pget<1>(hi), valpha, pget<1>(c1)))); + pcreate(pmadd(pg, pget<1>(lo), valpha, sme_c_value<Overwrite>(m, pget<0>(c1))), + pmadd(pg, pget<1>(hi), valpha, sme_c_value<Overwrite>(m, pget<1>(c1))))); pstoreu_x2(pn, p + 2 * ldc, - pcreate(pmadd(pg, pget<2>(lo), valpha, pget<0>(c2)), pmadd(pg, pget<2>(hi), valpha, pget<1>(c2)))); + pcreate(pmadd(pg, pget<2>(lo), valpha, sme_c_value<Overwrite>(m, pget<0>(c2))), + pmadd(pg, pget<2>(hi), valpha, sme_c_value<Overwrite>(m, pget<1>(c2))))); pstoreu_x2(pn, p + 3 * ldc, - pcreate(pmadd(pg, pget<3>(lo), valpha, pget<0>(c3)), pmadd(pg, pget<3>(hi), valpha, pget<1>(c3)))); + pcreate(pmadd(pg, pget<3>(lo), valpha, sme_c_value<Overwrite>(m, pget<0>(c3))), + pmadd(pg, pget<3>(hi), valpha, sme_c_value<Overwrite>(m, pget<1>(c3))))); } } -template <typename Scalar, typename Index> +template <bool Overwrite = false, typename Scalar, typename Index> EIGEN_ALWAYS_INLINE void sme_store_2x2_grid(Scalar* EIGEN_RESTRICT C, Index C_stride_row, Index C_stride_col, Scalar alpha, Index row_start, int rlo, int rhi, Index col_start, int clo, int chi) __arm_streaming __arm_inout("za") { const int svl = sme_packet_traits<Scalar>::size(); if (svl >= 4 && C_stride_row == 1 && rlo == svl && rhi == svl && clo == svl && chi == svl) { Scalar* c0 = C + row_start + col_start * C_stride_col; - sme_store_tile_pair_colmajor<0, 2>(c0, C_stride_col, alpha, svl); - sme_store_tile_pair_colmajor<1, 3>(c0 + Index(svl) * C_stride_col, C_stride_col, alpha, svl); + sme_store_tile_pair_colmajor<0, 2, Overwrite>(c0, C_stride_col, alpha, svl); + sme_store_tile_pair_colmajor<1, 3, Overwrite>(c0 + Index(svl) * C_stride_col, C_stride_col, alpha, svl); return; } - sme_store_za_tile<Scalar, 0>(C, C_stride_row, C_stride_col, alpha, row_start, rlo, col_start, clo); + sme_store_za_tile<Scalar, 0, Overwrite>(C, C_stride_row, C_stride_col, alpha, row_start, rlo, col_start, clo); if (chi > 0) { - sme_store_za_tile<Scalar, 1>(C, C_stride_row, C_stride_col, alpha, row_start, rlo, col_start + svl, chi); + sme_store_za_tile<Scalar, 1, Overwrite>(C, C_stride_row, C_stride_col, alpha, row_start, rlo, col_start + svl, chi); } if (rhi > 0) { - sme_store_za_tile<Scalar, 2>(C, C_stride_row, C_stride_col, alpha, row_start + svl, rhi, col_start, clo); + sme_store_za_tile<Scalar, 2, Overwrite>(C, C_stride_row, C_stride_col, alpha, row_start + svl, rhi, col_start, clo); if (chi > 0) { - sme_store_za_tile<Scalar, 3>(C, C_stride_row, C_stride_col, alpha, row_start + svl, rhi, col_start + svl, chi); + sme_store_za_tile<Scalar, 3, Overwrite>(C, C_stride_row, C_stride_col, alpha, row_start + svl, rhi, + col_start + svl, chi); } } } @@ -1844,8 +1956,8 @@ template <bool ConjLhs, bool ConjRhs, typename Scalar, typename Index> EIGEN_ALWAYS_INLINE void sme_process(Scalar* EIGEN_RESTRICT C, Index C_stride_row, Index C_stride_col, const Scalar* EIGEN_RESTRICT blA, const Scalar* EIGEN_RESTRICT blB, Index depth, - Scalar alpha, Index row_start, int pw, Index col_start, int cw, - Index a_step) __arm_streaming __arm_inout("za") { + Scalar alpha, Index row_start, int pw, Index col_start, int cw, Index a_step, + bool overwrite) __arm_streaming __arm_inout("za") { // Conjugation is the identity on real scalars, so this overload ignores it. // a_step is the distance between A's depth steps: pw for a packed panel, the // column stride for a ColMajor source read in place. @@ -1933,7 +2045,12 @@ } // Store the (up to) 2x2 grid of tiles for this sub-block pass. - sme_store_2x2_grid(C, C_stride_row, C_stride_col, alpha, row_start + rt, rlo, rhi, col_start + ct, clo, chi); + if (overwrite) + sme_store_2x2_grid<true>(C, C_stride_row, C_stride_col, alpha, row_start + rt, rlo, rhi, col_start + ct, clo, + chi); + else + sme_store_2x2_grid<false>(C, C_stride_row, C_stride_col, alpha, row_start + rt, rlo, rhi, col_start + ct, clo, + chi); } } } @@ -2055,7 +2172,8 @@ const Scalar* EIGEN_RESTRICT blA0, Index step0, const Scalar* EIGEN_RESTRICT blA1, Index step1, int pw1, const Scalar* EIGEN_RESTRICT blB, int cw, Index depth, Scalar alpha, - Index row_start, Index col_start) __arm_streaming __arm_inout("za") { + Index row_start, Index col_start, + bool overwrite) __arm_streaming __arm_inout("za") { using Traits = sme_packet_traits<Scalar>; using Vec = typename Traits::type; const int svl = Traits::size(); @@ -2075,12 +2193,28 @@ sme_mopa<2>(p1lo, pn, a1lo, b); sme_mopa<3>(p1hi, pn, a1hi, b); } - sme_store_za_tile<Scalar, 0>(C, C_stride_row, C_stride_col, alpha, row_start, svl, col_start, cw); - sme_store_za_tile<Scalar, 1>(C, C_stride_row, C_stride_col, alpha, row_start + svl, svl, col_start, cw); - sme_store_za_tile<Scalar, 2>(C, C_stride_row, C_stride_col, alpha, row_start + 2 * svl, sme_min(pw1, svl), col_start, - cw); - if (pw1 > svl) - sme_store_za_tile<Scalar, 3>(C, C_stride_row, C_stride_col, alpha, row_start + 3 * svl, pw1 - svl, col_start, cw); + if (overwrite) + sme_store_za_tile<Scalar, 0, true>(C, C_stride_row, C_stride_col, alpha, row_start, svl, col_start, cw); + else + sme_store_za_tile<Scalar, 0, false>(C, C_stride_row, C_stride_col, alpha, row_start, svl, col_start, cw); + if (overwrite) + sme_store_za_tile<Scalar, 1, true>(C, C_stride_row, C_stride_col, alpha, row_start + svl, svl, col_start, cw); + else + sme_store_za_tile<Scalar, 1, false>(C, C_stride_row, C_stride_col, alpha, row_start + svl, svl, col_start, cw); + if (overwrite) + sme_store_za_tile<Scalar, 2, true>(C, C_stride_row, C_stride_col, alpha, row_start + 2 * svl, sme_min(pw1, svl), + col_start, cw); + else + sme_store_za_tile<Scalar, 2, false>(C, C_stride_row, C_stride_col, alpha, row_start + 2 * svl, sme_min(pw1, svl), + col_start, cw); + if (pw1 > svl) { + if (overwrite) + sme_store_za_tile<Scalar, 3, true>(C, C_stride_row, C_stride_col, alpha, row_start + 3 * svl, pw1 - svl, + col_start, cw); + else + sme_store_za_tile<Scalar, 3, false>(C, C_stride_row, C_stride_col, alpha, row_start + 3 * svl, pw1 - svl, + col_start, cw); + } } // Tile Dst += tile Src, four columns per step. @@ -2101,7 +2235,7 @@ EIGEN_ALWAYS_INLINE void sme_process_split(Scalar* EIGEN_RESTRICT C, Index C_stride_row, Index C_stride_col, const Scalar* EIGEN_RESTRICT blA, const Scalar* EIGEN_RESTRICT blB, Index depth, Scalar alpha, Index row_start, int pw, Index col_start, int cw, - Index a_step) __arm_streaming __arm_inout("za") { + Index a_step, bool overwrite) __arm_streaming __arm_inout("za") { using Traits = sme_packet_traits<Scalar>; using Vec = typename Traits::type; const int svl = Traits::size(); @@ -2124,7 +2258,10 @@ sme_fold_tile<0, 1, Scalar>(); sme_fold_tile<2, 3, Scalar>(); sme_fold_tile<0, 2, Scalar>(); - sme_store_za_tile<Scalar, 0>(C, C_stride_row, C_stride_col, alpha, row_start, pw, col_start, cw); + if (overwrite) + sme_store_za_tile<Scalar, 0, true>(C, C_stride_row, C_stride_col, alpha, row_start, pw, col_start, cw); + else + sme_store_za_tile<Scalar, 0, false>(C, C_stride_row, C_stride_col, alpha, row_start, pw, col_start, cw); } else if (pw <= svl) { // Tiles 0 and 1 take the even depth steps, 2 and 3 the odd ones. for (; k + 2 <= depth; k += 2) { @@ -2143,7 +2280,10 @@ } sme_fold_tile<0, 2, Scalar>(); sme_fold_tile<1, 3, Scalar>(); - sme_store_2x2_grid(C, C_stride_row, C_stride_col, alpha, row_start, pw, 0, col_start, svl, cw - svl); + if (overwrite) + sme_store_2x2_grid<true>(C, C_stride_row, C_stride_col, alpha, row_start, pw, 0, col_start, svl, cw - svl); + else + sme_store_2x2_grid<false>(C, C_stride_row, C_stride_col, alpha, row_start, pw, 0, col_start, svl, cw - svl); } else { // cw <= svl < pw: tiles 0 and 2 take the even depth steps, 1 and 3 the odd ones. for (; k + 2 <= depth; k += 2) { @@ -2162,29 +2302,35 @@ } sme_fold_tile<0, 1, Scalar>(); sme_fold_tile<2, 3, Scalar>(); - sme_store_2x2_grid(C, C_stride_row, C_stride_col, alpha, row_start, svl, pw - svl, col_start, cw, 0); + if (overwrite) + sme_store_2x2_grid<true>(C, C_stride_row, C_stride_col, alpha, row_start, svl, pw - svl, col_start, cw, 0); + else + sme_store_2x2_grid<false>(C, C_stride_row, C_stride_col, alpha, row_start, svl, pw - svl, col_start, cw, 0); } } // One pw x cw block: sme_process_split when it fills at most two tiles of the 2 x 2 grid; complex blocks keep -// sme_process. `split_ok` (sme_split_ok) holds once per call, outside the block loops. +// sme_process. `split_ok` (sme_split_ok) holds once per call, outside the block loops. `overwrite` assigns the block +// instead of accumulating into it; a complex overwrite zeroes C up front (sme_zero_c) and then accumulates. The flag is +// a runtime argument and each store picks its compile-time form, so one kernel serves both modes, accumulation runs +// the plain accumulating stores, and only the store epilogues are compiled twice. template <bool ConjLhs, bool ConjRhs, bool FoldRealAlpha, typename Scalar, typename Index> EIGEN_ALWAYS_INLINE void sme_process_block(Scalar* C, Index rs, Index cs, const Scalar* blA, const Scalar* blB, Index depth, Scalar alpha, Index row_start, int pw, Index col_start, int cw, - Index a_step, bool split_ok, Index, - Index) __arm_streaming __arm_inout("za") { + Index a_step, bool split_ok, Index, Index, + bool overwrite) __arm_streaming __arm_inout("za") { const int svl = sme_packet_traits<Scalar>::size(); if (split_ok && (pw <= svl || cw <= svl)) - sme_process_split(C, rs, cs, blA, blB, depth, alpha, row_start, pw, col_start, cw, a_step); + sme_process_split(C, rs, cs, blA, blB, depth, alpha, row_start, pw, col_start, cw, a_step, overwrite); else - sme_process<ConjLhs, ConjRhs>(C, rs, cs, blA, blB, depth, alpha, row_start, pw, col_start, cw, a_step); + sme_process<ConjLhs, ConjRhs>(C, rs, cs, blA, blB, depth, alpha, row_start, pw, col_start, cw, a_step, overwrite); } template <bool ConjLhs, bool ConjRhs, bool FoldRealAlpha, typename RealScalar, typename Index> EIGEN_ALWAYS_INLINE void sme_process_block(std::complex<RealScalar>* C, Index rs, Index cs, const std::complex<RealScalar>* blA, const std::complex<RealScalar>* blB, Index depth, std::complex<RealScalar> alpha, Index row_start, int pw, - Index col_start, int cw, Index a_step, bool, Index rows, - Index cols) __arm_streaming __arm_inout("za") { + Index col_start, int cw, Index a_step, bool, Index rows, Index cols, + bool) __arm_streaming __arm_inout("za") { sme_process<ConjLhs, ConjRhs, FoldRealAlpha>(C, rs, cs, blA, blB, depth, alpha, row_start, pw, col_start, cw, a_step, rows, cols); } @@ -2206,16 +2352,17 @@ EIGEN_ALWAYS_INLINE void sme_process_narrow_dispatch(Scalar* C, Index rs, Index cs, const Scalar* blA0, Index step0, const Scalar* blA1, Index step1, int pw1, const Scalar* blB, int cw, Index depth, Scalar alpha, Index row_start, - Index col_start) __arm_streaming __arm_inout("za") { - sme_process_narrow(C, rs, cs, blA0, step0, blA1, step1, pw1, blB, cw, depth, alpha, row_start, col_start); + Index col_start, + bool overwrite) __arm_streaming __arm_inout("za") { + sme_process_narrow(C, rs, cs, blA0, step0, blA1, step1, pw1, blB, cw, depth, alpha, row_start, col_start, overwrite); } template <typename RealScalar, typename Index> EIGEN_ALWAYS_INLINE void sme_process_narrow_dispatch(std::complex<RealScalar>*, Index, Index, const std::complex<RealScalar>*, Index, const std::complex<RealScalar>*, Index, int, const std::complex<RealScalar>*, int, Index, - std::complex<RealScalar>, Index, - Index) __arm_streaming __arm_inout("za") {} + std::complex<RealScalar>, Index, Index, + bool) __arm_streaming __arm_inout("za") {} // Core-side prefetch into L2 of the column-major C block the next sme_process call reads and writes. template <typename Scalar, typename Index> @@ -2231,9 +2378,9 @@ } template <typename Scalar, bool ConjLhs, bool ConjRhs, bool FoldRealAlpha = false, typename Index> -EIGEN_DONT_INLINE __arm_locally_streaming __arm_new("za") void sme_gebp_impl( +EIGEN_DONT_INLINE EIGEN_GEMM_NO_CLONE __arm_locally_streaming __arm_new("za") void sme_gebp_impl( Scalar* C, Index C_stride_row, Index C_stride_col, const Scalar* blockA, const Scalar* blockB, Index rows, - Index depth, Index cols, Scalar alpha, Index strideA, Index strideB, Index offsetA, Index offsetB) { + Index depth, Index cols, Scalar alpha, Index strideA, Index strideB, Index offsetA, Index offsetB, bool overwrite) { constexpr int MR = sme_block<Scalar>::mr; constexpr int NR = sme_block<Scalar>::nr; @@ -2258,7 +2405,7 @@ const int pw1 = static_cast<int>(sme_min(rows - i - MR, Index(MR))); sme_process_narrow_dispatch(C, C_stride_row, C_stride_col, blockA + i * strideA + offsetA * MR, Index(MR), blockA + (i + MR) * strideA + offsetA * pw1, Index(pw1), pw1, blB, cw, depth, - alpha, i, j); + alpha, i, j, overwrite); } } for (; i < rows; i += MR) { @@ -2268,7 +2415,7 @@ sme_prefetch_next_c(C, C_stride_row, C_stride_col, i + MR < rows ? i + MR : Index(0), i + MR < rows ? j : j + NR, rows, cols, MR, NR); sme_process_block<ConjLhs, ConjRhs, FoldRealAlpha>(C, C_stride_row, C_stride_col, blA, blB, depth, alpha, i, pw, - j, cw, Index(pw), split_ok, rows, cols); + j, cw, Index(pw), split_ok, rows, cols, overwrite); } } } @@ -2282,9 +2429,9 @@ template <typename Index> static void run(Scalar* C, Index C_stride_row, Index C_stride_col, const Scalar* blockA, const Scalar* blockB, Index rows, Index depth, Index cols, Scalar alpha, Index strideA, Index strideB, Index offsetA, - Index offsetB) { + Index offsetB, bool overwrite = false) { sme_gebp_impl<Scalar, ConjLhs, ConjRhs, false>(C, C_stride_row, C_stride_col, blockA, blockB, rows, depth, cols, - alpha, strideA, strideB, offsetA, offsetB); + alpha, strideA, strideB, offsetA, offsetB, overwrite); } }; @@ -2293,14 +2440,14 @@ template <typename Index> static void run(Scalar* C, Index C_stride_row, Index C_stride_col, const Scalar* blockA, const Scalar* blockB, Index rows, Index depth, Index cols, Scalar alpha, Index strideA, Index strideB, Index offsetA, - Index offsetB) { + Index offsetB, bool overwrite = false) { using RealScalar = typename NumTraits<Scalar>::Real; if (numext::imag(alpha) == RealScalar(0) && numext::real(alpha) != RealScalar(1)) { sme_gebp_impl<Scalar, ConjLhs, ConjRhs, true>(C, C_stride_row, C_stride_col, blockA, blockB, rows, depth, cols, - alpha, strideA, strideB, offsetA, offsetB); + alpha, strideA, strideB, offsetA, offsetB, overwrite); } else { sme_gebp_impl<Scalar, ConjLhs, ConjRhs, false>(C, C_stride_row, C_stride_col, blockA, blockB, rows, depth, cols, - alpha, strideA, strideB, offsetA, offsetB); + alpha, strideA, strideB, offsetA, offsetB, overwrite); } } }; @@ -2308,9 +2455,9 @@ // gebp with the LHS read from a ColMajor source: rows i..i+pw of column k are // at lhs + i + k * lda, the packed layout with a_step = lda. Real scalars only. template <typename Scalar, typename Index> -EIGEN_DONT_INLINE __arm_locally_streaming __arm_new("za") void sme_gebp_impl_direct_lhs( +EIGEN_DONT_INLINE EIGEN_GEMM_NO_CLONE __arm_locally_streaming __arm_new("za") void sme_gebp_impl_direct_lhs( Scalar* C, Index C_stride_row, Index C_stride_col, const Scalar* lhs, Index lda, const Scalar* blockB, Index rows, - Index depth, Index cols, Scalar alpha, Index strideB, Index offsetB) { + Index depth, Index cols, Scalar alpha, Index strideB, Index offsetB, bool overwrite) { constexpr int MR = sme_block<Scalar>::mr; constexpr int NR = sme_block<Scalar>::nr; const bool split_ok = sme_split_ok<Scalar>(), narrow_ok = sme_narrow_ok<Scalar>(); @@ -2319,13 +2466,15 @@ const Scalar* blB = blockB + j * strideB + offsetB * cw; Index i = 0; if (narrow_ok && cw <= sme_packet_traits<Scalar>::size()) - for (; i + MR < rows; i += 2 * MR) - sme_process_narrow(C, C_stride_row, C_stride_col, lhs + i, lda, lhs + i + MR, lda, - static_cast<int>(sme_min(rows - i - MR, Index(MR))), blB, cw, depth, alpha, i, j); + for (; i + MR < rows; i += 2 * MR) { + const int pw1 = static_cast<int>(sme_min(rows - i - MR, Index(MR))); + sme_process_narrow(C, C_stride_row, C_stride_col, lhs + i, lda, lhs + i + MR, lda, pw1, blB, cw, depth, alpha, + i, j, overwrite); + } for (; i < rows; i += MR) { const int pw = static_cast<int>(sme_min(rows - i, Index(MR))); sme_process_block<false, false, false>(C, C_stride_row, C_stride_col, lhs + i, blB, depth, alpha, i, pw, j, cw, - lda, split_ok, rows, cols); + lda, split_ok, rows, cols, overwrite); } } } @@ -2852,10 +3001,50 @@ template <typename Scalar, typename Index, typename DataMapper, int mr, int nr, bool ConjugateLhs, bool ConjugateRhs> struct sme_gebp_kernel { using ResScalar = Scalar; + // Real kernels overwrite in their stores (sme_c_mask). Complex ones zero their C block with streaming stores + // (sme_zero_c), then accumulate: on M4 that beats the driver's non-streaming zeroing for a small result. + static constexpr bool CanOverwrite = true; + // Whether overwriting a rows x cols result pays over zeroing it first (gebp_overwrite_pays). The NEON small-block + // path has nothing to gain: it zeroes and accumulates either way. Otherwise, on Apple M4, overwriting pays for + // 32-bit scalars at every size, and for 64-bit and complex ones up to a 128 KB result, past which zeroing up front + // measures faster. + static bool overwrite_pays(Index rows, Index cols, Index depth) { + if (rows <= 0 || cols <= 0 || sme_kernel_with_neon<Scalar>(rows, cols, depth, depth, depth)) return false; + // rows * cols <= 128 KB / sizeof(Scalar), without forming a product a narrow Index cannot hold. + return sizeof(Scalar) == 4 || rows <= Index((128 * 1024) / sizeof(Scalar)) / cols; + } EIGEN_DONT_INLINE void operator()(const DataMapper& res, const Scalar* blockA, const Scalar* blockB, Index rows, Index depth, Index cols, ResScalar alpha, Index strideA = -1, Index strideB = -1, Index offsetA = 0, Index offsetB = 0) { + run(false, res, blockA, blockB, rows, depth, cols, alpha, strideA, strideB, offsetA, offsetB); + } + // C = alpha * A' * B' over the block, whatever C held (see gebp_overwrite_t). + EIGEN_DONT_INLINE void operator()(gebp_overwrite_t, const DataMapper& res, const Scalar* blockA, const Scalar* blockB, + Index rows, Index depth, Index cols, ResScalar alpha, Index strideA = -1, + Index strideB = -1, Index offsetA = 0, Index offsetB = 0) { + run(true, res, blockA, blockB, rows, depth, cols, alpha, strideA, strideB, offsetA, offsetB); + } + + // The LHS block read from its ColMajor source (see sme_direct_lhs_ok); only + // instantiated for real scalars. + EIGEN_DONT_INLINE void run_direct_lhs(const DataMapper& res, const Scalar* lhs, Index lhsStride, const Scalar* blockB, + Index rows, Index depth, Index cols, ResScalar alpha, Index strideB = -1, + Index offsetB = 0) { + run_direct_lhs_impl(false, res, lhs, lhsStride, blockB, rows, depth, cols, alpha, strideB, offsetB); + } + EIGEN_DONT_INLINE void run_direct_lhs(gebp_overwrite_t, const DataMapper& res, const Scalar* lhs, Index lhsStride, + const Scalar* blockB, Index rows, Index depth, Index cols, ResScalar alpha, + Index strideB = -1, Index offsetB = 0) { + run_direct_lhs_impl(true, res, lhs, lhsStride, blockB, rows, depth, cols, alpha, strideB, offsetB); + } + + private: + // One body for both modes, so that the kernels are compiled once (see sme_process_block). + EIGEN_DONT_INLINE EIGEN_GEMM_NO_CLONE void run(bool overwrite, const DataMapper& res, const Scalar* blockA, + const Scalar* blockB, Index rows, Index depth, Index cols, + ResScalar alpha, Index strideA, Index strideB, Index offsetA, + Index offsetB) { // Real scalars never reach the kernel conjugated (conj_helper folds it into // the identity long before), so the real path stays free of the flags. static_assert(NumTraits<Scalar>::IsComplex || (!ConjugateLhs && !ConjugateRhs), @@ -2866,36 +3055,50 @@ if (strideA == -1) strideA = depth; if (strideB == -1) strideB = depth; - if (rows <= 0 || cols <= 0 || depth <= 0) return; + if (rows <= 0 || cols <= 0) return; Scalar* C_base = const_cast<Scalar*>(&res(0, 0)); const Index C_stride_row = &res(1, 0) - &res(0, 0); const Index C_stride_col = &res(0, 1) - &res(0, 0); + if (depth <= 0) { + if (overwrite) gemm_zero_result(C_base, rows, cols, C_stride_row, C_stride_col); + return; + } if (sme_kernel_with_neon<Scalar>(rows, cols, depth, strideA, strideB)) { + if (overwrite) gemm_zero_result(C_base, rows, cols, C_stride_row, C_stride_col); sme_gebp_neon<Scalar, ConjugateLhs, ConjugateRhs>(C_base, C_stride_row, C_stride_col, blockA, blockB, rows, depth, cols, alpha, strideA, strideB, offsetA, offsetB); return; } sme_fpsr_guard fpsr; + const bool zero_first = overwrite && NumTraits<Scalar>::IsComplex; + if (zero_first) sme_zero_c(C_base, C_stride_row, C_stride_col, rows, cols); sme_gebp_dispatch<Scalar, ConjugateLhs, ConjugateRhs>::run(C_base, C_stride_row, C_stride_col, blockA, blockB, rows, - depth, cols, alpha, strideA, strideB, offsetA, offsetB); + depth, cols, alpha, strideA, strideB, offsetA, offsetB, + overwrite && !zero_first); } - // The LHS block read from its ColMajor source (see sme_direct_lhs_ok); only - // instantiated for real scalars. - EIGEN_DONT_INLINE void run_direct_lhs(const DataMapper& res, const Scalar* lhs, Index lhsStride, const Scalar* blockB, - Index rows, Index depth, Index cols, ResScalar alpha, Index strideB = -1, - Index offsetB = 0) { + EIGEN_DONT_INLINE EIGEN_GEMM_NO_CLONE void run_direct_lhs_impl(bool overwrite, const DataMapper& res, + const Scalar* lhs, Index lhsStride, + const Scalar* blockB, Index rows, Index depth, + Index cols, ResScalar alpha, Index strideB, + Index offsetB) { if (strideB == -1) strideB = depth; - if (rows <= 0 || cols <= 0 || depth <= 0) return; + if (rows <= 0 || cols <= 0) return; Scalar* C_base = const_cast<Scalar*>(&res(0, 0)); const Index C_stride_row = &res(1, 0) - &res(0, 0); const Index C_stride_col = &res(0, 1) - &res(0, 0); + if (depth <= 0) { + if (overwrite) gemm_zero_result(C_base, rows, cols, C_stride_row, C_stride_col); + return; + } sme_fpsr_guard fpsr; + const bool zero_first = overwrite && NumTraits<Scalar>::IsComplex; + if (zero_first) sme_zero_c(C_base, C_stride_row, C_stride_col, rows, cols); sme_gebp_impl_direct_lhs<Scalar, Index>(C_base, C_stride_row, C_stride_col, lhs, lhsStride, blockB, rows, depth, - cols, alpha, strideB, offsetB); + cols, alpha, strideB, offsetB, overwrite && !zero_first); } };
diff --git a/Eigen/src/Core/arch/SSE/Complex.h b/Eigen/src/Core/arch/SSE/Complex.h index c00d356..2ab00ae 100644 --- a/Eigen/src/Core/arch/SSE/Complex.h +++ b/Eigen/src/Core/arch/SSE/Complex.h
@@ -186,9 +186,7 @@ template <> EIGEN_STRONG_INLINE std::complex<float> pfirst<Packet2cf>(const Packet2cf& a) { - alignas(alignof(__m64)) std::complex<float> res; - _mm_storel_pi((__m64*)&res, a.v); - return res; + return std::complex<float>(_mm_cvtss_f32(a.v), _mm_cvtss_f32(_mm_shuffle_ps(a.v, a.v, _MM_SHUFFLE(1, 1, 1, 1)))); } template <>
diff --git a/Eigen/src/Core/arch/SVE/PacketMath.h b/Eigen/src/Core/arch/SVE/PacketMath.h index a6120dd..3fd0e8e 100644 --- a/Eigen/src/Core/arch/SVE/PacketMath.h +++ b/Eigen/src/Core/arch/SVE/PacketMath.h
@@ -1400,13 +1400,13 @@ // Predicates with lanes [begin, begin + count) active. begin is a literal 0 at the assignment // call sites, so that branch folds and the common case is a single whilelt. EIGEN_STRONG_INLINE svbool_t sve_segment_predicate_b32(Index begin, Index count) { - eigen_assert(begin >= 0 && count >= 0); + eigen_assert(begin >= 0 && count >= 0 && begin + count <= EIGEN_ARM64_SVE_VL / 32); if (begin == 0) return svwhilelt_b32(uint64_t(0), uint64_t(count)); return svnot_b_z(svwhilelt_b32(uint64_t(0), uint64_t(begin + count)), svwhilelt_b32(uint64_t(0), uint64_t(begin))); } EIGEN_STRONG_INLINE svbool_t sve_segment_predicate_b64(Index begin, Index count) { - eigen_assert(begin >= 0 && count >= 0); + eigen_assert(begin >= 0 && count >= 0 && begin + count <= EIGEN_ARM64_SVE_VL / 64); if (begin == 0) return svwhilelt_b64(uint64_t(0), uint64_t(count)); return svnot_b_z(svwhilelt_b64(uint64_t(0), uint64_t(begin + count)), svwhilelt_b64(uint64_t(0), uint64_t(begin))); }
diff --git a/Eigen/src/Core/arch/SYCL/InteropHeaders.h b/Eigen/src/Core/arch/SYCL/InteropHeaders.h index 12c30d4..6b4907b 100644 --- a/Eigen/src/Core/arch/SYCL/InteropHeaders.h +++ b/Eigen/src/Core/arch/SYCL/InteropHeaders.h
@@ -78,7 +78,7 @@ }; }; -#ifdef SYCL_DEVICE_ONLY +#ifdef EIGEN_SYCL_DEVICE_ONLY #define SYCL_PACKET_TRAITS(packet_type, unpacket_type, lengths) \ template <> \ struct packet_traits<unpacket_type> : sycl_packet_traits<lengths> { \ @@ -129,7 +129,7 @@ template <typename PacketReturnType, int PacketSize> struct PacketWrapper; // This function should never get called on the device -#ifndef SYCL_DEVICE_ONLY +#ifndef EIGEN_SYCL_DEVICE_ONLY template <typename PacketReturnType, int PacketSize> struct PacketWrapper { typedef typename ::Eigen::internal::unpacket_traits<PacketReturnType>::type Scalar; @@ -147,7 +147,7 @@ } }; -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) template <typename PacketReturnType> struct PacketWrapper<PacketReturnType, 4> { typedef typename ::Eigen::internal::unpacket_traits<PacketReturnType>::type Scalar;
diff --git a/Eigen/src/Core/arch/SYCL/MathFunctions.h b/Eigen/src/Core/arch/SYCL/MathFunctions.h index 239aee9..72862c5 100644 --- a/Eigen/src/Core/arch/SYCL/MathFunctions.h +++ b/Eigen/src/Core/arch/SYCL/MathFunctions.h
@@ -32,7 +32,7 @@ // Make sure this is only available when targeting a GPU: we don't want to // introduce conflicts between these packet_traits definitions and the ones // we'll use on the host side (SSE, AVX, ...) -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) // Generic macro for unary SYCL math functions. #define SYCL_PACKET_FUNCTION(EIGEN_FUNC, SYCL_FUNC, PACKET) \
diff --git a/Eigen/src/Core/arch/SYCL/PacketMath.h b/Eigen/src/Core/arch/SYCL/PacketMath.h index 0c1bba8..753a91f 100644 --- a/Eigen/src/Core/arch/SYCL/PacketMath.h +++ b/Eigen/src/Core/arch/SYCL/PacketMath.h
@@ -30,7 +30,7 @@ namespace Eigen { namespace internal { -#ifdef SYCL_DEVICE_ONLY +#ifdef EIGEN_SYCL_DEVICE_ONLY #define SYCL_PLOAD(packet_type, AlignedType) \ template <> \ EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE packet_type pload##AlignedType<packet_type>( \ @@ -559,7 +559,7 @@ kernel.packet[1].x() = tmp; } -#endif // SYCL_DEVICE_ONLY +#endif // EIGEN_SYCL_DEVICE_ONLY } // end namespace internal
diff --git a/Eigen/src/Core/arch/SYCL/TypeCasting.h b/Eigen/src/Core/arch/SYCL/TypeCasting.h index 2c8fd51..1c3073f 100644 --- a/Eigen/src/Core/arch/SYCL/TypeCasting.h +++ b/Eigen/src/Core/arch/SYCL/TypeCasting.h
@@ -29,7 +29,7 @@ namespace Eigen { namespace internal { -#ifdef SYCL_DEVICE_ONLY +#ifdef EIGEN_SYCL_DEVICE_ONLY template <> struct type_casting_traits<float, int> { enum { VectorizedCast = 1, SrcCoeffRatio = 1, TgtCoeffRatio = 1 };
diff --git a/Eigen/src/Core/functors/BinaryFunctors.h b/Eigen/src/Core/functors/BinaryFunctors.h index f184fa9..f1eb89c 100644 --- a/Eigen/src/Core/functors/BinaryFunctors.h +++ b/Eigen/src/Core/functors/BinaryFunctors.h
@@ -106,11 +106,37 @@ }; }; -// Multiplication commutes for arithmetic and complex scalars, but not for every user-defined -// Scalar (e.g. quaternion-like or matrix-like types). -template <typename Scalar> -struct functor_is_commutative<scalar_product_op<Scalar, Scalar>> - : bool_constant<is_arithmetic<Scalar>::value || NumTraits<Scalar>::IsComplex> {}; +template <typename LhsScalar, typename RhsScalar, typename = void> +struct scalar_product_commutativity + : bool_constant<is_arithmetic<LhsScalar>::value && is_arithmetic<RhsScalar>::value> {}; + +template <typename LhsScalar, typename RhsScalar> +struct scalar_product_commutativity< + LhsScalar, RhsScalar, + void_t< + decltype(ScalarBinaryOpTraits<LhsScalar, RhsScalar, scalar_product_op<LhsScalar, RhsScalar>>::IsCommutative)>> + : bool_constant< + ScalarBinaryOpTraits<LhsScalar, RhsScalar, scalar_product_op<LhsScalar, RhsScalar>>::IsCommutative> {}; + +// IsComplex does not imply commutativity; custom scalars opt in through ScalarBinaryOpTraits. +template <typename LhsScalar, typename RhsScalar> +struct functor_is_commutative<scalar_product_op<LhsScalar, RhsScalar>> + : scalar_product_commutativity<LhsScalar, RhsScalar> {}; + +template <typename RealScalar> +struct functor_is_commutative<scalar_product_op<std::complex<RealScalar>, std::complex<RealScalar>>> + : bool_constant<is_arithmetic<RealScalar>::value || + scalar_product_commutativity<std::complex<RealScalar>, std::complex<RealScalar>>::value> {}; + +template <typename RealScalar> +struct functor_is_commutative<scalar_product_op<std::complex<RealScalar>, RealScalar>> + : bool_constant<is_arithmetic<RealScalar>::value || + scalar_product_commutativity<std::complex<RealScalar>, RealScalar>::value> {}; + +template <typename RealScalar> +struct functor_is_commutative<scalar_product_op<RealScalar, std::complex<RealScalar>>> + : bool_constant<is_arithmetic<RealScalar>::value || + scalar_product_commutativity<RealScalar, std::complex<RealScalar>>::value> {}; // Same as scalar_product_op, but its scalar path uses pmul instead of operator*. For complex scalars // pmul is Eigen's explicit (non-Annex-G) complex multiply, so this avoids std::complex::operator*,
diff --git a/Eigen/src/Core/functors/UnaryFunctors.h b/Eigen/src/Core/functors/UnaryFunctors.h index c25cef0..62f88cc 100644 --- a/Eigen/src/Core/functors/UnaryFunctors.h +++ b/Eigen/src/Core/functors/UnaryFunctors.h
@@ -541,9 +541,7 @@ */ template <typename Scalar> struct scalar_log10_op { - EIGEN_DEVICE_FUNC constexpr inline Scalar operator()(const Scalar& a) const { - EIGEN_USING_STD(log10) return log10(a); - } + EIGEN_DEVICE_FUNC constexpr inline Scalar operator()(const Scalar& a) const { return numext::log10(a); } template <typename Packet> EIGEN_DEVICE_FUNC inline Packet packetOp(const Packet& a) const { return internal::plog10(a); @@ -1090,7 +1088,7 @@ template <typename Scalar, bool UseTypedPredicate = false> struct scalar_isnan_op { EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE bool operator()(const Scalar& a) const { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return numext::isnan(a); #else return numext::isnan EIGEN_NOT_A_MACRO(a); @@ -1101,7 +1099,7 @@ template <typename Scalar> struct scalar_isnan_op<Scalar, true> { EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE Scalar operator()(const Scalar& a) const { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return (numext::isnan(a) ? ptrue(a) : pzero(a)); #else return (numext::isnan EIGEN_NOT_A_MACRO(a) ? ptrue(a) : pzero(a)); @@ -1125,7 +1123,7 @@ template <typename Scalar, bool UseTypedPredicate = false> struct scalar_isinf_op { EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE bool operator()(const Scalar& a) const { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return numext::isinf(a); #else return (numext::isinf)(a); @@ -1136,7 +1134,7 @@ template <typename Scalar> struct scalar_isinf_op<Scalar, true> { EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE Scalar operator()(const Scalar& a) const { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return (numext::isinf(a) ? ptrue(a) : pzero(a)); #else return (numext::isinf EIGEN_NOT_A_MACRO(a) ? ptrue(a) : pzero(a)); @@ -1159,7 +1157,7 @@ template <typename Scalar, bool UseTypedPredicate = false> struct scalar_isfinite_op { EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE bool operator()(const Scalar& a) const { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return numext::isfinite(a); #else return (numext::isfinite)(a); @@ -1170,7 +1168,7 @@ template <typename Scalar> struct scalar_isfinite_op<Scalar, true> { EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE Scalar operator()(const Scalar& a) const { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return (numext::isfinite(a) ? ptrue(a) : pzero(a)); #else return (numext::isfinite EIGEN_NOT_A_MACRO(a) ? ptrue(a) : pzero(a)); @@ -1471,8 +1469,7 @@ } EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE Scalar operator()(const Scalar& a) const { - EIGEN_USING_STD(pow); - return static_cast<Scalar>(pow(a, m_exponent)); + return static_cast<Scalar>(numext::pow(a, m_exponent)); } template <typename Packet> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE Packet packetOp(const Packet& a) const { @@ -1495,6 +1492,7 @@ unary_pow::use_repeated_squaring<Scalar>(exponent)) {} EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Scalar operator()(const Scalar& a) const { if (m_use_repeated_squaring) return unary_pow::int_pow(a, m_exponent); + // std::pow can promote the complex component for a wider exponent before casting back to Scalar. EIGEN_USING_STD(pow); return static_cast<Scalar>(pow(a, m_exponent)); }
diff --git a/Eigen/src/Core/products/GeneralBlockPanelKernel.h b/Eigen/src/Core/products/GeneralBlockPanelKernel.h index 0a4bec3..b29edc3 100644 --- a/Eigen/src/Core/products/GeneralBlockPanelKernel.h +++ b/Eigen/src/Core/products/GeneralBlockPanelKernel.h
@@ -132,6 +132,41 @@ * * \sa setCpuCacheSizes */ +// GCC clones a function for constant arguments (IPA-CP, IPA-SRA), which would compile a runtime overwrite flag's +// body twice again; clang 18 and Apple clang 21 did not in our builds. +#if EIGEN_COMP_GNUC_STRICT +#define EIGEN_GEMM_NO_CLONE __attribute__((noclone)) +#else +#define EIGEN_GEMM_NO_CLONE +#endif + +// Tag for the gebp_kernel call that assigns C = alpha * A' * B' over the block instead of accumulating into it, +// independently of C's prior contents. A kernel offers it by declaring `static constexpr bool CanOverwrite = true` +// next to that overload; every other kernel, including user specializations, keeps the accumulate-only contract and +// the GEMM driver zeroes the result before calling it. A specialization inheriting a true CanOverwrite while hiding +// the tagged overload must declare it false. +struct gebp_overwrite_t {}; +template <typename Gebp, typename = void> +struct gebp_kernel_overwrites : std::false_type {}; +template <typename Gebp> +struct gebp_kernel_overwrites<Gebp, std::enable_if_t<Gebp::CanOverwrite>> : std::true_type {}; + +// Whether an overwriting kernel should overwrite a rows x cols x depth product rather than have the driver zero the +// result first: Gebp::overwrite_pays where the kernel declares it, true otherwise. +template <typename Gebp, typename Index> +EIGEN_ALWAYS_INLINE bool gebp_overwrite_pays_impl(int, Index rows, Index cols, Index depth, + decltype(Gebp::overwrite_pays(rows, cols, depth))* = nullptr) { + return Gebp::overwrite_pays(rows, cols, depth); +} +template <typename Gebp, typename Index> +EIGEN_ALWAYS_INLINE bool gebp_overwrite_pays_impl(long, Index, Index, Index) { + return true; +} +template <typename Gebp, typename Index> +EIGEN_ALWAYS_INLINE bool gebp_overwrite_pays(Index rows, Index cols, Index depth) { + return gebp_overwrite_pays_impl<Gebp>(0, rows, cols, depth); +} + #ifdef EIGEN_VECTORIZE_SME // True for the scalar pairs the SME gebp_kernel specializes (see // arch/SME/GeneralBlockPanelKernel.h, which static_asserts that it agrees with @@ -484,14 +519,15 @@ computeProductBlockingSizes<LhsScalar, RhsScalar, 1, Index>(k, m, n, num_threads); } -template <typename RhsPacket, typename RhsPacketx4, int registers_taken> +// rhs_registers is the number of registers one RhsPacket takes, so RhsPacketx4 takes four times that. +template <typename RhsPacket, typename RhsPacketx4, int registers_taken, int rhs_registers = 1> struct RhsPanelHelper { private: static constexpr int remaining_registers = (std::max)(int(EIGEN_ARCH_DEFAULT_NUMBER_OF_REGISTERS) - registers_taken, 0); public: - using type = std::conditional_t<remaining_registers >= 4, RhsPacketx4, RhsPacket>; + using type = std::conditional_t<remaining_registers >= 4 * rhs_registers, RhsPacketx4, RhsPacket>; }; template <typename Packet> @@ -786,6 +822,11 @@ }; template <typename Packet> +struct is_double_packet : std::false_type {}; +template <typename Packet> +struct is_double_packet<DoublePacket<Packet>> : std::true_type {}; + +template <typename Packet> DoublePacket<Packet> padd(const DoublePacket<Packet>& a, const DoublePacket<Packet>& b) { DoublePacket<Packet> res; res.first = padd(a.first, b.first); @@ -866,7 +907,12 @@ NumberOfRegisters = EIGEN_ARCH_DEFAULT_NUMBER_OF_REGISTERS, nr = 4, +#ifdef EIGEN_HAS_SINGLE_INSTRUCTION_MADD + // Each DoublePacket accumulator takes two registers. + mr = plain_enum_max(1, NumberOfRegisters / 4 / nr) * ResPacketSize, +#else mr = (plain_enum_min(16, NumberOfRegisters) / 2 / nr) * ResPacketSize, +#endif LhsProgress = ResPacketSize, RhsProgress = 1 @@ -1096,6 +1142,13 @@ * |real |cplx | no vectorization yet, would require to pack A with duplication * |cplx |real | easy vectorization */ +// The result scalars whose kernel stores may overwrite: assigning one is known to give what adding it onto a zero +// does. Other scalars, including custom ones over a built-in real type, keep accumulating. +template <typename Scalar> +struct gebp_overwritable_scalar : bool_constant<std::is_arithmetic<Scalar>::value> {}; +template <typename Real> +struct gebp_overwritable_scalar<std::complex<Real>> : bool_constant<std::is_arithmetic<Real>::value> {}; + template <typename LhsScalar, typename RhsScalar, typename Index, typename DataMapper, int mr, int nr, bool ConjugateLhs, bool ConjugateRhs> struct gebp_kernel { @@ -1142,11 +1195,52 @@ ResPacketSize = Traits::ResPacketSize }; - EIGEN_DONT_INLINE void operator()(const DataMapper& res, const LhsScalar* blockA, const RhsScalar* blockB, Index rows, - Index depth, Index cols, ResScalar alpha, Index strideA = -1, Index strideB = -1, - Index offsetA = 0, Index offsetB = 0) const; + // Overwriting starts each result packet from zero instead of loading it, so it computes exactly what accumulating + // into a zeroed result does (see gebp_overwritable_scalar for the scalars that holds for). + static constexpr bool CanOverwrite = gebp_overwritable_scalar<ResScalar>::value; + + EIGEN_STRONG_INLINE void operator()(const DataMapper& res, const LhsScalar* blockA, const RhsScalar* blockB, + Index rows, Index depth, Index cols, ResScalar alpha, Index strideA = -1, + Index strideB = -1, Index offsetA = 0, Index offsetB = 0) const { + run(false, res, blockA, blockB, rows, depth, cols, alpha, strideA, strideB, offsetA, offsetB); + } + // C = alpha * A' * B' over the block, whatever C held (see gebp_overwrite_t). + EIGEN_STRONG_INLINE void operator()(gebp_overwrite_t, const DataMapper& res, const LhsScalar* blockA, + const RhsScalar* blockB, Index rows, Index depth, Index cols, ResScalar alpha, + Index strideA = -1, Index strideB = -1, Index offsetA = 0, + Index offsetB = 0) const { + run(true, res, blockA, blockB, rows, depth, cols, alpha, strideA, strideB, offsetA, offsetB); + } + + // One body for both: `overwrite` is a runtime flag, so that assignment and accumulation share one + // instantiation. + EIGEN_DONT_INLINE EIGEN_GEMM_NO_CLONE void run(bool overwrite, const DataMapper& res, const LhsScalar* blockA, + const RhsScalar* blockB, Index rows, Index depth, Index cols, + ResScalar alpha, Index strideA, Index strideB, Index offsetA, + Index offsetB) const; }; +// One scalar result, alpha * c: added to the destination, or, overwriting, assigned as the sum onto zero that +// accumulating into a zeroed destination computes. A scalar that cannot overwrite (CanOverwrite) takes an overload +// without the overwriting form, so that accumulation asks of it only the `dst += alpha * c` it always did. +template <typename ResScalar, typename Alpha, typename Acc> +EIGEN_ALWAYS_INLINE void gebp_store_scalar_impl(std::false_type, bool, ResScalar& dst, const Alpha& alpha, + const Acc& c) { + dst += alpha * c; +} +template <typename ResScalar, typename Alpha, typename Acc> +EIGEN_ALWAYS_INLINE void gebp_store_scalar_impl(std::true_type, bool overwrite, ResScalar& dst, const Alpha& alpha, + const Acc& c) { + if (overwrite) + dst = ResScalar(0) + alpha * c; + else + dst += alpha * c; +} +template <bool CanOverwrite, typename ResScalar, typename Alpha, typename Acc> +EIGEN_ALWAYS_INLINE void gebp_store_scalar(bool overwrite, ResScalar& dst, const Alpha& alpha, const Acc& c) { + gebp_store_scalar_impl(bool_constant<CanOverwrite>(), overwrite, dst, alpha, c); +} + template <typename LhsScalar, typename RhsScalar, typename Index, typename DataMapper, int mr, int nr, bool ConjugateLhs, bool ConjugateRhs, int SwappedLhsProgress = @@ -1163,7 +1257,8 @@ EIGEN_STRONG_INLINE void operator()(const DataMapper& res, SwappedTraits& straits, const LhsScalar* blA, const RhsScalar* blB, Index depth, const Index endk, Index i, Index j2, - ResScalar alpha, SAccPacket& C0) const { + ResScalar alpha, SAccPacket& C0, bool overwrite = false) const { + EIGEN_UNUSED_VARIABLE(overwrite); EIGEN_UNUSED_VARIABLE(res); EIGEN_UNUSED_VARIABLE(straits); EIGEN_UNUSED_VARIABLE(blA); @@ -1191,13 +1286,14 @@ EIGEN_STRONG_INLINE void operator()(const DataMapper& res, SwappedTraits& straits, const LhsScalar* blA, const RhsScalar* blB, Index depth, const Index endk, Index i, Index j2, - ResScalar alpha, SAccPacket& C0) const { + ResScalar alpha, SAccPacket& C0, bool overwrite = false) const { using SResPacketQuarter = typename unpacket_traits<typename unpacket_traits<SResPacket>::half>::half; using SLhsPacketQuarter = typename unpacket_traits<typename unpacket_traits<SLhsPacket>::half>::half; using SRhsPacketQuarter = typename unpacket_traits<typename unpacket_traits<SRhsPacket>::half>::half; using SAccPacketQuarter = typename unpacket_traits<typename unpacket_traits<SAccPacket>::half>::half; - SResPacketQuarter R = res.template gatherPacket<SResPacketQuarter>(i, j2); + SResPacketQuarter R = + overwrite ? pset1<SResPacketQuarter>(ResScalar(0)) : res.template gatherPacket<SResPacketQuarter>(i, j2); SResPacketQuarter alphav = pset1<SResPacketQuarter>(alpha); if (depth - endk > 0) { @@ -1258,6 +1354,17 @@ } }; +// EIGEN_GEBP_NEON_GCC_WORKAROUNDS: AArch64 NEON with GCC >= 9, where the micro-kernel needs help keeping its +// accumulators in registers. +// EIGEN_GEBP_UNROLL_ACCUMULATORS: fully unrolls a loop over the accumulators (at most 3 x 8 = 24), exposing constant +// indices before GCC's scalar replacement pass at -O2. +#if EIGEN_ARCH_ARM64 && defined(EIGEN_VECTORIZE_NEON) && EIGEN_GNUC_STRICT_AT_LEAST(9, 0, 0) +#define EIGEN_GEBP_NEON_GCC_WORKAROUNDS +#define EIGEN_GEBP_UNROLL_ACCUMULATORS _Pragma("GCC unroll 24") +#else +#define EIGEN_GEBP_UNROLL_ACCUMULATORS +#endif + // One step of the micro-kernel: loads MrPackets LHS packets at step K, // then processes NrCols RHS columns via gebp_rhs_cols. template <int K, int MrPackets, int NrCols> @@ -1273,10 +1380,27 @@ EIGEN_IF_CONSTEXPR (MrPackets >= 3) traits.loadLhs(&blA[(2 + MrPackets * K) * LhsProg], A[2]); gebp_rhs_cols<0, MrPackets, NrCols>::run(traits, blB, Index(NrCols * K), A, rhs_panel, T0, C); +#if defined(EIGEN_GEBP_NEON_GCC_WORKAROUNDS) && !defined(EIGEN_DONT_DISABLE_GEBP_INSN_SCHEDULING) + using AccPacket = typename GEBPTraits::AccPacket; + constexpr bool kRealPacket = std::is_same<AccPacket, Packet4f>::value || std::is_same<AccPacket, Packet2d>::value; + EIGEN_IF_CONSTEXPR (MrPackets == 3 && NrCols == 8 && kRealPacket) { + // Keep the next depth step's loads from extending live ranges beyond the 24 accumulators. + __asm__ volatile("" ::: "memory"); +#if EIGEN_GNUC_STRICT_AT_LEAST(16, 0, 0) + // GCC 16 also needs the double accumulator updates to stay on their own depth step. + EIGEN_IF_CONSTEXPR (std::is_same<AccPacket, Packet2d>::value) { + EIGEN_GEBP_UNROLL_ACCUMULATORS + for (int n = 0; n < MrPackets * NrCols; ++n) { + EIGEN_OPTIMIZATION_BARRIER(C[n]); + } + } +#endif + } +#endif } }; // Compiler register allocation workarounds for the GEBP micro-kernel. -// GCC can fail to keep array-based SIMD values in vector registers, causing +// GCC and clang can fail to keep array-based SIMD values in vector registers, causing // excessive spilling. These helpers use inline asm constraints to pin values. // Only applied when the scalar type is actually vectorizable (not custom types). // See Eigen bugs 935, 1637, and 3059. @@ -1326,6 +1450,48 @@ #endif } +// Clang AVX/AVX2 on x86-64: the 3pX4 tile needs all 16 ymm registers (12 accumulators, 3 LHS packets, 1 RHS +// broadcast), and clang keeps accumulators on the stack, reloading and storing them on every depth step: a dependency +// chain through store forwarding that roughly halves the kernel's throughput. Pinned accumulators make clang spill LHS +// packets instead, which are off those chains. 32-bit x86 has 8 ymm registers, so its accumulators spill anyway and +// pinning only adds stack traffic. A barrier sends a class through memory, so complex packets are pinned through their +// vector member. Not Packet8i: its madd needs a 17th register, and pinning made rare stack layouts up to 2.8x slower. +#if EIGEN_COMP_CLANG && EIGEN_ARCH_x86_64 && defined(EIGEN_VECTORIZE_AVX) && !defined(EIGEN_VECTORIZE_AVX512) +#define EIGEN_GEBP_AVX_CLANG_PIN_ACCUMULATORS +template <typename Packet> +EIGEN_ALWAYS_INLINE void gebp_avx_clang_pin_accumulator(Packet& x) { + pbarrier(x); +} +EIGEN_ALWAYS_INLINE void gebp_avx_clang_pin_accumulator(Packet4cf& x) { pbarrier(x.v); } +EIGEN_ALWAYS_INLINE void gebp_avx_clang_pin_accumulator(Packet2cd& x) { pbarrier(x.v); } +#endif +template <int MrPackets, int NrCols, typename GEBPTraits_, typename FullLhsPacket_, typename AccArray_> +EIGEN_ALWAYS_INLINE void gebp_avx_clang_spilling_workaround(AccArray_& ACC) { + EIGEN_UNUSED_VARIABLE(ACC); +#ifdef EIGEN_GEBP_AVX_CLANG_PIN_ACCUMULATORS + using AccElement = std::decay_t<decltype(ACC[0])>; + constexpr bool kFloatingPoint = + std::is_same<AccElement, Packet8f>::value || std::is_same<AccElement, Packet4d>::value || + std::is_same<AccElement, Packet4cf>::value || std::is_same<AccElement, Packet2cd>::value; + constexpr bool apply = GEBPTraits_::Vectorizable && MrPackets == 3 && NrCols == 4 && kFloatingPoint && + std::is_same<AccElement, FullLhsPacket_>::value; + EIGEN_IF_CONSTEXPR (apply) { + for (int n = 0; n < MrPackets * NrCols; ++n) gebp_avx_clang_pin_accumulator(ACC[n]); + } +#endif +} + +// Double-accumulation trick for 1pX4 path to break FMA dependency chains. A DoublePacket +// accumulator already holds two chains, and with 16 registers doubling it would spill. +template <typename AccPacket, int MrPackets, int NrCols> +struct gebp_use_double_accum { + static constexpr bool value = MrPackets == 1 && NrCols == 4 +#ifdef EIGEN_HAS_SINGLE_INSTRUCTION_MADD + && !(is_double_packet<AccPacket>::value && EIGEN_ARCH_DEFAULT_NUMBER_OF_REGISTERS < 32) +#endif + ; +}; + // Unrolled peeled loop body: calls gebp_micro_step for K=0..7, handling // double-accumulation for 1pX4, prefetches, and compiler workarounds. template <int MrPackets, int NrCols> @@ -1334,7 +1500,7 @@ typename RhsPacketType, typename AccArray, typename AccArrayD, typename FullLhsPacket> static EIGEN_ALWAYS_INLINE void run(GEBPTraits& traits, const LhsScalar_* blA, const RhsScalar_* blB, LhsArray& A, RhsPanelType& rhs_panel, RhsPacketType& T0, AccArray& C, AccArrayD& D) { - constexpr bool use_double_accum = (MrPackets == 1 && NrCols == 4); + constexpr bool use_double_accum = gebp_use_double_accum<typename GEBPTraits::AccPacket, MrPackets, NrCols>::value; // Prefetch for 4-col paths EIGEN_IF_CONSTEXPR (NrCols == 4) { @@ -1347,6 +1513,7 @@ gebp_micro_step<KVAL, MrPackets, NrCols>::run(traits, blA, blB, A, rhs_panel, T0, ACC); \ gebp_neon_3p_workaround<MrPackets, GEBPTraits, FullLhsPacket>(A); \ gebp_sse_spilling_workaround<MrPackets, NrCols, GEBPTraits, FullLhsPacket>(A, ACC); \ + gebp_avx_clang_spilling_workaround<MrPackets, NrCols, GEBPTraits, FullLhsPacket>(ACC); \ /* LHS prefetch for 2pX4 and 3pX4 */ \ EIGEN_IF_CONSTEXPR ((MrPackets == 2 || MrPackets == 3) && NrCols == 4) { \ internal::prefetch(blA + (MrPackets * KVAL + 16) * GEBPTraits::LhsProgress); \ @@ -1394,7 +1561,8 @@ EIGEN_ALWAYS_INLINE void gebp_micro_panel_impl(GEBPTraits& traits, const DataMapper_& res, const LhsScalar_* blockA, const RhsScalar_* blockB, ResScalar_ alpha, Index_ i, Index_ j2, Index_ depth, Index_ strideA, Index_ strideB, Index_ offsetA, - Index_ offsetB, int prefetch_res_offset, Index_ peeled_kc, int pk) { + Index_ offsetB, int prefetch_res_offset, Index_ peeled_kc, int pk, + bool overwrite) { using LhsPacketLocal = typename GEBPTraits::LhsPacket; using RhsPacketLocal = typename GEBPTraits::RhsPacket; using ResPacketLocal = typename GEBPTraits::ResPacket; @@ -1404,10 +1572,14 @@ constexpr int RhsProg = GEBPTraits::RhsProgress; constexpr int ResPacketSz = GEBPTraits::ResPacketSize; - // Determine RhsPanel type based on register pressure - using RhsPanelType = std::conditional_t< - NrCols == 1, RhsPacketLocal, - typename RhsPanelHelper<RhsPacketLocal, RhsPacketx4Local, MrPackets * NrCols + MrPackets>::type>; + // Determine RhsPanel type based on register pressure. A DoublePacket takes two registers. + constexpr bool use_double_accum = gebp_use_double_accum<AccPacketLocal, MrPackets, NrCols>::value; + constexpr int AccRegisters = is_double_packet<AccPacketLocal>::value ? 2 : 1; + constexpr int RhsRegisters = is_double_packet<RhsPacketLocal>::value ? 2 : 1; + constexpr int RegistersTaken = AccRegisters * (MrPackets + use_double_accum) * NrCols + MrPackets; + using RhsPanelType = + std::conditional_t<NrCols == 1, RhsPacketLocal, + typename RhsPanelHelper<RhsPacketLocal, RhsPacketx4Local, RegistersTaken, RhsRegisters>::type>; const LhsScalar_* blA = &blockA[i * strideA + offsetA * (MrPackets * LhsProg)]; prefetch(&blA[0]); @@ -1421,10 +1593,9 @@ constexpr int CSize = 3 * NrCols > MrPackets * NrCols ? 3 * NrCols : MrPackets * NrCols; #endif alignas(AccPacketLocal) AccPacketLocal C[CSize]; + EIGEN_GEBP_UNROLL_ACCUMULATORS for (int n = 0; n < MrPackets * NrCols; ++n) traits.initAcc(C[n]); - // Double-accumulation trick for 1pX4 path to break FMA dependency chains - constexpr bool use_double_accum = (MrPackets == 1 && NrCols == 4); #ifdef EIGEN_HAS_CXX17_IFCONSTEXPR alignas(AccPacketLocal) AccPacketLocal D[use_double_accum ? NrCols : 1]; #else @@ -1469,6 +1640,7 @@ // Merge double accumulators EIGEN_IF_CONSTEXPR (use_double_accum) { + EIGEN_GEBP_UNROLL_ACCUMULATORS for (int n = 0; n < NrCols; ++n) C[n] = padd(C[n], D[n]); } @@ -1485,15 +1657,21 @@ // ---- Store results: C[j + p * NrCols] -> res(i + p*ResPacketSz, j2 + j) ---- alignas(ResPacketLocal) ResPacketLocal alphav = pset1<ResPacketLocal>(alpha); + EIGEN_GEBP_UNROLL_ACCUMULATORS for (int j = 0; j < NrCols; ++j) { LinearMapper_ r = res.getLinearMapper(i, j2 + j); + EIGEN_GEBP_UNROLL_ACCUMULATORS for (int p = 0; p < MrPackets; ++p) { - alignas(ResPacketLocal) ResPacketLocal R = r.template loadPacket<ResPacketLocal>(p * ResPacketSz); + alignas(ResPacketLocal) ResPacketLocal R = + overwrite ? pset1<ResPacketLocal>(ResScalar_(0)) : r.template loadPacket<ResPacketLocal>(p * ResPacketSz); traits.acc(C[j + p * NrCols], alphav, R); r.storePacket(p * ResPacketSz, R); } } } +#undef EIGEN_GEBP_UNROLL_ACCUMULATORS +#undef EIGEN_GEBP_NEON_GCC_WORKAROUNDS +#undef EIGEN_GEBP_AVX_CLANG_PIN_ACCUMULATORS // Workaround a GCC/AArch64 register-allocation issue (through at least GCC 14.3): the float // kernel's mr=12 x nr=8 tile holds 24 of the 32 NEON registers, and GCC's pre-RA scheduler @@ -1501,7 +1679,8 @@ // ~halving GEMM throughput (~2.1x fp32 on Cortex-X925; spills 20 -> 2 with it off). Clang // allocates the same tile spill-free, so we keep nr=8 and just disable that GCC pass here. // This is a known bug in GCC: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100697 -// In-order cores can opt out via EIGEN_DONT_DISABLE_GEBP_INSN_SCHEDULING. +// In-order cores can opt out via EIGEN_DONT_DISABLE_GEBP_INSN_SCHEDULING, which also drops the +// depth-step barriers in gebp_micro_step. #if EIGEN_COMP_GNUC_STRICT && EIGEN_ARCH_ARM64 && !defined(EIGEN_DONT_DISABLE_GEBP_INSN_SCHEDULING) #pragma GCC push_options #pragma GCC optimize("no-schedule-insns") @@ -1509,11 +1688,12 @@ #endif template <typename LhsScalar, typename RhsScalar, typename Index, typename DataMapper, int mr, int nr, bool ConjugateLhs, bool ConjugateRhs> -EIGEN_DONT_INLINE void gebp_kernel<LhsScalar, RhsScalar, Index, DataMapper, mr, nr, ConjugateLhs, - ConjugateRhs>::operator()(const DataMapper& res, const LhsScalar* blockA, - const RhsScalar* blockB, Index rows, Index depth, - Index cols, ResScalar alpha, Index strideA, Index strideB, - Index offsetA, Index offsetB) const { +EIGEN_DONT_INLINE EIGEN_GEMM_NO_CLONE void +gebp_kernel<LhsScalar, RhsScalar, Index, DataMapper, mr, nr, ConjugateLhs, ConjugateRhs>::run( + bool overwrite_, const DataMapper& res, const LhsScalar* blockA, const RhsScalar* blockB, Index rows, Index depth, + Index cols, ResScalar alpha, Index strideA, Index strideB, Index offsetA, Index offsetB) const { + // Constant false where the scalar cannot overwrite, so that the overwriting stores fold away. + const bool overwrite = CanOverwrite && overwrite_; Traits traits; SwappedTraits straits; @@ -1548,7 +1728,7 @@ using LTraits = std::remove_reference_t<decltype(local_traits)>; gebp_micro_panel_impl<MrP, NrC, LTraits, LhsScalar, RhsScalar, ResScalar, Index, DataMapper, LinearMapper, LhsPacket>(local_traits, res, blockA, blockB, alpha, i, j2, depth, strideA, strideB, offsetA, - offsetB, prefetch_res_offset, peeled_kc, pk); + offsetB, prefetch_res_offset, peeled_kc, pk, overwrite); }; // Budget (in bytes) for co-residency of the RHS block and a strip of the @@ -1632,7 +1812,8 @@ } //---------- Process 1 * LhsProgress rows at once ---------- - EIGEN_IF_CONSTEXPR (mr >= 1 * Traits::LhsProgress) { + // With a wider mr, at most one row panel is left here. + EIGEN_IF_CONSTEXPR (mr > 1 * Traits::LhsProgress) { for (Index i = peeled_mc2; i < peeled_mc1; i += LhsProgress) { EIGEN_IF_CONSTEXPR (nr >= 8) { for (Index j2 = 0; j2 < packet_cols8; j2 += 8) { @@ -1646,6 +1827,36 @@ micro_panel(fix<1>, fix<1>, traits, i, j2); } } + } else EIGEN_IF_CONSTEXPR (mr == 1 * Traits::LhsProgress) { + // Every row comes here, blocked like the wider panels above: rows inside columns keep the rhs panel in cache + // and walk C down its columns. + const Index rhs_block1 = sizeof(ResScalar) * mr * nr + depth * nr * sizeof(RhsScalar); + const Index lhs_strip1 = depth * sizeof(LhsScalar) * LhsProgress; + const Index lhs_avail1 = (lhs_budget > rhs_block1) ? (lhs_budget - rhs_block1) : 0; + const Index mc1_range = peeled_mc1 - peeled_mc2; + const Index actual_panel_rows = (lhs_avail1 >= mc1_range * depth * static_cast<Index>(sizeof(LhsScalar))) + ? mc1_range + : LhsProgress * std::max<Index>(1, lhs_avail1 / lhs_strip1); + for (Index i1 = peeled_mc2; i1 < peeled_mc1; i1 += actual_panel_rows) { + const Index actual_panel_end = (std::min)(i1 + actual_panel_rows, peeled_mc1); + EIGEN_IF_CONSTEXPR (nr >= 8) { + for (Index j2 = 0; j2 < packet_cols8; j2 += 8) { + for (Index i = i1; i < actual_panel_end; i += LhsProgress) { + micro_panel(fix<1>, fix<8>, traits, i, j2); + } + } + } + for (Index j2 = packet_cols8; j2 < packet_cols4; j2 += 4) { + for (Index i = i1; i < actual_panel_end; i += LhsProgress) { + micro_panel(fix<1>, fix<4>, traits, i, j2); + } + } + for (Index j2 = packet_cols4; j2 < cols; j2++) { + for (Index i = i1; i < actual_panel_end; i += LhsProgress) { + micro_panel(fix<1>, fix<1>, traits, i, j2); + } + } + } } //---------- Process LhsProgressHalf rows at once ---------- @@ -1656,18 +1867,18 @@ for (Index j2 = 0; j2 < packet_cols8; j2 += 8) { gebp_micro_panel_impl<1, 8, HalfTraits, LhsScalar, RhsScalar, ResScalar, Index, DataMapper, LinearMapper, LhsPacket>(half_traits, res, blockA, blockB, alpha, i, j2, depth, strideA, strideB, - offsetA, offsetB, prefetch_res_offset, peeled_kc, pk); + offsetA, offsetB, prefetch_res_offset, peeled_kc, pk, overwrite); } } for (Index j2 = packet_cols8; j2 < packet_cols4; j2 += 4) { gebp_micro_panel_impl<1, 4, HalfTraits, LhsScalar, RhsScalar, ResScalar, Index, DataMapper, LinearMapper, LhsPacket>(half_traits, res, blockA, blockB, alpha, i, j2, depth, strideA, strideB, - offsetA, offsetB, prefetch_res_offset, peeled_kc, pk); + offsetA, offsetB, prefetch_res_offset, peeled_kc, pk, overwrite); } for (Index j2 = packet_cols4; j2 < cols; j2++) { gebp_micro_panel_impl<1, 1, HalfTraits, LhsScalar, RhsScalar, ResScalar, Index, DataMapper, LinearMapper, LhsPacket>(half_traits, res, blockA, blockB, alpha, i, j2, depth, strideA, strideB, - offsetA, offsetB, prefetch_res_offset, peeled_kc, pk); + offsetA, offsetB, prefetch_res_offset, peeled_kc, pk, overwrite); } } } @@ -1680,18 +1891,18 @@ for (Index j2 = 0; j2 < packet_cols8; j2 += 8) { gebp_micro_panel_impl<1, 8, QuarterTraits, LhsScalar, RhsScalar, ResScalar, Index, DataMapper, LinearMapper, LhsPacket>(quarter_traits, res, blockA, blockB, alpha, i, j2, depth, strideA, strideB, - offsetA, offsetB, prefetch_res_offset, peeled_kc, pk); + offsetA, offsetB, prefetch_res_offset, peeled_kc, pk, overwrite); } } for (Index j2 = packet_cols8; j2 < packet_cols4; j2 += 4) { gebp_micro_panel_impl<1, 4, QuarterTraits, LhsScalar, RhsScalar, ResScalar, Index, DataMapper, LinearMapper, LhsPacket>(quarter_traits, res, blockA, blockB, alpha, i, j2, depth, strideA, strideB, - offsetA, offsetB, prefetch_res_offset, peeled_kc, pk); + offsetA, offsetB, prefetch_res_offset, peeled_kc, pk, overwrite); } for (Index j2 = packet_cols4; j2 < cols; j2++) { gebp_micro_panel_impl<1, 1, QuarterTraits, LhsScalar, RhsScalar, ResScalar, Index, DataMapper, LinearMapper, LhsPacket>(quarter_traits, res, blockA, blockB, alpha, i, j2, depth, strideA, strideB, - offsetA, offsetB, prefetch_res_offset, peeled_kc, pk); + offsetA, offsetB, prefetch_res_offset, peeled_kc, pk, overwrite); } } } @@ -1738,14 +1949,14 @@ blB += 8; } - res(i, j2 + 0) += alpha * C0; - res(i, j2 + 1) += alpha * C1; - res(i, j2 + 2) += alpha * C2; - res(i, j2 + 3) += alpha * C3; - res(i, j2 + 4) += alpha * C4; - res(i, j2 + 5) += alpha * C5; - res(i, j2 + 6) += alpha * C6; - res(i, j2 + 7) += alpha * C7; + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 0), alpha, C0); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 1), alpha, C1); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 2), alpha, C2); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 3), alpha, C3); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 4), alpha, C4); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 5), alpha, C5); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 6), alpha, C6); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 7), alpha, C7); } } } @@ -1834,7 +2045,8 @@ SAccPacket> SAccPacketHalf; - SResPacketHalf R = res.template gatherPacket<SResPacketHalf>(i, j2); + SResPacketHalf R = + overwrite ? pset1<SResPacketHalf>(ResScalar(0)) : res.template gatherPacket<SResPacketHalf>(i, j2); SResPacketHalf alphav = pset1<SResPacketHalf>(alpha); if (depth - endk > 0) { @@ -1856,9 +2068,9 @@ // template form, so that LhsProgress < 16 paths don't // fail to compile last_row_process_16_packets<LhsScalar, RhsScalar, Index, DataMapper, mr, nr, ConjugateLhs, ConjugateRhs> p; - p(res, straits, blA, blB, depth, endk, i, j2, alpha, C0); + p(res, straits, blA, blB, depth, endk, i, j2, alpha, C0, overwrite); } else { - SResPacket R = res.template gatherPacket<SResPacket>(i, j2); + SResPacket R = overwrite ? pset1<SResPacket>(ResScalar(0)) : res.template gatherPacket<SResPacket>(i, j2); SResPacket alphav = pset1<SResPacket>(alpha); straits.acc(C0, alphav, R); res.scatterPacket(i, j2, R); @@ -1886,10 +2098,10 @@ blB += 4; } - res(i, j2 + 0) += alpha * C0; - res(i, j2 + 1) += alpha * C1; - res(i, j2 + 2) += alpha * C2; - res(i, j2 + 3) += alpha * C3; + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 0), alpha, C0); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 1), alpha, C1); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 2), alpha, C2); + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2 + 3), alpha, C3); } } } @@ -1907,7 +2119,7 @@ RhsScalar B_0 = blB[k]; C0 = cj.pmadd(A0, B_0, C0); } - res(i, j2) += alpha * C0; + gebp_store_scalar<CanOverwrite>(overwrite, res(i, j2), alpha, C0); } } } @@ -2521,7 +2733,7 @@ } // end namespace internal /** \returns the currently set level 1 cpu cache size (in bytes) used to estimate the ideal blocking size parameters. - * \sa setCpuCacheSize */ + * \sa setCpuCacheSizes */ inline std::ptrdiff_t l1CacheSize() { std::ptrdiff_t l1, l2, l3; internal::manage_caching_sizes(GetAction, &l1, &l2, &l3); @@ -2529,7 +2741,7 @@ } /** \returns the currently set level 2 cpu cache size (in bytes) used to estimate the ideal blocking size parameters. - * \sa setCpuCacheSize */ + * \sa setCpuCacheSizes */ inline std::ptrdiff_t l2CacheSize() { std::ptrdiff_t l1, l2, l3; internal::manage_caching_sizes(GetAction, &l1, &l2, &l3); @@ -2537,7 +2749,7 @@ } /** \returns the currently set level 3 cpu cache size (in bytes) used to estimate the ideal blocking size parameters. - * \sa setCpuCacheSize */ + * \sa setCpuCacheSizes */ inline std::ptrdiff_t l3CacheSize() { std::ptrdiff_t l1, l2, l3; internal::manage_caching_sizes(GetAction, &l1, &l2, &l3); @@ -2546,6 +2758,8 @@ /** Set the cpu L1, L2 and L3 cache sizes (in bytes). * These values are used to adjust blocking sizes and, on some backends, the size crossovers between kernels. + * By default they are queried from the CPU that runs the first product. Setting them explicitly makes the + * blocking, and therefore the rounding of the results, independent of that CPU; see \ref TopicPitfalls_reproducibility. * * \sa computeProductBlockingSizes */ inline void setCpuCacheSizes(std::ptrdiff_t l1, std::ptrdiff_t l2, std::ptrdiff_t l3) {
diff --git a/Eigen/src/Core/products/GeneralMatrixMatrix.h b/Eigen/src/Core/products/GeneralMatrixMatrix.h index 62c1e5f..b65a50d 100644 --- a/Eigen/src/Core/products/GeneralMatrixMatrix.h +++ b/Eigen/src/Core/products/GeneralMatrixMatrix.h
@@ -21,15 +21,98 @@ template <typename LhsScalar_, typename RhsScalar_> class level3_blocking; +// Coalesce two heap packing buffers so glibc's dynamic thresholds learn the whole workspace size (#3201). +// Leave stack allocations and caller-provided buffers on their existing paths. +template <typename LhsScalar, typename RhsScalar> +class gemm_packing_space { + LhsScalar* m_blockA; + RhsScalar* m_blockB; + aligned_stack_memory_handler<unsigned char> m_storage; + aligned_stack_memory_handler<LhsScalar> m_lhs_lifetime; + aligned_stack_memory_handler<RhsScalar> m_rhs_lifetime; + + static EIGEN_DONT_INLINE unsigned char* allocateCombined(std::size_t sizeA, std::size_t sizeB, LhsScalar*& blockA, + RhsScalar*& blockB) { + check_size_for_overflow<LhsScalar>(sizeA); + check_size_for_overflow<RhsScalar>(sizeB); + // Not numext::maxi: under SYCL_DEVICE_ONLY its cl_ulong (std::size_t) specialization is not constexpr. + constexpr std::size_t alignment = (std::max)(std::size_t{EIGEN_STACK_ALIGN_BYTES}, alignof(RhsScalar)); + const std::size_t max_bytes = (std::numeric_limits<std::ptrdiff_t>::max)(); + const std::size_t bytesA = sizeA * sizeof(LhsScalar); + const std::size_t bytesB = sizeB * sizeof(RhsScalar); + if (bytesA > max_bytes - (alignment - 1)) throw_std_bad_alloc(); + const std::size_t offsetB = first_multiple(bytesA, alignment); + if (bytesB > max_bytes - offsetB) throw_std_bad_alloc(); + void* storage = scratch_malloc(offsetB + bytesB); + void* storageB = static_cast<unsigned char*>(storage) + offsetB; + blockA = static_cast<LhsScalar*>(storage); + blockB = static_cast<RhsScalar*>(storageB); + return static_cast<unsigned char*>(storage); + } + + static unsigned char* allocate(std::size_t sizeA, std::size_t sizeB, LhsScalar*& blockA, RhsScalar*& blockB) { + if (blockA != nullptr || blockB != nullptr || sizeA == 0 || sizeB == 0) return nullptr; +#if defined(EIGEN_ALLOCA) && !defined(EIGEN_NO_ALLOCA) + if (sizeA <= EIGEN_STACK_ALLOCATION_LIMIT / sizeof(LhsScalar) || + sizeB <= EIGEN_STACK_ALLOCATION_LIMIT / sizeof(RhsScalar)) + return nullptr; +#endif + // scratch_malloc guarantees only EIGEN_STACK_ALIGN_BYTES, or malloc's alignment when that is 0. + constexpr std::size_t base_alignment = + EIGEN_STACK_ALIGN_BYTES > 0 ? EIGEN_STACK_ALIGN_BYTES : alignof(std::max_align_t); + if ((std::max)(alignof(LhsScalar), alignof(RhsScalar)) > base_alignment) return nullptr; + return allocateCombined(sizeA, sizeB, blockA, blockB); + } + + public: + gemm_packing_space(std::size_t sizeA, std::size_t sizeB, LhsScalar* blockA, RhsScalar* blockB) + : m_blockA(blockA), + m_blockB(blockB), + m_storage(allocate(sizeA, sizeB, m_blockA, m_blockB), 0, true), + m_lhs_lifetime(blockA == nullptr ? m_blockA : nullptr, sizeA, false), + m_rhs_lifetime(blockB == nullptr ? m_blockB : nullptr, sizeB, false) {} + + LhsScalar* blockA() const { return m_blockA; } + RhsScalar* blockB() const { return m_blockB; } +}; + +// Runs one gebp call, as C = alpha * A' * B' when `overwrite` holds and the kernel can (gebp_kernel_overwrites), and +// as C += alpha * A' * B' otherwise. It and the loop policies are forced inline into the driver: GCC 13 otherwise +// clones a policy (IPA-SRA) with the kernel call still in it, and emits kernels that only that call's dead branches +// reach (the AVX-512 kernel's non-unit-increment ones). +template <typename Gebp, typename... Args> +EIGEN_ALWAYS_INLINE void gebp_run(std::true_type, Gebp& gebp, bool overwrite, const Args&... args) { + if (overwrite) + gebp(gebp_overwrite_t(), args...); + else + gebp(args...); +} +template <typename Gebp, typename... Args> +EIGEN_ALWAYS_INLINE void gebp_run(std::false_type, Gebp& gebp, bool, const Args&... args) { + gebp(args...); +} + +// Zeroes the rows x cols column-major result block at `res`. +template <typename Scalar, typename Index> +void gemm_zero_result(Scalar* res, Index rows, Index cols, Index resIncr, Index resStride) { + using Block = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + if (resIncr == 1 && resStride == rows) + Map<Matrix<Scalar, Dynamic, 1>>(res, rows * cols).setZero(); + else if (resIncr == 1) + Map<Block, 0, OuterStride<>>(res, rows, cols, OuterStride<>(resStride)).setZero(); + else + Map<Block, 0, Stride<Dynamic, Dynamic>>(res, rows, cols, Stride<Dynamic, Dynamic>(resStride, resIncr)).setZero(); +} + // LHS-first loop order: mc -> kc -> nc. This is Eigen's usual sequential // schedule, with a fast path that can pack RHS once for tall skinny blocking. struct gemm_pack_lhs_first_loop_policy { template <typename Index, typename LhsScalar, typename RhsScalar, typename ResScalar, typename LhsMapper, typename RhsMapper, typename ResMapper, typename PackLhs, typename PackRhs, typename Gebp> - static EIGEN_STRONG_INLINE void run(Index rows, Index cols, Index depth, Index kc, Index mc, Index nc, + static EIGEN_ALWAYS_INLINE void run(Index rows, Index cols, Index depth, Index kc, Index mc, Index nc, const LhsMapper& lhs, const RhsMapper& rhs, ResMapper& res, PackLhs& pack_lhs, PackRhs& pack_rhs, Gebp& gebp, LhsScalar* blockA, RhsScalar* blockB, - ResScalar alpha) { + ResScalar alpha, bool overwrite) { const bool pack_rhs_once = mc != rows && kc == depth && nc == cols; // For each horizontal panel of the rhs, and corresponding panel of the lhs... @@ -55,7 +138,8 @@ if ((!pack_rhs_once) || i2 == 0) pack_rhs(blockB, rhs.getSubMapper(k2, j2), actual_kc, actual_nc); // Everything is packed, we can now call the panel * block kernel: - gebp(res.getSubMapper(i2, j2), blockA, blockB, actual_mc, actual_kc, actual_nc, alpha); + gebp_run(bool_constant<gebp_kernel_overwrites<Gebp>::value>(), gebp, overwrite && k2 == 0, + res.getSubMapper(i2, j2), blockA, blockB, actual_mc, actual_kc, actual_nc, alpha); } } } @@ -76,12 +160,25 @@ : bool_constant<!NumTraits<Scalar>::IsComplex> {}; // Runs the block on the in-place LHS when the kernel can take it; the false // overload keeps the call out of every other instantiation. +// The in-place call, tagged to overwrite only for a kernel that can (gebp_run). +template <typename Gebp, typename... Args> +EIGEN_ALWAYS_INLINE void sme_direct_lhs_call(std::true_type, Gebp& gebp, bool overwrite, const Args&... args) { + if (overwrite) + gebp.run_direct_lhs(gebp_overwrite_t(), args...); + else + gebp.run_direct_lhs(args...); +} +template <typename Gebp, typename... Args> +EIGEN_ALWAYS_INLINE void sme_direct_lhs_call(std::false_type, Gebp& gebp, bool, const Args&... args) { + gebp.run_direct_lhs(args...); +} template <typename Gebp, typename ResMapper, typename LhsMapper, typename Scalar, typename ResScalar, typename Index> EIGEN_ALWAYS_INLINE bool sme_run_direct_lhs(std::true_type, Gebp& gebp, const ResMapper& res, const LhsMapper& lhs, Index i2, Index k2, const Scalar* blockB, Index mc, Index kc, Index nc, - ResScalar alpha) { + ResScalar alpha, bool overwrite) { if (!sme_direct_lhs_ok<Scalar>(lhs.stride(), mc, kc, nc)) return false; - gebp.run_direct_lhs(res, &lhs(i2, k2), lhs.stride(), blockB, mc, kc, nc, alpha); + sme_direct_lhs_call(bool_constant<gebp_kernel_overwrites<Gebp>::value>(), gebp, overwrite, res, &lhs(i2, k2), + lhs.stride(), blockB, mc, kc, nc, alpha); return true; } template <typename Scalar, int LhsOrder, int RhsOrder, typename Index> @@ -125,7 +222,7 @@ } template <typename Gebp, typename ResMapper, typename LhsMapper, typename Scalar, typename ResScalar, typename Index> EIGEN_ALWAYS_INLINE bool sme_run_direct_lhs(std::false_type, Gebp&, const ResMapper&, const LhsMapper&, Index, Index, - const Scalar*, Index, Index, Index, ResScalar) { + const Scalar*, Index, Index, Index, ResScalar, bool) { return false; } template <typename Scalar, typename Index> @@ -137,10 +234,12 @@ // The test stays inline: a call into the NEON path's frame dirties stack lines the ZA path then packs into. template <int LhsOrder, int RhsOrder, bool ConjLhs, bool ConjRhs, int ResInnerStride, typename Scalar, typename Index, typename Blocking> -EIGEN_ALWAYS_INLINE bool sme_run_small_gemm(std::true_type, Index rows, Index cols, Index depth, const Scalar* lhs, - Index lhsStride, const Scalar* rhs, Index rhsStride, Scalar* res, - Index resIncr, Index resStride, Scalar alpha, Blocking& blocking) { +EIGEN_ALWAYS_INLINE bool sme_run_small_gemm(std::true_type, bool overwrite, Index rows, Index cols, Index depth, + const Scalar* lhs, Index lhsStride, const Scalar* rhs, Index rhsStride, + Scalar* res, Index resIncr, Index resStride, Scalar alpha, + Blocking& blocking) { if (!sme_small_gemm_wins<Scalar>(rows, cols, depth)) return false; + if (overwrite) gemm_zero_result(res, rows, cols, resIncr, resStride); const bool fits = depth <= blocking.kc() && rows <= blocking.mc() && cols <= blocking.nc(); sme_small_gemm<Scalar, LhsOrder, RhsOrder, ConjLhs, ConjRhs, ResInnerStride>( rows, cols, depth, lhs, lhsStride, rhs, rhsStride, res, resIncr, resStride, alpha, @@ -149,7 +248,7 @@ } template <int LhsOrder, int RhsOrder, bool ConjLhs, bool ConjRhs, int ResInnerStride, typename LhsScalar, typename RhsScalar, typename ResScalar, typename Index, typename Blocking> -EIGEN_ALWAYS_INLINE bool sme_run_small_gemm(std::false_type, Index, Index, Index, const LhsScalar*, Index, +EIGEN_ALWAYS_INLINE bool sme_run_small_gemm(std::false_type, bool, Index, Index, Index, const LhsScalar*, Index, const RhsScalar*, Index, ResScalar*, Index, Index, ResScalar, Blocking&) { return false; } @@ -161,10 +260,10 @@ struct gemm_pack_rhs_first_loop_policy { template <typename Index, typename LhsScalar, typename RhsScalar, typename ResScalar, typename LhsMapper, typename RhsMapper, typename ResMapper, typename PackLhs, typename PackRhs, typename Gebp> - static EIGEN_STRONG_INLINE void run(Index rows, Index cols, Index depth, Index kc, Index mc, Index nc, + static EIGEN_ALWAYS_INLINE void run(Index rows, Index cols, Index depth, Index kc, Index mc, Index nc, const LhsMapper& lhs, const RhsMapper& rhs, ResMapper& res, PackLhs& pack_lhs, PackRhs& pack_rhs, Gebp& gebp, LhsScalar* blockA, RhsScalar* blockB, - ResScalar alpha) { + ResScalar alpha, bool overwrite) { // Mirror of pack_rhs_once: reuse one full LHS panel across column blocks. The in-place read is decided per column // block, so a block that falls back to packing packs unless an earlier one already did. const bool pack_lhs_once = nc != cols && kc == depth && mc == rows; @@ -183,68 +282,164 @@ const Index actual_mc = (std::min)(i2 + mc, rows) - i2; #ifdef EIGEN_VECTORIZE_SME if (sme_run_direct_lhs(bool_constant<sme_direct_lhs_mapper<LhsMapper>::value>(), gebp, - res.getSubMapper(i2, j2), lhs, i2, k2, blockB, actual_mc, actual_kc, actual_nc, alpha)) + res.getSubMapper(i2, j2), lhs, i2, k2, blockB, actual_mc, actual_kc, actual_nc, alpha, + overwrite && k2 == 0)) continue; #endif if (!pack_lhs_once || !lhs_packed) { pack_lhs(blockA, lhs.getSubMapper(i2, k2), actual_kc, actual_mc); lhs_packed = true; } - gebp(res.getSubMapper(i2, j2), blockA, blockB, actual_mc, actual_kc, actual_nc, alpha); + gebp_run(bool_constant<gebp_kernel_overwrites<Gebp>::value>(), gebp, overwrite && k2 == 0, + res.getSubMapper(i2, j2), blockA, blockB, actual_mc, actual_kc, actual_nc, alpha); } } } } }; +// Whether Gemm::run_overwrite writes the result without zeroing it first (Gemm::CanOverwrite), so that assignment +// should take it instead of zeroing the destination and accumulating. +template <typename Gemm, typename = void> +struct gemm_can_overwrite : std::false_type {}; +template <typename Gemm> +struct gemm_can_overwrite<Gemm, std::enable_if_t<Gemm::CanOverwrite>> : std::true_type {}; + +// Overwrites the result through Gemm::run_overwrite, or, for a general_matrix_matrix_product specialization without +// one, by zeroing it and accumulating with Gemm::run. +template <typename Gemm, typename = void> +struct gemm_has_run_overwrite : std::false_type {}; +template <typename Gemm> +struct gemm_has_run_overwrite<Gemm, void_t<decltype(&Gemm::run_overwrite)>> : std::true_type {}; +template <typename Gemm, typename Index, typename LhsScalar, typename RhsScalar, typename ResScalar, typename Blocking> +EIGEN_STRONG_INLINE void gemm_run_overwrite_impl(std::true_type, Index rows, Index cols, Index depth, + const LhsScalar* lhs, Index lhsStride, const RhsScalar* rhs, + Index rhsStride, ResScalar* res, Index resIncr, Index resStride, + ResScalar alpha, Blocking& blocking, GemmParallelInfo<Index>* info) { + Gemm::run_overwrite(rows, cols, depth, lhs, lhsStride, rhs, rhsStride, res, resIncr, resStride, alpha, blocking, + info); +} +template <typename Gemm, typename Index, typename LhsScalar, typename RhsScalar, typename ResScalar, typename Blocking> +EIGEN_STRONG_INLINE void gemm_run_overwrite_impl(std::false_type, Index rows, Index cols, Index depth, + const LhsScalar* lhs, Index lhsStride, const RhsScalar* rhs, + Index rhsStride, ResScalar* res, Index resIncr, Index resStride, + ResScalar alpha, Blocking& blocking, GemmParallelInfo<Index>* info) { + // A cooperative session needs every thread in Gemm::run, so only a sequential call returns on an empty result. + if (info == nullptr && (rows <= 0 || cols <= 0)) return; + gemm_zero_result(res, rows, cols, resIncr, resStride); + if (numext::is_exactly_zero(alpha) || depth <= 0) return; + Gemm::run(rows, cols, depth, lhs, lhsStride, rhs, rhsStride, res, resIncr, resStride, alpha, blocking, info); +} +template <typename Gemm, typename Index, typename LhsScalar, typename RhsScalar, typename ResScalar, typename Blocking> +EIGEN_STRONG_INLINE void gemm_run_overwrite(Index rows, Index cols, Index depth, const LhsScalar* lhs, Index lhsStride, + const RhsScalar* rhs, Index rhsStride, ResScalar* res, Index resIncr, + Index resStride, ResScalar alpha, Blocking& blocking, + GemmParallelInfo<Index>* info) { + gemm_run_overwrite_impl<Gemm>(gemm_has_run_overwrite<Gemm>(), rows, cols, depth, lhs, lhsStride, rhs, rhsStride, res, + resIncr, resStride, alpha, blocking, info); +} + /* Specialization for a row-major destination matrix => simple transposition of the product */ template <typename Index, typename LhsScalar, int LhsStorageOrder, bool ConjugateLhs, typename RhsScalar, - int RhsStorageOrder, bool ConjugateRhs, int ResInnerStride> + int RhsStorageOrder, bool ConjugateRhs, int ResInnerStride, typename LhsStorage, typename RhsStorage> struct general_matrix_matrix_product<Index, LhsScalar, LhsStorageOrder, ConjugateLhs, RhsScalar, RhsStorageOrder, - ConjugateRhs, RowMajor, ResInnerStride> { + ConjugateRhs, RowMajor, ResInnerStride, LhsStorage, RhsStorage> { using Traits = gebp_traits<RhsScalar, LhsScalar>; + using Transposed = + general_matrix_matrix_product<Index, RhsScalar, RhsStorageOrder == RowMajor ? ColMajor : RowMajor, ConjugateRhs, + LhsScalar, LhsStorageOrder == RowMajor ? ColMajor : RowMajor, ConjugateLhs, + ColMajor, ResInnerStride, RhsStorage, LhsStorage>; using ResScalar = typename ScalarBinaryOpTraits<LhsScalar, RhsScalar>::ReturnType; - static EIGEN_STRONG_INLINE void run(Index rows, Index cols, Index depth, const LhsScalar* lhs, Index lhsStride, - const RhsScalar* rhs, Index rhsStride, ResScalar* res, Index resIncr, + static constexpr bool CanOverwrite = gemm_can_overwrite<Transposed>::value; + static EIGEN_STRONG_INLINE void run(Index rows, Index cols, Index depth, const LhsStorage* lhs, Index lhsStride, + const RhsStorage* rhs, Index rhsStride, ResScalar* res, Index resIncr, Index resStride, ResScalar alpha, level3_blocking<RhsScalar, LhsScalar>& blocking, GemmParallelInfo<Index>* info = 0) { // transpose the product such that the result is column major - general_matrix_matrix_product<Index, RhsScalar, RhsStorageOrder == RowMajor ? ColMajor : RowMajor, ConjugateRhs, - LhsScalar, LhsStorageOrder == RowMajor ? ColMajor : RowMajor, ConjugateLhs, ColMajor, - ResInnerStride>::run(cols, rows, depth, rhs, rhsStride, lhs, lhsStride, res, resIncr, - resStride, alpha, blocking, info); + Transposed::run(cols, rows, depth, rhs, rhsStride, lhs, lhsStride, res, resIncr, resStride, alpha, blocking, info); + } + // As run, but overwrites the result instead of accumulating into it. + static EIGEN_STRONG_INLINE void run_overwrite(Index rows, Index cols, Index depth, const LhsStorage* lhs, + Index lhsStride, const RhsStorage* rhs, Index rhsStride, ResScalar* res, + Index resIncr, Index resStride, ResScalar alpha, + level3_blocking<RhsScalar, LhsScalar>& blocking, + GemmParallelInfo<Index>* info = 0) { + gemm_run_overwrite<Transposed>(cols, rows, depth, rhs, rhsStride, lhs, lhsStride, res, resIncr, resStride, alpha, + blocking, info); } }; /* Specialization for a col-major destination matrix * => Blocking algorithm following Goto's paper */ template <typename Index, typename LhsScalar, int LhsStorageOrder, bool ConjugateLhs, typename RhsScalar, - int RhsStorageOrder, bool ConjugateRhs, int ResInnerStride> + int RhsStorageOrder, bool ConjugateRhs, int ResInnerStride, typename LhsStorage, typename RhsStorage> struct general_matrix_matrix_product<Index, LhsScalar, LhsStorageOrder, ConjugateLhs, RhsScalar, RhsStorageOrder, - ConjugateRhs, ColMajor, ResInnerStride> { + ConjugateRhs, ColMajor, ResInnerStride, LhsStorage, RhsStorage> { using Traits = gebp_traits<LhsScalar, RhsScalar>; + private: + using Gebp = gebp_kernel<LhsScalar, RhsScalar, Index, + blas_data_mapper<typename Traits::ResScalar, Index, ColMajor, Unaligned, ResInnerStride>, + Traits::mr, Traits::nr, ConjugateLhs, ConjugateRhs>; + + public: using ResScalar = typename ScalarBinaryOpTraits<LhsScalar, RhsScalar>::ReturnType; - static void run(Index rows, Index cols, Index depth, const LhsScalar* lhs_, Index lhsStride, const RhsScalar* rhs_, + static constexpr bool CanOverwrite = gebp_kernel_overwrites<Gebp>::value; + static void run(Index rows, Index cols, Index depth, const LhsStorage* lhs_, Index lhsStride, const RhsStorage* rhs_, Index rhsStride, ResScalar* res_, Index resIncr, Index resStride, ResScalar alpha, level3_blocking<LhsScalar, RhsScalar>& blocking, GemmParallelInfo<Index>* info = 0) { - // BLAS contract: if alpha == 0, the result is unchanged (and lhs/rhs need not be read). - if (numext::is_exactly_zero(alpha)) return; + run_impl(false, rows, cols, depth, lhs_, lhsStride, rhs_, rhsStride, res_, resIncr, resStride, alpha, blocking, + info); + } + // As run, but assigns the result instead of accumulating into it, independently of what res held before. + static void run_overwrite(Index rows, Index cols, Index depth, const LhsStorage* lhs_, Index lhsStride, + const RhsStorage* rhs_, Index rhsStride, ResScalar* res_, Index resIncr, Index resStride, + ResScalar alpha, level3_blocking<LhsScalar, RhsScalar>& blocking, + GemmParallelInfo<Index>* info = 0) { + run_impl(true, rows, cols, depth, lhs_, lhsStride, rhs_, rhsStride, res_, resIncr, resStride, alpha, blocking, + info); + } + + private: + // One body for both: `overwrite` is a runtime flag, so that assignment and accumulation share one + // instantiation. + static EIGEN_DONT_INLINE EIGEN_GEMM_NO_CLONE void run_impl(bool overwrite, Index rows, Index cols, Index depth, + const LhsStorage* lhs_, Index lhsStride, + const RhsStorage* rhs_, Index rhsStride, ResScalar* res_, + Index resIncr, Index resStride, ResScalar alpha, + level3_blocking<LhsScalar, RhsScalar>& blocking, + GemmParallelInfo<Index>* info) { + // A cooperative session needs every thread to pack and publish its part of A', so only a sequential call returns. + if (info == nullptr && (rows <= 0 || cols <= 0)) return; + // BLAS contract: if alpha == 0, the result is unchanged (and lhs/rhs need not be read); overwriting, it is zero. + // A kernel that only accumulates or finds it does not pay, and an empty depth, overwrite by zeroing first. + if (overwrite && (numext::is_exactly_zero(alpha) || depth == 0 || !gebp_kernel_overwrites<Gebp>::value || + !gebp_overwrite_pays<Gebp>(rows, cols, depth))) { + gemm_zero_result(res_, rows, cols, resIncr, resStride); + overwrite = false; + } + if (numext::is_exactly_zero(alpha) || depth == 0) return; #ifdef EIGEN_VECTORIZE_SME // A parallel session needs every thread to take the packed path; scaleAndAddTo runs tiny results on one thread. - if (info == nullptr && - (sme_run_tiny_gemm<LhsStorageOrder, RhsStorageOrder>( - bool_constant<sme_tiny_gemm_pair<LhsScalar, RhsScalar>::value>(), rows, cols, depth, lhs_, lhsStride, rhs_, - rhsStride, res_, resIncr, resStride, alpha) || - sme_run_small_gemm<LhsStorageOrder, RhsStorageOrder, ConjugateLhs, ConjugateRhs, ResInnerStride>( - bool_constant<sme_has_gebp_kernel<LhsScalar, RhsScalar>::value>(), rows, cols, depth, lhs_, lhsStride, - rhs_, rhsStride, res_, resIncr, resStride, alpha, blocking))) - return; + // The tiny-result and NEON kernels accumulate, so overwriting zeroes their result first; the tiny one runs exactly + // when sme_tiny_gemm_wins holds. + if (info == nullptr) { + constexpr bool tiny_pair = sme_tiny_gemm_pair<LhsScalar, RhsScalar>::value; + if (overwrite && tiny_pair && sme_tiny_gemm_wins<ResScalar>(rows, cols, depth)) + gemm_zero_result(res_, rows, cols, resIncr, resStride); + if (sme_run_tiny_gemm<LhsStorageOrder, RhsStorageOrder>(bool_constant<tiny_pair>(), rows, cols, depth, lhs_, + lhsStride, rhs_, rhsStride, res_, resIncr, resStride, + alpha) || + sme_run_small_gemm<LhsStorageOrder, RhsStorageOrder, ConjugateLhs, ConjugateRhs, ResInnerStride>( + bool_constant<sme_has_gebp_kernel<LhsScalar, RhsScalar>::value>(), overwrite, rows, cols, depth, lhs_, + lhsStride, rhs_, rhsStride, res_, resIncr, resStride, alpha, blocking)) + return; + } #endif - using LhsMapper = const_blas_data_mapper<LhsScalar, Index, LhsStorageOrder>; - using RhsMapper = const_blas_data_mapper<RhsScalar, Index, RhsStorageOrder>; + using LhsMapper = gemm_operand_mapper<LhsScalar, LhsStorage, Index, LhsStorageOrder>; + using RhsMapper = gemm_operand_mapper<RhsScalar, RhsStorage, Index, RhsStorageOrder>; using ResMapper = blas_data_mapper<typename Traits::ResScalar, Index, ColMajor, Unaligned, ResInnerStride>; LhsMapper lhs(lhs_, lhsStride); RhsMapper rhs(rhs_, rhsStride); @@ -258,9 +453,10 @@ LhsStorageOrder> pack_lhs; gemm_pack_rhs<RhsScalar, Index, RhsMapper, Traits::nr, RhsStorageOrder> pack_rhs; - gebp_kernel<LhsScalar, RhsScalar, Index, ResMapper, Traits::mr, Traits::nr, ConjugateLhs, ConjugateRhs> gebp; + Gebp gebp; #if !defined(EIGEN_USE_BLAS) && (defined(EIGEN_HAS_OPENMP) || defined(EIGEN_GEMM_THREADPOOL)) + constexpr bool kOverwrites = gebp_kernel_overwrites<Gebp>::value; if (info) { // this is the parallel version! int tid = info->logical_thread_id; @@ -311,8 +507,9 @@ } } - gebp(res.getSubMapper(info->task_info[i].lhs_start, 0), blockA + info->task_info[i].lhs_start * actual_kc, - blockB, info->task_info[i].lhs_length, actual_kc, nc, alpha); + gebp_run(bool_constant<kOverwrites>(), gebp, overwrite && k == 0, + res.getSubMapper(info->task_info[i].lhs_start, 0), blockA + info->task_info[i].lhs_start * actual_kc, + blockB, info->task_info[i].lhs_length, actual_kc, nc, alpha); } // Then keep going as usual with the remaining B' @@ -323,7 +520,8 @@ pack_rhs(blockB, rhs.getSubMapper(k, j), actual_kc, actual_nc); // C_j += A' * B' - gebp(res.getSubMapper(0, j), blockA, blockB, rows, actual_kc, actual_nc, alpha); + gebp_run(bool_constant<kOverwrites>(), gebp, overwrite && k == 0, res.getSubMapper(0, j), blockA, blockB, + rows, actual_kc, actual_nc, alpha); } // Release all the sub blocks A'_i of A' for the current thread, @@ -339,8 +537,9 @@ std::size_t sizeA = kc * mc; std::size_t sizeB = kc * nc; - ei_declare_aligned_stack_constructed_variable(LhsScalar, blockA, sizeA, blocking.blockA()); - ei_declare_aligned_stack_constructed_variable(RhsScalar, blockB, sizeB, blocking.blockB()); + gemm_packing_space<LhsScalar, RhsScalar> packing(sizeA, sizeB, blocking.blockA(), blocking.blockB()); + ei_declare_aligned_stack_constructed_variable(LhsScalar, blockA, sizeA, packing.blockA()); + ei_declare_aligned_stack_constructed_variable(RhsScalar, blockB, sizeB, packing.blockB()); // The SME kernel uses RHS-first order so consecutive gebp calls stream // through adjacent row panels of a ColMajor result. Other kernels -- @@ -354,7 +553,7 @@ #endif SequentialGemmLoop::run(rows, cols, depth, kc, mc, nc, lhs, rhs, res, pack_lhs, pack_rhs, gebp, blockA, blockB, - alpha); + alpha, overwrite); } } }; @@ -364,11 +563,18 @@ * implementation of the high level wrapper to general_matrix_matrix_product **********************************************************************************/ +// overwrite: the product is assigned to dest, independently of its prior contents, instead of added to it. template <typename Scalar, typename Index, typename Gemm, typename Lhs, typename Rhs, typename Dest, typename BlockingType> struct gemm_functor { - gemm_functor(const Lhs& lhs, const Rhs& rhs, Dest& dest, const Scalar& actualAlpha, BlockingType& blocking) - : m_lhs(lhs), m_rhs(rhs), m_dest(dest), m_actualAlpha(actualAlpha), m_blocking(blocking) {} + gemm_functor(const Lhs& lhs, const Rhs& rhs, Dest& dest, const Scalar& actualAlpha, BlockingType& blocking, + bool overwrite = false) + : m_lhs(lhs), + m_rhs(rhs), + m_dest(dest), + m_actualAlpha(actualAlpha), + m_blocking(blocking), + m_overwrite(overwrite) {} void initParallelSession(Index num_threads) const { m_blocking.initParallel(m_lhs.rows(), m_rhs.cols(), m_lhs.cols(), num_threads); @@ -381,9 +587,15 @@ void operator()(Index row, Index rows, Index col = 0, Index cols = -1, GemmParallelInfo<Index>* info = 0) const { if (cols == -1) cols = m_rhs.cols(); - Gemm::run(rows, cols, m_lhs.cols(), &m_lhs.coeffRef(row, 0), m_lhs.outerStride(), &m_rhs.coeffRef(0, col), - m_rhs.outerStride(), (Scalar*)&(m_dest.coeffRef(row, col)), m_dest.innerStride(), m_dest.outerStride(), - m_actualAlpha, m_blocking, info); + if (m_overwrite) { + gemm_run_overwrite<Gemm>(rows, cols, m_lhs.cols(), &m_lhs.coeffRef(row, 0), m_lhs.outerStride(), + &m_rhs.coeffRef(0, col), m_rhs.outerStride(), (Scalar*)&(m_dest.coeffRef(row, col)), + m_dest.innerStride(), m_dest.outerStride(), m_actualAlpha, m_blocking, info); + } else { + Gemm::run(rows, cols, m_lhs.cols(), &m_lhs.coeffRef(row, 0), m_lhs.outerStride(), &m_rhs.coeffRef(0, col), + m_rhs.outerStride(), (Scalar*)&(m_dest.coeffRef(row, col)), m_dest.innerStride(), m_dest.outerStride(), + m_actualAlpha, m_blocking, info); + } } using Traits = typename Gemm::Traits; @@ -394,6 +606,7 @@ Dest& m_dest; Scalar m_actualAlpha; BlockingType& m_blocking; + bool m_overwrite; }; template <int StorageOrder, typename LhsScalar, typename RhsScalar, int MaxRows, int MaxCols, int MaxDepth, @@ -440,7 +653,9 @@ // A zero-sized fixed dimension must not declare a zero-length array, which MSVC rejects (C2229); every caller // returns before it touches the blocks of an empty product. static constexpr int SizeA = plain_enum_max(int(ActualRows) * MaxDepth, 1); - static constexpr int SizeB = plain_enum_max(int(ActualCols) * MaxDepth, 1); + // One packet of slack: product_triangular_matrix_matrix with a triangular rhs realigns its packed general panel + // inside blockB to a packet boundary. + static constexpr int SizeB = plain_enum_max(int(ActualCols) * MaxDepth, 1) + int(packet_traits<RhsScalar>::size); #if EIGEN_MAX_STATIC_ALIGN_BYTES >= EIGEN_DEFAULT_ALIGN_BYTES EIGEN_ALIGN_MAX LhsScalar m_staticA[SizeA > 0 ? SizeA : 1]; @@ -540,6 +755,52 @@ namespace internal { +// Whether a GEMM of LhsScalar by RhsScalar runs in float on the converted operands and rounds each result coefficient +// once. Summed in the operand type, half by half and bfloat16 by bfloat16 round every multiply-add, or every kc block, +// to 11 or 8 significand bits (#3158). +template <typename LhsScalar, typename RhsScalar> +struct gemm_accumulates_in_float : std::false_type {}; +template <> +struct gemm_accumulates_in_float<half, half> : std::true_type {}; +// OpenBLAS's sbgemm already accumulates bfloat16 in float; Eigen/Core reaches it only under EIGEN_USE_BLAS. +#if !(defined(EIGEN_USE_BLAS) && defined(EIGEN_USE_OPENBLAS_BFLOAT16) && EIGEN_USE_OPENBLAS_BFLOAT16) +template <> +struct gemm_accumulates_in_float<bfloat16, bfloat16> : std::true_type {}; +#endif + +// blas_traits of a GEMM operand, which also sees through a cast of a real operand with direct access: the packers then +// read the operand's StorageScalar coefficients and convert them (gemm_operand_mapper), instead of the product +// materializing the cast. A scalar factor below the cast stays below it, so such an operand is materialized as before. +// So is a cast to a type whose GEMM sums in float, which would otherwise skip the rounding to that type. +// With EIGEN_USE_BLAS the cast is materialized, so that the product still reaches the external ?gemm, and so it is on +// SME, whose tiny, small and direct-packing paths read the operand in the kernel's scalar type. +template <typename Xpr, typename = void> +struct gemm_operand_traits : blas_traits<Xpr> { + using StorageScalar = typename traits<Xpr>::Scalar; +}; +#if !defined(EIGEN_USE_BLAS) && !defined(EIGEN_VECTORIZE_SME) +template <typename SrcScalar, typename DstScalar, typename NestedXpr> +struct gemm_operand_traits< + CwiseUnaryOp<core_cast_op<SrcScalar, DstScalar>, NestedXpr>, + std::enable_if_t<bool(blas_traits<remove_all_t<NestedXpr>>::HasUsableDirectAccess) && + !bool(blas_traits<remove_all_t<NestedXpr>>::HasScalarFactor) && !NumTraits<SrcScalar>::IsComplex && + !NumTraits<DstScalar>::IsComplex && !gemm_accumulates_in_float<DstScalar, DstScalar>::value>> + : blas_traits<remove_all_t<NestedXpr>> { + using Base = blas_traits<remove_all_t<NestedXpr>>; + using XprType = CwiseUnaryOp<core_cast_op<SrcScalar, DstScalar>, NestedXpr>; + using StorageScalar = SrcScalar; + using Scalar = DstScalar; + static constexpr bool IsComplex = false; + static constexpr bool NeedToConjugate = false; + static EIGEN_DEVICE_FUNC inline typename Base::ExtractType extract(const XprType& x) { + return Base::extract(x.nestedExpression()); + } + static EIGEN_DEVICE_FUNC inline Scalar extractScalarFactor(const XprType&) { return Scalar(1); } +}; +#endif +template <typename Xpr> +struct gemm_operand_traits<const Xpr> : gemm_operand_traits<Xpr> {}; + template <typename Lhs, typename Rhs> struct generic_product_impl<Lhs, Rhs, DenseShape, DenseShape, GemmProduct> : generic_product_impl_base<Lhs, Rhs, generic_product_impl<Lhs, Rhs, DenseShape, DenseShape, GemmProduct>> { @@ -547,11 +808,11 @@ using LhsScalar = typename Lhs::Scalar; using RhsScalar = typename Rhs::Scalar; - using LhsBlasTraits = internal::blas_traits<Lhs>; + using LhsBlasTraits = internal::gemm_operand_traits<Lhs>; using ActualLhsType = typename LhsBlasTraits::DirectLinearAccessType; using ActualLhsTypeCleaned = internal::remove_all_t<ActualLhsType>; - using RhsBlasTraits = internal::blas_traits<Rhs>; + using RhsBlasTraits = internal::gemm_operand_traits<Rhs>; using ActualRhsType = typename RhsBlasTraits::DirectLinearAccessType; using ActualRhsTypeCleaned = internal::remove_all_t<ActualRhsType>; @@ -620,8 +881,24 @@ // would evaluate the product, so the factor is tested before the dispatch // rather than inside either kernel. static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE bool scalarFactorIsZero(const Lhs& lhs, const Rhs& rhs) { + return scalarFactorIsZero(gemm_accumulates_in_float<LhsScalar, RhsScalar>(), lhs, rhs); + } + static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE bool scalarFactorIsZero(std::false_type, const Lhs& lhs, + const Rhs& rhs) { return numext::is_exactly_zero(combine_scalar_factors<Scalar>(lhs, rhs)); } + // The float GEMM combines the factors in float, where their product may not underflow. The coefficient-based path + // recombines them in Scalar and would then evaluate 0 * lazyProduct, so such a product takes the GEMM. + static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE bool scalarFactorIsZero(std::true_type, const Lhs& lhs, const Rhs& rhs) { + return numext::is_exactly_zero(float(LhsBlasTraits::extractScalarFactor(lhs)) * + float(RhsBlasTraits::extractScalarFactor(rhs))); + } + + // A factor that underflows in Scalar but not in float takes the float GEMM (see scalarFactorIsZero). + template <typename Dst> + static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE bool useCoeffBasedPath(const Dst& dst, const Lhs& lhs, const Rhs& rhs) { + return useRuntimeCoeffBasedProduct(dst, rhs) && !numext::is_exactly_zero(combine_scalar_factors<Scalar>(lhs, rhs)); + } template <typename Dst> static void evalTo(Dst& dst, const Lhs& lhs, const Rhs& rhs) { @@ -629,18 +906,16 @@ dst.setZero(); return; } - if (useRuntimeCoeffBasedProduct(dst, rhs)) + if (useCoeffBasedPath(dst, lhs, rhs)) lazyproduct::eval_dynamic(dst, lhs, rhs, internal::assign_op<typename Dst::Scalar, Scalar>()); - else { - dst.setZero(); - scaleAndAddTo(dst, lhs, rhs, Scalar(1)); - } + else + scaleAndEvalTo(dst, lhs, rhs, Scalar(1)); } template <typename Dst> static void addTo(Dst& dst, const Lhs& lhs, const Rhs& rhs) { if (scalarFactorIsZero(lhs, rhs)) return; - if (useRuntimeCoeffBasedProduct(dst, rhs)) + if (useCoeffBasedPath(dst, lhs, rhs)) lazyproduct::eval_dynamic(dst, lhs, rhs, internal::add_assign_op<typename Dst::Scalar, Scalar>()); else scaleAndAddTo(dst, lhs, rhs, Scalar(1)); @@ -649,7 +924,7 @@ template <typename Dst> static void subTo(Dst& dst, const Lhs& lhs, const Rhs& rhs) { if (scalarFactorIsZero(lhs, rhs)) return; - if (useRuntimeCoeffBasedProduct(dst, rhs)) + if (useCoeffBasedPath(dst, lhs, rhs)) lazyproduct::eval_dynamic(dst, lhs, rhs, internal::sub_assign_op<typename Dst::Scalar, Scalar>()); else scaleAndAddTo(dst, lhs, rhs, Scalar(-1)); @@ -657,20 +932,103 @@ template <typename Dest> static void scaleAndAddTo(Dest& dst, const Lhs& a_lhs, const Rhs& a_rhs, const Scalar& alpha) { + scaleAndAddOrEvalTo(dst, a_lhs, a_rhs, alpha, false); + } + + // dst = alpha * a_lhs * a_rhs, independently of dst's prior contents and, where the GEMM kernel can overwrite, + // without zeroing it first. + template <typename Dest> + static void scaleAndEvalTo(Dest& dst, const Lhs& a_lhs, const Rhs& a_rhs, const Scalar& alpha) { + scaleAndAddOrEvalTo(dst, a_lhs, a_rhs, alpha, true); + } + + private: + // One body for both: `overwrite` is a runtime flag, so that assignment and accumulation share one + // instantiation. + template <typename Dest> + static void scaleAndAddOrEvalTo(Dest& dst, const Lhs& a_lhs, const Rhs& a_rhs, const Scalar& alpha, bool overwrite) { + scaleAndAddOrEvalTo(gemm_accumulates_in_float<LhsScalar, RhsScalar>(), dst, a_lhs, a_rhs, alpha, overwrite); + } + + // The products the GEMM kernel does not take: empty operands, zero depth and a runtime vector result, which goes to + // GEMV. They run on the operand type also where the GEMM sums in float, so that they leave an empty product's dst + // untouched and do not allocate. Returns whether dst is done. + template <typename Dest> + static bool evalWithoutGemm(Dest& dst, const Lhs& a_lhs, const Rhs& a_rhs, const Scalar& alpha, bool overwrite) { + if (a_lhs.rows() == 0 || a_rhs.cols() == 0) return true; + if (a_lhs.cols() != 0 && dst.cols() != 1 && dst.rows() != 1) return false; + // Zero depth and GEMV accumulate, so overwriting zeroes first. + if (overwrite) dst.setZero(); + if (a_lhs.cols() == 0) return true; + if (dst.cols() == 1) { + typename Dest::ColXpr dst_vec(dst.col(0)); + internal::generic_product_impl<Lhs, typename Rhs::ConstColXpr, DenseShape, DenseShape, + GemvProduct>::scaleAndAddTo(dst_vec, a_lhs, a_rhs.col(0), alpha); + } else { + typename Dest::RowXpr dst_vec(dst.row(0)); + internal::generic_product_impl<typename Lhs::ConstRowXpr, Rhs, DenseShape, DenseShape, + GemvProduct>::scaleAndAddTo(dst_vec, a_lhs.row(0), a_rhs, alpha); + } + return true; + } + + // dst (+)= alpha * a_lhs * a_rhs as the float product of the converted operands, rounded once into dst. The packers + // convert the operands; only the result needs a float workspace, which goes where the packed blocks do: on the stack + // up to EIGEN_STACK_ALLOCATION_LIMIT, and on the heap above it, also for a fixed-size result. + template <typename Dest> + static void scaleAndAddOrEvalTo(std::true_type, Dest& dst, const Lhs& a_lhs, const Rhs& a_rhs, const Scalar& alpha, + bool overwrite) { EIGEN_INTERNAL_ASSERT_SAME_ROWS(dst, a_lhs); EIGEN_INTERNAL_ASSERT_SAME_COLS(dst, a_rhs); - if (a_lhs.cols() == 0 || a_lhs.rows() == 0 || a_rhs.cols() == 0) return; + if (evalWithoutGemm(dst, a_lhs, a_rhs, alpha, overwrite)) return; + using Real = float; + add_const_on_value_type_t<ActualLhsType> lhs = LhsBlasTraits::extract(a_lhs); + add_const_on_value_type_t<ActualRhsType> rhs = RhsBlasTraits::extract(a_rhs); + // A cast the float GEMM does not pack in place (under EIGEN_USE_BLAS or on SME) is materialized at run-time size, + // so that a fixed-size operand does not land on the stack. + using CastLhsXpr = std::remove_const_t<decltype(lhs.template cast<Real>())>; + using CastRhsXpr = std::remove_const_t<decltype(rhs.template cast<Real>())>; + using CastLhs = std::conditional_t< + std::is_same<typename gemm_operand_traits<CastLhsXpr>::StorageScalar, Real>::value, + Matrix<Real, Dynamic, Dynamic, (ActualLhsTypeCleaned::Flags & RowMajorBit) ? RowMajor : ColMajor>, CastLhsXpr>; + using CastRhs = std::conditional_t< + std::is_same<typename gemm_operand_traits<CastRhsXpr>::StorageScalar, Real>::value, + Matrix<Real, Dynamic, Dynamic, (ActualRhsTypeCleaned::Flags & RowMajorBit) ? RowMajor : ColMajor>, CastRhsXpr>; + CastLhs cast_lhs = lhs.template cast<Real>(); + CastRhs cast_rhs = rhs.template cast<Real>(); + // The factors combine in float, so that (s * A) * (t * B) keeps an s * t below half's range. + Real actualAlpha = + Real(alpha) * Real(LhsBlasTraits::extractScalarFactor(a_lhs)) * Real(RhsBlasTraits::extractScalarFactor(a_rhs)); - if (dst.cols() == 1) { - // Fallback to GEMV if either the lhs or rhs is a runtime vector - typename Dest::ColXpr dst_vec(dst.col(0)); - return internal::generic_product_impl<Lhs, typename Rhs::ConstColXpr, DenseShape, DenseShape, - GemvProduct>::scaleAndAddTo(dst_vec, a_lhs, a_rhs.col(0), alpha); - } else if (dst.rows() == 1) { - // Fallback to GEMV if either the lhs or rhs is a runtime vector - typename Dest::RowXpr dst_vec(dst.row(0)); - return internal::generic_product_impl<typename Lhs::ConstRowXpr, Rhs, DenseShape, DenseShape, - GemvProduct>::scaleAndAddTo(dst_vec, a_lhs.row(0), a_rhs, alpha); + using FloatGemm = generic_product_impl<CastLhs, CastRhs, DenseShape, DenseShape, GemmProduct>; + ei_declare_aligned_stack_constructed_variable(Real, acc_data, dst.size(), nullptr); + Map<Matrix<Real, Dynamic, Dynamic, (Dest::Flags & RowMajorBit) ? RowMajor : ColMajor>> acc(acc_data, dst.rows(), + dst.cols()); + if (overwrite) { + FloatGemm::scaleAndEvalTo(acc, cast_lhs, cast_rhs, actualAlpha); + } else { + acc = dst.template cast<Real>(); + FloatGemm::scaleAndAddTo(acc, cast_lhs, cast_rhs, actualAlpha); + } + dst = acc.template cast<typename Dest::Scalar>(); + } + + template <typename Dest> + static void scaleAndAddOrEvalTo(std::false_type, Dest& dst, const Lhs& a_lhs, const Rhs& a_rhs, const Scalar& alpha, + bool overwrite) { + EIGEN_INTERNAL_ASSERT_SAME_ROWS(dst, a_lhs); + EIGEN_INTERNAL_ASSERT_SAME_COLS(dst, a_rhs); + using Gemm = internal::general_matrix_matrix_product< + Index, LhsScalar, (ActualLhsTypeCleaned::Flags & RowMajorBit) ? RowMajor : ColMajor, + bool(LhsBlasTraits::NeedToConjugate), RhsScalar, + (ActualRhsTypeCleaned::Flags & RowMajorBit) ? RowMajor : ColMajor, bool(RhsBlasTraits::NeedToConjugate), + (Dest::Flags & RowMajorBit) ? RowMajor : ColMajor, Dest::InnerStrideAtCompileTime, + typename LhsBlasTraits::StorageScalar, typename RhsBlasTraits::StorageScalar>; + if (evalWithoutGemm(dst, a_lhs, a_rhs, alpha, overwrite)) return; + // A GEMM that cannot overwrite accumulates, so overwriting zeroes first. + if (overwrite && !internal::gemm_can_overwrite<Gemm>::value) { + dst.setZero(); + overwrite = false; } add_const_on_value_type_t<ActualLhsType> lhs = LhsBlasTraits::extract(a_lhs); @@ -682,24 +1040,18 @@ internal::gemm_blocking_space<(Dest::Flags & RowMajorBit) ? RowMajor : ColMajor, LhsScalar, RhsScalar, Dest::MaxRowsAtCompileTime, Dest::MaxColsAtCompileTime, MaxDepthAtCompileTime>; - using GemmFunctor = internal::gemm_functor< - Scalar, Index, - internal::general_matrix_matrix_product< - Index, LhsScalar, (ActualLhsTypeCleaned::Flags & RowMajorBit) ? RowMajor : ColMajor, - bool(LhsBlasTraits::NeedToConjugate), RhsScalar, - (ActualRhsTypeCleaned::Flags & RowMajorBit) ? RowMajor : ColMajor, bool(RhsBlasTraits::NeedToConjugate), - (Dest::Flags & RowMajorBit) ? RowMajor : ColMajor, Dest::InnerStrideAtCompileTime>, - ActualLhsTypeCleaned, ActualRhsTypeCleaned, Dest, BlockingType>; + using GemmFunctor = + internal::gemm_functor<Scalar, Index, Gemm, ActualLhsTypeCleaned, ActualRhsTypeCleaned, Dest, BlockingType>; BlockingType blocking(dst.rows(), dst.cols(), lhs.cols(), 1, true); #ifdef EIGEN_VECTORIZE_SME // A result the tiny-result kernel takes is never worth a parallel session. if (tinyKernelWins(dst, a_rhs)) - return internal::parallelize_gemm<false>(GemmFunctor(lhs, rhs, dst, actualAlpha, blocking), a_lhs.rows(), - a_rhs.cols(), a_lhs.cols(), Dest::Flags & RowMajorBit); + return internal::parallelize_gemm<false>(GemmFunctor(lhs, rhs, dst, actualAlpha, blocking, overwrite), + a_lhs.rows(), a_rhs.cols(), a_lhs.cols(), Dest::Flags & RowMajorBit); #endif internal::parallelize_gemm<(Dest::MaxRowsAtCompileTime > 32 || Dest::MaxRowsAtCompileTime == Dynamic)>( - GemmFunctor(lhs, rhs, dst, actualAlpha, blocking), a_lhs.rows(), a_rhs.cols(), a_lhs.cols(), + GemmFunctor(lhs, rhs, dst, actualAlpha, blocking, overwrite), a_lhs.rows(), a_rhs.cols(), a_lhs.cols(), Dest::Flags & RowMajorBit); } };
diff --git a/Eigen/src/Core/products/GeneralMatrixMatrixTriangular.h b/Eigen/src/Core/products/GeneralMatrixMatrixTriangular.h index e19d4fc..019150f 100644 --- a/Eigen/src/Core/products/GeneralMatrixMatrixTriangular.h +++ b/Eigen/src/Core/products/GeneralMatrixMatrixTriangular.h
@@ -95,8 +95,9 @@ std::size_t sizeA = kc * mc; std::size_t sizeB = kc * size; - ei_declare_aligned_stack_constructed_variable(LhsScalar, blockA, sizeA, blocking.blockA()); - ei_declare_aligned_stack_constructed_variable(RhsScalar, blockB, sizeB, blocking.blockB()); + gemm_packing_space<LhsScalar, RhsScalar> packing(sizeA, sizeB, blocking.blockA(), blocking.blockB()); + ei_declare_aligned_stack_constructed_variable(LhsScalar, blockA, sizeA, packing.blockA()); + ei_declare_aligned_stack_constructed_variable(RhsScalar, blockB, sizeB, packing.blockB()); gemm_pack_lhs<LhsScalar, Index, LhsMapper, Traits::mr, Traits::LhsProgress, typename Traits::LhsPacket4Packing, LhsStorageOrder>
diff --git a/Eigen/src/Core/products/GeneralMatrixMatrixTriangular_BLAS.h b/Eigen/src/Core/products/GeneralMatrixMatrixTriangular_BLAS.h index e7f824c..e0807d5 100644 --- a/Eigen/src/Core/products/GeneralMatrixMatrixTriangular_BLAS.h +++ b/Eigen/src/Core/products/GeneralMatrixMatrixTriangular_BLAS.h
@@ -78,25 +78,36 @@ // EIGEN_BLAS_RANKUPDATE_SPECIALIZE(scomplex) // SYRK for float/double -#define EIGEN_BLAS_RANKUPDATE_R(EIGTYPE, BLASTYPE, BLASFUNC) \ - template <typename Index, int AStorageOrder, bool ConjugateA, int UpLo> \ - struct general_matrix_matrix_rankupdate<Index, EIGTYPE, AStorageOrder, ConjugateA, ColMajor, UpLo> { \ - enum { \ - IsLower = (UpLo & Lower) == Lower, \ - LowUp = IsLower ? Lower : Upper, \ - conjA = ((AStorageOrder == ColMajor) && ConjugateA) ? 1 : 0 \ - }; \ - static EIGEN_STRONG_INLINE void run(Index size, Index depth, const EIGTYPE* lhs, Index lhsStride, \ - const EIGTYPE* /*rhs*/, Index /*rhsStride*/, EIGTYPE* res, Index resStride, \ - EIGTYPE alpha, level3_blocking<EIGTYPE, EIGTYPE>& /*blocking*/) { \ - if (size == 0 || depth == 0) return; \ - BlasIndex lda = convert_index<BlasIndex>(lhsStride), ldc = convert_index<BlasIndex>(resStride), \ - n = convert_index<BlasIndex>(size), k = convert_index<BlasIndex>(depth); \ - char uplo = ((IsLower) ? 'L' : 'U'), trans = ((AStorageOrder == RowMajor) ? 'T' : 'N'); \ - EIGTYPE beta(1); \ - BLASFUNC(&uplo, &trans, &n, &k, (const BLASTYPE*)&numext::real_ref(alpha), lhs, &lda, \ - (const BLASTYPE*)&numext::real_ref(beta), res, &ldc); \ - } \ +#define EIGEN_BLAS_RANKUPDATE_R(EIGTYPE, BLASTYPE, BLASFUNC) \ + template <typename Index, int AStorageOrder, bool ConjugateA, int UpLo> \ + struct general_matrix_matrix_rankupdate<Index, EIGTYPE, AStorageOrder, ConjugateA, ColMajor, UpLo> { \ + using Scalar = EIGTYPE; \ + enum { \ + IsLower = (UpLo & Lower) == Lower, \ + LowUp = IsLower ? Lower : Upper, \ + conjA = ((AStorageOrder == ColMajor) && ConjugateA) ? 1 : 0 \ + }; \ + static EIGEN_STRONG_INLINE void run(Index size, Index depth, const EIGTYPE* lhs, Index lhsStride, \ + const EIGTYPE* /*rhs*/, Index /*rhsStride*/, Scalar* res, Index resStride, \ + EIGTYPE alpha, level3_blocking<EIGTYPE, EIGTYPE>& blocking) { \ + if (size == 0 || depth == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<AStorageOrder>(size, depth, lhsStride) || \ + !blas_compatible_stride<ColMajor>(size, size, resStride)) { \ + general_matrix_matrix_triangular_product<Index, EIGTYPE, AStorageOrder, ConjugateA, EIGTYPE, \ + AStorageOrder == ColMajor ? RowMajor : ColMajor, ConjugateA, \ + ColMajor, 1, UpLo, BuiltIn>::run(size, depth, lhs, lhsStride, lhs, \ + lhsStride, res, Index(1), resStride, \ + alpha, blocking); \ + return; \ + } \ + BlasIndex lda = blas_leading_dim<AStorageOrder>(size, depth, lhsStride), \ + ldc = blas_leading_dim<ColMajor>(size, size, resStride), n = convert_index<BlasIndex>(size), \ + k = convert_index<BlasIndex>(depth); \ + char uplo = ((IsLower) ? 'L' : 'U'), trans = ((AStorageOrder == RowMajor) ? 'T' : 'N'); \ + EIGTYPE beta(1); \ + BLASFUNC(&uplo, &trans, &n, &k, (const BLASTYPE*)&numext::real_ref(alpha), lhs, &lda, \ + (const BLASTYPE*)&numext::real_ref(beta), res, &ldc); \ + } \ }; #ifdef EIGEN_USE_MKL
diff --git a/Eigen/src/Core/products/GeneralMatrixMatrix_BLAS.h b/Eigen/src/Core/products/GeneralMatrixMatrix_BLAS.h index d93df65..7d74c60 100644 --- a/Eigen/src/Core/products/GeneralMatrixMatrix_BLAS.h +++ b/Eigen/src/Core/products/GeneralMatrixMatrix_BLAS.h
@@ -50,59 +50,68 @@ // gemm specialization -#define EIGEN_BLAS_GEMM_SPECIALIZATION(EIGTYPE, EIGPREFIX, BLASTYPE, BLASFUNC) \ - template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ - struct general_matrix_matrix_product<Index, EIGTYPE, LhsStorageOrder, ConjugateLhs, EIGTYPE, RhsStorageOrder, \ - ConjugateRhs, ColMajor, 1> { \ - typedef gebp_traits<EIGTYPE, EIGTYPE> Traits; \ - \ - static void run(Index rows, Index cols, Index depth, const EIGTYPE* lhs_, Index lhsStride, const EIGTYPE* rhs_, \ - Index rhsStride, EIGTYPE* res, Index resIncr, Index resStride, EIGTYPE alpha, \ - level3_blocking<EIGTYPE, EIGTYPE>& /*blocking*/, GemmParallelInfo<Index>* /*info = 0*/) { \ - using std::conj; \ - if (rows == 0 || cols == 0 || depth == 0) return; \ - EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ - eigen_assert(resIncr == 1); \ - char transa, transb; \ - BlasIndex m, n, k, lda, ldb, ldc; \ - const EIGTYPE *a, *b; \ - EIGTYPE beta(1); \ - MatrixX##EIGPREFIX a_tmp, b_tmp; \ - \ - /* Set transpose options */ \ - transa = (LhsStorageOrder == RowMajor) ? ((ConjugateLhs) ? 'C' : 'T') : 'N'; \ - transb = (RhsStorageOrder == RowMajor) ? ((ConjugateRhs) ? 'C' : 'T') : 'N'; \ - \ - /* Set m, n, k */ \ - m = convert_index<BlasIndex>(rows); \ - n = convert_index<BlasIndex>(cols); \ - k = convert_index<BlasIndex>(depth); \ - \ - /* Set lda, ldb, ldc */ \ - lda = convert_index<BlasIndex>(lhsStride); \ - ldb = convert_index<BlasIndex>(rhsStride); \ - ldc = convert_index<BlasIndex>(resStride); \ - \ - /* Set a, b, c */ \ - EIGEN_IF_CONSTEXPR ((LhsStorageOrder == ColMajor) && (ConjugateLhs)) { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(lhs_, m, k, OuterStride<>(lhsStride)); \ - a_tmp = lhs.conjugate(); \ - a = a_tmp.data(); \ - lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ - } else \ - a = lhs_; \ - \ - EIGEN_IF_CONSTEXPR ((RhsStorageOrder == ColMajor) && (ConjugateRhs)) { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(rhs_, k, n, OuterStride<>(rhsStride)); \ - b_tmp = rhs.conjugate(); \ - b = b_tmp.data(); \ - ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ - } else \ - b = rhs_; \ - \ - BLASFUNC(&transa, &transb, &m, &n, &k, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ - (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ - } \ +#define EIGEN_BLAS_GEMM_SPECIALIZATION(EIGTYPE, EIGPREFIX, BLASTYPE, BLASFUNC) \ + template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ + struct general_matrix_matrix_product<Index, EIGTYPE, LhsStorageOrder, ConjugateLhs, EIGTYPE, RhsStorageOrder, \ + ConjugateRhs, ColMajor, 1> { \ + typedef gebp_traits<EIGTYPE, EIGTYPE> Traits; \ + using Scalar = EIGTYPE; \ + \ + static void run(Index rows, Index cols, Index depth, const EIGTYPE* lhs_, Index lhsStride, const EIGTYPE* rhs_, \ + Index rhsStride, Scalar* res, Index resIncr, Index resStride, EIGTYPE alpha, \ + level3_blocking<EIGTYPE, EIGTYPE>& blocking, GemmParallelInfo<Index>* info) { \ + if (rows == 0 || cols == 0 || depth == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<LhsStorageOrder>(rows, depth, lhsStride) || \ + !blas_compatible_stride<RhsStorageOrder>(depth, cols, rhsStride) || \ + !blas_compatible_stride<ColMajor>(rows, cols, resStride)) { \ + general_matrix_matrix_product<Index, EIGTYPE, LhsStorageOrder, ConjugateLhs, EIGTYPE, RhsStorageOrder, \ + ConjugateRhs, ColMajor, Dynamic>::run(rows, cols, depth, lhs_, lhsStride, rhs_, \ + rhsStride, res, resIncr, resStride, alpha, \ + blocking, info); \ + return; \ + } \ + EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ + eigen_assert(resIncr == 1); \ + char transa, transb; \ + BlasIndex m, n, k, lda, ldb, ldc; \ + const EIGTYPE *a, *b; \ + EIGTYPE beta(1); \ + MatrixX##EIGPREFIX a_tmp, b_tmp; \ + \ + /* Set transpose options */ \ + transa = (LhsStorageOrder == RowMajor) ? ((ConjugateLhs) ? 'C' : 'T') : 'N'; \ + transb = (RhsStorageOrder == RowMajor) ? ((ConjugateRhs) ? 'C' : 'T') : 'N'; \ + \ + /* Set m, n, k */ \ + m = convert_index<BlasIndex>(rows); \ + n = convert_index<BlasIndex>(cols); \ + k = convert_index<BlasIndex>(depth); \ + \ + /* Set lda, ldb, ldc */ \ + lda = blas_leading_dim<LhsStorageOrder>(rows, depth, lhsStride); \ + ldb = blas_leading_dim<RhsStorageOrder>(depth, cols, rhsStride); \ + ldc = blas_leading_dim<ColMajor>(rows, cols, resStride); \ + \ + /* Set a, b, c */ \ + EIGEN_IF_CONSTEXPR ((LhsStorageOrder == ColMajor) && (ConjugateLhs)) { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(lhs_, m, k, OuterStride<>(lhsStride)); \ + a_tmp = lhs.conjugate(); \ + a = a_tmp.data(); \ + lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ + } else \ + a = lhs_; \ + \ + EIGEN_IF_CONSTEXPR ((RhsStorageOrder == ColMajor) && (ConjugateRhs)) { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(rhs_, k, n, OuterStride<>(rhsStride)); \ + b_tmp = rhs.conjugate(); \ + b = b_tmp.data(); \ + ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ + } else \ + b = rhs_; \ + \ + BLASFUNC(&transa, &transb, &m, &n, &k, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ + (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ + } \ }; #ifdef EIGEN_USE_MKL @@ -136,10 +145,18 @@ static void run(Index rows, Index cols, Index depth, const Eigen::bfloat16* lhs_, Index lhsStride, const Eigen::bfloat16* rhs_, Index rhsStride, Eigen::bfloat16* res, Index resIncr, Index resStride, - Eigen::bfloat16 alpha, level3_blocking<Eigen::bfloat16, Eigen::bfloat16>& /*blocking*/, - GemmParallelInfo<Index>* /*info = 0*/) { - using std::conj; - if (rows == 0 || cols == 0 || depth == 0) return; + Eigen::bfloat16 alpha, level3_blocking<Eigen::bfloat16, Eigen::bfloat16>& blocking, + GemmParallelInfo<Index>* info) { + if (rows == 0 || cols == 0 || depth == 0 || numext::is_exactly_zero(alpha)) return; + if (!blas_compatible_stride<LhsStorageOrder>(rows, depth, lhsStride) || + !blas_compatible_stride<RhsStorageOrder>(depth, cols, rhsStride)) { + general_matrix_matrix_product<Index, Eigen::bfloat16, LhsStorageOrder, ConjugateLhs, Eigen::bfloat16, + RhsStorageOrder, ConjugateRhs, ColMajor, Dynamic>::run(rows, cols, depth, lhs_, + lhsStride, rhs_, rhsStride, + res, resIncr, resStride, + alpha, blocking, info); + return; + } EIGEN_ONLY_USED_FOR_DEBUG(resIncr); eigen_assert(resIncr == 1); char transa, transb; @@ -163,8 +180,8 @@ k = convert_index<BlasIndex>(depth); /* Set lda, ldb, ldc */ - lda = convert_index<BlasIndex>(lhsStride); - ldb = convert_index<BlasIndex>(rhsStride); + lda = blas_leading_dim<LhsStorageOrder>(rows, depth, lhsStride); + ldb = blas_leading_dim<RhsStorageOrder>(depth, cols, rhsStride); ldc = convert_index<BlasIndex>(m); /* Set a, b, c */
diff --git a/Eigen/src/Core/products/GeneralMatrixVector_BLAS.h b/Eigen/src/Core/products/GeneralMatrixVector_BLAS.h index 23b0695..a59509b 100644 --- a/Eigen/src/Core/products/GeneralMatrixVector_BLAS.h +++ b/Eigen/src/Core/products/GeneralMatrixVector_BLAS.h
@@ -89,38 +89,53 @@ EIGEN_BLAS_GEMV_SPECIALIZE(dcomplex) EIGEN_BLAS_GEMV_SPECIALIZE(scomplex) -#define EIGEN_BLAS_GEMV_SPECIALIZATION(EIGTYPE, BLASTYPE, BLASFUNC) \ - template <typename Index, int LhsStorageOrder, bool ConjugateLhs, bool ConjugateRhs> \ - struct general_matrix_vector_product_gemv<Index, EIGTYPE, LhsStorageOrder, ConjugateLhs, EIGTYPE, ConjugateRhs> { \ - typedef Matrix<EIGTYPE, Dynamic, 1, ColMajor> GEMVVector; \ - \ - static void run(Index rows, Index cols, const EIGTYPE* lhs, Index lhsStride, const EIGTYPE* rhs, Index rhsIncr, \ - EIGTYPE* res, Index resIncr, EIGTYPE alpha) { \ - if (rows == 0 || cols == 0) return; \ - BlasIndex m = convert_index<BlasIndex>(rows), n = convert_index<BlasIndex>(cols), \ - lda = convert_index<BlasIndex>(lhsStride), incx = convert_index<BlasIndex>(rhsIncr), \ - incy = convert_index<BlasIndex>(resIncr); \ - const EIGTYPE beta(1); \ - const EIGTYPE* x_ptr; \ - char trans = (LhsStorageOrder == ColMajor) ? 'N' : (ConjugateLhs) ? 'C' : 'T'; \ - EIGEN_IF_CONSTEXPR (LhsStorageOrder == RowMajor) { \ - m = convert_index<BlasIndex>(cols); \ - n = convert_index<BlasIndex>(rows); \ - } \ - GEMVVector x_tmp; \ - EIGEN_IF_CONSTEXPR (ConjugateRhs) { \ - Map<const GEMVVector, 0, InnerStride<> > map_x(rhs, cols, 1, InnerStride<>(incx)); \ - x_tmp = map_x.conjugate(); \ - x_ptr = x_tmp.data(); \ - incx = 1; \ - } else { \ - /* BLAS reads a negative increment from the lowest address, Eigen from entry 0. */ \ - x_ptr = rhsIncr < 0 ? rhs + (cols - 1) * rhsIncr : rhs; \ - } \ - auto* y_ptr = resIncr < 0 ? res + (rows - 1) * resIncr : res; \ - BLASFUNC(&trans, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)lhs, &lda, \ - (const BLASTYPE*)x_ptr, &incx, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)y_ptr, &incy); \ - } \ +#define EIGEN_BLAS_GEMV_SPECIALIZATION(EIGTYPE, BLASTYPE, BLASFUNC) \ + template <typename Index, int LhsStorageOrder, bool ConjugateLhs, bool ConjugateRhs> \ + struct general_matrix_vector_product_gemv<Index, EIGTYPE, LhsStorageOrder, ConjugateLhs, EIGTYPE, ConjugateRhs> { \ + using Scalar = EIGTYPE; \ + typedef Matrix<EIGTYPE, Dynamic, 1, ColMajor> GEMVVector; \ + \ + static void run(Index rows, Index cols, const EIGTYPE* lhs, Index lhsStride, const EIGTYPE* rhs, Index rhsIncr, \ + Scalar* res, Index resIncr, EIGTYPE alpha) { \ + if (rows == 0 || cols == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<LhsStorageOrder>(rows, cols, lhsStride) || (cols > 1 && rhsIncr == 0) || \ + (rows > 1 && resIncr == 0)) { \ + using LhsMapper = const_blas_data_mapper<EIGTYPE, Index, LhsStorageOrder>; \ + using RhsMapper = const_blas_data_mapper<EIGTYPE, Index, LhsStorageOrder == ColMajor ? RowMajor : ColMajor>; \ + const GEMVVector rhs_copy = Map<const GEMVVector, 0, InnerStride<>>(rhs, cols, InnerStride<>(rhsIncr)); \ + GEMVVector result = GEMVVector::Zero(rows); \ + general_matrix_vector_product<Index, EIGTYPE, LhsMapper, LhsStorageOrder, ConjugateLhs, EIGTYPE, RhsMapper, \ + ConjugateRhs, BuiltIn>::run(rows, cols, LhsMapper(lhs, lhsStride), \ + RhsMapper(rhs_copy.data(), Index(1)), result.data(), \ + Index(1), alpha); \ + Map<GEMVVector, 0, InnerStride<>>(res, rows, InnerStride<>(resIncr)) += result; \ + return; \ + } \ + BlasIndex m = convert_index<BlasIndex>(rows), n = convert_index<BlasIndex>(cols); \ + const EIGTYPE beta(1); \ + const EIGTYPE* x_ptr; \ + char trans = (LhsStorageOrder == ColMajor) ? 'N' : (ConjugateLhs) ? 'C' : 'T'; \ + EIGEN_IF_CONSTEXPR (LhsStorageOrder == RowMajor) { \ + m = convert_index<BlasIndex>(cols); \ + n = convert_index<BlasIndex>(rows); \ + } \ + const BlasIndex lda = blas_leading_dim<LhsStorageOrder>(rows, cols, lhsStride); \ + BlasIndex incx = blas_vector_increment(cols, rhsIncr); \ + const BlasIndex incy = blas_vector_increment(rows, resIncr); \ + GEMVVector x_tmp; \ + EIGEN_IF_CONSTEXPR (ConjugateRhs) { \ + Map<const GEMVVector, 0, InnerStride<>> map_x(rhs, cols, 1, InnerStride<>(incx)); \ + x_tmp = map_x.conjugate(); \ + x_ptr = x_tmp.data(); \ + incx = 1; \ + } else { \ + /* BLAS reads a negative increment from the lowest address, Eigen from entry 0. */ \ + x_ptr = rhsIncr < 0 ? rhs + (cols - 1) * rhsIncr : rhs; \ + } \ + auto* y_ptr = resIncr < 0 ? res + (rows - 1) * resIncr : res; \ + BLASFUNC(&trans, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)lhs, &lda, \ + (const BLASTYPE*)x_ptr, &incx, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)y_ptr, &incy); \ + } \ }; #ifdef EIGEN_USE_MKL
diff --git a/Eigen/src/Core/products/SelfadjointMatrixMatrix_BLAS.h b/Eigen/src/Core/products/SelfadjointMatrixMatrix_BLAS.h index 5398b7d..d8f298c 100644 --- a/Eigen/src/Core/products/SelfadjointMatrixMatrix_BLAS.h +++ b/Eigen/src/Core/products/SelfadjointMatrixMatrix_BLAS.h
@@ -43,112 +43,130 @@ /* Optimized selfadjoint matrix * matrix (?SYMM/?HEMM) product */ -#define EIGEN_BLAS_SYMM_L(EIGTYPE, BLASTYPE, EIGPREFIX, BLASFUNC) \ - template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ - struct product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, true, ConjugateLhs, RhsStorageOrder, false, \ - ConjugateRhs, ColMajor, 1> { \ - static void run(Index rows, Index cols, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ - Index rhsStride, EIGTYPE* res, Index resIncr, Index resStride, EIGTYPE alpha, \ - level3_blocking<EIGTYPE, EIGTYPE>& /*blocking*/) { \ - if (rows == 0 || cols == 0) return; \ - EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ - eigen_assert(resIncr == 1); \ - char side = 'L', uplo = 'L'; \ - BlasIndex m, n, lda, ldb, ldc; \ - const EIGTYPE *a, *b; \ - EIGTYPE beta(1); \ - MatrixX##EIGPREFIX b_tmp; \ - \ - /* Set transpose options */ \ - /* Set m, n, k */ \ - m = convert_index<BlasIndex>(rows); \ - n = convert_index<BlasIndex>(cols); \ - \ - /* Set lda, ldb, ldc */ \ - lda = convert_index<BlasIndex>(lhsStride); \ - ldb = convert_index<BlasIndex>(rhsStride); \ - ldc = convert_index<BlasIndex>(resStride); \ - \ - /* Set a, b, c */ \ - EIGEN_IF_CONSTEXPR (LhsStorageOrder == RowMajor) { \ - uplo = 'U'; \ - } \ - a = _lhs; \ - \ - EIGEN_IF_CONSTEXPR (RhsStorageOrder == RowMajor) { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(_rhs, n, m, OuterStride<>(rhsStride)); \ - b_tmp = rhs.adjoint(); \ - b = b_tmp.data(); \ - ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ - } else \ - b = _rhs; \ - \ - BLASFUNC(&side, &uplo, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ - (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ - } \ +#define EIGEN_BLAS_SYMM_L(EIGTYPE, BLASTYPE, EIGPREFIX, BLASFUNC) \ + template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ + struct product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, true, ConjugateLhs, RhsStorageOrder, false, \ + ConjugateRhs, ColMajor, 1> { \ + using Scalar = EIGTYPE; \ + static void run(Index rows, Index cols, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ + Index rhsStride, Scalar* res, Index resIncr, Index resStride, EIGTYPE alpha, \ + level3_blocking<EIGTYPE, EIGTYPE>& blocking) { \ + if (rows == 0 || cols == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<LhsStorageOrder>(rows, rows, lhsStride) || \ + !blas_compatible_stride<RhsStorageOrder>(rows, cols, rhsStride) || \ + !blas_compatible_stride<ColMajor>(rows, cols, resStride)) { \ + product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, true, ConjugateLhs, RhsStorageOrder, false, \ + ConjugateRhs, ColMajor, Dynamic>::run(rows, cols, _lhs, lhsStride, _rhs, rhsStride, \ + res, resIncr, resStride, alpha, blocking); \ + return; \ + } \ + EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ + eigen_assert(resIncr == 1); \ + char side = 'L', uplo = 'L'; \ + BlasIndex m, n, lda, ldb, ldc; \ + const EIGTYPE *a, *b; \ + EIGTYPE beta(1); \ + MatrixX##EIGPREFIX b_tmp; \ + \ + /* Set transpose options */ \ + /* Set m, n, k */ \ + m = convert_index<BlasIndex>(rows); \ + n = convert_index<BlasIndex>(cols); \ + \ + /* Set lda, ldb, ldc */ \ + lda = blas_leading_dim<LhsStorageOrder>(rows, rows, lhsStride); \ + ldb = blas_leading_dim<RhsStorageOrder>(rows, cols, rhsStride); \ + ldc = blas_leading_dim<ColMajor>(rows, cols, resStride); \ + \ + /* Set a, b, c */ \ + EIGEN_IF_CONSTEXPR (LhsStorageOrder == RowMajor) { \ + uplo = 'U'; \ + } \ + a = _lhs; \ + \ + EIGEN_IF_CONSTEXPR (RhsStorageOrder == RowMajor) { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(_rhs, n, m, OuterStride<>(rhsStride)); \ + b_tmp = rhs.adjoint(); \ + b = b_tmp.data(); \ + ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ + } else \ + b = _rhs; \ + \ + BLASFUNC(&side, &uplo, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ + (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ + } \ }; -#define EIGEN_BLAS_HEMM_L(EIGTYPE, BLASTYPE, EIGPREFIX, BLASFUNC) \ - template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ - struct product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, true, ConjugateLhs, RhsStorageOrder, false, \ - ConjugateRhs, ColMajor, 1> { \ - static void run(Index rows, Index cols, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ - Index rhsStride, EIGTYPE* res, Index resIncr, Index resStride, EIGTYPE alpha, \ - level3_blocking<EIGTYPE, EIGTYPE>& /*blocking*/) { \ - if (rows == 0 || cols == 0) return; \ - EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ - eigen_assert(resIncr == 1); \ - char side = 'L', uplo = 'L'; \ - BlasIndex m, n, lda, ldb, ldc; \ - const EIGTYPE *a, *b; \ - EIGTYPE beta(1); \ - MatrixX##EIGPREFIX b_tmp; \ - Matrix<EIGTYPE, Dynamic, Dynamic, LhsStorageOrder> a_tmp; \ - \ - /* Set transpose options */ \ - /* Set m, n, k */ \ - m = convert_index<BlasIndex>(rows); \ - n = convert_index<BlasIndex>(cols); \ - \ - /* Set lda, ldb, ldc */ \ - lda = convert_index<BlasIndex>(lhsStride); \ - ldb = convert_index<BlasIndex>(rhsStride); \ - ldc = convert_index<BlasIndex>(resStride); \ - \ - /* Set a, b, c */ \ - EIGEN_IF_CONSTEXPR (((LhsStorageOrder == ColMajor) && ConjugateLhs) || \ - ((LhsStorageOrder == RowMajor) && (!ConjugateLhs))) { \ - Map<const Matrix<EIGTYPE, Dynamic, Dynamic, LhsStorageOrder>, 0, OuterStride<> > lhs( \ - _lhs, m, m, OuterStride<>(lhsStride)); \ - a_tmp = lhs.conjugate(); \ - a = a_tmp.data(); \ - lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ - } else \ - a = _lhs; \ - EIGEN_IF_CONSTEXPR (LhsStorageOrder == RowMajor) { \ - uplo = 'U'; \ - } \ - \ - EIGEN_IF_CONSTEXPR (RhsStorageOrder == ColMajor && (!ConjugateRhs)) { \ - b = _rhs; \ - } else { \ - EIGEN_IF_CONSTEXPR (RhsStorageOrder == ColMajor && ConjugateRhs) { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(_rhs, m, n, OuterStride<>(rhsStride)); \ - b_tmp = rhs.conjugate(); \ - } else EIGEN_IF_CONSTEXPR (ConjugateRhs) { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(_rhs, n, m, OuterStride<>(rhsStride)); \ - b_tmp = rhs.adjoint(); \ - } else { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(_rhs, n, m, OuterStride<>(rhsStride)); \ - b_tmp = rhs.transpose(); \ - } \ - b = b_tmp.data(); \ - ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ - } \ - \ - BLASFUNC(&side, &uplo, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ - (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ - } \ +#define EIGEN_BLAS_HEMM_L(EIGTYPE, BLASTYPE, EIGPREFIX, BLASFUNC) \ + template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ + struct product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, true, ConjugateLhs, RhsStorageOrder, false, \ + ConjugateRhs, ColMajor, 1> { \ + using Scalar = EIGTYPE; \ + static void run(Index rows, Index cols, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ + Index rhsStride, Scalar* res, Index resIncr, Index resStride, EIGTYPE alpha, \ + level3_blocking<EIGTYPE, EIGTYPE>& blocking) { \ + if (rows == 0 || cols == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<LhsStorageOrder>(rows, rows, lhsStride) || \ + !blas_compatible_stride<RhsStorageOrder>(rows, cols, rhsStride) || \ + !blas_compatible_stride<ColMajor>(rows, cols, resStride)) { \ + product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, true, ConjugateLhs, RhsStorageOrder, false, \ + ConjugateRhs, ColMajor, Dynamic>::run(rows, cols, _lhs, lhsStride, _rhs, rhsStride, \ + res, resIncr, resStride, alpha, blocking); \ + return; \ + } \ + EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ + eigen_assert(resIncr == 1); \ + char side = 'L', uplo = 'L'; \ + BlasIndex m, n, lda, ldb, ldc; \ + const EIGTYPE *a, *b; \ + EIGTYPE beta(1); \ + MatrixX##EIGPREFIX b_tmp; \ + Matrix<EIGTYPE, Dynamic, Dynamic, LhsStorageOrder> a_tmp; \ + \ + /* Set transpose options */ \ + /* Set m, n, k */ \ + m = convert_index<BlasIndex>(rows); \ + n = convert_index<BlasIndex>(cols); \ + \ + /* Set lda, ldb, ldc */ \ + lda = blas_leading_dim<LhsStorageOrder>(rows, rows, lhsStride); \ + ldb = blas_leading_dim<RhsStorageOrder>(rows, cols, rhsStride); \ + ldc = blas_leading_dim<ColMajor>(rows, cols, resStride); \ + \ + /* Set a, b, c */ \ + EIGEN_IF_CONSTEXPR (((LhsStorageOrder == ColMajor) && ConjugateLhs) || \ + ((LhsStorageOrder == RowMajor) && (!ConjugateLhs))) { \ + Map<const Matrix<EIGTYPE, Dynamic, Dynamic, LhsStorageOrder>, 0, OuterStride<> > lhs( \ + _lhs, m, m, OuterStride<>(lhsStride)); \ + a_tmp = lhs.conjugate(); \ + a = a_tmp.data(); \ + lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ + } else \ + a = _lhs; \ + EIGEN_IF_CONSTEXPR (LhsStorageOrder == RowMajor) { \ + uplo = 'U'; \ + } \ + \ + EIGEN_IF_CONSTEXPR (RhsStorageOrder == ColMajor && (!ConjugateRhs)) { \ + b = _rhs; \ + } else { \ + EIGEN_IF_CONSTEXPR (RhsStorageOrder == ColMajor && ConjugateRhs) { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(_rhs, m, n, OuterStride<>(rhsStride)); \ + b_tmp = rhs.conjugate(); \ + } else EIGEN_IF_CONSTEXPR (ConjugateRhs) { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(_rhs, n, m, OuterStride<>(rhsStride)); \ + b_tmp = rhs.adjoint(); \ + } else { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > rhs(_rhs, n, m, OuterStride<>(rhsStride)); \ + b_tmp = rhs.transpose(); \ + } \ + b = b_tmp.data(); \ + ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ + } \ + \ + BLASFUNC(&side, &uplo, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ + (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ + } \ }; #ifdef EIGEN_USE_MKL @@ -165,109 +183,128 @@ /* Optimized matrix * selfadjoint matrix (?SYMM/?HEMM) product */ -#define EIGEN_BLAS_SYMM_R(EIGTYPE, BLASTYPE, EIGPREFIX, BLASFUNC) \ - template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ - struct product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, false, ConjugateLhs, RhsStorageOrder, true, \ - ConjugateRhs, ColMajor, 1> { \ - static void run(Index rows, Index cols, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ - Index rhsStride, EIGTYPE* res, Index resIncr, Index resStride, EIGTYPE alpha, \ - level3_blocking<EIGTYPE, EIGTYPE>& /*blocking*/) { \ - if (rows == 0 || cols == 0) return; \ - EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ - eigen_assert(resIncr == 1); \ - char side = 'R', uplo = 'L'; \ - BlasIndex m, n, lda, ldb, ldc; \ - const EIGTYPE *a, *b; \ - EIGTYPE beta(1); \ - MatrixX##EIGPREFIX b_tmp; \ - \ - /* Set m, n, k */ \ - m = convert_index<BlasIndex>(rows); \ - n = convert_index<BlasIndex>(cols); \ - \ - /* Set lda, ldb, ldc */ \ - lda = convert_index<BlasIndex>(rhsStride); \ - ldb = convert_index<BlasIndex>(lhsStride); \ - ldc = convert_index<BlasIndex>(resStride); \ - \ - /* Set a, b, c */ \ - EIGEN_IF_CONSTEXPR (RhsStorageOrder == RowMajor) { \ - uplo = 'U'; \ - } \ - a = _rhs; \ - \ - EIGEN_IF_CONSTEXPR (LhsStorageOrder == RowMajor) { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(_lhs, n, m, OuterStride<>(rhsStride)); \ - b_tmp = lhs.adjoint(); \ - b = b_tmp.data(); \ - ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ - } else \ - b = _lhs; \ - \ - BLASFUNC(&side, &uplo, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ - (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ - } \ +#define EIGEN_BLAS_SYMM_R(EIGTYPE, BLASTYPE, EIGPREFIX, BLASFUNC) \ + template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ + struct product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, false, ConjugateLhs, RhsStorageOrder, true, \ + ConjugateRhs, ColMajor, 1> { \ + using Scalar = EIGTYPE; \ + static void run(Index rows, Index cols, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ + Index rhsStride, Scalar* res, Index resIncr, Index resStride, EIGTYPE alpha, \ + level3_blocking<EIGTYPE, EIGTYPE>& blocking) { \ + if (rows == 0 || cols == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<LhsStorageOrder>(rows, cols, lhsStride) || \ + !blas_compatible_stride<RhsStorageOrder>(cols, cols, rhsStride) || \ + !blas_compatible_stride<ColMajor>(rows, cols, resStride)) { \ + product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, false, ConjugateLhs, RhsStorageOrder, true, \ + ConjugateRhs, ColMajor, Dynamic>::run(rows, cols, _lhs, lhsStride, _rhs, rhsStride, \ + res, resIncr, resStride, alpha, blocking); \ + return; \ + } \ + EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ + eigen_assert(resIncr == 1); \ + char side = 'R', uplo = 'L'; \ + BlasIndex m, n, lda, ldb, ldc; \ + const EIGTYPE *a, *b; \ + EIGTYPE beta(1); \ + MatrixX##EIGPREFIX b_tmp; \ + \ + /* Set m, n, k */ \ + m = convert_index<BlasIndex>(rows); \ + n = convert_index<BlasIndex>(cols); \ + \ + /* Set lda, ldb, ldc */ \ + lda = blas_leading_dim<RhsStorageOrder>(cols, cols, rhsStride); \ + ldb = blas_leading_dim<LhsStorageOrder>(rows, cols, lhsStride); \ + ldc = blas_leading_dim<ColMajor>(rows, cols, resStride); \ + \ + /* Set a, b, c */ \ + EIGEN_IF_CONSTEXPR (RhsStorageOrder == RowMajor) { \ + uplo = 'U'; \ + } \ + a = _rhs; \ + \ + EIGEN_IF_CONSTEXPR (LhsStorageOrder == RowMajor) { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(_lhs, n, m, OuterStride<>(lhsStride)); \ + b_tmp = lhs.adjoint(); \ + b = b_tmp.data(); \ + ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ + } else \ + b = _lhs; \ + \ + BLASFUNC(&side, &uplo, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ + (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ + } \ }; -#define EIGEN_BLAS_HEMM_R(EIGTYPE, BLASTYPE, EIGPREFIX, BLASFUNC) \ - template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ - struct product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, false, ConjugateLhs, RhsStorageOrder, true, \ - ConjugateRhs, ColMajor, 1> { \ - static void run(Index rows, Index cols, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ - Index rhsStride, EIGTYPE* res, Index resIncr, Index resStride, EIGTYPE alpha, \ - level3_blocking<EIGTYPE, EIGTYPE>& /*blocking*/) { \ - EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ - eigen_assert(resIncr == 1); \ - char side = 'R', uplo = 'L'; \ - BlasIndex m, n, lda, ldb, ldc; \ - const EIGTYPE *a, *b; \ - EIGTYPE beta(1); \ - MatrixX##EIGPREFIX b_tmp; \ - Matrix<EIGTYPE, Dynamic, Dynamic, RhsStorageOrder> a_tmp; \ - \ - /* Set m, n, k */ \ - m = convert_index<BlasIndex>(rows); \ - n = convert_index<BlasIndex>(cols); \ - \ - /* Set lda, ldb, ldc */ \ - lda = convert_index<BlasIndex>(rhsStride); \ - ldb = convert_index<BlasIndex>(lhsStride); \ - ldc = convert_index<BlasIndex>(resStride); \ - \ - /* Set a, b, c */ \ - EIGEN_IF_CONSTEXPR (((RhsStorageOrder == ColMajor) && ConjugateRhs) || \ - ((RhsStorageOrder == RowMajor) && (!ConjugateRhs))) { \ - Map<const Matrix<EIGTYPE, Dynamic, Dynamic, RhsStorageOrder>, 0, OuterStride<> > rhs( \ - _rhs, n, n, OuterStride<>(rhsStride)); \ - a_tmp = rhs.conjugate(); \ - a = a_tmp.data(); \ - lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ - } else \ - a = _rhs; \ - EIGEN_IF_CONSTEXPR (RhsStorageOrder == RowMajor) { \ - uplo = 'U'; \ - } \ - \ - EIGEN_IF_CONSTEXPR (LhsStorageOrder == ColMajor && (!ConjugateLhs)) { \ - b = _lhs; \ - } else { \ - EIGEN_IF_CONSTEXPR (LhsStorageOrder == ColMajor && ConjugateLhs) { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(_lhs, m, n, OuterStride<>(lhsStride)); \ - b_tmp = lhs.conjugate(); \ - } else EIGEN_IF_CONSTEXPR (ConjugateLhs) { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(_lhs, n, m, OuterStride<>(lhsStride)); \ - b_tmp = lhs.adjoint(); \ - } else { \ - Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(_lhs, n, m, OuterStride<>(lhsStride)); \ - b_tmp = lhs.transpose(); \ - } \ - b = b_tmp.data(); \ - ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ - } \ - \ - BLASFUNC(&side, &uplo, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ - (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ - } \ +#define EIGEN_BLAS_HEMM_R(EIGTYPE, BLASTYPE, EIGPREFIX, BLASFUNC) \ + template <typename Index, int LhsStorageOrder, bool ConjugateLhs, int RhsStorageOrder, bool ConjugateRhs> \ + struct product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, false, ConjugateLhs, RhsStorageOrder, true, \ + ConjugateRhs, ColMajor, 1> { \ + using Scalar = EIGTYPE; \ + static void run(Index rows, Index cols, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ + Index rhsStride, Scalar* res, Index resIncr, Index resStride, EIGTYPE alpha, \ + level3_blocking<EIGTYPE, EIGTYPE>& blocking) { \ + if (rows == 0 || cols == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<LhsStorageOrder>(rows, cols, lhsStride) || \ + !blas_compatible_stride<RhsStorageOrder>(cols, cols, rhsStride) || \ + !blas_compatible_stride<ColMajor>(rows, cols, resStride)) { \ + product_selfadjoint_matrix<EIGTYPE, Index, LhsStorageOrder, false, ConjugateLhs, RhsStorageOrder, true, \ + ConjugateRhs, ColMajor, Dynamic>::run(rows, cols, _lhs, lhsStride, _rhs, rhsStride, \ + res, resIncr, resStride, alpha, blocking); \ + return; \ + } \ + EIGEN_ONLY_USED_FOR_DEBUG(resIncr); \ + eigen_assert(resIncr == 1); \ + char side = 'R', uplo = 'L'; \ + BlasIndex m, n, lda, ldb, ldc; \ + const EIGTYPE *a, *b; \ + EIGTYPE beta(1); \ + MatrixX##EIGPREFIX b_tmp; \ + Matrix<EIGTYPE, Dynamic, Dynamic, RhsStorageOrder> a_tmp; \ + \ + /* Set m, n, k */ \ + m = convert_index<BlasIndex>(rows); \ + n = convert_index<BlasIndex>(cols); \ + \ + /* Set lda, ldb, ldc */ \ + lda = blas_leading_dim<RhsStorageOrder>(cols, cols, rhsStride); \ + ldb = blas_leading_dim<LhsStorageOrder>(rows, cols, lhsStride); \ + ldc = blas_leading_dim<ColMajor>(rows, cols, resStride); \ + \ + /* Set a, b, c */ \ + EIGEN_IF_CONSTEXPR (((RhsStorageOrder == ColMajor) && ConjugateRhs) || \ + ((RhsStorageOrder == RowMajor) && (!ConjugateRhs))) { \ + Map<const Matrix<EIGTYPE, Dynamic, Dynamic, RhsStorageOrder>, 0, OuterStride<> > rhs( \ + _rhs, n, n, OuterStride<>(rhsStride)); \ + a_tmp = rhs.conjugate(); \ + a = a_tmp.data(); \ + lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ + } else \ + a = _rhs; \ + EIGEN_IF_CONSTEXPR (RhsStorageOrder == RowMajor) { \ + uplo = 'U'; \ + } \ + \ + EIGEN_IF_CONSTEXPR (LhsStorageOrder == ColMajor && (!ConjugateLhs)) { \ + b = _lhs; \ + } else { \ + EIGEN_IF_CONSTEXPR (LhsStorageOrder == ColMajor && ConjugateLhs) { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(_lhs, m, n, OuterStride<>(lhsStride)); \ + b_tmp = lhs.conjugate(); \ + } else EIGEN_IF_CONSTEXPR (ConjugateLhs) { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(_lhs, n, m, OuterStride<>(lhsStride)); \ + b_tmp = lhs.adjoint(); \ + } else { \ + Map<const MatrixX##EIGPREFIX, 0, OuterStride<> > lhs(_lhs, n, m, OuterStride<>(lhsStride)); \ + b_tmp = lhs.transpose(); \ + } \ + b = b_tmp.data(); \ + ldb = convert_index<BlasIndex>(b_tmp.outerStride()); \ + } \ + \ + BLASFUNC(&side, &uplo, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, \ + (const BLASTYPE*)b, &ldb, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &ldc); \ + } \ }; #ifdef EIGEN_USE_MKL
diff --git a/Eigen/src/Core/products/SelfadjointMatrixVector_BLAS.h b/Eigen/src/Core/products/SelfadjointMatrixVector_BLAS.h index 71f1dbb..f8db2cf 100644 --- a/Eigen/src/Core/products/SelfadjointMatrixVector_BLAS.h +++ b/Eigen/src/Core/products/SelfadjointMatrixVector_BLAS.h
@@ -72,29 +72,36 @@ EIGEN_BLAS_SYMV_SPECIALIZE(dcomplex) EIGEN_BLAS_SYMV_SPECIALIZE(scomplex) -#define EIGEN_BLAS_SYMV_SPECIALIZATION(EIGTYPE, BLASTYPE, BLASFUNC) \ - template <typename Index, int StorageOrder, int UpLo, bool ConjugateLhs, bool ConjugateRhs> \ - struct selfadjoint_matrix_vector_product_symv<EIGTYPE, Index, StorageOrder, UpLo, ConjugateLhs, ConjugateRhs> { \ - typedef Matrix<EIGTYPE, Dynamic, 1, ColMajor> SYMVVector; \ - \ - static void run(Index size, const EIGTYPE* lhs, Index lhsStride, const EIGTYPE* _rhs, EIGTYPE* res, \ - EIGTYPE alpha) { \ - if (size == 0) return; \ - enum { IsRowMajor = StorageOrder == RowMajor ? 1 : 0, IsLower = UpLo == Lower ? 1 : 0 }; \ - BlasIndex n = convert_index<BlasIndex>(size), lda = convert_index<BlasIndex>(lhsStride), incx = 1, incy = 1; \ - EIGTYPE beta(1); \ - const EIGTYPE* x_ptr; \ - char uplo = (IsRowMajor) ? (IsLower ? 'U' : 'L') : (IsLower ? 'L' : 'U'); \ - SYMVVector x_tmp; \ - EIGEN_IF_CONSTEXPR (ConjugateRhs) { \ - Map<const SYMVVector, 0> map_x(_rhs, size, 1); \ - x_tmp = map_x.conjugate(); \ - x_ptr = x_tmp.data(); \ - } else \ - x_ptr = _rhs; \ - BLASFUNC(&uplo, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)lhs, &lda, \ - (const BLASTYPE*)x_ptr, &incx, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &incy); \ - } \ +#define EIGEN_BLAS_SYMV_SPECIALIZATION(EIGTYPE, BLASTYPE, BLASFUNC) \ + template <typename Index, int StorageOrder, int UpLo, bool ConjugateLhs, bool ConjugateRhs> \ + struct selfadjoint_matrix_vector_product_symv<EIGTYPE, Index, StorageOrder, UpLo, ConjugateLhs, ConjugateRhs> { \ + using Scalar = EIGTYPE; \ + typedef Matrix<EIGTYPE, Dynamic, 1, ColMajor> SYMVVector; \ + \ + static void run(Index size, const EIGTYPE* lhs, Index lhsStride, const EIGTYPE* _rhs, Scalar* res, \ + EIGTYPE alpha) { \ + if (size == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<StorageOrder>(size, size, lhsStride)) { \ + selfadjoint_matrix_vector_product<EIGTYPE, Index, StorageOrder, UpLo, ConjugateLhs, ConjugateRhs, \ + BuiltIn>::run(size, lhs, lhsStride, _rhs, res, alpha); \ + return; \ + } \ + enum { IsRowMajor = StorageOrder == RowMajor ? 1 : 0, IsLower = UpLo == Lower ? 1 : 0 }; \ + BlasIndex n = convert_index<BlasIndex>(size), lda = blas_leading_dim<StorageOrder>(size, size, lhsStride), \ + incx = 1, incy = 1; \ + EIGTYPE beta(1); \ + const EIGTYPE* x_ptr; \ + char uplo = (IsRowMajor) ? (IsLower ? 'U' : 'L') : (IsLower ? 'L' : 'U'); \ + SYMVVector x_tmp; \ + EIGEN_IF_CONSTEXPR (ConjugateRhs) { \ + Map<const SYMVVector, 0> map_x(_rhs, size, 1); \ + x_tmp = map_x.conjugate(); \ + x_ptr = x_tmp.data(); \ + } else \ + x_ptr = _rhs; \ + BLASFUNC(&uplo, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)lhs, &lda, \ + (const BLASTYPE*)x_ptr, &incx, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)res, &incy); \ + } \ }; #ifdef EIGEN_USE_MKL
diff --git a/Eigen/src/Core/products/TriangularMatrixMatrix.h b/Eigen/src/Core/products/TriangularMatrixMatrix.h index 04a657f..8c04cb9 100644 --- a/Eigen/src/Core/products/TriangularMatrixMatrix.h +++ b/Eigen/src/Core/products/TriangularMatrixMatrix.h
@@ -48,6 +48,9 @@ using Traits = gebp_traits<Scalar, Scalar>; enum { SmallPanelWidth = 2 * plain_enum_max(Traits::mr, Traits::nr), + // A multiple of mr, so that horizontal panels start on lhs micro-panel boundaries. It exceeds SmallPanelWidth only + // if nr > mr, e.g. for NEON double (18 vs 16), whose vertical panels ran 1-3% slower at 18 on Cortex-A76. + HorizontalPanelWidth = numext::div_ceil(int(SmallPanelWidth), int(Traits::mr)) * int(Traits::mr), IsLower = (Mode & Lower) == Lower, SetDiag = (Mode & (ZeroDiag | UnitDiag)) ? 0 : 1 }; @@ -79,10 +82,22 @@ Index kc = blocking.kc(); // cache block size along the K direction Index mc = (std::min)(rows, blocking.mc()); // cache block size along the M direction +#ifdef EIGEN_VECTORIZE_SME + // The SME kernel moves each output block through ZA on every call, which a panelWidth-deep update does not amortize. + constexpr bool HorizontalBelowNr = sme_has_gebp_kernel<Scalar, Scalar>::value; +#else + constexpr bool HorizontalBelowNr = false; +#endif + // P horizontal panels update each output panel once instead of (P+1)/2 times on average, with the same packed + // coefficients and flops. Below nr columns, the generic kernel's partial register blocks run faster over the short + // depth of vertical panels. + const bool horizontal = + (HorizontalBelowNr || cols >= Traits::nr) && Index(HorizontalPanelWidth) <= (std::min)(kc, mc); // The small panel size must not be larger than blocking size. // Usually this should never be the case because SmallPanelWidth^2 is very small // compared to L2 cache size, but let's be safe: - Index panelWidth = (std::min)(Index(SmallPanelWidth), (std::min)(kc, mc)); + const Index panelWidth = + horizontal ? Index(HorizontalPanelWidth) : (std::min)(Index(SmallPanelWidth), (std::min)(kc, mc)); std::size_t sizeA = kc * mc; std::size_t sizeB = kc * cols; @@ -92,19 +107,21 @@ static constexpr int PanelCapacity = (EIGEN_STACK_ALLOCATION_LIMIT == 0 || - SmallPanelWidth * SmallPanelWidth * sizeof(Scalar) <= EIGEN_STACK_ALLOCATION_LIMIT) - ? int(SmallPanelWidth) + HorizontalPanelWidth * HorizontalPanelWidth * sizeof(Scalar) <= EIGEN_STACK_ALLOCATION_LIMIT) + ? int(HorizontalPanelWidth) : Dynamic; - Matrix<Scalar, PanelCapacity, PanelCapacity, LhsStorageOrder> triangularBuffer(SmallPanelWidth, SmallPanelWidth); - triangularBuffer.setZero(); + Matrix<Scalar, PanelCapacity, PanelCapacity, LhsStorageOrder> triangularBuffer(HorizontalPanelWidth, + HorizontalPanelWidth); + triangularBuffer.topLeftCorner(panelWidth, panelWidth).setZero(); EIGEN_IF_CONSTEXPR ((Mode & ZeroDiag) == ZeroDiag) triangularBuffer.diagonal().setZero(); else triangularBuffer.diagonal().setOnes(); gebp_kernel<Scalar, Scalar, Index, ResMapper, Traits::mr, Traits::nr, ConjugateLhs, ConjugateRhs> gebp_kernel; + // Panel mode; with stride == depth and offset 0 it packs exactly as plain mode does. gemm_pack_lhs<Scalar, Index, LhsMapper, Traits::mr, Traits::LhsProgress, typename Traits::LhsPacket4Packing, - LhsStorageOrder> + LhsStorageOrder, false, true> pack_lhs; gemm_pack_rhs<Scalar, Index, RhsMapper, Traits::nr, RhsStorageOrder> pack_rhs; @@ -129,35 +146,63 @@ // the block diagonal, if any: if (IsLower || actual_k2 < rows) { - // for each small vertical panels of lhs - for (Index k1 = 0; k1 < actual_kc; k1 += panelWidth) { - Index actualPanelWidth = std::min<Index>(actual_kc - k1, panelWidth); - Index lengthTarget = IsLower ? actual_kc - k1 - actualPanelWidth : k1; - Index startBlock = actual_k2 + k1; - Index blockBOffset = k1; - - // => GEBP with the micro triangular block - // The trick is to pack this micro block while filling the opposite triangular part with zeros. - // To this end we do an extra triangular copy to a small temporary buffer - for (Index k = 0; k < actualPanelWidth; ++k) { - EIGEN_IF_CONSTEXPR (SetDiag) triangularBuffer.coeffRef(k, k) = lhs(startBlock + k, startBlock + k); - for (Index i = IsLower ? k + 1 : 0; IsLower ? i < actualPanelWidth : i < k; ++i) - triangularBuffer.coeffRef(i, k) = lhs(startBlock + i, startBlock + k); + // The trick is to pack each micro triangular block while filling the opposite triangular part with zeros. + // To this end we do an extra triangular copy to a small temporary buffer + const auto packTriangle = [&](Scalar* dst, Index k1, Index width, Index stride, Index offset) { + const Index start = actual_k2 + k1; + for (Index k = 0; k < width; ++k) { + EIGEN_IF_CONSTEXPR (SetDiag) triangularBuffer.coeffRef(k, k) = lhs(start + k, start + k); + for (Index i = IsLower ? k + 1 : 0; IsLower ? i < width : i < k; ++i) + triangularBuffer.coeffRef(i, k) = lhs(start + i, start + k); } - pack_lhs(blockA, LhsMapper(triangularBuffer.data(), triangularBuffer.outerStride()), actualPanelWidth, - actualPanelWidth); + pack_lhs(dst, LhsMapper(triangularBuffer.data(), triangularBuffer.outerStride()), width, width, stride, offset); + }; + // Each group of horizontal panels that fits blockA is packed in the vertical order, whose tall column segments + // stream from memory, at stride actual_kc. + if (horizontal) { + const Index groupRows = mc / panelWidth * panelWidth; + for (Index g0 = 0; g0 < actual_kc; g0 += groupRows) { + const Index g1 = (std::min)(actual_kc, g0 + groupRows); + for (Index k1 = IsLower ? 0 : g0; k1 < (IsLower ? g1 : actual_kc); k1 += panelWidth) { + const Index width = (std::min)(actual_kc - k1, panelWidth); + if (k1 >= g0 && k1 < g1) packTriangle(blockA + (k1 - g0) * actual_kc, k1, width, actual_kc, k1); + const Index begin = IsLower ? (std::max)(k1 + width, g0) : g0; + const Index end = IsLower ? g1 : (std::min)(k1, g1); + if (end > begin) + pack_lhs(blockA + (begin - g0) * actual_kc, lhs.getSubMapper(actual_k2 + begin, actual_k2 + k1), width, + end - begin, actual_kc, k1); + } + for (Index k1 = g0; k1 < g1; k1 += panelWidth) { + const Index width = (std::min)(actual_kc - k1, panelWidth); + const Index offset = IsLower ? 0 : k1; + gebp_kernel(res.getSubMapper(actual_k2 + k1, 0), blockA + (k1 - g0) * actual_kc, blockB, width, + IsLower ? k1 + width : actual_kc - k1, cols, alpha, actual_kc, actual_kc, offset, offset); + } + } + } else { + // for each small vertical panels of lhs + for (Index k1 = 0; k1 < actual_kc; k1 += panelWidth) { + Index actualPanelWidth = std::min<Index>(actual_kc - k1, panelWidth); + Index lengthTarget = IsLower ? actual_kc - k1 - actualPanelWidth : k1; + Index startBlock = actual_k2 + k1; + Index blockBOffset = k1; - gebp_kernel(res.getSubMapper(startBlock, 0), blockA, blockB, actualPanelWidth, actualPanelWidth, cols, alpha, - actualPanelWidth, actual_kc, 0, blockBOffset); + // => GEBP with the micro triangular block + packTriangle(blockA, k1, actualPanelWidth, actualPanelWidth, 0); - // GEBP with remaining micro panel - if (lengthTarget > 0) { - Index startTarget = IsLower ? actual_k2 + k1 + actualPanelWidth : actual_k2; - - pack_lhs(blockA, lhs.getSubMapper(startTarget, startBlock), actualPanelWidth, lengthTarget); - - gebp_kernel(res.getSubMapper(startTarget, 0), blockA, blockB, lengthTarget, actualPanelWidth, cols, alpha, + gebp_kernel(res.getSubMapper(startBlock, 0), blockA, blockB, actualPanelWidth, actualPanelWidth, cols, alpha, actualPanelWidth, actual_kc, 0, blockBOffset); + + // GEBP with remaining micro panel + if (lengthTarget > 0) { + Index startTarget = IsLower ? actual_k2 + k1 + actualPanelWidth : actual_k2; + + pack_lhs(blockA, lhs.getSubMapper(startTarget, startBlock), actualPanelWidth, lengthTarget, + actualPanelWidth, 0); + + gebp_kernel(res.getSubMapper(startTarget, 0), blockA, blockB, lengthTarget, actualPanelWidth, cols, alpha, + actualPanelWidth, actual_kc, 0, blockBOffset); + } } } } @@ -218,7 +263,9 @@ Index mc = (std::min)(rows, blocking.mc()); // cache block size along the M direction std::size_t sizeA = kc * mc; - std::size_t sizeB = kc * cols + EIGEN_MAX_ALIGN_BYTES / sizeof(Scalar); + // Room for realigning geb below, which moves it by less than one packet: more than EIGEN_MAX_ALIGN_BYTES when + // packets are wider than the allocation alignment (RVV with LMUL > 1, or a lowered EIGEN_MAX_ALIGN_BYTES). + std::size_t sizeB = kc * cols + PacketBytes / sizeof(Scalar); ei_declare_aligned_stack_constructed_variable(Scalar, blockA, sizeA, blocking.blockA()); ei_declare_aligned_stack_constructed_variable(Scalar, blockB, sizeB, blocking.blockB()); @@ -323,6 +370,31 @@ using RhsScalar = typename Rhs::Scalar; using Scalar = typename Dest::Scalar; + if (a_lhs.rows() == 0 || a_lhs.cols() == 0 || a_rhs.cols() == 0) return; + // Reuse GEMM's generic crossover: below it, packing costs more than a coefficient-based product. + if (a_lhs.rows() + a_lhs.cols() + a_rhs.cols() < EIGEN_GEMM_TO_COEFFBASED_THRESHOLD) { + constexpr bool lower = (Mode & Lower) != 0; + constexpr bool implicitDiag = (Mode & (UnitDiag | ZeroDiag)) != 0; + evaluator<Lhs> lhs(a_lhs); + evaluator<Rhs> rhs(a_rhs); + const Index depth = a_lhs.cols(); + for (Index j = 0; j < a_rhs.cols(); ++j) { + for (Index i = 0; i < a_lhs.rows(); ++i) { + const Index diagonal = LhsIsTriangular ? i : j; + const bool prefix = LhsIsTriangular ? lower : !lower; + const Index begin = prefix ? 0 : diagonal + Index(implicitDiag); + const Index end = prefix ? (numext::mini)(depth, diagonal + Index(!implicitDiag)) : depth; + Scalar sum(0); + for (Index k = begin; k < end; ++k) sum += lhs.coeff(i, k) * rhs.coeff(k, j); + EIGEN_IF_CONSTEXPR ((Mode & UnitDiag) != 0) { + if (diagonal < depth) sum += LhsIsTriangular ? rhs.coeff(i, j) : lhs.coeff(i, j); + } + dst.coeffRef(i, j) += alpha * sum; + } + } + return; + } + using LhsBlasTraits = internal::blas_traits<Lhs>; using ActualLhsType = typename LhsBlasTraits::DirectLinearAccessType; using ActualLhsTypeCleaned = internal::remove_all_t<ActualLhsType>; @@ -333,12 +405,6 @@ internal::add_const_on_value_type_t<ActualLhsType> lhs = LhsBlasTraits::extract(a_lhs); internal::add_const_on_value_type_t<ActualRhsType> rhs = RhsBlasTraits::extract(a_rhs); - // Empty product, return early. Otherwise, we get `nullptr` use errors below when we try to access - // coeffRef(0,0). - if (lhs.size() == 0 || rhs.size() == 0) { - return; - } - LhsScalar lhs_alpha = LhsBlasTraits::extractScalarFactor(a_lhs); RhsScalar rhs_alpha = RhsBlasTraits::extractScalarFactor(a_rhs); Scalar actualAlpha = alpha * lhs_alpha * rhs_alpha; @@ -353,19 +419,40 @@ Index stripedDepth = LhsIsTriangular ? ((!IsLower) ? lhs.cols() : (std::min)(lhs.cols(), lhs.rows())) : ((IsLower) ? rhs.rows() : (std::min)(rhs.rows(), rhs.cols())); - BlockingType blocking(stripedRows, stripedCols, stripedDepth, 1, false); - - internal::product_triangular_matrix_matrix< + using Trmm = internal::product_triangular_matrix_matrix< Scalar, Index, Mode, LhsIsTriangular, (internal::traits<ActualLhsTypeCleaned>::Flags & RowMajorBit) ? RowMajor : ColMajor, LhsBlasTraits::NeedToConjugate, (internal::traits<ActualRhsTypeCleaned>::Flags & RowMajorBit) ? RowMajor : ColMajor, RhsBlasTraits::NeedToConjugate, (internal::traits<Dest>::Flags & RowMajorBit) ? RowMajor : ColMajor, - Dest::InnerStrideAtCompileTime>::run(stripedRows, stripedCols, stripedDepth, // sizes - &lhs.coeffRef(0, 0), lhs.outerStride(), // lhs info - &rhs.coeffRef(0, 0), rhs.outerStride(), // rhs info - &dst.coeffRef(0, 0), dst.innerStride(), dst.outerStride(), // result info - actualAlpha, blocking); + Dest::InnerStrideAtCompileTime>; + const auto runWith = [&](level3_blocking<Scalar, Scalar>& blocking) { + Trmm::run(stripedRows, stripedCols, stripedDepth, // sizes + &lhs.coeffRef(0, 0), lhs.outerStride(), // lhs info + &rhs.coeffRef(0, 0), rhs.outerStride(), // rhs info + &dst.coeffRef(0, 0), dst.innerStride(), dst.outerStride(), // result info + actualAlpha, blocking); + }; + +#ifdef EIGEN_VECTORIZE_AVX512 + // The AVX512 physical-L1 cap on kc already keeps the micro-panels in L1, so on large outputs BlockingType's + // extra KcFactor = 4 makes depth blocks too short to amortize result loads/stores; small and narrow products + // still favor it. Fixed-size blocking packs the whole product into inline buffers and ignores KcFactor, so only + // dynamic blocking can change; a second fixed-size blocking object would double the stack frame. + constexpr bool DynamicBlocking = Lhs::MaxRowsAtCompileTime == Dynamic || Rhs::MaxColsAtCompileTime == Dynamic || + Lhs::MaxColsAtCompileTime == Dynamic; + if (DynamicBlocking && (is_same<Scalar, float>::value || is_same<Scalar, double>::value) && + gebp_traits<Scalar, Scalar>::nr == 8 && stripedRows >= 512 && stripedCols >= 512) { + using LargeBlockingType = internal::gemm_blocking_space<(Dest::Flags & RowMajorBit) ? RowMajor : ColMajor, Scalar, + Scalar, Dynamic, Dynamic, Dynamic, 1>; + LargeBlockingType blocking(stripedRows, stripedCols, stripedDepth, 1, false); + runWith(blocking); + } else +#endif + { + BlockingType blocking(stripedRows, stripedCols, stripedDepth, 1, false); + runWith(blocking); + } // A factor s nested in a unit triangle reached the kernel through actualAlpha, so the unit diagonal // contributed alpha * s * other instead of alpha * other: subtract alpha * (s - 1) * other.
diff --git a/Eigen/src/Core/products/TriangularMatrixMatrix_BLAS.h b/Eigen/src/Core/products/TriangularMatrixMatrix_BLAS.h index 00730e6..7e0ad3e 100644 --- a/Eigen/src/Core/products/TriangularMatrixMatrix_BLAS.h +++ b/Eigen/src/Core/products/TriangularMatrixMatrix_BLAS.h
@@ -91,7 +91,17 @@ static void run(Index _rows, Index _cols, Index _depth, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ Index rhsStride, EIGTYPE* res, Index resStride, EIGTYPE alpha, \ level3_blocking<EIGTYPE, EIGTYPE>& blocking) { \ - if (_rows == 0 || _cols == 0 || _depth == 0) return; \ + if (_rows == 0 || _cols == 0 || _depth == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<LhsStorageOrder>(_rows, _depth, lhsStride) || \ + !blas_compatible_stride<RhsStorageOrder>(_depth, _cols, rhsStride) || \ + !blas_compatible_stride<ColMajor>(_rows, _cols, resStride)) { \ + product_triangular_matrix_matrix<EIGTYPE, Index, Mode, true, LhsStorageOrder, ConjugateLhs, RhsStorageOrder, \ + ConjugateRhs, ColMajor, 1, BuiltIn>::run(_rows, _cols, _depth, _lhs, \ + lhsStride, _rhs, rhsStride, res, \ + Index(1), resStride, alpha, \ + blocking); \ + return; \ + } \ Index diagSize = (std::min)(_rows, _depth); \ Index rows = IsLower ? _rows : diagSize; \ Index depth = IsLower ? diagSize : _depth; \ @@ -173,7 +183,7 @@ lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ } else { \ a = _lhs; \ - lda = convert_index<BlasIndex>(lhsStride); \ + lda = blas_leading_dim<LhsStorageOrder>(rows, depth, lhsStride); \ } \ /*std::cout << "TRMM_L: A is square! Go to BLAS TRMM implementation! \n";*/ \ /* call ?trmm*/ \ @@ -215,7 +225,17 @@ static void run(Index _rows, Index _cols, Index _depth, const EIGTYPE* _lhs, Index lhsStride, const EIGTYPE* _rhs, \ Index rhsStride, EIGTYPE* res, Index resStride, EIGTYPE alpha, \ level3_blocking<EIGTYPE, EIGTYPE>& blocking) { \ - if (_rows == 0 || _cols == 0 || _depth == 0) return; \ + if (_rows == 0 || _cols == 0 || _depth == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<LhsStorageOrder>(_rows, _depth, lhsStride) || \ + !blas_compatible_stride<RhsStorageOrder>(_depth, _cols, rhsStride) || \ + !blas_compatible_stride<ColMajor>(_rows, _cols, resStride)) { \ + product_triangular_matrix_matrix<EIGTYPE, Index, Mode, false, LhsStorageOrder, ConjugateLhs, RhsStorageOrder, \ + ConjugateRhs, ColMajor, 1, BuiltIn>::run(_rows, _cols, _depth, _lhs, \ + lhsStride, _rhs, rhsStride, res, \ + Index(1), resStride, alpha, \ + blocking); \ + return; \ + } \ Index diagSize = (std::min)(_cols, _depth); \ Index rows = _rows; \ Index depth = IsLower ? _depth : diagSize; \ @@ -299,7 +319,7 @@ lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ } else { \ a = _rhs; \ - lda = convert_index<BlasIndex>(rhsStride); \ + lda = blas_leading_dim<RhsStorageOrder>(depth, cols, rhsStride); \ } \ /*std::cout << "TRMM_R: A is square! Go to BLAS TRMM implementation! \n";*/ \ /* call ?trmm*/ \
diff --git a/Eigen/src/Core/products/TriangularMatrixVector.h b/Eigen/src/Core/products/TriangularMatrixVector.h index d7bba5c..cf69b23 100644 --- a/Eigen/src/Core/products/TriangularMatrixVector.h +++ b/Eigen/src/Core/products/TriangularMatrixVector.h
@@ -18,6 +18,16 @@ namespace internal { +// GCC 15.1 and 15.2 at -O3 with AVX2 miscompile run() for a real matrix times a complex vector: the SLP vectorizer +// breaks copies of it specialized for a constant size (#3214). GCC 15.3 is correct. MinGW's GCC 15.2 fails the same +// way, but EIGEN_COMP_GNUC_STRICT leaves MinGW out. +#if (EIGEN_COMP_GNUC_STRICT || (EIGEN_COMP_MINGW && !EIGEN_COMP_CLANG)) && EIGEN_COMP_GNUC >= 1500 && \ + EIGEN_COMP_GNUC < 1530 +#define EIGEN_TRMV_DONT_INLINE EIGEN_DONT_INLINE __attribute__((optimize("no-tree-slp-vectorize"))) +#else +#define EIGEN_TRMV_DONT_INLINE EIGEN_DONT_INLINE +#endif + template <typename Index, int Mode, typename LhsScalar, bool ConjLhs, typename RhsScalar, bool ConjRhs, int StorageOrder, int Version = Specialized> struct triangular_matrix_vector_product; @@ -28,9 +38,9 @@ static constexpr bool IsLower = (Mode & Lower) == Lower; static constexpr bool HasUnitDiag = (Mode & UnitDiag) == UnitDiag; static constexpr bool HasZeroDiag = (Mode & ZeroDiag) == ZeroDiag; - static EIGEN_DONT_INLINE void run(Index _rows, Index _cols, const LhsScalar* lhs_, Index lhsStride, - const RhsScalar* rhs_, Index rhsIncr, ResScalar* res_, Index resIncr, - const RhsScalar& alpha); + static EIGEN_TRMV_DONT_INLINE void run(Index _rows, Index _cols, const LhsScalar* lhs_, Index lhsStride, + const RhsScalar* rhs_, Index rhsIncr, ResScalar* res_, Index resIncr, + const RhsScalar& alpha); }; template <typename Index, int Mode, typename LhsScalar, bool ConjLhs, typename RhsScalar, bool ConjRhs, int Version> @@ -151,9 +161,9 @@ static constexpr bool IsLower = (Mode & Lower) == Lower; static constexpr bool HasUnitDiag = (Mode & UnitDiag) == UnitDiag; static constexpr bool HasZeroDiag = (Mode & ZeroDiag) == ZeroDiag; - static EIGEN_DONT_INLINE void run(Index _rows, Index _cols, const LhsScalar* lhs_, Index lhsStride, - const RhsScalar* rhs_, Index rhsIncr, ResScalar* res_, Index resIncr, - const ResScalar& alpha); + static EIGEN_TRMV_DONT_INLINE void run(Index _rows, Index _cols, const LhsScalar* lhs_, Index lhsStride, + const RhsScalar* rhs_, Index rhsIncr, ResScalar* res_, Index resIncr, + const ResScalar& alpha); }; template <typename Index, int Mode, typename LhsScalar, bool ConjLhs, typename RhsScalar, bool ConjRhs, int Version> @@ -214,6 +224,8 @@ } } +#undef EIGEN_TRMV_DONT_INLINE + /*************************************************************************** * Wrapper to product_triangular_vector ***************************************************************************/
diff --git a/Eigen/src/Core/products/TriangularMatrixVector_BLAS.h b/Eigen/src/Core/products/TriangularMatrixVector_BLAS.h index f6e3a8f..f8d7d49 100644 --- a/Eigen/src/Core/products/TriangularMatrixVector_BLAS.h +++ b/Eigen/src/Core/products/TriangularMatrixVector_BLAS.h
@@ -52,6 +52,34 @@ struct triangular_matrix_vector_product_trmv : triangular_matrix_vector_product<Index, Mode, LhsScalar, ConjLhs, RhsScalar, ConjRhs, StorageOrder, BuiltIn> {}; +template <typename Index, int Mode, typename Scalar, bool ConjLhs, bool ConjRhs, int Order> +void trmv_builtin(Index rows, Index cols, const Scalar* lhs, Index lhsStride, const Scalar* rhs, Index rhsIncr, + Scalar* res, Index resIncr, Scalar alpha) { + using Vector = Matrix<Scalar, Dynamic, 1>; + Vector rhs_copy, result; + Scalar* output = res; + Index output_stride = resIncr; + const Index size = (std::min)(rows, cols); + const Index output_size = (Mode & Lower) ? rows : size; + // The row-major kernel needs contiguous input; the column-major kernel needs contiguous output. + EIGEN_IF_CONSTEXPR (Order == RowMajor) { + if (rhsIncr != 1) { + rhs_copy = Map<const Vector, 0, InnerStride<>>(rhs, (Mode & Lower) ? size : cols, InnerStride<>(rhsIncr)); + rhs = rhs_copy.data(); + rhsIncr = 1; + } + } else { + if (resIncr != 1) { + result = Vector::Zero(output_size); + output = result.data(); + output_stride = 1; + } + } + triangular_matrix_vector_product<Index, Mode, Scalar, ConjLhs, Scalar, ConjRhs, Order, BuiltIn>::run( + rows, cols, lhs, lhsStride, rhs, rhsIncr, output, output_stride, alpha); + if (result.size() != 0) Map<Vector, 0, InnerStride<>>(res, output_size, InnerStride<>(resIncr)) += result; +} + #define EIGEN_BLAS_TRMV_SPECIALIZE(Scalar) \ template <typename Index, int Mode, bool ConjLhs, bool ConjRhs> \ struct triangular_matrix_vector_product<Index, Mode, Scalar, ConjLhs, Scalar, ConjRhs, ColMajor, Specialized> { \ @@ -76,92 +104,98 @@ EIGEN_BLAS_TRMV_SPECIALIZE(scomplex) // implements col-major: res += alpha * op(triangular) * vector -#define EIGEN_BLAS_TRMV_CM(EIGTYPE, BLASTYPE, EIGPREFIX, BLASPREFIX, BLASPOSTFIX) \ - template <typename Index, int Mode, bool ConjLhs, bool ConjRhs> \ - struct triangular_matrix_vector_product_trmv<Index, Mode, EIGTYPE, ConjLhs, EIGTYPE, ConjRhs, ColMajor> { \ - enum { \ - IsLower = (Mode & Lower) == Lower, \ - SetDiag = (Mode & (ZeroDiag | UnitDiag)) ? 0 : 1, \ - IsUnitDiag = (Mode & UnitDiag) ? 1 : 0, \ - IsZeroDiag = (Mode & ZeroDiag) ? 1 : 0, \ - LowUp = IsLower ? Lower : Upper \ - }; \ - static void run(Index rows_, Index cols_, const EIGTYPE* lhs_, Index lhsStride, const EIGTYPE* rhs_, \ - Index rhsIncr, EIGTYPE* res_, Index resIncr, EIGTYPE alpha) { \ - if (rows_ == 0 || cols_ == 0) return; \ - EIGEN_IF_CONSTEXPR (ConjLhs || IsZeroDiag) { \ - triangular_matrix_vector_product<Index, Mode, EIGTYPE, ConjLhs, EIGTYPE, ConjRhs, ColMajor, BuiltIn>::run( \ - rows_, cols_, lhs_, lhsStride, rhs_, rhsIncr, res_, resIncr, alpha); \ - return; \ - } \ - Index size = (std::min)(rows_, cols_); \ - Index rows = IsLower ? rows_ : size; \ - Index cols = IsLower ? size : cols_; \ - \ - typedef VectorX##EIGPREFIX VectorRhs; \ - EIGTYPE *x, *y; \ - \ - /* Set x*/ \ - Map<const VectorRhs, 0, InnerStride<> > rhs(rhs_, cols, InnerStride<>(rhsIncr)); \ - VectorRhs x_tmp; \ - EIGEN_IF_CONSTEXPR (ConjRhs) \ - x_tmp = rhs.conjugate(); \ - else \ - x_tmp = rhs; \ - x = x_tmp.data(); \ - \ - /* Square part handling */ \ - \ - char trans, uplo, diag; \ - BlasIndex m, n, lda, incx, incy; \ - EIGTYPE const* a; \ - EIGTYPE beta(1); \ - \ - /* Set m, n */ \ - n = convert_index<BlasIndex>(size); \ - lda = convert_index<BlasIndex>(lhsStride); \ - incx = 1; \ - incy = convert_index<BlasIndex>(resIncr); \ - \ - /* Set uplo, trans and diag*/ \ - trans = 'N'; \ - uplo = IsLower ? 'L' : 'U'; \ - diag = IsUnitDiag ? 'U' : 'N'; \ - \ - /* call ?TRMV*/ \ - EIGEN_CAT(EIGEN_CAT(BLASPREFIX, trmv), BLASPOSTFIX) \ - (&uplo, &trans, &diag, &n, (const BLASTYPE*)lhs_, &lda, (BLASTYPE*)x, &incx); \ - \ - /* Add op(a_tr)rhs into res*/ \ - /* BLAS addresses a vector with a negative increment from its lowest address, Eigen from entry 0. */ \ - EIGEN_CAT(EIGEN_CAT(BLASPREFIX, axpy), BLASPOSTFIX) \ - (&n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)x, &incx, \ - (BLASTYPE*)(resIncr < 0 ? res_ + (size - 1) * resIncr : res_), &incy); \ - /* Non-square case - doesn't fit to BLAS ?TRMV. Fall to default triangular product*/ \ - if (size < (std::max)(rows, cols)) { \ - EIGEN_IF_CONSTEXPR (ConjRhs) \ - x_tmp = rhs.conjugate(); \ - else \ - x_tmp = rhs; \ - x = x_tmp.data(); \ - if (size < rows) { \ - y = res_ + size * resIncr; \ - a = lhs_ + size; \ - m = convert_index<BlasIndex>(rows - size); \ - n = convert_index<BlasIndex>(size); \ - } else { \ - x += size; \ - y = res_; \ - a = lhs_ + size * lda; \ - m = convert_index<BlasIndex>(size); \ - n = convert_index<BlasIndex>(cols - size); \ - } \ - if (resIncr < 0) y += (m - 1) * resIncr; \ - EIGEN_CAT(EIGEN_CAT(BLASPREFIX, gemv), BLASPOSTFIX) \ - (&trans, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, (const BLASTYPE*)x, \ - &incx, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)y, &incy); \ - } \ - } \ +#define EIGEN_BLAS_TRMV_CM(EIGTYPE, BLASTYPE, EIGPREFIX, BLASPREFIX, BLASPOSTFIX) \ + template <typename Index, int Mode, bool ConjLhs, bool ConjRhs> \ + struct triangular_matrix_vector_product_trmv<Index, Mode, EIGTYPE, ConjLhs, EIGTYPE, ConjRhs, ColMajor> { \ + using Scalar = EIGTYPE; \ + enum { \ + IsLower = (Mode & Lower) == Lower, \ + SetDiag = (Mode & (ZeroDiag | UnitDiag)) ? 0 : 1, \ + IsUnitDiag = (Mode & UnitDiag) ? 1 : 0, \ + IsZeroDiag = (Mode & ZeroDiag) ? 1 : 0, \ + LowUp = IsLower ? Lower : Upper \ + }; \ + static void run(Index rows_, Index cols_, const EIGTYPE* lhs_, Index lhsStride, const EIGTYPE* rhs_, \ + Index rhsIncr, Scalar* res_, Index resIncr, EIGTYPE alpha) { \ + if (rows_ == 0 || cols_ == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<ColMajor>(rows_, cols_, lhsStride) || (rows_ > 1 && resIncr == 0)) { \ + trmv_builtin<Index, Mode, EIGTYPE, ConjLhs, ConjRhs, ColMajor>(rows_, cols_, lhs_, lhsStride, rhs_, rhsIncr, \ + res_, resIncr, alpha); \ + return; \ + } \ + EIGEN_IF_CONSTEXPR (ConjLhs || IsZeroDiag) { \ + trmv_builtin<Index, Mode, EIGTYPE, ConjLhs, ConjRhs, ColMajor>(rows_, cols_, lhs_, lhsStride, rhs_, rhsIncr, \ + res_, resIncr, alpha); \ + return; \ + } \ + Index size = (std::min)(rows_, cols_); \ + Index rows = IsLower ? rows_ : size; \ + Index cols = IsLower ? size : cols_; \ + \ + typedef VectorX##EIGPREFIX VectorRhs; \ + Scalar *x, *y; \ + \ + /* Set x*/ \ + Map<const VectorRhs, 0, InnerStride<>> rhs(rhs_, cols, InnerStride<>(rhsIncr)); \ + VectorRhs x_tmp; \ + EIGEN_IF_CONSTEXPR (ConjRhs) \ + x_tmp = rhs.conjugate(); \ + else \ + x_tmp = rhs; \ + x = x_tmp.data(); \ + \ + /* Square part handling */ \ + \ + char trans, uplo, diag; \ + BlasIndex m, n, lda, incx, incy; \ + EIGTYPE const* a; \ + EIGTYPE beta(1); \ + \ + /* Set m, n */ \ + n = convert_index<BlasIndex>(size); \ + lda = blas_leading_dim<ColMajor>(rows, cols, lhsStride); \ + incx = 1; \ + incy = blas_vector_increment(rows, resIncr); \ + \ + /* Set uplo, trans and diag*/ \ + trans = 'N'; \ + uplo = IsLower ? 'L' : 'U'; \ + diag = IsUnitDiag ? 'U' : 'N'; \ + \ + /* call ?TRMV*/ \ + EIGEN_CAT(EIGEN_CAT(BLASPREFIX, trmv), BLASPOSTFIX) \ + (&uplo, &trans, &diag, &n, (const BLASTYPE*)lhs_, &lda, (BLASTYPE*)x, &incx); \ + \ + /* Add op(a_tr)rhs into res*/ \ + /* BLAS addresses a vector with a negative increment from its lowest address, Eigen from entry 0. */ \ + EIGEN_CAT(EIGEN_CAT(BLASPREFIX, axpy), BLASPOSTFIX) \ + (&n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)x, &incx, \ + (BLASTYPE*)(resIncr < 0 ? res_ + (size - 1) * resIncr : res_), &incy); \ + /* Non-square case - doesn't fit to BLAS ?TRMV. Fall to default triangular product*/ \ + if (size < (std::max)(rows, cols)) { \ + EIGEN_IF_CONSTEXPR (ConjRhs) \ + x_tmp = rhs.conjugate(); \ + else \ + x_tmp = rhs; \ + x = x_tmp.data(); \ + if (size < rows) { \ + y = res_ + size * resIncr; \ + a = lhs_ + size; \ + m = convert_index<BlasIndex>(rows - size); \ + n = convert_index<BlasIndex>(size); \ + } else { \ + x += size; \ + y = res_; \ + a = lhs_ + size * lda; \ + m = convert_index<BlasIndex>(size); \ + n = convert_index<BlasIndex>(cols - size); \ + } \ + if (resIncr < 0) y += (m - 1) * resIncr; \ + EIGEN_CAT(EIGEN_CAT(BLASPREFIX, gemv), BLASPOSTFIX) \ + (&trans, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, (const BLASTYPE*)x, \ + &incx, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)y, &incy); \ + } \ + } \ }; #ifdef EIGEN_USE_MKL @@ -177,92 +211,98 @@ #endif // implements row-major: res += alpha * op(triangular) * vector -#define EIGEN_BLAS_TRMV_RM(EIGTYPE, BLASTYPE, EIGPREFIX, BLASPREFIX, BLASPOSTFIX) \ - template <typename Index, int Mode, bool ConjLhs, bool ConjRhs> \ - struct triangular_matrix_vector_product_trmv<Index, Mode, EIGTYPE, ConjLhs, EIGTYPE, ConjRhs, RowMajor> { \ - enum { \ - IsLower = (Mode & Lower) == Lower, \ - SetDiag = (Mode & (ZeroDiag | UnitDiag)) ? 0 : 1, \ - IsUnitDiag = (Mode & UnitDiag) ? 1 : 0, \ - IsZeroDiag = (Mode & ZeroDiag) ? 1 : 0, \ - LowUp = IsLower ? Lower : Upper \ - }; \ - static void run(Index rows_, Index cols_, const EIGTYPE* lhs_, Index lhsStride, const EIGTYPE* rhs_, \ - Index rhsIncr, EIGTYPE* res_, Index resIncr, EIGTYPE alpha) { \ - if (rows_ == 0 || cols_ == 0) return; \ - EIGEN_IF_CONSTEXPR (IsZeroDiag) { \ - triangular_matrix_vector_product<Index, Mode, EIGTYPE, ConjLhs, EIGTYPE, ConjRhs, RowMajor, BuiltIn>::run( \ - rows_, cols_, lhs_, lhsStride, rhs_, rhsIncr, res_, resIncr, alpha); \ - return; \ - } \ - Index size = (std::min)(rows_, cols_); \ - Index rows = IsLower ? rows_ : size; \ - Index cols = IsLower ? size : cols_; \ - \ - typedef VectorX##EIGPREFIX VectorRhs; \ - EIGTYPE *x, *y; \ - \ - /* Set x*/ \ - Map<const VectorRhs, 0, InnerStride<> > rhs(rhs_, cols, InnerStride<>(rhsIncr)); \ - VectorRhs x_tmp; \ - EIGEN_IF_CONSTEXPR (ConjRhs) \ - x_tmp = rhs.conjugate(); \ - else \ - x_tmp = rhs; \ - x = x_tmp.data(); \ - \ - /* Square part handling */ \ - \ - char trans, uplo, diag; \ - BlasIndex m, n, lda, incx, incy; \ - EIGTYPE const* a; \ - EIGTYPE beta(1); \ - \ - /* Set m, n */ \ - n = convert_index<BlasIndex>(size); \ - lda = convert_index<BlasIndex>(lhsStride); \ - incx = 1; \ - incy = convert_index<BlasIndex>(resIncr); \ - \ - /* Set uplo, trans and diag*/ \ - trans = ConjLhs ? 'C' : 'T'; \ - uplo = IsLower ? 'U' : 'L'; \ - diag = IsUnitDiag ? 'U' : 'N'; \ - \ - /* call ?TRMV*/ \ - EIGEN_CAT(EIGEN_CAT(BLASPREFIX, trmv), BLASPOSTFIX) \ - (&uplo, &trans, &diag, &n, (const BLASTYPE*)lhs_, &lda, (BLASTYPE*)x, &incx); \ - \ - /* Add op(a_tr)rhs into res*/ \ - /* BLAS addresses a vector with a negative increment from its lowest address, Eigen from entry 0. */ \ - EIGEN_CAT(EIGEN_CAT(BLASPREFIX, axpy), BLASPOSTFIX) \ - (&n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)x, &incx, \ - (BLASTYPE*)(resIncr < 0 ? res_ + (size - 1) * resIncr : res_), &incy); \ - /* Non-square case - doesn't fit to BLAS ?TRMV. Fall to default triangular product*/ \ - if (size < (std::max)(rows, cols)) { \ - EIGEN_IF_CONSTEXPR (ConjRhs) \ - x_tmp = rhs.conjugate(); \ - else \ - x_tmp = rhs; \ - x = x_tmp.data(); \ - if (size < rows) { \ - y = res_ + size * resIncr; \ - a = lhs_ + size * lda; \ - m = convert_index<BlasIndex>(rows - size); \ - n = convert_index<BlasIndex>(size); \ - } else { \ - x += size; \ - y = res_; \ - a = lhs_ + size; \ - m = convert_index<BlasIndex>(size); \ - n = convert_index<BlasIndex>(cols - size); \ - } \ - if (resIncr < 0) y += (m - 1) * resIncr; \ - EIGEN_CAT(EIGEN_CAT(BLASPREFIX, gemv), BLASPOSTFIX) \ - (&trans, &n, &m, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, (const BLASTYPE*)x, \ - &incx, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)y, &incy); \ - } \ - } \ +#define EIGEN_BLAS_TRMV_RM(EIGTYPE, BLASTYPE, EIGPREFIX, BLASPREFIX, BLASPOSTFIX) \ + template <typename Index, int Mode, bool ConjLhs, bool ConjRhs> \ + struct triangular_matrix_vector_product_trmv<Index, Mode, EIGTYPE, ConjLhs, EIGTYPE, ConjRhs, RowMajor> { \ + using Scalar = EIGTYPE; \ + enum { \ + IsLower = (Mode & Lower) == Lower, \ + SetDiag = (Mode & (ZeroDiag | UnitDiag)) ? 0 : 1, \ + IsUnitDiag = (Mode & UnitDiag) ? 1 : 0, \ + IsZeroDiag = (Mode & ZeroDiag) ? 1 : 0, \ + LowUp = IsLower ? Lower : Upper \ + }; \ + static void run(Index rows_, Index cols_, const EIGTYPE* lhs_, Index lhsStride, const EIGTYPE* rhs_, \ + Index rhsIncr, Scalar* res_, Index resIncr, EIGTYPE alpha) { \ + if (rows_ == 0 || cols_ == 0 || numext::is_exactly_zero(alpha)) return; \ + if (!blas_compatible_stride<RowMajor>(rows_, cols_, lhsStride) || (rows_ > 1 && resIncr == 0)) { \ + trmv_builtin<Index, Mode, EIGTYPE, ConjLhs, ConjRhs, RowMajor>(rows_, cols_, lhs_, lhsStride, rhs_, rhsIncr, \ + res_, resIncr, alpha); \ + return; \ + } \ + EIGEN_IF_CONSTEXPR (IsZeroDiag) { \ + trmv_builtin<Index, Mode, EIGTYPE, ConjLhs, ConjRhs, RowMajor>(rows_, cols_, lhs_, lhsStride, rhs_, rhsIncr, \ + res_, resIncr, alpha); \ + return; \ + } \ + Index size = (std::min)(rows_, cols_); \ + Index rows = IsLower ? rows_ : size; \ + Index cols = IsLower ? size : cols_; \ + \ + typedef VectorX##EIGPREFIX VectorRhs; \ + Scalar *x, *y; \ + \ + /* Set x*/ \ + Map<const VectorRhs, 0, InnerStride<>> rhs(rhs_, cols, InnerStride<>(rhsIncr)); \ + VectorRhs x_tmp; \ + EIGEN_IF_CONSTEXPR (ConjRhs) \ + x_tmp = rhs.conjugate(); \ + else \ + x_tmp = rhs; \ + x = x_tmp.data(); \ + \ + /* Square part handling */ \ + \ + char trans, uplo, diag; \ + BlasIndex m, n, lda, incx, incy; \ + EIGTYPE const* a; \ + EIGTYPE beta(1); \ + \ + /* Set m, n */ \ + n = convert_index<BlasIndex>(size); \ + lda = blas_leading_dim<RowMajor>(rows, cols, lhsStride); \ + incx = 1; \ + incy = blas_vector_increment(rows, resIncr); \ + \ + /* Set uplo, trans and diag*/ \ + trans = ConjLhs ? 'C' : 'T'; \ + uplo = IsLower ? 'U' : 'L'; \ + diag = IsUnitDiag ? 'U' : 'N'; \ + \ + /* call ?TRMV*/ \ + EIGEN_CAT(EIGEN_CAT(BLASPREFIX, trmv), BLASPOSTFIX) \ + (&uplo, &trans, &diag, &n, (const BLASTYPE*)lhs_, &lda, (BLASTYPE*)x, &incx); \ + \ + /* Add op(a_tr)rhs into res*/ \ + /* BLAS addresses a vector with a negative increment from its lowest address, Eigen from entry 0. */ \ + EIGEN_CAT(EIGEN_CAT(BLASPREFIX, axpy), BLASPOSTFIX) \ + (&n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)x, &incx, \ + (BLASTYPE*)(resIncr < 0 ? res_ + (size - 1) * resIncr : res_), &incy); \ + /* Non-square case - doesn't fit to BLAS ?TRMV. Fall to default triangular product*/ \ + if (size < (std::max)(rows, cols)) { \ + EIGEN_IF_CONSTEXPR (ConjRhs) \ + x_tmp = rhs.conjugate(); \ + else \ + x_tmp = rhs; \ + x = x_tmp.data(); \ + if (size < rows) { \ + y = res_ + size * resIncr; \ + a = lhs_ + size * lda; \ + m = convert_index<BlasIndex>(rows - size); \ + n = convert_index<BlasIndex>(size); \ + } else { \ + x += size; \ + y = res_; \ + a = lhs_ + size; \ + m = convert_index<BlasIndex>(size); \ + n = convert_index<BlasIndex>(cols - size); \ + } \ + if (resIncr < 0) y += (m - 1) * resIncr; \ + EIGEN_CAT(EIGEN_CAT(BLASPREFIX, gemv), BLASPOSTFIX) \ + (&trans, &n, &m, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, &lda, (const BLASTYPE*)x, \ + &incx, (const BLASTYPE*)&numext::real_ref(beta), (BLASTYPE*)y, &incy); \ + } \ + } \ }; #ifdef EIGEN_USE_MKL
diff --git a/Eigen/src/Core/products/TriangularSolverMatrix.h b/Eigen/src/Core/products/TriangularSolverMatrix.h index 0ec89dd..b8787ea 100644 --- a/Eigen/src/Core/products/TriangularSolverMatrix.h +++ b/Eigen/src/Core/products/TriangularSolverMatrix.h
@@ -546,7 +546,19 @@ using Traits = gebp_traits<Scalar, Scalar>; - enum { SmallPanelWidth = plain_enum_max(Traits::mr, Traits::nr), IsLower = (Mode & Lower) == Lower }; + // The diagonal panels are updated by gebp with a depth of SmallPanelWidth. The DoublePacket kernel that FMA targets + // with 16 registers use is one packet tall; where that is narrower than nr (complex<double> with AVX2+FMA), such + // shallow updates run slowly, so its panels are twice as wide. +#ifdef EIGEN_HAS_SINGLE_INSTRUCTION_MADD + constexpr bool WidePanels = is_double_packet<typename Traits::AccPacket>::value && Traits::NumberOfRegisters < 32 && + int(Traits::mr) < int(Traits::nr); +#else + constexpr bool WidePanels = false; +#endif + enum { + SmallPanelWidth = WidePanels ? 2 * Traits::nr : plain_enum_max(Traits::mr, Traits::nr), + IsLower = (Mode & Lower) == Lower + }; // Every k-block updates the rows of the right-hand side beyond it through gebp, so a right-hand side // solved whole streams through the caches size/kc times (issue #3162); column panels that stay in
diff --git a/Eigen/src/Core/products/TriangularSolverMatrix_BLAS.h b/Eigen/src/Core/products/TriangularSolverMatrix_BLAS.h index 9e7407e..3a65da1 100644 --- a/Eigen/src/Core/products/TriangularSolverMatrix_BLAS.h +++ b/Eigen/src/Core/products/TriangularSolverMatrix_BLAS.h
@@ -42,54 +42,61 @@ namespace internal { // implements LeftSide op(triangular)^-1 * general -#define EIGEN_BLAS_TRSM_L(EIGTYPE, BLASTYPE, BLASFUNC) \ - template <typename Index, int Mode, bool Conjugate, int TriStorageOrder> \ - struct triangular_solve_matrix<EIGTYPE, Index, OnTheLeft, Mode, Conjugate, TriStorageOrder, ColMajor, 1> { \ - enum { \ - IsLower = (Mode & Lower) == Lower, \ - IsUnitDiag = (Mode & UnitDiag) ? 1 : 0, \ - IsZeroDiag = (Mode & ZeroDiag) ? 1 : 0, \ - conjA = ((TriStorageOrder == ColMajor) && Conjugate) ? 1 : 0 \ - }; \ - static void run(Index size, Index otherSize, const EIGTYPE* _tri, Index triStride, EIGTYPE* _other, \ - Index otherIncr, Index otherStride, level3_blocking<EIGTYPE, EIGTYPE>& /*blocking*/) { \ - if (size == 0 || otherSize == 0) return; \ - EIGEN_ONLY_USED_FOR_DEBUG(otherIncr); \ - eigen_assert(otherIncr == 1); \ - BlasIndex m = convert_index<BlasIndex>(size), n = convert_index<BlasIndex>(otherSize), lda, ldb; \ - char side = 'L', uplo, diag = 'N', transa; \ - /* Set alpha_ */ \ - EIGTYPE alpha(1); \ - ldb = convert_index<BlasIndex>(otherStride); \ - \ - const EIGTYPE* a; \ - /* Set trans */ \ - transa = (TriStorageOrder == RowMajor) ? ((Conjugate) ? 'C' : 'T') : 'N'; \ - /* Set uplo */ \ - uplo = IsLower ? 'L' : 'U'; \ - EIGEN_IF_CONSTEXPR (TriStorageOrder == RowMajor) { \ - uplo = (uplo == 'L') ? 'U' : 'L'; \ - } \ - /* Set a, lda */ \ - typedef Matrix<EIGTYPE, Dynamic, Dynamic, TriStorageOrder> MatrixTri; \ - Map<const MatrixTri, 0, OuterStride<> > tri(_tri, size, size, OuterStride<>(triStride)); \ - MatrixTri a_tmp; \ - \ - EIGEN_IF_CONSTEXPR (conjA) { \ - a_tmp = tri.conjugate(); \ - a = a_tmp.data(); \ - lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ - } else { \ - a = _tri; \ - lda = convert_index<BlasIndex>(triStride); \ - } \ - EIGEN_IF_CONSTEXPR (IsUnitDiag) { \ - diag = 'U'; \ - } \ - /* call ?trsm*/ \ - BLASFUNC(&side, &uplo, &transa, &diag, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, \ - &lda, (BLASTYPE*)_other, &ldb); \ - } \ +#define EIGEN_BLAS_TRSM_L(EIGTYPE, BLASTYPE, BLASFUNC) \ + template <typename Index, int Mode, bool Conjugate, int TriStorageOrder> \ + struct triangular_solve_matrix<EIGTYPE, Index, OnTheLeft, Mode, Conjugate, TriStorageOrder, ColMajor, 1> { \ + using Scalar = EIGTYPE; \ + enum { \ + IsLower = (Mode & Lower) == Lower, \ + IsUnitDiag = (Mode & UnitDiag) ? 1 : 0, \ + IsZeroDiag = (Mode & ZeroDiag) ? 1 : 0, \ + conjA = ((TriStorageOrder == ColMajor) && Conjugate) ? 1 : 0 \ + }; \ + static void run(Index size, Index otherSize, const EIGTYPE* _tri, Index triStride, Scalar* _other, \ + Index otherIncr, Index otherStride, level3_blocking<EIGTYPE, EIGTYPE>& blocking) { \ + if (size == 0 || otherSize == 0) return; \ + if (!blas_compatible_stride<TriStorageOrder>(size, size, triStride) || \ + !blas_compatible_stride<ColMajor>(size, otherSize, otherStride)) { \ + triangular_solve_matrix<EIGTYPE, Index, OnTheLeft, Mode, Conjugate, TriStorageOrder, ColMajor, Dynamic>::run( \ + size, otherSize, _tri, triStride, _other, otherIncr, otherStride, blocking); \ + return; \ + } \ + EIGEN_ONLY_USED_FOR_DEBUG(otherIncr); \ + eigen_assert(otherIncr == 1); \ + BlasIndex m = convert_index<BlasIndex>(size), n = convert_index<BlasIndex>(otherSize), lda, ldb; \ + char side = 'L', uplo, diag = 'N', transa; \ + /* Set alpha_ */ \ + EIGTYPE alpha(1); \ + ldb = blas_leading_dim<ColMajor>(size, otherSize, otherStride); \ + \ + const EIGTYPE* a; \ + /* Set trans */ \ + transa = (TriStorageOrder == RowMajor) ? ((Conjugate) ? 'C' : 'T') : 'N'; \ + /* Set uplo */ \ + uplo = IsLower ? 'L' : 'U'; \ + EIGEN_IF_CONSTEXPR (TriStorageOrder == RowMajor) { \ + uplo = (uplo == 'L') ? 'U' : 'L'; \ + } \ + /* Set a, lda */ \ + typedef Matrix<EIGTYPE, Dynamic, Dynamic, TriStorageOrder> MatrixTri; \ + Map<const MatrixTri, 0, OuterStride<> > tri(_tri, size, size, OuterStride<>(triStride)); \ + MatrixTri a_tmp; \ + \ + EIGEN_IF_CONSTEXPR (conjA) { \ + a_tmp = tri.conjugate(); \ + a = a_tmp.data(); \ + lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ + } else { \ + a = _tri; \ + lda = blas_leading_dim<TriStorageOrder>(size, size, triStride); \ + } \ + EIGEN_IF_CONSTEXPR (IsUnitDiag) { \ + diag = 'U'; \ + } \ + /* call ?trsm*/ \ + BLASFUNC(&side, &uplo, &transa, &diag, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, \ + &lda, (BLASTYPE*)_other, &ldb); \ + } \ }; #ifdef EIGEN_USE_MKL @@ -105,55 +112,62 @@ #endif // implements RightSide general * op(triangular)^-1 -#define EIGEN_BLAS_TRSM_R(EIGTYPE, BLASTYPE, BLASFUNC) \ - template <typename Index, int Mode, bool Conjugate, int TriStorageOrder> \ - struct triangular_solve_matrix<EIGTYPE, Index, OnTheRight, Mode, Conjugate, TriStorageOrder, ColMajor, 1> { \ - enum { \ - IsLower = (Mode & Lower) == Lower, \ - IsUnitDiag = (Mode & UnitDiag) ? 1 : 0, \ - IsZeroDiag = (Mode & ZeroDiag) ? 1 : 0, \ - conjA = ((TriStorageOrder == ColMajor) && Conjugate) ? 1 : 0 \ - }; \ - static void run(Index size, Index otherSize, const EIGTYPE* _tri, Index triStride, EIGTYPE* _other, \ - Index otherIncr, Index otherStride, level3_blocking<EIGTYPE, EIGTYPE>& /*blocking*/) { \ - if (size == 0 || otherSize == 0) return; \ - EIGEN_ONLY_USED_FOR_DEBUG(otherIncr); \ - eigen_assert(otherIncr == 1); \ - BlasIndex m = convert_index<BlasIndex>(otherSize), n = convert_index<BlasIndex>(size), lda, ldb; \ - char side = 'R', uplo, diag = 'N', transa; \ - /* Set alpha_ */ \ - EIGTYPE alpha(1); \ - ldb = convert_index<BlasIndex>(otherStride); \ - \ - const EIGTYPE* a; \ - /* Set trans */ \ - transa = (TriStorageOrder == RowMajor) ? ((Conjugate) ? 'C' : 'T') : 'N'; \ - /* Set uplo */ \ - uplo = IsLower ? 'L' : 'U'; \ - EIGEN_IF_CONSTEXPR (TriStorageOrder == RowMajor) { \ - uplo = (uplo == 'L') ? 'U' : 'L'; \ - } \ - /* Set a, lda */ \ - typedef Matrix<EIGTYPE, Dynamic, Dynamic, TriStorageOrder> MatrixTri; \ - Map<const MatrixTri, 0, OuterStride<> > tri(_tri, size, size, OuterStride<>(triStride)); \ - MatrixTri a_tmp; \ - \ - EIGEN_IF_CONSTEXPR (conjA) { \ - a_tmp = tri.conjugate(); \ - a = a_tmp.data(); \ - lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ - } else { \ - a = _tri; \ - lda = convert_index<BlasIndex>(triStride); \ - } \ - EIGEN_IF_CONSTEXPR (IsUnitDiag) { \ - diag = 'U'; \ - } \ - /* call ?trsm*/ \ - BLASFUNC(&side, &uplo, &transa, &diag, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, \ - &lda, (BLASTYPE*)_other, &ldb); \ - /*std::cout << "TRMS_L specialization!\n";*/ \ - } \ +#define EIGEN_BLAS_TRSM_R(EIGTYPE, BLASTYPE, BLASFUNC) \ + template <typename Index, int Mode, bool Conjugate, int TriStorageOrder> \ + struct triangular_solve_matrix<EIGTYPE, Index, OnTheRight, Mode, Conjugate, TriStorageOrder, ColMajor, 1> { \ + using Scalar = EIGTYPE; \ + enum { \ + IsLower = (Mode & Lower) == Lower, \ + IsUnitDiag = (Mode & UnitDiag) ? 1 : 0, \ + IsZeroDiag = (Mode & ZeroDiag) ? 1 : 0, \ + conjA = ((TriStorageOrder == ColMajor) && Conjugate) ? 1 : 0 \ + }; \ + static void run(Index size, Index otherSize, const EIGTYPE* _tri, Index triStride, Scalar* _other, \ + Index otherIncr, Index otherStride, level3_blocking<EIGTYPE, EIGTYPE>& blocking) { \ + if (size == 0 || otherSize == 0) return; \ + if (!blas_compatible_stride<TriStorageOrder>(size, size, triStride) || \ + !blas_compatible_stride<ColMajor>(otherSize, size, otherStride)) { \ + triangular_solve_matrix<EIGTYPE, Index, OnTheRight, Mode, Conjugate, TriStorageOrder, ColMajor, Dynamic>::run( \ + size, otherSize, _tri, triStride, _other, otherIncr, otherStride, blocking); \ + return; \ + } \ + EIGEN_ONLY_USED_FOR_DEBUG(otherIncr); \ + eigen_assert(otherIncr == 1); \ + BlasIndex m = convert_index<BlasIndex>(otherSize), n = convert_index<BlasIndex>(size), lda, ldb; \ + char side = 'R', uplo, diag = 'N', transa; \ + /* Set alpha_ */ \ + EIGTYPE alpha(1); \ + ldb = blas_leading_dim<ColMajor>(otherSize, size, otherStride); \ + \ + const EIGTYPE* a; \ + /* Set trans */ \ + transa = (TriStorageOrder == RowMajor) ? ((Conjugate) ? 'C' : 'T') : 'N'; \ + /* Set uplo */ \ + uplo = IsLower ? 'L' : 'U'; \ + EIGEN_IF_CONSTEXPR (TriStorageOrder == RowMajor) { \ + uplo = (uplo == 'L') ? 'U' : 'L'; \ + } \ + /* Set a, lda */ \ + typedef Matrix<EIGTYPE, Dynamic, Dynamic, TriStorageOrder> MatrixTri; \ + Map<const MatrixTri, 0, OuterStride<> > tri(_tri, size, size, OuterStride<>(triStride)); \ + MatrixTri a_tmp; \ + \ + EIGEN_IF_CONSTEXPR (conjA) { \ + a_tmp = tri.conjugate(); \ + a = a_tmp.data(); \ + lda = convert_index<BlasIndex>(a_tmp.outerStride()); \ + } else { \ + a = _tri; \ + lda = blas_leading_dim<TriStorageOrder>(size, size, triStride); \ + } \ + EIGEN_IF_CONSTEXPR (IsUnitDiag) { \ + diag = 'U'; \ + } \ + /* call ?trsm*/ \ + BLASFUNC(&side, &uplo, &transa, &diag, &m, &n, (const BLASTYPE*)&numext::real_ref(alpha), (const BLASTYPE*)a, \ + &lda, (BLASTYPE*)_other, &ldb); \ + /*std::cout << "TRMS_L specialization!\n";*/ \ + } \ }; #ifdef EIGEN_USE_MKL
diff --git a/Eigen/src/Core/util/BlasUtil.h b/Eigen/src/Core/util/BlasUtil.h index af49412..a123323 100644 --- a/Eigen/src/Core/util/BlasUtil.h +++ b/Eigen/src/Core/util/BlasUtil.h
@@ -21,6 +21,28 @@ namespace internal { +// A singleton stored column never uses its outer stride, but BLAS still checks ld >= max(1, inner). +// Normalize before narrowing: an unused stride need not fit BlasIndex. +template <int StorageOrder, typename Index> +EIGEN_STRONG_INLINE BlasIndex blas_leading_dim(Index rows, Index cols, Index stride) { + const Index inner = StorageOrder == ColMajor ? rows : cols; + const Index outer = StorageOrder == ColMajor ? cols : rows; + return convert_index<BlasIndex>(outer <= 1 ? numext::maxi(inner, Index(1)) : stride); +} + +template <int StorageOrder, typename Index> +EIGEN_STRONG_INLINE bool blas_compatible_stride(Index rows, Index cols, Index stride) { + const Index inner = StorageOrder == ColMajor ? rows : cols; + const Index outer = StorageOrder == ColMajor ? cols : rows; + return outer <= 1 || stride >= numext::maxi(inner, Index(1)); +} + +// BLAS rejects zero increments even when the vector has only one coefficient. +template <typename Index> +EIGEN_STRONG_INLINE BlasIndex blas_vector_increment(Index size, Index stride) { + return convert_index<BlasIndex>(size <= 1 ? Index(1) : stride); +} + // forward declarations template <typename LhsScalar, typename RhsScalar, typename Index, typename DataMapper, int mr, int nr, bool ConjugateLhs = false, bool ConjugateRhs = false> @@ -34,8 +56,12 @@ bool Conjugate = false, bool PanelMode = false> struct gemm_pack_lhs; +// LhsStorage and RhsStorage are the scalar types the operands are stored in. The packers convert each coefficient to +// LhsScalar or RhsScalar as they read it, so that A.cast<float>() * B.cast<float>() runs the float kernel on half +// operands without materializing the casts. template <typename Index, typename LhsScalar, int LhsStorageOrder, bool ConjugateLhs, typename RhsScalar, - int RhsStorageOrder, bool ConjugateRhs, int ResStorageOrder, int ResInnerStride> + int RhsStorageOrder, bool ConjugateRhs, int ResStorageOrder, int ResInnerStride, + typename LhsStorage = LhsScalar, typename RhsStorage = RhsScalar> struct general_matrix_matrix_product; template <typename Index, typename LhsScalar, typename LhsMapper, int LhsStorageOrder, bool ConjugateLhs, @@ -456,6 +482,102 @@ } }; +// Loads a Packet from the SrcScalar coefficients at `from`, each converted as cast<SrcScalar, Scalar> converts it. +// Vectorized when a source packet of as many coefficients casts to Packet, as in the core_cast_op evaluator. +template <typename SrcScalar, typename Packet, typename = void> +struct cast_ploadu_impl { + using Scalar = typename unpacket_traits<Packet>::type; + static constexpr int Size = unpacket_traits<Packet>::size; + static EIGEN_ALWAYS_INLINE Packet run(const SrcScalar* from) { + Scalar to[Size]; + for (int i = 0; i < Size; ++i) to[i] = cast<SrcScalar, Scalar>(from[i]); + return ploadu<Packet>(to); + } +}; +template <typename SrcScalar, typename Packet> +struct cast_ploadu_impl< + SrcScalar, Packet, + std::enable_if_t<type_casting_traits<SrcScalar, typename unpacket_traits<Packet>::type>::VectorizedCast && + find_packet_by_size<SrcScalar, unpacket_traits<Packet>::size>::value>> { + using SrcPacket = typename find_packet_by_size<SrcScalar, unpacket_traits<Packet>::size>::type; + static EIGEN_ALWAYS_INLINE Packet run(const SrcScalar* from) { + return pcast<SrcPacket, Packet>(ploadu<SrcPacket>(from)); + } +}; +#ifdef EIGEN_VECTORIZE_VSX +// Packet8bf is its own half, so no bfloat16 packet holds four coefficients. pcast<Packet8bf, Packet4f> widens the +// first four, so splatting them as one 64-bit lane suffices. +template <> +struct cast_ploadu_impl<bfloat16, Packet4f> { + static EIGEN_ALWAYS_INLINE Packet4f run(const bfloat16* from) { + unsigned long long bits; + std::memcpy(&bits, from, sizeof(bits)); + return pcast<Packet8bf, Packet4f>(Packet8bf(reinterpret_cast<Packet8us>(vec_splats(bits)))); + } +}; +#endif + +template <typename Scalar, typename SrcScalar, typename Index> +class BlasCastLinearMapper { + using SrcMapper = BlasLinearMapper<const SrcScalar, Index, Unaligned>; + + public: + EIGEN_ALWAYS_INLINE explicit BlasCastLinearMapper(const SrcMapper& src) : m_src(src) {} + + EIGEN_ALWAYS_INLINE void prefetch(Index i) const { m_src.prefetch(i); } + + EIGEN_ALWAYS_INLINE Scalar operator()(Index i) const { return cast<SrcScalar, Scalar>(m_src(i)); } + + template <typename PacketType> + EIGEN_ALWAYS_INLINE PacketType loadPacket(Index i) const { + return cast_ploadu_impl<SrcScalar, PacketType>::run(&m_src(i)); + } + + private: + SrcMapper m_src; +}; + +// Read-only counterpart of const_blas_data_mapper over SrcScalar storage that reads each coefficient as Scalar. It +// provides what the GEMM packers read through, so that they pack A.cast<Scalar>() straight from A. It holds the +// SrcScalar mapper rather than deriving from it, so that a packer cannot reach the unconverted loads of the base. +template <typename Scalar, typename SrcScalar, typename Index, int StorageOrder> +class blas_cast_data_mapper { + using SrcMapper = const_blas_data_mapper<SrcScalar, Index, StorageOrder>; + + public: + using LinearMapper = BlasCastLinearMapper<Scalar, SrcScalar, Index>; + using SubMapper = blas_cast_data_mapper; + + EIGEN_ALWAYS_INLINE blas_cast_data_mapper(const SrcScalar* data, Index stride) : m_src(data, stride) {} + EIGEN_ALWAYS_INLINE explicit blas_cast_data_mapper(const SrcMapper& src) : m_src(src) {} + + EIGEN_ALWAYS_INLINE SubMapper getSubMapper(Index i, Index j) const { return SubMapper(m_src.getSubMapper(i, j)); } + + EIGEN_ALWAYS_INLINE LinearMapper getLinearMapper(Index i, Index j) const { + return LinearMapper(m_src.getLinearMapper(i, j)); + } + + EIGEN_ALWAYS_INLINE void prefetch(Index i, Index j) const { m_src.prefetch(i, j); } + + EIGEN_ALWAYS_INLINE Scalar operator()(Index i, Index j) const { return cast<SrcScalar, Scalar>(m_src(i, j)); } + + template <typename PacketType> + EIGEN_ALWAYS_INLINE PacketType loadPacket(Index i, Index j) const { + return cast_ploadu_impl<SrcScalar, PacketType>::run(&m_src(i, j)); + } + + EIGEN_ALWAYS_INLINE Index stride() const { return m_src.stride(); } + + private: + SrcMapper m_src; +}; + +// The mapper the GEMM packers read an operand stored as StorageScalar through, as Scalar. +template <typename Scalar, typename StorageScalar, typename Index, int StorageOrder> +using gemm_operand_mapper = + std::conditional_t<std::is_same<Scalar, StorageScalar>::value, const_blas_data_mapper<Scalar, Index, StorageOrder>, + blas_cast_data_mapper<Scalar, StorageScalar, Index, StorageOrder>>; + /* Helper class to analyze the factors of a Product expression. * In particular it allows to pop out operator-, scalar multiples, * and conjugate */ @@ -493,7 +615,7 @@ enum { IsComplex = NumTraits<Scalar>::IsComplex, NeedToConjugate = Base::NeedToConjugate ? 0 : IsComplex }; EIGEN_DEVICE_FUNC static inline ExtractType extract(const XprType& x) { return Base::extract(x.nestedExpression()); } EIGEN_DEVICE_FUNC static inline Scalar extractScalarFactor(const XprType& x) { - return conj(Base::extractScalarFactor(x.nestedExpression())); + return numext::conj(Base::extractScalarFactor(x.nestedExpression())); } };
diff --git a/Eigen/src/Core/util/DisableStupidWarnings.h b/Eigen/src/Core/util/DisableStupidWarnings.h index 8091ad4..b993f2d 100644 --- a/Eigen/src/Core/util/DisableStupidWarnings.h +++ b/Eigen/src/Core/util/DisableStupidWarnings.h
@@ -11,7 +11,6 @@ // 4181 - qualifier applied to reference type ignored // 4211 - nonstandard extension used : redefined extern to static // 4244 - 'argument' : conversion from 'type1' to 'type2', possible loss of data -// 4273 - QtAlignedMalloc, inconsistent DLL linkage // 4324 - structure was padded due to declspec(align()) // 4503 - decorated name length exceeded, name was truncated // 4512 - assignment operator could not be generated @@ -21,7 +20,7 @@ // 4717 - 'function' : recursive on all control paths, function will cause runtime stack overflow // 4800 - 'type' : forcing value to bool 'true' or 'false' (performance warning) #pragma warning(push) -#pragma warning(disable : 4100 4101 4127 4181 4211 4244 4273 4324 4503 4512 4522 4700 4714 4717 4800) +#pragma warning(disable : 4100 4101 4127 4181 4211 4244 4324 4503 4512 4522 4700 4714 4717 4800) // We currently rely on has_denorm in tests, and need it defined correctly for half/bfloat16. #ifndef _SILENCE_CXX23_DENORM_DEPRECATION_WARNING #define EIGEN_REENABLE_CXX23_DENORM_DEPRECATION_WARNING 1
diff --git a/Eigen/src/Core/util/Macros.h b/Eigen/src/Core/util/Macros.h index 474a185..1084f6f 100644 --- a/Eigen/src/Core/util/Macros.h +++ b/Eigen/src/Core/util/Macros.h
@@ -565,12 +565,14 @@ #error "NVCC as the target platform for HIPCC is currently not supported." #endif -#if defined(__CUDACC__) && !defined(EIGEN_NO_CUDA) && !defined(__SYCL_DEVICE_ONLY__) +// SYCL CUDA kernels still need CUDA scalar math unless Eigen's SYCL backend is selected. +#if defined(__CUDACC__) && !defined(EIGEN_NO_CUDA) && (!defined(__SYCL_DEVICE_ONLY__) || !defined(EIGEN_USE_SYCL)) // Means the compiler is either nvcc or clang with CUDA enabled #define EIGEN_CUDACC __CUDACC__ #endif -#if defined(__CUDA_ARCH__) && !defined(EIGEN_NO_CUDA) && !defined(__SYCL_DEVICE_ONLY__) +#if defined(__CUDA_ARCH__) && !defined(EIGEN_NO_CUDA) && \ + (!defined(__SYCL_DEVICE_ONLY__) || (defined(__CUDACC__) && !defined(EIGEN_USE_SYCL))) // Means we are generating code for the device #define EIGEN_CUDA_ARCH __CUDA_ARCH__ #endif @@ -587,12 +589,10 @@ #error "Eigen requires CUDA 11.8 or later." #endif -// Native FP16 packet math intrinsics (e.g. __hfma2, h2exp, h2log) are only -// declared by the CUDA headers when __CUDA_ARCH__ >= 530. Guard the device -// pass with a clear error rather than surfacing as "identifier `__hfma2` is -// undefined" deep inside PacketMath.h. -#if defined(EIGEN_CUDA_ARCH) && EIGEN_CUDA_ARCH < 600 -#error "Eigen requires CUDA compute capability >= 6.0 (sm_60). Compile with -arch=sm_60 or higher." +// Guard the CUDA device pass with a clear error when targeting architectures +// older than Maxwell (sm_50). +#if defined(EIGEN_CUDA_ARCH) && EIGEN_CUDA_ARCH < 500 +#error "Eigen requires CUDA compute capability >= 5.0 (sm_50). Compile with -arch=sm_50 or higher." #endif #if defined(__HIPCC__) && !defined(EIGEN_NO_HIP) && !defined(__SYCL_DEVICE_ONLY__) @@ -725,10 +725,12 @@ #endif #if defined(EIGEN_USE_SYCL) && defined(__SYCL_DEVICE_ONLY__) -// EIGEN_USE_SYCL is a user-defined macro while __SYCL_DEVICE_ONLY__ is a compiler-defined macro. -// In most cases we want to check if both macros are defined which can be done using the define below. +// Eigen's SYCL dispatch flag. SYCL_DEVICE_ONLY is kept for users, but AdaptiveCpp's headers also define it. +#define EIGEN_SYCL_DEVICE_ONLY +#ifndef SYCL_DEVICE_ONLY #define SYCL_DEVICE_ONLY #endif +#endif // Under fast-math flags (-ffinite-math-only in particular), clang attaches the // `nofpclass(nan inf)` attribute to every function argument and return value of floating-point @@ -739,7 +741,7 @@ // store/load pair where it is materialized. It uses a memory operand so that it works for any // packet type, including aggregates, and expands to nothing outside affected builds. #if EIGEN_COMP_CLANG && defined(__FINITE_MATH_ONLY__) && __FINITE_MATH_ONLY__ && !defined(EIGEN_GPU_COMPILE_PHASE) && \ - !defined(SYCL_DEVICE_ONLY) + !defined(__SYCL_DEVICE_ONLY__) #define EIGEN_FAST_MATH_CONSTANT_BARRIER(X) __asm__("" : "+m"(X)) #else #define EIGEN_FAST_MATH_CONSTANT_BARRIER(X) @@ -940,7 +942,7 @@ // adds attributes to maximize inlining. This should only be used when really necessary: in particular, the // __attribute__((always_inline)) used on GCC is often unnecessary and can severely harm compile times. #ifndef EIGEN_ALWAYS_INLINE -#if EIGEN_COMP_GNUC && !defined(SYCL_DEVICE_ONLY) +#if EIGEN_COMP_GNUC && !defined(EIGEN_SYCL_DEVICE_ONLY) #define EIGEN_ALWAYS_INLINE __attribute__((always_inline)) inline #else #define EIGEN_ALWAYS_INLINE EIGEN_STRONG_INLINE @@ -953,7 +955,7 @@ // (those with auto parameters), so we leave it empty and rely on the // optimizer to inline small lambda bodies at /O2. #ifndef EIGEN_LAMBDA_ALWAYS_INLINE -#if EIGEN_COMP_GNUC && !defined(SYCL_DEVICE_ONLY) +#if EIGEN_COMP_GNUC && !defined(EIGEN_SYCL_DEVICE_ONLY) #define EIGEN_LAMBDA_ALWAYS_INLINE __attribute__((always_inline)) #else #define EIGEN_LAMBDA_ALWAYS_INLINE @@ -979,7 +981,7 @@ // GPU stuff // Disable some features when compiling with GPU compilers (SYCL/HIPCC) -#if defined(SYCL_DEVICE_ONLY) || defined(EIGEN_HIP_DEVICE_COMPILE) +#if defined(__SYCL_DEVICE_ONLY__) || defined(EIGEN_HIP_DEVICE_COMPILE) // Do not try asserts on device code #ifndef EIGEN_NO_DEBUG #define EIGEN_NO_DEBUG @@ -991,13 +993,13 @@ #endif // No exceptions on device. -#if defined(SYCL_DEVICE_ONLY) || defined(EIGEN_GPU_COMPILE_PHASE) +#if defined(__SYCL_DEVICE_ONLY__) || defined(EIGEN_GPU_COMPILE_PHASE) #ifdef EIGEN_EXCEPTIONS #undef EIGEN_EXCEPTIONS #endif #endif -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) && !defined(EIGEN_GPUCC) #ifndef EIGEN_DONT_VECTORIZE #define EIGEN_DONT_VECTORIZE #endif @@ -1129,6 +1131,7 @@ // x: (SSE) Any SSE register. // (AArch64) Like w, but restricted to registers 0 to 15 inclusive. // v: (PowerPC) An Altivec vector register. +// (x86) Any EVEX-encodable SSE register, including registers 16-31. // wa:(PowerPC) A VSX register. // // "X" (uppercase) should work for all cases, though this seems to fail for @@ -1147,9 +1150,48 @@ // gcc for ppc64le, // both vectors and scalars. // -// Note that this is restricted to plain types - this will not work -// directly for std::complex<T>, Eigen::half, Eigen::bfloat16. For these, -// you will need to apply to the underlying POD type. +// GCC (including MinGW) and Clang on x86, Arm and PPC pass class, union, +// complex and array operands through memory. Elsewhere, apply the barrier to +// the underlying plain type of std::complex<T>, Eigen::half or Eigen::bfloat16. +#if (EIGEN_COMP_GNUC_STRICT || EIGEN_COMP_CLANG || EIGEN_COMP_MINGW) && \ + (EIGEN_ARCH_PPC || EIGEN_ARCH_ARM_OR_ARM64 || EIGEN_ARCH_i386_OR_x86_64) +namespace Eigen { +namespace internal { +// Macros.h precedes Meta.h and <type_traits>. Operands are classified without cv-qualifiers: __builtin_classify_type +// does not evaluate its operand, but GCC rejects a volatile one in a constant expression. +template <typename T> +struct optimization_barrier_unqualified { + using type = T; +}; +template <typename T> +struct optimization_barrier_unqualified<const T> { + using type = T; +}; +template <typename T> +struct optimization_barrier_unqualified<volatile T> { + using type = T; +}; +template <typename T> +struct optimization_barrier_unqualified<const volatile T> { + using type = T; +}; +// Aggregates go to memory. GCC rejects register and general constraints for a struct of three doubles or a double[8] +// on every target, and for an int[4] or a struct of two doubles on 32-bit Arm; on PPC, complex<float> comes back +// corrupted at -O3 even with "m" added as an alternative. A memory operand hides the object's own bytes, not the +// storage it points to. +template <typename T> +struct optimization_barrier_in_memory { + // 9 = complex, 12 = class, 13 = union. Arrays decay to pointers here and are matched below. + static constexpr int kTypeClass = __builtin_classify_type(*static_cast<T*>(nullptr)); + static constexpr bool value = kTypeClass == 9 || kTypeClass == 12 || kTypeClass == 13; +}; +template <typename T, decltype(sizeof(0)) N> +struct optimization_barrier_in_memory<T[N]> { + static constexpr bool value = true; +}; +} // namespace internal +} // namespace Eigen +#endif #if EIGEN_ARCH_PPC && EIGEN_COMP_GNUC_STRICT // These register alternatives are broken on Clang. Packet4f is loaded into a single register rather than a vector, // zeroing out some entries, and integer types generate a compile error. @@ -1162,9 +1204,8 @@ #endif namespace Eigen { namespace internal { -// Complex, class, and union operands go to memory: a class that must stay in memory is a compile error with the -// register constraints, and complex<float> comes back corrupted at -O3, even with "m" added as an alternative. The two -// constraints need separate instantiations because GCC rejects an impossible constraint even on a branch not taken. +// The two constraints need separate instantiations because GCC rejects an impossible constraint even on a branch not +// taken. template <bool InMemory> struct ppc_optimization_barrier_impl { template <typename T> @@ -1181,9 +1222,8 @@ }; template <typename T> EIGEN_ALWAYS_INLINE void ppc_optimization_barrier(T& x) { - // __builtin_classify_type does not evaluate its operand: 9 = complex, 12 = class, 13 = union. - constexpr int kTypeClass = __builtin_classify_type(*static_cast<T*>(nullptr)); - ppc_optimization_barrier_impl<kTypeClass == 9 || kTypeClass == 12 || kTypeClass == 13>::run(x); + using Operand = typename optimization_barrier_unqualified<T>::type; + ppc_optimization_barrier_impl<optimization_barrier_in_memory<Operand>::value>::run(x); } } // namespace internal } // namespace Eigen @@ -1193,12 +1233,121 @@ // Clang's PPC backend does not accept one register constraint covering all scalar and vector operands. In particular, // "wa" crashes the backend for scalar integers narrower than 64 bits. #define EIGEN_OPTIMIZATION_BARRIER(X) __asm__("" : "+m"(X)); +#elif (EIGEN_COMP_GNUC_STRICT || EIGEN_COMP_CLANG || EIGEN_COMP_MINGW) && \ + (EIGEN_ARCH_ARM_OR_ARM64 || EIGEN_ARCH_i386_OR_x86_64) +// Clang chooses memory for mixed general/vector constraints. Use a register-only constraint for float, double, +// and builtin vectors that fit in one enabled register, and memory for aggregates. Other operands retain the +// general/memory alternatives. +#if EIGEN_ARCH_ARM_OR_ARM64 && defined(__ARM_FP) +#define EIGEN_BARRIER_GENERAL_CONSTRAINT "+g,w" +#define EIGEN_BARRIER_REGISTER_CONSTRAINT "+w" +#elif EIGEN_ARCH_ARM_OR_ARM64 +// Arm without VFP or NEON: "w" will not compile, and no operand takes the register path. +#define EIGEN_BARRIER_GENERAL_CONSTRAINT "+g" +#define EIGEN_BARRIER_REGISTER_CONSTRAINT "+g" +#else +#define EIGEN_BARRIER_GENERAL_CONSTRAINT "+g,x" +#ifdef __AVX512F__ +// Unlike "x", "v" also allows registers 16-31. +#define EIGEN_BARRIER_REGISTER_CONSTRAINT "+v" +#else +#define EIGEN_BARRIER_REGISTER_CONSTRAINT "+x" +#endif +#endif +#if EIGEN_ARCH_ARM_OR_ARM64 && defined(__ARM_NEON) +#define EIGEN_BARRIER_VECTOR_BYTES 16 +#elif EIGEN_ARCH_i386_OR_x86_64 && defined(__AVX512F__) +#define EIGEN_BARRIER_VECTOR_BYTES 64 +#elif EIGEN_ARCH_i386_OR_x86_64 && defined(__AVX__) +#define EIGEN_BARRIER_VECTOR_BYTES 32 +#elif EIGEN_ARCH_i386_OR_x86_64 && defined(__SSE2__) +#define EIGEN_BARRIER_VECTOR_BYTES 16 +#else +#define EIGEN_BARRIER_VECTOR_BYTES 0 +#endif +// The SME kernels use the barrier in streaming functions with ZA state. GCC refuses to inline a helper without these +// attributes there, and Clang 23 calls it out of line, switching streaming mode and lazily saving ZA around each call. +// Clang 20 and later have __arm_agnostic; for GCC, __arm_streaming_compatible suffices. +#if EIGEN_ARCH_ARM64 && defined(__ARM_FEATURE_SME) +#define EIGEN_BARRIER_SME_ATTRIBUTES __arm_streaming_compatible +#if EIGEN_COMP_CLANG +#if !__is_identifier(__arm_agnostic) +#undef EIGEN_BARRIER_SME_ATTRIBUTES +#define EIGEN_BARRIER_SME_ATTRIBUTES __arm_streaming_compatible __arm_agnostic("sme_za_state") +#endif +#endif +#else +#define EIGEN_BARRIER_SME_ATTRIBUTES +#endif +namespace Eigen { +namespace internal { +// GCC before 14 classifies builtin vectors as -1; newer GCC and Clang use 19. nullptr_t also has category -1 and is +// excluded below. Scalars have other categories. Sizeless SVE types have no sizeof and keep the general constraint. +template <typename T, typename = void> +struct optimization_barrier_in_register { + static constexpr bool value = false; +}; +template <typename T> +struct optimization_barrier_in_register<T, decltype(void(sizeof(T)))> { + static constexpr int kTypeClass = __builtin_classify_type(*static_cast<T*>(nullptr)); + static constexpr bool value = (kTypeClass == -1 || kTypeClass == 19) && sizeof(T) <= EIGEN_BARRIER_VECTOR_BYTES; +}; +template <> +struct optimization_barrier_in_register<decltype(nullptr)> { + static constexpr bool value = false; +}; +#if (EIGEN_ARCH_ARM_OR_ARM64 && defined(__ARM_FP) && (__ARM_FP & 4)) || (EIGEN_ARCH_i386_OR_x86_64 && defined(__SSE2__)) +template <> +struct optimization_barrier_in_register<float> { + static constexpr bool value = true; +}; +#endif +#if (EIGEN_ARCH_ARM_OR_ARM64 && defined(__ARM_FP) && (__ARM_FP & 8)) || (EIGEN_ARCH_i386_OR_x86_64 && defined(__SSE2__)) +template <> +struct optimization_barrier_in_register<double> { + static constexpr bool value = true; +}; +#endif +enum class optimization_barrier_operand_kind { General, Register, Memory }; +template <optimization_barrier_operand_kind Kind> +struct optimization_barrier_impl { + template <typename T> + static EIGEN_ALWAYS_INLINE void run(T& x) EIGEN_BARRIER_SME_ATTRIBUTES { + __asm__("" : EIGEN_BARRIER_GENERAL_CONSTRAINT(x)); + } +}; +template <> +struct optimization_barrier_impl<optimization_barrier_operand_kind::Register> { + template <typename T> + static EIGEN_ALWAYS_INLINE void run(T& x) EIGEN_BARRIER_SME_ATTRIBUTES { + __asm__("" : EIGEN_BARRIER_REGISTER_CONSTRAINT(x)); + } +}; +template <> +struct optimization_barrier_impl<optimization_barrier_operand_kind::Memory> { + template <typename T> + static EIGEN_ALWAYS_INLINE void run(T& x) EIGEN_BARRIER_SME_ATTRIBUTES { + __asm__("" : "+m"(x)); + } +}; +template <typename T> +EIGEN_ALWAYS_INLINE void optimization_barrier(T& x) EIGEN_BARRIER_SME_ATTRIBUTES { + using Operand = typename optimization_barrier_unqualified<T>::type; + constexpr optimization_barrier_operand_kind kKind = + optimization_barrier_in_memory<Operand>::value ? optimization_barrier_operand_kind::Memory + : optimization_barrier_in_register<Operand>::value ? optimization_barrier_operand_kind::Register + : optimization_barrier_operand_kind::General; + optimization_barrier_impl<kKind>::run(x); +} +} // namespace internal +} // namespace Eigen +#undef EIGEN_BARRIER_GENERAL_CONSTRAINT +#undef EIGEN_BARRIER_REGISTER_CONSTRAINT +#undef EIGEN_BARRIER_VECTOR_BYTES +#undef EIGEN_BARRIER_SME_ATTRIBUTES +#define EIGEN_OPTIMIZATION_BARRIER(X) Eigen::internal::optimization_barrier(X); #elif EIGEN_ARCH_ARM_OR_ARM64 #ifdef __ARM_FP -// General, VFP or NEON. -// Clang doesn't like "r", -// error: non-trivial scalar-to-vector conversion, possible invalid -// constraint for vector typ #define EIGEN_OPTIMIZATION_BARRIER(X) __asm__("" : "+g,w"(X)); #else // Arm without VFP or NEON. @@ -1222,12 +1371,35 @@ #elif EIGEN_ARCH_i386_OR_x86_64 // General, SSE. #define EIGEN_OPTIMIZATION_BARRIER(X) __asm__("" : "+g,x"(X)); -#elif !defined(SYCL_DEVICE_ONLY) +#elif !defined(__SYCL_DEVICE_ONLY__) // Other architectures: a memory operand needs no target-specific register class. #define EIGEN_OPTIMIZATION_BARRIER(X) __asm__("" : "+m"(X)); #else #define EIGEN_OPTIMIZATION_BARRIER(X) #endif +#elif EIGEN_COMP_MSVC && defined(_M_FP_FAST) +// MSVC has no inline asm on x64 or Arm64, and /fp:fast folds across an empty barrier, e.g. x - q*c1 - q*c2 into +// x - q*(c1 + c2). Reloading x from an address OR-ed with a volatile zero hides its value from the optimizer at the +// cost of a store and a store-forwarded load. +namespace Eigen { +namespace internal { +template <typename Dummy = void> +struct msvc_optimization_barrier_zero { + static volatile decltype(sizeof(0)) value; +}; +template <typename Dummy> +volatile decltype(sizeof(0)) msvc_optimization_barrier_zero<Dummy>::value = 0; +template <typename T> +EIGEN_ALWAYS_INLINE void msvc_optimization_barrier(T& x) { + x = *reinterpret_cast<T*>(reinterpret_cast<decltype(sizeof(0))>(&x) | msvc_optimization_barrier_zero<>::value); +} +template <typename T, decltype(sizeof(0)) N> +EIGEN_ALWAYS_INLINE void msvc_optimization_barrier(T (&x)[N]) { + for (auto& element : x) msvc_optimization_barrier(element); +} +} // namespace internal +} // namespace Eigen +#define EIGEN_OPTIMIZATION_BARRIER(X) Eigen::internal::msvc_optimization_barrier(X); #else // Not implemented for other compilers. #define EIGEN_OPTIMIZATION_BARRIER(X) @@ -1238,17 +1410,16 @@ namespace internal { // A volatile asm that the optimizer must execute exactly where it is written, so a value passed through it is defined // only on its branch, and an operation that consumes it stays there instead of being evaluated speculatively. -// EIGEN_OPTIMIZATION_BARRIER is not volatile, and its "+g,w" and "+g,x" alternatives put a scalar float or double in -// memory on clang; the float and double overloads keep the value in a register. +// EIGEN_OPTIMIZATION_BARRIER is not volatile and can move across branches. template <typename T> EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE void volatile_barrier(T& x) { EIGEN_UNUSED_VARIABLE(x); -#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(SYCL_DEVICE_ONLY) && \ +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) && \ (EIGEN_COMP_GNUC || (defined(__clang__) && defined(_MSC_VER))) __asm__ __volatile__("" : "+m"(x)); #endif } -#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(SYCL_DEVICE_ONLY) && \ +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) && \ (EIGEN_COMP_GNUC || (defined(__clang__) && defined(_MSC_VER))) #if EIGEN_ARCH_i386_OR_x86_64 && EIGEN_COMP_NVHPC // nvc++ rejects a long double asm operand under every constraint, so its bytes go through memory instead. @@ -1492,7 +1663,7 @@ #define EIGEN_FINAL final // Wrapping #pragma unroll in a macro since it is required for SYCL -#if defined(SYCL_DEVICE_ONLY) +#if defined(__SYCL_DEVICE_ONLY__) #if defined(_MSC_VER) #define EIGEN_UNROLL_LOOP __pragma(unroll) #else
diff --git a/Eigen/src/Core/util/Memory.h b/Eigen/src/Core/util/Memory.h index 904bf3f..3b5ebb4 100644 --- a/Eigen/src/Core/util/Memory.h +++ b/Eigen/src/Core/util/Memory.h
@@ -474,20 +474,22 @@ // Otherwise, objects that contain internal pointers like mpfr or // AnnoyingScalar can be pointing to the wrong thing. T* result = static_cast<T*>(conditional_aligned_malloc<Align>(sizeof(T) * new_size)); + const std::size_t copy_size = (std::min)(old_size, new_size); + bool tail_constructed = false; EIGEN_TRY { - // Move-construct initial elements. - std::size_t copy_size = (std::min)(old_size, new_size); - move_construct_elements_of_array(result, pts, copy_size); - - // Default-construct remaining elements. - if (new_size > old_size) { - default_construct_elements_of_array(result + copy_size, new_size - old_size); + // A failing default constructor must not leave the source elements moved-from. + if (new_size > copy_size) { + default_construct_elements_of_array(result + copy_size, new_size - copy_size); + tail_constructed = true; } + move_construct_elements_of_array(result, pts, copy_size); // Delete old elements. conditional_aligned_delete<T, Align>(pts, old_size); } EIGEN_CATCH(...) { + // The construction helpers clean up their own partial ranges, but not the completed tail. + if (tail_constructed) destruct_elements_of_array(result + copy_size, new_size - copy_size); conditional_aligned_free<Align>(result); EIGEN_THROW; } @@ -537,13 +539,13 @@ * \param array the address of the start of the array * \param size the size of the array * - * \note If no element of the array is well aligned or the requested alignment is not a multiple of a scalar, - * the size of the array is returned. For example with SSE, the requested alignment is typically 16-bytes. If - * packet size for the given scalar type is 1, then everything is considered well-aligned. + * \note Returns 0 if \a Alignment <= alignof(Scalar). If alignof(Scalar) < \a Alignment <= sizeof(Scalar), all + * elements share the alignment of the first one, so 0 or \a size is returned. Otherwise \a size is returned if no + * element is well aligned or \a Alignment is not a multiple of sizeof(Scalar). * - * \note Otherwise, if the Alignment is larger that the scalar size, we rely on the assumptions that sizeof(Scalar) is a - * power of 2. On the other hand, we do not assume that the array address is a multiple of sizeof(Scalar), as that fails - * for example with Scalar=double on certain 32-bit platforms, see bug #79. + * \note We rely on the assumption that sizeof(Scalar) is a power of 2. On the other hand, we do not assume that the + * array address is a multiple of sizeof(Scalar), as that fails for example with Scalar=double on certain 32-bit + * platforms (see bug #79) or with Scalar=std::complex<double> when alignof(Scalar) < sizeof(Scalar). * * There is also the variant first_aligned(const MatrixBase&) defined in DenseCoeffsBase.h. * \sa first_default_aligned() @@ -554,10 +556,12 @@ constexpr Index AlignmentSize = Alignment / ScalarSize; constexpr Index AlignmentMask = AlignmentSize - 1; - EIGEN_IF_CONSTEXPR (AlignmentSize <= 1) { - // Either the requested alignment if smaller than a scalar, or it exactly match a 1 scalar - // so that all elements of the array have the same alignment. + EIGEN_IF_CONSTEXPR (Alignment <= alignof(Scalar)) { + // A valid Scalar* is already aligned to at least alignof(Scalar). return 0; + } else EIGEN_IF_CONSTEXPR (Alignment <= ScalarSize) { + // Stepping by sizeof(Scalar) preserves address modulo Alignment, so either element 0 is aligned or none is. + return (std::uintptr_t(array) & (Alignment - 1)) ? size : 0; } else EIGEN_IF_CONSTEXPR ((Alignment % ScalarSize) != 0) { // The requested alignment is not a multiple of the scalar size. Consequently, no element of the array is well // aligned. @@ -694,14 +698,51 @@ #endif } -/** \internal Like aligned_new, at the temporaries' alignment (EIGEN_STACK_ALIGN_BYTES); release with scratch_delete. */ +/** \internal Allocates size bytes without constructing T; release with scratch_free<T>. */ +template <typename T> +EIGEN_DEVICE_FUNC inline T* scratch_malloc(std::size_t size) { + EIGEN_IF_CONSTEXPR (alignof(T) <= EIGEN_STACK_ALIGN_BYTES || + (EIGEN_STACK_ALIGN_BYTES == 0 && alignof(T) <= alignof(std::max_align_t))) { + return static_cast<T*>(scratch_malloc(size)); + } else { + // Unlike the byte-offset allocator, this also supports alignments greater than 256. + constexpr std::size_t overhead = alignof(T) - 1 + sizeof(void*); + size += overhead; + if (size < overhead) throw_std_bad_alloc(); + check_that_malloc_is_allowed(); + void* original = std::malloc(size); + if (!original) throw_std_bad_alloc(); + // std::align is host-only with CUDA. + char* start = static_cast<char*>(original) + sizeof(void*); + const std::uintptr_t offset = reinterpret_cast<std::uintptr_t>(start) % alignof(T); + const std::uintptr_t shift = offset > 0 ? alignof(T) - offset : 0; + void* aligned = start + shift; + std::memcpy(static_cast<char*>(aligned) - sizeof(void*), reinterpret_cast<const void*>(&original), sizeof(void*)); + return static_cast<T*>(aligned); + } +} + +template <typename T> +EIGEN_DEVICE_FUNC inline void scratch_free(void* ptr) { + EIGEN_IF_CONSTEXPR (alignof(T) <= EIGEN_STACK_ALIGN_BYTES || + (EIGEN_STACK_ALIGN_BYTES == 0 && alignof(T) <= alignof(std::max_align_t))) { + scratch_free(ptr); + } else if (ptr) { + void* original; + std::memcpy(reinterpret_cast<void*>(&original), static_cast<char*>(ptr) - sizeof(void*), sizeof(void*)); + check_that_free_is_allowed(); + std::free(original); + } +} + +/** \internal Like aligned_new, respecting both type and scratch alignment; release with scratch_delete. */ template <typename T> EIGEN_DEVICE_FUNC inline T* scratch_new(std::size_t size) { check_size_for_overflow<T>(size); - T* result = static_cast<T*>(scratch_malloc(sizeof(T) * size)); + T* result = scratch_malloc<T>(sizeof(T) * size); EIGEN_TRY { default_construct_elements_of_array(result, size); } EIGEN_CATCH(...) { - scratch_free(result); + scratch_free<T>(result); EIGEN_THROW; } return result; @@ -711,7 +752,7 @@ template <typename T> EIGEN_DEVICE_FUNC inline void scratch_delete(T* ptr, std::size_t size) { destruct_elements_of_array<T>(ptr, size); - scratch_free(ptr); + scratch_free<T>(ptr); } // This helper class construct the allocated memory, and takes care of destructing and freeing the handled data @@ -738,7 +779,7 @@ EIGEN_IF_CONSTEXPR (NumTraits<T>::RequireInitialization) { if (m_ptr) Eigen::internal::destruct_elements_of_array<T>(m_ptr, m_size); } - if (m_deallocate) Eigen::internal::scratch_free(m_ptr); + if (m_deallocate) Eigen::internal::scratch_free<T>(m_ptr); } protected: @@ -772,8 +813,8 @@ ObjectType object; EIGEN_DEVICE_FUNC local_nested_eval_wrapper(const Xpr& xpr, Scalar* ptr) - : object(ptr == 0 ? reinterpret_cast<Scalar*>(Eigen::internal::aligned_malloc(sizeof(Scalar) * xpr.size())) : ptr, - xpr.rows(), xpr.cols()), + : object(ptr == 0 ? Eigen::internal::scratch_malloc<Scalar>(sizeof(Scalar) * xpr.size()) : ptr, xpr.rows(), + xpr.cols()), m_deallocate(ptr == 0) { EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::RequireInitialization) { if (object.data()) Eigen::internal::default_construct_elements_of_array(object.data(), object.size()); @@ -785,7 +826,7 @@ EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::RequireInitialization) { if (object.data()) Eigen::internal::destruct_elements_of_array(object.data(), object.size()); } - if (m_deallocate) Eigen::internal::aligned_free(object.data()); + if (m_deallocate) Eigen::internal::scratch_free<Scalar>(object.data()); } private: @@ -799,7 +840,9 @@ /** \internal * * The macro ei_declare_aligned_stack_constructed_variable(TYPE,NAME,SIZE,BUFFER) declares, allocates, - * and construct an aligned buffer named NAME of SIZE elements of type TYPE on the stack + * and construct an aligned buffer named NAME of SIZE elements of type TYPE on the stack. + * Allocated storage respects both alignof(TYPE) and EIGEN_STACK_ALIGN_BYTES, even when SIMD alignment is disabled. + * Storage is allocated on the stack * if the size in bytes is smaller than EIGEN_STACK_ALLOCATION_LIMIT, and if stack allocation is supported by the * platform (currently, this is Linux, OSX and Visual Studio only). Otherwise the memory is allocated on the heap; * in particular, a zero EIGEN_STACK_ALLOCATION_LIMIT disables stack allocation and every buffer is heap-allocated. The @@ -843,32 +886,46 @@ #define EIGEN_ALIGNED_ALLOCA(SIZE) EIGEN_ALLOCA(SIZE) #endif -#define ei_declare_aligned_stack_constructed_variable(TYPE, NAME, SIZE, BUFFER) \ - Eigen::internal::check_size_for_overflow<TYPE>(SIZE); \ - TYPE* NAME = (BUFFER) != 0 ? (BUFFER) \ - : reinterpret_cast<TYPE*>((sizeof(TYPE) * (SIZE) <= EIGEN_STACK_ALLOCATION_LIMIT) \ - ? EIGEN_ALIGNED_ALLOCA(sizeof(TYPE) * (SIZE)) \ - : Eigen::internal::scratch_malloc(sizeof(TYPE) * (SIZE))); \ - Eigen::internal::aligned_stack_memory_handler<TYPE> EIGEN_CAT(NAME, _stack_memory_destructor)( \ - (BUFFER) == 0 ? NAME : 0, SIZE, sizeof(TYPE) * (SIZE) > EIGEN_STACK_ALLOCATION_LIMIT) +// Preserve the existing allocation path unless the scalar needs stronger alignment. +#define EIGEN_TYPED_ALIGNED_ALLOCA(TYPE, SIZE) \ + (alignof(TYPE) <= EIGEN_STACK_ALIGN_BYTES \ + ? EIGEN_ALIGNED_ALLOCA(SIZE) \ + : reinterpret_cast<void*>( \ + (reinterpret_cast<std::uintptr_t>(EIGEN_ALLOCA((SIZE) + alignof(TYPE) - 1)) + alignof(TYPE) - 1) & \ + ~(std::uintptr_t(alignof(TYPE)) - 1))) -#define ei_declare_local_nested_eval(XPR_T, XPR, N, NAME) \ - Eigen::internal::local_nested_eval_wrapper<XPR_T, N> EIGEN_CAT(NAME, _wrapper)( \ - XPR, reinterpret_cast<typename XPR_T::Scalar*>( \ - ((Eigen::internal::local_nested_eval_wrapper<XPR_T, N>::NeedExternalBuffer) && \ - ((sizeof(typename XPR_T::Scalar) * XPR.size()) <= EIGEN_STACK_ALLOCATION_LIMIT)) \ - ? EIGEN_ALIGNED_ALLOCA(sizeof(typename XPR_T::Scalar) * XPR.size()) \ - : 0)); \ +// TYPE and NAME form declarations, not expressions. +// NOLINTBEGIN(bugprone-macro-parentheses) +#define ei_declare_aligned_stack_constructed_variable(TYPE, NAME, SIZE, BUFFER) \ + Eigen::internal::check_size_for_overflow<TYPE>(SIZE); \ + TYPE* NAME = (BUFFER) != 0 \ + ? (BUFFER) \ + : reinterpret_cast<TYPE*>((sizeof(TYPE) * (SIZE) <= EIGEN_STACK_ALLOCATION_LIMIT) \ + ? EIGEN_TYPED_ALIGNED_ALLOCA(TYPE, sizeof(TYPE) * (SIZE)) \ + : Eigen::internal::scratch_malloc<TYPE>(sizeof(TYPE) * (SIZE))); \ + Eigen::internal::aligned_stack_memory_handler<TYPE> EIGEN_CAT(NAME, _stack_memory_destructor)( \ + (BUFFER) == 0 ? NAME : 0, SIZE, sizeof(TYPE) * (SIZE) > EIGEN_STACK_ALLOCATION_LIMIT) +// NOLINTEND(bugprone-macro-parentheses) + +#define ei_declare_local_nested_eval(XPR_T, XPR, N, NAME) \ + Eigen::internal::local_nested_eval_wrapper<XPR_T, N> EIGEN_CAT(NAME, _wrapper)( \ + XPR, reinterpret_cast<typename XPR_T::Scalar*>( \ + ((Eigen::internal::local_nested_eval_wrapper<XPR_T, N>::NeedExternalBuffer) && \ + ((sizeof(typename XPR_T::Scalar) * (XPR).size()) <= EIGEN_STACK_ALLOCATION_LIMIT)) \ + ? EIGEN_TYPED_ALIGNED_ALLOCA(typename XPR_T::Scalar, sizeof(typename XPR_T::Scalar) * (XPR).size()) \ + : 0)); \ typename Eigen::internal::local_nested_eval_wrapper<XPR_T, N>::ObjectType NAME(EIGEN_CAT(NAME, _wrapper).object) #else -#define ei_declare_aligned_stack_constructed_variable(TYPE, NAME, SIZE, BUFFER) \ - Eigen::internal::check_size_for_overflow<TYPE>(SIZE); \ - TYPE* NAME = \ - (BUFFER) != 0 ? BUFFER : reinterpret_cast<TYPE*>(Eigen::internal::scratch_malloc(sizeof(TYPE) * (SIZE))); \ - Eigen::internal::aligned_stack_memory_handler<TYPE> EIGEN_CAT(NAME, _stack_memory_destructor)( \ +// TYPE and NAME form declarations, not expressions. +// NOLINTBEGIN(bugprone-macro-parentheses) +#define ei_declare_aligned_stack_constructed_variable(TYPE, NAME, SIZE, BUFFER) \ + Eigen::internal::check_size_for_overflow<TYPE>(SIZE); \ + TYPE* NAME = (BUFFER) != 0 ? (BUFFER) : Eigen::internal::scratch_malloc<TYPE>(sizeof(TYPE) * (SIZE)); \ + Eigen::internal::aligned_stack_memory_handler<TYPE> EIGEN_CAT(NAME, _stack_memory_destructor)( \ (BUFFER) == 0 ? NAME : 0, SIZE, true) +// NOLINTEND(bugprone-macro-parentheses) #define ei_declare_local_nested_eval(XPR_T, XPR, N, NAME) \ typename Eigen::internal::nested_eval<XPR_T, N>::type NAME(XPR)
diff --git a/Eigen/src/Core/util/ShapeChecks.h b/Eigen/src/Core/util/ShapeChecks.h index d725eef..2c5f8fa 100644 --- a/Eigen/src/Core/util/ShapeChecks.h +++ b/Eigen/src/Core/util/ShapeChecks.h
@@ -10,6 +10,16 @@ namespace Eigen { namespace internal { +/** \internal Compile-time product of two dimensions: Dynamic when either factor is unknown, and also when the + * product would not fit in \c int -- no fixed-size dimension or size that large is usable anyway, and the + * overflowing multiplication would be ill-formed in a constant expression. */ +constexpr int size_at_compile_time(int rows, int cols) { + if (rows == 0 || cols == 0) return 0; + if (rows == Dynamic || cols == Dynamic) return Dynamic; + if (rows > (std::numeric_limits<int>::max)() / cols) return Dynamic; + return rows * cols; +} + template <typename T, typename = void> struct shape_expression { using type = T; @@ -32,17 +42,35 @@ struct shape_cols<T, void_t<decltype(traits<remove_all_t<T>>::ColsAtCompileTime)>> : std::integral_constant<int, traits<remove_all_t<T>>::ColsAtCompileTime> {}; +template <typename Expr, typename = void> +struct shape_max_dimensions { + static constexpr int Rows = Dynamic; + static constexpr int Cols = Dynamic; +}; + +template <typename Expr> +struct shape_max_dimensions<Expr, void_t<decltype(traits<remove_all_t<Expr>>::MaxRowsAtCompileTime), + decltype(traits<remove_all_t<Expr>>::MaxColsAtCompileTime)>> { + static constexpr int Rows = traits<remove_all_t<Expr>>::MaxRowsAtCompileTime; + static constexpr int Cols = traits<remove_all_t<Expr>>::MaxColsAtCompileTime; +}; + template <typename T> struct compile_time_shape { static constexpr int Rows = shape_rows<T>::value; static constexpr int Cols = shape_cols<T>::value; static constexpr int Size = size_at_compile_time(Rows, Cols); + static constexpr int MaxRows = shape_max_dimensions<T>::Rows; + static constexpr int MaxCols = shape_max_dimensions<T>::Cols; }; -template <int Left, int Right> +template <int Left, int Right, int LeftMax = Dynamic, int RightMax = Dynamic> struct dimension_relation { static constexpr bool Known = Left != Dynamic && Right != Dynamic; - static constexpr bool Compatible = !Known || Left == Right; + // Bounds can disprove equality, but do not establish the actual dimensions. + static constexpr bool Compatible = (!Known || Left == Right) && + (Left == Dynamic || RightMax == Dynamic || Left <= RightMax) && + (Right == Dynamic || LeftMax == Dynamic || Right <= LeftMax); static constexpr bool Proven = Known && Compatible; };
diff --git a/Eigen/src/Core/util/XprHelper.h b/Eigen/src/Core/util/XprHelper.h index d0229af..97d0c0e 100644 --- a/Eigen/src/Core/util/XprHelper.h +++ b/Eigen/src/Core/util/XprHelper.h
@@ -69,6 +69,15 @@ struct valid_indexed_view_overload : bool_constant<!(internal::is_valid_index_type<RowIndices>::value && internal::is_valid_index_type<ColIndices>::value)> {}; +template <typename T, typename = void> +struct is_eigen_expression : std::false_type {}; +// The conversion check may instantiate EigenBase<X> (MSVC 14.29 does), which is a hard error unless +// traits<X>::StorageKind exists. Every class derived from EigenBase<X> has it, so requiring it rejects no expression. +template <typename T> +struct is_eigen_expression<T, void_t<decltype(std::declval<const T&>().derived()), + typename traits<typename shape_expression<T>::type>::StorageKind>> + : std::is_convertible<const remove_all_t<T>*, const EigenBase<typename shape_expression<T>::type>*> {}; + // promote_scalar_arg is an helper used in operation between an expression and a scalar, like: // expression * scalar // Its role is to determine how the type T of the scalar operand should be promoted given the scalar type ExprScalar of @@ -418,16 +427,6 @@ return DirectAccessBit | LvalueBit | NestByRefBit | row_major_bit; } -/** \internal Compile-time product of two dimensions: Dynamic when either factor is unknown, and also when the - * product would not fit in \c int -- no fixed-size dimension or size that large is usable anyway, and the - * overflowing multiplication would be ill-formed in a constant expression. */ -constexpr int size_at_compile_time(int rows, int cols) { - if (rows == 0 || cols == 0) return 0; - if (rows == Dynamic || cols == Dynamic) return Dynamic; - if (rows > (std::numeric_limits<int>::max)() / cols) return Dynamic; - return rows * cols; -} - template <typename XprType> struct size_of_xpr_at_compile_time : std::integral_constant<int, size_at_compile_time(traits<XprType>::RowsAtCompileTime, @@ -1041,6 +1040,20 @@ template<> struct ScalarBinaryOpTraits<U1,U2,internal::scalar_sum_op<U1,U2> > { typedef U1 ReturnType; }; \endcode + * Custom multiplication can opt into commutative product traversals and reductions by adding + * \c IsCommutative to its product specialization: + \code + template<> + struct ScalarBinaryOpTraits<MyScalar, MyScalar, + internal::scalar_product_op<MyScalar, MyScalar>> { + using ReturnType = MyScalar; + static constexpr bool IsCommutative = true; + }; + \endcode + * This promises that reversing the operands preserves the value and result type for all valid inputs. + * For mixed scalar types, opt in for both ordered pairs. Missing metadata keeps custom multiplication on the + * conservative path; built-in arithmetic and standard complex types with arithmetic components are recognized + * automatically. Commutative traversals may reorder and reassociate products, changing floating-point rounding. * * By default, the following generic combinations are supported: <table class="manual"> @@ -1103,6 +1116,136 @@ (Eigen::internal::has_ReturnType<ScalarBinaryOpTraits<LHS, RHS, BINOP>>::value), \ YOU_MIXED_DIFFERENT_NUMERIC_TYPES__YOU_NEED_TO_USE_THE_CAST_METHOD_OF_MATRIXBASE_TO_CAST_NUMERIC_TYPES_EXPLICITLY) +namespace internal { + +// EigenBase alone does not supply the generic scalar operators used here. +template <typename T, typename Derived, typename XprKind> +struct has_generic_scalar_binary_ops + : bool_constant<std::is_same<XprKind, ArrayXpr>::value + ? std::is_convertible<const remove_all_t<T>*, const ArrayBase<Derived>*>::value + : (std::is_convertible<const remove_all_t<T>*, const MatrixBase<Derived>*>::value || + std::is_convertible<const remove_all_t<T>*, const SparseMatrixBase<Derived>*>::value)> {}; + +namespace scalar_pow_adl { +using std::pow; + +template <bool Inspect, typename Lhs, typename Rhs, typename = void> +struct is_supported : std::false_type {}; +template <typename Lhs, typename Rhs> +struct is_supported<true, Lhs, Rhs, + void_t<typename ScalarBinaryOpTraits<Lhs, Rhs, scalar_pow_op<Lhs, Rhs>>::ReturnType, + decltype(pow(std::declval<const Lhs&>(), std::declval<const Rhs&>()))>> + : std::is_convertible<decltype(pow(std::declval<const Lhs&>(), std::declval<const Rhs&>())), + typename ScalarBinaryOpTraits<Lhs, Rhs, scalar_pow_op<Lhs, Rhs>>::ReturnType> {}; +} // namespace scalar_pow_adl + +template <typename Scalar, typename T, template <typename, typename> class BinaryOp, bool ScalarOnLeft> +using scalar_expression_binary_traits = + std::conditional_t<ScalarOnLeft, ScalarBinaryOpTraits<T, Scalar, BinaryOp<T, Scalar>>, + ScalarBinaryOpTraits<Scalar, T, BinaryOp<Scalar, T>>>; + +template <typename Expr, typename T, template <typename, typename> class BinaryOp, bool ScalarOnLeft, typename = void> +struct has_competing_expression_binary_op : std::false_type {}; + +template <typename Expr, typename T, template <typename, typename> class BinaryOp, bool ScalarOnLeft> +struct has_competing_expression_binary_op< + Expr, T, BinaryOp, ScalarOnLeft, + void_t<decltype(std::declval<const T&>().derived()), typename traits<typename shape_expression<T>::type>::Scalar, + typename traits<typename shape_expression<T>::type>::XprKind, + decltype(traits<typename shape_expression<T>::type>::RowsAtCompileTime), + decltype(traits<typename shape_expression<T>::type>::ColsAtCompileTime), + typename scalar_expression_binary_traits<typename traits<Expr>::Scalar, T, BinaryOp, + ScalarOnLeft>::ReturnType>> { + using Other = typename shape_expression<T>::type; + using Scalar = typename traits<Expr>::Scalar; + using XprKind = typename traits<Expr>::XprKind; + using Lhs = std::conditional_t<ScalarOnLeft, Other, Expr>; + using Rhs = std::conditional_t<ScalarOnLeft, Expr, Other>; + using LhsScalar = typename traits<Lhs>::Scalar; + using RhsScalar = typename traits<Rhs>::Scalar; + using LhsShape = compile_time_shape<Lhs>; + using RhsShape = compile_time_shape<Rhs>; + using Op = BinaryOp<LhsScalar, RhsScalar>; + static constexpr bool IsArray = std::is_same<XprKind, ArrayXpr>::value; + static constexpr bool HasExpressionOverload = has_generic_scalar_binary_ops<Expr, Expr, XprKind>::value && + has_generic_scalar_binary_ops<T, Other, XprKind>::value; + static constexpr bool ProductSizeMatches = + dimension_relation<LhsShape::Cols, RhsShape::Rows, LhsShape::MaxCols, RhsShape::MaxRows>::Compatible; + static constexpr bool ArraySizeMatches = + dimension_relation<LhsShape::Rows, RhsShape::Rows, LhsShape::MaxRows, RhsShape::MaxRows>::Compatible && + dimension_relation<LhsShape::Cols, RhsShape::Cols, LhsShape::MaxCols, RhsShape::MaxCols>::Compatible; + using OppositeLhs = std::conditional_t<ScalarOnLeft, typename traits<Other>::Scalar, Expr>; + using OppositeRhs = std::conditional_t<ScalarOnLeft, Expr, typename traits<Other>::Scalar>; + // Preserve a one-sided scalar registration. Only broad traits admitting both + // expression-as-scalar interpretations need to give way to an expression operation. + static constexpr bool HasOppositeScalarOverload = + has_ReturnType<ScalarBinaryOpTraits<OppositeLhs, OppositeRhs, BinaryOp<OppositeLhs, OppositeRhs>>>::value; + // Matrix division has no expression overload; matrix products also require compatible coefficient types. + static constexpr bool Competing = HasExpressionOverload && !is_eigen_expression<Scalar>::value && + std::is_same<XprKind, typename traits<Other>::XprKind>::value && + (IsArray ? ArraySizeMatches : ProductSizeMatches) && + (IsArray || std::is_same<Op, scalar_product_op<LhsScalar, RhsScalar>>::value) && + has_ReturnType<ScalarBinaryOpTraits<LhsScalar, RhsScalar, Op>>::value; + static constexpr bool IsPow = std::is_same<Op, scalar_pow_op<LhsScalar, RhsScalar>>::value; + using ScalarLhs = std::conditional_t<ScalarOnLeft, T, Scalar>; + using ScalarRhs = std::conditional_t<ScalarOnLeft, Scalar, T>; + using ScalarPower = scalar_pow_adl::is_supported<(Competing && IsPow), ScalarLhs, ScalarRhs>; + static constexpr bool value = Competing && (IsPow ? !ScalarPower::value : HasOppositeScalarOverload); +}; + +template <typename T> +using expression_binary_base_t = typename shape_expression<T>::type::StorageBaseType; + +// Preserve base-reference bindings; only concrete operands get a better match +// when broad traits admit both expression-as-scalar interpretations. +template <typename Lhs, typename Rhs, template <typename, typename> class BinaryOp, typename OwnerBase, typename = void> +struct has_ambiguous_expression_binary_op : std::false_type {}; + +template <typename Lhs, typename Rhs, template <typename, typename> class BinaryOp, typename OwnerBase> +struct has_ambiguous_expression_binary_op<Lhs, Rhs, BinaryOp, OwnerBase, + void_t<expression_binary_base_t<Lhs>, expression_binary_base_t<Rhs>>> + : bool_constant< + std::is_same<OwnerBase, expression_binary_base_t<Rhs>>::value && + std::is_convertible<const Rhs*, const OwnerBase*>::value && + !std::is_same<remove_all_t<Lhs>, expression_binary_base_t<Lhs>>::value && + !std::is_same<remove_all_t<Rhs>, expression_binary_base_t<Rhs>>::value && + has_competing_expression_binary_op<typename shape_expression<Lhs>::type, Rhs, BinaryOp, false>::value && + has_competing_expression_binary_op<typename shape_expression<Rhs>::type, Lhs, BinaryOp, true>::value> { + static constexpr bool DenseSparse = + std::is_same<typename traits<typename shape_expression<Lhs>::type>::StorageKind, Dense>::value && + std::is_same<typename traits<typename shape_expression<Rhs>::type>::StorageKind, Sparse>::value; +}; + +template <template <typename, typename> class BinaryOp, bool DenseSparse> +struct expression_binary_op { + template <typename Lhs, typename Rhs> + EIGEN_DEVICE_FUNC static constexpr EIGEN_STRONG_INLINE auto run(const Lhs& lhs, const Rhs& rhs) + -> CwiseBinaryOp<BinaryOp<typename Lhs::Scalar, typename Rhs::Scalar>, const typename shape_expression<Lhs>::type, + const typename shape_expression<Rhs>::type> { + return {lhs.derived(), rhs.derived()}; + } +}; + +template <> +struct expression_binary_op<scalar_product_op, false> { + template <typename Lhs, typename Rhs> + EIGEN_DEVICE_FUNC static constexpr EIGEN_STRONG_INLINE auto run(const Lhs& lhs, const Rhs& rhs) + -> decltype(lhs.operator*(rhs)) { + return lhs.operator*(rhs); + } +}; + +template <> +struct expression_binary_op<scalar_product_op, true> { + template <typename Lhs, typename Rhs> + EIGEN_DEVICE_FUNC static constexpr EIGEN_STRONG_INLINE auto run(const Lhs& lhs, const Rhs& rhs) + -> decltype(operator*(lhs, rhs)) { + return operator*(lhs, rhs); + } +}; + +} // namespace internal + } // end namespace Eigen #endif // EIGEN_XPRHELPER_H
diff --git a/Eigen/src/Eigenvalues/ComplexQZ.h b/Eigen/src/Eigenvalues/ComplexQZ.h index f3a0ff0..d177617 100644 --- a/Eigen/src/Eigenvalues/ComplexQZ.h +++ b/Eigen/src/Eigenvalues/ComplexQZ.h
@@ -242,7 +242,9 @@ return numext::abs(x) <= tol; } - void do_QZ_step(Index p, Index q, unsigned int iter); + void do_QZ_step(Index f, Index l, unsigned int iter); + + Vec3 shiftColumn(Index p, Index l, unsigned int iter) const; JacobiRotation<Scalar> computeZk2(const Row2& b); @@ -455,7 +457,7 @@ if (z >= f) { push_down_zero_ST(z, l); } else { - do_QZ_step(f, m_n - l - 1, local_iter); + do_QZ_step(f, l, local_iter); local_iter++; m_global_iter++; } @@ -476,34 +478,57 @@ return J.transpose(); } +/** \internal First column of the shift polynomial of the block (S, T)(p:l, p:l), up to a nonzero factor. */ template <typename MatrixType_> -void ComplexQZ<MatrixType_>::do_QZ_step(Index p, Index q, unsigned int iter) { - // This is certainly not the most efficient way of doing this, - // but a readable one. +typename ComplexQZ<MatrixType_>::Vec3 ComplexQZ<MatrixType_>::shiftColumn(Index p, Index l, unsigned int iter) const { const auto a = [p, this](Index i, Index j) { return m_S(p + i - 1, p + j - 1); }; const auto b = [p, this](Index i, Index j) { return m_T(p + i - 1, p + j - 1); }; - const Index m = m_n - p - q; // Size of the inner block - Scalar x, y, z; + const Index m = l - p + 1; // Size of the inner block if (iter > 0 && iter % 10 == 0) { // Break stalled double-shift iterations with a single shift displaced by the trailing subdiagonals. const RealScalar displacement = numext::abs(a(m, m - 1) / b(m - 1, m - 1)) + numext::abs(a(m - 1, m - 2) / b(m - 2, m - 2)); const Scalar shift = a(m, m) / b(m, m) + displacement; // (S*T^-1 - shift*I)*e1 has only two nonzero entries. - x = a(1, 1) / b(1, 1) - shift; - y = a(2, 1) / b(1, 1); - z = Scalar(0); - } else { - Scalar W1 = a(m - 1, m - 1) / b(m - 1, m - 1) - a(1, 1) / b(1, 1), W2 = a(m, m) / b(m, m) - a(1, 1) / b(1, 1), - W3 = a(m, m - 1) / b(m - 1, m - 1); - - x = (W1 * W2 - a(m - 1, m) / b(m, m) * W3 + W3 * b(m - 1, m) / b(m, m) * a(1, 1) / b(1, 1)) * b(1, 1) / a(2, 1) + - a(1, 2) / b(2, 2) - a(1, 1) / b(1, 1) * b(1, 2) / b(2, 2); - y = (a(2, 2) / b(2, 2) - a(1, 1) / b(1, 1)) - a(2, 1) / b(1, 1) * b(1, 2) / b(2, 2) - W1 - W2 + - W3 * (b(m - 1, m) / b(m, m)); - z = a(3, 2) / b(2, 2); + return Vec3(a(1, 1) / b(1, 1) - shift, a(2, 1) / b(1, 1), Scalar(0)); } + const Scalar W1 = a(m - 1, m - 1) / b(m - 1, m - 1) - a(1, 1) / b(1, 1), W2 = a(m, m) / b(m, m) - a(1, 1) / b(1, 1), + W3 = a(m, m - 1) / b(m - 1, m - 1); + return Vec3( + (W1 * W2 - a(m - 1, m) / b(m, m) * W3 + W3 * b(m - 1, m) / b(m, m) * a(1, 1) / b(1, 1)) * b(1, 1) / a(2, 1) + + a(1, 2) / b(2, 2) - a(1, 1) / b(1, 1) * b(1, 2) / b(2, 2), + (a(2, 2) / b(2, 2) - a(1, 1) / b(1, 1)) - a(2, 1) / b(1, 1) * b(1, 2) / b(2, 2) - W1 - W2 + + W3 * (b(m - 1, m) / b(m, m)), + a(3, 2) / b(2, 2)); +} + +/** \internal Performs one QZ sweep on the unreduced block (S, T)(f:l, f:l). */ +template <typename MatrixType_> +void ComplexQZ<MatrixType_>::do_QZ_step(Index f, Index l, unsigned int iter) { + // Start at the lowest row p > f where the bulge v spreads S(p, p-1) only into negligible entries (the test of + // EISPACK's hqr and LAPACK's xLAHQR, applied to S, with v scaled to |v(0)| + |v(1)| + |v(2)| = 1; rows where that + // sum is 0 or not finite are skipped): + // |S(p, p-1)| (|v(1)| + |v(2)|) <= eps |v(0)| (|S(p-1, p-1)| + |S(p, p)| + |S(p+1, p+1)|). + // From row f the bulge is proportional to S(f+1, f), and a tiny but not negligible entry, e.g. next to zero + // eigenvalues, can stall the iteration. A block of three rows has no such p, as in xLAHQR. + Index p = f; Vec3 X; + for (Index j = l - 2; j > f; --j) { + X = shiftColumn(j, l, iter); + const RealScalar scale = numext::abs(X(0)) + numext::abs(X(1)) + numext::abs(X(2)); + if (!(numext::isfinite)(scale) || scale == RealScalar(0)) continue; + X /= scale; + const RealScalar diagonal = + numext::abs(m_S(j - 1, j - 1)) + numext::abs(m_S(j, j)) + numext::abs(m_S(j + 1, j + 1)); + if (numext::abs(m_S(j, j - 1)) * (numext::abs(X(1)) + numext::abs(X(2))) <= + NumTraits<RealScalar>::epsilon() * numext::abs(X(0)) * diagonal) { + p = j; + break; + } + } + if (p == f) X = shiftColumn(f, l, iter); + const Index m = l - p + 1; // Size of the inner block + Scalar x = X(0), y = X(1), z = X(2); const PermutationMatrix<3, 3, int> S3(Vector3i(2, 0, 1)); for (Index k = p; k < p + m - 2; k++) { X << x, y, z; @@ -517,6 +542,11 @@ m_S.template middleRows<3>(k) .rightCols((std::min)(m_n, m_n - k + 1)) .applyHouseholderOnTheLeft(ess, tau, m_ws.data()); + if (k > f && k == p) { + // The reflector spread the negligible S(p, p-1) into the entries below it, which are dropped. + m_S.coeffRef(k + 1, k - 1) = Scalar(0); + m_S.coeffRef(k + 2, k - 1) = Scalar(0); + } m_T.template middleRows<3>(k).rightCols(m_n - k).applyHouseholderOnTheLeft(ess, tau, m_ws.data()); if (m_computeQZ) m_Q.template middleCols<3>(k).applyHouseholderOnTheRight(ess, numext::conj(tau), m_ws.data()); @@ -612,19 +642,73 @@ if (m_computeQZ) m_Z.applyOnTheLeft(i, i + 1, G); - Mat2 tildeSi = m_S.template block<2, 2>(i, i), tildeTi = m_T.template block<2, 2>(i, i); - Mat2 C = tildeSi.norm() < (lambda * tildeTi).norm() ? tildeSi : lambda * tildeTi; - G.makeGivens(C(0, 0), C(1, 0)); + // The left rotation from T's first column keeps T triangular; the residual ~eps (|S| + |lambda| |T|) of the null + // vector lands in S(i+1, i), where the refinement below removes it. + if (numext::abs(m_T(i, i)) + numext::abs(m_T(i + 1, i)) > tolT) + G.makeGivens(m_T(i, i), m_T(i + 1, i)); + else + G.makeGivens(m_S(i, i), m_S(i + 1, i)); m_S.applyOnTheLeft(i, i + 1, G.adjoint()); m_T.applyOnTheLeft(i, i + 1, G.adjoint()); if (m_computeQZ) m_Q.applyOnTheRight(i, i + 1, G); } - if (!is_negligible(m_S(i + 1, i), m_normOfS * NumTraits<RealScalar>::epsilon())) { + // Rotations formed from lambda can leave S(i+1, i) well above eps (|S(i, i)| + |S(i+1, i+1)|), e.g. for graded T. + // Refine with single-shift QZ steps. With K = T^-1 S = [a b; c d], the Rayleigh quotient shift d = S(i+1, i+1) / + // T(i+1, i+1) maps |c| to |c|^2 |b| / (|a - d|^2 + |c|^2): quadratic for separated eigenvalues, but not monotone. Up + // to four steps run on a copy of the block. Of the prefixes that pass the final test |S(i+1, i)| <= eps m_normOfS, or + // of all if none does, the one with the smallest |S(i+1, i)| / (|S(i, i)| + |S(i+1, i+1)|) is applied. + const auto relativeSubdiagonal = [](const Mat2& s) { + const RealScalar subdiagonal = numext::abs(s(1, 0)); + const RealScalar diagonal = numext::abs(s(0, 0)) + numext::abs(s(1, 1)); + return subdiagonal <= NumTraits<RealScalar>::epsilon() * diagonal ? RealScalar(0) : subdiagonal / diagonal; + }; + JacobiRotation<Scalar> left[4], right[4]; + Mat2 Sstep = m_S.template block<2, 2>(i, i), Tstep = m_T.template block<2, 2>(i, i); + const RealScalar tolS = m_normOfS * NumTraits<RealScalar>::epsilon(); + RealScalar current = relativeSubdiagonal(Sstep), best = current; + bool bestPasses = is_negligible(Sstep(1, 0), tolS); + int bestSteps = 0; + for (int k = 0; k < 4 && current > RealScalar(0); ++k) { + if (is_negligible(Tstep(1, 1), tolT)) { + // An infinite eigenvalue: deflate it with a rotation of the columns, as in the branch above. + left[k] = JacobiRotation<Scalar>(Scalar(1), Scalar(0)); + right[k].makeGivens(Sstep(1, 1), Sstep(1, 0)); + right[k] = right[k].adjoint(); + } else { + const Scalar shift = Sstep(1, 1) / Tstep(1, 1); + left[k].makeGivens(Sstep(0, 0) - shift * Tstep(0, 0), Sstep(1, 0)); + Sstep.applyOnTheLeft(0, 1, left[k].adjoint()); + Tstep.applyOnTheLeft(0, 1, left[k].adjoint()); + right[k] = computeZk2(Tstep.row(1)); + } + Sstep.applyOnTheRight(0, 1, right[k]); + Tstep.applyOnTheRight(0, 1, right[k]); + Tstep(1, 0) = Scalar(0); + current = relativeSubdiagonal(Sstep); + const bool passes = is_negligible(Sstep(1, 0), tolS); + if ((passes && !bestPasses) || (passes == bestPasses && current < best)) { + best = current; + bestPasses = passes; + bestSteps = k + 1; + } + } + for (int k = 0; k < bestSteps; ++k) { + m_S.rightCols(m_n - i).applyOnTheLeft(i, i + 1, left[k].adjoint()); + m_T.rightCols(m_n - i).applyOnTheLeft(i, i + 1, left[k].adjoint()); + if (m_computeQZ) m_Q.applyOnTheRight(i, i + 1, left[k]); + m_S.topRows(i + 2).applyOnTheRight(i, i + 1, right[k]); + m_T.topRows(i + 2).applyOnTheRight(i, i + 1, right[k]); + m_T(i + 1, i) = Scalar(0); + if (m_computeQZ) m_Z.applyOnTheLeft(i, i + 1, right[k].adjoint()); + } + + if (!is_negligible(m_S(i + 1, i), tolS)) { m_info = ComputationInfo::NumericalIssue; } else { m_S(i + 1, i) = Scalar(0); + m_T(i + 1, i) = Scalar(0); } }
diff --git a/Eigen/src/Eigenvalues/ComplexSchur.h b/Eigen/src/Eigenvalues/ComplexSchur.h index c27d813..acd6afa 100644 --- a/Eigen/src/Eigenvalues/ComplexSchur.h +++ b/Eigen/src/Eigenvalues/ComplexSchur.h
@@ -310,11 +310,10 @@ /** Compute the shift in the current QR iteration. */ template <typename MatrixType> typename ComplexSchur<MatrixType>::ComplexScalar ComplexSchur<MatrixType>::computeShift(Index iu, Index iter) { - using std::abs; if ((iter == 10 || iter == 20) && iu > 1) { // exceptional shift, taken from http://www.netlib.org/eispack/comqr.f - return ComplexSchur<MatrixType>::ComplexScalar(abs(numext::real(m_matT.coeff(iu, iu - 1))) + - abs(numext::real(m_matT.coeff(iu - 1, iu - 2)))); + return ComplexSchur<MatrixType>::ComplexScalar(numext::abs(numext::real(m_matT.coeff(iu, iu - 1))) + + numext::abs(numext::real(m_matT.coeff(iu - 1, iu - 2)))); } // compute the shift as one of the eigenvalues of t, the 2x2 @@ -325,7 +324,7 @@ ComplexScalar b = t.coeff(0, 1) * t.coeff(1, 0); ComplexScalar c = t.coeff(0, 0) - t.coeff(1, 1); - ComplexScalar disc = sqrt(c * c + RealScalar(4) * b); + ComplexScalar disc = numext::sqrt(c * c + RealScalar(4) * b); ComplexScalar det = t.coeff(0, 0) * t.coeff(1, 1) - b; ComplexScalar trace = t.coeff(0, 0) + t.coeff(1, 1); ComplexScalar eival1 = (trace + disc) / RealScalar(2);
diff --git a/Eigen/src/Eigenvalues/EigenSolver.h b/Eigen/src/Eigenvalues/EigenSolver.h index 1d434cd..342663e 100644 --- a/Eigen/src/Eigenvalues/EigenSolver.h +++ b/Eigen/src/Eigenvalues/EigenSolver.h
@@ -405,8 +405,6 @@ * m_realSchur. */ template <typename MatrixType> EigenSolver<MatrixType>& EigenSolver<MatrixType>::computeFromSchur(bool computeEigenvectors) { - using numext::isfinite; - m_info = m_realSchur.info(); if (m_info == Success) { @@ -419,7 +417,7 @@ while (i < n) { if (i == n - 1 || matT.coeff(i + 1, i) == Scalar(0)) { m_eivalues.coeffRef(i) = matT.coeff(i, i); - if (!(isfinite)(m_eivalues.coeffRef(i))) { + if (!(numext::isfinite)(m_eivalues.coeffRef(i))) { m_isInitialized = true; m_eigenvectorsOk = false; m_info = NumericalIssue; @@ -443,7 +441,7 @@ m_eivalues.coeffRef(i) = ComplexScalar(matT.coeff(i + 1, i + 1) + p, z); m_eivalues.coeffRef(i + 1) = ComplexScalar(matT.coeff(i + 1, i + 1) + p, -z); - if (!((isfinite)(m_eivalues.coeffRef(i)) && (isfinite)(m_eivalues.coeffRef(i + 1)))) { + if (!((numext::isfinite)(m_eivalues.coeffRef(i)) && (numext::isfinite)(m_eivalues.coeffRef(i + 1)))) { m_isInitialized = true; m_eigenvectorsOk = false; m_info = NumericalIssue;
diff --git a/Eigen/src/Eigenvalues/GeneralizedEigenSolver.h b/Eigen/src/Eigenvalues/GeneralizedEigenSolver.h index 7dfa043..49c4686 100644 --- a/Eigen/src/Eigenvalues/GeneralizedEigenSolver.h +++ b/Eigen/src/Eigenvalues/GeneralizedEigenSolver.h
@@ -347,7 +347,7 @@ // For singular eigenvalues do nothing more if (numext::abs(m_betas.coeffRef(i)) >= (std::numeric_limits<RealScalar>::min)()) { // Non-singular eigenvalue - const Scalar alpha = real(m_alphas.coeffRef(i)); + const Scalar alpha = numext::real(m_alphas.coeffRef(i)); const Scalar beta = m_betas.coeffRef(i); for (Index j = i - 1; j >= 0; j--) { const Index st = j + 1; @@ -391,7 +391,7 @@ Scalar p = Scalar(0.5) * (S2.coeff(0, 0) - S2.coeff(1, 1)); Scalar z = numext::sqrt(numext::abs(p * p + S2.coeff(1, 0) * S2.coeff(0, 1))); const ComplexScalar alpha = ComplexScalar(S2.coeff(1, 1) + p, (beta > 0) ? z : -z); - m_alphas.coeffRef(i) = conj(alpha); + m_alphas.coeffRef(i) = numext::conj(alpha); m_alphas.coeffRef(i + 1) = alpha; if (computeEigenvectors) {
diff --git a/Eigen/src/Eigenvalues/MatrixBaseEigenvalues.h b/Eigen/src/Eigenvalues/MatrixBaseEigenvalues.h index b452692..cf14154 100644 --- a/Eigen/src/Eigenvalues/MatrixBaseEigenvalues.h +++ b/Eigen/src/Eigenvalues/MatrixBaseEigenvalues.h
@@ -110,10 +110,9 @@ */ template <typename Derived> inline typename MatrixBase<Derived>::RealScalar MatrixBase<Derived>::operatorNorm() const { - using std::sqrt; typename Derived::PlainObject m_eval(derived()); // FIXME: if eigenvalues are guaranteed to be sorted, comparing the first and last is sufficient. - return sqrt((m_eval * m_eval.adjoint()).eval().template selfadjointView<Lower>().eigenvalues().maxCoeff()); + return numext::sqrt((m_eval * m_eval.adjoint()).eval().template selfadjointView<Lower>().eigenvalues().maxCoeff()); } /** \brief Computes the L2 operator norm
diff --git a/Eigen/src/Eigenvalues/RealQZ.h b/Eigen/src/Eigenvalues/RealQZ.h index e44ff26..cf43314 100644 --- a/Eigen/src/Eigenvalues/RealQZ.h +++ b/Eigen/src/Eigenvalues/RealQZ.h
@@ -301,12 +301,11 @@ /** \internal Look for single small sub-diagonal element S(res, res-1) and return res (or 0) */ template <typename MatrixType> inline Index RealQZ<MatrixType>::findSmallSubdiagEntry(Index iu) { - using std::abs; Index res = iu; while (res > 0) { - Scalar s = abs(m_S.coeff(res - 1, res - 1)) + abs(m_S.coeff(res, res)); + Scalar s = numext::abs(m_S.coeff(res - 1, res - 1)) + numext::abs(m_S.coeff(res, res)); if (numext::is_exactly_zero(s)) s = m_normOfS; - if (abs(m_S.coeff(res, res - 1)) < NumTraits<Scalar>::epsilon() * s) break; + if (numext::abs(m_S.coeff(res, res - 1)) < NumTraits<Scalar>::epsilon() * s) break; res--; } return res; @@ -315,10 +314,9 @@ /** \internal Look for single small diagonal element T(res, res) for res between f and l, and return res (or f-1) */ template <typename MatrixType> inline Index RealQZ<MatrixType>::findSmallDiagEntry(Index f, Index l) { - using std::abs; Index res = l; while (res >= f) { - if (abs(m_T.coeff(res, res)) <= NumTraits<Scalar>::epsilon() * m_normOfT) break; + if (numext::abs(m_T.coeff(res, res)) <= NumTraits<Scalar>::epsilon() * m_normOfT) break; res--; } return res; @@ -327,10 +325,8 @@ /** \internal decouple 2x2 diagonal block in rows i, i+1 if eigenvalues are real */ template <typename MatrixType> inline void RealQZ<MatrixType>::splitOffTwoRows(Index i) { - using std::abs; - using std::sqrt; const Index dim = m_S.cols(); - if (numext::is_exactly_zero(abs(m_S.coeff(i + 1, i)))) return; + if (numext::is_exactly_zero(numext::abs(m_S.coeff(i + 1, i)))) return; Index j = findSmallDiagEntry(i, i + 1); if (j == i - 1) { // block of (S T^{-1}) @@ -339,7 +335,7 @@ Scalar p = Scalar(0.5) * (STi(0, 0) - STi(1, 1)); Scalar q = p * p + STi(1, 0) * STi(0, 1); if (q >= 0) { - Scalar z = sqrt(q); + Scalar z = numext::sqrt(q); // one QR-like iteration for ABi - lambda I // is enough - when we know exact eigenvalue in advance, // convergence is immediate @@ -403,7 +399,6 @@ /** \internal QR-like iterative step for block f..l */ template <typename MatrixType> inline void RealQZ<MatrixType>::step(Index f, Index l, Index iter) { - using std::abs; const Index dim = m_S.cols(); // x, y, z @@ -415,7 +410,7 @@ b11i = Scalar(1.0) / m_T.coeff(f + 0, f + 0), b22i = Scalar(1.0) / m_T.coeff(f + 1, f + 1), a87 = m_S.coeff(l - 1, l - 2), a98 = m_S.coeff(l - 0, l - 1), b77i = Scalar(1.0) / m_T.coeff(l - 2, l - 2), b88i = Scalar(1.0) / m_T.coeff(l - 1, l - 1); - Scalar ss = abs(a87 * b77i) + abs(a98 * b88i), lpl = Scalar(1.5) * ss, ll = ss * ss; + Scalar ss = numext::abs(a87 * b77i) + numext::abs(a98 * b88i), lpl = Scalar(1.5) * ss, ll = ss * ss; x = ll + a11 * a11 * b11i * b11i - lpl * a11 * b11i + a12 * a21 * b11i * b22i - a11 * a21 * b12 * b11i * b11i * b22i; y = a11 * a21 * b11i * b11i - lpl * a21 * b11i + a21 * a22 * b11i * b22i - a21 * a21 * b12 * b11i * b11i * b22i;
diff --git a/Eigen/src/Eigenvalues/RealSchur.h b/Eigen/src/Eigenvalues/RealSchur.h index a47fb3a..f9f3acb 100644 --- a/Eigen/src/Eigenvalues/RealSchur.h +++ b/Eigen/src/Eigenvalues/RealSchur.h
@@ -472,25 +472,33 @@ template <typename MatrixType> template <typename TMatrix> inline Index RealSchur<MatrixType>::findSmallSubdiagEntry(TMatrix& matT, Index iu, const Scalar& considerAsZero) { - using std::abs; Index res = iu; while (res > 0) { - Scalar s = abs(matT.coeff(res - 1, res - 1)) + abs(matT.coeff(res, res)); + Scalar s = numext::abs(matT.coeff(res - 1, res - 1)) + numext::abs(matT.coeff(res, res)); s = numext::maxi<Scalar>(s * NumTraits<Scalar>::epsilon(), considerAsZero); - if (abs(matT.coeff(res, res - 1)) <= s) break; + if (numext::abs(matT.coeff(res, res - 1)) <= s) break; res--; } return res; } +// GCC 10 and 11 record on each vzeroupper the SSE registers live across it. Post-reload GCSE can extend a live range +// across a vzeroupper without updating that record, and cprop_hardreg then deletes a copy that reads the unrecorded +// register (#3196). GCC 12 dropped the record in the fix for https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82735. +// Compile this helper out of line without that pass; to protect other code, build with -fno-gcse-after-reload. +#if EIGEN_GNUC_STRICT_AT_LEAST(10, 0, 0) && EIGEN_GNUC_STRICT_LESS_THAN(12, 0, 0) && defined(__AVX__) +#define EIGEN_REALSCHUR_SPLIT_INLINE EIGEN_DONT_INLINE __attribute__((optimize("no-gcse-after-reload"))) +#else +#define EIGEN_REALSCHUR_SPLIT_INLINE inline +#endif + /** \internal Update T given that rows iu-1 and iu decouple from the rest. */ template <typename MatrixType> template <typename TMatrix> -inline void RealSchur<MatrixType>::splitOffTwoRows(TMatrix& matT, Index iu, bool computeU, const Scalar& exshift) { - using std::abs; - using std::sqrt; +EIGEN_REALSCHUR_SPLIT_INLINE void RealSchur<MatrixType>::splitOffTwoRows(TMatrix& matT, Index iu, bool computeU, + const Scalar& exshift) { const Index size = matT.cols(); // The eigenvalues of the 2x2 matrix [a b; c d] are @@ -502,7 +510,7 @@ if (q >= Scalar(0)) // Two real eigenvalues { - Scalar z = sqrt(abs(q)); + Scalar z = numext::sqrt(numext::abs(q)); JacobiRotation<Scalar> rot; if (p >= Scalar(0)) rot.makeGivens(p + z, matT.coeff(iu, iu - 1)); @@ -518,13 +526,13 @@ if (iu > 1) matT.coeffRef(iu - 1, iu - 2) = Scalar(0); } +#undef EIGEN_REALSCHUR_SPLIT_INLINE + /** \internal Form shift in shiftInfo, and update exshift if an exceptional shift is performed. */ template <typename MatrixType> template <typename TMatrix> inline void RealSchur<MatrixType>::computeShift(TMatrix& matT, Index iu, Index iter, Scalar& exshift, Vector3s& shiftInfo) { - using std::abs; - using std::sqrt; shiftInfo.coeffRef(0) = matT.coeff(iu, iu); shiftInfo.coeffRef(1) = matT.coeff(iu - 1, iu - 1); shiftInfo.coeffRef(2) = matT.coeff(iu, iu - 1) * matT.coeff(iu - 1, iu); @@ -535,7 +543,7 @@ if (iter % 32 != 0) { exshift += shiftInfo.coeff(0); matT.diagonal().head(iu + 1).array() -= shiftInfo.coeff(0); - Scalar s = abs(matT.coeff(iu, iu - 1)) + abs(matT.coeff(iu - 1, iu - 2)); + Scalar s = numext::abs(matT.coeff(iu, iu - 1)) + numext::abs(matT.coeff(iu - 1, iu - 2)); shiftInfo.coeffRef(0) = Scalar(0.75) * s; shiftInfo.coeffRef(1) = Scalar(0.75) * s; shiftInfo.coeffRef(2) = Scalar(-0.4375) * s * s; @@ -544,7 +552,7 @@ Scalar s = (shiftInfo.coeff(1) - shiftInfo.coeff(0)) / Scalar(2.0); s = s * s + shiftInfo.coeff(2); if (s > Scalar(0)) { - s = sqrt(s); + s = numext::sqrt(s); if (shiftInfo.coeff(1) < shiftInfo.coeff(0)) s = -s; s = s + (shiftInfo.coeff(1) - shiftInfo.coeff(0)) / Scalar(2.0); s = shiftInfo.coeff(0) - shiftInfo.coeff(2) / s; @@ -561,7 +569,6 @@ template <typename TMatrix> inline void RealSchur<MatrixType>::initFrancisQRStep(TMatrix& matT, Index il, Index iu, const Vector3s& shiftInfo, Index& im, Vector3s& firstHouseholderVector) { - using std::abs; Vector3s& v = firstHouseholderVector; // alias to save typing for (im = iu - 2; im >= il; --im) { @@ -574,9 +581,10 @@ if (im == il) { break; } - const Scalar lhs = matT.coeff(im, im - 1) * (abs(v.coeff(1)) + abs(v.coeff(2))); - const Scalar rhs = v.coeff(0) * (abs(matT.coeff(im - 1, im - 1)) + abs(Tmm) + abs(matT.coeff(im + 1, im + 1))); - if (abs(lhs) < NumTraits<Scalar>::epsilon() * rhs) break; + const Scalar lhs = matT.coeff(im, im - 1) * (numext::abs(v.coeff(1)) + numext::abs(v.coeff(2))); + const Scalar rhs = v.coeff(0) * (numext::abs(matT.coeff(im - 1, im - 1)) + numext::abs(Tmm) + + numext::abs(matT.coeff(im + 1, im + 1))); + if (numext::abs(lhs) < NumTraits<Scalar>::epsilon() * rhs) break; } }
diff --git a/Eigen/src/Eigenvalues/SelfAdjointEigenSolver.h b/Eigen/src/Eigenvalues/SelfAdjointEigenSolver.h index de6a142..e49e1b6 100644 --- a/Eigen/src/Eigenvalues/SelfAdjointEigenSolver.h +++ b/Eigen/src/Eigenvalues/SelfAdjointEigenSolver.h
@@ -732,12 +732,8 @@ * For numerical stability m.trace() should be near zero and to avoid over- or underflow m should be normalized. */ EIGEN_DEVICE_FUNC static inline void computeRoots(const MatrixType& m, VectorType& roots) { - EIGEN_USING_STD(sqrt) - EIGEN_USING_STD(atan2) - EIGEN_USING_STD(cos) - EIGEN_USING_STD(sin) const Scalar s_inv3 = Scalar(1) / Scalar(3); - const Scalar s_sqrt3 = sqrt(Scalar(3)); + const Scalar s_sqrt3 = numext::sqrt(Scalar(3)); // The characteristic equation is x^3 - c2*x^2 + c1*x - c0 = 0. The // eigenvalues are the roots to this equation, all guaranteed to be @@ -760,10 +756,11 @@ q = numext::maxi(q, Scalar(0)); // Compute the eigenvalues by solving for the roots of the polynomial. - Scalar rho = sqrt(a_over_3); - Scalar theta = atan2(sqrt(q), half_b) * s_inv3; // since sqrt(q) > 0, atan2 is in [0, pi] and theta is in [0, pi/3] - Scalar cos_theta = cos(theta); - Scalar sin_theta = sin(theta); + Scalar rho = numext::sqrt(a_over_3); + Scalar theta = numext::atan2(numext::sqrt(q), half_b) * + s_inv3; // since sqrt(q) > 0, atan2 is in [0, pi] and theta is in [0, pi/3] + Scalar cos_theta = numext::cos(theta); + Scalar sin_theta = numext::sin(theta); // roots are already sorted, since cos is monotonically decreasing on [0, pi] roots(0) = c2_over_3 - rho * (cos_theta + s_sqrt3 * sin_theta); // == 2*rho*cos(theta+2pi/3) roots(1) = c2_over_3 - rho * (cos_theta - s_sqrt3 * sin_theta); // == 2*rho*cos(theta+ pi/3) @@ -772,8 +769,6 @@ EIGEN_DEVICE_FUNC static inline bool extract_kernel(PlainMatrixType& mat, Ref<VectorType> res, Ref<VectorType> representative) { - EIGEN_USING_STD(abs); - EIGEN_USING_STD(sqrt); Index i0; // Find non-zero column i0 (by construction, there must exist a non zero coefficient on the diagonal): mat.diagonal().cwiseAbs().maxCoeff(&i0); @@ -785,9 +780,9 @@ n0 = (c0 = representative.cross(mat.col((i0 + 1) % 3))).squaredNorm(); n1 = (c1 = representative.cross(mat.col((i0 + 2) % 3))).squaredNorm(); if (n0 > n1) - res = c0 / sqrt(n0); + res = c0 / numext::sqrt(n0); else - res = c1 / sqrt(n1); + res = c1 / numext::sqrt(n1); return true; } @@ -860,8 +855,8 @@ using PlainMatrixType = typename SolverType::PlainMatrixType; EIGEN_DEVICE_FUNC static inline void computeRoots(const MatrixType& m, VectorType& roots) { - EIGEN_USING_STD(sqrt); - const Scalar t0 = Scalar(0.5) * sqrt(numext::abs2(m(0, 0) - m(1, 1)) + Scalar(4) * numext::abs2(m(1, 0))); + const Scalar t0 = + Scalar(0.5) * numext::sqrt<Scalar>(numext::abs2(m(0, 0) - m(1, 1)) + Scalar(4) * numext::abs2(m(1, 0))); const Scalar t1 = Scalar(0.5) * (m(0, 0) + m(1, 1)); roots(0) = t1 - t0; roots(1) = t1 + t0; @@ -870,15 +865,12 @@ EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE static void run(PlainMatrixType& scaledMat, VectorType& eivals, EigenvectorsType& eivecs, bool computeEigenvectors, const Scalar&) { - EIGEN_USING_STD(sqrt); - EIGEN_USING_STD(abs); - // Compute the eigenvalues computeRoots(scaledMat, eivals); // compute the eigen vectors if (computeEigenvectors) { - if ((eivals(1) - eivals(0)) <= abs(eivals(1)) * Eigen::NumTraits<Scalar>::epsilon()) { + if ((eivals(1) - eivals(0)) <= numext::abs(eivals(1)) * Eigen::NumTraits<Scalar>::epsilon()) { eivecs.setIdentity(); } else { scaledMat.diagonal().array() -= eivals(1); @@ -887,10 +879,10 @@ Scalar b2 = numext::abs2(scaledMat(1, 0)); if (a2 > c2) { eivecs.col(1) << -scaledMat(1, 0), scaledMat(0, 0); - eivecs.col(1) /= sqrt(a2 + b2); + eivecs.col(1) /= numext::sqrt<Scalar>(a2 + b2); } else { eivecs.col(1) << -scaledMat(1, 1), scaledMat(1, 0); - eivecs.col(1) /= sqrt(c2 + b2); + eivecs.col(1) /= numext::sqrt<Scalar>(c2 + b2); } // The partner is already normalized; this rotation preserves its norm.
diff --git a/Eigen/src/Eigenvalues/Tridiagonalization.h b/Eigen/src/Eigenvalues/Tridiagonalization.h index ba640c6..2712ca0 100644 --- a/Eigen/src/Eigenvalues/Tridiagonalization.h +++ b/Eigen/src/Eigenvalues/Tridiagonalization.h
@@ -350,7 +350,6 @@ */ template <typename MatrixType, typename CoeffVectorType> EIGEN_DEVICE_FUNC void tridiagonalization_inplace_unblocked(MatrixType& matA, CoeffVectorType& hCoeffs) { - using numext::conj; using Scalar = typename MatrixType::Scalar; using RealScalar = typename MatrixType::RealScalar; Index n = matA.rows(); @@ -369,10 +368,10 @@ hCoeffs.tail(n - i - 1).noalias() = (matA.bottomRightCorner(remainingSize, remainingSize).template selfadjointView<Lower>() * - (conj(h) * matA.col(i).tail(remainingSize))); + (numext::conj(h) * matA.col(i).tail(remainingSize))); hCoeffs.tail(n - i - 1) += - (conj(h) * RealScalar(-0.5) * (hCoeffs.tail(remainingSize).dot(matA.col(i).tail(remainingSize)))) * + (numext::conj(h) * RealScalar(-0.5) * (hCoeffs.tail(remainingSize).dot(matA.col(i).tail(remainingSize)))) * matA.col(i).tail(n - i - 1); matA.bottomRightCorner(remainingSize, remainingSize) @@ -393,7 +392,6 @@ */ template <typename MatrixType, typename CoeffVectorType> void tridiagonalization_inplace_blocked(MatrixType& matA, CoeffVectorType& hCoeffs, Index nb = 16) { - using numext::conj; using Scalar = typename MatrixType::Scalar; using RealScalar = typename MatrixType::RealScalar; const Index n = matA.rows(); @@ -455,8 +453,8 @@ // Step 3: Compute w = conj(h) * A_eff * v where A_eff accounts for deferred updates. // Start with SYMV on the stored (not yet updated) trailing submatrix. - w.noalias() = - matA.bottomRightCorner(remainingSize, remainingSize).template selfadjointView<Lower>() * (conj(h) * v); + w.noalias() = matA.bottomRightCorner(remainingSize, remainingSize).template selfadjointView<Lower>() * + (numext::conj(h) * v); // GEMV corrections for deferred rank-2 updates within this panel. if (local_j > 0) { @@ -465,15 +463,15 @@ // w -= conj(h) * V_prev * (W_prev^H * v) temp.head(local_j).noalias() = W_prev.adjoint() * v; - w.noalias() -= conj(h) * (V_prev * temp.head(local_j)); + w.noalias() -= numext::conj(h) * (V_prev * temp.head(local_j)); // w -= conj(h) * W_prev * (V_prev^H * v) temp.head(local_j).noalias() = V_prev.adjoint() * v; - w.noalias() -= conj(h) * (W_prev * temp.head(local_j)); + w.noalias() -= numext::conj(h) * (W_prev * temp.head(local_j)); } // Step 4: Half-dot correction: w -= 0.5 * conj(h) * (w^H * v) * v - w += (conj(h) * RealScalar(-0.5) * w.dot(v)) * v; + w += (numext::conj(h) * RealScalar(-0.5) * w.dot(v)) * v; hCoeffs.coeffRef(j) = h; } @@ -626,7 +624,6 @@ template <typename DiagonalType, typename SubDiagonalType, typename CoeffVectorType, typename WorkSpaceType> static EIGEN_DEVICE_FUNC void run(MatrixType& mat, DiagonalType& diag, SubDiagonalType& subdiag, CoeffVectorType&, WorkSpaceType&, bool extractQ) { - using std::sqrt; const RealScalar tol = (std::numeric_limits<RealScalar>::min)(); diag[0] = mat(0, 0); RealScalar v1norm2 = numext::abs2(mat(2, 0)); @@ -637,7 +634,7 @@ subdiag[1] = mat(2, 1); if (extractQ) mat.setIdentity(); } else { - RealScalar beta = sqrt(numext::abs2(mat(1, 0)) + v1norm2); + RealScalar beta = numext::sqrt<RealScalar>(numext::abs2(mat(1, 0)) + v1norm2); RealScalar invBeta = RealScalar(1) / beta; Scalar m01 = mat(1, 0) * invBeta; Scalar m02 = mat(2, 0) * invBeta;
diff --git a/Eigen/src/Geometry/AlignedBox.h b/Eigen/src/Geometry/AlignedBox.h index 210c0f9..a13f692 100644 --- a/Eigen/src/Geometry/AlignedBox.h +++ b/Eigen/src/Geometry/AlignedBox.h
@@ -317,7 +317,7 @@ */ template <typename Derived> EIGEN_DEVICE_FUNC inline NonInteger exteriorDistance(const MatrixBase<Derived>& p) const { - EIGEN_USING_STD(sqrt) return sqrt(NonInteger(squaredExteriorDistance(p))); + return numext::sqrt(NonInteger(squaredExteriorDistance(p))); } /** \returns the distance between the boxes \a b and \c *this, @@ -325,7 +325,7 @@ * \sa squaredExteriorDistance(const AlignedBox&), exteriorDistance(const MatrixBase&) */ EIGEN_DEVICE_FUNC inline NonInteger exteriorDistance(const AlignedBox& b) const { - EIGEN_USING_STD(sqrt) return sqrt(NonInteger(squaredExteriorDistance(b))); + return numext::sqrt(NonInteger(squaredExteriorDistance(b))); } /**
diff --git a/Eigen/src/Geometry/AngleAxis.h b/Eigen/src/Geometry/AngleAxis.h index 7bd7a6a..bed4a54 100644 --- a/Eigen/src/Geometry/AngleAxis.h +++ b/Eigen/src/Geometry/AngleAxis.h
@@ -135,14 +135,12 @@ * without constructing the full rotation matrix. */ template <typename OtherVectorType> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Vector3 _transformVector(const OtherVectorType& v) const { - EIGEN_USING_STD(sin) - EIGEN_USING_STD(cos) EIGEN_INTERNAL_ASSERT_SAME_SIZE(v, m_axis); // Rodrigues' rotation formula: v' = v*cos(θ) + (k×v)*sin(θ) + k*(k·v)*(1-cos(θ)) const Vector3 v3(v); - const Scalar c = cos(m_angle); - const Scalar s = sin(m_angle); + const Scalar c = numext::cos(m_angle); + const Scalar s = numext::sin(m_angle); return v3 * c + m_axis.cross(v3) * s + m_axis * (m_axis.dot(v3) * (Scalar(1) - c)); } @@ -192,13 +190,11 @@ template <typename Scalar> template <typename QuatDerived> EIGEN_DEVICE_FUNC AngleAxis<Scalar>& AngleAxis<Scalar>::operator=(const QuaternionBase<QuatDerived>& q) { - EIGEN_USING_STD(atan2) - EIGEN_USING_STD(abs) Scalar n = q.vec().norm(); if (n < NumTraits<Scalar>::epsilon()) n = q.vec().stableNorm(); if (n != Scalar(0)) { - m_angle = Scalar(2) * atan2(n, abs(q.w())); + m_angle = Scalar(2) * numext::atan2(n, numext::abs(q.w())); if (q.w() < Scalar(0)) n = -n; m_axis = q.vec() / n; } else { @@ -248,11 +244,9 @@ */ template <typename Scalar> typename AngleAxis<Scalar>::Matrix3 EIGEN_DEVICE_FUNC AngleAxis<Scalar>::toRotationMatrix(void) const { - EIGEN_USING_STD(sin) - EIGEN_USING_STD(cos) Matrix3 res; - Vector3 sin_axis = sin(m_angle) * m_axis; - Scalar c = cos(m_angle); + Vector3 sin_axis = numext::sin(m_angle) * m_axis; + Scalar c = numext::cos(m_angle); Vector3 cos1_axis = (Scalar(1) - c) * m_axis; Scalar tmp;
diff --git a/Eigen/src/Geometry/ParametrizedLine.h b/Eigen/src/Geometry/ParametrizedLine.h index 8c59ffb..8ed8ab5 100644 --- a/Eigen/src/Geometry/ParametrizedLine.h +++ b/Eigen/src/Geometry/ParametrizedLine.h
@@ -87,9 +87,7 @@ /** \returns the distance of a point \a p to its projection onto the line \c *this. * \sa squaredDistance() */ - EIGEN_DEVICE_FUNC RealScalar distance(const VectorType& p) const { - EIGEN_USING_STD(sqrt) return sqrt(squaredDistance(p)); - } + EIGEN_DEVICE_FUNC RealScalar distance(const VectorType& p) const { return numext::sqrt(squaredDistance(p)); } /** \returns the projection of a point \a p onto the line \c *this. */ EIGEN_DEVICE_FUNC VectorType projection(const VectorType& p) const {
diff --git a/Eigen/src/Geometry/Quaternion.h b/Eigen/src/Geometry/Quaternion.h index 4efc0e7..8dfa33f 100644 --- a/Eigen/src/Geometry/Quaternion.h +++ b/Eigen/src/Geometry/Quaternion.h
@@ -621,11 +621,9 @@ */ template <class Derived> EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Derived& QuaternionBase<Derived>::operator=(const AngleAxisType& aa) { - EIGEN_USING_STD(cos) - EIGEN_USING_STD(sin) Scalar ha = Scalar(0.5) * aa.angle(); // Scalar(0.5) to suppress precision loss warnings - this->w() = cos(ha); - this->vec() = sin(ha) * aa.axis(); + this->w() = numext::cos(ha); + this->vec() = numext::sin(ha) * aa.axis(); return derived(); } @@ -694,7 +692,6 @@ template <typename Derived1, typename Derived2> EIGEN_DEVICE_FUNC inline Derived& QuaternionBase<Derived>::setFromTwoVectors(const MatrixBase<Derived1>& a, const MatrixBase<Derived2>& b) { - EIGEN_USING_STD(sqrt) Vector3 v0 = a.normalized(); Vector3 v1 = b.normalized(); Scalar c = v1.dot(v0); @@ -708,7 +705,7 @@ return derived(); } Vector3 axis = v0.cross(v1); - Scalar s = sqrt((Scalar(1) + c) * Scalar(2)); + Scalar s = numext::sqrt<Scalar>((Scalar(1) + c) * Scalar(2)); Scalar invs = Scalar(1) / s; this->vec() = axis * invs; this->w() = s * Scalar(0.5); @@ -731,8 +728,6 @@ template <typename OtherDerived> EIGEN_DEVICE_FUNC inline Derived& QuaternionBase<Derived>::setFromScaledAxis( const MatrixBase<OtherDerived>& scaled_axis) { - EIGEN_USING_STD(sin) - EIGEN_USING_STD(cos) EIGEN_STATIC_ASSERT_VECTOR_SPECIFIC_SIZE(OtherDerived, 3) // Materialise the input once so the caller's expression (e.g. omega*dt) is not @@ -751,8 +746,8 @@ } const Scalar half_angle = Scalar(0.5) * angle; - this->w() = cos(half_angle); - this->vec() = (sin(half_angle) / angle) * v; + this->w() = numext::cos(half_angle); + this->vec() = (numext::sin(half_angle) / angle) * v; return derived(); } @@ -771,7 +766,6 @@ */ template <class Derived> EIGEN_DEVICE_FUNC inline typename QuaternionBase<Derived>::Vector3 QuaternionBase<Derived>::toScaledAxis() const { - EIGEN_USING_STD(atan2) eigen_assert(internal::isApprox(this->squaredNorm(), Scalar(1), Scalar(8) * NumTraits<Scalar>::dummy_precision()) && "QuaternionBase::toScaledAxis(): the quaternion must be approximately a unit quaternion"); @@ -783,7 +777,7 @@ // |w| in atan2 prevents -0.0 from flipping the sign; recover it via the comparison below // (false for both +0.0 and -0.0). // Cannot overflow: for unit q, atan2(s, |c|) is O(s) as s -> 0, so k stays in [2, pi]. - Scalar k = Scalar(2) * atan2(sin_half_angle, numext::abs(this->w())) / sin_half_angle; + Scalar k = Scalar(2) * numext::atan2(sin_half_angle, numext::abs(this->w())) / sin_half_angle; if (this->w() < Scalar(0)) k = -k; return k * this->vec(); } @@ -794,13 +788,10 @@ */ template <typename Scalar, int Options> EIGEN_DEVICE_FUNC Quaternion<Scalar, Options> Quaternion<Scalar, Options>::UnitRandom() { - EIGEN_USING_STD(sqrt) - EIGEN_USING_STD(sin) - EIGEN_USING_STD(cos) const Scalar u1 = internal::random<Scalar>(0, 1), u2 = internal::random<Scalar>(0, 2 * EIGEN_PI), u3 = internal::random<Scalar>(0, 2 * EIGEN_PI); - const Scalar a = sqrt(Scalar(1) - u1), b = sqrt(u1); - return Quaternion(a * sin(u2), a * cos(u2), b * sin(u3), b * cos(u3)); + const Scalar a = numext::sqrt<Scalar>(Scalar(1) - u1), b = numext::sqrt(u1); + return Quaternion(a * numext::sin(u2), a * numext::cos(u2), b * numext::sin(u3), b * numext::cos(u3)); } /** Constructs a quaternion from its coefficients in the order [\c x, \c y, \c z, \c w], i.e. vector part [\c x, \c y, @@ -914,9 +905,8 @@ template <class OtherDerived> EIGEN_DEVICE_FUNC inline typename internal::traits<Derived>::Scalar QuaternionBase<Derived>::angularDistance( const QuaternionBase<OtherDerived>& other) const { - EIGEN_USING_STD(atan2) Quaternion<Scalar> d = (*this) * other.conjugate(); - return Scalar(2) * atan2(d.vec().norm(), numext::abs(d.w())); + return Scalar(2) * numext::atan2(d.vec().norm(), numext::abs(d.w())); } /** \returns the spherical linear interpolation between the two quaternions @@ -929,8 +919,6 @@ template <class OtherDerived> EIGEN_DEVICE_FUNC Quaternion<typename internal::traits<Derived>::Scalar> QuaternionBase<Derived>::slerp( const Scalar& t, const QuaternionBase<OtherDerived>& other) const { - EIGEN_USING_STD(acos) - EIGEN_USING_STD(sin) const Scalar one = Scalar(1) - NumTraits<Scalar>::epsilon(); Scalar d = this->dot(other); Scalar absD = numext::abs(d); @@ -944,11 +932,11 @@ scale1 = t; } else { // theta is the angle between the 2 quaternions - Scalar theta = acos(absD); - Scalar sinTheta = numext::sqrt(Scalar(1) - absD * absD); + Scalar theta = numext::acos(absD); + Scalar sinTheta = numext::sqrt<Scalar>(Scalar(1) - absD * absD); - scale0 = sin((Scalar(1) - t) * theta) / sinTheta; - scale1 = sin(t * theta) / sinTheta; + scale0 = numext::sin<Scalar>((Scalar(1) - t) * theta) / sinTheta; + scale1 = numext::sin<Scalar>(t * theta) / sinTheta; } if (d < Scalar(0)) scale1 = -scale1;
diff --git a/Eigen/src/Geometry/Rotation2D.h b/Eigen/src/Geometry/Rotation2D.h index 31107b0..1554401 100644 --- a/Eigen/src/Geometry/Rotation2D.h +++ b/Eigen/src/Geometry/Rotation2D.h
@@ -179,10 +179,9 @@ template <typename Scalar> template <typename Derived> EIGEN_DEVICE_FUNC Rotation2D<Scalar>& Rotation2D<Scalar>::fromRotationMatrix(const MatrixBase<Derived>& mat) { - EIGEN_USING_STD(atan2) EIGEN_STATIC_ASSERT(Derived::RowsAtCompileTime == 2 && Derived::ColsAtCompileTime == 2, YOU_MADE_A_PROGRAMMING_MISTAKE) - m_angle = atan2(mat.coeff(1, 0), mat.coeff(0, 0)); + m_angle = numext::atan2(Scalar(mat.coeff(1, 0)), Scalar(mat.coeff(0, 0))); return *this; } @@ -190,10 +189,8 @@ */ template <typename Scalar> typename Rotation2D<Scalar>::Matrix2 EIGEN_DEVICE_FUNC Rotation2D<Scalar>::toRotationMatrix(void) const { - EIGEN_USING_STD(sin) - EIGEN_USING_STD(cos) - Scalar sinA = sin(m_angle); - Scalar cosA = cos(m_angle); + Scalar sinA = numext::sin(m_angle); + Scalar cosA = numext::cos(m_angle); return (Matrix2() << cosA, -sinA, sinA, cosA).finished(); }
diff --git a/Eigen/src/Geometry/Transform.h b/Eigen/src/Geometry/Transform.h index 2a2150f..4d1133a 100644 --- a/Eigen/src/Geometry/Transform.h +++ b/Eigen/src/Geometry/Transform.h
@@ -211,7 +211,7 @@ * Note that there is zero overhead. * * Conversion methods from/to Qt's QMatrix and QTransform are available if the - * preprocessor token EIGEN_QT_SUPPORT is defined. + * preprocessor token EIGEN_QT_SUPPORT is defined. They are deprecated and will be removed. * * This class can be extended with the help of the plugin mechanism described on the page * \ref TopicCustomizing_Plugins by defining the preprocessor symbol \c EIGEN_TRANSFORM_PLUGIN. @@ -296,7 +296,8 @@ using take_affine_part = internal::transform_take_affine_part<Transform>; - /** Constructs and initializes a transformation from a Dim^2 or a (Dim+1)^2 matrix. */ + /** Constructs and initializes a transformation from a Dim^2 or a (Dim+1)^2 matrix. + * If Mode==Affine or Mode==Isometry, then the last row is set to [0 ... 0 1] */ template <typename OtherDerived> EIGEN_DEVICE_FUNC inline explicit Transform(const EigenBase<OtherDerived>& other) { EIGEN_STATIC_ASSERT( @@ -307,7 +308,8 @@ internal::transform_construct_from_matrix<OtherDerived, Mode, Options, Dim, HDim>::run(this, other.derived()); } - /** Set \c *this from a Dim^2 or (Dim+1)^2 matrix. */ + /** Set \c *this from a Dim^2 or (Dim+1)^2 matrix. + * If Mode==Affine or Mode==Isometry, then the last row is set to [0 ... 0 1] */ template <typename OtherDerived> EIGEN_DEVICE_FUNC inline Transform& operator=(const EigenBase<OtherDerived>& other) { EIGEN_STATIC_ASSERT( @@ -376,13 +378,14 @@ #ifdef EIGEN_QT_SUPPORT #if (QT_VERSION < QT_VERSION_CHECK(6, 0, 0)) - inline Transform(const QMatrix& other); - inline Transform& operator=(const QMatrix& other); - inline QMatrix toQMatrix(void) const; + EIGEN_DEPRECATED_WITH_REASON("Copy the coefficients explicitly.") inline Transform(const QMatrix& other); + EIGEN_DEPRECATED_WITH_REASON("Copy the coefficients explicitly.") inline Transform& operator=(const QMatrix& other); + EIGEN_DEPRECATED_WITH_REASON("Copy the coefficients explicitly.") inline QMatrix toQMatrix(void) const; #endif - inline Transform(const QTransform& other); + EIGEN_DEPRECATED_WITH_REASON("Copy the coefficients explicitly.") inline Transform(const QTransform& other); + EIGEN_DEPRECATED_WITH_REASON("Copy the coefficients explicitly.") inline Transform& operator=(const QTransform& other); - inline QTransform toQTransform(void) const; + EIGEN_DEPRECATED_WITH_REASON("Copy the coefficients explicitly.") inline QTransform toQTransform(void) const; #endif EIGEN_DEVICE_FUNC constexpr Index rows() const noexcept { @@ -723,10 +726,16 @@ #ifdef EIGEN_QT_SUPPORT +// The deprecated constructors call the deprecated assignment operators. +EIGEN_DIAGNOSTICS(push) +EIGEN_DIAGNOSTICS_OFF(disable : 4996, ignored "-Wdeprecated-declarations") + #if (QT_VERSION < QT_VERSION_CHECK(6, 0, 0)) /** Initializes \c *this from a QMatrix assuming the dimension is 2. * * This function is available only if the token EIGEN_QT_SUPPORT is defined. + * + * \deprecated Copy the coefficients explicitly. */ template <typename Scalar, int Dim, int Mode, int Options> Transform<Scalar, Dim, Mode, Options>::Transform(const QMatrix& other) { @@ -737,6 +746,8 @@ /** Set \c *this from a QMatrix assuming the dimension is 2. * * This function is available only if the token EIGEN_QT_SUPPORT is defined. + * + * \deprecated Copy the coefficients explicitly. */ template <typename Scalar, int Dim, int Mode, int Options> Transform<Scalar, Dim, Mode, Options>& Transform<Scalar, Dim, Mode, Options>::operator=(const QMatrix& other) { @@ -753,6 +764,8 @@ * \warning this conversion might lose data if \c *this is not affine * * This function is available only if the token EIGEN_QT_SUPPORT is defined. + * + * \deprecated Copy the coefficients explicitly. */ template <typename Scalar, int Dim, int Mode, int Options> QMatrix Transform<Scalar, Dim, Mode, Options>::toQMatrix(void) const { @@ -766,6 +779,8 @@ /** Initializes \c *this from a QTransform assuming the dimension is 2. * * This function is available only if the token EIGEN_QT_SUPPORT is defined. + * + * \deprecated Copy the coefficients explicitly. */ template <typename Scalar, int Dim, int Mode, int Options> Transform<Scalar, Dim, Mode, Options>::Transform(const QTransform& other) { @@ -776,6 +791,8 @@ /** Set \c *this from a QTransform assuming the dimension is 2. * * This function is available only if the token EIGEN_QT_SUPPORT is defined. + * + * \deprecated Copy the coefficients explicitly. */ template <typename Scalar, int Dim, int Mode, int Options> Transform<Scalar, Dim, Mode, Options>& Transform<Scalar, Dim, Mode, Options>::operator=(const QTransform& other) { @@ -792,6 +809,8 @@ /** \returns a QTransform from \c *this assuming the dimension is 2. * * This function is available only if the token EIGEN_QT_SUPPORT is defined. + * + * \deprecated Copy the coefficients explicitly. */ template <typename Scalar, int Dim, int Mode, int Options> QTransform Transform<Scalar, Dim, Mode, Options>::toQTransform(void) const { @@ -804,6 +823,8 @@ m_matrix.coeff(1, 1), m_matrix.coeff(2, 1), m_matrix.coeff(0, 2), m_matrix.coeff(1, 2), m_matrix.coeff(2, 2)); } + +EIGEN_DIAGNOSTICS(pop) #endif /********************* @@ -1343,6 +1364,9 @@ static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void run( Transform<typename Other::Scalar, Dim, Mode, Options>* transform, const Other& other) { transform->matrix() = other; + // Affine and Isometry transforms require the last row to be [0 ... 0 1]; Projective keeps it as given. + transform_make_affine<(int(Mode) == Affine || int(Mode) == Isometry) ? Affine : AffineCompact>::run( + transform->matrix()); } };
diff --git a/Eigen/src/Householder/Householder.h b/Eigen/src/Householder/Householder.h index 6e98552..f02cfbf 100644 --- a/Eigen/src/Householder/Householder.h +++ b/Eigen/src/Householder/Householder.h
@@ -138,8 +138,6 @@ template <typename EssentialPart> EIGEN_DEVICE_FUNC void MatrixBase<Derived>::makeHouseholder(EssentialPart& essential, Scalar& tau, RealScalar& beta) const { - using numext::conj; - EIGEN_STATIC_ASSERT_VECTOR_ONLY(EssentialPart) const VectorBlock<const Derived, EssentialPart::SizeAtCompileTime> tail(derived(), 1, size() - 1); @@ -227,13 +225,13 @@ if (numext::real(c0) >= RealScalar(0)) scaledBeta = -scaledBeta; beta = RealScalar(scale * Accumulator(scaledBeta)); internal::householder_rescale<Scalar, Accumulator>::run(essential, tailView, scale, scaledC0 - scaledBeta); - tau = conj(Scalar(RealScalar(1)) - scaledC0 / scaledBeta); + tau = numext::conj(Scalar(RealScalar(1)) - scaledC0 / scaledBeta); return; } beta = numext::sqrt<RealScalar>(numext::abs2(c0) + tailSqNorm); if (numext::real(c0) >= RealScalar(0)) beta = -beta; essential = tail.unwind() / (c0 - beta); - tau = conj((beta - c0) / beta); + tau = numext::conj((beta - c0) / beta); } namespace internal {
diff --git a/Eigen/src/Householder/HouseholderSequence.h b/Eigen/src/Householder/HouseholderSequence.h index 39e985c..3c8b8aa 100644 --- a/Eigen/src/Householder/HouseholderSequence.h +++ b/Eigen/src/Householder/HouseholderSequence.h
@@ -104,9 +104,11 @@ } }; -template <typename OtherScalarType, typename MatrixType> +template <typename OtherScalarType, typename MatrixType, bool ScalarOnLeft = true> struct matrix_type_times_scalar_type { - using ResultScalar = typename ScalarBinaryOpTraits<OtherScalarType, typename MatrixType::Scalar>::ReturnType; + using ResultScalar = + typename std::conditional_t<ScalarOnLeft, ScalarBinaryOpTraits<OtherScalarType, typename MatrixType::Scalar>, + ScalarBinaryOpTraits<typename MatrixType::Scalar, OtherScalarType>>::ReturnType; using Type = Matrix<ResultScalar, MatrixType::RowsAtCompileTime, MatrixType::ColsAtCompileTime, (MatrixType::MaxRowsAtCompileTime == 1 && MatrixType::MaxColsAtCompileTime != 1) ? RowMajor : ColMajor, @@ -591,11 +593,11 @@ * Householder sequence represented by \p h. */ template <typename OtherDerived, typename VectorsType, typename CoeffsType, int Side> -typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived>::Type operator*( +typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived, false>::Type operator*( const MatrixBase<OtherDerived>& other, const HouseholderSequence<VectorsType, CoeffsType, Side>& h) { - typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived>::Type res( - other.template cast<typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, - OtherDerived>::ResultScalar>()); + typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived, false>::Type res( + other.template cast<typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived, + false>::ResultScalar>()); h.applyThisOnTheRight(res); return res; } @@ -605,9 +607,10 @@ * \f$ DH \f$ is \f$ H \f$ with its rows scaled: the sequence is evaluated and scaled in place. */ template <typename OtherDerived, typename VectorsType, typename CoeffsType, int Side> -typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived>::Type operator*( +typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived, false>::Type operator*( const DiagonalBase<OtherDerived>& other, const HouseholderSequence<VectorsType, CoeffsType, Side>& h) { - using ResultType = typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived>::Type; + using ResultType = + typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived, false>::Type; EIGEN_INTERNAL_ASSERT_PRODUCT_SHAPE(other, h); ResultType res = ResultType::Identity(h.rows(), h.rows()); h.applyThisOnTheLeft(res, true); @@ -623,9 +626,10 @@ * structured path. */ template <typename OtherDerived, typename VectorsType, typename CoeffsType, int Side> -typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived>::Type operator*( +typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived, false>::Type operator*( const TriangularBase<OtherDerived>& other, const HouseholderSequence<VectorsType, CoeffsType, Side>& h) { - using ResultType = typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived>::Type; + using ResultType = + typename internal::matrix_type_times_scalar_type<typename VectorsType::Scalar, OtherDerived, false>::Type; ResultType res; internal::assign_householder_operand(res, other); h.applyThisOnTheRight(res);
diff --git a/Eigen/src/IterativeLinearSolvers/GMRES.h b/Eigen/src/IterativeLinearSolvers/GMRES.h index caecf5b..f86e6bb 100644 --- a/Eigen/src/IterativeLinearSolvers/GMRES.h +++ b/Eigen/src/IterativeLinearSolvers/GMRES.h
@@ -60,9 +60,6 @@ template <typename MatrixType, typename Rhs, typename Dest, typename Preconditioner> bool gmres(const MatrixType& mat, const Rhs& rhs, Dest& x, const Preconditioner& precond, Index& iters, const Index& restart_, typename Dest::RealScalar& tol_error) { - using std::abs; - using std::sqrt; - using RealScalar = typename Dest::RealScalar; using Scalar = typename Dest::Scalar; using VectorType = Matrix<Scalar, Dynamic, 1>; @@ -166,7 +163,7 @@ // insert coefficients into upper matrix triangle H.col(k - 1).head(k) = v.head(k); - tol_error = abs(w(k)) / rhsNorm; + tol_error = numext::abs(w(k)) / rhsNorm; bool stop = (k == m || tol_error < tol || iters == maxIters); if (stop || k == restart) {
diff --git a/Eigen/src/IterativeLinearSolvers/IDRS.h b/Eigen/src/IterativeLinearSolvers/IDRS.h index 7819ae1..5b4d938 100644 --- a/Eigen/src/IterativeLinearSolvers/IDRS.h +++ b/Eigen/src/IterativeLinearSolvers/IDRS.h
@@ -35,12 +35,11 @@ */ template <typename Vector, typename RealScalar> typename Vector::Scalar omega(const Vector& t, const Vector& s, RealScalar angle) { - using numext::abs; using Scalar = typename Vector::Scalar; const RealScalar ns = s.stableNorm(); const RealScalar nt = t.stableNorm(); const Scalar ts = t.dot(s); - const RealScalar rho = abs(ts / (nt * ns)); + const RealScalar rho = numext::abs(ts / (nt * ns)); if (rho < angle) { if (ts == Scalar(0)) { @@ -50,7 +49,7 @@ // om = om * angle / rho; // To alleviate potential (near) division by zero this can be rewritten as // om = angle * (ns / nt) * (ts / abs(ts)) = angle * (ns / nt) * sgn(ts) - return angle * (ns / nt) * (ts / abs(ts)); + return angle * (ns / nt) * (ts / numext::abs(ts)); } return ts / (nt * nt); }
diff --git a/Eigen/src/IterativeLinearSolvers/IDRSTABL.h b/Eigen/src/IterativeLinearSolvers/IDRSTABL.h index c22c06b..1231589 100644 --- a/Eigen/src/IterativeLinearSolvers/IDRSTABL.h +++ b/Eigen/src/IterativeLinearSolvers/IDRSTABL.h
@@ -52,7 +52,6 @@ /* Setup and type definitions. */ - using numext::abs; using Scalar = typename Dest::Scalar; using RealScalar = typename Dest::RealScalar; using VectorType = Matrix<Scalar, Dynamic, 1>; @@ -134,7 +133,7 @@ u.col(0) = w; h_FOM(col_index, col_index - 1) = u.col(0).stableNorm(); - if (abs(h_FOM(col_index, col_index - 1)) != RealScalar(0)) { + if (numext::abs(h_FOM(col_index, col_index - 1)) != RealScalar(0)) { /* This only happens if u is NOT exactly zero. In case it is exactly zero it would imply that that this u has no component in the direction of the @@ -258,8 +257,8 @@ bool break_normalization = false; for (Index q = 1; q <= S; ++q) { if (q == 1) { - // u = r; - u.leftCols(j + 1) = r.leftCols(j + 1); + // u = [r_0, ..., r_{j-1}]; u_j is computed below. + u.leftCols(j) = r.leftCols(j); } else { // u=[u_1;u_2;...;u_j] u.leftCols(j) = u.middleCols(1, j); @@ -303,7 +302,11 @@ } if (!break_normalization) { - U = V; + U.topRows(N * (j + 1)) = V.topRows(N * (j + 1)); + } else { + // Zero block row j rather than leaving it uninitialized (k == 0) or keeping a stale + // block from the previous cycle (k >= 1), so the next step's sigma and alpha are zero. + U.middleRows(N * j, N).setZero(); } } if (reset_while) {
diff --git a/Eigen/src/IterativeLinearSolvers/IncompleteCholesky.h b/Eigen/src/IterativeLinearSolvers/IncompleteCholesky.h index aa06c70..e592e96 100644 --- a/Eigen/src/IterativeLinearSolvers/IncompleteCholesky.h +++ b/Eigen/src/IterativeLinearSolvers/IncompleteCholesky.h
@@ -204,7 +204,6 @@ template <typename Scalar, int UpLo_, typename OrderingType> template <typename MatrixType_> void IncompleteCholesky<Scalar, UpLo_, OrderingType>::factorize(const MatrixType_& mat) { - using std::sqrt; eigen_assert(m_analysisIsOk && "analyzePattern() should be called first"); // Dropping strategy : Keep only the p largest elements per column, where p is the number of elements in the column of @@ -342,7 +341,7 @@ break; } - RealScalar rdiag = sqrt(numext::real(diag)); + RealScalar rdiag = numext::sqrt(numext::real(diag)); vals[colPtr[j]] = rdiag; for (Index k = 0; k < col_nnz; ++k) { Index i = col_irow[k];
diff --git a/Eigen/src/IterativeLinearSolvers/IncompleteLUT.h b/Eigen/src/IterativeLinearSolvers/IncompleteLUT.h index 11fed19..97d7036 100644 --- a/Eigen/src/IterativeLinearSolvers/IncompleteLUT.h +++ b/Eigen/src/IterativeLinearSolvers/IncompleteLUT.h
@@ -31,7 +31,6 @@ template <typename VectorV, typename VectorI> Index QuickSplit(VectorV& row, VectorI& ind, Index ncut) { using RealScalar = typename VectorV::RealScalar; - using std::abs; using std::swap; Index mid; Index n = row.size(); /* length of the vector */ @@ -44,9 +43,9 @@ do { mid = first; - RealScalar abskey = abs(row(mid)); + RealScalar abskey = numext::abs(row(mid)); for (Index j = first + 1; j <= last; j++) { - if (abs(row(j)) > abskey) { + if (numext::abs(row(j)) > abskey) { ++mid; swap(row(mid), row(j)); swap(ind(mid), ind(j)); @@ -429,8 +428,6 @@ template <typename MatrixType_> void IncompleteLUT<Scalar, StorageIndex>::factorize(const MatrixType_& amat) { using internal::convert_index; - using std::abs; - using std::sqrt; using std::swap; EIGEN_INTERNAL_ASSERT_SQUARE_MSG(amat, "The factorization should be done on a square matrix"); @@ -502,7 +499,7 @@ return; } // Take the 2-norm of the current row as a relative tolerance - rownorm = sqrt(rownorm); + rownorm = numext::sqrt(rownorm); // 3 - eliminate the previous nonzero rows Index jj = 0; @@ -531,7 +528,7 @@ Scalar fact = u(jj) / ki_it.value(); // drop too small elements - if (abs(fact) <= m_droptol) { + if (numext::abs(fact) <= m_droptol) { jj++; continue; } @@ -589,7 +586,7 @@ // store the diagonal element // apply a shifting rule to avoid zero pivots (we are doing an incomplete factorization) if (u(ii) == Scalar(0)) { - u(ii) = sqrt(m_droptol) * rownorm; + u(ii) = numext::sqrt(m_droptol) * rownorm; ++zero_pivots; } m_lu.insertBackByOuterInnerUnordered(ii, ii) = u(ii); @@ -598,7 +595,7 @@ // apply the dropping rule first len = 0; for (Index k = 1; k < sizeu; k++) { - if (abs(u(ii + k)) > m_droptol * rownorm) { + if (numext::abs(u(ii + k)) > m_droptol * rownorm) { ++len; u(ii + len) = u(ii + k); ju(ii + len) = ju(ii + k);
diff --git a/Eigen/src/IterativeLinearSolvers/LSMR.h b/Eigen/src/IterativeLinearSolvers/LSMR.h index 114837d..77b896e 100644 --- a/Eigen/src/IterativeLinearSolvers/LSMR.h +++ b/Eigen/src/IterativeLinearSolvers/LSMR.h
@@ -78,8 +78,6 @@ Index& iters, typename Dest::RealScalar& tol_error, const typename Dest::RealScalar& atol, const typename Dest::RealScalar& btol, const typename Dest::RealScalar& lambda, const typename Dest::RealScalar& conlim) { - using numext::abs; - using numext::sqrt; using RealScalar = typename Dest::RealScalar; using Scalar = typename Dest::Scalar; using VectorType = Matrix<Scalar, Dynamic, 1>; @@ -215,11 +213,11 @@ tautildeold = (zetaold - thetatildeold * tautildeold) / rhotildeold; const RealScalar taud = (zeta - thetatilde * tautildeold) / rhodold; d += betacheck * betacheck; - const RealScalar normr = sqrt(d + numext::abs2(betad - taud) + numext::abs2(betadd)); + const RealScalar normr = numext::sqrt<RealScalar>(d + numext::abs2(betad - taud) + numext::abs2(betadd)); // Estimate ||A||. normA2 += beta * beta; - const RealScalar normA = sqrt(normA2); + const RealScalar normA = numext::sqrt(normA2); normA2 += alpha * alpha; // Estimate cond(A). @@ -228,7 +226,7 @@ const RealScalar condA = numext::maxi(maxrbar, rhotemp) / numext::mini(minrbar, rhotemp); // Compute the norms needed for the stopping rules. - const RealScalar normAr = abs(zetabar); + const RealScalar normAr = numext::abs(zetabar); const RealScalar normx = dx.stableNorm(); const RealScalar test1 = normr / normb;
diff --git a/Eigen/src/IterativeLinearSolvers/LeastSquareConjugateGradient.h b/Eigen/src/IterativeLinearSolvers/LeastSquareConjugateGradient.h index 1e39928..2153e4a 100644 --- a/Eigen/src/IterativeLinearSolvers/LeastSquareConjugateGradient.h +++ b/Eigen/src/IterativeLinearSolvers/LeastSquareConjugateGradient.h
@@ -31,7 +31,6 @@ EIGEN_DONT_INLINE void least_square_conjugate_gradient(const MatrixType& mat, const Rhs& rhs, Dest& x, const Preconditioner& precond, Index& iters, typename Dest::RealScalar& tol_error) { - using std::sqrt; using RealScalar = typename Dest::RealScalar; using Scalar = typename Dest::Scalar; using VectorType = Matrix<Scalar, Dynamic, 1>;
diff --git a/Eigen/src/IterativeLinearSolvers/MINRES.h b/Eigen/src/IterativeLinearSolvers/MINRES.h index da132b3..27fddef 100644 --- a/Eigen/src/IterativeLinearSolvers/MINRES.h +++ b/Eigen/src/IterativeLinearSolvers/MINRES.h
@@ -32,7 +32,6 @@ template <typename MatrixType, typename Rhs, typename Dest, typename Preconditioner> EIGEN_DONT_INLINE void minres(const MatrixType& mat, const Rhs& rhs, Dest& x, const Preconditioner& precond, Index& iters, typename Dest::RealScalar& tol_error) { - using std::sqrt; using RealScalar = typename Dest::RealScalar; using Scalar = typename Dest::Scalar; using VectorType = Matrix<Scalar, Dynamic, 1>; @@ -69,7 +68,7 @@ VectorType w_new(precond.solve(v_new)); // initialize w_new RealScalar beta_new2(v_new.dot(w_new)); eigen_assert(beta_new2 >= 0.0 && "PRECONDITIONER IS NOT POSITIVE DEFINITE"); - RealScalar beta_new(sqrt(beta_new2)); + RealScalar beta_new(numext::sqrt(beta_new2)); const RealScalar beta_one(beta_new); // Initialize other variables RealScalar c(1.0); // the cosine of the Givens rotation @@ -105,7 +104,7 @@ w_new = precond.solve(v_new); // overwrite w_new beta_new2 = v_new.dot(w_new); // compute beta_new eigen_assert(beta_new2 >= 0.0 && "PRECONDITIONER IS NOT POSITIVE DEFINITE"); - beta_new = sqrt(beta_new2); // compute beta_new + beta_new = numext::sqrt(beta_new2); // compute beta_new // Givens rotation const RealScalar r2 = s * alpha + c * c_old * beta; // s, s_old, c and c_old are still from previous iteration
diff --git a/Eigen/src/IterativeLinearSolvers/Scaling.h b/Eigen/src/IterativeLinearSolvers/Scaling.h index f190c9f..898d2cf 100644 --- a/Eigen/src/IterativeLinearSolvers/Scaling.h +++ b/Eigen/src/IterativeLinearSolvers/Scaling.h
@@ -70,7 +70,6 @@ * \sa LeftScaling() RightScaling() */ void compute(const MatrixType& mat) { - using std::abs; int m = mat.rows(); int n = mat.cols(); eigen_assert((m > 0 && m == n) && "Please give a non - empty matrix"); @@ -89,9 +88,9 @@ Dc.setZero(); for (int k = 0; k < m_matrix.outerSize(); ++k) { for (typename MatrixType::InnerIterator it(m_matrix, k); it; ++it) { - if (Dr(it.row()) < abs(it.value())) Dr(it.row()) = abs(it.value()); + if (Dr(it.row()) < numext::abs(it.value())) Dr(it.row()) = numext::abs(it.value()); - if (Dc(it.col()) < abs(it.value())) Dc(it.col()) = abs(it.value()); + if (Dc(it.col()) < numext::abs(it.value())) Dc(it.col()) = numext::abs(it.value()); } } for (int i = 0; i < m; ++i) { @@ -114,9 +113,9 @@ for (typename MatrixType::InnerIterator it(m_matrix, k); it; ++it) { it.valueRef() = it.value() / (Dr(it.row()) * Dc(it.col())); // Accumulate the norms of the row and column vectors - if (DrRes(it.row()) < abs(it.value())) DrRes(it.row()) = abs(it.value()); + if (DrRes(it.row()) < numext::abs(it.value())) DrRes(it.row()) = numext::abs(it.value()); - if (DcRes(it.col()) < abs(it.value())) DcRes(it.col()) = abs(it.value()); + if (DcRes(it.col()) < numext::abs(it.value())) DcRes(it.col()) = numext::abs(it.value()); } } DrRes.array() = (1 - DrRes.array()).abs();
diff --git a/Eigen/src/Jacobi/Jacobi.h b/Eigen/src/Jacobi/Jacobi.h index d40ac70..8eb671a 100644 --- a/Eigen/src/Jacobi/Jacobi.h +++ b/Eigen/src/Jacobi/Jacobi.h
@@ -53,22 +53,15 @@ /** Concatenates two planar rotations */ EIGEN_DEVICE_FUNC JacobiRotation operator*(const JacobiRotation& other) { - using numext::conj; - return JacobiRotation(m_c * other.m_c - conj(m_s) * other.m_s, - conj(m_c * conj(other.m_s) + conj(m_s) * conj(other.m_c))); + return JacobiRotation(m_c * other.m_c - numext::conj(m_s) * other.m_s, + numext::conj(m_c * numext::conj(other.m_s) + numext::conj(m_s) * numext::conj(other.m_c))); } /** Returns the transposed transformation */ - EIGEN_DEVICE_FUNC JacobiRotation transpose() const { - using numext::conj; - return JacobiRotation(m_c, -conj(m_s)); - } + EIGEN_DEVICE_FUNC JacobiRotation transpose() const { return JacobiRotation(m_c, -numext::conj(m_s)); } /** Returns the adjoint transformation */ - EIGEN_DEVICE_FUNC JacobiRotation adjoint() const { - using numext::conj; - return JacobiRotation(conj(m_c), -m_s); - } + EIGEN_DEVICE_FUNC JacobiRotation adjoint() const { return JacobiRotation(numext::conj(m_c), -m_s); } template <typename Derived> EIGEN_DEVICE_FUNC bool makeJacobi(const MatrixBase<Derived>&, Index p, Index q); @@ -93,10 +86,7 @@ */ template <typename Scalar> EIGEN_DEVICE_FUNC bool JacobiRotation<Scalar>::makeJacobi(const RealScalar& x, const Scalar& y, const RealScalar& z) { - using std::abs; - using std::sqrt; - - const RealScalar abs_y = abs(y); + const RealScalar abs_y = numext::abs(y); const RealScalar deno = RealScalar(2) * abs_y; if (deno < (std::numeric_limits<RealScalar>::min)()) { m_c = Scalar(1); @@ -104,7 +94,7 @@ return false; } else { const RealScalar delta = x - z; - const RealScalar abs_delta = abs(delta); + const RealScalar abs_delta = numext::abs(delta); // |t| = 1 / (|tau| + sqrt(1 + tau^2)), where tau = (x - z) / (2 * |y|). // Scale this numerator/denominator pair by min(1, 1/|tau|), taking scale = 1 at tau = 0. // Then numerator <= 1 and denominator <= 1 + sqrt(2); only a ratio <= 1 is squared. @@ -113,8 +103,9 @@ const bool large_delta = abs_delta > deno; const RealScalar ratio = (large_delta ? deno : abs_delta) / (large_delta ? abs_delta : deno); const RealScalar numerator = large_delta ? ratio : RealScalar(1); - const RealScalar denominator = sqrt(RealScalar(1) + numext::abs2(ratio)) + (large_delta ? RealScalar(1) : ratio); - const RealScalar n = RealScalar(1) / sqrt(numext::abs2(numerator) + numext::abs2(denominator)); + const RealScalar denominator = + numext::sqrt<RealScalar>(RealScalar(1) + numext::abs2(ratio)) + (large_delta ? RealScalar(1) : ratio); + const RealScalar n = RealScalar(1) / numext::sqrt<RealScalar>(numext::abs2(numerator) + numext::abs2(denominator)); const RealScalar sine = numerator * n; const RealScalar sign_t = delta > RealScalar(0) ? RealScalar(1) : RealScalar(-1); EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::IsComplex) { @@ -169,18 +160,17 @@ // specialization for complexes template <typename Scalar> EIGEN_DEVICE_FUNC void JacobiRotation<Scalar>::makeGivens(const Scalar& p, const Scalar& q, Scalar* r, std::true_type) { - using numext::conj; - using std::abs; - using std::sqrt; - if (q == Scalar(0)) { m_c = numext::real(p) < 0 ? Scalar(-1) : Scalar(1); m_s = 0; if (r) *r = m_c * p; } else if (p == Scalar(0)) { + // Scale by max(|Re q|, |Im q|) first: for a subnormal q, abs(q) has only a few bits and q / abs(q) is not + // unimodular. Unlike |q|_1, the max norm cannot overflow. + const Scalar qn = q / numext::maxi(numext::abs(numext::real(q)), numext::abs(numext::imag(q))); m_c = 0; - m_s = -q / abs(q); - if (r) *r = abs(q); + m_s = -qn / numext::abs(qn); + if (r) *r = numext::abs(q); } else { RealScalar p1 = numext::norm1(p); RealScalar q1 = numext::norm1(q); @@ -190,26 +180,29 @@ Scalar qs = q / p1; RealScalar q2 = numext::abs2(qs); - RealScalar u = sqrt(RealScalar(1) + q2 / p2); + RealScalar u = numext::sqrt<RealScalar>(RealScalar(1) + q2 / p2); if (numext::real(p) < RealScalar(0)) u = -u; m_c = Scalar(1) / u; - m_s = -qs * conj(ps) * (m_c / p2); + m_s = -qs * numext::conj(ps) * (m_c / p2); if (r) *r = p * u; } else { + // Divide by u = sqrt(p2 + q2), i.e. in the quantities scaled by 1/q1: dividing p and q by q1 u loses all but a + // few bits when q1 is subnormal, and the rotation is no longer unitary. The phase of p comes from p / p1, which + // is O(1) even when p is subnormal or p / q1 underflows. Scalar ps = p / q1; RealScalar p2 = numext::abs2(ps); Scalar qs = q / q1; RealScalar q2 = numext::abs2(qs); - RealScalar u = q1 * sqrt(p2 + q2); + RealScalar u = numext::sqrt<RealScalar>(p2 + q2); if (numext::real(p) < RealScalar(0)) u = -u; - p1 = abs(p); - ps = p / p1; - m_c = p1 / u; - m_s = -conj(ps) * (q / u); - if (r) *r = ps * u; + const Scalar pn = p / p1; + const Scalar phase = pn / numext::abs(pn); + m_c = numext::abs(ps) / u; + m_s = -numext::conj(phase) * (qs / u); + if (r) *r = phase * (q1 * u); } } } @@ -218,28 +211,26 @@ template <typename Scalar> EIGEN_DEVICE_FUNC void JacobiRotation<Scalar>::makeGivens(const Scalar& p, const Scalar& q, Scalar* r, std::false_type) { - using std::abs; - using std::sqrt; if (numext::is_exactly_zero_no_flush(q)) { // DAZ can hide the sign; MSVC's non-intrinsic abs(float) can flush the // result through a float->double->float conversion even with FTZ alone. - if (abs(p) < (std::numeric_limits<Scalar>::min)() && !numext::is_exactly_zero_no_flush(p)) { + if (numext::abs(p) < (std::numeric_limits<Scalar>::min)() && !numext::is_exactly_zero_no_flush(p)) { makeGivensScaled(p, q, r); return; } m_c = p < Scalar(0) ? Scalar(-1) : Scalar(1); m_s = Scalar(0); - if (r) *r = abs(p); + if (r) *r = numext::abs(p); return; } if (numext::is_exactly_zero_no_flush(p)) { - if (abs(q) < (std::numeric_limits<Scalar>::min)()) { + if (numext::abs(q) < (std::numeric_limits<Scalar>::min)()) { makeGivensScaled(p, q, r); return; } m_c = Scalar(0); m_s = q < Scalar(0) ? Scalar(1) : Scalar(-1); - if (r) *r = abs(q); + if (r) *r = numext::abs(q); return; } @@ -252,10 +243,17 @@ // existing Eigen sign convention (r >= 0, sign carried in c). const Scalar safmin = (std::numeric_limits<Scalar>::min)(); const Scalar safmax = Scalar(1) / safmin; +#if defined(EIGEN_SYCL_DEVICE_ONLY) + const Scalar rtmin = numext::sqrt(safmin); + const Scalar rtmax = numext::sqrt<Scalar>(safmax / Scalar(2)); +#else + // GCC does not fold the SSE numext::sqrt intrinsic for the constant range thresholds below. + EIGEN_USING_STD(sqrt); const Scalar rtmin = sqrt(safmin); const Scalar rtmax = sqrt(safmax / Scalar(2)); - const Scalar abs_p = abs(p); - const Scalar abs_q = abs(q); +#endif + const Scalar abs_p = numext::abs(p); + const Scalar abs_q = numext::abs(q); const Scalar mx = numext::maxi(abs_p, abs_q); const Scalar mn = numext::mini(abs_p, abs_q); @@ -263,14 +261,14 @@ // Safe range: existing direct formulas are stable. if (abs_p > abs_q) { Scalar t = q / p; - Scalar u = sqrt(Scalar(1) + numext::abs2(t)); + Scalar u = numext::sqrt<Scalar>(Scalar(1) + numext::abs2(t)); if (p < Scalar(0)) u = -u; m_c = Scalar(1) / u; m_s = -t * m_c; if (r) *r = p * u; } else { Scalar t = p / q; - Scalar u = sqrt(Scalar(1) + numext::abs2(t)); + Scalar u = numext::sqrt<Scalar>(Scalar(1) + numext::abs2(t)); if (q < Scalar(0)) u = -u; m_s = -Scalar(1) / u; m_c = -t * m_s;
diff --git a/Eigen/src/LU/FullPivLU.h b/Eigen/src/LU/FullPivLU.h index bf7a9a0..3112fbe 100644 --- a/Eigen/src/LU/FullPivLU.h +++ b/Eigen/src/LU/FullPivLU.h
@@ -335,10 +335,7 @@ Scalar signDeterminant() const; /** \returns the absolute value of the i-th pivot coefficient (for RankRevealingBase). */ - RealScalar pivotCoeff(Index i) const { - using std::abs; - return abs(m_lu.coeff(i, i)); - } + RealScalar pivotCoeff(Index i) const { return numext::abs(m_lu.coeff(i, i)); } /** \returns the inverse of the matrix of which *this is the LU decomposition. *
diff --git a/Eigen/src/LU/InverseImpl.h b/Eigen/src/LU/InverseImpl.h index aa4e05d..2295d0e 100644 --- a/Eigen/src/LU/InverseImpl.h +++ b/Eigen/src/LU/InverseImpl.h
@@ -53,9 +53,8 @@ const typename MatrixType::RealScalar& absDeterminantThreshold, ResultType& result, typename ResultType::Scalar& determinant, bool& invertible) { - using std::abs; determinant = matrix.coeff(0, 0); - invertible = abs(determinant) > absDeterminantThreshold; + invertible = numext::abs(determinant) > absDeterminantThreshold; if (invertible) result.coeffRef(0, 0) = typename ResultType::Scalar(1) / determinant; } }; @@ -90,10 +89,9 @@ const typename MatrixType::RealScalar& absDeterminantThreshold, ResultType& inverse, typename ResultType::Scalar& determinant, bool& invertible) { - using std::abs; using Scalar = typename ResultType::Scalar; determinant = matrix.determinant(); - invertible = abs(determinant) > absDeterminantThreshold; + invertible = numext::abs(determinant) > absDeterminantThreshold; if (!invertible) return; const Scalar invdet = Scalar(1) / determinant; compute_inverse_size2_helper(matrix, invdet, inverse); @@ -212,9 +210,8 @@ const typename MatrixType::RealScalar& absDeterminantThreshold, ResultType& inverse, typename ResultType::Scalar& determinant, bool& invertible) { - using std::abs; determinant = matrix.determinant(); - invertible = abs(determinant) > absDeterminantThreshold; + invertible = numext::abs(determinant) > absDeterminantThreshold; if (invertible && extract_data(matrix) != extract_data(inverse)) { compute_inverse<MatrixType, ResultType>::run(matrix, inverse); } else if (invertible) {
diff --git a/Eigen/src/OrderingMethods/Amd.h b/Eigen/src/OrderingMethods/Amd.h index 3027186..2eba79e 100644 --- a/Eigen/src/OrderingMethods/Amd.h +++ b/Eigen/src/OrderingMethods/Amd.h
@@ -80,13 +80,11 @@ template <typename Scalar, typename StorageIndex> void minimum_degree_ordering(SparseMatrix<Scalar, ColMajor, StorageIndex>& C, PermutationMatrix<Dynamic, Dynamic, StorageIndex>& perm) { - using std::sqrt; - StorageIndex d, dk, dext, lemax = 0, e, elenk, eln, i, j, k, k1, k2, k3, jlast, ln, dense, nzmax, mindeg = 0, nvi, nvj, nvk, mark, wnvi, ok, nel = 0, p, p1, p2, p3, p4, pj, pk, pk1, pk2, pn, q, t, h; StorageIndex n = StorageIndex(C.cols()); - dense = std::max<StorageIndex>(16, StorageIndex(10 * sqrt(double(n)))); /* find dense threshold */ + dense = std::max<StorageIndex>(16, StorageIndex(10 * numext::sqrt(double(n)))); /* find dense threshold */ dense = (std::min)(n - 2, dense); StorageIndex cnz = StorageIndex(C.nonZeros());
diff --git a/Eigen/src/PaStiXSupport/PaStiXSupport.h b/Eigen/src/PaStiXSupport/PaStiXSupport.h index 52b893e..7cf7f8b 100644 --- a/Eigen/src/PaStiXSupport/PaStiXSupport.h +++ b/Eigen/src/PaStiXSupport/PaStiXSupport.h
@@ -7,6 +7,7 @@ // Public License v. 2.0. If a copy of the MPL was not distributed // with this file, You can obtain one at http://mozilla.org/MPL/2.0/. // SPDX-License-Identifier: MPL-2.0 +// SPDX-FileCopyrightText: The Eigen Authors #ifndef EIGEN_PASTIXSUPPORT_H #define EIGEN_PASTIXSUPPORT_H @@ -16,14 +17,6 @@ namespace Eigen { -#if defined(DCOMPLEX) -#define PASTIX_COMPLEX COMPLEX -#define PASTIX_DCOMPLEX DCOMPLEX -#else -#define PASTIX_COMPLEX std::complex<float> -#define PASTIX_DCOMPLEX std::complex<double> -#endif - /** \ingroup PaStiXSupport_Module * \brief Interface to the PaStix solver * @@ -68,299 +61,154 @@ typedef typename MatrixType_::StorageIndex StorageIndex; }; -inline void eigen_pastix(pastix_data_t **pastix_data, int pastix_comm, int n, int *ptr, int *idx, float *vals, - int *perm, int *invp, float *x, int nbrhs, int *iparm, double *dparm) { - if (n == 0) { - ptr = nullptr; - idx = nullptr; - vals = nullptr; - } - if (nbrhs == 0) { - x = nullptr; - nbrhs = 1; - } - s_pastix(pastix_data, pastix_comm, n, ptr, idx, vals, perm, invp, x, nbrhs, iparm, dparm); -} - -inline void eigen_pastix(pastix_data_t **pastix_data, int pastix_comm, int n, int *ptr, int *idx, double *vals, - int *perm, int *invp, double *x, int nbrhs, int *iparm, double *dparm) { - if (n == 0) { - ptr = nullptr; - idx = nullptr; - vals = nullptr; - } - if (nbrhs == 0) { - x = nullptr; - nbrhs = 1; - } - d_pastix(pastix_data, pastix_comm, n, ptr, idx, vals, perm, invp, x, nbrhs, iparm, dparm); -} - -inline void eigen_pastix(pastix_data_t **pastix_data, int pastix_comm, int n, int *ptr, int *idx, - std::complex<float> *vals, int *perm, int *invp, std::complex<float> *x, int nbrhs, int *iparm, - double *dparm) { - if (n == 0) { - ptr = nullptr; - idx = nullptr; - vals = nullptr; - } - if (nbrhs == 0) { - x = nullptr; - nbrhs = 1; - } - c_pastix(pastix_data, pastix_comm, n, ptr, idx, reinterpret_cast<PASTIX_COMPLEX *>(vals), perm, invp, - reinterpret_cast<PASTIX_COMPLEX *>(x), nbrhs, iparm, dparm); -} - -inline void eigen_pastix(pastix_data_t **pastix_data, int pastix_comm, int n, int *ptr, int *idx, - std::complex<double> *vals, int *perm, int *invp, std::complex<double> *x, int nbrhs, - int *iparm, double *dparm) { - if (n == 0) { - ptr = nullptr; - idx = nullptr; - vals = nullptr; - } - if (nbrhs == 0) { - x = nullptr; - nbrhs = 1; - } - z_pastix(pastix_data, pastix_comm, n, ptr, idx, reinterpret_cast<PASTIX_DCOMPLEX *>(vals), perm, invp, - reinterpret_cast<PASTIX_DCOMPLEX *>(x), nbrhs, iparm, dparm); -} - -// Convert the matrix to Fortran-style Numbering -template <typename MatrixType> -void c_to_fortran_numbering(MatrixType &mat) { - if (!(mat.outerIndexPtr()[0])) { - int i; - for (i = 0; i <= mat.rows(); ++i) ++mat.outerIndexPtr()[i]; - for (i = 0; i < mat.nonZeros(); ++i) ++mat.innerIndexPtr()[i]; - } -} } // namespace internal -// This is the base class to interface with PaStiX functions. -// Users should not use this class directly. -template <class Derived> +// PaStiX 6 task interface: +// https://solverstack.gitlabpages.inria.fr/pastix/group__pastix__users.html +template <typename Derived> class PastixBase : public SparseSolverBase<Derived> { protected: - typedef SparseSolverBase<Derived> Base; - using Base::derived; + using Base = SparseSolverBase<Derived>; using Base::m_isInitialized; public: using Base::_solve_impl; + using MatrixType = typename internal::pastix_traits<Derived>::MatrixType; + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename MatrixType::RealScalar; + using StorageIndex = typename MatrixType::StorageIndex; + using Vector = Matrix<Scalar, Dynamic, 1>; + using ColSpMatrix = SparseMatrix<Scalar, ColMajor, pastix_int_t>; + static constexpr int ColsAtCompileTime = MatrixType::ColsAtCompileTime; + static constexpr int MaxColsAtCompileTime = MatrixType::MaxColsAtCompileTime; - typedef typename internal::pastix_traits<Derived>::MatrixType MatrixType_; - typedef MatrixType_ MatrixType; - typedef typename MatrixType::Scalar Scalar; - typedef typename MatrixType::RealScalar RealScalar; - typedef typename MatrixType::StorageIndex StorageIndex; - typedef Matrix<Scalar, Dynamic, 1> Vector; - typedef SparseMatrix<Scalar, ColMajor> ColSpMatrix; - enum { ColsAtCompileTime = MatrixType::ColsAtCompileTime, MaxColsAtCompileTime = MatrixType::MaxColsAtCompileTime }; - - public: - PastixBase() : m_initisOk(false), m_analysisIsOk(false), m_factorizationIsOk(false), m_pastixdata(0), m_size(0) { - init(); + PastixBase() { + EIGEN_STATIC_ASSERT( + (std::is_same<Scalar, float>::value || std::is_same<Scalar, double>::value || + std::is_same<Scalar, std::complex<float>>::value || std::is_same<Scalar, std::complex<double>>::value), + THIS_TYPE_IS_NOT_SUPPORTED) + pastixInitParam(m_iparm.data(), m_dparm.data()); + m_iparm(IPARM_VERBOSE) = PastixVerboseNot; } ~PastixBase() { clean(); } - template <typename Rhs, typename Dest> - bool _solve_impl(const MatrixBase<Rhs> &b, MatrixBase<Dest> &x) const; - - /** Returns a reference to the integer vector IPARM of PaStiX parameters - * to modify the default parameters. - * The statistics related to the different phases of factorization and solve are saved here as well - * \sa analyzePattern() factorize() - */ - Array<StorageIndex, IPARM_SIZE, 1> &iparm() { return m_iparm; } - - /** Return a reference to a particular index parameter of the IPARM vector - * \sa iparm() - */ - - int &iparm(int idxparam) { return m_iparm(idxparam); } - - /** Returns a reference to the double vector DPARM of PaStiX parameters - * The statistics related to the different phases of factorization and solve are saved here as well - * \sa analyzePattern() factorize() - */ + /** Returns the integer PaStiX parameters. Set input parameters before analyzePattern(). */ + Array<pastix_int_t, IPARM_SIZE, 1> &iparm() { return m_iparm; } + /** Returns the integer PaStiX parameter at \p parameter. */ + pastix_int_t &iparm(int parameter) { return m_iparm(parameter); } + /** Returns the floating-point PaStiX parameters. */ Array<double, DPARM_SIZE, 1> &dparm() { return m_dparm; } - - /** Return a reference to a particular index parameter of the DPARM vector - * \sa dparm() + /** Returns the floating-point PaStiX parameter at \p parameter. */ + double &dparm(int parameter) { return m_dparm(parameter); } + Index rows() const { return m_matrix.rows(); } + Index cols() const { return m_matrix.cols(); } + /** Returns Success if the last task succeeded, NumericalIssue for a task error or a non-finite solution, + * and InvalidInput before computation. PaStiX may not detect singularity during factorization. + * Some library errors terminate the process instead of returning an error code; in particular, + * PastixLLT requires a positive definite matrix. */ - double &dparm(int idxparam) { return m_dparm(idxparam); } + ComputationInfo info() const { return m_info; } - inline Index cols() const { return m_size; } - inline Index rows() const { return m_size; } - - /** \brief Reports whether previous computation was successful. - * - * \returns \c Success if computation was successful, - * \c NumericalIssue if the PaStiX reports a problem - * \c InvalidInput if the input matrix is invalid - * - * \sa iparm() - */ - ComputationInfo info() const { - eigen_assert(m_isInitialized && "Decomposition is not initialized."); - return m_info; + template <typename Rhs, typename Dest> + bool _solve_impl(const MatrixBase<Rhs> &b, MatrixBase<Dest> &x) const { + eigen_assert(m_isInitialized && "The matrix should be factorized first"); + eigen_assert(b.rows() == rows()); + // PaStiX overwrites a contiguous, column-major RHS. Refine needs the original RHS as well. + Matrix<Scalar, Dynamic, Dynamic, ColMajor> solution = b; + Matrix<Scalar, Dynamic, Dynamic, ColMajor> rhs = b; + const pastix_int_t n = internal::convert_index<pastix_int_t>(rows()); + const pastix_int_t nrhs = internal::convert_index<pastix_int_t>(b.cols()); + int error = 0; + if (n != 0 && nrhs != 0) { +#if PASTIX_VERSION_MINOR >= 3 + error = pastix_task_solve(m_pastixdata, n, nrhs, solution.data(), n); +#else + error = pastix_task_solve(m_pastixdata, nrhs, solution.data(), n); +#endif + // PaStiX refinement accepts only one right-hand side, even when solve accepts several. + for (pastix_int_t col = 0; error == PASTIX_SUCCESS && col < nrhs; ++col) { + if (!solution.col(col).allFinite()) break; + error = pastix_task_refine(m_pastixdata, n, 1, rhs.col(col).data(), n, solution.col(col).data(), n); + } + } + m_info = error == PASTIX_SUCCESS && solution.allFinite() ? Success : NumericalIssue; + x = solution; + return m_info == Success; } protected: - // Initialize the Pastix data structure, check the matrix - void init(); - - // Compute the ordering and the symbolic factorization - void analyzePattern(ColSpMatrix &mat); - - // Compute the numerical factorization - void factorize(ColSpMatrix &mat); - - // Free all the data allocated by Pastix void clean() { - eigen_assert(m_initisOk && "The Pastix structure should be allocated first"); - m_iparm(IPARM_START_TASK) = API_TASK_CLEAN; - m_iparm(IPARM_END_TASK) = API_TASK_CLEAN; - internal::eigen_pastix(&m_pastixdata, MPI_COMM_WORLD, 0, 0, 0, (Scalar *)0, m_perm.data(), m_invp.data(), 0, 0, - m_iparm.data(), m_dparm.data()); - } - - void compute(ColSpMatrix &mat); - - int m_initisOk; - int m_analysisIsOk; - int m_factorizationIsOk; - mutable ComputationInfo m_info; - mutable pastix_data_t *m_pastixdata; // Data structure for pastix - mutable int m_comm; // The MPI communicator identifier - mutable Array<int, IPARM_SIZE, 1> m_iparm; // integer vector for the input parameters - mutable Array<double, DPARM_SIZE, 1> m_dparm; // Scalar vector for the input parameters - mutable Matrix<StorageIndex, Dynamic, 1> m_perm; // Permutation vector - mutable Matrix<StorageIndex, Dynamic, 1> m_invp; // Inverse permutation vector - mutable int m_size; // Size of the matrix -}; - -/** Initialize the PaStiX data structure. - *A first call to this function fills iparm and dparm with the default PaStiX parameters - * \sa iparm() dparm() - */ -template <class Derived> -void PastixBase<Derived>::init() { - m_size = 0; - m_iparm.setZero(IPARM_SIZE); - m_dparm.setZero(DPARM_SIZE); - - m_iparm(IPARM_MODIFY_PARAMETER) = API_NO; - pastix(&m_pastixdata, MPI_COMM_WORLD, 0, 0, 0, 0, 0, 0, 0, 1, m_iparm.data(), m_dparm.data()); - - m_iparm[IPARM_MATRIX_VERIFICATION] = API_NO; - m_iparm[IPARM_VERBOSE] = API_VERBOSE_NOT; - m_iparm[IPARM_ORDERING] = API_ORDER_SCOTCH; - m_iparm[IPARM_INCOMPLETE] = API_NO; - m_iparm[IPARM_OOC_LIMIT] = 2000; - m_iparm[IPARM_RHS_MAKING] = API_RHS_B; - m_iparm(IPARM_MATRIX_VERIFICATION) = API_NO; - - m_iparm(IPARM_START_TASK) = API_TASK_INIT; - m_iparm(IPARM_END_TASK) = API_TASK_INIT; - internal::eigen_pastix(&m_pastixdata, MPI_COMM_WORLD, 0, 0, 0, (Scalar *)0, 0, 0, 0, 0, m_iparm.data(), - m_dparm.data()); - - // Check the returned error - if (m_iparm(IPARM_ERROR_NUMBER)) { - m_info = InvalidInput; - m_initisOk = false; - } else { - m_info = Success; - m_initisOk = true; - } -} - -template <class Derived> -void PastixBase<Derived>::compute(ColSpMatrix &mat) { - eigen_assert(mat.rows() == mat.cols() && "The input matrix should be squared"); - - analyzePattern(mat); - factorize(mat); - - m_iparm(IPARM_MATRIX_VERIFICATION) = API_NO; -} - -template <class Derived> -void PastixBase<Derived>::analyzePattern(ColSpMatrix &mat) { - eigen_assert(m_initisOk && "The initialization of PaSTiX failed"); - - // clean previous calls - if (m_size > 0) clean(); - - m_size = internal::convert_index<int>(mat.rows()); - m_perm.resize(m_size); - m_invp.resize(m_size); - - m_iparm(IPARM_START_TASK) = API_TASK_ORDERING; - m_iparm(IPARM_END_TASK) = API_TASK_ANALYSE; - internal::eigen_pastix(&m_pastixdata, MPI_COMM_WORLD, m_size, mat.outerIndexPtr(), mat.innerIndexPtr(), - mat.valuePtr(), m_perm.data(), m_invp.data(), 0, 0, m_iparm.data(), m_dparm.data()); - - // Check the returned error - if (m_iparm(IPARM_ERROR_NUMBER)) { - m_info = NumericalIssue; - m_analysisIsOk = false; - } else { - m_info = Success; - m_analysisIsOk = true; - } -} - -template <class Derived> -void PastixBase<Derived>::factorize(ColSpMatrix &mat) { - eigen_assert(m_analysisIsOk && "The analysis phase should be called before the factorization phase"); - m_iparm(IPARM_START_TASK) = API_TASK_NUMFACT; - m_iparm(IPARM_END_TASK) = API_TASK_NUMFACT; - m_size = internal::convert_index<int>(mat.rows()); - - internal::eigen_pastix(&m_pastixdata, MPI_COMM_WORLD, m_size, mat.outerIndexPtr(), mat.innerIndexPtr(), - mat.valuePtr(), m_perm.data(), m_invp.data(), 0, 0, m_iparm.data(), m_dparm.data()); - - // Check the returned error - if (m_iparm(IPARM_ERROR_NUMBER)) { - m_info = NumericalIssue; - m_factorizationIsOk = false; + if (m_pastixdata) pastixFinalize(&m_pastixdata); m_isInitialized = false; - } else { - m_info = Success; - m_factorizationIsOk = true; - m_isInitialized = true; - } -} - -/* Solve the system */ -template <typename Base> -template <typename Rhs, typename Dest> -bool PastixBase<Base>::_solve_impl(const MatrixBase<Rhs> &b, MatrixBase<Dest> &x) const { - eigen_assert(m_isInitialized && "The matrix should be factorized first"); - EIGEN_STATIC_ASSERT((Dest::Flags & RowMajorBit) == 0, THIS_METHOD_IS_ONLY_FOR_COLUMN_MAJOR_MATRICES); - int rhs = 1; - - x = b; /* on return, x is overwritten by the computed solution */ - - for (int i = 0; i < b.cols(); i++) { - m_iparm[IPARM_START_TASK] = API_TASK_SOLVE; - m_iparm[IPARM_END_TASK] = API_TASK_REFINE; - - internal::eigen_pastix(&m_pastixdata, MPI_COMM_WORLD, internal::convert_index<int>(x.rows()), 0, 0, 0, - m_perm.data(), m_invp.data(), &x(0, i), rhs, m_iparm.data(), m_dparm.data()); + m_analysisIsOk = false; } - // Check the returned error - m_info = m_iparm(IPARM_ERROR_NUMBER) == 0 ? Success : NumericalIssue; + void compute(ColSpMatrix &matrix) { + analyzePattern(matrix); + if (m_analysisIsOk) factorize(m_matrix); + } - return m_iparm(IPARM_ERROR_NUMBER) == 0; -} + void analyzePattern(ColSpMatrix &matrix) { + eigen_assert(matrix.rows() == matrix.cols() && "The input matrix should be square"); + clean(); + m_matrix.swap(matrix); + if (rows() == 0) { + m_analysisIsOk = true; + m_info = Success; + return; + } + setMatrix(); + pastixInit(&m_pastixdata, MPI_COMM_WORLD, m_iparm.data(), m_dparm.data()); + const int error = pastix_task_analyze(m_pastixdata, &m_spm); + m_analysisIsOk = error == PASTIX_SUCCESS; + m_info = m_analysisIsOk ? Success : NumericalIssue; + } + + void factorize(ColSpMatrix &matrix) { + eigen_assert(m_analysisIsOk && "The analysis phase should be called before the factorization phase"); + eigen_assert(matrix.rows() == rows() && matrix.cols() == cols()); + if (&matrix != &m_matrix) m_matrix.swap(matrix); + if (rows() == 0) { + m_isInitialized = true; + m_info = Success; + return; + } + setMatrix(); + const int error = pastix_task_numfact(m_pastixdata, &m_spm); + m_isInitialized = error == PASTIX_SUCCESS; + m_info = m_isInitialized ? Success : NumericalIssue; + } + + Array<pastix_int_t, IPARM_SIZE, 1> m_iparm; + Array<double, DPARM_SIZE, 1> m_dparm; + + private: + void setMatrix() { + // Eigen owns the CSC arrays: do not pass this descriptor to spmExit(). + m_iparm(IPARM_FREE_CSCUSER) = 0; + spmInit(&m_spm); + m_spm.mtxtype = static_cast<spm_mtxtype_t>(m_iparm(IPARM_MTX_TYPE)); + m_spm.flttype = NumTraits<Scalar>::IsComplex + ? (std::is_same<RealScalar, float>::value ? SpmComplex32 : SpmComplex64) + : (std::is_same<Scalar, float>::value ? SpmFloat : SpmDouble); + m_spm.fmttype = SpmCSC; + m_spm.baseval = 0; + m_spm.n = internal::convert_index<spm_int_t>(m_matrix.rows()); + m_spm.nnz = internal::convert_index<spm_int_t>(m_matrix.nonZeros()); + m_spm.dof = 1; + m_spm.colptr = m_matrix.outerIndexPtr(); + m_spm.rowptr = m_matrix.innerIndexPtr(); + m_spm.values = m_matrix.valuePtr(); + spmUpdateComputedFields(&m_spm); + } + + ColSpMatrix m_matrix; + spmatrix_t m_spm; + pastix_data_t *m_pastixdata = nullptr; + bool m_analysisIsOk = false; + mutable ComputationInfo m_info = InvalidInput; +}; /** \ingroup PaStiXSupport_Module * \class PastixLU @@ -435,8 +283,8 @@ protected: void init() { m_structureIsUptodate = false; - m_iparm(IPARM_SYM) = API_SYM_NO; - m_iparm(IPARM_FACTORIZATION) = API_FACT_LU; + m_iparm(IPARM_MTX_TYPE) = PastixGeneral; + m_iparm(IPARM_FACTORIZATION) = PastixFactGETRF; } void grabMatrix(const MatrixType &matrix, ColSpMatrix &out) { @@ -454,9 +302,10 @@ m_structureIsUptodate = true; } - out = m_transposedStructure + matrix; + const ColSpMatrix &converted = matrix; + out = m_transposedStructure + converted; } - internal::c_to_fortran_numbering(out); + out.makeCompressed(); } using Base::m_dparm; @@ -470,9 +319,9 @@ * \class PastixLLT * \brief A sparse direct supernodal Cholesky (LLT) factorization and solver based on the PaStiX library * - * This class is used to solve the linear systems A.X = B via a LL^T supernodal Cholesky factorization - * available in the PaStiX library. The matrix A should be symmetric and positive definite - * WARNING Selfadjoint complex matrices are not supported in the current version of PaStiX + * This class solves A.X = B via a supernodal LL^T (real) or LL^* (complex) Cholesky factorization. + * The matrix A must be symmetric (real) or Hermitian (complex) and positive definite. + * \warning PaStiX can terminate the process on non-positive-definite input rather than return an error. * The vectors or matrices X and B can be either dense or sparse * * \tparam MatrixType the type of the sparse matrix A, it must be a SparseMatrix<> @@ -530,25 +379,25 @@ using Base::m_iparm; void init() { - m_iparm(IPARM_SYM) = API_SYM_YES; - m_iparm(IPARM_FACTORIZATION) = API_FACT_LLT; + m_iparm(IPARM_MTX_TYPE) = NumTraits<typename Base::Scalar>::IsComplex ? PastixHermitian : PastixSymmetric; + m_iparm(IPARM_FACTORIZATION) = PastixFactPOTRF; } void grabMatrix(const MatrixType &matrix, ColSpMatrix &out) { out.resize(matrix.rows(), matrix.cols()); // Pastix supports only lower, column-major matrices - out.template selfadjointView<Lower>() = matrix.template selfadjointView<UpLo>(); - internal::c_to_fortran_numbering(out); + const ColSpMatrix &converted = matrix; + out.template selfadjointView<Lower>() = converted.template selfadjointView<UpLo>(); + out.makeCompressed(); } }; /** \ingroup PaStiXSupport_Module * \class PastixLDLT - * \brief A sparse direct supernodal Cholesky (LLT) factorization and solver based on the PaStiX library + * \brief A sparse direct supernodal LDLT factorization and solver based on the PaStiX library * - * This class is used to solve the linear systems A.X = B via a LDL^T supernodal Cholesky factorization - * available in the PaStiX library. The matrix A should be symmetric and positive definite - * WARNING Selfadjoint complex matrices are not supported in the current version of PaStiX + * This class solves A.X = B via a supernodal LDL^T (real) or LDL^* (complex) factorization. + * The matrix A should be symmetric (real) or Hermitian (complex) and positive definite. * The vectors or matrices X and B can be either dense or sparse * * \tparam MatrixType the type of the sparse matrix A, it must be a SparseMatrix<> @@ -606,21 +455,19 @@ using Base::m_iparm; void init() { - m_iparm(IPARM_SYM) = API_SYM_YES; - m_iparm(IPARM_FACTORIZATION) = API_FACT_LDLT; + m_iparm(IPARM_MTX_TYPE) = NumTraits<typename Base::Scalar>::IsComplex ? PastixHermitian : PastixSymmetric; + m_iparm(IPARM_FACTORIZATION) = NumTraits<typename Base::Scalar>::IsComplex ? PastixFactHETRF : PastixFactSYTRF; } void grabMatrix(const MatrixType &matrix, ColSpMatrix &out) { // Pastix supports only lower, column-major matrices out.resize(matrix.rows(), matrix.cols()); - out.template selfadjointView<Lower>() = matrix.template selfadjointView<UpLo>(); - internal::c_to_fortran_numbering(out); + const ColSpMatrix &converted = matrix; + out.template selfadjointView<Lower>() = converted.template selfadjointView<UpLo>(); + out.makeCompressed(); } }; } // end namespace Eigen -#undef PASTIX_COMPLEX -#undef PASTIX_DCOMPLEX - #endif
diff --git a/Eigen/src/QR/ColPivHouseholderQR.h b/Eigen/src/QR/ColPivHouseholderQR.h index dca45f7..fd53e1d 100644 --- a/Eigen/src/QR/ColPivHouseholderQR.h +++ b/Eigen/src/QR/ColPivHouseholderQR.h
@@ -264,10 +264,7 @@ typename MatrixType::Scalar signDeterminant() const; /** \returns the absolute value of the i-th pivot coefficient (for RankRevealingBase). */ - RealScalar pivotCoeff(Index i) const { - using std::abs; - return abs(m_qr.coeff(i, i)); - } + RealScalar pivotCoeff(Index i) const { return numext::abs(m_qr.coeff(i, i)); } /** \returns the inverse of the matrix of which *this is the QR decomposition. * @@ -336,10 +333,9 @@ template <typename MatrixType, typename PermutationIndex> typename MatrixType::RealScalar ColPivHouseholderQR<MatrixType, PermutationIndex>::absDeterminant() const { - using std::abs; eigen_assert(m_isInitialized && "ColPivHouseholderQR is not initialized."); EIGEN_INTERNAL_ASSERT_SQUARE_MSG(m_qr, "You can't take the determinant of a non-square matrix!"); - return isInjective() ? abs(m_qr.diagonal().prod()) : RealScalar(0); + return isInjective() ? numext::abs(m_qr.diagonal().prod()) : RealScalar(0); } template <typename MatrixType, typename PermutationIndex> @@ -377,8 +373,6 @@ void ColPivHouseholderQR<MatrixType, PermutationIndex>::computeInPlace() { eigen_assert(m_qr.cols() <= NumTraits<PermutationIndex>::highest()); - using std::abs; - Index rows = m_qr.rows(); Index cols = m_qr.cols(); Index size = m_qr.diagonalSize(); @@ -431,7 +425,7 @@ m_qr.coeffRef(k, k) = beta; // remember the maximum absolute value of diagonal coefficients - if (abs(beta) > this->m_maxpivot) this->m_maxpivot = abs(beta); + if (numext::abs(beta) > this->m_maxpivot) this->m_maxpivot = numext::abs(beta); // apply the householder transformation m_qr.bottomRightCorner(rows - k, cols - k - 1) @@ -444,7 +438,7 @@ // and used in LAPACK routines xGEQPF and xGEQP3. // See lines 278-297 in http://www.netlib.org/lapack/explore-html/dc/df4/sgeqpf_8f_source.html if (!numext::is_exactly_zero(m_colNormsUpdated.coeffRef(j))) { - RealScalar temp = abs(m_qr.coeffRef(k, j)) / m_colNormsUpdated.coeffRef(j); + RealScalar temp = numext::abs(m_qr.coeffRef(k, j)) / m_colNormsUpdated.coeffRef(j); temp = (RealScalar(1) + temp) * (RealScalar(1) - temp); temp = temp < RealScalar(0) ? RealScalar(0) : temp; RealScalar temp2 =
diff --git a/Eigen/src/QR/FullPivHouseholderQR.h b/Eigen/src/QR/FullPivHouseholderQR.h index 3de6e0c..4124965 100644 --- a/Eigen/src/QR/FullPivHouseholderQR.h +++ b/Eigen/src/QR/FullPivHouseholderQR.h
@@ -287,10 +287,7 @@ typename MatrixType::Scalar signDeterminant() const; /** \returns the absolute value of the i-th pivot coefficient (for RankRevealingBase). */ - RealScalar pivotCoeff(Index i) const { - using std::abs; - return abs(m_qr.coeff(i, i)); - } + RealScalar pivotCoeff(Index i) const { return numext::abs(m_qr.coeff(i, i)); } /** \returns the inverse of the matrix of which *this is the QR decomposition. * @@ -346,10 +343,9 @@ template <typename MatrixType, typename PermutationIndex> typename MatrixType::RealScalar FullPivHouseholderQR<MatrixType, PermutationIndex>::absDeterminant() const { - using std::abs; eigen_assert(m_isInitialized && "FullPivHouseholderQR is not initialized."); EIGEN_INTERNAL_ASSERT_SQUARE_MSG(m_qr, "You can't take the determinant of a non-square matrix!"); - return isInjective() ? abs(m_qr.diagonal().prod()) : RealScalar(0); + return isInjective() ? numext::abs(m_qr.diagonal().prod()) : RealScalar(0); } template <typename MatrixType, typename PermutationIndex> @@ -386,7 +382,6 @@ template <typename MatrixType, typename PermutationIndex> void FullPivHouseholderQR<MatrixType, PermutationIndex>::computeInPlace() { eigen_assert(m_qr.cols() <= NumTraits<PermutationIndex>::highest()); - using std::abs; Index rows = m_qr.rows(); Index cols = m_qr.cols(); Index size = (std::min)(rows, cols); @@ -447,7 +442,7 @@ m_qr.coeffRef(k, k) = beta; // remember the maximum absolute value of diagonal coefficients - if (abs(beta) > this->m_maxpivot) this->m_maxpivot = abs(beta); + if (numext::abs(beta) > this->m_maxpivot) this->m_maxpivot = numext::abs(beta); m_qr.bottomRightCorner(rows - k, cols - k - 1) .applyHouseholderOnTheLeft(m_qr.col(k).tail(rows - k - 1), m_hCoeffs.coeffRef(k), &m_temp.coeffRef(k + 1)); @@ -567,7 +562,6 @@ template <typename ResultType> void evalTo(ResultType& result, WorkVectorType& workspace) const { - using numext::conj; // compute the product H'_0 H'_1 ... H'_n-1, // where H_k is the k-th Householder transformation I - h_k v_k v_k' // and v_k is the k-th Householder vector [1,m_qr(k+1,k), m_qr(k+2,k), ...] @@ -578,7 +572,8 @@ result.setIdentity(rows, rows); for (Index k = size - 1; k >= 0; k--) { result.block(k, k, rows - k, rows - k) - .applyHouseholderOnTheLeft(m_qr.col(k).tail(rows - k - 1), conj(m_hCoeffs.coeff(k)), &workspace.coeffRef(k)); + .applyHouseholderOnTheLeft(m_qr.col(k).tail(rows - k - 1), numext::conj(m_hCoeffs.coeff(k)), + &workspace.coeffRef(k)); result.row(k).swap(result.row(m_rowsTranspositions.coeff(k))); } }
diff --git a/Eigen/src/QR/HouseholderQR.h b/Eigen/src/QR/HouseholderQR.h index 4a48441..7bf2e74 100644 --- a/Eigen/src/QR/HouseholderQR.h +++ b/Eigen/src/QR/HouseholderQR.h
@@ -346,10 +346,9 @@ template <typename MatrixType> typename MatrixType::RealScalar HouseholderQR<MatrixType>::absDeterminant() const { - using std::abs; eigen_assert(m_isInitialized && "HouseholderQR is not initialized."); EIGEN_INTERNAL_ASSERT_SQUARE_MSG(m_qr, "You can't take the determinant of a non-square matrix!"); - return abs(m_qr.diagonal().prod()); + return numext::abs(m_qr.diagonal().prod()); } template <typename MatrixType>
diff --git a/Eigen/src/QR/RandColPivHouseholderQR.h b/Eigen/src/QR/RandColPivHouseholderQR.h index ea4f3b3..3586dc1 100644 --- a/Eigen/src/QR/RandColPivHouseholderQR.h +++ b/Eigen/src/QR/RandColPivHouseholderQR.h
@@ -57,9 +57,8 @@ template <typename RealScalar, typename Scalar, typename RecomputeFn> EIGEN_STRONG_INLINE void lawn176_norm_downdate(RealScalar& norm_updated, RealScalar& norm_direct, Scalar pivot_entry, RealScalar downdate_threshold, RecomputeFn&& tail_norm_fn) { - using std::abs; if (numext::is_exactly_zero(norm_updated)) return; - RealScalar t = abs(pivot_entry) / norm_updated; + RealScalar t = numext::abs(pivot_entry) / norm_updated; t = (RealScalar(1) + t) * (RealScalar(1) - t); if (t < RealScalar(0)) t = RealScalar(0); RealScalar t2 = t * numext::abs2<RealScalar>(norm_updated / norm_direct); @@ -290,10 +289,7 @@ typename MatrixType::RealScalar logAbsDeterminant() const; typename MatrixType::Scalar signDeterminant() const; - RealScalar pivotCoeff(Index i) const { - using std::abs; - return abs(m_qr.coeff(i, i)); - } + RealScalar pivotCoeff(Index i) const { return numext::abs(m_qr.coeff(i, i)); } inline Inverse<RandColPivHouseholderQR> inverse() const { eigen_assert(m_isInitialized && "RandColPivHouseholderQR is not initialized."); @@ -406,10 +402,9 @@ template <typename MatrixType, typename PermutationIndex> typename MatrixType::RealScalar RandColPivHouseholderQR<MatrixType, PermutationIndex>::absDeterminant() const { - using std::abs; eigen_assert(m_isInitialized && "RandColPivHouseholderQR is not initialized."); EIGEN_INTERNAL_ASSERT_SQUARE_MSG(m_qr, "You can't take the determinant of a non-square matrix!"); - return isInjective() ? abs(m_qr.diagonal().prod()) : RealScalar(0); + return isInjective() ? numext::abs(m_qr.diagonal().prod()) : RealScalar(0); } template <typename MatrixType, typename PermutationIndex> @@ -440,7 +435,6 @@ template <typename MatrixType, typename PermutationIndex> Index RandColPivHouseholderQR<MatrixType, PermutationIndex>::unblocked_pivoted_qr(Index row0, Index col0, Index ncols, RealScalar threshold_helper) { - using std::abs; const Index rows = m_qr.rows(); const Index col_end = col0 + ncols; const Index sub_rows = rows - row0; @@ -470,7 +464,7 @@ RealScalar beta; m_qr.col(col0 + k).tail(sub_rows - k).makeHouseholderInPlace(m_hCoeffs.coeffRef(col0 + k), beta); m_qr.coeffRef(row0 + k, col0 + k) = beta; - if (abs(beta) > this->m_maxpivot) this->m_maxpivot = abs(beta); + if (numext::abs(beta) > this->m_maxpivot) this->m_maxpivot = numext::abs(beta); // Apply the reflector only to the remaining panel columns. Columns at // indices >= col_end are owned by the outer caller and updated via a @@ -494,7 +488,6 @@ template <typename MatrixType, typename PermutationIndex> void RandColPivHouseholderQR<MatrixType, PermutationIndex>::finalize_rank(RealScalar threshold_helper) { - using std::abs; const Index rows = m_qr.rows(); const Index size = (std::min)(rows, m_qr.cols()); // m_maxpivot is already up-to-date: the blocked path tracks it per @@ -506,7 +499,7 @@ // unblocked tail. if (this->m_nonzero_pivots == size) { for (Index i = 0; i < size; ++i) { - RealScalar a = abs(m_qr.coeff(i, i)); + RealScalar a = numext::abs(m_qr.coeff(i, i)); if (numext::abs2(a) < threshold_helper * RealScalar(rows - i)) { this->m_nonzero_pivots = i; break; @@ -675,10 +668,9 @@ // rank() applies to the final diagonal. Index panel_rank = b; { - using std::abs; const RealScalar relative_cutoff = this->m_maxpivot * NumTraits<RealScalar>::epsilon() * RealScalar(4 * size); for (Index i = 0; i < b; ++i) { - RealScalar a = abs(m_qr.coeff(k + i, k + i)); + RealScalar a = numext::abs(m_qr.coeff(k + i, k + i)); if (a <= relative_cutoff) { panel_rank = i; break;
diff --git a/Eigen/src/SVD/BDCSVD.h b/Eigen/src/SVD/BDCSVD.h index 3f0b0f0..38cad6a 100644 --- a/Eigen/src/SVD/BDCSVD.h +++ b/Eigen/src/SVD/BDCSVD.h
@@ -362,8 +362,6 @@ template <typename Derived> EIGEN_DONT_INLINE BDCSVD<MatrixType, Options>& BDCSVD<MatrixType, Options>::compute_impl( const MatrixBase<Derived>& matrix, unsigned int computationOptions, internal::false_type) { - using std::abs; - allocate(matrix.rows(), matrix.cols(), computationOptions); const RealScalar considerZero = (std::numeric_limits<RealScalar>::min)(); @@ -432,7 +430,7 @@ //**** step 3 - Copy singular values and vectors for (int i = 0; i < diagSize(); i++) { - RealScalar a = abs(m_impl.computed().coeff(i, i)); + RealScalar a = numext::abs(m_impl.computed().coeff(i, i)); m_singularValues.coeffRef(i) = a; if (a < considerZero) { m_nonzeroSingularValues = i; @@ -509,7 +507,6 @@ EIGEN_STATIC_ASSERT((NumTraits<typename DerivedE::Scalar>::IsComplex == 0), THIS_FUNCTION_IS_NOT_FOR_COMPLEX_VALUED_MATRICES); - using std::abs; const Index n = diagonal.size(); eigen_assert((n == 0 || superdiagonal.size() == n - 1) && "superdiagonal must have size diagonal.size() - 1"); @@ -590,7 +587,7 @@ //**** Extract singular values. for (int i = 0; i < diagSize(); i++) { - RealScalar a = abs(m_impl.computed().coeff(i, i)); + RealScalar a = numext::abs(m_impl.computed().coeff(i, i)); m_singularValues.coeffRef(i) = a; if (a < considerZero) { m_nonzeroSingularValues = i;
diff --git a/Eigen/src/SVD/BDCSVDImpl.h b/Eigen/src/SVD/BDCSVDImpl.h index 5f0a7b7..0b3ddc6 100644 --- a/Eigen/src/SVD/BDCSVDImpl.h +++ b/Eigen/src/SVD/BDCSVDImpl.h
@@ -348,7 +348,6 @@ template <typename RealScalar_> void bdcsvd_impl<RealScalar_>::computeSVDofM(Index firstCol, Index n, MatrixXr& U, VectorType& singVals, MatrixXr& V) { const RealScalar considerZero = (std::numeric_limits<RealScalar>::min)(); - using std::abs; ArrayRef col0 = m_computed.col(firstCol).segment(firstCol, n); m_workspace.head(n) = m_computed.block(firstCol, firstCol, n, n).diagonal(); ArrayRef diag = m_workspace.head(n); @@ -369,7 +368,7 @@ } Index m = 0; // size of the deflated problem for (Index k = 0; k < actual_n; ++k) - if (abs(col0(k)) > considerZero) m_workspaceI(m++) = k; + if (numext::abs(col0(k)) > considerZero) m_workspaceI(m++) = k; Map<ArrayXi> perm(m_workspaceI.data(), m); Map<ArrayXr> shifts(m_workspace.data() + 1 * n, n); @@ -412,8 +411,8 @@ RealScalar secondQuotient = secondNumerator / secondDenominator; #if defined(__FAST_MATH__) || EIGEN_COMP_NVHPC // NVHPC does not expose a preprocessor macro for -fast, so retain the barriers in all NVHPC builds. - EIGEN_OPTIMIZATION_BARRIER(firstQuotient) - EIGEN_OPTIMIZATION_BARRIER(secondQuotient) + pbarrier(firstQuotient); + pbarrier(secondQuotient); #endif return firstQuotient * secondQuotient; } @@ -423,7 +422,7 @@ RealScalar numerator, RealScalar firstDenominator, RealScalar secondDenominator) { RealScalar firstQuotient = numerator / firstDenominator; #if defined(__FAST_MATH__) || EIGEN_COMP_NVHPC - EIGEN_OPTIMIZATION_BARRIER(firstQuotient) + pbarrier(firstQuotient); #endif return firstQuotient / secondDenominator; } @@ -448,7 +447,6 @@ // See Ren-Cang Li, "Solving Secular Equations Stably and Efficiently", // LAPACK Working Note 89 (1994), and LAPACK's xLASD4/xLASD5 for the // stability rationale behind pole-relative shifts and safeguarded steps. - using std::abs; using std::swap; Index n = col0.size(); @@ -507,14 +505,15 @@ diagShifted = activeDiag - shift; if (k != actual_n - 1) { - // check that after the shift, f(mid) is still negative: + // mid rounds to working precision, but midShifted is exact relative to the pole. With close poles, fMid can + // therefore misplace the root, and the bisection below only reaches 0.51 * (right - left) from the shift. RealScalar midShifted = (right - left) / RealScalar(2); // we can test exact equality here, because shift comes from `... ? left : right` if (numext::equal_strict(shift, right)) midShifted = -midShifted; RealScalar fMidShifted = secularEq(midShifted, activeCol0, activeDiag, shiftedRef, shift); - if (fMidShifted > 0) { - // fMid was erroneous, fix it: - shift = fMidShifted > Literal(0) ? left : right; + const RealScalar rootSideShift = fMidShifted > Literal(0) ? left : right; + if (!numext::equal_strict(shift, rootSideShift)) { + shift = rootSideShift; diagShifted = activeDiag - shift; } } @@ -535,7 +534,7 @@ RealScalar fPrev = secularEq(muPrev, activeCol0, activeDiag, shiftedRef, shift); RealScalar fCur = secularEq(muCur, activeCol0, activeDiag, shiftedRef, shift); - if (abs(fPrev) < abs(fCur)) { + if (numext::abs(fPrev) < numext::abs(fCur)) { swap(fPrev, fCur); swap(muPrev, muCur); } @@ -544,13 +543,14 @@ // samples whose reciprocals may overflow stay on bisection. The interval and residual checks below // safeguard each interpolation step. const RealScalar minNormal = (std::numeric_limits<RealScalar>::min)(); - bool useBisection = fPrev * fCur > Literal(0) && !(abs(fCur - fPrev) > NumTraits<RealScalar>::epsilon() && - abs(fPrev) <= (std::numeric_limits<RealScalar>::max)() && - abs(muPrev) >= minNormal && abs(muCur) >= minNormal); + bool useBisection = + fPrev * fCur > Literal(0) && !(numext::abs(fCur - fPrev) > NumTraits<RealScalar>::epsilon() && + numext::abs(fPrev) <= (std::numeric_limits<RealScalar>::max)() && + numext::abs(muPrev) >= minNormal && numext::abs(muCur) >= minNormal); while (!numext::is_exactly_zero(fCur) && - abs(muCur - muPrev) > - Literal(8) * NumTraits<RealScalar>::epsilon() * numext::maxi<RealScalar>(abs(muCur), abs(muPrev)) && - abs(fCur - fPrev) > NumTraits<RealScalar>::epsilon() && !useBisection) { + numext::abs(muCur - muPrev) > Literal(8) * NumTraits<RealScalar>::epsilon() * + numext::maxi<RealScalar>(numext::abs(muCur), numext::abs(muPrev)) && + numext::abs(fCur - fPrev) > NumTraits<RealScalar>::epsilon() && !useBisection) { ++m_numIters; // Find a and b such that the function f(mu) = a / mu + b matches the current and previous samples. @@ -568,7 +568,7 @@ // we can test exact equality here, because shift comes from `... ? left : right` if (numext::equal_strict(shift, left) && (muCur < Literal(0) || muCur > right - left)) useBisection = true; if (numext::equal_strict(shift, right) && (muCur < -(right - left) || muCur > Literal(0))) useBisection = true; - if (abs(fCur) > abs(fPrev)) useBisection = true; + if (numext::abs(fCur) > numext::abs(fPrev)) useBisection = true; } // fall back on bisection method if rational interpolation did not work @@ -580,7 +580,7 @@ // the factor 2 is to be more conservative leftShifted = numext::maxi<RealScalar>( (std::numeric_limits<RealScalar>::min)(), - Literal(2) * abs(col0(k)) / numext::sqrt((std::numeric_limits<RealScalar>::max)())); + Literal(2) * numext::abs(col0(k)) / numext::sqrt((std::numeric_limits<RealScalar>::max)())); // check that we did it right: eigen_internal_assert( @@ -591,9 +591,9 @@ } else { leftShifted = -(right - left) * RealScalar(0.51); if (k + 1 < n) - rightShifted = - -numext::maxi<RealScalar>((std::numeric_limits<RealScalar>::min)(), - abs(col0(k + 1)) / numext::sqrt((std::numeric_limits<RealScalar>::max)())); + rightShifted = -numext::maxi<RealScalar>( + (std::numeric_limits<RealScalar>::min)(), + numext::abs(col0(k + 1)) / numext::sqrt((std::numeric_limits<RealScalar>::max)())); else rightShifted = -(std::numeric_limits<RealScalar>::min)(); } @@ -601,12 +601,20 @@ eigen_internal_assert(fLeft < Literal(0)); if (fLeft < Literal(0)) { - while (rightShifted - leftShifted > Literal(2) * NumTraits<RealScalar>::epsilon() * - numext::maxi<RealScalar>(abs(leftShifted), abs(rightShifted))) { + while (rightShifted - leftShifted > + Literal(2) * NumTraits<RealScalar>::epsilon() * + numext::maxi<RealScalar>(numext::abs(leftShifted), numext::abs(rightShifted))) { RealScalar midShifted = (leftShifted + rightShifted) / Literal(2); fMid = secularEq(midShifted, activeCol0, activeDiag, shiftedRef, shift); eigen_internal_assert((numext::isfinite)(fMid)); + // An exact root must end the search: with fLeft = 0, every later sign test would fail and drive the search to + // rightShifted. + if (numext::is_exactly_zero(fMid)) { + leftShifted = rightShifted = midShifted; + break; + } + if (fLeft * fMid < Literal(0)) { rightShifted = midShifted; } else { @@ -637,7 +645,6 @@ void bdcsvd_impl<RealScalar_>::perturbCol0(const ArrayRef& col0, const ArrayRef& diag, const IndicesRef& perm, const VectorType& singVals, const ArrayRef& shifts, const ArrayRef& mus, ArrayRef zhat) { - using std::abs; Index n = col0.size(); Index m = perm.size(); if (m == 0) { @@ -656,7 +663,7 @@ // Unsafe FP reassociation may otherwise turn `mus + (shift - dk)` into // `(mus + shift) - dk`, losing `mus` when `shift` and `dk` cancel. RealScalar diff = shifts(lastIdx) - dk; - EIGEN_OPTIMIZATION_BARRIER(diff) + pbarrier(diff); RealScalar prod = (singVals(lastIdx) + dk) * (mus(lastIdx) + diff); for (Index l = 0; l < m; ++l) { @@ -671,7 +678,7 @@ } Index j = i < k ? i : perm(l - 1); diff = shifts(j) - dk; - EIGEN_OPTIMIZATION_BARRIER(diff) + pbarrier(diff); prod *= productOfQuotients(singVals(j) + dk, diag(i) + dk, mus(j) + diff, diag(i) - dk); } } @@ -706,9 +713,9 @@ const RealScalar diagonal = diag(i); const RealScalar z = zhat(i); RealScalar diff = diagonal - shift; - EIGEN_OPTIMIZATION_BARRIER(diff) + pbarrier(diff); diff -= mu; - EIGEN_OPTIMIZATION_BARRIER(diff) + pbarrier(diff); const RealScalar sum = diagonal + singularValue; U(i, k) = sequentialQuotient(z, diff, sum); if (m_compV && l > 0) V(i, k) = sequentialQuotient(diagonal * z, diff, sum); @@ -784,7 +791,6 @@ template <typename RealScalar_> void bdcsvd_impl<RealScalar_>::deflation(Index firstCol, Index lastCol, Index k, Index firstRowW, Index firstColW, Index shift) { - using std::abs; const Index length = lastCol + 1 - firstCol; Block<MatrixXr, Dynamic, 1> col0(m_computed, firstCol + shift, firstCol + shift, length, 1); @@ -804,7 +810,7 @@ // condition 4.2 for (Index i = 1; i < length; ++i) - if (abs(col0(i)) < epsilon_strict) { + if (numext::abs(col0(i)) < epsilon_strict) { col0(i) = Literal(0); } @@ -899,7 +905,7 @@ { Index i = length - 1; // Find last non-deflated entry. - while (i > 0 && (diag(i) < considerZero || abs(col0(i)) < considerZero)) --i; + while (i > 0 && (diag(i) < considerZero || numext::abs(col0(i)) < considerZero)) --i; for (; i > 1; --i) if ((diag(i) - diag(i - 1)) < epsilon_coarse) {
diff --git a/Eigen/src/SVD/BDCSVD_LAPACKE.h b/Eigen/src/SVD/BDCSVD_LAPACKE.h index bcd6bac..618f068 100644 --- a/Eigen/src/SVD/BDCSVD_LAPACKE.h +++ b/Eigen/src/SVD/BDCSVD_LAPACKE.h
@@ -64,9 +64,20 @@ SVD::allocate(matrix.rows(), matrix.cols(), computationOptions); SVD::m_nonzeroSingularValues = SVD::m_diagSize; + // LAPACK rejects the zero leading dimensions of an empty input, and given valid ones it returns without writing + // U or VT. Set the identities the native path returns. + if (SVD::diagSize() == 0) { + if (SVD::computeU()) SVD::m_matrixU.setIdentity(); + if (SVD::computeV()) SVD::m_matrixV.setIdentity(); + SVD::m_isInitialized = true; + return; + } - // prepare arguments to ?gesdd - const lapack_int matrix_order = lapack_storage_of(matrix); + // prepare arguments to ?gesdd. A, U and VT all have MatrixType's layout, which the input + // expression need not share. + MatrixType temp; + temp = matrix; + const lapack_int matrix_order = lapack_storage_of(temp); const char jobz = (SVD::m_computeFullU || SVD::m_computeFullV) ? 'A' : (SVD::m_computeThinU || SVD::m_computeThinV) ? 'S' : 'N'; @@ -95,8 +106,6 @@ ldvt = 1; vt = &dummy; } - MatrixType temp; - temp = matrix; // actual call to ?gesdd lapack_int info = gesdd(matrix_order, jobz, to_lapack(SVD::rows()), to_lapack(SVD::cols()), to_lapack(temp.data()),
diff --git a/Eigen/src/SVD/JacobiSVD.h b/Eigen/src/SVD/JacobiSVD.h index b0b424e..4eddb77 100644 --- a/Eigen/src/SVD/JacobiSVD.h +++ b/Eigen/src/SVD/JacobiSVD.h
@@ -378,11 +378,10 @@ using Scalar = typename MatrixType::Scalar; using RealScalar = typename MatrixType::RealScalar; static bool run(typename SVD::WorkMatrixType& work_matrix, SVD& svd, Index p, Index q, RealScalar& maxDiagEntry) { - using numext::abs; - using numext::sqrt; Scalar z; JacobiRotation<Scalar> rot; - RealScalar n = sqrt(numext::abs2(work_matrix.coeff(p, p)) + numext::abs2(work_matrix.coeff(q, p))); + RealScalar n = + numext::sqrt<RealScalar>(numext::abs2(work_matrix.coeff(p, p)) + numext::abs2(work_matrix.coeff(q, p))); const RealScalar considerAsZero = (std::numeric_limits<RealScalar>::min)(); const RealScalar precision = NumTraits<Scalar>::epsilon(); @@ -391,42 +390,43 @@ // make sure first column is zero work_matrix.coeffRef(p, p) = work_matrix.coeffRef(q, p) = Scalar(0); - if (abs(numext::imag(work_matrix.coeff(p, q))) > considerAsZero) { + if (numext::abs(numext::imag(work_matrix.coeff(p, q))) > considerAsZero) { // work_matrix.coeff(p,q) can be zero if work_matrix.coeff(q,p) is not zero but small enough to underflow when // computing n - z = abs(work_matrix.coeff(p, q)) / work_matrix.coeff(p, q); + z = numext::abs(work_matrix.coeff(p, q)) / work_matrix.coeff(p, q); work_matrix.row(p) *= z; - if (svd.computeU()) svd.m_matrixU.col(p) *= conj(z); + if (svd.computeU()) svd.m_matrixU.col(p) *= numext::conj(z); } - if (abs(numext::imag(work_matrix.coeff(q, q))) > considerAsZero) { - z = abs(work_matrix.coeff(q, q)) / work_matrix.coeff(q, q); + if (numext::abs(numext::imag(work_matrix.coeff(q, q))) > considerAsZero) { + z = numext::abs(work_matrix.coeff(q, q)) / work_matrix.coeff(q, q); work_matrix.row(q) *= z; - if (svd.computeU()) svd.m_matrixU.col(q) *= conj(z); + if (svd.computeU()) svd.m_matrixU.col(q) *= numext::conj(z); } // otherwise the second row is already zero, so we have nothing to do. } else { - rot.c() = conj(work_matrix.coeff(p, p)) / n; + rot.c() = numext::conj(work_matrix.coeff(p, p)) / n; rot.s() = work_matrix.coeff(q, p) / n; work_matrix.applyOnTheLeft(p, q, rot); if (svd.computeU()) svd.m_matrixU.applyOnTheRight(p, q, rot.adjoint()); - if (abs(numext::imag(work_matrix.coeff(p, q))) > considerAsZero) { - z = abs(work_matrix.coeff(p, q)) / work_matrix.coeff(p, q); + if (numext::abs(numext::imag(work_matrix.coeff(p, q))) > considerAsZero) { + z = numext::abs(work_matrix.coeff(p, q)) / work_matrix.coeff(p, q); work_matrix.col(q) *= z; if (svd.computeV()) svd.m_matrixV.col(q) *= z; } - if (abs(numext::imag(work_matrix.coeff(q, q))) > considerAsZero) { - z = abs(work_matrix.coeff(q, q)) / work_matrix.coeff(q, q); + if (numext::abs(numext::imag(work_matrix.coeff(q, q))) > considerAsZero) { + z = numext::abs(work_matrix.coeff(q, q)) / work_matrix.coeff(q, q); work_matrix.row(q) *= z; - if (svd.computeU()) svd.m_matrixU.col(q) *= conj(z); + if (svd.computeU()) svd.m_matrixU.col(q) *= numext::conj(z); } } // update largest diagonal entry maxDiagEntry = numext::maxi<RealScalar>( - maxDiagEntry, numext::maxi<RealScalar>(abs(work_matrix.coeff(p, p)), abs(work_matrix.coeff(q, q)))); + maxDiagEntry, + numext::maxi<RealScalar>(numext::abs(work_matrix.coeff(p, p)), numext::abs(work_matrix.coeff(q, q)))); // and check whether the 2x2 block is already diagonal RealScalar threshold = numext::maxi<RealScalar>(considerAsZero, precision * maxDiagEntry); - return abs(work_matrix.coeff(p, q)) > threshold || abs(work_matrix.coeff(q, p)) > threshold; + return numext::abs(work_matrix.coeff(p, q)) > threshold || numext::abs(work_matrix.coeff(q, p)) > threshold; } }; @@ -436,8 +436,6 @@ typename WorkMatrixType::RealScalar considerAsZero, typename WorkMatrixType::RealScalar precision, typename WorkMatrixType::RealScalar& maxDiagEntry) { - using numext::abs; - using numext::sqrt; using Scalar = typename WorkMatrixType::Scalar; using RealScalar = typename WorkMatrixType::RealScalar; const Index n = work_matrix.rows(); @@ -448,25 +446,26 @@ for (Index p = 1; p < n; ++p) { for (Index q = 0; q < p; ++q) { RealScalar threshold = numext::maxi<RealScalar>(considerAsZero, precision * maxDiagEntry); - if (abs(work_matrix.coeff(p, q)) > threshold || abs(work_matrix.coeff(q, p)) > threshold) { + if (numext::abs(work_matrix.coeff(p, q)) > threshold || numext::abs(work_matrix.coeff(q, p)) > threshold) { notFinished = true; Scalar z; bool doRealSvd = true; - RealScalar nn = sqrt(numext::abs2(work_matrix.coeff(p, p)) + numext::abs2(work_matrix.coeff(q, p))); + RealScalar nn = + numext::sqrt<RealScalar>(numext::abs2(work_matrix.coeff(p, p)) + numext::abs2(work_matrix.coeff(q, p))); if (numext::is_exactly_zero(nn)) { // Make sure first column is zero. work_matrix.coeffRef(p, p) = work_matrix.coeffRef(q, p) = Scalar(0); - if (abs(numext::imag(work_matrix.coeff(p, q))) > considerAsZero) { + if (numext::abs(numext::imag(work_matrix.coeff(p, q))) > considerAsZero) { // work_matrix.coeff(p, q) can be zero if work_matrix.coeff(q, p) is not zero but small enough to // underflow when computing nn. - z = abs(work_matrix.coeff(p, q)) / work_matrix.coeff(p, q); + z = numext::abs(work_matrix.coeff(p, q)) / work_matrix.coeff(p, q); work_matrix.row(p) *= z; if (compute_u) matrix_u.col(p) *= numext::conj(z); } - if (abs(numext::imag(work_matrix.coeff(q, q))) > considerAsZero) { - z = abs(work_matrix.coeff(q, q)) / work_matrix.coeff(q, q); + if (numext::abs(numext::imag(work_matrix.coeff(q, q))) > considerAsZero) { + z = numext::abs(work_matrix.coeff(q, q)) / work_matrix.coeff(q, q); work_matrix.row(q) *= z; if (compute_u) matrix_u.col(q) *= numext::conj(z); } @@ -476,22 +475,24 @@ rot.s() = work_matrix.coeff(q, p) / nn; work_matrix.applyOnTheLeft(p, q, rot); if (compute_u) matrix_u.applyOnTheRight(p, q, rot.adjoint()); - if (abs(numext::imag(work_matrix.coeff(p, q))) > considerAsZero) { - z = abs(work_matrix.coeff(p, q)) / work_matrix.coeff(p, q); + if (numext::abs(numext::imag(work_matrix.coeff(p, q))) > considerAsZero) { + z = numext::abs(work_matrix.coeff(p, q)) / work_matrix.coeff(p, q); work_matrix.col(q) *= z; if (compute_v) matrix_v.col(q) *= z; } - if (abs(numext::imag(work_matrix.coeff(q, q))) > considerAsZero) { - z = abs(work_matrix.coeff(q, q)) / work_matrix.coeff(q, q); + if (numext::abs(numext::imag(work_matrix.coeff(q, q))) > considerAsZero) { + z = numext::abs(work_matrix.coeff(q, q)) / work_matrix.coeff(q, q); work_matrix.row(q) *= z; if (compute_u) matrix_u.col(q) *= numext::conj(z); } } maxDiagEntry = numext::maxi<RealScalar>( - maxDiagEntry, numext::maxi<RealScalar>(abs(work_matrix.coeff(p, p)), abs(work_matrix.coeff(q, q)))); + maxDiagEntry, + numext::maxi<RealScalar>(numext::abs(work_matrix.coeff(p, p)), numext::abs(work_matrix.coeff(q, q)))); threshold = numext::maxi<RealScalar>(considerAsZero, NumTraits<Scalar>::epsilon() * maxDiagEntry); - doRealSvd = abs(work_matrix.coeff(p, q)) > threshold || abs(work_matrix.coeff(q, p)) > threshold; + doRealSvd = + numext::abs(work_matrix.coeff(p, q)) > threshold || numext::abs(work_matrix.coeff(q, p)) > threshold; if (doRealSvd) { JacobiRotation<RealScalar> j_left, j_right; @@ -501,7 +502,8 @@ work_matrix.applyOnTheRight(p, q, j_right); if (compute_v) matrix_v.applyOnTheRight(p, q, j_right); maxDiagEntry = numext::maxi<RealScalar>( - maxDiagEntry, numext::maxi<RealScalar>(abs(work_matrix.coeff(p, p)), abs(work_matrix.coeff(q, q)))); + maxDiagEntry, + numext::maxi<RealScalar>(numext::abs(work_matrix.coeff(p, p)), numext::abs(work_matrix.coeff(q, q)))); } } } @@ -511,7 +513,7 @@ RealScalar threshold = numext::maxi<RealScalar>(considerAsZero, precision * maxDiagEntry); for (Index p = 1; p < n; ++p) { for (Index q = 0; q < p; ++q) { - if (abs(work_matrix.coeff(p, q)) > threshold || abs(work_matrix.coeff(q, p)) > threshold) { + if (numext::abs(work_matrix.coeff(p, q)) > threshold || numext::abs(work_matrix.coeff(q, p)) > threshold) { notFinished = true; JacobiRotation<RealScalar> j_left, j_right; internal::real_2x2_jacobi_svd(work_matrix, p, q, &j_left, &j_right); @@ -520,7 +522,8 @@ work_matrix.applyOnTheRight(p, q, j_right); if (compute_v) matrix_v.applyOnTheRight(p, q, j_right); maxDiagEntry = numext::maxi<RealScalar>( - maxDiagEntry, numext::maxi<RealScalar>(abs(work_matrix.coeff(p, p)), abs(work_matrix.coeff(q, q)))); + maxDiagEntry, + numext::maxi<RealScalar>(numext::abs(work_matrix.coeff(p, p)), numext::abs(work_matrix.coeff(q, q)))); threshold = numext::maxi<RealScalar>(considerAsZero, precision * maxDiagEntry); } } @@ -832,8 +835,6 @@ EIGEN_STATIC_ASSERT((std::is_same<typename Derived::Scalar, typename MatrixType::Scalar>::value), Input matrix must have the same Scalar type as the JacobiSVD object.); - using numext::abs; - allocate(matrix.rows(), matrix.cols(), computationOptions); // currently we stop when we reach precision 2*epsilon as the last bit of precision can require an unreasonable number @@ -921,11 +922,11 @@ // of some diagonal entry might not be null. bool diagonal_has_imaginary_part = false; EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::IsComplex) { - diagonal_has_imaginary_part = abs(numext::imag(m_workMatrix.coeff(i, i))) > considerAsZero; + diagonal_has_imaginary_part = numext::abs(numext::imag(m_workMatrix.coeff(i, i))) > considerAsZero; } if (diagonal_has_imaginary_part) { - RealScalar a = abs(m_workMatrix.coeff(i, i)); - m_singularValues.coeffRef(i) = abs(a); + RealScalar a = numext::abs(m_workMatrix.coeff(i, i)); + m_singularValues.coeffRef(i) = numext::abs(a); if (computeU()) m_matrixU.col(i) *= m_workMatrix.coeff(i, i) / a; } else { // m_workMatrix.coeff(i,i) is already real. Its magnitude and sign are read from the representation for float @@ -990,8 +991,6 @@ template <typename MatrixType, int Options> EIGEN_DONT_INLINE bool JacobiSVD<MatrixType, Options>::blocked_sweep(RealScalar considerAsZero, RealScalar precision, RealScalar& maxDiagEntry) { - using numext::abs; - using numext::sqrt; const Index n = diagSize(); RealScalar threshold = numext::maxi<RealScalar>(considerAsZero, precision * maxDiagEntry); bool notFinished = false; @@ -1029,7 +1028,8 @@ bool blockDirty = false; for (Index qq = 0; qq < kBlockSize; ++qq) { - if (abs(blockBuffer.coeff(kBlockSize, qq)) > threshold || abs(blockBuffer.coeff(qq, kBlockSize)) > threshold) { + if (numext::abs(blockBuffer.coeff(kBlockSize, qq)) > threshold || + numext::abs(blockBuffer.coeff(qq, kBlockSize)) > threshold) { notFinished = true; blockDirty = true; @@ -1047,8 +1047,8 @@ EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::IsComplex) { Scalar z; // nn = ||(w_pp, w_qp)||_2, the norm of the first column of the 2x2 block. - RealScalar nn = sqrt(numext::abs2(blockBuffer.coeff(kBlockSize, kBlockSize)) + - numext::abs2(blockBuffer.coeff(qq, kBlockSize))); + RealScalar nn = numext::sqrt<RealScalar>(numext::abs2(blockBuffer.coeff(kBlockSize, kBlockSize)) + + numext::abs2(blockBuffer.coeff(qq, kBlockSize))); if (numext::is_exactly_zero(nn)) { // First column is zero => block is already upper triangular. @@ -1056,14 +1056,14 @@ blockBuffer.coeffRef(qq, kBlockSize) = Scalar(0); // Scale rows by z = e^{-i*arg(w)} to make remaining entries real. - if (abs(numext::imag(blockBuffer.coeff(kBlockSize, qq))) > considerAsZero) { - z = abs(blockBuffer.coeff(kBlockSize, qq)) / blockBuffer.coeff(kBlockSize, qq); + if (numext::abs(numext::imag(blockBuffer.coeff(kBlockSize, qq))) > considerAsZero) { + z = numext::abs(blockBuffer.coeff(kBlockSize, qq)) / blockBuffer.coeff(kBlockSize, qq); blockBuffer.row(kBlockSize) *= z; accum.row(kBlockSize) *= z; // accumulate left op if (computeU()) m_matrixU.col(p) *= numext::conj(z); } - if (abs(numext::imag(blockBuffer.coeff(qq, qq))) > considerAsZero) { - z = abs(blockBuffer.coeff(qq, qq)) / blockBuffer.coeff(qq, qq); + if (numext::abs(numext::imag(blockBuffer.coeff(qq, qq))) > considerAsZero) { + z = numext::abs(blockBuffer.coeff(qq, qq)) / blockBuffer.coeff(qq, qq); blockBuffer.row(qq) *= z; accum.row(qq) *= z; // accumulate left op if (computeU()) m_matrixU.col(q + qq) *= numext::conj(z); @@ -1081,15 +1081,15 @@ if (computeU()) m_matrixU.applyOnTheRight(p, q + qq, rot.adjoint()); // Scale column qq by z = e^{-i*arg(w_pq)} to make w_pq real. - if (abs(numext::imag(blockBuffer.coeff(kBlockSize, qq))) > considerAsZero) { - z = abs(blockBuffer.coeff(kBlockSize, qq)) / blockBuffer.coeff(kBlockSize, qq); + if (numext::abs(numext::imag(blockBuffer.coeff(kBlockSize, qq))) > considerAsZero) { + z = numext::abs(blockBuffer.coeff(kBlockSize, qq)) / blockBuffer.coeff(kBlockSize, qq); blockBuffer.col(qq) *= z; m_workMatrix.col(q + qq) *= z; // right op: apply directly if (computeV()) m_matrixV.col(q + qq) *= z; } // Scale row qq by z = e^{-i*arg(w_qq)} to make w_qq real. - if (abs(numext::imag(blockBuffer.coeff(qq, qq))) > considerAsZero) { - z = abs(blockBuffer.coeff(qq, qq)) / blockBuffer.coeff(qq, qq); + if (numext::abs(numext::imag(blockBuffer.coeff(qq, qq))) > considerAsZero) { + z = numext::abs(blockBuffer.coeff(qq, qq)) / blockBuffer.coeff(qq, qq); blockBuffer.row(qq) *= z; accum.row(qq) *= z; // accumulate left op if (computeU()) m_matrixU.col(q + qq) *= numext::conj(z); @@ -1097,12 +1097,12 @@ } // Update maxDiagEntry from preconditioning. maxDiagEntry = numext::maxi<RealScalar>( - maxDiagEntry, numext::maxi<RealScalar>(abs(blockBuffer.coeff(kBlockSize, kBlockSize)), - abs(blockBuffer.coeff(qq, qq)))); + maxDiagEntry, numext::maxi<RealScalar>(numext::abs(blockBuffer.coeff(kBlockSize, kBlockSize)), + numext::abs(blockBuffer.coeff(qq, qq)))); // Check if 2x2 block still needs diagonalizing. RealScalar precondThreshold = numext::maxi<RealScalar>(considerAsZero, precision * maxDiagEntry); - doRealSvd = abs(blockBuffer.coeff(kBlockSize, qq)) > precondThreshold || - abs(blockBuffer.coeff(qq, kBlockSize)) > precondThreshold; + doRealSvd = numext::abs(blockBuffer.coeff(kBlockSize, qq)) > precondThreshold || + numext::abs(blockBuffer.coeff(qq, kBlockSize)) > precondThreshold; } if (doRealSvd) { @@ -1121,8 +1121,8 @@ if (computeV()) m_matrixV.applyOnTheRight(p, q + qq, j_right); maxDiagEntry = numext::maxi<RealScalar>( - maxDiagEntry, numext::maxi<RealScalar>(abs(blockBuffer.coeff(kBlockSize, kBlockSize)), - abs(blockBuffer.coeff(qq, qq)))); + maxDiagEntry, numext::maxi<RealScalar>(numext::abs(blockBuffer.coeff(kBlockSize, kBlockSize)), + numext::abs(blockBuffer.coeff(qq, qq)))); } threshold = numext::maxi<RealScalar>(considerAsZero, precision * maxDiagEntry); } @@ -1154,7 +1154,7 @@ // Scalar loop for remaining pairs after blocked processing. for (; q < p; ++q) { - if (abs(m_workMatrix.coeff(p, q)) > threshold || abs(m_workMatrix.coeff(q, p)) > threshold) { + if (numext::abs(m_workMatrix.coeff(p, q)) > threshold || numext::abs(m_workMatrix.coeff(q, p)) > threshold) { notFinished = true; bool doRealSvd = true; @@ -1171,7 +1171,8 @@ m_workMatrix.applyOnTheRight(p, q, j_right); if (computeV()) m_matrixV.applyOnTheRight(p, q, j_right); maxDiagEntry = numext::maxi<RealScalar>( - maxDiagEntry, numext::maxi<RealScalar>(abs(m_workMatrix.coeff(p, p)), abs(m_workMatrix.coeff(q, q)))); + maxDiagEntry, + numext::maxi<RealScalar>(numext::abs(m_workMatrix.coeff(p, p)), numext::abs(m_workMatrix.coeff(q, q)))); } threshold = numext::maxi<RealScalar>(considerAsZero, precision * maxDiagEntry); }
diff --git a/Eigen/src/SVD/JacobiSVD_LAPACKE.h b/Eigen/src/SVD/JacobiSVD_LAPACKE.h index 5d72db2..d90a17a 100644 --- a/Eigen/src/SVD/JacobiSVD_LAPACKE.h +++ b/Eigen/src/SVD/JacobiSVD_LAPACKE.h
@@ -53,8 +53,16 @@ \ /*const RealScalar precision = RealScalar(2) * NumTraits<Scalar>::epsilon();*/ \ m_nonzeroSingularValues = diagSize(); \ + /* LAPACK rejects the zero leading dimensions of an empty input, and given valid ones it returns without */ \ + /* writing U or VT. Set the identities the native path returns. */ \ + if (diagSize() == 0) { \ + if (computeU()) m_matrixU.setIdentity(); \ + if (computeV()) m_matrixV.setIdentity(); \ + m_isInitialized = true; \ + return *this; \ + } \ \ - lapack_int lda = internal::convert_index<lapack_int>(matrix.outerStride()), ldu, ldvt; \ + lapack_int ldu, ldvt; \ lapack_int matrix_order = LAPACKE_COLROW; \ char jobu, jobvt; \ LAPACKE_TYPE *u, *vt, dummy; \ @@ -83,6 +91,8 @@ superb.resize(diagSize(), 1); \ MatrixType m_temp; \ m_temp = matrix; \ + /* LAPACK reads the copy, whose layout the input expression need not share. */ \ + lapack_int lda = internal::convert_index<lapack_int>(m_temp.outerStride()); \ lapack_int info = LAPACKE_##LAPACKE_PREFIX##gesvd( \ matrix_order, jobu, jobvt, internal::convert_index<lapack_int>(rows()), \ internal::convert_index<lapack_int>(cols()), (LAPACKE_TYPE*)m_temp.data(), lda, \
diff --git a/Eigen/src/SVD/SVDBase.h b/Eigen/src/SVD/SVDBase.h index 724c6d3..7496308 100644 --- a/Eigen/src/SVD/SVDBase.h +++ b/Eigen/src/SVD/SVDBase.h
@@ -218,7 +218,6 @@ * setThreshold(const RealScalar&). */ inline Index rank() const { - using std::abs; _check_compute_assertions(); if (m_singularValues.size() == 0) return 0; RealScalar premultiplied_threshold =
diff --git a/Eigen/src/SparseCholesky/SimplicialCholesky_impl.h b/Eigen/src/SparseCholesky/SimplicialCholesky_impl.h index 779a5d1..c17db9c 100644 --- a/Eigen/src/SparseCholesky/SimplicialCholesky_impl.h +++ b/Eigen/src/SparseCholesky/SimplicialCholesky_impl.h
@@ -321,7 +321,6 @@ template <typename Derived> template <bool DoLDLT, bool NonHermitian, bool UsePackets> void SimplicialCholeskyBase<Derived>::factorize_preordered_impl(const CholMatrixType& ap) { - using std::sqrt; const StorageIndex size = StorageIndex(ap.rows()); eigen_assert(m_analysisIsOk && "You must first call analyzePattern()"); @@ -406,7 +405,7 @@ ok = false; /* failure, matrix is not positive definite */ break; } - Lx[p] = sqrt(d); + Lx[p] = numext::sqrt(d); } }
diff --git a/Eigen/src/SparseCore/AmbiVector.h b/Eigen/src/SparseCore/AmbiVector.h index 9c20262..25ad024 100644 --- a/Eigen/src/SparseCore/AmbiVector.h +++ b/Eigen/src/SparseCore/AmbiVector.h
@@ -321,7 +321,6 @@ * are skipped. */ explicit Iterator(const AmbiVector& vec, const RealScalar& epsilon = 0) : m_vector(vec) { - using std::abs; m_epsilon = epsilon; m_isDense = m_vector.m_mode == IsDense; if (m_isDense) { @@ -332,7 +331,7 @@ } else { const ListEl* EIGEN_RESTRICT llElements = m_vector.listElements(); m_currentEl = m_vector.m_llStart; - while (m_currentEl >= 0 && abs(llElements[m_currentEl].value) <= m_epsilon) + while (m_currentEl >= 0 && numext::abs(llElements[m_currentEl].value) <= m_epsilon) m_currentEl = llElements[m_currentEl].next; if (m_currentEl < 0) { m_cachedValue = 0; // this is to avoid a compilation warning @@ -350,11 +349,10 @@ operator bool() const { return m_cachedIndex >= 0; } Iterator& operator++() { - using std::abs; if (m_isDense) { do { ++m_cachedIndex; - } while (m_cachedIndex < m_vector.m_end && abs(m_vector.m_buffer[m_cachedIndex]) <= m_epsilon); + } while (m_cachedIndex < m_vector.m_end && numext::abs(m_vector.m_buffer[m_cachedIndex]) <= m_epsilon); if (m_cachedIndex < m_vector.m_end) m_cachedValue = m_vector.m_buffer[m_cachedIndex]; else @@ -363,7 +361,7 @@ const ListEl* EIGEN_RESTRICT llElements = m_vector.listElements(); do { m_currentEl = llElements[m_currentEl].next; - } while (m_currentEl >= 0 && abs(llElements[m_currentEl].value) <= m_epsilon); + } while (m_currentEl >= 0 && numext::abs(llElements[m_currentEl].value) <= m_epsilon); if (m_currentEl < 0) { m_cachedIndex = -1; } else {
diff --git a/Eigen/src/SparseCore/CompressedStorage.h b/Eigen/src/SparseCore/CompressedStorage.h index 72ceaf2..15f1b1f 100644 --- a/Eigen/src/SparseCore/CompressedStorage.h +++ b/Eigen/src/SparseCore/CompressedStorage.h
@@ -155,10 +155,7 @@ if (id >= m_size || m_indices[id] != key) { if (m_allocatedSize < m_size + 1) { Index newAllocatedSize = 2 * (m_size + 1); - m_values = conditional_aligned_realloc_new_auto<Scalar, true>(m_values, newAllocatedSize, m_allocatedSize); - m_indices = - conditional_aligned_realloc_new_auto<StorageIndex, true>(m_indices, newAllocatedSize, m_allocatedSize); - m_allocatedSize = newAllocatedSize; + reallocate(newAllocatedSize); } if (m_size > id) { internal::smart_memmove(m_values + id, m_values + m_size, m_values + id + 1); @@ -183,8 +180,32 @@ EIGEN_SPARSE_COMPRESSED_STORAGE_REALLOCATE_PLUGIN #endif eigen_internal_assert(size != m_allocatedSize); - m_values = conditional_aligned_realloc_new_auto<Scalar, true>(m_values, size, m_allocatedSize); - m_indices = conditional_aligned_realloc_new_auto<StorageIndex, true>(m_indices, size, m_allocatedSize); + EIGEN_IF_CONSTEXPR (!NumTraits<Scalar>::RequireInitialization && !NumTraits<StorageIndex>::RequireInitialization && + std::is_trivially_copyable<Scalar>::value && std::is_trivially_copyable<StorageIndex>::value) { + const Index old_capacity = m_allocatedSize; + m_values = conditional_aligned_realloc_new_auto<Scalar, true>(m_values, size, old_capacity); + EIGEN_TRY { m_indices = conditional_aligned_realloc_new_auto<StorageIndex, true>(m_indices, size, old_capacity); } + EIGEN_CATCH(...) { + // The value buffer may have shrunk; never advertise more capacity than either buffer holds. + m_allocatedSize = (std::min)(old_capacity, size); + EIGEN_THROW; + } + m_allocatedSize = size; + return; + } + // Prepare indices first: once values have moved, no allocation may throw before committing the capacity. + StorageIndex* new_indices = conditional_aligned_new_auto<StorageIndex, true>(size); + EIGEN_TRY { + const Index copy_size = (std::min)(m_size, size); + if (copy_size > 0) internal::smart_copy(m_indices, m_indices + copy_size, new_indices); + m_values = conditional_aligned_realloc_new_auto<Scalar, true>(m_values, size, m_allocatedSize); + } + EIGEN_CATCH(...) { + conditional_aligned_delete_auto<StorageIndex, true>(new_indices, size); + EIGEN_THROW; + } + conditional_aligned_delete_auto<StorageIndex, true>(m_indices, m_allocatedSize); + m_indices = new_indices; m_allocatedSize = size; }
diff --git a/Eigen/src/SparseCore/ConservativeSparseSparseProduct.h b/Eigen/src/SparseCore/ConservativeSparseSparseProduct.h index 010e795..2b47d83 100644 --- a/Eigen/src/SparseCore/ConservativeSparseSparseProduct.h +++ b/Eigen/src/SparseCore/ConservativeSparseSparseProduct.h
@@ -18,12 +18,13 @@ namespace internal { -template <typename Lhs, typename Rhs, typename ResultType> +template <typename Lhs, typename Rhs, typename ResultType, bool ReverseProduct = false> static void conservative_sparse_sparse_product_impl(const Lhs& lhs, const Rhs& rhs, ResultType& res, bool sortedInsertion = false) { using LhsScalar = typename remove_all_t<Lhs>::Scalar; using RhsScalar = typename remove_all_t<Rhs>::Scalar; using ResScalar = typename remove_all_t<ResultType>::Scalar; + const sparse_scalar_product_op<LhsScalar, RhsScalar, ReverseProduct> product; // make sure to call innerSize/outerSize since we fake the storage order. Index rows = lhs.innerSize(); @@ -61,11 +62,11 @@ LhsScalar x = lhsIt.value(); if (!mask[i]) { mask[i] = true; - values[i] = x * y; + values[i] = product(x, y); indices[nnz] = i; ++nnz; } else - values[i] += x * y; + product.accumulate(values[i], x, y); } } if (!sortedInsertion) { @@ -149,7 +150,7 @@ using RowMajorRes = WithStorageOrder<ResultType, RowMajor>; RowMajorRhs rhsRow = rhs; RowMajorRes resRow(lhs.rows(), rhs.cols()); - internal::conservative_sparse_sparse_product_impl<RowMajorRhs, Lhs, RowMajorRes>(rhsRow, lhs, resRow); + internal::conservative_sparse_sparse_product_impl<RowMajorRhs, Lhs, RowMajorRes, true>(rhsRow, lhs, resRow); res = resRow; } }; @@ -161,7 +162,7 @@ using RowMajorRes = WithStorageOrder<ResultType, RowMajor>; RowMajorLhs lhsRow = lhs; RowMajorRes resRow(lhs.rows(), rhs.cols()); - internal::conservative_sparse_sparse_product_impl<Rhs, RowMajorLhs, RowMajorRes>(rhs, lhsRow, resRow); + internal::conservative_sparse_sparse_product_impl<Rhs, RowMajorLhs, RowMajorRes, true>(rhs, lhsRow, resRow); res = resRow; } }; @@ -171,7 +172,7 @@ static void run(const Lhs& lhs, const Rhs& rhs, ResultType& res) { using RowMajorRes = WithStorageOrder<ResultType, RowMajor>; RowMajorRes resRow(lhs.rows(), rhs.cols()); - internal::conservative_sparse_sparse_product_impl<Rhs, Lhs, RowMajorRes>(rhs, lhs, resRow); + internal::conservative_sparse_sparse_product_impl<Rhs, Lhs, RowMajorRes, true>(rhs, lhs, resRow); res = resRow; } }; @@ -216,17 +217,18 @@ using ColMajorRes = WithStorageOrder<ResultType, ColMajor>; using RowMajorRes = WithStorageOrder<ResultType, RowMajor>; RowMajorRes resRow(lhs.rows(), rhs.cols()); - internal::conservative_sparse_sparse_product_impl<Rhs, Lhs, RowMajorRes>(rhs, lhs, resRow); + internal::conservative_sparse_sparse_product_impl<Rhs, Lhs, RowMajorRes, true>(rhs, lhs, resRow); // sort the non zeros: ColMajorRes resCol(resRow); res = resCol; } }; -template <typename Lhs, typename Rhs, typename ResultType> +template <typename Lhs, typename Rhs, typename ResultType, bool ReverseProduct = false> static void sparse_sparse_to_dense_product_impl(const Lhs& lhs, const Rhs& rhs, ResultType& res) { using LhsScalar = typename remove_all_t<Lhs>::Scalar; using RhsScalar = typename remove_all_t<Rhs>::Scalar; + const sparse_scalar_product_op<LhsScalar, RhsScalar, ReverseProduct> product; Index cols = rhs.outerSize(); eigen_assert(lhs.outerSize() == rhs.innerSize()); @@ -240,7 +242,7 @@ for (typename evaluator<Lhs>::InnerIterator lhsIt(lhsEval, k); lhsIt; ++lhsIt) { Index i = lhsIt.index(); LhsScalar x = lhsIt.value(); - res.coeffRef(i, j) += x * y; + product.accumulate(res.coeffRef(i, j), x, y); } } } @@ -284,7 +286,7 @@ struct sparse_sparse_to_dense_product_selector<Lhs, Rhs, ResultType, RowMajor, RowMajor> { static void run(const Lhs& lhs, const Rhs& rhs, ResultType& res) { Transpose<ResultType> trRes(res); - internal::sparse_sparse_to_dense_product_impl<Rhs, Lhs, Transpose<ResultType>>(rhs, lhs, trRes); + internal::sparse_sparse_to_dense_product_impl<Rhs, Lhs, Transpose<ResultType>, true>(rhs, lhs, trRes); } };
diff --git a/Eigen/src/SparseCore/SparseCwiseBinaryOp.h b/Eigen/src/SparseCore/SparseCwiseBinaryOp.h index a80e19e..477c456 100644 --- a/Eigen/src/SparseCore/SparseCwiseBinaryOp.h +++ b/Eigen/src/SparseCore/SparseCwiseBinaryOp.h
@@ -88,11 +88,11 @@ ++m_rhsIter; } else if (m_lhsIter && (!m_rhsIter || (m_lhsIter.index() < m_rhsIter.index()))) { m_id = convert_index<StorageIndex>(m_lhsIter.index()); - m_value = m_functor(m_lhsIter.value(), Scalar(0)); + m_value = m_functor(m_lhsIter.value(), typename Rhs::Scalar(0)); ++m_lhsIter; } else if (m_rhsIter && (!m_lhsIter || (m_lhsIter.index() > m_rhsIter.index()))) { m_id = convert_index<StorageIndex>(m_rhsIter.index()); - m_value = m_functor(Scalar(0), m_rhsIter.value()); + m_value = m_functor(typename Lhs::Scalar(0), m_rhsIter.value()); ++m_rhsIter; } else { m_id = -1; @@ -164,12 +164,13 @@ EIGEN_STRONG_INLINE InnerIterator& operator++() { ++m_id; if (m_id < m_innerSize) { - Scalar lhsVal = m_lhsEval.coeff(IsRowMajor ? m_rhsIter.outer() : m_id, IsRowMajor ? m_id : m_rhsIter.outer()); + typename Lhs::Scalar lhsVal = + m_lhsEval.coeff(IsRowMajor ? m_rhsIter.outer() : m_id, IsRowMajor ? m_id : m_rhsIter.outer()); if (m_rhsIter && m_rhsIter.index() == m_id) { m_value = m_functor(lhsVal, m_rhsIter.value()); ++m_rhsIter; } else - m_value = m_functor(lhsVal, Scalar(0)); + m_value = m_functor(lhsVal, typename Rhs::Scalar(0)); } return *this; @@ -245,12 +246,13 @@ EIGEN_STRONG_INLINE InnerIterator& operator++() { ++m_id; if (m_id < m_innerSize) { - Scalar rhsVal = m_rhsEval.coeff(IsRowMajor ? m_lhsIter.outer() : m_id, IsRowMajor ? m_id : m_lhsIter.outer()); + typename Rhs::Scalar rhsVal = + m_rhsEval.coeff(IsRowMajor ? m_lhsIter.outer() : m_id, IsRowMajor ? m_id : m_lhsIter.outer()); if (m_lhsIter && m_lhsIter.index() == m_id) { m_value = m_functor(m_lhsIter.value(), rhsVal); ++m_lhsIter; } else - m_value = m_functor(Scalar(0), rhsVal); + m_value = m_functor(typename Lhs::Scalar(0), rhsVal); } return *this;
diff --git a/Eigen/src/SparseCore/SparseDenseProduct.h b/Eigen/src/SparseCore/SparseDenseProduct.h index f0dfeae..9186a90 100644 --- a/Eigen/src/SparseCore/SparseDenseProduct.h +++ b/Eigen/src/SparseCore/SparseDenseProduct.h
@@ -18,6 +18,28 @@ namespace internal { +template <typename Lhs, typename Rhs, typename Dest, typename Alpha> +void ordered_sparse_dense_add(Dest& dst, const Lhs& lhs, const Rhs& rhs, const Alpha& alpha, true_type) { + if (numext::is_exactly_zero(alpha)) return; + evaluator<Lhs> lhsEval(lhs); + evaluator<Rhs> rhsEval(rhs); + for (Index outer = 0; outer < lhs.outerSize(); ++outer) + for (typename evaluator<Lhs>::InnerIterator it(lhsEval, outer); it; ++it) + for (Index col = 0; col < rhs.cols(); ++col) + dst.coeffRef(it.row(), col) += alpha * (it.value() * rhsEval.coeff(it.col(), col)); +} + +template <typename Lhs, typename Rhs, typename Dest, typename Alpha> +void ordered_sparse_dense_add(Dest& dst, const Lhs& lhs, const Rhs& rhs, const Alpha& alpha, false_type) { + if (numext::is_exactly_zero(alpha)) return; + evaluator<Lhs> lhsEval(lhs); + evaluator<Rhs> rhsEval(rhs); + for (Index outer = 0; outer < rhs.outerSize(); ++outer) + for (typename evaluator<Rhs>::InnerIterator it(rhsEval, outer); it; ++it) + for (Index row = 0; row < lhs.rows(); ++row) + dst.coeffRef(row, it.col()) += alpha * (lhsEval.coeff(row, it.row()) * it.value()); +} + template <> struct product_promote_storage_type<Sparse, Dense, OuterProduct> { using ret = Sparse; @@ -462,7 +484,19 @@ using RhsNested = typename nested_eval<Rhs, ((Lhs::Flags & RowMajorBit) == 0) ? 1 : Dynamic>::type; LhsNested lhsNested(lhs); RhsNested rhsNested(rhs); - internal::sparse_time_dense_product(lhsNested, rhsNested, dst, alpha); + run(dst, lhsNested, rhsNested, alpha, + functor_is_commutative<scalar_product_op<typename Lhs::Scalar, typename Rhs::Scalar>>()); + } + + private: + template <typename Dest, typename NestedLhs, typename NestedRhs> + static void run(Dest& dst, const NestedLhs& lhs, const NestedRhs& rhs, const Scalar& alpha, true_type) { + internal::sparse_time_dense_product(lhs, rhs, dst, alpha); + } + + template <typename Dest, typename NestedLhs, typename NestedRhs> + static void run(Dest& dst, const NestedLhs& lhs, const NestedRhs& rhs, const Scalar& alpha, false_type) { + ordered_sparse_dense_add(dst, lhs, rhs, alpha, true_type()); } }; @@ -483,9 +517,21 @@ LhsNested lhsNested(lhs); RhsNested rhsNested(rhs); - // transpose everything + run(dst, lhsNested, rhsNested, alpha, + functor_is_commutative<scalar_product_op<typename Lhs::Scalar, typename Rhs::Scalar>>()); + } + + private: + template <typename Dst, typename NestedLhs, typename NestedRhs> + static void run(Dst& dst, const NestedLhs& lhs, const NestedRhs& rhs, const Scalar& alpha, true_type) { + // Transposition reverses coefficient multiplication, so this needs commutative scalars. Transpose<Dst> dstT(dst); - internal::sparse_time_dense_product(rhsNested.transpose(), lhsNested.transpose(), dstT, alpha); + internal::sparse_time_dense_product(rhs.transpose(), lhs.transpose(), dstT, alpha); + } + + template <typename Dst, typename NestedLhs, typename NestedRhs> + static void run(Dst& dst, const NestedLhs& lhs, const NestedRhs& rhs, const Scalar& alpha, false_type) { + ordered_sparse_dense_add(dst, lhs, rhs, alpha, false_type()); } }; @@ -511,6 +557,8 @@ using RhsEval = evaluator<ActualRhs>; using LhsIterator = typename evaluator<ActualLhs>::InnerIterator; using Scalar = typename ProdXprType::Scalar; + using FactorScalar = typename ActualRhs::Scalar; + using ProductOp = sparse_scalar_product_op<typename ActualLhs::Scalar, FactorScalar, NeedToTranspose>; public: enum { Flags = NeedToTranspose ? RowMajorBit : 0, CoeffReadCost = HugeCost }; @@ -527,22 +575,22 @@ EIGEN_STRONG_INLINE Index row() const { return NeedToTranspose ? m_outer : LhsIterator::index(); } EIGEN_STRONG_INLINE Index col() const { return NeedToTranspose ? LhsIterator::index() : m_outer; } - EIGEN_STRONG_INLINE Scalar value() const { return LhsIterator::value() * m_factor; } + EIGEN_STRONG_INLINE Scalar value() const { return ProductOp()(LhsIterator::value(), m_factor); } EIGEN_STRONG_INLINE operator bool() const { return LhsIterator::operator bool() && (!m_empty); } protected: - Scalar get(const RhsEval& rhs, Index outer, Dense = Dense()) const { return rhs.coeff(outer); } + FactorScalar get(const RhsEval& rhs, Index outer, Dense = Dense()) const { return rhs.coeff(outer); } - Scalar get(const RhsEval& rhs, Index outer, Sparse = Sparse()) { + FactorScalar get(const RhsEval& rhs, Index outer, Sparse = Sparse()) { typename RhsEval::InnerIterator it(rhs, outer); - if (it && it.index() == 0 && it.value() != Scalar(0)) return it.value(); + if (it && it.index() == 0 && it.value() != FactorScalar(0)) return it.value(); m_empty = true; - return Scalar(0); + return FactorScalar(0); } Index m_outer; bool m_empty; - Scalar m_factor; + FactorScalar m_factor; }; sparse_dense_outer_product_evaluator(const Lhs1& lhs, const ActualRhs& rhs)
diff --git a/Eigen/src/SparseCore/SparseDiagonalProduct.h b/Eigen/src/SparseCore/SparseDiagonalProduct.h index 10bac4a..62bd047 100644 --- a/Eigen/src/SparseCore/SparseDiagonalProduct.h +++ b/Eigen/src/SparseCore/SparseDiagonalProduct.h
@@ -32,13 +32,14 @@ enum { SDP_AsScalarProduct, SDP_AsCwiseProduct }; -template <typename SparseXprType, typename DiagonalCoeffType, int SDP_Tag> +template <typename SparseXprType, typename DiagonalCoeffType, int SDP_Tag, bool DiagonalOnLeft = true> struct sparse_diagonal_product_evaluator; template <typename Lhs, typename Rhs, int ProductTag> struct product_evaluator<Product<Lhs, Rhs, DefaultProduct>, ProductTag, DiagonalShape, SparseShape> : public sparse_diagonal_product_evaluator<Rhs, typename Lhs::DiagonalVectorType, - Rhs::Flags & RowMajorBit ? SDP_AsScalarProduct : SDP_AsCwiseProduct> { + Rhs::Flags & RowMajorBit ? SDP_AsScalarProduct : SDP_AsCwiseProduct, + true> { using XprType = Product<Lhs, Rhs, DefaultProduct>; enum { CoeffReadCost = HugeCost, @@ -46,15 +47,17 @@ Alignment = 0 }; // FIXME: compute proper CoeffReadCost and propagate Flags. - using Base = sparse_diagonal_product_evaluator<Rhs, typename Lhs::DiagonalVectorType, - Rhs::Flags & RowMajorBit ? SDP_AsScalarProduct : SDP_AsCwiseProduct>; + using Base = + sparse_diagonal_product_evaluator<Rhs, typename Lhs::DiagonalVectorType, + Rhs::Flags & RowMajorBit ? SDP_AsScalarProduct : SDP_AsCwiseProduct, true>; explicit product_evaluator(const XprType& xpr) : Base(xpr.rhs(), xpr.lhs().diagonal()) {} }; template <typename Lhs, typename Rhs, int ProductTag> struct product_evaluator<Product<Lhs, Rhs, DefaultProduct>, ProductTag, SparseShape, DiagonalShape> : public sparse_diagonal_product_evaluator<Lhs, Transpose<const typename Rhs::DiagonalVectorType>, - Lhs::Flags & RowMajorBit ? SDP_AsCwiseProduct : SDP_AsScalarProduct> { + Lhs::Flags & RowMajorBit ? SDP_AsCwiseProduct : SDP_AsScalarProduct, + false> { using XprType = Product<Lhs, Rhs, DefaultProduct>; enum { CoeffReadCost = HugeCost, @@ -62,8 +65,9 @@ Alignment = 0 }; // FIXME: compute proper CoeffReadCost and propagate Flags. - using Base = sparse_diagonal_product_evaluator<Lhs, Transpose<const typename Rhs::DiagonalVectorType>, - Lhs::Flags & RowMajorBit ? SDP_AsCwiseProduct : SDP_AsScalarProduct>; + using Base = + sparse_diagonal_product_evaluator<Lhs, Transpose<const typename Rhs::DiagonalVectorType>, + Lhs::Flags & RowMajorBit ? SDP_AsCwiseProduct : SDP_AsScalarProduct, false>; explicit product_evaluator(const XprType& xpr) : Base(xpr.lhs(), xpr.rhs().diagonal().transpose()) {} }; @@ -210,11 +214,15 @@ using materialized_right_sparse_product_evaluator_base<Lhs, Rhs>::materialized_right_sparse_product_evaluator_base; }; -template <typename SparseXprType, typename DiagonalCoeffType> -struct sparse_diagonal_product_evaluator<SparseXprType, DiagonalCoeffType, SDP_AsScalarProduct> { +template <typename SparseXprType, typename DiagonalCoeffType, bool DiagonalOnLeft> +struct sparse_diagonal_product_evaluator<SparseXprType, DiagonalCoeffType, SDP_AsScalarProduct, DiagonalOnLeft> { protected: using SparseXprInnerIterator = typename evaluator<SparseXprType>::InnerIterator; - using Scalar = typename SparseXprType::Scalar; + using SparseScalar = typename SparseXprType::Scalar; + using DiagonalScalar = typename DiagonalCoeffType::Scalar; + using Scalar = typename std::conditional_t<DiagonalOnLeft, ScalarBinaryOpTraits<DiagonalScalar, SparseScalar>, + ScalarBinaryOpTraits<SparseScalar, DiagonalScalar>>::ReturnType; + using ProductOp = sparse_scalar_product_op<DiagonalScalar, SparseScalar, !DiagonalOnLeft>; public: class InnerIterator : public SparseXprInnerIterator { @@ -222,7 +230,7 @@ InnerIterator(const sparse_diagonal_product_evaluator& xprEval, Index outer) : SparseXprInnerIterator(xprEval.m_sparseXprImpl, outer), m_coeff(xprEval.m_diagCoeffImpl.coeff(outer)) {} - EIGEN_STRONG_INLINE Scalar value() const { return m_coeff * SparseXprInnerIterator::value(); } + EIGEN_STRONG_INLINE Scalar value() const { return ProductOp()(m_coeff, SparseXprInnerIterator::value()); } protected: typename DiagonalCoeffType::Scalar m_coeff; @@ -238,9 +246,13 @@ evaluator<DiagonalCoeffType> m_diagCoeffImpl; }; -template <typename SparseXprType, typename DiagCoeffType> -struct sparse_diagonal_product_evaluator<SparseXprType, DiagCoeffType, SDP_AsCwiseProduct> { - using Scalar = typename SparseXprType::Scalar; +template <typename SparseXprType, typename DiagCoeffType, bool DiagonalOnLeft> +struct sparse_diagonal_product_evaluator<SparseXprType, DiagCoeffType, SDP_AsCwiseProduct, DiagonalOnLeft> { + using SparseScalar = typename SparseXprType::Scalar; + using DiagonalScalar = typename DiagCoeffType::Scalar; + using Scalar = typename std::conditional_t<DiagonalOnLeft, ScalarBinaryOpTraits<DiagonalScalar, SparseScalar>, + ScalarBinaryOpTraits<SparseScalar, DiagonalScalar>>::ReturnType; + using ProductOp = sparse_scalar_product_op<SparseScalar, DiagonalScalar, DiagonalOnLeft>; using StorageIndex = typename SparseXprType::StorageIndex; using DiagCoeffNested = @@ -254,7 +266,9 @@ InnerIterator(const sparse_diagonal_product_evaluator& xprEval, Index outer) : m_sparseIter(xprEval.m_sparseXprEval, outer), m_diagCoeffNested(xprEval.m_diagCoeffNested) {} - inline Scalar value() const { return m_sparseIter.value() * m_diagCoeffNested.coeff(index()); } + EIGEN_STRONG_INLINE Scalar value() const { + return ProductOp()(m_sparseIter.value(), m_diagCoeffNested.coeff(index())); + } inline StorageIndex index() const { return m_sparseIter.index(); } inline Index outer() const { return m_sparseIter.outer(); } inline Index col() const { return SparseXprType::IsRowMajor ? m_sparseIter.index() : m_sparseIter.outer(); }
diff --git a/Eigen/src/SparseCore/SparseDot.h b/Eigen/src/SparseCore/SparseDot.h index bcd4916..5d22094 100644 --- a/Eigen/src/SparseCore/SparseDot.h +++ b/Eigen/src/SparseCore/SparseDot.h
@@ -86,8 +86,7 @@ template <typename Derived> inline typename NumTraits<typename internal::traits<Derived>::Scalar>::Real SparseMatrixBase<Derived>::norm() const { - using std::sqrt; - return sqrt(squaredNorm()); + return numext::sqrt(squaredNorm()); } template <typename Derived>
diff --git a/Eigen/src/SparseCore/SparseMatrixBase.h b/Eigen/src/SparseCore/SparseMatrixBase.h index 6506967..d2c6b86 100644 --- a/Eigen/src/SparseCore/SparseMatrixBase.h +++ b/Eigen/src/SparseCore/SparseMatrixBase.h
@@ -331,10 +331,9 @@ template <typename OtherDerived> struct CwiseProductDenseReturnType { - using Type = CwiseBinaryOp< - internal::scalar_product_op<typename ScalarBinaryOpTraits< - typename internal::traits<Derived>::Scalar, typename internal::traits<OtherDerived>::Scalar>::ReturnType>, - const Derived, const OtherDerived>; + using Type = CwiseBinaryOp<internal::scalar_product_op<typename internal::traits<Derived>::Scalar, + typename internal::traits<OtherDerived>::Scalar>, + const Derived, const OtherDerived>; }; template <typename OtherDerived>
diff --git a/Eigen/src/SparseCore/SparseProduct.h b/Eigen/src/SparseCore/SparseProduct.h index 2ad7708..62c75c3 100644 --- a/Eigen/src/SparseCore/SparseProduct.h +++ b/Eigen/src/SparseCore/SparseProduct.h
@@ -151,7 +151,6 @@ using Base = evaluator<PlainObject>; explicit unary_evaluator(const XprType& xpr) : m_result(xpr.rows(), xpr.cols()) { - using std::abs; internal::construct_at<Base>(this, m_result); using LhsNested = typename nested_eval<Lhs, Dynamic>::type; using RhsNested = typename nested_eval<Rhs, Dynamic>::type; @@ -160,7 +159,8 @@ internal::sparse_sparse_product_with_pruning_selector<remove_all_t<LhsNested>, remove_all_t<RhsNested>, PlainObject>::run(lhsNested, rhsNested, m_result, - abs(xpr.reference()) * xpr.epsilon()); + numext::abs(xpr.reference()) * + xpr.epsilon()); } protected:
diff --git a/Eigen/src/SparseCore/SparseSparseProductWithPruning.h b/Eigen/src/SparseCore/SparseSparseProductWithPruning.h index 3df0f42..773511f 100644 --- a/Eigen/src/SparseCore/SparseSparseProductWithPruning.h +++ b/Eigen/src/SparseCore/SparseSparseProductWithPruning.h
@@ -19,11 +19,13 @@ namespace internal { // perform a pseudo in-place sparse * sparse product assuming all matrices are col major -template <typename Lhs, typename Rhs, typename ResultType> +template <typename Lhs, typename Rhs, typename ResultType, bool ReverseProduct = false> static void sparse_sparse_product_with_pruning_impl(const Lhs& lhs, const Rhs& rhs, ResultType& res, const typename ResultType::RealScalar& tolerance) { + using LhsScalar = typename remove_all_t<Lhs>::Scalar; using RhsScalar = typename remove_all_t<Rhs>::Scalar; using ResScalar = typename remove_all_t<ResultType>::Scalar; + const sparse_scalar_product_op<LhsScalar, RhsScalar, ReverseProduct> product; using StorageIndex = typename remove_all_t<Lhs>::StorageIndex; // make sure to call innerSize/outerSize since we fake the storage order. @@ -63,7 +65,7 @@ tempVector.restart(); RhsScalar x = rhsIt.value(); for (typename evaluator<Lhs>::InnerIterator lhsIt(lhsEval, rhsIt.index()); lhsIt; ++lhsIt) { - tempVector.coeffRef(lhsIt.index()) += lhsIt.value() * x; + product.accumulate(tempVector.coeffRef(lhsIt.index()), lhsIt.value(), x); } } res.startVec(j); @@ -107,7 +109,7 @@ static void run(const Lhs& lhs, const Rhs& rhs, ResultType& res, const RealScalar& tolerance) { // let's transpose the product to get a column x column product remove_all_t<ResultType> res_{res.rows(), res.cols()}; - internal::sparse_sparse_product_with_pruning_impl<Rhs, Lhs, ResultType>(rhs, lhs, res_, tolerance); + internal::sparse_sparse_product_with_pruning_impl<Rhs, Lhs, ResultType, true>(rhs, lhs, res_, tolerance); res.swap(res_); } };
diff --git a/Eigen/src/SparseCore/SparseUtil.h b/Eigen/src/SparseCore/SparseUtil.h index f747028..6395172 100644 --- a/Eigen/src/SparseCore/SparseUtil.h +++ b/Eigen/src/SparseCore/SparseUtil.h
@@ -59,6 +59,31 @@ namespace internal { +// Retain the existing evaluation order for products declared commutative. +template <typename LhsScalar, typename RhsScalar, bool Reverse, + bool Commutative = functor_is_commutative<scalar_product_op<LhsScalar, RhsScalar>>::value> +struct sparse_scalar_product_op { + EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE auto operator()(const LhsScalar& lhs, const RhsScalar& rhs) const { + return lhs * rhs; + } + // Keep multiply and addition in one expression for -ffp-contract=on. + template <typename Acc> + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void accumulate(Acc& acc, const LhsScalar& lhs, const RhsScalar& rhs) const { + acc += lhs * rhs; + } +}; + +template <typename LhsScalar, typename RhsScalar> +struct sparse_scalar_product_op<LhsScalar, RhsScalar, true, false> { + EIGEN_DEVICE_FUNC constexpr EIGEN_STRONG_INLINE auto operator()(const LhsScalar& lhs, const RhsScalar& rhs) const { + return rhs * lhs; + } + template <typename Acc> + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void accumulate(Acc& acc, const LhsScalar& lhs, const RhsScalar& rhs) const { + acc += rhs * lhs; + } +}; + /** \internal * Compile-time test for whether a sparse expression exposes its storage directly * through outerIndexPtr() / innerIndexPtr() / valuePtr() / innerNonZeroPtr()
diff --git a/Eigen/src/SparseLU/SparseLU.h b/Eigen/src/SparseLU/SparseLU.h index 7316cf6..c1842b0 100644 --- a/Eigen/src/SparseLU/SparseLU.h +++ b/Eigen/src/SparseLU/SparseLU.h
@@ -370,7 +370,6 @@ * \sa logAbsDeterminant(), signDeterminant() */ Scalar absDeterminant() const { - using std::abs; eigen_assert(m_factorizationIsOk && "The matrix should be factorized first."); // Initialize with the determinant of the row matrix Scalar det = Scalar(1.); @@ -379,7 +378,7 @@ for (Index j = 0; j < this->cols(); ++j) { for (typename SCMatrix::InnerIterator it(m_Lstore, j); it; ++it) { if (it.index() == j) { - det *= abs(it.value()); + det *= numext::abs(it.value()); break; } } @@ -398,16 +397,13 @@ * \sa absDeterminant(), signDeterminant() */ Scalar logAbsDeterminant() const { - using std::abs; - using std::log; - eigen_assert(m_factorizationIsOk && "The matrix should be factorized first."); Scalar det = Scalar(0.); for (Index j = 0; j < this->cols(); ++j) { for (typename SCMatrix::InnerIterator it(m_Lstore, j); it; ++it) { if (it.row() < j) continue; if (it.row() == j) { - det += log(abs(it.value())); + det += numext::log(numext::abs(it.value())); break; } } @@ -898,7 +894,6 @@ template <bool Conjugate, typename Dest> void solveTransposedInPlace(MatrixBase<Dest>& X) const { - using numext::conj; Index nrhs = X.cols(); // Forward solve with U for (Index k = 0; k <= m_mapL.nsuper(); k++) { @@ -912,13 +907,13 @@ typename MatrixUType::InnerIterator it(m_mapU, jcol); for (; it; ++it) { Index irow = it.index(); - X(jcol, j) -= X(irow, j) * (Conjugate ? conj(it.value()) : it.value()); + X(jcol, j) -= X(irow, j) * (Conjugate ? numext::conj(it.value()) : it.value()); } } } if (nsupc == 1) { for (Index j = 0; j < nrhs; j++) { - X(fsupc, j) /= (Conjugate ? conj(m_mapL.valuePtr()[luptr]) : m_mapL.valuePtr()[luptr]); + X(fsupc, j) /= (Conjugate ? numext::conj(m_mapL.valuePtr()[luptr]) : m_mapL.valuePtr()[luptr]); } } else { Map<const Matrix<Scalar, Dynamic, Dynamic, ColMajor>, 0, OuterStride<>> A(&(m_mapL.valuePtr()[luptr]), nsupc,
diff --git a/Eigen/src/SparseLU/SparseLU_SupernodalMatrix.h b/Eigen/src/SparseLU/SparseLU_SupernodalMatrix.h index be92743..bbcdc17 100644 --- a/Eigen/src/SparseLU/SparseLU_SupernodalMatrix.h +++ b/Eigen/src/SparseLU/SparseLU_SupernodalMatrix.h
@@ -248,7 +248,6 @@ template <typename Scalar, typename Index_> template <bool Conjugate, typename Dest> void MappedSuperNodalMatrix<Scalar, Index_>::solveTransposedInPlace(MatrixBase<Dest>& X) const { - using numext::conj; Index n = int(X.rows()); Index nrhs = Index(X.cols()); const Scalar* Lval = valuePtr(); // Nonzero values @@ -268,7 +267,7 @@ ++it; // Skip the diagonal element for (; it; ++it) { irow = it.row(); - X(fsupc, j) -= X(irow, j) * (Conjugate ? conj(it.value()) : it.value()); + X(fsupc, j) -= X(irow, j) * (Conjugate ? numext::conj(it.value()) : it.value()); } } } else {
diff --git a/Eigen/src/SparseLU/SparseLU_pivotL.h b/Eigen/src/SparseLU/SparseLU_pivotL.h index e781801..ddc2943 100644 --- a/Eigen/src/SparseLU/SparseLU_pivotL.h +++ b/Eigen/src/SparseLU/SparseLU_pivotL.h
@@ -81,8 +81,7 @@ RealScalar rtemp; Index isub, icol, itemp, k; for (isub = nsupc; isub < nsupr; ++isub) { - using std::abs; - rtemp = abs(lu_col_ptr[isub]); + rtemp = numext::abs(lu_col_ptr[isub]); if (rtemp > pivmax) { pivmax = rtemp; pivptr = isub; @@ -106,8 +105,7 @@ // Test if the diagonal element can be used as a pivot (given the threshold value) if (diag >= 0) { // Diagonal element exists - using std::abs; - rtemp = abs(lu_col_ptr[diag]); + rtemp = numext::abs(lu_col_ptr[diag]); if (rtemp != RealScalar(0.0) && rtemp >= thresh) pivptr = diag; } pivrow = lsub_ptr[pivptr];
diff --git a/Eigen/src/SparseQR/SparseQR.h b/Eigen/src/SparseQR/SparseQR.h index 9940138..331b8b4 100644 --- a/Eigen/src/SparseQR/SparseQR.h +++ b/Eigen/src/SparseQR/SparseQR.h
@@ -389,8 +389,6 @@ */ template <typename MatrixType, typename OrderingType> void SparseQR<MatrixType, OrderingType>::factorize(const MatrixType& mat) { - using std::abs; - eigen_assert(m_analysisIsok && "analyzePattern() should be called before this step"); // Reset state from any prior factorize() so info() and lastErrorMessage() @@ -579,8 +577,7 @@ beta = numext::real(c0); tval(Qidx(0)) = Scalar(1); } else { - using std::sqrt; - beta = sqrt(numext::abs2(c0) + sqrNorm); + beta = numext::sqrt<RealScalar>(numext::abs2(c0) + sqrNorm); if (numext::real(c0) >= RealScalar(0)) beta = -beta; tval(Qidx(0)) = Scalar(1); for (Index itq = 1; itq < nzcolQ; ++itq) tval(Qidx(itq)) /= (c0 - beta); @@ -597,7 +594,7 @@ } } - const RealScalar absBeta = abs(beta); + const RealScalar absBeta = numext::abs(beta); bool hasReplacement = false; const bool canRejectColumn = nonzeroCol + (n - col - 1) >= diagSize; // Gate for examining a pivot as replaceable. The Householder pivot `beta` of a @@ -609,8 +606,7 @@ // *examined* (and never rejects a pivot that lacks a replacement). A threshold at // the residual's noise floor would otherwise make the pivoting decision -- and // hence the result -- depend on floating-point contraction. - using std::sqrt; - const RealScalar weakPivotTolerance = sqrt(NumTraits<RealScalar>::epsilon()); + const RealScalar weakPivotTolerance = numext::sqrt(NumTraits<RealScalar>::epsilon()); const RealScalar maxReplaceablePivotThreshold = max2Norm * weakPivotTolerance; if (nonzeroCol < diagSize && canRejectColumn && m_useDefaultThreshold && absBeta >= pivotThreshold && absBeta < maxReplaceablePivotThreshold) {
diff --git a/Eigen/src/ThreadPool/CoreThreadPoolDevice.h b/Eigen/src/ThreadPool/CoreThreadPoolDevice.h index 30be008..28e381a 100644 --- a/Eigen/src/ThreadPool/CoreThreadPoolDevice.h +++ b/Eigen/src/ThreadPool/CoreThreadPoolDevice.h
@@ -296,28 +296,31 @@ struct dense_assignment_loop_with_device<Kernel, CoreThreadPoolDevice, LinearVectorizedTraversal, NoUnrolling> { using Scalar = typename Kernel::Scalar; using PacketType = typename Kernel::PacketType; - static constexpr Index XprEvaluationCost = cost_helper<Kernel>::Cost, - RequestedAlignment = Kernel::AssignmentTraits::LinearRequiredAlignment, - PacketSize = unpacket_traits<PacketType>::size, - DstIsAligned = Kernel::AssignmentTraits::DstAlignment >= RequestedAlignment, - DstAlignment = packet_traits<Scalar>::AlignedOnScalar ? RequestedAlignment - : Kernel::AssignmentTraits::DstAlignment, - SrcAlignment = Kernel::AssignmentTraits::JointAlignment; + static constexpr int PacketSize = unpacket_traits<PacketType>::size; + static constexpr int SrcAlignment = Kernel::AssignmentTraits::JointAlignment; + static constexpr int DstAlignment = plain_enum_max(Kernel::AssignmentTraits::DstAlignment, alignof(Scalar)); + static constexpr int RequestedAlignment = unpacket_traits<PacketType>::alignment; + static constexpr bool Alignable = (DstAlignment >= RequestedAlignment) || + (static_cast<std::size_t>(RequestedAlignment - DstAlignment) % sizeof(Scalar) == 0); + static constexpr int Alignment = Alignable ? RequestedAlignment : DstAlignment; + static constexpr bool DstIsAligned = DstAlignment >= Alignment; + static constexpr bool UsePacketSegment = Kernel::AssignmentTraits::UsePacketSegment; + static constexpr Index XprEvaluationCost = cost_helper<Kernel>::Cost; + struct AssignmentFunctor : public Kernel { EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE AssignmentFunctor(Kernel& kernel) : Kernel(kernel) {} EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void operator()(Index index) { - this->template assignPacket<DstAlignment, SrcAlignment, PacketType>(index); + this->template assignPacket<Alignment, SrcAlignment, PacketType>(index); } }; - static constexpr bool UsePacketSegment = Kernel::AssignmentTraits::UsePacketSegment; + using head_loop = unaligned_dense_assignment_loop<PacketType, DstAlignment, SrcAlignment, UsePacketSegment, DstIsAligned>; - using tail_loop = unaligned_dense_assignment_loop<PacketType, DstAlignment, SrcAlignment, UsePacketSegment, false>; + using tail_loop = unaligned_dense_assignment_loop<PacketType, Alignment, SrcAlignment, UsePacketSegment, false>; static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE void run(Kernel& kernel, CoreThreadPoolDevice& device) { const Index size = kernel.size(); - const Index alignedStart = - DstIsAligned ? 0 : internal::first_aligned<RequestedAlignment>(kernel.dstDataPtr(), size); + const Index alignedStart = DstIsAligned ? 0 : internal::first_aligned<Alignment>(kernel.dstDataPtr(), size); const Index alignedEnd = alignedStart + numext::round_down(size - alignedStart, PacketSize); head_loop::run(kernel, 0, alignedStart);
diff --git a/LICENSES/BSD-3-Clause-Open-MPI.txt b/LICENSES/BSD-3-Clause-Open-MPI.txt new file mode 100644 index 0000000..166a95b --- /dev/null +++ b/LICENSES/BSD-3-Clause-Open-MPI.txt
@@ -0,0 +1,34 @@ +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +- Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer listed + in this license in the documentation and/or other materials + provided with the distribution. + +- Neither the name of the copyright holders nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +The copyright holders provide no reassurances that the source code +provided does not infringe any patent, copyright, or any other +intellectual property rights of third parties. The copyright holders +disclaim any liability to any recipient for claims brought against +recipient by any third party for infringement of that parties +intellectual property rights. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/REUSE.toml b/REUSE.toml index d8a1a69..3adc938 100644 --- a/REUSE.toml +++ b/REUSE.toml
@@ -137,16 +137,18 @@ SPDX-FileCopyrightText = "The Eigen Authors" SPDX-License-Identifier = "MPL-2.0" -# Reference BLAS test data files (netlib reference distribution: Univ. of -# Tennessee / Berkeley / Colorado Denver — modified BSD-3-Clause). +# Reference BLAS test inputs: LAPACK's BLAS/*.in, identical to 3.6.0's apart +# from output file names and, in [cz]blat3.dat, disabled error-exit tests. +# LAPACK's license is BSD-3-Clause plus an IP disclaimer, i.e. +# BSD-3-Clause-Open-MPI. [[annotations]] path = "blas/testing/*.dat" SPDX-FileCopyrightText = [ - "1992-2013 The University of Tennessee and The University of Tennessee Research Foundation", - "2000-2013 The University of California Berkeley", - "2006-2013 The University of Colorado Denver", + "1992-2015 The University of Tennessee and The University of Tennessee Research Foundation", + "2000-2015 The University of California Berkeley", + "2006-2015 The University of Colorado Denver", ] -SPDX-License-Identifier = "BSD-3-Clause" +SPDX-License-Identifier = "BSD-3-Clause-Open-MPI" # MSVC autoexp data — mechanical pretty-print description, MPL-2.0. [[annotations]]
diff --git a/benchmarks/Cholesky/CMakeLists.txt b/benchmarks/Cholesky/CMakeLists.txt index ec05343..134f305 100644 --- a/benchmarks/Cholesky/CMakeLists.txt +++ b/benchmarks/Cholesky/CMakeLists.txt
@@ -9,3 +9,5 @@ eigen_add_benchmark(bench_llt_inverse_cplx bench_llt_inverse.cpp DEFINITIONS "SCALAR=std::complex<double>") eigen_add_benchmark(bench_llt_inverse_float bench_llt_inverse.cpp DEFINITIONS SCALAR=float) eigen_add_benchmark(bench_llt_inverse_cplxf bench_llt_inverse.cpp DEFINITIONS "SCALAR=std::complex<float>") + +eigen_add_benchmark(bench_cholesky_reconstruction bench_cholesky_reconstruction.cpp)
diff --git a/benchmarks/Cholesky/bench_cholesky.cpp b/benchmarks/Cholesky/bench_cholesky.cpp index a901aba..e010f6d 100644 --- a/benchmarks/Cholesky/bench_cholesky.cpp +++ b/benchmarks/Cholesky/bench_cholesky.cpp
@@ -15,20 +15,30 @@ static void BM_LDLT(benchmark::State& state) { int n = state.range(0); - typedef Matrix<Scalar, Dynamic, Dynamic> MatrixType; - typedef Matrix<Scalar, Dynamic, Dynamic> SquareMatrixType; + using MatrixType = Matrix<Scalar, Dynamic, Dynamic>; MatrixType a = MatrixType::Random(n, n); - SquareMatrixType covMat = a * a.adjoint(); + MatrixType covMat = a * a.adjoint(); + // In place: each compute() copies covMat into the bound storage once and allocates no factorization storage. + MatrixType storage = covMat; + LDLT<Ref<MatrixType> > cholnosqrt(storage); int r = internal::random<int>(0, n - 1); int c = internal::random<int>(0, n - 1); Scalar acc = 0; for (auto _ : state) { - LDLT<SquareMatrixType> cholnosqrt(covMat); + cholnosqrt.compute(covMat); acc += cholnosqrt.matrixL().coeff(r, c); benchmark::DoNotOptimize(acc); } + double cost = 0; + for (int j = 0; j < n; ++j) { + int rem = std::max(n - j - 1, 0); + cost += 2 * (rem * j + rem + j); + } + state.counters["GFLOPS"] = + benchmark::Counter(cost, benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); } -BENCHMARK(BM_LDLT)->RangeMultiplier(2)->Range(4, 1500); +// 96 and 97 bracket the unblocked/blocked crossover. +BENCHMARK(BM_LDLT)->RangeMultiplier(2)->Range(4, 1500)->Arg(96)->Arg(97); static void BM_LLT(benchmark::State& state) { int n = state.range(0);
diff --git a/benchmarks/Cholesky/bench_cholesky_reconstruction.cpp b/benchmarks/Cholesky/bench_cholesky_reconstruction.cpp new file mode 100644 index 0000000..138e9ae --- /dev/null +++ b/benchmarks/Cholesky/bench_cholesky_reconstruction.cpp
@@ -0,0 +1,54 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Cholesky> +#include <cstdlib> +#include <type_traits> + +namespace { +using namespace Eigen; + +template <typename Scalar, int Order, int UpLo, bool Indefinite> +void BM_CholeskyReconstruction(benchmark::State& state) { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + using RealScalar = typename MatrixType::RealScalar; + using Factorization = + typename std::conditional<Indefinite, BunchKaufman<MatrixType, UpLo>, LDLT<MatrixType, UpLo>>::type; + const Index size = state.range(0); + std::srand(3395); + const MatrixType random = MatrixType::Random(size, size); + MatrixType input(size, size); + if (Indefinite) { + input = random + random.adjoint(); + } else { + input.noalias() = random * random.adjoint(); + input.diagonal().array() += Scalar(1); + } + const Factorization factorization(input); + const MatrixType reconstructed = factorization.reconstructedMatrix(); + const RealScalar tolerance = RealScalar(64) * RealScalar(size) * NumTraits<RealScalar>::epsilon() * input.norm(); + if (factorization.info() != Success || !reconstructed.allFinite() || !((reconstructed - input).norm() <= tolerance)) { + state.SkipWithError("Incorrect Cholesky reconstruction"); + return; + } + for (auto _ : state) { + const MatrixType result = factorization.reconstructedMatrix(); + benchmark::DoNotOptimize(result.data()); + benchmark::ClobberMemory(); + } +} + +// Indefinite=false uses LDLT; true uses BunchKaufman. Factorization is outside timing. +#define EIGEN_BENCH_RECONSTRUCTION(Scalar, Order, UpLo) \ + BENCHMARK_TEMPLATE(BM_CholeskyReconstruction, Scalar, Order, UpLo, false) \ + ->ArgsProduct({{4, 16, 64, 128, 256, 1024}}); \ + BENCHMARK_TEMPLATE(BM_CholeskyReconstruction, Scalar, Order, UpLo, true)->ArgsProduct({{4, 16, 64, 128, 256, 1024}}) + +EIGEN_BENCH_RECONSTRUCTION(double, ColMajor, Lower); +EIGEN_BENCH_RECONSTRUCTION(double, RowMajor, Upper); +EIGEN_BENCH_RECONSTRUCTION(std::complex<double>, ColMajor, Upper); +EIGEN_BENCH_RECONSTRUCTION(std::complex<double>, RowMajor, Lower); + +#undef EIGEN_BENCH_RECONSTRUCTION +} // namespace
diff --git a/benchmarks/Core/CMakeLists.txt b/benchmarks/Core/CMakeLists.txt index e87a4c8..6869484 100644 --- a/benchmarks/Core/CMakeLists.txt +++ b/benchmarks/Core/CMakeLists.txt
@@ -11,6 +11,11 @@ eigen_add_benchmark(bench_gemm bench_gemm.cpp) eigen_add_benchmark(bench_gemm_double bench_gemm.cpp DEFINITIONS SCALAR=double) endif() +# Keep -O2 visible: GCC's fixed-loop unrolling changes GEBP register allocation. +eigen_add_benchmark(bench_gemm_o2 bench_gemm.cpp) +eigen_add_benchmark(bench_gemm_double_o2 bench_gemm.cpp DEFINITIONS SCALAR=double) +target_compile_options(bench_gemm_o2 PRIVATE $<$<NOT:$<CXX_COMPILER_ID:MSVC>>:-O2>) +target_compile_options(bench_gemm_double_o2 PRIVATE $<$<NOT:$<CXX_COMPILER_ID:MSVC>>:-O2>) eigen_add_benchmark(bench_gemm_complex bench_gemm.cpp DEFINITIONS "SCALAR=std::complex<float>" EIGEN_BENCH_COMPLEX_SCALAR) eigen_add_benchmark(bench_gemm_complex_double bench_gemm.cpp @@ -21,14 +26,28 @@ eigen_add_benchmark(bench_gemm_scaled_double bench_gemm_scaled.cpp DEFINITIONS SCALAR=double) eigen_add_benchmark(bench_gemm_scaled_complex bench_gemm_scaled.cpp DEFINITIONS "SCALAR=std::complex<float>") eigen_add_benchmark(bench_gemm_scaled_complex_double bench_gemm_scaled.cpp DEFINITIONS "SCALAR=std::complex<double>") +eigen_add_benchmark(bench_gemm_cast bench_gemm_cast.cpp) +eigen_add_benchmark(bench_gemm_cast_bfloat16 bench_gemm_cast.cpp DEFINITIONS SCALAR=Eigen::bfloat16) eigen_add_benchmark(bench_gemv bench_gemv.cpp) +eigen_add_benchmark(bench_gemv_degenerate bench_gemv_degenerate.cpp) eigen_add_benchmark(bench_vecadd bench_vecadd.cpp) +foreach(scalar float double) + foreach(operation abs:0 square:1 add:2 copy:3 exp:4) + string(REPLACE ":" ";" operation ${operation}) + list(GET operation 0 name) + list(GET operation 1 index) + eigen_add_benchmark(bench_assignment_${name}_${scalar} bench_assignment.cpp + DEFINITIONS EIGEN_BENCH_ASSIGN_SCALAR=${scalar} EIGEN_BENCH_ASSIGN_OP=${index}) + endforeach() +endforeach() eigen_add_benchmark(bench_trsm bench_trsm.cpp) eigen_add_benchmark(bench_trsm_packet bench_trsm_packet.cpp) eigen_add_benchmark(bench_reverse bench_reverse.cpp) +eigen_add_benchmark(bench_stl_iterators bench_stl_iterators.cpp) eigen_add_benchmark(bench_permutation_product bench_permutation_product.cpp) eigen_add_benchmark(bench_permutation_transposition bench_permutation_transposition.cpp) eigen_add_benchmark(bench_move_semantics bench_move_semantics.cpp) +eigen_add_benchmark(bench_conservative_resize bench_conservative_resize.cpp) eigen_add_benchmark(bench_reductions bench_reductions.cpp) eigen_add_benchmark(bench_vectorwise_norm bench_vectorwise_norm.cpp) eigen_add_benchmark(bench_complex_components bench_complex_components.cpp) @@ -69,7 +88,10 @@ eigen_add_benchmark(bench_stable_norm bench_stable_norm.cpp) eigen_add_benchmark(bench_approx_comparisons bench_approx_comparisons.cpp) eigen_add_benchmark(bench_selfadjoint_l1norm bench_selfadjoint_l1norm.cpp) +eigen_add_benchmark(bench_selfadjoint_permutation bench_selfadjoint_permutation.cpp) eigen_add_benchmark(bench_cwise_math bench_cwise_math.cpp) +eigen_add_benchmark(bench_trig_reduction bench_trig_reduction.cpp) +eigen_add_benchmark(bench_optimization_barrier bench_optimization_barrier.cpp) eigen_add_benchmark(bench_complex_sqrt bench_complex_sqrt.cpp) eigen_add_benchmark(bench_cast bench_cast.cpp) eigen_add_benchmark(bench_complex_log bench_complex_log.cpp) @@ -84,6 +106,7 @@ eigen_add_benchmark(bench_diagonal_move bench_diagonal_move.cpp) eigen_add_benchmark(bench_triangular_product bench_triangular_product.cpp) eigen_add_benchmark(bench_selfadjoint_product bench_selfadjoint_product.cpp) +eigen_add_benchmark(bench_gemm_packing bench_gemm_packing.cpp) eigen_add_benchmark(bench_symv bench_symv.cpp) eigen_add_benchmark(bench_trmv bench_trmv.cpp) eigen_add_benchmark(bench_syr bench_syr.cpp) @@ -104,6 +127,7 @@ eigen_add_benchmark(bench_small_dynamic_product bench_small_dynamic_product.cpp) eigen_add_benchmark(bench_scaled_product bench_scaled_product.cpp) eigen_add_benchmark(bench_outer_product bench_outer_product.cpp) +eigen_add_benchmark(bench_product_storage_order bench_product_storage_order.cpp) eigen_add_benchmark(bench_selfadjoint_diagonal bench_selfadjoint_diagonal.cpp) eigen_add_benchmark(bench_segment_tail bench_segment_tail.cpp) eigen_add_benchmark(bench_boolean_ops bench_boolean_ops.cpp)
diff --git a/benchmarks/Core/bench_assignment.cpp b/benchmarks/Core/bench_assignment.cpp new file mode 100644 index 0000000..5738b30 --- /dev/null +++ b/benchmarks/Core/bench_assignment.cpp
@@ -0,0 +1,75 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> +#include <cstdint> + +#ifndef EIGEN_BENCH_ASSIGN_SCALAR +#define EIGEN_BENCH_ASSIGN_SCALAR double +#endif +#ifndef EIGEN_BENCH_ASSIGN_OP +// 0: abs, 1: square, 2: add, 3: copy, 4: exp. +#define EIGEN_BENCH_ASSIGN_OP 0 +#endif + +using namespace Eigen; + +static void BM_Assignment(benchmark::State& state) { + using Scalar = EIGEN_BENCH_ASSIGN_SCALAR; + using Vector = Array<Scalar, Dynamic, 1>; + // Control the low address bits independently: matching page offsets can trigger false store/load dependencies. + constexpr Index pageSize = 4096; + const Index size = state.range(0); + Vector storage(3 * size + 3 * pageSize / sizeof(Scalar)); + Scalar* source = storage.data() + internal::first_aligned<pageSize>(storage.data(), storage.size()); + const Index separation = (((size + 1) * sizeof(Scalar) + pageSize - 1) / pageSize) * pageSize / sizeof(Scalar); + Map<Vector> input(source + state.range(1), size); + Map<Vector> output(source + separation + state.range(2), size); + Vector other(size); + for (Index i = 0; i < size; ++i) { + input(i) = Scalar(i % 127 - 63) / Scalar(32); + other(i) = Scalar(i % 61 - 30) / Scalar(16); + } + for (auto _ : state) { + benchmark::DoNotOptimize(input.data()); + benchmark::DoNotOptimize(other.data()); +#if EIGEN_BENCH_ASSIGN_OP == 0 + output = input.abs(); +#elif EIGEN_BENCH_ASSIGN_OP == 1 + output = input.square(); +#elif EIGEN_BENCH_ASSIGN_OP == 2 + output = input + other; +#elif EIGEN_BENCH_ASSIGN_OP == 3 + output = input; +#else + output = input.exp(); +#endif + benchmark::DoNotOptimize(output.data()); + benchmark::ClobberMemory(); + } + for (Index i = 0; i < size; ++i) { +#if EIGEN_BENCH_ASSIGN_OP == 0 + const Scalar expected = numext::abs(input(i)); +#elif EIGEN_BENCH_ASSIGN_OP == 1 + const Scalar expected = input(i) * input(i); +#elif EIGEN_BENCH_ASSIGN_OP == 2 + const Scalar expected = input(i) + other(i); +#elif EIGEN_BENCH_ASSIGN_OP == 3 + const Scalar expected = input(i); +#else + const Scalar expected = numext::exp(input(i)); +#endif + if (!(numext::abs(output(i) - expected) <= Scalar(8) * NumTraits<Scalar>::epsilon() * numext::abs(expected))) { + state.SkipWithError("assignment differs from the scalar reference"); + return; + } + } + state.SetBytesProcessed(state.iterations() * size * sizeof(Scalar) * (EIGEN_BENCH_ASSIGN_OP == 2 ? 3 : 2)); +} + +BENCHMARK(BM_Assignment) + ->ArgsProduct({{31, 32, 33, 127, 128, 129, 255, 256, 257, 1023, 1024, 1025, 4095, 4096, 4097, 16384, 65536, + 1048576}, + {0, 1}, + {0, 16}});
diff --git a/benchmarks/Core/bench_block_ops.cpp b/benchmarks/Core/bench_block_ops.cpp index 1e3e137..41d8e9a 100644 --- a/benchmarks/Core/bench_block_ops.cpp +++ b/benchmarks/Core/bench_block_ops.cpp
@@ -72,6 +72,44 @@ state.SetBytesProcessed(state.iterations() * n * k * sizeof(Scalar)); } +// Copy and sum a fixed block spanning whole columns, which is contiguous like middleCols<4>. +static void BM_FixedFullColumnsCopy(benchmark::State& state) { + Matrix<float, 6, 6> src = Matrix<float, 6, 6>::Random(); + Matrix<float, 6, 4> dst; + Index j = 1; + for (auto _ : state) { + benchmark::DoNotOptimize(j); + dst = src.block<6, 4>(0, j); + benchmark::DoNotOptimize(dst.data()); + } +} + +static void BM_FixedFullColumnsSum(benchmark::State& state) { + Matrix<float, 6, 6> src = Matrix<float, 6, 6>::Random(); + Index j = 1; + for (auto _ : state) { + benchmark::DoNotOptimize(j); + benchmark::DoNotOptimize(src.block<6, 4>(0, j).sum()); + } +} + +// Copy a block of a product, which forwards packets to the product's temporary. +static void BM_ProductBlockCopy(benchmark::State& state) { + Matrix<float, 8, 8> a = Matrix<float, 8, 8>::Random(), b = Matrix<float, 8, 8>::Random(); + Matrix<float, 8, 4> dst; + Index j = 1; + for (auto _ : state) { + benchmark::DoNotOptimize(j); + benchmark::DoNotOptimize(a.data()); + dst = (a * b).block<8, 4>(0, j); + benchmark::DoNotOptimize(dst.data()); + } +} + +BENCHMARK(BM_FixedFullColumnsCopy); +BENCHMARK(BM_FixedFullColumnsSum); +BENCHMARK(BM_ProductBlockCopy); + // (matrix_size, block_size) // clang-format off #define BLOCK_SIZES ->ArgsProduct({{256, 512, 1024}, {16, 64, 128}})
diff --git a/benchmarks/Core/bench_conservative_resize.cpp b/benchmarks/Core/bench_conservative_resize.cpp new file mode 100644 index 0000000..fcf04c0 --- /dev/null +++ b/benchmarks/Core/bench_conservative_resize.cpp
@@ -0,0 +1,59 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> +#include "../../test/MovableScalar.h" +#include <complex> + +using Complex = std::complex<double>; +using NonTrivialScalar = Eigen::MovableScalar<Complex>; + +template <typename Scalar, int Capacity> +static void BM_ConservativeResizeAppend(benchmark::State& state) { + using Vector = Eigen::Array<Scalar, Eigen::Dynamic, 1, Eigen::ColMajor, Capacity, 1>; + const Eigen::Index size = state.range(0); + const Scalar value(Complex(1, 2)); + Vector v; + for (auto _ : state) { + v.resize(0); + for (Eigen::Index i = 0; i < size; ++i) { + v.conservativeResize(i + 1); + v(i) = value; + benchmark::DoNotOptimize(v.data()); + benchmark::ClobberMemory(); + } + } + for (Eigen::Index i = 0; i < size; ++i) { + if (Complex(v(i)) != Complex(1, 2)) state.SkipWithError("Appended coefficient was not preserved"); + } + state.SetItemsProcessed(state.iterations() * size); +} + +template <int Storage> +static void BM_ConservativeResizeOuter(benchmark::State& state) { + using BoundedMatrix = Eigen::Matrix<NonTrivialScalar, Eigen::Dynamic, Eigen::Dynamic, Storage, + Storage == Eigen::RowMajor ? 256 : 3, Storage == Eigen::RowMajor ? 3 : 256>; + const Eigen::Index size = state.range(0); + const NonTrivialScalar value(Complex(1, 2)); + BoundedMatrix m; + for (auto _ : state) { + m.resize(Storage == Eigen::RowMajor ? 0 : 3, Storage == Eigen::RowMajor ? 3 : 0); + for (Eigen::Index i = 0; i < size; ++i) { + m.conservativeResize(Storage == Eigen::RowMajor ? i + 1 : 3, Storage == Eigen::RowMajor ? 3 : i + 1); + for (Eigen::Index j = 0; j < 3; ++j) m.data()[3 * i + j] = value; + benchmark::DoNotOptimize(m.data()); + benchmark::ClobberMemory(); + } + } + for (Eigen::Index i = 0; i < m.size(); ++i) { + if (Complex(m.data()[i]) != Complex(1, 2)) state.SkipWithError("Appended coefficient was not preserved"); + } + state.SetItemsProcessed(state.iterations() * size * 3); +} + +BENCHMARK_TEMPLATE(BM_ConservativeResizeAppend, Complex, 256)->Arg(16)->Arg(64)->Arg(200)->Arg(256); +BENCHMARK_TEMPLATE(BM_ConservativeResizeAppend, NonTrivialScalar, 256)->Arg(16)->Arg(64)->Arg(200)->Arg(256); +BENCHMARK_TEMPLATE(BM_ConservativeResizeAppend, NonTrivialScalar, Eigen::Dynamic)->Arg(16)->Arg(64)->Arg(200)->Arg(256); +BENCHMARK_TEMPLATE(BM_ConservativeResizeOuter, Eigen::RowMajor)->Arg(16)->Arg(64)->Arg(200)->Arg(256); +BENCHMARK_TEMPLATE(BM_ConservativeResizeOuter, Eigen::ColMajor)->Arg(16)->Arg(64)->Arg(200)->Arg(256);
diff --git a/benchmarks/Core/bench_cwise_math.cpp b/benchmarks/Core/bench_cwise_math.cpp index 2428af4..19104e7 100644 --- a/benchmarks/Core/bench_cwise_math.cpp +++ b/benchmarks/Core/bench_cwise_math.cpp
@@ -260,6 +260,9 @@ BENCH_CWISE_BINARY(Min, a.min(b), -100, 100) BENCH_CWISE_BINARY(Max, a.max(b), -100, 100) +BENCH_CWISE_BINARY(Add, a + b, -100, 100) +BENCH_CWISE_BINARY(Mul, (a * b), -100, 100) +BENCH_CWISE_BINARY(Div, a / b, 1, 100) // Power: array^scalar template <typename Scalar> @@ -356,6 +359,14 @@ ->ArgNames({"size", "mode"})->Name("Erf_float"); BENCHMARK(BM_Erf<bfloat16>)->ArgsProduct({{1024, 4096, 16384, 65536, 262144, 1048576}, {0, 1, 2}}) ->ArgNames({"size", "mode"})->Name("Erf_bfloat16"); +BENCHMARK(BM_Add<bfloat16>) CWISE_SIZES ->Name("Add_bfloat16"); +BENCHMARK(BM_Mul<bfloat16>) CWISE_SIZES ->Name("Mul_bfloat16"); +BENCHMARK(BM_Div<bfloat16>) CWISE_SIZES ->Name("Div_bfloat16"); +BENCHMARK(BM_Exp<bfloat16>) CWISE_SIZES ->Name("Exp_bfloat16"); +BENCHMARK(BM_Log<bfloat16>) CWISE_SIZES ->Name("Log_bfloat16"); +BENCHMARK(BM_Tanh<bfloat16>) CWISE_SIZES ->Name("Tanh_bfloat16"); +BENCHMARK(BM_Sqrt<bfloat16>) CWISE_SIZES ->Name("Sqrt_bfloat16"); +BENCHMARK(BM_Rsqrt<bfloat16>) CWISE_SIZES ->Name("Rsqrt_bfloat16"); BENCHMARK(BM_LogSpecialInputs<float>)->ArgsProduct({{4096, 65536}, {0, 1, 2}}) ->ArgNames({"size", "mode"})->Name("LogSpecialInputs_float"); BENCHMARK(BM_Log1pSpecialInputs<float>)->ArgsProduct({{4096, 65536}, {0, 1, 2}})
diff --git a/benchmarks/Core/bench_gemm.cpp b/benchmarks/Core/bench_gemm.cpp index 35ec76c..99019d6 100644 --- a/benchmarks/Core/bench_gemm.cpp +++ b/benchmarks/Core/bench_gemm.cpp
@@ -21,6 +21,11 @@ c.noalias() += a * b; } +template <typename A, typename B, typename C> +EIGEN_DONT_INLINE void gemm_assign(const A& a, const B& b, C& c) { + c.noalias() = a * b; +} + static void add_gflops_counter(benchmark::State& state) { const double m = state.range(0), n = state.range(1), p = state.range(2); state.counters["GFLOPS"] = benchmark::Counter( @@ -44,8 +49,12 @@ return true; } -// c += a * b with a stored in LhsOrder and b in RhsOrder; c is column-major. -template <int LhsOrder, int RhsOrder = ColMajor> +// How the timed loop forms c = a * b: zeroing c then accumulating (the cost assignment had before it could +// overwrite), assigning, or accumulating onto c as it is. +enum GemmMode { ZeroThenAdd, Assign, Accumulate }; + +// c (op)= a * b with a stored in LhsOrder and b in RhsOrder; c is column-major. +template <int LhsOrder, int RhsOrder = ColMajor, GemmMode Mode = ZeroThenAdd> static void eigen_gemm(benchmark::State& state) { int m = state.range(0); int n = state.range(1); @@ -54,15 +63,23 @@ a.setRandom(); Matrix<Scalar, Dynamic, Dynamic, RhsOrder> b(p, n); b.setRandom(); - Mat c = Mat::Zero(m, n); - gemm(a, b, c); + // Validated through the timed operation, assignment from a nonzero c so that a coefficient it misses shows. + Mat c = Mat::Constant(m, n, Scalar(Mode == Assign ? 7 : 0)); + if (Mode == Assign) + gemm_assign(a, b, c); + else + gemm(a, b, c); if (!gemm_result_ok(a, b, c)) { state.SkipWithError("GEMM result check failed"); return; } for (auto _ : state) { - c.setZero(); - gemm(a, b, c); + if (Mode == Assign) { + gemm_assign(a, b, c); + } else { + if (Mode == ZeroThenAdd) c.setZero(); + gemm(a, b, c); + } benchmark::DoNotOptimize(c.data()); benchmark::ClobberMemory(); } @@ -70,6 +87,8 @@ } static void BM_EigenGemm(benchmark::State& state) { eigen_gemm<ColMajor>(state); } +static void BM_EigenGemmAssign(benchmark::State& state) { eigen_gemm<ColMajor, ColMajor, Assign>(state); } +static void BM_EigenGemmAccumulate(benchmark::State& state) { eigen_gemm<ColMajor, ColMajor, Accumulate>(state); } static void BM_EigenGemmRowMajorLhs(benchmark::State& state) { eigen_gemm<RowMajor>(state); } static void BM_EigenGemmRowMajorRhs(benchmark::State& state) { eigen_gemm<ColMajor, RowMajor>(state); } @@ -93,7 +112,14 @@ /* Small and thin products on both sides of the SME backend's NEON small-product bound. */ \ ->Args({17, 17, 17})->Args({20, 20, 20})->Args({25, 25, 25})->Args({28, 28, 28})->Args({31, 31, 31}) \ ->Args({4, 32, 32})->Args({8, 32, 32})->Args({32, 4, 32})->Args({32, 8, 32})->Args({16, 16, 32}) \ - ->Args({24, 24, 48})->Args({12, 12, 64})->Args({2, 64, 64})->Args({8, 64, 128})->Args({64, 8, 128}) + ->Args({24, 24, 48})->Args({12, 12, 64})->Args({2, 64, 64})->Args({8, 64, 128})->Args({64, 8, 128}) \ + /* Full tiles and narrow panels affected by SME store code generation. */ \ + ->Args({32, 8, 128})->Args({32, 16, 128})->Args({40, 40, 40})->Args({48, 48, 48}) \ + /* Ragged C columns at SVL 512, beyond the NEON small-product bound. */ \ + ->Args({9, 17, 256})->Args({13, 17, 256})->Args({15, 17, 256}) \ + ->Args({17, 17, 256})->Args({23, 17, 256})->Args({31, 17, 256}) \ + /* Scalar ARM32 double kernel: full 3x4 tiles and row/column remainders. */ \ + ->ArgsProduct({{3, 6, 7, 13}, {4, 5, 9}, {128}}) // Depth sweeps through the packers' transposer, which packs b and a row-major a; their leading dimension is k, so // these also sweep the source stride. At SVL 512, k = 41 (complex<double>) and 81 (double) end in a paired step and a @@ -107,10 +133,18 @@ // floats; 12 doubles for 60 and 252), and n = 60 leaves b's. A column-major b and a row-major a pack through the ZA // transposer, a row-major b through the copy; a column-major a is mostly read in place. #define TAIL_PANEL_SIZES \ + ->Args({15, 17, 1024})->Args({17, 15, 1024})->Args({31, 31, 1024}) \ ->Args({60, 60, 1024})->Args({60, 64, 1024})->Args({64, 60, 1024})->Args({120, 120, 1024})->Args({252, 252, 252}) + +// Shallow depth: loading and storing c costs about as much as the k multiply-adds, so these measure the order in which +// gebp's micro panels walk c. +#define SHALLOW_DEPTH_SIZES \ + ->ArgsProduct({{256}, {256}, {4, 8, 16}})->Args({1024, 256, 8})->Args({1024, 1024, 4}) // clang-format on -BENCHMARK(BM_EigenGemm) GEMM_SIZES DEPTH_SWEEP_SIZES TAIL_PANEL_SIZES; +BENCHMARK(BM_EigenGemm) GEMM_SIZES DEPTH_SWEEP_SIZES TAIL_PANEL_SIZES SHALLOW_DEPTH_SIZES; +BENCHMARK(BM_EigenGemmAssign) GEMM_SIZES DEPTH_SWEEP_SIZES TAIL_PANEL_SIZES SHALLOW_DEPTH_SIZES; +BENCHMARK(BM_EigenGemmAccumulate) GEMM_SIZES DEPTH_SWEEP_SIZES TAIL_PANEL_SIZES SHALLOW_DEPTH_SIZES; BENCHMARK(BM_EigenGemmRowMajorLhs)->Args({256, 256, 256})->Args({1024, 1024, 1024}) DEPTH_SWEEP_SIZES TAIL_PANEL_SIZES; BENCHMARK(BM_EigenGemmRowMajorRhs) ->Args({64, 64, 1024})
diff --git a/benchmarks/Core/bench_gemm_cast.cpp b/benchmarks/Core/bench_gemm_cast.cpp new file mode 100644 index 0000000..8934817 --- /dev/null +++ b/benchmarks/Core/bench_gemm_cast.cpp
@@ -0,0 +1,160 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +// GEMMs of 16-bit operands: cast to float (lazily converted while packing), mixed with a float operand, and the +// native product, which accumulates in float. + +#include <benchmark/benchmark.h> +#include <Eigen/Core> + +using namespace Eigen; + +#ifndef SCALAR +#define SCALAR Eigen::half +#endif + +using Scalar = SCALAR; +using Mat = Matrix<Scalar, Dynamic, Dynamic>; + +template <typename Dst, typename Expected> +bool close_enough(const Dst& dst, const Expected& expected, float tol) { + return (dst.template cast<float>() - expected).norm() <= tol * expected.norm(); +} + +struct Operands { + MatrixXf af, bf, expected; + Mat a, b; + Operands(Index m, Index n, Index k) { + a = MatrixXf::Random(m, k).cast<Scalar>(); + b = MatrixXf::Random(k, n).cast<Scalar>(); + af = a.cast<float>(); + bf = b.cast<float>(); + expected = af * bf; + } +}; + +template <typename Dst, typename Xpr> +EIGEN_DONT_INLINE void assign(Dst& dst, const Xpr& xpr) { + dst.noalias() = xpr; +} + +template <typename Dst, typename Xpr> +EIGEN_DONT_INLINE void add(Dst& dst, const Xpr& xpr) { + dst.noalias() += xpr; +} + +static void set_counters(benchmark::State& state) { + state.counters["GFLOPS"] = + benchmark::Counter(2.0 * state.range(0) * state.range(1) * state.range(2), + benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); +} + +// Kernel speed of light: the float product of float operands. +static void BM_Float(benchmark::State& state) { + Operands op(state.range(0), state.range(1), state.range(2)); + MatrixXf c(op.af.rows(), op.bf.cols()); + for (auto _ : state) { + assign(c, op.af * op.bf); + benchmark::DoNotOptimize(c.data()); + benchmark::ClobberMemory(); + } + set_counters(state); +} + +// C (float) = A.cast<float>() * B.cast<float>(). +static void BM_CastCast(benchmark::State& state) { + Operands op(state.range(0), state.range(1), state.range(2)); + MatrixXf c(op.af.rows(), op.bf.cols()); + assign(c, op.a.cast<float>() * op.b.cast<float>()); + if (!close_enough(c, op.expected, 1e-5f)) return state.SkipWithError("validation failed"); + for (auto _ : state) { + assign(c, op.a.cast<float>() * op.b.cast<float>()); + benchmark::DoNotOptimize(c.data()); + benchmark::ClobberMemory(); + } + set_counters(state); +} + +// C (float) = A (float) * B.cast<float>(). +static void BM_FloatCast(benchmark::State& state) { + Operands op(state.range(0), state.range(1), state.range(2)); + MatrixXf c(op.af.rows(), op.bf.cols()); + assign(c, op.af * op.b.cast<float>()); + if (!close_enough(c, op.expected, 1e-5f)) return state.SkipWithError("validation failed"); + for (auto _ : state) { + assign(c, op.af * op.b.cast<float>()); + benchmark::DoNotOptimize(c.data()); + benchmark::ClobberMemory(); + } + set_counters(state); +} + +// C (float) = A.cast<float>() * B (float). +static void BM_CastFloat(benchmark::State& state) { + Operands op(state.range(0), state.range(1), state.range(2)); + MatrixXf c(op.af.rows(), op.bf.cols()); + assign(c, op.a.cast<float>() * op.bf); + if (!close_enough(c, op.expected, 1e-5f)) return state.SkipWithError("validation failed"); + for (auto _ : state) { + assign(c, op.a.cast<float>() * op.bf); + benchmark::DoNotOptimize(c.data()); + benchmark::ClobberMemory(); + } + set_counters(state); +} + +// C (16-bit) = (A.cast<float>() * B.cast<float>()).cast<Scalar>(): widening spelled out by the user. +static void BM_CastCastNarrow(benchmark::State& state) { + Operands op(state.range(0), state.range(1), state.range(2)); + Mat c(op.a.rows(), op.b.cols()); + assign(c, (op.a.cast<float>() * op.b.cast<float>()).cast<Scalar>()); + if (!close_enough(c, op.expected, 1e-2f)) return state.SkipWithError("validation failed"); + for (auto _ : state) { + assign(c, (op.a.cast<float>() * op.b.cast<float>()).cast<Scalar>()); + benchmark::DoNotOptimize(c.data()); + benchmark::ClobberMemory(); + } + set_counters(state); +} + +// C (16-bit) = A * B, the native product. The loose bound admits a product summed in the 16-bit type. +static void BM_Native(benchmark::State& state) { + Operands op(state.range(0), state.range(1), state.range(2)); + Mat c(op.a.rows(), op.b.cols()); + assign(c, op.a * op.b); + if (!close_enough(c, op.expected, 1e-1f)) return state.SkipWithError("validation failed"); + for (auto _ : state) { + assign(c, op.a * op.b); + benchmark::DoNotOptimize(c.data()); + benchmark::ClobberMemory(); + } + set_counters(state); +} + +// C (16-bit) += A * B. +static void BM_NativeAdd(benchmark::State& state) { + Operands op(state.range(0), state.range(1), state.range(2)); + Mat c = Mat::Zero(op.a.rows(), op.b.cols()); + add(c, op.a * op.b); + if (!close_enough(c, op.expected, 1e-1f)) return state.SkipWithError("validation failed"); + for (auto _ : state) { + add(c, op.a * op.b); + benchmark::DoNotOptimize(c.data()); + benchmark::ClobberMemory(); + } + set_counters(state); +} + +// clang-format off +#define EIGEN_GEMM_CAST_SHAPES \ + ->Args({32, 32, 32})->Args({64, 64, 64})->Args({128, 128, 128})->Args({256, 256, 256}) \ + ->Args({512, 512, 512})->Args({1024, 1024, 1024}) \ + ->Args({4096, 128, 128})->Args({128, 4096, 128})->Args({128, 128, 4096}) +BENCHMARK(BM_Float) EIGEN_GEMM_CAST_SHAPES; +BENCHMARK(BM_CastCast) EIGEN_GEMM_CAST_SHAPES; +BENCHMARK(BM_FloatCast) EIGEN_GEMM_CAST_SHAPES; +BENCHMARK(BM_CastFloat) EIGEN_GEMM_CAST_SHAPES; +BENCHMARK(BM_CastCastNarrow) EIGEN_GEMM_CAST_SHAPES; +BENCHMARK(BM_Native) EIGEN_GEMM_CAST_SHAPES; +BENCHMARK(BM_NativeAdd) EIGEN_GEMM_CAST_SHAPES; +// clang-format on
diff --git a/benchmarks/Core/bench_gemm_packing.cpp b/benchmarks/Core/bench_gemm_packing.cpp new file mode 100644 index 0000000..1708459 --- /dev/null +++ b/benchmarks/Core/bench_gemm_packing.cpp
@@ -0,0 +1,72 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> + +#if defined(__linux__) +#include <sys/resource.h> +#endif + +template <typename Scalar, bool RankUpdate> +static void BM_Packing(benchmark::State& state) { + Eigen::setNbThreads(1); + using MatrixType = Eigen::Matrix<Scalar, Eigen::Dynamic, Eigen::Dynamic>; + using Real = typename Eigen::NumTraits<Scalar>::Real; + const Eigen::Index n = state.range(0); + const MatrixType a = MatrixType::Random(n, n); + const MatrixType b = MatrixType::Random(n, n); + MatrixType c = MatrixType::Zero(n, n); + const auto product = [&]() { + if (RankUpdate) + c.template selfadjointView<Eigen::Lower>().rankUpdate(a); + else + c.noalias() += a * b; + }; + product(); + // Validate sampled coefficients without a reference allocation changing malloc's history. + for (Eigen::Index i : {Eigen::Index(0), n / 2, n - 1}) { + for (Eigen::Index j : {Eigen::Index(0), n / 2, n - 1}) { + long double expected = 0, scale = 0; + if (!RankUpdate || i >= j) { + for (Eigen::Index k = 0; k < n; ++k) { + const long double term = static_cast<long double>(a(i, k)) * (RankUpdate ? a(j, k) : b(k, j)); + expected += term; + scale += Eigen::numext::abs(term); + } + } + const long double bound = 8 * n * Eigen::NumTraits<Real>::epsilon() * scale; + if (!(Eigen::numext::abs(static_cast<long double>(c(i, j)) - expected) <= bound)) { + state.SkipWithError("matrix product validation failed"); + return; + } + } + } + c.setZero(); +#if defined(__linux__) + rusage before, after; + getrusage(RUSAGE_SELF, &before); +#endif + for (auto _ : state) { + product(); + benchmark::DoNotOptimize(c.data()); + benchmark::ClobberMemory(); + } +#if defined(__linux__) + getrusage(RUSAGE_SELF, &after); + state.counters["minor_faults_per_call"] = double(after.ru_minflt - before.ru_minflt) / state.iterations(); +#endif + const double flops = RankUpdate ? double(n) * (n + 1) * n : 2.0 * n * n * n; + state.counters["GFLOPS"] = + benchmark::Counter(flops, benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); +} + +// clang-format off +#define PACKING_SIZES ->Arg(128)->Arg(144)->Arg(160)->Arg(176)->Arg(184)->Arg(192)->Arg(200)->Arg(240) \ + ->Arg(256)->Arg(384)->Arg(448)->Arg(512)->Arg(768) +BENCHMARK(BM_Packing<float, false>)->Name("GEMM_float") PACKING_SIZES; +BENCHMARK(BM_Packing<double, false>)->Name("GEMM_double") PACKING_SIZES; +BENCHMARK(BM_Packing<float, true>)->Name("SYRK_float") PACKING_SIZES; +BENCHMARK(BM_Packing<double, true>)->Name("SYRK_double") PACKING_SIZES; +#undef PACKING_SIZES +// clang-format on
diff --git a/benchmarks/Core/bench_gemv_degenerate.cpp b/benchmarks/Core/bench_gemv_degenerate.cpp new file mode 100644 index 0000000..f172b9e --- /dev/null +++ b/benchmarks/Core/bench_gemv_degenerate.cpp
@@ -0,0 +1,77 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> + +template <typename Scalar, int StorageOrder = Eigen::ColMajor> +static void BM_GemvTranspose(benchmark::State& state) { + using Matrix = Eigen::Matrix<Scalar, Eigen::Dynamic, Eigen::Dynamic, StorageOrder>; + using Vector = Eigen::Matrix<Scalar, Eigen::Dynamic, 1>; + const Eigen::Index rows = state.range(0), cols = state.range(1); + const Matrix a = Matrix::Random(rows, cols); + const Vector x = Vector::Random(rows); + Vector y = Vector::Zero(cols); + const Scalar alpha(0.75); + y.noalias() += alpha * a.transpose() * x; + using Wide = std::conditional_t<Eigen::NumTraits<Scalar>::IsComplex, std::complex<long double>, long double>; + using Real = typename Eigen::NumTraits<Scalar>::Real; + for (Eigen::Index j = 0; j < cols; ++j) { + Wide expected(0); + long double magnitude = 0; + for (Eigen::Index i = 0; i < rows; ++i) { + const Wide term = Wide(alpha) * Wide(a(i, j)) * Wide(x[i]); + expected += term; + magnitude += Eigen::numext::abs(term); + } + const long double bound = 8 * static_cast<long double>(rows + 1) * + static_cast<long double>(Eigen::NumTraits<Real>::epsilon()) * magnitude; + if (!(Eigen::numext::abs(Wide(y[j]) - expected) <= bound)) { + state.SkipWithError("GEMV differs from scalar reference"); + return; + } + } + benchmark::DoNotOptimize(a.data()); + benchmark::DoNotOptimize(x.data()); + for (auto _ : state) { + y.noalias() += alpha * a.transpose() * x; + benchmark::DoNotOptimize(y.data()); + benchmark::ClobberMemory(); + } +} + +BENCHMARK_TEMPLATE(BM_GemvTranspose, float)->ArgsProduct({{1, 2, 4}, {16, 256, 1024}}); +BENCHMARK_TEMPLATE(BM_GemvTranspose, double)->ArgsProduct({{1, 2, 4}, {16, 256, 1024}}); +BENCHMARK_TEMPLATE(BM_GemvTranspose, std::complex<float>)->ArgsProduct({{1, 2, 4}, {16, 256, 1024}}); +BENCHMARK_TEMPLATE(BM_GemvTranspose, float, Eigen::RowMajor)->ArgsProduct({{1, 2, 4}, {16, 256, 1024}}); +BENCHMARK_TEMPLATE(BM_GemvTranspose, double, Eigen::RowMajor)->ArgsProduct({{1, 2, 4}, {16, 256, 1024}}); +BENCHMARK_TEMPLATE(BM_GemvTranspose, std::complex<float>, Eigen::RowMajor)->ArgsProduct({{1, 2, 4}, {16, 256, 1024}}); + +template <typename Scalar> +static void BM_GemvStrided(benchmark::State& state) { + using Matrix = Eigen::Matrix<Scalar, Eigen::Dynamic, Eigen::Dynamic>; + using Vector = Eigen::Matrix<Scalar, Eigen::Dynamic, 1>; + const Eigen::Index rows = state.range(0), cols = state.range(1); + const Matrix a = Matrix::Random(rows, cols); + const Vector x = Vector::Random(cols); + Vector storage = Vector::Zero(2 * rows); + Eigen::Map<Vector, Eigen::Unaligned, Eigen::InnerStride<2>> y(storage.data(), rows); + const Scalar alpha(0.75); + const Vector expected = alpha * a * x; + y.noalias() += alpha * a * x; + if (!y.isApprox(expected)) { + state.SkipWithError("Strided GEMV differs from contiguous result"); + return; + } + benchmark::DoNotOptimize(a.data()); + benchmark::DoNotOptimize(x.data()); + for (auto _ : state) { + y.noalias() += alpha * a * x; + benchmark::DoNotOptimize(y.data()); + benchmark::ClobberMemory(); + } +} + +BENCHMARK_TEMPLATE(BM_GemvStrided, float)->ArgsProduct({{16, 256, 1024}, {1, 2, 4}}); +BENCHMARK_TEMPLATE(BM_GemvStrided, double)->ArgsProduct({{16, 256, 1024}, {1, 2, 4}}); +BENCHMARK_TEMPLATE(BM_GemvStrided, std::complex<float>)->ArgsProduct({{16, 256, 1024}, {1, 2, 4}});
diff --git a/benchmarks/Core/bench_optimization_barrier.cpp b/benchmarks/Core/bench_optimization_barrier.cpp new file mode 100644 index 0000000..eebc49a --- /dev/null +++ b/benchmarks/Core/bench_optimization_barrier.cpp
@@ -0,0 +1,60 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> + +using namespace Eigen; + +template <typename Scalar, typename Packet, bool TwoSum> +EIGEN_DONT_INLINE void barrier_kernel(const Scalar* x, const Scalar* y, Scalar* hi, Scalar* lo, Index size) { + constexpr int width = internal::unpacket_traits<Packet>::size; + for (Index i = 0; i < size; i += width) { + const Packet value = internal::ploadu<Packet>(x + i); + Packet high = value; + Packet low = internal::pset1<Packet>(Scalar(0)); + if (TwoSum) { + internal::twosum(value, internal::ploadu<Packet>(y + i), high, low); + } else { + EIGEN_OPTIMIZATION_BARRIER(high); + } + internal::pstoreu(hi + i, high); + internal::pstoreu(lo + i, low); + } +} + +template <typename Scalar, typename Packet, bool TwoSum> +static void BM_OptimizationBarrier(benchmark::State& state) { + const Index size = state.range(0); + const Scalar tiny = numext::ldexp(Scalar(1), -NumTraits<Scalar>::digits() - 1); + Array<Scalar, Dynamic, 1> x(size), y(size), hi(size), lo(size); + for (Index i = 0; i < size; ++i) { + x[i] = Scalar(1) + Scalar(i % 16) / Scalar(16); + y[i] = tiny; + } + barrier_kernel<Scalar, Packet, TwoSum>(x.data(), y.data(), hi.data(), lo.data(), size); + for (Index i = 0; i < size; ++i) { + if (hi[i] != x[i] || lo[i] != (TwoSum ? tiny : Scalar(0))) { + state.SkipWithError("optimization barrier changed the result"); + return; + } + } + for (auto _ : state) { + barrier_kernel<Scalar, Packet, TwoSum>(x.data(), y.data(), hi.data(), lo.data(), size); + benchmark::DoNotOptimize(hi.data()); + benchmark::DoNotOptimize(lo.data()); + benchmark::ClobberMemory(); + } + state.SetItemsProcessed(state.iterations() * size); +} + +using FloatPacket = internal::packet_traits<float>::type; +using DoublePacket = internal::packet_traits<double>::type; +BENCHMARK_TEMPLATE(BM_OptimizationBarrier, float, float, false)->Name("MacroFloatScalar")->Arg(4096); +BENCHMARK_TEMPLATE(BM_OptimizationBarrier, float, FloatPacket, false)->Name("MacroFloatPacket")->Arg(4096); +BENCHMARK_TEMPLATE(BM_OptimizationBarrier, double, double, false)->Name("MacroDoubleScalar")->Arg(4096); +BENCHMARK_TEMPLATE(BM_OptimizationBarrier, double, DoublePacket, false)->Name("MacroDoublePacket")->Arg(4096); +BENCHMARK_TEMPLATE(BM_OptimizationBarrier, float, float, true)->Name("TwoSumFloatScalar")->Arg(4096); +BENCHMARK_TEMPLATE(BM_OptimizationBarrier, float, FloatPacket, true)->Name("TwoSumFloatPacket")->Arg(4096); +BENCHMARK_TEMPLATE(BM_OptimizationBarrier, double, double, true)->Name("TwoSumDoubleScalar")->Arg(4096); +BENCHMARK_TEMPLATE(BM_OptimizationBarrier, double, DoublePacket, true)->Name("TwoSumDoublePacket")->Arg(4096);
diff --git a/benchmarks/Core/bench_packet_ops.cpp b/benchmarks/Core/bench_packet_ops.cpp index a84387d..fe7fabc 100644 --- a/benchmarks/Core/bench_packet_ops.cpp +++ b/benchmarks/Core/bench_packet_ops.cpp
@@ -1,5 +1,5 @@ // Benchmarks for the PacketMath implementations of `plset`, `ploaddup`, -// `ploadquad`, `psign`, `predux_mul`, `ptranspose`, and `pldexp`, at the packet-op +// `ploadquad`, `psign`, `predux`, `predux_mul`, `ptranspose`, and `pldexp`, at the packet-op // level and shared across whichever architecture backend the build targets. To // compare against a prior implementation, build and run this same file // against the Eigen checkout in question -- it only calls the public @@ -12,6 +12,7 @@ #include <Eigen/Core> #include <cmath> +#include <complex> #include <cstdint> #include <type_traits> @@ -226,6 +227,34 @@ BENCHMARK(BM_ReduxMul<float>)->Name("ReduxMul_float"); BENCHMARK(BM_ReduxMul<double>)->Name("ReduxMul_double"); +template <typename Scalar, bool Half> +void BM_ComplexRedux(benchmark::State& state) { + using FullPacket = typename packet_traits<Scalar>::type; + using Packet = std::conditional_t<Half, typename internal::unpacket_traits<FullPacket>::half, FullPacket>; + constexpr int size = internal::unpacket_traits<Packet>::size; + Scalar input[size]; + Scalar expected(0); + for (int i = 0; i < size; ++i) { + input[i] = Scalar(i + 1, 2 * i - 1); + expected += input[i]; + } + Packet packet = ploadu<Packet>(input); + if (internal::predux(packet) != expected) { + state.SkipWithError("complex predux differs from scalar reference"); + return; + } + for (auto _ : state) { + benchmark::DoNotOptimize(packet); + Scalar result = internal::predux(packet); + benchmark::DoNotOptimize(result); + } + state.SetItemsProcessed(state.iterations()); +} +BENCHMARK(BM_ComplexRedux<std::complex<float>, false>)->Name("Redux_complex_float"); +BENCHMARK(BM_ComplexRedux<std::complex<float>, true>)->Name("Redux_complex_float_half"); +BENCHMARK(BM_ComplexRedux<std::complex<double>, false>)->Name("Redux_complex_double"); +BENCHMARK(BM_ComplexRedux<std::complex<double>, true>)->Name("Redux_complex_double_half"); + // ---- ptranspose ---- // Benchmarked at N == the type's packet width, i.e. a full square transpose, // so the expected result is simply new_packet[i][k] == old_packet[k][i].
diff --git a/benchmarks/Core/bench_product_storage_order.cpp b/benchmarks/Core/bench_product_storage_order.cpp new file mode 100644 index 0000000..e6e92c5 --- /dev/null +++ b/benchmarks/Core/bench_product_storage_order.cpp
@@ -0,0 +1,107 @@ +// Small fixed-size products for every storage order of the operands and the destination. An assignment evaluates the +// coefficient-based product along the destination's storage order, which vectorizes when the operand that runs along +// it is contiguous: the lhs for a column-major destination, the rhs for a row-major one. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> + +using namespace Eigen; + +#ifndef SCALAR +#define SCALAR float +#endif + +using Scalar = SCALAR; +using RealScalar = NumTraits<Scalar>::Real; + +template <typename Dst, typename Lhs, typename Rhs> +static bool matches_reference(const Dst& dst, const Dst& before, const Lhs& lhs, const Rhs& rhs) { + for (Index i = 0; i < dst.rows(); ++i) { + for (Index j = 0; j < dst.cols(); ++j) { + Scalar sum = before(i, j); + RealScalar scale = numext::abs(before(i, j)); + for (Index k = 0; k < lhs.cols(); ++k) { + sum += lhs(i, k) * rhs(k, j); + scale += numext::abs(lhs(i, k)) * numext::abs(rhs(k, j)); + } + if (!(numext::abs(dst(i, j) - sum) <= RealScalar(8) * NumTraits<RealScalar>::epsilon() * scale)) return false; + } + } + return true; +} + +// dst = lhs * rhs (Accumulate == false) or dst += lhs * rhs. +template <int N, int LhsOrder, int RhsOrder, int DstOrder, bool Accumulate> +static void BM_ProductOrder(benchmark::State& state) { + const Matrix<Scalar, N, N, LhsOrder> lhs = Matrix<Scalar, N, N, LhsOrder>::Random(); + const Matrix<Scalar, N, N, RhsOrder> rhs = Matrix<Scalar, N, N, RhsOrder>::Random(); + using Dst = Matrix<Scalar, N, N, DstOrder>; + Dst dst = Accumulate ? Dst(Dst::Random()) : Dst(Dst::Zero()); + const Dst before = dst; + if (Accumulate) + dst.noalias() += lhs * rhs; + else + dst.noalias() = lhs * rhs; + if (!matches_reference(dst, before, lhs, rhs)) { + state.SkipWithError("Product disagrees with the scalar reference"); + return; + } + for (auto _ : state) { + benchmark::DoNotOptimize(lhs.data()); + benchmark::DoNotOptimize(rhs.data()); + if (Accumulate) + dst.noalias() += lhs * rhs; + else + dst.noalias() = lhs * rhs; + benchmark::DoNotOptimize(dst.data()); + benchmark::ClobberMemory(); + } + state.SetItemsProcessed(state.iterations() * N * N * N); +} + +// dst -= src.col(k) * src.row(k): a rank-1 update with a strided column and a contiguous row when Order is RowMajor. +template <int N, int Order> +static void BM_RankOneUpdate(benchmark::State& state) { + using Mat = Matrix<Scalar, N, N, Order>; + const Mat src = Mat::Random(); + Mat dst = Mat::Random(); + const Mat before = dst; + dst.noalias() -= src.col(1) * src.row(2); + if (!matches_reference(dst, before, -src.col(1), src.row(2))) { + state.SkipWithError("Rank-1 update disagrees with the scalar reference"); + return; + } + for (auto _ : state) { + benchmark::DoNotOptimize(src.data()); + dst.noalias() -= src.col(1) * src.row(2); + benchmark::DoNotOptimize(dst.data()); + benchmark::ClobberMemory(); + } + state.SetItemsProcessed(state.iterations() * N * N); +} + +#define PRODUCT_ORDERS(N, ACC) \ + BENCHMARK_TEMPLATE(BM_ProductOrder, N, ColMajor, ColMajor, ColMajor, ACC); \ + BENCHMARK_TEMPLATE(BM_ProductOrder, N, ColMajor, ColMajor, RowMajor, ACC); \ + BENCHMARK_TEMPLATE(BM_ProductOrder, N, ColMajor, RowMajor, ColMajor, ACC); \ + BENCHMARK_TEMPLATE(BM_ProductOrder, N, ColMajor, RowMajor, RowMajor, ACC); \ + BENCHMARK_TEMPLATE(BM_ProductOrder, N, RowMajor, ColMajor, ColMajor, ACC); \ + BENCHMARK_TEMPLATE(BM_ProductOrder, N, RowMajor, ColMajor, RowMajor, ACC); \ + BENCHMARK_TEMPLATE(BM_ProductOrder, N, RowMajor, RowMajor, ColMajor, ACC); \ + BENCHMARK_TEMPLATE(BM_ProductOrder, N, RowMajor, RowMajor, RowMajor, ACC) + +PRODUCT_ORDERS(3, false); +PRODUCT_ORDERS(4, false); +PRODUCT_ORDERS(8, false); +PRODUCT_ORDERS(3, true); +PRODUCT_ORDERS(4, true); +PRODUCT_ORDERS(8, true); + +BENCHMARK_TEMPLATE(BM_RankOneUpdate, 4, ColMajor); +BENCHMARK_TEMPLATE(BM_RankOneUpdate, 4, RowMajor); +BENCHMARK_TEMPLATE(BM_RankOneUpdate, 8, ColMajor); +BENCHMARK_TEMPLATE(BM_RankOneUpdate, 8, RowMajor); +BENCHMARK_TEMPLATE(BM_RankOneUpdate, 16, ColMajor); +BENCHMARK_TEMPLATE(BM_RankOneUpdate, 16, RowMajor);
diff --git a/benchmarks/Core/bench_segment_tail.cpp b/benchmarks/Core/bench_segment_tail.cpp index 85f6de0..2f8efe6 100644 --- a/benchmarks/Core/bench_segment_tail.cpp +++ b/benchmarks/Core/bench_segment_tail.cpp
@@ -318,6 +318,30 @@ state.SetItemsProcessed(state.iterations() * count); } +#ifdef EIGEN_VECTORIZE_AVX +void BM_ComplexHalfSegmentLoad(benchmark::State& state) { + using Packet = typename internal::find_packet_by_size<std::complex<float>, 2>::type; + const Index begin = state.range(0), count = state.range(1); + Buffers<std::complex<float>> buf; + const std::complex<float>* input = buf.src + 1; + std::complex<float> output[2]; + internal::pstoreu(output, internal::ploaduSegment<Packet>(input, begin, count)); + for (Index i = 0; i < 2; ++i) { + const std::complex<float> expected = i >= begin && i < begin + count ? input[i] : std::complex<float>(0); + if (output[i] != expected) { + state.SkipWithError("complex half-packet segment differs from scalar reference"); + return; + } + } + for (auto _ : state) { + benchmark::DoNotOptimize(buf.src); + Packet packet = internal::ploaduSegment<Packet>(input, begin, count); + benchmark::DoNotOptimize(packet); + } + state.SetItemsProcessed(state.iterations()); +} +#endif + // --- compile-time-count store: matches the completely-unrolled tail -------- // Here the scalar alternative is itself fully unrolled (no loop), which is why // it can beat a masked store for tiny tails. @@ -650,6 +674,15 @@ add_prim<int64_t>(); add_prim<std::complex<float>>(); add_prim<std::complex<double>>(); +#ifdef EIGEN_VECTORIZE_AVX + benchmark::RegisterBenchmark("Prim/Load/seg/half_cf32", &BM_ComplexHalfSegmentLoad) + ->Args({0, 0}) + ->Args({0, 1}) + ->Args({0, 2}) + ->Args({1, 0}) + ->Args({1, 1}) + ->Args({2, 0}); +#endif add_all_prim_ct<float>(); add_all_prim_ct<double>();
diff --git a/benchmarks/Core/bench_selfadjoint_permutation.cpp b/benchmarks/Core/bench_selfadjoint_permutation.cpp new file mode 100644 index 0000000..292d42c --- /dev/null +++ b/benchmarks/Core/bench_selfadjoint_permutation.cpp
@@ -0,0 +1,83 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> +#include <algorithm> +#include <cstdlib> +#include <random> + +namespace { +using namespace Eigen; + +template <bool InPlace, unsigned int UpLo, typename MatrixType, typename PermutationType> +void permute(MatrixType& matrix, const PermutationType& permutation, bool inverse) { + EIGEN_IF_CONSTEXPR (InPlace) { + if (inverse) + matrix.template selfadjointView<UpLo>().applyPermutation(permutation.inverse()); + else + matrix.template selfadjointView<UpLo>().applyPermutation(permutation); + } else { + if (inverse) + matrix.template selfadjointView<UpLo>() = + (permutation.inverse() * matrix.template selfadjointView<UpLo>() * permutation).eval(); + else + matrix.template selfadjointView<UpLo>() = + (permutation * matrix.template selfadjointView<UpLo>() * permutation.transpose()).eval(); + } +} + +template <typename Scalar, int Order, unsigned int UpLo, bool InPlace> +void BM_SelfAdjointPermutation(benchmark::State& state) { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + const Index size = state.range(0); + std::srand(42); + MatrixType matrix = MatrixType::Random(size, size); + matrix.diagonal() = matrix.diagonal().real().template cast<Scalar>(); + PermutationMatrix<Dynamic> permutation(size); + permutation.setIdentity(); + std::mt19937 random(42); + if (state.range(1) == 0) { + std::shuffle(permutation.indices().data(), permutation.indices().data() + size, random); + } else { + permutation.indices().reverseInPlace(); + } + const MatrixType full = matrix.template selfadjointView<UpLo>(); + const bool inverse = state.range(2) != 0; + MatrixType reference(size, size); + if (inverse) + reference = permutation.inverse() * full * permutation; + else + reference = permutation * full * permutation.transpose(); + MatrixType expected = matrix; + expected.template triangularView<UpLo>() = reference; + permute<InPlace, UpLo>(matrix, permutation, inverse); + if (matrix != expected) { + state.SkipWithError("Incorrect self-adjoint permutation or modified opposite triangle"); + return; + } + for (auto _ : state) { + permute<InPlace, UpLo>(matrix, permutation, inverse); + benchmark::DoNotOptimize(matrix.data()); + benchmark::ClobberMemory(); + } +} + +// Permutation kind: 0 = shuffled (seed 42), 1 = reversal. Last argument selects the inverse. +#define EIGEN_BENCH_SELFADJOINT_PERMUTATION(Scalar, Order, UpLo) \ + BENCHMARK_TEMPLATE(BM_SelfAdjointPermutation, Scalar, Order, UpLo, false) \ + ->ArgsProduct({{4, 16, 64, 256, 1024}, {0, 1}, {0, 1}}); \ + BENCHMARK_TEMPLATE(BM_SelfAdjointPermutation, Scalar, Order, UpLo, true) \ + ->ArgsProduct({{4, 16, 64, 256, 1024}, {0, 1}, {0, 1}}) + +EIGEN_BENCH_SELFADJOINT_PERMUTATION(float, ColMajor, Lower); +EIGEN_BENCH_SELFADJOINT_PERMUTATION(float, RowMajor, Upper); +EIGEN_BENCH_SELFADJOINT_PERMUTATION(double, ColMajor, Lower); +EIGEN_BENCH_SELFADJOINT_PERMUTATION(double, RowMajor, Upper); +EIGEN_BENCH_SELFADJOINT_PERMUTATION(std::complex<float>, ColMajor, Upper); +EIGEN_BENCH_SELFADJOINT_PERMUTATION(std::complex<float>, RowMajor, Lower); +EIGEN_BENCH_SELFADJOINT_PERMUTATION(std::complex<double>, ColMajor, Upper); +EIGEN_BENCH_SELFADJOINT_PERMUTATION(std::complex<double>, RowMajor, Lower); + +#undef EIGEN_BENCH_SELFADJOINT_PERMUTATION +} // namespace
diff --git a/benchmarks/Core/bench_stl_iterators.cpp b/benchmarks/Core/bench_stl_iterators.cpp new file mode 100644 index 0000000..ec4464c --- /dev/null +++ b/benchmarks/Core/bench_stl_iterators.cpp
@@ -0,0 +1,48 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> +#include <algorithm> +#include <numeric> + +template <int Stride, bool Reverse> +static void BM_StridedIterator(benchmark::State& state) { + const Eigen::Index size = state.range(0); + const Eigen::Index stride = state.range(1); + // Padding also makes the pre-fix iterator's end pointer valid for the comparison build. + Eigen::VectorXd storage(size * Eigen::numext::abs(stride) + 1); + Eigen::Map<Eigen::VectorXd, Eigen::Unaligned, Eigen::InnerStride<Stride>> values( + storage.data() + (stride < 0 ? storage.size() - 1 : 0), size, Eigen::InnerStride<Stride>(stride)); + for (Eigen::Index i = 0; i < size; ++i) values[i] = double(i + 1); + const double expected = double(size) * double(size + 1) / 2; + if (std::accumulate(values.begin(), values.end(), 0.0) != expected) { + state.SkipWithError("Incorrect iterator sum"); + return; + } + for (auto _ : state) { + benchmark::ClobberMemory(); + EIGEN_IF_CONSTEXPR (Reverse) { + std::reverse(values.begin(), values.end()); + benchmark::DoNotOptimize(values.data()); + } else { + double sum = std::accumulate(values.begin(), values.end(), 0.0); + benchmark::DoNotOptimize(sum); + } + } + const bool reversed = Reverse && state.iterations() % 2 != 0; + for (Eigen::Index i = 0; i < size; ++i) { + if (values[i] != double(reversed ? size - i : i + 1)) { + state.SkipWithError("Incorrect iterator traversal"); + return; + } + } + state.SetItemsProcessed(state.iterations() * size); +} + +BENCHMARK_TEMPLATE(BM_StridedIterator, 1, false)->ArgsProduct({{16, 256, 4096}, {1}}); +BENCHMARK_TEMPLATE(BM_StridedIterator, 2, false)->ArgsProduct({{16, 256, 4096}, {2}}); +BENCHMARK_TEMPLATE(BM_StridedIterator, Eigen::Dynamic, false)->ArgsProduct({{16, 256, 4096}, {-2, 1, 2}}); +BENCHMARK_TEMPLATE(BM_StridedIterator, 1, true)->ArgsProduct({{16, 256, 4096}, {1}}); +BENCHMARK_TEMPLATE(BM_StridedIterator, 2, true)->ArgsProduct({{16, 256, 4096}, {2}}); +BENCHMARK_TEMPLATE(BM_StridedIterator, Eigen::Dynamic, true)->ArgsProduct({{16, 256, 4096}, {-2, 1, 2}});
diff --git a/benchmarks/Core/bench_triangular_product.cpp b/benchmarks/Core/bench_triangular_product.cpp index 922a205..08c9863 100644 --- a/benchmarks/Core/bench_triangular_product.cpp +++ b/benchmarks/Core/bench_triangular_product.cpp
@@ -9,40 +9,71 @@ using namespace Eigen; +// C and reference are two evaluations of n-term inner products; || |T|*|B| ||_F <= ||T||_F * ||B||_F. +template <typename Mat> +static bool MatchesDenseProduct(benchmark::State& state, const Mat& C, const Mat& reference, const Mat& T, + const Mat& B) { + using Scalar = typename Mat::Scalar; + const Scalar bound = Scalar(2) * NumTraits<Scalar>::epsilon() * Scalar(T.rows()) * T.norm() * B.norm(); + if (!numext::isfinite(bound) || !((C - reference).norm() <= bound)) { + state.SkipWithError("TRMM result differs from dense product"); + return false; + } + return true; +} + // C = triangularView<Mode>(A) * B -template <typename Scalar, unsigned int Mode> +template <typename Scalar, unsigned int Mode, int Order = ColMajor> static void BM_TRMM_Left(benchmark::State& state) { const Index n = state.range(0); - using Mat = Matrix<Scalar, Dynamic, Dynamic>; + const Index cols = state.range(1); + using Mat = Matrix<Scalar, Dynamic, Dynamic, Order>; Mat A = Mat::Random(n, n); - Mat B = Mat::Random(n, n); - Mat C(n, n); + Mat B = Mat::Random(n, cols); + Mat C(n, cols); + const Mat dense = A.template triangularView<Mode>(); + const Mat reference = dense * B; + C.noalias() = A.template triangularView<Mode>() * B; + if (!MatchesDenseProduct(state, C, reference, dense, B)) return; for (auto _ : state) { C.noalias() = A.template triangularView<Mode>() * B; benchmark::DoNotOptimize(C.data()); + benchmark::ClobberMemory(); } - state.counters["GFLOPS"] = - benchmark::Counter(1.0 * n * n * n, benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); + state.counters["GFLOPS"] = benchmark::Counter( + double(n) * (n + 1) * cols, benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); } // C = B * triangularView<Mode>(A) -template <typename Scalar, unsigned int Mode> +template <typename Scalar, unsigned int Mode, int Order = ColMajor> static void BM_TRMM_Right(benchmark::State& state) { const Index n = state.range(0); - using Mat = Matrix<Scalar, Dynamic, Dynamic>; + const Index rows = state.range(1); + using Mat = Matrix<Scalar, Dynamic, Dynamic, Order>; Mat A = Mat::Random(n, n); - Mat B = Mat::Random(n, n); - Mat C(n, n); + Mat B = Mat::Random(rows, n); + Mat C(rows, n); + const Mat dense = A.template triangularView<Mode>(); + const Mat reference = B * dense; + C.noalias() = B * A.template triangularView<Mode>(); + if (!MatchesDenseProduct(state, C, reference, dense, B)) return; for (auto _ : state) { C.noalias() = B * A.template triangularView<Mode>(); benchmark::DoNotOptimize(C.data()); + benchmark::ClobberMemory(); } - state.counters["GFLOPS"] = - benchmark::Counter(1.0 * n * n * n, benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); + state.counters["GFLOPS"] = benchmark::Counter( + double(n) * (n + 1) * rows, benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); } // clang-format off -#define TRMM_SIZES ->Arg(64)->Arg(128)->Arg(256)->Arg(512)->Arg(1024) +#define TRMM_SIZES \ + ->Args({2, 2})->Args({4, 4})->Args({8, 8})->Args({16, 16})->Args({17, 17})->Args({32, 32}) \ + ->Args({64, 64})->Args({65, 65})->Args({128, 128})->Args({256, 256})->Args({257, 257}) \ + ->Args({512, 512})->Args({1001, 1001})->Args({1024, 1024})->Args({2048, 2048})->Args({4096, 4096}) \ + ->Args({1024, 8})->Args({1024, 64})->Args({4096, 8})->Args({4096, 16})->Args({4096, 64}) \ + ->Args({4096, 128})->Args({4096, 256})->Args({4096, 1024}) \ + ->Args({64, 4096})->Args({128, 4096})->Args({256, 4096})->Args({1024, 4096}) // Left product BENCHMARK(BM_TRMM_Left<float, Lower>) TRMM_SIZES ->Name("TRMM_Left_float_Lower"); BENCHMARK(BM_TRMM_Left<float, Upper>) TRMM_SIZES ->Name("TRMM_Left_float_Upper"); @@ -53,5 +84,13 @@ BENCHMARK(BM_TRMM_Right<float, Upper>) TRMM_SIZES ->Name("TRMM_Right_float_Upper"); BENCHMARK(BM_TRMM_Right<double, Lower>) TRMM_SIZES ->Name("TRMM_Right_double_Lower"); BENCHMARK(BM_TRMM_Right<double, Upper>) TRMM_SIZES ->Name("TRMM_Right_double_Upper"); +BENCHMARK(BM_TRMM_Left<float, Lower, RowMajor>) TRMM_SIZES ->Name("TRMM_Left_float_Lower_RowMajor"); +BENCHMARK(BM_TRMM_Left<float, Upper, RowMajor>) TRMM_SIZES ->Name("TRMM_Left_float_Upper_RowMajor"); +BENCHMARK(BM_TRMM_Left<double, Lower, RowMajor>) TRMM_SIZES ->Name("TRMM_Left_double_Lower_RowMajor"); +BENCHMARK(BM_TRMM_Left<double, Upper, RowMajor>) TRMM_SIZES ->Name("TRMM_Left_double_Upper_RowMajor"); +BENCHMARK(BM_TRMM_Right<float, Lower, RowMajor>) TRMM_SIZES ->Name("TRMM_Right_float_Lower_RowMajor"); +BENCHMARK(BM_TRMM_Right<float, Upper, RowMajor>) TRMM_SIZES ->Name("TRMM_Right_float_Upper_RowMajor"); +BENCHMARK(BM_TRMM_Right<double, Lower, RowMajor>) TRMM_SIZES ->Name("TRMM_Right_double_Lower_RowMajor"); +BENCHMARK(BM_TRMM_Right<double, Upper, RowMajor>) TRMM_SIZES ->Name("TRMM_Right_double_Upper_RowMajor"); #undef TRMM_SIZES // clang-format on
diff --git a/benchmarks/Core/bench_trig_reduction.cpp b/benchmarks/Core/bench_trig_reduction.cpp new file mode 100644 index 0000000..4a92abf --- /dev/null +++ b/benchmarks/Core/bench_trig_reduction.cpp
@@ -0,0 +1,66 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> + +#include <cmath> +#include <random> + +using namespace Eigen; + +// Domains exercise the small reduction, double-word reduction, and libm fallback. +template <int Function> +static void BM_TrigReduction(benchmark::State& state) { + const Index size = state.range(0); + const int domain = static_cast<int>(state.range(1)); + const double lower[] = {-10.0, 15.0, 1e4, 1e15}; + const double upper[] = {10.0, 1e4, 1e13, 2e15}; + // MPFR 4.2.1 at 256 bits, rounded to binary64; checked outside the timed loop. + const double reference[][4] = { + {0.3, 0.29552020666133955, 0.95533648912560598, 0.30933624960962325}, + {1000.0, 0.82687954053200252, 0.56237907629070294, 1.4703241557027185}, + {1e12, -0.61123870237688949, 0.79144630185289022, -0.77230596813187613}, + {1e15, 0.85827279317023586, -0.51319373778697031, -1.672414782127583}, + }; + std::mt19937_64 random(0x3389); + ArrayXd x(size), y(size); + for (Index i = 0; i < size; ++i) { + const double fraction = double(random() >> 11) / 9007199254740992.0; + x(i) = lower[domain] + (upper[domain] - lower[domain]) * fraction; + if (i % 2) x(i) = -x(i); + } + x(0) = reference[domain][0]; + auto evaluate = [&]() { + EIGEN_IF_CONSTEXPR (Function == 0) y = x.sin(); + EIGEN_IF_CONSTEXPR (Function == 1) y = x.cos(); + EIGEN_IF_CONSTEXPR (Function == 2) y = x.tan(); + }; + evaluate(); + const double expected = reference[domain][Function + 1]; + const double bound = 4 * NumTraits<double>::epsilon() * (std::max)(1.0, std::abs(expected)); + if (!(std::abs(y(0) - expected) <= bound)) { + state.SkipWithError("trig result disagrees with MPFR reference"); + return; + } + for (auto _ : state) { + benchmark::DoNotOptimize(x.data()); + evaluate(); + benchmark::DoNotOptimize(y.data()); + benchmark::ClobberMemory(); + } + state.SetItemsProcessed(state.iterations() * size); +} + +BENCHMARK_TEMPLATE(BM_TrigReduction, 0) + ->Name("SinDoubleReduction") + ->ArgsProduct({{256, 4096, 65536}, {0, 1, 2, 3}}) + ->ArgNames({"size", "domain"}); +BENCHMARK_TEMPLATE(BM_TrigReduction, 1) + ->Name("CosDoubleReduction") + ->ArgsProduct({{256, 4096, 65536}, {0, 1, 2, 3}}) + ->ArgNames({"size", "domain"}); +BENCHMARK_TEMPLATE(BM_TrigReduction, 2) + ->Name("TanDoubleReduction") + ->ArgsProduct({{256, 4096, 65536}, {0, 1, 2, 3}}) + ->ArgNames({"size", "domain"});
diff --git a/benchmarks/Core/bench_trmv.cpp b/benchmarks/Core/bench_trmv.cpp index ecc0676..8713995 100644 --- a/benchmarks/Core/bench_trmv.cpp +++ b/benchmarks/Core/bench_trmv.cpp
@@ -1,6 +1,7 @@ // Benchmarks for triangular matrix-vector product (TRMV). // -// Tests y += triangularView(A) * x for various modes and sizes. +// Tests y += triangularView(A) * x for various modes and sizes, with real, complex and mixed (real matrix, complex +// vector) scalars, for column-major and row-major A. // Exercises TriangularMatrixVector.h kernel. // SPDX-FileCopyrightText: The Eigen Authors // SPDX-License-Identifier: MPL-2.0 @@ -10,39 +11,58 @@ using namespace Eigen; -template <typename Scalar> +template <typename LhsScalar, typename RhsScalar> double trmvFlops(Index n) { - // TRMV: ~n^2 multiply-adds - return (NumTraits<Scalar>::IsComplex ? 8.0 : 2.0) * n * n; + // TRMV: ~n^2 multiply-adds, of 2 flops (real), 4 (real times complex) or 8 (complex) each + const int complex_operands = int(NumTraits<LhsScalar>::IsComplex) + int(NumTraits<RhsScalar>::IsComplex); + return double(2 << complex_operands) * n * n; } // y = triangularView<Mode>(A) * x -template <typename Scalar, unsigned int Mode> +template <typename LhsScalar, typename RhsScalar, unsigned int Mode, int StorageOrder = ColMajor> static void BM_TRMV(benchmark::State& state) { const Index n = state.range(0); - using Mat = Matrix<Scalar, Dynamic, Dynamic>; - using Vec = Matrix<Scalar, Dynamic, 1>; + using Mat = Matrix<LhsScalar, Dynamic, Dynamic, StorageOrder>; + using Vec = Matrix<RhsScalar, Dynamic, 1>; + using ResVec = Matrix<typename ScalarBinaryOpTraits<LhsScalar, RhsScalar>::ReturnType, Dynamic, 1>; Mat A = Mat::Random(n, n); Vec x = Vec::Random(n); - Vec y(n); + ResVec y(n); for (auto _ : state) { y.noalias() = A.template triangularView<Mode>() * x; benchmark::DoNotOptimize(y.data()); benchmark::ClobberMemory(); } - state.counters["GFLOPS"] = benchmark::Counter(trmvFlops<Scalar>(n), benchmark::Counter::kIsIterationInvariantRate, - benchmark::Counter::kIs1000); + state.counters["GFLOPS"] = benchmark::Counter( + trmvFlops<LhsScalar, RhsScalar>(n), benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); } +using cfloat = std::complex<float>; +using cdouble = std::complex<double>; + // clang-format off #define TRMV_SIZES ->Arg(8)->Arg(16)->Arg(32)->Arg(64)->Arg(128)->Arg(256)->Arg(512)->Arg(1024)->Arg(2048) -BENCHMARK(BM_TRMV<float, Lower>) TRMV_SIZES ->Name("TRMV_float_Lower"); -BENCHMARK(BM_TRMV<float, Upper>) TRMV_SIZES ->Name("TRMV_float_Upper"); -BENCHMARK(BM_TRMV<float, UnitLower>) TRMV_SIZES ->Name("TRMV_float_UnitLower"); -BENCHMARK(BM_TRMV<float, UnitUpper>) TRMV_SIZES ->Name("TRMV_float_UnitUpper"); -BENCHMARK(BM_TRMV<double, Lower>) TRMV_SIZES ->Name("TRMV_double_Lower"); -BENCHMARK(BM_TRMV<double, Upper>) TRMV_SIZES ->Name("TRMV_double_Upper"); -BENCHMARK(BM_TRMV<double, UnitLower>) TRMV_SIZES ->Name("TRMV_double_UnitLower"); -BENCHMARK(BM_TRMV<double, UnitUpper>) TRMV_SIZES ->Name("TRMV_double_UnitUpper"); +BENCHMARK(BM_TRMV<float, float, Lower>) TRMV_SIZES ->Name("TRMV_float_Lower"); +BENCHMARK(BM_TRMV<float, float, Upper>) TRMV_SIZES ->Name("TRMV_float_Upper"); +BENCHMARK(BM_TRMV<float, float, UnitLower>) TRMV_SIZES ->Name("TRMV_float_UnitLower"); +BENCHMARK(BM_TRMV<float, float, UnitUpper>) TRMV_SIZES ->Name("TRMV_float_UnitUpper"); +BENCHMARK(BM_TRMV<double, double, Lower>) TRMV_SIZES ->Name("TRMV_double_Lower"); +BENCHMARK(BM_TRMV<double, double, Upper>) TRMV_SIZES ->Name("TRMV_double_Upper"); +BENCHMARK(BM_TRMV<double, double, UnitLower>) TRMV_SIZES ->Name("TRMV_double_UnitLower"); +BENCHMARK(BM_TRMV<double, double, UnitUpper>) TRMV_SIZES ->Name("TRMV_double_UnitUpper"); +BENCHMARK(BM_TRMV<cfloat, cfloat, Lower>) TRMV_SIZES ->Name("TRMV_cfloat_Lower"); +BENCHMARK(BM_TRMV<cfloat, cfloat, Upper>) TRMV_SIZES ->Name("TRMV_cfloat_Upper"); +BENCHMARK(BM_TRMV<cdouble, cdouble, Lower>) TRMV_SIZES ->Name("TRMV_cdouble_Lower"); +BENCHMARK(BM_TRMV<cdouble, cdouble, Upper>) TRMV_SIZES ->Name("TRMV_cdouble_Upper"); +BENCHMARK(BM_TRMV<float, cfloat, Lower>) TRMV_SIZES ->Name("TRMV_float_cfloat_Lower"); +BENCHMARK(BM_TRMV<float, cfloat, Upper>) TRMV_SIZES ->Name("TRMV_float_cfloat_Upper"); +BENCHMARK(BM_TRMV<double, cdouble, Lower>) TRMV_SIZES ->Name("TRMV_double_cdouble_Lower"); +BENCHMARK(BM_TRMV<double, cdouble, Upper>) TRMV_SIZES ->Name("TRMV_double_cdouble_Upper"); +BENCHMARK(BM_TRMV<double, double, Lower, RowMajor>) TRMV_SIZES ->Name("TRMV_double_Lower_RowMajor"); +BENCHMARK(BM_TRMV<double, double, Upper, RowMajor>) TRMV_SIZES ->Name("TRMV_double_Upper_RowMajor"); +BENCHMARK(BM_TRMV<cdouble, cdouble, Lower, RowMajor>) TRMV_SIZES ->Name("TRMV_cdouble_Lower_RowMajor"); +BENCHMARK(BM_TRMV<cdouble, cdouble, Upper, RowMajor>) TRMV_SIZES ->Name("TRMV_cdouble_Upper_RowMajor"); +BENCHMARK(BM_TRMV<double, cdouble, Lower, RowMajor>) TRMV_SIZES ->Name("TRMV_double_cdouble_Lower_RowMajor"); +BENCHMARK(BM_TRMV<double, cdouble, Upper, RowMajor>) TRMV_SIZES ->Name("TRMV_double_cdouble_Upper_RowMajor"); #undef TRMV_SIZES // clang-format on
diff --git a/benchmarks/Sparse/CMakeLists.txt b/benchmarks/Sparse/CMakeLists.txt index f777457..d1d38b1 100644 --- a/benchmarks/Sparse/CMakeLists.txt +++ b/benchmarks/Sparse/CMakeLists.txt
@@ -6,6 +6,7 @@ eigen_add_benchmark(bench_sparseview_assign bench_sparseview_assign.cpp) eigen_add_benchmark(bench_sparse_solvers bench_sparse_solvers.cpp) eigen_add_benchmark(bench_sparse_scatter bench_sparse_scatter.cpp) +eigen_add_benchmark(bench_sparse_storage_reallocation bench_sparse_storage_reallocation.cpp) eigen_add_benchmark(bench_simplicial_factorize bench_simplicial_factorize.cpp) eigen_add_benchmark(bench_sparselu_factorize bench_sparselu_factorize.cpp) eigen_add_benchmark(bench_sparseqr_factorize bench_sparseqr_factorize.cpp)
diff --git a/benchmarks/Sparse/bench_sparse_storage_reallocation.cpp b/benchmarks/Sparse/bench_sparse_storage_reallocation.cpp new file mode 100644 index 0000000..7d87d7d --- /dev/null +++ b/benchmarks/Sparse/bench_sparse_storage_reallocation.cpp
@@ -0,0 +1,23 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/SparseCore> + +static void BM_SparseStorageReallocation(benchmark::State& state) { + const Eigen::Index size = state.range(0); + Eigen::SparseVector<double> vector(size); + vector.reserve(size); + for (Eigen::Index i = 0; i < size; ++i) vector.insertBack(i) = double(i + 1); + for (auto _ : state) { + vector.reserve(size); + vector.data().squeeze(); + benchmark::DoNotOptimize(vector.valuePtr()); + benchmark::DoNotOptimize(vector.innerIndexPtr()); + benchmark::ClobberMemory(); + } + if (vector.nonZeros() != size || vector.coeff(size - 1) != double(size)) + state.SkipWithError("reallocation changed the sparse coefficients"); +} + +BENCHMARK(BM_SparseStorageReallocation)->Arg(16)->Arg(256)->Arg(4096)->Arg(65536);
diff --git a/blas/common.h b/blas/common.h index f4bf358..d6d641c 100644 --- a/blas/common.h +++ b/blas/common.h
@@ -89,44 +89,45 @@ template <typename T> Eigen::Map<Eigen::Matrix<T, Eigen::Dynamic, Eigen::Dynamic, Eigen::ColMajor>, 0, Eigen::OuterStride<>> matrix( - T* data, EIGEN_BLAS_INT rows, EIGEN_BLAS_INT cols, EIGEN_BLAS_INT stride) { + T* data, Eigen::Index rows, Eigen::Index cols, Eigen::Index stride) { return Eigen::Map<Eigen::Matrix<T, Eigen::Dynamic, Eigen::Dynamic, Eigen::ColMajor>, 0, Eigen::OuterStride<> >( data, rows, cols, Eigen::OuterStride<>(stride)); } template <typename T> Eigen::Map<const Eigen::Matrix<T, Eigen::Dynamic, Eigen::Dynamic, Eigen::ColMajor>, 0, Eigen::OuterStride<>> matrix( - const T* data, EIGEN_BLAS_INT rows, EIGEN_BLAS_INT cols, EIGEN_BLAS_INT stride) { + const T* data, Eigen::Index rows, Eigen::Index cols, Eigen::Index stride) { return Eigen::Map<const Eigen::Matrix<T, Eigen::Dynamic, Eigen::Dynamic, Eigen::ColMajor>, 0, Eigen::OuterStride<> >( data, rows, cols, Eigen::OuterStride<>(stride)); } template <typename T> -Eigen::Map<Eigen::Matrix<T, Eigen::Dynamic, 1>, 0, Eigen::InnerStride<Eigen::Dynamic>> make_vector( - T* data, EIGEN_BLAS_INT size, EIGEN_BLAS_INT incr) { +Eigen::Map<Eigen::Matrix<T, Eigen::Dynamic, 1>, 0, Eigen::InnerStride<Eigen::Dynamic>> make_vector(T* data, + Eigen::Index size, + Eigen::Index incr) { return Eigen::Map<Eigen::Matrix<T, Eigen::Dynamic, 1>, 0, Eigen::InnerStride<Eigen::Dynamic> >( data, size, Eigen::InnerStride<Eigen::Dynamic>(incr)); } template <typename T> Eigen::Map<const Eigen::Matrix<T, Eigen::Dynamic, 1>, 0, Eigen::InnerStride<Eigen::Dynamic>> make_vector( - const T* data, EIGEN_BLAS_INT size, EIGEN_BLAS_INT incr) { + const T* data, Eigen::Index size, Eigen::Index incr) { return Eigen::Map<const Eigen::Matrix<T, Eigen::Dynamic, 1>, 0, Eigen::InnerStride<Eigen::Dynamic> >( data, size, Eigen::InnerStride<Eigen::Dynamic>(incr)); } template <typename T> -Eigen::Map<Eigen::Matrix<T, Eigen::Dynamic, 1>> make_vector(T* data, EIGEN_BLAS_INT size) { +Eigen::Map<Eigen::Matrix<T, Eigen::Dynamic, 1>> make_vector(T* data, Eigen::Index size) { return Eigen::Map<Eigen::Matrix<T, Eigen::Dynamic, 1> >(data, size); } template <typename T> -Eigen::Map<const Eigen::Matrix<T, Eigen::Dynamic, 1>> make_vector(const T* data, EIGEN_BLAS_INT size) { +Eigen::Map<const Eigen::Matrix<T, Eigen::Dynamic, 1>> make_vector(const T* data, Eigen::Index size) { return Eigen::Map<const Eigen::Matrix<T, Eigen::Dynamic, 1> >(data, size); } template <typename T> -T* get_compact_vector(T* x, EIGEN_BLAS_INT n, EIGEN_BLAS_INT incx) { +T* get_compact_vector(T* x, Eigen::Index n, Eigen::Index incx) { if (incx == 1) return x; std::remove_const_t<T>* ret = new Scalar[n]; @@ -138,7 +139,7 @@ } template <typename T> -T* copy_back(T* x_cpy, T* x, EIGEN_BLAS_INT n, EIGEN_BLAS_INT incx) { +T* copy_back(T* x_cpy, T* x, Eigen::Index n, Eigen::Index incx) { if (x_cpy == x) return 0; if (incx < 0)
diff --git a/blas/double.cpp b/blas/double.cpp index 0e0393c..4ca430c 100644 --- a/blas/double.cpp +++ b/blas/double.cpp
@@ -20,25 +20,29 @@ #include "level2_real_impl.h" #include "level3_impl.h" -extern "C" double EIGEN_BLAS_FUNC_NAME(sdot)(EIGEN_BLAS_INT* n, float* x, EIGEN_BLAS_INT* incx, float* y, - EIGEN_BLAS_INT* incy) { - if (*n <= 0) return 0; +extern "C" double EIGEN_BLAS_FUNC_NAME(sdot)(EIGEN_BLAS_INT* n_in, float* x, EIGEN_BLAS_INT* incx_in, float* y, + EIGEN_BLAS_INT* incy_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; - if (*incx == 1 && *incy == 1) - return (make_vector(x, *n).cast<double>().cwiseProduct(make_vector(y, *n).cast<double>())).sum(); - else if (*incx >= 0 && *incy >= 0) - return (make_vector(x, *n, *incx).cast<double>().cwiseProduct(make_vector(y, *n, *incy).cast<double>())).sum(); - else if (*incx < 0 && *incy >= 0) - return (make_vector(x, *n, -*incx).reverse().cast<double>().cwiseProduct(make_vector(y, *n, *incy).cast<double>())) + if (n <= 0) return 0; + + if (incx == 1 && incy == 1) + return (make_vector(x, n).cast<double>().cwiseProduct(make_vector(y, n).cast<double>())).sum(); + else if (incx >= 0 && incy >= 0) + return (make_vector(x, n, incx).cast<double>().cwiseProduct(make_vector(y, n, incy).cast<double>())).sum(); + else if (incx < 0 && incy >= 0) + return (make_vector(x, n, -incx).reverse().cast<double>().cwiseProduct(make_vector(y, n, incy).cast<double>())) .sum(); - else if (*incx >= 0 && *incy < 0) - return (make_vector(x, *n, *incx).cast<double>().cwiseProduct(make_vector(y, *n, -*incy).reverse().cast<double>())) + else if (incx >= 0 && incy < 0) + return (make_vector(x, n, incx).cast<double>().cwiseProduct(make_vector(y, n, -incy).reverse().cast<double>())) .sum(); - else if (*incx < 0 && *incy < 0) - return (make_vector(x, *n, -*incx) + else if (incx < 0 && incy < 0) + return (make_vector(x, n, -incx) .reverse() .cast<double>() - .cwiseProduct(make_vector(y, *n, -*incy).reverse().cast<double>())) + .cwiseProduct(make_vector(y, n, -incy).reverse().cast<double>())) .sum(); else return 0;
diff --git a/blas/level1_cplx_impl.h b/blas/level1_cplx_impl.h index b9cf1d0..f17b685 100644 --- a/blas/level1_cplx_impl.h +++ b/blas/level1_cplx_impl.h
@@ -31,56 +31,67 @@ // computes the sum of magnitudes of all vector elements or, for a complex vector x, the sum // res = |Rex1| + |Imx1| + |Rex2| + |Imx2| + ... + |Rexn| + |Imxn|, where x is a vector of order n -extern "C" RealScalar EIGEN_CAT(REAL_SCALAR_SUFFIX, EIGEN_BLAS_FUNC_NAME(asum))(EIGEN_BLAS_INT *n, RealScalar *px, - EIGEN_BLAS_INT *incx) { - if (*n <= 0 || *incx <= 0) return 0; +extern "C" RealScalar EIGEN_CAT(REAL_SCALAR_SUFFIX, EIGEN_BLAS_FUNC_NAME(asum))(EIGEN_BLAS_INT* n_in, RealScalar* px, + EIGEN_BLAS_INT* incx_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + if (n <= 0 || incx <= 0) return 0; // std::complex<T> is layout-compatible with T[2], so we can reinterpret // a complex vector of length n as a real vector of length 2*n and use // the fully vectorized cwiseAbs().sum() path. - if (*incx == 1) - return make_vector(px, 2 * *n).cwiseAbs().sum(); + if (incx == 1) + return make_vector(px, 2 * n).cwiseAbs().sum(); else { // For non-unit stride, fall back to the scalar_norm1_op approach since // the real components are not contiguous across complex elements. Complex *x = reinterpret_cast<Complex *>(px); - return make_vector(x, *n, std::abs(*incx)).unaryExpr<scalar_norm1_op>().sum(); + return make_vector(x, n, std::abs(incx)).unaryExpr<scalar_norm1_op>().sum(); } } -extern "C" EIGEN_BLAS_INT EIGEN_CAT(i, EIGEN_BLAS_FUNC_NAME(amax))(EIGEN_BLAS_INT *n, RealScalar *px, - EIGEN_BLAS_INT *incx) { - if (*n <= 0 || *incx <= 0) return 0; +extern "C" EIGEN_BLAS_INT EIGEN_CAT(i, EIGEN_BLAS_FUNC_NAME(amax))(EIGEN_BLAS_INT* n_in, RealScalar* px, + EIGEN_BLAS_INT* incx_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + if (n <= 0 || incx <= 0) return 0; Scalar *x = reinterpret_cast<Scalar *>(px); Eigen::DenseIndex ret; - if (*incx == 1) - make_vector(x, *n).unaryExpr<scalar_norm1_op>().maxCoeff(&ret); + if (incx == 1) + make_vector(x, n).unaryExpr<scalar_norm1_op>().maxCoeff(&ret); else - make_vector(x, *n, std::abs(*incx)).unaryExpr<scalar_norm1_op>().maxCoeff(&ret); + make_vector(x, n, std::abs(incx)).unaryExpr<scalar_norm1_op>().maxCoeff(&ret); return EIGEN_BLAS_INT(ret) + 1; } -extern "C" EIGEN_BLAS_INT EIGEN_CAT(i, EIGEN_BLAS_FUNC_NAME(amin))(EIGEN_BLAS_INT *n, RealScalar *px, - EIGEN_BLAS_INT *incx) { - if (*n <= 0) return 0; +extern "C" EIGEN_BLAS_INT EIGEN_CAT(i, EIGEN_BLAS_FUNC_NAME(amin))(EIGEN_BLAS_INT* n_in, RealScalar* px, + EIGEN_BLAS_INT* incx_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + if (n <= 0) return 0; Scalar *x = reinterpret_cast<Scalar *>(px); Eigen::DenseIndex ret; - if (*incx == 1) - make_vector(x, *n).unaryExpr<scalar_norm1_op>().minCoeff(&ret); + if (incx == 1) + make_vector(x, n).unaryExpr<scalar_norm1_op>().minCoeff(&ret); else - make_vector(x, *n, std::abs(*incx)).unaryExpr<scalar_norm1_op>().minCoeff(&ret); + make_vector(x, n, std::abs(incx)).unaryExpr<scalar_norm1_op>().minCoeff(&ret); return EIGEN_BLAS_INT(ret) + 1; } // computes a dot product of a conjugated vector with another vector. EIGEN_BLAS_FUNC(dotcw) -(EIGEN_BLAS_INT *n, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *py, EIGEN_BLAS_INT *incy, RealScalar *pres) { +(EIGEN_BLAS_INT* n_in, RealScalar* px, EIGEN_BLAS_INT* incx_in, RealScalar* py, EIGEN_BLAS_INT* incy_in, + RealScalar* pres) { // std::cerr << "_dotc " << *n << " " << *incx << " " << *incy << "\n"; Scalar *res = reinterpret_cast<Scalar *>(pres); - if (*n <= 0) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; + + if (n <= 0) { *res = Scalar(0); return; } @@ -88,24 +99,29 @@ Scalar *x = reinterpret_cast<Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); - if (*incx == 1 && *incy == 1) - *res = (make_vector(x, *n).dot(make_vector(y, *n))); - else if (*incx >= 0 && *incy >= 0) - *res = (make_vector(x, *n, *incx).dot(make_vector(y, *n, *incy))); - else if (*incx < 0 && *incy >= 0) - *res = (make_vector(x, *n, -*incx).reverse().dot(make_vector(y, *n, *incy))); - else if (*incx >= 0 && *incy < 0) - *res = (make_vector(x, *n, *incx).dot(make_vector(y, *n, -*incy).reverse())); - else if (*incx < 0 && *incy < 0) - *res = (make_vector(x, *n, -*incx).reverse().dot(make_vector(y, *n, -*incy).reverse())); + if (incx == 1 && incy == 1) + *res = (make_vector(x, n).dot(make_vector(y, n))); + else if (incx >= 0 && incy >= 0) + *res = (make_vector(x, n, incx).dot(make_vector(y, n, incy))); + else if (incx < 0 && incy >= 0) + *res = (make_vector(x, n, -incx).reverse().dot(make_vector(y, n, incy))); + else if (incx >= 0 && incy < 0) + *res = (make_vector(x, n, incx).dot(make_vector(y, n, -incy).reverse())); + else if (incx < 0 && incy < 0) + *res = (make_vector(x, n, -incx).reverse().dot(make_vector(y, n, -incy).reverse())); } // computes a vector-vector dot product without complex conjugation. EIGEN_BLAS_FUNC(dotuw) -(EIGEN_BLAS_INT *n, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *py, EIGEN_BLAS_INT *incy, RealScalar *pres) { +(EIGEN_BLAS_INT* n_in, RealScalar* px, EIGEN_BLAS_INT* incx_in, RealScalar* py, EIGEN_BLAS_INT* incy_in, + RealScalar* pres) { Scalar *res = reinterpret_cast<Scalar *>(pres); - if (*n <= 0) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; + + if (n <= 0) { *res = Scalar(0); return; } @@ -113,66 +129,73 @@ Scalar *x = reinterpret_cast<Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); - if (*incx == 1 && *incy == 1) - *res = (make_vector(x, *n).cwiseProduct(make_vector(y, *n))).sum(); - else if (*incx >= 0 && *incy >= 0) - *res = (make_vector(x, *n, *incx).cwiseProduct(make_vector(y, *n, *incy))).sum(); - else if (*incx < 0 && *incy >= 0) - *res = (make_vector(x, *n, -*incx).reverse().cwiseProduct(make_vector(y, *n, *incy))).sum(); - else if (*incx >= 0 && *incy < 0) - *res = (make_vector(x, *n, *incx).cwiseProduct(make_vector(y, *n, -*incy).reverse())).sum(); - else if (*incx < 0 && *incy < 0) - *res = (make_vector(x, *n, -*incx).reverse().cwiseProduct(make_vector(y, *n, -*incy).reverse())).sum(); + if (incx == 1 && incy == 1) + *res = (make_vector(x, n).cwiseProduct(make_vector(y, n))).sum(); + else if (incx >= 0 && incy >= 0) + *res = (make_vector(x, n, incx).cwiseProduct(make_vector(y, n, incy))).sum(); + else if (incx < 0 && incy >= 0) + *res = (make_vector(x, n, -incx).reverse().cwiseProduct(make_vector(y, n, incy))).sum(); + else if (incx >= 0 && incy < 0) + *res = (make_vector(x, n, incx).cwiseProduct(make_vector(y, n, -incy).reverse())).sum(); + else if (incx < 0 && incy < 0) + *res = (make_vector(x, n, -incx).reverse().cwiseProduct(make_vector(y, n, -incy).reverse())).sum(); } -extern "C" RealScalar EIGEN_CAT(REAL_SCALAR_SUFFIX, EIGEN_BLAS_FUNC_NAME(nrm2))(EIGEN_BLAS_INT *n, RealScalar *px, - EIGEN_BLAS_INT *incx) { +extern "C" RealScalar EIGEN_CAT(REAL_SCALAR_SUFFIX, EIGEN_BLAS_FUNC_NAME(nrm2))(EIGEN_BLAS_INT* n_in, RealScalar* px, + EIGEN_BLAS_INT* incx_in) { // std::cerr << "__nrm2 " << *n << " " << *incx << "\n"; - if (*n <= 0) return 0; + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + if (n <= 0) return 0; Scalar *x = reinterpret_cast<Scalar *>(px); - if (*incx == 1) return make_vector(x, *n).stableNorm(); + if (incx == 1) return make_vector(x, n).stableNorm(); - return make_vector(x, *n, std::abs(*incx)).stableNorm(); + return make_vector(x, n, std::abs(incx)).stableNorm(); } EIGEN_BLAS_FUNC(EIGEN_CAT(REAL_SCALAR_SUFFIX, rot)) -(EIGEN_BLAS_INT *n, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *py, EIGEN_BLAS_INT *incy, RealScalar *pc, - RealScalar *ps) { - if (*n <= 0) return; +(EIGEN_BLAS_INT* n_in, RealScalar* px, EIGEN_BLAS_INT* incx_in, RealScalar* py, EIGEN_BLAS_INT* incy_in, RealScalar* pc, + RealScalar* ps) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; + if (n <= 0) return; Scalar *x = reinterpret_cast<Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); RealScalar c = *pc; RealScalar s = *ps; - StridedVectorType vx(make_vector(x, *n, std::abs(*incx))); - StridedVectorType vy(make_vector(y, *n, std::abs(*incy))); + StridedVectorType vx(make_vector(x, n, std::abs(incx))); + StridedVectorType vy(make_vector(y, n, std::abs(incy))); Eigen::Reverse<StridedVectorType> rvx(vx); Eigen::Reverse<StridedVectorType> rvy(vy); // TODO implement mixed real-scalar rotations - if (*incx < 0 && *incy > 0) + if (incx < 0 && incy > 0) Eigen::internal::apply_rotation_in_the_plane(rvx, vy, Eigen::JacobiRotation<Scalar>(c, s)); - else if (*incx > 0 && *incy < 0) + else if (incx > 0 && incy < 0) Eigen::internal::apply_rotation_in_the_plane(vx, rvy, Eigen::JacobiRotation<Scalar>(c, s)); else Eigen::internal::apply_rotation_in_the_plane(vx, vy, Eigen::JacobiRotation<Scalar>(c, s)); } EIGEN_BLAS_FUNC(EIGEN_CAT(REAL_SCALAR_SUFFIX, scal)) -(EIGEN_BLAS_INT *n, RealScalar *palpha, RealScalar *px, EIGEN_BLAS_INT *incx) { - if (*n <= 0 || *incx <= 0) return; +(EIGEN_BLAS_INT* n_in, RealScalar* palpha, RealScalar* px, EIGEN_BLAS_INT* incx_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + if (n <= 0 || incx <= 0) return; Scalar *x = reinterpret_cast<Scalar *>(px); RealScalar alpha = *palpha; // std::cerr << "__scal " << *n << " " << alpha << " " << *incx << "\n"; - if (*incx == 1) - make_vector(x, *n) *= alpha; + if (incx == 1) + make_vector(x, n) *= alpha; else - make_vector(x, *n, std::abs(*incx)) *= alpha; + make_vector(x, n, std::abs(incx)) *= alpha; }
diff --git a/blas/level1_impl.h b/blas/level1_impl.h index 6d96234..1a7eefc 100644 --- a/blas/level1_impl.h +++ b/blas/level1_impl.h
@@ -11,24 +11,28 @@ #include "common.h" EIGEN_BLAS_FUNC(axpy) -(const EIGEN_BLAS_INT *n, const RealScalar *palpha, const RealScalar *px, const EIGEN_BLAS_INT *incx, RealScalar *py, - const EIGEN_BLAS_INT *incy) { +(const EIGEN_BLAS_INT *n_in, const RealScalar *palpha, const RealScalar *px, const EIGEN_BLAS_INT *incx_in, + RealScalar *py, const EIGEN_BLAS_INT *incy_in) { const Scalar *x = reinterpret_cast<const Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); Scalar alpha = *reinterpret_cast<const Scalar *>(palpha); - if (*n <= 0) return; + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; - if (*incx == 1 && *incy == 1) - make_vector(y, *n) += alpha * make_vector(x, *n); - else if (*incx >= 0 && *incy >= 0) - make_vector(y, *n, *incy) += alpha * make_vector(x, *n, *incx); - else if (*incx >= 0 && *incy < 0) - make_vector(y, *n, -*incy) += alpha * make_vector(x, *n, *incx).reverse(); - else if (*incx < 0 && *incy >= 0) - make_vector(y, *n, *incy) += alpha * make_vector(x, *n, -*incx).reverse(); - else if (*incx < 0 && *incy < 0) - make_vector(y, *n, -*incy) += alpha * make_vector(x, *n, -*incx); + if (n <= 0) return; + + if (incx == 1 && incy == 1) + make_vector(y, n) += alpha * make_vector(x, n); + else if (incx >= 0 && incy >= 0) + make_vector(y, n, incy) += alpha * make_vector(x, n, incx); + else if (incx >= 0 && incy < 0) + make_vector(y, n, -incy) += alpha * make_vector(x, n, incx).reverse(); + else if (incx < 0 && incy >= 0) + make_vector(y, n, incy) += alpha * make_vector(x, n, -incx).reverse(); + else if (incx < 0 && incy < 0) + make_vector(y, n, -incy) += alpha * make_vector(x, n, -incx); } EIGEN_BLAS_FUNC(axpby) @@ -38,7 +42,7 @@ Scalar *y = reinterpret_cast<Scalar *>(py); const Scalar alpha = *reinterpret_cast<const Scalar *>(palpha); const Scalar beta = *reinterpret_cast<const Scalar *>(pbeta); - const EIGEN_BLAS_INT n = *pn; + const Eigen::Index n = *pn; if (n <= 0) return; @@ -47,8 +51,8 @@ return; } - const EIGEN_BLAS_INT incx = *pincx; - const EIGEN_BLAS_INT incy = *pincy; + const Eigen::Index incx = *pincx; + const Eigen::Index incy = *pincy; if (incx == 1 && incy == 1) make_vector(y, n) = alpha * make_vector(x, n) + beta * make_vector(y, n); @@ -62,30 +66,35 @@ make_vector(y, n, -incy) = alpha * make_vector(x, n, -incx) + beta * make_vector(y, n, -incy); } -EIGEN_BLAS_FUNC(copy)(EIGEN_BLAS_INT *n, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *py, EIGEN_BLAS_INT *incy) { - if (*n <= 0) return; +EIGEN_BLAS_FUNC(copy) +(EIGEN_BLAS_INT *n_in, RealScalar *px, EIGEN_BLAS_INT *incx_in, RealScalar *py, EIGEN_BLAS_INT *incy_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; + + if (n <= 0) return; Scalar *x = reinterpret_cast<Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); // be careful, *incx==0 is allowed !! - if (*incx == 1 && *incy == 1) - make_vector(y, *n) = make_vector(x, *n); - else if (*incx == 0) { + if (incx == 1 && incy == 1) + make_vector(y, n) = make_vector(x, n); + else if (incx == 0) { // Broadcast: copy x[0] to all elements of y. - if (*incy < 0) y = y - (*n - 1) * (*incy); - for (EIGEN_BLAS_INT i = 0; i < *n; ++i) { + if (incy < 0) y = y - (n - 1) * incy; + for (Eigen::Index i = 0; i < n; ++i) { *y = *x; - y += *incy; + y += incy; } - } else if (*incx > 0 && *incy > 0) - make_vector(y, *n, *incy) = make_vector(x, *n, *incx); - else if (*incx > 0 && *incy < 0) - make_vector(y, *n, -*incy).reverse() = make_vector(x, *n, *incx); - else if (*incx < 0 && *incy > 0) - make_vector(y, *n, *incy) = make_vector(x, *n, -*incx).reverse(); - else if (*incx < 0 && *incy < 0) - make_vector(y, *n, -*incy) = make_vector(x, *n, -*incx); + } else if (incx > 0 && incy > 0) + make_vector(y, n, incy) = make_vector(x, n, incx); + else if (incx > 0 && incy < 0) + make_vector(y, n, -incy).reverse() = make_vector(x, n, incx); + else if (incx < 0 && incy > 0) + make_vector(y, n, incy) = make_vector(x, n, -incx).reverse(); + else if (incx < 0 && incy < 0) + make_vector(y, n, -incy) = make_vector(x, n, -incx); } EIGEN_BLAS_FUNC(rotg)(RealScalar *pa, RealScalar *pb, RealScalar *pc, RealScalar *ps) { @@ -141,32 +150,38 @@ // *s = r.s(); } -EIGEN_BLAS_FUNC(scal)(EIGEN_BLAS_INT *n, RealScalar *palpha, RealScalar *px, EIGEN_BLAS_INT *incx) { - if (*n <= 0 || *incx <= 0) return; +EIGEN_BLAS_FUNC(scal)(EIGEN_BLAS_INT *n_in, RealScalar *palpha, RealScalar *px, EIGEN_BLAS_INT *incx_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + if (n <= 0 || incx <= 0) return; Scalar *x = reinterpret_cast<Scalar *>(px); Scalar alpha = *reinterpret_cast<Scalar *>(palpha); - if (*incx == 1) - make_vector(x, *n) *= alpha; + if (incx == 1) + make_vector(x, n) *= alpha; else - make_vector(x, *n, std::abs(*incx)) *= alpha; + make_vector(x, n, std::abs(incx)) *= alpha; } -EIGEN_BLAS_FUNC(swap)(EIGEN_BLAS_INT *n, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *py, EIGEN_BLAS_INT *incy) { - if (*n <= 0) return; +EIGEN_BLAS_FUNC(swap) +(EIGEN_BLAS_INT *n_in, RealScalar *px, EIGEN_BLAS_INT *incx_in, RealScalar *py, EIGEN_BLAS_INT *incy_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; + if (n <= 0) return; Scalar *x = reinterpret_cast<Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); - if (*incx == 1 && *incy == 1) - make_vector(y, *n).swap(make_vector(x, *n)); - else if (*incx > 0 && *incy > 0) - make_vector(y, *n, *incy).swap(make_vector(x, *n, *incx)); - else if (*incx > 0 && *incy < 0) - make_vector(y, *n, -*incy).reverse().swap(make_vector(x, *n, *incx)); - else if (*incx < 0 && *incy > 0) - make_vector(y, *n, *incy).swap(make_vector(x, *n, -*incx).reverse()); - else if (*incx < 0 && *incy < 0) - make_vector(y, *n, -*incy).reverse().swap(make_vector(x, *n, -*incx).reverse()); + if (incx == 1 && incy == 1) + make_vector(y, n).swap(make_vector(x, n)); + else if (incx > 0 && incy > 0) + make_vector(y, n, incy).swap(make_vector(x, n, incx)); + else if (incx > 0 && incy < 0) + make_vector(y, n, -incy).reverse().swap(make_vector(x, n, incx)); + else if (incx < 0 && incy > 0) + make_vector(y, n, incy).swap(make_vector(x, n, -incx).reverse()); + else if (incx < 0 && incy < 0) + make_vector(y, n, -incy).reverse().swap(make_vector(x, n, -incx).reverse()); }
diff --git a/blas/level1_real_impl.h b/blas/level1_real_impl.h index 5e4b2ae..251f679 100644 --- a/blas/level1_real_impl.h +++ b/blas/level1_real_impl.h
@@ -12,100 +12,117 @@ // computes the sum of magnitudes of all vector elements or, for a complex vector x, the sum // res = |Rex1| + |Imx1| + |Rex2| + |Imx2| + ... + |Rexn| + |Imxn|, where x is a vector of order n -extern "C" RealScalar EIGEN_BLAS_FUNC_NAME(asum)(EIGEN_BLAS_INT *n, Scalar *px, EIGEN_BLAS_INT *incx) { +extern "C" RealScalar EIGEN_BLAS_FUNC_NAME(asum)(EIGEN_BLAS_INT* n_in, Scalar* px, EIGEN_BLAS_INT* incx_in) { // std::cerr << "_asum " << *n << " " << *incx << "\n"; + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + Scalar *x = reinterpret_cast<Scalar *>(px); - if (*n <= 0 || *incx <= 0) return 0; + if (n <= 0 || incx <= 0) return 0; - if (*incx == 1) - return make_vector(x, *n).cwiseAbs().sum(); + if (incx == 1) + return make_vector(x, n).cwiseAbs().sum(); else - return make_vector(x, *n, std::abs(*incx)).cwiseAbs().sum(); + return make_vector(x, n, std::abs(incx)).cwiseAbs().sum(); } -extern "C" EIGEN_BLAS_INT EIGEN_CAT(i, EIGEN_BLAS_FUNC_NAME(amax))(EIGEN_BLAS_INT *n, Scalar *px, - EIGEN_BLAS_INT *incx) { - if (*n <= 0 || *incx <= 0) return 0; +extern "C" EIGEN_BLAS_INT EIGEN_CAT(i, EIGEN_BLAS_FUNC_NAME(amax))(EIGEN_BLAS_INT* n_in, Scalar* px, + EIGEN_BLAS_INT* incx_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + if (n <= 0 || incx <= 0) return 0; Scalar *x = reinterpret_cast<Scalar *>(px); Eigen::DenseIndex ret; - if (*incx == 1) - make_vector(x, *n).cwiseAbs().maxCoeff(&ret); + if (incx == 1) + make_vector(x, n).cwiseAbs().maxCoeff(&ret); else - make_vector(x, *n, std::abs(*incx)).cwiseAbs().maxCoeff(&ret); + make_vector(x, n, std::abs(incx)).cwiseAbs().maxCoeff(&ret); return EIGEN_BLAS_INT(ret) + 1; } -extern "C" EIGEN_BLAS_INT EIGEN_CAT(i, EIGEN_BLAS_FUNC_NAME(amin))(EIGEN_BLAS_INT *n, Scalar *px, - EIGEN_BLAS_INT *incx) { - if (*n <= 0) return 0; +extern "C" EIGEN_BLAS_INT EIGEN_CAT(i, EIGEN_BLAS_FUNC_NAME(amin))(EIGEN_BLAS_INT* n_in, Scalar* px, + EIGEN_BLAS_INT* incx_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + if (n <= 0) return 0; Scalar *x = reinterpret_cast<Scalar *>(px); Eigen::DenseIndex ret; - if (*incx == 1) - make_vector(x, *n).cwiseAbs().minCoeff(&ret); + if (incx == 1) + make_vector(x, n).cwiseAbs().minCoeff(&ret); else - make_vector(x, *n, std::abs(*incx)).cwiseAbs().minCoeff(&ret); + make_vector(x, n, std::abs(incx)).cwiseAbs().minCoeff(&ret); return EIGEN_BLAS_INT(ret) + 1; } // computes a vector-vector dot product. -extern "C" Scalar EIGEN_BLAS_FUNC_NAME(dot)(EIGEN_BLAS_INT *n, Scalar *px, EIGEN_BLAS_INT *incx, Scalar *py, - EIGEN_BLAS_INT *incy) { +extern "C" Scalar EIGEN_BLAS_FUNC_NAME(dot)(EIGEN_BLAS_INT* n_in, Scalar* px, EIGEN_BLAS_INT* incx_in, Scalar* py, + EIGEN_BLAS_INT* incy_in) { // std::cerr << "_dot " << *n << " " << *incx << " " << *incy << "\n"; - if (*n <= 0) return 0; + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; + + if (n <= 0) return 0; Scalar *x = reinterpret_cast<Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); - if (*incx == 1 && *incy == 1) - return make_vector(x, *n).dot(make_vector(y, *n)); - else if (*incx >= 0 && *incy >= 0) - return make_vector(x, *n, *incx).dot(make_vector(y, *n, *incy)); - else if (*incx < 0 && *incy >= 0) - return make_vector(x, *n, -*incx).reverse().dot(make_vector(y, *n, *incy)); - else if (*incx >= 0 && *incy < 0) - return make_vector(x, *n, *incx).dot(make_vector(y, *n, -*incy).reverse()); - else if (*incx < 0 && *incy < 0) - return make_vector(x, *n, -*incx).reverse().dot(make_vector(y, *n, -*incy).reverse()); + if (incx == 1 && incy == 1) + return make_vector(x, n).dot(make_vector(y, n)); + else if (incx >= 0 && incy >= 0) + return make_vector(x, n, incx).dot(make_vector(y, n, incy)); + else if (incx < 0 && incy >= 0) + return make_vector(x, n, -incx).reverse().dot(make_vector(y, n, incy)); + else if (incx >= 0 && incy < 0) + return make_vector(x, n, incx).dot(make_vector(y, n, -incy).reverse()); + else if (incx < 0 && incy < 0) + return make_vector(x, n, -incx).reverse().dot(make_vector(y, n, -incy).reverse()); else return 0; } // computes the Euclidean norm of a vector. -extern "C" Scalar EIGEN_BLAS_FUNC_NAME(nrm2)(EIGEN_BLAS_INT *n, Scalar *px, EIGEN_BLAS_INT *incx) { - if (*n <= 0) return 0; +extern "C" Scalar EIGEN_BLAS_FUNC_NAME(nrm2)(EIGEN_BLAS_INT* n_in, Scalar* px, EIGEN_BLAS_INT* incx_in) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + if (n <= 0) return 0; Scalar *x = reinterpret_cast<Scalar *>(px); - if (*incx == 1) - return make_vector(x, *n).stableNorm(); + if (incx == 1) + return make_vector(x, n).stableNorm(); else - return make_vector(x, *n, std::abs(*incx)).stableNorm(); + return make_vector(x, n, std::abs(incx)).stableNorm(); } EIGEN_BLAS_FUNC(rot) -(EIGEN_BLAS_INT *n, Scalar *px, EIGEN_BLAS_INT *incx, Scalar *py, EIGEN_BLAS_INT *incy, Scalar *pc, Scalar *ps) { +(EIGEN_BLAS_INT* n_in, Scalar* px, EIGEN_BLAS_INT* incx_in, Scalar* py, EIGEN_BLAS_INT* incy_in, Scalar* pc, + Scalar* ps) { // std::cerr << "_rot " << *n << " " << *incx << " " << *incy << "\n"; - if (*n <= 0) return; + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; + if (n <= 0) return; Scalar *x = reinterpret_cast<Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); Scalar c = *reinterpret_cast<Scalar *>(pc); Scalar s = *reinterpret_cast<Scalar *>(ps); - StridedVectorType vx(make_vector(x, *n, std::abs(*incx))); - StridedVectorType vy(make_vector(y, *n, std::abs(*incy))); + StridedVectorType vx(make_vector(x, n, std::abs(incx))); + StridedVectorType vy(make_vector(y, n, std::abs(incy))); Eigen::Reverse<StridedVectorType> rvx(vx); Eigen::Reverse<StridedVectorType> rvy(vy); - if (*incx < 0 && *incy > 0) + if (incx < 0 && incy > 0) Eigen::internal::apply_rotation_in_the_plane(rvx, vy, Eigen::JacobiRotation<Scalar>(c, s)); - else if (*incx > 0 && *incy < 0) + else if (incx > 0 && incy < 0) Eigen::internal::apply_rotation_in_the_plane(vx, rvy, Eigen::JacobiRotation<Scalar>(c, s)); else Eigen::internal::apply_rotation_in_the_plane(vx, vy, Eigen::JacobiRotation<Scalar>(c, s)); @@ -119,12 +136,16 @@ // -2: H = identity (no-op) // param[1..4] = h11, h21, h12, h22 EIGEN_BLAS_FUNC(rotm) -(EIGEN_BLAS_INT *n, Scalar *px, EIGEN_BLAS_INT *incx, Scalar *py, EIGEN_BLAS_INT *incy, Scalar *param) { +(EIGEN_BLAS_INT* n_in, Scalar* px, EIGEN_BLAS_INT* incx_in, Scalar* py, EIGEN_BLAS_INT* incy_in, Scalar* param) { + const Eigen::Index n = *n_in; + const Eigen::Index incx = *incx_in; + const Eigen::Index incy = *incy_in; + Scalar *x = reinterpret_cast<Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); Scalar flag = param[0]; - if (*n <= 0 || flag == Scalar(-2)) return; + if (n <= 0 || flag == Scalar(-2)) return; Scalar h11, h12, h21, h22; if (flag < Scalar(0)) { @@ -144,16 +165,16 @@ h22 = param[4]; } - EIGEN_BLAS_INT kx = *incx > 0 ? 0 : (1 - *n) * *incx; - EIGEN_BLAS_INT ky = *incy > 0 ? 0 : (1 - *n) * *incy; + Eigen::Index kx = incx > 0 ? 0 : (1 - n) * incx; + Eigen::Index ky = incy > 0 ? 0 : (1 - n) * incy; - for (EIGEN_BLAS_INT i = 0; i < *n; ++i) { + for (Eigen::Index i = 0; i < n; ++i) { Scalar w = x[kx]; Scalar z = y[ky]; x[kx] = h11 * w + h12 * z; y[ky] = h21 * w + h22 * z; - kx += *incx; - ky += *incy; + kx += incx; + ky += incy; } }
diff --git a/blas/level2_cplx_impl.h b/blas/level2_cplx_impl.h index 9c69fb0..e63de88 100644 --- a/blas/level2_cplx_impl.h +++ b/blas/level2_cplx_impl.h
@@ -21,13 +21,13 @@ (const char *uplo, const EIGEN_BLAS_INT *n, const RealScalar *palpha, const RealScalar *pa, const EIGEN_BLAS_INT *lda, const RealScalar *px, const EIGEN_BLAS_INT *incx, const RealScalar *pbeta, RealScalar *py, const EIGEN_BLAS_INT *incy) { - typedef void (*functype)(EIGEN_BLAS_INT, const Scalar *, EIGEN_BLAS_INT, const Scalar *, Scalar *, Scalar); + using functype = void (*)(Eigen::Index, const Scalar *, Eigen::Index, const Scalar *, Scalar *, Scalar); static const functype func[2] = { // array index: UP - (Eigen::internal::selfadjoint_matrix_vector_product<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, Eigen::Upper, false, + (Eigen::internal::selfadjoint_matrix_vector_product<Scalar, Eigen::Index, Eigen::ColMajor, Eigen::Upper, false, false>::run), // array index: LO - (Eigen::internal::selfadjoint_matrix_vector_product<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, Eigen::Lower, false, + (Eigen::internal::selfadjoint_matrix_vector_product<Scalar, Eigen::Index, Eigen::ColMajor, Eigen::Lower, false, false>::run), }; @@ -83,22 +83,25 @@ * Diagonal elements are real; off-diagonal contributions use conjugation. */ EIGEN_BLAS_FUNC(hbmv) -(char *uplo, EIGEN_BLAS_INT *n, EIGEN_BLAS_INT *k, RealScalar *palpha, RealScalar *pa, EIGEN_BLAS_INT *lda, +(char *uplo, EIGEN_BLAS_INT *n_in, EIGEN_BLAS_INT *k_in, RealScalar *palpha, RealScalar *pa, EIGEN_BLAS_INT *lda_in, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *pbeta, RealScalar *py, EIGEN_BLAS_INT *incy) { const Scalar alpha = *reinterpret_cast<const Scalar *>(palpha); const Scalar beta = *reinterpret_cast<const Scalar *>(pbeta); const Scalar *a = reinterpret_cast<const Scalar *>(pa); const Scalar *x = reinterpret_cast<const Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); + const Eigen::Index n = *n_in; + const Eigen::Index k = *k_in; + const Eigen::Index lda = *lda_in; EIGEN_BLAS_INT info = 0; if (UPLO(*uplo) == INVALID) info = 1; - else if (*n < 0) + else if (n < 0) info = 2; - else if (*k < 0) + else if (k < 0) info = 3; - else if (*lda < *k + 1) + else if (lda < k + 1) info = 6; else if (*incx == 0) info = 8; @@ -106,43 +109,43 @@ info = 11; if (info) return xerbla_(SCALAR_SUFFIX_UP "HBMV ", &info, kBlasNameLength); - if (*n == 0 || (alpha == Scalar(0) && beta == Scalar(1))) return; + if (n == 0 || (alpha == Scalar(0) && beta == Scalar(1))) return; - const Scalar *actual_x = get_compact_vector(x, *n, *incx); - Scalar *actual_y = get_compact_vector(y, *n, *incy); + const Scalar *actual_x = get_compact_vector(x, n, *incx); + Scalar *actual_y = get_compact_vector(y, n, *incy); // First form y := beta*y. if (beta != Scalar(1)) { if (beta == Scalar(0)) - make_vector(actual_y, *n).setZero(); + make_vector(actual_y, n).setZero(); else - make_vector(actual_y, *n) *= beta; + make_vector(actual_y, n) *= beta; } if (alpha == Scalar(0)) { if (actual_x != x) delete[] actual_x; - if (actual_y != y) delete[] copy_back(actual_y, y, *n, *incy); + if (actual_y != y) delete[] copy_back(actual_y, y, n, *incy); return; } - if (*k >= 8) { + if (k >= 8) { // Vectorized path: use Eigen Map segments for the inner band operations. - ConstMatrixType band(a, *k + 1, *n, *lda); + ConstMatrixType band(a, k + 1, n, lda); if (UPLO(*uplo) == UP) { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { - EIGEN_BLAS_INT start = std::max<EIGEN_BLAS_INT>(0, j - *k); - EIGEN_BLAS_INT len = j - start; - EIGEN_BLAS_INT offset = *k - (j - start); + for (Eigen::Index j = 0; j < n; ++j) { + Eigen::Index start = (std::max)(Eigen::Index(0), j - k); + Eigen::Index len = j - start; + Eigen::Index offset = k - (j - start); Scalar temp1 = alpha * actual_x[j]; - actual_y[j] += Scalar(Eigen::numext::real(band(*k, j))) * temp1; + actual_y[j] += Scalar(Eigen::numext::real(band(k, j))) * temp1; if (len > 0) { make_vector(actual_y + start, len) += temp1 * band.col(j).segment(offset, len); actual_y[j] += alpha * band.col(j).segment(offset, len).dot(make_vector(actual_x + start, len)); } } } else { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { - EIGEN_BLAS_INT len = std::min(*n - 1, j + *k) - j; + for (Eigen::Index j = 0; j < n; ++j) { + Eigen::Index len = (std::min)(n - 1, j + k) - j; Scalar temp1 = alpha * actual_x[j]; actual_y[j] += Scalar(Eigen::numext::real(band(0, j))) * temp1; if (len > 0) { @@ -154,23 +157,23 @@ } else { // Scalar path: for narrow bandwidth, avoid Map overhead. if (UPLO(*uplo) == UP) { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { + for (Eigen::Index j = 0; j < n; ++j) { Scalar temp1 = alpha * actual_x[j]; Scalar temp2 = Scalar(0); - for (EIGEN_BLAS_INT i = std::max<EIGEN_BLAS_INT>(0, j - *k); i < j; ++i) { - Scalar aij = a[(*k + i - j) + j * *lda]; + for (Eigen::Index i = (std::max)(Eigen::Index(0), j - k); i < j; ++i) { + Scalar aij = a[(k + i - j) + j * lda]; actual_y[i] += temp1 * aij; temp2 += Eigen::numext::conj(aij) * actual_x[i]; } - actual_y[j] += Scalar(Eigen::numext::real(a[*k + j * *lda])) * temp1 + alpha * temp2; + actual_y[j] += Scalar(Eigen::numext::real(a[k + j * lda])) * temp1 + alpha * temp2; } } else { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { + for (Eigen::Index j = 0; j < n; ++j) { Scalar temp1 = alpha * actual_x[j]; Scalar temp2 = Scalar(0); - actual_y[j] += Scalar(Eigen::numext::real(a[j * *lda])) * temp1; - for (EIGEN_BLAS_INT i = j + 1; i <= std::min(*n - 1, j + *k); ++i) { - Scalar aij = a[(i - j) + j * *lda]; + actual_y[j] += Scalar(Eigen::numext::real(a[j * lda])) * temp1; + for (Eigen::Index i = j + 1; i <= (std::min)(n - 1, j + k); ++i) { + Scalar aij = a[(i - j) + j * lda]; actual_y[i] += temp1 * aij; temp2 += Eigen::numext::conj(aij) * actual_x[i]; } @@ -180,7 +183,7 @@ } if (actual_x != x) delete[] actual_x; - if (actual_y != y) delete[] copy_back(actual_y, y, *n, *incy); + if (actual_y != y) delete[] copy_back(actual_y, y, n, *incy); } /** HPMV performs the matrix-vector operation @@ -230,10 +233,10 @@ return; } - EIGEN_BLAS_INT kk = 0; + Eigen::Index kk = 0; if (UPLO(*uplo) == UP) { // Upper triangle packed: column j occupies ap[kk..kk+j]. - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { + for (Eigen::Index j = 0; j < *n; ++j) { Scalar temp1 = alpha * actual_x[j]; // Diagonal is real. actual_y[j] += Scalar(Eigen::numext::real(ap[kk + j])) * temp1; @@ -245,8 +248,8 @@ } } else { // Lower triangle packed: column j occupies ap[kk..kk+(n-j-1)]. - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { - EIGEN_BLAS_INT len = *n - j - 1; + for (Eigen::Index j = 0; j < *n; ++j) { + Eigen::Index len = Eigen::Index(*n) - j - 1; Scalar temp1 = alpha * actual_x[j]; // Diagonal is real. actual_y[j] += Scalar(Eigen::numext::real(ap[kk])) * temp1; @@ -254,7 +257,7 @@ make_vector(actual_y + j + 1, len) += temp1 * make_vector(ap + kk + 1, len); actual_y[j] += alpha * make_vector(ap + kk + 1, len).dot(make_vector(actual_x + j + 1, len)); } - kk += *n - j; + kk += Eigen::Index(*n) - j; } } @@ -271,13 +274,13 @@ */ EIGEN_BLAS_FUNC(hpr) (char *uplo, EIGEN_BLAS_INT *n, RealScalar *palpha, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *pap) { - typedef void (*functype)(EIGEN_BLAS_INT, Scalar *, const Scalar *, RealScalar); + using functype = void (*)(Eigen::Index, Scalar *, const Scalar *, RealScalar); static const functype func[2] = { // array index: UP - (Eigen::internal::selfadjoint_packed_rank1_update<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, Eigen::Upper, false, + (Eigen::internal::selfadjoint_packed_rank1_update<Scalar, Eigen::Index, Eigen::ColMajor, Eigen::Upper, false, Conj>::run), // array index: LO - (Eigen::internal::selfadjoint_packed_rank1_update<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, Eigen::Lower, false, + (Eigen::internal::selfadjoint_packed_rank1_update<Scalar, Eigen::Index, Eigen::ColMajor, Eigen::Lower, false, Conj>::run), }; @@ -316,12 +319,12 @@ EIGEN_BLAS_FUNC(hpr2) (char *uplo, EIGEN_BLAS_INT *n, RealScalar *palpha, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *py, EIGEN_BLAS_INT *incy, RealScalar *pap) { - typedef void (*functype)(EIGEN_BLAS_INT, Scalar *, const Scalar *, const Scalar *, Scalar); + using functype = void (*)(Eigen::Index, Scalar *, const Scalar *, const Scalar *, Scalar); static const functype func[2] = { // array index: UP - (Eigen::internal::packed_rank2_update_selector<Scalar, EIGEN_BLAS_INT, Eigen::Upper>::run), + (Eigen::internal::packed_rank2_update_selector<Scalar, Eigen::Index, Eigen::Upper>::run), // array index: LO - (Eigen::internal::packed_rank2_update_selector<Scalar, EIGEN_BLAS_INT, Eigen::Lower>::run), + (Eigen::internal::packed_rank2_update_selector<Scalar, Eigen::Index, Eigen::Lower>::run), }; Scalar *x = reinterpret_cast<Scalar *>(px); @@ -364,12 +367,12 @@ EIGEN_BLAS_FUNC(her) (char *uplo, EIGEN_BLAS_INT *n, RealScalar *palpha, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *pa, EIGEN_BLAS_INT *lda) { - typedef void (*functype)(EIGEN_BLAS_INT, Scalar *, EIGEN_BLAS_INT, const Scalar *, const Scalar *, const Scalar &); + using functype = void (*)(Eigen::Index, Scalar *, Eigen::Index, const Scalar *, const Scalar *, const Scalar &); static const functype func[2] = { // array index: UP - (Eigen::selfadjoint_rank1_update<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, Eigen::Upper, false, Conj>::run), + (Eigen::selfadjoint_rank1_update<Scalar, Eigen::Index, Eigen::ColMajor, Eigen::Upper, false, Conj>::run), // array index: LO - (Eigen::selfadjoint_rank1_update<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, Eigen::Lower, false, Conj>::run), + (Eigen::selfadjoint_rank1_update<Scalar, Eigen::Index, Eigen::ColMajor, Eigen::Lower, false, Conj>::run), }; Scalar *x = reinterpret_cast<Scalar *>(px); @@ -411,12 +414,12 @@ EIGEN_BLAS_FUNC(her2) (char *uplo, EIGEN_BLAS_INT *n, RealScalar *palpha, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *py, EIGEN_BLAS_INT *incy, RealScalar *pa, EIGEN_BLAS_INT *lda) { - typedef void (*functype)(EIGEN_BLAS_INT, Scalar *, EIGEN_BLAS_INT, const Scalar *, const Scalar *, Scalar); + using functype = void (*)(Eigen::Index, Scalar *, Eigen::Index, const Scalar *, const Scalar *, Scalar); static const functype func[2] = { // array index: UP - (Eigen::internal::rank2_update_selector<Scalar, EIGEN_BLAS_INT, Eigen::Upper>::run), + (Eigen::internal::rank2_update_selector<Scalar, Eigen::Index, Eigen::Upper>::run), // array index: LO - (Eigen::internal::rank2_update_selector<Scalar, EIGEN_BLAS_INT, Eigen::Lower>::run), + (Eigen::internal::rank2_update_selector<Scalar, Eigen::Index, Eigen::Lower>::run), }; Scalar *x = reinterpret_cast<Scalar *>(px); @@ -486,8 +489,8 @@ Scalar *x_cpy = get_compact_vector(x, *m, *incx); Scalar *y_cpy = get_compact_vector(y, *n, *incy); - Eigen::internal::general_rank1_update<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, false, false>::run( - *m, *n, a, *lda, x_cpy, y_cpy, alpha); + Eigen::internal::general_rank1_update<Scalar, Eigen::Index, Eigen::ColMajor, false, false>::run(*m, *n, a, *lda, + x_cpy, y_cpy, alpha); if (x_cpy != x) delete[] x_cpy; if (y_cpy != y) delete[] y_cpy; @@ -526,8 +529,8 @@ Scalar *x_cpy = get_compact_vector(x, *m, *incx); Scalar *y_cpy = get_compact_vector(y, *n, *incy); - Eigen::internal::general_rank1_update<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, false, Conj>::run(*m, *n, a, *lda, - x_cpy, y_cpy, alpha); + Eigen::internal::general_rank1_update<Scalar, Eigen::Index, Eigen::ColMajor, false, Conj>::run(*m, *n, a, *lda, x_cpy, + y_cpy, alpha); if (x_cpy != x) delete[] x_cpy; if (y_cpy != y) delete[] y_cpy;
diff --git a/blas/level2_impl.h b/blas/level2_impl.h index 1688f59..4596f34 100644 --- a/blas/level2_impl.h +++ b/blas/level2_impl.h
@@ -28,15 +28,15 @@ (const char *opa, const EIGEN_BLAS_INT *m, const EIGEN_BLAS_INT *n, const RealScalar *palpha, const RealScalar *pa, const EIGEN_BLAS_INT *lda, const RealScalar *pb, const EIGEN_BLAS_INT *incb, const RealScalar *pbeta, RealScalar *pc, const EIGEN_BLAS_INT *incc) { - typedef void (*functype)(EIGEN_BLAS_INT, EIGEN_BLAS_INT, const Scalar *, EIGEN_BLAS_INT, const Scalar *, - EIGEN_BLAS_INT, Scalar *, EIGEN_BLAS_INT, Scalar); + using functype = void (*)(Eigen::Index, Eigen::Index, const Scalar *, Eigen::Index, const Scalar *, Eigen::Index, + Scalar *, Eigen::Index, Scalar); static const functype func[4] = { // array index: NOTR - (general_matrix_vector_product_wrapper<EIGEN_BLAS_INT, Scalar, Eigen::ColMajor, false, false>::run), + (general_matrix_vector_product_wrapper<Eigen::Index, Scalar, Eigen::ColMajor, false, false>::run), // array index: TR - (general_matrix_vector_product_wrapper<EIGEN_BLAS_INT, Scalar, Eigen::RowMajor, false, false>::run), + (general_matrix_vector_product_wrapper<Eigen::Index, Scalar, Eigen::RowMajor, false, false>::run), // array index: ADJ - (general_matrix_vector_product_wrapper<EIGEN_BLAS_INT, Scalar, Eigen::RowMajor, Conj, false>::run), 0}; + (general_matrix_vector_product_wrapper<Eigen::Index, Scalar, Eigen::RowMajor, Conj, false>::run), 0}; const Scalar *a = reinterpret_cast<const Scalar *>(pa); const Scalar *b = reinterpret_cast<const Scalar *>(pb); @@ -62,8 +62,8 @@ if (*m == 0 || *n == 0 || (alpha == Scalar(0) && beta == Scalar(1))) return; - EIGEN_BLAS_INT actual_m = *m; - EIGEN_BLAS_INT actual_n = *n; + Eigen::Index actual_m = *m; + Eigen::Index actual_n = *n; int code = OP(*opa); if (code != NOTR) std::swap(actual_m, actual_n); @@ -88,7 +88,7 @@ EIGEN_BLAS_FUNC(trsv) (const char *uplo, const char *opa, const char *diag, const EIGEN_BLAS_INT *n, const RealScalar *pa, const EIGEN_BLAS_INT *lda, RealScalar *pb, const EIGEN_BLAS_INT *incb) { - typedef void (*functype)(EIGEN_BLAS_INT, const Scalar *, EIGEN_BLAS_INT, Scalar *); + using functype = void (*)(Eigen::Index, const Scalar *, Eigen::Index, Scalar *); using Eigen::ColMajor; using Eigen::Lower; using Eigen::OnTheLeft; @@ -96,43 +96,43 @@ using Eigen::UnitDiag; using Eigen::Upper; static const functype func[16] = {// array index: NOTR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | 0, false, ColMajor>::run), // array index: TR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | 0, false, RowMajor>::run), // array index: ADJ | (UP << 2) | (NUNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | 0, Conj, RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | 0, false, ColMajor>::run), // array index: TR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | 0, false, RowMajor>::run), // array index: ADJ | (LO << 2) | (NUNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | 0, Conj, RowMajor>::run), 0, // array index: NOTR | (UP << 2) | (UNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | UnitDiag, false, ColMajor>::run), // array index: TR | (UP << 2) | (UNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | UnitDiag, false, RowMajor>::run), // array index: ADJ | (UP << 2) | (UNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | UnitDiag, Conj, RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (UNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | UnitDiag, false, ColMajor>::run), // array index: TR | (LO << 2) | (UNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | UnitDiag, false, RowMajor>::run), // array index: ADJ | (LO << 2) | (UNIT << 3) - (Eigen::internal::triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, + (Eigen::internal::triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | UnitDiag, Conj, RowMajor>::run), 0}; @@ -165,8 +165,8 @@ EIGEN_BLAS_FUNC(trmv) (const char *uplo, const char *opa, const char *diag, const EIGEN_BLAS_INT *n, const RealScalar *pa, const EIGEN_BLAS_INT *lda, RealScalar *pb, const EIGEN_BLAS_INT *incb) { - typedef void (*functype)(EIGEN_BLAS_INT, EIGEN_BLAS_INT, const Scalar *, EIGEN_BLAS_INT, const Scalar *, - EIGEN_BLAS_INT, Scalar *, EIGEN_BLAS_INT, const Scalar &); + using functype = void (*)(Eigen::Index, Eigen::Index, const Scalar *, Eigen::Index, const Scalar *, Eigen::Index, + Scalar *, Eigen::Index, const Scalar &); using Eigen::ColMajor; using Eigen::Lower; using Eigen::OnTheLeft; @@ -175,43 +175,43 @@ using Eigen::Upper; static const functype func[16] = { // array index: NOTR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | 0, Scalar, false, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Upper | 0, Scalar, false, Scalar, false, ColMajor>::run), // array index: TR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | 0, Scalar, false, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Lower | 0, Scalar, false, Scalar, false, RowMajor>::run), // array index: ADJ | (UP << 2) | (NUNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | 0, Scalar, Conj, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Lower | 0, Scalar, Conj, Scalar, false, RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | 0, Scalar, false, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Lower | 0, Scalar, false, Scalar, false, ColMajor>::run), // array index: TR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | 0, Scalar, false, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Upper | 0, Scalar, false, Scalar, false, RowMajor>::run), // array index: ADJ | (LO << 2) | (NUNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | 0, Scalar, Conj, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Upper | 0, Scalar, Conj, Scalar, false, RowMajor>::run), 0, // array index: NOTR | (UP << 2) | (UNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | UnitDiag, Scalar, false, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Upper | UnitDiag, Scalar, false, Scalar, false, ColMajor>::run), // array index: TR | (UP << 2) | (UNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | UnitDiag, Scalar, false, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Lower | UnitDiag, Scalar, false, Scalar, false, RowMajor>::run), // array index: ADJ | (UP << 2) | (UNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | UnitDiag, Scalar, Conj, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Lower | UnitDiag, Scalar, Conj, Scalar, false, RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (UNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | UnitDiag, Scalar, false, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Lower | UnitDiag, Scalar, false, Scalar, false, ColMajor>::run), // array index: TR | (LO << 2) | (UNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | UnitDiag, Scalar, false, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Upper | UnitDiag, Scalar, false, Scalar, false, RowMajor>::run), // array index: ADJ | (LO << 2) | (UNIT << 3) - (Eigen::internal::triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | UnitDiag, Scalar, Conj, Scalar, false, + (Eigen::internal::triangular_matrix_vector_product<Eigen::Index, Upper | UnitDiag, Scalar, Conj, Scalar, false, RowMajor>::run), 0}; @@ -264,7 +264,7 @@ Scalar *y = reinterpret_cast<Scalar *>(py); Scalar alpha = *reinterpret_cast<const Scalar *>(palpha); Scalar beta = *reinterpret_cast<const Scalar *>(pbeta); - EIGEN_BLAS_INT coeff_rows = *kl + *ku + 1; + Eigen::Index coeff_rows = Eigen::Index(*kl) + Eigen::Index(*ku) + 1; EIGEN_BLAS_INT info = 0; if (OP(*trans) == INVALID) @@ -277,7 +277,7 @@ info = 4; else if (*ku < 0) info = 5; - else if (*lda < coeff_rows) + else if (Eigen::Index(*lda) < coeff_rows) info = 8; else if (*incx == 0) info = 10; @@ -287,8 +287,8 @@ if (*m == 0 || *n == 0 || (alpha == Scalar(0) && beta == Scalar(1))) return; - EIGEN_BLAS_INT actual_m = *m; - EIGEN_BLAS_INT actual_n = *n; + Eigen::Index actual_m = *m; + Eigen::Index actual_n = *n; if (OP(*trans) != NOTR) std::swap(actual_m, actual_n); const Scalar *actual_x = get_compact_vector(x, actual_n, *incx); @@ -303,12 +303,12 @@ ConstMatrixType mat_coeffs(a, coeff_rows, *n, *lda); - EIGEN_BLAS_INT nb = alpha == Scalar(0) ? 0 : std::min(*n, (*m) + (*ku)); - for (EIGEN_BLAS_INT j = 0; j < nb; ++j) { - EIGEN_BLAS_INT start = std::max<EIGEN_BLAS_INT>(0, j - *ku); - EIGEN_BLAS_INT end = std::min((*m) - 1, j + *kl); - EIGEN_BLAS_INT len = end - start + 1; - EIGEN_BLAS_INT offset = (*ku) - j + start; + Eigen::Index nb = alpha == Scalar(0) ? 0 : (std::min)(Eigen::Index(*n), Eigen::Index(*m) + Eigen::Index(*ku)); + for (Eigen::Index j = 0; j < nb; ++j) { + Eigen::Index start = (std::max)(Eigen::Index(0), j - Eigen::Index(*ku)); + Eigen::Index end = (std::min)(Eigen::Index(*m) - 1, j + Eigen::Index(*kl)); + Eigen::Index len = end - start + 1; + Eigen::Index offset = Eigen::Index(*ku) - j + start; if (OP(*trans) == NOTR) make_vector(actual_y + start, len) += (alpha * actual_x[j]) * mat_coeffs.col(j).segment(offset, len); else if (OP(*trans) == TR) @@ -334,10 +334,13 @@ * lower triangle stores A[i,j] at a[(i-j) + j*lda]. */ EIGEN_BLAS_FUNC(tbmv) -(char *uplo, char *opa, char *diag, EIGEN_BLAS_INT *n, EIGEN_BLAS_INT *k, RealScalar *pa, EIGEN_BLAS_INT *lda, +(char *uplo, char *opa, char *diag, EIGEN_BLAS_INT *n_in, EIGEN_BLAS_INT *k_in, RealScalar *pa, EIGEN_BLAS_INT *lda_in, RealScalar *px, EIGEN_BLAS_INT *incx) { Scalar *a = reinterpret_cast<Scalar *>(pa); Scalar *x = reinterpret_cast<Scalar *>(px); + const Eigen::Index n = *n_in; + const Eigen::Index k = *k_in; + const Eigen::Index lda = *lda_in; EIGEN_BLAS_INT info = 0; if (UPLO(*uplo) == INVALID) @@ -346,43 +349,43 @@ info = 2; else if (DIAG(*diag) == INVALID) info = 3; - else if (*n < 0) + else if (n < 0) info = 4; - else if (*k < 0) + else if (k < 0) info = 5; - else if (*lda < *k + 1) + else if (lda < k + 1) info = 7; else if (*incx == 0) info = 9; if (info) return xerbla_(SCALAR_SUFFIX_UP "TBMV ", &info, kBlasNameLength); - if (*n == 0) return; + if (n == 0) return; - Scalar *actual_x = get_compact_vector(x, *n, *incx); + Scalar *actual_x = get_compact_vector(x, n, *incx); bool upper = (UPLO(*uplo) == UP); int op = OP(*opa); bool unit = (DIAG(*diag) == UNIT); - if (*k >= 8) { + if (k >= 8) { // Vectorized path: use Eigen Map segments for the inner band operations. - ConstMatrixType band(a, *k + 1, *n, *lda); + ConstMatrixType band(a, k + 1, n, lda); if (op == NOTR) { if (upper) { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { + for (Eigen::Index j = 0; j < n; ++j) { if (actual_x[j] != Scalar(0)) { - EIGEN_BLAS_INT start = std::max<EIGEN_BLAS_INT>(0, j - *k); - EIGEN_BLAS_INT len = j - start; - EIGEN_BLAS_INT offset = *k - (j - start); + Eigen::Index start = (std::max)(Eigen::Index(0), j - k); + Eigen::Index len = j - start; + Eigen::Index offset = k - (j - start); Scalar temp = actual_x[j]; if (len > 0) make_vector(actual_x + start, len) += temp * band.col(j).segment(offset, len); - if (!unit) actual_x[j] = temp * band(*k, j); + if (!unit) actual_x[j] = temp * band(k, j); } } } else { - for (EIGEN_BLAS_INT j = *n - 1; j >= 0; --j) { + for (Eigen::Index j = n - 1; j >= 0; --j) { if (actual_x[j] != Scalar(0)) { - EIGEN_BLAS_INT len = std::min(*n - 1, j + *k) - j; + Eigen::Index len = (std::min)(n - 1, j + k) - j; Scalar temp = actual_x[j]; if (len > 0) make_vector(actual_x + j + 1, len) += temp * band.col(j).segment(1, len); if (!unit) actual_x[j] = temp * band(0, j); @@ -391,19 +394,19 @@ } } else if (op == TR) { if (upper) { - for (EIGEN_BLAS_INT j = *n - 1; j >= 0; --j) { - EIGEN_BLAS_INT start = std::max<EIGEN_BLAS_INT>(0, j - *k); - EIGEN_BLAS_INT len = j - start; - EIGEN_BLAS_INT offset = *k - (j - start); + for (Eigen::Index j = n - 1; j >= 0; --j) { + Eigen::Index start = (std::max)(Eigen::Index(0), j - k); + Eigen::Index len = j - start; + Eigen::Index offset = k - (j - start); Scalar temp = actual_x[j]; - if (!unit) temp *= band(*k, j); + if (!unit) temp *= band(k, j); if (len > 0) temp += (band.col(j).segment(offset, len).cwiseProduct(make_vector(actual_x + start, len))).sum(); actual_x[j] = temp; } } else { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { - EIGEN_BLAS_INT len = std::min(*n - 1, j + *k) - j; + for (Eigen::Index j = 0; j < n; ++j) { + Eigen::Index len = (std::min)(n - 1, j + k) - j; Scalar temp = actual_x[j]; if (!unit) temp *= band(0, j); if (len > 0) temp += (band.col(j).segment(1, len).cwiseProduct(make_vector(actual_x + j + 1, len))).sum(); @@ -413,18 +416,18 @@ } else { // Conjugate transpose: .dot() computes conj(lhs) . rhs. if (upper) { - for (EIGEN_BLAS_INT j = *n - 1; j >= 0; --j) { - EIGEN_BLAS_INT start = std::max<EIGEN_BLAS_INT>(0, j - *k); - EIGEN_BLAS_INT len = j - start; - EIGEN_BLAS_INT offset = *k - (j - start); + for (Eigen::Index j = n - 1; j >= 0; --j) { + Eigen::Index start = (std::max)(Eigen::Index(0), j - k); + Eigen::Index len = j - start; + Eigen::Index offset = k - (j - start); Scalar temp = actual_x[j]; - if (!unit) temp *= Eigen::numext::conj(band(*k, j)); + if (!unit) temp *= Eigen::numext::conj(band(k, j)); if (len > 0) temp += band.col(j).segment(offset, len).dot(make_vector(actual_x + start, len)); actual_x[j] = temp; } } else { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { - EIGEN_BLAS_INT len = std::min(*n - 1, j + *k) - j; + for (Eigen::Index j = 0; j < n; ++j) { + Eigen::Index len = (std::min)(n - 1, j + k) - j; Scalar temp = actual_x[j]; if (!unit) temp *= Eigen::numext::conj(band(0, j)); if (len > 0) temp += band.col(j).segment(1, len).dot(make_vector(actual_x + j + 1, len)); @@ -436,21 +439,20 @@ // Scalar path: for narrow bandwidth, avoid Map overhead. if (op == NOTR) { if (upper) { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { + for (Eigen::Index j = 0; j < n; ++j) { if (actual_x[j] != Scalar(0)) { Scalar temp = actual_x[j]; - for (EIGEN_BLAS_INT i = std::max<EIGEN_BLAS_INT>(0, j - *k); i < j; ++i) - actual_x[i] += temp * a[(*k + i - j) + j * *lda]; - if (!unit) actual_x[j] = temp * a[*k + j * *lda]; + for (Eigen::Index i = (std::max)(Eigen::Index(0), j - k); i < j; ++i) + actual_x[i] += temp * a[(k + i - j) + j * lda]; + if (!unit) actual_x[j] = temp * a[k + j * lda]; } } } else { - for (EIGEN_BLAS_INT j = *n - 1; j >= 0; --j) { + for (Eigen::Index j = n - 1; j >= 0; --j) { if (actual_x[j] != Scalar(0)) { Scalar temp = actual_x[j]; - for (EIGEN_BLAS_INT i = j + 1; i <= std::min(*n - 1, j + *k); ++i) - actual_x[i] += temp * a[(i - j) + j * *lda]; - if (!unit) actual_x[j] = temp * a[j * *lda]; + for (Eigen::Index i = j + 1; i <= (std::min)(n - 1, j + k); ++i) actual_x[i] += temp * a[(i - j) + j * lda]; + if (!unit) actual_x[j] = temp * a[j * lda]; } } } @@ -458,26 +460,26 @@ // Transpose or conjugate transpose. auto maybe_conj = [op](Scalar val) -> Scalar { return op == ADJ ? Eigen::numext::conj(val) : val; }; if (upper) { - for (EIGEN_BLAS_INT j = *n - 1; j >= 0; --j) { + for (Eigen::Index j = n - 1; j >= 0; --j) { Scalar temp = actual_x[j]; - if (!unit) temp *= maybe_conj(a[*k + j * *lda]); - for (EIGEN_BLAS_INT i = std::max<EIGEN_BLAS_INT>(0, j - *k); i < j; ++i) - temp += maybe_conj(a[(*k + i - j) + j * *lda]) * actual_x[i]; + if (!unit) temp *= maybe_conj(a[k + j * lda]); + for (Eigen::Index i = (std::max)(Eigen::Index(0), j - k); i < j; ++i) + temp += maybe_conj(a[(k + i - j) + j * lda]) * actual_x[i]; actual_x[j] = temp; } } else { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { + for (Eigen::Index j = 0; j < n; ++j) { Scalar temp = actual_x[j]; - if (!unit) temp *= maybe_conj(a[j * *lda]); - for (EIGEN_BLAS_INT i = j + 1; i <= std::min(*n - 1, j + *k); ++i) - temp += maybe_conj(a[(i - j) + j * *lda]) * actual_x[i]; + if (!unit) temp *= maybe_conj(a[j * lda]); + for (Eigen::Index i = j + 1; i <= (std::min)(n - 1, j + k); ++i) + temp += maybe_conj(a[(i - j) + j * lda]) * actual_x[i]; actual_x[j] = temp; } } } } - if (actual_x != x) delete[] copy_back(actual_x, x, *n, *incx); + if (actual_x != x) delete[] copy_back(actual_x, x, n, *incx); } /** DTBSV solves one of the systems of equations @@ -494,7 +496,7 @@ EIGEN_BLAS_FUNC(tbsv) (char *uplo, char *op, char *diag, EIGEN_BLAS_INT *n, EIGEN_BLAS_INT *k, RealScalar *pa, EIGEN_BLAS_INT *lda, RealScalar *px, EIGEN_BLAS_INT *incx) { - typedef void (*functype)(EIGEN_BLAS_INT, EIGEN_BLAS_INT, const Scalar *, EIGEN_BLAS_INT, Scalar *); + using functype = void (*)(Eigen::Index, Eigen::Index, const Scalar *, Eigen::Index, Scalar *); using Eigen::ColMajor; using Eigen::Lower; using Eigen::OnTheLeft; @@ -503,48 +505,44 @@ using Eigen::Upper; static const functype func[16] = { // array index: NOTR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Upper | 0, Scalar, false, Scalar, - ColMajor>::run), + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Upper | 0, Scalar, false, Scalar, ColMajor>::run), // array index: TR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Lower | 0, Scalar, false, Scalar, - RowMajor>::run), + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Lower | 0, Scalar, false, Scalar, RowMajor>::run), // array index: ADJ | (UP << 2) | (NUNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Lower | 0, Scalar, Conj, Scalar, RowMajor>::run), + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Lower | 0, Scalar, Conj, Scalar, RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Lower | 0, Scalar, false, Scalar, - ColMajor>::run), + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Lower | 0, Scalar, false, Scalar, ColMajor>::run), // array index: TR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Upper | 0, Scalar, false, Scalar, - RowMajor>::run), + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Upper | 0, Scalar, false, Scalar, RowMajor>::run), // array index: ADJ | (LO << 2) | (NUNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Upper | 0, Scalar, Conj, Scalar, RowMajor>::run), + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Upper | 0, Scalar, Conj, Scalar, RowMajor>::run), 0, // array index: NOTR | (UP << 2) | (UNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Upper | UnitDiag, Scalar, false, Scalar, + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Upper | UnitDiag, Scalar, false, Scalar, ColMajor>::run), // array index: TR | (UP << 2) | (UNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Lower | UnitDiag, Scalar, false, Scalar, + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Lower | UnitDiag, Scalar, false, Scalar, RowMajor>::run), // array index: ADJ | (UP << 2) | (UNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Lower | UnitDiag, Scalar, Conj, Scalar, + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Lower | UnitDiag, Scalar, Conj, Scalar, RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (UNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Lower | UnitDiag, Scalar, false, Scalar, + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Lower | UnitDiag, Scalar, false, Scalar, ColMajor>::run), // array index: TR | (LO << 2) | (UNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Upper | UnitDiag, Scalar, false, Scalar, + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Upper | UnitDiag, Scalar, false, Scalar, RowMajor>::run), // array index: ADJ | (LO << 2) | (UNIT << 3) - (Eigen::internal::band_solve_triangular_selector<EIGEN_BLAS_INT, Upper | UnitDiag, Scalar, Conj, Scalar, + (Eigen::internal::band_solve_triangular_selector<Eigen::Index, Upper | UnitDiag, Scalar, Conj, Scalar, RowMajor>::run), 0, }; Scalar *a = reinterpret_cast<Scalar *>(pa); Scalar *x = reinterpret_cast<Scalar *>(px); - EIGEN_BLAS_INT coeff_rows = *k + 1; + Eigen::Index coeff_rows = Eigen::Index(*k) + 1; EIGEN_BLAS_INT info = 0; if (UPLO(*uplo) == INVALID) @@ -557,7 +555,7 @@ info = 4; else if (*k < 0) info = 5; - else if (*lda < coeff_rows) + else if (Eigen::Index(*lda) < coeff_rows) info = 7; else if (*incx == 0) info = 9; @@ -565,7 +563,7 @@ if (*n == 0 || (*k == 0 && DIAG(*diag) == UNIT)) return; - EIGEN_BLAS_INT actual_n = *n; + Eigen::Index actual_n = *n; Scalar *actual_x = get_compact_vector(x, actual_n, *incx); @@ -586,7 +584,7 @@ */ EIGEN_BLAS_FUNC(tpmv) (char *uplo, char *opa, char *diag, EIGEN_BLAS_INT *n, RealScalar *pap, RealScalar *px, EIGEN_BLAS_INT *incx) { - typedef void (*functype)(EIGEN_BLAS_INT, const Scalar *, const Scalar *, Scalar *, Scalar); + using functype = void (*)(Eigen::Index, const Scalar *, const Scalar *, Scalar *, Scalar); using Eigen::ColMajor; using Eigen::Lower; using Eigen::OnTheLeft; @@ -595,43 +593,43 @@ using Eigen::Upper; static const functype func[16] = { // array index: NOTR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | 0, Scalar, false, Scalar, false, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Upper | 0, Scalar, false, Scalar, false, ColMajor>::run), // array index: TR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | 0, Scalar, false, Scalar, false, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Lower | 0, Scalar, false, Scalar, false, RowMajor>::run), // array index: ADJ | (UP << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | 0, Scalar, Conj, Scalar, false, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Lower | 0, Scalar, Conj, Scalar, false, RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | 0, Scalar, false, Scalar, false, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Lower | 0, Scalar, false, Scalar, false, ColMajor>::run), // array index: TR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | 0, Scalar, false, Scalar, false, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Upper | 0, Scalar, false, Scalar, false, RowMajor>::run), // array index: ADJ | (LO << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | 0, Scalar, Conj, Scalar, false, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Upper | 0, Scalar, Conj, Scalar, false, RowMajor>::run), 0, // array index: NOTR | (UP << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | UnitDiag, Scalar, false, Scalar, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Upper | UnitDiag, Scalar, false, Scalar, false, ColMajor>::run), // array index: TR | (UP << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | UnitDiag, Scalar, false, Scalar, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Lower | UnitDiag, Scalar, false, Scalar, false, RowMajor>::run), // array index: ADJ | (UP << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | UnitDiag, Scalar, Conj, Scalar, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Lower | UnitDiag, Scalar, Conj, Scalar, false, RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Lower | UnitDiag, Scalar, false, Scalar, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Lower | UnitDiag, Scalar, false, Scalar, false, ColMajor>::run), // array index: TR | (LO << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | UnitDiag, Scalar, false, Scalar, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Upper | UnitDiag, Scalar, false, Scalar, false, RowMajor>::run), // array index: ADJ | (LO << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_matrix_vector_product<EIGEN_BLAS_INT, Upper | UnitDiag, Scalar, Conj, Scalar, + (Eigen::internal::packed_triangular_matrix_vector_product<Eigen::Index, Upper | UnitDiag, Scalar, Conj, Scalar, false, RowMajor>::run), 0}; @@ -678,7 +676,7 @@ */ EIGEN_BLAS_FUNC(tpsv) (char *uplo, char *opa, char *diag, EIGEN_BLAS_INT *n, RealScalar *pap, RealScalar *px, EIGEN_BLAS_INT *incx) { - typedef void (*functype)(EIGEN_BLAS_INT, const Scalar *, Scalar *); + using functype = void (*)(Eigen::Index, const Scalar *, Scalar *); using Eigen::ColMajor; using Eigen::Lower; using Eigen::OnTheLeft; @@ -687,44 +685,44 @@ using Eigen::Upper; static const functype func[16] = { // array index: NOTR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Upper | 0, false, + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | 0, false, ColMajor>::run), // array index: TR | (UP << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Lower | 0, false, + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | 0, false, RowMajor>::run), // array index: ADJ | (UP << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Lower | 0, Conj, + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | 0, Conj, RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Lower | 0, false, + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | 0, false, ColMajor>::run), // array index: TR | (LO << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Upper | 0, false, + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | 0, false, RowMajor>::run), // array index: ADJ | (LO << 2) | (NUNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Upper | 0, Conj, + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | 0, Conj, RowMajor>::run), 0, // array index: NOTR | (UP << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Upper | UnitDiag, - false, ColMajor>::run), + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | UnitDiag, false, + ColMajor>::run), // array index: TR | (UP << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Lower | UnitDiag, - false, RowMajor>::run), + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | UnitDiag, false, + RowMajor>::run), // array index: ADJ | (UP << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Lower | UnitDiag, - Conj, RowMajor>::run), + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | UnitDiag, Conj, + RowMajor>::run), 0, // array index: NOTR | (LO << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Lower | UnitDiag, - false, ColMajor>::run), + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Lower | UnitDiag, false, + ColMajor>::run), // array index: TR | (LO << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Upper | UnitDiag, - false, RowMajor>::run), + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | UnitDiag, false, + RowMajor>::run), // array index: ADJ | (LO << 2) | (UNIT << 3) - (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, EIGEN_BLAS_INT, OnTheLeft, Upper | UnitDiag, - Conj, RowMajor>::run), + (Eigen::internal::packed_triangular_solve_vector<Scalar, Scalar, Eigen::Index, OnTheLeft, Upper | UnitDiag, Conj, + RowMajor>::run), 0}; Scalar *ap = reinterpret_cast<Scalar *>(pap);
diff --git a/blas/level2_real_impl.h b/blas/level2_real_impl.h index 8ca64eb..c0cda93 100644 --- a/blas/level2_real_impl.h +++ b/blas/level2_real_impl.h
@@ -15,15 +15,15 @@ (const char *uplo, const EIGEN_BLAS_INT *n, const RealScalar *palpha, const RealScalar *pa, const EIGEN_BLAS_INT *lda, const RealScalar *px, const EIGEN_BLAS_INT *incx, const RealScalar *pbeta, RealScalar *py, const EIGEN_BLAS_INT *incy) { - typedef void (*functype)(EIGEN_BLAS_INT, const Scalar *, EIGEN_BLAS_INT, const Scalar *, Scalar *, Scalar); + using functype = void (*)(Eigen::Index, const Scalar *, Eigen::Index, const Scalar *, Scalar *, Scalar); using Eigen::ColMajor; using Eigen::Lower; using Eigen::Upper; static const functype func[2] = { // array index: UP - (Eigen::internal::selfadjoint_matrix_vector_product<Scalar, EIGEN_BLAS_INT, ColMajor, Upper, false, false>::run), + (Eigen::internal::selfadjoint_matrix_vector_product<Scalar, Eigen::Index, ColMajor, Upper, false, false>::run), // array index: LO - (Eigen::internal::selfadjoint_matrix_vector_product<Scalar, EIGEN_BLAS_INT, ColMajor, Lower, false, false>::run), + (Eigen::internal::selfadjoint_matrix_vector_product<Scalar, Eigen::Index, ColMajor, Lower, false, false>::run), }; const Scalar *a = reinterpret_cast<const Scalar *>(pa); @@ -71,15 +71,15 @@ EIGEN_BLAS_FUNC(syr) (const char *uplo, const EIGEN_BLAS_INT *n, const RealScalar *palpha, const RealScalar *px, const EIGEN_BLAS_INT *incx, RealScalar *pc, const EIGEN_BLAS_INT *ldc) { - typedef void (*functype)(EIGEN_BLAS_INT, Scalar *, EIGEN_BLAS_INT, const Scalar *, const Scalar *, const Scalar &); + using functype = void (*)(Eigen::Index, Scalar *, Eigen::Index, const Scalar *, const Scalar *, const Scalar &); using Eigen::ColMajor; using Eigen::Lower; using Eigen::Upper; static const functype func[2] = { // array index: UP - (Eigen::selfadjoint_rank1_update<Scalar, EIGEN_BLAS_INT, ColMajor, Upper, false, Conj>::run), + (Eigen::selfadjoint_rank1_update<Scalar, Eigen::Index, ColMajor, Upper, false, Conj>::run), // array index: LO - (Eigen::selfadjoint_rank1_update<Scalar, EIGEN_BLAS_INT, ColMajor, Lower, false, Conj>::run), + (Eigen::selfadjoint_rank1_update<Scalar, Eigen::Index, ColMajor, Lower, false, Conj>::run), }; const Scalar *x = reinterpret_cast<const Scalar *>(px); @@ -114,12 +114,12 @@ EIGEN_BLAS_FUNC(syr2) (const char *uplo, const EIGEN_BLAS_INT *n, const RealScalar *palpha, const RealScalar *px, const EIGEN_BLAS_INT *incx, const RealScalar *py, const EIGEN_BLAS_INT *incy, RealScalar *pc, const EIGEN_BLAS_INT *ldc) { - typedef void (*functype)(EIGEN_BLAS_INT, Scalar *, EIGEN_BLAS_INT, const Scalar *, const Scalar *, Scalar); + using functype = void (*)(Eigen::Index, Scalar *, Eigen::Index, const Scalar *, const Scalar *, Scalar); static const functype func[2] = { // array index: UP - (Eigen::internal::rank2_update_selector<Scalar, EIGEN_BLAS_INT, Eigen::Upper>::run), + (Eigen::internal::rank2_update_selector<Scalar, Eigen::Index, Eigen::Upper>::run), // array index: LO - (Eigen::internal::rank2_update_selector<Scalar, EIGEN_BLAS_INT, Eigen::Lower>::run), + (Eigen::internal::rank2_update_selector<Scalar, Eigen::Index, Eigen::Lower>::run), }; const Scalar *x = reinterpret_cast<const Scalar *>(px); @@ -171,22 +171,25 @@ * lower triangle stores A[i,j] at a[(i-j) + j*lda]. */ EIGEN_BLAS_FUNC(sbmv) -(char *uplo, EIGEN_BLAS_INT *n, EIGEN_BLAS_INT *k, RealScalar *palpha, RealScalar *pa, EIGEN_BLAS_INT *lda, +(char *uplo, EIGEN_BLAS_INT *n_in, EIGEN_BLAS_INT *k_in, RealScalar *palpha, RealScalar *pa, EIGEN_BLAS_INT *lda_in, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *pbeta, RealScalar *py, EIGEN_BLAS_INT *incy) { const Scalar alpha = *reinterpret_cast<const Scalar *>(palpha); const Scalar beta = *reinterpret_cast<const Scalar *>(pbeta); const Scalar *a = reinterpret_cast<const Scalar *>(pa); const Scalar *x = reinterpret_cast<const Scalar *>(px); Scalar *y = reinterpret_cast<Scalar *>(py); + const Eigen::Index n = *n_in; + const Eigen::Index k = *k_in; + const Eigen::Index lda = *lda_in; EIGEN_BLAS_INT info = 0; if (UPLO(*uplo) == INVALID) info = 1; - else if (*n < 0) + else if (n < 0) info = 2; - else if (*k < 0) + else if (k < 0) info = 3; - else if (*lda < *k + 1) + else if (lda < k + 1) info = 6; else if (*incx == 0) info = 8; @@ -194,43 +197,43 @@ info = 11; if (info) return xerbla_(SCALAR_SUFFIX_UP "SBMV ", &info, kBlasNameLength); - if (*n == 0 || (alpha == Scalar(0) && beta == Scalar(1))) return; + if (n == 0 || (alpha == Scalar(0) && beta == Scalar(1))) return; - const Scalar *actual_x = get_compact_vector(x, *n, *incx); - Scalar *actual_y = get_compact_vector(y, *n, *incy); + const Scalar *actual_x = get_compact_vector(x, n, *incx); + Scalar *actual_y = get_compact_vector(y, n, *incy); // First form y := beta*y. if (beta != Scalar(1)) { if (beta == Scalar(0)) - make_vector(actual_y, *n).setZero(); + make_vector(actual_y, n).setZero(); else - make_vector(actual_y, *n) *= beta; + make_vector(actual_y, n) *= beta; } if (alpha == Scalar(0)) { if (actual_x != x) delete[] actual_x; - if (actual_y != y) delete[] copy_back(actual_y, y, *n, *incy); + if (actual_y != y) delete[] copy_back(actual_y, y, n, *incy); return; } - if (*k >= 8) { + if (k >= 8) { // Vectorized path: use Eigen Map segments for the inner band operations. - ConstMatrixType band(a, *k + 1, *n, *lda); + ConstMatrixType band(a, k + 1, n, lda); if (UPLO(*uplo) == UP) { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { - EIGEN_BLAS_INT start = std::max<EIGEN_BLAS_INT>(0, j - *k); - EIGEN_BLAS_INT len = j - start; - EIGEN_BLAS_INT offset = *k - (j - start); + for (Eigen::Index j = 0; j < n; ++j) { + Eigen::Index start = (std::max)(Eigen::Index(0), j - k); + Eigen::Index len = j - start; + Eigen::Index offset = k - (j - start); Scalar temp1 = alpha * actual_x[j]; - actual_y[j] += temp1 * band(*k, j); + actual_y[j] += temp1 * band(k, j); if (len > 0) { make_vector(actual_y + start, len) += temp1 * band.col(j).segment(offset, len); actual_y[j] += alpha * band.col(j).segment(offset, len).dot(make_vector(actual_x + start, len)); } } } else { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { - EIGEN_BLAS_INT len = std::min(*n - 1, j + *k) - j; + for (Eigen::Index j = 0; j < n; ++j) { + Eigen::Index len = (std::min)(n - 1, j + k) - j; Scalar temp1 = alpha * actual_x[j]; actual_y[j] += temp1 * band(0, j); if (len > 0) { @@ -242,23 +245,23 @@ } else { // Scalar path: for narrow bandwidth, avoid Map overhead. if (UPLO(*uplo) == UP) { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { + for (Eigen::Index j = 0; j < n; ++j) { Scalar temp1 = alpha * actual_x[j]; Scalar temp2 = Scalar(0); - for (EIGEN_BLAS_INT i = std::max<EIGEN_BLAS_INT>(0, j - *k); i < j; ++i) { - Scalar aij = a[(*k + i - j) + j * *lda]; + for (Eigen::Index i = (std::max)(Eigen::Index(0), j - k); i < j; ++i) { + Scalar aij = a[(k + i - j) + j * lda]; actual_y[i] += temp1 * aij; temp2 += aij * actual_x[i]; } - actual_y[j] += temp1 * a[*k + j * *lda] + alpha * temp2; + actual_y[j] += temp1 * a[k + j * lda] + alpha * temp2; } } else { - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { + for (Eigen::Index j = 0; j < n; ++j) { Scalar temp1 = alpha * actual_x[j]; Scalar temp2 = Scalar(0); - actual_y[j] += temp1 * a[j * *lda]; - for (EIGEN_BLAS_INT i = j + 1; i <= std::min(*n - 1, j + *k); ++i) { - Scalar aij = a[(i - j) + j * *lda]; + actual_y[j] += temp1 * a[j * lda]; + for (Eigen::Index i = j + 1; i <= (std::min)(n - 1, j + k); ++i) { + Scalar aij = a[(i - j) + j * lda]; actual_y[i] += temp1 * aij; temp2 += aij * actual_x[i]; } @@ -268,7 +271,7 @@ } if (actual_x != x) delete[] actual_x; - if (actual_y != y) delete[] copy_back(actual_y, y, *n, *incy); + if (actual_y != y) delete[] copy_back(actual_y, y, n, *incy); } /** SPMV performs the matrix-vector operation @@ -321,10 +324,10 @@ return; } - EIGEN_BLAS_INT kk = 0; + Eigen::Index kk = 0; if (UPLO(*uplo) == UP) { // Upper triangle packed: column j occupies ap[kk..kk+j]. - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { + for (Eigen::Index j = 0; j < *n; ++j) { Scalar temp1 = alpha * actual_x[j]; actual_y[j] += temp1 * ap[kk + j]; if (j > 0) { @@ -335,15 +338,15 @@ } } else { // Lower triangle packed: column j occupies ap[kk..kk+(n-j-1)]. - for (EIGEN_BLAS_INT j = 0; j < *n; ++j) { - EIGEN_BLAS_INT len = *n - j - 1; + for (Eigen::Index j = 0; j < *n; ++j) { + Eigen::Index len = Eigen::Index(*n) - j - 1; Scalar temp1 = alpha * actual_x[j]; actual_y[j] += temp1 * ap[kk]; if (len > 0) { make_vector(actual_y + j + 1, len) += temp1 * make_vector(ap + kk + 1, len); actual_y[j] += alpha * make_vector(ap + kk + 1, len).dot(make_vector(actual_x + j + 1, len)); } - kk += *n - j; + kk += Eigen::Index(*n) - j; } } @@ -359,13 +362,13 @@ * n by n symmetric matrix, supplied in packed form. */ EIGEN_BLAS_FUNC(spr)(char *uplo, EIGEN_BLAS_INT *n, Scalar *palpha, Scalar *px, EIGEN_BLAS_INT *incx, Scalar *pap) { - typedef void (*functype)(EIGEN_BLAS_INT, Scalar *, const Scalar *, Scalar); + using functype = void (*)(Eigen::Index, Scalar *, const Scalar *, Scalar); static const functype func[2] = { // array index: UP - (Eigen::internal::selfadjoint_packed_rank1_update<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, Eigen::Upper, false, + (Eigen::internal::selfadjoint_packed_rank1_update<Scalar, Eigen::Index, Eigen::ColMajor, Eigen::Upper, false, false>::run), // array index: LO - (Eigen::internal::selfadjoint_packed_rank1_update<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, Eigen::Lower, false, + (Eigen::internal::selfadjoint_packed_rank1_update<Scalar, Eigen::Index, Eigen::ColMajor, Eigen::Lower, false, false>::run), }; @@ -404,12 +407,12 @@ EIGEN_BLAS_FUNC(spr2) (char *uplo, EIGEN_BLAS_INT *n, RealScalar *palpha, RealScalar *px, EIGEN_BLAS_INT *incx, RealScalar *py, EIGEN_BLAS_INT *incy, RealScalar *pap) { - typedef void (*functype)(EIGEN_BLAS_INT, Scalar *, const Scalar *, const Scalar *, Scalar); + using functype = void (*)(Eigen::Index, Scalar *, const Scalar *, const Scalar *, Scalar); static const functype func[2] = { // array index: UP - (Eigen::internal::packed_rank2_update_selector<Scalar, EIGEN_BLAS_INT, Eigen::Upper>::run), + (Eigen::internal::packed_rank2_update_selector<Scalar, Eigen::Index, Eigen::Upper>::run), // array index: LO - (Eigen::internal::packed_rank2_update_selector<Scalar, EIGEN_BLAS_INT, Eigen::Lower>::run), + (Eigen::internal::packed_rank2_update_selector<Scalar, Eigen::Index, Eigen::Lower>::run), }; Scalar *x = reinterpret_cast<Scalar *>(px); @@ -475,8 +478,8 @@ Scalar *x_cpy = get_compact_vector(x, *m, *incx); Scalar *y_cpy = get_compact_vector(y, *n, *incy); - Eigen::internal::general_rank1_update<Scalar, EIGEN_BLAS_INT, Eigen::ColMajor, false, false>::run( - *m, *n, a, *lda, x_cpy, y_cpy, alpha); + Eigen::internal::general_rank1_update<Scalar, Eigen::Index, Eigen::ColMajor, false, false>::run(*m, *n, a, *lda, + x_cpy, y_cpy, alpha); if (x_cpy != x) delete[] x_cpy; if (y_cpy != y) delete[] y_cpy;
diff --git a/blas/testing/CMakeLists.txt b/blas/testing/CMakeLists.txt index 7f9a78e..9cb4b85 100644 --- a/blas/testing/CMakeLists.txt +++ b/blas/testing/CMakeLists.txt
@@ -10,6 +10,12 @@ set(EIGEN_Fortran_COMPILER_WORKS OFF) endif() +if(TARGET eigen_blas) + set(eigen_blas_lib eigen_blas) +else() + set(eigen_blas_lib eigen_blas_static) +endif() + if(EIGEN_BUILD_BLAS_ILP64) if(TARGET eigen_blas_ilp64) set(eigen_blas_ilp64_lib eigen_blas_ilp64) @@ -141,3 +147,14 @@ add_dependencies(buildtests blas_ilp64) endif() +add_executable(blas_large_stride blas_large_stride.cpp) +target_include_directories(blas_large_stride PRIVATE "${Eigen_SOURCE_DIR}" "${Eigen_SOURCE_DIR}/test") +target_link_libraries(blas_large_stride ${eigen_blas_lib}) +if(EIGEN_STANDARD_LIBRARIES_TO_LINK_TO) + target_link_libraries(blas_large_stride ${EIGEN_STANDARD_LIBRARIES_TO_LINK_TO}) +endif() +# Windows has no rpath: the loader finds the DLL only beside the executable or on PATH. +set_target_properties(blas_large_stride PROPERTIES RUNTIME_OUTPUT_DIRECTORY "$<TARGET_FILE_DIR:${eigen_blas_lib}>") +add_test(NAME blas_large_stride COMMAND blas_large_stride) +add_dependencies(buildtests blas_large_stride) +
diff --git a/blas/testing/blas_large_stride.cpp b/blas/testing/blas_large_stride.cpp new file mode 100644 index 0000000..7556740 --- /dev/null +++ b/blas/testing/blas_large_stride.cpp
@@ -0,0 +1,241 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// This Source Code Form is subject to the terms of the Mozilla +// Public License v. 2.0. If a copy of the MPL was not distributed +// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include "main.h" +#include "../blas.h" + +namespace { + +// With a singleton outer dimension (n = 1 column or 1 vector element), BLAS permits arbitrarily +// large leading dimensions and vector increments up to INT_MAX while only accessing the first +// column or element on the stack. This exercises internal index widening (such as +// lhs.stride() * sizeof(Scalar) in GEMV) across all platforms and compilers without allocating +// large buffers. +void test_level1_large_stride() { + EIGEN_BLAS_INT one = 1; + EIGEN_BLAS_INT zero = 0; + EIGEN_BLAS_INT large_inc = (std::numeric_limits<EIGEN_BLAS_INT>::max)(); + EIGEN_BLAS_INT neg_inc = -large_inc; + + // Broadcast and negative-stride copy. + double x = 42.0, y = 0.0; + BLASFUNC(dcopy)(&one, &x, &zero, &y, &neg_inc); + VERIFY_IS_EQUAL(y, 42.0); + + y = 0.0; + BLASFUNC(dcopy)(&one, &x, &large_inc, &y, &neg_inc); + VERIFY_IS_EQUAL(y, 42.0); + + // Modified Givens rotation with extreme positive and negative increments. + double param[5] = {-1.0, 0.0, 1.0, 1.0, 0.0}; + x = 3.0; + y = 7.0; + BLASFUNC(drotm)(&one, &x, &neg_inc, &y, &large_inc, param); + VERIFY_IS_EQUAL(x, 7.0); + VERIFY_IS_EQUAL(y, 3.0); + + // Dot products, axpy, scal, swap, reductions. + VERIFY_IS_EQUAL(BLASFUNC(ddot)(&one, &x, &large_inc, &y, &neg_inc), 21.0); + + float sx = 3.0f, sy = 7.0f, sb = 1.0f; + VERIFY_IS_EQUAL(BLASFUNC(dsdot)(&one, &sx, &large_inc, &sy, &neg_inc), 21.0); + VERIFY_IS_EQUAL(BLASFUNC(sdsdot)(&one, &sb, &sx, &large_inc, &sy, &neg_inc), 22.0f); + + double alpha = 2.0; + BLASFUNC(daxpy)(&one, &alpha, &x, &large_inc, &y, &neg_inc); + VERIFY_IS_EQUAL(y, 17.0); + + BLASFUNC(dscal)(&one, &alpha, &x, &large_inc); + VERIFY_IS_EQUAL(x, 14.0); + + BLASFUNC(dswap)(&one, &x, &large_inc, &y, &neg_inc); + VERIFY_IS_EQUAL(x, 17.0); + VERIFY_IS_EQUAL(y, 14.0); + + VERIFY_IS_EQUAL(BLASFUNC(dasum)(&one, &x, &large_inc), 17.0); + VERIFY_IS_EQUAL(BLASFUNC(dnrm2)(&one, &x, &large_inc), 17.0); + VERIFY_IS_EQUAL(BLASFUNC(idamax)(&one, &x, &large_inc), 1); +} + +void test_level2_real_large_stride() { + char uplo = 'U', trans = 'N', trans_t = 'T', diag = 'N'; + EIGEN_BLAS_INT one = 1, two = 2, eight = 8, k8 = 8; + EIGEN_BLAS_INT large_lda = (std::numeric_limits<EIGEN_BLAS_INT>::max)(); + EIGEN_BLAS_INT stride_28 = EIGEN_BLAS_INT(1) << 28; // stride_28 * sizeof(double) == 2^31 overflows 32-bit int. + EIGEN_BLAS_INT neg_inc = -large_lda; + double alpha = 2.0, beta = 0.0; + + double a[12] = {1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0}; + double x[8] = {1.0, 2.0, 1.0, 2.0, 1.0, 2.0, 1.0, 2.0}; + double y[8] = {}; + + for (EIGEN_BLAS_INT lda : {stride_28, large_lda}) { + // dgemv "N" and "T" for m = 2 (run_small_cols) and m = 8 (full packet path). + BLASFUNC(dgemv)("N", &two, &one, &alpha, a, &lda, x, &large_lda, &beta, y, &one); + VERIFY_IS_EQUAL(y[0], 2.0); + VERIFY_IS_EQUAL(y[1], 4.0); + + BLASFUNC(dgemv)("T", &two, &one, &alpha, a, &lda, x, &one, &beta, y, &large_lda); + VERIFY_IS_EQUAL(y[0], 10.0); + + BLASFUNC(dgemv)("T", &eight, &one, &alpha, a, &lda, x, &one, &beta, y, &large_lda); + VERIFY_IS_EQUAL(y[0], 112.0); + } + + // dtrmv & dtrsv (1x1 with large_lda and large_inc). + double tx = 3.0; + BLASFUNC(dtrmv)("U", "N", "N", &one, a, &large_lda, &tx, &large_lda); + VERIFY_IS_EQUAL(tx, 3.0); + a[0] = 2.0; + tx = 6.0; + BLASFUNC(dtrsv)("U", "N", "N", &one, a, &large_lda, &tx, &neg_inc); + VERIFY_IS_EQUAL(tx, 3.0); + + // Banded routines: dgbmv, dtbmv, dtbsv, dsbmv (k = 1 scalar and k = 8 vectorized paths). + a[0] = 2.0; + a[1] = 3.0; + BLASFUNC(dgbmv)(&trans, &two, &one, &one, &one, &alpha, a, &large_lda, x, &large_lda, &beta, y, &one); + VERIFY_IS_EQUAL(y[0], 6.0); + BLASFUNC(dgbmv)(&trans_t, &two, &one, &one, &one, &alpha, a, &large_lda, x, &one, &beta, y, &large_lda); + VERIFY_IS_EQUAL(y[0], 18.0); + + for (EIGEN_BLAS_INT k : {one, k8}) { + a[k] = 2.0; + tx = 3.0; + BLASFUNC(dtbmv)(&uplo, &trans, &diag, &one, &k, a, &large_lda, &tx, &large_lda); + VERIFY_IS_EQUAL(tx, 6.0); + BLASFUNC(dtbsv)(&uplo, &trans, &diag, &one, &k, a, &large_lda, &tx, &large_lda); + VERIFY_IS_EQUAL(tx, 3.0); + + y[0] = 0.0; + BLASFUNC(dsbmv)(&uplo, &one, &k, &alpha, a, &large_lda, &tx, &large_lda, &beta, y, &large_lda); + VERIFY_IS_EQUAL(y[0], 12.0); + } + + // Symmetric & rank updates: dsymv, dger, dsyr, dsyr2. + a[0] = 3.0; + y[0] = 0.0; + BLASFUNC(dsymv)(&uplo, &one, &alpha, a, &large_lda, x, &large_lda, &beta, y, &large_lda); + VERIFY_IS_EQUAL(y[0], 6.0); + + a[0] = 0.0; + a[1] = 0.0; + BLASFUNC(dger)(&two, &one, &alpha, x, &one, &tx, &large_lda, a, &large_lda); + VERIFY_IS_EQUAL(a[0], 6.0); + VERIFY_IS_EQUAL(a[1], 12.0); + + BLASFUNC(dsyr)(&uplo, &one, &alpha, &tx, &large_lda, a, &large_lda); + VERIFY_IS_EQUAL(a[0], 24.0); + + BLASFUNC(dsyr2)(&uplo, &one, &alpha, &tx, &large_lda, x, &large_lda, a, &large_lda); + VERIFY_IS_EQUAL(a[0], 36.0); + + // Packed routines: dspmv, dtpmv, dtpsv, dspr, dspr2. + double ap = 2.0; + tx = 3.0; + y[0] = 0.0; + BLASFUNC(dspmv)(&uplo, &one, &alpha, &ap, &tx, &large_lda, &beta, y, &large_lda); + VERIFY_IS_EQUAL(y[0], 12.0); + + BLASFUNC(dtpmv)(&uplo, &trans, &diag, &one, &ap, &tx, &large_lda); + VERIFY_IS_EQUAL(tx, 6.0); + BLASFUNC(dtpsv)(&uplo, &trans, &diag, &one, &ap, &tx, &large_lda); + VERIFY_IS_EQUAL(tx, 3.0); + + BLASFUNC(dspr)(&uplo, &one, &alpha, &tx, &large_lda, &ap); + VERIFY_IS_EQUAL(ap, 20.0); + BLASFUNC(dspr2)(&uplo, &one, &alpha, &tx, &large_lda, x, &large_lda, &ap); + VERIFY_IS_EQUAL(ap, 32.0); +} + +void test_level2_complex_large_stride() { + char uplo = 'U'; + EIGEN_BLAS_INT one = 1, two = 2, eight = 8, k8 = 8; + EIGEN_BLAS_INT large_lda = (std::numeric_limits<EIGEN_BLAS_INT>::max)(); + EIGEN_BLAS_INT stride_27 = EIGEN_BLAS_INT(1) << 27; // stride_27 * sizeof(complex<double>) == 2^31. + std::complex<double> alpha(2.0, 0.0), beta(0.0, 0.0); + double ralpha = 2.0; + + std::complex<double> a[12] = {{1.0, 1.0}, {2.0, -1.0}, {3.0, 0.0}, {4.0, 2.0}, {1.0, 0.0}, {2.0, 0.0}, + {3.0, 0.0}, {4.0, 0.0}, {5.0, 0.0}, {6.0, 0.0}, {7.0, 0.0}, {8.0, 0.0}}; + std::complex<double> x[8] = {{1.0, 0.0}, {2.0, 0.0}, {1.0, 0.0}, {1.0, 0.0}, + {1.0, 0.0}, {1.0, 0.0}, {1.0, 0.0}, {1.0, 0.0}}; + std::complex<double> y[8] = {}; + + for (EIGEN_BLAS_INT lda : {stride_27, large_lda}) { + BLASFUNC(zgemv) + ("N", &two, &one, reinterpret_cast<const double*>(&alpha), reinterpret_cast<const double*>(a), &lda, + reinterpret_cast<const double*>(x), &large_lda, reinterpret_cast<const double*>(&beta), + reinterpret_cast<double*>(y), &one); + VERIFY_IS_EQUAL(y[0], std::complex<double>(2.0, 2.0)); + VERIFY_IS_EQUAL(y[1], std::complex<double>(4.0, -2.0)); + + BLASFUNC(zgemv) + ("C", &two, &one, reinterpret_cast<const double*>(&alpha), reinterpret_cast<const double*>(a), &lda, + reinterpret_cast<const double*>(x), &one, reinterpret_cast<const double*>(&beta), reinterpret_cast<double*>(y), + &large_lda); + VERIFY_IS_EQUAL(y[0], std::complex<double>(10.0, 2.0)); + + BLASFUNC(zgemv) + ("C", &eight, &one, reinterpret_cast<const double*>(&alpha), reinterpret_cast<const double*>(a), &lda, + reinterpret_cast<const double*>(x), &one, reinterpret_cast<const double*>(&beta), reinterpret_cast<double*>(y), + &large_lda); + VERIFY_IS_EQUAL(y[0], std::complex<double>(44.0, -2.0)); + } + + a[0] = {3.0, 0.0}; + BLASFUNC(zhemv) + (&uplo, &one, reinterpret_cast<const double*>(&alpha), reinterpret_cast<const double*>(a), &large_lda, + reinterpret_cast<const double*>(x), &large_lda, reinterpret_cast<const double*>(&beta), reinterpret_cast<double*>(y), + &large_lda); + VERIFY_IS_EQUAL(y[0], std::complex<double>(6.0, 0.0)); + + for (EIGEN_BLAS_INT k : {one, k8}) { + a[k] = {3.0, 0.0}; + y[0] = {0.0, 0.0}; + BLASFUNC(zhbmv) + (&uplo, &one, &k, reinterpret_cast<double*>(&alpha), reinterpret_cast<double*>(a), &large_lda, + reinterpret_cast<double*>(x), &large_lda, reinterpret_cast<double*>(&beta), reinterpret_cast<double*>(y), + &large_lda); + VERIFY_IS_EQUAL(y[0], std::complex<double>(6.0, 0.0)); + } + + a[0] = {0.0, 0.0}; + std::complex<double> zx(2.0, 1.0), zy(3.0, -1.0); + BLASFUNC(zher) + (&uplo, &one, &ralpha, reinterpret_cast<double*>(&zx), &large_lda, reinterpret_cast<double*>(a), &large_lda); + VERIFY_IS_EQUAL(a[0], std::complex<double>(10.0, 0.0)); + + BLASFUNC(zher2) + (&uplo, &one, reinterpret_cast<double*>(&alpha), reinterpret_cast<double*>(&zx), &large_lda, + reinterpret_cast<double*>(&zy), &large_lda, reinterpret_cast<double*>(a), &large_lda); + VERIFY_IS_EQUAL(a[0], std::complex<double>(30.0, 0.0)); + + a[0] = {0.0, 0.0}; + a[1] = {0.0, 0.0}; + BLASFUNC(zgeru) + (&two, &one, reinterpret_cast<double*>(&alpha), reinterpret_cast<double*>(x), &one, reinterpret_cast<double*>(&zy), + &large_lda, reinterpret_cast<double*>(a), &large_lda); + VERIFY_IS_EQUAL(a[0], std::complex<double>(6.0, -2.0)); + VERIFY_IS_EQUAL(a[1], std::complex<double>(12.0, -4.0)); + + BLASFUNC(zgerc) + (&two, &one, reinterpret_cast<double*>(&alpha), reinterpret_cast<double*>(x), &one, reinterpret_cast<double*>(&zy), + &large_lda, reinterpret_cast<double*>(a), &large_lda); + VERIFY_IS_EQUAL(a[0], std::complex<double>(12.0, 0.0)); + VERIFY_IS_EQUAL(a[1], std::complex<double>(24.0, 0.0)); +} + +} // namespace + +EIGEN_DECLARE_TEST(blas_large_stride) { + CALL_SUBTEST(test_level1_large_stride()); + CALL_SUBTEST(test_level2_real_large_stride()); + CALL_SUBTEST(test_level2_complex_large_stride()); +}
diff --git a/blas/testing/cblat1.f b/blas/testing/cblat1.f index 0637f9e..4e151cf 100644 --- a/blas/testing/cblat1.f +++ b/blas/testing/cblat1.f
@@ -35,8 +35,10 @@ *> \ingroup complex_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM CBLAT1 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/cblat2.f b/blas/testing/cblat2.f index 5b483d2..bd19a7f 100644 --- a/blas/testing/cblat2.f +++ b/blas/testing/cblat2.f
@@ -101,8 +101,10 @@ *> \ingroup complex_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM CBLAT2 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/cblat3.f b/blas/testing/cblat3.f index 2ea3880..9ce4ea4 100644 --- a/blas/testing/cblat3.f +++ b/blas/testing/cblat3.f
@@ -83,8 +83,10 @@ *> \ingroup complex_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM CBLAT3 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/dblat1.f b/blas/testing/dblat1.f index 693c817..db14191 100644 --- a/blas/testing/dblat1.f +++ b/blas/testing/dblat1.f
@@ -35,8 +35,10 @@ *> \ingroup double_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM DBLAT1 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/dblat2.f b/blas/testing/dblat2.f index 7783ae2..1e71eab 100644 --- a/blas/testing/dblat2.f +++ b/blas/testing/dblat2.f
@@ -100,8 +100,10 @@ *> \ingroup double_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM DBLAT2 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/dblat3.f b/blas/testing/dblat3.f index 62fc7fa..edada59 100644 --- a/blas/testing/dblat3.f +++ b/blas/testing/dblat3.f
@@ -80,8 +80,10 @@ *> \ingroup double_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM DBLAT3 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/sblat1.f b/blas/testing/sblat1.f index 5cc2cda..97097a7 100644 --- a/blas/testing/sblat1.f +++ b/blas/testing/sblat1.f
@@ -35,8 +35,10 @@ *> \ingroup single_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM SBLAT1 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/sblat2.f b/blas/testing/sblat2.f index 6b800d5..48c354d 100644 --- a/blas/testing/sblat2.f +++ b/blas/testing/sblat2.f
@@ -100,8 +100,10 @@ *> \ingroup single_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM SBLAT2 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/sblat3.f b/blas/testing/sblat3.f index 4033e39..f1d2d06 100644 --- a/blas/testing/sblat3.f +++ b/blas/testing/sblat3.f
@@ -80,8 +80,10 @@ *> \ingroup single_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM SBLAT3 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/zblat1.f b/blas/testing/zblat1.f index ed01519..20f9c0b 100644 --- a/blas/testing/zblat1.f +++ b/blas/testing/zblat1.f
@@ -35,8 +35,10 @@ *> \ingroup complex16_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM ZBLAT1 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/zblat2.f b/blas/testing/zblat2.f index 8dbdbee..6880e76 100644 --- a/blas/testing/zblat2.f +++ b/blas/testing/zblat2.f
@@ -101,8 +101,10 @@ *> \ingroup complex16_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM ZBLAT2 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/blas/testing/zblat3.f b/blas/testing/zblat3.f index 7ca0de1..a0923c8 100644 --- a/blas/testing/zblat3.f +++ b/blas/testing/zblat3.f
@@ -84,8 +84,10 @@ *> \ingroup complex16_blas_testing * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI PROGRAM ZBLAT3 * * -- Reference BLAS test routine (version 3.4.1) --
diff --git a/ci/build.linux.gitlab-ci.yml b/ci/build.linux.gitlab-ci.yml index bc62bfa..c5f5cb6 100644 --- a/ci/build.linux.gitlab-ci.yml +++ b/ci/build.linux.gitlab-ci.yml
@@ -278,8 +278,16 @@ EIGEN_CI_CXX_COMPILER: nvc++ EIGEN_CI_INSTALL: "" EIGEN_CI_CROSS_INSTALL: "" - # Override CMake's NVHPC -fast default at both compilation and linking. + # -tp=px: nvc++ otherwise targets the CPU it runs on, but the test runner, + # or the builder behind a compiler-cache hit, can have a different one; + # tests then die with SIGILL (nightly pipeline 2986047532). px, nvc++'s + # generic x86-64 target, enables AVX, which every x86-64 runner has, but + # not AVX2 or FMA. CMAKE_<LANG>_FLAGS reach every target, the BLAS library + # included, at compilation and linking. + # -Kieee overrides CMake's NVHPC -fast default at compilation and linking. EIGEN_CI_ADDITIONAL_ARGS: >- + -DCMAKE_C_FLAGS=-tp=px + -DCMAKE_CXX_FLAGS=-tp=px -DEIGEN_TEST_CUSTOM_CXX_FLAGS=-Kieee -DEIGEN_TEST_CUSTOM_LINKER_FLAGS=-Kieee # NVHPC's compiler frontend is very memory-hungry with template-heavy code. @@ -401,9 +409,9 @@ EIGEN_CI_CUDA_COMPUTE_ARCH: "75" EIGEN_CI_CUDA_CXX_FLAGS: "-Wno-unknown-cuda-version" -# ---- Latest CUDA + newest compilers, sm_89;sm_90 --------------------------- +# ---- CUDA 12.6 + newest compilers, sm_50;sm_53;sm_89;sm_90 ----------------- # These jobs run natively on the L4 at sm_89 and retain compile-only coverage -# for sm_90. +# for sm_50, sm_53, and sm_90 (CUDA 13 dropped sm_50..sm_70). # GCC-13, CUDA-12.6 (nvcc) # CUDA 12.6 nvcc only supports gcc up to 13. @@ -413,7 +421,7 @@ variables: EIGEN_CI_C_COMPILER: gcc-13 EIGEN_CI_CXX_COMPILER: g++-13 - EIGEN_CI_CUDA_COMPUTE_ARCH: "89;90" + EIGEN_CI_CUDA_COMPUTE_ARCH: "50;53;89;90" # Clang-19, CUDA-12.6 (clang as CUDA compiler) # Clang CUDA support lags behind nvcc; clang-19 supports up to CUDA 12.x. @@ -425,7 +433,7 @@ EIGEN_CI_C_COMPILER: clang-19 EIGEN_CI_CXX_COMPILER: clang++-19 EIGEN_CI_TEST_CUDA_CLANG: "on" - EIGEN_CI_CUDA_COMPUTE_ARCH: "89;90" + EIGEN_CI_CUDA_COMPUTE_ARCH: "50;53;89;90" # ---- CUDA 13 ---------------------------------------------------------------- # CUDA 13 dropped offline compilation for sm_70, relocated the CCCL headers and
diff --git a/ci/checkformat.gitlab-ci.yml b/ci/checkformat.gitlab-ci.yml index 03bae1e..99965b8 100644 --- a/ci/checkformat.gitlab-ci.yml +++ b/ci/checkformat.gitlab-ci.yml
@@ -14,6 +14,7 @@ - merge_requests allow_failure: exit_codes: [ 3 ] + before_script: !reference [.fetch:diff-base, script] script: - sh ci/lint/lint.sh ${CI_MERGE_REQUEST_DIFF_BASE_SHA} artifacts: @@ -36,6 +37,7 @@ timeout: 1h tags: - saas-linux-medium-amd64 + before_script: !reference [.fetch:diff-base, script] script: - mkdir -p .tidy-build - cmake -G Ninja -B .tidy-build
diff --git a/ci/common.gitlab-ci.yml b/ci/common.gitlab-ci.yml index d9448da..614ad09 100644 --- a/ci/common.gitlab-ci.yml +++ b/ci/common.gitlab-ci.yml
@@ -298,6 +298,19 @@ before_script: - . ci/scripts/common.windows.before_script.ps1 +######## Checkout ############################################################# +# Adds the merge request's diff base, the merge-base with the target branch, to +# the depth-1 checkout, for jobs that diff against it: +# before_script: !reference [.fetch:diff-base, script] +# Two shallow commits suffice for a two-dot `git diff BASE HEAD`; a three-dot +# diff would need the history between them. A failed fetch is left to the job: +# the lint checks then fail as advisory, and the selector widens to the full +# suite. +.fetch:diff-base: + script: + - git fetch --quiet --depth=1 origin "${CI_MERGE_REQUEST_DIFF_BASE_SHA}" + || echo "Could not fetch the diff base ${CI_MERGE_REQUEST_DIFF_BASE_SHA}." >&2 + ######## Artifacts ############################################################# # Kept out of the .common:* bases so that publishing is a choice a job makes # rather than something it inherits.
diff --git a/ci/scripts/run-clang-tidy.sh b/ci/scripts/run-clang-tidy.sh index 8f3bcd4..ad0f1ee 100755 --- a/ci/scripts/run-clang-tidy.sh +++ b/ci/scripts/run-clang-tidy.sh
@@ -56,6 +56,10 @@ fi done +# set -e does not see a failed git diff inside the process substitution below, +# so a missing base would otherwise pass as "No changed files to check." +git cat-file -e "${BASE_SHA}^{commit}" || { echo "Diff base ${BASE_SHA} is not in the checkout."; exit 1; } + # Get changed files (Added, Modified, Renamed) without losing whitespace in # repository paths. mapfile -d '' -t CHANGED_FILES < <(git diff --name-only -z --diff-filter=AMR "${BASE_SHA}" HEAD)
diff --git a/ci/scripts/test.buildsystem.script.sh b/ci/scripts/test.buildsystem.script.sh index dfcba1d..4aac9bc 100755 --- a/ci/scripts/test.buildsystem.script.sh +++ b/ci/scripts/test.buildsystem.script.sh
@@ -30,7 +30,6 @@ "-DCMAKE_CXX_COMPILER=${EIGEN_CI_CXX_COMPILER}" \ -DEIGEN_BUILD_TESTING=ON \ -DEIGEN_BUILD_DOC=OFF \ - -DEIGEN_BUILD_DEMOS=OFF \ -DEIGEN_BUILD_BLAS=OFF \ -DEIGEN_BUILD_LAPACK=OFF \ ${EIGEN_CI_ADDITIONAL_ARGS} "${rootdir}"
diff --git a/ci/scripts/test.linux.script.sh b/ci/scripts/test.linux.script.sh index 7a9d577..014bcf2 100755 --- a/ci/scripts/test.linux.script.sh +++ b/ci/scripts/test.linux.script.sh
@@ -38,6 +38,16 @@ cd ${EIGEN_CI_BUILDDIR} +# Verify GPU availability for GPU test suites. Never allow non-GPU runners +# to silently skip tests and report false-positive success. +if [[ "${EIGEN_CI_CTEST_LABEL}" == "gpu" || "${CI_JOB_NAME:-}" == *":cuda"* ]]; then + if ! command -v nvidia-smi >/dev/null 2>&1 || ! nvidia-smi >/dev/null 2>&1; then + echo "ERROR: GPU test suite requested, but no functional NVIDIA device or driver was detected by nvidia-smi." >&2 + echo "Refusing to report a false-positive pass without GPU hardware." >&2 + exit 1 + fi +fi + target="" if [[ ${EIGEN_CI_CTEST_REGEX} ]]; then target="-R ${EIGEN_CI_CTEST_REGEX}"
diff --git a/ci/scripts/test.windows.script.ps1 b/ci/scripts/test.windows.script.ps1 index 10cd043..3e618a2 100644 --- a/ci/scripts/test.windows.script.ps1 +++ b/ci/scripts/test.windows.script.ps1
@@ -32,6 +32,39 @@ cd $EIGEN_CI_BUILDDIR +# CTestTestfile.cmake and DartConfiguration.tcl name the build job's checkout by +# absolute path. The Windows host runs several runner registrations, each with +# its own checkout directory, so a test job that lands on another one would look +# for the test executables in a directory that does not exist -- or, when it +# does, run whatever build another job left there. Point them at this +# checkout; -ne and -replace ignore case, as Windows paths do. +$configured_root = Select-String -Path CMakeCache.txt -Pattern '^CMAKE_HOME_DIRECTORY:INTERNAL=(.+)$' | + Select-Object -First 1 | ForEach-Object { $_.Matches[0].Groups[1].Value } +$job_root = $rootdir.Path -replace '\\', '/' +if ($configured_root -and $configured_root -ne $job_root) { + Write-Host "Relocating the CTest files from $configured_root to $job_root" + $pattern = [regex]::Escape($configured_root) + $replacement = $job_root.Replace('$', '$$') + Get-ChildItem -Recurse -File -Include CTestTestfile.cmake, DartConfiguration.tcl | ForEach-Object { + $text = [System.IO.File]::ReadAllText($_.FullName) + [System.IO.File]::WriteAllText($_.FullName, ($text -replace $pattern, $replacement)) + } +} + +# Verify GPU availability for GPU test suites. Never allow non-GPU runners +# to silently skip tests and report false-positive success. +if ($EIGEN_CI_CTEST_LABEL -eq "gpu" -or $Env:CI_JOB_NAME -like "*:cuda*") { + $has_gpu = $false + if (Get-Command nvidia-smi -ErrorAction SilentlyContinue) { + & nvidia-smi > $null 2>&1 + if ($LASTEXITCODE -eq 0) { $has_gpu = $true } + } + if (-not $has_gpu) { + Write-Error "ERROR: GPU test suite requested, but no functional NVIDIA device or driver was detected by nvidia-smi." + exit 1 + } +} + # Determine number of processors for parallel tests. $NPROC=${Env:NUMBER_OF_PROCESSORS}
diff --git a/ci/select.gitlab-ci.yml b/ci/select.gitlab-ci.yml index 07c7724..86bc534 100644 --- a/ci/select.gitlab-ci.yml +++ b/ci/select.gitlab-ci.yml
@@ -11,10 +11,7 @@ image: ${EIGEN_CI_IMAGE_LINUX_LINT} needs: [] rules: !reference [.rules:libeigen:affected-tests, rules] - variables: - # The selector diffs against the merge-base, which a shallow clone may not - # contain. - GIT_DEPTH: 0 + before_script: !reference [.fetch:diff-base, script] script: - python3 scripts/test_affected_tests.py - python3 scripts/affected_tests.py
diff --git a/ci/tidy/Dockerfile b/ci/tidy/Dockerfile index e11e784..019e1cc 100644 --- a/ci/tidy/Dockerfile +++ b/ci/tidy/Dockerfile
@@ -33,7 +33,7 @@ # cpack and ctest. RUN apt-get update -y \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ - git cmake ninja-build clang-tidy clang python3 \ + git ca-certificates cmake ninja-build clang-tidy clang python3 \ && rm -rf /var/lib/apt/lists/* /var/cache/apt/* /var/log/* \ && find /usr/lib/llvm-18/bin -mindepth 1 \ ! -name clang ! -name clang++ ! -name clang-18 ! -name clang-tidy \ @@ -52,6 +52,7 @@ # something a tool needs. RUN clang-tidy --version | grep -q 'LLVM version 18\.1\.3' \ && git --version && cmake --version \ + && test -s /etc/ssl/certs/ca-certificates.crt \ && for f in /usr/bin/* /usr/lib/llvm-18/bin/* /usr/lib/git-core/*; do \ if ldd "$f" 2>/dev/null | grep -q 'not found'; then \ echo "missing shared library: $f"; exit 1; fi; \
diff --git a/cmake/EigenSmokeTestList.cmake b/cmake/EigenSmokeTestList.cmake index d21c252..0d65626 100644 --- a/cmake/EigenSmokeTestList.cmake +++ b/cmake/EigenSmokeTestList.cmake
@@ -152,6 +152,7 @@ triangular_1 tridiagonal_eigensolver_1 tridiagonal_eigensolver_3 + trig_reduction_fastmath type_alias lu_9 umeyama_3
diff --git a/cmake/EigenTestPartGroups.cmake b/cmake/EigenTestPartGroups.cmake index 2e30999..4cadedc 100644 --- a/cmake/EigenTestPartGroups.cmake +++ b/cmake/EigenTestPartGroups.cmake
@@ -36,7 +36,7 @@ set(ei_test_part_groups_complex_components "1-3") set(ei_test_part_groups_complex_qz "1-3" "4-6" "7-8" "9-10") set(ei_test_part_groups_concat "1-15") -set(ei_test_part_groups_condition_estimator "1-4" "5-6") +set(ei_test_part_groups_condition_estimator "1-2" "3-4" "5-6") set(ei_test_part_groups_conjugate_gradient "1-2" "3-6") set(ei_test_part_groups_conservative_resize "2-6") set(ei_test_part_groups_corners "2-5") @@ -126,7 +126,7 @@ set(ei_test_part_groups_sparse_ref "2-4") set(ei_test_part_groups_sparse_scatter "1-3") set(ei_test_part_groups_sparse_threaded_product "3-4") -set(ei_test_part_groups_sparselu "4-6" "7-10") +set(ei_test_part_groups_sparselu "5-6" "7-10") set(ei_test_part_groups_sparseqr "1-7") set(ei_test_part_groups_special_functions "2-5") set(ei_test_part_groups_special_members "1-2") @@ -156,7 +156,7 @@ set(ei_test_part_groups_tridiagonal_eigensolver "4-5") set(ei_test_part_groups_trsm_packet_cancellation "1-2") set(ei_test_part_groups_umeyama "1-2" "4-7") -set(ei_test_part_groups_upperbidiagonalization "1-3" "4-5" "6-7" "8-10" "11-13") +set(ei_test_part_groups_upperbidiagonalization "1-2" "4-5" "6-7" "8-10" "11-13") set(ei_test_part_groups_vectorwise_norm "1-9" "10-13") set(ei_test_part_groups_vectorwiseop "2-4" "5-8" "9-12") set(ei_test_part_groups_visitor "2-14")
diff --git a/cmake/EigenTesting.cmake b/cmake/EigenTesting.cmake index ae21d7e..74550fc 100644 --- a/cmake/EigenTesting.cmake +++ b/cmake/EigenTesting.cmake
@@ -531,9 +531,6 @@ set_property(GLOBAL PROPERTY EIGEN_FAILTEST_FAILURE_COUNT "0") set_property(GLOBAL PROPERTY EIGEN_FAILTEST_COUNT "0") - - # uncomment anytime you change the ei_get_compilerver_from_cxx_version_string macro - # ei_test_get_compilerver_from_cxx_version_string() endmacro() macro(ei_set_sitename) @@ -553,208 +550,127 @@ endif() endmacro() -macro(ei_get_compilerver VAR) - if (NOT CMAKE_CXX_COMPILER_ID) - set(CMAKE_CXX_COMPILER_ID "<unknown>") - endif() - if(MSVC) - set(${VAR} "${CMAKE_CXX_COMPILER_VERSION}") - elseif(${CMAKE_CXX_COMPILER_ID} MATCHES "PGI" OR ${CMAKE_CXX_COMPILER_ID} MATCHES "NVHPC") - set(${VAR} "${CMAKE_CXX_COMPILER_ID}-${CMAKE_CXX_COMPILER_VERSION}") - else() - # on all other system we rely on ${CMAKE_CXX_COMPILER} - # supporting a "--version" or "/version" flag - - if(WIN32 AND ${CMAKE_CXX_COMPILER_ID} EQUAL "Intel") - set(EIGEN_CXX_FLAG_VERSION "/version") - else() - set(EIGEN_CXX_FLAG_VERSION "--version") - endif() - - execute_process(COMMAND ${CMAKE_CXX_COMPILER} ${EIGEN_CXX_FLAG_VERSION} - OUTPUT_VARIABLE eigen_cxx_compiler_version_string OUTPUT_STRIP_TRAILING_WHITESPACE) - string(REGEX REPLACE "^[ \n\r]+" "" eigen_cxx_compiler_version_string ${eigen_cxx_compiler_version_string}) - string(REGEX REPLACE "[\n\r].*" "" eigen_cxx_compiler_version_string ${eigen_cxx_compiler_version_string}) - - ei_get_compilerver_from_cxx_version_string("${eigen_cxx_compiler_version_string}" CNAME CVER) - set(${VAR} "${CNAME}-${CVER}") - +function(ei_get_compilerver VAR) + set(compiler "${CMAKE_CXX_COMPILER_ID}") + if(compiler STREQUAL "") + set(compiler "unknown") endif() -endmacro() - -# Extract compiler name and version from a raw version string -# WARNING: if you edit this macro, then please test it by uncommenting -# the testing macro call in ei_init_testing() of the EigenTesting.cmake file. -# See also the ei_test_get_compilerver_from_cxx_version_string macro at the end -# of the file -macro(ei_get_compilerver_from_cxx_version_string VERSTRING CNAME CVER) - # extract possible compiler names - string(REGEX MATCH "g\\+\\+" ei_has_gpp ${VERSTRING}) - string(REGEX MATCH "llvm|LLVM" ei_has_llvm ${VERSTRING}) - string(REGEX MATCH "gcc|GCC" ei_has_gcc ${VERSTRING}) - string(REGEX MATCH "icpc|ICC" ei_has_icpc ${VERSTRING}) - string(REGEX MATCH "clang|CLANG" ei_has_clang ${VERSTRING}) - string(REGEX MATCH "mingw32" ei_has_mingw ${VERSTRING}) - - # combine them - if((ei_has_llvm) AND (ei_has_gpp OR ei_has_gcc)) - set(${CNAME} "llvm-g++") - elseif((ei_has_llvm) AND (ei_has_clang)) - set(${CNAME} "llvm-clang++") - elseif(ei_has_clang) - set(${CNAME} "clang++") - elseif ((ei_has_mingw) AND (ei_has_gpp OR ei_has_gcc)) - set(${CNAME} "mingw32-g++") - elseif(ei_has_icpc) - set(${CNAME} "icpc") - elseif(ei_has_gpp OR ei_has_gcc) - set(${CNAME} "g++") - else() - set(${CNAME} "_") + string(TOLOWER "${compiler}" compiler) + if(CMAKE_CXX_COMPILER_FRONTEND_VARIANT STREQUAL "MSVC" AND + NOT CMAKE_CXX_COMPILER_ID STREQUAL "MSVC") + string(APPEND compiler "-msvc") endif() - # extract possible version numbers - # first try to extract 3 isolated numbers: - string(REGEX MATCH " [0-9]+\\.[0-9]+\\.[0-9]+" eicver ${VERSTRING}) - if(NOT eicver) - # try to extract 2 isolated ones: - string(REGEX MATCH " [0-9]+\\.[0-9]+" eicver ${VERSTRING}) - if(NOT eicver) - # try to extract 3: - string(REGEX MATCH "[^0-9][0-9]+\\.[0-9]+\\.[0-9]+" eicver ${VERSTRING}) - if(NOT eicver) - # try to extract 2: - string(REGEX MATCH "[^0-9][0-9]+\\.[0-9]+" eicver ${VERSTRING}) - if (NOT eicver AND ei_has_mingw) - # try to extract 1 number plus suffix: - string(REGEX MATCH "[^0-9][0-9]+-win32" eicver ${VERSTRING}) - endif() + set(version "${CMAKE_CXX_COMPILER_VERSION}") + if(version STREQUAL "") + set(version "unknown") + endif() + set(${VAR} "${compiler}-${version}" PARENT_SCOPE) +endfunction() + +function(ei_get_cxxflags VAR) + set(flags "") + # Report requested test options, not an ISA inferred from pointer width or host. + foreach(feature IN ITEMS NEON NEON64 ZVECTOR Z14 VSX ALTIVEC MSA LSX SVE SME + SSE2 SSE3 SSSE3 SSE4_1 SSE4_2 AVX AVX2 AVX512 AVX512DQ AVX512FP16 FMA F16C) + if(EIGEN_TEST_${feature}) + if(feature STREQUAL "SVE" AND NOT "${EIGEN_TEST_SVE_VL}" STREQUAL "") + list(APPEND flags "SVE${EIGEN_TEST_SVE_VL}") + else() + list(APPEND flags "${feature}") endif() endif() + endforeach() + if(EIGEN_TEST_NO_EXPLICIT_VECTORIZATION) + list(APPEND flags NOVEC) endif() - - if (NOT eicver) - set(eicver " _") - endif() - - string(REGEX REPLACE ".(.*)" "\\1" ${CVER} ${eicver}) - -endmacro() - -macro(ei_get_cxxflags VAR) - set(${VAR} "") - ei_is_64bit_env(IS_64BIT_ENV) - if(EIGEN_TEST_NEON) - set(${VAR} NEON) - elseif(EIGEN_TEST_NEON64) - set(${VAR} NEON) - elseif(EIGEN_TEST_ZVECTOR) - set(${VAR} ZVECTOR) - elseif(EIGEN_TEST_VSX) - set(${VAR} VSX) - elseif(EIGEN_TEST_ALTIVEC) - set(${VAR} ALVEC) - elseif(EIGEN_TEST_FMA) - set(${VAR} FMA) - elseif(EIGEN_TEST_AVX) - set(${VAR} AVX) - elseif(EIGEN_TEST_SSE4_2) - set(${VAR} SSE42) - elseif(EIGEN_TEST_SSE4_1) - set(${VAR} SSE41) - elseif(EIGEN_TEST_SSSE3) - set(${VAR} SSSE3) - elseif(EIGEN_TEST_SSE3) - set(${VAR} SSE3) - elseif(EIGEN_TEST_SSE2 OR IS_64BIT_ENV) - set(${VAR} SSE2) - elseif(EIGEN_TEST_MSA) - set(${VAR} MSA) + if(NOT flags) + list(APPEND flags default) endif() if(EIGEN_TEST_OPENMP) - if (${VAR} STREQUAL "") - set(${VAR} OMP) - else() - set(${VAR} ${${VAR}}-OMP) + list(APPEND flags OMP) + endif() + if(EIGEN_DEFAULT_TO_ROW_MAJOR) + list(APPEND flags ROWMAJ) + endif() + list(JOIN flags "-" flags) + set(${VAR} "${flags}" PARENT_SCOPE) +endfunction() + +function(ei_get_target_architecture VAR) + # Apple universal builds and Visual Studio targets may differ from the host. + if(APPLE AND NOT "${CMAKE_OSX_ARCHITECTURES}" STREQUAL "") + set(architectures "${CMAKE_OSX_ARCHITECTURES}") + elseif(NOT "${CMAKE_CXX_COMPILER_TARGET}" STREQUAL "") + # Keep the full target: QCC selectors are not necessarily target triples. + set(architectures "${CMAKE_CXX_COMPILER_TARGET}") + elseif(NOT "${CMAKE_CXX_COMPILER_ARCHITECTURE_ID}" STREQUAL "") + set(architectures "${CMAKE_CXX_COMPILER_ARCHITECTURE_ID}") + elseif(NOT "${CMAKE_GENERATOR_PLATFORM}" STREQUAL "") + string(REGEX REPLACE ",.*$" "" architectures "${CMAKE_GENERATOR_PLATFORM}") + if(architectures STREQUAL "") + set(architectures "${CMAKE_SYSTEM_PROCESSOR}") endif() + else() + set(architectures "${CMAKE_SYSTEM_PROCESSOR}") endif() - if(EIGEN_DEFAULT_TO_ROW_MAJOR) - if (${VAR} STREQUAL "") - set(${VAR} ROW) - else() - set(${VAR} ${${VAR}}-ROWMAJ) + set(target_architectures "") + foreach(architecture IN LISTS architectures) + string(TOLOWER "${architecture}" architecture) + if(architecture MATCHES "^(amd64|x64|x86_64)$") + set(architecture x86_64) + elseif(architecture MATCHES "^(win32|x86|i[3-6]86)$") + set(architecture x86) + elseif(architecture MATCHES "^(arm64|aarch64)$") + set(architecture aarch64) endif() + # Match the GNU-only -m32 handling in the top-level test configuration. + if(EIGEN_TEST_32BIT AND CMAKE_COMPILER_IS_GNUCXX AND architecture STREQUAL "x86_64") + set(architecture x86) + endif() + list(APPEND target_architectures "${architecture}") + endforeach() + if(NOT target_architectures) + list(APPEND target_architectures unknown) endif() -endmacro() + list(REMOVE_DUPLICATES target_architectures) + list(SORT target_architectures) + list(JOIN target_architectures "+" architecture) + set(${VAR} "${architecture}" PARENT_SCOPE) +endfunction() macro(ei_set_build_string) ei_get_compilerver(LOCAL_COMPILER_VERSION) ei_get_cxxflags(LOCAL_COMPILER_FLAGS) + ei_get_target_architecture(LOCAL_TARGET_ARCHITECTURE) - set(TMP_BUILD_STRING ${CMAKE_SYSTEM}-${LOCAL_COMPILER_VERSION}) - - if (NOT ${LOCAL_COMPILER_FLAGS} STREQUAL "") - set(TMP_BUILD_STRING ${TMP_BUILD_STRING}-${LOCAL_COMPILER_FLAGS}) + set(TMP_BUILD_STRING "${CMAKE_SYSTEM}-${LOCAL_TARGET_ARCHITECTURE}-${LOCAL_COMPILER_VERSION}") + if(NOT "${LOCAL_COMPILER_FLAGS}" STREQUAL "") + string(APPEND TMP_BUILD_STRING "-${LOCAL_COMPILER_FLAGS}") endif() if(EIGEN_TEST_EXTERNAL_BLAS) - set(TMP_BUILD_STRING ${TMP_BUILD_STRING}-external_blas) + string(APPEND TMP_BUILD_STRING "-external_blas") endif() - ei_is_64bit_env(IS_64BIT_ENV) - if(NOT IS_64BIT_ENV) - set(TMP_BUILD_STRING ${TMP_BUILD_STRING}-32bit) + if(EIGEN_TEST_32BIT AND CMAKE_COMPILER_IS_GNUCXX) + string(APPEND TMP_BUILD_STRING "-32bit") + elseif("${CMAKE_SIZEOF_VOID_P}" MATCHES "^[1-9][0-9]*$") + math(EXPR LOCAL_POINTER_BITS "8 * ${CMAKE_SIZEOF_VOID_P}") + string(APPEND TMP_BUILD_STRING "-${LOCAL_POINTER_BITS}bit") else() - set(TMP_BUILD_STRING ${TMP_BUILD_STRING}-64bit) + string(APPEND TMP_BUILD_STRING "-unknown-bits") endif() if(EIGEN_BUILD_STRING_SUFFIX) - set(TMP_BUILD_STRING ${TMP_BUILD_STRING}-${EIGEN_BUILD_STRING_SUFFIX}) + string(APPEND TMP_BUILD_STRING "-${EIGEN_BUILD_STRING_SUFFIX}") endif() - - string(TOLOWER ${TMP_BUILD_STRING} BUILDNAME) -endmacro() - -macro(ei_is_64bit_env VAR) - if(CMAKE_SIZEOF_VOID_P EQUAL 8) - set(${VAR} 1) - elseif(CMAKE_SIZEOF_VOID_P EQUAL 4) - set(${VAR} 0) - else() - message(WARNING "Unsupported pointer size. Please contact the authors.") - endif() + string(TOLOWER "${TMP_BUILD_STRING}" BUILDNAME) endmacro() -# helper macro for testing ei_get_compilerver_from_cxx_version_string -# STR: raw version string -# REFNAME: expected compiler name -# REFVER: expected compiler version -macro(ei_test1_get_compilerver_from_cxx_version_string STR REFNAME REFVER) - ei_get_compilerver_from_cxx_version_string(${STR} CNAME CVER) - if((NOT ${REFNAME} STREQUAL ${CNAME}) OR (NOT ${REFVER} STREQUAL ${CVER})) - message("STATUS ei_get_compilerver_from_cxx_version_string error:") - message("Expected \"${REFNAME}-${REFVER}\", got \"${CNAME}-${CVER}\"") - endif() -endmacro() - -# macro for testing ei_get_compilerver_from_cxx_version_string -# feel free to add more version strings -macro(ei_test_get_compilerver_from_cxx_version_string) - ei_test1_get_compilerver_from_cxx_version_string("g++ (SUSE Linux) 4.5.3 20110428 [gcc-4_5-branch revision 173117]" "g++" "4.5.3") - ei_test1_get_compilerver_from_cxx_version_string("c++ (GCC) 4.5.1 20100924 (Red Hat 4.5.1-4)" "g++" "4.5.1") - ei_test1_get_compilerver_from_cxx_version_string("icpc (ICC) 11.0 20081105" "icpc" "11.0") - ei_test1_get_compilerver_from_cxx_version_string("g++-3.4 (GCC) 3.4.6" "g++" "3.4.6") - ei_test1_get_compilerver_from_cxx_version_string("SUSE Linux clang version 3.0 (branches/release_30 145598) (based on LLVM 3.0)" "llvm-clang++" "3.0") - ei_test1_get_compilerver_from_cxx_version_string("icpc (ICC) 12.0.5 20110719" "icpc" "12.0.5") - ei_test1_get_compilerver_from_cxx_version_string("Apple clang version 2.1 (tags/Apple/clang-163.7.1) (based on LLVM 3.0svn)" "llvm-clang++" "2.1") - ei_test1_get_compilerver_from_cxx_version_string("i686-apple-darwin11-llvm-g++-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)" "llvm-g++" "4.2.1") - ei_test1_get_compilerver_from_cxx_version_string("g++-mp-4.4 (GCC) 4.4.6" "g++" "4.4.6") - ei_test1_get_compilerver_from_cxx_version_string("g++-mp-4.4 (GCC) 2011" "g++" "4.4") - ei_test1_get_compilerver_from_cxx_version_string("x86_64-w64-mingw32-g++ (GCC) 10-win32 20210110" "mingw32-g++" "10-win32") -endmacro() # Split all tests listed in EIGEN_TESTS_LIST into num_splits many targets # named buildtestspartN with N = { 0, ..., num_splits-1}.
diff --git a/cmake/FindBLASEXT.cmake b/cmake/FindBLASEXT.cmake deleted file mode 100644 index 43d3951..0000000 --- a/cmake/FindBLASEXT.cmake +++ /dev/null
@@ -1,385 +0,0 @@ -### -# -# @copyright (c) 2009-2014 The University of Tennessee and The University -# of Tennessee Research Foundation. -# All rights reserved. -# @copyright (c) 2012-2016 Inria. All rights reserved. -# @copyright (c) 2012-2014 Bordeaux INP, CNRS (LaBRI UMR 5800), Inria, Univ. Bordeaux. All rights reserved. -# -### -# -# - Find BLAS EXTENDED for MORSE projects: find include dirs and libraries -# -# This module allows to find BLAS libraries by calling the official FindBLAS module -# and handles the creation of different library lists whether the user wishes to link -# with a sequential BLAS or a multihreaded (BLAS_SEQ_LIBRARIES and BLAS_PAR_LIBRARIES). -# BLAS is detected with a FindBLAS call then if the BLAS vendor is Intel10_64lp, ACML -# or IBMESSLMT then the module attempts to find the corresponding multithreaded libraries. -# -# The following variables have been added to manage links with sequential or multithreaded -# versions: -# BLAS_INCLUDE_DIRS - BLAS include directories -# BLAS_LIBRARY_DIRS - Link directories for BLAS libraries -# BLAS_SEQ_LIBRARIES - BLAS component libraries to be linked (sequential) -# BLAS_PAR_LIBRARIES - BLAS component libraries to be linked (multithreaded) -# SPDX-License-Identifier: BSD-3-Clause - -#============================================================================= -# Copyright 2012-2013 Inria -# Copyright 2012-2013 Emmanuel Agullo -# Copyright 2012-2013 Mathieu Faverge -# Copyright 2012 Cedric Castagnede -# Copyright 2013-2016 Florent Pruvost -# -# Distributed under the OSI-approved BSD License (the "License"); -# see accompanying file MORSE-Copyright.txt for details. -# -# This software is distributed WITHOUT ANY WARRANTY; without even the -# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -# See the License for more information. -#============================================================================= -# (To distribute this file outside of Morse, substitute the full -# License text for the above reference.) - -# macro to factorize this call -include(CMakeFindDependencyMacro) -macro(find_package_blas) - if(BLASEXT_FIND_REQUIRED) - if(BLASEXT_FIND_QUIETLY) - find_dependency(BLAS REQUIRED QUIET) - else() - find_dependency(BLAS REQUIRED) - endif() - else() - if(BLASEXT_FIND_QUIETLY) - find_dependency(BLAS QUIET) - else() - find_dependency(BLAS) - endif() - endif() -endmacro() - -# add a cache variable to let the user specify the BLAS vendor -set(BLA_VENDOR "" CACHE STRING "list of possible BLAS vendor: - Open, Eigen, Goto, ATLAS PhiPACK, CXML, DXML, SunPerf, SCSL, SGIMATH, IBMESSL, IBMESSLMT, - Intel10_32 (intel mkl v10 32 bit), - Intel10_64lp (intel mkl v10 64 bit, lp thread model, lp64 model), - Intel10_64lp_seq (intel mkl v10 64 bit, sequential code, lp64 model), - Intel( older versions of mkl 32 and 64 bit), - ACML, ACML_MP, ACML_GPU, Apple, NAS, Generic") - -if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "In FindBLASEXT") - message(STATUS "If you want to force the use of one specific library, " - "\n please specify the BLAS vendor by setting -DBLA_VENDOR=blas_vendor_name" - "\n at cmake configure.") - message(STATUS "List of possible BLAS vendor: Goto, ATLAS PhiPACK, CXML, " - "\n DXML, SunPerf, SCSL, SGIMATH, IBMESSL, IBMESSLMT, Intel10_32 (intel mkl v10 32 bit)," - "\n Intel10_64lp (intel mkl v10 64 bit, lp thread model, lp64 model)," - "\n Intel10_64lp_seq (intel mkl v10 64 bit, sequential code, lp64 model)," - "\n Intel( older versions of mkl 32 and 64 bit)," - "\n ACML, ACML_MP, ACML_GPU, Apple, NAS, Generic") -endif() - -if (NOT BLAS_FOUND) - # First try to detect two cases: - # 1: only SEQ libs are handled - # 2: both SEQ and PAR libs are handled - find_package_blas() -endif () - -# detect the cases where SEQ and PAR libs are handled -if(BLA_VENDOR STREQUAL "All" AND - (BLAS_mkl_core_LIBRARY OR BLAS_mkl_core_dll_LIBRARY) - ) - set(BLA_VENDOR "Intel") - if(BLAS_mkl_intel_LIBRARY) - set(BLA_VENDOR "Intel10_32") - endif() - if(BLAS_mkl_intel_lp64_LIBRARY) - set(BLA_VENDOR "Intel10_64lp") - endif() - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "A BLAS library has been found (${BLAS_LIBRARIES}) but we" - "\n have also potentially detected some multithreaded BLAS libraries from the MKL." - "\n We try to find both libraries lists (Sequential/Multithreaded).") - endif() - set(BLAS_FOUND "") -elseif(BLA_VENDOR STREQUAL "All" AND BLAS_acml_LIBRARY) - set(BLA_VENDOR "ACML") - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "A BLAS library has been found (${BLAS_LIBRARIES}) but we" - "\n have also potentially detected some multithreaded BLAS libraries from the ACML." - "\n We try to find both libraries lists (Sequential/Multithreaded).") - endif() - set(BLAS_FOUND "") -elseif(BLA_VENDOR STREQUAL "All" AND BLAS_essl_LIBRARY) - set(BLA_VENDOR "IBMESSL") - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "A BLAS library has been found (${BLAS_LIBRARIES}) but we" - "\n have also potentially detected some multithreaded BLAS libraries from the ESSL." - "\n We try to find both libraries lists (Sequential/Multithreaded).") - endif() - set(BLAS_FOUND "") -endif() - -# Intel case -if(BLA_VENDOR MATCHES "Intel*") - - ### - # look for include path if the BLAS vendor is Intel - ### - - # gather system include paths - unset(_inc_env) - if(WIN32) - string(REPLACE ":" ";" _inc_env "$ENV{INCLUDE}") - else() - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{C_INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{CPATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - endif() - list(APPEND _inc_env "${CMAKE_PLATFORM_IMPLICIT_INCLUDE_DIRECTORIES}") - list(APPEND _inc_env "${CMAKE_C_IMPLICIT_INCLUDE_DIRECTORIES}") - set(ENV_MKLROOT "$ENV{MKLROOT}") - if (ENV_MKLROOT) - list(APPEND _inc_env "${ENV_MKLROOT}/include") - endif() - list(REMOVE_DUPLICATES _inc_env) - - # find mkl.h inside known include paths - set(BLAS_mkl.h_INCLUDE_DIRS "BLAS_mkl.h_INCLUDE_DIRS-NOTFOUND") - if(BLAS_INCDIR) - set(BLAS_mkl.h_INCLUDE_DIRS "BLAS_mkl.h_INCLUDE_DIRS-NOTFOUND") - find_path(BLAS_mkl.h_INCLUDE_DIRS - NAMES mkl.h - HINTS ${BLAS_INCDIR}) - else() - if(BLAS_DIR) - set(BLAS_mkl.h_INCLUDE_DIRS "BLAS_mkl.h_INCLUDE_DIRS-NOTFOUND") - find_path(BLAS_mkl.h_INCLUDE_DIRS - NAMES mkl.h - HINTS ${BLAS_DIR} - PATH_SUFFIXES include) - else() - set(BLAS_mkl.h_INCLUDE_DIRS "BLAS_mkl.h_INCLUDE_DIRS-NOTFOUND") - find_path(BLAS_mkl.h_INCLUDE_DIRS - NAMES mkl.h - HINTS ${_inc_env}) - endif() - endif() - mark_as_advanced(BLAS_mkl.h_INCLUDE_DIRS) - ## Print status if not found - ## ------------------------- - #if (NOT BLAS_mkl.h_INCLUDE_DIRS AND MORSE_VERBOSE) - # Print_Find_Header_Status(blas mkl.h) - #endif () - set(BLAS_INCLUDE_DIRS "") - if(BLAS_mkl.h_INCLUDE_DIRS) - list(APPEND BLAS_INCLUDE_DIRS "${BLAS_mkl.h_INCLUDE_DIRS}" ) - endif() - - ### - # look for libs - ### - # if Intel 10 64 bit -> look for sequential and multithreaded versions - if(BLA_VENDOR MATCHES "Intel10_64lp*") - - ## look for the sequential version - set(BLA_VENDOR "Intel10_64lp_seq") - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "Look for the sequential version Intel10_64lp_seq") - endif() - find_package_blas() - if(BLAS_FOUND) - set(BLAS_SEQ_LIBRARIES "${BLAS_LIBRARIES}") - else() - set(BLAS_SEQ_LIBRARIES "${BLAS_SEQ_LIBRARIES-NOTFOUND}") - endif() - - ## look for the multithreaded version - set(BLA_VENDOR "Intel10_64lp") - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "Look for the multithreaded version Intel10_64lp") - endif() - find_package_blas() - if(BLAS_FOUND) - set(BLAS_PAR_LIBRARIES "${BLAS_LIBRARIES}") - else() - set(BLAS_PAR_LIBRARIES "${BLAS_PAR_LIBRARIES-NOTFOUND}") - endif() - - else() - - if(BLAS_FOUND) - set(BLAS_SEQ_LIBRARIES "${BLAS_LIBRARIES}") - else() - set(BLAS_SEQ_LIBRARIES "${BLAS_SEQ_LIBRARIES-NOTFOUND}") - endif() - - endif() - - # ACML case -elseif(BLA_VENDOR MATCHES "ACML*") - - ## look for the sequential version - set(BLA_VENDOR "ACML") - find_package_blas() - if(BLAS_FOUND) - set(BLAS_SEQ_LIBRARIES "${BLAS_LIBRARIES}") - else() - set(BLAS_SEQ_LIBRARIES "${BLAS_SEQ_LIBRARIES-NOTFOUND}") - endif() - - ## look for the multithreaded version - set(BLA_VENDOR "ACML_MP") - find_package_blas() - if(BLAS_FOUND) - set(BLAS_PAR_LIBRARIES "${BLAS_LIBRARIES}") - else() - set(BLAS_PAR_LIBRARIES "${BLAS_PAR_LIBRARIES-NOTFOUND}") - endif() - - # IBMESSL case -elseif(BLA_VENDOR MATCHES "IBMESSL*") - - ## look for the sequential version - set(BLA_VENDOR "IBMESSL") - find_package_blas() - if(BLAS_FOUND) - set(BLAS_SEQ_LIBRARIES "${BLAS_LIBRARIES}") - else() - set(BLAS_SEQ_LIBRARIES "${BLAS_SEQ_LIBRARIES-NOTFOUND}") - endif() - - ## look for the multithreaded version - set(BLA_VENDOR "IBMESSLMT") - find_package_blas() - if(BLAS_FOUND) - set(BLAS_PAR_LIBRARIES "${BLAS_LIBRARIES}") - else() - set(BLAS_PAR_LIBRARIES "${BLAS_PAR_LIBRARIES-NOTFOUND}") - endif() - -else() - - if(BLAS_FOUND) - # define the SEQ libs as the BLAS_LIBRARIES - set(BLAS_SEQ_LIBRARIES "${BLAS_LIBRARIES}") - else() - set(BLAS_SEQ_LIBRARIES "${BLAS_SEQ_LIBRARIES-NOTFOUND}") - endif() - set(BLAS_PAR_LIBRARIES "${BLAS_PAR_LIBRARIES-NOTFOUND}") - -endif() - - -if(BLAS_SEQ_LIBRARIES) - set(BLAS_LIBRARIES "${BLAS_SEQ_LIBRARIES}") -endif() - -# extract libs paths -# remark: because it is not given by find_package(BLAS) -set(BLAS_LIBRARY_DIRS "") -string(REPLACE " " ";" BLAS_LIBRARIES "${BLAS_LIBRARIES}") -foreach(blas_lib ${BLAS_LIBRARIES}) - if (EXISTS "${blas_lib}") - get_filename_component(a_blas_lib_dir "${blas_lib}" PATH) - list(APPEND BLAS_LIBRARY_DIRS "${a_blas_lib_dir}" ) - else() - string(REPLACE "-L" "" blas_lib "${blas_lib}") - if (EXISTS "${blas_lib}") - list(APPEND BLAS_LIBRARY_DIRS "${blas_lib}" ) - else() - get_filename_component(a_blas_lib_dir "${blas_lib}" PATH) - if (EXISTS "${a_blas_lib_dir}") - list(APPEND BLAS_LIBRARY_DIRS "${a_blas_lib_dir}" ) - endif() - endif() - endif() -endforeach() -if (BLAS_LIBRARY_DIRS) - list(REMOVE_DUPLICATES BLAS_LIBRARY_DIRS) -endif () - -# check that BLAS has been found -# --------------------------------- -include(FindPackageHandleStandardArgs) -if(BLA_VENDOR MATCHES "Intel*") - if(BLA_VENDOR MATCHES "Intel10_64lp*") - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "BLAS found is Intel MKL:" - "\n we manage two lists of libs, one sequential and one parallel if found" - "\n (see BLAS_SEQ_LIBRARIES and BLAS_PAR_LIBRARIES)") - message(STATUS "BLAS sequential libraries stored in BLAS_SEQ_LIBRARIES") - endif() - find_package_handle_standard_args(BLASEXT DEFAULT_MSG - BLAS_SEQ_LIBRARIES - BLAS_LIBRARY_DIRS - BLAS_INCLUDE_DIRS) - if(BLAS_PAR_LIBRARIES) - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "BLAS parallel libraries stored in BLAS_PAR_LIBRARIES") - endif() - find_package_handle_standard_args(BLASEXT DEFAULT_MSG - BLAS_PAR_LIBRARIES) - endif() - else() - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "BLAS sequential libraries stored in BLAS_SEQ_LIBRARIES") - endif() - find_package_handle_standard_args(BLASEXT DEFAULT_MSG - BLAS_SEQ_LIBRARIES - BLAS_LIBRARY_DIRS - BLAS_INCLUDE_DIRS) - endif() -elseif(BLA_VENDOR MATCHES "ACML*") - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "BLAS found is ACML:" - "\n we manage two lists of libs, one sequential and one parallel if found" - "\n (see BLAS_SEQ_LIBRARIES and BLAS_PAR_LIBRARIES)") - message(STATUS "BLAS sequential libraries stored in BLAS_SEQ_LIBRARIES") - endif() - find_package_handle_standard_args(BLASEXT DEFAULT_MSG - BLAS_SEQ_LIBRARIES - BLAS_LIBRARY_DIRS) - if(BLAS_PAR_LIBRARIES) - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "BLAS parallel libraries stored in BLAS_PAR_LIBRARIES") - endif() - find_package_handle_standard_args(BLASEXT DEFAULT_MSG - BLAS_PAR_LIBRARIES) - endif() -elseif(BLA_VENDOR MATCHES "IBMESSL*") - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "BLAS found is ESSL:" - "\n we manage two lists of libs, one sequential and one parallel if found" - "\n (see BLAS_SEQ_LIBRARIES and BLAS_PAR_LIBRARIES)") - message(STATUS "BLAS sequential libraries stored in BLAS_SEQ_LIBRARIES") - endif() - find_package_handle_standard_args(BLASEXT DEFAULT_MSG - BLAS_SEQ_LIBRARIES - BLAS_LIBRARY_DIRS) - if(BLAS_PAR_LIBRARIES) - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "BLAS parallel libraries stored in BLAS_PAR_LIBRARIES") - endif() - find_package_handle_standard_args(BLASEXT DEFAULT_MSG - BLAS_PAR_LIBRARIES) - endif() -else() - if(NOT BLASEXT_FIND_QUIETLY) - message(STATUS "BLAS sequential libraries stored in BLAS_SEQ_LIBRARIES") - endif() - find_package_handle_standard_args(BLASEXT DEFAULT_MSG - BLAS_SEQ_LIBRARIES - BLAS_LIBRARY_DIRS) -endif() - -# Callers expect BLAS_FOUND to be set as well. -set(BLAS_FOUND BLASEXT_FOUND)
diff --git a/cmake/FindCLANG_FORMAT.cmake b/cmake/FindCLANG_FORMAT.cmake deleted file mode 100644 index 625e675..0000000 --- a/cmake/FindCLANG_FORMAT.cmake +++ /dev/null
@@ -1,62 +0,0 @@ -# SPDX-License-Identifier: MPL-2.0 - - -# Find clang-format -# -# CLANG_FORMAT_EXECUTABLE - Path to clang-format executable -# CLANG_FORMAT_FOUND - True if the clang-format executable was found. -# CLANG_FORMAT_VERSION - The version of clang-format found -# -# Copyright 2009-2020 The VOTCA Development Team (http://www.votca.org) -# -# Licensed under the Mozilla Public License Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# https://www.mozilla.org/en-US/MPL/2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -find_program(CLANG_FORMAT_EXECUTABLE - NAMES - clang-format-9 - clang-format - clang-format-11 - clang-format-10 - clang-format-8 - clang-format-7 - - DOC "clang-format executable") -mark_as_advanced(CLANG_FORMAT_EXECUTABLE) - -# Extract version from command "clang-format -version" -if(CLANG_FORMAT_EXECUTABLE) - execute_process(COMMAND ${CLANG_FORMAT_EXECUTABLE} -version - OUTPUT_VARIABLE clang_format_version - ERROR_QUIET OUTPUT_STRIP_TRAILING_WHITESPACE) - - if(clang_format_version MATCHES "^.*clang-format version .*") - # clang_format_version sample: "clang-format version 3.9.1-4ubuntu3~16.04.1 - # (tags/RELEASE_391/rc2)" - string(REGEX - REPLACE "^.*clang-format version ([.0-9]+).*" - "\\1" - CLANG_FORMAT_VERSION - "${clang_format_version}") - # CLANG_FORMAT_VERSION sample: "3.9.1" - else() - set(CLANG_FORMAT_VERSION 0.0) - endif() -else() - set(CLANG_FORMAT_VERSION 0.0) -endif() - -include(FindPackageHandleStandardArgs) -# handle the QUIETLY and REQUIRED arguments and set CLANG_FORMAT_FOUND to TRUE -# if all listed variables are TRUE -find_package_handle_standard_args(CLANG_FORMAT REQUIRED_VARS CLANG_FORMAT_EXECUTABLE VERSION_VAR CLANG_FORMAT_VERSION)
diff --git a/cmake/FindHWLOC.cmake b/cmake/FindHWLOC.cmake deleted file mode 100644 index 37f6df7..0000000 --- a/cmake/FindHWLOC.cmake +++ /dev/null
@@ -1,333 +0,0 @@ -### -# -# @copyright (c) 2009-2014 The University of Tennessee and The University -# of Tennessee Research Foundation. -# All rights reserved. -# @copyright (c) 2012-2014 Inria. All rights reserved. -# @copyright (c) 2012-2014 Bordeaux INP, CNRS (LaBRI UMR 5800), Inria, Univ. Bordeaux. All rights reserved. -# -### -# -# - Find HWLOC include dirs and libraries -# Use this module by invoking find_package with the form: -# find_package(HWLOC -# [REQUIRED]) # Fail with error if hwloc is not found -# -# This module finds headers and hwloc library. -# Results are reported in variables: -# HWLOC_FOUND - True if headers and requested libraries were found -# HWLOC_INCLUDE_DIRS - hwloc include directories -# HWLOC_LIBRARY_DIRS - Link directories for hwloc libraries -# HWLOC_LIBRARIES - hwloc component libraries to be linked -# -# The user can give specific paths where to find the libraries adding cmake -# options at configure (ex: cmake path/to/project -DHWLOC_DIR=path/to/hwloc): -# HWLOC_DIR - Where to find the base directory of hwloc -# HWLOC_INCDIR - Where to find the header files -# HWLOC_LIBDIR - Where to find the library files -# The module can also look for the following environment variables if paths -# are not given as cmake variable: HWLOC_DIR, HWLOC_INCDIR, HWLOC_LIBDIR -# SPDX-License-Identifier: BSD-3-Clause - -#============================================================================= -# Copyright 2012-2013 Inria -# Copyright 2012-2013 Emmanuel Agullo -# Copyright 2012-2013 Mathieu Faverge -# Copyright 2012 Cedric Castagnede -# Copyright 2013 Florent Pruvost -# -# Distributed under the OSI-approved BSD License (the "License"); -# see accompanying file MORSE-Copyright.txt for details. -# -# This software is distributed WITHOUT ANY WARRANTY; without even the -# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -# See the License for more information. -#============================================================================= -# (To distribute this file outside of Morse, substitute the full -# License text for the above reference.) - -include(CheckStructHasMember) -include(CheckCSourceCompiles) - -if (NOT HWLOC_FOUND) - set(HWLOC_DIR "" CACHE PATH "Installation directory of HWLOC library") - if (NOT HWLOC_FIND_QUIETLY) - message(STATUS "A cache variable, namely HWLOC_DIR, has been set to specify the install directory of HWLOC") - endif() -endif() - -set(ENV_HWLOC_DIR "$ENV{HWLOC_DIR}") -set(ENV_HWLOC_INCDIR "$ENV{HWLOC_INCDIR}") -set(ENV_HWLOC_LIBDIR "$ENV{HWLOC_LIBDIR}") -set(HWLOC_GIVEN_BY_USER "FALSE") -if ( HWLOC_DIR OR ( HWLOC_INCDIR AND HWLOC_LIBDIR) OR ENV_HWLOC_DIR OR (ENV_HWLOC_INCDIR AND ENV_HWLOC_LIBDIR) ) - set(HWLOC_GIVEN_BY_USER "TRUE") -endif() - -# Optionally use pkg-config to detect include/library dirs (if pkg-config is available) -# ------------------------------------------------------------------------------------- -include(CMakeFindDependencyMacro) -# include(FindPkgConfig) -find_dependency(PkgConfig QUIET) -if( PKG_CONFIG_EXECUTABLE AND NOT HWLOC_GIVEN_BY_USER ) - - pkg_search_module(HWLOC hwloc) - if (NOT HWLOC_FIND_QUIETLY) - if (HWLOC_FOUND AND HWLOC_LIBRARIES) - message(STATUS "Looking for HWLOC - found using PkgConfig") - #if(NOT HWLOC_INCLUDE_DIRS) - # message("${Magenta}HWLOC_INCLUDE_DIRS is empty using PkgConfig." - # "Perhaps the path to hwloc headers is already present in your" - # "C(PLUS)_INCLUDE_PATH environment variable.${ColourReset}") - #endif() - else() - message(STATUS "${Magenta}Looking for HWLOC - not found using PkgConfig." - "\n Perhaps you should add the directory containing hwloc.pc to" - "\n the PKG_CONFIG_PATH environment variable.${ColourReset}") - endif() - endif() - -endif() - -if( (NOT PKG_CONFIG_EXECUTABLE) OR (PKG_CONFIG_EXECUTABLE AND NOT HWLOC_FOUND) OR (HWLOC_GIVEN_BY_USER) ) - - if (NOT HWLOC_FIND_QUIETLY) - message(STATUS "Looking for HWLOC - PkgConfig not used") - endif() - - # Looking for include - # ------------------- - - # Add system include paths to search include - # ------------------------------------------ - unset(_inc_env) - if(ENV_HWLOC_INCDIR) - list(APPEND _inc_env "${ENV_HWLOC_INCDIR}") - elseif(ENV_HWLOC_DIR) - list(APPEND _inc_env "${ENV_HWLOC_DIR}") - list(APPEND _inc_env "${ENV_HWLOC_DIR}/include") - list(APPEND _inc_env "${ENV_HWLOC_DIR}/include/hwloc") - else() - if(WIN32) - string(REPLACE ":" ";" _inc_env "$ENV{INCLUDE}") - else() - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{C_INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{CPATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - endif() - endif() - list(APPEND _inc_env "${CMAKE_PLATFORM_IMPLICIT_INCLUDE_DIRECTORIES}") - list(APPEND _inc_env "${CMAKE_C_IMPLICIT_INCLUDE_DIRECTORIES}") - list(REMOVE_DUPLICATES _inc_env) - - # set paths where to look for - set(PATH_TO_LOOK_FOR "${_inc_env}") - - # Try to find the hwloc header in the given paths - # ------------------------------------------------- - # call cmake macro to find the header path - if(HWLOC_INCDIR) - set(HWLOC_hwloc.h_DIRS "HWLOC_hwloc.h_DIRS-NOTFOUND") - find_path(HWLOC_hwloc.h_DIRS - NAMES hwloc.h - HINTS ${HWLOC_INCDIR}) - else() - if(HWLOC_DIR) - set(HWLOC_hwloc.h_DIRS "HWLOC_hwloc.h_DIRS-NOTFOUND") - find_path(HWLOC_hwloc.h_DIRS - NAMES hwloc.h - HINTS ${HWLOC_DIR} - PATH_SUFFIXES "include" "include/hwloc") - else() - set(HWLOC_hwloc.h_DIRS "HWLOC_hwloc.h_DIRS-NOTFOUND") - find_path(HWLOC_hwloc.h_DIRS - NAMES hwloc.h - HINTS ${PATH_TO_LOOK_FOR} - PATH_SUFFIXES "hwloc") - endif() - endif() - mark_as_advanced(HWLOC_hwloc.h_DIRS) - - # Add path to cmake variable - # ------------------------------------ - if (HWLOC_hwloc.h_DIRS) - set(HWLOC_INCLUDE_DIRS "${HWLOC_hwloc.h_DIRS}") - else () - set(HWLOC_INCLUDE_DIRS "HWLOC_INCLUDE_DIRS-NOTFOUND") - if(NOT HWLOC_FIND_QUIETLY) - message(STATUS "Looking for hwloc -- hwloc.h not found") - endif() - endif () - - if (HWLOC_INCLUDE_DIRS) - list(REMOVE_DUPLICATES HWLOC_INCLUDE_DIRS) - endif () - - - # Looking for lib - # --------------- - - # Add system library paths to search lib - # -------------------------------------- - unset(_lib_env) - if(ENV_HWLOC_LIBDIR) - list(APPEND _lib_env "${ENV_HWLOC_LIBDIR}") - elseif(ENV_HWLOC_DIR) - list(APPEND _lib_env "${ENV_HWLOC_DIR}") - list(APPEND _lib_env "${ENV_HWLOC_DIR}/lib") - else() - if(WIN32) - string(REPLACE ":" ";" _lib_env "$ENV{LIB}") - else() - if(APPLE) - string(REPLACE ":" ";" _lib_env "$ENV{DYLD_LIBRARY_PATH}") - else() - string(REPLACE ":" ";" _lib_env "$ENV{LD_LIBRARY_PATH}") - endif() - list(APPEND _lib_env "${CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES}") - list(APPEND _lib_env "${CMAKE_C_IMPLICIT_LINK_DIRECTORIES}") - endif() - endif() - list(REMOVE_DUPLICATES _lib_env) - - # set paths where to look for - set(PATH_TO_LOOK_FOR "${_lib_env}") - - # Try to find the hwloc lib in the given paths - # ---------------------------------------------- - - # call cmake macro to find the lib path - if(HWLOC_LIBDIR) - set(HWLOC_hwloc_LIBRARY "HWLOC_hwloc_LIBRARY-NOTFOUND") - find_library(HWLOC_hwloc_LIBRARY - NAMES hwloc - HINTS ${HWLOC_LIBDIR}) - else() - if(HWLOC_DIR) - set(HWLOC_hwloc_LIBRARY "HWLOC_hwloc_LIBRARY-NOTFOUND") - find_library(HWLOC_hwloc_LIBRARY - NAMES hwloc - HINTS ${HWLOC_DIR} - PATH_SUFFIXES lib lib32 lib64) - else() - set(HWLOC_hwloc_LIBRARY "HWLOC_hwloc_LIBRARY-NOTFOUND") - find_library(HWLOC_hwloc_LIBRARY - NAMES hwloc - HINTS ${PATH_TO_LOOK_FOR}) - endif() - endif() - mark_as_advanced(HWLOC_hwloc_LIBRARY) - - # If found, add path to cmake variable - # ------------------------------------ - if (HWLOC_hwloc_LIBRARY) - get_filename_component(hwloc_lib_path ${HWLOC_hwloc_LIBRARY} PATH) - # set cmake variables (respects naming convention) - set(HWLOC_LIBRARIES "${HWLOC_hwloc_LIBRARY}") - set(HWLOC_LIBRARY_DIRS "${hwloc_lib_path}") - else () - set(HWLOC_LIBRARIES "HWLOC_LIBRARIES-NOTFOUND") - set(HWLOC_LIBRARY_DIRS "HWLOC_LIBRARY_DIRS-NOTFOUND") - if(NOT HWLOC_FIND_QUIETLY) - message(STATUS "Looking for hwloc -- lib hwloc not found") - endif() - endif () - - if (HWLOC_LIBRARY_DIRS) - list(REMOVE_DUPLICATES HWLOC_LIBRARY_DIRS) - endif () - - # check a function to validate the find - if(HWLOC_LIBRARIES) - - set(REQUIRED_INCDIRS) - set(REQUIRED_LIBDIRS) - set(REQUIRED_LIBS) - - # HWLOC - if (HWLOC_INCLUDE_DIRS) - set(REQUIRED_INCDIRS "${HWLOC_INCLUDE_DIRS}") - endif() - if (HWLOC_LIBRARY_DIRS) - set(REQUIRED_LIBDIRS "${HWLOC_LIBRARY_DIRS}") - endif() - set(REQUIRED_LIBS "${HWLOC_LIBRARIES}") - - # set required libraries for link - set(CMAKE_REQUIRED_INCLUDES "${REQUIRED_INCDIRS}") - set(CMAKE_REQUIRED_LIBRARIES) - foreach(lib_dir ${REQUIRED_LIBDIRS}) - list(APPEND CMAKE_REQUIRED_LIBRARIES "-L${lib_dir}") - endforeach() - list(APPEND CMAKE_REQUIRED_LIBRARIES "${REQUIRED_LIBS}") - string(REGEX REPLACE "^ -" "-" CMAKE_REQUIRED_LIBRARIES "${CMAKE_REQUIRED_LIBRARIES}") - - # test link - unset(HWLOC_WORKS CACHE) - include(CheckFunctionExists) - check_function_exists(hwloc_topology_init HWLOC_WORKS) - mark_as_advanced(HWLOC_WORKS) - - if(NOT HWLOC_WORKS) - if(NOT HWLOC_FIND_QUIETLY) - message(STATUS "Looking for hwloc : test of hwloc_topology_init with hwloc library fails") - message(STATUS "CMAKE_REQUIRED_LIBRARIES: ${CMAKE_REQUIRED_LIBRARIES}") - message(STATUS "CMAKE_REQUIRED_INCLUDES: ${CMAKE_REQUIRED_INCLUDES}") - message(STATUS "Check in CMakeFiles/CMakeError.log to figure out why it fails") - endif() - endif() - set(CMAKE_REQUIRED_INCLUDES) - set(CMAKE_REQUIRED_FLAGS) - set(CMAKE_REQUIRED_LIBRARIES) - endif() - -endif() - -if (HWLOC_LIBRARIES) - if (HWLOC_LIBRARY_DIRS) - list(GET HWLOC_LIBRARY_DIRS 0 first_lib_path) - else() - list(GET HWLOC_LIBRARIES 0 first_lib) - get_filename_component(first_lib_path "${first_lib}" PATH) - endif() - if (${first_lib_path} MATCHES "/lib(32|64)?$") - string(REGEX REPLACE "/lib(32|64)?$" "" not_cached_dir "${first_lib_path}") - set(HWLOC_DIR_FOUND "${not_cached_dir}" CACHE PATH "Installation directory of HWLOC library" FORCE) - else() - set(HWLOC_DIR_FOUND "${first_lib_path}" CACHE PATH "Installation directory of HWLOC library" FORCE) - endif() -endif() -mark_as_advanced(HWLOC_DIR) -mark_as_advanced(HWLOC_DIR_FOUND) - -# check that HWLOC has been found -# ------------------------------- -include(FindPackageHandleStandardArgs) -if (PKG_CONFIG_EXECUTABLE AND HWLOC_FOUND) - find_package_handle_standard_args(HWLOC DEFAULT_MSG - HWLOC_LIBRARIES) -else() - find_package_handle_standard_args(HWLOC DEFAULT_MSG - HWLOC_LIBRARIES - HWLOC_WORKS) -endif() - -if (HWLOC_FOUND) - set(HWLOC_SAVE_CMAKE_REQUIRED_INCLUDES ${CMAKE_REQUIRED_INCLUDES}) - list(APPEND CMAKE_REQUIRED_INCLUDES ${HWLOC_INCLUDE_DIRS}) - - # test headers to guess the version - check_struct_has_member( "struct hwloc_obj" parent hwloc.h HAVE_HWLOC_PARENT_MEMBER ) - check_struct_has_member( "struct hwloc_cache_attr_s" size hwloc.h HAVE_HWLOC_CACHE_ATTR ) - check_c_source_compiles( "#include <hwloc.h> - int main(void) { hwloc_obj_t o; o->type = HWLOC_OBJ_PU; return 0;}" HAVE_HWLOC_OBJ_PU) - include(CheckLibraryExists) - check_library_exists(${HWLOC_LIBRARIES} hwloc_bitmap_free "" HAVE_HWLOC_BITMAP) - - set(CMAKE_REQUIRED_INCLUDES ${HWLOC_SAVE_CMAKE_REQUIRED_INCLUDES}) -endif()
diff --git a/cmake/FindMETIS.cmake b/cmake/FindMETIS.cmake new file mode 100644 index 0000000..17cc4ac --- /dev/null +++ b/cmake/FindMETIS.cmake
@@ -0,0 +1,139 @@ +### +# +# @copyright (c) 2009-2014 The University of Tennessee and The University +# of Tennessee Research Foundation. +# All rights reserved. +# @copyright (c) 2012-2014 Inria. All rights reserved. +# @copyright (c) 2012-2014 Bordeaux INP, CNRS (LaBRI UMR 5800), Inria, Univ. Bordeaux. All rights reserved. +# +### +# +# - Find METIS include dirs and libraries +# Use this module by invoking find_package with the form: +# find_package(METIS +# [REQUIRED] # Fail with error if metis is not found +# ) +# +# This module finds headers and metis library. +# Results are reported in variables: +# METIS_FOUND - True if headers and requested libraries were found +# METIS_INCLUDE_DIRS - metis include directories +# METIS_LIBRARY_DIRS - Link directories for metis libraries +# METIS_LIBRARIES - metis component libraries to be linked +# +# The user can give specific paths where to find the libraries adding cmake +# options at configure (ex: cmake path/to/project -DMETIS_DIR=path/to/metis): +# METIS_DIR - Where to find the base directory of metis +# METIS_INCDIR - Where to find the header files +# METIS_LIBDIR - Where to find the library files +# The module can also look for the following environment variables if paths +# are not given as cmake variable: METIS_DIR, METIS_INCDIR, METIS_LIBDIR +# SPDX-License-Identifier: BSD-3-Clause + +#============================================================================= +# Copyright 2012-2013 Inria +# Copyright 2012-2013 Emmanuel Agullo +# Copyright 2012-2013 Mathieu Faverge +# Copyright 2012 Cedric Castagnede +# Copyright 2013 Florent Pruvost +# +# Distributed under the OSI-approved BSD License (the "License"); +# see accompanying file MORSE-Copyright.txt for details. +# +# This software is distributed WITHOUT ANY WARRANTY; without even the +# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. +# See the License for more information. +#============================================================================= +# (To distribute this file outside of Morse, substitute the full +# License text for the above reference.) + +set(METIS_DIR "" CACHE PATH "Installation directory of METIS library") + +# Preserve the historical environment hints in addition to CMake's normal search paths. +if(WIN32) + set(_metis_include_env INCLUDE) + set(_metis_library_env LIB) +else() + set(_metis_include_env INCLUDE C_INCLUDE_PATH CPATH INCLUDE_PATH) + if(APPLE) + set(_metis_library_env DYLD_LIBRARY_PATH) + else() + set(_metis_library_env LD_LIBRARY_PATH) + endif() +endif() +set(_metis_include_hints) +foreach(variable IN LISTS _metis_include_env) + file(TO_CMAKE_PATH "$ENV{${variable}}" paths) + list(APPEND _metis_include_hints ${paths}) +endforeach() +set(_metis_library_hints) +foreach(variable IN LISTS _metis_library_env) + file(TO_CMAKE_PATH "$ENV{${variable}}" paths) + list(APPEND _metis_library_hints ${paths}) +endforeach() +if(NOT WIN32) + list(APPEND _metis_library_hints ${CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES} + ${CMAKE_C_IMPLICIT_LINK_DIRECTORIES}) +endif() + +set(_metis_include_suffixes) +if(METIS_INCDIR) + set(_metis_include_hints "${METIS_INCDIR}") +elseif(METIS_DIR) + set(_metis_include_hints "${METIS_DIR}") + set(_metis_include_suffixes include include/metis) +elseif(DEFINED ENV{METIS_INCDIR} AND NOT "$ENV{METIS_INCDIR}" STREQUAL "") + set(_metis_include_hints "$ENV{METIS_INCDIR}") +elseif(DEFINED ENV{METIS_DIR} AND NOT "$ENV{METIS_DIR}" STREQUAL "") + set(_metis_include_hints "$ENV{METIS_DIR}" "$ENV{METIS_DIR}/include" "$ENV{METIS_DIR}/include/metis") +endif() +if(NOT METIS_INCDIR AND NOT METIS_DIR) + list(APPEND _metis_include_hints ${CMAKE_PLATFORM_IMPLICIT_INCLUDE_DIRECTORIES} + ${CMAKE_C_IMPLICIT_INCLUDE_DIRECTORIES}) +endif() + +set(_metis_library_suffixes) +if(METIS_LIBDIR) + set(_metis_library_hints "${METIS_LIBDIR}") +elseif(METIS_DIR) + set(_metis_library_hints "${METIS_DIR}") + set(_metis_library_suffixes lib lib32 lib64) +elseif(DEFINED ENV{METIS_LIBDIR} AND NOT "$ENV{METIS_LIBDIR}" STREQUAL "") + set(_metis_library_hints "$ENV{METIS_LIBDIR}") +elseif(DEFINED ENV{METIS_DIR} AND NOT "$ENV{METIS_DIR}" STREQUAL "") + set(_metis_library_hints "$ENV{METIS_DIR}" "$ENV{METIS_DIR}/lib") +endif() + +# Re-search on each call so changed installation hints also work with a warm cache. +set(METIS_metis.h_DIRS "METIS_metis.h_DIRS-NOTFOUND") +set(METIS_metis_LIBRARY "METIS_metis_LIBRARY-NOTFOUND") +find_path(METIS_metis.h_DIRS NAMES metis.h HINTS ${_metis_include_hints} + PATH_SUFFIXES ${_metis_include_suffixes}) +find_library(METIS_metis_LIBRARY NAMES metis HINTS ${_metis_library_hints} + PATH_SUFFIXES ${_metis_library_suffixes}) +set(METIS_INCLUDE_DIRS "${METIS_metis.h_DIRS}") +set(METIS_LIBRARIES "${METIS_metis_LIBRARY}") +set(METIS_LIBRARY_DIRS "METIS_LIBRARY_DIRS-NOTFOUND") + +unset(METIS_WORKS CACHE) +unset(METIS_WORKS) +if(METIS_LIBRARIES) + get_filename_component(METIS_LIBRARY_DIRS "${METIS_metis_LIBRARY}" DIRECTORY) + string(REGEX REPLACE "/lib(32|64)?$" "" _metis_prefix "${METIS_LIBRARY_DIRS}") + set(METIS_DIR_FOUND "${_metis_prefix}" CACHE PATH "Installation directory of METIS library" FORCE) + + include(CMakePushCheckState) + include(CheckFunctionExists) + cmake_push_check_state(RESET) + set(CMAKE_REQUIRED_LIBRARIES "${METIS_LIBRARIES}") + find_library(M_LIBRARY NAMES m) + if(M_LIBRARY) + list(APPEND CMAKE_REQUIRED_LIBRARIES "${M_LIBRARY}") + endif() + check_function_exists(METIS_NodeND METIS_WORKS) + cmake_pop_check_state() +endif() + +include(FindPackageHandleStandardArgs) +find_package_handle_standard_args(METIS DEFAULT_MSG METIS_LIBRARIES METIS_WORKS METIS_INCLUDE_DIRS) +mark_as_advanced(METIS_DIR METIS_DIR_FOUND METIS_metis.h_DIRS METIS_metis_LIBRARY METIS_WORKS M_LIBRARY)
diff --git a/cmake/FindMetis.cmake b/cmake/FindMetis.cmake deleted file mode 100644 index 371fc09..0000000 --- a/cmake/FindMetis.cmake +++ /dev/null
@@ -1,266 +0,0 @@ -### -# -# @copyright (c) 2009-2014 The University of Tennessee and The University -# of Tennessee Research Foundation. -# All rights reserved. -# @copyright (c) 2012-2014 Inria. All rights reserved. -# @copyright (c) 2012-2014 Bordeaux INP, CNRS (LaBRI UMR 5800), Inria, Univ. Bordeaux. All rights reserved. -# -### -# -# - Find METIS include dirs and libraries -# Use this module by invoking find_package with the form: -# find_package(METIS -# [REQUIRED] # Fail with error if metis is not found -# ) -# -# This module finds headers and metis library. -# Results are reported in variables: -# METIS_FOUND - True if headers and requested libraries were found -# METIS_INCLUDE_DIRS - metis include directories -# METIS_LIBRARY_DIRS - Link directories for metis libraries -# METIS_LIBRARIES - metis component libraries to be linked -# -# The user can give specific paths where to find the libraries adding cmake -# options at configure (ex: cmake path/to/project -DMETIS_DIR=path/to/metis): -# METIS_DIR - Where to find the base directory of metis -# METIS_INCDIR - Where to find the header files -# METIS_LIBDIR - Where to find the library files -# The module can also look for the following environment variables if paths -# are not given as cmake variable: METIS_DIR, METIS_INCDIR, METIS_LIBDIR -# SPDX-License-Identifier: BSD-3-Clause - -#============================================================================= -# Copyright 2012-2013 Inria -# Copyright 2012-2013 Emmanuel Agullo -# Copyright 2012-2013 Mathieu Faverge -# Copyright 2012 Cedric Castagnede -# Copyright 2013 Florent Pruvost -# -# Distributed under the OSI-approved BSD License (the "License"); -# see accompanying file MORSE-Copyright.txt for details. -# -# This software is distributed WITHOUT ANY WARRANTY; without even the -# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -# See the License for more information. -#============================================================================= -# (To distribute this file outside of Morse, substitute the full -# License text for the above reference.) - -if (NOT METIS_FOUND) - set(METIS_DIR "" CACHE PATH "Installation directory of METIS library") - if (NOT METIS_FIND_QUIETLY) - message(STATUS "A cache variable, namely METIS_DIR, has been set to specify the install directory of METIS") - endif() -endif() - -# Looking for include -# ------------------- - -# Add system include paths to search include -# ------------------------------------------ -unset(_inc_env) -set(ENV_METIS_DIR "$ENV{METIS_DIR}") -set(ENV_METIS_INCDIR "$ENV{METIS_INCDIR}") -if(ENV_METIS_INCDIR) - list(APPEND _inc_env "${ENV_METIS_INCDIR}") -elseif(ENV_METIS_DIR) - list(APPEND _inc_env "${ENV_METIS_DIR}") - list(APPEND _inc_env "${ENV_METIS_DIR}/include") - list(APPEND _inc_env "${ENV_METIS_DIR}/include/metis") -else() - if(WIN32) - string(REPLACE ":" ";" _inc_env "$ENV{INCLUDE}") - else() - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{C_INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{CPATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - endif() -endif() -list(APPEND _inc_env "${CMAKE_PLATFORM_IMPLICIT_INCLUDE_DIRECTORIES}") -list(APPEND _inc_env "${CMAKE_C_IMPLICIT_INCLUDE_DIRECTORIES}") -list(REMOVE_DUPLICATES _inc_env) - - -# Try to find the metis header in the given paths -# ------------------------------------------------- -# call cmake macro to find the header path -if(METIS_INCDIR) - set(METIS_metis.h_DIRS "METIS_metis.h_DIRS-NOTFOUND") - find_path(METIS_metis.h_DIRS - NAMES metis.h - HINTS ${METIS_INCDIR}) -else() - if(METIS_DIR) - set(METIS_metis.h_DIRS "METIS_metis.h_DIRS-NOTFOUND") - find_path(METIS_metis.h_DIRS - NAMES metis.h - HINTS ${METIS_DIR} - PATH_SUFFIXES "include" "include/metis") - else() - set(METIS_metis.h_DIRS "METIS_metis.h_DIRS-NOTFOUND") - find_path(METIS_metis.h_DIRS - NAMES metis.h - HINTS ${_inc_env}) - endif() -endif() -mark_as_advanced(METIS_metis.h_DIRS) - - -# If found, add path to cmake variable -# ------------------------------------ -if (METIS_metis.h_DIRS) - set(METIS_INCLUDE_DIRS "${METIS_metis.h_DIRS}") -else () - set(METIS_INCLUDE_DIRS "METIS_INCLUDE_DIRS-NOTFOUND") - if(NOT METIS_FIND_QUIETLY) - message(STATUS "Looking for metis -- metis.h not found") - endif() -endif() - - -# Looking for lib -# --------------- - -# Add system library paths to search lib -# -------------------------------------- -unset(_lib_env) -set(ENV_METIS_LIBDIR "$ENV{METIS_LIBDIR}") -if(ENV_METIS_LIBDIR) - list(APPEND _lib_env "${ENV_METIS_LIBDIR}") -elseif(ENV_METIS_DIR) - list(APPEND _lib_env "${ENV_METIS_DIR}") - list(APPEND _lib_env "${ENV_METIS_DIR}/lib") -else() - if(WIN32) - string(REPLACE ":" ";" _lib_env "$ENV{LIB}") - else() - if(APPLE) - string(REPLACE ":" ";" _lib_env "$ENV{DYLD_LIBRARY_PATH}") - else() - string(REPLACE ":" ";" _lib_env "$ENV{LD_LIBRARY_PATH}") - endif() - list(APPEND _lib_env "${CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES}") - list(APPEND _lib_env "${CMAKE_C_IMPLICIT_LINK_DIRECTORIES}") - endif() -endif() -list(REMOVE_DUPLICATES _lib_env) - -# Try to find the metis lib in the given paths -# ---------------------------------------------- -# call cmake macro to find the lib path -if(METIS_LIBDIR) - set(METIS_metis_LIBRARY "METIS_metis_LIBRARY-NOTFOUND") - find_library(METIS_metis_LIBRARY - NAMES metis - HINTS ${METIS_LIBDIR}) -else() - if(METIS_DIR) - set(METIS_metis_LIBRARY "METIS_metis_LIBRARY-NOTFOUND") - find_library(METIS_metis_LIBRARY - NAMES metis - HINTS ${METIS_DIR} - PATH_SUFFIXES lib lib32 lib64) - else() - set(METIS_metis_LIBRARY "METIS_metis_LIBRARY-NOTFOUND") - find_library(METIS_metis_LIBRARY - NAMES metis - HINTS ${_lib_env}) - endif() -endif() -mark_as_advanced(METIS_metis_LIBRARY) - - -# If found, add path to cmake variable -# ------------------------------------ -if (METIS_metis_LIBRARY) - get_filename_component(metis_lib_path "${METIS_metis_LIBRARY}" PATH) - # set cmake variables - set(METIS_LIBRARIES "${METIS_metis_LIBRARY}") - set(METIS_LIBRARY_DIRS "${metis_lib_path}") -else () - set(METIS_LIBRARIES "METIS_LIBRARIES-NOTFOUND") - set(METIS_LIBRARY_DIRS "METIS_LIBRARY_DIRS-NOTFOUND") - if(NOT METIS_FIND_QUIETLY) - message(STATUS "Looking for metis -- lib metis not found") - endif() -endif () - -# check a function to validate the find -if(METIS_LIBRARIES) - - set(REQUIRED_INCDIRS) - set(REQUIRED_LIBDIRS) - set(REQUIRED_LIBS) - - # METIS - if (METIS_INCLUDE_DIRS) - set(REQUIRED_INCDIRS "${METIS_INCLUDE_DIRS}") - endif() - if (METIS_LIBRARY_DIRS) - set(REQUIRED_LIBDIRS "${METIS_LIBRARY_DIRS}") - endif() - set(REQUIRED_LIBS "${METIS_LIBRARIES}") - # m - find_library(M_LIBRARY NAMES m) - mark_as_advanced(M_LIBRARY) - if(M_LIBRARY) - list(APPEND REQUIRED_LIBS "-lm") - endif() - - # set required libraries for link - set(CMAKE_REQUIRED_INCLUDES "${REQUIRED_INCDIRS}") - set(CMAKE_REQUIRED_LIBRARIES) - foreach(lib_dir ${REQUIRED_LIBDIRS}) - list(APPEND CMAKE_REQUIRED_LIBRARIES "-L${lib_dir}") - endforeach() - list(APPEND CMAKE_REQUIRED_LIBRARIES "${REQUIRED_LIBS}") - string(REGEX REPLACE "^ -" "-" CMAKE_REQUIRED_LIBRARIES "${CMAKE_REQUIRED_LIBRARIES}") - - # test link - unset(METIS_WORKS CACHE) - include(CheckFunctionExists) - check_function_exists(METIS_NodeND METIS_WORKS) - mark_as_advanced(METIS_WORKS) - - if(NOT METIS_WORKS) - if(NOT METIS_FIND_QUIETLY) - message(STATUS "Looking for METIS : test of METIS_NodeND with METIS library fails") - message(STATUS "CMAKE_REQUIRED_LIBRARIES: ${CMAKE_REQUIRED_LIBRARIES}") - message(STATUS "CMAKE_REQUIRED_INCLUDES: ${CMAKE_REQUIRED_INCLUDES}") - message(STATUS "Check in CMakeFiles/CMakeError.log to figure out why it fails") - endif() - endif() - set(CMAKE_REQUIRED_INCLUDES) - set(CMAKE_REQUIRED_FLAGS) - set(CMAKE_REQUIRED_LIBRARIES) -endif() - -if (METIS_LIBRARIES) - list(GET METIS_LIBRARIES 0 first_lib) - get_filename_component(first_lib_path "${first_lib}" PATH) - if (${first_lib_path} MATCHES "/lib(32|64)?$") - string(REGEX REPLACE "/lib(32|64)?$" "" not_cached_dir "${first_lib_path}") - set(METIS_DIR_FOUND "${not_cached_dir}" CACHE PATH "Installation directory of METIS library" FORCE) - else() - set(METIS_DIR_FOUND "${first_lib_path}" CACHE PATH "Installation directory of METIS library" FORCE) - endif() -endif() -mark_as_advanced(METIS_DIR) -mark_as_advanced(METIS_DIR_FOUND) - -# check that METIS has been found -# --------------------------------- -include(FindPackageHandleStandardArgs) -find_package_handle_standard_args(METIS DEFAULT_MSG - METIS_LIBRARIES - METIS_WORKS - METIS_INCLUDE_DIRS) -# -# TODO: Add possibility to check for specific functions in the library -#
diff --git a/cmake/FindPASTIX.cmake b/cmake/FindPASTIX.cmake index 0a91ccd..a02a834 100644 --- a/cmake/FindPASTIX.cmake +++ b/cmake/FindPASTIX.cmake
@@ -15,45 +15,20 @@ # [COMPONENTS <comp1> <comp2> ...] # dependencies # ) # -# PASTIX depends on the following libraries: -# - Threads, m, rt -# - MPI -# - HWLOC -# - BLAS -# -# COMPONENTS are optional libraries PASTIX could be linked with, -# Use it to drive detection of a specific compilation chain -# COMPONENTS can be some of the following: -# - MPI: to activate detection of the parallel MPI version (default) -# it looks for Threads, HWLOC, BLAS, MPI and ScaLAPACK libraries -# - SEQ: to activate detection of the sequential version (exclude MPI version) -# - STARPU: to activate detection of StarPU version -# it looks for MPI version of StarPU (default behaviour) -# if SEQ and STARPU are given, it looks for a StarPU without MPI -# - STARPU_CUDA: to activate detection of StarPU with CUDA -# - STARPU_FXT: to activate detection of StarPU with FxT -# - SCOTCH: to activate detection of PASTIX linked with SCOTCH -# - PTSCOTCH: to activate detection of PASTIX linked with SCOTCH -# - METIS: to activate detection of PASTIX linked with SCOTCH +# Requires PaStiX 6.x and its pkg-config metadata, including SpM and ordering dependencies. # # This module finds headers and pastix library. # Results are reported in variables: # PASTIX_FOUND - True if headers and requested libraries were found -# PASTIX_LINKER_FLAGS - list of required linker flags (excluding -l and -L) +# PASTIX_VERSION - Detected PaStiX version # PASTIX_INCLUDE_DIRS - pastix include directories -# PASTIX_LIBRARY_DIRS - Link directories for pastix libraries # PASTIX_LIBRARIES - pastix libraries # PASTIX_INCLUDE_DIRS_DEP - pastix + dependencies include directories -# PASTIX_LIBRARY_DIRS_DEP - pastix + dependencies link directories # PASTIX_LIBRARIES_DEP - pastix libraries + dependencies # # The user can give specific paths where to find the libraries adding cmake # options at configure (ex: cmake path/to/project -DPASTIX_DIR=path/to/pastix): # PASTIX_DIR - Where to find the base directory of pastix -# PASTIX_INCDIR - Where to find the header files -# PASTIX_LIBDIR - Where to find the library files -# The module can also look for the following environment variables if paths -# are not given as cmake variable: PASTIX_DIR, PASTIX_INCDIR, PASTIX_LIBDIR # SPDX-License-Identifier: BSD-3-Clause #============================================================================= @@ -74,632 +49,59 @@ # License text for the above reference.) -if (NOT PASTIX_FOUND) - set(PASTIX_DIR "" CACHE PATH "Installation directory of PASTIX library") - if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "A cache variable, namely PASTIX_DIR, has been set to specify the install directory of PASTIX") - endif() -endif() - -# Set the version to find -set(PASTIX_LOOK_FOR_MPI ON) -set(PASTIX_LOOK_FOR_SEQ OFF) -set(PASTIX_LOOK_FOR_STARPU OFF) -set(PASTIX_LOOK_FOR_STARPU_CUDA OFF) -set(PASTIX_LOOK_FOR_STARPU_FXT OFF) -set(PASTIX_LOOK_FOR_SCOTCH ON) -set(PASTIX_LOOK_FOR_PTSCOTCH OFF) -set(PASTIX_LOOK_FOR_METIS OFF) - -if( PASTIX_FIND_COMPONENTS ) - foreach( component ${PASTIX_FIND_COMPONENTS} ) - if (${component} STREQUAL "SEQ") - # means we look for the sequential version of PaStiX (without MPI) - set(PASTIX_LOOK_FOR_SEQ ON) - set(PASTIX_LOOK_FOR_MPI OFF) - endif() - if (${component} STREQUAL "MPI") - # means we look for the MPI version of PaStiX (default) - set(PASTIX_LOOK_FOR_SEQ OFF) - set(PASTIX_LOOK_FOR_MPI ON) - endif() - if (${component} STREQUAL "STARPU") - # means we look for PaStiX with StarPU - set(PASTIX_LOOK_FOR_STARPU ON) - endif() - if (${component} STREQUAL "STARPU_CUDA") - # means we look for PaStiX with StarPU + CUDA - set(PASTIX_LOOK_FOR_STARPU ON) - set(PASTIX_LOOK_FOR_STARPU_CUDA ON) - endif() - if (${component} STREQUAL "STARPU_FXT") - # means we look for PaStiX with StarPU + FxT - set(PASTIX_LOOK_FOR_STARPU_FXT ON) - endif() - if (${component} STREQUAL "SCOTCH") - set(PASTIX_LOOK_FOR_SCOTCH ON) - endif() - if (${component} STREQUAL "PTSCOTCH") - set(PASTIX_LOOK_FOR_PTSCOTCH ON) - endif() - if (${component} STREQUAL "METIS") - set(PASTIX_LOOK_FOR_METIS ON) - endif() - endforeach() -endif() - -# Dependencies detection -# ---------------------- - - -# Required dependencies -# --------------------- -include(CMakeFindDependencyMacro) -if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect pthread") -endif() -if (PASTIX_FIND_REQUIRED) - find_dependency(Threads REQUIRED QUIET) -else() - find_dependency(Threads QUIET) -endif() -set(PASTIX_EXTRA_LIBRARIES "") -if( THREADS_FOUND ) - list(APPEND PASTIX_EXTRA_LIBRARIES ${CMAKE_THREAD_LIBS_INIT}) -endif () - -# Add math library to the list of extra -# it normally exists on all common systems provided with a C compiler -if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect libm") -endif() -set(PASTIX_M_LIBRARIES "") -if(UNIX OR WIN32) - find_library( - PASTIX_M_m_LIBRARY - NAMES m - ) - mark_as_advanced(PASTIX_M_m_LIBRARY) - if (PASTIX_M_m_LIBRARY) - list(APPEND PASTIX_M_LIBRARIES "${PASTIX_M_m_LIBRARY}") - list(APPEND PASTIX_EXTRA_LIBRARIES "${PASTIX_M_m_LIBRARY}") - else() - if (PASTIX_FIND_REQUIRED) - message(FATAL_ERROR "Could NOT find libm on your system." - "Are you sure to a have a C compiler installed?") - endif() - endif() -endif() - -# Try to find librt (libposix4 - POSIX.1b Realtime Extensions library) -# on Unix systems except Apple ones because it does not exist on it -if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect librt") -endif() -set(PASTIX_RT_LIBRARIES "") -if(UNIX AND NOT APPLE) - find_library( - PASTIX_RT_rt_LIBRARY - NAMES rt - ) - mark_as_advanced(PASTIX_RT_rt_LIBRARY) - if (PASTIX_RT_rt_LIBRARY) - list(APPEND PASTIX_RT_LIBRARIES "${PASTIX_RT_rt_LIBRARY}") - list(APPEND PASTIX_EXTRA_LIBRARIES "${PASTIX_RT_rt_LIBRARY}") - else() - if (PASTIX_FIND_REQUIRED) - message(FATAL_ERROR "Could NOT find librt on your system") - endif() - endif() -endif() - -# PASTIX depends on HWLOC -#------------------------ -if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect HWLOC") -endif() -if (PASTIX_FIND_REQUIRED) - find_dependency(HWLOC REQUIRED QUIET) -else() - find_dependency(HWLOC QUIET) -endif() - -# PASTIX depends on BLAS -#----------------------- -if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect BLAS") -endif() -if (PASTIX_FIND_REQUIRED) - find_dependency(BLASEXT REQUIRED QUIET) -else() - find_dependency(BLASEXT QUIET) -endif() - -# Optional dependencies -# --------------------- - -# PASTIX may depend on MPI -#------------------------- -if (NOT MPI_FOUND AND PASTIX_LOOK_FOR_MPI) - if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect MPI") - endif() - # allows to use an external mpi compilation by setting compilers with - # -DMPI_C_COMPILER=path/to/mpicc -DMPI_Fortran_COMPILER=path/to/mpif90 - # at cmake configure - if(NOT MPI_C_COMPILER) - set(MPI_C_COMPILER mpicc) - endif() - if (PASTIX_FIND_REQUIRED AND PASTIX_FIND_REQUIRED_MPI) - find_dependency(MPI REQUIRED QUIET) - else() - find_dependency(MPI QUIET) - endif() - if (MPI_FOUND) - mark_as_advanced(MPI_LIBRARY) - mark_as_advanced(MPI_EXTRA_LIBRARY) - endif() -endif () - -# PASTIX may depend on STARPU -#---------------------------- -if( NOT STARPU_FOUND AND PASTIX_LOOK_FOR_STARPU) - - if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect StarPU") - endif() - - set(PASTIX_STARPU_VERSION "1.1" CACHE STRING "oldest STARPU version desired") - - # create list of components in order to make a single call to find_package(starpu...) - # we explicitly need a StarPU version built with hwloc - set(STARPU_COMPONENT_LIST "HWLOC") - - # StarPU may depend on MPI - # allows to use an external mpi compilation by setting compilers with - # -DMPI_C_COMPILER=path/to/mpicc -DMPI_Fortran_COMPILER=path/to/mpif90 - # at cmake configure - if (PASTIX_LOOK_FOR_MPI) - if(NOT MPI_C_COMPILER) - set(MPI_C_COMPILER mpicc) - endif() - list(APPEND STARPU_COMPONENT_LIST "MPI") - endif() - if (PASTIX_LOOK_FOR_STARPU_CUDA) - list(APPEND STARPU_COMPONENT_LIST "CUDA") - endif() - if (PASTIX_LOOK_FOR_STARPU_FXT) - list(APPEND STARPU_COMPONENT_LIST "FXT") - endif() - # set the list of optional dependencies we may discover - if (PASTIX_FIND_REQUIRED AND PASTIX_FIND_REQUIRED_STARPU) - find_dependency(STARPU ${PASTIX_STARPU_VERSION} REQUIRED - COMPONENTS ${STARPU_COMPONENT_LIST}) - else() - find_dependency(STARPU ${PASTIX_STARPU_VERSION} - COMPONENTS ${STARPU_COMPONENT_LIST}) - endif() - -endif() - -# PASTIX may depends on SCOTCH -#----------------------------- -if (NOT SCOTCH_FOUND AND PASTIX_LOOK_FOR_SCOTCH) - if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect SCOTCH") - endif() - if (PASTIX_FIND_REQUIRED AND PASTIX_FIND_REQUIRED_SCOTCH) - find_dependency(SCOTCH REQUIRED QUIET) - else() - find_dependency(SCOTCH QUIET) - endif() -endif() - -# PASTIX may depends on PTSCOTCH -#------------------------------- -if (NOT PTSCOTCH_FOUND AND PASTIX_LOOK_FOR_PTSCOTCH) - if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect PTSCOTCH") - endif() - if (PASTIX_FIND_REQUIRED AND PASTIX_FIND_REQUIRED_PTSCOTCH) - find_dependency(PTSCOTCH REQUIRED QUIET) - else() - find_dependency(PTSCOTCH QUIET) - endif() -endif() - -# PASTIX may depends on METIS -#---------------------------- -if (NOT METIS_FOUND AND PASTIX_LOOK_FOR_METIS) - if (NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX - Try to detect METIS") - endif() - if (PASTIX_FIND_REQUIRED AND PASTIX_FIND_REQUIRED_METIS) - find_dependency(METIS REQUIRED QUIET) - else() - find_dependency(METIS QUIET) - endif() -endif() - -# Error if pastix required and no partitioning lib found -if (PASTIX_FIND_REQUIRED AND NOT SCOTCH_FOUND AND NOT PTSCOTCH_FOUND AND NOT METIS_FOUND) - message(FATAL_ERROR "Could NOT find any partitioning library on your system" - " (install scotch, ptscotch or metis)") -endif() - - -# Looking for PaStiX -# ------------------ - -# Looking for include -# ------------------- - -# Add system include paths to search include -# ------------------------------------------ -unset(_inc_env) -set(ENV_PASTIX_DIR "$ENV{PASTIX_DIR}") -set(ENV_PASTIX_INCDIR "$ENV{PASTIX_INCDIR}") -if(ENV_PASTIX_INCDIR) - list(APPEND _inc_env "${ENV_PASTIX_INCDIR}") -elseif(ENV_PASTIX_DIR) - list(APPEND _inc_env "${ENV_PASTIX_DIR}") - list(APPEND _inc_env "${ENV_PASTIX_DIR}/include") - list(APPEND _inc_env "${ENV_PASTIX_DIR}/include/pastix") -else() - if(WIN32) - string(REPLACE ":" ";" _inc_env "$ENV{INCLUDE}") - else() - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{C_INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{CPATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - endif() -endif() -list(APPEND _inc_env "${CMAKE_PLATFORM_IMPLICIT_INCLUDE_DIRECTORIES}") -list(APPEND _inc_env "${CMAKE_C_IMPLICIT_INCLUDE_DIRECTORIES}") -list(REMOVE_DUPLICATES _inc_env) - - -# Try to find the pastix header in the given paths -# --------------------------------------------------- -# call cmake macro to find the header path -if(PASTIX_INCDIR) - set(PASTIX_pastix.h_DIRS "PASTIX_pastix.h_DIRS-NOTFOUND") - find_path(PASTIX_pastix.h_DIRS - NAMES pastix.h - HINTS ${PASTIX_INCDIR}) -else() +find_package(PkgConfig QUIET) +if(PKG_CONFIG_FOUND) + set(_eigen_pastix_prefix_path "${CMAKE_PREFIX_PATH}") if(PASTIX_DIR) - set(PASTIX_pastix.h_DIRS "PASTIX_pastix.h_DIRS-NOTFOUND") - find_path(PASTIX_pastix.h_DIRS - NAMES pastix.h - HINTS ${PASTIX_DIR} - PATH_SUFFIXES "include" "include/pastix") - else() - set(PASTIX_pastix.h_DIRS "PASTIX_pastix.h_DIRS-NOTFOUND") - find_path(PASTIX_pastix.h_DIRS - NAMES pastix.h - HINTS ${_inc_env} - PATH_SUFFIXES "pastix") + list(PREPEND CMAKE_PREFIX_PATH "${PASTIX_DIR}") endif() -endif() -mark_as_advanced(PASTIX_pastix.h_DIRS) - -# If found, add path to cmake variable -# ------------------------------------ -if (PASTIX_pastix.h_DIRS) - set(PASTIX_INCLUDE_DIRS "${PASTIX_pastix.h_DIRS}") -else () - set(PASTIX_INCLUDE_DIRS "PASTIX_INCLUDE_DIRS-NOTFOUND") - if(NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for pastix -- pastix.h not found") - endif() + pkg_check_modules(PASTIX6 QUIET IMPORTED_TARGET GLOBAL "pastix>=6") + set(CMAKE_PREFIX_PATH "${_eigen_pastix_prefix_path}") + unset(_eigen_pastix_prefix_path) endif() - -# Looking for lib -# --------------- - -# Add system library paths to search lib -# -------------------------------------- -unset(_lib_env) -set(ENV_PASTIX_LIBDIR "$ENV{PASTIX_LIBDIR}") -if(ENV_PASTIX_LIBDIR) - list(APPEND _lib_env "${ENV_PASTIX_LIBDIR}") -elseif(ENV_PASTIX_DIR) - list(APPEND _lib_env "${ENV_PASTIX_DIR}") - list(APPEND _lib_env "${ENV_PASTIX_DIR}/lib") -else() - if(WIN32) - string(REPLACE ":" ";" _lib_env "$ENV{LIB}") - else() - if(APPLE) - string(REPLACE ":" ";" _lib_env "$ENV{DYLD_LIBRARY_PATH}") - else() - string(REPLACE ":" ";" _lib_env "$ENV{LD_LIBRARY_PATH}") - endif() - list(APPEND _lib_env "${CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES}") - list(APPEND _lib_env "${CMAKE_C_IMPLICIT_LINK_DIRECTORIES}") - endif() -endif() -list(REMOVE_DUPLICATES _lib_env) - -# Try to find the pastix lib in the given paths -# ------------------------------------------------ - -# create list of libs to find -set(PASTIX_libs_to_find "pastix_murge;pastix") - -# call cmake macro to find the lib path -if(PASTIX_LIBDIR) - foreach(pastix_lib ${PASTIX_libs_to_find}) - set(PASTIX_${pastix_lib}_LIBRARY "PASTIX_${pastix_lib}_LIBRARY-NOTFOUND") - find_library(PASTIX_${pastix_lib}_LIBRARY - NAMES ${pastix_lib} - HINTS ${PASTIX_LIBDIR}) - endforeach() -else() - if(PASTIX_DIR) - foreach(pastix_lib ${PASTIX_libs_to_find}) - set(PASTIX_${pastix_lib}_LIBRARY "PASTIX_${pastix_lib}_LIBRARY-NOTFOUND") - find_library(PASTIX_${pastix_lib}_LIBRARY - NAMES ${pastix_lib} - HINTS ${PASTIX_DIR} - PATH_SUFFIXES lib lib32 lib64) - endforeach() - else() - foreach(pastix_lib ${PASTIX_libs_to_find}) - set(PASTIX_${pastix_lib}_LIBRARY "PASTIX_${pastix_lib}_LIBRARY-NOTFOUND") - find_library(PASTIX_${pastix_lib}_LIBRARY - NAMES ${pastix_lib} - HINTS ${_lib_env}) - endforeach() - endif() -endif() - -# If found, add path to cmake variable -# ------------------------------------ -foreach(pastix_lib ${PASTIX_libs_to_find}) - - get_filename_component(${pastix_lib}_lib_path ${PASTIX_${pastix_lib}_LIBRARY} PATH) - # set cmake variables (respects naming convention) - if (PASTIX_LIBRARIES) - list(APPEND PASTIX_LIBRARIES "${PASTIX_${pastix_lib}_LIBRARY}") - else() - set(PASTIX_LIBRARIES "${PASTIX_${pastix_lib}_LIBRARY}") - endif() - if (PASTIX_LIBRARY_DIRS) - list(APPEND PASTIX_LIBRARY_DIRS "${${pastix_lib}_lib_path}") - else() - set(PASTIX_LIBRARY_DIRS "${${pastix_lib}_lib_path}") - endif() - mark_as_advanced(PASTIX_${pastix_lib}_LIBRARY) - -endforeach() - -# check a function to validate the find -if(PASTIX_LIBRARIES) - - set(REQUIRED_LDFLAGS) - set(REQUIRED_INCDIRS) - set(REQUIRED_LIBDIRS) - set(REQUIRED_LIBS) - - # PASTIX - if (PASTIX_INCLUDE_DIRS) - set(REQUIRED_INCDIRS "${PASTIX_INCLUDE_DIRS}") - endif() - foreach(libdir ${PASTIX_LIBRARY_DIRS}) - if (libdir) - list(APPEND REQUIRED_LIBDIRS "${libdir}") - endif() - endforeach() - set(REQUIRED_LIBS "${PASTIX_LIBRARIES}") - # STARPU - if (PASTIX_LOOK_FOR_STARPU AND STARPU_FOUND) - if (STARPU_INCLUDE_DIRS_DEP) - list(APPEND REQUIRED_INCDIRS "${STARPU_INCLUDE_DIRS_DEP}") - elseif (STARPU_INCLUDE_DIRS) - list(APPEND REQUIRED_INCDIRS "${STARPU_INCLUDE_DIRS}") - endif() - if(STARPU_LIBRARY_DIRS_DEP) - list(APPEND REQUIRED_LIBDIRS "${STARPU_LIBRARY_DIRS_DEP}") - elseif(STARPU_LIBRARY_DIRS) - list(APPEND REQUIRED_LIBDIRS "${STARPU_LIBRARY_DIRS}") - endif() - if (STARPU_LIBRARIES_DEP) - list(APPEND REQUIRED_LIBS "${STARPU_LIBRARIES_DEP}") - elseif (STARPU_LIBRARIES) - foreach(lib ${STARPU_LIBRARIES}) - if (EXISTS ${lib} OR ${lib} MATCHES "^-") - list(APPEND REQUIRED_LIBS "${lib}") - else() - list(APPEND REQUIRED_LIBS "-l${lib}") - endif() +set(PASTIX_VERSION "${PASTIX6_VERSION}") +set(PASTIX_INCLUDE_DIRS ${PASTIX6_INCLUDE_DIRS}) +set(PASTIX_INCLUDE_DIRS_DEP ${PASTIX6_INCLUDE_DIRS}) +set(PASTIX_LIBRARIES "") +set(PASTIX_LIBRARIES_DEP "") +if(PASTIX6_FOUND AND PASTIX6_VERSION VERSION_LESS 7) + # FindPkgConfig's imported target uses only the shared-link metadata, even + # when find_library resolves PaStiX to a static archive. + foreach(_pastix_library IN LISTS PASTIX6_LINK_LIBRARIES) + if(_pastix_library MATCHES "[/\\\\]libpastix\\.a$") + set(_pastix_static_libraries "") + foreach(_pastix_dependency IN LISTS PASTIX6_STATIC_LIBRARIES) + if(_pastix_dependency STREQUAL "pastix") + list(APPEND _pastix_static_libraries "${_pastix_library}") + continue() + endif() + find_library(_pastix_dependency_path NAMES ${_pastix_dependency} + HINTS ${PASTIX6_STATIC_LIBRARY_DIRS}) + if(_pastix_dependency_path) + list(APPEND _pastix_static_libraries "${_pastix_dependency_path}") + else() + list(APPEND _pastix_static_libraries "${_pastix_dependency}") + endif() + unset(_pastix_dependency_path CACHE) endforeach() + set_target_properties(PkgConfig::PASTIX6 PROPERTIES + INTERFACE_LINK_LIBRARIES "${_pastix_static_libraries}" + INTERFACE_LINK_DIRECTORIES "${PASTIX6_STATIC_LIBRARY_DIRS}" + INTERFACE_LINK_OPTIONS "${PASTIX6_STATIC_LDFLAGS_OTHER}" + INTERFACE_INCLUDE_DIRECTORIES "${PASTIX6_STATIC_INCLUDE_DIRS}" + INTERFACE_COMPILE_OPTIONS "${PASTIX6_STATIC_CFLAGS_OTHER}") + unset(_pastix_static_libraries) + unset(_pastix_dependency) + break() endif() - endif() - # CUDA - if (PASTIX_LOOK_FOR_STARPU_CUDA AND CUDA_FOUND) - if (CUDA_INCLUDE_DIRS) - list(APPEND REQUIRED_INCDIRS "${CUDA_INCLUDE_DIRS}") - endif() - foreach(libdir ${CUDA_LIBRARY_DIRS}) - if (libdir) - list(APPEND REQUIRED_LIBDIRS "${libdir}") - endif() - endforeach() - list(APPEND REQUIRED_LIBS "${CUDA_CUBLAS_LIBRARIES};${CUDA_LIBRARIES}") - endif() - # MPI - if (PASTIX_LOOK_FOR_MPI AND MPI_FOUND) - if (MPI_C_INCLUDE_PATH) - list(APPEND REQUIRED_INCDIRS "${MPI_C_INCLUDE_PATH}") - endif() - if (MPI_C_LINK_FLAGS) - if (${MPI_C_LINK_FLAGS} MATCHES " -") - string(REGEX REPLACE " -" "-" MPI_C_LINK_FLAGS ${MPI_C_LINK_FLAGS}) - endif() - list(APPEND REQUIRED_LDFLAGS "${MPI_C_LINK_FLAGS}") - endif() - list(APPEND REQUIRED_LIBS "${MPI_C_LIBRARIES}") - endif() - # HWLOC - if (HWLOC_FOUND) - if (HWLOC_INCLUDE_DIRS) - list(APPEND REQUIRED_INCDIRS "${HWLOC_INCLUDE_DIRS}") - endif() - foreach(libdir ${HWLOC_LIBRARY_DIRS}) - if (libdir) - list(APPEND REQUIRED_LIBDIRS "${libdir}") - endif() - endforeach() - foreach(lib ${HWLOC_LIBRARIES}) - if (EXISTS ${lib} OR ${lib} MATCHES "^-") - list(APPEND REQUIRED_LIBS "${lib}") - else() - list(APPEND REQUIRED_LIBS "-l${lib}") - endif() - endforeach() - endif() - # BLAS - if (BLAS_FOUND) - if (BLAS_INCLUDE_DIRS) - list(APPEND REQUIRED_INCDIRS "${BLAS_INCLUDE_DIRS}") - endif() - foreach(libdir ${BLAS_LIBRARY_DIRS}) - if (libdir) - list(APPEND REQUIRED_LIBDIRS "${libdir}") - endif() - endforeach() - list(APPEND REQUIRED_LIBS "${BLAS_LIBRARIES}") - if (BLAS_LINKER_FLAGS) - list(APPEND REQUIRED_LDFLAGS "${BLAS_LINKER_FLAGS}") - endif() - endif() - # SCOTCH - if (PASTIX_LOOK_FOR_SCOTCH AND SCOTCH_FOUND) - if (SCOTCH_INCLUDE_DIRS) - list(APPEND REQUIRED_INCDIRS "${SCOTCH_INCLUDE_DIRS}") - endif() - foreach(libdir ${SCOTCH_LIBRARY_DIRS}) - if (libdir) - list(APPEND REQUIRED_LIBDIRS "${libdir}") - endif() - endforeach() - list(APPEND REQUIRED_LIBS "${SCOTCH_LIBRARIES}") - endif() - # PTSCOTCH - if (PASTIX_LOOK_FOR_PTSCOTCH AND PTSCOTCH_FOUND) - if (PTSCOTCH_INCLUDE_DIRS) - list(APPEND REQUIRED_INCDIRS "${PTSCOTCH_INCLUDE_DIRS}") - endif() - foreach(libdir ${PTSCOTCH_LIBRARY_DIRS}) - if (libdir) - list(APPEND REQUIRED_LIBDIRS "${libdir}") - endif() - endforeach() - list(APPEND REQUIRED_LIBS "${PTSCOTCH_LIBRARIES}") - endif() - # METIS - if (PASTIX_LOOK_FOR_METIS AND METIS_FOUND) - if (METIS_INCLUDE_DIRS) - list(APPEND REQUIRED_INCDIRS "${METIS_INCLUDE_DIRS}") - endif() - foreach(libdir ${METIS_LIBRARY_DIRS}) - if (libdir) - list(APPEND REQUIRED_LIBDIRS "${libdir}") - endif() - endforeach() - list(APPEND REQUIRED_LIBS "${METIS_LIBRARIES}") - endif() - # Fortran - if (CMAKE_C_COMPILER_ID MATCHES "GNU") - find_library( - FORTRAN_gfortran_LIBRARY - NAMES gfortran - HINTS ${_lib_env} - ) - mark_as_advanced(FORTRAN_gfortran_LIBRARY) - if (FORTRAN_gfortran_LIBRARY) - list(APPEND REQUIRED_LIBS "${FORTRAN_gfortran_LIBRARY}") - endif() - elseif (CMAKE_C_COMPILER_ID MATCHES "Intel") - find_library( - FORTRAN_ifcore_LIBRARY - NAMES ifcore - HINTS ${_lib_env} - ) - mark_as_advanced(FORTRAN_ifcore_LIBRARY) - if (FORTRAN_ifcore_LIBRARY) - list(APPEND REQUIRED_LIBS "${FORTRAN_ifcore_LIBRARY}") - endif() - endif() - # EXTRA LIBS such that pthread, m, rt - list(APPEND REQUIRED_LIBS ${PASTIX_EXTRA_LIBRARIES}) - - # set required libraries for link - set(CMAKE_REQUIRED_INCLUDES "${REQUIRED_INCDIRS}") - set(CMAKE_REQUIRED_LIBRARIES) - list(APPEND CMAKE_REQUIRED_LIBRARIES "${REQUIRED_LDFLAGS}") - foreach(lib_dir ${REQUIRED_LIBDIRS}) - list(APPEND CMAKE_REQUIRED_LIBRARIES "-L${lib_dir}") endforeach() - list(APPEND CMAKE_REQUIRED_LIBRARIES "${REQUIRED_LIBS}") - list(APPEND CMAKE_REQUIRED_FLAGS "${REQUIRED_FLAGS}") - string(REGEX REPLACE "^ -" "-" CMAKE_REQUIRED_LIBRARIES "${CMAKE_REQUIRED_LIBRARIES}") - - # test link - unset(PASTIX_WORKS CACHE) - include(CheckFunctionExists) - check_function_exists(pastix PASTIX_WORKS) - mark_as_advanced(PASTIX_WORKS) - - if(PASTIX_WORKS) - # save link with dependencies - set(PASTIX_LIBRARIES_DEP "${REQUIRED_LIBS}") - set(PASTIX_LIBRARY_DIRS_DEP "${REQUIRED_LIBDIRS}") - set(PASTIX_INCLUDE_DIRS_DEP "${REQUIRED_INCDIRS}") - set(PASTIX_LINKER_FLAGS "${REQUIRED_LDFLAGS}") - list(REMOVE_DUPLICATES PASTIX_LIBRARY_DIRS_DEP) - list(REMOVE_DUPLICATES PASTIX_INCLUDE_DIRS_DEP) - list(REMOVE_DUPLICATES PASTIX_LINKER_FLAGS) - else() - if(NOT PASTIX_FIND_QUIETLY) - message(STATUS "Looking for PASTIX : test of pastix() fails") - message(STATUS "CMAKE_REQUIRED_LIBRARIES: ${CMAKE_REQUIRED_LIBRARIES}") - message(STATUS "CMAKE_REQUIRED_INCLUDES: ${CMAKE_REQUIRED_INCLUDES}") - message(STATUS "Check in CMakeFiles/CMakeError.log to figure out why it fails") - message(STATUS "Maybe PASTIX is linked with specific libraries. " - "Have you tried with COMPONENTS (MPI/SEQ, STARPU, STARPU_CUDA, SCOTCH, PTSCOTCH, METIS)? " - "See the explanation in FindPASTIX.cmake.") - endif() - endif() - set(CMAKE_REQUIRED_INCLUDES) - set(CMAKE_REQUIRED_FLAGS) - set(CMAKE_REQUIRED_LIBRARIES) + unset(_pastix_library) + set(PASTIX_LIBRARIES PkgConfig::PASTIX6) + set(PASTIX_LIBRARIES_DEP PkgConfig::PASTIX6) endif() -if (PASTIX_LIBRARIES) - list(GET PASTIX_LIBRARIES 0 first_lib) - get_filename_component(first_lib_path "${first_lib}" PATH) - if (${first_lib_path} MATCHES "/lib(32|64)?$") - string(REGEX REPLACE "/lib(32|64)?$" "" not_cached_dir "${first_lib_path}") - set(PASTIX_DIR_FOUND "${not_cached_dir}" CACHE PATH "Installation directory of PASTIX library" FORCE) - else() - set(PASTIX_DIR_FOUND "${first_lib_path}" CACHE PATH "Installation directory of PASTIX library" FORCE) - endif() -endif() -mark_as_advanced(PASTIX_DIR) -mark_as_advanced(PASTIX_DIR_FOUND) - -# check that PASTIX has been found -# --------------------------------- include(FindPackageHandleStandardArgs) -find_package_handle_standard_args(PASTIX DEFAULT_MSG - PASTIX_LIBRARIES - PASTIX_WORKS) +find_package_handle_standard_args(PASTIX + REQUIRED_VARS PASTIX6_FOUND PASTIX_LIBRARIES + VERSION_VAR PASTIX_VERSION)
diff --git a/cmake/FindPTSCOTCH.cmake b/cmake/FindPTSCOTCH.cmake deleted file mode 100644 index 32587dc..0000000 --- a/cmake/FindPTSCOTCH.cmake +++ /dev/null
@@ -1,423 +0,0 @@ -### -# -# @copyright (c) 2009-2014 The University of Tennessee and The University -# of Tennessee Research Foundation. -# All rights reserved. -# @copyright (c) 2012-2016 Inria. All rights reserved. -# @copyright (c) 2012-2014 Bordeaux INP, CNRS (LaBRI UMR 5800), Inria, Univ. Bordeaux. All rights reserved. -# -### -# -# - Find PTSCOTCH include dirs and libraries -# Use this module by invoking find_package with the form: -# find_package(PTSCOTCH -# [REQUIRED] # Fail with error if ptscotch is not found -# [COMPONENTS <comp1> <comp2> ...] # dependencies -# ) -# -# PTSCOTCH depends on the following libraries: -# - Threads -# - MPI -# -# COMPONENTS can be some of the following: -# - ESMUMPS: to activate detection of PT-Scotch with the esmumps interface -# -# This module finds headers and ptscotch library. -# Results are reported in variables: -# PTSCOTCH_FOUND - True if headers and requested libraries were found -# PTSCOTCH_LINKER_FLAGS - list of required linker flags (excluding -l and -L) -# PTSCOTCH_INCLUDE_DIRS - ptscotch include directories -# PTSCOTCH_LIBRARY_DIRS - Link directories for ptscotch libraries -# PTSCOTCH_LIBRARIES - ptscotch component libraries to be linked -# PTSCOTCH_INCLUDE_DIRS_DEP - ptscotch + dependencies include directories -# PTSCOTCH_LIBRARY_DIRS_DEP - ptscotch + dependencies link directories -# PTSCOTCH_LIBRARIES_DEP - ptscotch libraries + dependencies -# PTSCOTCH_INTSIZE - Number of octets occupied by a SCOTCH_Num -# -# The user can give specific paths where to find the libraries adding cmake -# options at configure (ex: cmake path/to/project -DPTSCOTCH=path/to/ptscotch): -# PTSCOTCH_DIR - Where to find the base directory of ptscotch -# PTSCOTCH_INCDIR - Where to find the header files -# PTSCOTCH_LIBDIR - Where to find the library files -# The module can also look for the following environment variables if paths -# are not given as cmake variable: PTSCOTCH_DIR, PTSCOTCH_INCDIR, PTSCOTCH_LIBDIR -# SPDX-License-Identifier: BSD-3-Clause - -#============================================================================= -# Copyright 2012-2013 Inria -# Copyright 2012-2013 Emmanuel Agullo -# Copyright 2012-2013 Mathieu Faverge -# Copyright 2012 Cedric Castagnede -# Copyright 2013-2016 Florent Pruvost -# -# Distributed under the OSI-approved BSD License (the "License"); -# see accompanying file MORSE-Copyright.txt for details. -# -# This software is distributed WITHOUT ANY WARRANTY; without even the -# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -# See the License for more information. -#============================================================================= -# (To distribute this file outside of Morse, substitute the full -# License text for the above reference.) - -if (NOT PTSCOTCH_FOUND) - set(PTSCOTCH_DIR "" CACHE PATH "Installation directory of PTSCOTCH library") - if (NOT PTSCOTCH_FIND_QUIETLY) - message(STATUS "A cache variable, namely PTSCOTCH_DIR, has been set to specify the install directory of PTSCOTCH") - endif() -endif() - -# Set the version to find -set(PTSCOTCH_LOOK_FOR_ESMUMPS OFF) - -if( PTSCOTCH_FIND_COMPONENTS ) - foreach( component ${PTSCOTCH_FIND_COMPONENTS} ) - if (${component} STREQUAL "ESMUMPS") - # means we look for esmumps library - set(PTSCOTCH_LOOK_FOR_ESMUMPS ON) - endif() - endforeach() -endif() - -# PTSCOTCH depends on Threads, try to find it -include(CMakeFindDependencyMacro) -if (NOT THREADS_FOUND) - if (PTSCOTCH_FIND_REQUIRED) - find_dependency(Threads REQUIRED) - else() - find_dependency(Threads) - endif() -endif() - -# PTSCOTCH depends on MPI, try to find it -if (NOT MPI_FOUND) - if (PTSCOTCH_FIND_REQUIRED) - find_dependency(MPI REQUIRED) - else() - find_dependency(MPI) - endif() -endif() - -# Looking for include -# ------------------- - -# Add system include paths to search include -# ------------------------------------------ -unset(_inc_env) -set(ENV_PTSCOTCH_DIR "$ENV{PTSCOTCH_DIR}") -set(ENV_PTSCOTCH_INCDIR "$ENV{PTSCOTCH_INCDIR}") -if(ENV_PTSCOTCH_INCDIR) - list(APPEND _inc_env "${ENV_PTSCOTCH_INCDIR}") -elseif(ENV_PTSCOTCH_DIR) - list(APPEND _inc_env "${ENV_PTSCOTCH_DIR}") - list(APPEND _inc_env "${ENV_PTSCOTCH_DIR}/include") - list(APPEND _inc_env "${ENV_PTSCOTCH_DIR}/include/ptscotch") -else() - if(WIN32) - string(REPLACE ":" ";" _inc_env "$ENV{INCLUDE}") - else() - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{C_INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{CPATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - endif() -endif() -list(APPEND _inc_env "${CMAKE_PLATFORM_IMPLICIT_INCLUDE_DIRECTORIES}") -list(APPEND _inc_env "${CMAKE_C_IMPLICIT_INCLUDE_DIRECTORIES}") -list(REMOVE_DUPLICATES _inc_env) - - -# Try to find the ptscotch header in the given paths -# ------------------------------------------------- - -set(PTSCOTCH_hdrs_to_find "ptscotch.h;scotch.h") - -# call cmake macro to find the header path -if(PTSCOTCH_INCDIR) - foreach(ptscotch_hdr ${PTSCOTCH_hdrs_to_find}) - set(PTSCOTCH_${ptscotch_hdr}_DIRS "PTSCOTCH_${ptscotch_hdr}_DIRS-NOTFOUND") - find_path(PTSCOTCH_${ptscotch_hdr}_DIRS - NAMES ${ptscotch_hdr} - HINTS ${PTSCOTCH_INCDIR}) - mark_as_advanced(PTSCOTCH_${ptscotch_hdr}_DIRS) - endforeach() -else() - if(PTSCOTCH_DIR) - foreach(ptscotch_hdr ${PTSCOTCH_hdrs_to_find}) - set(PTSCOTCH_${ptscotch_hdr}_DIRS "PTSCOTCH_${ptscotch_hdr}_DIRS-NOTFOUND") - find_path(PTSCOTCH_${ptscotch_hdr}_DIRS - NAMES ${ptscotch_hdr} - HINTS ${PTSCOTCH_DIR} - PATH_SUFFIXES "include" "include/scotch") - mark_as_advanced(PTSCOTCH_${ptscotch_hdr}_DIRS) - endforeach() - else() - foreach(ptscotch_hdr ${PTSCOTCH_hdrs_to_find}) - set(PTSCOTCH_${ptscotch_hdr}_DIRS "PTSCOTCH_${ptscotch_hdr}_DIRS-NOTFOUND") - find_path(PTSCOTCH_${ptscotch_hdr}_DIRS - NAMES ${ptscotch_hdr} - HINTS ${_inc_env} - PATH_SUFFIXES "scotch") - mark_as_advanced(PTSCOTCH_${ptscotch_hdr}_DIRS) - endforeach() - endif() -endif() - -# If found, add path to cmake variable -# ------------------------------------ -foreach(ptscotch_hdr ${PTSCOTCH_hdrs_to_find}) - if (PTSCOTCH_${ptscotch_hdr}_DIRS) - list(APPEND PTSCOTCH_INCLUDE_DIRS "${PTSCOTCH_${ptscotch_hdr}_DIRS}") - else () - if (NOT PTSCOTCH_FIND_QUIETLY) - message(STATUS "Looking for ptscotch -- ${ptscotch_hdr} not found") - endif() - endif() -endforeach() -list(REMOVE_DUPLICATES PTSCOTCH_INCLUDE_DIRS) - -# Looking for lib -# --------------- - -# Add system library paths to search lib -# -------------------------------------- -unset(_lib_env) -set(ENV_PTSCOTCH_LIBDIR "$ENV{PTSCOTCH_LIBDIR}") -if(ENV_PTSCOTCH_LIBDIR) - list(APPEND _lib_env "${ENV_PTSCOTCH_LIBDIR}") -elseif(ENV_PTSCOTCH_DIR) - list(APPEND _lib_env "${ENV_PTSCOTCH_DIR}") - list(APPEND _lib_env "${ENV_PTSCOTCH_DIR}/lib") -else() - if(WIN32) - string(REPLACE ":" ";" _lib_env "$ENV{LIB}") - else() - if(APPLE) - string(REPLACE ":" ";" _lib_env "$ENV{DYLD_LIBRARY_PATH}") - else() - string(REPLACE ":" ";" _lib_env "$ENV{LD_LIBRARY_PATH}") - endif() - list(APPEND _lib_env "${CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES}") - list(APPEND _lib_env "${CMAKE_C_IMPLICIT_LINK_DIRECTORIES}") - endif() -endif() -list(REMOVE_DUPLICATES _lib_env) - -# Try to find the ptscotch lib in the given paths -# ---------------------------------------------- - -set(PTSCOTCH_libs_to_find "ptscotch;ptscotcherr") -if (PTSCOTCH_LOOK_FOR_ESMUMPS) - list(INSERT PTSCOTCH_libs_to_find 0 "ptesmumps") - list(APPEND PTSCOTCH_libs_to_find "esmumps" ) -endif() -list(APPEND PTSCOTCH_libs_to_find "scotch;scotcherr") - -# call cmake macro to find the lib path -if(PTSCOTCH_LIBDIR) - foreach(ptscotch_lib ${PTSCOTCH_libs_to_find}) - set(PTSCOTCH_${ptscotch_lib}_LIBRARY "PTSCOTCH_${ptscotch_lib}_LIBRARY-NOTFOUND") - find_library(PTSCOTCH_${ptscotch_lib}_LIBRARY - NAMES ${ptscotch_lib} - HINTS ${PTSCOTCH_LIBDIR}) - endforeach() -else() - if(PTSCOTCH_DIR) - foreach(ptscotch_lib ${PTSCOTCH_libs_to_find}) - set(PTSCOTCH_${ptscotch_lib}_LIBRARY "PTSCOTCH_${ptscotch_lib}_LIBRARY-NOTFOUND") - find_library(PTSCOTCH_${ptscotch_lib}_LIBRARY - NAMES ${ptscotch_lib} - HINTS ${PTSCOTCH_DIR} - PATH_SUFFIXES lib lib32 lib64) - endforeach() - else() - foreach(ptscotch_lib ${PTSCOTCH_libs_to_find}) - set(PTSCOTCH_${ptscotch_lib}_LIBRARY "PTSCOTCH_${ptscotch_lib}_LIBRARY-NOTFOUND") - find_library(PTSCOTCH_${ptscotch_lib}_LIBRARY - NAMES ${ptscotch_lib} - HINTS ${_lib_env}) - endforeach() - endif() -endif() - -set(PTSCOTCH_LIBRARIES "") -set(PTSCOTCH_LIBRARY_DIRS "") -# If found, add path to cmake variable -# ------------------------------------ -foreach(ptscotch_lib ${PTSCOTCH_libs_to_find}) - - if (PTSCOTCH_${ptscotch_lib}_LIBRARY) - get_filename_component(${ptscotch_lib}_lib_path "${PTSCOTCH_${ptscotch_lib}_LIBRARY}" PATH) - # set cmake variables - list(APPEND PTSCOTCH_LIBRARIES "${PTSCOTCH_${ptscotch_lib}_LIBRARY}") - list(APPEND PTSCOTCH_LIBRARY_DIRS "${${ptscotch_lib}_lib_path}") - else () - if (NOT PTSCOTCH_FIND_QUIETLY) - message(STATUS "Looking for ptscotch -- lib ${ptscotch_lib} not found") - endif() - endif () - - mark_as_advanced(PTSCOTCH_${ptscotch_lib}_LIBRARY) - -endforeach() -list(REMOVE_DUPLICATES PTSCOTCH_LIBRARY_DIRS) - -# check a function to validate the find -if(PTSCOTCH_LIBRARIES) - - set(REQUIRED_LDFLAGS) - set(REQUIRED_INCDIRS) - set(REQUIRED_LIBDIRS) - set(REQUIRED_LIBS) - - # PTSCOTCH - if (PTSCOTCH_INCLUDE_DIRS) - set(REQUIRED_INCDIRS "${PTSCOTCH_INCLUDE_DIRS}") - endif() - if (PTSCOTCH_LIBRARY_DIRS) - set(REQUIRED_LIBDIRS "${PTSCOTCH_LIBRARY_DIRS}") - endif() - set(REQUIRED_LIBS "${PTSCOTCH_LIBRARIES}") - # MPI - if (MPI_FOUND) - if (MPI_C_INCLUDE_PATH) - list(APPEND CMAKE_REQUIRED_INCLUDES "${MPI_C_INCLUDE_PATH}") - endif() - if (MPI_C_LINK_FLAGS) - if (${MPI_C_LINK_FLAGS} MATCHES " -") - string(REGEX REPLACE " -" "-" MPI_C_LINK_FLAGS ${MPI_C_LINK_FLAGS}) - endif() - list(APPEND REQUIRED_LDFLAGS "${MPI_C_LINK_FLAGS}") - endif() - list(APPEND REQUIRED_LIBS "${MPI_C_LIBRARIES}") - endif() - # THREADS - if(CMAKE_THREAD_LIBS_INIT) - list(APPEND REQUIRED_LIBS "${CMAKE_THREAD_LIBS_INIT}") - endif() - set(Z_LIBRARY "Z_LIBRARY-NOTFOUND") - find_library(Z_LIBRARY NAMES z) - mark_as_advanced(Z_LIBRARY) - if(Z_LIBRARY) - list(APPEND REQUIRED_LIBS "-lz") - endif() - set(M_LIBRARY "M_LIBRARY-NOTFOUND") - find_library(M_LIBRARY NAMES m) - mark_as_advanced(M_LIBRARY) - if(M_LIBRARY) - list(APPEND REQUIRED_LIBS "-lm") - endif() - set(RT_LIBRARY "RT_LIBRARY-NOTFOUND") - find_library(RT_LIBRARY NAMES rt) - mark_as_advanced(RT_LIBRARY) - if(RT_LIBRARY) - list(APPEND REQUIRED_LIBS "-lrt") - endif() - - # set required libraries for link - set(CMAKE_REQUIRED_INCLUDES "${REQUIRED_INCDIRS}") - set(CMAKE_REQUIRED_LIBRARIES) - list(APPEND CMAKE_REQUIRED_LIBRARIES "${REQUIRED_LDFLAGS}") - foreach(lib_dir ${REQUIRED_LIBDIRS}) - list(APPEND CMAKE_REQUIRED_LIBRARIES "-L${lib_dir}") - endforeach() - list(APPEND CMAKE_REQUIRED_LIBRARIES "${REQUIRED_LIBS}") - list(APPEND CMAKE_REQUIRED_FLAGS "${REQUIRED_FLAGS}") - string(REGEX REPLACE "^ -" "-" CMAKE_REQUIRED_LIBRARIES "${CMAKE_REQUIRED_LIBRARIES}") - - # test link - unset(PTSCOTCH_WORKS CACHE) - include(CheckFunctionExists) - check_function_exists(SCOTCH_dgraphInit PTSCOTCH_WORKS) - mark_as_advanced(PTSCOTCH_WORKS) - - if(PTSCOTCH_WORKS) - # save link with dependencies - set(PTSCOTCH_LIBRARIES_DEP "${REQUIRED_LIBS}") - set(PTSCOTCH_LIBRARY_DIRS_DEP "${REQUIRED_LIBDIRS}") - set(PTSCOTCH_INCLUDE_DIRS_DEP "${REQUIRED_INCDIRS}") - set(PTSCOTCH_LINKER_FLAGS "${REQUIRED_LDFLAGS}") - list(REMOVE_DUPLICATES PTSCOTCH_LIBRARY_DIRS_DEP) - list(REMOVE_DUPLICATES PTSCOTCH_INCLUDE_DIRS_DEP) - list(REMOVE_DUPLICATES PTSCOTCH_LINKER_FLAGS) - else() - if(NOT PTSCOTCH_FIND_QUIETLY) - message(STATUS "Looking for PTSCOTCH : test of SCOTCH_dgraphInit with PTSCOTCH library fails") - message(STATUS "CMAKE_REQUIRED_LIBRARIES: ${CMAKE_REQUIRED_LIBRARIES}") - message(STATUS "CMAKE_REQUIRED_INCLUDES: ${CMAKE_REQUIRED_INCLUDES}") - message(STATUS "Check in CMakeFiles/CMakeError.log to figure out why it fails") - endif() - endif() - set(CMAKE_REQUIRED_INCLUDES) - set(CMAKE_REQUIRED_FLAGS) - set(CMAKE_REQUIRED_LIBRARIES) -endif() - -if (PTSCOTCH_LIBRARIES) - list(GET PTSCOTCH_LIBRARIES 0 first_lib) - get_filename_component(first_lib_path "${first_lib}" PATH) - if (${first_lib_path} MATCHES "/lib(32|64)?$") - string(REGEX REPLACE "/lib(32|64)?$" "" not_cached_dir "${first_lib_path}") - set(PTSCOTCH_DIR_FOUND "${not_cached_dir}" CACHE PATH "Installation directory of PTSCOTCH library" FORCE) - else() - set(PTSCOTCH_DIR_FOUND "${first_lib_path}" CACHE PATH "Installation directory of PTSCOTCH library" FORCE) - endif() -endif() -mark_as_advanced(PTSCOTCH_DIR) -mark_as_advanced(PTSCOTCH_DIR_FOUND) - -# Check the size of SCOTCH_Num -# --------------------------------- -set(CMAKE_REQUIRED_INCLUDES ${PTSCOTCH_INCLUDE_DIRS}) - -include(CheckCSourceRuns) -#stdio.h and stdint.h should be included by scotch.h directly -set(PTSCOTCH_C_TEST_SCOTCH_Num_4 " -#include <stdio.h> -#include <stdint.h> -#include <ptscotch.h> -int main(int argc, char **argv) { - if (sizeof(SCOTCH_Num) == 4) - return 0; - else - return 1; -} -") - -set(PTSCOTCH_C_TEST_SCOTCH_Num_8 " -#include <stdio.h> -#include <stdint.h> -#include <ptscotch.h> -int main(int argc, char **argv) { - if (sizeof(SCOTCH_Num) == 8) - return 0; - else - return 1; -} -") -check_c_source_runs("${PTSCOTCH_C_TEST_SCOTCH_Num_4}" PTSCOTCH_Num_4) -if(NOT PTSCOTCH_Num_4) - check_c_source_runs("${PTSCOTCH_C_TEST_SCOTCH_Num_8}" PTSCOTCH_Num_8) - if(NOT PTSCOTCH_Num_8) - set(PTSCOTCH_INTSIZE -1) - else() - set(PTSCOTCH_INTSIZE 8) - endif() -else() - set(PTSCOTCH_INTSIZE 4) -endif() -set(CMAKE_REQUIRED_INCLUDES "") - -# check that PTSCOTCH has been found -# --------------------------------- -include(FindPackageHandleStandardArgs) -find_package_handle_standard_args(PTSCOTCH DEFAULT_MSG - PTSCOTCH_LIBRARIES - PTSCOTCH_WORKS) -# -# TODO: Add possibility to check for specific functions in the library -#
diff --git a/cmake/FindSCOTCH.cmake b/cmake/FindSCOTCH.cmake deleted file mode 100644 index e3a7b0f..0000000 --- a/cmake/FindSCOTCH.cmake +++ /dev/null
@@ -1,371 +0,0 @@ -### -# -# @copyright (c) 2009-2014 The University of Tennessee and The University -# of Tennessee Research Foundation. -# All rights reserved. -# @copyright (c) 2012-2014 Inria. All rights reserved. -# @copyright (c) 2012-2014 Bordeaux INP, CNRS (LaBRI UMR 5800), Inria, Univ. Bordeaux. All rights reserved. -# -### -# -# - Find SCOTCH include dirs and libraries -# Use this module by invoking find_package with the form: -# find_package(SCOTCH -# [REQUIRED] # Fail with error if scotch is not found -# [COMPONENTS <comp1> <comp2> ...] # dependencies -# ) -# -# COMPONENTS can be some of the following: -# - ESMUMPS: to activate detection of Scotch with the esmumps interface -# -# This module finds headers and scotch library. -# Results are reported in variables: -# SCOTCH_FOUND - True if headers and requested libraries were found -# SCOTCH_INCLUDE_DIRS - scotch include directories -# SCOTCH_LIBRARY_DIRS - Link directories for scotch libraries -# SCOTCH_LIBRARIES - scotch component libraries to be linked -# SCOTCH_INTSIZE - Number of octets occupied by a SCOTCH_Num -# -# The user can give specific paths where to find the libraries adding cmake -# options at configure (ex: cmake path/to/project -DSCOTCH=path/to/scotch): -# SCOTCH_DIR - Where to find the base directory of scotch -# SCOTCH_INCDIR - Where to find the header files -# SCOTCH_LIBDIR - Where to find the library files -# The module can also look for the following environment variables if paths -# are not given as cmake variable: SCOTCH_DIR, SCOTCH_INCDIR, SCOTCH_LIBDIR -# SPDX-License-Identifier: BSD-3-Clause - -#============================================================================= -# Copyright 2012-2013 Inria -# Copyright 2012-2013 Emmanuel Agullo -# Copyright 2012-2013 Mathieu Faverge -# Copyright 2012 Cedric Castagnede -# Copyright 2013 Florent Pruvost -# -# Distributed under the OSI-approved BSD License (the "License"); -# see accompanying file MORSE-Copyright.txt for details. -# -# This software is distributed WITHOUT ANY WARRANTY; without even the -# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -# See the License for more information. -#============================================================================= -# (To distribute this file outside of Morse, substitute the full -# License text for the above reference.) - -if (NOT SCOTCH_FOUND) - set(SCOTCH_DIR "" CACHE PATH "Installation directory of SCOTCH library") - if (NOT SCOTCH_FIND_QUIETLY) - message(STATUS "A cache variable, namely SCOTCH_DIR, has been set to specify the install directory of SCOTCH") - endif() -endif() - -# Set the version to find -set(SCOTCH_LOOK_FOR_ESMUMPS OFF) - -if( SCOTCH_FIND_COMPONENTS ) - foreach( component ${SCOTCH_FIND_COMPONENTS} ) - if (${component} STREQUAL "ESMUMPS") - # means we look for esmumps library - set(SCOTCH_LOOK_FOR_ESMUMPS ON) - endif() - endforeach() -endif() - -# SCOTCH may depend on Threads, try to find it -include(CMakeFindDependencyMacro) -if (NOT THREADS_FOUND) - if (SCOTCH_FIND_REQUIRED) - find_dependency(Threads REQUIRED) - else() - find_dependency(Threads) - endif() -endif() - -# Looking for include -# ------------------- - -# Add system include paths to search include -# ------------------------------------------ -unset(_inc_env) -set(ENV_SCOTCH_DIR "$ENV{SCOTCH_DIR}") -set(ENV_SCOTCH_INCDIR "$ENV{SCOTCH_INCDIR}") -if(ENV_SCOTCH_INCDIR) - list(APPEND _inc_env "${ENV_SCOTCH_INCDIR}") -elseif(ENV_SCOTCH_DIR) - list(APPEND _inc_env "${ENV_SCOTCH_DIR}") - list(APPEND _inc_env "${ENV_SCOTCH_DIR}/include") - list(APPEND _inc_env "${ENV_SCOTCH_DIR}/include/scotch") -else() - if(WIN32) - string(REPLACE ":" ";" _inc_env "$ENV{INCLUDE}") - else() - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{C_INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{CPATH}") - list(APPEND _inc_env "${_path_env}") - string(REPLACE ":" ";" _path_env "$ENV{INCLUDE_PATH}") - list(APPEND _inc_env "${_path_env}") - endif() -endif() -list(APPEND _inc_env "${CMAKE_PLATFORM_IMPLICIT_INCLUDE_DIRECTORIES}") -list(APPEND _inc_env "${CMAKE_C_IMPLICIT_INCLUDE_DIRECTORIES}") -list(REMOVE_DUPLICATES _inc_env) - - -# Try to find the scotch header in the given paths -# ------------------------------------------------- -# call cmake macro to find the header path -if(SCOTCH_INCDIR) - set(SCOTCH_scotch.h_DIRS "SCOTCH_scotch.h_DIRS-NOTFOUND") - find_path(SCOTCH_scotch.h_DIRS - NAMES scotch.h - HINTS ${SCOTCH_INCDIR}) -else() - if(SCOTCH_DIR) - set(SCOTCH_scotch.h_DIRS "SCOTCH_scotch.h_DIRS-NOTFOUND") - find_path(SCOTCH_scotch.h_DIRS - NAMES scotch.h - HINTS ${SCOTCH_DIR} - PATH_SUFFIXES "include" "include/scotch") - else() - set(SCOTCH_scotch.h_DIRS "SCOTCH_scotch.h_DIRS-NOTFOUND") - find_path(SCOTCH_scotch.h_DIRS - NAMES scotch.h - HINTS ${_inc_env} - PATH_SUFFIXES "scotch") - endif() -endif() -mark_as_advanced(SCOTCH_scotch.h_DIRS) - -# If found, add path to cmake variable -# ------------------------------------ -if (SCOTCH_scotch.h_DIRS) - set(SCOTCH_INCLUDE_DIRS "${SCOTCH_scotch.h_DIRS}") -else () - set(SCOTCH_INCLUDE_DIRS "SCOTCH_INCLUDE_DIRS-NOTFOUND") - if (NOT SCOTCH_FIND_QUIETLY) - message(STATUS "Looking for scotch -- scotch.h not found") - endif() -endif() -list(REMOVE_DUPLICATES SCOTCH_INCLUDE_DIRS) - -# Looking for lib -# --------------- - -# Add system library paths to search lib -# -------------------------------------- -unset(_lib_env) -set(ENV_SCOTCH_LIBDIR "$ENV{SCOTCH_LIBDIR}") -if(ENV_SCOTCH_LIBDIR) - list(APPEND _lib_env "${ENV_SCOTCH_LIBDIR}") -elseif(ENV_SCOTCH_DIR) - list(APPEND _lib_env "${ENV_SCOTCH_DIR}") - list(APPEND _lib_env "${ENV_SCOTCH_DIR}/lib") -else() - if(WIN32) - string(REPLACE ":" ";" _lib_env "$ENV{LIB}") - else() - if(APPLE) - string(REPLACE ":" ";" _lib_env "$ENV{DYLD_LIBRARY_PATH}") - else() - string(REPLACE ":" ";" _lib_env "$ENV{LD_LIBRARY_PATH}") - endif() - list(APPEND _lib_env "${CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES}") - list(APPEND _lib_env "${CMAKE_C_IMPLICIT_LINK_DIRECTORIES}") - endif() -endif() -list(REMOVE_DUPLICATES _lib_env) - -# Try to find the scotch lib in the given paths -# ---------------------------------------------- - -set(SCOTCH_libs_to_find "scotch;scotcherrexit") -if (SCOTCH_LOOK_FOR_ESMUMPS) - list(INSERT SCOTCH_libs_to_find 0 "esmumps") -endif() - -# call cmake macro to find the lib path -if(SCOTCH_LIBDIR) - foreach(scotch_lib ${SCOTCH_libs_to_find}) - set(SCOTCH_${scotch_lib}_LIBRARY "SCOTCH_${scotch_lib}_LIBRARY-NOTFOUND") - find_library(SCOTCH_${scotch_lib}_LIBRARY - NAMES ${scotch_lib} - HINTS ${SCOTCH_LIBDIR}) - endforeach() -else() - if(SCOTCH_DIR) - foreach(scotch_lib ${SCOTCH_libs_to_find}) - set(SCOTCH_${scotch_lib}_LIBRARY "SCOTCH_${scotch_lib}_LIBRARY-NOTFOUND") - find_library(SCOTCH_${scotch_lib}_LIBRARY - NAMES ${scotch_lib} - HINTS ${SCOTCH_DIR} - PATH_SUFFIXES lib lib32 lib64) - endforeach() - else() - foreach(scotch_lib ${SCOTCH_libs_to_find}) - set(SCOTCH_${scotch_lib}_LIBRARY "SCOTCH_${scotch_lib}_LIBRARY-NOTFOUND") - find_library(SCOTCH_${scotch_lib}_LIBRARY - NAMES ${scotch_lib} - HINTS ${_lib_env}) - endforeach() - endif() -endif() - -set(SCOTCH_LIBRARIES "") -set(SCOTCH_LIBRARY_DIRS "") -# If found, add path to cmake variable -# ------------------------------------ -foreach(scotch_lib ${SCOTCH_libs_to_find}) - - if (SCOTCH_${scotch_lib}_LIBRARY) - get_filename_component(${scotch_lib}_lib_path "${SCOTCH_${scotch_lib}_LIBRARY}" PATH) - # set cmake variables - list(APPEND SCOTCH_LIBRARIES "${SCOTCH_${scotch_lib}_LIBRARY}") - list(APPEND SCOTCH_LIBRARY_DIRS "${${scotch_lib}_lib_path}") - else () - list(APPEND SCOTCH_LIBRARIES "${SCOTCH_${scotch_lib}_LIBRARY}") - if (NOT SCOTCH_FIND_QUIETLY) - message(STATUS "Looking for scotch -- lib ${scotch_lib} not found") - endif() - endif () - - mark_as_advanced(SCOTCH_${scotch_lib}_LIBRARY) - -endforeach() -list(REMOVE_DUPLICATES SCOTCH_LIBRARY_DIRS) - -# check a function to validate the find -if(SCOTCH_LIBRARIES) - - set(REQUIRED_INCDIRS) - set(REQUIRED_LIBDIRS) - set(REQUIRED_LIBS) - - # SCOTCH - if (SCOTCH_INCLUDE_DIRS) - set(REQUIRED_INCDIRS "${SCOTCH_INCLUDE_DIRS}") - endif() - if (SCOTCH_LIBRARY_DIRS) - set(REQUIRED_LIBDIRS "${SCOTCH_LIBRARY_DIRS}") - endif() - set(REQUIRED_LIBS "${SCOTCH_LIBRARIES}") - # THREADS - if(CMAKE_THREAD_LIBS_INIT) - list(APPEND REQUIRED_LIBS "${CMAKE_THREAD_LIBS_INIT}") - endif() - set(Z_LIBRARY "Z_LIBRARY-NOTFOUND") - find_library(Z_LIBRARY NAMES z) - mark_as_advanced(Z_LIBRARY) - if(Z_LIBRARY) - list(APPEND REQUIRED_LIBS "-lz") - endif() - set(M_LIBRARY "M_LIBRARY-NOTFOUND") - find_library(M_LIBRARY NAMES m) - mark_as_advanced(M_LIBRARY) - if(M_LIBRARY) - list(APPEND REQUIRED_LIBS "-lm") - endif() - set(RT_LIBRARY "RT_LIBRARY-NOTFOUND") - find_library(RT_LIBRARY NAMES rt) - mark_as_advanced(RT_LIBRARY) - if(RT_LIBRARY) - list(APPEND REQUIRED_LIBS "-lrt") - endif() - - # set required libraries for link - set(CMAKE_REQUIRED_INCLUDES "${REQUIRED_INCDIRS}") - set(CMAKE_REQUIRED_LIBRARIES) - foreach(lib_dir ${REQUIRED_LIBDIRS}) - list(APPEND CMAKE_REQUIRED_LIBRARIES "-L${lib_dir}") - endforeach() - list(APPEND CMAKE_REQUIRED_LIBRARIES "${REQUIRED_LIBS}") - string(REGEX REPLACE "^ -" "-" CMAKE_REQUIRED_LIBRARIES "${CMAKE_REQUIRED_LIBRARIES}") - - # test link - unset(SCOTCH_WORKS CACHE) - include(CheckFunctionExists) - check_function_exists(SCOTCH_graphInit SCOTCH_WORKS) - mark_as_advanced(SCOTCH_WORKS) - - if(SCOTCH_WORKS) - # save link with dependencies - set(SCOTCH_LIBRARIES "${REQUIRED_LIBS}") - else() - if(NOT SCOTCH_FIND_QUIETLY) - message(STATUS "Looking for SCOTCH : test of SCOTCH_graphInit with SCOTCH library fails") - message(STATUS "CMAKE_REQUIRED_LIBRARIES: ${CMAKE_REQUIRED_LIBRARIES}") - message(STATUS "CMAKE_REQUIRED_INCLUDES: ${CMAKE_REQUIRED_INCLUDES}") - message(STATUS "Check in CMakeFiles/CMakeError.log to figure out why it fails") - endif() - endif() - set(CMAKE_REQUIRED_INCLUDES) - set(CMAKE_REQUIRED_FLAGS) - set(CMAKE_REQUIRED_LIBRARIES) -endif() - -if (SCOTCH_LIBRARIES) - list(GET SCOTCH_LIBRARIES 0 first_lib) - get_filename_component(first_lib_path "${first_lib}" PATH) - if (${first_lib_path} MATCHES "/lib(32|64)?$") - string(REGEX REPLACE "/lib(32|64)?$" "" not_cached_dir "${first_lib_path}") - set(SCOTCH_DIR_FOUND "${not_cached_dir}" CACHE PATH "Installation directory of SCOTCH library" FORCE) - else() - set(SCOTCH_DIR_FOUND "${first_lib_path}" CACHE PATH "Installation directory of SCOTCH library" FORCE) - endif() -endif() -mark_as_advanced(SCOTCH_DIR) -mark_as_advanced(SCOTCH_DIR_FOUND) - -# Check the size of SCOTCH_Num -# --------------------------------- -set(CMAKE_REQUIRED_INCLUDES ${SCOTCH_INCLUDE_DIRS}) - -include(CheckCSourceRuns) -#stdio.h and stdint.h should be included by scotch.h directly -set(SCOTCH_C_TEST_SCOTCH_Num_4 " -#include <stdio.h> -#include <stdint.h> -#include <scotch.h> -int main(int argc, char **argv) { - if (sizeof(SCOTCH_Num) == 4) - return 0; - else - return 1; -} -") - -set(SCOTCH_C_TEST_SCOTCH_Num_8 " -#include <stdio.h> -#include <stdint.h> -#include <scotch.h> -int main(int argc, char **argv) { - if (sizeof(SCOTCH_Num) == 8) - return 0; - else - return 1; -} -") -check_c_source_runs("${SCOTCH_C_TEST_SCOTCH_Num_4}" SCOTCH_Num_4) -if(NOT SCOTCH_Num_4) - check_c_source_runs("${SCOTCH_C_TEST_SCOTCH_Num_8}" SCOTCH_Num_8) - if(NOT SCOTCH_Num_8) - set(SCOTCH_INTSIZE -1) - else() - set(SCOTCH_INTSIZE 8) - endif() -else() - set(SCOTCH_INTSIZE 4) -endif() -set(CMAKE_REQUIRED_INCLUDES "") - -# check that SCOTCH has been found -# --------------------------------- -include(FindPackageHandleStandardArgs) -find_package_handle_standard_args(SCOTCH DEFAULT_MSG - SCOTCH_LIBRARIES - SCOTCH_WORKS) -# -# TODO: Add possibility to check for specific functions in the library -#
diff --git a/contrib/Eigen/CMakeLists.txt b/contrib/Eigen/CMakeLists.txt index ecf39bc..3b4a566 100644 --- a/contrib/Eigen/CMakeLists.txt +++ b/contrib/Eigen/CMakeLists.txt
@@ -24,6 +24,7 @@ NumericalDiff OpenGLSupport Polynomials + Riccati SparseExtra SpecialFunctions Splines
diff --git a/contrib/Eigen/GPU b/contrib/Eigen/GPU index 9e55590..233e9aa 100644 --- a/contrib/Eigen/GPU +++ b/contrib/Eigen/GPU
@@ -17,8 +17,8 @@ /** \defgroup GPU_Module GPU module * - * GPU-accelerated dense linear algebra using NVIDIA CUDA libraries - * (cuSOLVER and cuBLAS). + * GPU-accelerated dense and sparse linear algebra using NVIDIA CUDA libraries + * (cuBLAS, cuSOLVER, cuSPARSE, cuFFT, NPP, and optionally cuDSS). * * This module provides explicit GPU solver classes that coexist with Eigen's * CPU solvers. Unlike the LAPACKE dispatch (which replaces the CPU @@ -37,7 +37,7 @@ * auto X = llt_gpu.solve(B); // only B transferred per solve * \endcode * - * Requires CUDA 11.4+. + * Requires CUDA 11.8+. */ #ifdef EIGEN_USE_GPU
diff --git a/contrib/Eigen/NumericalDiff b/contrib/Eigen/NumericalDiff index ac98ce3..81fec45 100644 --- a/contrib/Eigen/NumericalDiff +++ b/contrib/Eigen/NumericalDiff
@@ -17,6 +17,7 @@ /** * \defgroup NumericalDiff_Module Numerical differentiation module + * \brief Finite-difference Jacobians and differentiation weights. * * \code * #include <contrib/Eigen/NumericalDiff> @@ -28,19 +29,13 @@ * is a different method and has its own module in Eigen : \ref * AutoDiff_Module. * - * NumericalDiff implements the "Forward" and "Central" schemes. Stencil - * computes finite-difference weights for arbitrary fixed-size grids and - * derivative orders; the caller supplies the points and chooses the - * step size. The "Forward" and "Central" schemes are basic methods, and - * there exist some more elaborated way of computing such approximates. - * They are implemented using both proprietary and free software, and - * usually requires linking to an external library. It is very easy for - * you to write a functor using such software, and the purpose is quite - * orthogonal to what we want to achieve with Eigen. - * - * This is why we will not provide wrappers for every great numerical - * differentiation software that exist, but should rather stick with those - * basic ones, that still are useful for testing. + * This module provides the following classes: + * - \ref NumericalDiff extends a functor with Jacobian computation using + * \c Forward or \c Central finite differences at a fixed step. + * - \ref Ridders stores a functor and computes its Jacobian using adaptive + * central differences and Richardson extrapolation, with optional error estimates. + * - \ref Stencil computes finite-difference weights for arbitrary fixed-size + * grids and derivative orders, using points and a step size chosen by the caller. * * Also, the \ref NonLinearOptimization_Module needs this in order to * provide full features compatibility with the original (c)minpack @@ -56,6 +51,7 @@ // IWYU pragma: begin_exports #include "src/NumericalDiff/NumericalDiff.h" #include "src/NumericalDiff/Stencil.h" +#include "src/NumericalDiff/Ridders.h" // IWYU pragma: end_exports #include "../../Eigen/src/Core/util/ReenableStupidWarnings.h"
diff --git a/contrib/Eigen/Riccati b/contrib/Eigen/Riccati new file mode 100644 index 0000000..61e1286 --- /dev/null +++ b/contrib/Eigen/Riccati
@@ -0,0 +1,49 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// This Source Code Form is subject to the terms of the Mozilla +// Public License v. 2.0. If a copy of the MPL was not distributed +// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_RICCATI_MODULE_H +#define EIGEN_RICCATI_MODULE_H + +#include "../../Eigen/Core" +#include "../../Eigen/LU" +#include "../../Eigen/QR" +#include "../../Eigen/Eigenvalues" + +#include <limits> + +namespace Eigen { + +/** + * \defgroup Riccati_Module Riccati module + * + * This module provides solvers for algebraic Riccati equations, as they arise in optimal control (LQR) and + * estimation (Kalman filtering): + * - DiscreteAlgebraicRiccatiSolver for the (generalized) discrete-time algebraic Riccati equation. + * + * \code + * #include <contrib/Eigen/Riccati> + * \endcode + * + * \note DiscreteAlgebraicRiccatiSolver incorporates code adapted from SciPy (BSD-3-Clause) and LAPACK + * (BSD-3-Clause-Open-MPI). Binary redistributions of software that uses it must reproduce the copyright + * notices, conditions and disclaimers in contrib/Eigen/src/Riccati/DiscreteAlgebraicRiccatiSolver.h. + */ + +} // namespace Eigen + +#include "../../Eigen/src/Core/util/DisableStupidWarnings.h" + +// IWYU pragma: begin_exports +#include "src/Riccati/GeneralizedSchurReordering.h" +#include "src/Riccati/DiscreteAlgebraicRiccatiSolver.h" +// IWYU pragma: end_exports + +#include "../../Eigen/src/Core/util/ReenableStupidWarnings.h" + +#endif // EIGEN_RICCATI_MODULE_H
diff --git a/contrib/Eigen/src/AutoDiff/AutoDiffScalar.h b/contrib/Eigen/src/AutoDiff/AutoDiffScalar.h index 382edac..b19bddd 100644 --- a/contrib/Eigen/src/AutoDiff/AutoDiffScalar.h +++ b/contrib/Eigen/src/AutoDiff/AutoDiffScalar.h
@@ -85,6 +85,7 @@ * - std::abs, std::sqrt, std::pow, std::exp, std::log, std::sin, std::cos, * - internal::abs, internal::sqrt, numext::pow, internal::exp, internal::log, internal::sin, internal::cos, * - internal::conj, internal::real, internal::imag, numext::abs2. + * - atan, log1p, expm1, log2, log10, cbrt, asinh, acosh, and atanh, through argument-dependent lookup or numext. * * AutoDiffScalar can be used as the scalar type of an Eigen::Matrix object. However, * in that case, the expression template mechanism only occurs at the top Matrix level, @@ -530,6 +531,22 @@ EIGEN_AUTODIFF_DECLARE_GLOBAL_UNARY(tan, using std::tan; using std::cos; return Eigen::MakeAutoDiffScalar( tan(x.value()), x.derivatives() * (Scalar(1) / numext::abs2(cos(x.value()))));) +template <typename DerType> +inline auto atan(const AutoDiffScalar<DerType>& x) { + using Scalar = typename AutoDiffScalar<DerType>::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + const Scalar& value = x.value(); + Scalar scale(1), denominator; + if (numext::abs(value) > RealScalar(1)) { + // dx / x / (x + 1/x) avoids overflowing x*x or underflowing its reciprocal before multiplying dx. + scale = value; + denominator = value + Scalar(1) / value; + } else { + denominator = Scalar(1) + value * value; + } + return MakeAutoDiffScalar(numext::atan(value), (x.derivatives() / scale) / denominator); +} + EIGEN_AUTODIFF_DECLARE_GLOBAL_UNARY(asin, using std::sqrt; using std::asin; return Eigen::MakeAutoDiffScalar( asin(x.value()), x.derivatives() * (Scalar(1) / sqrt(1 - numext::abs2(x.value()))));) @@ -550,6 +567,74 @@ return Eigen::MakeAutoDiffScalar(cosh(x.value()), x.derivatives() * sinh(x.value()));) +EIGEN_AUTODIFF_DECLARE_GLOBAL_UNARY(log1p, return MakeAutoDiffScalar(numext::log1p(x.value()), + x.derivatives() / (Scalar(1) + x.value()));) + +EIGEN_AUTODIFF_DECLARE_GLOBAL_UNARY(expm1, return MakeAutoDiffScalar(numext::expm1(x.value()), + x.derivatives() * numext::exp(x.value()));) + +EIGEN_AUTODIFF_DECLARE_GLOBAL_UNARY(log2, + return MakeAutoDiffScalar(numext::log2(x.value()), + (x.derivatives() / x.value()) / numext::log(Scalar(2)));) + +template <typename DerType> +inline auto log10(const AutoDiffScalar<DerType>& x) { + using Scalar = typename AutoDiffScalar<DerType>::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + const Scalar logarithm = numext::log(Scalar(10)); + EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::IsComplex) { + const bool below_one = numext::abs(x.value()) < RealScalar(1); + const Scalar first = below_one ? logarithm : x.value(); + const Scalar second = below_one ? x.value() : logarithm; + return MakeAutoDiffScalar(numext::log10(x.value()), (x.derivatives() / first) / second); + } + const RealScalar magnitude = numext::abs(x.value()); + const bool subnormal = magnitude < RealScalar((numext::numeric_limits<RealScalar>::min)()); + const bool combine = magnitude < RealScalar(1) && !subnormal; + // Combine normal small x with log(10) before dividing a tiny seed. For subnormal x, + // (x/eps)*(log(10)*eps) keeps both factors normal without rounding away the seed. + const Scalar scale = subnormal ? Scalar(NumTraits<Scalar>::epsilon()) : Scalar(1); + const Scalar first = (x.value() * (combine ? logarithm : Scalar(1))) / scale; + const Scalar second = (combine ? Scalar(1) : logarithm) * scale; + return MakeAutoDiffScalar(numext::log10(x.value()), (x.derivatives() / first) / second); +} + +EIGEN_AUTODIFF_DECLARE_GLOBAL_UNARY(cbrt, const Scalar root = numext::cbrt(x.value()); + return MakeAutoDiffScalar(root, x.derivatives() / (Scalar(3) * root * root));) + +EIGEN_AUTODIFF_DECLARE_GLOBAL_UNARY(asinh, + return MakeAutoDiffScalar(numext::asinh(x.value()), + x.derivatives() / numext::hypot(Scalar(1), x.value()));) + +template <typename DerType> +inline auto acosh(const AutoDiffScalar<DerType>& x) { + using Scalar = typename AutoDiffScalar<DerType>::Scalar; + // Separate square roots avoid overflowing x*x for large x. + const Scalar lower = numext::sqrt(Scalar(x.value() - Scalar(1))); + const Scalar upper = numext::sqrt(Scalar(x.value() + Scalar(1))); + return MakeAutoDiffScalar(numext::acosh(x.value()), (x.derivatives() / upper) / lower); +} + +template <typename DerType> +inline auto atanh(const AutoDiffScalar<DerType>& x) { + using Scalar = typename AutoDiffScalar<DerType>::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + const Scalar lower = Scalar(1) - x.value(); + const Scalar upper = Scalar(1) + x.value(); + const bool lower_first = numext::abs(lower) > numext::abs(upper); + const Scalar larger = lower_first ? lower : upper; + const Scalar smaller = lower_first ? upper : lower; + EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::IsComplex) { + return MakeAutoDiffScalar(numext::atanh(x.value()), (x.derivatives() / larger) / smaller); + } + // For real |x| <= 1 the product is in [0,1]: divide by it before rounding a tiny seed. + // Outside that interval, retain separate factors to avoid overflowing their product. + const bool combine = numext::abs(x.value()) <= RealScalar(1); + const Scalar first = larger * (combine ? smaller : Scalar(1)); + const Scalar second = combine ? Scalar(1) : smaller; + return MakeAutoDiffScalar(numext::atanh(x.value()), (x.derivatives() / first) / second); +} + #undef EIGEN_AUTODIFF_DECLARE_GLOBAL_UNARY template <typename DerType> @@ -567,6 +652,21 @@ }; namespace internal { +template <typename LhsDerivatives, typename RhsDerivatives> +struct functor_is_commutative<scalar_product_op<AutoDiffScalar<LhsDerivatives>, AutoDiffScalar<RhsDerivatives>>> + : functor_is_commutative<scalar_product_op<typename remove_all_t<LhsDerivatives>::Scalar, + typename remove_all_t<RhsDerivatives>::Scalar>> {}; + +template <typename Derivatives> +struct functor_is_commutative< + scalar_product_op<AutoDiffScalar<Derivatives>, typename remove_all_t<Derivatives>::Scalar>> + : functor_is_commutative<scalar_product_op<typename remove_all_t<Derivatives>::Scalar>> {}; + +template <typename Derivatives> +struct functor_is_commutative< + scalar_product_op<typename remove_all_t<Derivatives>::Scalar, AutoDiffScalar<Derivatives>>> + : functor_is_commutative<scalar_product_op<typename remove_all_t<Derivatives>::Scalar>> {}; + template <typename DerivativeType> struct is_identically_zero_impl<AutoDiffScalar<DerivativeType>> { static inline bool run(const AutoDiffScalar<DerivativeType>& s) {
diff --git a/contrib/Eigen/src/GPU/CuDssSupport.h b/contrib/Eigen/src/GPU/CuDssSupport.h index a718a8e..d41ab2d 100644 --- a/contrib/Eigen/src/GPU/CuDssSupport.h +++ b/contrib/Eigen/src/GPU/CuDssSupport.h
@@ -43,10 +43,12 @@ } while (0) // cuDSS 0.8 changed cudssMatrixCreateDn/Csr from cudaDataType_t to the -// value-compatible cudssDataType_t (CUDSS_R_32F == CUDA_R_32F, ...). Map the -// module's cudaDataType_t traits through this alias so both API generations -// compile. -#if defined(CUDSS_VERSION) && CUDSS_VERSION >= 800 +// value-compatible cudssDataType_t (CUDSS_R_32F == CUDA_R_32F, ...); cuDSS 1.0 +// changed them back and kept cudssDataType_t only as a deprecated typedef of +// cudaDataType_t. Map the module's cudaDataType_t traits through this alias so +// all three API generations compile without deprecation warnings. CUDSS_VERSION +// is major * 10000 + minor * 100 + patch: 0.8.0 is 800, 1.0.0 is 10000. +#if defined(CUDSS_VERSION) && CUDSS_VERSION >= 800 && CUDSS_VERSION < 10000 using cudss_value_type_t = cudssDataType_t; #else using cudss_value_type_t = cudaDataType_t;
diff --git a/contrib/Eigen/src/GPU/README.md b/contrib/Eigen/src/GPU/README.md index 07e2fdf..815e500 100644 --- a/contrib/Eigen/src/GPU/README.md +++ b/contrib/Eigen/src/GPU/README.md
@@ -1,7 +1,7 @@ # Eigen GPU Module (`contrib/Eigen/GPU`) GPU-accelerated linear algebra for Eigen users, dispatching to NVIDIA CUDA -Math Libraries (cuBLAS, cuSOLVER, cuFFT, cuSPARSE, cuDSS). Requires CUDA 11.4+; +Math Libraries (cuBLAS, cuSOLVER, cuFFT, cuSPARSE, cuDSS). Requires CUDA 11.8+; cuDSS features require CUDA 12.0+ and a separate cuDSS install. Header-only. This module dispatches rather than reimplements, so numerical behavior, @@ -21,26 +21,19 @@ ## Why this module -Eigen is the linear algebra foundation for a large ecosystem of C++ projects -in robotics (ROS, Drake, MoveIt, Pinocchio), computer vision (OpenCV, COLMAP, -Open3D), scientific computing (Ceres, Stan), and beyond. Many of these -projects run on GPU-equipped hardware but cannot use GPUs for Eigen operations -without dropping down to raw CUDA library APIs. +Eigen's own products and solvers run on the CPU. Code that keeps its data in +Eigen types and wants these operations on a GPU otherwise has to call the CUDA +libraries directly and convert to and from their formats. This module wraps +those calls in types that follow Eigen's API. It provides matrix products +(cuBLAS), dense solvers (cuSOLVER), sparse Cholesky, LDL^T, and LU factorization +(cuDSS), FFT (cuFFT), and sparse matrix-vector products (cuSPARSE). -GPU sparse solvers are a particularly acute gap. Sparse factorization is the -bottleneck in SLAM, bundle adjustment, FEM, and nonlinear optimization -- -exactly the workloads where GPU acceleration matters most. Downstream projects -like [Ceres](https://github.com/ceres-solver/ceres-solver/issues/1151) and -[COLMAP](https://github.com/colmap/colmap/issues/4018) have open requests for -GPU-accelerated sparse solvers, and third-party projects like -[cholespy](https://github.com/rgl-epfl/cholespy) exist specifically because -Eigen lacks them. The `contrib/Eigen/GPU` module provides GPU sparse Cholesky, LDL^T, -and LU factorization via cuDSS, alongside dense solvers (cuSOLVER), matrix -products (cuBLAS), FFT (cuFFT), and sparse matrix-vector products (cuSPARSE). - -Existing Eigen users should be able to move performance-critical dense or -sparse linear algebra to the GPU with minimal code changes and without -learning CUDA library APIs directly. +Some projects that use Eigen have integrated GPU solvers of their own: +[Ceres](https://github.com/ceres-solver/ceres-solver/blob/master/internal/ceres/cuda_sparse_cholesky.h) +calls cuDSS directly, and +[COLMAP](https://github.com/colmap/colmap/pull/4018) added GPU bundle adjustment +through Caspar. This module is an alternative for code that has not done that +work. ## Design philosophy @@ -50,7 +43,7 @@ `gpu::SparseLLT<double>` vs `SimplicialLLT<SparseMatrix<double>>` -- and both coexist in the same binary. This also lets users keep the factored matrix on device across multiple solves, -something impossible with compile-time replacement. +which compile-time replacement cannot do. **Familiar syntax.** GPU operations use the same expression patterns as CPU Eigen. Here is a side-by-side comparison: @@ -83,22 +76,23 @@ **Standalone module.** `contrib/Eigen/GPU` does not modify or depend on Eigen's Core expression template system (`MatrixBase`, `CwiseBinaryOp`, etc.). `DeviceMatrix` is not an Eigen expression type and does not inherit from -`MatrixBase`. The expression layer is a thin compile-time dispatch where every -supported expression maps to a single NVIDIA library call. There is no -coefficient-level evaluation, lazy fusion, or packet operations. +`MatrixBase`. The expression layer is a thin compile-time dispatch from each +supported expression to NVIDIA library calls, usually one: `norm()` is a dot +product and an NPP square root, and `llt().solve()` is a copy, a factorization +and a solve. There is no coefficient-level evaluation, lazy fusion, or packet +operations. **Interoperability where useful.** `DeviceMatrix` provides the same operator signatures as `Matrix` for common vector operations: `+=`, `-=`, `*=`, `/=`, `dot()`, `squaredNorm()`, `norm()`, `stableNorm()`, `setZero()`, `noalias()`, and copy construction and assignment (a device-to-device copy). This makes -`DeviceMatrix` usable as a drop-in `VectorType` in Eigen algorithm templates +`DeviceMatrix` usable as the `VectorType` in Eigen algorithm templates that rely on these operations, and `DeviceSparseView` is a matrix-free matrix type for Eigen's iterative solvers: `ConjugateGradient<gpu::DeviceSparseView<double>, Lower | Upper>` runs Eigen's own algorithm, unmodified, on device vectors (see [Eigen algorithm interop](#eigen_gpu_interop)). -Conjugate gradient is just the motivating example; we are open to expanding -operator coverage as needed to support other high-level Eigen algorithms on the -GPU. +Conjugate gradient is the algorithm this is tested with; other Eigen algorithms +might need more operators. **Explicit over implicit.** Host-device transfers, stream management, and library handle lifetimes are visible in the API. There are no hidden @@ -216,23 +210,26 @@ ### Linking {#eigen_gpu_linking} The module is header-only, but each feature pulls in the corresponding NVIDIA -library at link time. cuSOLVER, cuBLASLt, and cuSPARSE are created lazily on -first use, so a translation unit that only uses cuBLAS or cuFFT does not need -to link the others: +library at link time. Always link `-lcublas -lcublasLt -lcudart`, and add the +libraries of the features you use: -| Feature | Link flags | +| Feature | Additional link flags | |-----------------------------------------|---------------------------| -| `DeviceMatrix`, GEMM, TRSM, SYMM, SYRK | `-lcublas -lcublasLt` | -| Dense solvers (LLT, LU, QR, SVD, EVD) | `-lcusolver -lcublas` | -| FFT (`gpu::FFT`) | `-lcufft -lcublas` | -| SpMV / SpMM (`gpu::SparseContext`) | `-lcusparse -lcublas` | +| `DeviceMatrix`, GEMM, TRSM, SYMM, SYRK | none | +| Dense solvers (LLT, LU, QR, SVD, EVD) | `-lcusolver` | +| FFT (`gpu::FFT`) | `-lcufft` | +| SpMV / SpMM (`gpu::SparseContext`) | `-lcusparse` | | `norm()`, `DeviceScalar` arithmetic, `/=`, `cwiseProduct` | `-lnpps -lnppc` | -| Sparse direct solvers (cuDSS) | `-lcudss -lcublas` | +| `ConjugateGradient` on `DeviceSparseView` | `-lcusparse -lnpps -lnppc` | +| Sparse direct solvers (cuDSS) | `-lcudss` | -cuBLAS is required by `DeviceMatrix` itself (every `Context` creates a cuBLAS -handle eagerly) and is also a runtime dependency of cuDSS, so it is the one -constant. cuDSS additionally requires `EIGEN_CUDSS` to be defined before -including `contrib/Eigen/GPU`. +Every `Context` creates a cuBLAS handle eagerly, and cuDSS also depends on +cuBLAS at run time. The cuBLASLt handle is lazy, but `Context`'s destructor +references `cublasLtDestroy` directly, so any translation unit using a +`Context` needs cuBLASLt even without a product. The cuSOLVER and cuSPARSE +handles are lazy behind function-pointer deleters, so a translation unit that +does not use those features does not link them. cuDSS additionally requires +`EIGEN_CUDSS` to be defined before including `contrib/Eigen/GPU`. ## Usage @@ -314,7 +311,7 @@ when it reports a numerical failure. **One-shot expression syntax** -- "one-shot" means factorization and solve -run as a single fused call with no persistent factorization object; each +run as one expression with no persistent factorization object; each evaluation re-factorizes: ```cpp
diff --git a/contrib/Eigen/src/NumericalDiff/NumericalDiff.h b/contrib/Eigen/src/NumericalDiff/NumericalDiff.h index 2d0ffbe..f2aba41 100644 --- a/contrib/Eigen/src/NumericalDiff/NumericalDiff.h +++ b/contrib/Eigen/src/NumericalDiff/NumericalDiff.h
@@ -19,21 +19,6 @@ namespace Eigen { -namespace internal { - -// Keeps -ffast-math from folding the rounded evaluation points back into x and h, e.g. (x + h) - x -// into h. EIGEN_OPTIMIZATION_BARRIER is an asm operand constraint restricted to plain types (see -// Macros.h), so class-type scalars go unguarded. -template <typename Scalar, std::enable_if_t<std::is_floating_point<Scalar>::value, int> = 0> -EIGEN_STRONG_INLINE void numerical_diff_barrier(Scalar& x) { - EIGEN_UNUSED_VARIABLE(x); - EIGEN_OPTIMIZATION_BARRIER(x) -} -template <typename Scalar, std::enable_if_t<!std::is_floating_point<Scalar>::value, int> = 0> -EIGEN_STRONG_INLINE void numerical_diff_barrier(Scalar&) {} - -} // namespace internal - enum NumericalDiffMode { Forward, Central }; /** @@ -149,7 +134,8 @@ // of x[j] + h perturbs h by up to ulp(x[j]) <= epsilon |x[j]| <= (epsilon/rel_tol) * h <= h, comparable // to the error of the difference quotient itself. Scalar x_plus = _x[j] + h; - internal::numerical_diff_barrier(x_plus); + // Keeps -ffast-math from folding (x[j] + h) - x[j] back into h. + internal::pbarrier(x_plus); h = x_plus - _x[j]; switch (mode) { case Forward: @@ -166,7 +152,7 @@ // x[j] - h can round (a tie when x[j] < 0 has |x[j]| within h above a power of two), so // divide by the separation of the two evaluation points rather than by 2*h. Scalar x_minus = _x[j] - h; - internal::numerical_diff_barrier(x_minus); + internal::pbarrier(x_minus); x[j] = x_minus; Functor::operator()(x, val1); nfev++;
diff --git a/contrib/Eigen/src/NumericalDiff/Ridders.h b/contrib/Eigen/src/NumericalDiff/Ridders.h new file mode 100644 index 0000000..4b2109a --- /dev/null +++ b/contrib/Eigen/src/NumericalDiff/Ridders.h
@@ -0,0 +1,258 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_NUMERICALDIFF_RIDDERS_H +#define EIGEN_NUMERICALDIFF_RIDDERS_H + +// IWYU pragma: private +#include "./InternalHeaderCheck.h" + +namespace Eigen { + +namespace internal { + +template <typename F, typename = void> +struct ridders_solver_traits {}; + +template <typename F> +struct ridders_solver_traits<F, void_t<typename F::QRSolver>> { + using QRSolver = typename F::QRSolver; +}; + +} // namespace internal + +/** \class Ridders + * \ingroup NumericalDiff_Module + * \brief Adaptive numerical Jacobians using Ridders' extrapolation method. + * + * Stores a functor by value and computes its Jacobian from central differences at successively + * smaller steps. Richardson extrapolation in the squared step size reduces truncation error; + * the Jacobian column with the smallest estimated error is retained. The stopping test uses + * the maximum absolute componentwise difference over the entire column. + * + * The functor supplies \c Scalar, \c InputType, \c ValueType, \c JacobianType, + * \c InputsAtCompileTime, \c ValuesAtCompileTime, \c values(), and \c operator(), + * as for NumericalDiff. The scalar type must be real and non-integer. + * The functor's \c inputs() method is required only when calling inputs(). + * + * \warning The functor must be smooth and defined throughout the initial step interval. + * + * \code + * Eigen::Ridders<MyFunctor> derivative(functor); + * derivative.df(x, jacobian); // Returns the number of function evaluations. + * derivative.df(x, jacobian, error); // Also returns an absolute error estimate per input coordinate. + * \endcode + * + * \tparam Functor_ The vector-valued functor to differentiate. + */ +template <typename Functor_> +class Ridders : public internal::ridders_solver_traits<Functor_> { + public: + EIGEN_MAKE_ALIGNED_OPERATOR_NEW + + using Functor = Functor_; + using Scalar = typename Functor::Scalar; + using InputType = typename Functor::InputType; + using ValueType = typename Functor::ValueType; + using JacobianType = typename Functor::JacobianType; + + static constexpr int InputsAtCompileTime = Functor::InputsAtCompileTime; + static constexpr int ValuesAtCompileTime = Functor::ValuesAtCompileTime; + + /** \brief Parameters controlling the initial interval and the maximum refinement cost. */ + struct Parameters { + /** \brief Initial step scale, giving \f$h_j = \mathrm{initialStep}\,\max(|x_j|, 1)\f$. + * Must be positive and finite. */ + Scalar initialStep = Scalar(0.01); + /** \brief Step reduction factor applied at each refinement. + * The factor must be greater than one and its square must be finite. + * Values close to one amplify rounding errors in extrapolation and may repeat sampling points. + * Refinement stops when both sampling points repeat; this does not detect all ill-conditioning. */ + Scalar stepReduction = Scalar(2); + /** \brief Maximum number of central differences per input coordinate. + * Must be at least two and at most half the maximum representable Index. */ + int maxSteps = 10; + }; + + Ridders() = default; + + /** \brief Constructs from a copy of a functor and differentiation parameters. + * \param functor The functor to differentiate. + * \param parameters The differentiation parameters. */ + explicit Ridders(const Functor& functor, const Parameters& parameters = Parameters()) : m_functor(functor) { + setParameters(parameters); + } + + /** \brief Sets the differentiation parameters. + * \param parameters The differentiation parameters. */ + void setParameters(const Parameters& parameters) { + eigen_assert(parameters.initialStep > Scalar(0) && (numext::isfinite)(parameters.initialStep)); + eigen_assert(parameters.stepReduction > Scalar(1) && + (numext::isfinite)(parameters.stepReduction * parameters.stepReduction)); + eigen_assert(parameters.maxSteps >= 2); + eigen_assert(parameters.maxSteps <= NumTraits<Index>::highest() / 2); + m_parameters = parameters; + } + + /** \returns the differentiation parameters. */ + const Parameters& parameters() const { return m_parameters; } + /** \returns the stored functor. */ + Functor& functor() { return m_functor; } + /** \returns the stored functor. */ + const Functor& functor() const { return m_functor; } + + /** \brief Evaluates the stored functor. + * \param[in] x The evaluation point. + * \param[out] values The function values. + * \returns the value returned by the functor. */ + int operator()(const InputType& x, ValueType& values) const { return m_functor(x, values); } + /** \returns the stored functor's number of inputs. */ + int inputs() const { return m_functor.inputs(); } + /** \returns the stored functor's number of values. */ + int values() const { return m_functor.values(); } + + /** + * \brief Computes the Jacobian using adaptive central differences. + * + * The caller sizes \a jac to values() by x.size(). At most \c 2*maxSteps evaluations + * are made per coordinate. Difference quotients divide by the separation of the representable + * evaluation points. Refinement stops when successive tableau diagonals differ by at least + * twice the best estimated error, when evaluation points coincide or repeat the previous pair, + * or when a non-finite value is encountered. The best finite estimate is retained; + * if none exists, the column contains NaNs. + * + * \param[in] x The evaluation point. + * \param[out] jac The Jacobian at \a x. + * \returns the number of functor evaluations. + * \sa Parameters + */ + int df(const InputType& x, JacobianType& jac) const { return dfImpl(x, jac, nullptr); } + + /** + * \brief Computes the Jacobian and an absolute error estimate per input coordinate. + * + * The estimate is the maximum componentwise difference from the two parent tableau entries; + * it is a heuristic, not an error bound. An infinite estimate means no extrapolation was available. + * + * \param[in] x The evaluation point. + * \param[out] jac The Jacobian at \a x, sized by the caller to values() by x.size(). + * \param[out] error The error estimates, resized to x.size(). + * \returns the number of functor evaluations. + * \sa Parameters + */ + int df(const InputType& x, JacobianType& jac, InputType& error) const { + error.resize(x.size()); + return dfImpl(x, jac, &error); + } + + private: + Functor m_functor; + Parameters m_parameters; + + int dfImpl(const InputType& input, JacobianType& jac, InputType* error) const { + EIGEN_STATIC_ASSERT(!NumTraits<Scalar>::IsComplex && !NumTraits<Scalar>::IsInteger, + NUMERICAL_DIFF_REQUIRES_A_REAL_FLOATING_POINT_SCALAR); + // C. J. F. Ridders, "Accurate computation of F'(x) and F'(x)F''(x)", + // Advances in Engineering Software 4(2), 75-76 (1982). Extrapolate central differences in h^2. + using Tableau = Matrix<Scalar, Functor::ValuesAtCompileTime, Dynamic>; + const Index values = m_functor.values(); + eigen_assert(jac.rows() == values && jac.cols() == input.size()); + if (values == 0 || input.size() == 0) { + if (error) error->setZero(); + return 0; + } + + Tableau workspace(values, 2 * Index(m_parameters.maxSteps)); + auto first = workspace.leftCols(m_parameters.maxSteps); + auto second = workspace.rightCols(m_parameters.maxSteps); + auto current = &first; + auto previous = &second; + + // Functors may require ValueType& outputs, which cannot bind to column views. + ValueType plus(values), minus(values); + InputType x = input; + const Scalar reductionSquared = m_parameters.stepReduction * m_parameters.stepReduction; + int evaluations = 0; + + for (Index j = 0; j < input.size(); ++j) { + Scalar h = m_parameters.initialStep * numext::maxi(numext::abs(input[j]), Scalar(1)); + Scalar previousPlus = input[j], previousMinus = input[j]; + Scalar bestError = NumTraits<Scalar>::infinity(); + jac.col(j).setConstant(NumTraits<Scalar>::quiet_NaN()); + for (int step = 0; step < m_parameters.maxSteps; ++step) { + Scalar xPlus = input[j] + h; + Scalar xMinus = input[j] - h; + internal::pbarrier(xPlus); + internal::pbarrier(xMinus); + if (!(numext::isfinite)(xPlus) || !(numext::isfinite)(xMinus)) break; + if (step > 0 && xPlus == previousPlus && xMinus == previousMinus) break; + + const Scalar separation = xPlus - xMinus; + if (!(separation > Scalar(0)) || !(numext::isfinite)(separation)) break; + previousPlus = xPlus; + previousMinus = xMinus; + + x[j] = xPlus; + m_functor(x, plus); + x[j] = xMinus; + m_functor(x, minus); + x[j] = input[j]; + evaluations += 2; + + current->col(0) = (plus - minus) / separation; + if (!current->col(0).allFinite()) { + if (separation <= Scalar(1)) break; + // Divide before subtracting to avoid overflow, preserving already finite quotients. + auto quotient = current->col(0).array(); + quotient = quotient.isFiniteTyped().select(quotient, plus.array() / separation - minus.array() / separation); + if (!current->col(0).allFinite()) break; + } + Index bestOrder = step == 0 ? 0 : -1; + + Scalar factor = reductionSquared; + bool finite = true; + for (int order = 1; order <= step; ++order) { + // A(k,i) = (t^(2k) * A(k-1,i) - A(k-1,i-1)) / (t^(2k) - 1). + // Avoid forming `t^(2k) * A(k-1,i)` directly, since it can overflow. + current->col(order) = + current->col(order - 1) + (current->col(order - 1) - previous->col(order - 1)) / (factor - Scalar(1)); + if (!current->col(order).allFinite()) { + finite = false; + break; + } + + const Scalar estimate = + numext::maxi((current->col(order) - current->col(order - 1)).template lpNorm<Infinity>(), + (current->col(order) - previous->col(order - 1)).template lpNorm<Infinity>()); + if ((numext::isfinite)(estimate) && estimate <= bestError) { + bestError = estimate; + bestOrder = order; + } + + factor *= reductionSquared; + } + if (bestOrder >= 0) jac.col(j) = current->col(bestOrder); + if (!finite) break; + if (step > 0 && + (current->col(step) - previous->col(step - 1)).template lpNorm<Infinity>() >= Scalar(2) * bestError) + break; + + std::swap(current, previous); + h /= m_parameters.stepReduction; + } + if (error) (*error)[j] = bestError; + } + return evaluations; + } +}; + +#if EIGEN_COMP_CXXVER < 17 +template <typename Functor> +constexpr int Ridders<Functor>::InputsAtCompileTime; +template <typename Functor> +constexpr int Ridders<Functor>::ValuesAtCompileTime; +#endif + +} // namespace Eigen + +#endif // EIGEN_NUMERICALDIFF_RIDDERS_H
diff --git a/contrib/Eigen/src/Riccati/DiscreteAlgebraicRiccatiSolver.h b/contrib/Eigen/src/Riccati/DiscreteAlgebraicRiccatiSolver.h new file mode 100644 index 0000000..0652a10 --- /dev/null +++ b/contrib/Eigen/src/Riccati/DiscreteAlgebraicRiccatiSolver.h
@@ -0,0 +1,653 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// This Source Code Form is subject to the terms of the Mozilla +// Public License v. 2.0. If a copy of the MPL was not distributed +// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-FileCopyrightText: 2001-2002 Enthought, Inc. +// SPDX-FileCopyrightText: 2003 SciPy Developers +// SPDX-FileCopyrightText: 1992-2025 The University of Tennessee and The University of Tennessee Research Foundation +// SPDX-FileCopyrightText: 2000-2025 The University of California Berkeley +// SPDX-FileCopyrightText: 2006-2025 The University of Colorado Denver +// SPDX-License-Identifier: MPL-2.0 AND BSD-3-Clause AND BSD-3-Clause-Open-MPI +// +// This file incorporates work covered by the following copyright and +// permission notices: +// +// DiscreteAlgebraicRiccatiSolver is adapted from scipy.linalg.solve_discrete_are +// (scipy/linalg/_solvers.py, SciPy 1.18.0): +// +// Copyright (c) 2001-2002 Enthought, Inc. 2003, SciPy Developers. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions +// are met: +// +// 1. Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// +// 2. Redistributions in binary form must reproduce the above +// copyright notice, this list of conditions and the following +// disclaimer in the documentation and/or other materials provided +// with the distribution. +// +// 3. Neither the name of the copyright holder nor the names of its +// contributors may be used to endorse or promote products derived +// from this software without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +// +// internal::riccati_balance_exponents() is adapted from LAPACK's xGEBAL +// (https://github.com/Reference-LAPACK/lapack): +// +// Copyright (c) 1992-2025 The University of Tennessee and The University +// of Tennessee Research Foundation. All rights +// reserved. +// Copyright (c) 2000-2025 The University of California Berkeley. All +// rights reserved. +// Copyright (c) 2006-2025 The University of Colorado Denver. All rights +// reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// - Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// +// - Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer listed +// in this license in the documentation and/or other materials +// provided with the distribution. +// +// - Neither the name of the copyright holders nor the names of its +// contributors may be used to endorse or promote products derived from +// this software without specific prior written permission. +// +// The copyright holders provide no reassurances that the source code +// provided does not infringe any patent, copyright, or any other +// intellectual property rights of third parties. The copyright holders +// disclaim any liability to any recipient for claims brought against +// recipient by any third party for infringement of that parties +// intellectual property rights. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#ifndef EIGEN_DISCRETE_ALGEBRAIC_RICCATI_SOLVER_H +#define EIGEN_DISCRETE_ALGEBRAIC_RICCATI_SOLVER_H + +// IWYU pragma: private +#include "./InternalHeaderCheck.h" + +namespace Eigen { + +namespace internal { + +/** \internal The QZ decomposition used for a given scalar type. */ +template <typename MatrixType, bool IsComplex = NumTraits<typename MatrixType::Scalar>::IsComplex> +struct riccati_qz { + using type = RealQZ<MatrixType>; +}; +template <typename MatrixType> +struct riccati_qz<MatrixType, true> { + using type = ComplexQZ<MatrixType>; +}; + +/** \internal \returns the 1-norm (maximum absolute column sum) of \p m. */ +template <typename Derived> +typename NumTraits<typename Derived::Scalar>::Real riccati_norm1(const MatrixBase<Derived>& m) { + return m.size() == 0 ? typename NumTraits<typename Derived::Scalar>::Real(0) + : m.cwiseAbs().colwise().sum().maxCoeff(); +} + +/** \internal + * Computes the power-of-two scaling of the nonnegative square matrix \p M that balances the 2-norms of its rows and + * columns, as LAPACK's xGEBAL with JOB='S'. \p M is overwritten by the balanced matrix and the exponents of the + * scaling factors are returned in \p exponents, such that the balanced matrix is \f$ D^{-1} M D \f$ with + * \f$ D = \mathrm{diag}(2^{e_i}) \f$. + */ +template <typename RealMatrix> +void riccati_balance_exponents(RealMatrix& M, VectorXi& exponents) { + using RealScalar = typename RealMatrix::Scalar; + const Index n = M.rows(); + const RealScalar radix(2); + const RealScalar factor(0.95); + const RealScalar sfmin1 = (std::numeric_limits<RealScalar>::min)() / NumTraits<RealScalar>::epsilon(); + const RealScalar sfmax1 = RealScalar(1) / sfmin1; + const RealScalar sfmin2 = sfmin1 * radix; + const RealScalar sfmax2 = RealScalar(1) / sfmin2; + + exponents.setZero(n); + Matrix<RealScalar, Dynamic, 1> scale = Matrix<RealScalar, Dynamic, 1>::Ones(n); + bool noconv = true; + while (noconv) { + noconv = false; + for (Index i = 0; i < n; ++i) { + RealScalar c = M.col(i).stableNorm(); + RealScalar r = M.row(i).stableNorm(); + if (numext::is_exactly_zero(c) || numext::is_exactly_zero(r)) continue; + + RealScalar ca = M.col(i).maxCoeff(); + RealScalar ra = M.row(i).maxCoeff(); + + RealScalar g = r / radix; + RealScalar f(1); + const RealScalar s = c + r; + int e = 0; + while (c < g && numext::maxi(f, numext::maxi(c, ca)) < sfmax2 && numext::mini(r, numext::mini(g, ra)) > sfmin2) { + f *= radix; + c *= radix; + ca *= radix; + r /= radix; + g /= radix; + ra /= radix; + ++e; + } + g = c / radix; + while (g >= r && numext::maxi(r, ra) < sfmax2 && numext::mini(numext::mini(f, c), numext::mini(g, ca)) > sfmin2) { + f /= radix; + c /= radix; + g /= radix; + ca /= radix; + r *= radix; + ra *= radix; + --e; + } + + if (c + r >= factor * s) continue; + if (f < RealScalar(1) && scale(i) < RealScalar(1) && f * scale(i) <= sfmin1) continue; + if (f > RealScalar(1) && scale(i) > RealScalar(1) && scale(i) >= sfmax1 / f) continue; + scale(i) *= f; + exponents(i) += e; + noconv = true; + M.row(i) /= f; + M.col(i) *= f; + } + } +} + +/** \internal Rounds \p d / 2 to the nearest integer, ties to even (as NumPy's round()). */ +inline int riccati_half_round_even(int d) { + const int q = (d >= 0) ? d / 2 : -((-d + 1) / 2); // floor(d / 2) + if (d % 2 == 0) return q; + return (q % 2 == 0) ? q : q + 1; +} + +} // namespace internal + +/** \ingroup Riccati_Module + * + * \class DiscreteAlgebraicRiccatiSolver + * + * \brief Solves the discrete-time algebraic Riccati equation (DARE) + * + * \tparam MatrixType_ the type of the solution matrix \f$ X \f$, a square matrix of real or complex scalars; + * this is expected to be an instantiation of the Matrix class template. + * + * This class computes the stabilizing solution \f$ X \f$ of the discrete-time algebraic Riccati equation + * \f[ A^* X A - X - (A^* X B) (R + B^* X B)^{-1} (B^* X A) + Q = 0 \f] + * or of its generalized version + * \f[ A^* X A - E^* X E - (A^* X B + S) (R + B^* X B)^{-1} (B^* X A + S^*) + Q = 0, \f] + * where \f$ A \f$ and \f$ E \f$ are \f$ m \times m \f$, \f$ B \f$ and \f$ S \f$ are \f$ m \times n \f$, + * \f$ Q \f$ is \f$ m \times m \f$ selfadjoint, \f$ R \f$ is \f$ n \times n \f$ selfadjoint and \f$ E \f$ is + * nonsingular. \f$ R \f$ itself does not need to be invertible. + * + * For a solution to exist, the eigenvalues of \f$ A \f$ outside the unit disc have to be controllable and the + * associated symplectic pencil (see below) must not have eigenvalues on, or too close to, the unit circle. + * + * The solver follows the generalized eigenvalue approach of [1]: it forms the extended symplectic pencil + * \f[ + * H - \lambda J = + * \begin{bmatrix} A & 0 & B \\ -Q & E^* & -S \\ S^* & 0 & R \end{bmatrix} + * - \lambda + * \begin{bmatrix} E & 0 & 0 \\ 0 & A^* & 0 \\ 0 & -B^* & 0 \end{bmatrix}, + * \f] + * optionally balances it as in [3], deflates it by the columns of \f$ R \f$, computes its generalized Schur form + * (RealQZ or ComplexQZ) and reorders it such that the eigenvalues inside the unit circle come first. The + * leading \f$ m \f$ Schur vectors \f$ [U_1; U_2] \f$ span the stable deflating subspace and + * \f$ X = U_2 U_1^{-1} \f$ (for the generalized equation, \f$ U_1 \f$ is first replaced by \f$ E U_1 \f$). + * The computation is reported as failed if the pencil does not have exactly \f$ m \f$ eigenvalues inside the unit + * circle, if \f$ U_1 \f$ is numerically singular or so small that \f$ \|X\| \f$ (of the balanced problem) would + * exceed about \f$ 0.2 / \varepsilon \f$, or if \f$ U_1^* U_2 \f$ is far from selfadjoint, which happens when the + * pencil has eigenvalues too close to the unit circle [2]. A stabilizing solution that large cannot be told apart + * from an unstable mode that \f$ B \f$ does not reach, for which none exists. + * + * Balancing is essential for some badly scaled problems, but it can cost accuracy for others, notably when \f$ B \f$ + * is very small compared to the other matrices. With balancing enabled (the default), the solver therefore checks + * the backward error of the balanced solution whenever balancing changed the pencil; if it is large, or the balanced + * computation fails, the unbalanced pencil is solved as well and the more accurate solution is returned. + * compute(..., false) never balances. + * + * The algorithm follows SciPy's \c scipy.linalg.solve_discrete_are, with these differences: the generalized equation + * combines the stable subspace with the balanced \f$ E \f$ (SciPy uses the original one, which is wrong when + * balancing scales the pencil), the unbalanced pencil serves as a fallback (see above), and the number of stable + * eigenvalues is checked. + * + * Example: a discrete-time LQR controller for a sampled double integrator. + * \include DiscreteAlgebraicRiccatiSolver.cpp + * Output: \verbinclude DiscreteAlgebraicRiccatiSolver.out + * + * References: + * - [1] P. Van Dooren, "A Generalized Eigenvalue Approach For Solving Riccati Equations", SIAM J. Sci. Stat. Comput., + * 2(2), 1981. + * - [2] A. J. Laub, "A Schur Method for Solving Algebraic Riccati Equations", MIT LIDS-R-859, 1978. + * - [3] P. Benner, "Symplectic Balancing of Hamiltonian Matrices", SIAM J. Sci. Comput., 22(5), 2001. + * + * \sa class RealQZ, class ComplexQZ + */ +template <typename MatrixType_> +class DiscreteAlgebraicRiccatiSolver { + public: + using MatrixType = MatrixType_; + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + + /** \brief Default constructor; the solver is initialized by calling compute(). */ + DiscreteAlgebraicRiccatiSolver() = default; + + /** \brief Constructor; solves the discrete-time algebraic Riccati equation. + * + * This is equivalent to calling compute(const EigenBase<InputTypeA>&, const EigenBase<InputTypeB>&, + * const EigenBase<InputTypeQ>&, const EigenBase<InputTypeR>&, bool). + */ + template <typename InputTypeA, typename InputTypeB, typename InputTypeQ, typename InputTypeR> + DiscreteAlgebraicRiccatiSolver(const EigenBase<InputTypeA>& A, const EigenBase<InputTypeB>& B, + const EigenBase<InputTypeQ>& Q, const EigenBase<InputTypeR>& R, bool balance = true) { + compute(A, B, Q, R, balance); + } + + /** \brief Constructor; solves the generalized discrete-time algebraic Riccati equation. + * + * This is equivalent to calling compute(const EigenBase<InputTypeA>&, const EigenBase<InputTypeB>&, + * const EigenBase<InputTypeQ>&, const EigenBase<InputTypeR>&, const EigenBase<InputTypeE>&, + * const EigenBase<InputTypeS>&, bool). + */ + template <typename InputTypeA, typename InputTypeB, typename InputTypeQ, typename InputTypeR, typename InputTypeE, + typename InputTypeS> + DiscreteAlgebraicRiccatiSolver(const EigenBase<InputTypeA>& A, const EigenBase<InputTypeB>& B, + const EigenBase<InputTypeQ>& Q, const EigenBase<InputTypeR>& R, + const EigenBase<InputTypeE>& E, const EigenBase<InputTypeS>& S, bool balance = true) { + compute(A, B, Q, R, E, S, balance); + } + + /** \brief Solves \f$ A^* X A - X - (A^* X B) (R + B^* X B)^{-1} (B^* X A) + Q = 0 \f$. + * + * \param[in] A square \f$ m \times m \f$ matrix + * \param[in] B \f$ m \times n \f$ matrix + * \param[in] Q selfadjoint \f$ m \times m \f$ matrix + * \param[in] R selfadjoint \f$ n \times n \f$ matrix + * \param[in] balance whether the symplectic pencil is balanced before the QZ decomposition, see the class + * documentation + * + * \returns a reference to \c *this + * + * The result of the computation is reported by info(). + */ + template <typename InputTypeA, typename InputTypeB, typename InputTypeQ, typename InputTypeR> + DiscreteAlgebraicRiccatiSolver& compute(const EigenBase<InputTypeA>& A, const EigenBase<InputTypeB>& B, + const EigenBase<InputTypeQ>& Q, const EigenBase<InputTypeR>& R, + bool balance = true) { + return computeImpl(A.derived(), B.derived(), Q.derived(), R.derived(), nullptr, nullptr, balance); + } + + /** \brief Solves \f$ A^* X A - E^* X E - (A^* X B + S) (R + B^* X B)^{-1} (B^* X A + S^*) + Q = 0 \f$. + * + * \param[in] A square \f$ m \times m \f$ matrix + * \param[in] B \f$ m \times n \f$ matrix + * \param[in] Q selfadjoint \f$ m \times m \f$ matrix + * \param[in] R selfadjoint \f$ n \times n \f$ matrix + * \param[in] E nonsingular \f$ m \times m \f$ matrix + * \param[in] S \f$ m \times n \f$ matrix + * \param[in] balance whether the symplectic pencil is balanced before the QZ decomposition, see the class + * documentation + * + * \returns a reference to \c *this + * + * Pass the identity for \p E or zero for \p S to leave out one of the generalized terms. + * The result of the computation is reported by info(). + */ + template <typename InputTypeA, typename InputTypeB, typename InputTypeQ, typename InputTypeR, typename InputTypeE, + typename InputTypeS> + DiscreteAlgebraicRiccatiSolver& compute(const EigenBase<InputTypeA>& A, const EigenBase<InputTypeB>& B, + const EigenBase<InputTypeQ>& Q, const EigenBase<InputTypeR>& R, + const EigenBase<InputTypeE>& E, const EigenBase<InputTypeS>& S, + bool balance = true) { + const DynamicMatrix e = E.derived(); + const DynamicMatrix s = S.derived(); + return computeImpl(A.derived(), B.derived(), Q.derived(), R.derived(), &e, &s, balance); + } + + /** \returns the stabilizing solution \f$ X \f$ of the Riccati equation, which is selfadjoint. + * + * \pre The solver has been initialized and info() returns \c Success. + */ + const MatrixType& solution() const { + eigen_assert(m_isInitialized && "DiscreteAlgebraicRiccatiSolver is not initialized."); + eigen_assert(m_info == Success && "The Riccati equation could not be solved."); + return m_X; + } + + /** \brief Reports whether the last computation was successful. + * + * \returns \c Success if the computation was successful, + * \c InvalidInput if an input matrix is not finite, \p Q or \p R is not selfadjoint, or \p E is + * numerically singular, + * \c NoConvergence if the QZ iteration did not converge, and + * \c NumericalIssue if the stable deflating subspace of the symplectic pencil could not be isolated + * (no stabilizing solution exists, the pencil has eigenvalues too close to the unit circle, or the + * solution would be too large to compute, see the class documentation). + */ + ComputationInfo info() const { + eigen_assert(m_isInitialized && "DiscreteAlgebraicRiccatiSolver is not initialized."); + return m_info; + } + + protected: + using DynamicMatrix = Matrix<Scalar, Dynamic, Dynamic>; + using RealDynamicMatrix = Matrix<RealScalar, Dynamic, Dynamic>; + using RealVector = Matrix<RealScalar, Dynamic, 1>; + + static bool isSelfAdjoint(const DynamicMatrix& M) { + return internal::riccati_norm1(M - M.adjoint()) <= + RealScalar(100) * NumTraits<RealScalar>::epsilon() * internal::riccati_norm1(M); + } + + DiscreteAlgebraicRiccatiSolver& computeImpl(const DynamicMatrix& A, const DynamicMatrix& B, const DynamicMatrix& Q, + const DynamicMatrix& R, const DynamicMatrix* E, const DynamicMatrix* S, + bool balance); + + static ComputationInfo solvePencil(const DynamicMatrix& A, const DynamicMatrix& B, const DynamicMatrix& Q, + const DynamicMatrix& R, const DynamicMatrix* E, const DynamicMatrix* S, + bool balance, RealScalar rcondThreshold, DynamicMatrix& X, RealVector& balancing); + + // Backward error ratios of a solution on the original and on the balanced equation, see backwardErrorRatios(). + struct BackwardErrorRatios { + RealScalar original; + RealScalar balanced; + }; + + static BackwardErrorRatios backwardErrorRatios(const DynamicMatrix& A, const DynamicMatrix& B, const DynamicMatrix& Q, + const DynamicMatrix& R, const DynamicMatrix* E, const DynamicMatrix* S, + const DynamicMatrix& X, const RealVector& balancing); + + MatrixType m_X; + ComputationInfo m_info = InvalidInput; + bool m_isInitialized = false; +}; + +template <typename MatrixType_> +DiscreteAlgebraicRiccatiSolver<MatrixType_>& DiscreteAlgebraicRiccatiSolver<MatrixType_>::computeImpl( + const DynamicMatrix& A, const DynamicMatrix& B, const DynamicMatrix& Q, const DynamicMatrix& R, + const DynamicMatrix* E, const DynamicMatrix* S, bool balance) { + const Index m = B.rows(); + const Index n = B.cols(); + EIGEN_ONLY_USED_FOR_DEBUG(n); + eigen_assert(A.rows() == m && A.cols() == m && "A must be square with as many rows as B."); + eigen_assert(Q.rows() == m && Q.cols() == m && "Q must have the same size as A."); + eigen_assert(R.rows() == n && R.cols() == n && "R must be square with as many columns as B."); + eigen_assert((E == nullptr || (E->rows() == m && E->cols() == m)) && "E must have the same size as A."); + eigen_assert((S == nullptr || (S->rows() == m && S->cols() == n)) && "S must have the same size as B."); + + m_isInitialized = true; + m_info = InvalidInput; + + if (!A.allFinite() || !B.allFinite() || !Q.allFinite() || !R.allFinite()) return *this; + if ((E != nullptr && !E->allFinite()) || (S != nullptr && !S->allFinite())) return *this; + if (!isSelfAdjoint(Q) || !isSelfAdjoint(R)) return *this; + if (m == 0) { + m_X.resize(0, 0); + m_info = Success; + return *this; + } + // rcond() bounds 1 / kappa_1(E) from above and kappa_1 <= m kappa_2, so this rejects only kappa_2(E) > 1 / eps. + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + if (E != nullptr && !(RealScalar(m) * PartialPivLU<DynamicMatrix>(*E).rcond() >= eps)) return *this; + // U1 is held to the stricter rcond(U1) >= eps: the test above guarantees no false rejection but can accept + // kappa_2 > 1 / eps, and for U1 it accepts the non-stabilizable problem of dare_no_stabilizing_solution() with a + // meaningless ||X|| ~ 1e16. + const RealScalar rcondThreshold = eps; + + DynamicMatrix X; + RealVector balancing; + m_info = solvePencil(A, B, Q, R, E, S, balance, rcondThreshold, X, balancing); + if (balancing.size() > 0) { + // Balancing is essential for some badly scaled problems (e.g. DAREX 13 and 14), but costs accuracy for others, + // notably when B is small. The unbalanced pencil is solved as well if the balanced solve fails or looks + // inaccurate by either ratio of backwardErrorRatios(): above 10 on the original equation (balancing lost + // accuracy) or above 100 on the balanced equation (a poor solution that the kappa factor of the first ratio + // hides, e.g. 1e3 for the unbalanced solutions of DAREX 14, against 0.2 to 3 for good ones). Of the two + // solutions, the one with the smaller ratio on the original equation is kept. Against a long double reference, + // this kept the more accurate solution in 40000 random problems like those of dare_random() whenever the + // difference exceeded a factor 2, and triggered the second solve only for problems with B scaled by 10^-1 or + // less. If balancing left the pencil unchanged, there is nothing to fall back to. + const RealScalar inf = NumTraits<RealScalar>::infinity(); + const BackwardErrorRatios ratios = + m_info == Success ? backwardErrorRatios(A, B, Q, R, E, S, X, balancing) : BackwardErrorRatios{inf, inf}; + if (!(ratios.original <= RealScalar(10) && ratios.balanced <= RealScalar(100))) { + DynamicMatrix Xu; + RealVector none; + if (solvePencil(A, B, Q, R, E, S, false, rcondThreshold, Xu, none) == Success && + (m_info != Success || backwardErrorRatios(A, B, Q, R, E, S, Xu, balancing).original < ratios.original)) { + X.swap(Xu); + m_info = Success; + } + } + } + if (m_info == Success) m_X = X; + return *this; +} + +/** \internal + * \returns the backward error ratios of the solution \p X on the original equation, + * \f[ \rho = \frac{\|\mathrm{res}\|_F}{m\,\varepsilon\,(\|A^*XA\|_F + \|E^*XE\|_F + \|Q\|_F)\,\kappa},\qquad + * \kappa = \max(1, \|E^*XE\|_F / \|Q\|_F), \f] + * and on the balanced equation, + * \f[ \rho_b = \frac{\|D^{-1} \mathrm{res}\, D^{-1}\|_F}{\varepsilon\,(\|D^{-1} A^*XA\, D^{-1}\|_F + * + \|D^{-1} E^*XE\, D^{-1}\|_F + \|D^{-1} Q D^{-1}\|_F + \|D^{-1} G D^{-1}\|_F)}. \f] + * Here res \f$ = A^*XA - E^*XE - G + Q \f$ is the residual, \f$ G = (A^*XB + S)(R + B^*XB)^{-1}(B^*XA + S^*) \f$, and + * \f$ D = \f$ diag(\p balancing) relates \p X to the solution of the balanced equation by \f$ X = D X_b D \f$; its + * residual \f$ D^{-1} \mathrm{res}\, D^{-1} \f$ is exact in floating point, since \f$ D \f$ consists of powers of two. + * + * \f$ \kappa \f$ accounts for the residual that a large solution leaves even when computed accurately: for random + * problems with B scaled by \f$ 10^{-k} \f$, the residual without \f$ \kappa \f$ grows in proportion to + * \f$ \|X\|_F / \|Q\|_F \f$, for SciPy (LAPACK) as for this solver. When \f$ X \f$ is large only because the problem + * is badly scaled, \f$ \kappa \f$ also hides poor solutions, which \f$ \rho_b \f$ exposes. Non-finite ratios are + * reported as infinity. + */ +template <typename MatrixType_> +typename DiscreteAlgebraicRiccatiSolver<MatrixType_>::BackwardErrorRatios +DiscreteAlgebraicRiccatiSolver<MatrixType_>::backwardErrorRatios(const DynamicMatrix& A, const DynamicMatrix& B, + const DynamicMatrix& Q, const DynamicMatrix& R, + const DynamicMatrix* E, const DynamicMatrix* S, + const DynamicMatrix& X, const RealVector& balancing) { + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + const auto finite = [](RealScalar r) { return (numext::isfinite)(r) ? r : NumTraits<RealScalar>::infinity(); }; + const DynamicMatrix XA = X * A; + DynamicMatrix AXA = A.adjoint() * XA; + DynamicMatrix EXE = (E != nullptr) ? DynamicMatrix(E->adjoint() * X * *E) : X; + DynamicMatrix G = DynamicMatrix::Zero(A.rows(), A.rows()); + if (B.cols() > 0) { + DynamicMatrix AXBS = XA.adjoint() * B; + if (S != nullptr) AXBS += *S; + G = AXBS * (R + B.adjoint() * X * B).partialPivLu().solve(AXBS.adjoint()); + } + DynamicMatrix res = AXA - EXE - G + Q; + if (numext::is_exactly_zero(res.norm())) return BackwardErrorRatios{RealScalar(0), RealScalar(0)}; + + BackwardErrorRatios ratios; + const RealScalar normQ = Q.norm(), normEXE = EXE.norm(); + const RealScalar kappa = (normQ > RealScalar(0) && normEXE > normQ) ? normEXE / normQ : RealScalar(1); + ratios.original = finite(res.norm() / (RealScalar(A.rows()) * eps * (AXA.norm() + normEXE + normQ) * kappa)); + DynamicMatrix Qb = Q; + if (balancing.size() > 0) { + const RealVector inv = balancing.cwiseInverse(); + for (DynamicMatrix* M : {&res, &AXA, &EXE, &G, &Qb}) *M = inv.asDiagonal() * *M * inv.asDiagonal(); + } + ratios.balanced = finite(res.norm() / (eps * (AXA.norm() + EXE.norm() + Qb.norm() + G.norm()))); + return ratios; +} + +/** \internal + * Solves the equation through the extended symplectic pencil, balanced if \p balance is true; on success, \p X holds + * the selfadjoint solution. + */ +template <typename MatrixType_> +ComputationInfo DiscreteAlgebraicRiccatiSolver<MatrixType_>::solvePencil(const DynamicMatrix& A, const DynamicMatrix& B, + const DynamicMatrix& Q, const DynamicMatrix& R, + const DynamicMatrix* E, const DynamicMatrix* S, + bool balance, RealScalar rcondThreshold, + DynamicMatrix& X, RealVector& balancing) { + const Index m = B.rows(); + const Index n = B.cols(); + + // Form the extended symplectic pencil H - lambda J. + const Index p = 2 * m + n; + DynamicMatrix H = DynamicMatrix::Zero(p, p); + DynamicMatrix J = DynamicMatrix::Zero(p, p); + H.topLeftCorner(m, m) = A; + H.topRightCorner(m, n) = B; + H.block(m, 0, m, m) = -Q; + if (E != nullptr) + H.block(m, m, m, m) = E->adjoint(); + else + H.block(m, m, m, m).setIdentity(); + if (S != nullptr) { + H.block(m, 2 * m, m, n) = -*S; + H.bottomLeftCorner(n, m) = S->adjoint(); + } + H.bottomRightCorner(n, n) = R; + + if (E != nullptr) + J.topLeftCorner(m, m) = *E; + else + J.topLeftCorner(m, m).setIdentity(); + J.block(m, m, m, m) = A.adjoint(); + J.block(2 * m, m, n, m) = -B.adjoint(); + + // Balance the pencil without destroying its symplectic structure: the scaling is diag(D, D^-1, D_R), + // with D derived from the xGEBAL scaling of |H| + |J| with zeroed diagonal, see [3]. + RealVector scaling; + bool balanced = false; + balancing.resize(0); + if (balance) { + RealDynamicMatrix M = H.cwiseAbs() + J.cwiseAbs(); + M.diagonal().setZero(); + VectorXi exponents; + internal::riccati_balance_exponents(M, exponents); + if (!(exponents.array() == 0).all()) { + balanced = true; + scaling.resize(p); + for (Index i = 0; i < m; ++i) { + const int d = internal::riccati_half_round_even(exponents(m + i) - exponents(i)); + scaling(i) = numext::ldexp(RealScalar(1), d); + scaling(m + i) = numext::ldexp(RealScalar(1), -d); + } + // As in SciPy, the R block is scaled by D_R = diag(2^e), the inverse of the xGEBAL scaling D^-1 M D. The + // xGEBAL-consistent 2^-e loses accuracy: the DAREX 13 and 14 residuals grow from 7.7e-7 and 1.3e-7 to 1.6e-5 + // and 1.8e-6. + for (Index i = 2 * m; i < p; ++i) scaling(i) = numext::ldexp(RealScalar(1), exponents(i)); + // H <- D H D^-1, J <- D J D^-1 with D = diag(scaling), exact in floating point. + H = scaling.asDiagonal() * H * scaling.cwiseInverse().asDiagonal(); + J = scaling.asDiagonal() * J * scaling.cwiseInverse().asDiagonal(); + // Also reported when the solve fails, so that computeImpl() falls back to the unbalanced pencil. + balancing = scaling.head(m); + } + } + + // Deflate the pencil by the columns belonging to R, see [1]. + DynamicMatrix Hd, Jd; + if (n > 0) { + const HouseholderQR<DynamicMatrix> qr(H.rightCols(n)); + Hd = (qr.householderQ().adjoint() * H.leftCols(2 * m)).bottomRows(2 * m); + Jd = (qr.householderQ().adjoint() * J.leftCols(2 * m)).bottomRows(2 * m); + } else { + Hd = H; + Jd = J; + } + + // Generalized Schur form with the eigenvalues inside the unit circle first. + typename internal::riccati_qz<DynamicMatrix>::type qz(Hd, Jd, true); + if (qz.info() == NoConvergence) return NoConvergence; + DynamicMatrix SS = qz.matrixS(); + DynamicMatrix TT = qz.matrixT(); + DynamicMatrix ZZ = qz.matrixZ().adjoint(); + if (qz.info() != Success) { + const DynamicMatrix QQ = qz.matrixQ(); + // ComplexQZ reports NumericalIssue when its direct reduction of a 2x2 block leaves a subdiagonal entry of S above + // eps ||S||, although the reduction is only backward stable to a few eps ||S||. Accept the decomposition if + // ||Q (S, T) Z^* - (Hd, Jd)||_F <= 10 (2m) eps ||(Hd, Jd)||_F once that entry is dropped. + internal::clean_generalized_schur_form(SS, TT); + const RealScalar bound = RealScalar(10 * 2 * m) * NumTraits<RealScalar>::epsilon(); + if (!((QQ * SS * ZZ.adjoint() - Hd).norm() <= bound * Hd.norm() && + (QQ * TT * ZZ.adjoint() - Jd).norm() <= bound * Jd.norm())) + return NumericalIssue; + } + // The eigenvalues of the symplectic pencil come in pairs (lambda, 1 / conj(lambda)), so exactly m of them lie + // inside the unit circle if a stabilizing solution exists. (SciPy does not check this.) + if (internal::reorder_generalized_schur_inside_unit_circle<DynamicMatrix>(SS, TT, nullptr, ZZ) != m) + return NumericalIssue; + + // The leading m right Schur vectors span the stable deflating subspace. + DynamicMatrix U = ZZ.leftCols(m); + if (E != nullptr) { + // U belongs to the balanced pencil, so it is combined with the balanced E held in J. (SciPy combines it with the + // original E, which yields wrong solutions or spurious failures whenever balancing scales the pencil.) + U.topRows(m) = (J.topLeftCorner(m, m) * U.topRows(m)).eval(); + U = HouseholderQR<DynamicMatrix>(U).householderQ() * DynamicMatrix::Identity(2 * m, m); + } + const auto U1 = U.topRows(m); + const auto U2 = U.bottomRows(m); + + // [U1; U2] has orthonormal columns, so ||X||_2 = sqrt(1 - sigma_min(U1)^2) / sigma_min(U1). rcond(U1) is scale + // invariant and misses a uniformly small U1, which an unstable mode that B does not reach produces from rounding + // errors. ||U1||_1 rcond(U1) estimates 1 / ||U1^-1||_1 <= sqrt(m) sigma_min(U1), so the second test bounds the + // solution of the (balanced) pencil by about sqrt(m) / (5 rcondThreshold). Stabilizing solutions beyond that are + // numerically indistinguishable from the absence of one. The factor 5 rejected all of 14400 random problems with + // unstable modes that B does not reach; with 1, 10% of them returned a meaningless X. + const PartialPivLU<DynamicMatrix> lu(U1); + const RealScalar rcondU1 = lu.rcond(); + if (!(rcondU1 >= rcondThreshold && internal::riccati_norm1(U1) * rcondU1 >= RealScalar(5) * rcondThreshold)) + return NumericalIssue; + + // A large deviation of U1^* U2 from selfadjointness indicates failure, see Theorem 5 in [2]. + const DynamicMatrix uSym = U1.adjoint() * U2; + // SciPy's threshold max(spacing(1000), 0.1 ||U1^* U2||_1), with spacing(1000) = 512 eps. + const RealScalar symThreshold = + numext::maxi(RealScalar(512) * NumTraits<RealScalar>::epsilon(), RealScalar(0.1) * internal::riccati_norm1(uSym)); + if (!(internal::riccati_norm1(uSym - uSym.adjoint()) <= symThreshold)) return NumericalIssue; + + // X = U2 U1^-1 is computed as (U1^-* U2^*)^*. + X = lu.adjoint().solve(U2.adjoint()); + X.adjointInPlace(); + if (balanced) X = balancing.asDiagonal() * X * balancing.asDiagonal(); + X = (X + X.adjoint()).eval() / RealScalar(2); + // The solve, the unscaling and the symmetrization can overflow. + if (!X.allFinite()) return NumericalIssue; + return Success; +} + +} // namespace Eigen + +#endif // EIGEN_DISCRETE_ALGEBRAIC_RICCATI_SOLVER_H
diff --git a/contrib/Eigen/src/Riccati/GeneralizedSchurReordering.h b/contrib/Eigen/src/Riccati/GeneralizedSchurReordering.h new file mode 100644 index 0000000..75adf58 --- /dev/null +++ b/contrib/Eigen/src/Riccati/GeneralizedSchurReordering.h
@@ -0,0 +1,275 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// This Source Code Form is subject to the terms of the Mozilla +// Public License v. 2.0. If a copy of the MPL was not distributed +// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_GENERALIZED_SCHUR_REORDERING_H +#define EIGEN_GENERALIZED_SCHUR_REORDERING_H + +// IWYU pragma: private +#include "./InternalHeaderCheck.h" + +namespace Eigen { + +namespace internal { + +/** \internal + * Size of the diagonal block of the generalized (quasi-)triangular matrix \p S starting at row/column \p k. + * Complex Schur forms only have 1x1 blocks; real ones also have 2x2 blocks for complex conjugate eigenvalue pairs, + * which are recognized by a nonzero subdiagonal entry of \p S (block boundaries are exact zeros). + */ +template <typename MatrixType> +Index generalized_schur_block_size(const MatrixType& S, Index k) { + if (NumTraits<typename MatrixType::Scalar>::IsComplex) return 1; + return (k + 1 < S.rows() && !numext::is_exactly_zero(S.coeff(k + 1, k))) ? 2 : 1; +} + +/** \internal + * Swaps the adjacent 1x1 diagonal blocks at row/column \p j of the generalized Schur form (\p S, \p T). + * + * As in LAPACK's xTGEX2, the rotations are formed directly from the entries rather than from a Sylvester solve, which + * preserves the relative accuracy of small components: with a = S(j:j+1, j:j+1) and b = T(j:j+1, j:j+1), the right + * eigenvector of the lower eigenvalue a22 / b22 is v = [g; -f] with f = a22 b11 - b22 a11 and g = a22 b12 - b22 a12, + * and the first column of the left rotation is a v or b v, whichever is computed from the larger diagonal products. + */ +template <typename MatrixType> +bool swap_generalized_schur_1x1(MatrixType& S, MatrixType& T, MatrixType* Q, MatrixType& Z, Index j) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + using Matrix2 = Matrix<Scalar, 2, 2>; + const Matrix2 a = S.template block<2, 2>(j, j); + const Matrix2 b = T.template block<2, 2>(j, j); + const Scalar f = a(1, 1) * b(0, 0) - b(1, 1) * a(0, 0); + const Scalar g = a(1, 1) * b(0, 1) - b(1, 1) * a(0, 1); + // f = g = 0 only for equal eigenvalues, which never need swapping, or a singular pencil. + if (numext::is_exactly_zero(f) && numext::is_exactly_zero(g)) return false; + JacobiRotation<Scalar> gz, gq; + gz.makeGivens(g, -f); + Matrix2 an = a, bn = b; + an.applyOnTheRight(0, 1, gz); + bn.applyOnTheRight(0, 1, gz); + if (numext::abs(a(1, 1)) * numext::abs(b(0, 0)) >= numext::abs(a(0, 0)) * numext::abs(b(1, 1))) + gq.makeGivens(an(0, 0), an(1, 0)); + else + gq.makeGivens(bn(0, 0), bn(1, 0)); + an.applyOnTheLeft(0, 1, gq.adjoint()); + bn.applyOnTheLeft(0, 1, gq.adjoint()); + + // Weak and strong stability tests of xTGEX2, with tau_a = max(20 eps ||a||_F, smlnum) and likewise tau_b. + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + const RealScalar smlnum = (std::numeric_limits<RealScalar>::min)() / eps; + const RealScalar threshA = numext::maxi(RealScalar(20) * eps * a.stableNorm(), smlnum); + const RealScalar threshB = numext::maxi(RealScalar(20) * eps * b.stableNorm(), smlnum); + if (!(numext::abs(an(1, 0)) <= threshA && numext::abs(bn(1, 0)) <= threshB)) return false; + an(1, 0) = Scalar(0); + bn(1, 0) = Scalar(0); + Matrix2 ra = an, rb = bn; + ra.applyOnTheLeft(0, 1, gq); + rb.applyOnTheLeft(0, 1, gq); + ra.applyOnTheRight(0, 1, gz.adjoint()); + rb.applyOnTheRight(0, 1, gz.adjoint()); + if (!((ra - a).stableNorm() <= threshA && (rb - b).stableNorm() <= threshB)) return false; + + S.template block<2, 2>(j, j) = an; + T.template block<2, 2>(j, j) = bn; + const Index right = S.cols() - j - 2; + if (right > 0) { + S.rightCols(right).applyOnTheLeft(j, j + 1, gq.adjoint()); + T.rightCols(right).applyOnTheLeft(j, j + 1, gq.adjoint()); + } + if (j > 0) { + S.topRows(j).applyOnTheRight(j, j + 1, gz); + T.topRows(j).applyOnTheRight(j, j + 1, gz); + } + if (Q != nullptr) Q->applyOnTheRight(j, j + 1, gq); + Z.applyOnTheRight(j, j + 1, gz); + return true; +} + +/** \internal + * Swaps the adjacent diagonal blocks of sizes \p n1 (upper) and \p n2 (lower), one of them 2x2, at row/column \p j of + * the real generalized Schur form (\p S, \p T), following B. Kagstrom and P. Poromaa, "Computing eigenspaces with + * specified eigenvalues of a regular matrix pair (A, B) and condition estimation: theory, algorithms and software", + * Numer. Algorithms 12 (1996), and LAPACK's xTGEX2. + * + * The solution of the generalized Sylvester equation + * A11 X - Y A22 = -A12, B11 X - Y B22 = -B12 + * gives the deflating subspaces of the lower block, spanned by [X; I] (right) and [Y; I] (left); the leading columns + * of Zl and Ql are orthonormal bases of these subspaces. The swapped B is then made upper triangular by a QR (from the + * left) or an RQ (from the right) factorization, whichever leaves the smaller lower left block of A. + */ +template <typename MatrixType> +bool swap_generalized_schur_sylvester(MatrixType& S, MatrixType& T, MatrixType* Q, MatrixType& Z, Index j, Index n1, + Index n2) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + using LocalMatrix = Matrix<Scalar, Dynamic, Dynamic, ColMajor, 4, 4>; + using SylvesterMatrix = Matrix<Scalar, Dynamic, Dynamic, ColMajor, 8, 8>; + using SylvesterVector = Matrix<Scalar, Dynamic, 1, ColMajor, 8, 1>; + const Index m = n1 + n2; + const Index k = n1 * n2; + const LocalMatrix A = S.block(j, j, m, m); + const LocalMatrix B = T.block(j, j, m, m); + + // K [vec(X); vec(Y)] = -[vec(A12); vec(B12)] with + // K = [I (x) A11, -A22^T (x) I; I (x) B11, -B22^T (x) I]. + SylvesterMatrix K = SylvesterMatrix::Zero(2 * k, 2 * k); + for (Index c = 0; c < n2; ++c) { + K.block(c * n1, c * n1, n1, n1) = A.topLeftCorner(n1, n1); + K.block(k + c * n1, c * n1, n1, n1) = B.topLeftCorner(n1, n1); + for (Index i = 0; i < n2; ++i) { + K.block(c * n1, k + i * n1, n1, n1).diagonal().setConstant(-A(n1 + i, n1 + c)); + K.block(k + c * n1, k + i * n1, n1, n1).diagonal().setConstant(-B(n1 + i, n1 + c)); + } + } + SylvesterVector rhs(2 * k); + rhs << -A.topRightCorner(n1, n2).reshaped(), -B.topRightCorner(n1, n2).reshaped(); + const FullPivLU<SylvesterMatrix> sylvester(K); + if (!sylvester.isInvertible()) return false; + const SylvesterVector sol = sylvester.solve(rhs); + + LocalMatrix basis(m, n2); + basis.topRows(n1) = sol.head(k).reshaped(n1, n2); + basis.bottomRows(n2).setIdentity(); + const LocalMatrix Zl = HouseholderQR<LocalMatrix>(basis).householderQ(); + basis.topRows(n1) = sol.tail(k).reshaped(n1, n2); + const LocalMatrix Ql = HouseholderQR<LocalMatrix>(basis).householderQ(); + const LocalMatrix An = Ql.adjoint() * A * Zl; + const LocalMatrix Bn = Ql.adjoint() * B * Zl; + + // QR: Bn = H R, so H^* (An, Bn) with Ql <- Ql H. + const LocalMatrix H = HouseholderQR<LocalMatrix>(Bn).householderQ(); + LocalMatrix Aq = H.adjoint() * An; + // RQ: Bn V is upper triangular for V = W P, where P reverses the order and M^* P = W R' is a QR factorization of + // the columns of M^* in reverse order; then (An, Bn) V with Zl <- Zl V. + const LocalMatrix W = HouseholderQR<LocalMatrix>(Bn.adjoint().rowwise().reverse()).householderQ(); + const LocalMatrix V = W.rowwise().reverse(); + LocalMatrix Ar = An * V; + + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + const RealScalar smlnum = (std::numeric_limits<RealScalar>::min)() / eps; + const RealScalar threshA = numext::maxi(RealScalar(20) * eps * A.stableNorm(), smlnum); + const RealScalar threshB = numext::maxi(RealScalar(20) * eps * B.stableNorm(), smlnum); + const RealScalar qrA21 = Aq.bottomLeftCorner(n1, n2).stableNorm(); + const RealScalar rqA21 = Ar.bottomLeftCorner(n1, n2).stableNorm(); + + // Weak stability test: ||A21||_F <= tau_a after the swap (B21 vanishes by construction). + LocalMatrix Qn, Zn, Sn, Tn; + if (qrA21 <= rqA21 && qrA21 <= threshA) { + Qn.noalias() = Ql * H; + Zn = Zl; + Sn = Aq; + Tn.noalias() = H.adjoint() * Bn; + } else if (rqA21 <= threshA) { + Qn = Ql; + Zn.noalias() = Zl * V; + Sn = Ar; + Tn.noalias() = Bn * V; + } else { + return false; + } + Sn.bottomLeftCorner(n1, n2).setZero(); + Tn.template triangularView<StrictlyLower>().setZero(); + + // Strong stability test: ||Qn Sn Zn^* - A||_F <= tau_a and ||Qn Tn Zn^* - B||_F <= tau_b. + if (!((Qn * Sn * Zn.adjoint() - A).stableNorm() <= threshA && (Qn * Tn * Zn.adjoint() - B).stableNorm() <= threshB)) + return false; + + S.block(j, j, m, m) = Sn; + T.block(j, j, m, m) = Tn; + const Index right = S.cols() - j - m; + if (right > 0) { + S.block(j, j + m, m, right) = Qn.adjoint() * S.block(j, j + m, m, right); + T.block(j, j + m, m, right) = Qn.adjoint() * T.block(j, j + m, m, right); + } + if (j > 0) { + S.block(0, j, j, m) = S.block(0, j, j, m) * Zn; + T.block(0, j, j, m) = T.block(0, j, j, m) * Zn; + } + if (Q != nullptr) Q->middleCols(j, m) = Q->middleCols(j, m) * Qn; + Z.middleCols(j, m) = Z.middleCols(j, m) * Zn; + return true; +} + +/** \internal + * Swaps the adjacent diagonal blocks of sizes \p n1 (upper) and \p n2 (lower) starting at row/column \p j of the + * generalized Schur form (\p S, \p T), where \p S is upper (quasi-)triangular and \p T is upper triangular. + * + * \p Q, unless null, and \p Z are updated such that A = Q S Z^* and B = Q T Z^* keep holding (as in LAPACK, \p Z is the + * adjoint of the matrix returned by RealQZ::matrixZ()). + * + * \returns false, leaving all matrices untouched, if the swap fails the weak or strong stability test of Kagstrom and + * Poromaa (1996), typically because the eigenvalues of both blocks are too close. + */ +template <typename MatrixType> +bool swap_generalized_schur_blocks(MatrixType& S, MatrixType& T, MatrixType* Q, MatrixType& Z, Index j, Index n1, + Index n2) { + eigen_assert(n1 >= 1 && n1 <= 2 && n2 >= 1 && n2 <= 2 && j >= 0 && j + n1 + n2 <= S.rows()); + if (n1 == 1 && n2 == 1) return swap_generalized_schur_1x1(S, T, Q, Z, j); + return swap_generalized_schur_sylvester(S, T, Q, Z, j, n1, n2); +} + +/** \internal + * Sets the entries below the structure of the generalized Schur form (\p S, \p T) to zero: below the diagonal of \p T, + * and below the diagonal (complex) or the first subdiagonal (real) of \p S. These are rounding errors left by the QZ + * iteration. + */ +template <typename MatrixType> +void clean_generalized_schur_form(MatrixType& S, MatrixType& T) { + const Index n = S.rows(); + T.template triangularView<StrictlyLower>().setZero(); + if (NumTraits<typename MatrixType::Scalar>::IsComplex) + S.template triangularView<StrictlyLower>().setZero(); + else if (n > 2) + S.bottomLeftCorner(n - 2, n - 2).template triangularView<Lower>().setZero(); +} + +/** \internal + * Reorders the generalized Schur form (\p S, \p T) with transformations \p Q (optional) and \p Z (see + * swap_generalized_schur_blocks()) such that all eigenvalues strictly inside the unit circle lead the diagonal, keeping + * their relative order. This corresponds to SciPy's `ordqz(..., sort='iuc')`; infinite eigenvalues are outside the unit + * circle. + * + * \returns the number of eigenvalues inside the unit circle, or -1 if a swap was rejected. + */ +template <typename MatrixType> +Index reorder_generalized_schur_inside_unit_circle(MatrixType& S, MatrixType& T, MatrixType* Q, MatrixType& Z) { + using Scalar = typename MatrixType::Scalar; + const Index n = S.rows(); + clean_generalized_schur_form(S, T); + + Index selected = 0; + for (Index k = 0; k < n;) { + const Index bs = generalized_schur_block_size(S, k); + bool inside; + if (bs == 1) { + inside = numext::abs(S(k, k)) < numext::abs(T(k, k)); + } else { + // Complex conjugate pair of equal modulus: |lambda|^2 = det(S_kk) / det(T_kk), with T_kk triangular. + inside = numext::abs(S.template block<2, 2>(k, k).determinant()) < numext::abs(T(k, k) * T(k + 1, k + 1)); + } + if (inside) { + // Bubble the block up past the unselected ones. + for (Index pos = k; pos > selected;) { + const Index pbs = + (!NumTraits<Scalar>::IsComplex && pos - 2 >= selected && !numext::is_exactly_zero(S(pos - 1, pos - 2))) ? 2 + : 1; + if (!swap_generalized_schur_blocks(S, T, Q, Z, pos - pbs, pbs, bs)) return -1; + pos -= pbs; + } + selected += bs; + } + k += bs; + } + return selected; +} + +} // namespace internal + +} // namespace Eigen + +#endif // EIGEN_GENERALIZED_SCHUR_REORDERING_H
diff --git a/contrib/Eigen/src/Riccati/InternalHeaderCheck.h b/contrib/Eigen/src/Riccati/InternalHeaderCheck.h new file mode 100644 index 0000000..eb104a5 --- /dev/null +++ b/contrib/Eigen/src/Riccati/InternalHeaderCheck.h
@@ -0,0 +1,6 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_RICCATI_MODULE_H +#error "Please include contrib/Eigen/Riccati instead of including headers inside the src directory directly." +#endif
diff --git a/contrib/Eigen/src/SpecialFunctions/BesselFunctionsImpl.h b/contrib/Eigen/src/SpecialFunctions/BesselFunctionsImpl.h index 331211f..a3e91a1 100644 --- a/contrib/Eigen/src/SpecialFunctions/BesselFunctionsImpl.h +++ b/contrib/Eigen/src/SpecialFunctions/BesselFunctionsImpl.h
@@ -192,7 +192,7 @@ #if defined(__FAST_MATH__) || defined(__ASSOCIATIVE_MATH__) || EIGEN_COMP_NVHPC // Unfortunately fast-math reassociates (exp(|x|/2) * exp(|x|/2)) * i0e(x) which reintroduces the overflow, // so we need to tell it not to. - EIGEN_OPTIMIZATION_BARRIER(scaled) + pbarrier(scaled); #endif // i0e(inf) = 0 would turn exp(inf) * i0e(inf) into NaN; i0(+-inf) = +inf. const T inf = pinf<T>(); @@ -353,7 +353,7 @@ T scaled = pmul(half_exp, i1e); #if defined(__FAST_MATH__) || defined(__ASSOCIATIVE_MATH__) || EIGEN_COMP_NVHPC // Reassociating back to (exp(|x|/2) * exp(|x|/2)) * i1e(x) would reintroduce the overflow. - EIGEN_OPTIMIZATION_BARRIER(scaled) + pbarrier(scaled); #endif // i1(+-inf) = +-inf; the scaled product would be inf * 0 = NaN. return pselect(pcmp_eq(ax, pinf<T>()), x, pmul(half_exp, scaled));
diff --git a/contrib/Eigen/src/SpecialFunctions/SpecialFunctionsImpl.h b/contrib/Eigen/src/SpecialFunctions/SpecialFunctionsImpl.h index c8cfb06..ba67256 100644 --- a/contrib/Eigen/src/SpecialFunctions/SpecialFunctionsImpl.h +++ b/contrib/Eigen/src/SpecialFunctions/SpecialFunctionsImpl.h
@@ -67,10 +67,11 @@ template <> struct lgamma_impl<float> { EIGEN_DEVICE_FUNC static EIGEN_STRONG_INLINE float run(float x) { -#if !defined(EIGEN_GPU_COMPILE_PHASE) && defined(EIGEN_HAS_LGAMMA_R) && !defined(__APPLE__) +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) && defined(EIGEN_HAS_LGAMMA_R) && \ + !defined(__APPLE__) int dummy; return ::lgammaf_r(x, &dummy); -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::lgamma(x); #else return ::lgammaf(x); @@ -81,10 +82,11 @@ template <> struct lgamma_impl<double> { EIGEN_DEVICE_FUNC static EIGEN_STRONG_INLINE double run(double x) { -#if !defined(EIGEN_GPU_COMPILE_PHASE) && defined(EIGEN_HAS_LGAMMA_R) && !defined(__APPLE__) +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) && defined(EIGEN_HAS_LGAMMA_R) && \ + !defined(__APPLE__) int dummy; return ::lgamma_r(x, &dummy); -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::lgamma(x); #else return ::lgamma(x); @@ -474,7 +476,7 @@ template <> struct erfc_impl<float> { EIGEN_DEVICE_FUNC static EIGEN_STRONG_INLINE float run(const float x) { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::erfc(x); #else return generic_fast_erfc<float>::run(x); @@ -485,7 +487,7 @@ template <> struct erfc_impl<double> { EIGEN_DEVICE_FUNC static EIGEN_STRONG_INLINE double run(const double x) { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::erfc(x); #else return generic_fast_erfc<double>::run(x); @@ -594,7 +596,7 @@ template <> struct erf_impl<float> { EIGEN_DEVICE_FUNC static EIGEN_STRONG_INLINE float run(const float x) { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::erf(x); #else return generic_fast_erf<float>::run(x); @@ -605,7 +607,7 @@ template <> struct erf_impl<double> { EIGEN_DEVICE_FUNC static EIGEN_STRONG_INLINE double run(const double x) { -#if defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::erf(x); #else return generic_fast_erf<double>::run(x);
diff --git a/contrib/Eigen/src/SpecialFunctions/arch/NEON/BesselFunctions.h b/contrib/Eigen/src/SpecialFunctions/arch/NEON/BesselFunctions.h index cfaa48d..b0d4306 100644 --- a/contrib/Eigen/src/SpecialFunctions/arch/NEON/BesselFunctions.h +++ b/contrib/Eigen/src/SpecialFunctions/arch/NEON/BesselFunctions.h
@@ -39,6 +39,7 @@ #endif // EIGEN_ARCH_ARM64 && EIGEN_HAS_ARM64_FP16 EIGEN_INSTANTIATE_BESSEL_FUNCS_BF16(Packet4f, Packet4bf) +EIGEN_INSTANTIATE_BESSEL_FUNCS_BF16_WITH(BF16_PACKET_FUNCTION_HALVES, Packet4f, Packet8bf) } // namespace internal } // namespace Eigen
diff --git a/contrib/Eigen/src/SpecialFunctions/arch/NEON/SpecialFunctions.h b/contrib/Eigen/src/SpecialFunctions/arch/NEON/SpecialFunctions.h index 3b27b9c..a7307af 100644 --- a/contrib/Eigen/src/SpecialFunctions/arch/NEON/SpecialFunctions.h +++ b/contrib/Eigen/src/SpecialFunctions/arch/NEON/SpecialFunctions.h
@@ -65,6 +65,7 @@ #endif // EIGEN_ARCH_ARM64 && EIGEN_HAS_ARM64_FP16 EIGEN_INSTANTIATE_SPECIAL_FUNCS_BF16(Packet4f, Packet4bf) +EIGEN_INSTANTIATE_SPECIAL_FUNCS_BF16_WITH(BF16_PACKET_FUNCTION_HALVES, Packet4f, Packet8bf) } // namespace internal } // namespace Eigen
diff --git a/contrib/Eigen/src/Splines/Spline.h b/contrib/Eigen/src/Splines/Spline.h index 74eca8e..06d70be 100644 --- a/contrib/Eigen/src/Splines/Spline.h +++ b/contrib/Eigen/src/Splines/Spline.h
@@ -448,10 +448,10 @@ /* Multiply through by the correct factors */ /* (Eq. [2.9]) */ - r = p; + Scalar factor = Scalar(1); for (DenseIndex k = 1; k <= static_cast<DenseIndex>(n); ++k) { - for (j = p; j >= 0; --j) N_(k, j) *= r; - r *= p - k; + factor *= Scalar(p - k + 1); + for (j = p; j >= 0; --j) N_(k, j) *= factor; } }
diff --git a/contrib/Eigen/src/Tensor/README.md b/contrib/Eigen/src/Tensor/README.md index d0ca8ca..a21ffb4 100644 --- a/contrib/Eigen/src/Tensor/README.md +++ b/contrib/Eigen/src/Tensor/README.md
@@ -696,13 +696,15 @@ Returns the n-th dimension of the tensor. The actual type of the `dimension()` result is `<Tensor-Type>::Index`, but you can -always use it like an int. +always use it like an int. The requested dimension can also be +supplied as a template parameter. ```cpp Eigen::Tensor<float, 2> a(3, 4); +auto dim0 = a.dimension<0>(); int dim1 = a.dimension(1); -std::cout << "Dim 1: " << dim1; -// Dim 1: 4 +std::cout << "Dim 0: " << dim0 << " Dim 1: " << dim1; +// Dim 0: 3 Dim 1: 4 ``` ### Index size()
diff --git a/contrib/Eigen/src/Tensor/Tensor.h b/contrib/Eigen/src/Tensor/Tensor.h index f3c7d29..4973a24 100644 --- a/contrib/Eigen/src/Tensor/Tensor.h +++ b/contrib/Eigen/src/Tensor/Tensor.h
@@ -97,7 +97,15 @@ public: // Metadata EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index rank() const { return NumIndices; } - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension(std::size_t n) const { return m_storage.dimensions()[n]; } + template <Index N> + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension() const { + EIGEN_STATIC_ASSERT(0 <= N && N < NumIndices, INVALID_DIMENSION); + return m_storage.dimensions()[N]; + } + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension(Index n) const { + eigen_assert(0 <= n && n < NumIndices); + return m_storage.dimensions()[n]; + } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE const Dimensions& dimensions() const { return m_storage.dimensions(); } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index size() const { return m_storage.size(); } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Scalar* data() { return m_storage.data(); }
diff --git a/contrib/Eigen/src/Tensor/TensorBlock.h b/contrib/Eigen/src/Tensor/TensorBlock.h index 66a4c7d..11102bc 100644 --- a/contrib/Eigen/src/Tensor/TensorBlock.h +++ b/contrib/Eigen/src/Tensor/TensorBlock.h
@@ -278,7 +278,10 @@ IndexType offset() const { return m_offset; } const Dimensions& dimensions() const { return m_dimensions; } - IndexType dimension(int index) const { return m_dimensions[index]; } + IndexType dimension(Index n) const { + eigen_internal_assert(0 <= n && n < NumDims); + return m_dimensions[n]; + } IndexType size() const { return array_prod<IndexType>(m_dimensions); } const DestinationBuffer& destination() const { return m_destination; }
diff --git a/contrib/Eigen/src/Tensor/TensorChipping.h b/contrib/Eigen/src/Tensor/TensorChipping.h index 188ae92..4f00eb7 100644 --- a/contrib/Eigen/src/Tensor/TensorChipping.h +++ b/contrib/Eigen/src/Tensor/TensorChipping.h
@@ -50,6 +50,102 @@ const DenseIndex actual_dim; }; +template <typename EvalT, typename PacketT, typename IndexType, typename = void> +struct chipping_has_partial_packet : bool_constant<false> {}; + +template <typename EvalT, typename PacketT, typename IndexType> +struct chipping_has_partial_packet< + EvalT, PacketT, IndexType, + void_t<decltype(std::declval<const EvalT>().template partialPacket<PacketT>( + std::declval<IndexType>(), std::declval<typename unpacket_traits<PacketT>::mask_t>()))>> + : bool_constant<unpacket_traits<PacketT>::masked_load_available && + std::is_same<PacketT, decltype(std::declval<const EvalT>().template partialPacket<PacketT>( + std::declval<IndexType>(), + std::declval<typename unpacket_traits<PacketT>::mask_t>()))>::value> {}; + +template <typename EvalT, typename PacketT, typename IndexType, + bool HasPartial = chipping_has_partial_packet<EvalT, PacketT, IndexType>::value> +struct chipping_forward_partial_packet { + static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketT run(const EvalT& impl, IndexType index, + typename unpacket_traits<PacketT>::mask_t umask) { + return impl.template partialPacket<PacketT>(index, umask); + } +}; + +template <typename EvalT, typename PacketT, typename IndexType> +struct chipping_forward_partial_packet<EvalT, PacketT, IndexType, false> { + static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketT run(const EvalT&, IndexType, + typename unpacket_traits<PacketT>::mask_t) { + eigen_assert(false && "unreachable"); + return pset1<PacketT>(typename unpacket_traits<PacketT>::type(0)); + } +}; + +template <typename EvalT, int LoadMode, typename PacketT, typename IndexType, typename = void> +struct chipping_has_packet_t : bool_constant<false> {}; + +template <typename EvalT, int LoadMode, typename PacketT, typename IndexType> +struct chipping_has_packet_t< + EvalT, LoadMode, PacketT, IndexType, + void_t<decltype(std::declval<const EvalT>().template packet<LoadMode, PacketT>(std::declval<IndexType>()))>> + : bool_constant<std::is_same<PacketT, decltype(std::declval<const EvalT>().template packet<LoadMode, PacketT>( + std::declval<IndexType>()))>::value> {}; + +template <int LoadMode, typename PacketT, typename ArgEvaluator, + bool SamePacket = std::is_same<PacketT, typename ArgEvaluator::PacketReturnType>::value, + bool HasPacketT = chipping_has_packet_t<ArgEvaluator, LoadMode, PacketT, typename ArgEvaluator::Index>::value, + bool IsRawPointer = std::is_convertible< + typename ArgEvaluator::EvaluatorPointerType, + const std::remove_const_t<std::remove_reference_t<typename ArgEvaluator::CoeffReturnType>>*>::value> +struct chipping_packet_load { + using Index = typename ArgEvaluator::Index; + using Scalar = std::remove_const_t<std::remove_reference_t<typename ArgEvaluator::CoeffReturnType>>; + static constexpr int kSize = unpacket_traits<PacketT>::size; + + static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketT run(const ArgEvaluator& impl, Index index) { + EIGEN_ALIGN_TO_BOUNDARY(unpacket_traits<PacketT>::alignment) Scalar values[kSize]; + EIGEN_UNROLL_LOOP + for (int i = 0; i < kSize; ++i) { + values[i] = impl.coeff(index + i); + } + return pload<PacketT>(values); + } +}; + +template <int LoadMode, typename PacketT, typename ArgEvaluator, bool HasPacketT, bool IsRawPointer> +struct chipping_packet_load<LoadMode, PacketT, ArgEvaluator, true, HasPacketT, IsRawPointer> { + using Index = typename ArgEvaluator::Index; + + static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketT run(const ArgEvaluator& impl, Index index) { + return impl.template packet<LoadMode>(index); + } +}; + +template <int LoadMode, typename PacketT, typename ArgEvaluator, bool IsRawPointer> +struct chipping_packet_load<LoadMode, PacketT, ArgEvaluator, false, true, IsRawPointer> { + using Index = typename ArgEvaluator::Index; + + static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketT run(const ArgEvaluator& impl, Index index) { + return impl.template packet<LoadMode, PacketT>(index); + } +}; + +template <int LoadMode, typename PacketT, typename ArgEvaluator> +struct chipping_packet_load<LoadMode, PacketT, ArgEvaluator, false, false, true> { + using Index = typename ArgEvaluator::Index; + using Scalar = std::remove_const_t<std::remove_reference_t<typename ArgEvaluator::CoeffReturnType>>; + + static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketT run(const ArgEvaluator& impl, Index index) { + const Scalar* src_ptr = impl.data(); + EIGEN_IF_CONSTEXPR (!static_cast<bool>(ArgEvaluator::RawAccess)) { + if (src_ptr == nullptr) { + return chipping_packet_load<LoadMode, PacketT, ArgEvaluator, false, false, false>::run(impl, index); + } + } + return ploadt_ro<PacketT, Unaligned>(src_ptr + index); + } +}; + } // end namespace internal /** A chip is a thin slice, corresponding to a column or a row in a 2-d tensor. @@ -163,7 +259,7 @@ m_inputOffset = m_stride * op.offset(); // Check if chipping is effectively inner or outer: product of dimensions - // before or after the chipped dimension is `1`. + // before or after the chipped dimension is `1`, or the chipped dimension itself has size `1`. Index after_chipped_dim_product = 1; for (int i = static_cast<int>(m_dim.actualDim()) + 1; i < NumInputDims; ++i) { after_chipped_dim_product *= input_dims[i]; @@ -174,12 +270,13 @@ before_chipped_dim_product *= input_dims[i]; } + const bool is_trivial_chip_dim = input_dims[m_dim.actualDim()] == 1; EIGEN_IF_CONSTEXPR (static_cast<int>(Layout) == static_cast<int>(ColMajor)) { - m_isEffectivelyInnerChipping = before_chipped_dim_product == 1; - m_isEffectivelyOuterChipping = after_chipped_dim_product == 1; + m_isEffectivelyInnerChipping = (before_chipped_dim_product == 1) && !is_trivial_chip_dim; + m_isEffectivelyOuterChipping = (after_chipped_dim_product == 1) || is_trivial_chip_dim; } else { - m_isEffectivelyInnerChipping = after_chipped_dim_product == 1; - m_isEffectivelyOuterChipping = before_chipped_dim_product == 1; + m_isEffectivelyInnerChipping = (after_chipped_dim_product == 1) && !is_trivial_chip_dim; + m_isEffectivelyOuterChipping = (before_chipped_dim_product == 1) || is_trivial_chip_dim; } } @@ -203,59 +300,157 @@ return m_impl.coeff(srcCoeff(index)); } - template <int LoadMode> - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketReturnType packet(Index index) const { - eigen_assert(index + PacketSize - 1 < dimensions().TotalSize()); + template <int LoadMode, typename PacketReturnTypeT = PacketReturnType> + EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE PacketReturnTypeT packet(Index index) const { + constexpr int RequestedPacketSize = internal::unpacket_traits<PacketReturnTypeT>::size; + EIGEN_STATIC_ASSERT((RequestedPacketSize <= PacketSize), YOU_MADE_A_PROGRAMMING_MISTAKE); + eigen_assert(index >= 0 && index + RequestedPacketSize - 1 < dimensions().TotalSize()); EIGEN_IF_CONSTEXPR (IsInnerChipping) { - return packetInnerChipping<LoadMode>(index); + return packetInnerChipping<LoadMode, PacketReturnTypeT>(index); } else EIGEN_IF_CONSTEXPR (IsOuterChipping) { - return packetOuterChipping<LoadMode>(index); + return packetOuterChipping<LoadMode, PacketReturnTypeT>(index); } else if (m_isEffectivelyInnerChipping) { - return packetInnerChipping<LoadMode>(index); + // index * m_inputStride is linear in the caller's loop index; keep clang-19+ LSR + // from hoisting per-lane induction variables across this runtime branch. + Index opaqueIndex = index; +#if EIGEN_COMP_CLANG >= 1900 && !defined(__apple_build_version__) && !defined(EIGEN_GPU_COMPILE_PHASE) && \ + !defined(EIGEN_SYCL_DEVICE_ONLY) + __asm__("" : "+r"(opaqueIndex)); +#endif + return packetInnerChipping<LoadMode, PacketReturnTypeT>(opaqueIndex); } else if (m_isEffectivelyOuterChipping) { - return packetOuterChipping<LoadMode>(index); + return packetOuterChipping<LoadMode, PacketReturnTypeT>(index); } else { const Index idx = index / m_stride; const Index rem = index - idx * m_stride; - if (rem + PacketSize <= m_stride) { - Index inputIndex = idx * m_inputStride + m_inputOffset + rem; - return m_impl.template packet<LoadMode>(inputIndex); + Index inputIndex = idx * m_inputStride + m_inputOffset + rem; + if (EIGEN_PREDICT_TRUE(rem + RequestedPacketSize <= m_stride)) { + return internal::chipping_packet_load<LoadMode, PacketReturnTypeT, TensorEvaluator<ArgType, Device>>::run( + m_impl, inputIndex); } else { - // Cross the stride boundary. Fallback to slow path. - EIGEN_ALIGN_TO_BOUNDARY(internal::unpacket_traits<PacketReturnType>::alignment) - std::remove_const_t<CoeffReturnType> values[PacketSize]; + EIGEN_ALIGN_TO_BOUNDARY(internal::unpacket_traits<PacketReturnTypeT>::alignment) + std::remove_const_t<CoeffReturnType> values[RequestedPacketSize]; + const Index strideDelta = m_inputStride - m_stride; + Index lane_in_slice = rem; EIGEN_UNROLL_LOOP - for (int i = 0; i < PacketSize; ++i) { - values[i] = coeff(index); - ++index; + for (int i = 0; i < RequestedPacketSize; ++i) { + values[i] = m_impl.coeff(inputIndex); + ++inputIndex; + if (++lane_in_slice == m_stride) { + lane_in_slice = 0; + inputIndex += strideDelta; + } } - PacketReturnType rslt = internal::pload<PacketReturnType>(values); - return rslt; + return internal::pload<PacketReturnTypeT>(values); } } } + template < + typename PacketReturnTypeT, + std::enable_if_t<PacketAccess && internal::unpacket_traits<PacketReturnTypeT>::masked_load_available, int> = 0> + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketReturnTypeT + partialPacket(Index index, typename internal::unpacket_traits<PacketReturnTypeT>::mask_t umask) const { + using MaskT = typename internal::unpacket_traits<PacketReturnTypeT>::mask_t; + using RawCoeffType = std::remove_const_t<CoeffReturnType>; + using ArgEval = TensorEvaluator<ArgType, Device>; + constexpr int RequestedPacketSize = internal::unpacket_traits<PacketReturnTypeT>::size; + constexpr int kMaskBits = static_cast<int>(CHAR_BIT * sizeof(MaskT)); + constexpr bool kHasImplPartial = internal::chipping_has_partial_packet<ArgEval, PacketReturnTypeT, Index>::value; + + EIGEN_IF_CONSTEXPR (RequestedPacketSize < kMaskBits) { + umask &= static_cast<MaskT>((MaskT(1) << (RequestedPacketSize < kMaskBits ? RequestedPacketSize : 0)) - 1); + } + if (umask == 0) { + return internal::pset1<PacketReturnTypeT>(CoeffReturnType(0)); + } + + const Index first_lane = static_cast<Index>(internal::ctz(umask)); + const Index last_lane = static_cast<Index>(kMaskBits - 1 - internal::clz(umask)); + const Index active_start = index + first_lane; + const Index active_count = last_lane - first_lane + 1; + eigen_assert(active_start >= 0 && active_start + active_count - 1 < dimensions().TotalSize()); + + if (isOuterChipping()) { + const Index target_index = index + m_inputOffset; + if (kHasImplPartial && EIGEN_PREDICT_TRUE(target_index >= 0)) { + return internal::chipping_forward_partial_packet<ArgEval, PacketReturnTypeT, Index>::run(m_impl, target_index, + umask); + } + EIGEN_ALIGN_TO_BOUNDARY(internal::unpacket_traits<PacketReturnTypeT>::alignment) + RawCoeffType values[RequestedPacketSize]; + EIGEN_UNROLL_LOOP + for (int i = 0; i < RequestedPacketSize; ++i) { + values[i] = RawCoeffType(0); + } + const Index srcOffset = srcCoeffOuterChipping(active_start); + for (Index lane = first_lane; lane <= last_lane; ++lane) { + if ((umask >> lane) & MaskT(1)) { + values[lane] = m_impl.coeff(srcOffset + (lane - first_lane)); + } + } + return internal::pload<PacketReturnTypeT>(values); + } + + if (isInnerChipping()) { + EIGEN_ALIGN_TO_BOUNDARY(internal::unpacket_traits<PacketReturnTypeT>::alignment) + RawCoeffType values[RequestedPacketSize]; + EIGEN_UNROLL_LOOP + for (int i = 0; i < RequestedPacketSize; ++i) { + values[i] = RawCoeffType(0); + } + Index inputIndex = srcCoeffInnerChipping(active_start); + for (Index lane = first_lane; lane <= last_lane; ++lane) { + if ((umask >> lane) & MaskT(1)) { + values[lane] = m_impl.coeff(inputIndex); + } + inputIndex += m_inputStride; + } + return internal::pload<PacketReturnTypeT>(values); + } + + const Index idx = active_start / m_stride; + const Index rem = active_start - idx * m_stride; + const Index sliceBase = idx * m_inputStride + m_inputOffset; + if (EIGEN_PREDICT_TRUE(rem + active_count <= m_stride)) { + const Index target_index = sliceBase + rem - first_lane; + if (kHasImplPartial && EIGEN_PREDICT_TRUE(target_index >= 0)) { + return internal::chipping_forward_partial_packet<ArgEval, PacketReturnTypeT, Index>::run(m_impl, target_index, + umask); + } + } + + EIGEN_ALIGN_TO_BOUNDARY(internal::unpacket_traits<PacketReturnTypeT>::alignment) + RawCoeffType values[RequestedPacketSize]; + EIGEN_UNROLL_LOOP + for (int i = 0; i < RequestedPacketSize; ++i) { + values[i] = RawCoeffType(0); + } + const Index strideDelta = m_inputStride - m_stride; + Index inputIndex = sliceBase + rem; + Index lane_in_slice = rem; + for (Index lane = first_lane; lane <= last_lane; ++lane) { + if ((umask >> lane) & MaskT(1)) { + values[lane] = m_impl.coeff(inputIndex); + } + ++inputIndex; + if (++lane_in_slice == m_stride) { + lane_in_slice = 0; + inputIndex += strideDelta; + } + } + return internal::pload<PacketReturnTypeT>(values); + } + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE TensorOpCost costPerCoeff(bool vectorized) const { double cost = 0; - EIGEN_IF_CONSTEXPR (static_cast<int>(Layout) == static_cast<int>(ColMajor)) { - if (m_dim.actualDim() == 0) { - cost += TensorOpCost::MulCost<Index>() + TensorOpCost::AddCost<Index>(); - } else if (m_dim.actualDim() == NumInputDims - 1) { - cost += TensorOpCost::AddCost<Index>(); - } else { - cost += - 3 * TensorOpCost::MulCost<Index>() + TensorOpCost::DivCost<Index>() + 3 * TensorOpCost::AddCost<Index>(); - } + if (isOuterChipping()) { + cost += TensorOpCost::AddCost<Index>(); + } else if (isInnerChipping()) { + cost += TensorOpCost::MulCost<Index>() + TensorOpCost::AddCost<Index>(); } else { - if (m_dim.actualDim() == NumInputDims - 1) { - cost += TensorOpCost::MulCost<Index>() + TensorOpCost::AddCost<Index>(); - } else if (m_dim.actualDim() == 0) { - cost += TensorOpCost::AddCost<Index>(); - } else { - cost += - 3 * TensorOpCost::MulCost<Index>() + TensorOpCost::DivCost<Index>() + 3 * TensorOpCost::AddCost<Index>(); - } + cost += 3 * TensorOpCost::MulCost<Index>() + TensorOpCost::DivCost<Index>() + 3 * TensorOpCost::AddCost<Index>(); } return m_impl.costPerCoeff(vectorized) + TensorOpCost(0, 0, cost, vectorized, PacketSize); @@ -351,28 +546,29 @@ } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index srcCoeffOuterChipping(Index index) const { - // m_stride is always greater than index, so let's avoid the integer division. - eigen_assert(m_stride > index); + // m_stride > index (or m_stride == m_inputStride when chipped dim has size 1). + eigen_assert(m_stride > index || m_stride == m_inputStride); return index + m_inputOffset; } - template <int LoadMode> - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketReturnType packetInnerChipping(Index index) const { - EIGEN_ALIGN_TO_BOUNDARY(internal::unpacket_traits<PacketReturnType>::alignment) - std::remove_const_t<CoeffReturnType> values[PacketSize]; + template <int LoadMode, typename PacketReturnTypeT = PacketReturnType> + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketReturnTypeT packetInnerChipping(Index index) const { + constexpr int RequestedPacketSize = internal::unpacket_traits<PacketReturnTypeT>::size; + EIGEN_ALIGN_TO_BOUNDARY(internal::unpacket_traits<PacketReturnTypeT>::alignment) + std::remove_const_t<CoeffReturnType> values[RequestedPacketSize]; Index inputIndex = srcCoeffInnerChipping(index); EIGEN_UNROLL_LOOP - for (int i = 0; i < PacketSize; ++i) { + for (int i = 0; i < RequestedPacketSize; ++i) { values[i] = m_impl.coeff(inputIndex); inputIndex += m_inputStride; } - PacketReturnType rslt = internal::pload<PacketReturnType>(values); - return rslt; + return internal::pload<PacketReturnTypeT>(values); } - template <int LoadMode> - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketReturnType packetOuterChipping(Index index) const { - return m_impl.template packet<LoadMode>(srcCoeffOuterChipping(index)); + template <int LoadMode, typename PacketReturnTypeT = PacketReturnType> + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE PacketReturnTypeT packetOuterChipping(Index index) const { + return internal::chipping_packet_load<LoadMode, PacketReturnTypeT, TensorEvaluator<ArgType, Device>>::run( + m_impl, srcCoeffOuterChipping(index)); } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE bool isInnerChipping() const { @@ -392,7 +588,8 @@ const Device EIGEN_DEVICE_REF m_device; // If product of all dimensions after or before the chipped dimension is `1`, - // it is effectively the same as chipping innermost or outermost dimension. + // or the chipped dimension itself has size `1`, it is effectively the same + // as chipping innermost or outermost dimension. bool m_isEffectivelyInnerChipping; bool m_isEffectivelyOuterChipping; }; @@ -444,18 +641,23 @@ } else { const Index idx = index / this->m_stride; const Index rem = index - idx * this->m_stride; - if (rem + PacketSize <= this->m_stride) { - const Index inputIndex = idx * this->m_inputStride + this->m_inputOffset + rem; + Index inputIndex = idx * this->m_inputStride + this->m_inputOffset + rem; + if (EIGEN_PREDICT_TRUE(rem + PacketSize <= this->m_stride)) { this->m_impl.template writePacket<StoreMode>(inputIndex, x); } else { - // Cross stride boundary. Fallback to slow path. EIGEN_ALIGN_TO_BOUNDARY(internal::unpacket_traits<PacketReturnType>::alignment) std::remove_const_t<CoeffReturnType> values[PacketSize]; internal::pstore<CoeffReturnType, PacketReturnType>(values, x); + const Index strideDelta = this->m_inputStride - this->m_stride; + Index lane_in_slice = rem; EIGEN_UNROLL_LOOP for (int i = 0; i < PacketSize; ++i) { - this->coeffRef(index) = values[i]; - ++index; + this->m_impl.coeffRef(inputIndex) = values[i]; + ++inputIndex; + if (++lane_in_slice == this->m_stride) { + lane_in_slice = 0; + inputIndex += strideDelta; + } } } }
diff --git a/contrib/Eigen/src/Tensor/TensorConcatenation.h b/contrib/Eigen/src/Tensor/TensorConcatenation.h index 3fc2908..45ac5be 100644 --- a/contrib/Eigen/src/Tensor/TensorConcatenation.h +++ b/contrib/Eigen/src/Tensor/TensorConcatenation.h
@@ -182,7 +182,12 @@ // TODO(phli): Add short-circuit memcpy evaluation if underlying data are linear. EIGEN_STRONG_INLINE bool evalSubExprsIfNeeded(EvaluatorPointerType) { m_leftImpl.evalSubExprsIfNeeded(nullptr); - m_rightImpl.evalSubExprsIfNeeded(nullptr); + // The executor does not call cleanup() when evaluation throws, so release the operand evaluated first. + EIGEN_TRY { m_rightImpl.evalSubExprsIfNeeded(nullptr); } + EIGEN_CATCH(...) { + m_leftImpl.cleanup(); + EIGEN_THROW; + } return true; }
diff --git a/contrib/Eigen/src/Tensor/TensorContraction.h b/contrib/Eigen/src/Tensor/TensorContraction.h index e3e1f8c..3148e41 100644 --- a/contrib/Eigen/src/Tensor/TensorContraction.h +++ b/contrib/Eigen/src/Tensor/TensorContraction.h
@@ -60,7 +60,9 @@ }; // Helper class to allocate and deallocate temporary memory for packed buffers. -template <typename LhsScalar, typename RhsScalar> +template <typename LhsScalar, typename RhsScalar, + bool RequireInitialization = + NumTraits<LhsScalar>::RequireInitialization || NumTraits<RhsScalar>::RequireInitialization> struct TensorContractionBlockMemAllocator { using BlockMemHandle = void*; @@ -125,6 +127,87 @@ } }; +// Packed blocks of scalars that require initialization (e.g. AutoDiffScalar) are constructed after allocation and +// destructed before deallocation. The handle records the blocks so that deallocate() can destruct them. +template <typename LhsScalar, typename RhsScalar> +struct TensorContractionBlockMemAllocator<LhsScalar, RhsScalar, true> { + using RawAllocator = TensorContractionBlockMemAllocator<LhsScalar, RhsScalar, false>; + + struct BlockMemHandle { + void* mem = nullptr; + Index lhs_block_size = 0; + Index rhs_block_size = 0; + std::vector<LhsScalar*> lhs_blocks; + std::vector<RhsScalar*> rhs_blocks; + }; + + template <typename Device> + static BlockMemHandle allocate(Device& d, const Index bm, const Index bk, const Index bn, LhsScalar** lhs_block, + RhsScalar** rhs_block) { + BlockMemHandle handle; + handle.mem = RawAllocator::allocate(d, bm, bk, bn, lhs_block, rhs_block); + EIGEN_TRY { + handle.lhs_blocks.push_back(*lhs_block); + handle.rhs_blocks.push_back(*rhs_block); + construct(handle, bm * bk, bn * bk); + } + EIGEN_CATCH(...) { + RawAllocator::deallocate(d, handle.mem); + EIGEN_THROW; + } + return handle; + } + + template <typename Device> + static BlockMemHandle allocateSlices(Device& d, const Index bm, const Index bk, const Index bn, const Index num_lhs, + const Index num_rhs, const Index num_slices, std::vector<LhsScalar*>* lhs_blocks, + std::vector<RhsScalar*>* rhs_blocks) { + BlockMemHandle handle; + handle.mem = RawAllocator::allocateSlices(d, bm, bk, bn, num_lhs, num_rhs, num_slices, lhs_blocks, rhs_blocks); + EIGEN_TRY { + for (Index x = 0; x < num_slices; x++) { + if (num_lhs > 0) handle.lhs_blocks.insert(handle.lhs_blocks.end(), lhs_blocks[x].begin(), lhs_blocks[x].end()); + if (num_rhs > 0) handle.rhs_blocks.insert(handle.rhs_blocks.end(), rhs_blocks[x].begin(), rhs_blocks[x].end()); + } + construct(handle, bm * bk, bn * bk); + } + EIGEN_CATCH(...) { + RawAllocator::deallocate(d, handle.mem); + EIGEN_THROW; + } + return handle; + } + + template <typename Device> + static void deallocate(Device& d, const BlockMemHandle& handle) { + for (LhsScalar* block : handle.lhs_blocks) destruct_elements_of_array(block, handle.lhs_block_size); + for (RhsScalar* block : handle.rhs_blocks) destruct_elements_of_array(block, handle.rhs_block_size); + RawAllocator::deallocate(d, handle.mem); + } + + private: + // If a constructor throws, the blocks constructed so far are destructed before the exception propagates. + static void construct(BlockMemHandle& handle, const Index lhs_block_size, const Index rhs_block_size) { + std::size_t lhs_done = 0; + std::size_t rhs_done = 0; + EIGEN_TRY { + for (; lhs_done < handle.lhs_blocks.size(); ++lhs_done) { + default_construct_elements_of_array(handle.lhs_blocks[lhs_done], lhs_block_size); + } + for (; rhs_done < handle.rhs_blocks.size(); ++rhs_done) { + default_construct_elements_of_array(handle.rhs_blocks[rhs_done], rhs_block_size); + } + } + EIGEN_CATCH(...) { + while (rhs_done > 0) destruct_elements_of_array(handle.rhs_blocks[--rhs_done], rhs_block_size); + while (lhs_done > 0) destruct_elements_of_array(handle.lhs_blocks[--lhs_done], lhs_block_size); + EIGEN_THROW; + } + handle.lhs_block_size = lhs_block_size; + handle.rhs_block_size = rhs_block_size; + } +}; + // WARNING: In this code we assume that Lhs and Rhs tensor expressions are in // ColMajor storage order. This property is guaranteed by the // TensorContractionOp evaluator. TensorContractionKernel specifies how we pack @@ -196,7 +279,7 @@ } template <typename Device> - EIGEN_DEVICE_FUNC static void deallocate(Device& d, BlockMemHandle handle) { + EIGEN_DEVICE_FUNC static void deallocate(Device& d, const BlockMemHandle& handle) { BlockMemAllocator::deallocate(d, handle); } @@ -586,6 +669,7 @@ m_rhs_inner_dim_contiguous = false; } } + eigen_assert(!m_rhs_inner_dim_reordered || (ContractDims > 1 && !m_rhs_inner_dim_contiguous)); // After the sort above, eval_op_indices[*].first is in ascending order. The // contracted LHS dims are the contiguous prefix of storage iff that prefix @@ -620,15 +704,26 @@ EIGEN_STRONG_INLINE bool evalSubExprsIfNeeded(EvaluatorPointerType data) { m_leftImpl.evalSubExprsIfNeeded(nullptr); - m_rightImpl.evalSubExprsIfNeeded(nullptr); - if (data) { - evalTo(data); - return false; - } else { - m_result = static_cast<EvaluatorPointerType>(m_device.allocate(dimensions().TotalSize() * sizeof(Scalar))); - evalTo(m_result); - return true; + // The executor does not call cleanup() when evaluation throws, so release the operands and the result here. + EIGEN_TRY { m_rightImpl.evalSubExprsIfNeeded(nullptr); } + EIGEN_CATCH(...) { + m_leftImpl.cleanup(); + EIGEN_THROW; } + EIGEN_TRY { + if (data) { + evalTo(data); + } else { + m_result = static_cast<EvaluatorPointerType>(m_device.allocate(dimensions().TotalSize() * sizeof(Scalar))); + constructResult(); + evalTo(m_result); + } + } + EIGEN_CATCH(...) { + cleanup(); + EIGEN_THROW; + } + return data == nullptr; } #ifdef EIGEN_USE_THREADS @@ -636,12 +731,15 @@ EIGEN_STRONG_INLINE void evalSubExprsIfNeededAsync(EvaluatorPointerType dest, EvalSubExprsCallback done) { m_leftImpl.evalSubExprsIfNeededAsync(nullptr, [this, done, dest](bool) { m_rightImpl.evalSubExprsIfNeededAsync(nullptr, [this, done, dest](bool) { - if (dest) { - evalToAsync(dest, [done]() { done(false); }); - } else { + const bool need_assign = (dest == nullptr); + EvaluatorPointerType buffer = dest; + if (need_assign) { m_result = static_cast<EvaluatorPointerType>(m_device.allocate(dimensions().TotalSize() * sizeof(Scalar))); - evalToAsync(m_result, [done]() { done(true); }); + constructResult(); + buffer = m_result; } + // One callback type for both branches: each distinct closure type instantiates the whole evalProductAsync tree. + evalToAsync(buffer, [done, need_assign]() { done(need_assign); }); }); }); } @@ -658,6 +756,23 @@ } #endif // EIGEN_USE_THREADS + template <typename Func> + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE static void dispatchContraction(Func&& fn, bool lhs_inner_dim_contiguous, + bool rhs_inner_dim_contiguous, + bool rhs_inner_dim_reordered) { + // m_rhs_inner_dim_reordered is only ever set when ContractDims > 1 and + // !m_rhs_inner_dim_contiguous. Canonicalizing rhs_r at compile time avoids + // instantiating unreachable contractions (reducing 8 combinations to 4 when + // ContractDims <= 1, and to 6 when ContractDims > 1). + internal::tensor_contraction_dispatch( + [&](auto lhs_c, auto rhs_c, auto rhs_r) { + constexpr bool kRhsReordered = (ContractDims > 1) && !rhs_c() && rhs_r(); + fn(lhs_c, rhs_c, internal::bool_constant<kRhsReordered>{}); + }, + lhs_inner_dim_contiguous, rhs_inner_dim_contiguous, + (ContractDims > 1) && !rhs_inner_dim_contiguous && rhs_inner_dim_reordered); + } + template <bool lhs_inner_dim_contiguous, bool rhs_inner_dim_contiguous, bool rhs_inner_dim_reordered, int Alignment> void evalProductSequential(Scalar* buffer) const { if (this->m_i_size == 0 || this->m_j_size == 0) { @@ -906,6 +1021,13 @@ using BlockMemHandle = typename TensorContractionKernel::BlockMemHandle; const BlockMemHandle packed_mem = kernel.allocate(this->m_device, &blockA, &blockB); + // Releases the packed blocks also when packing, gebp or the output kernel throws. + struct PackedMemGuard { + TensorContractionKernel& kernel; + const Device& device; + const BlockMemHandle& mem; + EIGEN_DEVICE_FUNC ~PackedMemGuard() { kernel.deallocate(device, mem); } + } packed_mem_guard{kernel, this->m_device, packed_mem}; // If a contraction kernel does not support beta, explicitly initialize // output buffer with zeroes. @@ -943,18 +1065,12 @@ } } } - - kernel.deallocate(this->m_device, packed_mem); } EIGEN_STRONG_INLINE void cleanup() { m_leftImpl.cleanup(); m_rightImpl.cleanup(); - - if (m_result != nullptr) { - m_device.deallocate(m_result); - m_result = nullptr; - } + releaseResult(); } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE CoeffReturnType coeff(Index index) const { return m_result[index]; } @@ -982,6 +1098,32 @@ } protected: + // The temporary result is written by assignment (fill, gebp), so scalars that require initialization must be + // constructed first. The GPU evaluator allocates m_result itself without constructing it, so cleanup() destructs only + // what was constructed here. + EIGEN_STRONG_INLINE void constructResult() { + EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::RequireInitialization) { + EIGEN_TRY { internal::default_construct_elements_of_array(m_result, dimensions().TotalSize()); } + EIGEN_CATCH(...) { + m_device.deallocate(m_result); + m_result = nullptr; + EIGEN_THROW; + } + m_result_constructed = true; + } + } + + EIGEN_STRONG_INLINE void releaseResult() { + if (m_result != nullptr) { + EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::RequireInitialization) { + if (m_result_constructed) internal::destruct_elements_of_array(m_result, dimensions().TotalSize()); + m_result_constructed = false; + } + m_device.deallocate(m_result); + m_result = nullptr; + } + } + Dimensions m_dimensions; contract_t m_k_strides{}; @@ -1014,6 +1156,7 @@ const Device EIGEN_DEVICE_REF m_device; OutputKernelType m_output_kernel; EvaluatorPointerType m_result; + bool m_result_constructed = false; }; // evaluator for default device @@ -1058,7 +1201,7 @@ template <int Alignment> void evalProduct(Scalar* buffer) const { - internal::tensor_contraction_dispatch( + Base::dispatchContraction( [&](auto lhs_c, auto rhs_c, auto rhs_r) { this->template evalProductSequential<lhs_c(), rhs_c(), rhs_r(), Alignment>(buffer); },
diff --git a/contrib/Eigen/src/Tensor/TensorContractionThreadPool.h b/contrib/Eigen/src/Tensor/TensorContractionThreadPool.h index 2b40248..9f3af15 100644 --- a/contrib/Eigen/src/Tensor/TensorContractionThreadPool.h +++ b/contrib/Eigen/src/Tensor/TensorContractionThreadPool.h
@@ -83,6 +83,8 @@ evalProductImpl<EvalToCallback, Alignment>(buffer, std::move(done)); } + using Base::dispatchContraction; + template <typename DoneCallback, int Alignment> void evalProductImpl(Scalar* buffer, DoneCallback done) const { // This function computes a lot of heuristics in multiple steps, and it @@ -110,7 +112,7 @@ return; } if (k == 0) { - internal::tensor_contraction_dispatch( + dispatchContraction( [&](auto lhs_c, auto rhs_c, auto rhs_r) { this->template evalProductSequential<lhs_c(), rhs_c(), rhs_r(), Unaligned>(buffer); }, @@ -189,7 +191,7 @@ if (n == 1) num_threads = 1; if (num_threads == 1) { - internal::tensor_contraction_dispatch( + dispatchContraction( [&](auto lhs_c, auto rhs_c, auto rhs_r) { this->template evalProductSequential<lhs_c(), rhs_c(), rhs_r(), Unaligned>(buffer); }, @@ -275,15 +277,15 @@ // optimization. if (parallelize_by_sharding_dim_only) parallel_pack = false; - internal::tensor_contraction_dispatch( + dispatchContraction( [&](auto lhs_c, auto rhs_c, auto rhs_r) { EIGEN_IF_CONSTEXPR (IsEvalInSyncMode) { - EvalParallelContext<NoCallback, lhs_c(), rhs_c(), rhs_r(), Alignment> ctx( + EvalParallelContext<NoCallback, lhs_c(), rhs_c(), rhs_r()> ctx( this, num_threads, buffer, m, n, k, bm, bn, bk, nm, nn, nk, gm, gn, nm0, nn0, shard_by_col, parallel_pack, parallelize_by_sharding_dim_only, NoCallback()); ctx.run(); } else { - auto* ctx = new EvalParallelContext<DoneCallback, lhs_c(), rhs_c(), rhs_r(), Alignment>( + auto* ctx = new EvalParallelContext<DoneCallback, lhs_c(), rhs_c(), rhs_r()>( this, num_threads, buffer, m, n, k, bm, bn, bk, nm, nn, nk, gm, gn, nm0, nn0, shard_by_col, parallel_pack, parallelize_by_sharding_dim_only, std::move(done)); ctx->run(); @@ -348,7 +350,7 @@ // accessible by block packing and kernel tasks. template <typename DoneCallback, bool lhs_inner_dim_contiguous, bool rhs_inner_dim_contiguous, - bool rhs_inner_dim_reordered, int Alignment> + bool rhs_inner_dim_reordered> class EvalParallelContext { public: using LhsMapper = @@ -432,42 +434,59 @@ } } - // Allocate memory for packed rhs/lhs matrices. - packed_mem_ = kernel_.allocateSlices( // - device_, // - /*num_lhs=*/nm0_, // - /*num_rhs=*/nn0_, // - /*num_slices=*/std::min<Index>(nk_, P - 1), // - packed_lhs_, packed_rhs_); + // Constructing the packed blocks can throw for scalars that require initialization. The destructor does not run + // then, so release what was allocated here, and notify a synchronous waiter so that its destructor accepts it. + bool packed_mem_allocated = false; + can_use_thread_local_packed_ = nullptr; + EIGEN_TRY { + // Allocate memory for packed rhs/lhs matrices. + packed_mem_ = kernel_.allocateSlices( // + device_, // + /*num_lhs=*/nm0_, // + /*num_rhs=*/nn0_, // + /*num_slices=*/std::min<Index>(nk_, P - 1), // + packed_lhs_, packed_rhs_); + packed_mem_allocated = true; - if (parallelize_by_sharding_dim_only_) { - const int num_worker_threads = device_.numThreadsInPool(); + if (parallelize_by_sharding_dim_only_) { + const int num_worker_threads = device_.numThreadsInPool(); - if (shard_by_col) { - can_use_thread_local_packed_ = new std::atomic<bool>[nn_]; - for (int i = 0; i < nn_; ++i) can_use_thread_local_packed_[i].store(true, std::memory_order_relaxed); + if (shard_by_col) { + can_use_thread_local_packed_ = new std::atomic<bool>[nn_]; + for (int i = 0; i < nn_; ++i) can_use_thread_local_packed_[i].store(true, std::memory_order_relaxed); - Index num_blocks = num_worker_threads * gn_; - thread_local_pre_allocated_mem_ = kernel_.allocateSlices( // - device_, // - /*num_lhs=*/0, // - /*num_rhs=*/num_blocks, // - /*num_slices=*/1, // - /*lhs_blocks=*/nullptr, &rhs_thread_local_pre_allocated_); + Index num_blocks = num_worker_threads * gn_; + thread_local_pre_allocated_mem_ = kernel_.allocateSlices( // + device_, // + /*num_lhs=*/0, // + /*num_rhs=*/num_blocks, // + /*num_slices=*/1, // + /*lhs_blocks=*/nullptr, &rhs_thread_local_pre_allocated_); - } else { - can_use_thread_local_packed_ = new std::atomic<bool>[nm_]; - for (int i = 0; i < nm_; ++i) can_use_thread_local_packed_[i].store(true, std::memory_order_relaxed); + } else { + can_use_thread_local_packed_ = new std::atomic<bool>[nm_]; + for (int i = 0; i < nm_; ++i) can_use_thread_local_packed_[i].store(true, std::memory_order_relaxed); - Index num_blocks = num_worker_threads * gm_; - thread_local_pre_allocated_mem_ = kernel_.allocateSlices( // - device_, // - /*num_lhs=*/num_blocks, // - /*num_rhs=*/0, // - /*num_slices=*/1, &lhs_thread_local_pre_allocated_, // - /*rhs_blocks=*/nullptr); + Index num_blocks = num_worker_threads * gm_; + thread_local_pre_allocated_mem_ = kernel_.allocateSlices( // + device_, // + /*num_lhs=*/num_blocks, // + /*num_rhs=*/0, // + /*num_slices=*/1, &lhs_thread_local_pre_allocated_, // + /*rhs_blocks=*/nullptr); + } } } + EIGEN_CATCH(...) { + if (packed_mem_allocated) kernel_.deallocate(device_, packed_mem_); + delete[] can_use_thread_local_packed_; + for (Index x = 0; x < P; x++) { + for (Index m = 0; m < nm_; m++) delete[] state_kernel_[x][m]; + delete[] state_kernel_[x]; + } + EIGEN_IF_CONSTEXPR (std::is_same<DoneCallback, NoCallback>::value) done_.Notify(); + EIGEN_THROW; + } } ~EvalParallelContext() { @@ -482,7 +501,9 @@ } } - void run() { + // The calling thread runs part of the work after tasks are enqueued, and those tasks use this context. A throw here + // cannot unwind safely, so it terminates, as a throw on a worker does. + void run() noexcept { // Kick off packing of the first slice. signal_switch(0, 1); @@ -1062,15 +1083,37 @@ l0_state.emplace_back(internal::convert_index<int>(num_pending_tasks)); } - // Allocate temporary buffers for each block. - for (Index block_idx = 0; block_idx < num_blocks; ++block_idx) { - Scalar* buf = block_idx == 0 ? result : static_cast<Scalar*>(evaluator->m_device.allocate(buffer_size_bytes)); - block_buffers.emplace_back(buf); + // Allocate temporary buffers for each block. The destructor does not run if this throws, so release them here. + EIGEN_TRY { + for (Index block_idx = 0; block_idx < num_blocks; ++block_idx) { + block_buffers.emplace_back(block_idx == 0 ? result : allocateBlockBuffer()); + } + } + EIGEN_CATCH(...) { + releaseBlockBuffers(); + EIGEN_THROW; } } - ~EvalShardedByInnerDimContext() { - for (Index i = 1; i < num_blocks; ++i) { + ~EvalShardedByInnerDimContext() { releaseBlockBuffers(); } + + Scalar* allocateBlockBuffer() const { + Scalar* buf = static_cast<Scalar*>(evaluator->m_device.allocate(buffer_size_bytes)); + EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::RequireInitialization) { + EIGEN_TRY { internal::default_construct_elements_of_array(buf, m * n); } + EIGEN_CATCH(...) { + evaluator->m_device.deallocate(buf); + EIGEN_THROW; + } + } + return buf; + } + + void releaseBlockBuffers() { + for (size_t i = 1; i < block_buffers.size(); ++i) { + EIGEN_IF_CONSTEXPR (NumTraits<Scalar>::RequireInitialization) { + internal::destruct_elements_of_array(block_buffers[i], m * n); + } evaluator->m_device.deallocate(block_buffers[i]); } } @@ -1078,8 +1121,7 @@ template <int Alignment> void run() { Barrier barrier(internal::convert_index<int>(num_blocks)); - eval<Alignment>(barrier, 0, num_blocks); - barrier.Wait(); + evalBlocks<Alignment>(barrier); // Aggregate partial sums from l0 ranges. aggregateL0Blocks<Alignment>(); @@ -1088,8 +1130,16 @@ applyOutputKernel(); } + // The calling thread evaluates a block after tasks are enqueued, and those tasks use this context and the barrier. + // A throw here cannot unwind safely, so it terminates, as a throw on a worker does. template <int Alignment> - void runAsync() { + void evalBlocks(Barrier& barrier) noexcept { + eval<Alignment>(barrier, 0, num_blocks); + barrier.Wait(); + } + + template <int Alignment> + void runAsync() noexcept { evalAsync<Alignment>(0, num_blocks); } @@ -1154,7 +1204,7 @@ void processBlock(Index block_idx, Index begin, Index end) { Scalar* buf = block_buffers[block_idx]; - internal::tensor_contraction_dispatch( + dispatchContraction( [&](auto lhs_c, auto rhs_c, auto rhs_r) { evaluator->template evalGemmPartialWithoutOutputKernel<lhs_c(), rhs_c(), rhs_r(), Alignment>( buf, begin, end, /*num_threads=*/internal::convert_index<int>(num_blocks));
diff --git a/contrib/Eigen/src/Tensor/TensorConvolution.h b/contrib/Eigen/src/Tensor/TensorConvolution.h index 70357aa..8dab047 100644 --- a/contrib/Eigen/src/Tensor/TensorConvolution.h +++ b/contrib/Eigen/src/Tensor/TensorConvolution.h
@@ -653,6 +653,40 @@ } } +// Oversized kernels cannot stage even one output's input footprint in shared memory. +template <typename InputEvaluator, typename Index, typename InputDims> +__global__ void EigenConvolutionKernel3DGlobal( + InputEvaluator eval, const internal::IndexMapper<Index, InputDims, 3, InputEvaluator::Layout> indexMapper, + const float* kernel, Index total, Index numX, Index numY, Index numZ, Index kernelSizeX, Index kernelSizeY, + Index kernelSizeZ, float* buffer) { + const Index step = Index(blockDim.x) * gridDim.x; + for (Index output = Index(blockIdx.x) * blockDim.x + threadIdx.x; output < total;) { + Index remaining = output; + const Index x = remaining % numX; + remaining /= numX; + const Index y = remaining % numY; + remaining /= numY; + const Index z = remaining % numZ; + const Index plane = remaining / numZ; + const Index inputOffset = indexMapper.mapGpuInputPlaneToTensorInputOffset(plane); + float result = 0; + Index kernelIndex = 0; + for (Index k = 0; k < kernelSizeZ; ++k) { + for (Index j = 0; j < kernelSizeY; ++j) { + for (Index i = 0; i < kernelSizeX; ++i) { + result += eval.coeff(inputOffset + indexMapper.mapGpuInputKernelToTensorInputOffset(x + i, y + j, z + k)) * + kernel[kernelIndex++]; + } + } + } + buffer[indexMapper.mapGpuOutputPlaneToTensorOutputOffset(plane) + + indexMapper.mapGpuOutputKernelToTensorOutputOffset(x, y, z)] = result; + // Test the remaining distance before adding: even the final increment must not overflow Index. + if (step >= total - output) break; + output += step; + } +} + template <typename InputEvaluator, typename Index, typename InputDims> __global__ EIGEN_HIP_LAUNCH_BOUNDS_1024 void EigenConvolutionKernel3D( InputEvaluator eval, const internal::IndexMapper<Index, InputDims, 3, InputEvaluator::Layout> indexMapper, @@ -1014,8 +1048,28 @@ const int numX = dimensions()[m_indices[idxX]]; const int numY = dimensions()[m_indices[idxY]]; const int numZ = dimensions()[m_indices[idxZ]]; - const int numP = dimensions().TotalSize() / (numX * numY * numZ); + const Index total = dimensions().TotalSize(); + if (total == 0) return; + const array<Index, 3> indices{m_indices[idxX], m_indices[idxY], m_indices[idxZ]}; + const array<Index, 3> kernel_dims{m_kernelImpl.dimensions()[idxX], m_kernelImpl.dimensions()[idxY], + m_kernelImpl.dimensions()[idxZ]}; + internal::IndexMapper<Index, InputDims, 3, Layout> indexMapper(m_inputImpl.dimensions(), kernel_dims, indices); + + if (m_kernelImpl.dimensions().TotalSize() > Index(maxSharedMem / sizeof(Scalar))) { + constexpr int preferredBlockSize = 256; + const int threads = numext::mini(preferredBlockSize, maxThreadsPerBlock); + const int maxBlocks = static_cast<int>( + numext::mini<int64_t>(int64_t(numMultiProcessors) * 32, NumTraits<Index>::highest() / threads)); + const int blocks = static_cast<int>(numext::mini<Index>((total - 1) / threads + 1, Index(maxBlocks))); + LAUNCH_GPU_KERNEL( + (EigenConvolutionKernel3DGlobal<TensorEvaluator<InputArgType, GpuDevice>, Index, InputDims>), blocks, + threads, 0, m_device, m_inputImpl, indexMapper, m_kernel, total, Index(numX), Index(numY), Index(numZ), + Index(kernel_size_x), Index(kernel_size_y), Index(kernel_size_z), data); + break; + } + + const int numP = total / (Index(numX) * numY * numZ); const int maxX = numext::mini<int>( 128, numext::mini<int>(maxSharedMem / (sizeof(Scalar) * kernel_size_y * kernel_size_z) - kernel_size_x + 1, numX)); @@ -1039,11 +1093,6 @@ (maxX + kernel_size_x - 1) * (maxY + kernel_size_y - 1) * (maxZ + kernel_size_z - 1) * sizeof(Scalar); gpu_assert(shared_mem <= maxSharedMem); - const array<Index, 3> indices{m_indices[idxX], m_indices[idxY], m_indices[idxZ]}; - const array<Index, 3> kernel_dims{m_kernelImpl.dimensions()[idxX], m_kernelImpl.dimensions()[idxY], - m_kernelImpl.dimensions()[idxZ]}; - internal::IndexMapper<Index, InputDims, 3, Layout> indexMapper(m_inputImpl.dimensions(), kernel_dims, indices); - LAUNCH_GPU_KERNEL((EigenConvolutionKernel3D<TensorEvaluator<InputArgType, GpuDevice>, Index, InputDims>), num_blocks, block_size, shared_mem, m_device, m_inputImpl, indexMapper, m_kernel, numP, numX, maxX, numY, maxY, numZ, maxZ, kernel_size_x, kernel_size_y, kernel_size_z, data);
diff --git a/contrib/Eigen/src/Tensor/TensorDeviceDefault.h b/contrib/Eigen/src/Tensor/TensorDeviceDefault.h index efb4bb4..af5d272 100644 --- a/contrib/Eigen/src/Tensor/TensorDeviceDefault.h +++ b/contrib/Eigen/src/Tensor/TensorDeviceDefault.h
@@ -64,7 +64,7 @@ } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE size_t firstLevelCacheSize() const { -#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(SYCL_DEVICE_ONLY) +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) // Running on the host CPU return l1CacheSize(); #elif defined(EIGEN_HIP_DEVICE_COMPILE) @@ -77,7 +77,7 @@ } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE size_t lastLevelCacheSize() const { -#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(SYCL_DEVICE_ONLY) +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) // Running single threaded on the host CPU return l3CacheSize(); #elif defined(EIGEN_HIP_DEVICE_COMPILE)
diff --git a/contrib/Eigen/src/Tensor/TensorEvaluator.h b/contrib/Eigen/src/Tensor/TensorEvaluator.h index fc3687e..2e7b99c 100644 --- a/contrib/Eigen/src/Tensor/TensorEvaluator.h +++ b/contrib/Eigen/src/Tensor/TensorEvaluator.h
@@ -732,7 +732,12 @@ EIGEN_STRONG_INLINE bool evalSubExprsIfNeeded(EvaluatorPointerType) { m_leftImpl.evalSubExprsIfNeeded(nullptr); - m_rightImpl.evalSubExprsIfNeeded(nullptr); + // The executor does not call cleanup() when evaluation throws, so release the operand evaluated first. + EIGEN_TRY { m_rightImpl.evalSubExprsIfNeeded(nullptr); } + EIGEN_CATCH(...) { + m_leftImpl.cleanup(); + EIGEN_THROW; + } return true; } @@ -882,8 +887,18 @@ EIGEN_STRONG_INLINE bool evalSubExprsIfNeeded(EvaluatorPointerType) { m_arg1Impl.evalSubExprsIfNeeded(nullptr); - m_arg2Impl.evalSubExprsIfNeeded(nullptr); - m_arg3Impl.evalSubExprsIfNeeded(nullptr); + // The executor does not call cleanup() when evaluation throws, so release the operands evaluated first. + EIGEN_TRY { m_arg2Impl.evalSubExprsIfNeeded(nullptr); } + EIGEN_CATCH(...) { + m_arg1Impl.cleanup(); + EIGEN_THROW; + } + EIGEN_TRY { m_arg3Impl.evalSubExprsIfNeeded(nullptr); } + EIGEN_CATCH(...) { + m_arg1Impl.cleanup(); + m_arg2Impl.cleanup(); + EIGEN_THROW; + } return true; } EIGEN_STRONG_INLINE void cleanup() { @@ -1036,8 +1051,18 @@ EIGEN_STRONG_INLINE bool evalSubExprsIfNeeded(EvaluatorPointerType) { m_condImpl.evalSubExprsIfNeeded(nullptr); - m_thenImpl.evalSubExprsIfNeeded(nullptr); - m_elseImpl.evalSubExprsIfNeeded(nullptr); + // The executor does not call cleanup() when evaluation throws, so release the operands evaluated first. + EIGEN_TRY { m_thenImpl.evalSubExprsIfNeeded(nullptr); } + EIGEN_CATCH(...) { + m_condImpl.cleanup(); + EIGEN_THROW; + } + EIGEN_TRY { m_elseImpl.evalSubExprsIfNeeded(nullptr); } + EIGEN_CATCH(...) { + m_condImpl.cleanup(); + m_thenImpl.cleanup(); + EIGEN_THROW; + } return true; }
diff --git a/contrib/Eigen/src/Tensor/TensorExecutor.h b/contrib/Eigen/src/Tensor/TensorExecutor.h index 994abc5..95b6a7e 100644 --- a/contrib/Eigen/src/Tensor/TensorExecutor.h +++ b/contrib/Eigen/src/Tensor/TensorExecutor.h
@@ -53,9 +53,17 @@ TensorEvaluator<Expression, Device> evaluator(expr, device); const bool needs_assign = evaluator.evalSubExprsIfNeeded(nullptr); if (needs_assign) { - const StorageIndex size = static_cast<StorageIndex>(array_prod(evaluator.dimensions())); - for (StorageIndex i = 0; i < size; ++i) { - evaluator.evalScalar(i); + // Scalars that require initialization take this path, and their operations may throw. Release the temporaries + // that evalSubExprsIfNeeded allocated before the exception propagates. + EIGEN_TRY { + const StorageIndex size = static_cast<StorageIndex>(array_prod(evaluator.dimensions())); + for (StorageIndex i = 0; i < size; ++i) { + evaluator.evalScalar(i); + } + } + EIGEN_CATCH(...) { + evaluator.cleanup(); + EIGEN_THROW; } } evaluator.cleanup();
diff --git a/contrib/Eigen/src/Tensor/TensorFixedSize.h b/contrib/Eigen/src/Tensor/TensorFixedSize.h index 03af104..5ce519e 100644 --- a/contrib/Eigen/src/Tensor/TensorFixedSize.h +++ b/contrib/Eigen/src/Tensor/TensorFixedSize.h
@@ -62,7 +62,15 @@ public: constexpr EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index rank() const { return NumIndices; } - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension(std::size_t n) const { return m_storage.dimensions()[n]; } + template <Index N> + static constexpr EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension() { + EIGEN_STATIC_ASSERT(0 <= N && N < NumIndices, INVALID_DIMENSION); + return Dimensions{}[N]; + } + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension(Index n) const { + eigen_assert(0 <= n && n < static_cast<Index>(NumIndices)); + return m_storage.dimensions()[n]; + } constexpr EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE const Dimensions dimensions() const { return m_storage.dimensions(); } constexpr EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index size() const { return m_storage.size(); } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Scalar* data() { return m_storage.data(); }
diff --git a/contrib/Eigen/src/Tensor/TensorIntDiv.h b/contrib/Eigen/src/Tensor/TensorIntDiv.h index 73c7adf..0d1a553 100644 --- a/contrib/Eigen/src/Tensor/TensorIntDiv.h +++ b/contrib/Eigen/src/Tensor/TensorIntDiv.h
@@ -23,7 +23,7 @@ EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE std::enable_if_t<sizeof(T) == 4, int> count_leading_zeros(const T val) { #ifdef EIGEN_GPU_COMPILE_PHASE return __clz(val); -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::clz(val); #elif EIGEN_COMP_MSVC unsigned long index; @@ -39,7 +39,7 @@ EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE std::enable_if_t<sizeof(T) == 8, int> count_leading_zeros(const T val) { #ifdef EIGEN_GPU_COMPILE_PHASE return __clzll(val); -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) return static_cast<int>(cl::sycl::clz(val)); #elif EIGEN_COMP_MSVC && EIGEN_ARCH_x86_64 unsigned long index; @@ -76,7 +76,7 @@ EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE uint32_t muluh(const uint32_t a, const T b) { #if defined(EIGEN_GPU_COMPILE_PHASE) return __umulhi(a, b); -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::mul_hi(a, static_cast<uint32_t>(b)); #else return (static_cast<uint64_t>(a) * b) >> 32; @@ -87,7 +87,7 @@ EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE uint64_t muluh(const uint64_t a, const T b) { #if defined(EIGEN_GPU_COMPILE_PHASE) return __umul64hi(a, b); -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::mul_hi(a, static_cast<uint64_t>(b)); #elif EIGEN_COMP_MSVC && (EIGEN_ARCH_x86_64 || EIGEN_ARCH_ARM64) return __umulh(a, static_cast<uint64_t>(b)); @@ -111,7 +111,7 @@ template <typename T> struct DividerHelper<64, T> { static EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE uint64_t computeMultiplier(const int log_div, const T divider) { -#if EIGEN_HAS_BUILTIN_INT128 && !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(SYCL_DEVICE_ONLY) +#if EIGEN_HAS_BUILTIN_INT128 && !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) return static_cast<uint64_t>((static_cast<__uint128_t>(1) << (64 + log_div)) / static_cast<__uint128_t>(divider) - (static_cast<__uint128_t>(1) << 64) + 1); #else @@ -194,7 +194,7 @@ EIGEN_DEVICE_FUNC EIGEN_ALWAYS_INLINE int divide(const int32_t n) const { #ifdef EIGEN_GPU_COMPILE_PHASE return __umulhi(magic, n) >> shift; -#elif defined(SYCL_DEVICE_ONLY) +#elif defined(EIGEN_SYCL_DEVICE_ONLY) return cl::sycl::mul_hi(magic, static_cast<uint32_t>(n)) >> shift; #else uint64_t v = static_cast<uint64_t>(magic) * static_cast<uint64_t>(n);
diff --git a/contrib/Eigen/src/Tensor/TensorMacros.h b/contrib/Eigen/src/Tensor/TensorMacros.h index 1bdf913..a4e2824 100644 --- a/contrib/Eigen/src/Tensor/TensorMacros.h +++ b/contrib/Eigen/src/Tensor/TensorMacros.h
@@ -29,7 +29,7 @@ #define EIGEN_SFINAE_ENABLE_IF(__condition__) std::enable_if_t<(__condition__), int> = 0 // Define a macro to use a reference on the host but a value on the device -#if defined(SYCL_DEVICE_ONLY) +#if defined(__SYCL_DEVICE_ONLY__) #define EIGEN_DEVICE_REF #else #define EIGEN_DEVICE_REF &
diff --git a/contrib/Eigen/src/Tensor/TensorMap.h b/contrib/Eigen/src/Tensor/TensorMap.h index edaed0d..0c1ba11 100644 --- a/contrib/Eigen/src/Tensor/TensorMap.h +++ b/contrib/Eigen/src/Tensor/TensorMap.h
@@ -97,8 +97,16 @@ EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE TensorMap(PlainObjectType& tensor) : m_data(tensor.data()), m_dimensions(tensor.dimensions()) {} - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index rank() const { return m_dimensions.rank(); } - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension(Index n) const { return m_dimensions[n]; } + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index rank() const { return NumIndices; } + template <Index N> + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension() const { + EIGEN_STATIC_ASSERT(0 <= N && N < NumIndices, INVALID_DIMENSION); + return m_dimensions[N]; + } + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension(Index n) const { + eigen_assert(0 <= n && n < NumIndices); + return m_dimensions[n]; + } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE const Dimensions& dimensions() const { return m_dimensions; } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index size() const { return m_dimensions.TotalSize(); } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE StoragePointerType data() { return m_data; }
diff --git a/contrib/Eigen/src/Tensor/TensorMorphing.h b/contrib/Eigen/src/Tensor/TensorMorphing.h index eb7a364..94d71f0 100644 --- a/contrib/Eigen/src/Tensor/TensorMorphing.h +++ b/contrib/Eigen/src/Tensor/TensorMorphing.h
@@ -1030,7 +1030,7 @@ } static EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index clamp(Index value, Index min, Index max) { -#ifndef SYCL_DEVICE_ONLY +#ifndef EIGEN_SYCL_DEVICE_ONLY return numext::maxi(min, numext::mini(max, value)); #else return cl::sycl::clamp(value, min, max);
diff --git a/contrib/Eigen/src/Tensor/TensorReductionGpu.h b/contrib/Eigen/src/Tensor/TensorReductionGpu.h index cc4bb38..1dc0bf9 100644 --- a/contrib/Eigen/src/Tensor/TensorReductionGpu.h +++ b/contrib/Eigen/src/Tensor/TensorReductionGpu.h
@@ -156,10 +156,12 @@ atomicAdd(output, accum); } +#if !defined(EIGEN_CUDA_ARCH) || EIGEN_CUDA_ARCH >= 600 template <> __device__ inline void atomicReduce(double* output, double accum, SumReducer<double>&) { atomicAdd(output, accum); } +#endif template <typename CoeffType, typename Index> __global__ EIGEN_HIP_LAUNCH_BOUNDS_1024 void ReductionInitKernel(const CoeffType val, Index num_preserved_coeffs,
diff --git a/contrib/Eigen/src/Tensor/TensorRef.h b/contrib/Eigen/src/Tensor/TensorRef.h index 052af5f..9478bf7 100644 --- a/contrib/Eigen/src/Tensor/TensorRef.h +++ b/contrib/Eigen/src/Tensor/TensorRef.h
@@ -184,7 +184,15 @@ ~TensorRefBase() { unrefEvaluator(); } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index rank() const { return m_evaluator->dimensions().size(); } - EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension(Index n) const { return m_evaluator->dimensions()[n]; } + template <Index N> + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension() const { + EIGEN_STATIC_ASSERT(0 <= N && N < NumIndices, INVALID_DIMENSION); + return m_evaluator->dimensions()[N]; + } + EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index dimension(Index n) const { + eigen_assert(0 <= n && n < NumIndices); + return m_evaluator->dimensions()[n]; + } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE const Dimensions& dimensions() const { return m_evaluator->dimensions(); } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE Index size() const { return m_evaluator->dimensions().TotalSize(); } EIGEN_DEVICE_FUNC EIGEN_STRONG_INLINE const Scalar* data() const { return m_evaluator->data(); }
diff --git a/contrib/README.txt b/contrib/README.txt index f19488a..bb23648 100644 --- a/contrib/README.txt +++ b/contrib/README.txt
@@ -29,9 +29,7 @@ - should have some general purpose in the sense that it could potentially become a core Eigen module (or be merged into another one). -In doubt feel free to contact us. For instance, if your addon is very -specific but shows an interesting way of using Eigen, it could make for a -nice demo. +In doubt feel free to contact us. This directory is organized as follow:
diff --git a/contrib/benchmarks/AutoDiff/CMakeLists.txt b/contrib/benchmarks/AutoDiff/CMakeLists.txt index 00230d9..7e289c0 100644 --- a/contrib/benchmarks/AutoDiff/CMakeLists.txt +++ b/contrib/benchmarks/AutoDiff/CMakeLists.txt
@@ -2,3 +2,4 @@ # SPDX-License-Identifier: MPL-2.0 eigen_add_benchmark(bench_autodiff bench_autodiff.cpp) +eigen_add_benchmark(bench_autodiff_sparse_product bench_autodiff_sparse_product.cpp)
diff --git a/contrib/benchmarks/AutoDiff/bench_autodiff_sparse_product.cpp b/contrib/benchmarks/AutoDiff/bench_autodiff_sparse_product.cpp new file mode 100644 index 0000000..8036e65 --- /dev/null +++ b/contrib/benchmarks/AutoDiff/bench_autodiff_sparse_product.cpp
@@ -0,0 +1,58 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/SparseCore> +#include <contrib/Eigen/AutoDiff> + +template <int Order, bool SparseOnLeft> +static void BM_AutoDiffSparseProduct(benchmark::State& state) { + using AD = Eigen::AutoDiffScalar<Eigen::Vector4d>; + using Dense = Eigen::Matrix<AD, Eigen::Dynamic, Eigen::Dynamic>; + const Eigen::Index size = state.range(0), width = state.range(1); + Eigen::SparseMatrix<AD, Order> sparse(size, size); + for (Eigen::Index col = 0; col < size; ++col) + for (Eigen::Index k = 0; k < 8; ++k) { + AD value(double((col + k) % 5 - 2)); + value.derivatives() << double(k + 1), 0, 1, -1; + sparse.insert((col + 17 * k) % size, col) = value; + } + sparse.makeCompressed(); + Dense dense(SparseOnLeft ? size : width, SparseOnLeft ? width : size); + for (Eigen::Index col = 0; col < dense.cols(); ++col) + for (Eigen::Index row = 0; row < dense.rows(); ++row) { + AD value(double((row + 2 * col) % 7 - 3)); + value.derivatives() = Eigen::Vector4d(1, 2, 3, 4) * double((row - col) % 3); + dense(row, col) = value; + } + Dense result(SparseOnLeft ? size : width, SparseOnLeft ? width : size); + const auto evaluate = [&] { + EIGEN_IF_CONSTEXPR (SparseOnLeft) + result.noalias() = sparse * dense; + else + result.noalias() = dense * sparse; + }; + evaluate(); + AD expected(0); + expected.derivatives().setZero(); + for (Eigen::Index k = 0; k < size; ++k) { + EIGEN_IF_CONSTEXPR (SparseOnLeft) + expected += sparse.coeff(0, k) * dense(k, 0); + else + expected += dense(0, k) * sparse.coeff(k, 0); + } + if (result(0, 0).value() != expected.value() || !(result(0, 0).derivatives() - expected.derivatives()).isZero(0)) { + state.SkipWithError("incorrect AutoDiff sparse product"); + return; + } + for (auto _ : state) { + evaluate(); + benchmark::DoNotOptimize(result.data()); + benchmark::ClobberMemory(); + } +} + +BENCHMARK_TEMPLATE(BM_AutoDiffSparseProduct, Eigen::ColMajor, true)->Args({128, 16})->Args({1000, 50}); +BENCHMARK_TEMPLATE(BM_AutoDiffSparseProduct, Eigen::RowMajor, true)->Args({128, 16})->Args({1000, 50}); +BENCHMARK_TEMPLATE(BM_AutoDiffSparseProduct, Eigen::ColMajor, false)->Args({128, 16})->Args({1000, 50}); +BENCHMARK_TEMPLATE(BM_AutoDiffSparseProduct, Eigen::RowMajor, false)->Args({128, 16})->Args({1000, 50});
diff --git a/contrib/benchmarks/CMakeLists.txt b/contrib/benchmarks/CMakeLists.txt index 2bc57a4..c4e3b1c 100644 --- a/contrib/benchmarks/CMakeLists.txt +++ b/contrib/benchmarks/CMakeLists.txt
@@ -36,7 +36,9 @@ add_subdirectory(Splines) add_subdirectory(Polynomials) add_subdirectory(KroneckerProduct) +add_subdirectory(NumericalDiff) add_subdirectory(StructuredMatrices) +add_subdirectory(Riccati) # GPU benchmarks have their own CMake project (needs CUDAToolkit). # They can also be built standalone: cmake -B build -S contrib/benchmarks/GPU
diff --git a/contrib/benchmarks/NumericalDiff/CMakeLists.txt b/contrib/benchmarks/NumericalDiff/CMakeLists.txt new file mode 100644 index 0000000..6ae891e --- /dev/null +++ b/contrib/benchmarks/NumericalDiff/CMakeLists.txt
@@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: The Eigen Authors +# SPDX-License-Identifier: MPL-2.0 + +eigen_add_benchmark(bench_ridders bench_ridders.cpp)
diff --git a/contrib/benchmarks/NumericalDiff/bench_ridders.cpp b/contrib/benchmarks/NumericalDiff/bench_ridders.cpp new file mode 100644 index 0000000..85a51cd --- /dev/null +++ b/contrib/benchmarks/NumericalDiff/bench_ridders.cpp
@@ -0,0 +1,81 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <contrib/Eigen/NumericalDiff> +#include <algorithm> +#include <cmath> + +template <typename Scalar_, int Options> +struct CubicFunctor { + using Scalar = Scalar_; + using InputType = Eigen::Matrix<Scalar, 2, 1>; + using ValueType = Eigen::Matrix<Scalar, Eigen::Dynamic, 1>; + using JacobianType = Eigen::Matrix<Scalar, Eigen::Dynamic, 2, Options>; + + static constexpr int InputsAtCompileTime = 2; + static constexpr int ValuesAtCompileTime = Eigen::Dynamic; + + Eigen::Index count; + + Eigen::Index values() const { return count; } + + int operator()(const InputType& x, ValueType& y) const { + for (Eigen::Index i = 0; i < count; ++i) y[i] = Scalar(i + 1) * x[0] * x[0] * x[0] + x[1] * x[1] * x[1]; + return 0; + } +}; + +template <typename Derivative> +static void measureJacobian(benchmark::State& state, Derivative& derivative, typename Derivative::Scalar tolerance) { + using Functor = typename Derivative::Functor; + using Scalar = typename Functor::Scalar; + const Eigen::Index count = derivative.values(); + typename Functor::InputType x; + x << Scalar(0.5), Scalar(-1); + typename Functor::JacobianType jac(count, 2); + const int evaluations = derivative.df(x, jac); + long double relativeError = 0; + for (Eigen::Index i = 0; i < count; ++i) { + const Scalar expected = Scalar(i + 1) * Scalar(0.75); + relativeError = (std::max)(relativeError, std::abs(static_cast<long double>(jac(i, 0)) / (0.75L * (i + 1)) - 1)); + relativeError = (std::max)(relativeError, std::abs(static_cast<long double>(jac(i, 1)) / 3 - 1)); + const Scalar bound = tolerance * Eigen::numext::maxi(expected, Scalar(3)); + if (!(Eigen::numext::abs(jac(i, 0) - expected) <= bound && Eigen::numext::abs(jac(i, 1) - Scalar(3)) <= bound)) { + state.SkipWithError("Incorrect cubic Jacobian"); + return; + } + } + state.counters["evaluations"] = evaluations; + state.counters["max_relative_error"] = static_cast<double>(relativeError); + + for (auto _ : state) { + benchmark::DoNotOptimize(derivative.df(x, jac)); + benchmark::DoNotOptimize(jac.data()); + benchmark::ClobberMemory(); + } +} + +template <typename Scalar, int Options> +static void BM_Ridders(benchmark::State& state) { + using Functor = CubicFunctor<Scalar, Options>; + Eigen::Ridders<Functor> derivative(Functor{state.range(0)}); + typename Eigen::Ridders<Functor>::Parameters parameters; + parameters.maxSteps = static_cast<int>(state.range(1)); + derivative.setParameters(parameters); + measureJacobian(state, derivative, Scalar(512) * Eigen::NumTraits<Scalar>::epsilon()); +} + +template <typename Scalar, int Options> +static void BM_Central(benchmark::State& state) { + using Functor = CubicFunctor<Scalar, Options>; + Eigen::NumericalDiff<Functor, Eigen::Central> derivative(Functor{state.range(0)}); + measureJacobian(state, derivative, Scalar(16) * std::sqrt(Eigen::NumTraits<Scalar>::epsilon())); +} + +BENCHMARK_TEMPLATE(BM_Ridders, float, Eigen::ColMajor)->ArgsProduct({{1, 16, 128, 1024}, {2, 10}}); +BENCHMARK_TEMPLATE(BM_Ridders, double, Eigen::ColMajor)->ArgsProduct({{1, 16, 128, 1024}, {2, 10}}); +BENCHMARK_TEMPLATE(BM_Ridders, double, Eigen::RowMajor)->ArgsProduct({{1, 16, 128, 1024}, {2, 10}}); +BENCHMARK_TEMPLATE(BM_Central, float, Eigen::ColMajor)->Arg(1)->Arg(16)->Arg(128)->Arg(1024); +BENCHMARK_TEMPLATE(BM_Central, double, Eigen::ColMajor)->Arg(1)->Arg(16)->Arg(128)->Arg(1024); +BENCHMARK_TEMPLATE(BM_Central, double, Eigen::RowMajor)->Arg(1)->Arg(16)->Arg(128)->Arg(1024);
diff --git a/contrib/benchmarks/Riccati/CMakeLists.txt b/contrib/benchmarks/Riccati/CMakeLists.txt new file mode 100644 index 0000000..2380727 --- /dev/null +++ b/contrib/benchmarks/Riccati/CMakeLists.txt
@@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: The Eigen Authors +# SPDX-License-Identifier: MPL-2.0 + +eigen_add_benchmark(bench_dare bench_dare.cpp)
diff --git a/contrib/benchmarks/Riccati/bench_dare.cpp b/contrib/benchmarks/Riccati/bench_dare.cpp new file mode 100644 index 0000000..2176e29 --- /dev/null +++ b/contrib/benchmarks/Riccati/bench_dare.cpp
@@ -0,0 +1,101 @@ +// Benchmarks for DiscreteAlgebraicRiccatiSolver::compute on well-conditioned random problems. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <benchmark/benchmark.h> +#include <Eigen/Core> +#include <contrib/Eigen/Riccati> + +#include <complex> +#include <cstdlib> + +using namespace Eigen; + +namespace { + +template <typename Scalar> +struct DareProblem { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic>; + MatrixType A, B, Q, R, E, S; +}; + +// A = U T U^* with U unitary and T upper triangular with eigenvalue moduli in [0.1, 0.8], except for n/2 well +// controllable ones in [1.25, 2]; Q, R positive definite; E = I + O(1/2) and ||S||_F <= 1/2 for generalized problems. +template <typename Scalar> +DareProblem<Scalar> makeProblem(Index m, Index n, bool generalized) { + using RealScalar = typename NumTraits<Scalar>::Real; + using MatrixType = typename DareProblem<Scalar>::MatrixType; + std::srand(1); + MatrixType T = MatrixType::Random(m, m).template triangularView<StrictlyUpper>(); + T *= RealScalar(0.5) / numext::sqrt(RealScalar(m)); + const Index unstable = n / 2; + for (Index i = 0; i < m; ++i) { + const RealScalar modulus = + i < unstable ? internal::random<RealScalar>(1.25, 2) : internal::random<RealScalar>(0.1, 0.8); + T(i, i) = (i % 2 ? Scalar(modulus) : Scalar(-modulus)); + } + const MatrixType U = HouseholderQR<MatrixType>(MatrixType::Random(m, m)).householderQ(); + MatrixType Bu = MatrixType::Random(m, n); + for (Index i = 0; i < unstable; ++i) Bu(i, i) += Scalar(2); + const MatrixType C = MatrixType::Random(m, m), D = MatrixType::Random(n, n); + DareProblem<Scalar> p; + p.A = U * T * U.adjoint(); + p.B = U * Bu; + p.Q = C.adjoint() * C + MatrixType::Identity(m, m); + p.R = D.adjoint() * D + MatrixType::Identity(n, n); + p.E = MatrixType::Identity(m, m); + p.S = MatrixType::Zero(m, n); + if (generalized) { + p.E += MatrixType::Random(m, m) * (RealScalar(0.5) / numext::sqrt(RealScalar(m))); + p.S = MatrixType::Random(m, n) * (RealScalar(0.5) / numext::sqrt(RealScalar(m * n))); + p.A = p.E * p.A; + p.B = p.E * p.B; + } + return p; +} + +// ||res||_F / (m eps (||A^* X A||_F + ||E^* X E||_F + ||Q||_F)), the backward error measure of the unit test. +template <typename Scalar> +double normalizedResidual(const DareProblem<Scalar>& p, const typename DareProblem<Scalar>::MatrixType& X) { + using RealScalar = typename NumTraits<Scalar>::Real; + using MatrixType = typename DareProblem<Scalar>::MatrixType; + const MatrixType AXB = p.A.adjoint() * X * p.B + p.S; + const MatrixType res = p.A.adjoint() * X * p.A - p.E.adjoint() * X * p.E - + AXB * (p.R + p.B.adjoint() * X * p.B).fullPivLu().solve(AXB.adjoint()) + p.Q; + const RealScalar scale = (p.A.adjoint() * X * p.A).norm() + (p.E.adjoint() * X * p.E).norm() + p.Q.norm(); + return double(res.norm() / (RealScalar(p.A.rows()) * NumTraits<RealScalar>::epsilon() * scale)); +} + +// state.range(0) = m, n = max(1, m / 4); Generalized selects the (E, S) form. +template <typename Scalar, bool Generalized> +static void BM_DiscreteAlgebraicRiccatiSolver(benchmark::State& state) { + const Index m = state.range(0); + const Index n = numext::maxi(Index(1), m / 4); + const DareProblem<Scalar> p = makeProblem<Scalar>(m, n, Generalized); + DiscreteAlgebraicRiccatiSolver<typename DareProblem<Scalar>::MatrixType> dare; + const auto solve = [&] { + if (Generalized) + dare.compute(p.A, p.B, p.Q, p.R, p.E, p.S); + else + dare.compute(p.A, p.B, p.Q, p.R); + }; + solve(); + // The unit test bounds this measure by 1000; well-conditioned problems stay near 1. + if (dare.info() != Success || !(normalizedResidual(p, dare.solution()) <= 1000.0)) { + state.SkipWithError("DARE solution failed the backward error check"); + return; + } + for (auto _ : state) { + solve(); + benchmark::DoNotOptimize(dare.solution().data()); + } + state.counters["m"] = double(m); + state.counters["n"] = double(n); +} + +} // namespace + +BENCHMARK_TEMPLATE(BM_DiscreteAlgebraicRiccatiSolver, double, false)->RangeMultiplier(2)->Range(4, 256); +BENCHMARK_TEMPLATE(BM_DiscreteAlgebraicRiccatiSolver, double, true)->RangeMultiplier(4)->Range(4, 64); +BENCHMARK_TEMPLATE(BM_DiscreteAlgebraicRiccatiSolver, float, false)->RangeMultiplier(4)->Range(4, 64); +BENCHMARK_TEMPLATE(BM_DiscreteAlgebraicRiccatiSolver, std::complex<double>, false)->RangeMultiplier(4)->Range(4, 64);
diff --git a/contrib/benchmarks/Tensor/bench_contraction.cpp b/contrib/benchmarks/Tensor/bench_contraction.cpp index 88e9e97..13f06dc 100644 --- a/contrib/benchmarks/Tensor/bench_contraction.cpp +++ b/contrib/benchmarks/Tensor/bench_contraction.cpp
@@ -6,6 +6,7 @@ #define EIGEN_USE_THREADS #include <benchmark/benchmark.h> +#include <contrib/Eigen/AutoDiff> #include <contrib/Eigen/Tensor> #include <Eigen/ThreadPool> @@ -122,6 +123,56 @@ benchmark::Counter(2.0 * M * N * K, benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); } +// --- Scalar type and layout variants, on DefaultDevice (threads == 0) or ThreadPoolDevice --- +// AutoDiffScalar requires initialization, so the contraction constructs its packed blocks and temporaries. +template <typename T> +static T make_scalar(float value, Index /*i*/) { + return T(value); +} + +template <> +AutoDiffScalar<VectorXf> make_scalar<AutoDiffScalar<VectorXf>>(float value, Index i) { + const int num_derivatives = 4; + return AutoDiffScalar<VectorXf>(value, num_derivatives, static_cast<int>(i % num_derivatives)); +} + +template <typename T, int Layout> +static void BM_ContractionTyped(benchmark::State& state) { + const int M = state.range(0); + const int N = state.range(1); + const int K = state.range(2); + const int threads = state.range(3); + + Tensor<float, 2, Layout> A_float(M, K); + Tensor<float, 2, Layout> B_float(K, N); + A_float.setRandom(); + B_float.setRandom(); + Tensor<T, 2, Layout> A(M, K); + Tensor<T, 2, Layout> B(K, N); + Tensor<T, 2, Layout> C(M, N); + for (Index i = 0; i < A.size(); ++i) A.data()[i] = make_scalar<T>(A_float.data()[i], i); + for (Index i = 0; i < B.size(); ++i) B.data()[i] = make_scalar<T>(B_float.data()[i], i); + + ThreadPool tp(threads > 0 ? threads : 1); + ThreadPoolDevice dev(&tp, threads > 0 ? threads : 1); + + using ContractDims = typename Tensor<T, 2, Layout>::DimensionPair; + Eigen::array<ContractDims, 1> contract_dims = {ContractDims(1, 0)}; + + for (auto _ : state) { + if (threads > 0) { + C.device(dev) = A.contract(B, contract_dims); + } else { + C = A.contract(B, contract_dims); + } + benchmark::DoNotOptimize(C.data()); + benchmark::ClobberMemory(); + } + state.counters["GFLOPS"] = + benchmark::Counter(2.0 * M * N * K, benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::kIs1000); + state.counters["threads"] = threads; +} + // clang-format off #define CONTRACTION_SIZES \ ->Args({32, 32, 32})->Args({64, 64, 64})->Args({128, 128, 128}) \ @@ -138,6 +189,10 @@ ->Args({1024, 1024, 1024, 1})->Args({1024, 1024, 1024, 2})->Args({1024, 1024, 1024, 4}) \ ->Args({1024, 1024, 1024, 8})->Args({1024, 1024, 1024, 16}) +#define TYPED_SIZES \ + ->Args({64, 64, 64, 0})->Args({256, 256, 256, 0})->Args({256, 256, 256, 4}) \ + ->Args({2, 10, 30000, 4}) + #define BATCH_SIZES \ ->Args({1, 64, 64, 64})->Args({1, 256, 256, 256}) \ ->Args({8, 64, 64, 64})->Args({8, 256, 256, 256}) \ @@ -148,3 +203,7 @@ BENCHMARK(BM_Contraction_RowMajor) CONTRACTION_SIZES; BENCHMARK(BM_Contraction_ThreadPool) CONTRACTION_THREADPOOL_SIZES->UseRealTime(); BENCHMARK(BM_BatchContraction) BATCH_SIZES; +BENCHMARK_TEMPLATE(BM_ContractionTyped, float, ColMajor) TYPED_SIZES->UseRealTime(); +BENCHMARK_TEMPLATE(BM_ContractionTyped, float, RowMajor) TYPED_SIZES->UseRealTime(); +BENCHMARK_TEMPLATE(BM_ContractionTyped, AutoDiffScalar<VectorXf>, ColMajor) TYPED_SIZES->UseRealTime(); +BENCHMARK_TEMPLATE(BM_ContractionTyped, AutoDiffScalar<VectorXf>, RowMajor) TYPED_SIZES->UseRealTime();
diff --git a/contrib/benchmarks/Tensor/bench_morphing.cpp b/contrib/benchmarks/Tensor/bench_morphing.cpp index c33325f..5ce133e 100644 --- a/contrib/benchmarks/Tensor/bench_morphing.cpp +++ b/contrib/benchmarks/Tensor/bench_morphing.cpp
@@ -68,6 +68,68 @@ state.SetBytesProcessed(state.iterations() * D1 * D2 * sizeof(Scalar)); } +// Middle-dimension chip feeding a full reduction (BlockAccess = false, so +// InnerMostDimReducer evaluates TensorChippingOp::packet() across boundaries +// whenever Cg < PacketSize or Cg is not a multiple of PacketSize). +static void BM_ChipReduceRead(benchmark::State& state) { + const Index Cg = state.range(0); + const Index G = 16; + const Index M = 8192; + + Tensor<Scalar, 3> A(Cg, G, M); + A.setRandom(); + Tensor<Scalar, 0> B; + + for (auto _ : state) { + B = A.chip(1, 1).sum(); + benchmark::DoNotOptimize(B.data()); + benchmark::ClobberMemory(); + } + + Tensor<Scalar, 2> chip_ref = A.chip(1, 1); + Tensor<Scalar, 0> expected = chip_ref.sum(); + if (std::abs(B(0) - expected(0)) > 1e-2f * (1.0f + std::abs(expected(0)))) { + state.SkipWithError("validation failed"); + return; + } + state.SetBytesProcessed(state.iterations() * Cg * M * sizeof(Scalar)); +} + +// Middle-dimension lvalue chip assigned from a contraction (BlockAccess = +// false, as in grouped convolution output.chip(g, 1) = convolved, exercising +// TensorEvaluator<TensorChippingOp>::writePacket() across stride boundaries). +static void BM_ChipContractWrite(benchmark::State& state) { + const Index Cg = state.range(0); + const Index G = 16; + const Index K = 2; + const Index M = 4096; + + Tensor<Scalar, 2> lhs(Cg, K); + Tensor<Scalar, 2> rhs(K, M); + lhs.setRandom(); + rhs.setRandom(); + Tensor<Scalar, 3> dst(Cg, G, M); + dst.setZero(); + const Eigen::array<IndexPair<Index>, 1> contract_dims = {IndexPair<Index>(1, 0)}; + + for (auto _ : state) { + dst.chip(1, 1) = lhs.contract(rhs, contract_dims); + benchmark::DoNotOptimize(dst.data()); + benchmark::ClobberMemory(); + } + + Tensor<Scalar, 2> expected = lhs.contract(rhs, contract_dims); + for (Index m = 0; m < M; ++m) { + for (Index c = 0; c < Cg; ++c) { + if (std::abs(dst(c, 1, m) - expected(c, m)) > 1e-4f) { + state.SkipWithError("validation failed"); + return; + } + } + } + state.SetBytesProcessed(state.iterations() * Cg * M * sizeof(Scalar)); +} + // --- Pad --- static void BM_Pad(benchmark::State& state) { const int M = state.range(0); @@ -297,6 +359,9 @@ #define CHIP_SIZES \ ->Args({32, 256, 256})->Args({64, 128, 128})->Args({8, 512, 512}) +#define CHIP_BOUNDARY_SIZES \ + ->Arg(2)->Arg(3)->Arg(4)->Arg(6)->Arg(7)->Arg(8)->Arg(12)->Arg(16) + #define PAD_SIZES \ ->Args({256, 256, 1})->Args({256, 256, 4})->Args({256, 256, 16}) \ ->Args({1024, 1024, 1})->Args({1024, 1024, 4})->Args({1024, 1024, 16}) @@ -316,6 +381,8 @@ BENCHMARK(BM_Reshape) MORPH_SIZES; BENCHMARK(BM_Slice) MORPH_SIZES; BENCHMARK(BM_Chip) CHIP_SIZES; +BENCHMARK(BM_ChipReduceRead) CHIP_BOUNDARY_SIZES; +BENCHMARK(BM_ChipContractWrite) CHIP_BOUNDARY_SIZES; BENCHMARK(BM_Pad) PAD_SIZES; BENCHMARK(BM_StridedSliceRead) STRIDED_SLICE_SIZES; BENCHMARK(BM_StridedSliceExp) STRIDED_SLICE_SIZES;
diff --git a/contrib/doc/examples/DiscreteAlgebraicRiccatiSolver.cpp b/contrib/doc/examples/DiscreteAlgebraicRiccatiSolver.cpp new file mode 100644 index 0000000..bdcec88 --- /dev/null +++ b/contrib/doc/examples/DiscreteAlgebraicRiccatiSolver.cpp
@@ -0,0 +1,32 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <contrib/Eigen/Riccati> +#include <iostream> + +using namespace Eigen; + +int main() { + // Discrete-time LQR for a double integrator sampled with period h: + // minimize sum_k x_k^T Q x_k + u_k^T R u_k subject to x_{k+1} = A x_k + B u_k. + const double h = 0.1; + Matrix2d A; + A << 1, h, 0, 1; + Vector2d B(h * h / 2, h); + Matrix2d Q = Matrix2d::Identity(); + Matrix<double, 1, 1> R; + R << 0.1; + + DiscreteAlgebraicRiccatiSolver<Matrix2d> dare(A, B, Q, R); + if (dare.info() != Success) { + std::cout << "The Riccati equation could not be solved.\n"; + return 1; + } + const Matrix2d& X = dare.solution(); + std::cout << "The stabilizing solution X is:\n" << X << "\n\n"; + + // The optimal state feedback u_k = -K x_k. + const RowVector2d K = (R + B.transpose() * X * B).ldlt().solve(B.transpose() * X * A); + std::cout << "The optimal gain K is:\n" << K << "\n\n"; + std::cout << "The closed-loop eigenvalues of A - B K are:\n" << (A - B * K).eigenvalues() << "\n"; +}
diff --git a/contrib/test/CMakeLists.txt b/contrib/test/CMakeLists.txt index 3186340..c7ba51a 100644 --- a/contrib/test/CMakeLists.txt +++ b/contrib/test/CMakeLists.txt
@@ -39,6 +39,7 @@ ei_add_test(NumericalDiff) ei_add_test(stencil) +ei_add_test(ridders) ei_add_test(structured_dpr1) ei_add_test(autodiff_scalar) ei_add_test(autodiff) @@ -133,6 +134,7 @@ ei_add_test(structured_vandermonde_int_index) ei_add_test(polynomialsolver) ei_add_test(polynomialutils) +ei_add_test(riccati) ei_add_test(splines) ei_add_test(structured_dplr) ei_add_test(levenberg_marquardt) @@ -266,6 +268,7 @@ ei_add_test(tensor_device) ei_add_test(tensor_gpu_device_attributes) ei_add_test(tensor_gpu) + ei_add_test(tensor_convolution_large_gpu) ei_add_test(tensor_gpu_runtime_check) ei_add_test(tensor_gpu_runtime_check_ndebug) ei_add_test(tensor_contract_gpu)
diff --git a/contrib/test/GPU/CMakeLists.txt b/contrib/test/GPU/CMakeLists.txt index e7e66c7..5344dbe 100644 --- a/contrib/test/GPU/CMakeLists.txt +++ b/contrib/test/GPU/CMakeLists.txt
@@ -18,12 +18,13 @@ set(EIGEN_GPU_TEST_CXX_FLAGS "-Werror=exceptions") endif() -# ei_add_gpu_test(<name> [EXTRA_LIBS <lib>...] [EXTRA_INCLUDES <dir>...] [EXTRA_DEFINES <def>...]) +# ei_add_gpu_test(<name> [EXTRA_LIBS <lib>...] [EXTRA_INCLUDES <dir>...] [EXTRA_DEFINES <def>...] +# [EXTRA_FLAGS <flag>...]) # # Thin wrapper around ei_add_test for GPU module tests. Delegates to # ei_add_test so CALL_SUBTEST_N splitting and EIGEN_TEST_MAX_SIZE handling # work the same way as the rest of the test suite, then layers on the -# GPU-specific properties (EXTRA_{LIBS,INCLUDES,DEFINES}, the +# GPU-specific properties (EXTRA_{LIBS,INCLUDES,DEFINES,FLAGS}, the # "Official;gpu" labels, SKIP_RETURN_CODE=77, buildtests_gpu dependency). # # GPU module tests are plain .cpp (host-compiled) by design: they @@ -33,7 +34,7 @@ # triggers on EIGEN_ADD_TEST_FILENAME_EXTENSION=cu), so we set the # SKIP_RETURN_CODE and labels ourselves. function(ei_add_gpu_test test_name) - cmake_parse_arguments(ARG "" "" "EXTRA_LIBS;EXTRA_INCLUDES;EXTRA_DEFINES" ${ARGN}) + cmake_parse_arguments(ARG "" "" "EXTRA_LIBS;EXTRA_INCLUDES;EXTRA_DEFINES;EXTRA_FLAGS" ${ARGN}) # Delegate to ei_add_test. The third arg is a semicolon-separated library # list; CUDA::cudart_static is always needed for any host test that @@ -66,6 +67,9 @@ if(ARG_EXTRA_DEFINES) target_compile_definitions(${t} PRIVATE ${ARG_EXTRA_DEFINES}) endif() + if(ARG_EXTRA_FLAGS) + target_compile_options(${t} PRIVATE ${ARG_EXTRA_FLAGS}) + endif() add_dependencies(buildtests_gpu ${t}) set_property(TEST ${t} APPEND PROPERTY LABELS "Official;gpu") set_property(TEST ${t} PROPERTY SKIP_RETURN_CODE 77) @@ -136,11 +140,18 @@ HINTS ${CUDSS_DIR}/lib ${CUDSS_DIR}/lib64 ${CUDA_TOOLKIT_ROOT_DIR}/lib64 /usr/lib/x86_64-linux-gnu) if(CUDSS_INCLUDE_DIR AND CUDSS_LIBRARY) message(STATUS "cuDSS found: ${CUDSS_LIBRARY}") + # cuDSS renames API types between releases (1.0 deprecated 0.8's + # cudssDataType_t), so a deprecated name in the module fails these builds. + set(_cudss_flags "") + if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" OR CMAKE_CXX_COMPILER_ID MATCHES "Clang") + set(_cudss_flags "-Werror=deprecated-declarations") + endif() foreach(_cudss_test IN ITEMS cudss_llt cudss_ldlt cudss_lu) ei_add_gpu_test(${_cudss_test} EXTRA_LIBS CUDA::cusolver CUDA::cublas ${CUDSS_LIBRARY} EXTRA_INCLUDES ${CUDSS_INCLUDE_DIR} - EXTRA_DEFINES EIGEN_CUDSS=1) + EXTRA_DEFINES EIGEN_CUDSS=1 + EXTRA_FLAGS ${_cudss_flags}) endforeach() else() message(WARNING "EIGEN_TEST_CUDSS=ON but cuDSS not found. Set CUDSS_DIR.")
diff --git a/contrib/test/autodiff.cpp b/contrib/test/autodiff.cpp index 3e3c719..819cddd 100644 --- a/contrib/test/autodiff.cpp +++ b/contrib/test/autodiff.cpp
@@ -9,11 +9,14 @@ // SPDX-License-Identifier: MPL-2.0 #include "main.h" +#include <Eigen/Geometry> #include <Eigen/Jacobi> #include <Eigen/QR> #include <Eigen/SVD> +#include <Eigen/SparseCore> #include <contrib/Eigen/AutoDiff> #include <contrib/Eigen/Splines> +#include "NonCommutativeScalar.h" template <typename Scalar> EIGEN_DONT_INLINE Scalar foo(const Scalar& x, const Scalar& y) { @@ -188,15 +191,25 @@ VERIFY_IS_APPROX(j, jref); } -// TODO also check actual derivatives! -template <int> +template <typename Scalar> void test_autodiff_scalar() { - Vector2f p = Vector2f::Random(); - typedef AutoDiffScalar<Vector2f> AD; - AD ax(p.x(), Vector2f::UnitX()); - AD ay(p.y(), Vector2f::UnitY()); - AD res = foo<AD>(ax, ay); - VERIFY_IS_APPROX(res.value(), foo(p.x(), p.y())); + using Vector2 = Matrix<Scalar, 2, 1>; + using AD = AutoDiffScalar<Vector2>; + const Scalar x = internal::random<Scalar>(Scalar(-1), Scalar(1)); + // sqrt(y*y) is not differentiable at zero. + const Scalar magnitude = internal::random<Scalar>(Scalar(0.25), Scalar(1)); + for (Scalar sign : {Scalar(-1), Scalar(1)}) { + const Scalar y = sign * magnitude; + const AD ax(x, Vector2::UnitX()), ay(y, Vector2::UnitY()); + const AD res = foo<AD>(ax, ay); + VERIFY_IS_APPROX(res.value(), foo(x, y)); + + // dfoo/dx = 4 + 2*x - 4*cos(x) + x*exp(-x*x/2), dfoo/dy = 2*sign(y) - 2*sin(y). + const Vector2 expected(Scalar(4) + Scalar(2) * x - Scalar(4) * std::cos(x) + x * std::exp(-x * x / Scalar(2)), + Scalar(2) * sign - Scalar(2) * std::sin(y)); + const Scalar tolerance = Scalar(32) * NumTraits<Scalar>::epsilon() * (Scalar(1) + expected.norm()); + VERIFY((res.derivatives() - expected).norm() <= tolerance); + } } template <int> @@ -497,9 +510,99 @@ } } +template <typename Derivatives> +void test_autodiff_hypot() { + using AD = AutoDiffScalar<Derivatives>; + for (int swapped = 0; swapped < 2; ++swapped) { + AD x(swapped ? 4.0 : 3.0), y(swapped ? 3.0 : 4.0); + x.derivatives() = Vector2d::UnitX(); + y.derivatives() = Vector2d::UnitY(); + const AD result = numext::hypot(x, y); + const double tolerance = 8 * NumTraits<double>::epsilon(); + VERIFY(numext::abs(result.value() - 5.0) <= tolerance * 5.0); + VERIFY(numext::abs(result.derivatives()(0) - x.value() / 5.0) <= tolerance); + VERIFY(numext::abs(result.derivatives()(1) - y.value() / 5.0) <= tolerance); + } +} + +template <typename Derivatives> +void test_autodiff_geometry_atan2() { + using AD = AutoDiffScalar<Derivatives>; + using Vec3 = Matrix<AD, 3, 1>; + AD angle(0.6); + angle.derivatives() = Vector2d(1, 2); + const Quaternion<AD> rotation(AngleAxis<AD>(angle, Vec3::UnitZ())); + const AD distance = rotation.angularDistance(Quaternion<AD>::Identity()); + const Vec3 axis = rotation.toScaledAxis(); + AngleAxis<AD> aa_from_q; + aa_from_q = rotation; + const AD aa_angle = aa_from_q.angle(); + const Matrix<AD, 2, 2> matrix = Rotation2D<AD>(angle).toRotationMatrix(); + const AD recovered = Rotation2D<AD>(matrix).angle(); + const double tolerance = 64 * NumTraits<double>::epsilon(); + for (const AD* result : {&distance, &axis(2), &aa_angle, &recovered}) { + VERIFY(numext::abs(result->value() - angle.value()) <= tolerance); + VERIFY((result->derivatives() - angle.derivatives()).cwiseAbs().maxCoeff() <= tolerance); + } + for (Index i = 0; i < 2; ++i) { + VERIFY_IS_EQUAL(axis(i).value(), 0.0); + VERIFY(axis(i).derivatives().isZero()); + } +} + +template <typename Derivatives, int Order> +void test_autodiff_sparse_product_commutativity() { + using AD = AutoDiffScalar<Derivatives>; + using ADExpression = decltype(std::declval<AD>() * std::declval<AD>()); + using NoncommutativeAD = AutoDiffScalar<Matrix<noncommutative_scalar::Quaternion, 4, 1>>; + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<AD>>::value)); + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<AD, ADExpression>>::value)); + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<AD, double>>::value)); + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<double, AD>>::value)); + STATIC_CHECK((!internal::functor_is_commutative<internal::scalar_product_op<NoncommutativeAD>>::value)); + using Dense = Matrix<AD, Dynamic, Dynamic>; + SparseMatrix<AD, Order> sparse(3, 3); + Dense matrix(3, 3); + for (Index j = 0; j < 3; ++j) + for (Index i = 0; i < 3; ++i) { + AD value(double(i + 2 * j + 1)); + value.derivatives() = Derivatives::Constant(4, double(i - j + 1)); + matrix(i, j) = value; + if (i == j || (i + j) % 2) sparse.insert(i, j) = value; + } + AD alpha(2); + alpha.derivatives() = Derivatives::Constant(4, 1); + for (bool compressed : {false, true}) { + if (compressed) sparse.makeCompressed(); + Dense left = Dense::Zero(3, 3), right = Dense::Zero(3, 3); + left.noalias() += alpha * (sparse * matrix); + right.noalias() += alpha * (matrix * sparse); + for (Index j = 0; j < 3; ++j) + for (Index i = 0; i < 3; ++i) { + AD expectedLeft(0), expectedRight(0); + expectedLeft.derivatives() = Derivatives::Zero(4); + expectedRight.derivatives() = Derivatives::Zero(4); + for (Index k = 0; k < 3; ++k) { + expectedLeft += sparse.coeff(i, k) * matrix(k, j); + expectedRight += matrix(i, k) * sparse.coeff(k, j); + } + const AD scaledLeft = alpha * expectedLeft, scaledRight = alpha * expectedRight; + VERIFY_IS_EQUAL(left(i, j).value(), scaledLeft.value()); + VERIFY_IS_EQUAL(right(i, j).value(), scaledRight.value()); + VERIFY((left(i, j).derivatives() - scaledLeft.derivatives()).isZero(0)); + VERIFY((right(i, j).derivatives() - scaledRight.derivatives()).isZero(0)); + } + } +} + EIGEN_DECLARE_TEST(autodiff) { + CALL_SUBTEST_8((test_autodiff_sparse_product_commutativity<Vector4d, ColMajor>())); + CALL_SUBTEST_8((test_autodiff_sparse_product_commutativity<Vector4d, RowMajor>())); + CALL_SUBTEST_8((test_autodiff_sparse_product_commutativity<VectorXd, ColMajor>())); + CALL_SUBTEST_8((test_autodiff_sparse_product_commutativity<VectorXd, RowMajor>())); for (int i = 0; i < g_repeat; i++) { - CALL_SUBTEST_1(test_autodiff_scalar<1>()); + CALL_SUBTEST_1(test_autodiff_scalar<float>()); + CALL_SUBTEST_1(test_autodiff_scalar<double>()); CALL_SUBTEST_2(test_autodiff_vector<1>()); CALL_SUBTEST_2(test_autodiff_vector_derivative<1>()); CALL_SUBTEST_3(test_autodiff_jacobian<1>()); @@ -515,6 +618,10 @@ CALL_SUBTEST_5(test_autodiff_selfadjoint_l1norm()); CALL_SUBTEST_5(test_autodiff_padded_block_derivatives()); CALL_SUBTEST_5(test_autodiff_atan2_constant()); + CALL_SUBTEST_5(test_autodiff_hypot<Vector2d>()); + CALL_SUBTEST_5(test_autodiff_hypot<VectorXd>()); + CALL_SUBTEST_5(test_autodiff_geometry_atan2<Vector2d>()); + CALL_SUBTEST_5(test_autodiff_geometry_atan2<VectorXd>()); CALL_SUBTEST_6(test_autodiff_makegivens<Vector2d>()); CALL_SUBTEST_6(test_autodiff_makegivens<VectorXd>()); CALL_SUBTEST_7(test_autodiff_householder_qr<Vector4d>());
diff --git a/contrib/test/autodiff_scalar.cpp b/contrib/test/autodiff_scalar.cpp index b9fe84f..6004cd4 100644 --- a/contrib/test/autodiff_scalar.cpp +++ b/contrib/test/autodiff_scalar.cpp
@@ -9,7 +9,16 @@ // SPDX-License-Identifier: MPL-2.0 #include "main.h" +#include "fp_control.h" +#ifndef small +#define small char // Windows rpcndr.h defines this before users include AutoDiff. +#define EIGEN_TEST_DEFINED_SMALL +#endif #include <contrib/Eigen/AutoDiff> +#ifdef EIGEN_TEST_DEFINED_SMALL +#undef small +#undef EIGEN_TEST_DEFINED_SMALL +#endif /* * In this file scalar derivations are tested for correctness. @@ -107,6 +116,308 @@ VERIFY_IS_EQUAL((max)(b, a).derivatives(), b.derivatives()); } +template <typename Derivatives> +void check_atan() { + using Scalar = typename Derivatives::Scalar; + using AD = AutoDiffScalar<Derivatives>; + using std::atan; + const Scalar tolerance = Scalar(8) * NumTraits<Scalar>::epsilon(); + Derivatives seed(2); + seed << Scalar(2), Scalar(-3); + const Scalar values[] = {Scalar(-3), + Scalar(-1), + Scalar(-0.5), + Scalar(-0.0), + Scalar(0), + Scalar(0.5), + Scalar(1), + Scalar(3), + std::nextafter(Scalar(1), Scalar(0)), + std::nextafter(Scalar(1), Scalar(2)), + internal::random<Scalar>(Scalar(-8), Scalar(8))}; + for (Scalar value : values) { + AD x(value, seed); + const AD result = atan(x); + const AD dispatched = numext::atan(x); + VERIFY_IS_EQUAL(result.value(), atan(value)); + VERIFY_IS_EQUAL(bool((numext::signbit)(result.value())), bool((numext::signbit)(atan(value)))); + for (Index i = 0; i < seed.size(); ++i) { + const long double v = static_cast<long double>(value); + const Scalar expected = static_cast<Scalar>(static_cast<long double>(seed(i)) / (1 + v * v)); + VERIFY(numext::abs(result.derivatives()(i) - expected) <= tolerance * numext::abs(expected)); + } + VERIFY_IS_EQUAL(dispatched.value(), result.value()); + VERIFY_IS_EQUAL(dispatched.derivatives(), result.derivatives()); + const AutoDiffScalar<Derivatives&> referenced(value, seed); + const AD from_reference = atan(referenced); + VERIFY_IS_EQUAL(from_reference.derivatives(), result.derivatives()); + const AD composed = atan(Scalar(2) * x - x); + VERIFY_IS_EQUAL(composed.value(), result.value()); + VERIFY_IS_EQUAL(composed.derivatives(), result.derivatives()); + const auto deferred = atan(x); + const Derivatives derivatives = deferred.derivatives(); + VERIFY_IS_EQUAL(derivatives, result.derivatives()); + } + + // The derivative stays normal even though x*x overflows and its reciprocal underflows. + const Scalar largest = (NumTraits<Scalar>::highest)(); + for (Scalar sign : {Scalar(-1), Scalar(1)}) { + seed << largest, -largest; + const AD result = atan(AD(sign * (largest / Scalar(4)), seed)); + const Scalar expected = Scalar(16) / largest; + VERIFY((numext::isfinite)(result.derivatives()(0))); + VERIFY(numext::abs(result.derivatives()(0) / expected - Scalar(1)) <= tolerance); + VERIFY(numext::abs(result.derivatives()(1) / expected + Scalar(1)) <= tolerance); + } + + seed << Scalar(2), Scalar(-3); + for (Scalar sign : {Scalar(-1), Scalar(1)}) { + const AD result = atan(AD(sign * NumTraits<Scalar>::infinity(), seed)); + VERIFY_IS_EQUAL(result.value(), atan(sign * NumTraits<Scalar>::infinity())); + VERIFY(result.derivatives().isZero()); + } + const AD nan_result = atan(AD(NumTraits<Scalar>::quiet_NaN(), seed)); + VERIFY((numext::isnan)(nan_result.value())); + VERIFY((numext::isnan)(nan_result.derivatives()(0))); + VERIFY((numext::isnan)(nan_result.derivatives()(1))); + + const AD constant = atan(AD(Scalar(0.5))); + VERIFY(constant.derivatives().isZero()); +} + +template <typename Scalar> +void check_atan_second_derivative() { + using FirstDerivative = AutoDiffScalar<Matrix<Scalar, 1, 1>>; + using SecondDerivative = AutoDiffScalar<Matrix<FirstDerivative, 1, 1>>; + for (Scalar value : {Scalar(0.5), Scalar(2)}) { + SecondDerivative x; + x.value() = FirstDerivative(value); + x.value().derivatives()(0) = Scalar(1); + x.derivatives()(0) = FirstDerivative(Scalar(1)); + const SecondDerivative result = atan(x); + const Scalar denominator = Scalar(1) + value * value; + const Scalar expected = -Scalar(2) * value / (denominator * denominator); + const Scalar tolerance = Scalar(16) * NumTraits<Scalar>::epsilon(); + VERIFY(numext::abs(result.value().derivatives()(0) - Scalar(1) / denominator) <= tolerance); + VERIFY(numext::abs(result.derivatives()(0).value() - Scalar(1) / denominator) <= tolerance); + VERIFY(numext::abs(result.derivatives()(0).derivatives()(0) - expected) <= tolerance); + } +} + +template <typename Scalar> +void check_issue234_operations() { + using Derivatives = Matrix<Scalar, 2, 1>; + using AD = AutoDiffScalar<Derivatives>; + const AD x(Scalar(3), Derivatives::UnitX()); + const AD y = Scalar(1) - Scalar(2) * x; + const AD opposite = -x; + VERIFY(x < Scalar(4)); + VERIFY(x < AD(Scalar(4))); + VERIFY_IS_EQUAL(y.value(), Scalar(-5)); + VERIFY_IS_EQUAL(y.derivatives(), Derivatives(Scalar(-2), Scalar(0))); + VERIFY_IS_EQUAL(opposite.value(), Scalar(-3)); + VERIFY_IS_EQUAL(opposite.derivatives(), -x.derivatives()); + Matrix<AD, 2, 1> v; + v << x, AD(Scalar(4), Derivatives::UnitY()); + const AD length = v.norm(); + const Scalar tolerance = Scalar(8) * NumTraits<Scalar>::epsilon(); + VERIFY(numext::abs(length.value() - Scalar(5)) <= tolerance * Scalar(5)); + VERIFY(numext::abs(length.derivatives()(0) - Scalar(3) / Scalar(5)) <= tolerance); + VERIFY(numext::abs(length.derivatives()(1) - Scalar(4) / Scalar(5)) <= tolerance); +} + +template <typename Derivatives, typename Function, typename NumextFunction, typename Reference, + typename FirstDerivative, typename SecondDerivative> +void check_unary_math(Function function, NumextFunction numext_function, Reference reference, + FirstDerivative first_derivative, SecondDerivative second_derivative, + std::initializer_list<typename Derivatives::Scalar> values) { + using Scalar = typename Derivatives::Scalar; + using AD = AutoDiffScalar<Derivatives>; + // Up to two square roots and divisions, compared with long-double analytic derivatives. + const long double tolerance = 16.L * static_cast<long double>(NumTraits<Scalar>::epsilon()); + Derivatives seed(2); + seed << Scalar(1), Scalar(-2); + for (Scalar value : values) { + const AD x(value, seed); + const auto deferred = function(x); + const AD result = deferred; + VERIFY_IS_EQUAL(result.value(), reference(value)); + const AD dispatched = numext_function(x); + VERIFY_IS_EQUAL(dispatched.value(), result.value()); + for (Index i = 0; i < 2; ++i) { + const long double expected = + static_cast<long double>(seed(i)) * first_derivative(static_cast<long double>(value)); + const long double error = std::abs(static_cast<long double>(result.derivatives()(i)) - expected); + VERIFY(error <= tolerance * std::abs(expected)); + VERIFY_IS_EQUAL(dispatched.derivatives()(i), result.derivatives()(i)); + } + const AD expression = function(x + x); + const AD doubled = function(AD(Scalar(2) * value, Scalar(2) * seed)); + VERIFY_IS_EQUAL(expression.value(), doubled.value()); + VERIFY_IS_EQUAL(expression.derivatives(), doubled.derivatives()); + AutoDiffScalar<Derivatives&> reference_input(value, seed); + const AD referenced = function(reference_input); + VERIFY_IS_EQUAL(referenced.derivatives(), result.derivatives()); + const AD constant = function(AD(value)); + VERIFY(constant.derivatives().isZero()); + + using Inner = AutoDiffScalar<Matrix<Scalar, 1, 1>>; + using Outer = AutoDiffScalar<Matrix<Inner, 1, 1>>; + Outer nested; + nested.value() = Inner(value); + nested.value().derivatives()(0) = Scalar(1); + nested.derivatives()(0) = Inner(Scalar(1)); + const Outer twice = function(nested); + const long double expected_second = second_derivative(static_cast<long double>(value)); + VERIFY(std::abs(static_cast<long double>(twice.derivatives()(0).derivatives()(0)) - expected_second) <= + tolerance * std::abs(expected_second)); + } + const AD nan_result = function(AD(NumTraits<Scalar>::quiet_NaN(), seed)); + VERIFY((numext::isnan)(nan_result.value())); + VERIFY((numext::isnan)(nan_result.derivatives()(0))); +} + +template <typename Scalar, typename Function, typename Reference> +void check_unary_special_values(Function function, Reference reference) { + using AD = AutoDiffScalar<Matrix<Scalar, 1, 1>>; + const Scalar infinity = NumTraits<Scalar>::infinity(); + for (Scalar value : + {Scalar(0), -Scalar(0), Scalar(-1), Scalar(1), -infinity, infinity, numext::nextafter(Scalar(-1), -infinity), + numext::nextafter(Scalar(-1), Scalar(0)), numext::nextafter(Scalar(1), Scalar(0)), + numext::nextafter(Scalar(1), infinity), std::numeric_limits<Scalar>::denorm_min()}) { + const AD result = function(AD(value, Matrix<Scalar, 1, 1>::Ones())); + const Scalar expected = reference(value); + if ((numext::isnan)(expected)) { + VERIFY((numext::isnan)(result.value())); + } else { + VERIFY_IS_EQUAL(result.value(), expected); + VERIFY_IS_EQUAL(bool((numext::signbit)(result.value())), bool((numext::signbit)(expected))); + } + } +} + +template <typename Scalar, typename Function, typename FirstDerivative> +void check_unary_large_derivative(Function function, FirstDerivative first_derivative, + std::initializer_list<Scalar> values, + Scalar seed = NumTraits<Scalar>::highest() / Scalar(4)) { + using AD = AutoDiffScalar<Matrix<Scalar, 1, 1>>; + for (Scalar value : values) { + const AD result = function(AD(value, Matrix<Scalar, 1, 1>::Constant(seed))); + const long double expected = static_cast<long double>(seed) * first_derivative(static_cast<long double>(value)); + // The scaled seed keeps the reference normal even when the unscaled derivative is tiny. + VERIFY(std::abs(static_cast<long double>(result.derivatives()(0)) - expected) <= + 16.L * static_cast<long double>(NumTraits<Scalar>::epsilon()) * std::abs(expected)); + } +} + +template <typename Derivatives> +void check_subnormal_math_derivatives() { + using Scalar = typename Derivatives::Scalar; + using AD = AutoDiffScalar<Derivatives>; + // The scalar probes pass on ARMv7, whose VFP keeps gradual underflow, but derivatives divide in packets and its NEON + // flushes subnormals in float. + if (!(subnormalInputProbe<Scalar>() > Scalar(0)) || !(underflowProbe<Scalar>() > Scalar(0)) || + !subnormalDivisionIsExact<Scalar>()) { + return; + } + const Scalar denorm = std::numeric_limits<Scalar>::denorm_min(); + const Scalar normal = (std::numeric_limits<Scalar>::min)(); + const Scalar boundary = numext::nextafter(Scalar(1), Scalar(0)); + const Scalar inner = numext::nextafter(boundary, Scalar(0)); + const long double epsilon = static_cast<long double>(NumTraits<Scalar>::epsilon()); + const auto check = [&](const auto& function, const auto& reference, std::initializer_list<Scalar> values) { + for (Scalar value : values) { + for (Scalar seed : {denorm, Scalar(2) * denorm, normal / Scalar(2), normal}) { + Derivatives derivatives(2); + derivatives << seed, -seed; + const AD input(value, derivatives); + const auto deferred = function(input); + const AD result = deferred; + const AutoDiffScalar<Derivatives&> referenced(value, derivatives); + const AD from_reference = function(referenced); + VERIFY_IS_EQUAL(result.derivatives(), from_reference.derivatives()); + for (Index i = 0; i < derivatives.size(); ++i) { + const long double expected = + reference(static_cast<long double>(derivatives(i)), static_cast<long double>(value)); + if ((std::isinf)(expected)) { + VERIFY((numext::isinf)(result.derivatives()(i))); + VERIFY_IS_EQUAL((std::signbit)(result.derivatives()(i)), (std::signbit)(expected)); + } else { + // Allow ordinary rounding plus one subnormal quantum when the final result is tiny. + const long double bound = 16 * epsilon * std::abs(expected) + static_cast<long double>(denorm); + VERIFY(std::abs(static_cast<long double>(result.derivatives()(i)) - expected) <= bound); + } + } + } + } + }; + check([](const auto& x) { return log10(x); }, + [](long double seed, long double x) { return (seed / x) / std::log(10.L); }, + {denorm, Scalar(2) * denorm, normal / Scalar(2), numext::nextafter(normal, Scalar(0)), normal, + numext::nextafter(normal, Scalar(1)), Scalar(0.125), Scalar(0.5), Scalar(1)}); + check([](const auto& x) { return atanh(x); }, + [](long double seed, long double x) { return seed / ((1 - x) * (1 + x)); }, + {-Scalar(1), -boundary, -inner, Scalar(-0.5), Scalar(0), Scalar(0.5), inner, boundary, Scalar(1)}); +} + +template <typename Derivatives> +void check_additional_math() { + using Scalar = typename Derivatives::Scalar; +#define CHECK_AUTODIFF_MATH(NAME, FIRST, SECOND, ...) \ + check_unary_math<Derivatives>( \ + [](const auto& x) { \ + using std::NAME; \ + return NAME(x); \ + }, \ + [](const auto& x) { return numext::NAME(x); }, [](Scalar x) { return std::NAME(x); }, \ + [](long double x) { return FIRST; }, [](long double x) { return SECOND; }, {__VA_ARGS__}); \ + check_unary_special_values<Scalar>( \ + [](const auto& x) { \ + using std::NAME; \ + return NAME(x); \ + }, \ + [](Scalar x) { return std::NAME(x); }) + CHECK_AUTODIFF_MATH(log1p, 1 / (1 + x), -1 / ((1 + x) * (1 + x)), Scalar(-0.25), Scalar(0), Scalar(1e-8), Scalar(0.5), + Scalar(3)); + CHECK_AUTODIFF_MATH(expm1, std::exp(x), std::exp(x), Scalar(-10), Scalar(-0.5), Scalar(0), Scalar(1e-8), Scalar(3)); + CHECK_AUTODIFF_MATH(log2, 1 / (x * std::log(2.L)), -1 / (x * x * std::log(2.L)), Scalar(0.125), Scalar(0.5), + Scalar(1), Scalar(8)); + CHECK_AUTODIFF_MATH(log10, 1 / (x * std::log(10.L)), -1 / (x * x * std::log(10.L)), Scalar(0.125), Scalar(0.5), + Scalar(1), Scalar(10)); + CHECK_AUTODIFF_MATH(cbrt, 1 / (3 * std::cbrt(x) * std::cbrt(x)), -2 / (9 * x * std::cbrt(x) * std::cbrt(x)), + Scalar(-8), Scalar(-0.125), Scalar(0.125), Scalar(8)); + CHECK_AUTODIFF_MATH(asinh, 1 / std::sqrt(1 + x * x), -x / std::pow(1 + x * x, 1.5L), Scalar(-8), Scalar(-0.5), + Scalar(0), Scalar(0.5), Scalar(8)); + CHECK_AUTODIFF_MATH(acosh, 1 / std::sqrt(x * x - 1), -x / std::pow(x * x - 1, 1.5L), + numext::nextafter(Scalar(1), Scalar(2)), Scalar(2), Scalar(8)); + CHECK_AUTODIFF_MATH(atanh, 1 / (1 - x * x), 2 * x / ((1 - x * x) * (1 - x * x)), Scalar(-0.25), Scalar(0), + Scalar(0.25)); +#undef CHECK_AUTODIFF_MATH + const Scalar large = NumTraits<Scalar>::highest() / Scalar(4); + const Scalar large_seed = NumTraits<Scalar>::highest() * Scalar(0.6); + check_unary_large_derivative<Scalar>([](const auto& x) { return atanh(x); }, + [](long double x) { return 1 / (1 - x * x); }, {Scalar(-0.5), Scalar(0.5)}, + large_seed); + check_unary_large_derivative<Scalar>([](const auto& x) { return log10(x); }, + [](long double x) { return 1 / (x * std::log(10.L)); }, {Scalar(0.5)}, + large_seed); + check_unary_large_derivative<Scalar>([](const auto& x) { return log1p(x); }, + [](long double x) { return 1 / (1 + x); }, {large}); + check_unary_large_derivative<Scalar>([](const auto& x) { return log2(x); }, + [](long double x) { return 1 / (x * std::log(2.L)); }, {large}); + check_unary_large_derivative<Scalar>([](const auto& x) { return log10(x); }, + [](long double x) { return 1 / (x * std::log(10.L)); }, {large}); + check_unary_large_derivative<Scalar>([](const auto& x) { return cbrt(x); }, + [](long double x) { return 1 / (3 * std::cbrt(x) * std::cbrt(x)); }, + {-large, large}); + check_unary_large_derivative<Scalar>([](const auto& x) { return asinh(x); }, + [](long double x) { return 1 / std::hypot(1.L, x); }, {-large, large}); + check_unary_large_derivative<Scalar>([](const auto& x) { return acosh(x); }, + [](long double x) { return 1 / (std::sqrt(x - 1) * std::sqrt(x + 1)); }, + {large}); + check_subnormal_math_derivatives<Derivatives>(); +} + EIGEN_DECLARE_TEST(autodiff_scalar) { for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST_1(check_atan2<float>()); @@ -116,5 +427,17 @@ CALL_SUBTEST_5(check_limits_specialization<double>()); CALL_SUBTEST_6(check_min_max_equal_values<float>()); CALL_SUBTEST_7(check_min_max_equal_values<double>()); + CALL_SUBTEST_8(check_atan<Vector2f>()); + CALL_SUBTEST_8(check_atan<VectorXf>()); + CALL_SUBTEST_8(check_issue234_operations<float>()); + CALL_SUBTEST_8(check_atan_second_derivative<float>()); + CALL_SUBTEST_9(check_atan<Vector2d>()); + CALL_SUBTEST_9(check_atan<VectorXd>()); + CALL_SUBTEST_9(check_issue234_operations<double>()); + CALL_SUBTEST_9(check_atan_second_derivative<double>()); + CALL_SUBTEST_10(check_additional_math<Vector2f>()); + CALL_SUBTEST_10(check_additional_math<VectorXf>()); + CALL_SUBTEST_11(check_additional_math<Vector2d>()); + CALL_SUBTEST_11(check_additional_math<VectorXd>()); } }
diff --git a/contrib/test/kronecker_product.cpp b/contrib/test/kronecker_product.cpp index 2cc309e..ac0a509 100644 --- a/contrib/test/kronecker_product.cpp +++ b/contrib/test/kronecker_product.cpp
@@ -15,6 +15,7 @@ #include "sparse.h" #include <Eigen/SparseExtra> #include <Eigen/KroneckerProduct> +#include "structured_test_helpers.h" template <typename MatrixType> void check_dimension(const MatrixType& ab, const int rows, const int cols) { @@ -204,35 +205,43 @@ initSparse(density, dB, sB); sC = kroneckerProduct(sA, sB); - dC = kroneckerProduct(dA, dB); + dC = reference_kron<float>(dA, dB); + VERIFY_IS_APPROX(kroneckerProduct(dA, dB).eval(), dC); VERIFY_IS_APPROX(MatrixXf(sC), dC); sC = kroneckerProduct(sA.transpose(), sB); - dC = kroneckerProduct(dA.transpose(), dB); + dC = reference_kron<float>(dA.transpose(), dB); + VERIFY_IS_APPROX(kroneckerProduct(dA.transpose(), dB).eval(), dC); VERIFY_IS_APPROX(MatrixXf(sC), dC); sC = kroneckerProduct(sA.transpose(), sB.transpose()); - dC = kroneckerProduct(dA.transpose(), dB.transpose()); + dC = reference_kron<float>(dA.transpose(), dB.transpose()); + VERIFY_IS_APPROX(kroneckerProduct(dA.transpose(), dB.transpose()).eval(), dC); VERIFY_IS_APPROX(MatrixXf(sC), dC); sC = kroneckerProduct(sA, sB.transpose()); - dC = kroneckerProduct(dA, dB.transpose()); + dC = reference_kron<float>(dA, dB.transpose()); + VERIFY_IS_APPROX(kroneckerProduct(dA, dB.transpose()).eval(), dC); VERIFY_IS_APPROX(MatrixXf(sC), dC); sC2 = kroneckerProduct(sA, sB); - dC = kroneckerProduct(dA, dB); + dC = reference_kron<float>(dA, dB); + VERIFY_IS_APPROX(kroneckerProduct(dA, dB).eval(), dC); VERIFY_IS_APPROX(MatrixXf(sC2), dC); sC2 = kroneckerProduct(dA, sB); - dC = kroneckerProduct(dA, dB); + dC = reference_kron<float>(dA, dB); + VERIFY_IS_APPROX(kroneckerProduct(dA, dB).eval(), dC); VERIFY_IS_APPROX(MatrixXf(sC2), dC); sC2 = kroneckerProduct(sA, dB); - dC = kroneckerProduct(dA, dB); + dC = reference_kron<float>(dA, dB); + VERIFY_IS_APPROX(kroneckerProduct(dA, dB).eval(), dC); VERIFY_IS_APPROX(MatrixXf(sC2), dC); sC2 = kroneckerProduct(2 * sA, sB); - dC = kroneckerProduct(2 * dA, dB); + dC = reference_kron<float>(2 * dA, dB); + VERIFY_IS_APPROX(kroneckerProduct(2 * dA, dB).eval(), dC); VERIFY_IS_APPROX(MatrixXf(sC2), dC); } }
diff --git a/contrib/test/matrix_exponential.cpp b/contrib/test/matrix_exponential.cpp index 74cfa35..402d7b8 100644 --- a/contrib/test/matrix_exponential.cpp +++ b/contrib/test/matrix_exponential.cpp
@@ -122,6 +122,7 @@ VERIFY(A.exp().isApprox(expected, tol)); } +template <typename = void> void testCustomComplexScalingPath() { using Scalar = CustomComplex<double>; using MatrixType = Matrix<Scalar, 3, 3>; @@ -140,6 +141,7 @@ } // The 1-norm of an empty matrix was the maxCoeff() of no column sums. +template <typename = void> void testEmptyMatrix() { VERIFY_IS_EQUAL(MatrixXd(MatrixXd(0, 0).exp()).size(), 0); VERIFY_IS_EQUAL(MatrixXcd(MatrixXcd(0, 0).exp()).size(), 0); @@ -182,11 +184,11 @@ CALL_SUBTEST_3(randomTest(Matrix4cd(), 384 * NumTraits<std::complex<double>>::epsilon())); CALL_SUBTEST_3((testComplexScalingPath<ColMajor>())); CALL_SUBTEST_3((testComplexScalingPath<RowMajor>())); - CALL_SUBTEST_3(testCustomComplexScalingPath()); + CALL_SUBTEST_3(testCustomComplexScalingPath<>()); CALL_SUBTEST_4(randomTest(MatrixXd(8, 8), 384 * NumTraits<double>::epsilon())); - CALL_SUBTEST_4(testEmptyMatrix()); + CALL_SUBTEST_4(testEmptyMatrix<>()); CALL_SUBTEST_1(randomTest(Matrix2f(), 384 * NumTraits<float>::epsilon())); - CALL_SUBTEST_5(randomTest(Matrix3cf(), 384 * NumTraits<std::complex<float>>::epsilon())); + CALL_SUBTEST_10(randomTest(Matrix3cf(), 384 * NumTraits<std::complex<float>>::epsilon())); CALL_SUBTEST_1(randomTest(Matrix4f(), 384 * NumTraits<float>::epsilon())); CALL_SUBTEST_6(randomTest(MatrixXf(8, 8), 384 * NumTraits<float>::epsilon())); CALL_SUBTEST_9(randomTest(Matrix<long double, Dynamic, Dynamic>(7, 7), 384 * NumTraits<long double>::epsilon()));
diff --git a/contrib/test/matrix_function.cpp b/contrib/test/matrix_function.cpp index be7c5b8..673f1ee 100644 --- a/contrib/test/matrix_function.cpp +++ b/contrib/test/matrix_function.cpp
@@ -213,6 +213,7 @@ // The loop could also create an infinite entry: the first square root divides A(0, 1) by 2 sqrt(A(0, 0)) = 2e-10, // which overflows, as log(A)(0, 1) = A(0, 1) / A(0, 0) does. The three eigenvalues form one atomic block. +template <typename = void> void testMatrixLogarithmSquareRootOverflow() { MatrixXd A(3, 3); A << 1e-20, 1e300, 0, 0, 1e-20, 1, 0, 0, 1e-20; @@ -243,7 +244,7 @@ CALL_SUBTEST_3(testMatrixLogarithmSingular<MatrixXf>()); CALL_SUBTEST_7(testMatrixLogarithmSingular<MatrixXd>()); - CALL_SUBTEST_7(testMatrixLogarithmSquareRootOverflow()); + CALL_SUBTEST_7(testMatrixLogarithmSquareRootOverflow<>()); CALL_SUBTEST_3(testEmptyMatrix<MatrixXf>()); CALL_SUBTEST_7(testEmptyMatrix<MatrixXcd>()); }
diff --git a/contrib/test/matrix_power.cpp b/contrib/test/matrix_power.cpp index ad88a0c..f41fa30 100644 --- a/contrib/test/matrix_power.cpp +++ b/contrib/test/matrix_power.cpp
@@ -137,6 +137,7 @@ } // Zero eigenvalues found below the leading 2x2 block of the Schur factor were not all split off. +template <typename = void> void testSingularTrailingZero() { Matrix3d A; A << 1, 1, 1, 0, 4, 1, 0, 0, 0; @@ -147,6 +148,7 @@ } // A 2x2 base kept its zero eigenvalues in the atomic block, where a double one gave p * 0^(p-1) * 0 = NaN. +template <typename = void> void testSingularTwoByTwo() { VERIFY(Matrix2d(Matrix2d::Zero().pow(0.5)).isZero()); VERIFY(Matrix2d(Matrix2d::Zero().pow(1.5)).isZero()); @@ -159,6 +161,7 @@ // The coupling T11^(p-1) T12 of the nonsingular block T11 to the zero eigenvalues must not overflow or underflow when // the result does not. Forming T11^p T12 first failed for s = 1e300 and s = 1e-300, and forming T11^(p-1) first // failed for the tiny nonnormal T11 below. +template <typename = void> void testSingularExtremeScale() { // The complex pow(t, p) = exp(p log t) of a diagonal entry t near 1e+-300 can lose |p log t| ~ 345 ulp. const double tol = 512 * NumTraits<double>::epsilon(); @@ -185,7 +188,8 @@ } // MatrixPowerAtomic left the (1, 0) entry of a 2x2 result unset, and MatrixPower read it when it coupled a 2x2 block -// to a zero eigenvalue, as for the first matrix of testSingularTrailingZero(). +// to a zero eigenvalue, as for the first matrix of testSingularTrailingZero<>(). +template <typename = void> void testAtomicTwoByTwoLowerPart() { Matrix2d T, expected, res = Matrix2d::Constant(std::numeric_limits<double>::quiet_NaN()); T << 4, 1, 0, 9; @@ -196,6 +200,7 @@ } // MatrixPower::compute() did not size the result for 0x0 and 1x1 bases. +template <typename = void> void testComputeResultSize() { MatrixXd a = MatrixXd::Constant(1, 1, 4), r; MatrixPower<MatrixXd>(a).compute(r, 0.5); @@ -207,6 +212,7 @@ } // An infinite entry used to hang the square-rooting loop for atomic blocks larger than 2x2. +template <typename = void> void testInfiniteEntry() { Matrix3d a = Matrix3d::Identity(); a(0, 2) = std::numeric_limits<double>::infinity(); @@ -220,6 +226,7 @@ } // A nonfinite exponent used to hang the binary powering of the integral part. +template <typename = void> void testNonFiniteExponent() { Matrix3d a; a << 1, 2, 3, 4, 5, 6, 7, 8, 10; @@ -287,19 +294,19 @@ CALL_SUBTEST_7(testSingular(Matrix3dRowMajor(), 512 * NumTraits<double>::epsilon())); CALL_SUBTEST_3(testSingular(Matrix4cd(), 8 * NumTraits<std::complex<double>>::epsilon())); CALL_SUBTEST_4(testSingular(MatrixXd(8, 8), 128 * NumTraits<double>::epsilon())); - CALL_SUBTEST_4(testComputeResultSize()); + CALL_SUBTEST_4(testComputeResultSize<>()); CALL_SUBTEST_1(testSingular(Matrix2f(), 8 * NumTraits<float>::epsilon())); CALL_SUBTEST_5(testSingular(Matrix3cf(), 8 * NumTraits<std::complex<float>>::epsilon())); CALL_SUBTEST_8(testSingular(Matrix4f(), 256 * NumTraits<float>::epsilon())); CALL_SUBTEST_6(testSingular(MatrixXf(2, 2), 8 * NumTraits<float>::epsilon())); CALL_SUBTEST_9(testSingular(MatrixXe(7, 7), 256 * NumTraits<long double>::epsilon())); CALL_SUBTEST_10(testSingular(Matrix3d(), 1024 * NumTraits<double>::epsilon())); - CALL_SUBTEST_10(testSingularTrailingZero()); - CALL_SUBTEST_10(testSingularTwoByTwo()); - CALL_SUBTEST_10(testSingularExtremeScale()); - CALL_SUBTEST_10(testAtomicTwoByTwoLowerPart()); - CALL_SUBTEST_10(testInfiniteEntry()); - CALL_SUBTEST_10(testNonFiniteExponent()); + CALL_SUBTEST_10(testSingularTrailingZero<>()); + CALL_SUBTEST_10(testSingularTwoByTwo<>()); + CALL_SUBTEST_10(testSingularExtremeScale<>()); + CALL_SUBTEST_10(testAtomicTwoByTwoLowerPart<>()); + CALL_SUBTEST_10(testInfiniteEntry<>()); + CALL_SUBTEST_10(testNonFiniteExponent<>()); CALL_SUBTEST_11(testSingular(Matrix3f(), 2048 * NumTraits<float>::epsilon())); CALL_SUBTEST_12(testSingular(Matrix3e(), 1024 * NumTraits<long double>::epsilon()));
diff --git a/contrib/test/riccati.cpp b/contrib/test/riccati.cpp new file mode 100644 index 0000000..8924017 --- /dev/null +++ b/contrib/test/riccati.cpp
@@ -0,0 +1,751 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// This Source Code Form is subject to the terms of the Mozilla +// Public License v. 2.0. If a copy of the MPL was not distributed +// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-FileCopyrightText: 2001-2002 Enthought, Inc. +// SPDX-FileCopyrightText: 2003 SciPy Developers +// SPDX-License-Identifier: MPL-2.0 AND BSD-3-Clause +// +// This file incorporates work covered by the following copyright and +// permission notice: +// +// dare_scipy_complex_cases(), dare_scipy_real_cases() and dare_generalized_scipy_cases() +// are adapted from the solve_discrete_are tests of SciPy 1.18.0 (scipy/linalg/tests/test_solvers.py and +// scipy/linalg/tests/data/gendare_20170120_data.npz): +// +// Copyright (c) 2001-2002 Enthought, Inc. 2003, SciPy Developers. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions +// are met: +// +// 1. Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// +// 2. Redistributions in binary form must reproduce the above +// copyright notice, this list of conditions and the following +// disclaimer in the documentation and/or other materials provided +// with the distribution. +// +// 3. Neither the name of the copyright holder nor the names of its +// contributors may be used to endorse or promote products derived +// from this software without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include "main.h" +#include <contrib/Eigen/Riccati> + +#include <algorithm> +#include <vector> + +namespace { + +// Residual of A^* X A - E^* X E - (A^* X B + S) (R + B^* X B)^-1 (B^* X A + S^*) + Q. +template <typename MatrixType> +MatrixType dare_residual(const MatrixType& A, const MatrixType& B, const MatrixType& Q, const MatrixType& R, + const MatrixType& E, const MatrixType& S, const MatrixType& X) { + const MatrixType AXB = A.adjoint() * X * B + S; + const MatrixType gain = (R + B.adjoint() * X * B).fullPivLu().solve(AXB.adjoint()); + return A.adjoint() * X * A - E.adjoint() * X * E - AXB * gain + Q; +} + +template <typename MatrixType> +MatrixType dare_residual(const MatrixType& A, const MatrixType& B, const MatrixType& Q, const MatrixType& R, + const MatrixType& X) { + return dare_residual(A, B, Q, R, MatrixType::Identity(A.rows(), A.rows()).eval(), + MatrixType::Zero(B.rows(), B.cols()).eval(), X); +} + +// Solves the DARE and checks the residual entrywise against SciPy's absolute tolerance for the case. +template <typename MatrixType> +void check_dare(const MatrixType& A, const MatrixType& B, const MatrixType& Q, const MatrixType& R, + typename MatrixType::RealScalar atol) { + DiscreteAlgebraicRiccatiSolver<MatrixType> dare(A, B, Q, R); + VERIFY_IS_EQUAL(dare.info(), Success); + const MatrixType& X = dare.solution(); + VERIFY_IS_EQUAL(X, X.adjoint()); + VERIFY(dare_residual(A, B, Q, R, X).cwiseAbs().maxCoeff() <= atol); +} + +template <typename MatrixType> +void check_generalized_dare(const MatrixType& A, const MatrixType& B, const MatrixType& Q, const MatrixType& R, + const MatrixType& E, const MatrixType& S, typename MatrixType::RealScalar atol) { + DiscreteAlgebraicRiccatiSolver<MatrixType> dare(A, B, Q, R, E, S); + VERIFY_IS_EQUAL(dare.info(), Success); + const MatrixType& X = dare.solution(); + VERIFY_IS_EQUAL(X, X.adjoint()); + VERIFY(dare_residual(A, B, Q, R, E, S, X).cwiseAbs().maxCoeff() <= atol); +} + +// Solves a real test case in the arithmetic of MatrixType; the real cases are also solved in complex arithmetic to +// exercise the ComplexQZ code path. +template <typename MatrixType> +void check_real_dare(const MatrixXd& A, const MatrixXd& B, const MatrixXd& Q, const MatrixXd& R, double atol) { + using Scalar = typename MatrixType::Scalar; + check_dare<MatrixType>(A.cast<Scalar>(), B.cast<Scalar>(), Q.cast<Scalar>(), R.cast<Scalar>(), atol); +} + +// Backward error check for a computed solution X: +// rho = ||res||_F / (m eps (||A^* X A||_F + ||E^* X E||_F + ||Q||_F) kappa) <= 30, +// kappa = max(1, ||E^* X E||_F / ||Q||_F), +// where kappa accounts for the amplification of rounding errors by a large solution: for the problems with B scaled by +// 10^-k, the residual of an accurately computed solution grows in proportion to ||X|| / ||Q||, also with SciPy +// (LAPACK), up to 1e10 eps for double. Over 160000 solves of dare_random() problems in all four scalar types, rho had +// p99.9 <= 3.9 and maximum 9.7. +template <typename DynMatrix> +void verify_dare_backward_error(const DynMatrix& A, const DynMatrix& B, const DynMatrix& Q, const DynMatrix& R, + const DynMatrix& E, const DynMatrix& S, const DynMatrix& X) { + using RealScalar = typename NumTraits<typename DynMatrix::Scalar>::Real; + VERIFY(X.allFinite()); + const RealScalar normEXE = (E.adjoint() * X * E).norm(); + const RealScalar scale = (A.adjoint() * X * A).norm() + normEXE + Q.norm(); + const RealScalar kappa = numext::maxi(RealScalar(1), normEXE / Q.norm()); + const RealScalar bound = RealScalar(30) * RealScalar(A.rows()) * NumTraits<RealScalar>::epsilon() * scale * kappa; + VERIFY(dare_residual(A, B, Q, R, E, S, X).norm() <= bound); +} + +// The closed-loop pencil (A - B K, E), K = (R + B^* X B)^-1 (B^* X A + S^*), of the stabilizing solution is stable. +template <typename DynMatrix> +void verify_stabilizing(const DynMatrix& A, const DynMatrix& B, const DynMatrix& R, const DynMatrix& E, + const DynMatrix& S, const DynMatrix& X) { + using RealScalar = typename NumTraits<typename DynMatrix::Scalar>::Real; + using ComplexMatrix = Matrix<std::complex<RealScalar>, Dynamic, Dynamic>; + const DynMatrix K = (R + B.adjoint() * X * B).fullPivLu().solve(B.adjoint() * X * A + S.adjoint()); + const ComplexMatrix closedLoop = E.fullPivLu().solve(A - B * K).template cast<std::complex<RealScalar>>(); + VERIFY(ComplexEigenSolver<ComplexMatrix>(closedLoop).eigenvalues().cwiseAbs().maxCoeff() < RealScalar(1)); +} + +} // namespace + +#if defined(EIGEN_TEST_PART_7) || defined(EIGEN_TEST_PART_ALL) +// The test cases of scipy.linalg.solve_discrete_are with complex data. +void dare_scipy_complex_cases() { + using cd = std::complex<double>; + { // complex A; real B, Q, R + MatrixXcd A(2, 2), B(2, 1), Q(2, 2), R(1, 1); + A << cd(2, 0), cd(1, -2), cd(0, 0), cd(0, -3); + B << 0, 1; + Q << 1, 0, 0, 2; + R << 1; + check_dare(A, B, Q, R, 1.5e-12); + } + { // real A, Q, R; complex B + MatrixXcd A(2, 2), B(2, 1), Q(2, 2), R(1, 1); + A << 2, 1, 0, -1; + B << cd(0, -2), cd(0, 1); + Q << 1, 0, 0, 2; + R << 1; + check_dare(A, B, Q, R, 1.5e-14); + } + { // real A, B; complex Q, R + MatrixXcd A(2, 2), B(2, 2), Q(2, 2), R(2, 2); + A << 3, 1, 0, -1; + B << 1, 2, 1, 3; + Q << cd(1, 0), cd(1, 1), cd(1, -1), cd(2, 0); + R << cd(2, 0), cd(0, -2), cd(0, 2), cd(3, 0); + check_dare(A, B, Q, R, 1.5e-13); + } +} +#endif + +// The test cases of scipy.linalg.solve_discrete_are with real data, most of them from +// P. Benner, A. J. Laub, V. Mehrmann, "A Collection of Benchmark Examples for the Numerical Solution of Algebraic +// Riccati Equations II: Discrete-Time Case", Tech. Report SPC 95_23, TU Chemnitz-Zwickau, 1995 (DAREX). +template <typename MatrixType> +void dare_scipy_real_cases() { + { // SciPy gh-2251 + MatrixXd A(3, 3), B(3, 1); + A << 0.63399379, 0.54906824, 0.76253406, 0.5404729, 0.53745766, 0.08731853, 0.27524045, 0.84922129, 0.4681622; + B << 0.96861695, 0.05532739, 0.78934047; + check_real_dare<MatrixType>(A, B, MatrixXd::Identity(3, 3), MatrixXd::Identity(1, 1), 1.5e-14); + } + { // DAREX 1 + MatrixXd A(2, 2), B(2, 1), Q(2, 2), R(1, 1); + A << 4, 3, -4.5, -3.5; + B << 1, -1; + Q << 9, 6, 6, 4; + R << 1; + check_real_dare<MatrixType>(A, B, Q, R, 1.5e-13); + } + { // DAREX 2 + MatrixXd A(2, 2), B(2, 2), Q(2, 2), R(2, 2); + A << 0.9512, 0, 0, 0.9048; + B << 4.877, 4.877, -1.1895, 3.569; + Q << 0.005, 0, 0, 0.02; + R << 1. / 3, 0, 0, 3; + check_real_dare<MatrixType>(A, B, Q, R, 1.5e-14); + } + { // DAREX 3: singular R + MatrixXd A(2, 2), B(2, 1), Q(2, 2), R(1, 1); + A << 2, -1, 1, 0; + B << 1, 0; + Q << 0, 0, 0, 1; + R << 0; + // SciPy's bound of 1.5e-18 holds there only because LAPACK happens to round to the exact solution; X has O(1) + // entries, so a residual of a few eps is backward stable. + check_real_dare<MatrixType>(A, B, Q, R, 4 * NumTraits<double>::epsilon()); + } + { // DAREX 4 (without the generalized term S) + MatrixXd A(2, 2), B(2, 2), Q(2, 2), R(2, 2); + A << 0, 1, 0, -1; + B << 1, 0, 2, 1; + Q << -4, -4, -4, 7; + Q /= 11; + R << 9, 3, 3, 1; + check_real_dare<MatrixType>(A, B, Q, R, 1.5e-14); + } + { // DAREX 5 + MatrixXd A(2, 2), B(2, 1), Q(2, 2), R(1, 1); + A << 0, 1, 0, 0; + B << 0, 1; + Q << 1, 2, 2, 4; + R << 1; + check_real_dare<MatrixType>(A, B, Q, R, 1.5e-14); + } + { // DAREX 6 + MatrixXd A(4, 4), B(4, 2), Q(4, 4); + A << 0.998, 0.067, 0, 0, -.067, 0.998, 0, 0, 0, 0, 0.998, 0.153, 0, 0, -.153, 0.998; + B << 0.0033, 0.0200, 0.1000, -.0007, 0.0400, 0.0073, -.0028, 0.1000; + Q << 1.87, 0, 0, -0.244, 0, 0.744, 0.205, 0, 0, 0.205, 0.589, 0, -0.244, 0, 0, 1.048; + check_real_dare<MatrixType>(A, B, Q, MatrixXd::Identity(2, 2), 1.5e-13); + } + { // DAREX 7 + MatrixXd A(4, 4), B(4, 2); + A << 0.984750, -.079903, 0.0009054, -.0010765, 0.041588, 0.998990, -.0358550, 0.0126840, -.546620, 0.044916, + -.3299100, 0.1931800, 2.662400, -.100450, -.9245500, -.2632500; + B << 0.0037112, 0.0007361, -.0870510, 9.3411e-6, -1.198440, -4.1378e-4, -3.192700, 9.2535e-4; + check_real_dare<MatrixType>(A, B, MatrixXd::Identity(4, 4) * 1e-2, MatrixXd::Identity(2, 2), 1.5e-14); + } + { // DAREX 8 + MatrixXd A(4, 4), B(4, 4), Q(4, 4); + A << -0.6, -2.2, -3.6, -5.400018, 1.0, 0.6, 0.8, 3.399982, 0.0, 1.0, 1.8, 3.799982, 0.0, 0.0, 0.0, -0.999982; + B << 1.0, -1.0, -1.0, -1.0, 0.0, 1.0, -1.0, -1.0, 0.0, 0.0, 1.0, -1.0, 0.0, 0.0, 0.0, 1.0; + Q << 2, 1, 3, 6, 1, 2, 2, 5, 3, 2, 6, 11, 6, 5, 11, 22; + check_real_dare<MatrixType>(A, B, Q, MatrixXd::Identity(4, 4), 5.0e-13); + } + { // DAREX 9 + MatrixXd A(5, 5), B(5, 2); + A << 95.4070, 1.9643, 0.3597, 0.0673, 0.0190, 40.8490, 41.3170, 16.0840, 4.4679, 1.1971, 12.2170, 26.3260, 36.1490, + 15.9300, 12.3830, 4.1118, 12.8580, 27.2090, 21.4420, 40.9760, 0.1305, 0.5808, 1.8750, 3.6162, 94.2800; + A *= 0.01; + B << 0.0434, -0.0122, 2.6606, -1.0453, 3.7530, -5.5100, 3.6076, -6.6000, 0.4617, -0.9148; + B *= 0.01; + // SciPy's 1.5e-13 matches LAPACK's rounding; without vectorization the residual is 1.53e-13. + check_real_dare<MatrixType>(A, B, MatrixXd::Identity(5, 5), MatrixXd::Identity(2, 2), 3e-13); + } + { // DAREX 10 + MatrixXd A = MatrixXd::Zero(6, 6), B = MatrixXd::Zero(6, 2), Q(6, 6), R(2, 2); + A(0, 1) = A(1, 2) = A(3, 4) = A(4, 5) = 1; + B(2, 0) = B(5, 1) = 1; + Q << 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, -1, 0, 0, 0, 0, -1, 1, 0, 0, 0, 0, 0, 0, 0; + R << 3, 0, 0, 1; + check_real_dare<MatrixType>(A, B, Q, R, 1.5e-14); + } + { // DAREX 11 + MatrixXd A(9, 9), B(9, 3); + A << 870.1, 135.0, 11.59, .5014, -37.22, .3484, 0, 4.242, 7.249, // + 76.55, 897.4, 12.72, 0.5504, -40.16, .3743, 0, 4.53, 7.499, // + -127.2, 357.5, 817, 1.455, -102.8, .987, 0, 11.85, 18.72, // + -363.5, 633.9, 74.91, 796.6, -273.5, 2.653, 0, 31.72, 48.82, // + -960, 1645.9, -128.9, -5.597, 71.42, 7.108, 0, 84.52, 125.9, // + -664.4, 112.96, -88.89, -3.854, 84.47, 13.6, 0, 144.3, 101.6, // + -410.2, 693, -54.71, -2.371, 66.49, 12.49, .1063, 99.97, 69.67, // + -179.9, 301.7, -23.93, -1.035, 60.59, 22.16, 0, 213.9, 35.54, // + -345.1, 580.4, -45.96, -1.989, 105.6, 19.86, 0, 219.1, 215.2; + A *= 0.001; + B << 4.7600, -0.5701, -83.6800, 0.8790, -4.7730, -2.7300, 1.4820, -13.1200, 8.8760, 3.8920, -35.1300, 24.8000, + 10.3400, -92.7500, 66.8000, 7.2030, -61.5900, 38.3400, 4.4540, -36.8300, 20.2900, 1.9710, -15.5400, 6.9370, + 3.7730, -30.2800, 14.6900; + B *= 0.001; + VectorXd q = VectorXd::Zero(9); + q(0) = q(4) = 50; + check_real_dare<MatrixType>(A, B, MatrixXd(q.asDiagonal()), MatrixXd::Identity(3, 3), 1.5e-12); + } + { // DAREX 12: numerically the least accurate example + MatrixXd A(2, 2), B(2, 1); + A << 0, 1e6, 0, 0; + B << 0, 1; + check_real_dare<MatrixType>(A, B, MatrixXd::Identity(2, 2), MatrixXd::Identity(1, 1), 2.0e-2); + } + { // DAREX 13 + MatrixXd A(3, 3); + A << 16, 10, -2, 10, 13, -8, -2, -8, 7; + A /= 9; + check_real_dare<MatrixType>(A, MatrixXd::Identity(3, 3), 1e6 * MatrixXd::Identity(3, 3), + 1e6 * MatrixXd::Identity(3, 3), 2.0e-4); + } + { // DAREX 14 + MatrixXd A = MatrixXd::Zero(4, 4), B = MatrixXd::Zero(4, 1), Q = MatrixXd::Zero(4, 4), R(1, 1); + A(0, 0) = 1 - 1 / 1e8; + A(1, 0) = A(2, 1) = A(3, 2) = 1; + B(0, 0) = 1e-8; + Q(3, 3) = 1; + R << 0.25; + // In complex arithmetic the tolerance is raised: where ComplexQZ converges on the balanced pencil (GCC with SSE2), + // the residual is 7.6e-5, against 2.4e-8 with LAPACK's ZGGES. With AVX2 and FMA, or with Clang, ComplexQZ stalls + // on it, and the solver falls back to the unbalanced pencil, whose residual is close to this tolerance for any QZ + // (0.11 here, 5.0e-2 with ZGGES). To be tightened once ComplexQZ converges on this pencil (!3430). + check_real_dare<MatrixType>(A, B, Q, R, NumTraits<typename MatrixType::Scalar>::IsComplex ? 5e-2 : 1.5e-6); + } + { // DAREX 15 + const Index n = 100; + MatrixXd A = MatrixXd::Zero(n, n), B = MatrixXd::Zero(n, 1); + A.diagonal(1).setOnes(); + B(n - 1, 0) = 1; + check_real_dare<MatrixType>(A, B, MatrixXd::Identity(n, n), MatrixXd::Identity(1, 1), 1.5e-10); + } +} + +// The generalized test cases of scipy.linalg.solve_discrete_are. +template <typename MatrixType> +void dare_generalized_scipy_cases() { + using Scalar = typename MatrixType::Scalar; + MatrixXd A(3, 3), B(3, 2), E(3, 3); + A << 2.769230e-01, 8.234578e-01, 9.502220e-01, 4.617139e-02, 6.948286e-01, 3.444608e-02, 9.713178e-02, 3.170995e-01, + 4.387444e-01; + B << 3.815585e-01, 1.868726e-01, 7.655168e-01, 4.897644e-01, 7.951999e-01, 4.455862e-01; + E << 6.463130e-01, 2.760251e-01, 1.626117e-01, 7.093648e-01, 6.797027e-01, 1.189977e-01, 7.546867e-01, 6.550980e-01, + 4.983641e-01; + const MatrixXd Q = MatrixXd::Identity(3, 3), R = MatrixXd::Identity(2, 2); + // SciPy's 1.5e-11 matches LAPACK's rounding; depending on vectorization and scalar type, the residuals reach + // 1.8e-11. + const double atol = 3e-11; + check_generalized_dare<MatrixType>(A.cast<Scalar>(), B.cast<Scalar>(), Q.cast<Scalar>(), R.cast<Scalar>(), + E.cast<Scalar>(), MatrixType::Zero(3, 2), atol); + check_generalized_dare<MatrixType>(A.cast<Scalar>(), B.cast<Scalar>(), Q.cast<Scalar>(), R.cast<Scalar>(), + E.cast<Scalar>(), MatrixType::Ones(3, 2), atol); + if (NumTraits<Scalar>::IsComplex) return; + + // Reported under SciPy PR 6616: E omitted, S given. + MatrixXd A2(8, 8), B2(8, 2), Q2(8, 8), R2(2, 2), S2(8, 2); + A2 << 0.9790596124208226, 0.15428665211630566, -0.0399419325090604, 0.003911636788878224, -0.011803107183894452, + -0.02494092597437882, -0.008920788197667088, 0.014335952211974737, // + -0.14834721727344974, 0.9873863186572593, 0.006106682107487801, -0.0012366115496408269, 0.013575311081540167, + 0.020672662811912077, -0.0017998184559520581, 0.0017523155301884849, // + 0.07162947799991767, -0.010401328690850057, 0.9845726088881807, 0.04377286183324468, 0.03170819532733946, + 0.049510008354081356, -0.01203976564345253, 0.01388147319653372, // + -0.0019726741801273217, -0.024879350443434263, -0.12719519221321976, 0.9804305159866238, 0.021632354678924177, + 0.07905312980830177, -0.011451886896987448, 0.026787582574920674, // + -0.051702322194735015, 0.00969666725317041, 0.02104971125084225, 0.10007182592961279, 0.7141026413334377, + -0.6004593068202806, 0.004777582730886375, 0.03141438420954447, // + 0.040354013728618784, -0.015350748134960568, -0.05579631944984332, -0.10543722475599439, 0.6343856366274718, + 0.35004282911065215, -0.03776729806891735, 0.0074099601797860785, // + -0.03584359294207154, 0.0025900981016803853, -0.005668147016789041, -0.06465860121122835, -0.012656894372629887, + 0.04360315609168892, 0.7043110256864276, 0.6066975504180661, // + -0.02851398087932866, -0.0015631274113090954, -0.012577246358475958, -0.053341773444025395, -0.03769226823991803, + -0.007888383180591867, -0.6280157636492598, 0.34907000148948886; + B2 << -1.0436871916355555, 1.0931665872601304, 0.5133620925013083, -0.1514390795683095, 0.7861828230513443, + 0.3888125024790475, 0.9089599716479574, 0.7320641966547645, -0.6781510875985723, 0.12416864282640695, + 0.9048515879878576, -0.009000096506101762, -0.008637666934928385, 0.5564531625467788, 0.007083540587022204, + 0.6553914999236834; + Q2 << 1.7551250678842602e-06, -4.272252214687373e-07, -9.347904861120448e-07, -6.106412578819786e-07, + 1.6283957160863591e-06, 2.226201360118831e-06, 1.0890695975480443e-06, -1.3619304943943357e-06, // + -4.2722522146873673e-07, 5.11729442524146e-07, 1.0331612636389516e-06, 9.016609232641012e-07, + -9.306700600646157e-07, -1.4657838525589338e-06, 3.4201260458425485e-07, -4.2057289495481993e-07, // + -9.347904861120447e-07, 1.0331612636389514e-06, 2.0896827294140204e-06, 1.8133390743261377e-06, + -1.9225671817044196e-06, -3.0098180593831633e-06, 6.189175278904355e-07, -7.587103608495501e-07, // + -6.106412578819799e-07, 9.016609232641022e-07, 1.8133390743261417e-06, 1.6055691910525155e-06, + -1.5469792580036601e-06, -2.460263283152522e-06, 7.394613770930514e-07, -9.082211730936819e-07, // + 1.6283957160863587e-06, -9.30670060064614e-07, -1.9225671817044158e-06, -1.5469792580036527e-06, + 2.2352617341558302e-06, 3.3544676997154134e-06, 2.14575519310723e-07, -2.9072025953141554e-07, // + 2.226201360118829e-06, -1.465783852558932e-06, -3.0098180593831616e-06, -2.460263283152514e-06, + 3.3544676997154143e-06, 5.1697298143974225e-06, 4.095247131836552e-09, -6.178901942620202e-08, // + 1.089069597548046e-06, 3.4201260458425506e-07, 6.189175278904347e-07, 7.394613770930501e-07, + 2.145755193107224e-07, 4.095247131839212e-09, 1.58987128162285e-06, -2.001906554441249e-06, // + -1.361930494394337e-06, -4.2057289495481967e-07, -7.587103608495486e-07, -9.0822117309368e-07, + -2.9072025953141474e-07, -6.178901942620546e-08, -2.001906554441248e-06, 2.587024318215873e-06; + R2 << 6.86199839251629e-07, -1.4227980596335797e-08, -1.4227980596333721e-08, 1.811308664394915e-07; + S2 << -6.272561526745687e-07, 2.805546571188463e-07, 4.2782756682953457e-07, 8.54401685835999e-08, + 8.762219354284593e-07, 1.5285528673410845e-07, 7.043760079665012e-07, 1.8058957219255395e-07, + -1.0243476139042563e-06, 6.619287295779458e-08, -1.6830289137993863e-06, 2.9471233851484255e-08, + 2.409523332122225e-08, 4.306143497422768e-07, 1.2366259786610292e-08, -6.06921957130162e-07; + check_generalized_dare<MatrixType>(A2.cast<Scalar>(), B2.cast<Scalar>(), Q2.cast<Scalar>(), R2.cast<Scalar>(), + MatrixType::Identity(8, 8), S2.cast<Scalar>(), 3.5e-16); +} + +// All modes of A are unstable and B = 0, so no stabilizing solution exists. In a rotated basis, rounding errors leave +// U1 of the order of eps instead of zero, which the scale-invariant rcond(U1) does not detect: the solver returned +// Success with ||X|| = 2.5e20. +template <typename MatrixType> +void dare_unreachable_unstable_modes() { + using Scalar = typename MatrixType::Scalar; + const Vector3d v(1, 2, 3); + const Matrix3d P = Matrix3d::Identity() - 2 * v * v.transpose() / v.squaredNorm(); + const MatrixType A = (P * Vector3d(150, -200, 250).asDiagonal() * P).template cast<Scalar>(); + DiscreteAlgebraicRiccatiSolver<MatrixType> dare; + for (bool balance : {true, false}) + VERIFY_IS_EQUAL( + dare.compute(A, MatrixType::Zero(3, 1), MatrixType::Identity(3, 3), MatrixType::Identity(1, 1), balance).info(), + NumericalIssue); +} + +#if defined(EIGEN_TEST_PART_1) || defined(EIGEN_TEST_PART_ALL) +// Regression test: with E != I, the stable subspace of the balanced pencil must be combined with the balanced E. +// Combining it with the original E, as SciPy does, returns a solution with residual 3.9 for this problem. +void dare_generalized_balanced() { + MatrixXd A(2, 2), B(2, 1), Q(2, 2), R(1, 1), E(2, 2), S(2, 1); + A << 0.9, 0, 0, 2; + B << 1e-4, 1; + Q << 1, 0, 0, 1e4; + R << 1; + E << 1, 0.5, 0, 1; + S << 0, 0; + DiscreteAlgebraicRiccatiSolver<MatrixXd> dare(A, B, Q, R, E, S); + VERIFY_IS_EQUAL(dare.info(), Success); + verify_dare_backward_error<MatrixXd>(A, B, Q, R, E, S, dare.solution()); + verify_stabilizing<MatrixXd>(A, B, R, E, S, dare.solution()); +} +#endif + +#if defined(EIGEN_TEST_PART_4) || defined(EIGEN_TEST_PART_ALL) +// Regression test: ComplexQZ reports NumericalIssue for the deflated 2x2 pencil of this problem, whose subdiagonal +// entry its direct 2x2 reduction leaves at 2.1 eps ||S||; the decomposition is still backward stable. +void dare_complex_qz_2x2_block() { + using cf = std::complex<float>; + MatrixXcf A(1, 1), B(1, 1), Q(1, 1), R(1, 1), E(1, 1), S(1, 1); + A << cf(1.34278035f, -1.30643785f); + B << cf(-0.928204715f, 1.51564276f); + Q << cf(2.26285052f, 0); + R << cf(1.4296701f, 0); + E << cf(0.936707854f, 0.0122698545f); + S << cf(-0.0921620131f, -0.450842977f); + DiscreteAlgebraicRiccatiSolver<MatrixXcf> dare(A, B, Q, R, E, S); + VERIFY_IS_EQUAL(dare.info(), Success); + verify_dare_backward_error<MatrixXcf>(A, B, Q, R, E, S, dare.solution()); + verify_stabilizing<MatrixXcf>(A, B, R, E, S, dare.solution()); +} +#endif + +#if defined(EIGEN_TEST_PART_1) || defined(EIGEN_TEST_PART_ALL) +void dare_invalid_input() { + const MatrixXd Id = MatrixXd::Identity(3, 3); + DiscreteAlgebraicRiccatiSolver<MatrixXd> dare; + + // Q or R not symmetric. + MatrixXd nonsymmetric(3, 3); + nonsymmetric << 0, 1, 2, 3, 4, 5, 6, 7, 8; + VERIFY_IS_EQUAL(dare.compute(Id, Id, nonsymmetric, Id).info(), InvalidInput); + VERIFY_IS_EQUAL(dare.compute(Id, Id, Id, nonsymmetric).info(), InvalidInput); + + // Singular E. + MatrixXd singular = MatrixXd::Constant(3, 3, 1e12); + singular(2, 2) -= 1; + VERIFY_IS_EQUAL(dare.compute(Id, Id, Id, Id, singular, MatrixXd::Zero(3, 3)).info(), InvalidInput); + + // Non-finite input. + const MatrixXd nan = MatrixXd::Constant(3, 3, std::numeric_limits<double>::quiet_NaN()); + VERIFY_IS_EQUAL(dare.compute(nan, Id, Id, Id).info(), InvalidInput); + VERIFY_IS_EQUAL(dare.compute(Id, nan, Id, Id).info(), InvalidInput); + VERIFY_IS_EQUAL(dare.compute(Id, Id, nan, Id).info(), InvalidInput); + VERIFY_IS_EQUAL(dare.compute(Id, Id, Id, nan).info(), InvalidInput); + VERIFY_IS_EQUAL(dare.compute(Id, Id, Id, Id, nan, Id).info(), InvalidInput); + VERIFY_IS_EQUAL(dare.compute(Id, Id, Id, Id, Id, nan).info(), InvalidInput); + + // Incompatible sizes. + const MatrixXd rect = MatrixXd::Ones(3, 2); + VERIFY_RAISES_ASSERT(dare.compute(rect, Id, Id, Id)); + VERIFY_RAISES_ASSERT(dare.compute(Id, Id, rect, Id)); + VERIFY_RAISES_ASSERT(dare.compute(Id, Id, Id, rect)); + VERIFY_RAISES_ASSERT(dare.compute(Id, Id, Id, Id, rect, Id)); + VERIFY_RAISES_ASSERT(dare.compute(Id, Id, Id, Id, Id, MatrixXd::Ones(3, 2))); + + // The solver recovers from a failed computation. + VERIFY_IS_EQUAL(dare.compute(Id * 0.5, Id, Id, Id).info(), Success); +} + +void dare_no_stabilizing_solution() { + // The unstable mode 2 is not controllable, so no stabilizing solution exists. + MatrixXd A(2, 2), B(2, 1); + A << 2, 0, 0, 0.5; + B << 0, 1; + DiscreteAlgebraicRiccatiSolver<MatrixXd> dare; + for (bool balance : {true, false}) + VERIFY_IS_EQUAL(dare.compute(A, B, MatrixXd::Identity(2, 2), MatrixXd::Identity(1, 1), balance).info(), + NumericalIssue); + + // A Jordan block at 1 with B = 0 and Q = 0 gives a singular pencil, whose reordering rejects a swap. + A << 1, 1, 0, 1; + B.setZero(); + for (bool balance : {true, false}) + VERIFY_IS_EQUAL(dare.compute(A, B, MatrixXd::Zero(2, 2), MatrixXd::Identity(1, 1), balance).info(), NumericalIssue); +} + +void dare_empty() { + // m = 0: the solution is empty. + DiscreteAlgebraicRiccatiSolver<MatrixXd> dare(MatrixXd(0, 0), MatrixXd(0, 2), MatrixXd(0, 0), + MatrixXd::Identity(2, 2)); + VERIFY_IS_EQUAL(dare.info(), Success); + VERIFY_IS_EQUAL(dare.solution().rows(), 0); + dare.compute(MatrixXd(0, 0), MatrixXd(0, 2), MatrixXd(0, 0), MatrixXd::Identity(2, 2), MatrixXd(0, 0), + MatrixXd(0, 2)); + VERIFY_IS_EQUAL(dare.info(), Success); + + // n = 0: for stable A the equation reduces to the Stein equation A^* X A - X + Q = 0. + const Index m = 4; + MatrixXd A = MatrixXd::Random(m, m); + A *= 0.5 / A.operatorNorm(); + const MatrixXd Q = MatrixXd::Identity(m, m); + dare.compute(A, MatrixXd(m, 0), Q, MatrixXd(0, 0)); + VERIFY_IS_EQUAL(dare.info(), Success); + const MatrixXd& X = dare.solution(); + VERIFY((A.adjoint() * X * A - X + Q).norm() <= 100 * NumTraits<double>::epsilon() * X.norm()); +} + +#endif + +// Random problems with a unique stabilizing solution whose conditioning stays moderate: A = U T U^* with U unitary and +// T upper triangular with eigenvalue moduli in [0.1, 0.8], except for at most n well controllable ones in [1.25, 2]; Q +// and R are positive definite, and ||S||_F <= 1/2 keeps Q - S R^-1 S^* >= 3/4 I in the generalized problems. The sizes +// are m in [1, maxM] (m = maxM for fixed-size MatrixType) and n in [1, min(m, maxN)]. +template <typename MatrixType> +void dare_random(Index maxM, Index maxN) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename MatrixType::RealScalar; + using DynMatrix = Matrix<Scalar, Dynamic, Dynamic>; + const Index m = MatrixType::RowsAtCompileTime == Dynamic ? internal::random<Index>(1, maxM) : maxM; + const Index n = internal::random<Index>(1, numext::mini(m, maxN)); + + DynMatrix T = DynMatrix::Random(m, m).template triangularView<StrictlyUpper>(); + T *= RealScalar(0.5) / numext::sqrt(RealScalar(m)); + const Index unstable = internal::random<Index>(0, n); + for (Index i = 0; i < m; ++i) { + Scalar phase = internal::random<Scalar>(); + while (numext::is_exactly_zero(phase)) phase = internal::random<Scalar>(); + const RealScalar modulus = + i < unstable ? internal::random<RealScalar>(1.25, 2) : internal::random<RealScalar>(0.1, 0.8); + T(i, i) = phase / numext::abs(phase) * modulus; + } + const DynMatrix U = HouseholderQR<DynMatrix>(DynMatrix::Random(m, m)).householderQ(); + const MatrixType A = U * T * U.adjoint(); + // B = U Bu with Bu(i, i) = 2 + O(1) for the unstable modes i < unstable <= n keeps them well controllable. A third + // of the problems scale B by 10^-k, for which balancing alone would cost digits. + DynMatrix Bu = DynMatrix::Random(m, n); + for (Index i = 0; i < unstable; ++i) Bu(i, i) += Scalar(2); + const int k = internal::random<int>(0, 2) == 0 ? internal::random<int>(1, NumTraits<RealScalar>::digits10() / 3) : 0; + const DynMatrix B = U * Bu * numext::pow(RealScalar(10), RealScalar(-k)); + const DynMatrix C = DynMatrix::Random(m, m); + const DynMatrix D = DynMatrix::Random(n, n); + const MatrixType Q = C.adjoint() * C + DynMatrix::Identity(m, m); + const DynMatrix R = D.adjoint() * D + DynMatrix::Identity(n, n); + const DynMatrix Id = DynMatrix::Identity(m, m), Z = DynMatrix::Zero(m, n); + + DiscreteAlgebraicRiccatiSolver<MatrixType> dare(A, B, Q, R); + VERIFY_IS_EQUAL(dare.info(), Success); + const MatrixType& X = dare.solution(); + VERIFY_IS_EQUAL(X, X.adjoint()); + verify_dare_backward_error<DynMatrix>(A, B, Q, R, Id, Z, X); + verify_stabilizing<DynMatrix>(A, B, R, Id, Z, X); + // X >= Q > 0, since X = Q + A^* (X - X B (R + B^* X B)^-1 B^* X) A. + VERIFY(SelfAdjointEigenSolver<DynMatrix>(X).eigenvalues().minCoeff() > RealScalar(0)); + + DiscreteAlgebraicRiccatiSolver<MatrixType> unbalanced(A, B, Q, R, /*balance=*/false); + VERIFY_IS_EQUAL(unbalanced.info(), Success); + verify_dare_backward_error<DynMatrix>(A, B, Q, R, Id, Z, unbalanced.solution()); + + // The generalized equation with E = I and S = 0 is the standard one. + DiscreteAlgebraicRiccatiSolver<MatrixType> generalized; + generalized.compute(A, B, Q, R, MatrixType::Identity(m, m), Z); + VERIFY_IS_EQUAL(generalized.info(), Success); + verify_dare_backward_error<DynMatrix>(A, B, Q, R, Id, Z, generalized.solution()); + + // The generalized problem (E A, E B, Q, R, E, S) has the closed-loop dynamics of (A, B), and E^* X E is the solution + // of the standard problem with the cross term S. + const MatrixType E = + MatrixType::Identity(m, m) + MatrixType::Random(m, m) * (RealScalar(0.5) / numext::sqrt(RealScalar(m))); + const DynMatrix S = DynMatrix::Random(m, n) * (RealScalar(0.5) / numext::sqrt(RealScalar(m * n))); + const DynMatrix EA = E * A, EB = E * B; + for (bool balance : {true, false}) { + generalized.compute(EA, EB, Q, R, E, S, balance); + VERIFY_IS_EQUAL(generalized.info(), Success); + const DynMatrix Xg = generalized.solution(); + VERIFY_IS_EQUAL(Xg, Xg.adjoint()); + verify_dare_backward_error<DynMatrix>(EA, EB, Q, R, E, S, Xg); + verify_stabilizing<DynMatrix>(EA, EB, R, E, S, Xg); + } +} + +// Checks the reordering of a generalized Schur form directly. +template <typename MatrixType> +void generalized_schur_reordering(Index size) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename MatrixType::RealScalar; + const MatrixType A = MatrixType::Random(size, size) * RealScalar(1.5); + const MatrixType B = MatrixType::Random(size, size); + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + typename internal::riccati_qz<MatrixType>::type qz(A, B, true); + VERIFY_IS_EQUAL(qz.info(), Success); + MatrixType S = qz.matrixS(), T = qz.matrixT(), Q = qz.matrixQ(), Z = qz.matrixZ().adjoint(); + + Index inside = 0; + for (Index k = 0; k < size; ++k) { + // Count eigenvalues inside the unit circle through the 1x1 and 2x2 diagonal blocks. + const Index bs = internal::generalized_schur_block_size(S, k); + if (bs == 1) { + if (numext::abs(S(k, k)) < numext::abs(T(k, k))) ++inside; + } else { + if (numext::abs(S.template block<2, 2>(k, k).determinant()) < numext::abs(T(k, k) * T(k + 1, k + 1))) inside += 2; + ++k; + } + } + + const Index selected = internal::reorder_generalized_schur_inside_unit_circle(S, T, &Q, Z); + VERIFY_IS_EQUAL(selected, inside); + // Each swap is accepted only if it is backward stable to 20 eps ||(S, T)||_F locally, see the strong stability test. + const RealScalar bound = RealScalar(20) * RealScalar(size) * eps; + VERIFY((Q * S * Z.adjoint() - A).norm() <= bound * A.norm()); + VERIFY((Q * T * Z.adjoint() - B).norm() <= bound * B.norm()); + VERIFY((Q.adjoint() * Q - MatrixType::Identity(size, size)).norm() <= bound); + VERIFY((Z.adjoint() * Z - MatrixType::Identity(size, size)).norm() <= bound); + VERIFY_IS_EQUAL(T.template triangularView<StrictlyLower>().toDenseMatrix(), MatrixType::Zero(size, size)); + for (Index i = 0; i + 2 < size; ++i) VERIFY_IS_EQUAL(S(i + 2, i), Scalar(0)); + // The selected eigenvalues come first, and the boundary does not split a 2x2 block. + if (selected > 0 && selected < size) VERIFY_IS_EQUAL(S(selected, selected - 1), Scalar(0)); + for (Index k = 0; k < size;) { + const Index bs = internal::generalized_schur_block_size(S, k); + RealScalar modulus; + if (bs == 1) + modulus = numext::abs(S(k, k)) / numext::abs(T(k, k)); + else + modulus = numext::sqrt(numext::abs(S.template block<2, 2>(k, k).determinant()) / + numext::abs(T(k, k) * T(k + 1, k + 1))); + VERIFY((k < selected) == (modulus < RealScalar(1))); + k += bs; + } +} + +// Generalized eigenvalues of the diagonal block of size bs at (k, k) of (S, T), sorted. +template <typename MatrixType> +std::vector<std::complex<typename MatrixType::RealScalar>> block_eigenvalues(const MatrixType& S, const MatrixType& T, + Index k, Index bs) { + using Complex = std::complex<typename MatrixType::RealScalar>; + using ComplexMatrix = Matrix<Complex, Dynamic, Dynamic>; + const ComplexMatrix s = S.block(k, k, bs, bs).template cast<Complex>(); + const ComplexMatrix t = T.block(k, k, bs, bs).template cast<Complex>(); + const Matrix<Complex, Dynamic, 1> ev = ComplexEigenSolver<ComplexMatrix>(t.inverse() * s).eigenvalues(); + std::vector<Complex> v(ev.data(), ev.data() + ev.size()); + std::sort(v.begin(), v.end(), [](const Complex& a, const Complex& b) { + return a.imag() < b.imag() || (a.imag() == b.imag() && a.real() < b.real()); + }); + return v; +} + +// Swaps the diagonal blocks of sizes n1 and n2 at row/column j = 1 of a generalized Schur form of size n1 + n2 + 2. +template <typename MatrixType> +void generalized_schur_swap(Index n1, Index n2) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename MatrixType::RealScalar; + const Index j = 1, size = n1 + n2 + 2; + MatrixType S = MatrixType::Random(size, size).template triangularView<Upper>(); + MatrixType T = MatrixType::Random(size, size).template triangularView<Upper>(); + T.diagonal().array() += Scalar(2); + // A 2x2 block (s, t) with t = I and s = [a, 1; -1, a] has the eigenvalues a +- i. + const Index starts[] = {j, j + n1}; + const Index sizes[] = {n1, n2}; + for (int b = 0; b < 2; ++b) { + if (sizes[b] != 2) continue; + const Index k = starts[b]; + const Scalar a = Scalar(b == 0 ? 0.5 : -0.25); + S.template block<2, 2>(k, k) << a, Scalar(1), Scalar(-1), a; + T.template block<2, 2>(k, k).setIdentity(); + } + const MatrixType S0 = S, T0 = T; + MatrixType Q = MatrixType::Identity(size, size), Z = MatrixType::Identity(size, size); + const auto upper = block_eigenvalues(S0, T0, j, n1); + const auto lower = block_eigenvalues(S0, T0, j + n1, n2); + + VERIFY(internal::swap_generalized_schur_blocks(S, T, &Q, Z, j, n1, n2)); + const auto newUpper = block_eigenvalues(S, T, j, n2); + const auto newLower = block_eigenvalues(S, T, j + n2, n1); + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + for (Index i = 0; i < n2; ++i) + VERIFY(std::abs(newUpper[i] - lower[i]) <= RealScalar(100) * eps * (RealScalar(1) + std::abs(lower[i]))); + for (Index i = 0; i < n1; ++i) + VERIFY(std::abs(newLower[i] - upper[i]) <= RealScalar(100) * eps * (RealScalar(1) + std::abs(upper[i]))); + // Exact structure: zero block boundary, T upper triangular, S zero below its 2x2 diagonal blocks. + VERIFY_IS_EQUAL(S.block(j + n2, j, n1, n2), MatrixType::Zero(n1, n2)); + VERIFY_IS_EQUAL(T.template triangularView<StrictlyLower>().toDenseMatrix(), MatrixType::Zero(size, size)); + for (Index c = 0; c < size; ++c) + for (Index r = c + 1; r < size; ++r) + if (!((r == j + 1 && c == j && n2 == 2) || (r == j + n2 + 1 && c == j + n2 && n1 == 2))) + VERIFY_IS_EQUAL(S(r, c), Scalar(0)); + // Backward error and orthogonality. + const RealScalar bound = RealScalar(20) * RealScalar(size) * eps; + VERIFY((Q * S * Z.adjoint() - S0).norm() <= bound * S0.norm()); + VERIFY((Q * T * Z.adjoint() - T0).norm() <= bound * T0.norm()); + VERIFY((Q.adjoint() * Q - MatrixType::Identity(size, size)).norm() <= bound); + VERIFY((Z.adjoint() * Z - MatrixType::Identity(size, size)).norm() <= bound); +} + +#if defined(EIGEN_TEST_PART_6) || defined(EIGEN_TEST_PART_9) || defined(EIGEN_TEST_PART_10) || \ + defined(EIGEN_TEST_PART_ALL) +// A rejected swap leaves all matrices untouched, and the reordering then reports failure. +void generalized_schur_rejected_swap() { + // Equal 2x2 blocks make the generalized Sylvester equation singular. + MatrixXd S(4, 4), T = MatrixXd::Identity(4, 4); + S << 0.5, 1, 0.3, 0.2, // + -1, 0.5, 0.1, 0.4, // + 0, 0, 0.5, 1, // + 0, 0, -1, 0.5; + MatrixXd Q = MatrixXd::Identity(4, 4), Z = MatrixXd::Identity(4, 4); + const MatrixXd S0 = S, T0 = T, Q0 = Q, Z0 = Z; + VERIFY(!internal::swap_generalized_schur_blocks(S, T, &Q, Z, 0, 2, 2)); + VERIFY_IS_EQUAL(S, S0); + VERIFY_IS_EQUAL(T, T0); + VERIFY_IS_EQUAL(Q, Q0); + VERIFY_IS_EQUAL(Z, Z0); + + // The 1x1 blocks 0/0 and 1/2 of a singular pencil cannot be swapped, so the eigenvalue 1/2 cannot be selected. + MatrixXd S2(2, 2), T2(2, 2); + S2 << 0, 0, 0, 0.5; + T2 << 0, 0, 0, 1; + MatrixXd Q2 = MatrixXd::Identity(2, 2), Z2 = MatrixXd::Identity(2, 2); + const MatrixXd S20 = S2, T20 = T2; + VERIFY_IS_EQUAL(internal::reorder_generalized_schur_inside_unit_circle(S2, T2, &Q2, Z2), Index(-1)); + VERIFY_IS_EQUAL(S2, S20); + VERIFY_IS_EQUAL(T2, T20); +} +#endif + +EIGEN_DECLARE_TEST(riccati) { + CALL_SUBTEST_1(dare_scipy_real_cases<MatrixXd>()); + CALL_SUBTEST_1(dare_generalized_scipy_cases<MatrixXd>()); + CALL_SUBTEST_1(dare_generalized_balanced()); + CALL_SUBTEST_1(dare_invalid_input()); + CALL_SUBTEST_1(dare_no_stabilizing_solution()); + CALL_SUBTEST_1(dare_unreachable_unstable_modes<MatrixXd>()); + CALL_SUBTEST_7(dare_unreachable_unstable_modes<MatrixXcd>()); + CALL_SUBTEST_1(dare_empty()); + CALL_SUBTEST_6(generalized_schur_rejected_swap()); + for (Index n1 = 1; n1 <= 2; ++n1) + for (Index n2 = 1; n2 <= 2; ++n2) CALL_SUBTEST_6(generalized_schur_swap<MatrixXd>(n1, n2)); + CALL_SUBTEST_9(generalized_schur_swap<MatrixXcd>(1, 1)); + CALL_SUBTEST_4(dare_complex_qz_2x2_block()); + CALL_SUBTEST_7(dare_scipy_complex_cases()); + CALL_SUBTEST_7(dare_scipy_real_cases<MatrixXcd>()); + CALL_SUBTEST_7(dare_generalized_scipy_cases<MatrixXcd>()); + for (int i = 0; i < g_repeat; i++) { + CALL_SUBTEST_2(dare_random<MatrixXd>(20, 6)); + CALL_SUBTEST_3(dare_random<MatrixXcd>(20, 6)); + CALL_SUBTEST_11(dare_random<MatrixXf>(8, 4)); + CALL_SUBTEST_8(dare_random<MatrixXcf>(8, 4)); + CALL_SUBTEST_5((dare_random<Matrix<double, 4, 4>>(4, 4))); + CALL_SUBTEST_6(generalized_schur_reordering<MatrixXd>(internal::random<Index>(1, 30))); + CALL_SUBTEST_9(generalized_schur_reordering<MatrixXcd>(internal::random<Index>(1, 30))); + CALL_SUBTEST_10(generalized_schur_reordering<MatrixXf>(internal::random<Index>(1, 30))); + } +}
diff --git a/contrib/test/ridders.cpp b/contrib/test/ridders.cpp new file mode 100644 index 0000000..f8c5978 --- /dev/null +++ b/contrib/test/ridders.cpp
@@ -0,0 +1,556 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include "main.h" +#include <contrib/Eigen/NumericalDiff> +#include <contrib/Eigen/LevenbergMarquardt> + +template <typename Scalar_, int Inputs = Dynamic, int Values = Dynamic, int Options = ColMajor> +struct RiddersFunctor { + using Scalar = Scalar_; + using InputType = Matrix<Scalar, Inputs, 1>; + using ValueType = Matrix<Scalar, Values, 1>; + using JacobianType = Matrix<Scalar, Values, Inputs, Options>; + + static constexpr int InputsAtCompileTime = Inputs; + static constexpr int ValuesAtCompileTime = Values; + + mutable int evaluations = 0; + + int values() const { return 4; } + + int operator()(const InputType& x, ValueType& y) const { + ++evaluations; + y[0] = numext::exp(x[0]); + y[1] = numext::sin(Scalar(50) * x[0]); + y[2] = x[0] * x[1]; + y[3] = numext::exp(x[1]); + return 0; + } +}; + +template <typename Scalar, int Inputs, int Values, int Options> +void test_ridders_jacobian() { + using F = RiddersFunctor<Scalar, Inputs, Values, Options>; + + Ridders<F> diff; + typename F::InputType x(2), error(2); + typename F::JacobianType jac(4, 2), expected(4, 2), withoutError(4, 2); + + x << Scalar(1), Scalar(-2); + const typename F::InputType original = x; + + expected.row(0) << Scalar(std::exp(1.0L)), Scalar(0); + expected.row(1) << Scalar(50 * std::cos(50.0L)), Scalar(0); + expected.row(2) << x[1], x[0]; + expected.row(3) << Scalar(0), Scalar(std::exp(-2.0L)); + + const int count = diff.df(x, jac, error); + VERIFY_IS_EQUAL(count, diff.functor().evaluations); + VERIFY(count >= 8 && count <= 40 && count % 2 == 0); + + VERIFY(jac.allFinite()); + VERIFY(error.allFinite()); + VERIFY((error.array() >= Scalar(0)).all()); + + // The initial stencil spans half a radian in the oscillatory component; extrapolation + // must remove its truncation error to reach a roundoff-scale bound. + VERIFY((jac - expected).template lpNorm<Infinity>() <= + Scalar(256) * NumTraits<Scalar>::epsilon() * expected.template lpNorm<Infinity>()); + + VERIFY_IS_EQUAL(x, original); + VERIFY_IS_EQUAL(diff.df(x, withoutError), count); + VERIFY_IS_EQUAL(jac, withoutError); +} + +struct RiddersScalarFunctor { + using Scalar = double; + using InputType = Matrix<double, 1, 1>; + using ValueType = InputType; + using JacobianType = InputType; + + static constexpr int InputsAtCompileTime = 1; + static constexpr int ValuesAtCompileTime = 1; + + enum Function { Cubic, LargeCubic, Quintic, SquareRoot, NoisyExp, Identity, Invalid, InvalidRefinement }; + Function function = Cubic; + mutable int evaluations = 0; + + int values() const { return 1; } + + int operator()(const InputType& x, ValueType& y) const { + ++evaluations; + switch (function) { + case Cubic: + y[0] = x[0] * x[0] * x[0]; + break; + + case LargeCubic: + y[0] = 2e307 * x[0] * x[0] * x[0]; + break; + + case Quintic: + y[0] = x[0] * x[0] * x[0] * x[0] * x[0]; + break; + + case SquareRoot: + y[0] = std::sqrt(x[0]); + break; + + case NoisyExp: + y[0] = double(float(std::exp(x[0]))); + break; + + case Identity: + y[0] = x[0]; + break; + + case Invalid: + y[0] = NumTraits<double>::quiet_NaN(); + break; + + case InvalidRefinement: + y[0] = evaluations <= 2 ? x[0] : NumTraits<double>::quiet_NaN(); + break; + } + return 0; + } +}; + +void test_ridders_parameters() { + using DiffType = Ridders<RiddersScalarFunctor>; + + DiffType diff; + DiffType::Parameters parameters; + parameters.initialStep = 0.25; + parameters.stepReduction = 4; + parameters.maxSteps = 2; + diff.setParameters(parameters); + + RiddersScalarFunctor::InputType x, error; + RiddersScalarFunctor::JacobianType jac; + x[0] = 2; + VERIFY_IS_EQUAL(diff.df(x, jac, error), 4); + VERIFY_IS_EQUAL(jac[0], 12.0); + + // The cubic's central differences are 12 + h^2, with h = 1/2 and 1/8. + VERIFY_IS_EQUAL(error[0], 0.25); + + parameters.maxSteps = 10; + diff.setParameters(parameters); + VERIFY(diff.df(x, jac, error) < 20); + VERIFY_IS_EQUAL(jac[0], 12.0); + VERIFY_IS_EQUAL(error[0], 0.0); + + parameters.initialStep = 0; + VERIFY_RAISES_ASSERT(diff.setParameters(parameters)); + + parameters.initialStep = 0.01; + parameters.stepReduction = 1; + VERIFY_RAISES_ASSERT(diff.setParameters(parameters)); + + parameters.stepReduction = 2; + parameters.maxSteps = 1; + VERIFY_RAISES_ASSERT(diff.setParameters(parameters)); + + parameters.maxSteps = 10; + parameters.initialStep = NumTraits<double>::infinity(); + VERIFY_RAISES_ASSERT(diff.setParameters(parameters)); + + parameters.initialStep = 0.01; + parameters.stepReduction = NumTraits<double>::infinity(); + VERIFY_RAISES_ASSERT(diff.setParameters(parameters)); + + parameters = DiffType::Parameters(); + if (NumTraits<Index>::highest() / 2 < NumTraits<int>::highest()) { + parameters.maxSteps = static_cast<int>(NumTraits<Index>::highest() / 2) + 1; + VERIFY_RAISES_ASSERT(diff.setParameters(parameters)); + --parameters.maxSteps; + } else { + parameters.maxSteps = NumTraits<int>::highest(); + } + diff.setParameters(parameters); +} + +void test_ridders_repeated_stencil() { + using DiffType = Ridders<RiddersScalarFunctor>; + + for (int multiplier : {1, 16}) { + DiffType diff; + DiffType::Parameters parameters; + parameters.stepReduction = 1 + multiplier * NumTraits<double>::epsilon(); + diff.setParameters(parameters); + + RiddersScalarFunctor::InputType x, error; + RiddersScalarFunctor::JacobianType jac; + x[0] = 1; + VERIFY_IS_EQUAL(diff.df(x, jac, error), 2); + VERIFY_IS_EQUAL(diff.functor().evaluations, 2); + VERIFY((numext::isfinite)(jac[0])); + VERIFY(std::abs(jac[0] - 3.0001) <= 32 * NumTraits<double>::epsilon()); + VERIFY((numext::isinf)(error[0]) && error[0] > 0); + } +} + +void test_ridders_dimensions() { + Ridders<RiddersFunctor<double>> diff; + VectorXd x = VectorXd::Ones(2), error; + MatrixXd jac(3, 2); + VERIFY_RAISES_ASSERT(diff.df(x, jac)); + + jac.resize(4, 1); + VERIFY_RAISES_ASSERT(diff.df(x, jac, error)); + + jac.resize(4, 0); + VERIFY_RAISES_ASSERT(diff.df(x, jac)); + VERIFY_IS_EQUAL(diff.functor().evaluations, 0); +} + +template <typename Scalar_> +struct RiddersAffineFunctor { + using Scalar = Scalar_; + using InputType = Matrix<Scalar, 1, 1>; + using ValueType = Matrix<Scalar, Dynamic, 1>; + using JacobianType = Matrix<Scalar, Dynamic, 1>; + + static constexpr int InputsAtCompileTime = 1; + static constexpr int ValuesAtCompileTime = Dynamic; + + ValueType slopes, offsets; + Scalar center = Scalar(100); + Scalar inputScale = Scalar(1); + mutable int evaluations = 0; + int finiteEvaluations = 20; + + Index values() const { return slopes.size(); } + + int operator()(const InputType& x, ValueType& y) const { + ++evaluations; + y = slopes * ((x[0] - center) * inputScale) + offsets; + if (evaluations > finiteEvaluations) y.setConstant(NumTraits<Scalar>::quiet_NaN()); + return 0; + } +}; + +template <typename Scalar> +void test_ridders_central_overflow() { + using Functor = RiddersAffineFunctor<Scalar>; + using DiffType = Ridders<Functor>; + + const Scalar large = NumTraits<Scalar>::highest() * Scalar(0.75); + // 32-bit ARM NEON flushes subnormal lanes; there, keep tiny and small * h for the steps h = 1, 1/2 normal. + constexpr bool kPacketsFlush = EIGEN_ARCH_ARM && internal::packet_traits<Scalar>::Vectorizable; + const Scalar small = Scalar(kPacketsFlush ? 2 : 1) * (std::numeric_limits<Scalar>::min)(); + const Scalar tiny = kPacketsFlush ? small : (std::numeric_limits<Scalar>::denorm_min)(); + + for (Index count : {1, 7, 17}) { + Functor functor; + functor.slopes.resize(count); + functor.offsets.setZero(count); + + for (Index i = 0; i < count; ++i) { + functor.slopes[i] = i % 3 == 0 ? large : (i % 3 == 1 ? Scalar(-8) : small); + if (i % 6 == 3) functor.slopes[i] = -large; + } + + DiffType diff(functor); + typename Functor::InputType x, error; + typename Functor::JacobianType jac(count, 1); + x[0] = functor.center; + VERIFY_IS_EQUAL(diff.df(x, jac, error), 4); + VERIFY(jac.allFinite()); + for (Index i = 0; i < count; ++i) VERIFY_IS_EQUAL(jac[i], functor.slopes[i]); + VERIFY_IS_EQUAL(error[0], Scalar(0)); + + // A failed refinement must retain finite entries, including subnormals unless packets flush them, from the + // initial stencil. + diff.functor().evaluations = 0; + diff.functor().finiteEvaluations = 2; + for (Index i = 2; i < count; i += 3) diff.functor().slopes[i] = tiny; + VERIFY_IS_EQUAL(diff.df(x, jac, error), 4); + VERIFY_IS_EQUAL(jac, diff.functor().slopes); + VERIFY((numext::isinf)(error[0]) && error[0] > Scalar(0)); + } + + Functor functor; + functor.slopes.setZero(1); + functor.offsets.setConstant(1, large); + + DiffType diff(functor); + typename Functor::InputType x, error; + typename Functor::JacobianType jac(1, 1); + x[0] = Scalar(100); + VERIFY_IS_EQUAL(diff.df(x, jac, error), 4); + VERIFY_IS_EQUAL(jac[0], Scalar(0)); + + x[0] = Scalar(0); + diff.functor().center = Scalar(0); + + typename DiffType::Parameters parameters = diff.parameters(); + parameters.initialStep = (std::numeric_limits<Scalar>::min)(); + diff.setParameters(parameters); + VERIFY_IS_EQUAL(diff.df(x, jac, error), 4); + VERIFY_IS_EQUAL(jac[0], Scalar(0)); + + // Division first cannot repair a genuinely overflowing derivative or nonfinite samples. + diff.functor().slopes[0] = NumTraits<Scalar>::highest(); + diff.functor().offsets[0] = Scalar(0); + diff.functor().inputScale = Scalar(4); + parameters.initialStep = Scalar(0.25); + diff.setParameters(parameters); + VERIFY_IS_EQUAL(diff.df(x, jac, error), 2); + VERIFY((numext::isnan)(jac[0])); + VERIFY((numext::isinf)(error[0]) && error[0] > Scalar(0)); + + diff.functor().slopes[0] = NumTraits<Scalar>::infinity(); + VERIFY_IS_EQUAL(diff.df(x, jac, error), 2); + VERIFY((numext::isnan)(jac[0])); + VERIFY((numext::isinf)(error[0]) && error[0] > Scalar(0)); + + parameters.initialStep = Scalar(2); + diff.setParameters(parameters); + VERIFY_IS_EQUAL(diff.df(x, jac, error), 2); + VERIFY((numext::isnan)(jac[0])); +} + +void test_ridders_domain_and_noise() { + using DiffType = Ridders<RiddersScalarFunctor>; + + DiffType diff; + RiddersScalarFunctor::InputType x, error; + RiddersScalarFunctor::JacobianType jac; + + diff.functor().function = RiddersScalarFunctor::SquareRoot; + x[0] = 0.005; + + DiffType::Parameters parameters; + parameters.initialStep = 0.01 * x[0]; + diff.setParameters(parameters); + VERIFY(diff.df(x, jac, error) <= 20); + + const double expected = double(0.5L / std::sqrt(static_cast<long double>(x[0]))); + VERIFY(std::abs(jac[0] - expected) <= 256 * NumTraits<double>::epsilon() * expected); + VERIFY((numext::isfinite)(error[0])); + + diff.functor().function = RiddersScalarFunctor::NoisyExp; + x[0] = 1; + parameters.initialStep = 0.01; + diff.setParameters(parameters); + VERIFY(diff.df(x, jac, error) < 20); + VERIFY(std::abs(jac[0] - std::exp(1.0)) <= 256 * double(NumTraits<float>::epsilon()) * std::exp(1.0)); + VERIFY((numext::isfinite)(error[0])); +} + +void test_ridders_richardson_overflow() { + Ridders<RiddersScalarFunctor> diff; + diff.functor().function = RiddersScalarFunctor::LargeCubic; + RiddersScalarFunctor::InputType x, error; + RiddersScalarFunctor::JacobianType jac; + x[0] = 1; + + const int evaluations = diff.df(x, jac, error); + VERIFY_IS_EQUAL(evaluations, diff.functor().evaluations); + VERIFY(evaluations > 4 && evaluations <= 20); + + VERIFY((numext::isfinite)(jac[0])); + VERIFY((numext::isfinite)(error[0])); + VERIFY(error[0] >= 0); + + // The cubic's truncation error cancels on extrapolation; divide before comparing near the range limit. + VERIFY(std::abs(jac[0] / 6e307 - 1) <= 256 * NumTraits<double>::epsilon()); +} + +void test_ridders_stencil() { + using DiffType = Ridders<RiddersScalarFunctor>; + + DiffType diff; + diff.functor().function = RiddersScalarFunctor::Quintic; + + DiffType::Parameters parameters; + parameters.initialStep = 0.25; + parameters.maxSteps = 3; + diff.setParameters(parameters); + + RiddersScalarFunctor::InputType x, error, value; + RiddersScalarFunctor::JacobianType jac; + x[0] = 1; + VERIFY_IS_EQUAL(diff.df(x, jac, error), 6); + VERIFY_IS_EQUAL(jac[0], 5.0); + + // At depth three the extrapolation is the degree-five-exact stencil on these six nodes. + const Stencil<1, double, 6> stencil({-0.25, 0.25, -0.125, 0.125, -0.0625, 0.0625}); + double expected = 0; + for (Index i = 0; i < 6; ++i) { + RiddersScalarFunctor::InputType point; + point[0] = x[0] + stencil.point(i); + diff(point, value); + expected += stencil.weight(i) * value[0]; + } + + VERIFY(std::abs(jac[0] - expected) <= 32 * NumTraits<double>::epsilon() * std::abs(jac[0])); +} + +// Tests construction from a functor without a default constructor and optional inputs() forwarding. +struct OwnedRiddersFunctor : RiddersScalarFunctor { + explicit OwnedRiddersFunctor(Function selected) { function = selected; } + + int inputs() const { return 1; } +}; + +void test_ridders_composition() { + using Derivative = Ridders<OwnedRiddersFunctor>; + + static_assert(!std::is_base_of<OwnedRiddersFunctor, Derivative>::value, "Ridders owns its functor by composition"); + + OwnedRiddersFunctor functor(RiddersScalarFunctor::Cubic); + Derivative::Parameters parameters; + parameters.initialStep = 0.25; + parameters.maxSteps = 2; + Derivative derivative(functor, parameters); + VERIFY_IS_EQUAL(derivative.inputs(), 1); + VERIFY_IS_EQUAL(derivative.values(), 1); + VERIFY_IS_EQUAL(derivative.parameters().maxSteps, 2); + + const int* inputs = &Derivative::InputsAtCompileTime; + const int* values = &Derivative::ValuesAtCompileTime; + VERIFY_IS_EQUAL(*inputs, 1); + VERIFY_IS_EQUAL(*values, 1); + + OwnedRiddersFunctor::InputType x, y, error; + OwnedRiddersFunctor::JacobianType jac; + x[0] = 2; + VERIFY_IS_EQUAL(derivative(x, y), 0); + VERIFY_IS_EQUAL(y[0], 8.0); + + VERIFY_IS_EQUAL(derivative.df(x, jac, error), 4); + VERIFY_IS_EQUAL(jac[0], 12.0); + VERIFY_IS_EQUAL(functor.evaluations, 0); + + const Derivative& constDerivative = derivative; + VERIFY_IS_EQUAL(constDerivative.functor().evaluations, 5); +} + +struct LinearRiddersFunctor : RiddersFunctor<double, 7, 7> { + int values() const { return 7; } + + int operator()(const InputType& x, ValueType& y) const { + y = x * 8; + return 0; + } +}; + +void test_ridders_step_alignment() { + Ridders<LinearRiddersFunctor> derivative; + LinearRiddersFunctor::InputType x; + x << 0.0, 0.082, 1.13, -2350.7, 7.7e8 + 0.3, -1.0 - std::ldexp(1.0, -27), -1.0; + LinearRiddersFunctor::JacobianType jac; + + derivative.df(x, jac); + VERIFY_IS_EQUAL(jac, LinearRiddersFunctor::JacobianType::Identity() * 8); +} + +void test_ridders_fallback() { + using DiffType = Ridders<RiddersScalarFunctor>; + + DiffType diff; + RiddersScalarFunctor::InputType x, error; + RiddersScalarFunctor::JacobianType jac; + x[0] = 1; + + diff.functor().function = RiddersScalarFunctor::Invalid; + VERIFY_IS_EQUAL(diff.df(x, jac, error), 2); + VERIFY((numext::isnan)(jac[0])); + VERIFY((numext::isinf)(error[0]) && error[0] > 0); + + diff.functor().function = RiddersScalarFunctor::InvalidRefinement; + diff.functor().evaluations = 0; + VERIFY_IS_EQUAL(diff.df(x, jac, error), 4); + VERIFY_IS_EQUAL(jac[0], 1.0); + VERIFY((numext::isinf)(error[0])); + + diff.functor().function = RiddersScalarFunctor::Identity; + + DiffType::Parameters parameters; + parameters.initialStep = NumTraits<double>::epsilon() / 16; + diff.setParameters(parameters); + VERIFY_IS_EQUAL(diff.df(x, jac, error), 0); + VERIFY((numext::isnan)(jac[0])); + VERIFY((numext::isinf)(error[0])); + + parameters.initialStep = NumTraits<double>::epsilon(); + diff.setParameters(parameters); + VERIFY_IS_EQUAL(diff.df(x, jac, error), 4); + VERIFY_IS_EQUAL(jac[0], 1.0); + VERIFY_IS_EQUAL(error[0], 0.0); +} + +struct RiddersLevenbergMarquardtFunctor : DenseFunctor<double> { + RiddersLevenbergMarquardtFunctor() : DenseFunctor<double>(1, 1) {} + + int operator()(const InputType& x, ValueType& y) const { + y[0] = x[0] * x[0] - 2; + return 0; + } +}; + +void test_ridders_levenberg_marquardt() { + using Functor = RiddersLevenbergMarquardtFunctor; + using Derivative = Ridders<Functor>; + + static_assert(std::is_same<typename Derivative::QRSolver, typename Functor::QRSolver>::value, + "Ridders must preserve the functor's QRSolver type"); + + Derivative derivative; + LevenbergMarquardt<Derivative> solver(derivative); + VectorXd x = VectorXd::Ones(1); + + const LevenbergMarquardtSpace::Status status = solver.minimize(x); + VERIFY(status >= LevenbergMarquardtSpace::RelativeReductionTooSmall && + status <= LevenbergMarquardtSpace::CosinusTooSmall); + VERIFY(std::abs(x[0] - std::sqrt(2.0)) <= 256 * NumTraits<double>::epsilon()); +} + +struct EmptyRiddersFunctor : RiddersFunctor<double> { + int values() const { return 0; } +}; + +void test_ridders_empty() { + Ridders<EmptyRiddersFunctor> diff; + VectorXd x = VectorXd::Ones(2), error; + MatrixXd jac(0, 2); + VERIFY_IS_EQUAL(diff.df(x, jac, error), 0); + VERIFY_IS_EQUAL(error, VectorXd::Zero(2)); + VERIFY_IS_EQUAL(diff.functor().evaluations, 0); + + Ridders<RiddersFunctor<double>> noInputs; + x.resize(0); + jac.resize(4, 0); + VERIFY_IS_EQUAL(noInputs.df(x, jac, error), 0); + VERIFY_IS_EQUAL(error.size(), 0); + VERIFY_IS_EQUAL(noInputs.functor().evaluations, 0); +} + +EIGEN_DECLARE_TEST(ridders) { + CALL_SUBTEST((test_ridders_jacobian<float, 2, 4, ColMajor>())); + CALL_SUBTEST((test_ridders_jacobian<double, 2, 4, RowMajor>())); + CALL_SUBTEST((test_ridders_jacobian<double, Dynamic, Dynamic, ColMajor>())); + CALL_SUBTEST((test_ridders_jacobian<long double, Dynamic, Dynamic, RowMajor>())); + + CALL_SUBTEST(test_ridders_parameters()); + CALL_SUBTEST(test_ridders_repeated_stencil()); + CALL_SUBTEST(test_ridders_dimensions()); + + CALL_SUBTEST(test_ridders_central_overflow<float>()); + CALL_SUBTEST(test_ridders_central_overflow<double>()); + + CALL_SUBTEST(test_ridders_domain_and_noise()); + CALL_SUBTEST(test_ridders_richardson_overflow()); + CALL_SUBTEST(test_ridders_stencil()); + + CALL_SUBTEST(test_ridders_composition()); + CALL_SUBTEST(test_ridders_step_alignment()); + CALL_SUBTEST(test_ridders_fallback()); + + CALL_SUBTEST(test_ridders_levenberg_marquardt()); + CALL_SUBTEST(test_ridders_empty()); +}
diff --git a/contrib/test/special_functions.cpp b/contrib/test/special_functions.cpp index 1cd0b29..7a9beec 100644 --- a/contrib/test/special_functions.cpp +++ b/contrib/test/special_functions.cpp
@@ -295,10 +295,17 @@ ArrayType b = (m2 * Scalar(4)).exp(); ArrayType x = m3.abs(); - // betainc(a, 1, x) == x**a. One ulp of a subnormal x**a exceeds the relative tolerance, so the smallest normal is - // added to the scale as an absolute floor. + // ARMv7 NEON flushes subnormal float intermediates, unlike the scalar betainc implementation. Form the + // reference in at least double precision so x**a and the recurrence term retain subnormal float results. + using ReferenceScalar = std::conditional_t<(sizeof(Scalar) < sizeof(double)), double, Scalar>; + using ReferenceArray = Array<ReferenceScalar, Dynamic, 1>; + const ReferenceArray& ar = a.template cast<ReferenceScalar>(); + const ReferenceArray& br = b.template cast<ReferenceScalar>(); + const ReferenceArray& xr = x.template cast<ReferenceScalar>(); const Scalar tiny = (std::numeric_limits<Scalar>::min)(); - CALL_SUBTEST(ArrayType test = betainc(a, one, x); ArrayType expected = x.pow(a); + + // betainc(a, 1, x) == x**a. The absolute floor covers rounding of subnormal results. + CALL_SUBTEST(ArrayType test = betainc(a, one, x); ArrayType expected = xr.pow(ar).template cast<Scalar>(); for (Index i = 0; i < test.size(); ++i) VERIFY_IS_APPROX_SCALED(test.segment(i, 1), expected.segment(i, 1), expected(i) + tiny);); @@ -311,15 +318,13 @@ CALL_SUBTEST(ArrayType test = betainc(a, b, x) + betainc(b, a, one - x); ArrayType expected = one; verify_component_wise(test, expected);); - // betainc(a+1, b, x) = betainc(a, b, x) - x**a * (1 - x)**b / (a * beta(a, b)). The difference cancels, so the - // error is bounded relative to the larger of the two terms rather than to the result, plus the smallest normal as - // the floor for subnormal results. The subtracted term is evaluated in log space: in float x**a * (1 - x)**b - // underflows while the term itself, divided by a small beta(a, b), does not. - CALL_SUBTEST(ArrayType log_beta = a.lgamma() + b.lgamma() - (a + b).lgamma(); - ArrayType term = (a * x.log() + b * (-x).log1p() - a.log() - log_beta).exp(); - ArrayType lhs = betainc(a, b, x); ArrayType scale = (lhs.max)(term) + tiny; - ArrayType expected = lhs - term; ArrayType test = betainc(a + one, b, x); - for (Index i = 0; i < test.size(); ++i) + // betainc(a+1, b, x) = betainc(a, b, x) - x**a * (1 - x)**b / (a * beta(a, b)). Bound cancellation relative + // to the larger term. Log space avoids underflow of the numerator before division by a small beta(a, b). + CALL_SUBTEST(ReferenceArray log_beta = ar.lgamma() + br.lgamma() - (ar + br).lgamma(); + ReferenceArray term = (ar * xr.log() + br * (-xr).log1p() - ar.log() - log_beta).exp(); + ArrayType lhs = betainc(a, b, x); ArrayType scale = (lhs.max)(term.template cast<Scalar>()) + tiny; + ArrayType expected = (lhs.template cast<ReferenceScalar>() - term).template cast<Scalar>(); + ArrayType test = betainc(a + one, b, x); for (Index i = 0; i < test.size(); ++i) VERIFY_IS_APPROX_SCALED(test.segment(i, 1), expected.segment(i, 1), scale(i));); // betainc(a, b+1, x) = betainc(a, b, x) + x**a * (1 - x)**b / (b * beta(a, b)) @@ -571,6 +576,23 @@ VERIFY((numext::isnan)(numext::betainc(Scalar(0.5), Scalar(0), nan))); VERIFY((numext::isnan)(numext::betainc(nan, Scalar(0.5), Scalar(0)))); + if (sizeof(Scalar) < sizeof(double)) { + // MPFR-backed Boost.Math ibeta, 100 decimal digits. The result is exp(t) with + // t = b log(1 - x) + a' log(x) + lgamma(a' + b) - lgamma(a') - lgamma(b) + log(cf / a'), a' = a + 1, summed in + // Scalar: |t| ~ 90, but its first five terms add up to S ~ 330 in magnitude, so t carries an absolute rounding + // error of order S eps, which exp turns into relative error. How much of it is reached depends on the libm (the + // result is off by 62 eps with glibc 2.43), hence S eps rather than a measured constant, plus one subnormal ulp for + // the final rounding. + const Scalar a = Scalar(38.325092315673828), b = Scalar(5.0568113327026367), x = Scalar(0.07500767707824707); + const double ref = 6.17294757793914262e-40; + const double actual = static_cast<double>(numext::betainc(a + Scalar(1), b, x)); + const double ap = static_cast<double>(a) + 1, bd = static_cast<double>(b), xd = static_cast<double>(x); + const double S = bd * std::abs(std::log1p(-xd)) + ap * std::abs(std::log(xd)) + std::lgamma(ap + bd) + + std::lgamma(ap) + std::abs(std::lgamma(bd)); + VERIFY(std::abs(actual - ref) <= S * static_cast<double>(NumTraits<Scalar>::epsilon()) * ref + + static_cast<double>((std::numeric_limits<Scalar>::denorm_min)())); + } + // A negative q sums until n + q > 9. zeta(2, -20.5) = pi^2/2 + sum_{j=0}^{20} (j + 1/2)^-2. { double ref = EIGEN_PI * EIGEN_PI / 2;
diff --git a/contrib/test/special_packetmath.cpp b/contrib/test/special_packetmath.cpp index 516e52b..1806975 100644 --- a/contrib/test/special_packetmath.cpp +++ b/contrib/test/special_packetmath.cpp
@@ -93,6 +93,7 @@ void neon_erf_subnormals_bfloat16() { packet_erf_subnormals<bfloat16, internal::Packet4bf>(); + packet_erf_subnormals<bfloat16, internal::Packet8bf>(); // Exhaustive bfloat16 subnormals, with both signs, checked without floating-point subnormal conversions. for (unsigned int n = 1; n < 128; ++n) { const numext::uint16_t expectedBits =
diff --git a/contrib/test/splines.cpp b/contrib/test/splines.cpp index a6cd449..e57de4a 100644 --- a/contrib/test/splines.cpp +++ b/contrib/test/splines.cpp
@@ -233,7 +233,35 @@ } } +template <typename Scalar> +void check_high_degree_basis_derivatives() { + using SplineType = Spline<Scalar, 1>; + // 13! exceeds signed 32-bit indices; 21! exceeds signed 64-bit indices. + for (DenseIndex degree : {DenseIndex(13), DenseIndex(21), DenseIndex(22)}) { + typename SplineType::KnotVectorType knots(2 * (degree + 1)); + knots.head(degree + 1).setZero(); + knots.tail(degree + 1).setOnes(); + const auto derivatives = SplineType::BasisFunctionDerivatives(Scalar(0.5), degree, degree, knots); + long double factorial = 1; + for (DenseIndex k = 2; k <= degree; ++k) factorial *= static_cast<long double>(k); + long double binomial = 1; + for (DenseIndex j = 0; j <= degree; ++j) { + // Clamped knots give Bernstein polynomials: B_j^(p) = (-1)^(p-j) p! binomial(p,j). + const Scalar expected = static_cast<Scalar>(((degree - j) % 2 ? -factorial : factorial) * binomial); + const Scalar actual = derivatives(degree, j); + // At most p-1 factorial roundings, one final product, and reference rounding; 2p*eps leaves margin. + const Scalar bound = Scalar(2 * degree) * NumTraits<Scalar>::epsilon() * numext::abs(expected); + VERIFY((numext::isfinite)(actual)); + VERIFY((numext::isfinite)(bound)); + VERIFY(numext::abs(actual - expected) <= bound); + binomial = binomial * static_cast<long double>(degree - j) / static_cast<long double>(j + 1); + } + } +} + EIGEN_DECLARE_TEST(splines) { + CALL_SUBTEST(check_high_degree_basis_derivatives<float>()); + CALL_SUBTEST(check_high_degree_basis_derivatives<double>()); for (int i = 0; i < g_repeat; ++i) { CALL_SUBTEST(eval_spline3d()); CALL_SUBTEST(eval_spline3d_onbrks());
diff --git a/contrib/test/structured_kronecker.cpp b/contrib/test/structured_kronecker.cpp index be59af9..63269d1 100644 --- a/contrib/test/structured_kronecker.cpp +++ b/contrib/test/structured_kronecker.cpp
@@ -11,20 +11,10 @@ #include "fp_control.h" #include <contrib/Eigen/StructuredMatrices> +#include "structured_test_helpers.h" using namespace Eigen; -// Reference dense Kronecker product, built entry-wise and independently of both -// the operator under test and the KroneckerProduct module. -template <typename Scalar> -Matrix<Scalar, Dynamic, Dynamic> reference_kron(const Matrix<Scalar, Dynamic, Dynamic>& A, - const Matrix<Scalar, Dynamic, Dynamic>& B) { - Matrix<Scalar, Dynamic, Dynamic> K(A.rows() * B.rows(), A.cols() * B.cols()); - for (Index i = 0; i < K.rows(); ++i) - for (Index j = 0; j < K.cols(); ++j) K(i, j) = A(i / B.rows(), j / B.cols()) * B(i % B.rows(), j % B.cols()); - return K; -} - template <typename Scalar> void test_kron_product(Index m1, Index n1, Index m2, Index n2) { typedef Matrix<Scalar, Dynamic, 1> Vec; @@ -421,6 +411,7 @@ // so rank() is 3 -- matching the dense complete orthogonal decomposition. // leastSquaresSolve() decides rank per factor instead (it never forms the // pairwise products), so it applies A^+ (x) B^+ with both 1e-8 modes inverted. +template <typename = void> void test_kron_product_level_rank() { typedef Matrix<double, Dynamic, 1> Vec; typedef Matrix<double, Dynamic, Dynamic> Mat; @@ -445,6 +436,7 @@ // det(A (x) B) = det(A)^n2 * det(B)^n1 must be accumulated with exponent // balancing: det(A)^n2 or det(B)^n1 can overflow or underflow on their own even // when the result is representable. +template <typename = void> void test_kron_determinant_scaling() { typedef Matrix<double, Dynamic, Dynamic> Mat; @@ -487,6 +479,7 @@ // multiplication, so every pairwise singular-value product used to pass and // rank() returned 8 where the dense SVD says 1. The comparison must happen in // ratio space: (sa_i/sa_0) * (sb_j/sb_0) against min(rows,cols) * eps. +template <typename = void> void test_kron_rank_ratio_threshold() { typedef Matrix<double, Dynamic, 1> Vec; typedef Matrix<double, Dynamic, Dynamic> Mat; @@ -528,6 +521,7 @@ // zero. A = [1], B = [DBL_MIN/2] has the singular value DBL_MIN/2: the dense // rank is 0, but without the clamp the structured rank was 1. The boundary // DBL_MIN itself stays rank one, matching SVDBase::rank()'s >= convention. +template <typename = void> void test_kron_rank_min_normal_clamp() { typedef Matrix<double, Dynamic, 1> Vec; typedef Matrix<double, Dynamic, Dynamic> Mat; @@ -952,6 +946,7 @@ // Through diagonal factors, the balanced determinant survives partial products // that overflow on their own. +template <typename = void> void test_kron_diag_determinant_scaling() { typedef Matrix<double, Dynamic, 1> Vec; typedef DiagonalMatrix<double, Dynamic> Diag; @@ -1013,6 +1008,7 @@ // Matrix-free CG through a diagonal-factor operator: a positive diagonal // tensored with an SPD dense factor is SPD. +template <typename = void> void test_kron_diag_matrix_free_cg(Index n1, Index n2) { typedef Matrix<double, Dynamic, 1> Vec; typedef Matrix<double, Dynamic, Dynamic> Mat; @@ -1377,6 +1373,7 @@ // determinant is an exact zero. A NaN entry propagates through products, solves // and the determinant. An all-zero factor gives zero products, determinant and // rank, and an empty materialization. +template <typename = void> void test_kron_sparse_degenerate(Index n1, Index n2) { using Vec = Matrix<double, Dynamic, 1>; using Mat = Matrix<double, Dynamic, Dynamic>; @@ -1420,6 +1417,7 @@ // A factor handed over uncompressed (reserve + insert) is compressed on // construction, and its explicitly stored zeros count as stored entries of the // materialization, exactly as in kroneckerProduct(). +template <typename = void> void test_kron_sparse_stored_zeros(Index n1, Index n2) { using Vec = Matrix<double, Dynamic, 1>; using Mat = Matrix<double, Dynamic, Dynamic>; @@ -1449,6 +1447,7 @@ // An ill-conditioned sparse factor: the tridiagonal Laplacian of order n has // condition number ~0.4 n^2, so I_p (x) L is judged by its backward error // ||b - K x|| <= n eps ||L||_2 ||x||, with ||L||_2 < 4, not entrywise. +template <typename = void> void test_kron_sparse_solve_laplacian(Index n, Index p) { using Vec = Matrix<double, Dynamic, 1>; using Sparse = SparseMatrix<double>; @@ -1529,6 +1528,7 @@ // The balanced SparseLU determinant survives partial products that overflow on // their own. +template <typename = void> void test_kron_sparse_determinant_scaling() { using Vec = Matrix<double, Dynamic, 1>; using Mat = Matrix<double, Dynamic, Dynamic>; @@ -1576,6 +1576,7 @@ // Matrix-free CG through a sparse-factor operator: an SPD sparse factor // tensored with an SPD dense factor is SPD. +template <typename = void> void test_kron_sparse_matrix_free_cg(Index n1, Index n2) { using Vec = Matrix<double, Dynamic, 1>; using Mat = Matrix<double, Dynamic, Dynamic>; @@ -2152,10 +2153,10 @@ // Numerical boundaries: product-level rank truncation, balanced determinant // and ratio-space rank thresholds. - CALL_SUBTEST_7(test_kron_product_level_rank()); - CALL_SUBTEST_7(test_kron_determinant_scaling()); - CALL_SUBTEST_7(test_kron_rank_ratio_threshold()); - CALL_SUBTEST_7(test_kron_rank_min_normal_clamp()); + CALL_SUBTEST_7(test_kron_product_level_rank<>()); + CALL_SUBTEST_7(test_kron_determinant_scaling<>()); + CALL_SUBTEST_7(test_kron_rank_ratio_threshold<>()); + CALL_SUBTEST_7(test_kron_rank_min_normal_clamp<>()); // Diagonal and identity factors: products, transposition family, // mixed-scalar promotion, fixed sizes. @@ -2186,8 +2187,8 @@ CALL_SUBTEST_35((test_kron_diag_svd<std::complex<double>>(3, 4))); CALL_SUBTEST_36((test_kron_diag_rank<std::complex<double>>(3, 4))); CALL_SUBTEST_37((test_kron_diag_least_squares<std::complex<double>>(3, 4))); - CALL_SUBTEST_9(test_kron_diag_determinant_scaling()); - CALL_SUBTEST_9(test_kron_diag_matrix_free_cg(6, 7)); + CALL_SUBTEST_9(test_kron_diag_determinant_scaling<>()); + CALL_SUBTEST_9(test_kron_diag_matrix_free_cg<>(6, 7)); // Sparse factors: products, materialization, the identity-Kronecker // operators, the transposition family, mixed-scalar promotion. @@ -2203,7 +2204,7 @@ CALL_SUBTEST_40((test_kron_sparse_transpose<std::complex<double>>(3, 4, 2, 5))); CALL_SUBTEST_10((test_kron_sparse_mixed_scalar<double>(4, 3, 5))); CALL_SUBTEST_40((test_kron_sparse_mixed_scalar<float>(3, 4, 2))); - CALL_SUBTEST_10(test_kron_sparse_stored_zeros(4, 3)); + CALL_SUBTEST_10(test_kron_sparse_stored_zeros<>(4, 3)); // Sparse factors through the solvers, inverse/determinant, the // decomposition family and matrix-free CG. @@ -2212,18 +2213,18 @@ CALL_SUBTEST_22((test_kron_sparse_solve<std::complex<double>>(6, 4))); CALL_SUBTEST_11((test_kron_sparse_inverse_determinant<double>(4, 5))); CALL_SUBTEST_22((test_kron_sparse_inverse_determinant<std::complex<double>>(3, 4))); - CALL_SUBTEST_11((test_kron_sparse_eigen<double>(4, 5))); - CALL_SUBTEST_11((test_kron_sparse_svd<double>(4, 5))); - CALL_SUBTEST_11((test_kron_sparse_rank<double>(4, 5))); - CALL_SUBTEST_11((test_kron_sparse_least_squares<double>(4, 5))); + CALL_SUBTEST_57((test_kron_sparse_eigen<double>(4, 5))); + CALL_SUBTEST_57((test_kron_sparse_svd<double>(4, 5))); + CALL_SUBTEST_57((test_kron_sparse_rank<double>(4, 5))); + CALL_SUBTEST_57((test_kron_sparse_least_squares<double>(4, 5))); CALL_SUBTEST_23((test_kron_sparse_eigen<std::complex<double>>(3, 4))); CALL_SUBTEST_41((test_kron_sparse_svd<std::complex<double>>(3, 4))); CALL_SUBTEST_42((test_kron_sparse_rank<std::complex<double>>(3, 4))); CALL_SUBTEST_43((test_kron_sparse_least_squares<std::complex<double>>(3, 4))); - CALL_SUBTEST_11(test_kron_sparse_solve_laplacian(64, 4)); - CALL_SUBTEST_11(test_kron_sparse_degenerate(4, 3)); - CALL_SUBTEST_11(test_kron_sparse_determinant_scaling()); - CALL_SUBTEST_11(test_kron_sparse_matrix_free_cg(6, 7)); + CALL_SUBTEST_11(test_kron_sparse_solve_laplacian<>(64, 4)); + CALL_SUBTEST_11(test_kron_sparse_degenerate<>(4, 3)); + CALL_SUBTEST_11(test_kron_sparse_determinant_scaling<>()); + CALL_SUBTEST_11(test_kron_sparse_matrix_free_cg<>(6, 7)); CALL_SUBTEST_12((test_kron_sparse_determinant_subnormal<float, ColMajor>(1))); CALL_SUBTEST_12((test_kron_sparse_determinant_subnormal<double, RowMajor>(1)));
diff --git a/contrib/test/structured_kronecker_sparse_view.cpp b/contrib/test/structured_kronecker_sparse_view.cpp index f5c77ff..f3f83fe 100644 --- a/contrib/test/structured_kronecker_sparse_view.cpp +++ b/contrib/test/structured_kronecker_sparse_view.cpp
@@ -143,6 +143,7 @@ // The finite-difference assembly the view exists for: I + tau L built in one // sparse expression, and the Jacobi preconditioner read through the view. +template <typename = void> void test_view_finite_difference(Index n1, Index n2) { using Sparse = SparseMatrix<double>; using RowSparse = SparseMatrix<double, RowMajor>; @@ -179,6 +180,7 @@ // Factors outside the common case: a compressed factor with unsorted inner // vectors, signed zeros in a Kronecker sum, and 64-bit sparse indices. +template <typename = void> void test_view_factor_contracts() { using Sparse = SparseMatrix<double>; Sparse U(3, 3); @@ -238,7 +240,7 @@ CALL_SUBTEST_1((test_view_kronecker<double>(1, 1, 1, 1))); CALL_SUBTEST_2((test_view_kronecker_sum<double>(3, 4))); CALL_SUBTEST_2((test_view_kronecker_sum<std::complex<double>>(2, 3))); - CALL_SUBTEST_2(test_view_finite_difference(6, 5)); - CALL_SUBTEST_3(test_view_factor_contracts()); + CALL_SUBTEST_2(test_view_finite_difference<>(6, 5)); + CALL_SUBTEST_3(test_view_factor_contracts<>()); } }
diff --git a/contrib/test/structured_kronecker_sum.cpp b/contrib/test/structured_kronecker_sum.cpp index 67172db..ac343cb 100644 --- a/contrib/test/structured_kronecker_sum.cpp +++ b/contrib/test/structured_kronecker_sum.cpp
@@ -10,18 +10,11 @@ #include "main.h" #include <contrib/Eigen/StructuredMatrices> +#include "structured_test_helpers.h" using namespace Eigen; // Dense references built entry-wise, independently of the operators under test. -template <typename Scalar> -Matrix<Scalar, Dynamic, Dynamic> reference_kron(const Matrix<Scalar, Dynamic, Dynamic>& A, - const Matrix<Scalar, Dynamic, Dynamic>& B) { - Matrix<Scalar, Dynamic, Dynamic> K(A.rows() * B.rows(), A.cols() * B.cols()); - for (Index i = 0; i < K.rows(); ++i) - for (Index j = 0; j < K.cols(); ++j) K(i, j) = A(i / B.rows(), j / B.cols()) * B(i % B.rows(), j % B.cols()); - return K; -} template <typename Scalar> Matrix<Scalar, Dynamic, Dynamic> reference_ksum(const Matrix<Scalar, Dynamic, Dynamic>& A, @@ -694,6 +687,7 @@ // u_t = (d_yy + d_xx) u on an n1 x n2 grid with D = tridiag(-1, 2, -1) the SPD // negated second difference, (I + tau (Dy (+) Dx)) u = b, against SparseLU of // the materialized matrix; and matrix-free CG on Dy (+) Dx. +template <typename = void> void test_ksum_finite_difference(Index n1, Index n2) { using Sparse = SparseMatrix<double>; using Vec = VectorXd; @@ -729,6 +723,7 @@ } // Fixed-size factors keep the compile-time size. +template <typename = void> void test_ksum_fixed() { const Matrix3d A = Matrix3d::Random(); const Matrix2d B = Matrix2d::Random(); @@ -746,7 +741,7 @@ for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST_1((test_ksum_product<double>(3, 4))); CALL_SUBTEST_1((test_ksum_product<double>(1, 1))); - CALL_SUBTEST_1(test_ksum_fixed()); + CALL_SUBTEST_1(test_ksum_fixed<>()); CALL_SUBTEST_5((test_ksum_product<float>(4, 3))); CALL_SUBTEST_6((test_ksum_product<std::complex<double>>(3, 3))); CALL_SUBTEST_2((test_ksum_nested<double>(2, 3, 4))); @@ -763,7 +758,7 @@ CALL_SUBTEST_9((test_ksum_hermitian_gate<std::complex<double>>(4, 3))); CALL_SUBTEST_11((test_ksum_hermitian_gate<std::complex<float>>(3, 4))); CALL_SUBTEST_4((test_ksum_eigenvalues<double>(3, 4))); - CALL_SUBTEST_4((test_ksum_eigenvalues<std::complex<float>>(3, 2))); - CALL_SUBTEST_4(test_ksum_finite_difference(12, 9)); + CALL_SUBTEST_13((test_ksum_eigenvalues<std::complex<float>>(3, 2))); + CALL_SUBTEST_14(test_ksum_finite_difference<>(12, 9)); } }
diff --git a/contrib/test/structured_matrices.cpp b/contrib/test/structured_matrices.cpp index 459160a..a303a6e 100644 --- a/contrib/test/structured_matrices.cpp +++ b/contrib/test/structured_matrices.cpp
@@ -1671,22 +1671,22 @@ CALL_SUBTEST_7((test_circulant_svd<double>(24))); CALL_SUBTEST_7((test_circulant_svd<std::complex<double>>(18))); CALL_SUBTEST_7((test_circulant_svd<float>(12))); - CALL_SUBTEST_7((test_circulant_minnorm_solve<double>(20))); - CALL_SUBTEST_7((test_circulant_minnorm_solve<std::complex<double>>(15))); - CALL_SUBTEST_7((test_circulant_rank_deficient<std::complex<double>>(24, 3))); - CALL_SUBTEST_7((test_circulant_rank_deficient<std::complex<double>>(50, 5))); - CALL_SUBTEST_7((test_circulant_rank_deficient<std::complex<float>>(16, 2))); - CALL_SUBTEST_7((test_circulant_zero<double>(12))); - CALL_SUBTEST_7((test_circulant_zero<std::complex<double>>(7))); - CALL_SUBTEST_7(test_circulant_nan_propagation<>(20)); - CALL_SUBTEST_7((test_circulant_inverse<double>(1))); - CALL_SUBTEST_7((test_circulant_inverse<double>(50))); - CALL_SUBTEST_7((test_circulant_inverse<double>(97))); // prime: inverse rebuilds the padded product symbol - CALL_SUBTEST_7((test_circulant_inverse<std::complex<double>>(20))); - CALL_SUBTEST_7((test_circulant_inverse<float>(16))); - CALL_SUBTEST_7((test_circulant_determinant<double>(1))); - CALL_SUBTEST_7((test_circulant_determinant<double>(12))); - CALL_SUBTEST_7((test_circulant_determinant<std::complex<double>>(9))); + CALL_SUBTEST_19((test_circulant_minnorm_solve<double>(20))); + CALL_SUBTEST_19((test_circulant_minnorm_solve<std::complex<double>>(15))); + CALL_SUBTEST_19((test_circulant_rank_deficient<std::complex<double>>(24, 3))); + CALL_SUBTEST_19((test_circulant_rank_deficient<std::complex<double>>(50, 5))); + CALL_SUBTEST_19((test_circulant_rank_deficient<std::complex<float>>(16, 2))); + CALL_SUBTEST_20((test_circulant_zero<double>(12))); + CALL_SUBTEST_20((test_circulant_zero<std::complex<double>>(7))); + CALL_SUBTEST_20(test_circulant_nan_propagation<>(20)); + CALL_SUBTEST_20((test_circulant_inverse<double>(1))); + CALL_SUBTEST_20((test_circulant_inverse<double>(50))); + CALL_SUBTEST_20((test_circulant_inverse<double>(97))); // prime: inverse rebuilds the padded product symbol + CALL_SUBTEST_20((test_circulant_inverse<std::complex<double>>(20))); + CALL_SUBTEST_20((test_circulant_inverse<float>(16))); + CALL_SUBTEST_20((test_circulant_determinant<double>(1))); + CALL_SUBTEST_20((test_circulant_determinant<double>(12))); + CALL_SUBTEST_20((test_circulant_determinant<std::complex<double>>(9))); // Numerical and lifetime boundaries: value-nested (owning) delayed products, // aliased products across the dispatch tiers (scalar, direct, FFT), the @@ -1771,21 +1771,21 @@ CALL_SUBTEST_9((test_hankel_symmetry<double>(48))); CALL_SUBTEST_9((test_hankel_to_toeplitz<double>(12, 7))); CALL_SUBTEST_9((test_hankel_to_toeplitz<std::complex<double>>(40, 56))); - CALL_SUBTEST_9((test_hankel_solve<double>(1))); - CALL_SUBTEST_9((test_hankel_solve<double>(20))); - CALL_SUBTEST_9((test_hankel_solve<double>(45))); - CALL_SUBTEST_9((test_hankel_solve<std::complex<double>>(24))); - CALL_SUBTEST_9((test_hankel_solve<float>(16))); - CALL_SUBTEST_9((test_hankel_solve_lookahead<double>(20))); - CALL_SUBTEST_9((test_hankel_solve_lookahead<std::complex<double>>(15))); - CALL_SUBTEST_9(test_hankel_hilbert<>()); - CALL_SUBTEST_9((test_hankel_least_squares<double>(60, 40))); - CALL_SUBTEST_9((test_hankel_least_squares<std::complex<double>>(48, 32))); - CALL_SUBTEST_9((test_hankel_fixed<double, 4, 6>())); - CALL_SUBTEST_9((test_hankel_fixed<double, 40, 24>())); - CALL_SUBTEST_9((test_hankel_fixed<std::complex<float>, 6, 4>())); - CALL_SUBTEST_9((test_hankel_fixed_solve<double, 8>())); - CALL_SUBTEST_9((test_hankel_fixed_solve<std::complex<double>, 12>())); + CALL_SUBTEST_17((test_hankel_solve<double>(1))); + CALL_SUBTEST_17((test_hankel_solve<double>(20))); + CALL_SUBTEST_17((test_hankel_solve<double>(45))); + CALL_SUBTEST_23((test_hankel_solve<std::complex<double>>(24))); + CALL_SUBTEST_23((test_hankel_solve<float>(16))); + CALL_SUBTEST_21((test_hankel_solve_lookahead<double>(20))); + CALL_SUBTEST_21((test_hankel_solve_lookahead<std::complex<double>>(15))); + CALL_SUBTEST_21(test_hankel_hilbert<>()); + CALL_SUBTEST_22((test_hankel_least_squares<double>(60, 40))); + CALL_SUBTEST_22((test_hankel_least_squares<std::complex<double>>(48, 32))); + CALL_SUBTEST_18((test_hankel_fixed<double, 4, 6>())); + CALL_SUBTEST_18((test_hankel_fixed<double, 40, 24>())); + CALL_SUBTEST_18((test_hankel_fixed<std::complex<float>, 6, 4>())); + CALL_SUBTEST_18((test_hankel_fixed_solve<double, 8>())); + CALL_SUBTEST_18((test_hankel_fixed_solve<std::complex<double>, 12>())); // MR 2688 review regressions: delayed (value-nested) products, aliased right-hand sides across the // dispatch tiers, and mixed real/complex products.
diff --git a/contrib/test/structured_test_helpers.h b/contrib/test/structured_test_helpers.h index ef5dc9b..e57228a 100644 --- a/contrib/test/structured_test_helpers.h +++ b/contrib/test/structured_test_helpers.h
@@ -8,6 +8,19 @@ namespace Eigen { +// Coefficient-wise reference independent of StructuredMatrices and KroneckerProduct. +template <typename Scalar> +Matrix<Scalar, Dynamic, Dynamic> reference_kron(const Matrix<Scalar, Dynamic, Dynamic>& A, + const Matrix<Scalar, Dynamic, Dynamic>& B) { + Matrix<Scalar, Dynamic, Dynamic> K(A.rows() * B.rows(), A.cols() * B.cols()); + for (Index j = 0; j < A.cols(); ++j) { + for (Index i = 0; i < A.rows(); ++i) { + K.block(i * B.rows(), j * B.cols(), B.rows(), B.cols()) = A(i, j) * B; + } + } + return K; +} + // Vectorized complex products can introduce Inf - Inf across component accumulators. // Preserve scalar evaluation order for the non-finite reference results. template <typename Scalar>
diff --git a/contrib/test/structured_vandermonde.cpp b/contrib/test/structured_vandermonde.cpp index 271ae0d..b4ee51f 100644 --- a/contrib/test/structured_vandermonde.cpp +++ b/contrib/test/structured_vandermonde.cpp
@@ -13,6 +13,34 @@ using namespace Eigen; +struct VandermondeMatrixScalar : Matrix2d {}; + +double operator*(const VandermondeMatrixScalar& scale, double coefficient) { return scale.sum() * coefficient; } + +namespace Eigen { +template <> +struct ScalarBinaryOpTraits<VandermondeMatrixScalar, double, + internal::scalar_product_op<VandermondeMatrixScalar, double>> { + using ReturnType = double; +}; +} // namespace Eigen + +template <typename = void> +void test_vandermonde_matrix_scalar() { + static_assert(!internal::has_competing_expression_binary_op<Vandermonde<double, 2, 2>, VandermondeMatrixScalar, + internal::scalar_product_op, true>::HasExpressionOverload, + "a matrix operand alone does not provide left multiplication of a Vandermonde"); + Vector2d nodes; + nodes << 2, 3; + const Vandermonde<double, 2, 2> matrix(nodes); + VandermondeMatrixScalar scale; + scale.setIdentity(); + const Matrix2d result = scale * matrix; + Matrix2d expected; + expected << 2, 4, 2, 6; + VERIFY_IS_EQUAL(result, expected); +} + // Reference dense Vandermonde built entry-wise from the nodes. template <typename Scalar> Matrix<Scalar, Dynamic, Dynamic> reference_vandermonde(const Matrix<Scalar, Dynamic, 1>& x, Index n) { @@ -644,6 +672,7 @@ } EIGEN_DECLARE_TEST(structured_vandermonde) { + CALL_SUBTEST_3(test_vandermonde_matrix_scalar<>()); for (int i = 0; i < g_repeat; ++i) { // Horner products, dense assignment, coefficient access. CALL_SUBTEST_1((test_vandermonde_product<double>(1, 1)));
diff --git a/contrib/test/tensor_block_access.cpp b/contrib/test/tensor_block_access.cpp index d8dd0b8..f596a88 100644 --- a/contrib/test/tensor_block_access.cpp +++ b/contrib/test/tensor_block_access.cpp
@@ -15,6 +15,7 @@ #include <set> #include <Eigen/Tensor> +#include "tensor_test_helpers.h" using Eigen::ColMajor; using Eigen::Index; @@ -29,24 +30,6 @@ return layout == ColMajor ? col : row; } -static TensorBlockShapeType RandomShape() { - return internal::random<bool>() ? TensorBlockShapeType::kUniformAllDims : TensorBlockShapeType::kSkewedInnerDims; -} - -template <int NumDims> -static size_t RandomTargetSize(const DSizes<Index, NumDims>& dims) { - return internal::random<size_t>(1, dims.TotalSize()); -} - -template <int NumDims> -static DSizes<Index, NumDims> RandomDims() { - array<Index, NumDims> dims; - for (int i = 0; i < NumDims; ++i) { - dims[i] = internal::random<int>(1, 20); - } - return DSizes<Index, NumDims>(dims); -} - template <typename T> static T* GenerateRandomData(const Index& size) { T* data = new T[size]; @@ -118,14 +101,14 @@ static void test_block_mapper_maps_every_element() { typedef internal::TensorBlockMapper<NumDims, Layout> TensorBlockMapper; - DSizes<Index, NumDims> dims = RandomDims<NumDims>(); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims, int>(); DSizes<Index, NumDims> strides = internal::strides<Layout>(dims); // Keep track of elements indices available via block access. std::set<Index> coeff_set; // Try different combinations of block types and sizes. - TensorBlockMapper block_mapper(dims, {RandomShape(), RandomTargetSize(dims), zeroCost()}); + TensorBlockMapper block_mapper(dims, {randomTensorBlockShape(), randomTensorBlockSize(dims), zeroCost()}); for (int i = 0; i < block_mapper.blockCount(); ++i) { auto block = block_mapper.blockDescriptor(i);
diff --git a/contrib/test/tensor_block_eval.cpp b/contrib/test/tensor_block_eval.cpp index 6ed1242..1e61f1e 100644 --- a/contrib/test/tensor_block_eval.cpp +++ b/contrib/test/tensor_block_eval.cpp
@@ -10,6 +10,7 @@ // clang-format off #include "main.h" #include <Eigen/Tensor> +#include "tensor_test_helpers.h" #include "../Eigen/SpecialFunctions" // clang-format on @@ -28,15 +29,6 @@ // -------------------------------------------------------------------------- // // Utility functions to generate random tensors, blocks, and evaluate them. -template <int NumDims> -static DSizes<Index, NumDims> RandomDims(Index min, Index max) { - DSizes<Index, NumDims> dims; - for (int i = 0; i < NumDims; ++i) { - dims[i] = internal::random<Index>(min, max); - } - return DSizes<Index, NumDims>(dims); -} - // Block offsets and extents allows to construct a TensorSlicingOp corresponding // to a TensorBlockDescriptor. template <int NumDims> @@ -49,8 +41,8 @@ template <int Layout, int NumDims> static TensorBlockParams<NumDims> RandomBlock(DSizes<Index, NumDims> dims, Index min, Index max) { // Choose random offsets and sizes along all tensor dimensions. - DSizes<Index, NumDims> offsets(RandomDims<NumDims>(min, max)); - DSizes<Index, NumDims> sizes(RandomDims<NumDims>(min, max)); + DSizes<Index, NumDims> offsets(randomTensorDims<NumDims>(min, max)); + DSizes<Index, NumDims> sizes(randomTensorDims<NumDims>(min, max)); // Make sure that offset + size do not overflow dims. for (int i = 0; i < NumDims; ++i) { @@ -86,9 +78,8 @@ template <int Layout, int NumDims> static TensorBlockParams<NumDims> SkewedInnerBlock(DSizes<Index, NumDims> dims) { using BlockMapper = internal::TensorBlockMapper<NumDims, Layout, Index>; - BlockMapper block_mapper( - dims, - {internal::TensorBlockShapeType::kSkewedInnerDims, internal::random<size_t>(1, dims.TotalSize()), {0, 0, 0}}); + BlockMapper block_mapper(dims, + {internal::TensorBlockShapeType::kSkewedInnerDims, randomTensorBlockSize(dims), {0, 0, 0}}); Index total_blocks = block_mapper.blockCount(); Index block_index = internal::random<Index>(0, total_blocks - 1); @@ -228,7 +219,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_block() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -238,7 +229,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_unary_expr_block() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -247,7 +238,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_binary_expr_block() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> lhs(dims), rhs(dims); setRandomForBinaryProduct(lhs); setRandomForBinaryProduct(rhs); @@ -257,7 +248,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_binary_with_unary_expr_block() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> lhs(dims), rhs(dims); lhs.setRandom(); rhs.setRandom(); @@ -268,11 +259,11 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_broadcast() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(1, 10); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(1, 10); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); - DSizes<Index, NumDims> bcast = RandomDims<NumDims>(1, 5); + DSizes<Index, NumDims> bcast = randomTensorDims<NumDims>(1, 5); DSizes<Index, NumDims> bcasted_dims; for (int i = 0; i < NumDims; ++i) bcasted_dims[i] = dims[i] * bcast[i]; @@ -298,7 +289,7 @@ static void test_eval_tensor_scan() { // Scan eagerly materializes m_output during evalSubExprsIfNeeded; this // exercises the block() wrapper around that buffer. - DSizes<Index, NumDims> dims = RandomDims<NumDims>(4, 12); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(4, 12); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -315,7 +306,7 @@ // exercises the block() wrapper around that buffer. Use RealPart so // the output type matches the input type and the harness's slice // comparison stays straightforward. - DSizes<Index, NumDims> dims = RandomDims<NumDims>(4, 12); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(4, 12); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -392,7 +383,7 @@ // the input with the opposite layout and assert the block evaluator on the // resulting expression matches the slice-based reference. constexpr int InputLayout = (Layout == ColMajor) ? RowMajor : ColMajor; - DSizes<Index, NumDims> input_dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> input_dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, InputLayout> input(input_dims); setRandomForBinaryProduct(input); @@ -442,7 +433,7 @@ // over the composed expressions exercises both sides of that compile path. template <typename T, int NumDims, int Layout> static void test_eval_composed_block_ops() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(4, 8); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(4, 8); array<std::pair<Index, Index>, NumDims> paddings; DSizes<Index, NumDims> padded_dims; for (int i = 0; i < NumDims; ++i) { @@ -492,7 +483,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_reshape() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(1, 10); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(1, 10); DSizes<Index, NumDims> shuffled = dims; std::shuffle(&shuffled[0], &shuffled[NumDims - 1], std::mt19937(g_seed)); @@ -509,7 +500,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_cast() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -519,7 +510,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_ternary() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> a(dims); Tensor<T, NumDims, Layout> b(dims); Tensor<T, NumDims, Layout> x(dims); @@ -538,7 +529,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_nullary() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -556,7 +547,7 @@ // Each draw is a pure function of the generator's seed and the coefficient's // tensor-linear index, so a block must reproduce the coefficient evaluation // of the same expression exactly. - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); auto expr = input.random(); @@ -585,7 +576,7 @@ T operator()(Index i) const { return static_cast<T>(i); } }; - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); VerifyBlockEvaluator<T, NumDims, Layout>(input.nullaryExpr(BothOverloads()), @@ -596,7 +587,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_select() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> lhs(dims); Tensor<T, NumDims, Layout> rhs(dims); Tensor<bool, NumDims, Layout> cond(dims); @@ -612,12 +603,12 @@ static void test_eval_tensor_padding() { constexpr int inner_dim = Layout == static_cast<int>(ColMajor) ? 0 : NumDims - 1; - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); - DSizes<Index, NumDims> pad_before = RandomDims<NumDims>(0, 4); - DSizes<Index, NumDims> pad_after = RandomDims<NumDims>(0, 4); + DSizes<Index, NumDims> pad_before = randomTensorDims<NumDims>(0, 4); + DSizes<Index, NumDims> pad_after = randomTensorDims<NumDims>(0, 4); array<std::pair<Index, Index>, NumDims> paddings; for (int i = 0; i < NumDims; ++i) { paddings[i] = std::make_pair(pad_before[i], pad_after[i]); @@ -647,7 +638,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_chipping() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -679,7 +670,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_concatenation() { - DSizes<Index, NumDims> lhs_dims = RandomDims<NumDims>(5, 12); + DSizes<Index, NumDims> lhs_dims = randomTensorDims<NumDims>(5, 12); DSizes<Index, NumDims> rhs_dims = lhs_dims; const Index axis = internal::random<int>(0, NumDims - 1); @@ -728,7 +719,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_generator() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -742,7 +733,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_reverse() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -758,13 +749,13 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_slice() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); // Pick a random slice of an input tensor. - DSizes<Index, NumDims> slice_start = RandomDims<NumDims>(5, 10); - DSizes<Index, NumDims> slice_size = RandomDims<NumDims>(5, 10); + DSizes<Index, NumDims> slice_start = randomTensorDims<NumDims>(5, 10); + DSizes<Index, NumDims> slice_size = randomTensorDims<NumDims>(5, 10); // Make sure that slice start + size do not overflow tensor dims. for (int i = 0; i < NumDims; ++i) { @@ -815,7 +806,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_shuffle() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(5, 15); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(5, 15); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -840,14 +831,13 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_inflation() { // Keep the input small: inflation multiplies every dimension by its stride. - DSizes<Index, NumDims> dims = RandomDims<NumDims>(2, 5); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(2, 5); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); - DSizes<Index, NumDims> strides; + DSizes<Index, NumDims> strides = randomTensorDims<NumDims>(1, 3); DSizes<Index, NumDims> inflated_dims; for (int i = 0; i < NumDims; ++i) { - strides[i] = internal::random<Index>(1, 3); inflated_dims[i] = (dims[i] - 1) * strides[i] + 1; } @@ -863,7 +853,7 @@ template <typename T, int NumDims, int Layout> static void test_eval_tensor_strided_slice() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -970,7 +960,7 @@ template <typename T, int Layout> static void test_eval_tensor_patch() { - DSizes<Index, 3> dims = RandomDims<3>(4, 10); + DSizes<Index, 3> dims = randomTensorDims<3>(4, 10); Tensor<T, 3, Layout> input(dims); input.setRandom(); @@ -1250,7 +1240,7 @@ template <typename T, int NumDims, int Layout> static void test_assign_to_tensor() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> tensor(dims); TensorMap<Tensor<T, NumDims, Layout>> map(tensor.data(), dims); @@ -1261,7 +1251,7 @@ template <typename T, int NumDims, int Layout> static void test_assign_to_tensor_reshape() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> tensor(dims); TensorMap<Tensor<T, NumDims, Layout>> map(tensor.data(), dims); @@ -1281,7 +1271,7 @@ template <typename T, int NumDims, int Layout> static void test_assign_to_tensor_chipping() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> tensor(dims); Index chip_dim = internal::random<int>(0, NumDims - 1); @@ -1309,12 +1299,12 @@ template <typename T, int NumDims, int Layout> static void test_assign_to_tensor_slice() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> tensor(dims); // Pick a random slice of tensor. - DSizes<Index, NumDims> slice_start = RandomDims<NumDims>(5, 10); - DSizes<Index, NumDims> slice_size = RandomDims<NumDims>(5, 10); + DSizes<Index, NumDims> slice_start = randomTensorDims<NumDims>(5, 10); + DSizes<Index, NumDims> slice_size = randomTensorDims<NumDims>(5, 10); // Make sure that slice start + size do not overflow tensor dims. for (int i = 0; i < NumDims; ++i) { @@ -1336,7 +1326,7 @@ template <typename T, int NumDims, int Layout> static void test_assign_to_tensor_shuffle() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(5, 15); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(5, 15); Tensor<T, NumDims, Layout> tensor(dims); DSizes<Index, NumDims> shuffle; @@ -1363,7 +1353,7 @@ // static so the lambdas below can name it without capturing: MSVC 14.29 // rejects an uncaptured non-static constexpr local there (error C3493). static constexpr int SwappedLayout = (Layout == ColMajor) ? RowMajor : ColMajor; - DSizes<Index, NumDims> dims = RandomDims<NumDims>(5, 15); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(5, 15); Tensor<T, NumDims, Layout> tensor(dims); DSizes<Index, NumDims> swapped_dims; @@ -1380,7 +1370,7 @@ template <typename T, int NumDims, int Layout> static void test_assign_to_tensor_strided_slice() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> tensor(dims); TensorMap<Tensor<T, NumDims, Layout>> map(tensor.data(), dims); @@ -1417,7 +1407,7 @@ // writeBlock a materialized block. template <typename T, int NumDims, int Layout> static void test_assign_expr_to_tensor_strided_slice() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(10, 20); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(10, 20); Tensor<T, NumDims, Layout> tensor(dims); TensorMap<Tensor<T, NumDims, Layout>> map(tensor.data(), dims); @@ -1462,7 +1452,7 @@ template <typename T, int NumDims, int Layout> static void test_assign_to_tensor_reverse() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(5, 15); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(5, 15); Tensor<T, NumDims, Layout> tensor(dims); TensorMap<Tensor<T, NumDims, Layout>> map(tensor.data(), dims); @@ -1484,7 +1474,7 @@ template <typename T, int NumDims, int Layout> static void test_assign_to_tensor_roll() { - DSizes<Index, NumDims> dims = RandomDims<NumDims>(5, 15); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(5, 15); Tensor<T, NumDims, Layout> tensor(dims); TensorMap<Tensor<T, NumDims, Layout>> map(tensor.data(), dims); @@ -1508,7 +1498,7 @@ // scalar reference instead. This is also the only place the RowMajor lvalue // evaluator is instantiated: its coeffRef/writePacket are ColMajor-only, so // a slice-assignment reference would not compile. - DSizes<Index, NumDims> dims = RandomDims<NumDims>(5, 15); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(5, 15); const int axis = internal::random<int>(0, NumDims - 1); const Index left_axis_size = internal::random<Index>(1, dims[axis] - 1);
diff --git a/contrib/test/tensor_block_io.cpp b/contrib/test/tensor_block_io.cpp index 5a3efb6..87ad7e7 100644 --- a/contrib/test/tensor_block_io.cpp +++ b/contrib/test/tensor_block_io.cpp
@@ -10,30 +10,12 @@ // clang-format off #include "main.h" #include <Eigen/Tensor> +#include "tensor_test_helpers.h" // clang-format on // -------------------------------------------------------------------------- // // A set of tests for TensorBlockIO: copying data between tensor blocks. -template <int NumDims> -static DSizes<Index, NumDims> RandomDims(Index min, Index max) { - DSizes<Index, NumDims> dims; - for (int i = 0; i < NumDims; ++i) { - dims[i] = internal::random<Index>(min, max); - } - return DSizes<Index, NumDims>(dims); -} - -static internal::TensorBlockShapeType RandomBlockShape() { - return internal::random<bool>() ? internal::TensorBlockShapeType::kUniformAllDims - : internal::TensorBlockShapeType::kSkewedInnerDims; -} - -template <int NumDims> -static size_t RandomTargetBlockSize(const DSizes<Index, NumDims>& dims) { - return internal::random<size_t>(1, dims.TotalSize()); -} - template <int Layout, int NumDims> static Index GetInputIndex(Index output_index, const array<Index, NumDims>& output_to_input_dim_map, const array<Index, NumDims>& input_strides, const array<Index, NumDims>& output_strides) { @@ -62,7 +44,7 @@ using IOSrc = typename TensorBlockIO::Src; // Generate a random input Tensor. - DSizes<Index, NumDims> dims = RandomDims<NumDims>(1, 30); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(1, 30); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -71,7 +53,7 @@ // Construct a tensor block mapper. using TensorBlockMapper = internal::TensorBlockMapper<NumDims, Layout, Index>; - TensorBlockMapper block_mapper(dims, {RandomBlockShape(), RandomTargetBlockSize(dims), {0, 0, 0}}); + TensorBlockMapper block_mapper(dims, {randomTensorBlockShape(), randomTensorBlockSize(dims), {0, 0, 0}}); // We will copy data from input to output through this buffer. Tensor<T, NumDims, Layout> block(block_mapper.blockDimensions()); @@ -115,7 +97,7 @@ template <typename T, int NumDims, int Layout> static void test_block_io_copy_using_reordered_dimensions() { // Generate a random input Tensor. - DSizes<Index, NumDims> dims = RandomDims<NumDims>(1, 30); + DSizes<Index, NumDims> dims = randomTensorDims<NumDims>(1, 30); Tensor<T, NumDims, Layout> input(dims); input.setRandom(); @@ -141,7 +123,7 @@ // NOTE: Tensor block mapper works with shuffled dimensions. using TensorBlockMapper = internal::TensorBlockMapper<NumDims, Layout, Index>; TensorBlockMapper block_mapper(output_tensor_dims, - {RandomBlockShape(), RandomTargetBlockSize(output_tensor_dims), {0, 0, 0}}); + {randomTensorBlockShape(), randomTensorBlockSize(output_tensor_dims), {0, 0, 0}}); // We will copy data from input to output through this buffer. Tensor<T, NumDims, Layout> block(block_mapper.blockDimensions()); @@ -302,7 +284,7 @@ template <int Layout> static void test_block_io_zero_stride() { - DSizes<Index, 5> rnd_dims = RandomDims<5>(1, 30); + DSizes<Index, 5> rnd_dims = randomTensorDims<5>(1, 30); DSizes<Index, 5> input_tensor_dims = rnd_dims; input_tensor_dims[0] = 1;
diff --git a/contrib/test/tensor_cast_float16_gpu.cu b/contrib/test/tensor_cast_float16_gpu.cu index 530eb48..bf1ab2a 100644 --- a/contrib/test/tensor_cast_float16_gpu.cu +++ b/contrib/test/tensor_cast_float16_gpu.cu
@@ -16,43 +16,32 @@ #include "main.h" #include <contrib/Eigen/Tensor> +#include "tensor_gpu_test_helpers.h" using Eigen::Tensor; void test_gpu_conversion() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); + tensor_test::GpuTensorTestFixture fixture; int num_elem = 101; Tensor<float, 1> floats(num_elem); floats.setRandom(); - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::half* d_half = (Eigen::half*)gpu_device.allocate(num_elem * sizeof(Eigen::half)); - float* d_conv = (float*)gpu_device.allocate(num_elem * sizeof(float)); + auto gpu_float = fixture.tensor<float, 1, Eigen::Aligned>(num_elem); + auto gpu_half = fixture.tensor<Eigen::half, 1, Eigen::Aligned>(num_elem); + auto gpu_conv = fixture.tensor<float, 1, Eigen::Aligned>(num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float(d_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_half(d_half, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_conv(d_conv, num_elem); + fixture.device().memcpyHostToDevice(gpu_float.data(), floats.data(), num_elem * sizeof(float)); - gpu_device.memcpyHostToDevice(d_float, floats.data(), num_elem * sizeof(float)); + gpu_half.device(fixture.device()) = gpu_float.cast<Eigen::half>(); + gpu_conv.device(fixture.device()) = gpu_half.cast<float>(); - gpu_half.device(gpu_device) = gpu_float.cast<Eigen::half>(); - gpu_conv.device(gpu_device) = gpu_half.cast<float>(); - - Tensor<float, 1> initial(num_elem); - Tensor<float, 1> final(num_elem); - gpu_device.memcpyDeviceToHost(initial.data(), d_float, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(final.data(), d_conv, num_elem * sizeof(float)); - gpu_device.synchronize(); + const auto initial = fixture.toHost(gpu_float); + const auto final = fixture.toHost(gpu_conv); for (int i = 0; i < num_elem; ++i) { VERIFY_IS_APPROX(initial(i), final(i)); } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_half); - gpu_device.deallocate(d_conv); } void test_fallback_conversion() {
diff --git a/contrib/test/tensor_chipping.cpp b/contrib/test/tensor_chipping.cpp index 6e84c79..68382aa 100644 --- a/contrib/test/tensor_chipping.cpp +++ b/contrib/test/tensor_chipping.cpp
@@ -12,6 +12,10 @@ #include <Eigen/Tensor> +#if defined __GNUC__ && __GNUC__ >= 6 +#pragma GCC diagnostic ignored "-Wignored-attributes" +#endif + using Eigen::Tensor; template <int DataLayout> @@ -443,9 +447,315 @@ test_runtime_axis_block_chip<4, Layout, 3>(); } +template <typename Scalar, int Layout> +void test_chip_packet_boundary_for_stride(Index inner_dim) { + const Index chip_axis = (Layout == ColMajor) ? 1 : 2; + DSizes<Index, 4> dims; + if (Layout == ColMajor) { + dims = DSizes<Index, 4>(inner_dim, 3, 5, 7); + } else { + dims = DSizes<Index, 4>(7, 5, 3, inner_dim); + } + + Tensor<Scalar, 4, Layout> tensor(dims); + for (Index i = 0; i < tensor.size(); ++i) { + tensor.data()[i] = static_cast<Scalar>((i % 29) - 14); + } + + DefaultDevice device; + for (Index chip_offset = 0; chip_offset < 3; ++chip_offset) { + // 1. Raw storage evaluator (m_impl.data() != nullptr). + auto raw_chip_op = tensor.chip(chip_offset, chip_axis); + TensorEvaluator<const decltype(raw_chip_op), DefaultDevice> raw_eval(raw_chip_op, device); + const int packet_size = TensorEvaluator<const decltype(raw_chip_op), DefaultDevice>::PacketSize; + const Index total_size = raw_eval.dimensions().TotalSize(); + + EIGEN_ALIGN_MAX Scalar unpacked[64]; + for (Index idx = 0; idx + packet_size <= total_size; ++idx) { + auto pkt = raw_eval.template packet<Unaligned>(idx); + internal::pstore<Scalar>(unpacked, pkt); + for (int lane = 0; lane < packet_size; ++lane) { + VERIFY_IS_EQUAL(unpacked[lane], raw_eval.coeff(idx + lane)); + } + } + + // 2. Non-raw nested expression evaluator (m_impl.data() == nullptr) - exercises division-free stepping. + auto expr_chip_op = (tensor + tensor).chip(chip_offset, chip_axis); + TensorEvaluator<const decltype(expr_chip_op), DefaultDevice> expr_eval(expr_chip_op, device); + expr_eval.evalSubExprsIfNeeded(nullptr); + for (Index idx = 0; idx + packet_size <= total_size; ++idx) { + auto pkt = expr_eval.template packet<Unaligned>(idx); + internal::pstore<Scalar>(unpacked, pkt); + for (int lane = 0; lane < packet_size; ++lane) { + VERIFY_IS_EQUAL(unpacked[lane], expr_eval.coeff(idx + lane)); + } + } + expr_eval.cleanup(); + + // 3. Lvalue writePacket across stride boundaries. + for (Index start_phase = 0; start_phase < numext::mini<Index>(packet_size, total_size); ++start_phase) { + Tensor<Scalar, 4, Layout> dst = tensor; + Tensor<Scalar, 4, Layout> expected = tensor; + auto lval_op = dst.chip(chip_offset, chip_axis); + TensorEvaluator<decltype(lval_op), DefaultDevice> lval_eval(lval_op, device); + auto ref_op = expected.chip(chip_offset, chip_axis); + TensorEvaluator<decltype(ref_op), DefaultDevice> ref_eval(ref_op, device); + + for (Index idx = start_phase; idx + packet_size <= total_size; idx += packet_size) { + for (int lane = 0; lane < packet_size; ++lane) { + unpacked[lane] = static_cast<Scalar>(((idx + lane) % 19) + 3); + ref_eval.coeffRef(idx + lane) = unpacked[lane]; + } + auto pkt = + internal::pload<typename TensorEvaluator<decltype(lval_op), DefaultDevice>::PacketReturnType>(unpacked); + lval_eval.template writePacket<Unaligned>(idx, pkt); + } + for (Index i = 0; i < dst.size(); ++i) { + VERIFY_IS_EQUAL(dst.data()[i], expected.data()[i]); + } + } + } +} + +template <int Layout> +void test_chip_packet_boundaries() { + for (Index stride : {Index(2), Index(3), Index(4), Index(6), Index(8), Index(12), Index(18), Index(24)}) { + test_chip_packet_boundary_for_stride<float, Layout>(stride); + test_chip_packet_boundary_for_stride<double, Layout>(stride); + test_chip_packet_boundary_for_stride<Eigen::half, Layout>(stride); + test_chip_packet_boundary_for_stride<Eigen::bfloat16, Layout>(stride); + test_chip_packet_boundary_for_stride<std::complex<float>, Layout>(stride); + } +} + +template <typename EvalT, typename PacketT, + std::enable_if_t<internal::unpacket_traits<PacketT>::masked_load_available, int> = 0> +void verify_partial_packets_for_eval(const EvalT& eval) { + using Scalar = typename internal::unpacket_traits<PacketT>::type; + using MaskT = typename internal::unpacket_traits<PacketT>::mask_t; + constexpr int kPacketSize = internal::unpacket_traits<PacketT>::size; + const Index total_size = eval.dimensions().TotalSize(); + EIGEN_ALIGN_MAX Scalar unpacked[64]; + + const MaskT full_mask = kPacketSize < static_cast<int>(CHAR_BIT * sizeof(MaskT)) + ? static_cast<MaskT>((MaskT(1) << kPacketSize) - 1) + : (std::numeric_limits<MaskT>::max)(); + const MaskT sparse_mask = static_cast<MaskT>(0xA5A5u) & full_mask; + const MaskT upper_half_mask = static_cast<MaskT>(full_mask ^ ((MaskT(1) << (kPacketSize / 2)) - 1)); + + // Zero mask never touches underlying memory, even at boundary indices. + { + PacketT zero_pkt = eval.template partialPacket<PacketT>(0, MaskT(0)); + internal::pstore<Scalar>(unpacked, zero_pkt); + for (int lane = 0; lane < kPacketSize; ++lane) { + VERIFY_IS_EQUAL(unpacked[lane], Scalar(0)); + } + } + + // Negative starting index with upper lanes active (exercised by spatial convolution mappers). + if (total_size >= kPacketSize / 2) { + const Index neg_idx = -(kPacketSize / 2); + PacketT pkt = eval.template partialPacket<PacketT>(neg_idx, upper_half_mask); + internal::pstore<Scalar>(unpacked, pkt); + for (int lane = 0; lane < kPacketSize; ++lane) { + const Scalar expected = ((upper_half_mask >> lane) & MaskT(1)) ? eval.coeff(neg_idx + lane) : Scalar(0); + VERIFY_IS_EQUAL(unpacked[lane], expected); + } + } + + // Sweep contiguous prefix/suffix masks and non-contiguous masks across the chipped tensor. + for (Index idx = 0; idx < total_size; ++idx) { + const int avail = static_cast<int>(numext::mini<Index>(kPacketSize, total_size - idx)); + const MaskT prefix_mask = + avail == static_cast<int>(CHAR_BIT * sizeof(MaskT)) ? full_mask : static_cast<MaskT>((MaskT(1) << avail) - 1); + const MaskT masks_to_check[2] = {prefix_mask, static_cast<MaskT>(sparse_mask & prefix_mask)}; + for (MaskT m : masks_to_check) { + PacketT pkt = eval.template partialPacket<PacketT>(idx, m); + internal::pstore<Scalar>(unpacked, pkt); + for (int lane = 0; lane < kPacketSize; ++lane) { + const Scalar expected = ((m >> lane) & MaskT(1)) ? eval.coeff(idx + lane) : Scalar(0); + VERIFY_IS_EQUAL(unpacked[lane], expected); + } + } + } +} + +template <typename EvalT, typename PacketT, + std::enable_if_t<!internal::unpacket_traits<PacketT>::masked_load_available, int> = 0> +void verify_partial_packets_for_eval(const EvalT&) {} + +template <typename Scalar, int Layout> +void test_chip_partial_packets_for_type() { + using Packet = typename internal::packet_traits<Scalar>::type; + using HalfPacket = typename internal::unpacket_traits<Packet>::half; + DefaultDevice device; + + for (Index inner_dim : {Index(3), Index(4), Index(8), Index(12)}) { + DSizes<Index, 4> dims = + (Layout == ColMajor) ? DSizes<Index, 4>(inner_dim, 3, 5, 2) : DSizes<Index, 4>(2, 5, 3, inner_dim); + Tensor<Scalar, 4, Layout> tensor(dims); + for (Index i = 0; i < tensor.size(); ++i) { + tensor.data()[i] = static_cast<Scalar>((i % 31) + 1); + } + + for (int axis = 0; axis < 4; ++axis) { + const Index extent = tensor.dimension(axis); + for (Index offset = 0; offset < numext::mini<Index>(extent, 2); ++offset) { + auto raw_op = tensor.chip(offset, axis); + TensorEvaluator<const decltype(raw_op), DefaultDevice> raw_eval(raw_op, device); + verify_partial_packets_for_eval<decltype(raw_eval), Packet>(raw_eval); + if (!std::is_same<HalfPacket, Packet>::value) { + verify_partial_packets_for_eval<decltype(raw_eval), HalfPacket>(raw_eval); + } + + auto expr_op = (tensor + tensor).chip(offset, axis); + TensorEvaluator<const decltype(expr_op), DefaultDevice> expr_eval(expr_op, device); + expr_eval.evalSubExprsIfNeeded(nullptr); + verify_partial_packets_for_eval<decltype(expr_eval), Packet>(expr_eval); + expr_eval.cleanup(); + } + } + + // Nested middle-dimension chipping. + auto nested_op = tensor.chip(1, Layout == ColMajor ? 2 : 1).chip(1, 1); + TensorEvaluator<const decltype(nested_op), DefaultDevice> nested_eval(nested_op, device); + verify_partial_packets_for_eval<decltype(nested_eval), Packet>(nested_eval); + } +} + +template <int Layout> +void test_chip_partial_packets() { + test_chip_partial_packets_for_type<float, Layout>(); + test_chip_partial_packets_for_type<double, Layout>(); +} + +template <int Layout> +void test_chip_trivial_size_one_dim() { + DefaultDevice device; + for (int singleton_axis = 0; singleton_axis < 4; ++singleton_axis) { + DSizes<Index, 4> dims(5, 3, 7, 4); + dims[singleton_axis] = 1; + Tensor<float, 4, Layout> src(dims); + for (Index i = 0; i < src.size(); ++i) { + src.data()[i] = static_cast<float>(i + 1); + } + + auto chip_op = src.chip(0, singleton_axis); + TensorEvaluator<const decltype(chip_op), DefaultDevice> eval(chip_op, device); + VERIFY(eval.data() == src.data()); + + Tensor<float, 3, Layout> out = chip_op; + VERIFY_IS_EQUAL(out.size(), src.size()); + for (Index i = 0; i < out.size(); ++i) { + VERIFY_IS_EQUAL(out.data()[i], src.data()[i]); + } + + Tensor<float, 4, Layout> dst(dims); + dst.setZero(); + dst.chip(0, singleton_axis) = out * out.constant(2.0f); + for (Index i = 0; i < dst.size(); ++i) { + VERIFY_IS_EQUAL(dst.data()[i], 2.0f * src.data()[i]); + } + } + + // Static inner axis with runtime extent 1 is also contiguous. + DSizes<Index, 3> static_dims = (Layout == ColMajor) ? DSizes<Index, 3>(1, 9, 5) : DSizes<Index, 3>(5, 9, 1); + Tensor<float, 3, Layout> static_src(static_dims); + static_src.setRandom(); + constexpr int kInnerAxis = (Layout == ColMajor) ? 0 : 2; + auto static_chip = static_src.template chip<kInnerAxis>(0); + TensorEvaluator<const decltype(static_chip), DefaultDevice> static_eval(static_chip, device); + VERIFY(static_eval.data() == static_src.data()); + Tensor<float, 2, Layout> static_out = static_chip + static_chip; + for (Index i = 0; i < static_src.size(); ++i) { + VERIFY_IS_APPROX(static_out.data()[i], 2.0f * static_src.data()[i]); + } +} + +template <typename EvalT, typename SubPacket> +void verify_subpackets_for_eval(const EvalT& eval) { + using Scalar = std::remove_const_t<typename EvalT::CoeffReturnType>; + constexpr int kSubSize = internal::unpacket_traits<SubPacket>::size; + const Index total_size = eval.dimensions().TotalSize(); + if (total_size < kSubSize) return; + + EIGEN_ALIGN_TO_BOUNDARY(internal::unpacket_traits<SubPacket>::alignment) Scalar unpacked[kSubSize]; + for (Index idx = 0; idx + kSubSize <= total_size; ++idx) { + SubPacket pkt = eval.template packet<Unaligned, SubPacket>(idx); + internal::pstore<Scalar>(unpacked, pkt); + for (int lane = 0; lane < kSubSize; ++lane) { + VERIFY_IS_EQUAL(unpacked[lane], eval.coeff(idx + lane)); + } + } +} + +template <typename Scalar, int Layout> +void test_chip_subpackets_and_empty_for_type() { + using Packet = typename internal::packet_traits<Scalar>::type; + using HalfPacket = typename internal::unpacket_traits<Packet>::half; + using QuarterPacket = typename internal::unpacket_traits<HalfPacket>::half; + DefaultDevice device; + + // Zero-sized unchipped dimension exercises m_stride == 0 initialization. + DSizes<Index, 3> empty_dims = (Layout == ColMajor) ? DSizes<Index, 3>(0, 4, 5) : DSizes<Index, 3>(5, 4, 0); + Tensor<Scalar, 3, Layout> empty_src(empty_dims); + Tensor<Scalar, 2, Layout> empty_out = empty_src.chip(1, 1); + VERIFY_IS_EQUAL(empty_out.size(), Index(0)); + + for (Index inner_dim : {Index(3), Index(4), Index(8), Index(12)}) { + DSizes<Index, 4> dims = + (Layout == ColMajor) ? DSizes<Index, 4>(inner_dim, 3, 5, 2) : DSizes<Index, 4>(2, 5, 3, inner_dim); + Tensor<Scalar, 4, Layout> tensor(dims); + for (Index i = 0; i < tensor.size(); ++i) { + tensor.data()[i] = static_cast<Scalar>((i % 29) + 1); + } + + for (int axis = 0; axis < 4; ++axis) { + auto raw_op = tensor.chip(0, axis); + TensorEvaluator<const decltype(raw_op), DefaultDevice> raw_eval(raw_op, device); + verify_subpackets_for_eval<decltype(raw_eval), Packet>(raw_eval); + if (!std::is_same<HalfPacket, Packet>::value) { + verify_subpackets_for_eval<decltype(raw_eval), HalfPacket>(raw_eval); + } + if (!std::is_same<QuarterPacket, HalfPacket>::value) { + verify_subpackets_for_eval<decltype(raw_eval), QuarterPacket>(raw_eval); + } + + auto expr_op = (tensor + tensor).chip(0, axis); + TensorEvaluator<const decltype(expr_op), DefaultDevice> expr_eval(expr_op, device); + expr_eval.evalSubExprsIfNeeded(nullptr); + if (!std::is_same<HalfPacket, Packet>::value) { + verify_subpackets_for_eval<decltype(expr_eval), HalfPacket>(expr_eval); + } + expr_eval.cleanup(); + } + + auto nested_op = tensor.chip(1, Layout == ColMajor ? 2 : 1).chip(1, 1); + TensorEvaluator<const decltype(nested_op), DefaultDevice> nested_eval(nested_op, device); + if (!std::is_same<HalfPacket, Packet>::value) { + verify_subpackets_for_eval<decltype(nested_eval), HalfPacket>(nested_eval); + } + } +} + +template <int Layout> +void test_chip_subpackets_and_empty() { + test_chip_subpackets_and_empty_for_type<float, Layout>(); + test_chip_subpackets_and_empty_for_type<double, Layout>(); + test_chip_subpackets_and_empty_for_type<int, Layout>(); +} + EIGEN_DECLARE_TEST(tensor_chipping) { CALL_SUBTEST(test_runtime_axis_block_chips<ColMajor>()); CALL_SUBTEST(test_runtime_axis_block_chips<RowMajor>()); + CALL_SUBTEST(test_chip_packet_boundaries<ColMajor>()); + CALL_SUBTEST(test_chip_packet_boundaries<RowMajor>()); + CALL_SUBTEST(test_chip_partial_packets<ColMajor>()); + CALL_SUBTEST(test_chip_partial_packets<RowMajor>()); + CALL_SUBTEST(test_chip_subpackets_and_empty<ColMajor>()); + CALL_SUBTEST(test_chip_subpackets_and_empty<RowMajor>()); + CALL_SUBTEST(test_chip_trivial_size_one_dim<ColMajor>()); + CALL_SUBTEST(test_chip_trivial_size_one_dim<RowMajor>()); CALL_SUBTEST(test_simple_chip<ColMajor>()); CALL_SUBTEST(test_simple_chip<RowMajor>()); CALL_SUBTEST(test_dynamic_chip<ColMajor>());
diff --git a/contrib/test/tensor_contraction.cpp b/contrib/test/tensor_contraction.cpp index e6c2d33..2e27a62 100644 --- a/contrib/test/tensor_contraction.cpp +++ b/contrib/test/tensor_contraction.cpp
@@ -8,9 +8,14 @@ // with this file, You can obtain one at http://mozilla.org/MPL/2.0/. // SPDX-License-Identifier: MPL-2.0 +#define EIGEN_TEST_ANNOYING_SCALAR_DONT_THROW + #include "main.h" #include <Eigen/Tensor> +#include <contrib/Eigen/AutoDiff> + +#include "AnnoyingScalar.h" using Eigen::DefaultDevice; using Eigen::Tensor; @@ -292,6 +297,44 @@ mat1(0, 1, 1) * mat2(1, 1, 0) + mat1(1, 1, 1) * mat2(1, 1, 1)); } +template <int DataLayout, int ContractDims> +static void check_dispatch_pruning_for_rank() { + using Tensor3 = Tensor<float, 3, DataLayout>; + using Contraction = TensorContractionOp<const Eigen::array<DimPair, ContractDims>, const Tensor3, const Tensor3>; + using Evaluator = TensorEvaluator<const Contraction, DefaultDevice>; + + for (int lhs_c_int = 0; lhs_c_int < 2; ++lhs_c_int) { + for (int rhs_c_int = 0; rhs_c_int < 2; ++rhs_c_int) { + for (int rhs_r_int = 0; rhs_r_int < 2; ++rhs_r_int) { + const bool lhs_c_in = lhs_c_int != 0; + const bool rhs_c_in = rhs_c_int != 0; + const bool rhs_r_in = rhs_r_int != 0; + bool dispatched_lhs_c = false; + bool dispatched_rhs_c = false; + bool dispatched_rhs_r = false; + Evaluator::dispatchContraction( + [&](auto lhs_c, auto rhs_c, auto rhs_r) { + static_assert(!rhs_r() || (ContractDims > 1 && !rhs_c()), "unreachable contraction instantiated"); + dispatched_lhs_c = lhs_c(); + dispatched_rhs_c = rhs_c(); + dispatched_rhs_r = rhs_r(); + }, + lhs_c_in, rhs_c_in, rhs_r_in); + VERIFY_IS_EQUAL(dispatched_lhs_c, lhs_c_in); + VERIFY_IS_EQUAL(dispatched_rhs_c, rhs_c_in); + VERIFY_IS_EQUAL(dispatched_rhs_r, (ContractDims > 1) && !rhs_c_in && rhs_r_in); + } + } + } +} + +template <int DataLayout> +static void test_dispatch_pruning() { + check_dispatch_pruning_for_rank<DataLayout, 0>(); + check_dispatch_pruning_for_rank<DataLayout, 1>(); + check_dispatch_pruning_for_rank<DataLayout, 2>(); +} + template <int DataLayout> static void test_consistency() { // this does something like testing (A*B)^T = (B^T * A^T) @@ -938,11 +981,75 @@ } } +// Issue #2714: the packed blocks and the temporary result of a contraction hold scalars that require +// initialization; they must be constructed before the packers and gebp assign into them, and destructed afterwards. +template <int DataLayout> +static void test_scalar_require_initialization() { + Tensor<float, 2, DataLayout> left(37, 53); + Tensor<float, 2, DataLayout> right(53, 41); + Tensor<float, 2, DataLayout> vec(53, 1); + left.setRandom(); + right.setRandom(); + vec.setRandom(); + Eigen::array<DimPair, 1> dims = {{DimPair(1, 0)}}; + const Tensor<float, 2, DataLayout> expected = left.contract(right, dims); + const Tensor<float, 2, DataLayout> expected_gemv = left.contract(vec, dims); + + AnnoyingScalar::instances = 0; + { + const Tensor<AnnoyingScalar, 2, DataLayout> a_left = left.template cast<AnnoyingScalar>(); + const Tensor<AnnoyingScalar, 2, DataLayout> a_right = right.template cast<AnnoyingScalar>(); + const Tensor<AnnoyingScalar, 2, DataLayout> a_vec = vec.template cast<AnnoyingScalar>(); + Tensor<AnnoyingScalar, 2, DataLayout> result = a_left.contract(a_right, dims); + Tensor<float, 2, DataLayout> actual = result.template cast<float>(); + for (Index i = 0; i < actual.size(); ++i) VERIFY_IS_APPROX(actual.data()[i], expected.data()[i]); + + // A contraction nested in another expression evaluates into a temporary result buffer. + result = a_left.contract(a_right, dims) + a_left.contract(a_right, dims); + actual = result.template cast<float>(); + for (Index i = 0; i < actual.size(); ++i) VERIFY_IS_APPROX(actual.data()[i], 2 * expected.data()[i]); + + Tensor<AnnoyingScalar, 2, DataLayout> result_gemv = a_left.contract(a_vec, dims) + a_left.contract(a_vec, dims); + actual = result_gemv.template cast<float>(); + for (Index i = 0; i < actual.size(); ++i) VERIFY_IS_APPROX(actual.data()[i], 2 * expected_gemv.data()[i]); + } + VERIFY_IS_EQUAL(AnnoyingScalar::instances, 0); +} + +// The reproducer from issue #2714. +template <typename = void> +static void test_autodiff_contraction() { + using AD = AutoDiffScalar<VectorXf>; + Tensor<AD, 2> x(2, 2); + Tensor<AD, 2> w(2, 2); + for (int i = 0; i < 4; ++i) { + x.data()[i] = AD(1.0f + i, 8, i); + w.data()[i] = AD(5.0f + i, 8, 4 + i); + } + Eigen::array<DimPair, 1> dims = {{DimPair(1, 0)}}; + const Tensor<AD, 2> z = x.contract(w, dims); + for (Index i = 0; i < 2; ++i) { + for (Index j = 0; j < 2; ++j) { + VectorXf derivatives = VectorXf::Zero(8); + float value = 0; + for (Index l = 0; l < 2; ++l) { + value += x(i, l).value() * w(l, j).value(); + derivatives += x(i, l).value() * w(l, j).derivatives() + w(l, j).value() * x(i, l).derivatives(); + } + VERIFY_IS_APPROX(z(i, j).value(), value); + VERIFY_IS_APPROX(z(i, j).derivatives(), derivatives); + } + } +} + EIGEN_DECLARE_TEST(tensor_contraction) { CALL_SUBTEST_1(test_evals<ColMajor>()); CALL_SUBTEST_1(test_evals<RowMajor>()); CALL_SUBTEST_1(test_scalar<ColMajor>()); CALL_SUBTEST_1(test_scalar<RowMajor>()); + CALL_SUBTEST_1(test_scalar_require_initialization<ColMajor>()); + CALL_SUBTEST_1(test_scalar_require_initialization<RowMajor>()); + CALL_SUBTEST_1(test_autodiff_contraction<>()); CALL_SUBTEST_2(test_multidims<ColMajor>()); CALL_SUBTEST_2(test_multidims<RowMajor>()); CALL_SUBTEST_2(test_holes<ColMajor>()); @@ -955,6 +1062,8 @@ CALL_SUBTEST_4(test_expr<RowMajor>()); CALL_SUBTEST_4(test_out_of_order_contraction<ColMajor>()); CALL_SUBTEST_4(test_out_of_order_contraction<RowMajor>()); + CALL_SUBTEST_4(test_dispatch_pruning<ColMajor>()); + CALL_SUBTEST_4(test_dispatch_pruning<RowMajor>()); CALL_SUBTEST_5(test_consistency<ColMajor>()); CALL_SUBTEST_5(test_consistency<RowMajor>()); CALL_SUBTEST_5(test_large_contraction<ColMajor>());
diff --git a/contrib/test/tensor_convolution_large_gpu.cu b/contrib/test/tensor_convolution_large_gpu.cu new file mode 100644 index 0000000..a70d7ff --- /dev/null +++ b/contrib/test/tensor_convolution_large_gpu.cu
@@ -0,0 +1,108 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#define EIGEN_USE_GPU +#define EIGEN_TEST_NO_LONGDOUBLE +#define EIGEN_TEST_NO_COMPLEX +#include "main.h" +#include <contrib/Eigen/Tensor> + +template <int Layout, int NumDims, typename TensorIndex> +void verify_gpu_convolution(const Tensor<float, NumDims, Layout, TensorIndex>& input, + const Tensor<float, 3, Layout, TensorIndex>& kernel, const array<TensorIndex, 3>& axes, + const Tensor<float, NumDims, Layout, TensorIndex>& expected) { + GpuStreamDevice stream; + GpuDevice device(&stream); + bool nonzero = false, varying = false; + for (Index i = 0; i < expected.size(); ++i) { + nonzero |= expected.data()[i] != 0; + varying |= expected.data()[i] != expected.data()[0]; + } + VERIFY(nonzero); + VERIFY(varying); + Tensor<float, NumDims, Layout, TensorIndex> result; + result.resize(expected.dimensions()); + float* a = static_cast<float*>(device.allocate(input.size() * sizeof(float))); + float* k = static_cast<float*>(device.allocate(kernel.size() * sizeof(float))); + float* out = static_cast<float*>(device.allocate(result.size() * sizeof(float))); + device.memcpyHostToDevice(a, input.data(), input.size() * sizeof(float)); + device.memcpyHostToDevice(k, kernel.data(), kernel.size() * sizeof(float)); + // NaN sentinels make missing output writes fail even when the reference is zero. + device.memset(out, 0xff, result.size() * sizeof(float)); + TensorMap<Tensor<float, NumDims, Layout, TensorIndex>> gpu_input(a, input.dimensions()); + TensorMap<Tensor<float, 3, Layout, TensorIndex>> gpu_kernel(k, kernel.dimensions()); + TensorMap<Tensor<float, NumDims, Layout, TensorIndex>> gpu_result(out, result.dimensions()); + gpu_result.device(device) = gpu_input.convolve(gpu_kernel, axes); + device.memcpyDeviceToHost(result.data(), out, result.size() * sizeof(float)); + device.synchronize(); + for (Index i = 0; i < result.size(); ++i) VERIFY_IS_EQUAL(result.data()[i], expected.data()[i]); + device.deallocate(a); + device.deallocate(k); + device.deallocate(out); +} + +template <int Layout> +void check_large_convolution(const array<Index, 3>& kernel_dims, const array<Index, 3>& axes) { + array<Index, 5> input_dims{{2, 3, 2, 3, 2}}; + for (int i = 0; i < 3; ++i) input_dims[axes[i]] = kernel_dims[i] + i + 1; + Tensor<float, 5, Layout> input(input_dims); + Tensor<float, 3, Layout> kernel(kernel_dims); + // Small integers keep every product and partial sum exactly representable in float. + for (Index i = 0; i < input.size(); ++i) input.data()[i] = float((i * 17 + i / 7) % 19 - 9); + for (Index i = 0; i < kernel.size(); ++i) kernel.data()[i] = float((i * 7 + i / 5) % 7 - 2); + Tensor<float, 5, Layout> expected = input.convolve(kernel, axes); + verify_gpu_convolution(input, kernel, axes, expected); +} + +template <int Layout, typename TensorIndex = Index> +void check_convolution_grid_stride(Index output_size = 0) { + GpuStreamDevice stream; + GpuDevice device(&stream); + const Index kernel_size = device.sharedMemPerBlock() / sizeof(float) + 1; + const Index threads = numext::mini(256, device.maxGpuThreadsPerBlock()); + const Index grid_stride = Index(device.getNumGpuMultiProcessors()) * 32 * threads; + if (output_size == 0) output_size = 2 * grid_stride + 17; + const Index max_input = + static_cast<Index>(numext::mini<int64_t>(NumTraits<Index>::highest(), NumTraits<TensorIndex>::highest())); + const Index max_output = max_input - kernel_size + 1; + // Some devices cannot fit an oversized convolution's input in this narrow index type. + if (max_output < 2) return; + output_size = numext::mini(output_size, max_output); + const array<TensorIndex, 3> axes{{0, 1, 2}}; + Tensor<float, 3, Layout, TensorIndex> input(1, 1, TensorIndex(kernel_size + output_size - 1)); + Tensor<float, 3, Layout, TensorIndex> kernel(1, 1, TensorIndex(kernel_size)); + Tensor<float, 3, Layout, TensorIndex> expected(1, 1, TensorIndex(output_size)); + input.setConstant(1); + input.data()[kernel_size + output_size / 2 - 1] = 2; + kernel.setConstant(1); + // An independent sliding sum avoids a CPU convolution of O(kernel_size * output_size). + Index sum = kernel_size; + for (Index i = 0; i < output_size; ++i) { + expected.data()[i] = float(sum); + if (i + 1 < output_size) sum += Index(input.data()[i + kernel_size]) - Index(input.data()[i]); + } + if (std::is_same<TensorIndex, Index>::value) VERIFY(output_size > grid_stride); + verify_gpu_convolution(input, kernel, axes, expected); +} + +EIGEN_DECLARE_TEST(tensor_convolution_large_gpu) { + GpuStreamDevice stream; + GpuDevice device(&stream); + // Always exceed this device's shared-memory limit, including newer GPUs. + const Index oversized = device.sharedMemPerBlock() / (sizeof(float) * 3 * 50) + 1; + const Index capacity = device.sharedMemPerBlock() / sizeof(float); + const array<Index, 3> shapes[] = {{{3, 4, 2}}, {{3, 50, 120}}, {{3, 50, 145}}, + {{3, 50, oversized}}, {{1, 1, capacity - 1}}, {{1, 1, capacity}}, + {{1, 1, capacity + 1}}}; + const array<Index, 3> axes[] = {{{1, 2, 3}}, {{0, 2, 4}}, {{4, 0, 2}}}; + for (const auto& shape : shapes) { + for (const auto& axis : axes) { + CALL_SUBTEST(check_large_convolution<ColMajor>(shape, axis)); + CALL_SUBTEST(check_large_convolution<RowMajor>(shape, axis)); + } + } + CALL_SUBTEST(check_convolution_grid_stride<ColMajor>()); + CALL_SUBTEST(check_convolution_grid_stride<RowMajor>()); + CALL_SUBTEST((check_convolution_grid_stride<ColMajor, int16_t>(18000))); + CALL_SUBTEST((check_convolution_grid_stride<RowMajor, int16_t>(18000))); +}
diff --git a/contrib/test/tensor_dimension.cpp b/contrib/test/tensor_dimension.cpp index 60f58f3..d95db1e 100644 --- a/contrib/test/tensor_dimension.cpp +++ b/contrib/test/tensor_dimension.cpp
@@ -134,6 +134,36 @@ tensor_rank_layout_odr_use<TensorRef<TensorFixedSize<float, Sizes<>>>>(0, ColMajor); } +// Test the accessors on actual tensors/maps/refs +// Check the dynamic assessors for both in- and out-of-range +// Only test static for in-range here (out-of-range causes a compilation failure) +static void test_dimension_accessors() { + Tensor<float, 3> t(2, 3, 7); + Eigen::TensorFixedSize<float, Eigen::Sizes<2, 3, 7>> f; + Eigen::TensorMap<Tensor<float, 3>> m(t.data(), 2, 3, 7); + Eigen::TensorRef<Tensor<float, 3>> r(t); + + VERIFY_IS_EQUAL(t.dimension<1>(), 3); + VERIFY_IS_EQUAL(f.dimension<1>(), 3); + VERIFY_IS_EQUAL(m.dimension<1>(), 3); + VERIFY_IS_EQUAL(r.dimension<1>(), 3); + static_assert(Eigen::TensorFixedSize<float, Eigen::Sizes<2, 3, 7>>::dimension<2>() == 7, "constexpr"); + + VERIFY_IS_EQUAL(t.dimension(2), 7); + VERIFY_IS_EQUAL(f.dimension(2), 7); + VERIFY_IS_EQUAL(m.dimension(2), 7); + VERIFY_IS_EQUAL(r.dimension(2), 7); + + VERIFY_RAISES_ASSERT(t.dimension(3)); + VERIFY_RAISES_ASSERT(t.dimension(-1)); + VERIFY_RAISES_ASSERT(f.dimension(3)); + VERIFY_RAISES_ASSERT(f.dimension(-1)); + VERIFY_RAISES_ASSERT(m.dimension(3)); + VERIFY_RAISES_ASSERT(m.dimension(-1)); + VERIFY_RAISES_ASSERT(r.dimension(3)); + VERIFY_RAISES_ASSERT(r.dimension(-1)); +} + EIGEN_DECLARE_TEST(tensor_dimension) { CALL_SUBTEST(test_dimension_constants_odr_use()); CALL_SUBTEST(test_tensor_metadata_odr_use()); @@ -144,4 +174,5 @@ CALL_SUBTEST(test_index_type_promotion()); CALL_SUBTEST(test_unsigned_dimensions()); CALL_SUBTEST(test_narrowing_dimensions()); + CALL_SUBTEST(test_dimension_accessors()); }
diff --git a/contrib/test/tensor_executor.cpp b/contrib/test/tensor_executor.cpp index da6b2ac..df76d9f 100644 --- a/contrib/test/tensor_executor.cpp +++ b/contrib/test/tensor_executor.cpp
@@ -13,6 +13,7 @@ #include "main.h" #include <Eigen/Tensor> +#include "tensor_test_helpers.h" #include "../Eigen/SpecialFunctions" using Eigen::ColMajor; @@ -46,11 +47,7 @@ template <int NumDims> static array<Index, NumDims> RandomDims(int min_dim = 1, int max_dim = 20) { - array<Index, NumDims> dims; - for (int i = 0; i < NumDims; ++i) { - dims[i] = internal::random<int>(min_dim, max_dim); - } - return dims; + return randomTensorDims<NumDims, int>(min_dim, max_dim); } template <typename T, int NumDims, typename Device, bool Vectorizable, TiledEvaluation Tiling, int Layout>
diff --git a/contrib/test/tensor_gpu_test_helpers.h b/contrib/test/tensor_gpu_test_helpers.h new file mode 100644 index 0000000..86c3b4c --- /dev/null +++ b/contrib/test/tensor_gpu_test_helpers.h
@@ -0,0 +1,54 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_TENSOR_GPU_TEST_HELPERS_H +#define EIGEN_TENSOR_GPU_TEST_HELPERS_H + +#include <vector> + +#include <contrib/Eigen/Tensor> + +namespace tensor_test { + +using Eigen::Index; +using Eigen::Tensor; + +class GpuTensorTestFixture { + public: + GpuTensorTestFixture() : device_(&stream_) {} + GpuTensorTestFixture(const GpuTensorTestFixture&) = delete; + GpuTensorTestFixture& operator=(const GpuTensorTestFixture&) = delete; + + ~GpuTensorTestFixture() { + // Maps and pending expressions borrow these buffers from the fixture. + device_.synchronize(); + for (void* buffer : buffers_) device_.deallocate(buffer); + } + + const Eigen::GpuDevice& device() const { return device_; } + + template <typename Scalar, int NumDims, int Alignment = Eigen::Aligned, typename... Dimensions> + Eigen::TensorMap<Tensor<Scalar, NumDims>, Alignment> tensor(Dimensions... dimensions) { + const Eigen::DSizes<Index, NumDims> shape{dimensions...}; + void* buffer = device_.allocate(shape.TotalSize() * sizeof(Scalar)); + buffers_.push_back(buffer); + return Eigen::TensorMap<Tensor<Scalar, NumDims>, Alignment>(static_cast<Scalar*>(buffer), shape); + } + + template <typename Scalar, int NumDims, int Alignment> + Tensor<Scalar, NumDims> toHost(const Eigen::TensorMap<Tensor<Scalar, NumDims>, Alignment>& values) { + Tensor<Scalar, NumDims> host(values.dimensions()); + device_.memcpyDeviceToHost(host.data(), values.data(), values.size() * sizeof(Scalar)); + device_.synchronize(); + return host; + } + + private: + Eigen::GpuStreamDevice stream_; + Eigen::GpuDevice device_; + std::vector<void*> buffers_; +}; + +} // namespace tensor_test + +#endif // EIGEN_TENSOR_GPU_TEST_HELPERS_H
diff --git a/contrib/test/tensor_low_precision_gpu_test_helpers.h b/contrib/test/tensor_low_precision_gpu_test_helpers.h new file mode 100644 index 0000000..0320934 --- /dev/null +++ b/contrib/test/tensor_low_precision_gpu_test_helpers.h
@@ -0,0 +1,216 @@ +// SPDX-FileCopyrightText: 2016 Benoit Steiner <benoit.steiner.goog@gmail.com> +// SPDX-FileCopyrightText: 2021 Rohit Santhanam <rohit.santhanam@amd.com> +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_TENSOR_LOW_PRECISION_GPU_TEST_HELPERS_H +#define EIGEN_TENSOR_LOW_PRECISION_GPU_TEST_HELPERS_H + +#include <type_traits> + +#include "tensor_gpu_test_helpers.h" + +namespace tensor_test { + +template <typename Scalar, typename Input> +void round_gpu_test_input(Input& input, const Eigen::GpuDevice& device) { + // The bfloat16 unary/elementwise checks use inputs rounded to the working precision. + EIGEN_IF_CONSTEXPR ((std::is_same<Scalar, Eigen::bfloat16>::value)) { + input.device(device) = input.template cast<Scalar>().template cast<float>(); + } +} + +template <typename Scalar> +void test_gpu_numext() { + GpuTensorTestFixture fixture; + auto input = fixture.tensor<float, 1>(101); + auto reference = fixture.tensor<bool, 1>(101); + auto result = fixture.tensor<bool, 1>(101); + input.device(fixture.device()) = input.random() - input.constant(0.5f); + reference.device(fixture.device()) = input.unaryExpr(Eigen::internal::scalar_isnan_op<float>()); + result.device(fixture.device()) = input.template cast<Scalar>().unaryExpr(Eigen::internal::scalar_isnan_op<Scalar>()); + const auto expected = fixture.toHost(reference); + const auto actual = fixture.toHost(result); + for (Index i = 0; i < input.size(); ++i) VERIFY_IS_EQUAL(expected(i), actual(i)); +} + +template <typename Scalar> +void test_gpu_conversion() { + GpuTensorTestFixture fixture; + auto input = fixture.tensor<float, 1>(101); + auto converted = fixture.tensor<Scalar, 1>(101); + auto result = fixture.tensor<float, 1>(101); + input.device(fixture.device()) = input.random(); + converted.device(fixture.device()) = input.template cast<Scalar>(); + result.device(fixture.device()) = converted.template cast<float>(); + const auto expected = fixture.toHost(input); + const auto actual = fixture.toHost(result); + for (Index i = 0; i < input.size(); ++i) VERIFY_IS_APPROX(Scalar(expected(i)), Scalar(actual(i))); +} + +template <typename Scalar> +void test_gpu_unary() { + GpuTensorTestFixture fixture; + auto input = fixture.tensor<float, 1>(101); + auto reference = fixture.tensor<float, 1>(101); + auto result = fixture.tensor<float, 1>(101); + input.device(fixture.device()) = input.random() - input.constant(0.5f); + round_gpu_test_input<Scalar>(input, fixture.device()); + reference.device(fixture.device()) = input.abs(); + result.device(fixture.device()) = input.template cast<Scalar>().abs().template cast<float>(); + const auto expected = fixture.toHost(reference); + const auto actual = fixture.toHost(result); + for (Index i = 0; i < input.size(); ++i) VERIFY_IS_APPROX(expected(i), actual(i)); +} + +template <typename Scalar> +void test_gpu_elementwise() { + GpuTensorTestFixture fixture; + auto left = fixture.tensor<float, 1>(101); + auto right = fixture.tensor<float, 1>(101); + auto reference = fixture.tensor<float, 1>(101); + auto result = fixture.tensor<float, 1>(101); + left.device(fixture.device()) = left.random(); + round_gpu_test_input<Scalar>(left, fixture.device()); + right.device(fixture.device()) = right.random(); + round_gpu_test_input<Scalar>(right, fixture.device()); + reference.device(fixture.device()) = (left + right) * left; + result.device(fixture.device()) = + ((left.template cast<Scalar>() + right.template cast<Scalar>()) * left.template cast<Scalar>()) + .template cast<float>(); + const auto expected = fixture.toHost(reference); + const auto actual = fixture.toHost(result); + for (Index i = 0; i < left.size(); ++i) VERIFY_IS_APPROX(Scalar(expected(i)), Scalar(actual(i))); +} + +template <typename Scalar, typename Input, typename Operation> +void check_gpu_transcendental(GpuTensorTestFixture& fixture, const Input& input, const Operation& operation, + bool logarithm = false) { + auto reference = fixture.tensor<Scalar, 1>(input.size()); + auto result = fixture.tensor<Scalar, 1>(input.size()); + reference.device(fixture.device()) = operation(input).template cast<Scalar>(); + result.device(fixture.device()) = input.template cast<Scalar>(); + result.device(fixture.device()) = operation(result); + const auto host_input = fixture.toHost(input); + const auto expected = fixture.toHost(reference); + const auto actual = fixture.toHost(result); + for (Index i = 0; i < input.size(); ++i) { + if (logarithm && Eigen::numext::abs(host_input(i) - 1.f) < 0.05f) { + // Preserve the existing absolute floor for log near 1. + VERIFY_IS_APPROX(expected(i) + Scalar(0.1f), actual(i) + Scalar(0.1f)); + } else { + VERIFY_IS_APPROX(expected(i), actual(i)); + } + } +} + +template <typename Scalar> +void test_gpu_transcendental() { + GpuTensorTestFixture fixture; + auto input1 = fixture.tensor<float, 1>(101); + auto input2 = fixture.tensor<float, 1>(101); + auto input3 = fixture.tensor<float, 1>(101); + input1.device(fixture.device()) = input1.random() - input1.constant(0.5f); + round_gpu_test_input<Scalar>(input1, fixture.device()); + input2.device(fixture.device()) = input2.random() + input1.constant(0.5f); + round_gpu_test_input<Scalar>(input2, fixture.device()); + input3.device(fixture.device()) = input3.random(); + round_gpu_test_input<Scalar>(input3, fixture.device()); + check_gpu_transcendental<Scalar>(fixture, input1, [](const auto& input) { return input.exp(); }); + check_gpu_transcendental<Scalar>( + fixture, input2, [](const auto& input) { return input.log(); }, true); + check_gpu_transcendental<Scalar>(fixture, input3, [](const auto& input) { return input.log1p(); }); + check_gpu_transcendental<Scalar>(fixture, input3, [](const auto& input) { return input.expm1(); }); +} + +template <typename Scalar> +void test_gpu_contractions(float absolute_eps_factor) { + GpuTensorTestFixture fixture; + auto left = fixture.tensor<float, 2>(23, 23); + auto right = fixture.tensor<float, 2>(23, 23); + auto reference = fixture.tensor<Scalar, 2>(23, 23); + auto result = fixture.tensor<Scalar, 2>(23, 23); + left.device(fixture.device()) = left.random() - left.constant(0.5f); + right.device(fixture.device()) = right.random() - right.constant(0.5f); + using DimPair = typename Tensor<float, 2>::DimensionPair; + const Eigen::array<DimPair, 1> dims{DimPair(1, 0)}; + reference.device(fixture.device()) = left.contract(right, dims).template cast<Scalar>(); + result.device(fixture.device()) = left.template cast<Scalar>().contract(right.template cast<Scalar>(), dims); + const auto expected = fixture.toHost(reference); + const auto actual = fixture.toHost(result); + // Absolute slack for the 23-term dot product, then the existing relative comparison. + const Scalar absolute_slack = Scalar(absolute_eps_factor) * Eigen::NumTraits<Scalar>::epsilon(); + for (Index i = 0; i < result.size(); ++i) { + const Scalar error = Eigen::numext::abs(expected.data()[i] - actual.data()[i]); + VERIFY((Eigen::numext::isfinite)(error)); + if (error > absolute_slack) VERIFY_IS_APPROX(expected.data()[i], actual.data()[i]); + } +} + +template <typename Scalar> +void test_gpu_reductions(Index rows, Index cols, int axis) { + GpuTensorTestFixture fixture; + auto input = fixture.tensor<float, 2>(rows, cols); + const Index size = axis == 1 ? rows : cols; + auto reference = fixture.tensor<Scalar, 1>(size); + auto result = fixture.tensor<Scalar, 1>(size); + input.device(fixture.device()) = input.random() * 2.0f; + const Eigen::array<int, 1> dims{axis}; + reference.device(fixture.device()) = input.sum(dims).template cast<Scalar>(); + result.device(fixture.device()) = input.template cast<Scalar>().sum(dims); + const auto expected = fixture.toHost(reference); + const auto actual = fixture.toHost(result); + for (Index i = 0; i < size; ++i) VERIFY_IS_APPROX(expected(i), actual(i)); +} + +template <typename Scalar> +void test_gpu_reductions() { + test_gpu_reductions<Scalar>(13, 13, 0); + test_gpu_reductions<Scalar>(13, 13, 1); + test_gpu_reductions<Scalar>(35, 36, 0); + test_gpu_reductions<Scalar>(35, 36, 1); + test_gpu_reductions<Scalar>(36, 35, 0); + test_gpu_reductions<Scalar>(36, 35, 1); +} + +template <typename Scalar> +void test_gpu_full_reductions() { + GpuTensorTestFixture fixture; + auto input = fixture.tensor<float, 2>(13, 13); + auto reference = fixture.tensor<Scalar, 0>(); + auto result = fixture.tensor<Scalar, 0>(); + input.device(fixture.device()) = input.random(); + reference.device(fixture.device()) = input.sum().template cast<Scalar>(); + result.device(fixture.device()) = input.template cast<Scalar>().sum(); + VERIFY_IS_APPROX(fixture.toHost(reference)(), fixture.toHost(result)()); + reference.device(fixture.device()) = input.maximum().template cast<Scalar>(); + result.device(fixture.device()) = input.template cast<Scalar>().maximum(); + VERIFY_IS_APPROX(fixture.toHost(reference)(), fixture.toHost(result)()); +} + +template <typename Scalar> +void test_gpu_forced_evals() { + GpuTensorTestFixture fixture; + auto input = fixture.tensor<float, 1>(101); + auto reference = fixture.tensor<float, 1>(101); + auto aligned_result = fixture.tensor<float, 1>(101); + auto unaligned_result = fixture.tensor<float, 1, Eigen::Unaligned>(101); + const Eigen::array<int, 1> no_broadcast{1}; + input.device(fixture.device()) = input.random() - input.constant(0.5f); + round_gpu_test_input<Scalar>(input, fixture.device()); + reference.device(fixture.device()) = input.abs(); + aligned_result.device(fixture.device()) = input.template cast<Scalar>().abs().eval().template cast<float>(); + unaligned_result.device(fixture.device()) = + input.template cast<Scalar>().abs().broadcast(no_broadcast).eval().template cast<float>(); + const auto expected = fixture.toHost(reference); + const auto aligned = fixture.toHost(aligned_result); + const auto unaligned = fixture.toHost(unaligned_result); + for (Index i = 0; i < input.size(); ++i) { + VERIFY_IS_APPROX(expected(i), aligned(i)); + VERIFY_IS_APPROX(expected(i), unaligned(i)); + } +} + +} // namespace tensor_test + +#endif // EIGEN_TENSOR_LOW_PRECISION_GPU_TEST_HELPERS_H
diff --git a/contrib/test/tensor_of_bfloat16_gpu.cu b/contrib/test/tensor_of_bfloat16_gpu.cu index fc1cc27..5317c38 100644 --- a/contrib/test/tensor_of_bfloat16_gpu.cu +++ b/contrib/test/tensor_of_bfloat16_gpu.cu
@@ -15,450 +15,20 @@ #define EIGEN_USE_GPU #include "main.h" -#include <contrib/Eigen/Tensor> - -using Eigen::Tensor; - -template <typename> -void test_gpu_numext() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - bool* d_res_bfloat16 = (bool*)gpu_device.allocate(num_elem * sizeof(bool)); - bool* d_res_float = (bool*)gpu_device.allocate(num_elem * sizeof(bool)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float(d_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<bool, 1>, Eigen::Aligned> gpu_res_bfloat16(d_res_bfloat16, num_elem); - Eigen::TensorMap<Eigen::Tensor<bool, 1>, Eigen::Aligned> gpu_res_float(d_res_float, num_elem); - - gpu_float.device(gpu_device) = gpu_float.random() - gpu_float.constant(0.5f); - gpu_res_float.device(gpu_device) = gpu_float.unaryExpr(Eigen::internal::scalar_isnan_op<float>()); - // Test bfloat16 specific isnan op. - gpu_res_bfloat16.device(gpu_device) = - gpu_float.cast<Eigen::bfloat16>().unaryExpr(Eigen::internal::scalar_isnan_op<Eigen::bfloat16>()); - - Tensor<bool, 1> bfloat16_prec(num_elem); - Tensor<bool, 1> full_prec(num_elem); - gpu_device.memcpyDeviceToHost(bfloat16_prec.data(), d_res_bfloat16, num_elem * sizeof(bool)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(bool)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - VERIFY_IS_EQUAL(full_prec(i), bfloat16_prec(i)); - } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_bfloat16); - gpu_device.deallocate(d_res_float); -} - -#ifdef EIGEN_HAS_GPU_BF16 - -template <typename> -void test_gpu_conversion() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::bfloat16* d_bfloat16 = (Eigen::bfloat16*)gpu_device.allocate(num_elem * sizeof(Eigen::bfloat16)); - float* d_conv = (float*)gpu_device.allocate(num_elem * sizeof(float)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float(d_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_bfloat16(d_bfloat16, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_conv(d_conv, num_elem); - - gpu_float.device(gpu_device) = gpu_float.random(); - gpu_bfloat16.device(gpu_device) = gpu_float.cast<Eigen::bfloat16>(); - gpu_conv.device(gpu_device) = gpu_bfloat16.cast<float>(); - - Tensor<float, 1> initial(num_elem); - Tensor<float, 1> final(num_elem); - gpu_device.memcpyDeviceToHost(initial.data(), d_float, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(final.data(), d_conv, num_elem * sizeof(float)); - - for (int i = 0; i < num_elem; ++i) { - VERIFY_IS_APPROX(static_cast<Eigen::bfloat16>(initial(i)), static_cast<Eigen::bfloat16>(final(i))); - } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_bfloat16); - gpu_device.deallocate(d_conv); -} - -template <typename> -void test_gpu_unary() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_bfloat16 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float(d_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_bfloat16(d_res_bfloat16, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_float(d_res_float, num_elem); - - gpu_float.device(gpu_device) = gpu_float.random() - gpu_float.constant(0.5f); - gpu_float.device(gpu_device) = gpu_float.cast<Eigen::bfloat16>().cast<float>(); - gpu_res_float.device(gpu_device) = gpu_float.abs(); - gpu_res_bfloat16.device(gpu_device) = gpu_float.cast<Eigen::bfloat16>().abs().cast<float>(); - - Tensor<float, 1> bfloat16_prec(num_elem); - Tensor<float, 1> full_prec(num_elem); - gpu_device.memcpyDeviceToHost(bfloat16_prec.data(), d_res_bfloat16, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(float)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - VERIFY_IS_APPROX(full_prec(i), bfloat16_prec(i)); - } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_bfloat16); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_elementwise() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float1 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_float2 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_bfloat16 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float1(d_float1, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float2(d_float2, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_bfloat16(d_res_bfloat16, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_float(d_res_float, num_elem); - - gpu_float1.device(gpu_device) = gpu_float1.random(); - gpu_float1.device(gpu_device) = gpu_float1.cast<Eigen::bfloat16>().cast<float>(); - gpu_float2.device(gpu_device) = gpu_float2.random(); - gpu_float2.device(gpu_device) = gpu_float2.cast<Eigen::bfloat16>().cast<float>(); - gpu_res_float.device(gpu_device) = (gpu_float1 + gpu_float2) * gpu_float1; - gpu_res_bfloat16.device(gpu_device) = - ((gpu_float1.cast<Eigen::bfloat16>() + gpu_float2.cast<Eigen::bfloat16>()) * gpu_float1.cast<Eigen::bfloat16>()) - .cast<float>(); - - Tensor<float, 1> bfloat16_prec(num_elem); - Tensor<float, 1> full_prec(num_elem); - gpu_device.memcpyDeviceToHost(bfloat16_prec.data(), d_res_bfloat16, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(float)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - VERIFY_IS_APPROX(static_cast<Eigen::bfloat16>(full_prec(i)), static_cast<Eigen::bfloat16>(bfloat16_prec(i))); - } - - gpu_device.deallocate(d_float1); - gpu_device.deallocate(d_float2); - gpu_device.deallocate(d_res_bfloat16); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_trancendental() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float1 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_float2 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_float3 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::bfloat16* d_res1_bfloat16 = (Eigen::bfloat16*)gpu_device.allocate(num_elem * sizeof(Eigen::bfloat16)); - Eigen::bfloat16* d_res1_float = (Eigen::bfloat16*)gpu_device.allocate(num_elem * sizeof(Eigen::bfloat16)); - Eigen::bfloat16* d_res2_bfloat16 = (Eigen::bfloat16*)gpu_device.allocate(num_elem * sizeof(Eigen::bfloat16)); - Eigen::bfloat16* d_res2_float = (Eigen::bfloat16*)gpu_device.allocate(num_elem * sizeof(Eigen::bfloat16)); - Eigen::bfloat16* d_res3_bfloat16 = (Eigen::bfloat16*)gpu_device.allocate(num_elem * sizeof(Eigen::bfloat16)); - Eigen::bfloat16* d_res3_float = (Eigen::bfloat16*)gpu_device.allocate(num_elem * sizeof(Eigen::bfloat16)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float1(d_float1, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float2(d_float2, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float3(d_float3, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res1_bfloat16(d_res1_bfloat16, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res1_float(d_res1_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res2_bfloat16(d_res2_bfloat16, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res2_float(d_res2_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res3_bfloat16(d_res3_bfloat16, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res3_float(d_res3_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res4_bfloat16(d_res3_bfloat16, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res4_float(d_res3_float, num_elem); - - gpu_float1.device(gpu_device) = gpu_float1.random() - gpu_float1.constant(0.5f); - gpu_float1.device(gpu_device) = gpu_float1.cast<Eigen::bfloat16>().cast<float>(); - gpu_float2.device(gpu_device) = gpu_float2.random() + gpu_float1.constant(0.5f); - gpu_float2.device(gpu_device) = gpu_float2.cast<Eigen::bfloat16>().cast<float>(); - gpu_float3.device(gpu_device) = gpu_float3.random(); - gpu_float3.device(gpu_device) = gpu_float3.cast<Eigen::bfloat16>().cast<float>(); - gpu_res1_float.device(gpu_device) = gpu_float1.exp().cast<Eigen::bfloat16>(); - gpu_res2_float.device(gpu_device) = gpu_float2.log().cast<Eigen::bfloat16>(); - gpu_res3_float.device(gpu_device) = gpu_float3.log1p().cast<Eigen::bfloat16>(); - gpu_res4_float.device(gpu_device) = gpu_float3.expm1().cast<Eigen::bfloat16>(); - - gpu_res1_bfloat16.device(gpu_device) = gpu_float1.cast<Eigen::bfloat16>(); - gpu_res1_bfloat16.device(gpu_device) = gpu_res1_bfloat16.exp(); - - gpu_res2_bfloat16.device(gpu_device) = gpu_float2.cast<Eigen::bfloat16>(); - gpu_res2_bfloat16.device(gpu_device) = gpu_res2_bfloat16.log(); - - gpu_res3_bfloat16.device(gpu_device) = gpu_float3.cast<Eigen::bfloat16>(); - gpu_res3_bfloat16.device(gpu_device) = gpu_res3_bfloat16.log1p(); - - gpu_res3_bfloat16.device(gpu_device) = gpu_float3.cast<Eigen::bfloat16>(); - gpu_res3_bfloat16.device(gpu_device) = gpu_res3_bfloat16.expm1(); - - Tensor<float, 1> input1(num_elem); - Tensor<Eigen::bfloat16, 1> bfloat16_prec1(num_elem); - Tensor<Eigen::bfloat16, 1> full_prec1(num_elem); - Tensor<float, 1> input2(num_elem); - Tensor<Eigen::bfloat16, 1> bfloat16_prec2(num_elem); - Tensor<Eigen::bfloat16, 1> full_prec2(num_elem); - Tensor<float, 1> input3(num_elem); - Tensor<Eigen::bfloat16, 1> bfloat16_prec3(num_elem); - Tensor<Eigen::bfloat16, 1> full_prec3(num_elem); - gpu_device.memcpyDeviceToHost(input1.data(), d_float1, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(input2.data(), d_float2, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(input3.data(), d_float3, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(bfloat16_prec1.data(), d_res1_bfloat16, num_elem * sizeof(Eigen::bfloat16)); - gpu_device.memcpyDeviceToHost(full_prec1.data(), d_res1_float, num_elem * sizeof(Eigen::bfloat16)); - gpu_device.memcpyDeviceToHost(bfloat16_prec2.data(), d_res2_bfloat16, num_elem * sizeof(Eigen::bfloat16)); - gpu_device.memcpyDeviceToHost(full_prec2.data(), d_res2_float, num_elem * sizeof(Eigen::bfloat16)); - gpu_device.memcpyDeviceToHost(bfloat16_prec3.data(), d_res3_bfloat16, num_elem * sizeof(Eigen::bfloat16)); - gpu_device.memcpyDeviceToHost(full_prec3.data(), d_res3_float, num_elem * sizeof(Eigen::bfloat16)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - VERIFY_IS_APPROX(full_prec1(i), bfloat16_prec1(i)); - } - for (int i = 0; i < num_elem; ++i) { - if (std::abs(input2(i) - 1.f) < 0.05f) // log lacks accuracy nearby 1 - VERIFY_IS_APPROX(full_prec2(i) + Eigen::bfloat16(0.1f), bfloat16_prec2(i) + Eigen::bfloat16(0.1f)); - else - VERIFY_IS_APPROX(full_prec2(i), bfloat16_prec2(i)); - } - for (int i = 0; i < num_elem; ++i) { - VERIFY_IS_APPROX(full_prec3(i), bfloat16_prec3(i)); - } - gpu_device.deallocate(d_float1); - gpu_device.deallocate(d_float2); - gpu_device.deallocate(d_float3); - gpu_device.deallocate(d_res1_bfloat16); - gpu_device.deallocate(d_res1_float); - gpu_device.deallocate(d_res2_bfloat16); - gpu_device.deallocate(d_res2_float); - gpu_device.deallocate(d_res3_float); - gpu_device.deallocate(d_res3_bfloat16); -} - -template <typename> -void test_gpu_contractions() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int rows = 23; - int cols = 23; - int num_elem = rows * cols; - - float* d_float1 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_float2 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::bfloat16* d_res_bfloat16 = (Eigen::bfloat16*)gpu_device.allocate(num_elem * sizeof(Eigen::bfloat16)); - Eigen::bfloat16* d_res_float = (Eigen::bfloat16*)gpu_device.allocate(num_elem * sizeof(Eigen::bfloat16)); - - Eigen::TensorMap<Eigen::Tensor<float, 2>, Eigen::Aligned> gpu_float1(d_float1, rows, cols); - Eigen::TensorMap<Eigen::Tensor<float, 2>, Eigen::Aligned> gpu_float2(d_float2, rows, cols); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 2>, Eigen::Aligned> gpu_res_bfloat16(d_res_bfloat16, rows, cols); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 2>, Eigen::Aligned> gpu_res_float(d_res_float, rows, cols); - - gpu_float1.device(gpu_device) = gpu_float1.random() - gpu_float1.constant(0.5f); - gpu_float2.device(gpu_device) = gpu_float2.random() - gpu_float2.constant(0.5f); - - typedef Tensor<float, 2>::DimensionPair DimPair; - Eigen::array<DimPair, 1> dims{DimPair(1, 0)}; - gpu_res_float.device(gpu_device) = gpu_float1.contract(gpu_float2, dims).cast<Eigen::bfloat16>(); - gpu_res_bfloat16.device(gpu_device) = - gpu_float1.cast<Eigen::bfloat16>().contract(gpu_float2.cast<Eigen::bfloat16>(), dims); - - Tensor<Eigen::bfloat16, 2> bfloat16_prec(rows, cols); - Tensor<Eigen::bfloat16, 2> full_prec(rows, cols); - gpu_device.memcpyDeviceToHost(bfloat16_prec.data(), d_res_bfloat16, num_elem * sizeof(Eigen::bfloat16)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(Eigen::bfloat16)); - gpu_device.synchronize(); - - for (int i = 0; i < rows; ++i) { - for (int j = 0; j < cols; ++j) { - // 2 * eps_bfloat16 of absolute slack covers the typical accumulated rounding error of the 23-term bfloat16 - // dot product before falling back to the relative check. - if (numext::abs(full_prec(i, j) - bfloat16_prec(i, j)) > - Eigen::bfloat16(2.f) * NumTraits<Eigen::bfloat16>::epsilon()) { - VERIFY_IS_APPROX(full_prec(i, j), bfloat16_prec(i, j)); - } - } - } - - gpu_device.deallocate(d_float1); - gpu_device.deallocate(d_float2); - gpu_device.deallocate(d_res_bfloat16); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_reductions(int size1, int size2, int redux) { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = size1 * size2; - int result_size = (redux == 1 ? size1 : size2); - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::bfloat16* d_res_bfloat16 = (Eigen::bfloat16*)gpu_device.allocate(result_size * sizeof(Eigen::bfloat16)); - Eigen::bfloat16* d_res_float = (Eigen::bfloat16*)gpu_device.allocate(result_size * sizeof(Eigen::bfloat16)); - - Eigen::TensorMap<Eigen::Tensor<float, 2>, Eigen::Aligned> gpu_float(d_float, size1, size2); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res_bfloat16(d_res_bfloat16, result_size); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 1>, Eigen::Aligned> gpu_res_float(d_res_float, result_size); - - gpu_float.device(gpu_device) = gpu_float.random() * 2.0f; - - Eigen::array<int, 1> redux_dim = {redux}; - gpu_res_float.device(gpu_device) = gpu_float.sum(redux_dim).cast<Eigen::bfloat16>(); - gpu_res_bfloat16.device(gpu_device) = gpu_float.cast<Eigen::bfloat16>().sum(redux_dim); - - Tensor<Eigen::bfloat16, 1> bfloat16_prec(result_size); - Tensor<Eigen::bfloat16, 1> full_prec(result_size); - gpu_device.memcpyDeviceToHost(bfloat16_prec.data(), d_res_bfloat16, result_size * sizeof(Eigen::bfloat16)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, result_size * sizeof(Eigen::bfloat16)); - gpu_device.synchronize(); - - for (int i = 0; i < result_size; ++i) { - VERIFY_IS_APPROX(full_prec(i), bfloat16_prec(i)); - } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_bfloat16); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_reductions() { - test_gpu_reductions<void>(13, 13, 0); - test_gpu_reductions<void>(13, 13, 1); - - test_gpu_reductions<void>(35, 36, 0); - test_gpu_reductions<void>(35, 36, 1); - - test_gpu_reductions<void>(36, 35, 0); - test_gpu_reductions<void>(36, 35, 1); -} - -template <typename> -void test_gpu_full_reductions() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int size = 13; - int num_elem = size * size; - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::bfloat16* d_res_bfloat16 = (Eigen::bfloat16*)gpu_device.allocate(1 * sizeof(Eigen::bfloat16)); - Eigen::bfloat16* d_res_float = (Eigen::bfloat16*)gpu_device.allocate(1 * sizeof(Eigen::bfloat16)); - - Eigen::TensorMap<Eigen::Tensor<float, 2>, Eigen::Aligned> gpu_float(d_float, size, size); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 0>, Eigen::Aligned> gpu_res_bfloat16(d_res_bfloat16); - Eigen::TensorMap<Eigen::Tensor<Eigen::bfloat16, 0>, Eigen::Aligned> gpu_res_float(d_res_float); - - gpu_float.device(gpu_device) = gpu_float.random(); - - gpu_res_float.device(gpu_device) = gpu_float.sum().cast<Eigen::bfloat16>(); - gpu_res_bfloat16.device(gpu_device) = gpu_float.cast<Eigen::bfloat16>().sum(); - - Tensor<Eigen::bfloat16, 0> bfloat16_prec; - Tensor<Eigen::bfloat16, 0> full_prec; - gpu_device.memcpyDeviceToHost(bfloat16_prec.data(), d_res_bfloat16, sizeof(Eigen::bfloat16)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, sizeof(Eigen::bfloat16)); - gpu_device.synchronize(); - - VERIFY_IS_APPROX(full_prec(), bfloat16_prec()); - - gpu_res_float.device(gpu_device) = gpu_float.maximum().cast<Eigen::bfloat16>(); - gpu_res_bfloat16.device(gpu_device) = gpu_float.cast<Eigen::bfloat16>().maximum(); - gpu_device.memcpyDeviceToHost(bfloat16_prec.data(), d_res_bfloat16, sizeof(Eigen::bfloat16)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, sizeof(Eigen::bfloat16)); - gpu_device.synchronize(); - - VERIFY_IS_APPROX(full_prec(), bfloat16_prec()); - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_bfloat16); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_forced_evals() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_bfloat16_1 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_bfloat16_2 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float(d_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_bfloat16_1(d_res_bfloat16_1, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Unaligned> gpu_res_bfloat16_2(d_res_bfloat16_2, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_float(d_res_float, num_elem); - - Eigen::array<int, 1> no_bcast; - no_bcast[0] = 1; - - gpu_float.device(gpu_device) = gpu_float.random() - gpu_float.constant(0.5f); - gpu_float.device(gpu_device) = gpu_float.cast<Eigen::bfloat16>().cast<float>(); - gpu_res_float.device(gpu_device) = gpu_float.abs(); - gpu_res_bfloat16_1.device(gpu_device) = gpu_float.cast<Eigen::bfloat16>().abs().eval().cast<float>(); - gpu_res_bfloat16_2.device(gpu_device) = - gpu_float.cast<Eigen::bfloat16>().abs().broadcast(no_bcast).eval().cast<float>(); - - Tensor<float, 1> bfloat16_prec1(num_elem); - Tensor<float, 1> bfloat16_prec2(num_elem); - Tensor<float, 1> full_prec(num_elem); - gpu_device.memcpyDeviceToHost(bfloat16_prec1.data(), d_res_bfloat16_1, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(bfloat16_prec2.data(), d_res_bfloat16_2, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(float)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - VERIFY_IS_APPROX(full_prec(i), bfloat16_prec1(i)); - VERIFY_IS_APPROX(full_prec(i), bfloat16_prec2(i)); - } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_bfloat16_1); - gpu_device.deallocate(d_res_bfloat16_2); - gpu_device.deallocate(d_res_float); -} - -#endif +#include "tensor_low_precision_gpu_test_helpers.h" EIGEN_DECLARE_TEST(tensor_of_bfloat16_gpu) { - CALL_SUBTEST_1(test_gpu_numext<void>()); + CALL_SUBTEST_1(tensor_test::test_gpu_numext<Eigen::bfloat16>()); -// The reduction unit tests have been excluded until a working -// implementation to expand the accumulator data type to float32 -// is available. -// TODO: add reduction unit tests #ifdef EIGEN_HAS_GPU_BF16 - CALL_SUBTEST_2(test_gpu_conversion<void>()); - CALL_SUBTEST_3(test_gpu_unary<void>()); - CALL_SUBTEST_4(test_gpu_elementwise<void>()); - CALL_SUBTEST_5(test_gpu_trancendental<void>()); - CALL_SUBTEST_6(test_gpu_contractions<void>()); - CALL_SUBTEST_7(test_gpu_reductions<void>()); - CALL_SUBTEST_8(test_gpu_full_reductions<void>()); - CALL_SUBTEST_9(test_gpu_forced_evals<void>()); + CALL_SUBTEST_2(tensor_test::test_gpu_conversion<Eigen::bfloat16>()); + CALL_SUBTEST_3(tensor_test::test_gpu_unary<Eigen::bfloat16>()); + CALL_SUBTEST_4(tensor_test::test_gpu_elementwise<Eigen::bfloat16>()); + CALL_SUBTEST_5(tensor_test::test_gpu_transcendental<Eigen::bfloat16>()); + CALL_SUBTEST_6(tensor_test::test_gpu_contractions<Eigen::bfloat16>(2.f)); + CALL_SUBTEST_7(tensor_test::test_gpu_reductions<Eigen::bfloat16>()); + CALL_SUBTEST_8(tensor_test::test_gpu_full_reductions<Eigen::bfloat16>()); + CALL_SUBTEST_9(tensor_test::test_gpu_forced_evals<Eigen::bfloat16>()); #else // Without device bfloat16 support parts 2-9 have nothing to run. Report them as skipped rather than let an // empty part pass: under EIGEN_SPLIT_LARGE_TESTS each part is its own executable, and a green one would claim
diff --git a/contrib/test/tensor_of_float16_gpu.cu b/contrib/test/tensor_of_float16_gpu.cu index fa9b2da..22c76af 100644 --- a/contrib/test/tensor_of_float16_gpu.cu +++ b/contrib/test/tensor_of_float16_gpu.cu
@@ -15,414 +15,16 @@ #define EIGEN_USE_GPU #include "main.h" -#include <contrib/Eigen/Tensor> - -using Eigen::Tensor; - -template <typename> -void test_gpu_numext() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - bool* d_res_half = (bool*)gpu_device.allocate(num_elem * sizeof(bool)); - bool* d_res_float = (bool*)gpu_device.allocate(num_elem * sizeof(bool)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float(d_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<bool, 1>, Eigen::Aligned> gpu_res_half(d_res_half, num_elem); - Eigen::TensorMap<Eigen::Tensor<bool, 1>, Eigen::Aligned> gpu_res_float(d_res_float, num_elem); - - gpu_float.device(gpu_device) = gpu_float.random() - gpu_float.constant(0.5f); - gpu_res_float.device(gpu_device) = gpu_float.unaryExpr(Eigen::internal::scalar_isnan_op<float>()); - gpu_res_half.device(gpu_device) = - gpu_float.template cast<Eigen::half>().unaryExpr(Eigen::internal::scalar_isnan_op<Eigen::half>()); - - Tensor<bool, 1> half_prec(num_elem); - Tensor<bool, 1> full_prec(num_elem); - gpu_device.memcpyDeviceToHost(half_prec.data(), d_res_half, num_elem * sizeof(bool)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(bool)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - std::cout << "Checking numext " << i << std::endl; - VERIFY_IS_EQUAL(full_prec(i), half_prec(i)); - } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_half); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_unary() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_half = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float(d_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_half(d_res_half, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_float(d_res_float, num_elem); - - gpu_float.device(gpu_device) = gpu_float.random() - gpu_float.constant(0.5f); - gpu_res_float.device(gpu_device) = gpu_float.abs(); - gpu_res_half.device(gpu_device) = gpu_float.template cast<Eigen::half>().abs().template cast<float>(); - - Tensor<float, 1> half_prec(num_elem); - Tensor<float, 1> full_prec(num_elem); - gpu_device.memcpyDeviceToHost(half_prec.data(), d_res_half, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(float)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - std::cout << "Checking unary " << i << std::endl; - VERIFY_IS_APPROX(full_prec(i), half_prec(i)); - } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_half); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_elementwise() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float1 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_float2 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_half = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float1(d_float1, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float2(d_float2, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_half(d_res_half, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_float(d_res_float, num_elem); - - gpu_float1.device(gpu_device) = gpu_float1.random(); - gpu_float2.device(gpu_device) = gpu_float2.random(); - gpu_res_float.device(gpu_device) = (gpu_float1 + gpu_float2) * gpu_float1; - gpu_res_half.device(gpu_device) = - ((gpu_float1.template cast<Eigen::half>() + gpu_float2.template cast<Eigen::half>()) * - gpu_float1.template cast<Eigen::half>()) - .template cast<float>(); - - Tensor<float, 1> half_prec(num_elem); - Tensor<float, 1> full_prec(num_elem); - gpu_device.memcpyDeviceToHost(half_prec.data(), d_res_half, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(float)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - std::cout << "Checking elemwise " << i << ": full prec = " << full_prec(i) << " vs half prec = " << half_prec(i) - << std::endl; - VERIFY_IS_APPROX(static_cast<Eigen::half>(full_prec(i)), static_cast<Eigen::half>(half_prec(i))); - } - - gpu_device.deallocate(d_float1); - gpu_device.deallocate(d_float2); - gpu_device.deallocate(d_res_half); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_trancendental() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float1 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_float2 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_float3 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::half* d_res1_half = (Eigen::half*)gpu_device.allocate(num_elem * sizeof(Eigen::half)); - Eigen::half* d_res1_float = (Eigen::half*)gpu_device.allocate(num_elem * sizeof(Eigen::half)); - Eigen::half* d_res2_half = (Eigen::half*)gpu_device.allocate(num_elem * sizeof(Eigen::half)); - Eigen::half* d_res2_float = (Eigen::half*)gpu_device.allocate(num_elem * sizeof(Eigen::half)); - Eigen::half* d_res3_half = (Eigen::half*)gpu_device.allocate(num_elem * sizeof(Eigen::half)); - Eigen::half* d_res3_float = (Eigen::half*)gpu_device.allocate(num_elem * sizeof(Eigen::half)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float1(d_float1, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float2(d_float2, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float3(d_float3, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res1_half(d_res1_half, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res1_float(d_res1_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res2_half(d_res2_half, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res2_float(d_res2_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res3_half(d_res3_half, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res3_float(d_res3_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res4_half(d_res3_half, num_elem); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res4_float(d_res3_float, num_elem); - - gpu_float1.device(gpu_device) = gpu_float1.random() - gpu_float1.constant(0.5f); - gpu_float2.device(gpu_device) = gpu_float2.random() + gpu_float1.constant(0.5f); - gpu_float3.device(gpu_device) = gpu_float3.random(); - gpu_res1_float.device(gpu_device) = gpu_float1.exp().template cast<Eigen::half>(); - gpu_res2_float.device(gpu_device) = gpu_float2.log().template cast<Eigen::half>(); - gpu_res3_float.device(gpu_device) = gpu_float3.log1p().template cast<Eigen::half>(); - gpu_res4_float.device(gpu_device) = gpu_float3.expm1().template cast<Eigen::half>(); - - gpu_res1_half.device(gpu_device) = gpu_float1.template cast<Eigen::half>(); - gpu_res1_half.device(gpu_device) = gpu_res1_half.exp(); - - gpu_res2_half.device(gpu_device) = gpu_float2.template cast<Eigen::half>(); - gpu_res2_half.device(gpu_device) = gpu_res2_half.log(); - - gpu_res3_half.device(gpu_device) = gpu_float3.template cast<Eigen::half>(); - gpu_res3_half.device(gpu_device) = gpu_res3_half.log1p(); - - gpu_res3_half.device(gpu_device) = gpu_float3.template cast<Eigen::half>(); - gpu_res3_half.device(gpu_device) = gpu_res3_half.expm1(); - - Tensor<float, 1> input1(num_elem); - Tensor<Eigen::half, 1> half_prec1(num_elem); - Tensor<Eigen::half, 1> full_prec1(num_elem); - Tensor<float, 1> input2(num_elem); - Tensor<Eigen::half, 1> half_prec2(num_elem); - Tensor<Eigen::half, 1> full_prec2(num_elem); - Tensor<float, 1> input3(num_elem); - Tensor<Eigen::half, 1> half_prec3(num_elem); - Tensor<Eigen::half, 1> full_prec3(num_elem); - gpu_device.memcpyDeviceToHost(input1.data(), d_float1, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(input2.data(), d_float2, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(input3.data(), d_float3, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(half_prec1.data(), d_res1_half, num_elem * sizeof(Eigen::half)); - gpu_device.memcpyDeviceToHost(full_prec1.data(), d_res1_float, num_elem * sizeof(Eigen::half)); - gpu_device.memcpyDeviceToHost(half_prec2.data(), d_res2_half, num_elem * sizeof(Eigen::half)); - gpu_device.memcpyDeviceToHost(full_prec2.data(), d_res2_float, num_elem * sizeof(Eigen::half)); - gpu_device.memcpyDeviceToHost(half_prec3.data(), d_res3_half, num_elem * sizeof(Eigen::half)); - gpu_device.memcpyDeviceToHost(full_prec3.data(), d_res3_float, num_elem * sizeof(Eigen::half)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - std::cout << "Checking elemwise exp " << i << " input = " << input1(i) << " full = " << full_prec1(i) - << " half = " << half_prec1(i) << std::endl; - VERIFY_IS_APPROX(full_prec1(i), half_prec1(i)); - } - for (int i = 0; i < num_elem; ++i) { - std::cout << "Checking elemwise log " << i << " input = " << input2(i) << " full = " << full_prec2(i) - << " half = " << half_prec2(i) << std::endl; - if (std::abs(input2(i) - 1.f) < 0.05f) // log lacks accuracy nearby 1 - VERIFY_IS_APPROX(full_prec2(i) + Eigen::half(0.1f), half_prec2(i) + Eigen::half(0.1f)); - else - VERIFY_IS_APPROX(full_prec2(i), half_prec2(i)); - } - for (int i = 0; i < num_elem; ++i) { - std::cout << "Checking elemwise plog1 " << i << " input = " << input3(i) << " full = " << full_prec3(i) - << " half = " << half_prec3(i) << std::endl; - VERIFY_IS_APPROX(full_prec3(i), half_prec3(i)); - } - gpu_device.deallocate(d_float1); - gpu_device.deallocate(d_float2); - gpu_device.deallocate(d_float3); - gpu_device.deallocate(d_res1_half); - gpu_device.deallocate(d_res1_float); - gpu_device.deallocate(d_res2_half); - gpu_device.deallocate(d_res2_float); - gpu_device.deallocate(d_res3_float); - gpu_device.deallocate(d_res3_half); -} - -template <typename> -void test_gpu_contractions() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int rows = 23; - int cols = 23; - int num_elem = rows * cols; - - float* d_float1 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_float2 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::half* d_res_half = (Eigen::half*)gpu_device.allocate(num_elem * sizeof(Eigen::half)); - Eigen::half* d_res_float = (Eigen::half*)gpu_device.allocate(num_elem * sizeof(Eigen::half)); - - Eigen::TensorMap<Eigen::Tensor<float, 2>, Eigen::Aligned> gpu_float1(d_float1, rows, cols); - Eigen::TensorMap<Eigen::Tensor<float, 2>, Eigen::Aligned> gpu_float2(d_float2, rows, cols); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 2>, Eigen::Aligned> gpu_res_half(d_res_half, rows, cols); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 2>, Eigen::Aligned> gpu_res_float(d_res_float, rows, cols); - - gpu_float1.device(gpu_device) = gpu_float1.random() - gpu_float1.constant(0.5f); - gpu_float2.device(gpu_device) = gpu_float2.random() - gpu_float2.constant(0.5f); - - typedef Tensor<float, 2>::DimensionPair DimPair; - Eigen::array<DimPair, 1> dims{DimPair(1, 0)}; - gpu_res_float.device(gpu_device) = gpu_float1.contract(gpu_float2, dims).template cast<Eigen::half>(); - gpu_res_half.device(gpu_device) = - gpu_float1.template cast<Eigen::half>().contract(gpu_float2.template cast<Eigen::half>(), dims); - - Tensor<Eigen::half, 2> half_prec(rows, cols); - Tensor<Eigen::half, 2> full_prec(rows, cols); - gpu_device.memcpyDeviceToHost(half_prec.data(), d_res_half, num_elem * sizeof(Eigen::half)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(Eigen::half)); - gpu_device.synchronize(); - - for (int i = 0; i < rows; ++i) { - for (int j = 0; j < cols; ++j) { - std::cout << "Checking contract " << i << " " << j << full_prec(i, j) << " " << half_prec(i, j) << std::endl; - // 16 * eps_half of absolute slack covers the typical accumulated rounding error of the 23-term half dot - // product before falling back to the relative check. - if (numext::abs(full_prec(i, j) - half_prec(i, j)) > Eigen::half(16.f) * NumTraits<Eigen::half>::epsilon()) { - VERIFY_IS_APPROX(full_prec(i, j), half_prec(i, j)); - } - } - } - - gpu_device.deallocate(d_float1); - gpu_device.deallocate(d_float2); - gpu_device.deallocate(d_res_half); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_reductions(int size1, int size2, int redux) { - std::cout << "Reducing " << size1 << " by " << size2 << " tensor along dim " << redux << std::endl; - - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = size1 * size2; - int result_size = (redux == 1 ? size1 : size2); - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::half* d_res_half = (Eigen::half*)gpu_device.allocate(result_size * sizeof(Eigen::half)); - Eigen::half* d_res_float = (Eigen::half*)gpu_device.allocate(result_size * sizeof(Eigen::half)); - - Eigen::TensorMap<Eigen::Tensor<float, 2>, Eigen::Aligned> gpu_float(d_float, size1, size2); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res_half(d_res_half, result_size); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 1>, Eigen::Aligned> gpu_res_float(d_res_float, result_size); - - gpu_float.device(gpu_device) = gpu_float.random() * 2.0f; - - Eigen::array<int, 1> redux_dim = {redux}; - gpu_res_float.device(gpu_device) = gpu_float.sum(redux_dim).template cast<Eigen::half>(); - gpu_res_half.device(gpu_device) = gpu_float.template cast<Eigen::half>().sum(redux_dim); - - Tensor<Eigen::half, 1> half_prec(result_size); - Tensor<Eigen::half, 1> full_prec(result_size); - gpu_device.memcpyDeviceToHost(half_prec.data(), d_res_half, result_size * sizeof(Eigen::half)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, result_size * sizeof(Eigen::half)); - gpu_device.synchronize(); - - for (int i = 0; i < result_size; ++i) { - std::cout << "EXPECTED " << full_prec(i) << " GOT " << half_prec(i) << std::endl; - VERIFY_IS_APPROX(full_prec(i), half_prec(i)); - } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_half); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_reductions() { - test_gpu_reductions<void>(13, 13, 0); - test_gpu_reductions<void>(13, 13, 1); - - test_gpu_reductions<void>(35, 36, 0); - test_gpu_reductions<void>(35, 36, 1); - - test_gpu_reductions<void>(36, 35, 0); - test_gpu_reductions<void>(36, 35, 1); -} - -template <typename> -void test_gpu_full_reductions() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int size = 13; - int num_elem = size * size; - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - Eigen::half* d_res_half = (Eigen::half*)gpu_device.allocate(1 * sizeof(Eigen::half)); - Eigen::half* d_res_float = (Eigen::half*)gpu_device.allocate(1 * sizeof(Eigen::half)); - - Eigen::TensorMap<Eigen::Tensor<float, 2>, Eigen::Aligned> gpu_float(d_float, size, size); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 0>, Eigen::Aligned> gpu_res_half(d_res_half); - Eigen::TensorMap<Eigen::Tensor<Eigen::half, 0>, Eigen::Aligned> gpu_res_float(d_res_float); - - gpu_float.device(gpu_device) = gpu_float.random(); - - gpu_res_float.device(gpu_device) = gpu_float.sum().template cast<Eigen::half>(); - gpu_res_half.device(gpu_device) = gpu_float.template cast<Eigen::half>().sum(); - - Tensor<Eigen::half, 0> half_prec; - Tensor<Eigen::half, 0> full_prec; - gpu_device.memcpyDeviceToHost(half_prec.data(), d_res_half, sizeof(Eigen::half)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, sizeof(Eigen::half)); - gpu_device.synchronize(); - - VERIFY_IS_APPROX(full_prec(), half_prec()); - - gpu_res_float.device(gpu_device) = gpu_float.maximum().template cast<Eigen::half>(); - gpu_res_half.device(gpu_device) = gpu_float.template cast<Eigen::half>().maximum(); - gpu_device.memcpyDeviceToHost(half_prec.data(), d_res_half, sizeof(Eigen::half)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, sizeof(Eigen::half)); - gpu_device.synchronize(); - - VERIFY_IS_APPROX(full_prec(), half_prec()); - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_half); - gpu_device.deallocate(d_res_float); -} - -template <typename> -void test_gpu_forced_evals() { - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - int num_elem = 101; - - float* d_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_half1 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_half2 = (float*)gpu_device.allocate(num_elem * sizeof(float)); - float* d_res_float = (float*)gpu_device.allocate(num_elem * sizeof(float)); - - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_float(d_float, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_half1(d_res_half1, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Unaligned> gpu_res_half2(d_res_half2, num_elem); - Eigen::TensorMap<Eigen::Tensor<float, 1>, Eigen::Aligned> gpu_res_float(d_res_float, num_elem); - - Eigen::array<int, 1> no_bcast; - no_bcast[0] = 1; - - gpu_float.device(gpu_device) = gpu_float.random() - gpu_float.constant(0.5f); - gpu_res_float.device(gpu_device) = gpu_float.abs(); - gpu_res_half1.device(gpu_device) = gpu_float.template cast<Eigen::half>().abs().eval().template cast<float>(); - gpu_res_half2.device(gpu_device) = - gpu_float.template cast<Eigen::half>().abs().broadcast(no_bcast).eval().template cast<float>(); - - Tensor<float, 1> half_prec1(num_elem); - Tensor<float, 1> half_prec2(num_elem); - Tensor<float, 1> full_prec(num_elem); - gpu_device.memcpyDeviceToHost(half_prec1.data(), d_res_half1, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(half_prec2.data(), d_res_half2, num_elem * sizeof(float)); - gpu_device.memcpyDeviceToHost(full_prec.data(), d_res_float, num_elem * sizeof(float)); - gpu_device.synchronize(); - - for (int i = 0; i < num_elem; ++i) { - std::cout << "Checking forced eval " << i << full_prec(i) << " vs " << half_prec1(i) << " vs " << half_prec2(i) - << std::endl; - VERIFY_IS_APPROX(full_prec(i), half_prec1(i)); - VERIFY_IS_APPROX(full_prec(i), half_prec2(i)); - } - - gpu_device.deallocate(d_float); - gpu_device.deallocate(d_res_half1); - gpu_device.deallocate(d_res_half2); - gpu_device.deallocate(d_res_float); -} +#include "tensor_low_precision_gpu_test_helpers.h" EIGEN_DECLARE_TEST(tensor_of_float16_gpu) { - CALL_SUBTEST_1(test_gpu_numext<void>()); + CALL_SUBTEST_1(tensor_test::test_gpu_numext<Eigen::half>()); - CALL_SUBTEST_1(test_gpu_unary<void>()); - CALL_SUBTEST_1(test_gpu_elementwise<void>()); - CALL_SUBTEST_1(test_gpu_trancendental<void>()); - CALL_SUBTEST_2(test_gpu_contractions<void>()); - CALL_SUBTEST_3(test_gpu_reductions<void>()); - CALL_SUBTEST_4(test_gpu_full_reductions<void>()); - CALL_SUBTEST_5(test_gpu_forced_evals<void>()); + CALL_SUBTEST_1(tensor_test::test_gpu_unary<Eigen::half>()); + CALL_SUBTEST_1(tensor_test::test_gpu_elementwise<Eigen::half>()); + CALL_SUBTEST_1(tensor_test::test_gpu_transcendental<Eigen::half>()); + CALL_SUBTEST_2(tensor_test::test_gpu_contractions<Eigen::half>(16.f)); + CALL_SUBTEST_3(tensor_test::test_gpu_reductions<Eigen::half>()); + CALL_SUBTEST_4(tensor_test::test_gpu_full_reductions<Eigen::half>()); + CALL_SUBTEST_5(tensor_test::test_gpu_forced_evals<Eigen::half>()); }
diff --git a/contrib/test/tensor_random_gpu.cu b/contrib/test/tensor_random_gpu.cu index 9bb6fdb..b5348c3 100644 --- a/contrib/test/tensor_random_gpu.cu +++ b/contrib/test/tensor_random_gpu.cu
@@ -15,26 +15,19 @@ #define EIGEN_USE_GPU #include "main.h" -#include <Eigen/Tensor> +#include <contrib/Eigen/Tensor> +#include "tensor_gpu_test_helpers.h" void test_gpu_random_uniform() { Tensor<float, 2> out(72, 97); out.setZero(); - std::size_t out_bytes = out.size() * sizeof(float); + tensor_test::GpuTensorTestFixture fixture; + auto gpu_out = fixture.tensor<float, 2, Eigen::Unaligned>(72, 97); - float* d_out; - gpuMalloc((void**)(&d_out), out_bytes); + gpu_out.device(fixture.device()) = gpu_out.random(); - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - - Eigen::TensorMap<Eigen::Tensor<float, 2> > gpu_out(d_out, 72, 97); - - gpu_out.device(gpu_device) = gpu_out.random(); - - assert(gpuMemcpyAsync(out.data(), d_out, out_bytes, gpuMemcpyDeviceToHost, gpu_device.stream()) == gpuSuccess); - assert(gpuStreamSynchronize(gpu_device.stream()) == gpuSuccess); + out = fixture.toHost(gpu_out); // For now we just check this code doesn't crash. // TODO: come up with a valid test of randomness @@ -44,21 +37,13 @@ Tensor<float, 2> out(72, 97); out.setZero(); - std::size_t out_bytes = out.size() * sizeof(float); - - float* d_out; - gpuMalloc((void**)(&d_out), out_bytes); - - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - - Eigen::TensorMap<Eigen::Tensor<float, 2> > gpu_out(d_out, 72, 97); + tensor_test::GpuTensorTestFixture fixture; + auto gpu_out = fixture.tensor<float, 2, Eigen::Unaligned>(72, 97); Eigen::internal::NormalRandomGenerator<float> gen(true); - gpu_out.device(gpu_device) = gpu_out.random(gen); + gpu_out.device(fixture.device()) = gpu_out.random(gen); - assert(gpuMemcpyAsync(out.data(), d_out, out_bytes, gpuMemcpyDeviceToHost, gpu_device.stream()) == gpuSuccess); - assert(gpuStreamSynchronize(gpu_device.stream()) == gpuSuccess); + out = fixture.toHost(gpu_out); } template <typename Scalar> @@ -66,20 +51,12 @@ Tensor<Scalar, 2> out(rows, cols); out.setZero(); - std::size_t out_bytes = out.size() * sizeof(Scalar); + tensor_test::GpuTensorTestFixture fixture; + auto gpu_out = fixture.tensor<Scalar, 2, Eigen::Unaligned>(rows, cols); - Scalar* d_out; - gpuMalloc((void**)(&d_out), out_bytes); + gpu_out.device(fixture.device()) = gpu_out.random(); - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - - Eigen::TensorMap<Eigen::Tensor<Scalar, 2> > gpu_out(d_out, rows, cols); - - gpu_out.device(gpu_device) = gpu_out.random(); - - assert(gpuMemcpyAsync(out.data(), d_out, out_bytes, gpuMemcpyDeviceToHost, gpu_device.stream()) == gpuSuccess); - assert(gpuStreamSynchronize(gpu_device.stream()) == gpuSuccess); + out = fixture.toHost(gpu_out); // All uniform draws must lie in [0, 1). int num_out_of_range = 0; @@ -94,21 +71,13 @@ Tensor<Scalar, 2> out(rows, cols); out.setZero(); - std::size_t out_bytes = out.size() * sizeof(Scalar); - - Scalar* d_out; - gpuMalloc((void**)(&d_out), out_bytes); - - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - - Eigen::TensorMap<Eigen::Tensor<Scalar, 2> > gpu_out(d_out, rows, cols); + tensor_test::GpuTensorTestFixture fixture; + auto gpu_out = fixture.tensor<Scalar, 2, Eigen::Unaligned>(rows, cols); Eigen::internal::NormalRandomGenerator<Scalar> gen(true); - gpu_out.device(gpu_device) = gpu_out.random(gen); + gpu_out.device(fixture.device()) = gpu_out.random(gen); - assert(gpuMemcpyAsync(out.data(), d_out, out_bytes, gpuMemcpyDeviceToHost, gpu_device.stream()) == gpuSuccess); - assert(gpuStreamSynchronize(gpu_device.stream()) == gpuSuccess); + out = fixture.toHost(gpu_out); // Regression test for 16-bit types: the deviate must be computed in float, // otherwise log(0) in the rejection algorithm emits NaN/Inf. @@ -123,17 +92,10 @@ // takes the packet path, Unaligned the scalar one. template <typename Scalar, int MapOptions, typename Generator> static Tensor<Scalar, 1> device_random_fill(int n, const Generator& gen) { - const std::size_t bytes = n * sizeof(Scalar); - Eigen::GpuStreamDevice stream; - Eigen::GpuDevice gpu_device(&stream); - Scalar* d_out = static_cast<Scalar*>(gpu_device.allocate(bytes)); - Eigen::TensorMap<Eigen::Tensor<Scalar, 1>, MapOptions> gpu_out(d_out, n); - gpu_out.device(gpu_device) = gpu_out.random(gen); - Tensor<Scalar, 1> out(n); - gpu_device.memcpyDeviceToHost(out.data(), d_out, bytes); - gpu_device.synchronize(); - gpu_device.deallocate(d_out); - return out; + tensor_test::GpuTensorTestFixture fixture; + auto gpu_out = fixture.tensor<Scalar, 1, MapOptions>(n); + gpu_out.device(fixture.device()) = gpu_out.random(gen); + return fixture.toHost(gpu_out); } // The generator travels into the kernel by value, so every thread starts from the same state, yet each element has
diff --git a/contrib/test/tensor_test_helpers.h b/contrib/test/tensor_test_helpers.h new file mode 100644 index 0000000..b051ab2 --- /dev/null +++ b/contrib/test/tensor_test_helpers.h
@@ -0,0 +1,33 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_TENSOR_TEST_HELPERS_H +#define EIGEN_TENSOR_TEST_HELPERS_H + +#include <contrib/Eigen/Tensor> + +namespace Eigen { + +// Keep each suite's integer draw type so fixed seeds retain their inputs. +template <int NumDims, typename RandomIndex = Index> +DSizes<Index, NumDims> randomTensorDims(Index min = 1, Index max = 20) { + DSizes<Index, NumDims> dims; + for (int i = 0; i < NumDims; ++i) { + dims[i] = internal::random<RandomIndex>(RandomIndex(min), RandomIndex(max)); + } + return dims; +} + +inline internal::TensorBlockShapeType randomTensorBlockShape() { + return internal::random<bool>() ? internal::TensorBlockShapeType::kUniformAllDims + : internal::TensorBlockShapeType::kSkewedInnerDims; +} + +template <int NumDims> +size_t randomTensorBlockSize(const DSizes<Index, NumDims>& dims) { + return internal::random<size_t>(1, dims.TotalSize()); +} + +} // namespace Eigen + +#endif // EIGEN_TENSOR_TEST_HELPERS_H
diff --git a/contrib/test/tensor_thread_pool.cpp b/contrib/test/tensor_thread_pool.cpp index 83b5d49..86201fe 100644 --- a/contrib/test/tensor_thread_pool.cpp +++ b/contrib/test/tensor_thread_pool.cpp
@@ -13,6 +13,9 @@ #include "main.h" #include <atomic> #include <iostream> +#include <limits> +#include <stdexcept> +#include <thread> #include <Eigen/Tensor> using Eigen::Tensor; @@ -21,20 +24,21 @@ public: ~TestAllocator() EIGEN_OVERRIDE {} EIGEN_DEVICE_FUNC void* allocate(size_t num_bytes) const EIGEN_OVERRIDE { - const_cast<TestAllocator*>(this)->alloc_count_++; + alloc_count_++; return internal::aligned_malloc(num_bytes); } EIGEN_DEVICE_FUNC void deallocate(void* buffer) const EIGEN_OVERRIDE { - const_cast<TestAllocator*>(this)->dealloc_count_++; + dealloc_count_++; internal::aligned_free(buffer); } - int alloc_count() const { return alloc_count_; } - int dealloc_count() const { return dealloc_count_; } + int alloc_count() const { return alloc_count_.load(); } + int dealloc_count() const { return dealloc_count_.load(); } private: - int alloc_count_ = 0; - int dealloc_count_ = 0; + // Pool workers allocate through the device concurrently. + mutable std::atomic<int> alloc_count_{0}; + mutable std::atomic<int> dealloc_count_{0}; }; template <typename = void> @@ -555,6 +559,106 @@ } } +template <typename Contraction> +struct AsyncContractionEvalToProbe; + +namespace Eigen { +namespace internal { +template <typename Contraction> +struct traits<AsyncContractionEvalToProbe<Contraction>> : traits<TensorEvaluator<const Contraction, ThreadPoolDevice>> { +}; +} // namespace internal +} // namespace Eigen + +template <typename Contraction> +struct AsyncContractionEvalToProbe : Eigen::TensorContractionEvaluatorBase<AsyncContractionEvalToProbe<Contraction>> { + using Base = Eigen::TensorContractionEvaluatorBase<AsyncContractionEvalToProbe<Contraction>>; + using Scalar = typename Base::Scalar; + + AsyncContractionEvalToProbe(const Contraction& op, const Eigen::ThreadPoolDevice& device) : Base(op, device) {} + + template <int Alignment> + void evalProduct(Scalar*) const {} + + template <typename EvalToCallback, int Alignment> + void evalProductAsync(Scalar* buffer, EvalToCallback done) const { + static int shared_instantiation_calls = 0; + last_instantiation_calls = ++shared_instantiation_calls; + last_buffer = buffer; + done(); + } + + mutable Scalar* last_buffer = nullptr; + mutable int last_instantiation_calls = 0; +}; + +template <int DataLayout> +void test_async_contraction_eval_subexprs_dedup() { + using DimPair = typename Tensor<float, 1>::DimensionPair; + Eigen::ThreadPool tp(4); + Eigen::ThreadPoolDevice device(&tp, 4); + + Tensor<float, 2, DataLayout> left(4, 4); + Tensor<float, 2, DataLayout> right(4, 4); + Tensor<float, 2, DataLayout> dest(4, 4); + left.setRandom(); + right.setRandom(); + + Eigen::array<DimPair, 1> dims{{DimPair(1, 0)}}; + auto expr = left.contract(right, dims); + AsyncContractionEvalToProbe<decltype(expr)> probe(expr, device); + + bool need_assign = true; + auto on_done = [&need_assign](bool na) { need_assign = na; }; + + probe.evalSubExprsIfNeededAsync(dest.data(), on_done); + VERIFY_IS_EQUAL(need_assign, false); + VERIFY_IS_EQUAL(probe.last_buffer, dest.data()); + VERIFY_IS_EQUAL(probe.data(), static_cast<float*>(nullptr)); + VERIFY_IS_EQUAL(probe.last_instantiation_calls, 1); + + probe.evalSubExprsIfNeededAsync(nullptr, on_done); + VERIFY_IS_EQUAL(need_assign, true); + VERIFY(probe.data() != nullptr); + VERIFY_IS_EQUAL(probe.last_buffer, probe.data()); + VERIFY_IS_EQUAL(probe.last_instantiation_calls, 2); + probe.cleanup(); + VERIFY_IS_EQUAL(probe.data(), static_cast<float*>(nullptr)); + + // Verify real ThreadPoolDevice evaluation along the dest == nullptr branch + // (nested contraction subexpression) across parallel GEMM, inner-dim sharded, + // and k = 0 shapes. + const int shapes[][3] = {{64, 64, 64}, {2, 2048, 10}, {10, 0, 20}}; + for (const auto& shape : shapes) { + const int m = shape[0]; + const int k = shape[1]; + const int n = shape[2]; + Tensor<float, 2, DataLayout> t_left(m, k); + Tensor<float, 2, DataLayout> t_right(k, n); + Tensor<float, 2, DataLayout> t_bias(m, n); + t_left.setRandom(); + t_right.setRandom(); + t_bias.setRandom(); + t_left += t_left.constant(1.5f); + t_right += t_right.constant(1.5f); + t_bias += t_bias.constant(1.5f); + + Tensor<float, 2, DataLayout> st_result(m, n); + Tensor<float, 2, DataLayout> tp_result(m, n); + tp_result.setZero(); + + st_result = t_left.contract(t_right, dims) + t_bias; + + Eigen::Barrier barrier(1); + tp_result.device(device, [&barrier]() { barrier.Notify(); }) = t_left.contract(t_right, dims) + t_bias; + barrier.Wait(); + + for (Index i = 0; i < st_result.size(); ++i) { + VERIFY_IS_APPROX(tp_result.data()[i], st_result.data()[i]); + } + } +} + // We are triggering 'evalShardedByInnerDim' optimization. template <int DataLayout> static void test_sharded_by_inner_dim_contraction() { @@ -961,6 +1065,371 @@ for (int i = 0; i < 64; ++i) VERIFY_IS_EQUAL(out(i), in(i)); } +// A scalar that requires initialization: its constructors set a magic value that operator= checks, and it counts +// live instances so that missing destructor calls are detected. +struct TrackedScalar { + static constexpr unsigned kMagic = 0x5EEDu; + static std::atomic<int> instances; + static std::atomic<int> unconstructed_assignments; + // Assignments on threads other than test_thread, to check that the work ran on the pool. + static std::atomic<int> pool_assignments; + // When positive, the throw_countdown-th default construction on test_thread throws. + static std::atomic<int> throw_countdown; + // When positive, the multiply_countdown-th multiplication on test_thread throws. + static std::atomic<int> multiply_countdown; + static std::thread::id test_thread; + float v; + unsigned magic; + TrackedScalar() : v(0.0f), magic(kMagic) { + if (throw_countdown.load() > 0 && std::this_thread::get_id() == test_thread) { + const int remaining = --throw_countdown; + if (remaining == 0) EIGEN_THROW_X(std::runtime_error("TrackedScalar")); + } + ++instances; + } + TrackedScalar(float f) : v(f), magic(kMagic) { ++instances; } + TrackedScalar(const TrackedScalar& o) : TrackedScalar(o.v) {} + ~TrackedScalar() { + magic = 0; + --instances; + } + TrackedScalar& operator=(const TrackedScalar& o) { + if (magic != kMagic) ++unconstructed_assignments; + if (std::this_thread::get_id() != test_thread) ++pool_assignments; + v = o.v; + return *this; + } + TrackedScalar operator+(const TrackedScalar& o) const { return TrackedScalar(v + o.v); } + TrackedScalar operator-(const TrackedScalar& o) const { return TrackedScalar(v - o.v); } + TrackedScalar operator*(const TrackedScalar& o) const { + if (multiply_countdown.load() > 0 && std::this_thread::get_id() == test_thread) { + const int remaining = --multiply_countdown; + if (remaining == 0) EIGEN_THROW_X(std::runtime_error("TrackedScalar")); + } + return TrackedScalar(v * o.v); + } + TrackedScalar operator-() const { return TrackedScalar(-v); } + TrackedScalar& operator+=(const TrackedScalar& o) { return *this = *this + o; } + TrackedScalar& operator*=(const TrackedScalar& o) { return *this = *this * o; } + bool operator==(const TrackedScalar& o) const { return v == o.v; } + bool operator!=(const TrackedScalar& o) const { return v != o.v; } +}; +std::atomic<int> TrackedScalar::instances(0); +std::atomic<int> TrackedScalar::unconstructed_assignments(0); +std::atomic<int> TrackedScalar::pool_assignments(0); +std::atomic<int> TrackedScalar::throw_countdown(0); +std::atomic<int> TrackedScalar::multiply_countdown(0); +std::thread::id TrackedScalar::test_thread; + +namespace Eigen { +template <> +struct NumTraits<TrackedScalar> : GenericNumTraits<TrackedScalar> { + using Real = TrackedScalar; + using NonInteger = TrackedScalar; + using Literal = TrackedScalar; + using Nested = TrackedScalar; + static constexpr int IsComplex = 0; + static constexpr int IsInteger = 0; + static constexpr int IsSigned = 1; + static constexpr int RequireInitialization = 1; + static constexpr int ReadCost = 1; + static constexpr int AddCost = 1; + static constexpr int MulCost = 1; +}; +} // namespace Eigen + +// The thread pool contraction paths, identified by the allocations of one direct contraction: parallel packing +// allocates its packed blocks once, thread-local packing (parallelize_by_sharding_dim_only) also allocates the +// thread-local blocks, and sharding by the inner dimension allocates a buffer and packed blocks per block. +enum class ContractionPath { ParallelPack, ThreadLocalPack, ShardByInnerDim }; + +// The path depends on the cache sizes, so the tests that check it pin them to typical x86 values. +struct ScopedCacheSizes { + ScopedCacheSizes() : l1(l1CacheSize()), l2(l2CacheSize()), l3(l3CacheSize()) { + setCpuCacheSizes(32 * 1024, 1024 * 1024, 32 * 1024 * 1024); + } + ~ScopedCacheSizes() { setCpuCacheSizes(l1, l2, l3); } + std::ptrdiff_t l1, l2, l3; +}; + +// Issue #2714: the threaded contraction paths, async evaluation and temporary results construct and destruct scalars +// that require initialization. +template <int DataLayout> +static void test_multithread_contraction_require_initialization(Index m, Index k, Index n, ContractionPath path) { + using DimPair = typename Tensor<float, 1>::DimensionPair; + const ScopedCacheSizes cache_sizes; + TestAllocator test_allocator; + TrackedScalar::test_thread = std::this_thread::get_id(); + + Tensor<float, 2, DataLayout> left(m, k); + Tensor<float, 2, DataLayout> right(k, n); + left.setRandom(); + right.setRandom(); + Eigen::array<DimPair, 1> dims{{DimPair(1, 0)}}; + Tensor<float, 2, DataLayout> expected = left.contract(right, dims); + + TrackedScalar::instances = 0; + TrackedScalar::unconstructed_assignments = 0; + { + // Destroying the pool at the end of this block joins the workers before the checks below: the kernel task that + // sends the final notification still holds alpha and beta until its frame unwinds. + Eigen::ThreadPool tp(4); + Eigen::ThreadPoolDevice device(&tp, 4, &test_allocator); + Tensor<TrackedScalar, 2, DataLayout> t_left = left.template cast<TrackedScalar>(); + Tensor<TrackedScalar, 2, DataLayout> t_right = right.template cast<TrackedScalar>(); + Tensor<TrackedScalar, 2, DataLayout> result(m, n); + const auto check = [&](float scale) { + for (Index i = 0; i < result.size(); ++i) { + VERIFY_IS_APPROX(result.data()[i].v, scale * expected.data()[i]); + } + }; + + TrackedScalar::pool_assignments = 0; + const int allocations_before = test_allocator.alloc_count(); + result.device(device) = t_left.contract(t_right, dims); + check(1); + VERIFY(TrackedScalar::pool_assignments.load() > 0); + const int allocations = test_allocator.alloc_count() - allocations_before; + switch (path) { + case ContractionPath::ParallelPack: + VERIFY_IS_EQUAL(allocations, 1); + break; + case ContractionPath::ThreadLocalPack: + VERIFY_IS_EQUAL(allocations, 2); + break; + case ContractionPath::ShardByInnerDim: + VERIFY(allocations >= 3); + break; + } + // A contraction nested in another expression evaluates into a temporary result buffer. The buffer does not + // depend on the path, so the slowest shape skips it. + const bool nested = path != ContractionPath::ThreadLocalPack; + if (nested) { + result.device(device) = t_left.contract(t_right, dims) + t_left.contract(t_right, dims); + check(2); + } + + Eigen::Barrier done(1); + result.device(device, [&done]() { done.Notify(); }) = t_left.contract(t_right, dims); + done.Wait(); + check(1); + if (nested) { + Eigen::Barrier done_nested(1); + result.device(device, [&done_nested]() { done_nested.Notify(); }) = + t_left.contract(t_right, dims) + t_left.contract(t_right, dims); + done_nested.Wait(); + check(2); + } + } + VERIFY_IS_EQUAL(TrackedScalar::unconstructed_assignments.load(), 0); + VERIFY_IS_EQUAL(TrackedScalar::instances.load(), 0); +} + +// A TestAllocator whose throw_countdown-th allocation on TrackedScalar::test_thread throws. +class ThrowingAllocator : public TestAllocator { + public: + EIGEN_DEVICE_FUNC void* allocate(size_t num_bytes) const EIGEN_OVERRIDE { + if (throw_countdown.load() > 0 && std::this_thread::get_id() == TrackedScalar::test_thread) { + const int remaining = --throw_countdown; + if (remaining == 0) EIGEN_THROW_X(std::runtime_error("ThrowingAllocator")); + } + return TestAllocator::allocate(num_bytes); + } + mutable std::atomic<int> throw_countdown{0}; +}; + +// Issue #2714: if a constructor throws partway through constructing the contraction buffers, the elements constructed +// so far are destructed and the buffers are freed. +template <int DataLayout> +static void test_contraction_require_initialization_throws() { +#ifdef EIGEN_EXCEPTIONS + using DimPair = typename Tensor<float, 1>::DimensionPair; + const ScopedCacheSizes cache_sizes; + Eigen::ThreadPool tp(4); + ThrowingAllocator test_allocator; + Eigen::ThreadPoolDevice device(&tp, 4, &test_allocator); + TrackedScalar::test_thread = std::this_thread::get_id(); + + // Runs f with the count-th default construction on this thread throwing; returns whether it threw. + const auto throws_at = [](int count, const auto& f) { + TrackedScalar::throw_countdown = count; + bool threw = false; + try { + f(); + } catch (const std::runtime_error&) { + threw = true; + } + TrackedScalar::throw_countdown = 0; + return threw; + }; + const auto verify_released = [&](int instances) { + VERIFY_IS_EQUAL(TrackedScalar::instances.load(), instances); + VERIFY_IS_EQUAL(test_allocator.alloc_count(), test_allocator.dealloc_count()); + }; + + // Throws at every element the allocator constructs. With float lhs blocks, only the rhs blocks are constructed. + const auto sweep_allocator = [&](auto* lhs_tag, int lhs_constructed) { + using LhsScalar = std::remove_pointer_t<decltype(lhs_tag)>; + using BlockAllocator = internal::TensorContractionBlockMemAllocator<LhsScalar, TrackedScalar>; + const Index bm = 3, bk = 4, bn = 5; + const int block_elements = lhs_constructed * bm * bk + bn * bk; + for (int count = 1; count <= block_elements; ++count) { + LhsScalar* lhs = nullptr; + TrackedScalar* rhs = nullptr; + VERIFY(throws_at(count, [&]() { BlockAllocator::allocate(device, bm, bk, bn, &lhs, &rhs); })); + verify_released(0); + } + + const Index num_lhs = 2, num_rhs = 3, num_slices = 2; + const int slice_elements = (lhs_constructed * num_lhs * bm * bk + num_rhs * bn * bk) * num_slices; + for (int count = 1; count <= slice_elements; ++count) { + std::vector<LhsScalar*> lhs_blocks[num_slices]; + std::vector<TrackedScalar*> rhs_blocks[num_slices]; + VERIFY(throws_at(count, [&]() { + BlockAllocator::allocateSlices(device, bm, bk, bn, num_lhs, num_rhs, num_slices, lhs_blocks, rhs_blocks); + })); + verify_released(0); + } + }; + TrackedScalar::instances = 0; + sweep_allocator(static_cast<TrackedScalar*>(nullptr), 1); + sweep_allocator(static_cast<float*>(nullptr), 0); + + // Parallel evaluation: the EvalParallelContext constructor allocates the packed blocks and, with thread-local + // packing, the thread-local blocks, all on this thread and before any task starts. A throw while constructing the + // first block, or from either allocation, releases what the constructor allocated. + { + Eigen::array<DimPair, 1> dims{{DimPair(1, 0)}}; + const auto parallel_throws = [&](Index pm, Index pk, Index pn, int allocations) { + Tensor<TrackedScalar, 2, DataLayout> t_left(pm, pk); + Tensor<TrackedScalar, 2, DataLayout> t_right(pk, pn); + Tensor<TrackedScalar, 2, DataLayout> result(pm, pn); + t_left.setConstant(TrackedScalar(1.0f)); + t_right.setConstant(TrackedScalar(1.0f)); + const auto contract = [&]() { result.device(device) = t_left.contract(t_right, dims); }; + const int live = TrackedScalar::instances.load(); + VERIFY(throws_at(1, contract)); + verify_released(live); + for (int allocation = 1; allocation <= allocations; ++allocation) { + test_allocator.throw_countdown = allocation; + VERIFY(throws_at(0, contract)); + test_allocator.throw_countdown = 0; + verify_released(live); + } + }; + // The paths of these shapes are checked in test_multithread_contraction_require_initialization. + parallel_throws(131, 67, 509, /*allocations=*/1); + parallel_throws(3, 127, 20001, /*allocations=*/2); + } + + // Sharded by the inner dimension: the result of a nested contraction, then the per-block buffers. + const Index m = 3, k = 100003, n = 11; + Tensor<float, 2, DataLayout> left(m, k); + Tensor<float, 2, DataLayout> right(k, n); + left.setRandom(); + right.setRandom(); + Eigen::array<DimPair, 1> dims{{DimPair(1, 0)}}; + Tensor<float, 2, DataLayout> expected = left.contract(right, dims); + { + Tensor<TrackedScalar, 2, DataLayout> t_left = left.template cast<TrackedScalar>(); + Tensor<TrackedScalar, 2, DataLayout> t_right = right.template cast<TrackedScalar>(); + Tensor<TrackedScalar, 2, DataLayout> result(m, n); + const int live = TrackedScalar::instances.load(); + const int allocations_before = test_allocator.alloc_count(); + result.device(device) = t_left.contract(t_right, dims); + // Sharding by the inner dimension allocates a buffer and packed blocks per block (see ContractionPath). + VERIFY(test_allocator.alloc_count() - allocations_before >= 3); + const int result_elements = static_cast<int>(m * n); + for (int count : {1, result_elements, result_elements + 1}) { + VERIFY(throws_at(count, [&]() { result.device(device) = t_left.contract(t_right, dims); })); + verify_released(live); + VERIFY(throws_at( + count, [&]() { result.device(device) = t_left.contract(t_right, dims) + t_left.contract(t_right, dims); })); + verify_released(live); + } + + result.device(device) = t_left.contract(t_right, dims); + for (Index i = 0; i < result.size(); ++i) VERIFY_IS_APPROX(result.data()[i].v, expected.data()[i]); + verify_released(live); + } + VERIFY_IS_EQUAL(TrackedScalar::instances.load(), 0); + + // A throw in the multiplication of a sequential contraction releases the packed blocks, the temporary result and + // the operands, here a nested contraction with its own temporary result. A throw in a later operand of an + // expression releases the operands evaluated before it. + { + Tensor<TrackedScalar, 2, DataLayout> t_left(9, 10); + Tensor<TrackedScalar, 2, DataLayout> t_right(10, 11); + Tensor<TrackedScalar, 2, DataLayout> t_square(11, 11); + Tensor<TrackedScalar, 2, DataLayout> result(9, 11); + Tensor<TrackedScalar, 2, DataLayout> concatenated(18, 11); + t_left.setConstant(TrackedScalar(1.0f)); + t_right.setConstant(TrackedScalar(2.0f)); + t_square.setConstant(TrackedScalar(1.0f)); + // Runs f with the count-th multiplication on this thread throwing; returns whether it threw. + const auto throws_in_multiply = [](int count, const auto& f) { + TrackedScalar::multiply_countdown = count; + bool threw = false; + try { + f(); + } catch (const std::runtime_error&) { + threw = true; + } + TrackedScalar::multiply_countdown = 0; + return threw; + }; + const int live = TrackedScalar::instances.load(); + TrackedScalar::pool_assignments = 0; + const int countdown = std::numeric_limits<int>::max(); + TrackedScalar::multiply_countdown = countdown; + result.device(device) = t_left.contract(t_right, dims); + const int multiplications = countdown - TrackedScalar::multiply_countdown.load(); + TrackedScalar::multiply_countdown = 0; + VERIFY_IS_EQUAL(TrackedScalar::pool_assignments.load(), 0); + VERIFY_IS_EQUAL(result(0, 0).v, 20.0f); + + VERIFY(throws_in_multiply(1, [&]() { result.device(device) = t_left.contract(t_right, dims); })); + verify_released(live); + VERIFY(throws_in_multiply(1, [&]() { result.device(device) = t_left.contract(t_right, dims) + result; })); + verify_released(live); + // The nested contraction is the operand: it completes, then the outer contraction throws. + VERIFY(throws_in_multiply(multiplications + 1, [&]() { + result.device(device) = t_left.contract(t_right, dims).contract(t_square, dims); + })); + verify_released(live); + + // A later operand throws after an earlier one completed: the earlier operand's temporary result is released. + const auto contraction = [&]() { return t_left.contract(t_right, dims); }; + VERIFY(throws_in_multiply(multiplications + 1, [&]() { result.device(device) = contraction() + contraction(); })); + verify_released(live); + VERIFY(throws_in_multiply(multiplications + 1, [&]() { + result.device(device) = contraction().contract(t_square.contract(t_square, dims), dims); + })); + verify_released(live); + Tensor<bool, 2, DataLayout> t_cond(9, 11); + t_cond.setConstant(true); + VERIFY(throws_in_multiply(multiplications + 1, + [&]() { result.device(device) = t_cond.select(contraction(), contraction()); })); + verify_released(live); + VERIFY(throws_in_multiply(multiplications + 1, + [&]() { concatenated.device(device) = contraction().concatenate(contraction(), 0); })); + verify_released(live); + + // A throw in the DefaultDevice executor's own loop, after the operands were evaluated, releases them. The + // ThreadPoolDevice executor is not covered: its loop can still be running on other threads when this one throws. + Eigen::DefaultDevice default_device; + TrackedScalar::multiply_countdown = countdown; + result.device(default_device) = t_left.contract(t_right, dims); + const int default_multiplications = countdown - TrackedScalar::multiply_countdown.load(); + TrackedScalar::multiply_countdown = 0; + VERIFY(throws_in_multiply(default_multiplications + 1, + [&]() { result.device(default_device) = contraction() * result; })); + verify_released(live); + } + VERIFY_IS_EQUAL(TrackedScalar::instances.load(), 0); +#endif +} + template <typename = void> void test_enqueue_with_args() { Eigen::ThreadPool tp(2); @@ -1344,6 +1813,8 @@ CALL_SUBTEST_6(test_async_multithread_contraction_agrees_with_singlethread<ColMajor>()); CALL_SUBTEST_6(test_async_multithread_contraction_agrees_with_singlethread<RowMajor>()); + CALL_SUBTEST_6(test_async_contraction_eval_subexprs_dedup<ColMajor>()); + CALL_SUBTEST_6(test_async_contraction_eval_subexprs_dedup<RowMajor>()); // Test EvalShardedByInnerDimContext parallelization strategy. CALL_SUBTEST_7(test_sharded_by_inner_dim_contraction<ColMajor>()); @@ -1378,6 +1849,20 @@ CALL_SUBTEST_12(test_enqueue_with_args<>()); CALL_SUBTEST_12(test_async_forced_eval_non_pod<>()); CALL_SUBTEST_12(test_async_single_block_scratch_freed_before_done<>()); + CALL_SUBTEST_12( + test_multithread_contraction_require_initialization<ColMajor>(131, 67, 509, ContractionPath::ParallelPack)); + CALL_SUBTEST_12( + test_multithread_contraction_require_initialization<RowMajor>(131, 67, 509, ContractionPath::ParallelPack)); + CALL_SUBTEST_12( + test_multithread_contraction_require_initialization<ColMajor>(3, 127, 20001, ContractionPath::ThreadLocalPack)); + CALL_SUBTEST_12( + test_multithread_contraction_require_initialization<RowMajor>(3, 127, 20001, ContractionPath::ThreadLocalPack)); + CALL_SUBTEST_12( + test_multithread_contraction_require_initialization<ColMajor>(3, 100003, 11, ContractionPath::ShardByInnerDim)); + CALL_SUBTEST_12( + test_multithread_contraction_require_initialization<RowMajor>(3, 100003, 11, ContractionPath::ShardByInnerDim)); + CALL_SUBTEST_12(test_contraction_require_initialization_throws<ColMajor>()); + CALL_SUBTEST_12(test_contraction_require_initialization_throws<RowMajor>()); TestAllocator test_allocator; CALL_SUBTEST_13(test_multithread_shuffle<ColMajor>(nullptr));
diff --git a/demos/CMakeLists.txt b/demos/CMakeLists.txt deleted file mode 100644 index a27f047..0000000 --- a/demos/CMakeLists.txt +++ /dev/null
@@ -1,16 +0,0 @@ -# SPDX-FileCopyrightText: The Eigen Authors -# SPDX-License-Identifier: MPL-2.0 - -project(EigenDemos) - -add_custom_target(demos) - -if(NOT EIGEN_TEST_NOQT) - find_package(Qt4) - if(QT4_FOUND) - add_subdirectory(mandelbrot) - add_subdirectory(opengl) - else() - message(STATUS "Qt4 not found, so disabling the mandelbrot and opengl demos") - endif() -endif()
diff --git a/demos/mandelbrot/CMakeLists.txt b/demos/mandelbrot/CMakeLists.txt deleted file mode 100644 index 66dd13f..0000000 --- a/demos/mandelbrot/CMakeLists.txt +++ /dev/null
@@ -1,24 +0,0 @@ -# SPDX-FileCopyrightText: The Eigen Authors -# SPDX-License-Identifier: MPL-2.0 - -find_package(Qt4 REQUIRED) - -set(CMAKE_INCLUDE_CURRENT_DIR ON) - -if (CMAKE_COMPILER_IS_GNUCXX) - set ( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O2") - add_definitions ( "-DNDEBUG" ) -endif () - -include_directories( ${QT_INCLUDE_DIR} ) - -set(mandelbrot_SRCS - mandelbrot.cpp -) - -qt4_automoc(${mandelbrot_SRCS}) - -add_executable(mandelbrot ${mandelbrot_SRCS}) -add_dependencies(demos mandelbrot) - -target_link_libraries(mandelbrot ${QT_QTCORE_LIBRARY} ${QT_QTGUI_LIBRARY})
diff --git a/demos/mandelbrot/README b/demos/mandelbrot/README deleted file mode 100644 index a451d65..0000000 --- a/demos/mandelbrot/README +++ /dev/null
@@ -1,10 +0,0 @@ -*** Mandelbrot demo *** - -Controls: -* Left mouse button to center view at a point. -* Drag vertically with left mouse button to zoom in and out. - -Be sure to enable SSE2 or AltiVec to improve performance. - -The number of iterations, and the choice between single and double precision, are -determined at runtime depending on the zoom level.
diff --git a/demos/mandelbrot/mandelbrot.cpp b/demos/mandelbrot/mandelbrot.cpp deleted file mode 100644 index 9c192fe..0000000 --- a/demos/mandelbrot/mandelbrot.cpp +++ /dev/null
@@ -1,197 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Benoit Jacob <jacob.benoit.1@gmail.com> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#include "mandelbrot.h" -#include <iostream> -#include <QtGui/QPainter> -#include <QtGui/QImage> -#include <QtGui/QMouseEvent> -#include <QtCore/QTime> - -void MandelbrotWidget::resizeEvent(QResizeEvent *) { - if (size < width() * height()) { - std::cout << "reallocate buffer" << std::endl; - size = width() * height(); - if (buffer) delete[] buffer; - buffer = new unsigned char[4 * size]; - } -} - -template <typename T> -struct iters_before_test { - enum { ret = 8 }; -}; -template <> -struct iters_before_test<double> { - enum { ret = 16 }; -}; - -template <typename Real> -void MandelbrotThread::render(int img_width, int img_height) { - enum { packetSize = Eigen::internal::packet_traits<Real>::size }; // number of reals in a Packet - typedef Eigen::Array<Real, packetSize, 1> Packet; // wrap a Packet as a vector - - enum { iters_before_test = iters_before_test<Real>::ret }; - max_iter = (max_iter / iters_before_test) * iters_before_test; - const int alignedWidth = (img_width / packetSize) * packetSize; - unsigned char *const buffer = widget->buffer; - const double xradius = widget->xradius; - const double yradius = xradius * img_height / img_width; - const int threadcount = widget->threadcount; - typedef Eigen::Array<Real, 2, 1> Vector2; - Vector2 start(widget->center.x() - widget->xradius, widget->center.y() - yradius); - Vector2 step(2 * widget->xradius / img_width, 2 * yradius / img_height); - total_iter = 0; - - for (int y = id; y < img_height; y += threadcount) { - int pix = y * img_width; - - // for each pixel, we're going to do the iteration z := z^2 + c where z and c are complex numbers, - // starting with z = c = complex coord of the pixel. pzi and pzr denote the real and imaginary parts of z. - // pci and pcr denote the real and imaginary parts of c. - - Packet pzi_start, pci_start; - for (int i = 0; i < packetSize; i++) pzi_start[i] = pci_start[i] = start.y() + y * step.y(); - - for (int x = 0; x < alignedWidth; x += packetSize, pix += packetSize) { - Packet pcr, pci = pci_start, pzr, pzi = pzi_start, pzr_buf; - for (int i = 0; i < packetSize; i++) pzr[i] = pcr[i] = start.x() + (x + i) * step.x(); - - // do the iterations. Every iters_before_test iterations we check for divergence, - // in which case we can stop iterating. - int j = 0; - typedef Eigen::Matrix<int, packetSize, 1> Packeti; - Packeti pix_iter = Packeti::Zero(), // number of iteration per pixel in the packet - pix_dont_diverge; // whether or not each pixel has already diverged - do { - for (int i = 0; i < iters_before_test / 4; i++) // peel the inner loop by 4 - { -#define ITERATE \ - pzr_buf = pzr; \ - pzr = pzr.square(); \ - pzr -= pzi.square(); \ - pzr += pcr; \ - pzi = (2 * pzr_buf) * pzi; \ - pzi += pci; - ITERATE ITERATE ITERATE ITERATE - } - pix_dont_diverge = - ((pzr.square() + pzi.square()).eval() // temporary fix as what follows is not yet vectorized by Eigen - <= Packet::Constant(4)) - // the 4 here is not a magic value, it's a math fact that if - // the square modulus is >4 then divergence is inevitable. - .template cast<int>(); - pix_iter += iters_before_test * pix_dont_diverge; - j++; - total_iter += iters_before_test * packetSize; - } while (j < max_iter / iters_before_test && pix_dont_diverge.any()); // any() is not yet vectorized by Eigen - - // compute pixel colors - for (int i = 0; i < packetSize; i++) { - buffer[4 * (pix + i)] = 255 * pix_iter[i] / max_iter; - buffer[4 * (pix + i) + 1] = 0; - buffer[4 * (pix + i) + 2] = 0; - } - } - - // if the width is not a multiple of packetSize, fill the remainder in black - for (int x = alignedWidth; x < img_width; x++, pix++) - buffer[4 * pix] = buffer[4 * pix + 1] = buffer[4 * pix + 2] = 0; - } - return; -} - -void MandelbrotThread::run() { - setTerminationEnabled(true); - double resolution = widget->xradius * 2 / widget->width(); - max_iter = 128; - if (resolution < 1e-4f) max_iter += 128 * (-4 - std::log10(resolution)); - int img_width = widget->width() / widget->draft; - int img_height = widget->height() / widget->draft; - single_precision = resolution > 1e-7f; - - if (single_precision) - render<float>(img_width, img_height); - else - render<double>(img_width, img_height); -} - -void MandelbrotWidget::paintEvent(QPaintEvent *) { - static float max_speed = 0; - long long total_iter = 0; - - QTime time; - time.start(); - for (int th = 0; th < threadcount; th++) threads[th]->start(QThread::LowPriority); - for (int th = 0; th < threadcount; th++) { - threads[th]->wait(); - total_iter += threads[th]->total_iter; - } - int elapsed = time.elapsed(); - - if (draft == 1) { - float speed = elapsed ? float(total_iter) * 1000 / elapsed : 0; - max_speed = std::max(max_speed, speed); - std::cout << threadcount << " threads, " << elapsed << " ms, " << speed << " iters/s (max " << max_speed << ")" - << std::endl; - int packetSize = threads[0]->single_precision ? int(Eigen::internal::packet_traits<float>::size) - : int(Eigen::internal::packet_traits<double>::size); - setWindowTitle( - QString("resolution ") + QString::number(xradius * 2 / width(), 'e', 2) + - QString(", %1 iterations per pixel, ").arg(threads[0]->max_iter) + - (threads[0]->single_precision ? QString("single ") : QString("double ")) + QString("precision, ") + - (packetSize == 1 ? QString("no vectorization") : QString("vectorized (%1 per packet)").arg(packetSize))); - } - - QImage image(buffer, width() / draft, height() / draft, QImage::Format_RGB32); - QPainter painter(this); - painter.drawImage(QPoint(0, 0), image.scaled(width(), height())); - - if (draft > 1) { - draft /= 2; - setWindowTitle(QString("recomputing at 1/%1 resolution...").arg(draft)); - update(); - } -} - -void MandelbrotWidget::mousePressEvent(QMouseEvent *event) { - if (event->buttons() & Qt::LeftButton) { - lastpos = event->pos(); - double yradius = xradius * height() / width(); - center = Eigen::Vector2d(center.x() + (event->pos().x() - width() / 2) * xradius * 2 / width(), - center.y() + (event->pos().y() - height() / 2) * yradius * 2 / height()); - draft = 16; - for (int th = 0; th < threadcount; th++) threads[th]->terminate(); - update(); - } -} - -void MandelbrotWidget::mouseMoveEvent(QMouseEvent *event) { - QPoint delta = event->pos() - lastpos; - lastpos = event->pos(); - if (event->buttons() & Qt::LeftButton) { - double t = 1 + 5 * double(delta.y()) / height(); - if (t < 0.5) t = 0.5; - if (t > 2) t = 2; - xradius *= t; - draft = 16; - for (int th = 0; th < threadcount; th++) threads[th]->terminate(); - update(); - } -} - -int main(int argc, char *argv[]) { - QApplication app(argc, argv); - MandelbrotWidget w; - w.show(); - return app.exec(); -} - -#include "mandelbrot.moc"
diff --git a/demos/mandelbrot/mandelbrot.h b/demos/mandelbrot/mandelbrot.h deleted file mode 100644 index 0b7d9a6..0000000 --- a/demos/mandelbrot/mandelbrot.h +++ /dev/null
@@ -1,68 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Benoit Jacob <jacob.benoit.1@gmail.com> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#ifndef MANDELBROT_H -#define MANDELBROT_H - -#include <Eigen/Core> -#include <QtGui/QApplication> -#include <QtGui/QWidget> -#include <QtCore/QThread> - -class MandelbrotWidget; - -class MandelbrotThread : public QThread { - friend class MandelbrotWidget; - MandelbrotWidget *widget; - long long total_iter; - int id, max_iter; - bool single_precision; - - public: - MandelbrotThread(MandelbrotWidget *w, int i) : widget(w), id(i) {} - void run(); - template <typename Real> - void render(int img_width, int img_height); -}; - -class MandelbrotWidget : public QWidget { - Q_OBJECT - - friend class MandelbrotThread; - Eigen::Vector2d center; - double xradius; - int size; - unsigned char *buffer; - QPoint lastpos; - int draft; - MandelbrotThread **threads; - int threadcount; - - protected: - void resizeEvent(QResizeEvent *); - void paintEvent(QPaintEvent *); - void mousePressEvent(QMouseEvent *event); - void mouseMoveEvent(QMouseEvent *event); - - public: - MandelbrotWidget() : QWidget(), center(0, 0), xradius(2), size(0), buffer(0), draft(16) { - setAutoFillBackground(false); - threadcount = QThread::idealThreadCount(); - threads = new MandelbrotThread *[threadcount]; - for (int th = 0; th < threadcount; th++) threads[th] = new MandelbrotThread(this, th); - } - ~MandelbrotWidget() { - if (buffer) delete[] buffer; - for (int th = 0; th < threadcount; th++) delete threads[th]; - delete[] threads; - } -}; - -#endif // MANDELBROT_H
diff --git a/demos/mix_eigen_and_c/README b/demos/mix_eigen_and_c/README deleted file mode 100644 index d9cc927..0000000 --- a/demos/mix_eigen_and_c/README +++ /dev/null
@@ -1,9 +0,0 @@ -This is an example of how one can wrap some of Eigen into a C library. - -To try this with GCC, do: - - g++ -c binary_library.cpp -O2 -msse2 -I ../.. - gcc example.c binary_library.o -o example -lstdc++ - ./example - -TODO: add CMakeLists, add more explanations here
diff --git a/demos/mix_eigen_and_c/binary_library.cpp b/demos/mix_eigen_and_c/binary_library.cpp deleted file mode 100644 index 4da74b6..0000000 --- a/demos/mix_eigen_and_c/binary_library.cpp +++ /dev/null
@@ -1,104 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2009 Benoit Jacob <jacob.benoit.1@gmail.com> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -// This C++ file compiles to binary code that can be linked to by your C program, -// thanks to the extern "C" syntax used in the declarations in binary_library.h. - -#include "binary_library.h" - -#include <Eigen/Core> - -using namespace Eigen; - -/************************* pointer conversion methods **********************************************/ - -////// class MatrixXd ////// - -inline MatrixXd &c_to_eigen(C_MatrixXd *ptr) { return *reinterpret_cast<MatrixXd *>(ptr); } - -inline const MatrixXd &c_to_eigen(const C_MatrixXd *ptr) { return *reinterpret_cast<const MatrixXd *>(ptr); } - -inline C_MatrixXd *eigen_to_c(MatrixXd &ref) { return reinterpret_cast<C_MatrixXd *>(&ref); } - -inline const C_MatrixXd *eigen_to_c(const MatrixXd &ref) { return reinterpret_cast<const C_MatrixXd *>(&ref); } - -////// class Map<MatrixXd> ////// - -inline Map<MatrixXd> &c_to_eigen(C_Map_MatrixXd *ptr) { return *reinterpret_cast<Map<MatrixXd> *>(ptr); } - -inline const Map<MatrixXd> &c_to_eigen(const C_Map_MatrixXd *ptr) { - return *reinterpret_cast<const Map<MatrixXd> *>(ptr); -} - -inline C_Map_MatrixXd *eigen_to_c(Map<MatrixXd> &ref) { return reinterpret_cast<C_Map_MatrixXd *>(&ref); } - -inline const C_Map_MatrixXd *eigen_to_c(const Map<MatrixXd> &ref) { - return reinterpret_cast<const C_Map_MatrixXd *>(&ref); -} - -/************************* implementation of classes **********************************************/ - -////// class MatrixXd ////// - -C_MatrixXd *MatrixXd_new(int rows, int cols) { return eigen_to_c(*new MatrixXd(rows, cols)); } - -void MatrixXd_delete(C_MatrixXd *m) { delete &c_to_eigen(m); } - -double *MatrixXd_data(C_MatrixXd *m) { return c_to_eigen(m).data(); } - -void MatrixXd_set_zero(C_MatrixXd *m) { c_to_eigen(m).setZero(); } - -void MatrixXd_resize(C_MatrixXd *m, int rows, int cols) { c_to_eigen(m).resize(rows, cols); } - -void MatrixXd_copy(C_MatrixXd *dst, const C_MatrixXd *src) { c_to_eigen(dst) = c_to_eigen(src); } - -void MatrixXd_copy_map(C_MatrixXd *dst, const C_Map_MatrixXd *src) { c_to_eigen(dst) = c_to_eigen(src); } - -void MatrixXd_set_coeff(C_MatrixXd *m, int i, int j, double coeff) { c_to_eigen(m)(i, j) = coeff; } - -double MatrixXd_get_coeff(const C_MatrixXd *m, int i, int j) { return c_to_eigen(m)(i, j); } - -void MatrixXd_print(const C_MatrixXd *m) { std::cout << c_to_eigen(m) << std::endl; } - -void MatrixXd_multiply(const C_MatrixXd *m1, const C_MatrixXd *m2, C_MatrixXd *result) { - c_to_eigen(result) = c_to_eigen(m1) * c_to_eigen(m2); -} - -void MatrixXd_add(const C_MatrixXd *m1, const C_MatrixXd *m2, C_MatrixXd *result) { - c_to_eigen(result) = c_to_eigen(m1) + c_to_eigen(m2); -} - -////// class Map_MatrixXd ////// - -C_Map_MatrixXd *Map_MatrixXd_new(double *array, int rows, int cols) { - return eigen_to_c(*new Map<MatrixXd>(array, rows, cols)); -} - -void Map_MatrixXd_delete(C_Map_MatrixXd *m) { delete &c_to_eigen(m); } - -void Map_MatrixXd_set_zero(C_Map_MatrixXd *m) { c_to_eigen(m).setZero(); } - -void Map_MatrixXd_copy(C_Map_MatrixXd *dst, const C_Map_MatrixXd *src) { c_to_eigen(dst) = c_to_eigen(src); } - -void Map_MatrixXd_copy_matrix(C_Map_MatrixXd *dst, const C_MatrixXd *src) { c_to_eigen(dst) = c_to_eigen(src); } - -void Map_MatrixXd_set_coeff(C_Map_MatrixXd *m, int i, int j, double coeff) { c_to_eigen(m)(i, j) = coeff; } - -double Map_MatrixXd_get_coeff(const C_Map_MatrixXd *m, int i, int j) { return c_to_eigen(m)(i, j); } - -void Map_MatrixXd_print(const C_Map_MatrixXd *m) { std::cout << c_to_eigen(m) << std::endl; } - -void Map_MatrixXd_multiply(const C_Map_MatrixXd *m1, const C_Map_MatrixXd *m2, C_Map_MatrixXd *result) { - c_to_eigen(result) = c_to_eigen(m1) * c_to_eigen(m2); -} - -void Map_MatrixXd_add(const C_Map_MatrixXd *m1, const C_Map_MatrixXd *m2, C_Map_MatrixXd *result) { - c_to_eigen(result) = c_to_eigen(m1) + c_to_eigen(m2); -}
diff --git a/demos/mix_eigen_and_c/binary_library.h b/demos/mix_eigen_and_c/binary_library.h deleted file mode 100644 index 8cb1a8f..0000000 --- a/demos/mix_eigen_and_c/binary_library.h +++ /dev/null
@@ -1,56 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2009 Benoit Jacob <jacob.benoit.1@gmail.com> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -// This is a pure C header, no C++ here. -// The functions declared here will be implemented in C++ but -// we don't have to know, because thanks to the extern "C" syntax, -// they will be compiled to C object code. - -#ifdef __cplusplus -extern "C" { -#endif - -// just dummy empty structs to give different pointer types, -// instead of using void* which would be type unsafe -struct C_MatrixXd {}; -struct C_Map_MatrixXd {}; - -// the C_MatrixXd class, wraps some of the functionality -// of Eigen::MatrixXd. -struct C_MatrixXd *MatrixXd_new(int rows, int cols); -void MatrixXd_delete(struct C_MatrixXd *m); -double *MatrixXd_data(struct C_MatrixXd *m); -void MatrixXd_set_zero(struct C_MatrixXd *m); -void MatrixXd_resize(struct C_MatrixXd *m, int rows, int cols); -void MatrixXd_copy(struct C_MatrixXd *dst, const struct C_MatrixXd *src); -void MatrixXd_copy_map(struct C_MatrixXd *dst, const struct C_Map_MatrixXd *src); -void MatrixXd_set_coeff(struct C_MatrixXd *m, int i, int j, double coeff); -double MatrixXd_get_coeff(const struct C_MatrixXd *m, int i, int j); -void MatrixXd_print(const struct C_MatrixXd *m); -void MatrixXd_add(const struct C_MatrixXd *m1, const struct C_MatrixXd *m2, struct C_MatrixXd *result); -void MatrixXd_multiply(const struct C_MatrixXd *m1, const struct C_MatrixXd *m2, struct C_MatrixXd *result); - -// the C_Map_MatrixXd class, wraps some of the functionality -// of Eigen::Map<MatrixXd> -struct C_Map_MatrixXd *Map_MatrixXd_new(double *array, int rows, int cols); -void Map_MatrixXd_delete(struct C_Map_MatrixXd *m); -void Map_MatrixXd_set_zero(struct C_Map_MatrixXd *m); -void Map_MatrixXd_copy(struct C_Map_MatrixXd *dst, const struct C_Map_MatrixXd *src); -void Map_MatrixXd_copy_matrix(struct C_Map_MatrixXd *dst, const struct C_MatrixXd *src); -void Map_MatrixXd_set_coeff(struct C_Map_MatrixXd *m, int i, int j, double coeff); -double Map_MatrixXd_get_coeff(const struct C_Map_MatrixXd *m, int i, int j); -void Map_MatrixXd_print(const struct C_Map_MatrixXd *m); -void Map_MatrixXd_add(const struct C_Map_MatrixXd *m1, const struct C_Map_MatrixXd *m2, struct C_Map_MatrixXd *result); -void Map_MatrixXd_multiply(const struct C_Map_MatrixXd *m1, const struct C_Map_MatrixXd *m2, - struct C_Map_MatrixXd *result); - -#ifdef __cplusplus -} // end extern "C" -#endif
diff --git a/demos/mix_eigen_and_c/example.c b/demos/mix_eigen_and_c/example.c deleted file mode 100644 index a8e2e60..0000000 --- a/demos/mix_eigen_and_c/example.c +++ /dev/null
@@ -1,62 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2009 Benoit Jacob <jacob.benoit.1@gmail.com> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#include "binary_library.h" -#include "stdio.h" - -void demo_MatrixXd() { - struct C_MatrixXd *matrix1, *matrix2, *result; - printf("*** demo_MatrixXd ***\n"); - - matrix1 = MatrixXd_new(3, 3); - MatrixXd_set_zero(matrix1); - MatrixXd_set_coeff(matrix1, 0, 1, 2.5); - MatrixXd_set_coeff(matrix1, 1, 0, 1.4); - printf("Here is matrix1:\n"); - MatrixXd_print(matrix1); - - matrix2 = MatrixXd_new(3, 3); - MatrixXd_multiply(matrix1, matrix1, matrix2); - printf("Here is matrix1*matrix1:\n"); - MatrixXd_print(matrix2); - - MatrixXd_delete(matrix1); - MatrixXd_delete(matrix2); -} - -// this helper function takes a plain C array and prints it in one line -void print_array(double *array, int n) { - struct C_Map_MatrixXd *m = Map_MatrixXd_new(array, 1, n); - Map_MatrixXd_print(m); - Map_MatrixXd_delete(m); -} - -void demo_Map_MatrixXd() { - struct C_Map_MatrixXd *map; - double array[5]; - int i; - printf("*** demo_Map_MatrixXd ***\n"); - - for (i = 0; i < 5; ++i) array[i] = i; - printf("Initially, the array is:\n"); - print_array(array, 5); - - map = Map_MatrixXd_new(array, 5, 1); - Map_MatrixXd_add(map, map, map); - Map_MatrixXd_delete(map); - - printf("Now the array is:\n"); - print_array(array, 5); -} - -int main() { - demo_MatrixXd(); - demo_Map_MatrixXd(); -}
diff --git a/demos/opengl/CMakeLists.txt b/demos/opengl/CMakeLists.txt deleted file mode 100644 index 38bc5ad..0000000 --- a/demos/opengl/CMakeLists.txt +++ /dev/null
@@ -1,31 +0,0 @@ -# SPDX-FileCopyrightText: The Eigen Authors -# SPDX-License-Identifier: MPL-2.0 - -find_package(Qt4) -find_package(OpenGL) - -if(QT4_FOUND AND OPENGL_FOUND) - - set(QT_USE_QTOPENGL TRUE) - include(${QT_USE_FILE}) - - set(CMAKE_INCLUDE_CURRENT_DIR ON) - - include_directories( ${QT_INCLUDE_DIR} ) - - set(quaternion_demo_SRCS gpuhelper.cpp icosphere.cpp camera.cpp trackball.cpp quaternion_demo.cpp) - - qt4_automoc(${quaternion_demo_SRCS}) - - add_executable(quaternion_demo ${quaternion_demo_SRCS}) - add_dependencies(demos quaternion_demo) - - target_link_libraries(quaternion_demo - ${QT_QTCORE_LIBRARY} ${QT_QTGUI_LIBRARY} - ${QT_QTOPENGL_LIBRARY} ${OPENGL_LIBRARIES} ) - -else() - - message(STATUS "OpenGL demo disabled because Qt4 and/or OpenGL have not been found.") - -endif() \ No newline at end of file
diff --git a/demos/opengl/README b/demos/opengl/README deleted file mode 100644 index 8fb1649..0000000 --- a/demos/opengl/README +++ /dev/null
@@ -1,13 +0,0 @@ - -Navigation: - left button: rotate around the target - middle button: zoom - left button + ctrl quake rotate (rotate around camera position) - middle button + ctrl walk (progress along camera's z direction) - left button: pan (translate in the XY camera's plane) - -R : move the camera to initial position -A : start/stop animation -C : clear the animation -G : add a key frame -
diff --git a/demos/opengl/camera.cpp b/demos/opengl/camera.cpp deleted file mode 100644 index 20f2242..0000000 --- a/demos/opengl/camera.cpp +++ /dev/null
@@ -1,220 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#include "camera.h" - -#include "gpuhelper.h" -#include <GL/glu.h> - -#include "Eigen/LU" -using namespace Eigen; - -Camera::Camera() : mViewIsUptodate(false), mProjIsUptodate(false) { - mViewMatrix.setIdentity(); - - mFovY = EIGEN_PI / 3.; - mNearDist = 1.; - mFarDist = 50000.; - - mVpX = 0; - mVpY = 0; - - setPosition(Vector3f::Constant(100.)); - setTarget(Vector3f::Zero()); -} - -Camera& Camera::operator=(const Camera& other) { - mViewIsUptodate = false; - mProjIsUptodate = false; - - mVpX = other.mVpX; - mVpY = other.mVpY; - mVpWidth = other.mVpWidth; - mVpHeight = other.mVpHeight; - - mTarget = other.mTarget; - mFovY = other.mFovY; - mNearDist = other.mNearDist; - mFarDist = other.mFarDist; - - mViewMatrix = other.mViewMatrix; - mProjectionMatrix = other.mProjectionMatrix; - - return *this; -} - -Camera::Camera(const Camera& other) { *this = other; } - -Camera::~Camera() {} - -void Camera::setViewport(uint offsetx, uint offsety, uint width, uint height) { - mVpX = offsetx; - mVpY = offsety; - mVpWidth = width; - mVpHeight = height; - - mProjIsUptodate = false; -} - -void Camera::setViewport(uint width, uint height) { - mVpWidth = width; - mVpHeight = height; - - mProjIsUptodate = false; -} - -void Camera::setFovY(float value) { - mFovY = value; - mProjIsUptodate = false; -} - -Vector3f Camera::direction(void) const { return -(orientation() * Vector3f::UnitZ()); } -Vector3f Camera::up(void) const { return orientation() * Vector3f::UnitY(); } -Vector3f Camera::right(void) const { return orientation() * Vector3f::UnitX(); } - -void Camera::setDirection(const Vector3f& newDirection) { - // TODO implement it computing the rotation between newDirection and current dir ? - Vector3f up = this->up(); - - Matrix3f camAxes; - - camAxes.col(2) = (-newDirection).normalized(); - camAxes.col(0) = up.cross(camAxes.col(2)).normalized(); - camAxes.col(1) = camAxes.col(2).cross(camAxes.col(0)).normalized(); - setOrientation(Quaternionf(camAxes)); - - mViewIsUptodate = false; -} - -void Camera::setTarget(const Vector3f& target) { - mTarget = target; - if (!mTarget.isApprox(position())) { - Vector3f newDirection = mTarget - position(); - setDirection(newDirection.normalized()); - } -} - -void Camera::setPosition(const Vector3f& p) { - mFrame.position = p; - mViewIsUptodate = false; -} - -void Camera::setOrientation(const Quaternionf& q) { - mFrame.orientation = q; - mViewIsUptodate = false; -} - -void Camera::setFrame(const Frame& f) { - mFrame = f; - mViewIsUptodate = false; -} - -void Camera::rotateAroundTarget(const Quaternionf& q) { - Matrix4f mrot, mt, mtm; - - // update the transform matrix - updateViewMatrix(); - Vector3f t = mViewMatrix * mTarget; - - mViewMatrix = Translation3f(t) * q * Translation3f(-t) * mViewMatrix; - - Quaternionf qa(mViewMatrix.linear()); - qa = qa.conjugate(); - setOrientation(qa); - setPosition(-(qa * mViewMatrix.translation())); - - mViewIsUptodate = true; -} - -void Camera::localRotate(const Quaternionf& q) { - float dist = (position() - mTarget).norm(); - setOrientation(orientation() * q); - mTarget = position() + dist * direction(); - mViewIsUptodate = false; -} - -void Camera::zoom(float d) { - float dist = (position() - mTarget).norm(); - if (dist > d) { - setPosition(position() + direction() * d); - mViewIsUptodate = false; - } -} - -void Camera::localTranslate(const Vector3f& t) { - Vector3f trans = orientation() * t; - setPosition(position() + trans); - setTarget(mTarget + trans); - - mViewIsUptodate = false; -} - -void Camera::updateViewMatrix(void) const { - if (!mViewIsUptodate) { - Quaternionf q = orientation().conjugate(); - mViewMatrix.linear() = q.toRotationMatrix(); - mViewMatrix.translation() = -(mViewMatrix.linear() * position()); - - mViewIsUptodate = true; - } -} - -const Affine3f& Camera::viewMatrix(void) const { - updateViewMatrix(); - return mViewMatrix; -} - -void Camera::updateProjectionMatrix(void) const { - if (!mProjIsUptodate) { - mProjectionMatrix.setIdentity(); - float aspect = float(mVpWidth) / float(mVpHeight); - float theta = mFovY * 0.5; - float range = mFarDist - mNearDist; - float invtan = 1. / tan(theta); - - mProjectionMatrix(0, 0) = invtan / aspect; - mProjectionMatrix(1, 1) = invtan; - mProjectionMatrix(2, 2) = -(mNearDist + mFarDist) / range; - mProjectionMatrix(3, 2) = -1; - mProjectionMatrix(2, 3) = -2 * mNearDist * mFarDist / range; - mProjectionMatrix(3, 3) = 0; - - mProjIsUptodate = true; - } -} - -const Matrix4f& Camera::projectionMatrix(void) const { - updateProjectionMatrix(); - return mProjectionMatrix; -} - -void Camera::activateGL(void) { - glViewport(vpX(), vpY(), vpWidth(), vpHeight()); - gpu.loadMatrix(projectionMatrix(), GL_PROJECTION); - gpu.loadMatrix(viewMatrix().matrix(), GL_MODELVIEW); -} - -Vector3f Camera::unProject(const Vector2f& uv, float depth) const { - Matrix4f inv = mViewMatrix.inverse().matrix(); - return unProject(uv, depth, inv); -} - -Vector3f Camera::unProject(const Vector2f& uv, float depth, const Matrix4f& invModelview) const { - updateViewMatrix(); - updateProjectionMatrix(); - - Vector3f a(2. * uv.x() / float(mVpWidth) - 1., 2. * uv.y() / float(mVpHeight) - 1., 1.); - a.x() *= depth / mProjectionMatrix(0, 0); - a.y() *= depth / mProjectionMatrix(1, 1); - a.z() = -depth; - // FIXME: verify this transformation. - Vector4f b = invModelview * Vector4f(a.x(), a.y(), a.z(), 1.); - return Vector3f(b.x(), b.y(), b.z()); -}
diff --git a/demos/opengl/camera.h b/demos/opengl/camera.h deleted file mode 100644 index a8f99ca..0000000 --- a/demos/opengl/camera.h +++ /dev/null
@@ -1,112 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#ifndef EIGEN_CAMERA_H -#define EIGEN_CAMERA_H - -#include <Eigen/Geometry> -#include <QObject> -// #include <frame.h> - -class Frame { - public: - EIGEN_MAKE_ALIGNED_OPERATOR_NEW - - inline Frame(const Eigen::Vector3f& pos = Eigen::Vector3f::Zero(), const Eigen::Quaternionf& o = Eigen::Quaternionf()) - : orientation(o), position(pos) {} - Frame lerp(float alpha, const Frame& other) const { - return Frame((1.f - alpha) * position + alpha * other.position, orientation.slerp(alpha, other.orientation)); - } - - Eigen::Quaternionf orientation; - Eigen::Vector3f position; -}; - -class Camera { - public: - EIGEN_MAKE_ALIGNED_OPERATOR_NEW - - Camera(void); - - Camera(const Camera& other); - - virtual ~Camera(); - - Camera& operator=(const Camera& other); - - void setViewport(uint offsetx, uint offsety, uint width, uint height); - void setViewport(uint width, uint height); - - inline uint vpX(void) const { return mVpX; } - inline uint vpY(void) const { return mVpY; } - inline uint vpWidth(void) const { return mVpWidth; } - inline uint vpHeight(void) const { return mVpHeight; } - - inline float fovY(void) const { return mFovY; } - void setFovY(float value); - - void setPosition(const Eigen::Vector3f& pos); - inline const Eigen::Vector3f& position(void) const { return mFrame.position; } - - void setOrientation(const Eigen::Quaternionf& q); - inline const Eigen::Quaternionf& orientation(void) const { return mFrame.orientation; } - - void setFrame(const Frame& f); - const Frame& frame(void) const { return mFrame; } - - void setDirection(const Eigen::Vector3f& newDirection); - Eigen::Vector3f direction(void) const; - void setUp(const Eigen::Vector3f& vectorUp); - Eigen::Vector3f up(void) const; - Eigen::Vector3f right(void) const; - - void setTarget(const Eigen::Vector3f& target); - inline const Eigen::Vector3f& target(void) { return mTarget; } - - const Eigen::Affine3f& viewMatrix(void) const; - const Eigen::Matrix4f& projectionMatrix(void) const; - - void rotateAroundTarget(const Eigen::Quaternionf& q); - void localRotate(const Eigen::Quaternionf& q); - void zoom(float d); - - void localTranslate(const Eigen::Vector3f& t); - - /** Setup OpenGL matrices and viewport */ - void activateGL(void); - - Eigen::Vector3f unProject(const Eigen::Vector2f& uv, float depth, const Eigen::Matrix4f& invModelview) const; - Eigen::Vector3f unProject(const Eigen::Vector2f& uv, float depth) const; - - protected: - void updateViewMatrix(void) const; - void updateProjectionMatrix(void) const; - - protected: - uint mVpX, mVpY; - uint mVpWidth, mVpHeight; - - Frame mFrame; - - mutable Eigen::Affine3f mViewMatrix; - mutable Eigen::Matrix4f mProjectionMatrix; - - mutable bool mViewIsUptodate; - mutable bool mProjIsUptodate; - - // used by rotateAroundTarget - Eigen::Vector3f mTarget; - - float mFovY; - float mNearDist; - float mFarDist; -}; - -#endif // EIGEN_CAMERA_H
diff --git a/demos/opengl/gpuhelper.cpp b/demos/opengl/gpuhelper.cpp deleted file mode 100644 index 1e4086d..0000000 --- a/demos/opengl/gpuhelper.cpp +++ /dev/null
@@ -1,128 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#include "gpuhelper.h" -#include "icosphere.h" -#include <GL/glu.h> -// PLEASE don't look at this old code... ;) - -#include <fstream> -#include <algorithm> - -GpuHelper gpu; - -GpuHelper::GpuHelper() { - mVpWidth = mVpHeight = 0; - mCurrentMatrixTarget = 0; - mInitialized = false; -} - -GpuHelper::~GpuHelper() {} - -void GpuHelper::pushProjectionMode2D(ProjectionMode2D pm) { - // switch to 2D projection - pushMatrix(Matrix4f::Identity(), GL_PROJECTION); - - if (pm == PM_Normalized) { - // glOrtho(-1., 1., -1., 1., 0., 1.); - } else if (pm == PM_Viewport) { - GLint vp[4]; - glGetIntegerv(GL_VIEWPORT, vp); - glOrtho(0., vp[2], 0., vp[3], -1., 1.); - } - - pushMatrix(Matrix4f::Identity(), GL_MODELVIEW); -} - -void GpuHelper::popProjectionMode2D(void) { - popMatrix(GL_PROJECTION); - popMatrix(GL_MODELVIEW); -} - -void GpuHelper::drawVector(const Vector3f& position, const Vector3f& vec, const Color& color, float aspect /* = 50.*/) { - static GLUquadricObj* cylinder = gluNewQuadric(); - glColor4fv(color.data()); - float length = vec.norm(); - pushMatrix(GL_MODELVIEW); - glTranslatef(position.x(), position.y(), position.z()); - Vector3f ax = Matrix3f::Identity().col(2).cross(vec); - ax.normalize(); - Vector3f tmp = vec; - tmp.normalize(); - float angle = 180.f / EIGEN_PI * acos(tmp.z()); - if (angle > 1e-3) glRotatef(angle, ax.x(), ax.y(), ax.z()); - gluCylinder(cylinder, length / aspect, length / aspect, 0.8 * length, 10, 10); - glTranslatef(0.0, 0.0, 0.8 * length); - gluCylinder(cylinder, 2.0 * length / aspect, 0.0, 0.2 * length, 10, 10); - - popMatrix(GL_MODELVIEW); -} - -void GpuHelper::drawVectorBox(const Vector3f& position, const Vector3f& vec, const Color& color, float aspect) { - static GLUquadricObj* cylinder = gluNewQuadric(); - glColor4fv(color.data()); - float length = vec.norm(); - pushMatrix(GL_MODELVIEW); - glTranslatef(position.x(), position.y(), position.z()); - Vector3f ax = Matrix3f::Identity().col(2).cross(vec); - ax.normalize(); - Vector3f tmp = vec; - tmp.normalize(); - float angle = 180.f / EIGEN_PI * acos(tmp.z()); - if (angle > 1e-3) glRotatef(angle, ax.x(), ax.y(), ax.z()); - gluCylinder(cylinder, length / aspect, length / aspect, 0.8 * length, 10, 10); - glTranslatef(0.0, 0.0, 0.8 * length); - glScalef(4.0 * length / aspect, 4.0 * length / aspect, 4.0 * length / aspect); - drawUnitCube(); - popMatrix(GL_MODELVIEW); -} - -void GpuHelper::drawUnitCube(void) { - static float vertices[][3] = {{-0.5, -0.5, -0.5}, {0.5, -0.5, -0.5}, {-0.5, 0.5, -0.5}, {0.5, 0.5, -0.5}, - {-0.5, -0.5, 0.5}, {0.5, -0.5, 0.5}, {-0.5, 0.5, 0.5}, {0.5, 0.5, 0.5}}; - - glBegin(GL_QUADS); - glNormal3f(0, 0, -1); - glVertex3fv(vertices[0]); - glVertex3fv(vertices[2]); - glVertex3fv(vertices[3]); - glVertex3fv(vertices[1]); - glNormal3f(0, 0, 1); - glVertex3fv(vertices[4]); - glVertex3fv(vertices[5]); - glVertex3fv(vertices[7]); - glVertex3fv(vertices[6]); - glNormal3f(0, -1, 0); - glVertex3fv(vertices[0]); - glVertex3fv(vertices[1]); - glVertex3fv(vertices[5]); - glVertex3fv(vertices[4]); - glNormal3f(0, 1, 0); - glVertex3fv(vertices[2]); - glVertex3fv(vertices[6]); - glVertex3fv(vertices[7]); - glVertex3fv(vertices[3]); - glNormal3f(-1, 0, 0); - glVertex3fv(vertices[0]); - glVertex3fv(vertices[4]); - glVertex3fv(vertices[6]); - glVertex3fv(vertices[2]); - glNormal3f(1, 0, 0); - glVertex3fv(vertices[1]); - glVertex3fv(vertices[3]); - glVertex3fv(vertices[7]); - glVertex3fv(vertices[5]); - glEnd(); -} - -void GpuHelper::drawUnitSphere(int level) { - static IcoSphere sphere; - sphere.draw(level); -}
diff --git a/demos/opengl/gpuhelper.h b/demos/opengl/gpuhelper.h deleted file mode 100644 index e31ffb6..0000000 --- a/demos/opengl/gpuhelper.h +++ /dev/null
@@ -1,188 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#ifndef EIGEN_GPUHELPER_H -#define EIGEN_GPUHELPER_H - -#include <Eigen/Geometry> -#include <GL/gl.h> -#include <vector> - -using namespace Eigen; - -typedef Vector4f Color; - -class GpuHelper { - public: - GpuHelper(); - - ~GpuHelper(); - - enum ProjectionMode2D { PM_Normalized = 1, PM_Viewport = 2 }; - void pushProjectionMode2D(ProjectionMode2D pm); - void popProjectionMode2D(); - - /** Multiply the OpenGL matrix \a matrixTarget by the matrix \a mat. - Essentially, this helper function automatically calls glMatrixMode(matrixTarget) if required - and does a proper call to the right glMultMatrix*() function according to the scalar type - and storage order. - \warning glMatrixMode() must never be called directly. If you are unsure, use forceMatrixMode(). - \sa Matrix, loadMatrix(), forceMatrixMode() - */ - template <typename Scalar, int Flags_> - void multMatrix(const Matrix<Scalar, 4, 4, Flags_, 4, 4>& mat, GLenum matrixTarget); - - /** Load the matrix \a mat to the OpenGL matrix \a matrixTarget. - Essentially, this helper function automatically calls glMatrixMode(matrixTarget) if required - and does a proper call to the right glLoadMatrix*() or glLoadIdentity() function according to the scalar type - and storage order. - \warning glMatrixMode() must never be called directly. If you are unsure, use forceMatrixMode(). - \sa Matrix, multMatrix(), forceMatrixMode() - */ - template <typename Scalar, int Flags_> - void loadMatrix(const Eigen::Matrix<Scalar, 4, 4, Flags_, 4, 4>& mat, GLenum matrixTarget); - - template <typename Scalar, typename Derived> - void loadMatrix(const Eigen::CwiseNullaryOp<Eigen::internal::scalar_identity_op<Scalar>, Derived>&, - GLenum matrixTarget); - - /** Make the matrix \a matrixTarget the current OpenGL matrix target. - Call this function before loadMatrix() or multMatrix() if you cannot guarantee that glMatrixMode() - has never been called after the last loadMatrix() or multMatrix() calls. - \todo provides a debug mode checking the sanity of the cached matrix mode. - */ - inline void forceMatrixTarget(GLenum matrixTarget) { glMatrixMode(mCurrentMatrixTarget = matrixTarget); } - - inline void setMatrixTarget(GLenum matrixTarget); - - /** Push the OpenGL matrix \a matrixTarget and load \a mat. - */ - template <typename Scalar, int Flags_> - inline void pushMatrix(const Matrix<Scalar, 4, 4, Flags_, 4, 4>& mat, GLenum matrixTarget); - - template <typename Scalar, typename Derived> - void pushMatrix(const Eigen::CwiseNullaryOp<Eigen::internal::scalar_identity_op<Scalar>, Derived>&, - GLenum matrixTarget); - - /** Push and clone the OpenGL matrix \a matrixTarget - */ - inline void pushMatrix(GLenum matrixTarget); - - /** Pop the OpenGL matrix \a matrixTarget - */ - inline void popMatrix(GLenum matrixTarget); - - void drawVector(const Vector3f& position, const Vector3f& vec, const Color& color, float aspect = 50.); - void drawVectorBox(const Vector3f& position, const Vector3f& vec, const Color& color, float aspect = 50.); - void drawUnitCube(void); - void drawUnitSphere(int level = 0); - - /// draw the \a nofElement first elements - inline void draw(GLenum mode, uint nofElement); - - /// draw a range of elements - inline void draw(GLenum mode, uint start, uint end); - - /// draw an indexed subset - inline void draw(GLenum mode, const std::vector<uint>* pIndexes); - - protected: - void update(void); - - GLuint mColorBufferId; - int mVpWidth, mVpHeight; - GLenum mCurrentMatrixTarget; - bool mInitialized; -}; - -/** Singleton shortcut - */ -extern GpuHelper gpu; - -/** \internal - */ -template <bool RowMajor, int Flags_> -struct GlMatrixHelper; - -template <int Flags_> -struct GlMatrixHelper<false, Flags_> { - static void loadMatrix(const Matrix<float, 4, 4, Flags_, 4, 4>& mat) { glLoadMatrixf(mat.data()); } - static void loadMatrix(const Matrix<double, 4, 4, Flags_, 4, 4>& mat) { glLoadMatrixd(mat.data()); } - static void multMatrix(const Matrix<float, 4, 4, Flags_, 4, 4>& mat) { glMultMatrixf(mat.data()); } - static void multMatrix(const Matrix<double, 4, 4, Flags_, 4, 4>& mat) { glMultMatrixd(mat.data()); } -}; - -template <int Flags_> -struct GlMatrixHelper<true, Flags_> { - static void loadMatrix(const Matrix<float, 4, 4, Flags_, 4, 4>& mat) { glLoadMatrixf(mat.transpose().eval().data()); } - static void loadMatrix(const Matrix<double, 4, 4, Flags_, 4, 4>& mat) { - glLoadMatrixd(mat.transpose().eval().data()); - } - static void multMatrix(const Matrix<float, 4, 4, Flags_, 4, 4>& mat) { glMultMatrixf(mat.transpose().eval().data()); } - static void multMatrix(const Matrix<double, 4, 4, Flags_, 4, 4>& mat) { - glMultMatrixd(mat.transpose().eval().data()); - } -}; - -inline void GpuHelper::setMatrixTarget(GLenum matrixTarget) { - if (matrixTarget != mCurrentMatrixTarget) glMatrixMode(mCurrentMatrixTarget = matrixTarget); -} - -template <typename Scalar, int Flags_> -void GpuHelper::multMatrix(const Matrix<Scalar, 4, 4, Flags_, 4, 4>& mat, GLenum matrixTarget) { - setMatrixTarget(matrixTarget); - GlMatrixHelper<Flags_ & Eigen::RowMajorBit, Flags_>::multMatrix(mat); -} - -template <typename Scalar, typename Derived> -void GpuHelper::loadMatrix(const Eigen::CwiseNullaryOp<Eigen::internal::scalar_identity_op<Scalar>, Derived>&, - GLenum matrixTarget) { - setMatrixTarget(matrixTarget); - glLoadIdentity(); -} - -template <typename Scalar, int Flags_> -void GpuHelper::loadMatrix(const Eigen::Matrix<Scalar, 4, 4, Flags_, 4, 4>& mat, GLenum matrixTarget) { - setMatrixTarget(matrixTarget); - GlMatrixHelper<(Flags_ & Eigen::RowMajorBit) != 0, Flags_>::loadMatrix(mat); -} - -inline void GpuHelper::pushMatrix(GLenum matrixTarget) { - setMatrixTarget(matrixTarget); - glPushMatrix(); -} - -template <typename Scalar, int Flags_> -inline void GpuHelper::pushMatrix(const Matrix<Scalar, 4, 4, Flags_, 4, 4>& mat, GLenum matrixTarget) { - pushMatrix(matrixTarget); - GlMatrixHelper<Flags_ & Eigen::RowMajorBit, Flags_>::loadMatrix(mat); -} - -template <typename Scalar, typename Derived> -void GpuHelper::pushMatrix(const Eigen::CwiseNullaryOp<Eigen::internal::scalar_identity_op<Scalar>, Derived>&, - GLenum matrixTarget) { - pushMatrix(matrixTarget); - glLoadIdentity(); -} - -inline void GpuHelper::popMatrix(GLenum matrixTarget) { - setMatrixTarget(matrixTarget); - glPopMatrix(); -} - -inline void GpuHelper::draw(GLenum mode, uint nofElement) { glDrawArrays(mode, 0, nofElement); } - -inline void GpuHelper::draw(GLenum mode, const std::vector<uint>* pIndexes) { - glDrawElements(mode, pIndexes->size(), GL_UNSIGNED_INT, &(pIndexes->front())); -} - -inline void GpuHelper::draw(GLenum mode, uint start, uint end) { glDrawArrays(mode, start, end - start); } - -#endif // EIGEN_GPUHELPER_H
diff --git a/demos/opengl/icosphere.cpp b/demos/opengl/icosphere.cpp deleted file mode 100644 index e5abdea..0000000 --- a/demos/opengl/icosphere.cpp +++ /dev/null
@@ -1,106 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#include "icosphere.h" - -#include <GL/gl.h> -#include <map> - -using namespace Eigen; - -//-------------------------------------------------------------------------------- -// icosahedron data -//-------------------------------------------------------------------------------- -#define X .525731112119133606 -#define Z .850650808352039932 - -static GLfloat vdata[12][3] = {{-X, 0.0, Z}, {X, 0.0, Z}, {-X, 0.0, -Z}, {X, 0.0, -Z}, {0.0, Z, X}, {0.0, Z, -X}, - {0.0, -Z, X}, {0.0, -Z, -X}, {Z, X, 0.0}, {-Z, X, 0.0}, {Z, -X, 0.0}, {-Z, -X, 0.0}}; - -static GLint tindices[20][3] = {{0, 4, 1}, {0, 9, 4}, {9, 5, 4}, {4, 5, 8}, {4, 8, 1}, {8, 10, 1}, {8, 3, 10}, - {5, 3, 8}, {5, 2, 3}, {2, 7, 3}, {7, 10, 3}, {7, 6, 10}, {7, 11, 6}, {11, 0, 6}, - {0, 1, 6}, {6, 1, 10}, {9, 0, 11}, {9, 11, 2}, {9, 2, 5}, {7, 2, 11}}; -//-------------------------------------------------------------------------------- - -IcoSphere::IcoSphere(unsigned int levels) { - // init with an icosahedron - for (int i = 0; i < 12; i++) mVertices.push_back(Map<Vector3f>(vdata[i])); - mIndices.push_back(new std::vector<int>); - std::vector<int>& indices = *mIndices.back(); - for (int i = 0; i < 20; i++) { - for (int k = 0; k < 3; k++) indices.push_back(tindices[i][k]); - } - mListIds.push_back(0); - - while (mIndices.size() < levels) _subdivide(); -} - -const std::vector<int>& IcoSphere::indices(int level) const { - while (level >= int(mIndices.size())) const_cast<IcoSphere*>(this)->_subdivide(); - return *mIndices[level]; -} - -void IcoSphere::_subdivide(void) { - typedef unsigned long long Key; - std::map<Key, int> edgeMap; - const std::vector<int>& indices = *mIndices.back(); - mIndices.push_back(new std::vector<int>); - std::vector<int>& refinedIndices = *mIndices.back(); - int end = indices.size(); - for (int i = 0; i < end; i += 3) { - int ids0[3], // indices of outer vertices - ids1[3]; // indices of edge vertices - for (int k = 0; k < 3; ++k) { - int k1 = (k + 1) % 3; - int e0 = indices[i + k]; - int e1 = indices[i + k1]; - ids0[k] = e0; - if (e1 > e0) std::swap(e0, e1); - Key edgeKey = Key(e0) | (Key(e1) << 32); - std::map<Key, int>::iterator it = edgeMap.find(edgeKey); - if (it == edgeMap.end()) { - ids1[k] = mVertices.size(); - edgeMap[edgeKey] = ids1[k]; - mVertices.push_back((mVertices[e0] + mVertices[e1]).normalized()); - } else - ids1[k] = it->second; - } - refinedIndices.push_back(ids0[0]); - refinedIndices.push_back(ids1[0]); - refinedIndices.push_back(ids1[2]); - refinedIndices.push_back(ids0[1]); - refinedIndices.push_back(ids1[1]); - refinedIndices.push_back(ids1[0]); - refinedIndices.push_back(ids0[2]); - refinedIndices.push_back(ids1[2]); - refinedIndices.push_back(ids1[1]); - refinedIndices.push_back(ids1[0]); - refinedIndices.push_back(ids1[1]); - refinedIndices.push_back(ids1[2]); - } - mListIds.push_back(0); -} - -void IcoSphere::draw(int level) { - while (level >= int(mIndices.size())) const_cast<IcoSphere*>(this)->_subdivide(); - if (mListIds[level] == 0) { - mListIds[level] = glGenLists(1); - glNewList(mListIds[level], GL_COMPILE); - glVertexPointer(3, GL_FLOAT, 0, mVertices[0].data()); - glNormalPointer(GL_FLOAT, 0, mVertices[0].data()); - glEnableClientState(GL_VERTEX_ARRAY); - glEnableClientState(GL_NORMAL_ARRAY); - glDrawElements(GL_TRIANGLES, mIndices[level]->size(), GL_UNSIGNED_INT, &(mIndices[level]->at(0))); - glDisableClientState(GL_VERTEX_ARRAY); - glDisableClientState(GL_NORMAL_ARRAY); - glEndList(); - } - glCallList(mListIds[level]); -}
diff --git a/demos/opengl/icosphere.h b/demos/opengl/icosphere.h deleted file mode 100644 index 909710f..0000000 --- a/demos/opengl/icosphere.h +++ /dev/null
@@ -1,31 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#ifndef EIGEN_ICOSPHERE_H -#define EIGEN_ICOSPHERE_H - -#include <Eigen/Core> -#include <vector> - -class IcoSphere { - public: - IcoSphere(unsigned int levels = 1); - const std::vector<Eigen::Vector3f>& vertices() const { return mVertices; } - const std::vector<int>& indices(int level) const; - void draw(int level); - - protected: - void _subdivide(); - std::vector<Eigen::Vector3f> mVertices; - std::vector<std::vector<int>*> mIndices; - std::vector<int> mListIds; -}; - -#endif // EIGEN_ICOSPHERE_H
diff --git a/demos/opengl/quaternion_demo.cpp b/demos/opengl/quaternion_demo.cpp deleted file mode 100644 index 74f8f59..0000000 --- a/demos/opengl/quaternion_demo.cpp +++ /dev/null
@@ -1,588 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#include "quaternion_demo.h" -#include "icosphere.h" - -#include <Eigen/Geometry> -#include <Eigen/QR> -#include <Eigen/LU> - -#include <iostream> -#include <QEvent> -#include <QMouseEvent> -#include <QInputDialog> -#include <QGridLayout> -#include <QButtonGroup> -#include <QRadioButton> -#include <QDockWidget> -#include <QPushButton> -#include <QGroupBox> - -using namespace Eigen; - -class FancySpheres { - public: - EIGEN_MAKE_ALIGNED_OPERATOR_NEW - - FancySpheres() { - const int levels = 4; - const float scale = 0.33; - float radius = 100; - std::vector<int> parents; - - // leval 0 - mCenters.push_back(Vector3f::Zero()); - parents.push_back(-1); - mRadii.push_back(radius); - - // generate level 1 using icosphere vertices - radius *= 0.45; - { - float dist = mRadii[0] * 0.9; - for (int i = 0; i < 12; ++i) { - mCenters.push_back(mIcoSphere.vertices()[i] * dist); - mRadii.push_back(radius); - parents.push_back(0); - } - } - - static const float angles[10] = { - 0, 0, EIGEN_PI, 0. * EIGEN_PI, EIGEN_PI, 0.5 * EIGEN_PI, EIGEN_PI, 1. * EIGEN_PI, EIGEN_PI, 1.5 * EIGEN_PI}; - - // generate other levels - int start = 1; - for (int l = 1; l < levels; l++) { - radius *= scale; - int end = mCenters.size(); - for (int i = start; i < end; ++i) { - Vector3f c = mCenters[i]; - Vector3f ax0 = (c - mCenters[parents[i]]).normalized(); - Vector3f ax1 = ax0.unitOrthogonal(); - Quaternionf q; - q.setFromTwoVectors(Vector3f::UnitZ(), ax0); - Affine3f t = Translation3f(c) * q * Scaling(mRadii[i] + radius); - for (int j = 0; j < 5; ++j) { - Vector3f newC = - c + ((AngleAxisf(angles[j * 2 + 1], ax0) * AngleAxisf(angles[j * 2 + 0] * (l == 1 ? 0.35 : 0.5), ax1)) * - ax0) * - (mRadii[i] + radius * 0.8); - mCenters.push_back(newC); - mRadii.push_back(radius); - parents.push_back(i); - } - } - start = end; - } - } - - void draw() { - int end = mCenters.size(); - glEnable(GL_NORMALIZE); - for (int i = 0; i < end; ++i) { - Affine3f t = Translation3f(mCenters[i]) * Scaling(mRadii[i]); - gpu.pushMatrix(GL_MODELVIEW); - gpu.multMatrix(t.matrix(), GL_MODELVIEW); - mIcoSphere.draw(2); - gpu.popMatrix(GL_MODELVIEW); - } - glDisable(GL_NORMALIZE); - } - - protected: - std::vector<Vector3f> mCenters; - std::vector<float> mRadii; - IcoSphere mIcoSphere; -}; - -// generic linear interpolation method -template <typename T> -T lerp(float t, const T& a, const T& b) { - return a * (1 - t) + b * t; -} - -// quaternion slerp -template <> -Quaternionf lerp(float t, const Quaternionf& a, const Quaternionf& b) { - return a.slerp(t, b); -} - -// linear interpolation of a frame using the type OrientationType -// to perform the interpolation of the orientations -template <typename OrientationType> -inline static Frame lerpFrame(float alpha, const Frame& a, const Frame& b) { - return Frame(lerp(alpha, a.position, b.position), - Quaternionf(lerp(alpha, OrientationType(a.orientation), OrientationType(b.orientation)))); -} - -template <typename Scalar_> -class EulerAngles { - public: - enum { Dim = 3 }; - typedef Scalar_ Scalar; - typedef Matrix<Scalar, 3, 3> Matrix3; - typedef Matrix<Scalar, 3, 1> Vector3; - typedef Quaternion<Scalar> QuaternionType; - - protected: - Vector3 m_angles; - - public: - EulerAngles() {} - inline EulerAngles(Scalar a0, Scalar a1, Scalar a2) : m_angles(a0, a1, a2) {} - inline EulerAngles(const QuaternionType& q) { *this = q; } - - const Vector3& coeffs() const { return m_angles; } - Vector3& coeffs() { return m_angles; } - - EulerAngles& operator=(const QuaternionType& q) { - Matrix3 m = q.toRotationMatrix(); - return *this = m; - } - - EulerAngles& operator=(const Matrix3& m) { - // mat = cy*cz -cy*sz sy - // cz*sx*sy+cx*sz cx*cz-sx*sy*sz -cy*sx - // -cx*cz*sy+sx*sz cz*sx+cx*sy*sz cx*cy - m_angles.coeffRef(1) = std::asin(m.coeff(0, 2)); - m_angles.coeffRef(0) = std::atan2(-m.coeff(1, 2), m.coeff(2, 2)); - m_angles.coeffRef(2) = std::atan2(-m.coeff(0, 1), m.coeff(0, 0)); - return *this; - } - - Matrix3 toRotationMatrix(void) const { - Vector3 c = m_angles.array().cos(); - Vector3 s = m_angles.array().sin(); - Matrix3 res; - res << c.y() * c.z(), -c.y() * s.z(), s.y(), c.z() * s.x() * s.y() + c.x() * s.z(), - c.x() * c.z() - s.x() * s.y() * s.z(), -c.y() * s.x(), -c.x() * c.z() * s.y() + s.x() * s.z(), - c.z() * s.x() + c.x() * s.y() * s.z(), c.x() * c.y(); - return res; - } - - operator QuaternionType() { return QuaternionType(toRotationMatrix()); } -}; - -// Euler angles slerp -template <> -EulerAngles<float> lerp(float t, const EulerAngles<float>& a, const EulerAngles<float>& b) { - EulerAngles<float> res; - res.coeffs() = lerp(t, a.coeffs(), b.coeffs()); - return res; -} - -RenderingWidget::RenderingWidget() { - mAnimate = false; - mCurrentTrackingMode = TM_NO_TRACK; - mNavMode = NavTurnAround; - mLerpMode = LerpQuaternion; - mRotationMode = RotationStable; - mTrackball.setCamera(&mCamera); - - // required to capture key press events - setFocusPolicy(Qt::ClickFocus); -} - -void RenderingWidget::grabFrame(void) { - // ask user for a time - bool ok = false; - double t = 0; - if (!m_timeline.empty()) t = (--m_timeline.end())->first + 1.; - t = QInputDialog::getDouble(this, "Eigen's RenderingWidget", "time value: ", t, 0, 1e3, 1, &ok); - if (ok) { - Frame aux; - aux.orientation = mCamera.viewMatrix().linear(); - aux.position = mCamera.viewMatrix().translation(); - m_timeline[t] = aux; - } -} - -void RenderingWidget::drawScene() { - static FancySpheres sFancySpheres; - float length = 50; - gpu.drawVector(Vector3f::Zero(), length * Vector3f::UnitX(), Color(1, 0, 0, 1)); - gpu.drawVector(Vector3f::Zero(), length * Vector3f::UnitY(), Color(0, 1, 0, 1)); - gpu.drawVector(Vector3f::Zero(), length * Vector3f::UnitZ(), Color(0, 0, 1, 1)); - - // draw the fractal object - float sqrt3 = std::sqrt(3.); - glLightfv(GL_LIGHT0, GL_AMBIENT, Vector4f(0.5, 0.5, 0.5, 1).data()); - glLightfv(GL_LIGHT0, GL_DIFFUSE, Vector4f(0.5, 1, 0.5, 1).data()); - glLightfv(GL_LIGHT0, GL_SPECULAR, Vector4f(1, 1, 1, 1).data()); - glLightfv(GL_LIGHT0, GL_POSITION, Vector4f(-sqrt3, -sqrt3, sqrt3, 0).data()); - - glLightfv(GL_LIGHT1, GL_AMBIENT, Vector4f(0, 0, 0, 1).data()); - glLightfv(GL_LIGHT1, GL_DIFFUSE, Vector4f(1, 0.5, 0.5, 1).data()); - glLightfv(GL_LIGHT1, GL_SPECULAR, Vector4f(1, 1, 1, 1).data()); - glLightfv(GL_LIGHT1, GL_POSITION, Vector4f(-sqrt3, sqrt3, -sqrt3, 0).data()); - - glMaterialfv(GL_FRONT_AND_BACK, GL_AMBIENT, Vector4f(0.7, 0.7, 0.7, 1).data()); - glMaterialfv(GL_FRONT_AND_BACK, GL_DIFFUSE, Vector4f(0.8, 0.75, 0.6, 1).data()); - glMaterialfv(GL_FRONT_AND_BACK, GL_SPECULAR, Vector4f(1, 1, 1, 1).data()); - glMaterialf(GL_FRONT_AND_BACK, GL_SHININESS, 64); - - glEnable(GL_LIGHTING); - glEnable(GL_LIGHT0); - glEnable(GL_LIGHT1); - - sFancySpheres.draw(); - glVertexPointer(3, GL_FLOAT, 0, mVertices[0].data()); - glNormalPointer(GL_FLOAT, 0, mNormals[0].data()); - glEnableClientState(GL_VERTEX_ARRAY); - glEnableClientState(GL_NORMAL_ARRAY); - glDrawArrays(GL_TRIANGLES, 0, mVertices.size()); - glDisableClientState(GL_VERTEX_ARRAY); - glDisableClientState(GL_NORMAL_ARRAY); - - glDisable(GL_LIGHTING); -} - -void RenderingWidget::animate() { - m_alpha += double(m_timer.interval()) * 1e-3; - - TimeLine::const_iterator hi = m_timeline.upper_bound(m_alpha); - TimeLine::const_iterator lo = hi; - --lo; - - Frame currentFrame; - - if (hi == m_timeline.end()) { - // end - currentFrame = lo->second; - stopAnimation(); - } else if (hi == m_timeline.begin()) { - // start - currentFrame = hi->second; - } else { - float s = (m_alpha - lo->first) / (hi->first - lo->first); - if (mLerpMode == LerpEulerAngles) - currentFrame = ::lerpFrame<EulerAngles<float> >(s, lo->second, hi->second); - else if (mLerpMode == LerpQuaternion) - currentFrame = ::lerpFrame<Eigen::Quaternionf>(s, lo->second, hi->second); - else { - std::cerr << "Invalid rotation interpolation mode (abort)\n"; - exit(2); - } - currentFrame.orientation.coeffs().normalize(); - } - - currentFrame.orientation = currentFrame.orientation.inverse(); - currentFrame.position = -(currentFrame.orientation * currentFrame.position); - mCamera.setFrame(currentFrame); - - updateGL(); -} - -void RenderingWidget::keyPressEvent(QKeyEvent* e) { - switch (e->key()) { - case Qt::Key_Up: - mCamera.zoom(2); - break; - case Qt::Key_Down: - mCamera.zoom(-2); - break; - // add a frame - case Qt::Key_G: - grabFrame(); - break; - // clear the time line - case Qt::Key_C: - m_timeline.clear(); - break; - // move the camera to initial pos - case Qt::Key_R: - resetCamera(); - break; - // start/stop the animation - case Qt::Key_A: - if (mAnimate) { - stopAnimation(); - } else { - m_alpha = 0; - connect(&m_timer, SIGNAL(timeout()), this, SLOT(animate())); - m_timer.start(1000 / 30); - mAnimate = true; - } - break; - default: - break; - } - - updateGL(); -} - -void RenderingWidget::stopAnimation() { - disconnect(&m_timer, SIGNAL(timeout()), this, SLOT(animate())); - m_timer.stop(); - mAnimate = false; - m_alpha = 0; -} - -void RenderingWidget::mousePressEvent(QMouseEvent* e) { - mMouseCoords = Vector2i(e->pos().x(), e->pos().y()); - bool fly = (mNavMode == NavFly) || (e->modifiers() & Qt::ControlModifier); - switch (e->button()) { - case Qt::LeftButton: - if (fly) { - mCurrentTrackingMode = TM_LOCAL_ROTATE; - mTrackball.start(Trackball::Local); - } else { - mCurrentTrackingMode = TM_ROTATE_AROUND; - mTrackball.start(Trackball::Around); - } - mTrackball.track(mMouseCoords); - break; - case Qt::MidButton: - if (fly) - mCurrentTrackingMode = TM_FLY_Z; - else - mCurrentTrackingMode = TM_ZOOM; - break; - case Qt::RightButton: - mCurrentTrackingMode = TM_FLY_PAN; - break; - default: - break; - } -} -void RenderingWidget::mouseReleaseEvent(QMouseEvent*) { - mCurrentTrackingMode = TM_NO_TRACK; - updateGL(); -} - -void RenderingWidget::mouseMoveEvent(QMouseEvent* e) { - // tracking - if (mCurrentTrackingMode != TM_NO_TRACK) { - float dx = float(e->x() - mMouseCoords.x()) / float(mCamera.vpWidth()); - float dy = -float(e->y() - mMouseCoords.y()) / float(mCamera.vpHeight()); - - // speedup the transformations - if (e->modifiers() & Qt::ShiftModifier) { - dx *= 10.; - dy *= 10.; - } - - switch (mCurrentTrackingMode) { - case TM_ROTATE_AROUND: - case TM_LOCAL_ROTATE: - if (mRotationMode == RotationStable) { - // use the stable trackball implementation mapping - // the 2D coordinates to 3D points on a sphere. - mTrackball.track(Vector2i(e->pos().x(), e->pos().y())); - } else { - // standard approach mapping the x and y displacements as rotations - // around the camera's X and Y axes. - Quaternionf q = AngleAxisf(dx * EIGEN_PI, Vector3f::UnitY()) * AngleAxisf(-dy * EIGEN_PI, Vector3f::UnitX()); - if (mCurrentTrackingMode == TM_LOCAL_ROTATE) - mCamera.localRotate(q); - else - mCamera.rotateAroundTarget(q); - } - break; - case TM_ZOOM: - mCamera.zoom(dy * 100); - break; - case TM_FLY_Z: - mCamera.localTranslate(Vector3f(0, 0, -dy * 200)); - break; - case TM_FLY_PAN: - mCamera.localTranslate(Vector3f(dx * 200, dy * 200, 0)); - break; - default: - break; - } - - updateGL(); - } - - mMouseCoords = Vector2i(e->pos().x(), e->pos().y()); -} - -void RenderingWidget::paintGL() { - glEnable(GL_DEPTH_TEST); - glDisable(GL_CULL_FACE); - glPolygonMode(GL_FRONT_AND_BACK, GL_FILL); - glDisable(GL_COLOR_MATERIAL); - glDisable(GL_BLEND); - glDisable(GL_ALPHA_TEST); - glDisable(GL_TEXTURE_1D); - glDisable(GL_TEXTURE_2D); - glDisable(GL_TEXTURE_3D); - - // Clear buffers - glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); - - mCamera.activateGL(); - - drawScene(); -} - -void RenderingWidget::initializeGL() { - glClearColor(1., 1., 1., 0.); - glLightModeli(GL_LIGHT_MODEL_LOCAL_VIEWER, 1); - glDepthMask(GL_TRUE); - glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); - - mCamera.setPosition(Vector3f(-200, -200, -200)); - mCamera.setTarget(Vector3f(0, 0, 0)); - mInitFrame.orientation = mCamera.orientation().inverse(); - mInitFrame.position = mCamera.viewMatrix().translation(); -} - -void RenderingWidget::resizeGL(int width, int height) { mCamera.setViewport(width, height); } - -void RenderingWidget::setNavMode(int m) { mNavMode = NavMode(m); } - -void RenderingWidget::setLerpMode(int m) { mLerpMode = LerpMode(m); } - -void RenderingWidget::setRotationMode(int m) { mRotationMode = RotationMode(m); } - -void RenderingWidget::resetCamera() { - if (mAnimate) stopAnimation(); - m_timeline.clear(); - Frame aux0 = mCamera.frame(); - aux0.orientation = aux0.orientation.inverse(); - aux0.position = mCamera.viewMatrix().translation(); - m_timeline[0] = aux0; - - Vector3f currentTarget = mCamera.target(); - mCamera.setTarget(Vector3f::Zero()); - - // compute the rotation duration to move the camera to the target - Frame aux1 = mCamera.frame(); - aux1.orientation = aux1.orientation.inverse(); - aux1.position = mCamera.viewMatrix().translation(); - float duration = aux0.orientation.angularDistance(aux1.orientation) * 0.9; - if (duration < 0.1) duration = 0.1; - - // put the camera at that time step: - aux1 = aux0.lerp(duration / 2, mInitFrame); - // and make it look at the target again - aux1.orientation = aux1.orientation.inverse(); - aux1.position = -(aux1.orientation * aux1.position); - mCamera.setFrame(aux1); - mCamera.setTarget(Vector3f::Zero()); - - // add this camera keyframe - aux1.orientation = aux1.orientation.inverse(); - aux1.position = mCamera.viewMatrix().translation(); - m_timeline[duration] = aux1; - - m_timeline[2] = mInitFrame; - m_alpha = 0; - animate(); - connect(&m_timer, SIGNAL(timeout()), this, SLOT(animate())); - m_timer.start(1000 / 30); - mAnimate = true; -} - -QWidget* RenderingWidget::createNavigationControlWidget() { - QWidget* panel = new QWidget(); - QVBoxLayout* layout = new QVBoxLayout(); - - { - QPushButton* but = new QPushButton("reset"); - but->setToolTip("move the camera to initial position (with animation)"); - layout->addWidget(but); - connect(but, SIGNAL(clicked()), this, SLOT(resetCamera())); - } - { - // navigation mode - QGroupBox* box = new QGroupBox("navigation mode"); - QVBoxLayout* boxLayout = new QVBoxLayout; - QButtonGroup* group = new QButtonGroup(panel); - QRadioButton* but; - but = new QRadioButton("turn around"); - but->setToolTip("look around an object"); - group->addButton(but, NavTurnAround); - boxLayout->addWidget(but); - but = new QRadioButton("fly"); - but->setToolTip("free navigation like a spaceship\n(this mode can also be enabled pressing the \"shift\" key)"); - group->addButton(but, NavFly); - boxLayout->addWidget(but); - group->button(mNavMode)->setChecked(true); - connect(group, SIGNAL(buttonClicked(int)), this, SLOT(setNavMode(int))); - box->setLayout(boxLayout); - layout->addWidget(box); - } - { - // track ball, rotation mode - QGroupBox* box = new QGroupBox("rotation mode"); - QVBoxLayout* boxLayout = new QVBoxLayout; - QButtonGroup* group = new QButtonGroup(panel); - QRadioButton* but; - but = new QRadioButton("stable trackball"); - group->addButton(but, RotationStable); - boxLayout->addWidget(but); - but->setToolTip("use the stable trackball implementation mapping\nthe 2D coordinates to 3D points on a sphere"); - but = new QRadioButton("standard rotation"); - group->addButton(but, RotationStandard); - boxLayout->addWidget(but); - but->setToolTip( - "standard approach mapping the x and y displacements\nas rotations around the camera's X and Y axes"); - group->button(mRotationMode)->setChecked(true); - connect(group, SIGNAL(buttonClicked(int)), this, SLOT(setRotationMode(int))); - box->setLayout(boxLayout); - layout->addWidget(box); - } - { - // interpolation mode - QGroupBox* box = new QGroupBox("spherical interpolation"); - QVBoxLayout* boxLayout = new QVBoxLayout; - QButtonGroup* group = new QButtonGroup(panel); - QRadioButton* but; - but = new QRadioButton("quaternion slerp"); - group->addButton(but, LerpQuaternion); - boxLayout->addWidget(but); - but->setToolTip("use quaternion spherical interpolation\nto interpolate orientations"); - but = new QRadioButton("euler angles"); - group->addButton(but, LerpEulerAngles); - boxLayout->addWidget(but); - but->setToolTip("use Euler angles to interpolate orientations"); - group->button(mNavMode)->setChecked(true); - connect(group, SIGNAL(buttonClicked(int)), this, SLOT(setLerpMode(int))); - box->setLayout(boxLayout); - layout->addWidget(box); - } - layout->addItem(new QSpacerItem(0, 0, QSizePolicy::Minimum, QSizePolicy::Expanding)); - panel->setLayout(layout); - return panel; -} - -QuaternionDemo::QuaternionDemo() { - mRenderingWidget = new RenderingWidget(); - setCentralWidget(mRenderingWidget); - - QDockWidget* panel = new QDockWidget("navigation", this); - panel->setAllowedAreas((QFlags<Qt::DockWidgetArea>)(Qt::RightDockWidgetArea | Qt::LeftDockWidgetArea)); - addDockWidget(Qt::RightDockWidgetArea, panel); - panel->setWidget(mRenderingWidget->createNavigationControlWidget()); -} - -int main(int argc, char* argv[]) { - std::cout << "Navigation:\n"; - std::cout << " left button: rotate around the target\n"; - std::cout << " middle button: zoom\n"; - std::cout << " left button + ctrl quake rotate (rotate around camera position)\n"; - std::cout << " middle button + ctrl walk (progress along camera's z direction)\n"; - std::cout << " left button: pan (translate in the XY camera's plane)\n\n"; - std::cout << "R : move the camera to initial position\n"; - std::cout << "A : start/stop animation\n"; - std::cout << "C : clear the animation\n"; - std::cout << "G : add a key frame\n"; - - QApplication app(argc, argv); - QuaternionDemo demo; - demo.resize(600, 500); - demo.show(); - return app.exec(); -} - -#include "quaternion_demo.moc"
diff --git a/demos/opengl/quaternion_demo.h b/demos/opengl/quaternion_demo.h deleted file mode 100644 index 4c8f351..0000000 --- a/demos/opengl/quaternion_demo.h +++ /dev/null
@@ -1,101 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#ifndef EIGEN_QUATERNION_DEMO_H -#define EIGEN_QUATERNION_DEMO_H - -#include "gpuhelper.h" -#include "camera.h" -#include "trackball.h" -#include <map> -#include <QTimer> -#include <QtGui/QApplication> -#include <QtOpenGL/QGLWidget> -#include <QtGui/QMainWindow> - -class RenderingWidget : public QGLWidget { - Q_OBJECT - - typedef std::map<float, Frame> TimeLine; - TimeLine m_timeline; - Frame lerpFrame(float t); - - Frame mInitFrame; - bool mAnimate; - float m_alpha; - - enum TrackMode { TM_NO_TRACK = 0, TM_ROTATE_AROUND, TM_ZOOM, TM_LOCAL_ROTATE, TM_FLY_Z, TM_FLY_PAN }; - - enum NavMode { NavTurnAround, NavFly }; - - enum LerpMode { LerpQuaternion, LerpEulerAngles }; - - enum RotationMode { RotationStable, RotationStandard }; - - Camera mCamera; - TrackMode mCurrentTrackingMode; - NavMode mNavMode; - LerpMode mLerpMode; - RotationMode mRotationMode; - Vector2i mMouseCoords; - Trackball mTrackball; - - QTimer m_timer; - - void setupCamera(); - - std::vector<Vector3f> mVertices; - std::vector<Vector3f> mNormals; - std::vector<int> mIndices; - - protected slots: - - virtual void animate(void); - virtual void drawScene(void); - - virtual void grabFrame(void); - virtual void stopAnimation(); - - virtual void setNavMode(int); - virtual void setLerpMode(int); - virtual void setRotationMode(int); - virtual void resetCamera(); - - protected: - virtual void initializeGL(); - virtual void resizeGL(int width, int height); - virtual void paintGL(); - - //-------------------------------------------------------------------------------- - virtual void mousePressEvent(QMouseEvent* e); - virtual void mouseReleaseEvent(QMouseEvent* e); - virtual void mouseMoveEvent(QMouseEvent* e); - virtual void keyPressEvent(QKeyEvent* e); - //-------------------------------------------------------------------------------- - - public: - EIGEN_MAKE_ALIGNED_OPERATOR_NEW - - RenderingWidget(); - ~RenderingWidget() {} - - QWidget* createNavigationControlWidget(); -}; - -class QuaternionDemo : public QMainWindow { - Q_OBJECT - public: - QuaternionDemo(); - - protected: - RenderingWidget* mRenderingWidget; -}; - -#endif // EIGEN_QUATERNION_DEMO_H
diff --git a/demos/opengl/trackball.cpp b/demos/opengl/trackball.cpp deleted file mode 100644 index dfa22a5..0000000 --- a/demos/opengl/trackball.cpp +++ /dev/null
@@ -1,52 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#include "trackball.h" -#include "camera.h" - -using namespace Eigen; - -void Trackball::track(const Vector2i& point2D) { - if (mpCamera == 0) return; - Vector3f newPoint3D; - bool newPointOk = mapToSphere(point2D, newPoint3D); - - if (mLastPointOk && newPointOk) { - Vector3f axis = mLastPoint3D.cross(newPoint3D).normalized(); - float cos_angle = mLastPoint3D.dot(newPoint3D); - if (std::abs(cos_angle) < 1.0) { - float angle = 2. * acos(cos_angle); - if (mMode == Around) - mpCamera->rotateAroundTarget(Quaternionf(AngleAxisf(angle, axis))); - else - mpCamera->localRotate(Quaternionf(AngleAxisf(-angle, axis))); - } - } - - mLastPoint3D = newPoint3D; - mLastPointOk = newPointOk; -} - -bool Trackball::mapToSphere(const Vector2i& p2, Vector3f& v3) { - if ((p2.x() >= 0) && (p2.x() <= int(mpCamera->vpWidth())) && (p2.y() >= 0) && (p2.y() <= int(mpCamera->vpHeight()))) { - double x = (double)(p2.x() - 0.5 * mpCamera->vpWidth()) / (double)mpCamera->vpWidth(); - double y = (double)(0.5 * mpCamera->vpHeight() - p2.y()) / (double)mpCamera->vpHeight(); - double sinx = sin(EIGEN_PI * x * 0.5); - double siny = sin(EIGEN_PI * y * 0.5); - double sinx2siny2 = sinx * sinx + siny * siny; - - v3.x() = sinx; - v3.y() = siny; - v3.z() = sinx2siny2 < 1.0 ? sqrt(1.0 - sinx2siny2) : 0.0; - - return true; - } else - return false; -}
diff --git a/demos/opengl/trackball.h b/demos/opengl/trackball.h deleted file mode 100644 index a0026ba..0000000 --- a/demos/opengl/trackball.h +++ /dev/null
@@ -1,42 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#ifndef EIGEN_TRACKBALL_H -#define EIGEN_TRACKBALL_H - -#include <Eigen/Geometry> - -class Camera; - -class Trackball { - public: - enum Mode { Around, Local }; - - Trackball() : mpCamera(0) {} - - void start(Mode m = Around) { - mMode = m; - mLastPointOk = false; - } - - void setCamera(Camera* pCam) { mpCamera = pCam; } - - void track(const Eigen::Vector2i& newPoint2D); - - protected: - bool mapToSphere(const Eigen::Vector2i& p2, Eigen::Vector3f& v3); - - Camera* mpCamera; - Eigen::Vector3f mLastPoint3D; - Mode mMode; - bool mLastPointOk; -}; - -#endif // EIGEN_TRACKBALL_H
diff --git a/doc/CustomizingEigen_CustomScalar.dox b/doc/CustomizingEigen_CustomScalar.dox index c774009..a409684 100644 --- a/doc/CustomizingEigen_CustomScalar.dox +++ b/doc/CustomizingEigen_CustomScalar.dox
@@ -16,6 +16,21 @@ The math function should be defined in the same namespace than \c T, or in the \c std namespace though that second approach is not recommended. +%Eigen calls these functions through \c Eigen::numext, which dispatches on \c NumTraits<T>. When \c NumTraits<T>::IsComplex is nonzero, \c numext::abs(x) computes <tt>hypot(real(x), imag(x))</tt> and \c numext::abs2(x) computes the sum of the squares of \c real(x) and \c imag(x). Neither calls an \c abs or \c abs2 defined in the namespace of \c T. If \c real and \c imag do not determine the magnitude of \c T, as for a quaternion, specialize both. Otherwise %Eigen sees the wrong magnitude: for example, pruning a sparse product silently drops a nonzero coefficient whose \c real and \c imag are zero. + +\code +namespace Eigen { +namespace numext { +template <> inline double abs(const my::Quaternion& x) { return norm(x); } +} +namespace internal { +template <> struct abs2_impl<my::Quaternion> { + static inline double run(const my::Quaternion& x) { return squaredNorm(x); } +}; +} +} +\endcode + Here is a concrete example adding support for the Adolc's \c adouble type. <a href="https://github.com/coin-or/ADOL-C">Adolc</a> is an automatic differentiation library. The type \c adouble is basically a real value tracking the values of any number of partial derivatives. \code
diff --git a/doc/Doxyfile.in b/doc/Doxyfile.in index 26f0067..cd7600b 100644 --- a/doc/Doxyfile.in +++ b/doc/Doxyfile.in
@@ -129,6 +129,7 @@ EIGEN_QT_SUPPORT \ EIGEN_STRONG_INLINE=inline \ EIGEN_DEVICE_FUNC= \ + "EIGEN_DEPRECATED_WITH_REASON(message)=" \ "EIGEN_MAKE_CWISE_UNARY_OP(METHOD,FUNCTOR,RETURN_TYPE)=const CwiseUnaryOp<FUNCTOR<Scalar>,const Derived> METHOD() const; using RETURN_TYPE=CwiseUnaryOp<FUNCTOR<Scalar>,const Derived>;" \ "EIGEN_MAKE_CWISE_BINARY_OP(METHOD,FUNCTOR)=template<typename OtherDerived> const CwiseBinaryOp<FUNCTOR<Scalar,typename OtherDerived::Scalar>, const Derived, const OtherDerived> METHOD(const EIGEN_CURRENT_STORAGE_BASE_CLASS<OtherDerived> &other) const;" \ "EIGEN_CWISE_PRODUCT_RETURN_TYPE(LHS,RHS)=CwiseBinaryOp<internal::scalar_product_op<LHS::Scalar,RHS::Scalar>, const LHS, const RHS>" \
diff --git a/doc/Manual.dox b/doc/Manual.dox index 0238307..7151a72 100644 --- a/doc/Manual.dox +++ b/doc/Manual.dox
@@ -26,6 +26,7 @@ - \subpage TopicUsingIntelMKL - \subpage TopicUsingAOCL - \subpage TopicCUDA + - \subpage TopicGPUModule - \subpage TopicPitfalls - \subpage TopicTemplateKeyword - \subpage TopicWritingEfficientProductExpression
diff --git a/doc/Pitfalls.dox b/doc/Pitfalls.dox index b3f49bc..c8434e3 100644 --- a/doc/Pitfalls.dox +++ b/doc/Pitfalls.dox
@@ -146,5 +146,24 @@ For coefficient-wise Boolean negation use <tt>!A.array()</tt>; for coefficient-wise XOR use <tt>A.array() != B.array()</tt>. +\section TopicPitfalls_reproducibility Bitwise reproducibility of products + +%Eigen does not promise bitwise identical floating-point results across machines, builds, or runs. Matrix +products, triangular solves, and the decompositions built on them split the summation into blocks sized to the +CPU caches, and floating-point addition is not associative, so a different blocking can change the last bits of +the result. The cache sizes are queried once, on the thread that runs the first such operation, and are kept for +the life of the process. On CPUs that mix core types with different caches, such as Intel processors with +P-cores and E-cores, the results of a program can therefore depend on which kind of core happened to run its +first product. The number of threads and the instruction set enabled at compile time change the results in the +same way. + +Where bitwise reproducibility matters, either: + - pin the process to one core type, for example with \c taskset on Linux or <tt>start /affinity</tt> on Windows, + or + - call \c Eigen::setCpuCacheSizes() with fixed values before the first product, so that the blocking no longer + depends on the core that is running. + +In both cases, also fix the number of threads and the build configuration. + */ }
diff --git a/doc/QuickReference.dox b/doc/QuickReference.dox index 91f006a..7f5ad2d 100644 --- a/doc/QuickReference.dox +++ b/doc/QuickReference.dox
@@ -732,6 +732,15 @@ m3 -= s1 * m3.adjoint() * m1.selfadjointView<Eigen::Lower>();\endcode </td></tr> <tr><td> +Symmetric permutation in place: \n +\f$ S := P S P^T \f$ (only the selected triangle is accessed) +</td><td>\code +M.selfadjointView<Eigen::Lower>().applyPermutation(P); +M.selfadjointView<Eigen::Upper>().applyPermutation(P.inverse()); +// P can also be a Transpositions sequence. +M.selfadjointView<Eigen::Lower>().applyTransposition(i,j);\endcode +</td></tr> +<tr><td> Rank 1 and rank K update: \n \f$ upper(M_1) \mathrel{{+}{=}} s_1 M_2 M_2^* \f$ \n \f$ lower(M_1) \mathbin{{-}{=}} M_2^* M_2 \f$
diff --git a/doc/TopicLinearAlgebraDecompositions.dox b/doc/TopicLinearAlgebraDecompositions.dox index 2280c67..78c2802 100644 --- a/doc/TopicLinearAlgebraDecompositions.dox +++ b/doc/TopicLinearAlgebraDecompositions.dox
@@ -144,7 +144,7 @@ <td>-</td> <td>Yes</td> <td>Excellent</td> - <td>-</td> + <td>Blocking</td> </tr> <tr> @@ -335,10 +335,10 @@ \b Notes: <ul> -<li><a name="note1">\b 1: </a>There exist two variants of the LDLT algorithm. %Eigen's LDLT produces a pure diagonal D matrix and therefore cannot handle indefinite matrices; it uses symmetric pivoting, which keeps the factorization stable when pivots are tiny or zero. The other variant, which produces the block-diagonal D needed for indefinite self-adjoint matrices, is provided by the BunchKaufman class.</li> +<li><a name="note1">\b 1: </a>There exist two variants of the LDLT algorithm. %Eigen's LDLT produces a pure diagonal D matrix and therefore is not stable on indefinite matrices; it pivots on the diagonal entry of largest magnitude of the remaining Schur complement, which keeps the factorization stable when pivots are tiny or zero and puts the negligible pivots of a semidefinite matrix last. The other variant, which produces the block-diagonal D needed for indefinite self-adjoint matrices, is provided by the BunchKaufman class.</li> <li><a name="note2">\b 2: </a>Eigenvalues, SVD and Schur decompositions rely on iterative algorithms. Their convergence speed depends on how well the eigenvalues are separated.</li> <li><a name="note3">\b 3: </a>Our JacobiSVD is two-sided, which buys high relative accuracy on square matrices: where a bidiagonalizing algorithm only computes each singular value accurately relative to the largest one, Jacobi computes it accurately relative to itself, so that tiny singular values are still meaningful. That property is conditional rather than absolute -- the relative error is governed by the condition number of the matrix obtained by scaling its columns to unit norm, so a matrix that stays ill-conditioned under any such scaling loses it (J. Demmel and K. Veselic, "Jacobi's method is more accurate than QR", SIAM J. Matrix Anal. Appl. 13 (1992), pp. 1204-1245). For non-square matrices, we have to use a QR preconditioner first; \c PreconditionSquareMatrix applies it to square matrices too. Householder QR is backward stable with or without pivoting, so this choice is about rank revelation rather than about the accuracy of the factorization itself: the default \c ColPivHouseholderQRPreconditioner reveals rank reliably in practice but without a guarantee, \c FullPivHouseholderQRPreconditioner inspects the whole trailing submatrix at a substantially higher cost, and \c HouseholderQRPreconditioner and \c NoQRPreconditioner do not pivot at all. See the guidance below on complete pivoting.</li> -<li><a name="note4">\b 4: </a>These factorizations are all backward stable, but pivoting buys something different in each. For LU the backward error bound is proportional to the growth factor: partial pivoting bounds it only by \f$ 2^{n-1} \f$, whereas complete pivoting admits a far smaller bound that grows only slightly faster than polynomially in \c n. That partial-pivoting worst case is not observed on real matrices, however, and partial pivoting is more than sufficient in practice. Householder QR is backward stable irrespective of pivoting, with a bound that involves no growth factor at all; there, pivoting serves rank revelation rather than stability, and what separates column from complete pivoting is how reliably the trailing diagonal of R exposes a rank deficiency. Since complete pivoting also rules out the cache-friendly blocked algorithms, LAPACK and %Eigen both treat partial and column pivoting as the defaults for LU and QR; see the guidance below. This is a judgement about those two factorizations rather than about complete pivoting in general: LAPACK does provide a complete-pivoting Cholesky (\c pstrf) for positive semidefinite matrices, where the pivoting also determines the rank.</li> +<li><a name="note4">\b 4: </a>These factorizations are all backward stable, but pivoting buys something different in each. For LU the backward error bound is proportional to the growth factor: partial pivoting bounds it only by \f$ 2^{n-1} \f$, whereas complete pivoting admits a far smaller bound that grows only slightly faster than polynomially in \c n. That partial-pivoting worst case is not observed on real matrices, however, and partial pivoting is more than sufficient in practice. Householder QR is backward stable irrespective of pivoting, with a bound that involves no growth factor at all; there, pivoting serves rank revelation rather than stability, and what separates column from complete pivoting is how reliably the trailing diagonal of R exposes a rank deficiency. Since complete pivoting also rules out the cache-friendly blocked algorithms, LAPACK and %Eigen both treat partial and column pivoting as the defaults for LU and QR; see the guidance below. This is a judgement about those two factorizations rather than about complete pivoting in general: a symmetric factorization searches only the diagonal of the Schur complement for its pivot, which blocking leaves cheap to maintain. LAPACK thus provides a blocked complete-pivoting Cholesky (\c pstrf) for positive semidefinite matrices, where the pivoting also determines the rank, and %Eigen's LDLT pivots the same way.</li> <li><a name="note5">\b 5: </a>Cholesky needs no pivoting for stability: on a positive definite matrix the growth factor is bounded by 1, so the unpivoted factorization is backward stable as it stands. What conditioning governs is instead whether it completes: on a matrix that is positive definite but nearly singular, rounding can drive a computed pivot to zero or below, and %Eigen reports that through info() rather than returning a meaningless factor. Use LDLT, or a rank-revealing decomposition, for such matrices.</li> </ul>
diff --git a/doc/TopicScalarTypes.dox b/doc/TopicScalarTypes.dox index d6e3d2e..2ade8f7 100644 --- a/doc/TopicScalarTypes.dox +++ b/doc/TopicScalarTypes.dox
@@ -40,7 +40,15 @@ - <b>\c Eigen::half and \c Eigen::bfloat16</b> are primarily storage and interchange formats: on hardware without native arithmetic, operations are emulated by converting to \c float and back. Both are vectorized on instruction sets with hardware support (see - \ref TopicVectorization). + \ref TopicVectorization). A matrix-matrix product of two \c half or two \c bfloat16 matrices + that runs on the general matrix-matrix kernel sums in \c float over the whole depth and rounds + each result coefficient once; products small enough for the coefficient-based path, products + with a vector operand or result, and triangular and self-adjoint products still round in the + operand type. On hardware with native \c half arithmetic, such as Armv8.2-A with FP16, this + makes a \c half product about twice as slow as the native \c half kernel it replaces. To keep a + \c float result, write <tt>A.cast<float>() * B.cast<float>()</tt>: the kernel converts the + operands while it reads them, without first copying them to \c float, except with + \c EIGEN_USE_BLAS and on SME, where the casts are materialized. The scalar types with vectorized kernels for a given instruction set are listed on the \ref TopicVectorization "vectorization page"; every supported scalar type also works through the
diff --git a/doc/TutorialSparse.dox b/doc/TutorialSparse.dox index 2f750a4..6e25ace 100644 --- a/doc/TutorialSparse.dox +++ b/doc/TutorialSparse.dox
@@ -101,7 +101,7 @@ The last step consists of effectively solving the assembled problem. Since the resulting matrix \c A is symmetric by construction, we can perform a direct Cholesky factorization via the SimplicialLDLT class which behaves like its LDLT counterpart for dense objects. -The resulting vector \c x contains the pixel values as a 1D array which is saved to a jpeg file shown on the right of the code above. +The resulting vector \c x contains the pixel values as a 1D array which is saved to a grayscale PGM image file. Describing the \a buildProblem and \a save functions is out of the scope of this tutorial. They are given \ref TutorialSparse_example_details "here" for the curious and reproducibility purpose.
diff --git a/doc/UsingGPUModule.dox b/doc/UsingGPUModule.dox new file mode 100644 index 0000000..f051c16 --- /dev/null +++ b/doc/UsingGPUModule.dox
@@ -0,0 +1,186 @@ +namespace Eigen { + +/** \page TopicGPUModule Offloading linear algebra to a GPU with the GPU module + +The contrib module <a href="contrib/group__GPU__Module.html"><tt>contrib/Eigen/GPU</tt></a> runs dense and sparse linear +algebra on NVIDIA GPUs from ordinary host code. It dispatches to the CUDA math libraries: cuBLAS, cuSOLVER, cuSPARSE, +cuFFT, NPP, and optionally cuDSS. You write no kernels, and your source files stay regular \c .cpp files. + +This is a different feature from \ref TopicCUDA "using Eigen in CUDA kernels", where %Eigen's fixed-size objects run +inside your own kernels, one small problem per thread. Use this module to run large operations on the GPU, such as a +matrix product or a factorization. + +\warning Like the other contrib modules, this module has looser API-stability guarantees than the main library; see +\ref Experimental. It supports NVIDIA CUDA only, not HIP or SYCL. + +\section TopicGPUModule_Setup Setup + +The module needs the CUDA Toolkit, version 11.8 or later. Sparse direct solvers also need cuDSS, which is installed +separately and requires CUDA 12.0 or later. Define \c EIGEN_USE_GPU before including the header, because without it the +header declares none of the module: + +\code +#define EIGEN_USE_GPU +#include <Eigen/Dense> +#include <contrib/Eigen/GPU> +\endcode + +The module is header-only, but your program must link NVIDIA libraries. Always link +<tt>-lcublas -lcublasLt -lcudart</tt>, and add the libraries of the features you use: + +<table class="manual"> +<tr><th>Feature</th><th>Additional link flags</th></tr> +<tr><td>\c DeviceMatrix, products, triangular solves</td><td>none</td></tr> +<tr class="alt"><td>Dense solvers</td><td><tt>-lcusolver</tt></td></tr> +<tr><td>\c norm(), \c DeviceScalar arithmetic, \c /=, \c cwiseProduct</td><td><tt>-lnpps -lnppc</tt></td></tr> +<tr class="alt"><td>\c gpu::SparseContext, \c gpu::DeviceSparseView</td><td><tt>-lcusparse</tt></td></tr> +<tr><td>\c ConjugateGradient on a \c gpu::DeviceSparseView</td><td><tt>-lcusparse -lnpps -lnppc</tt></td></tr> +<tr class="alt"><td>\c gpu::FFT</td><td><tt>-lcufft</tt></td></tr> +<tr><td>Sparse direct solvers (define \c EIGEN_CUDSS before the include)</td><td><tt>-lcudss</tt></td></tr> +</table> + +With CMake, the \c CUDAToolkit package provides these libraries as targets. This program uses the dense solvers: + +\code{.cmake} +find_package(CUDAToolkit REQUIRED) +target_link_libraries(app PRIVATE CUDA::cublas CUDA::cublasLt CUDA::cusolver CUDA::cudart) +\endcode + +For other features, add \c CUDA::cusparse, \c CUDA::cufft, \c CUDA::npps and \c CUDA::nppc as the table requires. cuDSS +is not part of that package: find it with <tt>find_package(cudss)</tt> and link its \c cudss target. + +\section TopicGPUModule_Dense Dense matrices + +GPU and CPU code coexist in one program, because there is no global switch: you choose the GPU by using the types in +the \c Eigen::gpu namespace. A \c gpu::DeviceMatrix holds a dense, column-major matrix in device memory. Move data with +\c fromHost() and \c toHost(). Between those calls, expressions look like their CPU counterparts: + +\code +MatrixXd A = ..., B = ...; // A is symmetric positive definite + +auto d_A = gpu::DeviceMatrix<double>::fromHost(A); +auto d_B = gpu::DeviceMatrix<double>::fromHost(B); + +gpu::DeviceMatrix<double> d_C = d_A * d_B; // cuBLAS +gpu::DeviceMatrix<double> d_X = d_A.llt().solve(d_B); // cuSOLVER + +MatrixXd X = d_X.toHost(); +\endcode + +The expression \c d_A.llt().solve(d_B) factorizes on every evaluation. To reuse a factorization, or to check for +numerical failure, use a solver class instead. The classes are \c gpu::LLT, \c gpu::LU, \c gpu::QR, \c gpu::SVD and +\c gpu::SelfAdjointEigenSolver: + +\code +gpu::LLT<double> llt; +llt.compute(d_A); +if (llt.info() != Success) { ... } + +gpu::DeviceMatrix<double> d_X1 = llt.solve(d_B1); // reuses the factor on the device +gpu::DeviceMatrix<double> d_X2 = llt.solve(d_B2); +\endcode + +The solvers also accept host matrices and handle the transfers themselves, which makes them the smallest change from +CPU code: + +\code +gpu::LLT<double> llt(A); // instead of Eigen::LLT<MatrixXd> llt(A); +MatrixXd X = llt.solve(B); +\endcode + +\section TopicGPUModule_Sparse Sparse matrices + +Define \c EIGEN_CUDSS to enable \c gpu::SparseLLT, \c gpu::SparseLDLT and \c gpu::SparseLU. They take an ordinary +\c SparseMatrix and host vectors: + +\code +#define EIGEN_USE_GPU +#define EIGEN_CUDSS +#include <Eigen/Sparse> +#include <contrib/Eigen/GPU> + +SparseMatrix<double> A = ...; +VectorXd b = ...; + +gpu::SparseLLT<double> llt(A); // instead of SimplicialLLT<SparseMatrix<double>> llt(A); +if (llt.info() != Success) { ... } +VectorXd x = llt.solve(b); +\endcode + +As with the CPU solvers, a sequence of matrices with one sparsity pattern can share the symbolic analysis, by calling +\c analyzePattern() once and \c factorize() for each matrix. + +For sparse matrix-vector products, \c gpu::SparseContext uses cuSPARSE. Its \c deviceView() also lets %Eigen's own +\c ConjugateGradient run on the GPU, with the identity preconditioner and real scalars: + +\code +#include <Eigen/IterativeLinearSolvers> + +gpu::SparseContext<double> spmv; +auto mat = spmv.deviceView(A); +auto d_b = gpu::DeviceMatrix<double>::fromHost(b); +gpu::DeviceMatrix<double> d_x(A.rows(), 1); +d_x.setZero(); + +ConjugateGradient<gpu::DeviceSparseView<double>, Lower | Upper, IdentityPreconditioner> cg; +cg.compute(mat); // stores a pointer: mat and spmv must outlive cg +cg.solveWithGuessInPlace(d_b, d_x); // d_x holds the initial guess +\endcode + +A \c gpu::SparseContext caches one matrix at a time, so another \c deviceView() or a \c multiply() with a host matrix +invalidates its earlier views. Use one context per matrix. + +\section TopicGPUModule_Execution Asynchronous execution and threads + +Operations are asynchronous by default. The host waits only where it must: in \c fromHost(), \c toHost(), \c info(), +when a \c gpu::DeviceScalar, the result type of \c dot() and \c norm(), converts to a host scalar, and in the forms that +return host data, such as \c solve() on a host matrix, and \c gpu::FFT and \c gpu::SparseContext::multiply() on host +vectors. The dense solvers also wait when \c compute() takes a host matrix, and an internal workspace that has to grow +waits for the work using it. Debug builds also wait to check the factorization, on the first +\c solve() after \c compute() and on each \c llt().solve(). Transfers are usually the main cost, so keep intermediate +results on the device and download only what the host needs. + +Each thread gets a default \c gpu::Context, which holds a CUDA stream and the library handles. One \c Context is not +thread-safe. To run work on separate streams, or to share a stream with your own CUDA code, create contexts explicitly. +\c DeviceMatrix synchronizes across streams automatically: + +\code +gpu::Context ctx1, ctx2; +d_C1.device(ctx1) = d_A1 * d_B1; // runs on stream 1 +d_C2.device(ctx2) = d_A2 * d_B2; // runs on stream 2, concurrently + +gpu::Context ctx3(my_existing_stream); // borrows the stream without owning it +\endcode + +\section TopicGPUModule_Errors Errors + +There are two kinds of failure: + +- A failed CUDA or library call stops the program, in release builds as well as debug builds. To handle such failures + yourself, for instance by throwing an exception, define the macro <tt>EIGEN_GPU_CHECK_FAILED(error, expression, file, + line)</tt> before including the module. +- A numerical failure, such as a matrix that is not positive definite, is reported by \c info(). Release builds do not + check it for you, and the one-shot \c llt().solve() form has no \c info(), so use a solver class and call \c info() + where failure is possible. cuSOLVER 11.4.1 and 11.4.2, from CUDA 11.8 and 12.0, do not detect a non-positive-definite + input to \c gpu::LLT: \c info() reports success and the factor contains NaNs. CUDA 12.1 and later report it. + +\section TopicGPUModule_Limitations Limitations + +- \c DeviceMatrix is not an %Eigen expression type and does not derive from \c MatrixBase, so it cannot be passed to + code templated on \c MatrixBase. It also has no coefficient access. +- Each expression dispatches to NVIDIA library calls, usually one: \c norm() is a dot product and an NPP square root, + and \c llt().solve() is a copy, a factorization and a solve. Unlike CPU expressions, GPU expressions do not fuse into + a single loop, and an unsupported expression is a compile error. +- Scalars are \c float, \c double, \c std::complex<float> and \c std::complex<double>, unless a class states otherwise. +- Numerical behavior and performance are those of the NVIDIA libraries. In particular, the default compute types let + cuBLAS choose faster algorithms that might not be bit-reproducible; define \c EIGEN_NO_CUDA_TENSOR_OPS to disable them. + +\section TopicGPUModule_Reference Further reference + +The <a href="https://gitlab.com/libeigen/eigen/-/blob/master/contrib/Eigen/src/GPU/README.md">module README</a> is the +complete reference. It covers the BLAS operations, FFT, solver configuration, stream control, and the mapping from each +expression to its library calls. + +*/ + +}
diff --git a/doc/UsingNVCC.dox b/doc/UsingNVCC.dox index c0c5a73..f6ef0bd 100644 --- a/doc/UsingNVCC.dox +++ b/doc/UsingNVCC.dox
@@ -15,6 +15,8 @@ before including any %Eigen header. This might be useful to disable some warnings when a \c .cu file makes use of %Eigen on the host side only. +To run large operations on the GPU from host code, without writing kernels, see \ref TopicGPUModule. + %Eigen requires CUDA 11.8 or later and compute capability 6.0 (\c sm_60) or later; CUDA 13 itself requires \c sm_75. For HIP the floor is ROCm 5.6.
diff --git a/doc/special_examples/CMakeLists.txt b/doc/special_examples/CMakeLists.txt index b0e90b0..0b77540 100644 --- a/doc/special_examples/CMakeLists.txt +++ b/doc/special_examples/CMakeLists.txt
@@ -1,26 +1,16 @@ # SPDX-FileCopyrightText: The Eigen Authors # SPDX-License-Identifier: MPL-2.0 -if(NOT EIGEN_TEST_NOQT) - find_package(Qt4) - if(QT4_FOUND) - include(${QT_USE_FILE}) - endif() -endif() +add_executable(Tutorial_sparse_example Tutorial_sparse_example.cpp Tutorial_sparse_example_details.cpp) +target_link_libraries(Tutorial_sparse_example ${EIGEN_STANDARD_LIBRARIES_TO_LINK_TO} Eigen3::Eigen) +add_dependencies(all_examples Tutorial_sparse_example) -if(QT4_FOUND) - add_executable(Tutorial_sparse_example Tutorial_sparse_example.cpp Tutorial_sparse_example_details.cpp) - target_link_libraries(Tutorial_sparse_example ${EIGEN_STANDARD_LIBRARIES_TO_LINK_TO} ${QT_QTCORE_LIBRARY} ${QT_QTGUI_LIBRARY} Eigen3::Eigen) - - add_custom_command( - TARGET Tutorial_sparse_example - POST_BUILD - COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/../html/ - COMMAND Tutorial_sparse_example ARGS ${CMAKE_CURRENT_BINARY_DIR}/../html/Tutorial_sparse_example.jpeg - ) - - add_dependencies(all_examples Tutorial_sparse_example) -endif() +add_custom_command( + TARGET Tutorial_sparse_example + POST_BUILD + COMMAND Tutorial_sparse_example + ARGS ${CMAKE_CURRENT_BINARY_DIR}/Tutorial_sparse_example.pgm +) add_executable(random_generators random_generators.cpp) target_link_libraries(random_generators ${EIGEN_STANDARD_LIBRARIES_TO_LINK_TO} Eigen3::Eigen)
diff --git a/doc/special_examples/Tutorial_sparse_example_details.cpp b/doc/special_examples/Tutorial_sparse_example_details.cpp index 81ef47a..833cb50 100644 --- a/doc/special_examples/Tutorial_sparse_example_details.cpp +++ b/doc/special_examples/Tutorial_sparse_example_details.cpp
@@ -2,8 +2,8 @@ // SPDX-License-Identifier: MPL-2.0 #include <Eigen/Sparse> +#include <fstream> #include <vector> -#include <QImage> typedef Eigen::SparseMatrix<double> SpMat; // declares a column-major sparse matrix type of double typedef Eigen::Triplet<double> T; @@ -36,10 +36,10 @@ } } +// Writes the n x n pixel values in x, which lie in [0,1], as a binary PGM (grayscale) image. void saveAsBitmap(const Eigen::VectorXd& x, int n, const char* filename) { Eigen::Array<unsigned char, Eigen::Dynamic, Eigen::Dynamic> bits = (x * 255).cast<unsigned char>(); - QImage img(bits.data(), n, n, QImage::Format_Indexed8); - img.setColorCount(256); - for (int i = 0; i < 256; i++) img.setColor(i, qRgb(i, i, i)); - img.save(filename); + std::ofstream file(filename, std::ios::binary); + file << "P5\n" << n << ' ' << n << "\n255\n"; + file.write(reinterpret_cast<const char*>(bits.data()), bits.size()); }
diff --git a/failtest/CMakeLists.txt b/failtest/CMakeLists.txt index c99ddc8..8ab7247 100644 --- a/failtest/CMakeLists.txt +++ b/failtest/CMakeLists.txt
@@ -125,4 +125,6 @@ ei_add_failtest("shape_size_rows") ei_add_failtest("shape_size_cols") +ei_add_failtest("numtraits_unknown_binary_precision") +ei_add_failtest("block_linear_coeff_runtime_vector") ei_add_failtest_fixture()
diff --git a/failtest/block_linear_coeff_runtime_vector.cpp b/failtest/block_linear_coeff_runtime_vector.cpp new file mode 100644 index 0000000..9f64697 --- /dev/null +++ b/failtest/block_linear_coeff_runtime_vector.cpp
@@ -0,0 +1,18 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include "../Eigen/Core" + +// A block of an expression without direct access has linear coefficient access only if it is a vector or an inner +// panel at compile time, not merely a vector at runtime. +#ifdef EIGEN_SHOULD_FAIL_TO_BUILD +#define BLOCK_ROWS Dynamic +#else +#define BLOCK_ROWS 1 +#endif + +using namespace Eigen; + +float foo(const MatrixXf &m) { return (m + m).block<BLOCK_ROWS, 6>(1, 0, 1, 6).coeff(3); } + +int main() {}
diff --git a/failtest/numtraits_unknown_binary_precision.cpp b/failtest/numtraits_unknown_binary_precision.cpp new file mode 100644 index 0000000..6ccfdca --- /dev/null +++ b/failtest/numtraits_unknown_binary_precision.cpp
@@ -0,0 +1,14 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include <Eigen/Core> + +struct Scalar {}; + +int main() { +#ifdef EIGEN_SHOULD_FAIL_TO_BUILD + return Eigen::NumTraits<Scalar>::digits(); +#else + return Eigen::NumTraits<Scalar>::digits10() + Eigen::NumTraits<Scalar>::max_digits10(); +#endif +}
diff --git a/lapack/CMakeLists.txt b/lapack/CMakeLists.txt index 27faa45..da000e1 100644 --- a/lapack/CMakeLists.txt +++ b/lapack/CMakeLists.txt
@@ -14,7 +14,7 @@ enable_language(Fortran) if("${CMAKE_Fortran_COMPILER_ID}" STREQUAL "GNU") if ("${CMAKE_Fortran_COMPILER_VERSION}" VERSION_GREATER_EQUAL 10.0) - # We use an old version of LAPACK with argument type mismatches. + # Reference LAPACK sources contain argument type mismatches. # Allow them to compile anyway with newer GNU versions. set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -fallow-argument-mismatch") endif() @@ -42,34 +42,18 @@ if(EIGEN_ENABLE_LAPACK_TESTS) - get_filename_component(eigen_full_path_to_reference_lapack "./reference/" ABSOLUTE) - if(NOT EXISTS ${eigen_full_path_to_reference_lapack}) - # Download lapack and install sources and testing at the right place - message(STATUS "Download lapack_addons_3.4.1.tgz...") - - file(DOWNLOAD "http://downloads.tuxfamily.org/eigen/lapack_addons_3.4.1.tgz" - "${CMAKE_CURRENT_SOURCE_DIR}/lapack_addons_3.4.1.tgz" - INACTIVITY_TIMEOUT 15 - TIMEOUT 240 - STATUS download_status - EXPECTED_MD5 ab5742640617e3221a873aba44bbdc93 - SHOW_PROGRESS) - - message(STATUS ${download_status}) - list(GET download_status 0 download_status_num) - set(download_status_num 0) - if(download_status_num EQUAL 0) - message(STATUS "Setup lapack reference and lapack unit tests") - execute_process(COMMAND tar xzf "lapack_addons_3.4.1.tgz" WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}) - else() - message(STATUS "Download of lapack_addons_3.4.1.tgz failed, LAPACK unit tests won't be enabled") - set(EIGEN_ENABLE_LAPACK_TESTS false) - endif() - + include(FetchContent) + FetchContent_Declare(eigen_lapack_reference + URL https://codeload.github.com/Reference-LAPACK/lapack/tar.gz/refs/tags/v3.12.1 + URL_HASH SHA256=2ca6407a001a474d4d4d35f3a61550156050c48016d949f0da0529c0aa052422 + TLS_VERIFY ON) + FetchContent_GetProperties(eigen_lapack_reference) + if(NOT eigen_lapack_reference_POPULATED) + FetchContent_Populate(eigen_lapack_reference) endif() + set(LAPACK_SOURCE_DIR "${eigen_lapack_reference_SOURCE_DIR}") + set(LAPACK_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}") - get_filename_component(eigen_full_path_to_reference_lapack "./reference/" ABSOLUTE) - if(EXISTS ${eigen_full_path_to_reference_lapack}) set(EigenLapack_funcfilenames ssyev.f dsyev.f csyev.f zsyev.f spotrf.f dpotrf.f cpotrf.f zpotrf.f @@ -88,18 +72,21 @@ slamch.f dlamch.f ilaver.f) - file(GLOB ReferenceLapack_SRCS0 RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "reference/*.f") + file(GLOB ReferenceLapack_SRCS0 "${LAPACK_SOURCE_DIR}/SRC/*.f" + "${LAPACK_SOURCE_DIR}/SRC/*.F" "${LAPACK_SOURCE_DIR}/SRC/*.f90" + "${LAPACK_SOURCE_DIR}/SRC/*.F90" + "${LAPACK_SOURCE_DIR}/SRC/DEPRECATED/*.f") + list(APPEND ReferenceLapack_SRCS0 + "${LAPACK_SOURCE_DIR}/INSTALL/sroundup_lwork.f" + "${LAPACK_SOURCE_DIR}/INSTALL/droundup_lwork.f") foreach(filename1 IN LISTS ReferenceLapack_SRCS0) - string(REPLACE "reference/" "" filename ${filename1}) + get_filename_component(filename "${filename1}" NAME) list(FIND EigenLapack_SRCS ${filename} id1) list(FIND EigenLapack_funcfilenames ${filename} id2) if((id1 EQUAL -1) AND (id2 EQUAL -1)) - set(ReferenceLapack_SRCS ${ReferenceLapack_SRCS} reference/${filename}) + list(APPEND ReferenceLapack_SRCS "${filename1}") endif() endforeach() - endif() - - endif() set(EIGEN_LAPACK_TARGETS "") @@ -134,19 +121,8 @@ endforeach() -get_filename_component(eigen_full_path_to_testing_lapack "./testing/" ABSOLUTE) -if(EIGEN_ENABLE_LAPACK_TESTS AND EXISTS ${eigen_full_path_to_testing_lapack}) +if(EIGEN_ENABLE_LAPACK_TESTS) - # The following comes from lapack/TESTING/CMakeLists.txt - # Get Python - find_package(PythonInterp) - message(STATUS "Looking for Python found - ${PYTHONINTERP_FOUND}") - if (PYTHONINTERP_FOUND) - message(STATUS "Using Python version ${PYTHON_VERSION_STRING}") - endif() - - set(LAPACK_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}) - set(LAPACK_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}) set(BUILD_SINGLE true) set(BUILD_DOUBLE true) set(BUILD_COMPLEX true) @@ -159,23 +135,39 @@ CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS}") endif() file(MAKE_DIRECTORY "${LAPACK_BINARY_DIR}/TESTING") - add_subdirectory(testing/MATGEN) - add_subdirectory(testing/LIN) - add_subdirectory(testing/EIG) + set(LAPACK_LIBRARIES eigen_lapack_static) + set(BLAS_LIBRARIES eigen_blas_static) + set(TMGLIB tmglib) + set(LAPACK_VERSION 3.12.1) + set(LAPACK_MAJOR_VERSION 3) + # Upstream's test libraries are build-only dependencies. + function(lapack_install_library target) + endfunction() + add_subdirectory("${LAPACK_SOURCE_DIR}/TESTING/MATGEN" testing/MATGEN) + add_subdirectory("${LAPACK_SOURCE_DIR}/TESTING/LIN" testing/LIN) + add_subdirectory("${LAPACK_SOURCE_DIR}/TESTING/EIG" testing/EIG) + # Some LAPACK 3.12.1 checks are expected to fail: obsolete bitwise eigenvalue + # comparisons, unnormalized RFP errors, a balance-test denominator missing ABS, + # and a reference-CSTEIN orthogonality limitation. These are LAPACK-side issues; + # keep the tests enabled and let the summary report their failures. + set(EIGEN_LAPACK_TEST_NAMES "") + set(EIGEN_LAPACK_TEST_OUTPUTS "") macro(add_lapack_test output input target) - set(TEST_INPUT "${LAPACK_SOURCE_DIR}/testing/${input}") + set(TEST_INPUT "${LAPACK_SOURCE_DIR}/TESTING/${input}") set(TEST_OUTPUT "${LAPACK_BINARY_DIR}/TESTING/${output}") string(REPLACE "." "_" input_name ${input}) set(testName "${target}_${input_name}") if(EXISTS "${TEST_INPUT}") add_dependencies(buildtests ${target}) + list(APPEND EIGEN_LAPACK_TEST_NAMES "LAPACK-${testName}") + list(APPEND EIGEN_LAPACK_TEST_OUTPUTS "${TEST_OUTPUT}") add_test(NAME LAPACK-${testName} COMMAND "${CMAKE_COMMAND}" -DTEST=$<TARGET_FILE:${target}> -DINPUT=${TEST_INPUT} -DOUTPUT=${TEST_OUTPUT} -DINTDIR=${CMAKE_CFG_INTDIR} - -P "${LAPACK_SOURCE_DIR}/testing/runtest.cmake") + -P "${LAPACK_SOURCE_DIR}/TESTING/runtest.cmake") endif() endmacro() @@ -465,16 +457,20 @@ # ============================================================================== - if(EXISTS "${LAPACK_SOURCE_DIR}/lapack_testing.py") - execute_process(COMMAND ${CMAKE_COMMAND} -E copy ${LAPACK_SOURCE_DIR}/lapack_testing.py ${LAPACK_BINARY_DIR}) - else() - execute_process(COMMAND ${CMAKE_COMMAND} -E copy ${LAPACK_SOURCE_DIR}/testing/lapack_testing.py ${LAPACK_BINARY_DIR}) + if(EIGEN_LAPACK_TEST_NAMES) + string(REPLACE ";" "\n" lapack_outputs "${EIGEN_LAPACK_TEST_OUTPUTS}") + file(WRITE "${LAPACK_BINARY_DIR}/lapack_test_outputs.txt" "${lapack_outputs}\n") + add_test( + NAME LAPACK_Test_Summary + WORKING_DIRECTORY ${LAPACK_BINARY_DIR} + COMMAND "${CMAKE_COMMAND}" + "-DOUTPUT_MANIFEST=${LAPACK_BINARY_DIR}/lapack_test_outputs.txt" + "-DREPORT_FILE=${LAPACK_BINARY_DIR}/TESTING/testing_results.txt" + -P "${CMAKE_CURRENT_SOURCE_DIR}/testing_summary.cmake" + ) + set_tests_properties(${EIGEN_LAPACK_TEST_NAMES} PROPERTIES FIXTURES_SETUP lapack_outputs) + set_tests_properties(LAPACK_Test_Summary PROPERTIES FIXTURES_REQUIRED lapack_outputs) endif() - add_test( - NAME LAPACK_Test_Summary - WORKING_DIRECTORY ${LAPACK_BINARY_DIR} - COMMAND ${PYTHON_EXECUTABLE} "lapack_testing.py" - ) endif()
diff --git a/lapack/README.md b/lapack/README.md new file mode 100644 index 0000000..d96310a --- /dev/null +++ b/lapack/README.md
@@ -0,0 +1,21 @@ +The LAPACK compatibility library is enabled with `EIGEN_BUILD_LAPACK` and +requires `EIGEN_BUILD_BLAS`. Its optional numerical tests +(`EIGEN_ENABLE_LAPACK_TESTS`) require a Fortran compiler and the LAPACK test data. + +The tests and their summary run through CMake/CTest without Python. +`LAPACK_Test_Summary` waits for the registered numerical tests and combines +only their expected output files into `TESTING/testing_results.txt` in the +LAPACK build directory. It fails when the output reports numerical errors, +illegal-argument or INFO diagnostics, when an expected output is missing, or +when no numerical tests were analyzed. Selecting `LAPACK_Test_Summary` alone also +runs its numerical tests through a CTest fixture. If a numerical driver fails to +execute successfully, CTest skips the summary. + +Balancing tests contribute their example counts and fail on nonzero INFO/ILO +error counts or non-finite maximum errors. Their finite maximum errors remain +in the combined report; these drivers do not specify a pass/fail threshold. + +The numerical tests use unmodified LAPACK 3.12.1 sources, downloaded with a pinned +SHA-256 hash. They exercise Eigen's BLAS/LAPACK implementations, using reference +LAPACK for the remaining routines. All test cases remain enabled; known numerical +and test-driver failures are reported by the summary.
diff --git a/lapack/dsecnd_NONE.f b/lapack/dsecnd_NONE.f index 518d4aa..a283e5d 100644 --- a/lapack/dsecnd_NONE.f +++ b/lapack/dsecnd_NONE.f
@@ -35,8 +35,10 @@ *> \ingroup auxOTHERauxiliary * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI DOUBLE PRECISION FUNCTION DSECND( ) * * -- LAPACK auxiliary routine (version 3.4.0) --
diff --git a/lapack/lapack_testing.py b/lapack/lapack_testing.py deleted file mode 100644 index 1e43e3a..0000000 --- a/lapack/lapack_testing.py +++ /dev/null
@@ -1,325 +0,0 @@ -#! /usr/bin/env python -# -*- coding: utf-8 -*- -# -# SPDX-FileCopyrightText: 1992-2013 The University of Tennessee and The University of Tennessee Research Foundation -# SPDX-FileCopyrightText: 2000-2013 The University of California Berkeley -# SPDX-FileCopyrightText: 2006-2013 The University of Colorado Denver -# SPDX-FileCopyrightText: The Eigen Authors -# SPDX-License-Identifier: BSD-3-Clause -# - -############################################################################### -# lapack_testing.py -############################################################################### - - -from __future__ import print_function -from subprocess import Popen, STDOUT, PIPE -import os, sys, math -import getopt -# Arguments -try: - opts, args = getopt.getopt(sys.argv[1:], "hd:srep:t:n", - ["help", "dir", "short", "run", "error","prec=","test=","number"]) - -except getopt.error as msg: - print(msg) - print("for help use --help") - sys.exit(2) - -short_summary=0 -with_file=1 -just_errors = 0 -prec='x' -test='all' -only_numbers=0 -dir="TESTING" -for o, a in opts: - if o in ("-h", "--help"): - print(sys.argv[0]+" [-h|--help] [-d dir |--dir dir] [-s |--short] [-r |--run] [-e |--error] [-p p |--prec p] [-t test |--test test] [-n | --number]") - print(" - h is to print this message") - print(" - r is to use to run the LAPACK tests then analyse the output (.out files). By default, the script will not run all the LAPACK tests") - print(" - d [dir] is to indicate where is the LAPACK testing directory (.out files). By default, the script will use .") - print(" LEVEL OF OUTPUT") - print(" - x is to print a detailed summary") - print(" - e is to print only the error summary") - print(" - s is to print a short summary") - print(" - n is to print the numbers of failing tests (turn on summary mode)") - print(" SECLECTION OF TESTS:") - print(" - p [s/c/d/z/x] is to indicate the PRECISION to run:") - print(" s=single") - print(" d=double") - print(" sd=single/double") - print(" c=complex") - print(" z=double complex") - print(" cz=complex/double complex") - print(" x=all [DEFAULT]") - print(" - t [lin/eig/mixed/rfp/all] is to indicate which TEST FAMILY to run:") - print(" lin=Linear Equation") - print(" eig=Eigen Problems") - print(" mixed=mixed-precision") - print(" rfp=rfp format") - print(" all=all tests [DEFAULT]") - print(" EXAMPLES:") - print(" ./lapack_testing.py -n") - print(" Will return the numbers of failed tests by analyzing the LAPACK output") - print(" ./lapack_testing.py -n -r -p s") - print(" Will return the numbers of failed tests in REAL precision by running the LAPACK Tests then analyzing the output") - print(" ./lapack_testing.py -n -p s -t eig ") - print(" Will return the numbers of failed tests in REAL precision by analyzing only the LAPACK output of EIGEN testings") - print("Written by Julie Langou (June 2011) ") - sys.exit(0) - else: - if o in ("-s", "--short"): - short_summary = 1 - if o in ("-r", "--run"): - with_file = 0 - if o in ("-e", "--error"): - just_errors = 1 - if o in ( '-p', '--prec' ): - prec = a - if o in ( '-d', '--dir' ): - dir = a - if o in ( '-t', '--test' ): - test = a - if o in ( '-n', '--number' ): - only_numbers = 1 - short_summary = 1 - -# process options -os.chdir(dir) -execution=1 -summary="\n\t\t\t--> LAPACK TESTING SUMMARY <--\n"; -if with_file: summary+= "\t\tProcessing LAPACK Testing output found in the "+dir+" directory\n"; -summary+="SUMMARY \tnb test run \tnumerical error \tother error \n"; -summary+="================ \t===========\t=================\t================ \n"; -nb_of_test=0 - -# Add current directory to the path for subshells of this shell -# Allows the popen to find local files in both windows and unixes -os.environ["PATH"] = os.environ["PATH"]+":." - -# Define a function to open the executable (different filenames on unix and Windows) -def run_summary_test( f, cmdline, short_summary): - nb_test_run=0 - nb_test_fail=0 - nb_test_illegal=0 - nb_test_info=0 - if (with_file): - if not os.path.exists(cmdline): - error_message=cmdline+" file not found" - r=1 - if short_summary: return [nb_test_run,nb_test_fail,nb_test_illegal,nb_test_info] - else: - pipe = open(cmdline,'r') - r=0 - else: - if os.name != 'nt': - cmdline="./" + cmdline - - outfile=cmdline.split()[4] - pipe = open(outfile,'w') - p = Popen(cmdline, shell=True, stdout=pipe) - p.wait() - pipe.close() - r=p.returncode - pipe = open(outfile,'r') - error_message=cmdline+" did not work" - - if r != 0 and not with_file: - print("---- TESTING " + cmdline.split()[0] + "... FAILED(" + error_message +") !") - for line in pipe.readlines(): - f.write(str(line)) - elif r != 0 and with_file and not short_summary: - print("---- WARNING: please check that you have the LAPACK output : "+cmdline+"!") - print("---- WARNING: with the option -r, we can run the LAPACK testing for you") - # print("---- "+error_message) - else: - for line in pipe.readlines(): - f.write(str(line)) - words_in_line=line.split() - if (line.find("run")!=-1): -# print(line) - whereisrun=words_in_line.index("run)") - nb_test_run+=int(words_in_line[whereisrun-2]) - if (line.find("out of")!=-1): - if (short_summary==0): print(line, end="") - whereisout= words_in_line.index("out") - nb_test_fail+=int(words_in_line[whereisout-1]) - if (line.find("illegal")!=-1): - if (short_summary==0):print(line, end="") - nb_test_illegal+=1 - if (line.find(" INFO")!=-1): - if (short_summary==0):print(line, end="") - nb_test_info+=1 - if (with_file==1): - pipe.close() - - f.flush(); - - return [nb_test_run,nb_test_fail,nb_test_illegal,nb_test_info] - - -# If filename cannot be opened, send output to sys.stderr -filename = "testing_results.txt" -try: - f = open(filename, 'w') -except IOError: - f = sys.stdout - -if (short_summary==0): - print(" ") - print("---------------- Testing LAPACK Routines ----------------") - print(" ") - print("-- Detailed results are stored in", filename) - -dtypes = ( -("s", "d", "c", "z"), -("REAL ", "DOUBLE PRECISION", "COMPLEX ", "COMPLEX16 "), -) - -if prec=='s': - range_prec=[0] -elif prec=='d': - range_prec=[1] -elif prec=='sd': - range_prec=[0,1] -elif prec=='c': - range_prec=[2] -elif prec=='z': - range_prec=[3] -elif prec=='cz': - range_prec=[2,3] -else: - prec='x'; - range_prec=range(4) - -if test=='lin': - range_test=[15] -elif test=='mixed': - range_test=[16] - range_prec=[1,3] -elif test=='rfp': - range_test=[17] -elif test=='eig': - range_test=range(15) -else: - range_test=range(18) - -list_results = [ -[0, 0, 0, 0, 0], -[0, 0, 0, 0, 0], -[0, 0, 0, 0, 0], -[0, 0, 0, 0, 0], -] - -for dtype in range_prec: - letter = dtypes[0][dtype] - name = dtypes[1][dtype] - - if (short_summary==0): - print(" ") - print("------------------------- %s ------------------------" % name) - print(" ") - sys.stdout.flush() - - - dtests = ( - ("nep", "sep", "svd", - letter+"ec",letter+"ed",letter+"gg", - letter+"gd",letter+"sb",letter+"sg", - letter+"bb","glm","gqr", - "gsv","csd","lse", - letter+"test", letter+dtypes[0][dtype-1]+"test",letter+"test_rfp"), - ("Nonsymmetric Eigenvalue Problem", "Symmetric Eigenvalue Problem", "Singular Value Decomposition", - "Eigen Condition","Nonsymmetric Eigenvalue","Nonsymmetric Generalized Eigenvalue Problem", - "Nonsymmetric Generalized Eigenvalue Problem driver", "Symmetric Eigenvalue Problem", "Symmetric Eigenvalue Generalized Problem", - "Banded Singular Value Decomposition routines", "Generalized Linear Regression Model routines", "Generalized QR and RQ factorization routines", - "Generalized Singular Value Decomposition routines", "CS Decomposition routines", "Constrained Linear Least Squares routines", - "Linear Equation routines", "Mixed Precision linear equation routines","RFP linear equation routines"), - (letter+"nep", letter+"sep", letter+"svd", - letter+"ec",letter+"ed",letter+"gg", - letter+"gd",letter+"sb",letter+"sg", - letter+"bb",letter+"glm",letter+"gqr", - letter+"gsv",letter+"csd",letter+"lse", - letter+"test", letter+dtypes[0][dtype-1]+"test",letter+"test_rfp"), - ) - - - for dtest in range_test: - nb_of_test=0 - # NEED TO SKIP SOME PRECISION (namely s and c) FOR PROTO MIXED PRECISION TESTING - if dtest==16 and (letter=="s" or letter=="c"): - continue - if (with_file==1): - cmdbase=dtests[2][dtest]+".out" - else: - if dtest==15: - # LIN TESTS - cmdbase="xlintst"+letter+" < "+dtests[0][dtest]+".in > "+dtests[2][dtest]+".out" - elif dtest==16: - # PROTO LIN TESTS - cmdbase="xlintst"+letter+dtypes[0][dtype-1]+" < "+dtests[0][dtest]+".in > "+dtests[2][dtest]+".out" - elif dtest==17: - # PROTO LIN TESTS - cmdbase="xlintstrf"+letter+" < "+dtests[0][dtest]+".in > "+dtests[2][dtest]+".out" - else: - # EIG TESTS - cmdbase="xeigtst"+letter+" < "+dtests[0][dtest]+".in > "+dtests[2][dtest]+".out" - if (not just_errors and not short_summary): - print("--> Testing "+name+" "+dtests[1][dtest]+" [ "+cmdbase+" ]") - # Run the process: either to read the file or run the LAPACK testing - nb_test = run_summary_test(f, cmdbase, short_summary) - list_results[0][dtype]+=nb_test[0] - list_results[1][dtype]+=nb_test[1] - list_results[2][dtype]+=nb_test[2] - list_results[3][dtype]+=nb_test[3] - got_error=nb_test[1]+nb_test[2]+nb_test[3] - - if (not short_summary): - if (nb_test[0]>0 and just_errors==0): - print("--> Tests passed: "+str(nb_test[0])) - if (nb_test[1]>0): - print("--> Tests failing to pass the threshold: "+str(nb_test[1])) - if (nb_test[2]>0): - print("--> Illegal Error: "+str(nb_test[2])) - if (nb_test[3]>0): - print("--> Info Error: "+str(nb_test[3])) - if (got_error>0 and just_errors==1): - print("ERROR IS LOCATED IN "+name+" "+dtests[1][dtest]+" [ "+cmdbase+" ]") - print("") - if (just_errors==0): - print("") -# elif (got_error>0): -# print(dtests[2][dtest]+".out \t"+str(nb_test[1])+"\t"+str(nb_test[2])+"\t"+str(nb_test[3])) - - sys.stdout.flush() - if (list_results[0][dtype] > 0 ): - percent_num_error=float(list_results[1][dtype])/float(list_results[0][dtype])*100 - percent_error=float(list_results[2][dtype]+list_results[3][dtype])/float(list_results[0][dtype])*100 - else: - percent_num_error=0 - percent_error=0 - summary+=name+"\t"+str(list_results[0][dtype])+"\t\t"+str(list_results[1][dtype])+"\t("+"%.3f" % percent_num_error+"%)\t"+str(list_results[2][dtype]+list_results[3][dtype])+"\t("+"%.3f" % percent_error+"%)\t""\n" - list_results[0][4]+=list_results[0][dtype] - list_results[1][4]+=list_results[1][dtype] - list_results[2][4]+=list_results[2][dtype] - list_results[3][4]+=list_results[3][dtype] - -if only_numbers==1: - print(str(list_results[1][4])+"\n"+str(list_results[2][4]+list_results[3][4])) -else: - print(summary) - if (list_results[0][4] > 0 ): - percent_num_error=float(list_results[1][4])/float(list_results[0][4])*100 - percent_error=float(list_results[2][4]+list_results[3][4])/float(list_results[0][4])*100 - else: - percent_num_error=0 - percent_error=0 - if (prec=='x'): - print("--> ALL PRECISIONS\t"+str(list_results[0][4])+"\t\t"+str(list_results[1][4])+"\t("+"%.3f" % percent_num_error+"%)\t"+str(list_results[2][4]+list_results[3][4])+"\t("+"%.3f" % percent_error+"%)\t""\n") - if list_results[0][4] == 0: - print("NO TESTS WERE ANALYZED, please use the -r option to run the LAPACK TESTING") - -# This may close the sys.stdout stream, so make it the last statement -f.close()
diff --git a/lapack/second_NONE.f b/lapack/second_NONE.f index 7c0d793..9725bfe 100644 --- a/lapack/second_NONE.f +++ b/lapack/second_NONE.f
@@ -35,8 +35,10 @@ *> \ingroup auxOTHERauxiliary * * ===================================================================== -*> SPDX-FileCopyrightText: The Eigen Authors -*> SPDX-License-Identifier: BSD-3-Clause +*> SPDX-FileCopyrightText: 1992-2015 The University of Tennessee and The University of Tennessee Research Foundation +*> SPDX-FileCopyrightText: 2000-2015 The University of California Berkeley +*> SPDX-FileCopyrightText: 2006-2015 The University of Colorado Denver +*> SPDX-License-Identifier: BSD-3-Clause-Open-MPI REAL FUNCTION SECOND( ) * * -- LAPACK auxiliary routine (version 3.4.0) --
diff --git a/lapack/testing_summary.cmake b/lapack/testing_summary.cmake new file mode 100644 index 0000000..14d4982 --- /dev/null +++ b/lapack/testing_summary.cmake
@@ -0,0 +1,98 @@ +# SPDX-FileCopyrightText: The Eigen Authors +# SPDX-License-Identifier: MPL-2.0 + +cmake_minimum_required(VERSION 3.18) + +if(NOT DEFINED OUTPUT_MANIFEST OR NOT EXISTS "${OUTPUT_MANIFEST}") + message(FATAL_ERROR "Missing LAPACK output manifest") +endif() +file(STRINGS "${OUTPUT_MANIFEST}" OUTPUT_FILES) +if(NOT OUTPUT_FILES) + message(FATAL_ERROR "No LAPACK test output files were specified") +endif() +if(NOT DEFINED REPORT_FILE) + set(REPORT_FILE "${CMAKE_CURRENT_BINARY_DIR}/testing_results.txt") +endif() +file(WRITE "${REPORT_FILE}" "") +set(total_run 0) +set(total_failed 0) +set(total_other 0) +message(STATUS "LAPACK TESTING SUMMARY") +foreach(output IN LISTS OUTPUT_FILES) + if(NOT EXISTS "${output}") + message(FATAL_ERROR "Missing LAPACK test output: ${output}") + endif() + file(READ "${output}" contents) + file(APPEND "${REPORT_FILE}" "${contents}") + file(STRINGS "${output}" lines) + set(run 0) + set(failed 0) + set(other 0) + set(routine_errors 0) + set(balance_errors 0) + foreach(line IN LISTS lines) + string(TOLOWER "${line}" lower) + # Failed groups give their total after "out of"; successful groups give "tests run". + if(line MATCHES "([0-9]+)[ \t]+out of[ \t]+([0-9]+)") + math(EXPR failed "${failed} + ${CMAKE_MATCH_1}") + math(EXPR run "${run} + ${CMAKE_MATCH_2}") + elseif(line MATCHES "([0-9]+)[ \t]+tests[ \t]+run\\)") + math(EXPR run "${run} + ${CMAKE_MATCH_1}") + elseif(lower MATCHES "total number of examples tested[ \t]*=[ \t]*([0-9]+)") + math(EXPR run "${run} + ${CMAKE_MATCH_1}") + math(EXPR other "${other} + ${balance_errors}") + set(balance_errors 0) + endif() + # xCHKEQ reports failing routines without an "out of" summary. + if(lower MATCHES "^[ \t]*[sdcz](ge|gb|po|pp|pb)equ[ \t]+failed test with value[ \t]+") + math(EXPR failed "${failed} + 1") + endif() + # Balance tests report example indices as well as a total error count. + # An index is only evidence of an error, not the number of errors. + if(lower MATCHES "number of examples where info is not 0[ \t]*=[ \t]*([0-9]+)") + if(CMAKE_MATCH_1 GREATER balance_errors) + set(balance_errors "${CMAKE_MATCH_1}") + endif() + elseif(lower MATCHES "example number where (.*info is not (zero|0)|ilo or ihi( is)? wrong)[ \t]*=[ \t]*([0-9]+)") + if(CMAKE_MATCH_4 GREATER 0 AND balance_errors EQUAL 0) + set(balance_errors 1) + endif() + endif() + if(lower MATCHES "(value|ratio) of largest test error[ \t]*=[ \t]*([^ \t]+)") + if(CMAKE_MATCH_2 MATCHES "inf|nan|\\*") + math(EXPR failed "${failed} + 1") + endif() + endif() + if(lower MATCHES "error code from" OR line MATCHES "(^|[ \t])INFO([ \t=]|$)") + math(EXPR routine_errors "${routine_errors} + 1") + elseif(lower MATCHES "([0-9]+)[ \t]+error messages recorded") + # ALASUM summarizes the preceding ALAERH diagnostics; count each error once. + if(CMAKE_MATCH_1 GREATER routine_errors) + set(routine_errors "${CMAKE_MATCH_1}") + endif() + math(EXPR other "${other} + ${routine_errors}") + set(routine_errors 0) + endif() + if(lower MATCHES "illegal|xerbla was called|failed the tests of the error exits" + OR lower MATCHES "^[ \t]*error in[ \t]+") + math(EXPR other "${other} + 1") + endif() + endforeach() + math(EXPR other "${other} + ${routine_errors} + ${balance_errors}") + get_filename_component(name "${output}" NAME) + message(STATUS "${name}: ${run} tests, ${failed} numerical errors, ${other} other errors") + # xCHKEC failure output contains diagnostics without a test-count summary. + if(run EQUAL 0 AND failed EQUAL 0 AND other EQUAL 0) + message(FATAL_ERROR "No LAPACK numerical tests were analyzed in: ${output}") + endif() + math(EXPR total_run "${total_run} + ${run}") + math(EXPR total_failed "${total_failed} + ${failed}") + math(EXPR total_other "${total_other} + ${other}") +endforeach() +message(STATUS "TOTAL: ${total_run} tests, ${total_failed} numerical errors, ${total_other} other errors") +if(total_failed GREATER 0 OR total_other GREATER 0) + message(FATAL_ERROR "LAPACK test output reports failures") +endif() +if(total_run EQUAL 0) + message(FATAL_ERROR "No LAPACK numerical tests were analyzed") +endif()
diff --git a/scripts/add_spdx_headers.py b/scripts/add_spdx_headers.py index 47b3d33..0bbcc53 100755 --- a/scripts/add_spdx_headers.py +++ b/scripts/add_spdx_headers.py
@@ -18,7 +18,7 @@ The license id assigned to each file is determined by `classify(path)`: - LAPACKE/BLAS bridges with Intel copyright -> BSD-3-Clause - MORSE-derived cmake/Find*.cmake modules -> BSD-3-Clause -- Reference-BLAS Fortran tests blas/testing/*.f -> BSD-3-Clause +- netlib Reference-BLAS / LAPACK files -> BSD-3-Clause-Open-MPI - LevenbergMarquardt MINPACK-derived headers -> MPL-2.0 AND LicenseRef-MINPACK - Everything else under repo -> MPL-2.0 @@ -68,11 +68,18 @@ # Header marker for the LevenbergMarquardt MINPACK-derived files. MINPACK_MARKER = b"Minpack license" -# Reference-BLAS / LAPACK Fortran (Univ. of Tennessee/Berkeley/Colorado Denver). +# netlib Reference-BLAS / LAPACK files. LAPACK's LICENSE is BSD-3-Clause plus an IP disclaimer, +# i.e. BSD-3-Clause-Open-MPI. These files carry no Copyright line, so the holders named in that +# LICENSE are tagged rather than The Eigen Authors. REFBLAS_MARKERS = ( b"Reference BLAS is a software package", b"LAPACK is a software package provided by Univ. of Tennessee", ) +NETLIB_COPYRIGHT_HOLDERS = ( + "The University of Tennessee and The University of Tennessee Research Foundation", + "The University of California Berkeley", + "The University of Colorado Denver", +) # Apache-2.0 — TensorFlow-derived BFloat16, Codeplay FindComputeCpp.cmake. APACHE_MARKER = b"Licensed under the Apache License" @@ -159,7 +166,7 @@ if MINPACK_MARKER in content: return "MPL-2.0 AND LicenseRef-MINPACK" if any(m in content for m in REFBLAS_MARKERS): - return "BSD-3-Clause" + return "BSD-3-Clause-Open-MPI" if APACHE_MARKER in content: return "Apache-2.0" return "MPL-2.0" @@ -241,7 +248,9 @@ # REUSE-IgnoreStart spdx_lines = [] if not has_copyright: - spdx_lines.append(f"{comment} SPDX-FileCopyrightText: The Eigen Authors{eol}") + is_netlib = spdx_id == "BSD-3-Clause-Open-MPI" + for holder in NETLIB_COPYRIGHT_HOLDERS if is_netlib else ("The Eigen Authors",): + spdx_lines.append(f"{comment} SPDX-FileCopyrightText: {holder}{eol}") spdx_lines.append(f"{comment} SPDX-License-Identifier: {spdx_id}{eol}") spdx_line = "".join(spdx_lines).encode() # REUSE-IgnoreEnd
diff --git a/scripts/affected_tests.py b/scripts/affected_tests.py index 1a99707..ccb6f27 100644 --- a/scripts/affected_tests.py +++ b/scripts/affected_tests.py
@@ -119,7 +119,6 @@ "ci/tidy/*", "ci/lint/*", "debug/*", - "demos/*", "doc/*", "contrib/benchmarks/*", "contrib/doc/*", @@ -470,9 +469,18 @@ def changed_files_from_git(source_dir, base_sha, head="HEAD"): - """Paths changed between ``base_sha`` and ``head``.""" + """Paths changed between merge-base(``base_sha``, ``head``) and ``head``. + + Falls back to ``base_sha`` itself when git cannot compute the merge-base, as + in CI's depth-1 checkout, which fetches only the merge request's diff base: + that commit already is the merge-base. The same fallback as + style_common.diff_added_lines.""" + merge_base = subprocess.run( + ["git", "merge-base", base_sha, head], cwd=source_dir, capture_output=True, text=True + ) + base = merge_base.stdout.strip() if merge_base.returncode == 0 else base_sha result = subprocess.run( - ["git", "diff", "--no-renames", "--name-only", "%s...%s" % (base_sha, head)], + ["git", "diff", "--no-renames", "--name-only", base, head], cwd=source_dir, capture_output=True, text=True,
diff --git a/scripts/clang_tidy_hook.py b/scripts/clang_tidy_hook.py index b4fe1de..5c75315 100644 --- a/scripts/clang_tidy_hook.py +++ b/scripts/clang_tidy_hook.py
@@ -49,7 +49,7 @@ hook_post_image_and_added, is_cxx_path, line_filter_json) # Parsing an umbrella costs about a second, so keep the hook off trees whose -# conventions these checks do not describe (bench/, demos/, doc/ examples). +# conventions these checks do not describe (bench/, doc/ examples). # unsupported/Eigen/ is kept for the legacy umbrella shims that still live # there; the implementation tree itself is contrib/, which has no shims and # so needs the src-tree rules.
diff --git a/scripts/test_affected_tests.py b/scripts/test_affected_tests.py index a6d302f..8a77391 100644 --- a/scripts/test_affected_tests.py +++ b/scripts/test_affected_tests.py
@@ -451,6 +451,58 @@ shutil.rmtree(root, ignore_errors=True) +def test_git_shallow_base(): + """The diff works in a depth-1 clone that fetched only the base, as CI does.""" + root = tempfile.mkdtemp(prefix="eigen-affected-shallow-") + try: + source = os.path.join(root, "source") + os.makedirs(source) + git = ["git", "-c", "user.name=Eigen Tests", "-c", "user.email=eigen@example.com"] + subprocess.run(["git", "init", "-q"], cwd=source, check=True) + for name in ("Base.h", "Middle.h", "Head.h"): + with open(os.path.join(source, name), "w") as handle: + handle.write("// test\n") + subprocess.run(["git", "add", name], cwd=source, check=True) + subprocess.run(git + ["commit", "-qm", name], cwd=source, check=True) + base = subprocess.run( + ["git", "rev-parse", "HEAD~2"], cwd=source, check=True, capture_output=True, text=True + ).stdout.strip() + clone = os.path.join(root, "clone") + subprocess.run(["git", "clone", "-q", "--depth=1", "file://" + source, clone], check=True) + # Fetching an unadvertised SHA over file:// needs protocol v2, whatever the user's git config says. + subprocess.run(["git", "-c", "protocol.version=2", "fetch", "-q", "--depth=1", "origin", base], cwd=clone, + check=True) + changed = changed_files_from_git(clone, base) + check(changed == ["Head.h", "Middle.h"], + "a shallow clone with only the base diffs against it, got %s" % changed) + finally: + shutil.rmtree(root, ignore_errors=True) + + +def test_git_merge_base(): + """With history, the diff is three-dot: a commit on the base branch alone does not count.""" + root = tempfile.mkdtemp(prefix="eigen-affected-merge-base-") + try: + git = ["git", "-c", "user.name=Eigen Tests", "-c", "user.email=eigen@example.com"] + + def commit(name): + with open(os.path.join(root, name), "w") as handle: + handle.write("// test\n") + subprocess.run(["git", "add", name], cwd=root, check=True) + subprocess.run(git + ["commit", "-qm", name], cwd=root, check=True) + + subprocess.run(["git", "init", "-q", "-b", "target"], cwd=root, check=True) + commit("Base.h") + subprocess.run(["git", "checkout", "-qb", "topic"], cwd=root, check=True) + commit("Topic.h") + subprocess.run(["git", "checkout", "-q", "target"], cwd=root, check=True) + commit("Target.h") + changed = changed_files_from_git(root, "target", "topic") + check(changed == ["Topic.h"], "changes on the base branch alone are not selected, got %s" % changed) + finally: + shutil.rmtree(root, ignore_errors=True) + + def test_real_tree(): """Properties that must hold against the checked-out tree.""" root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) @@ -572,6 +624,8 @@ test_foreach_registrations() test_output_encoding() test_git_rename_paths() + test_git_shallow_base() + test_git_merge_base() test_real_tree() if FAILURES:
diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 323dd1c..4f3e561 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt
@@ -177,28 +177,29 @@ endif() -find_package(PASTIX QUIET COMPONENTS METIS SEQ) -# check that the PASTIX found is a version without MPI -find_path(PASTIX_pastix_nompi.h_INCLUDE_DIRS - NAMES pastix_nompi.h - HINTS ${PASTIX_INCLUDE_DIRS} -) -if (NOT PASTIX_pastix_nompi.h_INCLUDE_DIRS) - message(STATUS "A version of Pastix has been found but pastix_nompi.h does not exist in the include directory." - " Because Eigen tests require a version without MPI, we disable the Pastix backend.") +find_package(PASTIX QUIET) +set(PASTIX_NO_MPI FALSE) +if(PASTIX_FOUND) + include(CMakePushCheckState) + cmake_push_check_state(RESET) + set(CMAKE_REQUIRED_INCLUDES ${PASTIX_INCLUDE_DIRS_DEP}) + unset(EIGEN_PASTIX_NO_MPI CACHE) + check_cxx_source_compiles(" + #include <pastix/config.h> + #ifdef PASTIX_WITH_MPI + #error Eigen tests require PaStiX without MPI + #endif + int main() { return 0; } + " EIGEN_PASTIX_NO_MPI) + set(PASTIX_NO_MPI ${EIGEN_PASTIX_NO_MPI}) + cmake_pop_check_state() endif() -if(PASTIX_FOUND AND PASTIX_pastix_nompi.h_INCLUDE_DIRS) +if(PASTIX_FOUND AND NOT PASTIX_NO_MPI) + message(STATUS "Eigen tests require PaStiX compiled without MPI; disabling the PaStiX backend.") +endif() +if(PASTIX_FOUND AND PASTIX_NO_MPI) add_definitions("-DEIGEN_PASTIX_SUPPORT") include_directories(${PASTIX_INCLUDE_DIRS_DEP}) - if(SCOTCH_FOUND) - include_directories(${SCOTCH_INCLUDE_DIRS}) - set(PASTIX_LIBRARIES ${PASTIX_LIBRARIES} ${SCOTCH_LIBRARIES}) - elseif(METIS_FOUND) - include_directories(${METIS_INCLUDE_DIRS}) - set(PASTIX_LIBRARIES ${PASTIX_LIBRARIES} ${METIS_LIBRARIES}) - else() - ei_add_property(EIGEN_MISSING_BACKENDS "PaStiX, ") - endif() set(SPARSE_LIBS ${SPARSE_LIBS} ${PASTIX_LIBRARIES_DEP} ${ORDERING_LIBRARIES}) set(PASTIX_ALL_LIBS ${PASTIX_LIBRARIES_DEP}) ei_add_property(EIGEN_TESTED_BACKENDS "PaStiX, ") @@ -206,6 +207,7 @@ ei_add_property(EIGEN_MISSING_BACKENDS "PaStiX, ") endif() +find_package(METIS QUIET) if(METIS_FOUND) add_definitions("-DEIGEN_METIS_SUPPORT") include_directories(${METIS_INCLUDE_DIRS}) @@ -236,17 +238,6 @@ ei_add_property(EIGEN_MISSING_BACKENDS "Accelerate, ") endif() -option(EIGEN_TEST_NOQT "Disable Qt support in unit tests" OFF) -if(NOT EIGEN_TEST_NOQT) - find_package(Qt4) - if(QT4_FOUND) - include(${QT_USE_FILE}) - ei_add_property(EIGEN_TESTED_BACKENDS "Qt4 support, ") - else() - ei_add_property(EIGEN_MISSING_BACKENDS "Qt4 support, ") - endif() -endif() - if(TEST_LIB) add_definitions("-DEIGEN_EXTERN_INSTANTIATIONS=1") endif() @@ -307,6 +298,7 @@ ei_add_test(type_alias) ei_add_test(nullary) ei_add_test(mixingtypes) +ei_add_test(scalar_binary_op_overloads) ei_add_test(float_conversion) ei_add_test(io) ei_add_test(packetmath "-DEIGEN_FAST_MATH=1") @@ -367,6 +359,7 @@ endif() ei_add_test(redux) ei_add_test(redux_bounded) +ei_add_test(redux_unrolling) if(CMAKE_COMPILER_IS_GNUCXX) # Check optimized bounds diagnostics: scalar with assertions, vectorized without. add_executable(redux_bounded_compile EXCLUDE_FROM_ALL redux_bounded_compile.cpp) @@ -396,7 +389,17 @@ ei_add_test(no_automatic_resizing) ei_add_test(conservative_resize) ei_add_test(product_small) +ei_add_test(gemv_degenerate) ei_add_test(product_large) +ei_add_test(product_cast) +ei_add_test(gemm_double) +ei_add_test(gemm_packing) +if(CMAKE_SYSTEM_NAME STREQUAL "Linux" AND CMAKE_CXX_COMPILER_ID MATCHES "^(GNU|Clang)$") + target_compile_definitions(gemm_packing PRIVATE EIGEN_TEST_WRAP_MALLOC) + target_link_libraries(gemm_packing "-Wl,--wrap=malloc") +endif() +ei_add_test(blas_singleton_stride) +ei_add_test(product_assign) ei_add_test(product_threshold) if(EIGEN_TEST_SME) # EIGEN_TEST_SME defines EIGEN_ARM64_USE_SME (root CMakeLists.txt), which turns @@ -414,6 +417,7 @@ endif() ei_add_test(product_extra) ei_add_test(product_evaluators) +ei_add_test(product_storage_order) ei_add_test(diagonalmatrices) ei_add_test(diagonalview) ei_add_test(skew_symmetric_matrix3) @@ -442,6 +446,7 @@ ei_add_test(trsm_packet_cancellation) ei_add_test(triangular_in_place) ei_add_test(selfadjoint) +ei_add_test(selfadjoint_permutation) ei_add_test(product_selfadjoint) ei_add_test(product_symm) ei_add_test(product_syrk) @@ -458,6 +463,7 @@ ei_add_test(bandmatrix) ei_add_test(cholesky) ei_add_test(bunchkaufman) +ei_add_test(cholesky_reconstruction) ei_add_test(condition_estimator) ei_add_test(lu) ei_add_test(determinant) @@ -541,8 +547,14 @@ ei_add_test(rvalue_types) ei_add_test(dense_storage) ei_add_test(ctorleak) +ei_add_test(memory_reallocation) +if(CMAKE_SYSTEM_NAME STREQUAL "Linux" AND CMAKE_CXX_COMPILER_ID MATCHES "^(GNU|Clang)$") + target_compile_definitions(memory_reallocation PRIVATE EIGEN_TEST_WRAP_MALLOC) + target_link_libraries(memory_reallocation "-Wl,--wrap=malloc" "-Wl,--wrap=realloc") +endif() ei_add_test(inplace_decomposition) ei_add_test(inplace_decomposition_lapacke) +ei_add_test(svd_lapacke) ei_add_test(half_float) ei_add_test(bfloat16_float) ei_add_test(bfloat16_classification) @@ -609,16 +621,14 @@ ei_add_test(householder_fastmath "${EIGEN_FASTMATH_FLAGS}") ei_add_test(trsm_packet_cancellation_fastmath "${EIGEN_FASTMATH_FLAGS}") ei_add_test(trig_sign_fastmath "${EIGEN_FASTMATH_FLAGS}") - -if(QT4_FOUND) - ei_add_test(qtvector "" "${QT_QTCORE_LIBRARY}") -endif() +ei_add_test(trig_reduction_fastmath "${EIGEN_FASTMATH_FLAGS}") +ei_add_test(optimization_barrier "${EIGEN_FASTMATH_FLAGS}") if(PARDISO_FOUND) ei_add_test(pardiso_support "" "${PARDISO_ALL_LIBS}") endif() -if(PASTIX_FOUND AND (SCOTCH_FOUND OR METIS_FOUND)) +if(PASTIX_FOUND AND PASTIX_NO_MPI) ei_add_test(pastix_support "" "${PASTIX_ALL_LIBS}") endif() @@ -674,11 +684,15 @@ unset(EIGEN_ADD_TEST_FILENAME_EXTENSION) endif() +ei_add_test(sycl_macro_collision) + if(EIGEN_TEST_SYCL) set(EIGEN_SYCL ON) include(SyclConfigureTesting) ei_add_test(sycl_basic) + ei_add_test(sycl_math) + ei_add_test(sycl_include_order) set(EIGEN_SYCL OFF) endif()
diff --git a/test/NonCommutativeScalar.h b/test/NonCommutativeScalar.h index 5745553..c8e0171 100644 --- a/test/NonCommutativeScalar.h +++ b/test/NonCommutativeScalar.h
@@ -26,6 +26,7 @@ bool operator!=(const Quaternion& b) const { return !(*this == b); } }; +inline double abs(const Quaternion& a) { return Eigen::numext::sqrt(a.r * a.r + a.i * a.i + a.j * a.j + a.k * a.k); } inline Quaternion conj(const Quaternion& a) { return Quaternion(a.r, -a.i, -a.j, -a.k); } inline double real(const Quaternion& a) { return a.r; } inline double imag(const Quaternion& a) { return a.i; } @@ -40,6 +41,13 @@ static constexpr bool IsComplex = true; static constexpr bool RequireInitialization = true; }; +namespace numext { +// The generic numext::abs of a complex scalar is hypot(real, imag), which would ignore the j and k components. +template <> +inline double abs(const noncommutative_scalar::Quaternion& a) { + return noncommutative_scalar::abs(a); +} +} // namespace numext } // namespace Eigen #endif
diff --git a/test/assignment_threaded.cpp b/test/assignment_threaded.cpp index b7ffdff..b702616 100644 --- a/test/assignment_threaded.cpp +++ b/test/assignment_threaded.cpp
@@ -44,7 +44,7 @@ PlainObject dst(rows, cols), ref(rows, cols), rhs(rows, cols); rhs.setRandom(); - const auto rhs_xpr = rhs.cwiseAbs2(); + const auto rhs_xpr = rhs + rhs; // linear access dst.setRandom(); @@ -57,6 +57,15 @@ dst.device(threadPoolDevice) = rhs_xpr.unaryExpr(VectorizationOn()); VERIFY_IS_CWISE_EQUAL(ref, dst); + EIGEN_IF_CONSTEXPR (PlainObject::SizeAtCompileTime == Dynamic) { + using RealScalar = typename NumTraits<Scalar>::Real; + Matrix<RealScalar, Dynamic, 1> raw_buf(2 * rows * cols + 2); + Map<PlainObject, Unaligned> unaligned_dst(reinterpret_cast<Scalar*>(raw_buf.data() + 1), rows, cols); + unaligned_dst.setZero(); + unaligned_dst.device(threadPoolDevice) = rhs_xpr.unaryExpr(VectorizationOn()); + VERIFY_IS_CWISE_EQUAL(ref, unaligned_dst); + } + // outer-inner access Index blockRows = numext::maxi(Index(1), rows - 1); Index blockCols = numext::maxi(Index(1), cols - 1); @@ -80,6 +89,8 @@ EIGEN_DECLARE_TEST(assignment_threaded) { for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST(test_threaded_assignment(MatrixXd(), 123, 123)); + CALL_SUBTEST(test_threaded_assignment(MatrixXcf(), 123, 123)); + CALL_SUBTEST(test_threaded_assignment(MatrixXcd(), 123, 123)); CALL_SUBTEST(test_threaded_assignment(Matrix<float, 16, 16>())); } }
diff --git a/test/bdcsvd.cpp b/test/bdcsvd.cpp index f152727..15a2964 100644 --- a/test/bdcsvd.cpp +++ b/test/bdcsvd.cpp
@@ -442,6 +442,66 @@ } } +// The top-level merge of this bidiagonal has the exact poles p = 1 + eps and q = p + w, w = 9 eps, the smallest odd gap +// above the 8 eps deflation threshold. With mu = sigma - p, the secular function between them is about +// f(mu) = 1 - A w / mu + B w / (w - mu), A = z1^2 / (2 w), B = z2^2 / (2 w). +// For some of these weights the root lies between the exact midpoint p + 4.5 eps and the p + 5 eps it rounds to, and +// the interpolation falls back on bisection, whose search must therefore start from q. Rounding moves that set of +// weights between compilers, so sweep them. +template <typename RealScalar> +void bdcsvd_secular_close_poles() { + using Mat = Matrix<RealScalar, Dynamic, Dynamic>; + using Vec = Matrix<RealScalar, Dynamic, 1>; + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + const RealScalar p = RealScalar(1) + eps, w = RealScalar(9) * eps; + // Reconstruction accumulates O(n * epsilon) rounding. + const RealScalar bound = RealScalar(64) * eps; + BDCSVD<Mat, ComputeFullU | ComputeFullV> svd; + svd.setSwitchSize(3); + Vec diagonal(3), superdiagonal(2); + Mat bidiagonal = Mat::Zero(3, 3); + for (int ia = 60; ia <= 120; ++ia) { + for (int ib = 10; ib <= 50; ++ib) { + diagonal << RealScalar(0), numext::sqrt(RealScalar(2) * w * RealScalar(ia) / RealScalar(100)), p + w; + superdiagonal << p, numext::sqrt(RealScalar(2) * w * RealScalar(ib) / RealScalar(100)); + bidiagonal.diagonal() = diagonal; + bidiagonal.template diagonal<1>() = superdiagonal; + svd.compute(diagonal, superdiagonal); + VERIFY_IS_EQUAL(svd.info(), Success); + VERIFY((bidiagonal - svd.matrixU() * svd.singularValues().asDiagonal() * svd.matrixV().transpose()).norm() <= + bound * bidiagonal.norm()); + } + } +} + +// A bisection midpoint that is an exact root of the secular equation must end the search. About 1 in 20 of these small +// problems reaches one, and about 1 in 10^4 reaches one early enough that overshooting it shows in the reconstruction. +// Which ones do depends on rounding, so this relies on the number of problems, not on a fixed input. A build that +// contracts the update in secularEq into an FMA, as CI's aarch64 gcc-10 flags do, met no exact zero in 2 * 10^6 +// problems, so this test does not exercise the stop there. +template <typename RealScalar> +void bdcsvd_secular_exact_root() { + using Mat = Matrix<RealScalar, Dynamic, Dynamic>; + using Vec = Matrix<RealScalar, Dynamic, 1>; + const RealScalar bound = RealScalar(64) * NumTraits<RealScalar>::epsilon(); + BDCSVD<Mat, ComputeFullU | ComputeFullV> svd; + svd.setSwitchSize(3); + Mat bidiagonal; + Vec diagonal, superdiagonal; + for (int i = 0; i < 100000; ++i) { + const Index n = 3 + i % 6; + diagonal = Vec::Random(n); + superdiagonal = Vec::Random(n - 1); + bidiagonal.setZero(n, n); + bidiagonal.diagonal() = diagonal; + bidiagonal.template diagonal<1>() = superdiagonal; + svd.compute(diagonal, superdiagonal); + VERIFY_IS_EQUAL(svd.info(), Success); + VERIFY((bidiagonal - svd.matrixU() * svd.singularValues().asDiagonal() * svd.matrixV().transpose()).norm() <= + bound * bidiagonal.norm()); + } +} + template <typename Scalar, int StorageOrder> void bdcsvd_qr_crossover() { using RealScalar = typename NumTraits<Scalar>::Real; @@ -475,9 +535,9 @@ EIGEN_DECLARE_TEST(bdcsvd) { CALL_SUBTEST_60((svd_normal_equation_roundoff<float, ColMajor>())); - CALL_SUBTEST_60((svd_normal_equation_roundoff<double, RowMajor>())); + CALL_SUBTEST_72((svd_normal_equation_roundoff<double, RowMajor>())); CALL_SUBTEST_61((svd_normal_equation_roundoff<std::complex<float>, RowMajor>())); - CALL_SUBTEST_61((svd_normal_equation_roundoff<std::complex<double>, ColMajor>())); + CALL_SUBTEST_73((svd_normal_equation_roundoff<std::complex<double>, ColMajor>())); CALL_SUBTEST_2((bdcsvd_asserts<Matrix4d>())); CALL_SUBTEST_3((bdcsvd_asserts<Matrix<float, 10, 7>>())); @@ -532,24 +592,24 @@ TEST_SET_BUT_UNUSED_VARIABLE(smallSize); CALL_SUBTEST_62((bdcsvd_precondition_square_matrix_bidiagonal<double>(smallSize))); CALL_SUBTEST_62((bdcsvd_precondition_square_matrix_bidiagonal<float>(smallSize))); - CALL_SUBTEST_62((svd_precondition_square_matrix_accuracy<Matrix2f>(2))); - CALL_SUBTEST_62((svd_precondition_square_matrix_accuracy<Matrix4d>(4))); - CALL_SUBTEST_62((svd_precondition_square_matrix_accuracy<MatrixXf>(12))); - CALL_SUBTEST_62((svd_precondition_square_matrix_accuracy<MatrixXd>(12))); - CALL_SUBTEST_62((svd_precondition_square_matrix_accuracy<Matrix<double, Dynamic, Dynamic, RowMajor>>(12))); + CALL_SUBTEST_74((svd_precondition_square_matrix_accuracy<Matrix2f>(2))); + CALL_SUBTEST_75((svd_precondition_square_matrix_accuracy<Matrix4d>(4))); + CALL_SUBTEST_74((svd_precondition_square_matrix_accuracy<MatrixXf>(12))); + CALL_SUBTEST_75((svd_precondition_square_matrix_accuracy<MatrixXd>(12))); + CALL_SUBTEST_75((svd_precondition_square_matrix_accuracy<Matrix<double, Dynamic, Dynamic, RowMajor>>(12))); CALL_SUBTEST_63((svd_precondition_square_matrix_accuracy<Matrix2cd>(2))); - CALL_SUBTEST_63((svd_precondition_square_matrix_accuracy<Matrix4cf>(4))); - CALL_SUBTEST_63((svd_precondition_square_matrix_accuracy<MatrixXcf>(12))); + CALL_SUBTEST_76((svd_precondition_square_matrix_accuracy<Matrix4cf>(4))); + CALL_SUBTEST_76((svd_precondition_square_matrix_accuracy<MatrixXcf>(12))); CALL_SUBTEST_63((svd_precondition_square_matrix_accuracy<MatrixXcd>(12))); CALL_SUBTEST_64((svd_precondition_square_matrix_runtime_options<MatrixXf>(12))); - CALL_SUBTEST_64((svd_precondition_square_matrix_runtime_options<MatrixXcd>(internal::random<Index>(1, 30)))); + CALL_SUBTEST_77((svd_precondition_square_matrix_runtime_options<MatrixXcd>(internal::random<Index>(1, 30)))); CALL_SUBTEST_64((svd_precondition_square_matrix_runtime_options<Matrix2f>(2))); - CALL_SUBTEST_64((bdcsvd_thin_full_options<Matrix<double, Dynamic, Dynamic, RowMajor>>( + CALL_SUBTEST_78((bdcsvd_thin_full_options<Matrix<double, Dynamic, Dynamic, RowMajor>>( Matrix<double, Dynamic, Dynamic, RowMajor>(8, 8)))); // smallSvd keeps its QR preconditioner, so small non-square inputs work with DisableQRDecomposition too. - CALL_SUBTEST_64( + CALL_SUBTEST_79( (svd_thin_full_option_checks<MatrixXd, DisableQRDecomposition | PreconditionSquareMatrix>(MatrixXd(10, 5)))); - CALL_SUBTEST_64( + CALL_SUBTEST_79( (svd_thin_full_option_checks<MatrixXd, DisableQRDecomposition | PreconditionSquareMatrix>(MatrixXd(5, 10)))); } @@ -562,10 +622,10 @@ // CALL_SUBTEST_9( svd_preallocate<void>() ); CALL_SUBTEST_45(svd_underoverflow<void>()); - CALL_SUBTEST_45(bdcsvd_power_of_two_scaling<>()); - CALL_SUBTEST_45((svd_zero_matrix_solvers<Matrix3d>(3, 3))); - CALL_SUBTEST_45((svd_zero_matrix_solvers<Matrix<float, 2, 5>>(2, 5))); - CALL_SUBTEST_45((svd_zero_matrix_solvers<MatrixXcf>(5, 3))); + CALL_SUBTEST_70(bdcsvd_power_of_two_scaling<>()); + CALL_SUBTEST_71((svd_zero_matrix_solvers<Matrix3d>(3, 3))); + CALL_SUBTEST_80((svd_zero_matrix_solvers<Matrix<float, 2, 5>>(2, 5))); + CALL_SUBTEST_81((svd_zero_matrix_solvers<MatrixXcf>(5, 3))); // Without total deflation issues. CALL_SUBTEST_46((compare_bdc_jacobi_instance<>(true))); @@ -585,6 +645,10 @@ CALL_SUBTEST_54((bdcsvd_fast_math_regression_1588<>())); CALL_SUBTEST_55((bdcsvd_secular_extrapolation<float>())); CALL_SUBTEST_56((bdcsvd_secular_extrapolation<double>())); + CALL_SUBTEST_66((bdcsvd_secular_close_poles<float>())); + CALL_SUBTEST_66((bdcsvd_secular_close_poles<double>())); + CALL_SUBTEST_67((bdcsvd_secular_exact_root<float>())); + CALL_SUBTEST_67((bdcsvd_secular_exact_root<double>())); CALL_SUBTEST_57((bdcsvd_qr_crossover<float, ColMajor>())); CALL_SUBTEST_58((bdcsvd_qr_crossover<double, RowMajor>())); CALL_SUBTEST_59((bdcsvd_qr_crossover<std::complex<double>, ColMajor>()));
diff --git a/test/bfloat16_float.cpp b/test/bfloat16_float.cpp index 7e9edf1..5f96bd2 100644 --- a/test/bfloat16_float.cpp +++ b/test/bfloat16_float.cpp
@@ -12,6 +12,7 @@ #include <math.h> #include "main.h" +#include "product_test_helpers.h" #define VERIFY_BFLOAT16_BITS_EQUAL(h, bits) \ VERIFY_IS_EQUAL((numext::bit_cast<numext::uint16_t>(h)), (static_cast<numext::uint16_t>(bits))) @@ -391,7 +392,7 @@ } void test_product() { - typedef Matrix<bfloat16, Dynamic, Dynamic> MatrixXh; + using MatrixXh = Matrix<bfloat16, Dynamic, Dynamic>; Index rows = internal::random<Index>(1, EIGEN_TEST_MAX_SIZE); Index cols = internal::random<Index>(1, EIGEN_TEST_MAX_SIZE); Index depth = internal::random<Index>(1, EIGEN_TEST_MAX_SIZE); @@ -404,6 +405,26 @@ VERIFY_IS_APPROX(Ch.noalias() += Ah * Bh, (Cf.noalias() += Af * Bf).cast<bfloat16>()); } +// The blocked GEMM sums in float and rounds each coefficient once; the sizes keep it off the GEMV and coefficient-based +// paths, which still round in bfloat16. +void test_gemm_float_accumulation() { + using MatrixXh = Matrix<bfloat16, Dynamic, Dynamic>; + Index rows = internal::random<Index>(2, EIGEN_TEST_MAX_SIZE); + Index cols = internal::random<Index>(2, EIGEN_TEST_MAX_SIZE); + Index depth = internal::random<Index>(20, EIGEN_TEST_MAX_SIZE); + MatrixXh A = MatrixXh::Random(rows, depth); + MatrixXh B = MatrixXh::Random(depth, cols); + MatrixXh C0 = MatrixXh::Random(rows, cols); + // References in double, whose rounding error is negligible next to the bound. + MatrixXd exact = A.cast<double>().eval() * B.cast<double>().eval(); + MatrixXd exact_update = C0.cast<double>() + exact; + MatrixXh C = C0; + C.noalias() += A * B; + VERIFY(verifyProductUpdate<float>(C.cast<double>(), exact_update, bfloat16(1), A, B, C0, 1)); + C.noalias() = A * B; + VERIFY(verifyProduct<float>(C.cast<double>(), exact, A, B, 1)); +} + void test_nextafter() { VERIFY((numext::isnan)(numext::nextafter(std::numeric_limits<bfloat16>::quiet_NaN(), bfloat16(1.0f)))); VERIFY((numext::isnan)(numext::nextafter(bfloat16(1.0f), std::numeric_limits<bfloat16>::quiet_NaN()))); @@ -450,6 +471,7 @@ CALL_SUBTEST(test_trigonometric_functions()); CALL_SUBTEST(test_array()); CALL_SUBTEST(test_product()); + CALL_SUBTEST(test_gemm_float_accumulation()); CALL_SUBTEST(test_nextafter()); } }
diff --git a/test/bicgstabl.cpp b/test/bicgstabl.cpp index bc737af..695858d 100644 --- a/test/bicgstabl.cpp +++ b/test/bicgstabl.cpp
@@ -34,6 +34,7 @@ } // A zero right-hand side is solved by x = 0 without iterating; iterations() used to report the cap. +template <typename = void> void test_zero_rhs() { SparseMatrix<double> A(4, 4); for (Index i = 0; i < 4; ++i) A.insert(i, i) = 2.0; @@ -48,5 +49,5 @@ CALL_SUBTEST_1(test_bicgstabl_T<double>()); CALL_SUBTEST_2(test_bicgstabl_T<std::complex<double> >()); CALL_SUBTEST_3(test_bicgstabl_solve_with_guess_restart<>()); - CALL_SUBTEST_3(test_zero_rhs()); + CALL_SUBTEST_3(test_zero_rhs<>()); }
diff --git a/test/blas_singleton_stride.cpp b/test/blas_singleton_stride.cpp new file mode 100644 index 0000000..31abc55 --- /dev/null +++ b/test/blas_singleton_stride.cpp
@@ -0,0 +1,394 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include "main.h" + +std::vector<Index> blas_singleton_strides(Index count) { + std::vector<Index> strides{-1, 0, 1, count - 1, count, count + 3}; + EIGEN_IF_CONSTEXPR (std::numeric_limits<Index>::digits > std::numeric_limits<int>::digits) { + Index large_stride = (std::numeric_limits<int>::max)(); + strides.push_back(++large_stride); + } + return strides; +} + +void blas_stride_helpers() { + constexpr Index count = 24; + for (Index stride : blas_singleton_strides(count)) { + VERIFY_IS_EQUAL(internal::blas_leading_dim<ColMajor>(count, Index(1), stride), count); + VERIFY_IS_EQUAL(internal::blas_leading_dim<RowMajor>(Index(1), count, stride), count); + VERIFY_IS_EQUAL(internal::blas_vector_increment(Index(1), stride), 1); + VERIFY(internal::blas_compatible_stride<ColMajor>(count, Index(1), stride)); + VERIFY(internal::blas_compatible_stride<RowMajor>(Index(1), count, stride)); + VERIFY_IS_EQUAL(internal::blas_compatible_stride<ColMajor>(count, count, stride), stride >= count); + } + VERIFY_IS_EQUAL(internal::blas_leading_dim<ColMajor>(Index(0), Index(1), Index(0)), 1); + VERIFY_IS_EQUAL(internal::blas_leading_dim<RowMajor>(Index(1), Index(0), Index(0)), 1); + VERIFY_IS_EQUAL(internal::blas_leading_dim<ColMajor>(count, count, count + 3), count + 3); + VERIFY_IS_EQUAL(internal::blas_vector_increment(count, Index(-2)), -2); +} + +template <typename Scalar> +Scalar blas_stride_value(Index i) { + return Scalar(i + 1); +} +template <> +std::complex<float> blas_stride_value<std::complex<float>>(Index i) { + return {float(i + 1), float(i % 3 - 1)}; +} +template <> +std::complex<double> blas_stride_value<std::complex<double>>(Index i) { + return {double(i + 1), double(i % 3 - 1)}; +} + +template <typename Scalar, int Order> +void blas_singleton_stride() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + using VectorType = Matrix<Scalar, Dynamic, 1>; + using RealScalar = typename NumTraits<Scalar>::Real; + constexpr Index count = 24; + const Index rows = Order == ColMajor ? count : 1; + const Index cols = Order == ColMajor ? 1 : count; + Scalar storage[count]; + for (Index i = 0; i < count; ++i) storage[i] = Scalar(i + 1); + + for (Index stride : blas_singleton_strides(count)) { + // With one outer vector, even a zero or negative outer stride is unused. + Map<const MatrixType, 0, OuterStride<>> matrix(storage, rows, cols, OuterStride<>(stride)); + const Scalar alpha = internal::random<Scalar>(); + const VectorType rhs = VectorType::Constant(cols, Scalar(2)); + VectorType expected(rows); + for (Index i = 0; i < rows; ++i) { + Scalar value = Scalar(0); + for (Index j = 0; j < cols; ++j) value += matrix(i, j) * rhs(j); + expected(i) = value; + } + VectorType result = matrix * rhs; + VERIFY_IS_CWISE_EQUAL(result, expected); + EIGEN_IF_CONSTEXPR (Order == RowMajor) { + Scalar singleton_storage = Scalar(0); + Map<VectorType, 0, InnerStride<>> singleton_result(&singleton_storage, 1, InnerStride<>(stride)); + singleton_result.noalias() = matrix * rhs; + VERIFY_IS_EQUAL(singleton_storage, expected(0)); + } + result.setOnes(); + result.noalias() += alpha * matrix * rhs; + const RealScalar bound = RealScalar(8 * count) * NumTraits<RealScalar>::epsilon(); + VERIFY((result - (VectorType::Ones(rows) + alpha * expected)).norm() <= bound * (RealScalar(1) + expected.norm())); + + const VectorType transpose_rhs = VectorType::Constant(rows, Scalar(2)); + VectorType transpose_expected(cols); + for (Index j = 0; j < cols; ++j) { + Scalar value = Scalar(0); + for (Index i = 0; i < rows; ++i) value += matrix(i, j) * transpose_rhs(i); + transpose_expected(j) = value; + } + result = matrix.transpose() * transpose_rhs; + VERIFY_IS_CWISE_EQUAL(result, transpose_expected); + } +} + +template <typename Scalar, int Order> +void blas_singleton_outer_product() { + using ColMatrix = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + using RowMatrix = Matrix<Scalar, Dynamic, Dynamic, RowMajor>; + using DstMatrix = Matrix<Scalar, Dynamic, Dynamic, Order>; + constexpr Index count = 24; + Scalar lhs_storage[count], rhs_storage[count]; + for (Index i = 0; i < count; ++i) { + lhs_storage[i] = Scalar(i + 1); + rhs_storage[i] = Scalar(2 * i + 1); + } + DstMatrix expected(count, count), result(count, count); + for (Index j = 0; j < count; ++j) + for (Index i = 0; i < count; ++i) expected(i, j) = lhs_storage[i] * rhs_storage[j]; + for (Index stride : blas_singleton_strides(count)) { + Map<const ColMatrix, 0, OuterStride<>> lhs(lhs_storage, count, 1, OuterStride<>(stride)); + Map<const RowMatrix, 0, OuterStride<>> rhs(rhs_storage, 1, count, OuterStride<>(stride)); + result.noalias() = lhs * rhs; + VERIFY_IS_CWISE_EQUAL(result, expected); + result.setOnes(); + result.noalias() += Scalar(2) * lhs * rhs; + VERIFY_IS_CWISE_EQUAL(result, (DstMatrix::Ones(count, count) + Scalar(2) * expected).eval()); + + Scalar scalar_storage = Scalar(2); + Map<const RowMatrix, 0, OuterStride<>> scalar(&scalar_storage, 1, 1, OuterStride<>(stride)); + result.noalias() = lhs * scalar; + for (Index i = 0; i < count; ++i) VERIFY_IS_EQUAL(result(i, 0), Scalar(2) * lhs_storage[i]); + } +} + +#ifdef EIGEN_USE_BLAS +// Call the adapters directly: public products redirect singleton outputs through dot products or GEMV. +template <typename Scalar, int Order> +void blas_direct_singleton_gemv() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + using VectorType = Matrix<Scalar, Dynamic, 1>; + constexpr Index count = 24; + const Index rows = Order == ColMajor ? count : 1; + const Index cols = Order == ColMajor ? 1 : count; + MatrixType lhs(rows, cols); + VectorType rhs(cols); + for (Index i = 0; i < lhs.size(); ++i) lhs.data()[i] = blas_stride_value<Scalar>(i); + for (Index i = 0; i < cols; ++i) rhs(i) = blas_stride_value<Scalar>(i % 3); + for (Index stride : blas_singleton_strides(count)) { + VectorType result = VectorType::Ones(rows), expected = result; + for (Index i = 0; i < rows; ++i) + for (Index j = 0; j < cols; ++j) expected(i) += Scalar(2) * lhs(i, j) * rhs(j); + internal::general_matrix_vector_product_gemv<Index, Scalar, Order, false, Scalar, false>::run( + rows, cols, lhs.data(), stride, rhs.data(), cols == 1 ? stride : Index(1), result.data(), + rows == 1 ? stride : Index(1), Scalar(2)); + VERIFY_IS_CWISE_EQUAL(result, expected); + } + using RealScalar = typename NumTraits<Scalar>::Real; + RealScalar nan = NumTraits<RealScalar>::quiet_NaN(); + EIGEN_FAST_MATH_CONSTANT_BARRIER(nan); + lhs.setConstant(Scalar(nan)); + for (Index stride : {Index(0), lhs.outerStride()}) { + VectorType result = VectorType::Ones(rows); + internal::general_matrix_vector_product_gemv<Index, Scalar, Order, false, Scalar, false>::run( + rows, cols, lhs.data(), stride, rhs.data(), Index(1), result.data(), Index(1), Scalar(0)); + VERIFY_IS_CWISE_EQUAL(result, VectorType::Ones(rows)); + } +} + +template <typename Scalar, int LhsOrder, int RhsOrder> +void blas_direct_singleton_gemm() { + using Lhs = Matrix<Scalar, Dynamic, Dynamic, LhsOrder>; + using Rhs = Matrix<Scalar, Dynamic, Dynamic, RhsOrder>; + using Dst = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + const Index shapes[][3] = {{24, 1, 24}, {1, 24, 24}, {24, 24, 1}, {1, 1, 24}}; + for (const auto& shape : shapes) { + const Index rows = shape[0], cols = shape[1], depth = shape[2]; + Lhs lhs(rows, depth); + Rhs rhs(depth, cols); + for (Index i = 0; i < lhs.size(); ++i) lhs.data()[i] = blas_stride_value<Scalar>(i % 3); + for (Index i = 0; i < rhs.size(); ++i) rhs.data()[i] = blas_stride_value<Scalar>(i % 5); + Dst expected = Dst::Ones(rows, cols); + for (Index j = 0; j < cols; ++j) + for (Index i = 0; i < rows; ++i) + for (Index k = 0; k < depth; ++k) expected(i, j) += Scalar(2) * lhs(i, k) * rhs(k, j); + for (Index stride : blas_singleton_strides(Index(24))) { + Dst result = Dst::Ones(rows, cols); + const Index lhs_stride = lhs.outerSize() == 1 ? stride : lhs.outerStride(); + const Index rhs_stride = rhs.outerSize() == 1 ? stride : rhs.outerStride(); + internal::gemm_blocking_space<ColMajor, Scalar, Scalar, Dynamic, Dynamic, Dynamic> blocking(rows, cols, depth, 1, + false); + internal::general_matrix_matrix_product<Index, Scalar, LhsOrder, false, Scalar, RhsOrder, false, ColMajor, + 1>::run(rows, cols, depth, lhs.data(), lhs_stride, rhs.data(), rhs_stride, + result.data(), Index(1), + cols == 1 ? stride : result.outerStride(), Scalar(2), blocking, + nullptr); + VERIFY_IS_CWISE_EQUAL(result, expected); + } + using RealScalar = typename NumTraits<Scalar>::Real; + RealScalar nan = NumTraits<RealScalar>::quiet_NaN(); + EIGEN_FAST_MATH_CONSTANT_BARRIER(nan); + lhs.setConstant(Scalar(nan)); + for (Index stride : {Index(0), lhs.outerStride()}) { + Dst result = Dst::Ones(rows, cols); + internal::gemm_blocking_space<ColMajor, Scalar, Scalar, Dynamic, Dynamic, Dynamic> blocking(rows, cols, depth, 1, + false); + internal::general_matrix_matrix_product<Index, Scalar, LhsOrder, false, Scalar, RhsOrder, false, ColMajor, + 1>::run(rows, cols, depth, lhs.data(), stride, rhs.data(), + rhs.outerStride(), result.data(), Index(1), result.outerStride(), + Scalar(0), blocking, nullptr); + VERIFY_IS_CWISE_EQUAL(result, Dst::Ones(rows, cols)); + } + } +} +template <typename Scalar, int Order> +void blas_direct_trimmed_trmv() { + using Vector = Matrix<Scalar, Dynamic, 1>; + constexpr Index count = 24; + constexpr int mode = Order == ColMajor ? Lower : Upper; + const Index rows = Order == ColMajor ? 1 : count; + const Index cols = Order == ColMajor ? count : 1; + // Only (0,0) belongs to the triangular part; the other coefficients are implicit zeros. + const Scalar coefficient = blas_stride_value<Scalar>(0); + const Vector rhs = Vector::Ones(cols); + Vector expected = Vector::Ones(rows); + expected(0) += Scalar(2) * coefficient; + for (Index stride : blas_singleton_strides(count)) { + Vector result = Vector::Ones(rows); + internal::triangular_matrix_vector_product_trmv<Index, mode, Scalar, false, Scalar, false, Order>::run( + rows, cols, &coefficient, stride, rhs.data(), Index(1), result.data(), Index(1), Scalar(2)); + VERIFY_IS_CWISE_EQUAL(result, expected); + } +} + +template <typename Scalar, int Order, int Mode, bool Conjugate> +void blas_direct_strided_trmv() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + using Vector = Matrix<Scalar, Dynamic, 1>; + const Index shapes[][2] = {{3, 3}, {3, 5}, {5, 3}, {19, 21}, {21, 19}}; + for (const auto& shape : shapes) { + const Index rows = shape[0], cols = shape[1]; + const Index inner = Order == ColMajor ? rows : cols; + const Index outer = Order == ColMajor ? cols : rows; + for (Index stride : {Index(-2), Index(0), Index(1), inner + 2}) { + Vector storage((outer - 1) * numext::abs(stride) + inner); + for (Index i = 0; i < storage.size(); ++i) storage(i) = blas_stride_value<Scalar>(i % 5); + const Scalar* data = storage.data() + (stride < 0 ? (outer - 1) * -stride : 0); + Map<const MatrixType, 0, OuterStride<>> matrix(data, rows, cols, OuterStride<>(stride)); + for (Index rhs_incr : {Index(-2), Index(0), Index(1), Index(2)}) { + Vector rhs_storage((cols - 1) * numext::abs(rhs_incr) + 1); + for (Index i = 0; i < rhs_storage.size(); ++i) rhs_storage(i) = blas_stride_value<Scalar>(i % 3); + const Scalar* rhs = rhs_storage.data() + (rhs_incr < 0 ? rhs_storage.size() - 1 : 0); + for (Index res_incr : {Index(-2), Index(0), Index(1), Index(2)}) { + Vector result = Vector::Constant((rows - 1) * numext::abs(res_incr) + 1, Scalar(-17)); + Vector expected = result; + const Index offset = res_incr < 0 ? result.size() - 1 : 0; + for (Index i = 0; i < rows; ++i) { + Scalar sum = Scalar(0); + for (Index j = 0; j < cols; ++j) { + if ((Mode & Lower) ? i < j : i > j) continue; + const Scalar value = i == j && (Mode & (UnitDiag | ZeroDiag)) ? Scalar((Mode & UnitDiag) ? 1 : 0) + : Conjugate ? numext::conj(matrix(i, j)) + : matrix(i, j); + sum += value * (Conjugate ? numext::conj(rhs[j * rhs_incr]) : rhs[j * rhs_incr]); + } + expected(offset + i * res_incr) += Scalar(2) * sum; + } + internal::triangular_matrix_vector_product_trmv<Index, Mode, Scalar, Conjugate, Scalar, Conjugate, + Order>::run(rows, cols, data, stride, rhs, rhs_incr, + result.data() + offset, res_incr, Scalar(2)); + // Compare the entire backing store, including gaps and implicit-zero output rows. + VERIFY_IS_CWISE_EQUAL(result, expected); + } + } + } + } +} + +#endif + +template <typename Scalar, int Order> +void blas_structured_singleton_stride() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + using ColMatrix = Matrix<Scalar, Dynamic, Dynamic>; + constexpr Index count = 24; + const ColMatrix diagonal = Scalar(2) * ColMatrix::Identity(count, count); + MatrixType storage(Order == ColMajor ? count : 1, Order == ColMajor ? 1 : count); + for (Index i = 0; i < count; ++i) storage.data()[i] = blas_stride_value<Scalar>(i); + for (Index stride : blas_singleton_strides(count)) { + Map<const MatrixType, 0, OuterStride<>> input(storage.data(), storage.rows(), storage.cols(), + OuterStride<>(stride)); + MatrixType output_storage = storage; + Map<MatrixType, 0, OuterStride<>> output(output_storage.data(), storage.rows(), storage.cols(), + OuterStride<>(stride)); + EIGEN_IF_CONSTEXPR (Order == ColMajor) { + diagonal.template triangularView<Lower>().solveInPlace(output); + VERIFY_IS_CWISE_EQUAL(output, (storage * Scalar(0.5)).eval()); + ColMatrix result = diagonal.template selfadjointView<Lower>() * input; + VERIFY_IS_CWISE_EQUAL(result, (storage * Scalar(2)).eval()); + result = ColMatrix::Zero(count, count); + result.template selfadjointView<Lower>().rankUpdate(input); + for (Index j = 0; j < count; ++j) + for (Index i = j; i < count; ++i) VERIFY_IS_EQUAL(result(i, j), storage(i, 0) * numext::conj(storage(j, 0))); + } else { + diagonal.template triangularView<Lower>().template solveInPlace<OnTheRight>(output); + VERIFY_IS_CWISE_EQUAL(output, (storage * Scalar(0.5)).eval()); + ColMatrix result = input * diagonal.template selfadjointView<Lower>(); + VERIFY_IS_CWISE_EQUAL(result, (storage * Scalar(2)).eval()); + } + } +} + +template <typename Scalar, int Order> +void blas_overlapping_input_stride() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + using VectorType = Matrix<Scalar, Dynamic, 1>; + constexpr Index count = 24; + VectorType storage(2 * count * count); + for (Index i = 0; i < storage.size(); ++i) storage(i) = blas_stride_value<Scalar>(i % 5); + const MatrixType rhs = MatrixType::Ones(count, count); + for (Index stride : {-25, -1, 0, 1, 23}) { + const Scalar* data = storage.data() + (stride < 0 ? (count - 1) * -stride : 0); + Map<const MatrixType, 0, OuterStride<>> lhs(data, count, count, OuterStride<>(stride)); + VectorType expected(count); + for (Index i = 0; i < count; ++i) { + expected(i) = Scalar(0); + for (Index j = 0; j < count; ++j) expected(i) += lhs(i, j); + } + VectorType result = lhs * VectorType::Ones(count); + VERIFY_IS_CWISE_EQUAL(result, expected); + MatrixType product = lhs * rhs; + for (Index j = 0; j < count; ++j) VERIFY_IS_CWISE_EQUAL(product.col(j), expected); + + VectorType hermitian_rows = VectorType::Zero(count), hermitian_cols = VectorType::Zero(count); + VectorType lower_rows = VectorType::Zero(count), lower_cols = VectorType::Zero(count); + for (Index j = 0; j < count; ++j) + for (Index i = 0; i < count; ++i) { + const Scalar value = i == j ? Scalar(numext::real(lhs(i, i))) : i > j ? lhs(i, j) : numext::conj(lhs(j, i)); + hermitian_rows(i) += value; + hermitian_cols(j) += value; + if (i >= j) { + lower_rows(i) += lhs(i, j); + lower_cols(j) += lhs(i, j); + } + } + result = lhs.template selfadjointView<Lower>() * VectorType::Ones(count); + VERIFY_IS_CWISE_EQUAL(result, hermitian_rows); + product = lhs.template selfadjointView<Lower>() * rhs; + for (Index j = 0; j < count; ++j) VERIFY_IS_CWISE_EQUAL(product.col(j), hermitian_rows); + product = rhs * lhs.template selfadjointView<Lower>(); + for (Index i = 0; i < count; ++i) VERIFY_IS_CWISE_EQUAL(product.row(i).transpose(), hermitian_cols); + result = lhs.template triangularView<Lower>() * VectorType::Ones(count); + VERIFY_IS_CWISE_EQUAL(result, lower_rows); + product = lhs.template triangularView<Lower>() * rhs; + for (Index j = 0; j < count; ++j) VERIFY_IS_CWISE_EQUAL(product.col(j), lower_rows); + product = rhs * lhs.template triangularView<Lower>(); + for (Index i = 0; i < count; ++i) VERIFY_IS_CWISE_EQUAL(product.row(i).transpose(), lower_cols); + product.setZero(); + product.template selfadjointView<Lower>().rankUpdate(lhs); + for (Index j = 0; j < count; ++j) + for (Index i = j; i < count; ++i) { + Scalar value = Scalar(0); + for (Index k = 0; k < count; ++k) value += lhs(i, k) * numext::conj(lhs(j, k)); + VERIFY_IS_EQUAL(product(i, j), value); + } + + MatrixType solved = MatrixType::Ones(4, 2); + for (Index j = 0; j < solved.cols(); ++j) + for (Index i = 0; i < solved.rows(); ++i) { + solved(i, j) = Scalar(0); + for (Index k = 0; k <= i; ++k) solved(i, j) += lhs(i, k); + } + lhs.topLeftCorner(4, 4).template triangularView<Lower>().solveInPlace(solved); + VERIFY_IS_APPROX(solved, MatrixType::Ones(4, 2)); + } + Scalar two = Scalar(2); + Map<const VectorType, 0, InnerStride<>> broadcast(&two, count, InnerStride<>(0)); + const VectorType result = rhs * broadcast; + VERIFY_IS_CWISE_EQUAL(result, VectorType::Constant(count, Scalar(2 * count))); +} + +template <typename Scalar, int Order> +void blas_stride_tests() { + blas_singleton_stride<Scalar, Order>(); + blas_singleton_outer_product<Scalar, Order>(); + blas_structured_singleton_stride<Scalar, Order>(); + blas_overlapping_input_stride<Scalar, Order>(); +#ifdef EIGEN_USE_BLAS + blas_direct_singleton_gemv<Scalar, Order>(); + blas_direct_trimmed_trmv<Scalar, Order>(); + blas_direct_strided_trmv<Scalar, Order, Lower, false>(); + blas_direct_strided_trmv<Scalar, Order, Upper | UnitDiag, true>(); + blas_direct_strided_trmv<Scalar, Order, Lower | ZeroDiag, false>(); + blas_direct_singleton_gemm<Scalar, Order, ColMajor>(); + blas_direct_singleton_gemm<Scalar, Order, RowMajor>(); +#endif +} + +EIGEN_DECLARE_TEST(blas_singleton_stride) { + blas_stride_helpers(); + blas_stride_tests<float, ColMajor>(); + blas_stride_tests<float, RowMajor>(); + blas_stride_tests<double, ColMajor>(); + blas_stride_tests<double, RowMajor>(); + blas_stride_tests<std::complex<float>, ColMajor>(); + blas_stride_tests<std::complex<float>, RowMajor>(); + blas_stride_tests<std::complex<double>, ColMajor>(); + blas_stride_tests<std::complex<double>, RowMajor>(); +}
diff --git a/test/block.cpp b/test/block.cpp index e56f799..3c6c178 100644 --- a/test/block.cpp +++ b/test/block.cpp
@@ -437,6 +437,65 @@ VERIFY_IS_CWISE_EQUAL(inner_rows_ref, inner_rows_unwound); } +template <typename Xpr, typename Reference> +void check_inner_panel_like(const Xpr& xpr, const Reference& reference) { + VERIFY_IS_EQUAL(int(internal::evaluator<Xpr>::Flags), int(internal::evaluator<Reference>::Flags)); + VERIFY_IS_EQUAL(int(internal::evaluator<Xpr>::Alignment), int(internal::evaluator<Reference>::Alignment)); + VERIFY_IS_EQUAL(xpr.eval(), reference.eval()); + VERIFY_IS_EQUAL(xpr.sum(), reference.sum()); +} + +// A fixed-size block spanning the whole fixed inner size of its parent is an inner +// panel even when it is not built with InnerPanel = true. +void inner_panel_inference_test() { + using ColMajor8 = Matrix<float, 8, 8, ColMajor>; + using RowMajor8 = Matrix<float, 8, 8, RowMajor>; + ColMajor8 a = ColMajor8::Random(); + RowMajor8 b = RowMajor8::Random(); + Index j = internal::random<Index>(0, 4); + VERIFY(internal::evaluator<decltype(a.block<8, 4>(0, j))>::Flags & LinearAccessBit); + check_inner_panel_like(a.block<8, 4>(0, j), a.middleCols<4>(j)); + check_inner_panel_like(b.block<4, 8>(j, 0), b.middleRows<4>(j)); + check_inner_panel_like((a + a).block<8, 4>(0, j), (a + a).middleCols<4>(j)); + // The inferred alignment follows the parent's outer stride: 48-byte columns keep 16-byte + // alignment, 40-byte columns do not. + alignas(16) float buffer[8 * 12] = {}; + Map<ColMajor8, Aligned16, OuterStride<12>> stride12(buffer); + Map<ColMajor8, Aligned16, OuterStride<10>> stride10(buffer); + stride12.setRandom(); + check_inner_panel_like(stride12.block<8, 3>(0, j), stride12.middleCols<3>(j)); + check_inner_panel_like(stride10.block<8, 3>(0, j), stride10.middleCols<3>(j)); + // Not spanning the inner size, or spanning the outer one, is not an inner panel. + VERIFY(!(internal::evaluator<decltype(a.block<7, 4>(0, 0))>::Flags & LinearAccessBit)); + VERIFY(!(internal::evaluator<decltype(a.block<4, 8>(0, 0))>::Flags & LinearAccessBit)); +} + +// An empty block has a null data pointer whichever constructor built it: its start can +// lie past the end of the parent's storage. +void empty_block_data_test() { + float buffer[16] = {}; + Map<MatrixXf, 0, OuterStride<>> no_rows(buffer, 0, 3, OuterStride<>(100)); + VERIFY(no_rows.col(2).data() == nullptr); + VERIFY(no_rows.block(0, 2, 0, 1).data() == nullptr); + Map<Matrix<float, Dynamic, Dynamic, RowMajor>, 0, OuterStride<>> no_cols(buffer, 3, 0, OuterStride<>(100)); + VERIFY(no_cols.row(2).data() == nullptr); + VERIFY(no_cols.block(2, 0, 1, 0).data() == nullptr); +} + +// A product evaluates into a temporary with direct access, but the product +// expression itself has none, so its blocks forward packets to that temporary. +void product_block_packet_access_test() { + using Matrix8 = Matrix<float, 8, 8>; + Matrix8 a = Matrix8::Random(), b = Matrix8::Random(); + Matrix8 ab = a * b; + Index j = internal::random<Index>(0, 4); + VERIFY_IS_EQUAL(int(internal::evaluator<decltype((a * b).block<8, 4>(0, j))>::Flags & PacketAccessBit), + int(internal::evaluator<Matrix8>::Flags & PacketAccessBit)); + VERIFY(!(internal::evaluator<decltype((a * b).block<8, 4>(0, j))>::Flags & DirectAccessBit)); + VERIFY_IS_APPROX((Matrix<float, 8, 4>((a * b).block<8, 4>(0, j))), (ab.block<8, 4>(0, j))); + VERIFY_IS_APPROX(MatrixXf((a * b).block(1, j, 6, 3)), MatrixXf(ab.block(1, j, 6, 3))); +} + EIGEN_DECLARE_TEST(block) { for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST_1(block(Matrix<float, 1, 1>())); @@ -453,6 +512,9 @@ CALL_SUBTEST_9(unwind_test(MatrixXf())); CALL_SUBTEST_9(unwind_inner_panel_test(MatrixXf())); CALL_SUBTEST_9(unwind_inner_panel_test(Matrix<float, Dynamic, Dynamic, RowMajor>())); + CALL_SUBTEST_12(inner_panel_inference_test()); + CALL_SUBTEST_12(empty_block_data_test()); + CALL_SUBTEST_12(product_block_packet_access_test()); #ifndef EIGEN_DEFAULT_TO_ROW_MAJOR CALL_SUBTEST_6(data_and_stride(MatrixXf(internal::random(5, 50), internal::random(5, 50))));
diff --git a/test/block_sparse_matrix.cpp b/test/block_sparse_matrix.cpp index 9bd76bb..b9bfcbb 100644 --- a/test/block_sparse_matrix.cpp +++ b/test/block_sparse_matrix.cpp
@@ -300,6 +300,7 @@ // Non-square block product: A(BR x BC) * B(BC x BC2) -> C(BR x BC2) // --------------------------------------------------------------------------- +template <typename = void> void test_nonsquare_block_product() { using Scalar = double; using StorageIndex = int; @@ -673,6 +674,7 @@ // --------------------------------------------------------------------------- // In-place division of an integer matrix must divide, not multiply by the integer 1/s. +template <typename = void> void test_block_sparse_integer_division() { using BSM = BlockSparseMatrix<int, ColMajor, 2, 2>; std::vector<BSM::TripletType> trips; @@ -716,6 +718,7 @@ } // Custom scalars keep the squared comparison, so isApprox() must not widen them to std::complex. +template <typename = void> void test_block_sparse_isapprox_custom_complex() { using Scalar = CustomComplex<double>; using BSM = BlockSparseMatrix<Scalar, ColMajor, 2, 2>; @@ -733,6 +736,7 @@ VERIFY(!sparse.isApprox(-sparse)); } +template <typename = void> void test_block_triplet_traits() { // Flat scalar array means BlockTriplet should be trivially copyable and // standard-layout for any trivially-copyable Scalar and StorageIndex. @@ -783,7 +787,7 @@ CALL_SUBTEST_10((test_block_sparse_product<3, RowMajor>(3, 4, 5))); // Non-square block product - CALL_SUBTEST_11(test_nonsquare_block_product()); + CALL_SUBTEST_11(test_nonsquare_block_product<>()); // Block-sparse * dense and dense * block-sparse (ColMajor) CALL_SUBTEST_12((test_block_sparse_dense_product<2, 2, ColMajor>(4, 5))); @@ -812,17 +816,17 @@ CALL_SUBTEST_16((test_block_sparse_selfadjoint<2, RowMajor>(5))); // BlockTriplet type traits - CALL_SUBTEST_17(test_block_triplet_traits()); + CALL_SUBTEST_17(test_block_triplet_traits<>()); CALL_SUBTEST_17((test_block_sparse_isapprox_extreme_scales<float, ColMajor>())); CALL_SUBTEST_17((test_block_sparse_isapprox_extreme_scales<double, RowMajor>())); CALL_SUBTEST_17((test_block_sparse_isapprox_extreme_scales<std::complex<double>, ColMajor>())); - CALL_SUBTEST_17(test_block_sparse_isapprox_custom_complex()); + CALL_SUBTEST_17(test_block_sparse_isapprox_custom_complex<>()); // Complex scalar coverage: conjugation paths in adjoint, selfadjoint, triangular products CALL_SUBTEST_18((test_block_sparse<2, 2, ColMajor, std::complex<double>>(4, 6))); CALL_SUBTEST_18((test_block_sparse<3, 3, ColMajor, std::complex<double>>(4, 5))); CALL_SUBTEST_18((test_block_sparse<2, 2, RowMajor, std::complex<double>>(4, 6))); - CALL_SUBTEST_18(test_block_sparse_integer_division()); + CALL_SUBTEST_18(test_block_sparse_integer_division<>()); CALL_SUBTEST_18((test_block_sparse_dense_product<2, 2, ColMajor, std::complex<double>>(4, 5))); CALL_SUBTEST_18((test_block_sparse_dense_product<2, 2, RowMajor, std::complex<double>>(4, 5))); CALL_SUBTEST_18((test_block_sparse_transpose<2, 2, ColMajor, std::complex<double>>(4, 5)));
diff --git a/test/buildsystem/CMakeLists.txt b/test/buildsystem/CMakeLists.txt index f2197f5..df33dce 100644 --- a/test/buildsystem/CMakeLists.txt +++ b/test/buildsystem/CMakeLists.txt
@@ -20,7 +20,16 @@ return() endif() +add_test(NAME buildsystem_compiler_version + COMMAND ${CMAKE_COMMAND} -P "${CMAKE_CURRENT_SOURCE_DIR}/compiler_version.cmake") +set_tests_properties(buildsystem_compiler_version PROPERTIES LABELS "buildsystem" TIMEOUT 30) + +add_test(NAME buildsystem_build_name + COMMAND ${CMAKE_COMMAND} -P "${CMAKE_CURRENT_SOURCE_DIR}/build_name.cmake") +set_tests_properties(buildsystem_build_name PROPERTIES LABELS "buildsystem" TIMEOUT 30) + set(EIGEN_BUILDSYSTEM_SCENARIOS + lapack_summary gpu_architectures gpu_compile_options gpu_flags
diff --git a/test/buildsystem/build_name.cmake b/test/buildsystem/build_name.cmake new file mode 100644 index 0000000..ff37e43 --- /dev/null +++ b/test/buildsystem/build_name.cmake
@@ -0,0 +1,133 @@ +# SPDX-FileCopyrightText: The Eigen Authors +# SPDX-License-Identifier: MPL-2.0 + +cmake_minimum_required(VERSION 3.18) +include("${CMAKE_CURRENT_LIST_DIR}/../../cmake/EigenTesting.cmake") + +set(CMAKE_SYSTEM Linux-test) +set(CMAKE_CXX_COMPILER_ID GNU) +set(CMAKE_CXX_COMPILER_VERSION 13.3.0) +set(CMAKE_CXX_COMPILER_FRONTEND_VARIANT GNU) + +function(assert_equal actual expected) + if(NOT actual STREQUAL expected) + message(FATAL_ERROR "Expected ${expected}, got ${actual}") + endif() +endfunction() + +function(check_flags expected) + foreach(feature IN LISTS ARGN) + set(EIGEN_TEST_${feature} ON) + endforeach() + ei_get_cxxflags(actual) + assert_equal("${actual}" "${expected}") +endfunction() + +# A 64-bit target does not imply SSE2. Host metadata must not affect the label. +set(CMAKE_SIZEOF_VOID_P 8) +set(CMAKE_HOST_SYSTEM_PROCESSOR AMD64) +set(CMAKE_SYSTEM_PROCESSOR aarch64) +check_flags(default) +set(CMAKE_SYSTEM_PROCESSOR mips64) +check_flags(MSA MSA) +set(CMAKE_SIZEOF_VOID_P 4) +check_flags(default) +foreach(feature IN ITEMS NEON NEON64 ZVECTOR Z14 VSX ALTIVEC MSA LSX SME + SSE2 SSE3 SSSE3 SSE4_1 SSE4_2 AVX AVX2 AVX512 AVX512DQ AVX512FP16 FMA F16C) + check_flags("${feature}" "${feature}") +endforeach() +check_flags(AVX-AVX2-AVX512-FMA AVX AVX2 AVX512 FMA) +check_flags(AVX512DQ-AVX512FP16-F16C AVX512FP16 AVX512DQ F16C) +check_flags(NOVEC NO_EXPLICIT_VECTORIZATION) +check_flags(AVX-NOVEC AVX NO_EXPLICIT_VECTORIZATION) +check_flags(AVX2-NOVEC AVX2 NO_EXPLICIT_VECTORIZATION) +check_flags(F16C-NOVEC F16C NO_EXPLICIT_VECTORIZATION) +check_flags(AVX-FMA-F16C-NOVEC AVX FMA F16C NO_EXPLICIT_VECTORIZATION) +check_flags(default-OMP OPENMP) +set(EIGEN_TEST_SVE_VL 512) +check_flags(SVE512 SVE) +set(EIGEN_DEFAULT_TO_ROW_MAJOR ON) +check_flags(default-ROWMAJ) +check_flags(AVX2-OMP-ROWMAJ AVX2 OPENMP) +unset(EIGEN_DEFAULT_TO_ROW_MAJOR) + +function(check_arch processor compiler_arch generator_platform expected) + set(CMAKE_SYSTEM_PROCESSOR "${processor}") + set(CMAKE_CXX_COMPILER_ARCHITECTURE_ID "${compiler_arch}") + set(CMAKE_GENERATOR_PLATFORM "${generator_platform}") + ei_get_target_architecture(actual) + assert_equal("${actual}" "${expected}") +endfunction() + +check_arch(aarch64 "" "" aarch64) +check_arch(AMD64 "" "" x86_64) +check_arch(i686 "" "" x86) +check_arch(ppc64le "" "" ppc64le) +check_arch(s390x "" "" s390x) +check_arch(riscv64 "" "" riscv64) +check_arch(loongarch64 "" "" loongarch64) +check_arch(AMD64 ARM64 "" aarch64) +check_arch(AMD64 "" "ARM64,version=10.0" aarch64) +check_arch(AMD64 "" ",version=10.0" x86_64) +check_arch(AMD64 "" Win32 x86) +check_arch(AMD64 ARM64EC ARM64 arm64ec) +check_arch("" "" "" unknown) + +function(check_build expected) + ei_set_build_string() + assert_equal("${BUILDNAME}" "linux-test-${expected}") +endfunction() + +function(check_target target) + set(CMAKE_CXX_COMPILER_TARGET "${target}") + check_arch(x86_64 "" "" "${target}") + set(CMAKE_SYSTEM_PROCESSOR x86_64) + set(CMAKE_SIZEOF_VOID_P 8) + set(CMAKE_CXX_COMPILER_ID Clang) + set(CMAKE_CXX_COMPILER_VERSION 18.1.3) + check_build("${target}-clang-18.1.3-default-64bit") +endfunction() + +foreach(target IN ITEMS aarch64-linux-gnu arm-linux-gnueabihf riscv64-linux-gnu + gcc_ntoaarch64le "8.3.0,gcc_ntoaarch64le") + check_target("${target}") +endforeach() + +function(check_apple_arch processor osx_arch expected) + set(APPLE TRUE) + set(CMAKE_OSX_ARCHITECTURES "${osx_arch}") + check_arch("${processor}" x86_64 "" "${expected}") +endfunction() + +check_apple_arch(x86_64 arm64 aarch64) +check_apple_arch(arm64 x86_64 x86_64) +check_apple_arch(x86_64 "x86_64;arm64" aarch64+x86_64) +check_apple_arch(arm64 "arm64;x86_64" aarch64+x86_64) +check_apple_arch(x86_64 "arm64;arm64" aarch64) +check_apple_arch(x86_64 arm64e arm64e) +check_apple_arch(x86_64 "" x86_64) + +function(check_width bytes expected) + set(CMAKE_SIZEOF_VOID_P "${bytes}") + set(CMAKE_SYSTEM_PROCESSOR aarch64) + set(EIGEN_TEST_EXTERNAL_BLAS ON) + set(EIGEN_BUILD_STRING_SUFFIX probe) + check_build("aarch64-gnu-13.3.0-default-external_blas-${expected}-probe") +endfunction() + +check_width(4 32bit) +check_width(8 64bit) +check_width("" unknown-bits) + +function(check_32bit gnu expected_arch expected_bits) + set(CMAKE_COMPILER_IS_GNUCXX "${gnu}") + set(EIGEN_TEST_32BIT ON) + set(CMAKE_SYSTEM_PROCESSOR x86_64) + set(CMAKE_SIZEOF_VOID_P 8) + ei_get_target_architecture(actual) + assert_equal("${actual}" "${expected_arch}") + check_build("${expected_arch}-gnu-13.3.0-default-${expected_bits}bit") +endfunction() + +check_32bit(TRUE x86 32) +check_32bit(FALSE x86_64 64)
diff --git a/test/buildsystem/compiler_version.cmake b/test/buildsystem/compiler_version.cmake new file mode 100644 index 0000000..d872809 --- /dev/null +++ b/test/buildsystem/compiler_version.cmake
@@ -0,0 +1,45 @@ +# SPDX-FileCopyrightText: The Eigen Authors +# SPDX-License-Identifier: MPL-2.0 + +cmake_minimum_required(VERSION 3.18) +include("${CMAKE_CURRENT_LIST_DIR}/../../cmake/EigenTesting.cmake") + +function(check_compiler id version frontend expected) + set(CMAKE_CXX_COMPILER_ID "${id}") + set(CMAKE_CXX_COMPILER_VERSION "${version}") + set(CMAKE_CXX_COMPILER_FRONTEND_VARIANT "${frontend}") + # No driver invocation, including for missing metadata. + set(CMAKE_CXX_COMPILER "${CMAKE_CURRENT_LIST_DIR}/compiler-must-not-be-invoked") + ei_get_compilerver(actual) + if(NOT actual STREQUAL expected) + message(FATAL_ERROR "Expected ${expected}, got ${actual}") + endif() + if(NOT CMAKE_CXX_COMPILER_ID STREQUAL id OR NOT CMAKE_CXX_COMPILER_VERSION STREQUAL version) + message(FATAL_ERROR "Compiler metadata was modified") + endif() +endfunction() + +check_compiler(GNU 13.3.0 GNU gnu-13.3.0) +check_compiler(Clang 18.1.3 GNU clang-18.1.3) +check_compiler(Clang 18.1.3 MSVC clang-msvc-18.1.3) +check_compiler(AppleClang 15.0.0 GNU appleclang-15.0.0) +check_compiler(MSVC 19.40.33811 MSVC msvc-19.40.33811) +check_compiler(Intel 2021.10.0 "" intel-2021.10.0) +check_compiler(IntelLLVM 2025.0.0 GNU intelllvm-2025.0.0) +check_compiler(IntelLLVM 2025.0.0 MSVC intelllvm-msvc-2025.0.0) +check_compiler(PGI 19.10 "" pgi-19.10) +check_compiler(NVHPC 25.1 "" nvhpc-25.1) +foreach(version IN ITEMS 5.4.0 8.3.0 12.2.0) + check_compiler(QCC "${version}" "" "qcc-${version}") +endforeach() +check_compiler(CustomCompiler 1.2 "" customcompiler-1.2) +check_compiler(GNU "" "" gnu-unknown) +check_compiler("" "" "" unknown-unknown) +check_compiler("" 1.2 "" unknown-1.2) + +unset(CMAKE_CXX_COMPILER_ID) +unset(CMAKE_CXX_COMPILER_VERSION) +ei_get_compilerver(actual) +if(NOT actual STREQUAL "unknown-unknown" OR DEFINED CMAKE_CXX_COMPILER_ID) + message(FATAL_ERROR "Unset metadata was not preserved") +endif()
diff --git a/test/buildsystem/consumers/lapack_summary/CMakeLists.txt b/test/buildsystem/consumers/lapack_summary/CMakeLists.txt new file mode 100644 index 0000000..4e61395 --- /dev/null +++ b/test/buildsystem/consumers/lapack_summary/CMakeLists.txt
@@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: The Eigen Authors +# SPDX-License-Identifier: MPL-2.0 + +cmake_minimum_required(VERSION 3.18) +project(LapackSummary CXX) +enable_testing() +if(POLICY CMP0148) + cmake_policy(SET CMP0148 NEW) +endif() + +# Exercise LAPACK's real registration without a Fortran compiler or downloads. +include(CheckLanguage) +macro(check_language language) + set(CMAKE_Fortran_COMPILER "unused-fortran") +endmacro() +macro(enable_language language) +endmacro() +set(EIGEN_BUILD_LAPACK ON) +set(EIGEN_BUILD_BLAS ON) +set(EIGEN_ENABLE_LAPACK_TESTS ON) +set(EIGEN_INSTALL OFF) +add_library(eigen_blas_static INTERFACE) +add_library(Eigen3::Eigen INTERFACE IMPORTED) +add_custom_target(buildtests) +foreach(precision s d) + add_executable(xlintst${precision} IMPORTED) + set_target_properties(xlintst${precision} PROPERTIES IMPORTED_LOCATION "${CMAKE_COMMAND}") +endforeach() + +set(fixture "${CMAKE_CURRENT_BINARY_DIR}/lapack") +set(reference "${CMAKE_CURRENT_BINARY_DIR}/lapack-reference") +file(MAKE_DIRECTORY "${reference}/SRC" "${reference}/TESTING" "${reference}/INSTALL") +foreach(precision s d) + file(WRITE "${reference}/INSTALL/${precision}roundup_lwork.f" "") +endforeach() +set(FETCHCONTENT_SOURCE_DIR_EIGEN_LAPACK_REFERENCE "${reference}") +file(GLOB sources "${EIGEN_SOURCE_DIR}/lapack/*.cpp") +file(COPY ${sources} "${EIGEN_SOURCE_DIR}/lapack/CMakeLists.txt" + "${EIGEN_SOURCE_DIR}/lapack/testing_summary.cmake" DESTINATION "${fixture}") +foreach(directory MATGEN LIN EIG) + file(MAKE_DIRECTORY "${reference}/TESTING/${directory}") + file(WRITE "${reference}/TESTING/${directory}/CMakeLists.txt" "") +endforeach() +file(WRITE "${reference}/TESTING/stest.in" "All tests passed ( 100 tests run)\n") +file(WRITE "${reference}/TESTING/dtest.in" "All tests passed ( 20 tests run)\n") +# LAPACK 3.4.1 TESTING/EIG/?CHKBL, ?CHKBK, ?CHKGL and ?CHKGK summary formats. +foreach(precision s d c z) + add_executable(xeigtst${precision} IMPORTED) + set_target_properties(xeigtst${precision} PROPERTIES IMPORTED_LOCATION "${CMAKE_COMMAND}") + foreach(kind bal bak gbal gbak) + if(kind MATCHES "^g") + string(TOUPPER "${precision}g${kind}" routine) + else() + string(TOUPPER "${precision}ge${kind}" routine) + endif() + set(output " .. test output of ${routine} ..\n value of largest test error = 0.000D+00\n") + if(kind STREQUAL "gbak") + string(TOUPPER "${precision}" prefix) + string(APPEND output + " example number where ${prefix}GGBAL info is not 0 = 0\n" + " example number where ${prefix}GGBAK(L) info is not 0 = 0\n" + " example number where ${prefix}GGBAK(R) info is not 0 = 0\n") + else() + string(APPEND output " example number where info is not zero = 0\n") + if(kind MATCHES "bal$") + string(APPEND output " example number where ILO or IHI wrong = 0\n") + endif() + endif() + string(APPEND output + " example number having largest error = 0\n" + " number of examples where info is not 0 = 0\n" + " total number of examples tested = 13\n") + file(WRITE "${reference}/TESTING/${precision}${kind}.in" "${output}") + endforeach() +endforeach() +file(WRITE "${reference}/TESTING/runtest.cmake" + "configure_file(\"\${INPUT}\" \"\${OUTPUT}\" COPYONLY)\n") + +# Fail if configuration attempts to discover any Python interpreter. +file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/modules") +foreach(module Python PythonInterp Python2 Python3) + file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/modules/Find${module}.cmake" + "message(FATAL_ERROR \"LAPACK tests must not require Python\")\n") +endforeach() +list(PREPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_BINARY_DIR}/modules") +add_subdirectory("${fixture}" lapack-build)
diff --git a/test/buildsystem/consumers/subproject/CMakeLists.txt b/test/buildsystem/consumers/subproject/CMakeLists.txt index 10df8d6..2731ef7 100644 --- a/test/buildsystem/consumers/subproject/CMakeLists.txt +++ b/test/buildsystem/consumers/subproject/CMakeLists.txt
@@ -21,7 +21,6 @@ set(EIGEN_BUILD_TESTING OFF CACHE BOOL "" FORCE) set(EIGEN_BUILD_DOC OFF CACHE BOOL "" FORCE) -set(EIGEN_BUILD_DEMOS OFF CACHE BOOL "" FORCE) set(BUILD_TESTING OFF CACHE BOOL "" FORCE) if(EIGEN_USE_EXCLUDE_FROM_ALL)
diff --git a/test/buildsystem/run_scenario.cmake b/test/buildsystem/run_scenario.cmake index feb98c1..57d0710 100644 --- a/test/buildsystem/run_scenario.cmake +++ b/test/buildsystem/run_scenario.cmake
@@ -27,7 +27,6 @@ set(BS_EIGEN_QUIET_OPTIONS -DEIGEN_BUILD_TESTING=OFF -DEIGEN_BUILD_DOC=OFF - -DEIGEN_BUILD_DEMOS=OFF -DEIGEN_BUILD_BLAS=OFF -DEIGEN_BUILD_LAPACK=OFF )
diff --git a/test/buildsystem/scenarios/install_off_all_options.cmake b/test/buildsystem/scenarios/install_off_all_options.cmake index 6f9fb39..353163c 100644 --- a/test/buildsystem/scenarios/install_off_all_options.cmake +++ b/test/buildsystem/scenarios/install_off_all_options.cmake
@@ -16,7 +16,6 @@ "-DCMAKE_INSTALL_PREFIX=${BS_PREFIX}" -DEIGEN_BUILD_TESTING=OFF -DEIGEN_BUILD_DOC=OFF - -DEIGEN_BUILD_DEMOS=OFF -DEIGEN_BUILD_BLAS=ON -DEIGEN_BUILD_LAPACK=ON -DEIGEN_BUILD_CMAKE_PACKAGE=ON @@ -45,7 +44,6 @@ "-DCMAKE_INSTALL_PREFIX=${BS_PREFIX}" -DEIGEN_BUILD_TESTING=OFF -DEIGEN_BUILD_DOC=ON - -DEIGEN_BUILD_DEMOS=OFF -DEIGEN_BUILD_BLAS=OFF -DEIGEN_BUILD_LAPACK=OFF -DEIGEN_INSTALL=OFF)
diff --git a/test/buildsystem/scenarios/lapack_summary.cmake b/test/buildsystem/scenarios/lapack_summary.cmake new file mode 100644 index 0000000..1ab43b5 --- /dev/null +++ b/test/buildsystem/scenarios/lapack_summary.cmake
@@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: The Eigen Authors +# SPDX-License-Identifier: MPL-2.0 + +bs_configure("LAPACK summary without Python" "${BS_CONSUMER_DIR}/lapack_summary" + "${WORK_DIR}/build" "-DEIGEN_SOURCE_DIR=${EIGEN_SOURCE_DIR}") +# Selecting only the summary must generate its inputs, including under -j. +bs_run(WHAT "summary selects registered LAPACK tests without Python" + COMMAND "${CMAKE_COMMAND}" -E chdir "${WORK_DIR}/build" + "${CMAKE_CTEST_COMMAND}" -C "${BUILD_CONFIG}" --parallel 2 + -R "^LAPACK_Test_Summary$" --output-on-failure --no-tests=error) + +set(summary "${EIGEN_SOURCE_DIR}/lapack/testing_summary.cmake") +set(report "${WORK_DIR}/report.txt") +set(manifest "${WORK_DIR}/outputs.txt") +set(first "${WORK_DIR}/first output.out") +set(second "${WORK_DIR}/second.out") +file(WRITE "${first}" "Note; unrelated run information\nAll tests passed ( 100 tests run)\n") +file(WRITE "${second}" "All tests passed ( 20 tests run)\n") +file(WRITE "${WORK_DIR}/stale.out" "999 out of 999 tests failed\n") +file(WRITE "${manifest}" "${first}\n${second}\n") +bs_run(WHAT "successful LAPACK summary" OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" + "-DREPORT_FILE=${report}" -P "${summary}") +if(NOT output MATCHES "TOTAL: 120 tests, 0 numerical errors, 0 other errors") + bs_fail("incorrect successful summary: ${output}") +endif() +file(READ "${report}" contents) +bs_assert_streq("${contents}" "Note; unrelated run information\nAll tests passed ( 100 tests run)\nAll tests passed ( 20 tests run)\n" + "combined LAPACK report") + +bs_run(WHAT "registered LAPACK example counts" OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" + "-DOUTPUT_MANIFEST=${WORK_DIR}/build/lapack-build/lapack_test_outputs.txt" + "-DREPORT_FILE=${report}" -P "${summary}") +if(NOT output MATCHES "TOTAL: 328 tests, 0 numerical errors, 0 other errors") + bs_fail("incorrect registered summary: ${output}") +endif() + +file(WRITE "${manifest}" "${first}\n") +file(READ "${WORK_DIR}/build/lapack-reference/TESTING/dgbak.in" balancing) +string(REPLACE "0.000D+00" "0.125D+00" finite "${balancing}") +string(REPLACE "example number having largest error = 0" + "example number having largest error = 7" finite "${finite}") +file(WRITE "${first}" "${finite}") +bs_run(WHAT "finite balancing error maximum" OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" + "-DREPORT_FILE=${report}" -P "${summary}") +if(NOT output MATCHES "13 tests, 0 numerical errors, 0 other errors") + bs_fail("incorrect finite balancing summary: ${output}") +endif() + +foreach(kind bal bak gbal gbak) + file(READ "${WORK_DIR}/build/lapack-reference/TESTING/d${kind}.in" balancing) + string(REPLACE "number of examples where info is not 0 = 0" + "number of examples where info is not 0 = 2" failing "${balancing}") + file(WRITE "${first}" "${failing}") + bs_run(WHAT "balancing ${kind} routine errors" EXPECT_RESULT FAILURE OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" + "-DREPORT_FILE=${report}" -P "${summary}") + if(NOT output MATCHES "13 tests, 0 numerical errors, 2 other errors") + bs_fail("incorrect balancing ${kind} error count: ${output}") + endif() +endforeach() + +foreach(kind bal gbal) + file(READ "${WORK_DIR}/build/lapack-reference/TESTING/d${kind}.in" balancing) + string(REPLACE "example number where ILO or IHI wrong = 0" + "example number where ILO or IHI wrong = 7" failing "${balancing}") + string(REPLACE "number of examples where info is not 0 = 0" + "number of examples where info is not 0 = 1" failing "${failing}") + file(WRITE "${first}" "${failing}") + bs_run(WHAT "balancing ${kind} ILO/IHI error" EXPECT_RESULT FAILURE OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" + "-DREPORT_FILE=${report}" -P "${summary}") + if(NOT output MATCHES "13 tests, 0 numerical errors, 1 other errors") + bs_fail("incorrect balancing ${kind} ILO/IHI error count: ${output}") + endif() +endforeach() + +foreach(error Infinity -Infinity NaN "************") + string(REPLACE "0.000D+00" "${error}" failing "${balancing}") + file(WRITE "${first}" "${failing}") + bs_run(WHAT "balancing error ${error}" EXPECT_RESULT FAILURE OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" + "-DREPORT_FILE=${report}" -P "${summary}") + if(NOT output MATCHES "13 tests, 1 numerical errors, 0 other errors") + bs_fail("unrecognized balancing error ${error}: ${output}") + endif() +endforeach() + +file(WRITE "${manifest}" "${first}\n${second}\n") +set(balance " .. test output of SGEBAL ..\n value of largest test error = 0.197D+01\n example number where info is not zero = 0\n example number where ILO or IHI wrong = 0\n example number having largest error = 11\n number of examples where info is not 0 = 0\n total number of examples tested = 13\n") +file(WRITE "${first}" "${balance}") +bs_run(WHAT "successful balance examples with ordinary tests" OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" + "-DREPORT_FILE=${report}" -P "${summary}") +if(NOT output MATCHES "TOTAL: 33 tests, 0 numerical errors, 0 other errors") + bs_fail("incorrect balance summary: ${output}") +endif() + +file(WRITE "${manifest}" "${first}\n") +foreach(case infinity nan overflow info bounds unrecorded_info) + set(contents "${balance}") + set(expected "13 tests, 1 numerical errors, 0 other errors") + if(case STREQUAL "infinity") + string(REPLACE "0.197D+01" "Infinity" contents "${contents}") + elseif(case STREQUAL "nan") + string(REPLACE "0.197D+01" "NaN" contents "${contents}") + string(REPLACE "value of largest" "ratio of largest" contents "${contents}") + elseif(case STREQUAL "overflow") + string(REPLACE "0.197D+01" "************" contents "${contents}") + elseif(case STREQUAL "info") + string(REPLACE "info is not zero = 0" "DGGBAL info is not 0 = 7" contents "${contents}") + string(REPLACE "info is not 0 = 0" "info is not 0 = 3" contents "${contents}") + set(expected "13 tests, 0 numerical errors, 3 other errors") + elseif(case STREQUAL "bounds") + string(REPLACE "ILO or IHI wrong = 0" "ILO or IHI is wrong = 4" contents "${contents}") + string(REPLACE "info is not 0 = 0" "info is not 0 = 1" contents "${contents}") + set(expected "13 tests, 0 numerical errors, 1 other errors") + else() + string(REPLACE "info is not zero = 0" "DGGBAK(L) info is not 0 = 5" contents "${contents}") + set(expected "13 tests, 0 numerical errors, 1 other errors") + endif() + file(WRITE "${first}" "${contents}") + bs_run(WHAT "LAPACK balance summary: ${case}" EXPECT_RESULT FAILURE OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" + "-DREPORT_FILE=${report}" -P "${summary}") + if(NOT output MATCHES "${expected}") + bs_fail("wrong balance failure for ${case}: ${output}") + endif() +endforeach() + +set(ec "${WORK_DIR}/dec.out") +file(WRITE "${ec}" " Error in DLALN2: RMAX = 0.123E+03\n Error in DLANV2: RMAX = 0.456E+03\n") +foreach(mode alone mixed) + file(WRITE "${manifest}" "${ec}\n") + if(mode STREQUAL "mixed") + file(APPEND "${manifest}" "${second}\n") + endif() + bs_run(WHAT "LAPACK diagnostic-only xCHKEC output: ${mode}" EXPECT_RESULT FAILURE OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" + "-DREPORT_FILE=${report}" -P "${summary}") + if(NOT output MATCHES "dec.out: 0 tests, 0 numerical errors, 2 other errors" + OR output MATCHES "No LAPACK numerical tests were analyzed") + bs_fail("incorrect xCHKEC failure diagnostic: ${output}") + endif() +endforeach() + +foreach(precision S D C Z) + foreach(routine GEEQU GBEQU POEQU PPEQU PBEQU) + foreach(mode mixed alone) + file(WRITE "${manifest}" "${first}\n") + file(WRITE "${first}" " ${precision}${routine} failed test with value 0.100D+03 exceeding threshold 0.200D+02\n") + set(tests_run 0) + if(mode STREQUAL "mixed") + file(APPEND "${first}" " All tests for ${precision}GE routines passed the threshold ( 100 tests run)\n") + set(tests_run 100) + endif() + bs_run(WHAT "LAPACK equilibration failure: ${precision}${routine}, ${mode}" + EXPECT_RESULT FAILURE OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" + "-DREPORT_FILE=${report}" -P "${summary}") + if(NOT output MATCHES "TOTAL: ${tests_run} tests, 1 numerical errors, 0 other errors" + OR output MATCHES "No LAPACK numerical tests were analyzed") + bs_fail("incorrect equilibration failure diagnostic: ${output}") + endif() + endforeach() + endforeach() +endforeach() + +foreach(case numerical diagnostics xerbla error_exits info_summary error_code recorded combined_errors empty mixed_empty zero_examples missing) + file(WRITE "${manifest}" "${first}\n") + if(case STREQUAL "numerical") + file(WRITE "${first}" "3 out of 20 tests failed ( 20 tests run)\n") + set(expected "20 tests, 3 numerical errors, 0 other errors") + elseif(case STREQUAL "diagnostics") + file(WRITE "${first}" "100 tests run)\nillegal argument\nDGETRF returned INFO = 1\n") + set(expected "100 tests, 0 numerical errors, 2 other errors") + elseif(case STREQUAL "xerbla") + file(WRITE "${first}" "100 tests run)\n*** XERBLA was called with SRNAME = DGETRS instead of DGETRF ***\n") + set(expected "100 tests, 0 numerical errors, 1 other errors") + elseif(case STREQUAL "error_exits") + file(WRITE "${first}" "100 tests run)\n*** DGE routines failed the tests of the error exits ***\n") + set(expected "100 tests, 0 numerical errors, 1 other errors") + elseif(case STREQUAL "info_summary") + file(WRITE "${first}" "*** DGETRF returned with INFO = 2 instead of 1\nAll tests for DGE routines passed the threshold ( 100 tests run)\n1 error messages recorded\n") + set(expected "100 tests, 0 numerical errors, 1 other errors") + elseif(case STREQUAL "error_code" OR case STREQUAL "recorded" OR case STREQUAL "combined_errors") + file(WRITE "${first}" "") + if(NOT case STREQUAL "recorded") + file(APPEND "${first}" " *** Error code from DGETRF = 1 for M = 2, N = 2, type 1\n") + endif() + file(APPEND "${first}" " All tests for DGE routines passed the threshold ( 100 tests run)\n") + if(NOT case STREQUAL "error_code") + file(APPEND "${first}" " 1 error messages recorded\n") + endif() + set(expected "100 tests, 0 numerical errors, 1 other errors") + elseif(case STREQUAL "zero_examples") + string(REPLACE "total number of examples tested = 13" + "total number of examples tested = 0" empty "${balancing}") + file(WRITE "${first}" "${empty}") + set(expected "No LAPACK numerical tests were analyzed") + elseif(case STREQUAL "empty" OR case STREQUAL "mixed_empty") + file(WRITE "${first}" "") + if(case STREQUAL "mixed_empty") + file(WRITE "${manifest}" "${second}\n${first}\n") + endif() + set(expected "No LAPACK numerical tests were analyzed") + else() + file(REMOVE "${first}") + set(expected "Missing LAPACK test output") + endif() + bs_run(WHAT "LAPACK summary: ${case}" EXPECT_RESULT FAILURE OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" "-DOUTPUT_MANIFEST=${manifest}" "-DREPORT_FILE=${report}" -P "${summary}") + if(NOT output MATCHES "${expected}") + bs_fail("wrong failure for ${case}: ${output}") + endif() +endforeach() + +file(WRITE "${WORK_DIR}/build/lapack-reference/TESTING/runtest.cmake" [=[ +get_filename_component(name "${INPUT}" NAME) +if(name STREQUAL "stest.in") + message(FATAL_ERROR "simulated LAPACK driver failure") +endif() +configure_file("${INPUT}" "${OUTPUT}" COPYONLY) +]=]) +set(registered_report "${WORK_DIR}/build/lapack-build/TESTING/testing_results.txt") +file(WRITE "${registered_report}" "previous summary\n") +bs_run(WHAT "failed LAPACK fixture skips summary" EXPECT_RESULT FAILURE OUTPUT_VARIABLE output + COMMAND "${CMAKE_COMMAND}" -E chdir "${WORK_DIR}/build" + "${CMAKE_CTEST_COMMAND}" -C "${BUILD_CONFIG}" --parallel 2 + -R "^LAPACK_Test_Summary$" --output-on-failure --no-tests=error) +if(NOT output MATCHES "LAPACK_Test_Summary[^\n]*Not Run") + bs_fail("summary was not skipped after a failed driver: ${output}") +endif() +file(READ "${registered_report}" contents) +bs_assert_streq("${contents}" "previous summary\n" "skipped LAPACK summary preserves its report")
diff --git a/test/cholesky.cpp b/test/cholesky.cpp index d313dae..c4da9e9 100644 --- a/test/cholesky.cpp +++ b/test/cholesky.cpp
@@ -486,21 +486,27 @@ VERIFY_IS_NOT_APPROX(mat, ldlt.reconstructedMatrix()); VERIFY(ldlt.info() == NumericalIssue); } + + // Pivoting on the input's diagonal took a zero of the Schur complement as a pivot of each of the next three + // indefinite matrices, above a non-zero column, and failed. Pivoting on the Schur complement's diagonal passes over + // that zero: they factor, and D carries their inertia (Sylvester's law of inertia). #if (!EIGEN_ARCH_i386) || defined(EIGEN_VECTORIZE_SSE2) { mat.resize(3, 3); mat << -1, -3, 3, -3, -8.9999999999999999999, 1, 3, 1, 0; ldlt.compute(mat); - VERIFY(ldlt.info() == NumericalIssue); - VERIFY_IS_NOT_APPROX(mat, ldlt.reconstructedMatrix()); + VERIFY(ldlt.info() == Success); + VERIFY_IS_APPROX(mat, ldlt.reconstructedMatrix()); + VERIFY_IS_EQUAL((ldlt.vectorD().array() < 0).count(), Index(2)); } #endif { mat.resize(3, 3); mat << 1, 2, 3, 2, 4, 1, 3, 1, 0; ldlt.compute(mat); - VERIFY(ldlt.info() == NumericalIssue); - VERIFY_IS_NOT_APPROX(mat, ldlt.reconstructedMatrix()); + VERIFY(ldlt.info() == Success); + VERIFY_IS_APPROX(mat, ldlt.reconstructedMatrix()); + VERIFY_IS_EQUAL((ldlt.vectorD().array() < 0).count(), Index(1)); } { @@ -509,8 +515,9 @@ 0, 8.49333, 0, 2.00333, 0, 0, 0, 0, -0.1, 0, 0.1, 0, 0, 1, 0, 0, 0, 2.00333, 0, 4.24667, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0; ldlt.compute(mat); - VERIFY(ldlt.info() == NumericalIssue); - VERIFY_IS_NOT_APPROX(mat, ldlt.reconstructedMatrix()); + VERIFY(ldlt.info() == Success); + VERIFY_IS_APPROX(mat, ldlt.reconstructedMatrix()); + VERIFY_IS_EQUAL((ldlt.vectorD().array() < 0).count(), Index(2)); } // bug 1479 @@ -523,6 +530,228 @@ } } +// Semidefinite matrices of rank r whose arithmetic is exact, so that their zero pivots stay exactly zero. LDLT pivots +// on the Schur complement's diagonal, so the n - r zeros of D come last. Pivoting on the input's diagonal took the zero +// of [1 1 0; 1 1 0; 0 0 1] second, left it inside D, and reported NumericalIssue. +template <typename MatrixType, int UpLo> +void check_ldlt_zero_pivots_last(const MatrixType& A, Index r) { + using RealScalar = typename NumTraits<typename MatrixType::Scalar>::Real; + using RealVectorType = Matrix<RealScalar, Dynamic, 1>; + const Index n = A.rows(); + const MatrixType triangle = A.template triangularView<UpLo>(); + LDLT<MatrixType, UpLo> ldlt(triangle); + VERIFY(ldlt.info() == Success); + VERIFY(ldlt.isPositive()); + const RealVectorType d = ldlt.vectorD().real(); + VERIFY_IS_CWISE_EQUAL(d.head(r), RealVectorType::Ones(r)); + VERIFY_IS_CWISE_EQUAL(d.tail(n - r), RealVectorType::Zero(n - r)); + VERIFY_IS_CWISE_EQUAL(ldlt.reconstructedMatrix(), A); +} + +template <typename MatrixType> +void cholesky_ldlt_zero_pivots_last(const MatrixType& A, Index r) { + check_ldlt_zero_pivots_last<MatrixType, Lower>(A, r); + check_ldlt_zero_pivots_last<MatrixType, Upper>(A, r); +} + +// The panel width at the smallest size that LDLT factors in panels, or 0 if it factors none, as with +// EIGEN_LDLT_BLOCKSIZE=0. The width grows with the size, so the blocked tests below take their sizes from this one. +template <typename MatrixType> +Index ldlt_smallest_blocksize() { + for (Index n = 1; n <= 4096; ++n) + if (internal::ldlt_blocksize<MatrixType>(n) > 0) return internal::ldlt_blocksize<MatrixType>(n); + return 0; +} + +// P (I_m kron [1 1; 1 1]) P^T has rank m, at a size that takes the blocked factorization. +template <typename MatrixType> +void cholesky_ldlt_zero_pivots_last_blocked() { + const Index nb = ldlt_smallest_blocksize<MatrixType>(); + if (nb == 0) return; + const Index m = 2 * nb + 1, n = 2 * m; + VERIFY(internal::ldlt_blocksize<MatrixType>(n) > 0); + MatrixType A = MatrixType::Zero(n, n); + for (Index i = 0; i < m; ++i) A.template block<2, 2>(2 * i, 2 * i).setOnes(); + PermutationMatrix<Dynamic> perm(n); + perm.setIdentity(); + for (Index i = n - 1; i > 0; --i) + std::swap(perm.indices().coeffRef(i), perm.indices().coeffRef(internal::random<Index>(0, i))); + cholesky_ldlt_zero_pivots_last(MatrixType(perm * A * perm.transpose()), m); +} + +// For a positive semidefinite A of rank r, the trailing n - r pivots of complete diagonal pivoting are at rounding +// level and the leading r are not (N. J. Higham, "Analysis of the Cholesky decomposition of a semi-definite matrix", +// 1990, for LAPACK's xPSTRF). +template <typename MatrixType, int UpLo> +void check_ldlt_semidefinite(const MatrixType& A, Index r) { + using RealScalar = typename NumTraits<typename MatrixType::Scalar>::Real; + const Index n = A.rows(); + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + const MatrixType triangle = A.template triangularView<UpLo>(); + // info() is not checked: rounding can leave an exactly zero pivot above a column of rounding errors, which + // NumericalIssue reports as it would for an indefinite matrix. + LDLT<MatrixType, UpLo> ldlt(triangle); + + // Complete pivoting keeps |L| <= 1 on a semidefinite matrix, so each trailing pivot is a sum of about n rounding + // errors of size eps d_0. Over 20 seeds and n <= 300, in single and double precision, they stayed below + // 1.2 n eps d_0, and the reconstruction error below 0.25 n eps |A|. With r = 1 or n/2 random columns in B, the + // leading pivots stay far above the tail bound. + const Matrix<RealScalar, Dynamic, 1> d = ldlt.vectorD().real().cwiseAbs(); + const RealScalar tail_bound = RealScalar(8 * n) * eps * d(0); + if (r < n) VERIFY(d.tail(n - r).maxCoeff() <= tail_bound); + VERIFY(d.head(r).minCoeff() > tail_bound); + VERIFY((ldlt.reconstructedMatrix() - A).norm() <= RealScalar(2 * n) * eps * A.norm()); +} + +// The two rows of B of largest norm are parallel, so that pivoting by the input's diagonal takes a rounding-level pivot +// second. +template <typename MatrixType> +void cholesky_ldlt_semidefinite(Index n, Index r) { + using RealScalar = typename NumTraits<typename MatrixType::Scalar>::Real; + MatrixType B = MatrixType::Random(n, r); + if (n >= 2) { + B.row(0) *= RealScalar(4); + B.row(1) = RealScalar(0.5) * B.row(0); + } + const MatrixType A = B * B.adjoint(); + check_ldlt_semidefinite<MatrixType, Lower>(A, r); + check_ldlt_semidefinite<MatrixType, Upper>(A, r); +} + +// A = G M G with M = B B^* + I and G a shuffled diagonal over 2 * decades orders of magnitude. Complete diagonal +// pivoting follows the grading, so the factorization is componentwise accurate, +// |A - P^T L D L^* P|_ij <= c n eps sqrt(a_ii a_jj). Forming |L_ik|^2 before D_k |L_ik|^2 in the diagonal downdate +// underflowed for |L_ik| < sqrt(min) and lost that accuracy. +template <typename MatrixType, int UpLo> +void check_ldlt_graded(const MatrixType& A) { + using RealScalar = typename NumTraits<typename MatrixType::Scalar>::Real; + using RealVectorType = Matrix<RealScalar, Dynamic, 1>; + const Index n = A.rows(); + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + const MatrixType triangle = A.template triangularView<UpLo>(); + LDLT<MatrixType, UpLo> ldlt(triangle); + VERIFY(ldlt.info() == Success); + const RealVectorType root = A.diagonal().real().cwiseSqrt(); + const RealScalar error = + ((ldlt.reconstructedMatrix() - A).cwiseAbs().array() / (root * root.transpose()).array()).maxCoeff(); + // Measured below 0.1 n eps, against above 1e10 n eps with the underflowing downdate. + VERIFY(error <= RealScalar(2 * n) * eps); +} + +template <typename MatrixType> +void cholesky_ldlt_graded(Index n, int decades) { + using RealScalar = typename NumTraits<typename MatrixType::Scalar>::Real; + Matrix<RealScalar, Dynamic, 1> g(n); + for (Index i = 0; i < n; ++i) + g(i) = std::pow(RealScalar(10), RealScalar(decades) * (RealScalar(2 * i) / RealScalar(n - 1) - RealScalar(1))); + for (Index i = n - 1; i > 0; --i) std::swap(g(i), g(internal::random<Index>(0, i))); + const MatrixType B = MatrixType::Random(n, n); + const MatrixType A = g.asDiagonal() * (B * B.adjoint() + MatrixType::Identity(n, n)) * g.asDiagonal(); + check_ldlt_graded<MatrixType, Lower>(A); + check_ldlt_graded<MatrixType, Upper>(A); +} + +// Definite and KKT matrices of size n, which give pivots of both signs. +template <typename MatrixType> +void check_ldlt_blocked(Index n) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + using RealMatrixType = Matrix<RealScalar, Dynamic, Dynamic>; + const RealScalar eps = NumTraits<RealScalar>::epsilon(); + const MatrixType B = MatrixType::Random(n, n); + const MatrixType pd = B * B.adjoint() + MatrixType::Identity(n, n); + for (RealScalar sign : {RealScalar(1), RealScalar(-1)}) { + const MatrixType A = sign * pd; + LDLT<MatrixType> ldlt(A); + VERIFY(ldlt.info() == Success); + VERIFY(ldlt.isPositive() == (sign > 0) && ldlt.isNegative() == (sign < 0)); + VERIFY((ldlt.reconstructedMatrix() - A).norm() <= RealScalar(2 * n) * eps * A.norm()); + } + + // [H J^*; J 0] with H positive definite and J of full row rank k has k negative eigenvalues. The matrix is + // indefinite, so the backward error is bounded by |L| |D| |L^*| rather than by |A|: |dA| <= c n eps |L| |D| |L^*| + // (Higham, ASNA 2nd ed., section 11.1). Measured below 0.01 n eps |A|, with |L| |D| |L^*| <= 3 |A|. + const Index k = n / 4; + MatrixType kkt = MatrixType::Zero(n, n); + kkt.topLeftCorner(n - k, n - k) = pd.topLeftCorner(n - k, n - k); + kkt.bottomLeftCorner(k, n - k) = MatrixType::Random(k, n - k); + kkt.topRightCorner(n - k, k) = kkt.bottomLeftCorner(k, n - k).adjoint(); + LDLT<MatrixType, Upper> ldlt(kkt); + VERIFY(ldlt.info() == Success); + const RealMatrixType absL = MatrixType(ldlt.matrixL()).cwiseAbs(); + const RealScalar growth = (absL * ldlt.vectorD().real().cwiseAbs().asDiagonal() * absL.transpose()).norm(); + VERIFY((ldlt.reconstructedMatrix() - kkt).norm() <= RealScalar(2 * n) * eps * growth); + VERIFY_IS_EQUAL((ldlt.vectorD().real().array() < 0).count(), k); +} + +// Above three panel widths, LDLT factors panels and applies each to the trailing Schur complement in one level-3 +// product. At the smallest panel width, the sizes leave a trailing panel of one column, none, and a partial one; a +// fourth size is the first with a wider panel. Fixed-size and bounded matrices stay unblocked, so that they allocate +// no panel workspace. +template <typename MatrixType> +void cholesky_ldlt_blocked() { + using Scalar = typename MatrixType::Scalar; + const Index nb = ldlt_smallest_blocksize<MatrixType>(); + if (nb == 0) return; + VERIFY_IS_EQUAL(internal::ldlt_blocksize<MatrixType>(3 * nb), Index(0)); + VERIFY_IS_EQUAL((internal::ldlt_blocksize<Matrix<Scalar, Dynamic, Dynamic, 0, 512, 512> >(4 * nb + 1)), Index(0)); + for (Index n : {4 * nb + 1, 5 * nb, 5 * nb + nb / 2}) { + VERIFY_IS_EQUAL(internal::ldlt_blocksize<MatrixType>(n), nb); + check_ldlt_blocked<MatrixType>(n); + } +#ifndef EIGEN_LDLT_BLOCKSIZE + Index n = 5 * nb + nb / 2; + while (internal::ldlt_blocksize<MatrixType>(n) == nb) ++n; + check_ldlt_blocked<MatrixType>(n); +#endif +} + +// A NaN or an infinity anywhere in the matrix reaches a pivot, so the factorization reports it instead of Success. +template <typename MatrixType> +void check_ldlt_nonfinite(Index n, const typename MatrixType::Scalar& value) { + const Index positions[][2] = {{0, 0}, {n / 2, n / 2}, {n - 1, n - 1}, {n - 1, 0}, {n - 1, n - 2}}; + for (const auto& position : positions) { + const Index i = position[0], j = position[1]; + if (j < 0 || (i == j && NumTraits<typename MatrixType::Scalar>::IsComplex && numext::real(value) == 0)) continue; + MatrixType A = MatrixType::Identity(n, n); + A(i, j) = value; + A(j, i) = numext::conj(value); + VERIFY((LDLT<MatrixType, Lower>(A).info() == NumericalIssue)); + VERIFY((LDLT<MatrixType, Upper>(A).info() == NumericalIssue)); + } +} + +template <typename MatrixType> +void cholesky_ldlt_nonfinite(Index n) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + check_ldlt_nonfinite<MatrixType>(n, Scalar(std::numeric_limits<RealScalar>::quiet_NaN())); + check_ldlt_nonfinite<MatrixType>(n, Scalar(std::numeric_limits<RealScalar>::infinity())); +} + +// A NaN in the imaginary part of an off-diagonal entry; the diagonal of a Hermitian matrix has none to read. +template <typename MatrixType> +void cholesky_ldlt_imaginary_nan(Index n) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + check_ldlt_nonfinite<MatrixType>(n, Scalar(RealScalar(0), std::numeric_limits<RealScalar>::quiet_NaN())); +} + +// The blocked factorization takes its panel workspace from the decomposition, so a preallocated one allocates nothing +// at this size, where the trailing product still packs its operands on the stack. +template <typename Scalar> +void cholesky_ldlt_blocked_preallocated_no_malloc() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic>; + const Index size = 4 * numext::maxi(ldlt_smallest_blocksize<MatrixType>(), Index(32)) + 3; + const MatrixType A = MatrixType::Random(size, size); + const MatrixType spd = A * A.adjoint() + MatrixType::Identity(size, size); + LDLT<MatrixType> ldlt(size); + internal::set_is_malloc_allowed(false); + ldlt.compute(spd); + internal::set_is_malloc_allowed(true); + VERIFY_IS_EQUAL(ldlt.info(), Success); +} + // Accumulates I + sum_k w_k w_k^* out of rank-1 updates alone, which is the path rankUpdate() takes while the // decomposition holds no factorization, and checks the reported status against the factorization it produced. template <typename MatrixType, int UpLo> @@ -699,7 +928,7 @@ MatrixType q = MatrixType::Random(size, size).householderQr().householderQ(); RealVectorType d(size); - for (Index i = 0; i < size; ++i) d(i) = internal::random<RealScalar>(RealScalar(1.2), RealScalar(2.8)); + setRandomInRange(d, RealScalar(1.2), RealScalar(2.8)); const RealScalar absdet = d.prod(); const RealScalar logabsdet = d.array().log().sum(); @@ -998,6 +1227,40 @@ CALL_SUBTEST_2(cholesky_faillure_cases<void>()); + CALL_SUBTEST_3(cholesky_ldlt_zero_pivots_last((Matrix3d() << 1, 1, 0, 1, 1, 0, 0, 0, 1).finished(), 2)); + CALL_SUBTEST_2(cholesky_ldlt_zero_pivots_last(MatrixXd((Matrix3d() << 1, 1, 0, 1, 1, 0, 0, 0, 1).finished()), 2)); + CALL_SUBTEST_6(cholesky_ldlt_zero_pivots_last( + (Matrix3cd() << 1, std::complex<double>(0, 1), 0, std::complex<double>(0, -1), 1, 0, 0, 0, 1).finished(), 2)); + CALL_SUBTEST_2(cholesky_ldlt_zero_pivots_last_blocked<MatrixXd>()); + CALL_SUBTEST_6(cholesky_ldlt_zero_pivots_last_blocked<MatrixXcd>()); + + CALL_SUBTEST_2(cholesky_ldlt_blocked<MatrixXd>()); + CALL_SUBTEST_2((cholesky_ldlt_blocked<Matrix<double, Dynamic, Dynamic, RowMajor> >())); + CALL_SUBTEST_4(cholesky_ldlt_blocked<MatrixXf>()); + CALL_SUBTEST_6(cholesky_ldlt_blocked<MatrixXcd>()); + CALL_SUBTEST_1(cholesky_ldlt_blocked_preallocated_no_malloc<double>()); + CALL_SUBTEST_1(cholesky_ldlt_blocked_preallocated_no_malloc<std::complex<double> >()); + + { + // Two unblocked sizes and two blocked ones, unless EIGEN_LDLT_BLOCKSIZE=0 disables blocking. + const Index nb = numext::maxi(ldlt_smallest_blocksize<MatrixXd>(), Index(32)); + for (Index n : {Index(3), Index(17), 4 * nb + 1, 5 * nb + nb / 2}) { + CALL_SUBTEST_2(cholesky_ldlt_semidefinite<MatrixXd>(n, 1)); + CALL_SUBTEST_2(cholesky_ldlt_semidefinite<MatrixXd>(n, n / 2)); + CALL_SUBTEST_4(cholesky_ldlt_semidefinite<MatrixXf>(n, n / 2)); + CALL_SUBTEST_6(cholesky_ldlt_semidefinite<MatrixXcd>(n, n / 2)); + CALL_SUBTEST_2(cholesky_ldlt_graded<MatrixXd>(n, 80)); + CALL_SUBTEST_4(cholesky_ldlt_graded<MatrixXf>(n, 12)); + CALL_SUBTEST_6(cholesky_ldlt_graded<MatrixXcd>(n, 80)); + CALL_SUBTEST_2(cholesky_ldlt_nonfinite<MatrixXd>(n)); + CALL_SUBTEST_6(cholesky_ldlt_nonfinite<MatrixXcd>(n)); + CALL_SUBTEST_6(cholesky_ldlt_imaginary_nan<MatrixXcd>(n)); + TEST_SET_BUT_UNUSED_VARIABLE(n); + } + CALL_SUBTEST_2(cholesky_ldlt_nonfinite<MatrixXd>(1)); + CALL_SUBTEST_5(cholesky_ldlt_nonfinite<Matrix4d>(4)); + } + // Blocking and vectorization boundary tests (deterministic, outside g_repeat). CALL_SUBTEST_2(cholesky_blocking_boundary<double>()); CALL_SUBTEST_8(cholesky_blocking_boundary<float>());
diff --git a/test/cholesky_reconstruction.cpp b/test/cholesky_reconstruction.cpp new file mode 100644 index 0000000..4a2f769 --- /dev/null +++ b/test/cholesky_reconstruction.cpp
@@ -0,0 +1,78 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#define EIGEN_RUNTIME_NO_MALLOC +#include "main.h" +#include <Eigen/Cholesky> + +template <typename MatrixType, typename Factorization> +void check_cholesky_reconstruction(const MatrixType& input, const Factorization& factorization) { + using RealScalar = typename MatrixType::RealScalar; + VERIFY_IS_EQUAL(factorization.info(), Success); + const MatrixType reconstructed = factorization.reconstructedMatrix(); + // Two triangular products and the factorization each accumulate O(n) rounding errors per coefficient. + const RealScalar tolerance = + RealScalar(64) * RealScalar((std::max)(Index(1), input.rows())) * NumTraits<RealScalar>::epsilon() * input.norm(); + VERIFY(reconstructed.allFinite()); + VERIFY((reconstructed - input).norm() <= tolerance); + VERIFY_IS_EQUAL(MatrixType(reconstructed.template triangularView<StrictlyUpper>()), + MatrixType(reconstructed.adjoint().template triangularView<StrictlyUpper>())); +} + +template <typename Scalar, int Order, int UpLo> +void cholesky_reconstruction() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + for (Index size : {0, 1, 2, 5, 17, 63, 64, 65}) { + const MatrixType random = MatrixType::Random(size, size); + MatrixType input = random * random.adjoint(); + input.diagonal().array() += Scalar(1); + check_cholesky_reconstruction(input, LDLT<MatrixType, UpLo>(input)); + input = random + random.adjoint(); + check_cholesky_reconstruction(input, BunchKaufman<MatrixType, UpLo>(input)); + } +} + +template <typename Scalar, int Order, int UpLo> +void cholesky_reconstruction_fixed() { + using MatrixType = Matrix<Scalar, 4, 4, Order>; + MatrixType input = MatrixType::Zero(); + input.diagonal() << Scalar(1), Scalar(4), Scalar(2), Scalar(3); + LDLT<MatrixType, UpLo> ldlt(input); + VERIFY_IS_EQUAL(ldlt.transpositionsP().coeff(0), 1); + internal::set_is_malloc_allowed(false); + check_cholesky_reconstruction(input, ldlt); + internal::set_is_malloc_allowed(true); + + // The off-diagonal entry forces a 2x2 pivot and a non-adjacent symmetric interchange. + const Scalar offDiagonal = Scalar(2) + Scalar(0.5) * internal::random<Scalar>(); + input.setZero(); + input(1, 1) = Scalar(2); + input(2, 2) = Scalar(-3); + input(3, 0) = offDiagonal; + input(0, 3) = numext::conj(offDiagonal); + BunchKaufman<MatrixType, UpLo> bk(input); + VERIFY_IS_EQUAL(bk.transpositionsP().coeff(1), 3); + VERIFY(!numext::is_exactly_zero(bk.subDiagonal().coeff(0))); + internal::set_is_malloc_allowed(false); + check_cholesky_reconstruction(input, bk); + internal::set_is_malloc_allowed(true); +} + +EIGEN_DECLARE_TEST(cholesky_reconstruction) { + CALL_SUBTEST_1((cholesky_reconstruction<double, ColMajor, Lower>())); + CALL_SUBTEST_1((cholesky_reconstruction<double, ColMajor, Upper>())); + CALL_SUBTEST_2((cholesky_reconstruction<double, RowMajor, Lower>())); + CALL_SUBTEST_2((cholesky_reconstruction<double, RowMajor, Upper>())); + CALL_SUBTEST_3((cholesky_reconstruction<std::complex<double>, ColMajor, Lower>())); + CALL_SUBTEST_3((cholesky_reconstruction<std::complex<double>, ColMajor, Upper>())); + CALL_SUBTEST_4((cholesky_reconstruction<std::complex<double>, RowMajor, Lower>())); + CALL_SUBTEST_4((cholesky_reconstruction<std::complex<double>, RowMajor, Upper>())); + CALL_SUBTEST_5((cholesky_reconstruction<float, ColMajor, Lower>())); + CALL_SUBTEST_5((cholesky_reconstruction<std::complex<float>, RowMajor, Upper>())); + for (int i = 0; i < g_repeat; ++i) { + CALL_SUBTEST_1((cholesky_reconstruction_fixed<double, ColMajor, Lower>())); + CALL_SUBTEST_2((cholesky_reconstruction_fixed<double, RowMajor, Upper>())); + CALL_SUBTEST_3((cholesky_reconstruction_fixed<std::complex<double>, ColMajor, Upper>())); + CALL_SUBTEST_4((cholesky_reconstruction_fixed<std::complex<double>, RowMajor, Lower>())); + } +}
diff --git a/test/complex_qz.cpp b/test/complex_qz.cpp index 8018ca5..20bb66a 100644 --- a/test/complex_qz.cpp +++ b/test/complex_qz.cpp
@@ -60,17 +60,18 @@ template <typename MatrixType, typename QZType> void verify_complex_qz_convergence(const MatrixType& a, const MatrixType& b, const QZType& qz) { using RealScalar = typename MatrixType::RealScalar; - // Accumulated rounding in the unitary transformations is O(n * epsilon). + // Accumulated rounding in the unitary transformations is O(n * epsilon). stableNorm, since the squared entries of + // complex_qz_scaled's pencils underflow. const RealScalar tolerance = RealScalar(128 * a.rows()) * NumTraits<RealScalar>::epsilon(); VERIFY_IS_EQUAL(qz.info(), Success); - VERIFY((a - qz.matrixQ() * qz.matrixS() * qz.matrixZ()).norm() <= tolerance * a.norm()); - VERIFY((b - qz.matrixQ() * qz.matrixT() * qz.matrixZ()).norm() <= tolerance * b.norm()); + VERIFY((a - qz.matrixQ() * qz.matrixS() * qz.matrixZ()).stableNorm() <= tolerance * a.stableNorm()); + VERIFY((b - qz.matrixQ() * qz.matrixT() * qz.matrixZ()).stableNorm() <= tolerance * b.stableNorm()); VERIFY(qz.matrixQ().isUnitary(tolerance)); VERIFY(qz.matrixZ().isUnitary(tolerance)); const MatrixType lowerS = qz.matrixS().template triangularView<StrictlyLower>(); const MatrixType lowerT = qz.matrixT().template triangularView<StrictlyLower>(); - VERIFY(lowerS.norm() <= tolerance * a.norm()); - VERIFY(lowerT.norm() <= tolerance * b.norm()); + VERIFY(lowerS.stableNorm() <= tolerance * a.stableNorm()); + VERIFY(lowerT.stableNorm() <= tolerance * b.stableNorm()); } template <typename MatrixType> @@ -137,6 +138,131 @@ } } +// The DARE example "darex #14" of SciPy's solve_discrete_are tests, from Benner, Laub and Mehrmann, "A Collection of +// Benchmark Examples for the Numerical Solution of Algebraic Riccati Equations II: Discrete-Time Case", TU +// Chemnitz-Zwickau SPC 95_23 (1995): the balanced symplectic pencil with the column of R deflated. Eigenvalues 0 (x3, +// one Jordan chain), infinity (x3) and 1 -+ sqrt(5) 1e-8. With AVX2 and FMA, an iterate had S(1, 0) = 1.8e-17 next to +// diagonal entries 2.4e-15 and 3e-3; sweeps starting at row 0 then stalled for 400 iterations. +template <typename MatrixType> +void complex_qz_darex14() { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename MatrixType::RealScalar; + const Index m = 4, p = 2 * m + 1; + // H = [A 0 B; -Q I 0; 0 0 R], J = [I 0 0; 0 A^* 0; 0 -B^* 0] with A = (1 - 1e-8) e0 e0^T + shift down, + // B = 1e-8 e0, Q = e3 e3^T and R = 1/4. + MatrixType H = MatrixType::Zero(p, p), J = MatrixType::Zero(p, p); + H(0, 0) = J(m, m) = Scalar(RealScalar(1) - RealScalar(1e-8)); + for (Index i = 1; i < m; ++i) H(i, i - 1) = J(m + i - 1, m + i) = Scalar(1); + H(0, 2 * m) = Scalar(RealScalar(1e-8)); + H(m + 3, 3) = Scalar(-1); + H.block(m, m, m, m).setIdentity(); + H(2 * m, 2 * m) = Scalar(RealScalar(0.25)); + J.topLeftCorner(m, m).setIdentity(); + J(2 * m, m) = Scalar(RealScalar(-1e-8)); + // Symplectic balancing (Benner, SIAM J. Sci. Comput. 22(5), 2001) by diag(D, D^-1, 1/2) with + // D = diag(2^20, 2^14, 2^8, 2^2). + Matrix<RealScalar, Dynamic, 1> scale(p); + const int exponents[m] = {20, 14, 8, 2}; + for (Index i = 0; i < m; ++i) { + scale(i) = std::ldexp(RealScalar(1), exponents[i]); + scale(m + i) = std::ldexp(RealScalar(1), -exponents[i]); + } + scale(2 * m) = RealScalar(0.5); + H = scale.template cast<Scalar>().asDiagonal() * H * scale.cwiseInverse().template cast<Scalar>().asDiagonal(); + J = scale.template cast<Scalar>().asDiagonal() * J * scale.cwiseInverse().template cast<Scalar>().asDiagonal(); + const HouseholderQR<MatrixType> qr(H.rightCols(1)); + const MatrixType a = (qr.householderQ().adjoint() * H.leftCols(2 * m)).bottomRows(2 * m); + const MatrixType b = (qr.householderQ().adjoint() * J.leftCols(2 * m)).bottomRows(2 * m); + + ComplexQZ<MatrixType> qz(a, b); + verify_complex_qz_convergence(a, b, qz); +} + +// a = diag(A1, s A2), b = I with A2 = U diag(lambda) U^* normal and |lambda_k| in [1/2, 1]. Relative deflation +// decomposes the small block as if it were alone, so by Bauer-Fike its eigenvalues move by at most its backward error, +// ~128 n eps s. A normwise deflation threshold eps ||a|| would allow errors ~eps ||A1|| >> s. +template <typename MatrixType> +void complex_qz_reducible(typename MatrixType::RealScalar s) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename MatrixType::RealScalar; + using Vector = Matrix<Scalar, Dynamic, 1>; + const Index n1 = 8, n2 = 8, n = n1 + n2; + Vector lambda(n2); + for (Index k = 0; k < n2; ++k) + lambda(k) = std::polar(internal::random<RealScalar>(RealScalar(0.5), RealScalar(1)), + internal::random<RealScalar>(-RealScalar(EIGEN_PI), RealScalar(EIGEN_PI))); + const MatrixType U = HouseholderQR<MatrixType>(MatrixType::Random(n2, n2)).householderQ(); + MatrixType a = MatrixType::Zero(n, n); + a.topLeftCorner(n1, n1).setRandom(); + a.bottomRightCorner(n2, n2) = s * U * lambda.asDiagonal() * U.adjoint(); + const MatrixType b = MatrixType::Identity(n, n); + + ComplexQZ<MatrixType> qz(a, b); + verify_complex_qz_convergence(a, b, qz); + const Vector computed = qz.matrixS().diagonal().cwiseQuotient(qz.matrixT().diagonal()); + const RealScalar tolerance = RealScalar(128 * n) * NumTraits<RealScalar>::epsilon() * s; + for (Index k = 0; k < n2; ++k) VERIFY((computed.array() - s * lambda(k)).abs().minCoeff() <= tolerance); +} + +// Graded pencils leave 2x2 blocks that the closed-form reduction in reduceDiagonal2x2block does not deflate on its own. +template <typename MatrixType> +void complex_qz_graded() { + using RealScalar = typename MatrixType::RealScalar; + for (Index dim : {4, 8}) { + for (int trial = 0; trial < 50; ++trial) { + MatrixType a = MatrixType::Random(dim, dim), b = MatrixType::Random(dim, dim); + for (Index i = 0; i < dim; ++i) { + const RealScalar grade = std::pow(RealScalar(10), -RealScalar(8 * i) / RealScalar(dim)); + a.row(i) *= grade; + b.col(i) *= grade; + } + ComplexQZ<MatrixType> qz(a, b); + verify_complex_qz_convergence(a, b, qz); + } + } +} + +// a = s N for the down-shift N, b = I. At s = 1 the 2x2 blocks have the eigenvalue 0, for which a left rotation formed +// from lambda T is the identity and leaves T(i+1, i) in place. At s = sqrt(min) eps the products of the sweep-start +// test underflow to 0, and an unscaled test accepted the first row it tried (0 <= 0) and dropped fill of size s. +template <typename MatrixType> +void complex_qz_scaled() { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename MatrixType::RealScalar; + const RealScalar tiny = std::sqrt((std::numeric_limits<RealScalar>::min)()) * NumTraits<RealScalar>::epsilon(); + for (RealScalar s : {RealScalar(1), tiny}) { + for (Index dim : {4, 5, 8}) { + MatrixType a = MatrixType::Zero(dim, dim); + a.diagonal(-1).setConstant(Scalar(s)); + const MatrixType b = MatrixType::Identity(dim, dim); + ComplexQZ<MatrixType> qz(a, b); + verify_complex_qz_convergence(a, b, qz); + } + } +} + +// a = V J W^*, b = V W^* with J the 2x2 Jordan block of the eigenvalue 0. On a defective block, refinement steps in +// reduceDiagonal2x2block can lower |S(1, 0)| / (|S(0, 0)| + |S(1, 1)|) by growing the diagonal while |S(1, 0)| grows +// past eps ||S||. +template <typename MatrixType> +void complex_qz_jordan() { + using Scalar = typename MatrixType::Scalar; + MatrixType J = MatrixType::Zero(2, 2); + J(0, 1) = Scalar(1); + for (int trial = 0; trial < 100; ++trial) { + for (bool real : {true, false}) { + MatrixType V = MatrixType::Random(2, 2), W = MatrixType::Random(2, 2); + if (real) { + V = V.real().template cast<Scalar>(); + W = W.real().template cast<Scalar>(); + } + const MatrixType a = V * J * W.adjoint(), b = V * W.adjoint(); + ComplexQZ<MatrixType> qz(a, b); + verify_complex_qz_convergence(a, b, qz); + } + } +} + // Block Householder operations still allocate at larger sizes (issue #3163). template <typename MatrixType> void complex_qz_no_malloc() { @@ -181,6 +307,24 @@ CALL_SUBTEST_8((complex_qz_exceptional_shift<MatrixXcd>())); CALL_SUBTEST_7((complex_qz_small_b<MatrixXcf>())); CALL_SUBTEST_8((complex_qz_small_b<MatrixXcd>())); + CALL_SUBTEST_8((complex_qz_darex14<MatrixXcd>())); + CALL_SUBTEST_10((complex_qz_darex14<Matrix<std::complex<double>, Dynamic, Dynamic, RowMajor>>())); + CALL_SUBTEST_7((complex_qz_reducible<MatrixXcf>(1e-6f))); + CALL_SUBTEST_8((complex_qz_reducible<MatrixXcd>(1e-10))); + CALL_SUBTEST_9((complex_qz_reducible<Matrix<std::complex<float>, Dynamic, Dynamic, RowMajor>>(1e-6f))); + CALL_SUBTEST_10((complex_qz_reducible<Matrix<std::complex<double>, Dynamic, Dynamic, RowMajor>>(1e-10))); + CALL_SUBTEST_7((complex_qz_graded<MatrixXcf>())); + CALL_SUBTEST_8((complex_qz_graded<MatrixXcd>())); + CALL_SUBTEST_9((complex_qz_graded<Matrix<std::complex<float>, Dynamic, Dynamic, RowMajor>>())); + CALL_SUBTEST_10((complex_qz_graded<Matrix<std::complex<double>, Dynamic, Dynamic, RowMajor>>())); + CALL_SUBTEST_7((complex_qz_scaled<MatrixXcf>())); + CALL_SUBTEST_8((complex_qz_scaled<MatrixXcd>())); + CALL_SUBTEST_9((complex_qz_scaled<Matrix<std::complex<float>, Dynamic, Dynamic, RowMajor>>())); + CALL_SUBTEST_10((complex_qz_scaled<Matrix<std::complex<double>, Dynamic, Dynamic, RowMajor>>())); + CALL_SUBTEST_7((complex_qz_jordan<MatrixXcf>())); + CALL_SUBTEST_8((complex_qz_jordan<MatrixXcd>())); + CALL_SUBTEST_9((complex_qz_jordan<Matrix<std::complex<float>, Dynamic, Dynamic, RowMajor>>())); + CALL_SUBTEST_10((complex_qz_jordan<Matrix<std::complex<double>, Dynamic, Dynamic, RowMajor>>())); CALL_SUBTEST_9((complex_qz_exceptional_shift<Matrix<std::complex<float>, Dynamic, Dynamic, RowMajor>>())); CALL_SUBTEST_10((complex_qz_exceptional_shift<Matrix<std::complex<double>, Dynamic, Dynamic, RowMajor>>()));
diff --git a/test/conservative_resize.cpp b/test/conservative_resize.cpp index 137fa72..28c5f8a 100644 --- a/test/conservative_resize.cpp +++ b/test/conservative_resize.cpp
@@ -8,6 +8,7 @@ // with this file, You can obtain one at http://mozilla.org/MPL/2.0/. // SPDX-License-Identifier: MPL-2.0 +#define EIGEN_RUNTIME_NO_MALLOC #include "main.h" #include <Eigen/Core> @@ -15,9 +16,9 @@ using namespace Eigen; -template <typename Scalar, int Storage> +template <typename Scalar, int Storage, int MaxSize = Dynamic> void run_matrix_tests() { - typedef Matrix<Scalar, Eigen::Dynamic, Eigen::Dynamic, Storage> MatrixType; + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Storage, MaxSize, MaxSize>; MatrixType m, n; @@ -156,7 +157,232 @@ VERIFY_IS_EQUAL(v.size(), 4); } +struct ResizeScalar { + int value = 0; + static Index copies; + static Index moves; + + ResizeScalar() = default; + ResizeScalar(const ResizeScalar& other) : value(other.value) { ++copies; } + ResizeScalar(ResizeScalar&& other) noexcept : value(other.value) { ++moves; } + ResizeScalar& operator=(const ResizeScalar& other) { + value = other.value; + ++copies; + return *this; + } + ResizeScalar& operator=(ResizeScalar&& other) noexcept { + value = other.value; + ++moves; + return *this; + } + static void reset() { copies = moves = 0; } +}; + +Index ResizeScalar::copies = 0; +Index ResizeScalar::moves = 0; + +template <typename VectorType> +void fixed_capacity_vector_resize() { + STATIC_CHECK((!std::is_trivially_copyable<ResizeScalar>::value)); + STATIC_CHECK((internal::conservative_resize_like_impl<VectorType>::IsRelocatable)); + STATIC_CHECK((!internal::conservative_resize_like_impl<Matrix<ResizeScalar, Dynamic, 1>>::IsRelocatable)); + VectorType v; + const ResizeScalar* data = v.data(); + ResizeScalar::reset(); + for (Index size = 1; size <= VectorType::MaxSizeAtCompileTime; ++size) { + v.conservativeResize(size); + v(size - 1).value = int(size); + } + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + VERIFY(v.data() == data); + for (Index i = 0; i < v.size(); ++i) VERIFY_IS_EQUAL(v(i).value, i + 1); + + v.conservativeResize(2); + v.conservativeResize(2); + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + + // A source with the opposite orientation exercises the vector resizeLike specialization. + Matrix<ResizeScalar, VectorType::IsRowMajor ? Dynamic : 1, VectorType::IsRowMajor ? 1 : Dynamic> other(5); + for (Index i = 0; i < other.size(); ++i) other(i).value = -int(i + 1); + v.conservativeResizeLike(other); + VERIFY_IS_EQUAL(ResizeScalar::copies, 3); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + VERIFY_IS_EQUAL(v.rows(), VectorType::IsRowMajor ? 1 : 5); + VERIFY_IS_EQUAL(v.cols(), VectorType::IsRowMajor ? 5 : 1); + for (Index i = 0; i < v.size(); ++i) VERIFY_IS_EQUAL(v(i).value, i < 2 ? i + 1 : -(i + 1)); + + ResizeScalar::reset(); + v.conservativeResizeLike(other.head(1)); + VERIFY_IS_EQUAL(v.size(), 1); + VERIFY_IS_EQUAL(v(0).value, 1); + v.conservativeResize(0); + VERIFY_IS_EQUAL(v.size(), 0); + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + VERIFY(v.data() == data); +} + +template <typename MatrixType> +void fixed_capacity_matrix_resize() { + STATIC_CHECK((internal::conservative_resize_like_impl<MatrixType>::IsRelocatable)); + STATIC_CHECK((!internal::conservative_resize_like_impl<Matrix<ResizeScalar, Dynamic, Dynamic>>::IsRelocatable)); + const Index inner = 3; + MatrixType m(MatrixType::IsRowMajor ? 0 : inner, MatrixType::IsRowMajor ? inner : 0); + const ResizeScalar* data = m.data(); + ResizeScalar::reset(); + for (Index outer = 1; outer <= 5; ++outer) { + m.conservativeResize(MatrixType::IsRowMajor ? outer : inner, MatrixType::IsRowMajor ? inner : outer); + for (Index i = (outer - 1) * inner; i < m.size(); ++i) m.data()[i].value = int(i + 1); + } + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + VERIFY(m.data() == data); + for (Index i = 0; i < m.size(); ++i) VERIFY_IS_EQUAL(m.data()[i].value, i + 1); + + m.conservativeResize(MatrixType::IsRowMajor ? 2 : inner, MatrixType::IsRowMajor ? inner : 2); + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + MatrixType other(MatrixType::IsRowMajor ? 4 : inner, MatrixType::IsRowMajor ? inner : 4); + for (Index i = 0; i < other.size(); ++i) other.data()[i].value = -int(i + 1); + m.conservativeResizeLike(other); + VERIFY_IS_EQUAL(ResizeScalar::copies, 2 * inner); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + VERIFY_IS_EQUAL(m.rows(), other.rows()); + VERIFY_IS_EQUAL(m.cols(), other.cols()); + for (Index i = 0; i < m.size(); ++i) VERIFY_IS_EQUAL(m.data()[i].value, i < 2 * inner ? i + 1 : -(i + 1)); + + ResizeScalar::reset(); + other.resize(MatrixType::IsRowMajor ? 1 : inner, MatrixType::IsRowMajor ? inner : 1); + m.conservativeResizeLike(other); + VERIFY_IS_EQUAL(m.size(), inner); + for (Index i = 0; i < m.size(); ++i) VERIFY_IS_EQUAL(m.data()[i].value, i + 1); + m.conservativeResize(MatrixType::IsRowMajor ? 0 : inner, MatrixType::IsRowMajor ? inner : 0); + VERIFY_IS_EQUAL(m.size(), 0); + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + VERIFY(m.data() == data); +} + +struct InitializedResizeScalar { + int value; + InitializedResizeScalar() : value(0) {} +}; + +template <typename Vector> +void same_shape_vector_resize() { + using Scalar = typename Vector::Scalar; + STATIC_CHECK(std::is_trivially_copyable<Scalar>::value); + STATIC_CHECK(NumTraits<Scalar>::RequireInitialization); + Vector v(8); + for (Index i = 0; i < v.size(); ++i) v(i).value = int(i + 1); + const auto* data = v.data(); + const bool malloc_allowed = internal::is_malloc_allowed(); + internal::set_is_malloc_allowed(false); + // Every overload must leave unchanged storage alone, including scalars that require initialization. + v.conservativeResize(v.size()); + v.conservativeResize(v.rows(), v.cols()); + v.conservativeResize(NoChange, v.cols()); + v.conservativeResize(v.rows(), NoChange); + internal::set_is_malloc_allowed(malloc_allowed); + VERIFY(v.data() == data); + for (Index i = 0; i < v.size(); ++i) VERIFY_IS_EQUAL(v(i).value, i + 1); +} + +template <typename Vector> +void two_dimensional_vector_resize() { + STATIC_CHECK(!std::is_trivially_copyable<typename Vector::Scalar>::value); + Vector v; + const auto* data = v.data(); + ResizeScalar::reset(); + for (Index i = 0; i < 16; ++i) { + if (i % 3 == 0) { + v.conservativeResize(i + 1); + } else if (i % 3 == 1) { + v.conservativeResize(Vector::RowsAtCompileTime == 1 ? 1 : i + 1, Vector::RowsAtCompileTime == 1 ? i + 1 : 1); + } else { + if (Vector::RowsAtCompileTime == 1) + v.conservativeResize(NoChange, i + 1); + else + v.conservativeResize(i + 1, NoChange); + } + v(i).value = int(i + 1); + } + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + VERIFY(v.data() == data); + for (Index i = 0; i < v.size(); ++i) VERIFY_IS_EQUAL(v(i).value, i + 1); + v.conservativeResize(Vector::RowsAtCompileTime == 1 ? 1 : 8, Vector::RowsAtCompileTime == 1 ? 8 : 1); + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + Vector other(12); + for (Index i = 0; i < other.size(); ++i) other(i).value = int(100 + i); + v.conservativeResizeLike(other); + VERIFY_IS_EQUAL(ResizeScalar::copies, 4); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + for (Index i = 0; i < v.size(); ++i) VERIFY_IS_EQUAL(v(i).value, i < 8 ? i + 1 : 100 + i); + other.resize(4); + ResizeScalar::reset(); + v.conservativeResizeLike(other); + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + for (Index i = 0; i < v.size(); ++i) VERIFY_IS_EQUAL(v(i).value, i + 1); + VERIFY_RAISES_ASSERT(v.conservativeResize(2, 2)); + VERIFY_RAISES_ASSERT( + v.conservativeResize(Vector::RowsAtCompileTime == 1 ? 1 : -1, Vector::RowsAtCompileTime == 1 ? -1 : 1)); + VERIFY_RAISES_ASSERT( + v.conservativeResize(Vector::RowsAtCompileTime == 1 ? 1 : 17, Vector::RowsAtCompileTime == 1 ? 17 : 1)); + v.conservativeResize(Vector::RowsAtCompileTime == 1 ? 1 : 0, Vector::RowsAtCompileTime == 1 ? 0 : 1); + VERIFY_IS_EQUAL(ResizeScalar::copies, 0); + VERIFY_IS_EQUAL(ResizeScalar::moves, 0); + VERIFY(v.data() == data); +} + +template <typename Vector> +void heap_vector_resize() { + for (bool no_change : {false, true}) { + Vector v = Vector::LinSpaced(3, 1, 3); + for (Index size : {5, 2, 0, 4}) { + const Index common_size = (std::min)(size, v.size()); + if (!no_change) { + v.conservativeResize(Vector::RowsAtCompileTime == 1 ? 1 : size, Vector::RowsAtCompileTime == 1 ? size : 1); + } else if (Vector::RowsAtCompileTime == 1) { + v.conservativeResize(NoChange, size); + } else { + v.conservativeResize(size, NoChange); + } + VERIFY_IS_EQUAL(v.rows(), Vector::RowsAtCompileTime == 1 ? 1 : size); + VERIFY_IS_EQUAL(v.cols(), Vector::RowsAtCompileTime == 1 ? size : 1); + for (Index i = 0; i < common_size; ++i) VERIFY_IS_EQUAL(v(i), i + 1); + for (Index i = common_size; i < size; ++i) v(i) = typename Vector::Scalar(i + 1); + } + // A zero fixed dimension used to free the buffer without changing the vector's length. + if (Vector::RowsAtCompileTime == 1) { + VERIFY_RAISES_ASSERT(v.conservativeResize(0, v.cols())); + VERIFY_RAISES_ASSERT(v.conservativeResize(0, NoChange)); + } else { + VERIFY_RAISES_ASSERT(v.conservativeResize(v.rows(), 0)); + VERIFY_RAISES_ASSERT(v.conservativeResize(NoChange, 0)); + } + VERIFY_IS_EQUAL(v.size(), 4); + for (Index i = 0; i < v.size(); ++i) VERIFY_IS_EQUAL(v(i), i + 1); + } +} + EIGEN_DECLARE_TEST(conservative_resize) { + CALL_SUBTEST_7((same_shape_vector_resize<Matrix<InitializedResizeScalar, Dynamic, 1>>())); + CALL_SUBTEST_7((same_shape_vector_resize<Matrix<InitializedResizeScalar, 1, Dynamic>>())); + CALL_SUBTEST_7((same_shape_vector_resize<Array<InitializedResizeScalar, Dynamic, 1>>())); + CALL_SUBTEST_7((same_shape_vector_resize<Array<InitializedResizeScalar, 1, Dynamic>>())); + CALL_SUBTEST_7((two_dimensional_vector_resize<Matrix<ResizeScalar, Dynamic, 1, ColMajor, 16, 1>>())); + CALL_SUBTEST_7((two_dimensional_vector_resize<Matrix<ResizeScalar, 1, Dynamic, RowMajor, 1, 16>>())); + CALL_SUBTEST_7((two_dimensional_vector_resize<Array<ResizeScalar, Dynamic, 1, ColMajor, 16, 1>>())); + CALL_SUBTEST_7((two_dimensional_vector_resize<Array<ResizeScalar, 1, Dynamic, RowMajor, 1, 16>>())); + CALL_SUBTEST_7(heap_vector_resize<VectorXd>()); + CALL_SUBTEST_7(heap_vector_resize<RowVectorXd>()); + CALL_SUBTEST_7(heap_vector_resize<ArrayXd>()); + CALL_SUBTEST_7((heap_vector_resize<Array<double, 1, Dynamic>>())); for (int i = 0; i < g_repeat; ++i) { CALL_SUBTEST_1((run_matrix_tests<int, Eigen::RowMajor>())); CALL_SUBTEST_1((run_matrix_tests<int, Eigen::ColMajor>())); @@ -182,5 +408,14 @@ #endif CALL_SUBTEST_6((run_vector_tests<AnnoyingScalar>())); CALL_SUBTEST_6((noncopyable<0>())); + + CALL_SUBTEST_7((fixed_capacity_vector_resize<Array<ResizeScalar, Dynamic, 1, ColMajor, 8, 1>>())); + CALL_SUBTEST_7((fixed_capacity_vector_resize<Matrix<ResizeScalar, 1, Dynamic, RowMajor, 1, 8>>())); + CALL_SUBTEST_7((fixed_capacity_matrix_resize<Matrix<ResizeScalar, Dynamic, Dynamic, ColMajor, 5, 5>>())); + CALL_SUBTEST_7((fixed_capacity_matrix_resize<Array<ResizeScalar, Dynamic, Dynamic, RowMajor, 5, 5>>())); + CALL_SUBTEST_7((fixed_capacity_matrix_resize<Matrix<ResizeScalar, 3, Dynamic, ColMajor, 3, 5>>())); + CALL_SUBTEST_7((fixed_capacity_matrix_resize<Array<ResizeScalar, Dynamic, 3, RowMajor, 5, 3>>())); + CALL_SUBTEST_7((run_matrix_tests<AnnoyingScalar, RowMajor, 75>())); + CALL_SUBTEST_7((run_matrix_tests<AnnoyingScalar, ColMajor, 75>())); } }
diff --git a/test/dgmres.cpp b/test/dgmres.cpp index 9650de4..6f95ebc 100644 --- a/test/dgmres.cpp +++ b/test/dgmres.cpp
@@ -146,6 +146,7 @@ // The Arnoldi coefficient is h(i,j) = v_i^H A v_j; computing its conjugate broke the orthogonalization for complex // scalars, so restarted DGMRES needed about twice the iterations of the equivalent GMRES(30). +template <typename = void> void test_dgmres_complex_arnoldi() { using T = std::complex<double>; const Index n = 60; @@ -233,6 +234,7 @@ // A real conjugate pair of Ritz values is deflated whole. The smallest eigenvalues here are about 0.002 +- 0.04i, so // setEigenv(1) with room for one update deflates a two-dimensional subspace. +template <typename = void> void test_dgmres_deflation_conjugate_pair() { const Index g = 20, n = g * g; SparseMatrix<double> A(n, n); @@ -265,6 +267,7 @@ // The deflation subspace never exceeds the dimension: with n = 3 and restart 2 it reached four vectors, which made T // singular. +template <typename = void> void test_dgmres_deflation_small() { const Index n = 3; SparseMatrix<double> A(n, n); @@ -293,11 +296,11 @@ CALL_SUBTEST_4(test_dgmres_breakdown_T<std::complex<double> >()); CALL_SUBTEST_5(test_dgmres_iterations_T<double>()); CALL_SUBTEST_6(test_dgmres_iterations_T<std::complex<double> >()); - CALL_SUBTEST_6(test_dgmres_complex_arnoldi()); + CALL_SUBTEST_6(test_dgmres_complex_arnoldi<>()); CALL_SUBTEST_7((test_dgmres_deflation<double, IdentityPreconditioner>(0.0))); CALL_SUBTEST_7((test_dgmres_deflation<double, DiagonalPreconditioner<double> >(0.0))); - CALL_SUBTEST_7(test_dgmres_deflation_conjugate_pair()); - CALL_SUBTEST_7(test_dgmres_deflation_small()); + CALL_SUBTEST_7(test_dgmres_deflation_conjugate_pair<>()); + CALL_SUBTEST_7(test_dgmres_deflation_small<>()); CALL_SUBTEST_8((test_dgmres_deflation<std::complex<double>, IdentityPreconditioner>(std::complex<double>(0, 1)))); CALL_SUBTEST_8((test_dgmres_deflation<std::complex<double>, DiagonalPreconditioner<std::complex<double> > >( std::complex<double>(0, 1))));
diff --git a/test/diagonalmatrices.cpp b/test/diagonalmatrices.cpp index e4e461e..15a79de 100644 --- a/test/diagonalmatrices.cpp +++ b/test/diagonalmatrices.cpp
@@ -16,6 +16,7 @@ #include "main.h" #include <Eigen/Core> #include "MovableScalar.h" +#include "NonCommutativeScalar.h" #include "random_for_arithmetic.h" using namespace std; template <typename MatrixType> @@ -800,8 +801,12 @@ rm_res.noalias() -= rm * umap.asDiagonal(); VERIFY_IS_APPROX(rm_res, rm_sub_ref); - VERIFY_IS_APPROX((umap.asDiagonal() * cm).sum(), dm_cm_ref.sum()); - VERIFY_IS_APPROX((rm * umap.asDiagonal()).sum(), rm_dm_ref.sum()); + // Random signed sums can nearly cancel, so bound the difference relative to the summed magnitudes, not to the sum. + // Each of two differently associated sums of N = n^2 products rounds by (N - 1 + sqrt(5)) u, with u = eps / 2. + using RealScalar = typename NumTraits<Scalar>::Real; + const RealScalar sum_tolerance = RealScalar(n * n + 2) * NumTraits<RealScalar>::epsilon(); + VERIFY(numext::abs((umap.asDiagonal() * cm).sum() - dm_cm_ref.sum()) <= sum_tolerance * dm_cm_ref.cwiseAbs().sum()); + VERIFY(numext::abs((rm * umap.asDiagonal()).sum() - rm_dm_ref.sum()) <= sum_tolerance * rm_dm_ref.cwiseAbs().sum()); internal::set_is_malloc_allowed(true); } } @@ -820,6 +825,129 @@ } } +template <int Tag> +struct DiagonalProductScalar { + explicit DiagonalProductScalar(int v = 0) : value(v) {} + int value; +}; + +DiagonalProductScalar<2> operator*(const DiagonalProductScalar<0>& lhs, const DiagonalProductScalar<1>& rhs) { + return DiagonalProductScalar<2>(10 * lhs.value + rhs.value); +} + +DiagonalProductScalar<3> operator*(const DiagonalProductScalar<1>& lhs, const DiagonalProductScalar<0>& rhs) { + return DiagonalProductScalar<3>(100 * lhs.value + rhs.value); +} + +namespace Eigen { +template <> +struct ScalarBinaryOpTraits<DiagonalProductScalar<0>, DiagonalProductScalar<1>, + internal::scalar_product_op<DiagonalProductScalar<0>, DiagonalProductScalar<1>>> { + using ReturnType = DiagonalProductScalar<2>; +}; + +template <> +struct ScalarBinaryOpTraits<DiagonalProductScalar<1>, DiagonalProductScalar<0>, + internal::scalar_product_op<DiagonalProductScalar<1>, DiagonalProductScalar<0>>> { + using ReturnType = DiagonalProductScalar<3>; +}; +} // namespace Eigen + +template <int Mode, int Order> +void diagonal_product_result_types() { + DiagonalMatrix<DiagonalProductScalar<0>, 2> diagonal; + diagonal.diagonal()(0) = DiagonalProductScalar<0>(2); + diagonal.diagonal()(1) = DiagonalProductScalar<0>(3); + Matrix<DiagonalProductScalar<1>, 2, 2, Order> matrix; + for (Index j = 0; j < 2; ++j) + for (Index i = 0; i < 2; ++i) matrix(i, j) = DiagonalProductScalar<1>(int(1 + i + 2 * j)); + const Matrix<DiagonalProductScalar<2>, 2, 2> denseLeft = diagonal * matrix; + const Matrix<DiagonalProductScalar<3>, 2, 2> denseRight = matrix * diagonal; + const auto triangularLeft = diagonal * matrix.template triangularView<Mode>(); + const auto triangularRight = matrix.template triangularView<Mode>() * diagonal; + internal::evaluator<decltype(triangularLeft)> left(triangularLeft); + internal::evaluator<decltype(triangularRight)> right(triangularRight); + static_assert(std::is_same<typename decltype(left)::Scalar, DiagonalProductScalar<2>>::value, "left result type"); + static_assert(std::is_same<typename decltype(right)::Scalar, DiagonalProductScalar<3>>::value, "right result type"); + const auto selfadjointLeft = diagonal * matrix.template selfadjointView<Mode&(Upper | Lower)>(); + const auto selfadjointRight = matrix.template selfadjointView<Mode&(Upper | Lower)>() * diagonal; + internal::evaluator<decltype(selfadjointLeft)> symmetricLeft(selfadjointLeft); + internal::evaluator<decltype(selfadjointRight)> symmetricRight(selfadjointRight); + for (Index j = 0; j < 2; ++j) + for (Index i = 0; i < 2; ++i) { + VERIFY_IS_EQUAL(denseLeft(i, j).value, 10 * diagonal.diagonal()(i).value + matrix(i, j).value); + VERIFY_IS_EQUAL(denseRight(i, j).value, 100 * matrix(i, j).value + diagonal.diagonal()(j).value); + const bool active = (Mode & Upper) ? i <= j : i >= j; + const bool zero = !active || (i == j && (Mode & ZeroDiag)); + const int triangular = i == j && (Mode & UnitDiag) ? 1 : matrix(i, j).value; + VERIFY_IS_EQUAL(left.coeff(i, j).value, zero ? 0 : 10 * diagonal.diagonal()(i).value + triangular); + VERIFY_IS_EQUAL(right.coeff(i, j).value, zero ? 0 : 100 * triangular + diagonal.diagonal()(j).value); + const int symmetric = active ? matrix(i, j).value : matrix(j, i).value; + VERIFY_IS_EQUAL(symmetricLeft.coeff(i, j).value, 10 * diagonal.diagonal()(i).value + symmetric); + VERIFY_IS_EQUAL(symmetricRight.coeff(i, j).value, 100 * symmetric + diagonal.diagonal()(j).value); + } +} + +template <typename = void> +void diagonal_product_quaternion_order() { + using Scalar = noncommutative_scalar::Quaternion; + Matrix<Scalar, 1, 1> matrix; + DiagonalMatrix<Scalar, 1> diagonal; + matrix(0) = Scalar(0, 1, 0, 0); + diagonal.diagonal()(0) = Scalar(0, 0, 1, 0); + VERIFY((matrix * diagonal).coeff(0) == Scalar(0, 0, 0, 1)); + VERIFY((diagonal * matrix).coeff(0) == Scalar(0, 0, 0, -1)); +} + +struct NoncommutingDiagonalScalar { + explicit NoncommutingDiagonalScalar(int diagonal = 0) : values{{diagonal, 0, 0, diagonal}} {} + std::array<int, 4> values; +}; + +NoncommutingDiagonalScalar operator*(const NoncommutingDiagonalScalar& lhs, const NoncommutingDiagonalScalar& rhs) { + NoncommutingDiagonalScalar result; + for (int i = 0; i < 2; ++i) + for (int j = 0; j < 2; ++j) + for (int k = 0; k < 2; ++k) result.values[2 * i + j] += lhs.values[2 * i + k] * rhs.values[2 * k + j]; + return result; +} + +template <int Size> +void diagonal_product_operand_order() { + using Scalar = NoncommutingDiagonalScalar; + const Index size = Size == Dynamic ? 2 : Size; + Matrix<Scalar, 1, Size> row(size); + DiagonalMatrix<Scalar, Size> diagonal(size); + for (Index i = 0; i < size; ++i) { + row(i).values = {{int(i) + 1, 2, 0, 1}}; + diagonal.diagonal()(i).values = {{1, 0, 3, int(i) + 1}}; + } + const auto right = row * diagonal; + const auto left = diagonal * row.transpose(); + const Matrix<Scalar, 1, Size> materializedRight = right; + const Matrix<Scalar, Size, 1> materializedLeft = left; + for (Index i = 0; i < size; ++i) { + const Scalar expectedRight = row(i) * diagonal.diagonal()(i); + const Scalar expectedLeft = diagonal.diagonal()(i) * row(i); + VERIFY(expectedRight.values != expectedLeft.values); + VERIFY(right.coeff(i).values == expectedRight.values); + VERIFY(left.coeff(i).values == expectedLeft.values); + VERIFY(materializedRight(i).values == expectedRight.values); + VERIFY(materializedLeft(i).values == expectedLeft.values); + } + + DiagonalMatrix<Scalar, Dynamic> singleton(1); + singleton.diagonal()(0) = diagonal.diagonal()(0); + const Matrix<Scalar, Size, 1> column = row.transpose(); + const auto scaled = column * singleton; + const Matrix<Scalar, Size, 1> materializedScaled = scaled; + for (Index i = 0; i < size; ++i) { + const Scalar expected = column(i) * singleton.diagonal()(0); + VERIFY(scaled.coeff(i).values == expected.values); + VERIFY(materializedScaled(i).values == expected.values); + } +} + EIGEN_DECLARE_TEST(diagonalmatrices) { for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST_1(diagonalmatrices(Matrix<float, 1, 1>())); @@ -858,4 +986,12 @@ CALL_SUBTEST_11(diagonal_matrix_moves<int>()); CALL_SUBTEST_11(diagonal_matrix_moves<std::complex<double>>()); CALL_SUBTEST_11(diagonal_matrix_moves<MovableScalar<float>>()); + CALL_SUBTEST_12((diagonal_product_result_types<Lower, ColMajor>())); + CALL_SUBTEST_12((diagonal_product_result_types<Upper, RowMajor>())); + CALL_SUBTEST_12((diagonal_product_result_types<UnitLower, RowMajor>())); + CALL_SUBTEST_12((diagonal_product_result_types<StrictlyUpper, ColMajor>())); + CALL_SUBTEST_13((diagonal_product_quaternion_order<>())); + CALL_SUBTEST_13((diagonal_product_operand_order<1>())); + CALL_SUBTEST_13((diagonal_product_operand_order<2>())); + CALL_SUBTEST_13((diagonal_product_operand_order<Dynamic>())); }
diff --git a/test/dynalloc.cpp b/test/dynalloc.cpp index df44818..321d148 100644 --- a/test/dynalloc.cpp +++ b/test/dynalloc.cpp
@@ -140,7 +140,87 @@ #endif } +template <int Alignment> +struct alignas(Alignment) ScratchScalar { + static int alive; + int value; + ScratchScalar() : value(17) { + VERIFY(std::uintptr_t(this) % Alignment == 0); + ++alive; + } + ~ScratchScalar() { --alive; } +}; +template <int Alignment> +int ScratchScalar<Alignment>::alive = 0; + +template <int Alignment> +void check_typed_scratch() { + using T = ScratchScalar<Alignment>; + STATIC_CHECK((std::is_same<decltype(internal::scratch_malloc<T>(sizeof(T))), T *>::value)); + const Index sizes[] = {1, 3, EIGEN_STACK_ALLOCATION_LIMIT / sizeof(T) + 1}; + for (Index size : sizes) { + T *raw = internal::scratch_malloc<T>(sizeof(T) * size); + VERIFY(std::uintptr_t(raw) % Alignment == 0); + VERIFY_IS_EQUAL(T::alive, 0); + std::memset(static_cast<void *>(raw), 0, sizeof(T) * size); + internal::scratch_free<T>(raw); + { + ei_declare_aligned_stack_constructed_variable(T, data, size, nullptr); + VERIFY_IS_EQUAL(T::alive, size); + for (Index i = 0; i < size; ++i) VERIFY_IS_EQUAL(data[i].value, 17); + } + VERIFY_IS_EQUAL(T::alive, 0); + T *data = internal::scratch_new<T>(size); + VERIFY_IS_EQUAL(T::alive, size); + internal::scratch_delete(data, size); + VERIFY_IS_EQUAL(T::alive, 0); + } + internal::scratch_free<T>(nullptr); +#ifdef EIGEN_EXCEPTIONS + EIGEN_IF_CONSTEXPR (alignof(T) > EIGEN_STACK_ALIGN_BYTES && + (EIGEN_STACK_ALIGN_BYTES != 0 || alignof(T) > alignof(std::max_align_t))) { + const std::size_t overhead = alignof(T) - 1 + sizeof(void *); + const std::size_t max_size = (std::numeric_limits<std::size_t>::max)(); + for (std::size_t bytes : {max_size, max_size - overhead + 1}) { + bool threw = false; + try { + T *raw = internal::scratch_malloc<T>(bytes); + internal::scratch_free<T>(raw); + } catch (const std::bad_alloc &) { + threw = true; + } + VERIFY(threw); + } + } +#endif + { + T supplied[2]; + { + ei_declare_aligned_stack_constructed_variable(T, data, 2, supplied); + VERIFY(data == supplied); + VERIFY_IS_EQUAL(T::alive, 2); + } + VERIFY_IS_EQUAL(T::alive, 2); + } + VERIFY_IS_EQUAL(T::alive, 0); +#if defined(EIGEN_ALLOCA) && !defined(EIGEN_NO_ALLOCA) + { + T supplied[3]; + using Vector = Matrix<T, Dynamic, 1>; + using View = Map<Vector, Unaligned>; + View input(supplied, 3); + internal::local_nested_eval_wrapper<View, 2, true> copy(input, nullptr); + VERIFY(std::uintptr_t(copy.object.data()) % Alignment == 0); + VERIFY_IS_EQUAL(copy.object(0).value, 17); + } + VERIFY_IS_EQUAL(T::alive, 0); +#endif +} + EIGEN_DECLARE_TEST(dynalloc) { + CALL_SUBTEST(check_typed_scratch<16>()); + CALL_SUBTEST(check_typed_scratch<64>()); + CALL_SUBTEST(check_typed_scratch<512>()); // low level dynamic memory allocation CALL_SUBTEST(check_handmade_aligned_malloc()); CALL_SUBTEST(check_aligned_malloc());
diff --git a/test/eigensolver_selfadjoint.cpp b/test/eigensolver_selfadjoint.cpp index f22b822..e329cdd 100644 --- a/test/eigensolver_selfadjoint.cpp +++ b/test/eigensolver_selfadjoint.cpp
@@ -644,9 +644,8 @@ // Random diagonal matrix. MatrixType diag = MatrixType::Zero(n, n); - for (Index i = 0; i < n; ++i) { - diag(i, i) = internal::random<RealScalar>(-100, 100); - } + auto diagonal = diag.diagonal(); + setRandomInRange(diagonal, RealScalar(-100), RealScalar(100)); selfadjointeigensolver_essential_check(diag); // The eigenvalues should be the diagonal entries, sorted. @@ -976,7 +975,8 @@ } } -#if defined(EIGEN_TEST_PART_20) || defined(EIGEN_TEST_PART_ALL) +#if defined(EIGEN_TEST_PART_20) || defined(EIGEN_TEST_PART_22) || defined(EIGEN_TEST_PART_23) || \ + defined(EIGEN_TEST_PART_ALL) template <typename = void> void selfadjoint_iterative_scaling_rounding() { Matrix4f matrix = Matrix4f::Zero(); @@ -1341,9 +1341,9 @@ EIGEN_DECLARE_TEST(eigensolver_selfadjoint) { CALL_SUBTEST_20(selfadjoint_iterative_scaling_rounding<>()); CALL_SUBTEST_20((selfadjoint_iterative_scaling_blocks<float, ColMajor>())); - CALL_SUBTEST_20((selfadjoint_iterative_scaling_blocks<double, RowMajor>())); - CALL_SUBTEST_20((selfadjoint_iterative_scaling_blocks<std::complex<float>, RowMajor>())); - CALL_SUBTEST_20((selfadjoint_iterative_scaling_blocks<std::complex<double>, ColMajor>())); + CALL_SUBTEST_22((selfadjoint_iterative_scaling_blocks<double, RowMajor>())); + CALL_SUBTEST_23((selfadjoint_iterative_scaling_blocks<std::complex<float>, RowMajor>())); + CALL_SUBTEST_23((selfadjoint_iterative_scaling_blocks<std::complex<double>, ColMajor>())); CALL_SUBTEST_21((direct_selfadjoint_clustered_boundary<float, ColMajor>())); CALL_SUBTEST_21((direct_selfadjoint_rotated_boundary<float, ColMajor>())); CALL_SUBTEST_21((direct_selfadjoint_rotated_boundary<double, RowMajor>()));
diff --git a/test/fastmath.cpp b/test/fastmath.cpp index 98fcbea..3ce5b7a 100644 --- a/test/fastmath.cpp +++ b/test/fastmath.cpp
@@ -232,9 +232,12 @@ VERIFY_IS_APPROX(conjugate_product, expected_conjugate_product); // numext::divide guards against intermediate overflow (c^2 + d^2) near max representable bounds. + // The divisor stays below max_val / 4: above that its reciprocal is subnormal, and GCC's + // -ffast-math vector division (rcpps plus a Newton step) flushes it to zero. const RealScalar max_val = (std::numeric_limits<RealScalar>::max)(); - const Scalar big_num(RealScalar(0.9) * max_val, RealScalar(0)); - const Scalar big_denom(max_val, RealScalar(0)); + const RealScalar big_val = max_val / RealScalar(16); + const Scalar big_num(RealScalar(0.9) * big_val, RealScalar(0)); + const Scalar big_denom(big_val, RealScalar(0)); const Scalar safe_div_result = numext::divide(big_num, big_denom); VERIFY((numext::isfinite)(safe_div_result.real())); VERIFY_IS_APPROX(safe_div_result, Scalar(RealScalar(0.9), RealScalar(0)));
diff --git a/test/first_aligned.cpp b/test/first_aligned.cpp index b3b545e..1f8744f 100644 --- a/test/first_aligned.cpp +++ b/test/first_aligned.cpp
@@ -20,7 +20,7 @@ void test_none_aligned_helper(Scalar *array, int size) { EIGEN_UNUSED_VARIABLE(array); EIGEN_UNUSED_VARIABLE(size); - VERIFY(internal::packet_traits<Scalar>::size == 1 || internal::first_default_aligned(array, size) == size); + VERIFY(!internal::packet_traits<Scalar>::Vectorizable || internal::first_default_aligned(array, size) == size); } struct some_non_vectorizable_type { @@ -45,6 +45,18 @@ test_none_aligned_helper(array_double_plus_4_bytes, 50); test_none_aligned_helper(array_double_plus_4_bytes + 1, 50); + // alignof(std::complex<double>) == 8 while sizeof(std::complex<double>) == 16; an 8-byte-offset buffer + // cannot reach >= 16-byte alignment at any element stride. + EIGEN_ALIGN16 std::complex<double> array_cdouble[100]; + test_first_aligned_helper(array_cdouble, 50); + test_first_aligned_helper(array_cdouble + 1, 50); + std::complex<double> *array_cdouble_plus_8_bytes = + reinterpret_cast<std::complex<double> *>(std::uintptr_t(array_cdouble) + 8); + test_none_aligned_helper(array_cdouble_plus_8_bytes, 50); + test_none_aligned_helper(array_cdouble_plus_8_bytes + 1, 50); + VERIFY_IS_EQUAL(internal::first_aligned<16>(array_cdouble_plus_8_bytes, 50), 50); + VERIFY_IS_EQUAL(internal::first_aligned<8>(array_cdouble_plus_8_bytes, 50), 0); + some_non_vectorizable_type array_nonvec[100]; test_first_aligned_helper(array_nonvec, 100); test_none_aligned_helper(array_nonvec, 100);
diff --git a/test/fp_control.h b/test/fp_control.h index e0e9d5d..63f8d84 100644 --- a/test/fp_control.h +++ b/test/fp_control.h
@@ -11,7 +11,7 @@ #include <Eigen/Core> -#if defined(EIGEN_GPU_COMPILE_PHASE) || defined(SYCL_DEVICE_ONLY) +#if defined(EIGEN_GPU_COMPILE_PHASE) || defined(__SYCL_DEVICE_ONLY__) #define EIGEN_TEST_DEVICE_COMPILE 1 #else #define EIGEN_TEST_DEVICE_COMPILE 0
diff --git a/test/gemm_double.cpp b/test/gemm_double.cpp new file mode 100644 index 0000000..0875931 --- /dev/null +++ b/test/gemm_double.cpp
@@ -0,0 +1,101 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include "main.h" +#include <Eigen/Core> + +template <int LhsOrder, int RhsOrder, int ResOrder> +void double_gemm_tiles() { + using Lhs = Matrix<double, Dynamic, Dynamic, LhsOrder>; + using Rhs = Matrix<double, Dynamic, Dynamic, RhsOrder>; + using Res = Matrix<double, Dynamic, Dynamic, ResOrder>; + for (Index m : {2, 3, 4, 6, 7, 13}) { + for (Index n : {3, 4, 5, 8, 9}) { + for (Index k : {7, 8, 9, 17, 65}) { + const Lhs lhs = Lhs::Random(m + 2, k + 3); + const Rhs rhs = Rhs::Random(k + 2, n + 3); + const auto a = lhs.block(1, 1, m, k); + const auto b = rhs.block(1, 1, k, n); + const Index outer_stride = 2 * (ResOrder == ColMajor ? m : n) + 3; + const Index size = outer_stride * (ResOrder == ColMajor ? n : m) + 2; + for (double alpha : {1.0, -0.75}) { + VectorXd storage = VectorXd::Random(size); + VectorXd reference = storage; + Map<Res, Unaligned, Stride<Dynamic, Dynamic>> c(storage.data() + 1, m, n, + Stride<Dynamic, Dynamic>(outer_stride, 2)); + Map<Res, Unaligned, Stride<Dynamic, Dynamic>> expected(reference.data() + 1, m, n, + Stride<Dynamic, Dynamic>(outer_stride, 2)); + VectorXd bounds = VectorXd::Zero(size); + for (Index j = 0; j < n; ++j) { + for (Index i = 0; i < m; ++i) { + double sum = 0; + double scale = numext::abs(expected(i, j)); + for (Index p = 0; p < k; ++p) { + sum += a(i, p) * b(p, j); + scale += numext::abs(alpha) * numext::abs(a(i, p)) * numext::abs(b(p, j)); + } + expected(i, j) += alpha * sum; + // Bound both evaluations of C0 + alpha*A*B, including packing and accumulation. + bounds(&expected(i, j) - reference.data()) = 8 * (k + 1) * NumTraits<double>::epsilon() * scale; + } + } + c.noalias() += alpha * a * b; + VERIFY(((storage - reference).cwiseAbs().array() <= bounds.array()).all()); + } + } + } + } +} + +template <int Order, int UpLo> +void double_syrk_tiles() { + using Mat = Matrix<double, Dynamic, Dynamic, Order>; + // mr=3 and nr=4 also change the triangular kernel's least-common-multiple block size to 12. + for (Index n : {3, 4, 5, 11, 12, 13, 23, 24, 25, 49}) { + for (Index k : {7, 8, 9, 17, 65}) { + const Mat a = Mat::Random(n, k); + Mat c = Mat::Random(n, n); + Mat expected = c; + Mat bounds = Mat::Zero(n, n); + const double alpha = -0.75; + for (Index j = 0; j < n; ++j) { + for (Index i = 0; i < n; ++i) { + if ((UpLo == Lower && i < j) || (UpLo == Upper && i > j)) continue; + double sum = 0; + double scale = numext::abs(c(i, j)); + for (Index p = 0; p < k; ++p) { + sum += a(i, p) * a(j, p); + scale += numext::abs(alpha) * numext::abs(a(i, p)) * numext::abs(a(j, p)); + } + expected(i, j) += alpha * sum; + bounds(i, j) = 8 * (k + 1) * NumTraits<double>::epsilon() * scale; + } + } + c.template selfadjointView<UpLo>().rankUpdate(a, alpha); + VERIFY(((c - expected).cwiseAbs().array() <= bounds.array()).all()); + } + } +} + +EIGEN_DECLARE_TEST(gemm_double) { +#if EIGEN_ARCH_ARM && defined(EIGEN_VECTORIZE_NEON) + using Traits = internal::gebp_traits<double, double>; + STATIC_CHECK(Traits::mr == 3); + STATIC_CHECK((std::is_same<typename Traits::RhsPacketx4, double>::value)); + using IntTraits = internal::gebp_traits<int, int>; + using GenericIntTraits = internal::gebp_traits<int, int, false, false, Architecture::Generic>; + STATIC_CHECK((std::is_same<typename IntTraits::RhsPacketx4, typename GenericIntTraits::RhsPacketx4>::value)); +#endif + CALL_SUBTEST((double_gemm_tiles<ColMajor, ColMajor, ColMajor>())); + CALL_SUBTEST((double_gemm_tiles<ColMajor, ColMajor, RowMajor>())); + CALL_SUBTEST((double_gemm_tiles<ColMajor, RowMajor, ColMajor>())); + CALL_SUBTEST((double_gemm_tiles<ColMajor, RowMajor, RowMajor>())); + CALL_SUBTEST((double_gemm_tiles<RowMajor, ColMajor, ColMajor>())); + CALL_SUBTEST((double_gemm_tiles<RowMajor, ColMajor, RowMajor>())); + CALL_SUBTEST((double_gemm_tiles<RowMajor, RowMajor, ColMajor>())); + CALL_SUBTEST((double_gemm_tiles<RowMajor, RowMajor, RowMajor>())); + CALL_SUBTEST((double_syrk_tiles<ColMajor, Lower>())); + CALL_SUBTEST((double_syrk_tiles<ColMajor, Upper>())); + CALL_SUBTEST((double_syrk_tiles<RowMajor, Lower>())); + CALL_SUBTEST((double_syrk_tiles<RowMajor, Upper>())); +}
diff --git a/test/gemm_packing.cpp b/test/gemm_packing.cpp new file mode 100644 index 0000000..21f6a0c --- /dev/null +++ b/test/gemm_packing.cpp
@@ -0,0 +1,186 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +// Real cache blocking lets these products exceed the stack allocation limit. +#define EIGEN_NO_DEBUG_SMALL_PRODUCT_BLOCKS +#include "main.h" +#include <Eigen/Core> + +#ifdef EIGEN_TEST_WRAP_MALLOC +static volatile bool count_packing_allocations = false; +static volatile Index packing_allocations = 0; +// Preserve the linker's --wrap symbols without reserved C++ identifiers. +extern "C" void* real_malloc(std::size_t) asm("__real_malloc"); +extern "C" void* wrap_malloc(std::size_t) asm("__wrap_malloc"); +extern "C" void* wrap_malloc(std::size_t bytes) { + if (count_packing_allocations) packing_allocations = packing_allocations + 1; + return real_malloc(bytes); +} +#endif + +template <typename LhsScalar, typename RhsScalar> +void packing_storage() { + const std::size_t sizeA = EIGEN_STACK_ALLOCATION_LIMIT / sizeof(LhsScalar) + 3; + const std::size_t sizeB = EIGEN_STACK_ALLOCATION_LIMIT / sizeof(RhsScalar) + 7; + internal::gemm_packing_space<LhsScalar, RhsScalar> packing(sizeA, sizeB, nullptr, nullptr); + VERIFY(packing.blockA() != nullptr); + VERIFY(packing.blockB() != nullptr); + const auto first = reinterpret_cast<std::uintptr_t>(packing.blockA()); + const auto second = reinterpret_cast<std::uintptr_t>(packing.blockB()); + const std::size_t alignment = (std::max)(std::size_t(EIGEN_STACK_ALIGN_BYTES), std::size_t(alignof(RhsScalar))); + VERIFY_IS_EQUAL(second - first, (sizeA * sizeof(LhsScalar) + alignment - 1) / alignment * alignment); + VERIFY_IS_EQUAL(first % alignof(LhsScalar), std::uintptr_t(0)); + VERIFY_IS_EQUAL(second % alignment, std::uintptr_t(0)); + packing.blockA()[0] = LhsScalar(1); + packing.blockA()[sizeA - 1] = LhsScalar(2); + packing.blockB()[0] = RhsScalar(3); + packing.blockB()[sizeB - 1] = RhsScalar(4); + VERIFY_IS_EQUAL(packing.blockA()[0], LhsScalar(1)); + VERIFY_IS_EQUAL(packing.blockA()[sizeA - 1], LhsScalar(2)); + + std::vector<LhsScalar> suppliedA(sizeA); + std::vector<RhsScalar> suppliedB(sizeB); + internal::gemm_packing_space<LhsScalar, RhsScalar> supplied(sizeA, sizeB, suppliedA.data(), suppliedB.data()); + VERIFY(supplied.blockA() == suppliedA.data()); + VERIFY(supplied.blockB() == suppliedB.data()); + internal::gemm_packing_space<LhsScalar, RhsScalar> partial(sizeA, sizeB, suppliedA.data(), nullptr); + VERIFY(partial.blockA() == suppliedA.data()); + VERIFY(partial.blockB() == nullptr); +#if defined(EIGEN_ALLOCA) && !defined(EIGEN_NO_ALLOCA) && EIGEN_STACK_ALLOCATION_LIMIT > 0 + internal::gemm_packing_space<LhsScalar, RhsScalar> small(1, 1, nullptr, nullptr); + VERIFY(small.blockA() == nullptr); + VERIFY(small.blockB() == nullptr); + internal::gemm_packing_space<LhsScalar, RhsScalar> mixed(1, sizeB, nullptr, nullptr); + VERIFY(mixed.blockA() == nullptr); + VERIFY(mixed.blockB() == nullptr); +#endif +} + +// Stricter than the alignment scratch_malloc guarantees, so either panel of this type keeps its own allocation. +constexpr std::size_t kScratchAlignment = EIGEN_STACK_ALIGN_BYTES > 0 ? EIGEN_STACK_ALIGN_BYTES + : alignof(std::max_align_t); +struct alignas(2 * kScratchAlignment) OverAlignedScalar { + double value = 0; +}; + +template <typename LhsScalar, typename RhsScalar> +void packing_alignment_fallback() { + const std::size_t sizeA = EIGEN_STACK_ALLOCATION_LIMIT / sizeof(LhsScalar) + 3; + const std::size_t sizeB = EIGEN_STACK_ALLOCATION_LIMIT / sizeof(RhsScalar) + 7; + internal::gemm_packing_space<LhsScalar, RhsScalar> packing(sizeA, sizeB, nullptr, nullptr); + VERIFY(packing.blockA() == nullptr); + VERIFY(packing.blockB() == nullptr); +} + +struct PackingScalar { + static Index live; + static Index remaining; + int value = 0; + PackingScalar() { +#ifdef EIGEN_EXCEPTIONS + if (remaining == 0) throw std::bad_alloc(); +#endif + --remaining; + ++live; + } + ~PackingScalar() { --live; } +}; +Index PackingScalar::live = 0; +Index PackingScalar::remaining = 0; + +namespace Eigen { +template <> +struct NumTraits<PackingScalar> : GenericNumTraits<PackingScalar> { + static constexpr bool RequireInitialization = true; +}; +} // namespace Eigen + +#ifdef EIGEN_EXCEPTIONS +template <typename LhsScalar, typename RhsScalar> +bool packing_throws_bad_alloc(std::size_t sizeA, std::size_t sizeB) { + try { + internal::gemm_packing_space<LhsScalar, RhsScalar> packing(sizeA, sizeB, nullptr, nullptr); + } catch (const std::bad_alloc&) { + return true; + } + return false; +} +#endif + +void packing_lifetimes() { + const std::size_t size = EIGEN_STACK_ALLOCATION_LIMIT / sizeof(PackingScalar) + 3; + PackingScalar::remaining = Index(2 * size); + { + internal::gemm_packing_space<PackingScalar, PackingScalar> packing(size, size, nullptr, nullptr); + VERIFY_IS_EQUAL(PackingScalar::live, Index(2 * size)); + } + VERIFY_IS_EQUAL(PackingScalar::live, Index(0)); +#ifdef EIGEN_EXCEPTIONS + // Throw while constructing A's first element, A's second, B's first, and B's second; size >= 3 even when + // EIGEN_STACK_ALLOCATION_LIMIT is 0. + for (Index count : {Index(0), Index(1), Index(size), Index(size + 1)}) { + PackingScalar::remaining = count; + VERIFY((packing_throws_bad_alloc<PackingScalar, PackingScalar>(size, size))); + VERIFY_IS_EQUAL(PackingScalar::live, Index(0)); + } + const std::size_t max_bytes = (std::numeric_limits<std::ptrdiff_t>::max)(); + for (std::size_t sizeA : {max_bytes, max_bytes / 2}) + VERIFY((packing_throws_bad_alloc<char, char>(sizeA, max_bytes / 2 + 2))); +#endif +} + +// Runs product(); where malloc is wrapped, returns whether it allocated at most once. +template <typename Product> +bool allocates_at_most_once(Product product) { +#ifdef EIGEN_TEST_WRAP_MALLOC + packing_allocations = 0; + count_packing_allocations = true; +#endif + product(); +#ifdef EIGEN_TEST_WRAP_MALLOC + count_packing_allocations = false; + return packing_allocations <= 1; +#else + return true; +#endif +} + +template <typename Scalar, int Order> +void packing_products() { + Eigen::setNbThreads(1); + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + using Real = typename NumTraits<Scalar>::Real; + for (Index n : {Index(145), Index(201)}) { + const MatrixType a = MatrixType::Random(n, n + 3); + const MatrixType b = MatrixType::Random(n + 3, n + 1); + MatrixType result = MatrixType::Zero(n, n + 1); + MatrixType reference = a.lazyProduct(b); + VERIFY(allocates_at_most_once([&] { result.noalias() += a * b; })); + const Real gemm_bound = Real(8) * Real(n + 3) * NumTraits<Real>::epsilon() * a.norm() * b.norm(); + VERIFY((result - reference).norm() <= gemm_bound); + + result.setZero(n, n); + reference = a.lazyProduct(a.adjoint()); + const Real syrk_bound = Real(8) * Real(n + 3) * NumTraits<Real>::epsilon() * a.squaredNorm(); + VERIFY(allocates_at_most_once([&] { result.template selfadjointView<Lower>().rankUpdate(a); })); + reference.template triangularView<StrictlyUpper>().setZero(); + VERIFY((result - reference).norm() <= syrk_bound); + result.setZero(); + result.template selfadjointView<Upper>().rankUpdate(a.conjugate()); + reference = a.conjugate().lazyProduct(a.transpose()); + reference.template triangularView<StrictlyLower>().setZero(); + VERIFY((result - reference).norm() <= syrk_bound); + } +} + +EIGEN_DECLARE_TEST(gemm_packing) { + CALL_SUBTEST((packing_storage<float, double>())); + CALL_SUBTEST((packing_storage<double, std::complex<double>>())); + CALL_SUBTEST((packing_alignment_fallback<OverAlignedScalar, float>())); + CALL_SUBTEST((packing_alignment_fallback<float, OverAlignedScalar>())); + CALL_SUBTEST(packing_lifetimes()); + CALL_SUBTEST((packing_products<float, ColMajor>())); + CALL_SUBTEST((packing_products<double, RowMajor>())); + CALL_SUBTEST((packing_products<std::complex<double>, ColMajor>())); + CALL_SUBTEST((packing_products<std::complex<double>, RowMajor>())); +}
diff --git a/test/gemv_degenerate.cpp b/test/gemv_degenerate.cpp new file mode 100644 index 0000000..61716cc --- /dev/null +++ b/test/gemv_degenerate.cpp
@@ -0,0 +1,230 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#define EIGEN_RUNTIME_NO_MALLOC +#include "main.h" +#define EIGEN_TEST_ANNOYING_SCALAR_DONT_THROW +#include "AnnoyingScalar.h" + +template <typename Lhs, typename Rhs> +void check_degenerate_gemv(const Lhs& lhs, const Rhs& rhs) { + using Scalar = typename ScalarBinaryOpTraits<typename Lhs::Scalar, typename Rhs::Scalar>::ReturnType; + using Real = typename NumTraits<Scalar>::Real; + using Wide = std::conditional_t<NumTraits<Scalar>::IsComplex, std::complex<long double>, long double>; + using Vector = Matrix<Scalar, Dynamic, 1>; + const Scalar alpha = internal::random<Scalar>(); + const Vector initial = Vector::Random(lhs.rows()); + Vector contiguous(initial); + Vector storage = Vector::Constant(2 * lhs.rows() + 1, Scalar(7)); + Map<Vector, Unaligned, InnerStride<2>> result(storage.data(), lhs.rows()); + Vector dynamicStorage(storage); + Map<Vector, Unaligned, InnerStride<Dynamic>> dynamicResult(dynamicStorage.data(), lhs.rows(), + InnerStride<Dynamic>(2)); + for (int operation = 0; operation < 3; ++operation) { + result = initial; + contiguous = initial; + dynamicResult = initial; + auto evaluate = [&](auto& target) { + if (operation == 0) target.noalias() = alpha * lhs * rhs; + if (operation == 1) target.noalias() += alpha * lhs * rhs; + if (operation == 2) target.noalias() -= alpha * lhs * rhs; + }; + evaluate(result); + evaluate(contiguous); + evaluate(dynamicResult); + for (Index i = 0; i < lhs.rows(); ++i) { + Wide sum(0); + long double magnitude = 0; + for (Index k = 0; k < lhs.cols(); ++k) { + const Wide term = Wide(alpha) * Wide(lhs.coeff(i, k)) * Wide(rhs.coeff(k, 0)); + sum += term; + magnitude += numext::abs(term); + } + const Wide expected = operation == 0 ? sum : Wide(initial[i]) + (operation == 1 ? sum : -sum); + const long double bound = + 32 * static_cast<long double>(NumTraits<Real>::epsilon()) * (magnitude + numext::abs(Wide(initial[i]))); + VERIFY(numext::abs(Wide(result[i]) - expected) <= bound); + VERIFY(numext::abs(Wide(contiguous[i]) - expected) <= bound); + VERIFY(numext::abs(Wide(dynamicResult[i]) - expected) <= bound); + } + for (Index i = 1; i < storage.size(); i += 2) VERIFY_IS_EQUAL(storage[i], Scalar(7)); + VERIFY_IS_EQUAL(storage[storage.size() - 1], Scalar(7)); + for (Index i = 1; i < dynamicStorage.size(); i += 2) VERIFY_IS_EQUAL(dynamicStorage[i], Scalar(7)); + VERIFY_IS_EQUAL(dynamicStorage[dynamicStorage.size() - 1], Scalar(7)); + } +} + +template <typename LhsScalar, typename RhsScalar = LhsScalar> +void degenerate_gemv() { + using MatrixType = Matrix<LhsScalar, Dynamic, Dynamic>; + using RhsVector = Matrix<RhsScalar, Dynamic, 1>; + for (Index n : {0, 1, 3, 8, 17, 257}) { + for (Index depth : {0, 1, 2, 4}) { + MatrixType a = MatrixType::Random(depth, n); + RhsVector rhsStorage = RhsVector::Random(3 * depth + 1); + Map<RhsVector, Unaligned, InnerStride<3>> rhs(rhsStorage.data(), depth); + check_degenerate_gemv(a.transpose(), rhs); + check_degenerate_gemv(a.adjoint(), rhs.conjugate()); + check_degenerate_gemv(LhsScalar(0.75) * a.adjoint(), RhsScalar(0.5) * rhs); + MatrixType columnMajor = a.transpose(); + check_degenerate_gemv(columnMajor, rhs); + check_degenerate_gemv(columnMajor.conjugate(), rhs.conjugate()); + check_degenerate_gemv(LhsScalar(0.75) * columnMajor.conjugate(), RhsScalar(0.5) * rhs); + } + Matrix<LhsScalar, Dynamic, 1> lhsStorage = Matrix<LhsScalar, Dynamic, 1>::Random(3 * n + 1); + using RowMatrix = Matrix<LhsScalar, Dynamic, Dynamic, RowMajor>; + Map<RowMatrix, Unaligned, OuterStride<3>> lhs(lhsStorage.data(), n, 1); + RhsVector rhs = RhsVector::Random(1); + check_degenerate_gemv(lhs, rhs); + Map<RowMatrix, Unaligned, OuterStride<>> runtimeLhs(lhsStorage.data(), n, 1, OuterStride<>(3)); + check_degenerate_gemv(runtimeLhs, rhs); + Matrix<LhsScalar, 1, Dynamic> row = Matrix<LhsScalar, 1, Dynamic>::Random(n); + check_degenerate_gemv(row.transpose(), rhs); + Matrix<LhsScalar, 2, Dynamic> fixedDepth = Matrix<LhsScalar, 2, Dynamic>::Random(2, n); + RhsVector fixedDepthRhs = RhsVector::Random(2); + check_degenerate_gemv(fixedDepth.transpose(), fixedDepthRhs); + } +} + +template <typename Scalar, int StorageOrder> +void degenerate_gemv_zero_alpha() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, StorageOrder>; + using Vector = Matrix<Scalar, Dynamic, 1>; + using Real = typename NumTraits<Scalar>::Real; + MatrixType lhs(17, 1); + Vector rhs(1); + Vector backing(2 * lhs.rows() + 1); + Map<Vector, Unaligned, InnerStride<2>> strided(backing.data(), lhs.rows()); + Map<Vector, Unaligned, InnerStride<Dynamic>> dynamicStrided(backing.data(), lhs.rows(), InnerStride<Dynamic>(2)); + Vector contiguous(lhs.rows()); + for (const Real exceptional : {NumTraits<Real>::quiet_NaN(), NumTraits<Real>::infinity()}) { + for (int operand = 0; operand < 2; ++operand) { + lhs.setConstant(operand == 0 ? Scalar(exceptional) : Scalar(2)); + rhs.setConstant(operand == 1 ? Scalar(exceptional) : Scalar(3)); + for (const Real zero : {Real(0), -Real(0)}) { + for (int factor = 0; factor < 3; ++factor) { + for (int operation = 0; operation < 2; ++operation) { + auto check = [&](auto& destination) { + backing.setConstant(Scalar(7)); + contiguous.setConstant(Scalar(7)); + auto update = [&](const auto& product) { + if (operation == 0) + destination.noalias() += product; + else + destination.noalias() -= product; + }; + if (factor == 0) update(Scalar(zero) * (lhs * rhs)); + if (factor == 1) update((Scalar(zero) * lhs) * rhs); + if (factor == 2) update(lhs * (Scalar(zero) * rhs)); + for (Index i = 0; i < destination.size(); ++i) VERIFY_IS_EQUAL(destination[i], Scalar(7)); + for (Index i = 1; i < backing.size(); i += 2) VERIFY_IS_EQUAL(backing[i], Scalar(7)); + VERIFY_IS_EQUAL(backing[backing.size() - 1], Scalar(7)); + }; + check(contiguous); + check(strided); + check(dynamicStrided); + } + } + } + } + } +} + +template <typename Scalar, int StorageOrder> +void degenerate_gemv_array_destinations() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, StorageOrder>; + using Vector = Matrix<Scalar, Dynamic, 1>; + using ArrayVector = Array<Scalar, Dynamic, 1>; + const Index size = 17; + MatrixType lhs(size, 1); + ArrayVector expected(size); + for (Index i = 0; i < size; ++i) { + lhs(i, 0) = Scalar(2 * (i + 1)); + expected[i] = Scalar(3 * (i + 1)); + } + const Vector rhs = Vector::Constant(1, Scalar(3)); + const Scalar alpha(0.5); + const ArrayVector contiguous = alpha * lhs * rhs; + VERIFY((contiguous == expected).all()); + + ArrayVector backing = ArrayVector::Constant(2 * size + 1, Scalar(7)); + Map<ArrayVector, Unaligned, InnerStride<2>> strided(backing.data(), size); + Map<ArrayVector, Unaligned, InnerStride<Dynamic>> dynamicStrided(backing.data(), size, InnerStride<Dynamic>(2)); + auto check = [&](auto& destination) { + backing.setConstant(Scalar(7)); + destination = alpha * lhs * rhs; + VERIFY((destination == expected).all()); + for (Index i = 1; i < backing.size(); i += 2) VERIFY_IS_EQUAL(backing[i], Scalar(7)); + VERIFY_IS_EQUAL(backing[backing.size() - 1], Scalar(7)); + }; + check(strided); + check(dynamicStrided); + + const Array<Scalar, 1, Dynamic> row = alpha * rhs.transpose() * lhs.transpose(); + VERIFY((row.transpose() == expected).all()); + Array<Scalar, Dynamic, Dynamic> rows = Array<Scalar, Dynamic, Dynamic>::Constant(2, size + 2, Scalar(7)); + rows.row(1).segment(1, size) = alpha * rhs.transpose() * lhs.transpose(); + VERIFY((rows.row(1).segment(1, size).transpose() == expected).all()); + VERIFY((rows.row(0) == Scalar(7)).all()); + VERIFY_IS_EQUAL(rows(1, 0), Scalar(7)); + VERIFY_IS_EQUAL(rows(1, size + 1), Scalar(7)); +} + +EIGEN_DECLARE_TEST(gemv_degenerate) { + // The disabled C++14 dispatch must not require coefficient access or instantiate the update. + int placeholder = 0; + VERIFY((!internal::gemv_single_column<false, false, false>::run(placeholder, placeholder, placeholder, placeholder, + true))); + degenerate_gemv_array_destinations<float, RowMajor>(); + degenerate_gemv_array_destinations<float, ColMajor>(); + degenerate_gemv_array_destinations<double, RowMajor>(); + degenerate_gemv_array_destinations<double, ColMajor>(); + degenerate_gemv_array_destinations<std::complex<float>, RowMajor>(); + degenerate_gemv_array_destinations<std::complex<float>, ColMajor>(); + degenerate_gemv_array_destinations<std::complex<double>, RowMajor>(); + degenerate_gemv_array_destinations<std::complex<double>, ColMajor>(); + + degenerate_gemv_zero_alpha<float, RowMajor>(); + degenerate_gemv_zero_alpha<double, RowMajor>(); + degenerate_gemv_zero_alpha<std::complex<float>, RowMajor>(); + degenerate_gemv_zero_alpha<std::complex<double>, RowMajor>(); + degenerate_gemv_zero_alpha<float, ColMajor>(); + degenerate_gemv_zero_alpha<double, ColMajor>(); + STATIC_CHECK(!std::is_trivially_copyable<AnnoyingScalar>::value); + for (int repeat = 0; repeat < g_repeat; ++repeat) { + degenerate_gemv<float>(); + degenerate_gemv<double>(); + degenerate_gemv<half>(); + degenerate_gemv<bfloat16>(); + degenerate_gemv<std::complex<float>>(); + degenerate_gemv<std::complex<double>>(); + degenerate_gemv<double, std::complex<double>>(); + degenerate_gemv<std::complex<double>, double>(); + } + Matrix<double, Dynamic, Dynamic, RowMajor> lhs = MatrixXd::Constant(2, 1, 1e-200); + VectorXd rhs = VectorXd::Constant(1, 1e308); + VectorXd result = VectorXd::Zero(2); + result.noalias() += 2.0 * lhs * rhs; + for (Index i = 0; i < result.size(); ++i) { + VERIFY((numext::isfinite)(result[i])); + VERIFY(numext::abs(result[i] / 2e108 - 1.0) <= 8 * NumTraits<double>::epsilon()); + } + Matrix<AnnoyingScalar, Dynamic, Dynamic, RowMajor> custom = + Matrix<AnnoyingScalar, Dynamic, Dynamic, RowMajor>::Constant(5, 1, AnnoyingScalar(2)); + Matrix<AnnoyingScalar, Dynamic, 1> customRhs = Matrix<AnnoyingScalar, Dynamic, 1>::Constant(1, AnnoyingScalar(3)); + Matrix<AnnoyingScalar, Dynamic, 1> customResult(5); + customResult.noalias() = custom * customRhs; + for (Index i = 0; i < customResult.size(); ++i) VERIFY_IS_EQUAL(customResult[i], AnnoyingScalar(6)); + + MatrixXd column = MatrixXd::Constant(20000, 1, 2.0); + VectorXd factor = VectorXd::Constant(1, 3.0); + VectorXd backing = VectorXd::Zero(2 * column.rows()); + Map<VectorXd, Unaligned, InnerStride<2>> strided(backing.data(), column.rows()); + internal::set_is_malloc_allowed(false); + strided.noalias() += 0.75 * column * factor; + internal::set_is_malloc_allowed(true); + for (Index i = 0; i < column.rows(); ++i) { + VERIFY_IS_EQUAL(strided[i], 4.5); + VERIFY_IS_EQUAL(backing[2 * i + 1], 0.0); + } +}
diff --git a/test/geo_transformations.cpp b/test/geo_transformations.cpp index d762eaa..a56978f 100644 --- a/test/geo_transformations.cpp +++ b/test/geo_transformations.cpp
@@ -671,6 +671,11 @@ Rotation2D<double> r2d1d = r2d1.template cast<double>(); VERIFY_IS_APPROX(r2d1d.template cast<Scalar>(), r2d1); + Matrix2i quarter_turn; + quarter_turn << 0, -1, 1, 0; + const Scalar quarter_turn_angle = Rotation2D<Scalar>(quarter_turn).angle(); + VERIFY(numext::abs(quarter_turn_angle - Scalar(EIGEN_PI) / Scalar(2)) <= Scalar(4) * NumTraits<Scalar>::epsilon()); + for (int k = 0; k < 100; ++k) { Scalar angle = internal::random<Scalar>(-100, 100); Rotation2D<Scalar> rot2(angle); @@ -951,12 +956,34 @@ VERIFY((s0 - s1).norm() < c * eps); } +// Part of #2715: constructing or assigning from a (Dim+1)^2 matrix must respect Mode. +template <typename Scalar, int Mode, int Options> +void transform_from_full_matrix() { + using Transform3 = Transform<Scalar, 3, Mode, Options>; + using Matrix4 = Matrix<Scalar, 4, 4>; + Matrix4 m = Matrix4::Random(); + Transform3 t1(m); + Transform3 t2; + t2 = m; + Matrix4 expected = m; + if (Mode != int(Projective)) expected.row(3) << 0, 0, 0, 1; + // AffineCompact stores only the top three rows. + const Index r = t1.matrix().rows(); + VERIFY_IS_EQUAL(t1.matrix(), expected.topRows(r)); + VERIFY_IS_EQUAL(t2.matrix(), expected.topRows(r)); +} + EIGEN_DECLARE_TEST(geo_transformations) { for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST_1((transformations<double, Affine, AutoAlign>())); CALL_SUBTEST_1((non_projective_only<double, Affine, AutoAlign>())); CALL_SUBTEST_1((transformations_computed_scaling_continuity<double, Affine, AutoAlign>())); CALL_SUBTEST_1(custom_scalar_test<>()); + CALL_SUBTEST_1((transform_from_full_matrix<double, Affine, AutoAlign>())); + CALL_SUBTEST_1((transform_from_full_matrix<double, Isometry, AutoAlign>())); + CALL_SUBTEST_1((transform_from_full_matrix<double, AffineCompact, AutoAlign>())); + CALL_SUBTEST_1((transform_from_full_matrix<double, Projective, AutoAlign>())); + CALL_SUBTEST_1((transform_from_full_matrix<float, Affine, RowMajor>())); CALL_SUBTEST_2((transformations<float, AffineCompact, AutoAlign>())); CALL_SUBTEST_2((non_projective_only<float, AffineCompact, AutoAlign>()));
diff --git a/test/gpu_basic.cu b/test/gpu_basic.cu index a32c66e..4fd41a5 100644 --- a/test/gpu_basic.cu +++ b/test/gpu_basic.cu
@@ -34,6 +34,23 @@ // } // }; +template <typename MatrixType, int UpLo> +struct selfadjoint_permutation { + EIGEN_DEVICE_FUNC void operator()(int i, const typename MatrixType::Scalar* in, + typename MatrixType::Scalar* out) const { + using namespace Eigen; + MatrixType matrix = Map<const MatrixType>(in + i); + PermutationMatrix<4> permutation; + permutation.indices() << 1, 3, 0, 2; + matrix.template selfadjointView<UpLo>().applyPermutation(permutation); + Map<MatrixType>(out + 48 * i) = matrix; + matrix.template selfadjointView<UpLo>().applyPermutation(permutation.inverse()); + Map<MatrixType>(out + 48 * i + 16) = matrix; + matrix.template selfadjointView<UpLo>().applyTransposition(3, 0); + Map<MatrixType>(out + 48 * i + 32) = matrix; + } +}; + template <typename Scalar> struct scaled_permutation_product { EIGEN_DEVICE_FUNC void operator()(int i, const float* in, float* out) const { @@ -767,6 +784,64 @@ #endif } +struct bit_cast_non_literal_bits16 { + EIGEN_DEVICE_FUNC explicit bit_cast_non_literal_bits16(numext::uint16_t v = 0) : bits(v) {} + numext::uint16_t bits; +}; + +struct bit_cast_test { + EIGEN_DEVICE_FUNC void operator()(int i, const numext::uint16_t* in, numext::uint16_t* out) const { + const numext::uint16_t raw = in[i]; + const Eigen::half h = numext::bit_cast<Eigen::half>(raw); + const Eigen::bfloat16 bfloat = numext::bit_cast<Eigen::bfloat16>(h); + const Eigen::half h_from_bfloat = numext::bit_cast<Eigen::half>(bfloat); + const bit_cast_non_literal_bits16 non_literal = numext::bit_cast<bit_cast_non_literal_bits16>(h); + const Eigen::half h_from_non_literal = numext::bit_cast<Eigen::half>(non_literal); + const numext::uint32_t raw32 = (static_cast<numext::uint32_t>(raw & 0x8000u) << 16) | 0x3f800000u | + static_cast<numext::uint32_t>(raw & 0x7fffu); + const float f = numext::bit_cast<float>(raw32); + const numext::uint32_t f_bits = numext::bit_cast<numext::uint32_t>(f); + + out[6 * i + 0] = numext::bit_cast<numext::uint16_t>(bfloat); + out[6 * i + 1] = numext::bit_cast<numext::uint16_t>(h_from_bfloat); + out[6 * i + 2] = non_literal.bits; + out[6 * i + 3] = numext::bit_cast<numext::uint16_t>(h_from_non_literal); + out[6 * i + 4] = static_cast<numext::uint16_t>(f_bits & 0xffffu); + out[6 * i + 5] = static_cast<numext::uint16_t>(f_bits >> 16); + } +}; + +void test_bit_cast() { + const numext::uint16_t patterns[] = {0x0000, 0x8000, 0x0001, 0x3c00, 0x3f80, 0x7c00, 0x7e00, 0xfbff}; + constexpr int n = sizeof(patterns) / sizeof(patterns[0]); + Eigen::Array<numext::uint16_t, Eigen::Dynamic, 1> in(n), out_ref(6 * n), out_gpu(6 * n); + for (int i = 0; i < n; ++i) { + in(i) = patterns[i]; + } + out_ref.setConstant(0xa5a5); + out_gpu.setConstant(0x5a5a); + + run_on_cpu(bit_cast_test(), n, in, out_ref); + run_on_gpu(bit_cast_test(), n, in, out_gpu); + +#if !defined(EIGEN_GPU_COMPILE_PHASE) + VERIFY_IS_CWISE_EQUAL(out_ref, out_gpu); + for (int i = 0; i < n; ++i) { + const numext::uint16_t raw = patterns[i]; + const numext::uint16_t expected[6] = {raw, + raw, + raw, + raw, + static_cast<numext::uint16_t>(raw & 0x7fffu), + static_cast<numext::uint16_t>((raw & 0x8000u) | 0x3f80u)}; + for (int k = 0; k < 6; ++k) { + VERIFY_IS_EQUAL(out_ref(6 * i + k), expected[k]); + VERIFY_IS_EQUAL(out_gpu(6 * i + k), expected[k]); + } + } +#endif +} + // Eigen::half != is true when either operand is NaN, on the device as on the host. struct half_nan_not_equal_test { EIGEN_DEVICE_FUNC void operator()(int i, const float* in, float* out) const { @@ -1027,6 +1102,7 @@ CALL_SUBTEST(test_custom_less_scalar_minmax()); CALL_SUBTEST(test_float_nan_minmax()); CALL_SUBTEST(test_bit_count()); + CALL_SUBTEST(test_bit_cast()); CALL_SUBTEST(test_half_nan_not_equal()); CALL_SUBTEST(test_complex_exp_overflow()); @@ -1045,6 +1121,13 @@ CALL_SUBTEST(run_and_compare_to_gpu(selfadjoint_rank2_update<Matrix6f, Lower>(), nthreads, in, out)); CALL_SUBTEST(run_and_compare_to_gpu(selfadjoint_rank2_update<Matrix6f, Upper>(), nthreads, in, out)); + CALL_SUBTEST((run_and_compare_to_gpu(selfadjoint_permutation<Matrix4f, Lower>(), nthreads, in, out))); + CALL_SUBTEST( + (run_and_compare_to_gpu(selfadjoint_permutation<Matrix<float, 4, 4, RowMajor>, Upper>(), nthreads, in, out))); + CALL_SUBTEST((run_and_compare_to_gpu(selfadjoint_permutation<Matrix4cf, Upper>(), nthreads, cfin, cfout))); + CALL_SUBTEST((run_and_compare_to_gpu(selfadjoint_permutation<Matrix<std::complex<float>, 4, 4, RowMajor>, Lower>(), + nthreads, cfin, cfout))); + CALL_SUBTEST(run_and_compare_to_gpu(selfadjoint_l1_norm<Matrix4f, Lower>(), nthreads, in, out)); CALL_SUBTEST(run_and_compare_to_gpu(selfadjoint_l1_norm<Matrix4f, Upper>(), nthreads, in, out)); CALL_SUBTEST(run_and_compare_to_gpu(selfadjoint_l1_norm<Matrix6f, Lower>(), nthreads, in, out));
diff --git a/test/half_float.cpp b/test/half_float.cpp index 819c1e8..b6f8ac2 100644 --- a/test/half_float.cpp +++ b/test/half_float.cpp
@@ -10,6 +10,7 @@ #include <sstream> #include "main.h" +#include "product_test_helpers.h" #define VERIFY_HALF_BITS_EQUAL(h, bits) \ VERIFY_IS_EQUAL((numext::bit_cast<numext::uint16_t>(h)), (static_cast<numext::uint16_t>(bits))) @@ -360,7 +361,7 @@ } void test_product() { - typedef Matrix<half, Dynamic, Dynamic> MatrixXh; + using MatrixXh = Matrix<half, Dynamic, Dynamic>; Index rows = internal::random<Index>(1, EIGEN_TEST_MAX_SIZE); Index cols = internal::random<Index>(1, EIGEN_TEST_MAX_SIZE); Index depth = internal::random<Index>(1, EIGEN_TEST_MAX_SIZE); @@ -373,6 +374,26 @@ VERIFY_IS_APPROX(Ch.noalias() += Ah * Bh, (Cf.noalias() += Af * Bf).cast<half>()); } +// The blocked GEMM sums in float and rounds each coefficient once; the sizes keep it off the GEMV and coefficient-based +// paths, which still round in half. +void test_gemm_float_accumulation() { + using MatrixXh = Matrix<half, Dynamic, Dynamic>; + Index rows = internal::random<Index>(2, EIGEN_TEST_MAX_SIZE); + Index cols = internal::random<Index>(2, EIGEN_TEST_MAX_SIZE); + Index depth = internal::random<Index>(20, EIGEN_TEST_MAX_SIZE); + MatrixXh A = MatrixXh::Random(rows, depth); + MatrixXh B = MatrixXh::Random(depth, cols); + MatrixXh C0 = MatrixXh::Random(rows, cols); + // References in double, whose rounding error is negligible next to the bound. + MatrixXd exact = A.cast<double>().eval() * B.cast<double>().eval(); + MatrixXd exact_update = C0.cast<double>() + exact; + MatrixXh C = C0; + C.noalias() += A * B; + VERIFY(verifyProductUpdate<float>(C.cast<double>(), exact_update, half(1), A, B, C0, 1)); + C.noalias() = A * B; + VERIFY(verifyProduct<float>(C.cast<double>(), exact, A, B, 1)); +} + void test_nextafter() { VERIFY((numext::isnan)(numext::nextafter(std::numeric_limits<half>::quiet_NaN(), half(1.0f)))); VERIFY((numext::isnan)(numext::nextafter(half(1.0f), std::numeric_limits<half>::quiet_NaN()))); @@ -418,6 +439,7 @@ CALL_SUBTEST(test_trigonometric_functions()); CALL_SUBTEST(test_array()); CALL_SUBTEST(test_product()); + CALL_SUBTEST(test_gemm_float_accumulation()); CALL_SUBTEST(test_nextafter()); } }
diff --git a/test/householder.cpp b/test/householder.cpp index c3d35a3..6b79bd8 100644 --- a/test/householder.cpp +++ b/test/householder.cpp
@@ -13,6 +13,44 @@ #include "NonCommutativeScalar.h" #include <Eigen/QR> +template <int Tag> +struct HouseholderResultScalar {}; + +namespace Eigen { +template <> +struct ScalarBinaryOpTraits<HouseholderResultScalar<0>, HouseholderResultScalar<1>> { + using ReturnType = HouseholderResultScalar<2>; +}; +template <> +struct ScalarBinaryOpTraits<HouseholderResultScalar<1>, HouseholderResultScalar<0>> { + using ReturnType = HouseholderResultScalar<3>; +}; +} // namespace Eigen + +template <int Side> +void householder_ordered_result_types() { + using Vectors = Matrix<HouseholderResultScalar<0>, 2, 2>; + using Coefficients = Matrix<HouseholderResultScalar<0>, 2, 1>; + using Sequence = HouseholderSequence<Vectors, Coefficients, Side>; + using Dense = Matrix<HouseholderResultScalar<1>, 2, 2>; + using Diagonal = DiagonalMatrix<HouseholderResultScalar<1>, 2>; + using Triangular = TriangularView<Dense, Upper>; + using Left = decltype(std::declval<const Sequence&>() * std::declval<const Dense&>()); + using Right = decltype(std::declval<const Dense&>() * std::declval<const Sequence&>()); + using DiagonalLeft = decltype(std::declval<const Sequence&>() * std::declval<const Diagonal&>()); + using DiagonalRight = decltype(std::declval<const Diagonal&>() * std::declval<const Sequence&>()); + using TriangularLeft = decltype(std::declval<const Sequence&>() * std::declval<const Triangular&>()); + using TriangularRight = decltype(std::declval<const Triangular&>() * std::declval<const Sequence&>()); + static_assert(std::is_same<typename Left::Scalar, HouseholderResultScalar<2>>::value, "sequence * dense"); + static_assert(std::is_same<typename Right::Scalar, HouseholderResultScalar<3>>::value, "dense * sequence"); + static_assert(std::is_same<typename DiagonalLeft::Scalar, HouseholderResultScalar<2>>::value, "sequence * diagonal"); + static_assert(std::is_same<typename DiagonalRight::Scalar, HouseholderResultScalar<3>>::value, "diagonal * sequence"); + static_assert(std::is_same<typename TriangularLeft::Scalar, HouseholderResultScalar<2>>::value, + "sequence * triangular"); + static_assert(std::is_same<typename TriangularRight::Scalar, HouseholderResultScalar<3>>::value, + "triangular * sequence"); +} + namespace reordered_complex { template <typename Real> @@ -1225,6 +1263,8 @@ } EIGEN_DECLARE_TEST(householder) { + CALL_SUBTEST_26(householder_ordered_result_types<OnTheLeft>()); + CALL_SUBTEST_26(householder_ordered_result_types<OnTheRight>()); for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST_1(householder(Matrix<double, 2, 2>())); CALL_SUBTEST_2(householder(Matrix<float, 2, 3>()));
diff --git a/test/idrstabl.cpp b/test/idrstabl.cpp index 6fd5c3c..95b3842 100644 --- a/test/idrstabl.cpp +++ b/test/idrstabl.cpp
@@ -8,6 +8,8 @@ // with this file, You can obtain one at http://mozilla.org/MPL/2.0/. // SPDX-License-Identifier: MPL-2.0 +// Uninitialized workspace entries read by the solver show up as NaN in the solution. +#define EIGEN_INITIALIZE_MATRICES_BY_NAN #include "sparse_solver.h" #include <Eigen/IterativeLinearSolvers> @@ -24,6 +26,7 @@ } // Neither a zero right-hand side nor the direct solve for n <= S iterates; iterations() used to report the cap. +template <typename = void> void test_zero_rhs() { SparseMatrix<double> A(4, 4); for (Index i = 0; i < 4; ++i) A.insert(i, i) = 2.0; @@ -38,8 +41,28 @@ VERIFY_IS_EQUAL(solver.iterations(), Index(0)); } +// An exact zero residual with zero tolerance breaks down normalization in the first IDR step. +template <typename = void> +void test_normalization_breakdown() { + const Index n = 10; + SparseMatrix<double> A(n, n); + for (Index i = 0; i < n; ++i) A.insert(i, i) = 1.0; + const VectorXd b = VectorXd::Unit(n, 0); + for (Index L = 1; L <= 3; ++L) { + IDRSTABL<SparseMatrix<double> > solver(A); + solver.setL(L); + solver.setS(2); + solver.setTolerance(0.0); + solver.setMaxIterations(2); + VectorXd x = solver.solve(b); + VERIFY(x.allFinite()); + VERIFY_IS_EQUAL(x, b); + } +} + EIGEN_DECLARE_TEST(idrstabl) { CALL_SUBTEST_1((test_idrstabl_T<double>())); CALL_SUBTEST_2((test_idrstabl_T<std::complex<double> >())); - CALL_SUBTEST_1(test_zero_rhs()); + CALL_SUBTEST_1(test_zero_rhs<>()); + CALL_SUBTEST_1(test_normalization_breakdown<>()); }
diff --git a/test/inplace_decomposition.cpp b/test/inplace_decomposition.cpp index 144937d..b0a723b 100644 --- a/test/inplace_decomposition.cpp +++ b/test/inplace_decomposition.cpp
@@ -403,8 +403,8 @@ CALL_SUBTEST_4((inplace_fullpivlu_subspaces<Matrix<double, Dynamic, Dynamic, ColMajor>>())); CALL_SUBTEST_4((inplace_fullpivlu_subspaces<Matrix<double, Dynamic, Dynamic, RowMajor>>())); CALL_SUBTEST_4((inplace_fullpivlu_subspaces<Matrix<double, 3, 3, RowMajor | DontAlign>>())); - CALL_SUBTEST_10((inplace_special_values<MatrixXd>(2))); - CALL_SUBTEST_10((inplace_special_values<MatrixXd>(1))); + CALL_SUBTEST_19((inplace_special_values<MatrixXd>(2))); + CALL_SUBTEST_19((inplace_special_values<MatrixXd>(1))); CALL_SUBTEST_11(inplace_plain_lower_triangle<>()); CALL_SUBTEST_12((inplace_eigensolver<MatrixXd, EigenSolver>(128))); CALL_SUBTEST_12((inplace_eigensolver<MatrixXd, EigenSolver>(129))); @@ -451,31 +451,31 @@ CALL_SUBTEST_10((inplace_schur<MatrixXd, RealSchur>(size))); CALL_SUBTEST_10((inplace_schur<MatrixXd, RealSchur>(1))); CALL_SUBTEST_10((inplace_schur<Matrix4f, RealSchur>(4))); - CALL_SUBTEST_10((inplace_schur<MatrixXcd, ComplexSchur>(size))); - CALL_SUBTEST_10((inplace_schur<MatrixXcd, ComplexSchur>(1))); - CALL_SUBTEST_10((inplace_schur<Matrix4cf, ComplexSchur>(4))); - CALL_SUBTEST_10((inplace_special_values<MatrixXd>(size))); - CALL_SUBTEST_10((inplace_special_values<Matrix4f>(4))); + CALL_SUBTEST_18((inplace_schur<MatrixXcd, ComplexSchur>(size))); + CALL_SUBTEST_18((inplace_schur<MatrixXcd, ComplexSchur>(1))); + CALL_SUBTEST_18((inplace_schur<Matrix4cf, ComplexSchur>(4))); + CALL_SUBTEST_19((inplace_special_values<MatrixXd>(size))); + CALL_SUBTEST_19((inplace_special_values<Matrix4f>(4))); CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<MatrixXd>(size))); - CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<MatrixXcd>(size))); + CALL_SUBTEST_16((inplace_selfadjoint_eigensolver<MatrixXcd>(size))); CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<MatrixXd>(1))); // Fixed sizes whose columns are packet-aligned take the vectorized rotation kernel. - CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<Matrix2d>(2))); - CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<Matrix3d>(3))); - CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<Matrix4f>(4))); - CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<Matrix4d>(4))); - CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<Matrix<float, Dynamic, Dynamic, RowMajor>>(size))); + CALL_SUBTEST_15((inplace_selfadjoint_eigensolver<Matrix2d>(2))); + CALL_SUBTEST_15((inplace_selfadjoint_eigensolver<Matrix3d>(3))); + CALL_SUBTEST_17((inplace_selfadjoint_eigensolver<Matrix4f>(4))); + CALL_SUBTEST_15((inplace_selfadjoint_eigensolver<Matrix4d>(4))); + CALL_SUBTEST_17((inplace_selfadjoint_eigensolver<Matrix<float, Dynamic, Dynamic, RowMajor>>(size))); // Large enough for the blocked tridiagonalization, on strided and row-major storage. CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<MatrixXd>(100))); - CALL_SUBTEST_11((inplace_selfadjoint_eigensolver<Matrix<float, Dynamic, Dynamic, RowMajor>>(100))); + CALL_SUBTEST_17((inplace_selfadjoint_eigensolver<Matrix<float, Dynamic, Dynamic, RowMajor>>(100))); CALL_SUBTEST_12((inplace_eigensolver<MatrixXd, EigenSolver>(size))); CALL_SUBTEST_12((inplace_eigensolver<MatrixXd, EigenSolver>(1))); CALL_SUBTEST_12((inplace_eigensolver<Matrix4f, EigenSolver>(4))); - CALL_SUBTEST_12((inplace_eigensolver<MatrixXcd, ComplexEigenSolver>(size))); - CALL_SUBTEST_12((inplace_eigensolver<MatrixXcd, ComplexEigenSolver>(1))); - CALL_SUBTEST_12((inplace_eigensolver<Matrix4cf, ComplexEigenSolver>(4))); + CALL_SUBTEST_20((inplace_eigensolver<MatrixXcd, ComplexEigenSolver>(size))); + CALL_SUBTEST_20((inplace_eigensolver<MatrixXcd, ComplexEigenSolver>(1))); + CALL_SUBTEST_20((inplace_eigensolver<Matrix4cf, ComplexEigenSolver>(4))); CALL_SUBTEST_13((inplace_qz<MatrixXd, RealQZ>(size))); CALL_SUBTEST_13((inplace_generalized_eigensolver<MatrixXd>(size)));
diff --git a/test/io.cpp b/test/io.cpp index 155b253..1343022 100644 --- a/test/io.cpp +++ b/test/io.cpp
@@ -12,6 +12,97 @@ #include "main.h" +namespace stream_precision_test { +struct Rational { + int numerator; + int denominator; + Rational(int n = 0, int d = 1) : numerator(n), denominator(d) {} +}; + +struct BinaryScalar { + double value; +}; +double log2(const BinaryScalar& value) { return std::log2(value.value); } + +struct IncompleteScalar {}; +struct UnsupportedLog {}; +UnsupportedLog log2(const IncompleteScalar&) { return {}; } + +struct ExplicitPrecision {}; + +struct MetadataReal {}; +struct MetadataScalar { + double value; + MetadataScalar(double x = 0) : value(x) {} +}; +std::ostream& operator<<(std::ostream& stream, const MetadataScalar& value) { return stream << value.value; } + +std::ostream& operator<<(std::ostream& stream, const Rational& value) { + return stream << (std::to_string(value.numerator) + '/' + std::to_string(value.denominator)); +} +} // namespace stream_precision_test + +namespace std { +template <> +class numeric_limits<stream_precision_test::MetadataReal> : public numeric_limits<double> {}; +} // namespace std + +namespace Eigen { +template <> +struct NumTraits<stream_precision_test::Rational> : GenericNumTraits<stream_precision_test::Rational> {}; +template <> +struct NumTraits<stream_precision_test::BinaryScalar> : GenericNumTraits<stream_precision_test::BinaryScalar> { + static stream_precision_test::BinaryScalar epsilon() { return {1.0 / 1024}; } +}; +template <> +struct NumTraits<stream_precision_test::ExplicitPrecision> + : GenericNumTraits<stream_precision_test::ExplicitPrecision> { + static constexpr int digits() { return 11; } + static constexpr int digits10() { return 3; } + static constexpr int max_digits10() { return 5; } +}; +template <> +struct NumTraits<stream_precision_test::MetadataScalar> : GenericNumTraits<stream_precision_test::MetadataScalar> { + using Real = stream_precision_test::MetadataReal; +}; +} // namespace Eigen + +void check_custom_scalar_precision() { + using MetadataScalar = stream_precision_test::MetadataScalar; + VERIFY_IS_EQUAL(NumTraits<MetadataScalar>::digits10(), 15); + VERIFY_IS_EQUAL(NumTraits<MetadataScalar>::max_digits10(), 17); + Matrix<MetadataScalar, 1, 1> decimal; + decimal(0) = MetadataScalar(1.2345678901234567); + std::ostringstream full_precision; + full_precision << decimal.format(IOFormat(FullPrecision)); + VERIFY(full_precision.str() == "1.2345678901234567"); + VERIFY_IS_EQUAL(NumTraits<stream_precision_test::BinaryScalar>::digits(), 11); + VERIFY_IS_EQUAL(NumTraits<stream_precision_test::BinaryScalar>::digits10(), 3); + VERIFY_IS_EQUAL(NumTraits<stream_precision_test::BinaryScalar>::max_digits10(), 5); + VERIFY_IS_EQUAL(NumTraits<stream_precision_test::IncompleteScalar>::digits10(), 0); + VERIFY_IS_EQUAL(NumTraits<stream_precision_test::IncompleteScalar>::max_digits10(), 0); + VERIFY_IS_EQUAL(NumTraits<stream_precision_test::ExplicitPrecision>::digits(), 11); + VERIFY_IS_EQUAL(NumTraits<stream_precision_test::ExplicitPrecision>::digits10(), 3); + VERIFY_IS_EQUAL(NumTraits<stream_precision_test::ExplicitPrecision>::max_digits10(), 5); + VERIFY_IS_EQUAL(NumTraits<double>::max_digits10(), std::numeric_limits<double>::max_digits10); + using Scalar = stream_precision_test::Rational; + VERIFY_IS_EQUAL(NumTraits<Scalar>::digits10(), 0); + VERIFY_IS_EQUAL(NumTraits<Scalar>::max_digits10(), 0); + Matrix<Scalar, 2, 1> values; + values << Scalar(1, 2), Scalar(3, 4); + const int precisions[] = {StreamPrecision, FullPrecision, 4}; + for (int precision : precisions) { + std::ostringstream stream; + stream.precision(7); + stream << values.format(IOFormat(precision)); + VERIFY(stream.str() == "1/2\n3/4"); + VERIFY_IS_EQUAL(stream.precision(), 7); + } + std::ostringstream stream; + stream << values; + VERIFY(stream.str() == "1/2\n3/4"); +} + template <typename Scalar> struct check_ostream_impl { static void run() { @@ -49,7 +140,121 @@ check_ostream_impl<Scalar>::run(); } +static std::string format_2x2(const std::string& rowSeparator = "\n", const std::string& rowPrefix = "[", + const std::string& rowSuffix = "]", const std::string& matPrefix = "", + const std::string& matSuffix = "", int flags = 0) { + const auto matrix = (Matrix2i() << 6473, 27, 1, 22334).finished(); + std::ostringstream ss; + const IOFormat fmt(StreamPrecision, flags, ", ", rowSeparator, rowPrefix, rowSuffix, matPrefix, matSuffix); + ss << matrix.format(fmt); + return ss.str(); +} + +static void check_ioformat_standard() { + const std::string found = format_2x2(); + const std::string expected = + "[ 6473, 27]\n" + "[ 1, 22334]"; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_row_separator_newline() { + const std::string found = format_2x2("\nNext\n"); + const std::string expected = + "[ 6473, 27]\n" + "Next\n" + "[ 1, 22334]"; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_row_separator_multiple_newlines() { + const std::string found = format_2x2("\nNext\n ", "[", "]", "Matrix = [", "]"); + const std::string expected = + "Matrix = [[ 6473, 27]\n" + "Next\n" + " [ 1, 22334]]"; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_row_prefix_newline() { + const std::string found = format_2x2("\n", "[\n", "]"); + const std::string expected = + "[\n" + " 6473, 27]\n" + "[\n" + " 1, 22334]"; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_row_suffix_newline() { + const std::string found = format_2x2("\n", "[", "\n]"); + const std::string expected = + "[ 6473, 27\n" + "]\n" + "[ 1, 22334\n" + "]"; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_rows_same_line() { + const std::string found = format_2x2(", ", "[", "]", "[", "]"); + const std::string expected = "[[ 6473, 27], [ 1, 22334]]"; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_row_separator_indent() { + const std::string found = format_2x2(",\n ", "[", "]", "Matrix = [", "]"); + const std::string expected = + "Matrix = [[ 6473, 27],\n" + " [ 1, 22334]]"; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_row_prefix_only_newline() { + const std::string found = format_2x2("", "\n[", "]", "Matrix ="); + const std::string expected = + "Matrix =\n" + "[ 6473, 27]\n" + "[ 1, 22334]"; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_mat_prefix_newline() { + const std::string found = format_2x2("\n", "[", "]", "Matrix =\n [", "]."); + const std::string expected = + "Matrix =\n" + " [[ 6473, 27]\n" + " [ 1, 22334]]."; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_row_suffix_separator() { + const std::string found = format_2x2(", ", "[", "\n]", "Matrix = [", "]"); + const std::string expected = + "Matrix = [[ 6473, 27\n" + "], [ 1, 22334\n" + "]]"; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_dont_align_cols() { + const std::string found = format_2x2("\n", "[", "]", "Matrix = [", "].", DontAlignCols); + const std::string expected = + "Matrix = [[6473, 27]\n" + "[1, 22334]]."; + VERIFY_IS_EQUAL(found, expected); +} + +static void check_ioformat_carried_exceeds_indent() { + const std::string found = format_2x2("\n ---carried--- ", "[", "]", "Matrix [", "]"); + const std::string expected = + "Matrix [[ 6473, 27]\n" + " ---carried--- [ 1, 22334]]"; + VERIFY_IS_EQUAL(found, expected); +} + EIGEN_DECLARE_TEST(io) { + CALL_SUBTEST(check_custom_scalar_precision()); CALL_SUBTEST(check_ostream<bool>()); CALL_SUBTEST(check_ostream<float>()); CALL_SUBTEST(check_ostream<double>()); @@ -61,4 +266,16 @@ CALL_SUBTEST(check_ostream<Eigen::numext::uint32_t>()); CALL_SUBTEST(check_ostream<Eigen::numext::int64_t>()); CALL_SUBTEST(check_ostream<Eigen::numext::uint64_t>()); + CALL_SUBTEST(check_ioformat_standard()); + CALL_SUBTEST(check_ioformat_row_separator_newline()); + CALL_SUBTEST(check_ioformat_row_separator_multiple_newlines()); + CALL_SUBTEST(check_ioformat_row_prefix_newline()); + CALL_SUBTEST(check_ioformat_row_suffix_newline()); + CALL_SUBTEST(check_ioformat_rows_same_line()); + CALL_SUBTEST(check_ioformat_row_separator_indent()); + CALL_SUBTEST(check_ioformat_row_prefix_only_newline()); + CALL_SUBTEST(check_ioformat_mat_prefix_newline()); + CALL_SUBTEST(check_ioformat_row_suffix_separator()); + CALL_SUBTEST(check_ioformat_dont_align_cols()); + CALL_SUBTEST(check_ioformat_carried_exceeds_indent()); }
diff --git a/test/jacobi.cpp b/test/jacobi.cpp index 70868aa..8ceba1f 100644 --- a/test/jacobi.cpp +++ b/test/jacobi.cpp
@@ -53,6 +53,27 @@ b.applyOnTheRight(p, q, rot); VERIFY_IS_APPROX(b.col(p), c * a.col(p) - s * a.col(q)); VERIFY_IS_APPROX(b.col(q), numext::conj(s) * a.col(p) + numext::conj(c) * a.col(q)); + + EIGEN_IF_CONSTEXPR (RowsAtCompileTime == Dynamic) { + using Scalar = typename MatrixType::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + using Vec = Matrix<Scalar, Dynamic, 1>; + Matrix<RealScalar, Dynamic, 1> raw_x(2 * rows + 1); + Matrix<RealScalar, Dynamic, 1> raw_y(2 * rows + 1); + Map<Vec, Unaligned> ux(reinterpret_cast<Scalar*>(raw_x.data() + 1), rows); + Map<Vec, Unaligned> uy(reinterpret_cast<Scalar*>(raw_y.data() + 1), rows); + ux = a.col(p); + uy = a.col(q); + internal::apply_rotation_in_the_plane(ux, uy, rot.transpose()); + VERIFY_IS_APPROX(ux, b.col(p)); + VERIFY_IS_APPROX(uy, b.col(q)); + + ux = a.col(p); + Vec aligned_y = a.col(q); + internal::apply_rotation_in_the_plane(ux, aligned_y, rot.transpose()); + VERIFY_IS_APPROX(ux, b.col(p)); + VERIFY_IS_APPROX(aligned_y, b.col(q)); + } } } @@ -145,7 +166,7 @@ for (int mode = 0; mode < 4; ++mode) { JacobiRotation<Scalar> rotation, withoutR; Scalar r; -#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(SYCL_DEVICE_ONLY) && EIGEN_ARCH_i386_OR_x86_64 && \ +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) && EIGEN_ARCH_i386_OR_x86_64 && \ (defined(__SSE__) || defined(_M_X64) || (defined(_M_IX86_FP) && _M_IX86_FP >= 1)) { struct RestoreMxcsr { @@ -202,6 +223,81 @@ Scalar(8) * NumTraits<Scalar>::epsilon()); } +// Checks that makeGivens(p, q) is unitary and maps (p, q) to (r, 0), in long double with (P, Q) = (p, q) / scale exact. +template <typename Scalar> +void check_complex_givens(const Scalar& p, const Scalar& q, typename NumTraits<Scalar>::Real scale, + bool checkR = true) { + using RealScalar = typename NumTraits<Scalar>::Real; + using Wide = std::complex<long double>; + const long double tolerance = 16 * static_cast<long double>(NumTraits<RealScalar>::epsilon()); + const auto widen = [](const Scalar& x) { + return Wide(static_cast<long double>(numext::real(x)), static_cast<long double>(numext::imag(x))); + }; + JacobiRotation<Scalar> rotation, withoutR; + Scalar r; + rotation.makeGivens(p, q, &r); + withoutR.makeGivens(p, q); + VERIFY_IS_EQUAL(rotation.c(), withoutR.c()); + VERIFY_IS_EQUAL(rotation.s(), withoutR.s()); + VERIFY_IS_EQUAL(numext::imag(rotation.c()), RealScalar(0)); + + // J^* (P, Q) = (c P - conj(s) Q, s P + c Q) = (r / scale, 0). + const long double c = static_cast<long double>(numext::real(rotation.c())); + const Wide s = widen(rotation.s()); + const Wide P = widen(p) / static_cast<long double>(scale), Q = widen(q) / static_cast<long double>(scale); + const long double expected = std::sqrt(std::norm(P) + std::norm(Q)); + VERIFY(numext::abs(c * c + std::norm(s) - 1) <= tolerance); + VERIFY(std::abs(s * P + c * Q) <= tolerance * expected); + const Wide rotated = c * P - std::conj(s) * Q; + VERIFY(numext::abs(std::abs(rotated) - expected) <= tolerance * expected); + if (checkR) { + // r = phase * (q1 * u) is rounded twice, so allow two steps of the subnormal grid. + const long double quantum = + static_cast<long double>(std::numeric_limits<RealScalar>::denorm_min()) / static_cast<long double>(scale); + VERIFY(std::abs(widen(r) / static_cast<long double>(scale) - rotated) <= tolerance * expected + 2 * quantum); + } +} + +// Complex inputs whose components are small integers times a subnormal scale, e.g. p = -4.9e-324i, q = 2e-323i. The +// rotation must stay unitary and annihilate q although |p| and |q| carry only a few bits. +template <typename Scalar> +void jacobi_makegivens_complex_subnormal() { + using RealScalar = typename NumTraits<Scalar>::Real; + const RealScalar minimum = (std::numeric_limits<RealScalar>::min)(); + const RealScalar maximum = (std::numeric_limits<RealScalar>::max)(); + const RealScalar tiny = std::numeric_limits<RealScalar>::denorm_min(); + const int qValues[] = {-4, -1, 0, 1, 4}; + for (RealScalar scale : {minimum, RealScalar(minimum / RealScalar(16)), tiny}) { + for (int pr = -2; pr <= 2; ++pr) { + for (int pi = -2; pi <= 2; ++pi) { + for (int qr : qValues) { + for (int qi : qValues) { + if (pr == 0 && pi == 0 && qr == 0 && qi == 0) continue; + check_complex_givens(Scalar(RealScalar(pr) * scale, RealScalar(pi) * scale), + Scalar(RealScalar(qr) * scale, RealScalar(qi) * scale), scale); + } + } + } + } + } + + // Mixed scales: a subnormal p with a normal q, and |p| < |q| with |r| > max, where r overflows. + const int units[][2] = {{1, 0}, {0, 1}, {-1, 0}, {0, -1}}; + for (const auto& up : units) { + for (const auto& uq : units) { + const Scalar q = Scalar(RealScalar(uq[0]), RealScalar(uq[1])); + check_complex_givens(Scalar(RealScalar(up[0] - up[1]) * tiny, RealScalar(up[0] + up[1]) * tiny), q, + RealScalar(1)); + check_complex_givens(Scalar(RealScalar(up[0]) * (maximum / 2), RealScalar(up[1]) * (maximum / 2)), q * maximum, + maximum, false); + } + } + + // p = 0 where |Re q| + |Im q| overflows although |q| is finite. + const RealScalar big = maximum / RealScalar(1.5); + for (const Scalar q : {Scalar(big, big), Scalar(-big, big)}) check_complex_givens(Scalar(0), q, big); +} + template <typename Scalar> EIGEN_DONT_INLINE void jacobi_makejacobi_bounded_ratio_impl(Scalar x, Scalar y, JacobiRotation<Scalar>& rotation) { rotation.makeJacobi(x, y, Scalar(0)); @@ -352,6 +448,8 @@ CALL_SUBTEST_7((jacobi_makejacobi_ratio_boundaries<bfloat16>())); CALL_SUBTEST_8((jacobi_makejacobi_complex<std::complex<float>>())); CALL_SUBTEST_8((jacobi_makejacobi_complex<std::complex<double>>())); + CALL_SUBTEST_8((jacobi_makegivens_complex_subnormal<std::complex<float>>())); + CALL_SUBTEST_8((jacobi_makegivens_complex_subnormal<std::complex<double>>())); CALL_SUBTEST_1((jacobi<Matrix3f, float>())); CALL_SUBTEST_2((jacobi<Matrix4d, double>()));
diff --git a/test/jacobisvd.cpp b/test/jacobisvd.cpp index 5e4ca87..7fe6648 100644 --- a/test/jacobisvd.cpp +++ b/test/jacobisvd.cpp
@@ -368,9 +368,9 @@ CALL_SUBTEST_63((svd_precondition_square_matrix_accuracy<MatrixXcd>(12))); CALL_SUBTEST_64((svd_precondition_square_matrix_runtime_options<MatrixXf>(12))); CALL_SUBTEST_64((svd_precondition_square_matrix_runtime_options<MatrixXcd>(r))); - CALL_SUBTEST_64((jacobisvd_fullpiv_runtime_thin_asserts<0>(8, 5))); - CALL_SUBTEST_64((jacobisvd_fullpiv_runtime_thin_asserts<PreconditionSquareMatrix>(6, 6))); - CALL_SUBTEST_64((jacobisvd_thin_full_options<Matrix<double, Dynamic, Dynamic, RowMajor>>( + CALL_SUBTEST_68((jacobisvd_fullpiv_runtime_thin_asserts<0>(8, 5))); + CALL_SUBTEST_68((jacobisvd_fullpiv_runtime_thin_asserts<PreconditionSquareMatrix>(6, 6))); + CALL_SUBTEST_69((jacobisvd_thin_full_options<Matrix<double, Dynamic, Dynamic, RowMajor>>( Matrix<double, Dynamic, Dynamic, RowMajor>(8, 8)))); MatrixXcd noQRTest = MatrixXcd(r, r); @@ -424,18 +424,18 @@ CALL_SUBTEST_56(svd_underoverflow<void>()); CALL_SUBTEST_56(jacobisvd_power_of_two_scaling<>()); - CALL_SUBTEST_56((jacobisvd_flushed_subnormal_matrix<MatrixXf>(5, 4))); - CALL_SUBTEST_56((jacobisvd_flushed_subnormal_matrix<MatrixXd>(4, 5))); - CALL_SUBTEST_56((jacobisvd_flushed_subnormal_matrix<MatrixXcf>(4, 4))); - CALL_SUBTEST_56((jacobisvd_flushed_subnormal_matrix<MatrixXcd>(3, 5))); - CALL_SUBTEST_56(jacobisvd_subnormal_diagonals<float>()); - CALL_SUBTEST_56(jacobisvd_subnormal_diagonals<double>()); - CALL_SUBTEST_56((svd_zero_matrix_solvers<Matrix3d>(3, 3))); - CALL_SUBTEST_56((svd_zero_matrix_solvers<Matrix<float, 2, 5>>(2, 5))); - CALL_SUBTEST_56((svd_zero_matrix_solvers<MatrixXcf>(5, 3))); - CALL_SUBTEST_56((svd_zero_matrix_solvers<MatrixXd>(0, 0))); - CALL_SUBTEST_56((svd_zero_matrix_solvers<MatrixXd>(0, 3))); - CALL_SUBTEST_56((svd_zero_matrix_solvers<MatrixXd>(3, 0))); + CALL_SUBTEST_65((jacobisvd_flushed_subnormal_matrix<MatrixXf>(5, 4))); + CALL_SUBTEST_65((jacobisvd_flushed_subnormal_matrix<MatrixXd>(4, 5))); + CALL_SUBTEST_65((jacobisvd_flushed_subnormal_matrix<MatrixXcf>(4, 4))); + CALL_SUBTEST_65((jacobisvd_flushed_subnormal_matrix<MatrixXcd>(3, 5))); + CALL_SUBTEST_66(jacobisvd_subnormal_diagonals<float>()); + CALL_SUBTEST_66(jacobisvd_subnormal_diagonals<double>()); + CALL_SUBTEST_67((svd_zero_matrix_solvers<Matrix3d>(3, 3))); + CALL_SUBTEST_67((svd_zero_matrix_solvers<Matrix<float, 2, 5>>(2, 5))); + CALL_SUBTEST_67((svd_zero_matrix_solvers<MatrixXcf>(5, 3))); + CALL_SUBTEST_67((svd_zero_matrix_solvers<MatrixXd>(0, 0))); + CALL_SUBTEST_67((svd_zero_matrix_solvers<MatrixXd>(0, 3))); + CALL_SUBTEST_67((svd_zero_matrix_solvers<MatrixXd>(3, 0))); // Check that the TriangularBase constructor works CALL_SUBTEST_57((svd_triangular_matrix<Matrix3d>()));
diff --git a/test/mapped_matrix.cpp b/test/mapped_matrix.cpp index a921c21..6b19dc3 100644 --- a/test/mapped_matrix.cpp +++ b/test/mapped_matrix.cpp
@@ -217,7 +217,7 @@ typedef Matrix<Scalar, Dynamic, Dynamic> Mat; // Test various inner dimensions around packet size const Index inner_sizes[] = {1, PS - 1, PS, PS + 1, 2 * PS, 2 * PS + 1}; - const Index outer_stride = 64; // large enough for any inner size + const Index outer_stride = numext::maxi<Index>(64, 2 * PS + 1); // large enough for any inner size const Index cols = 4; for (int si = 0; si < 6; ++si) {
diff --git a/test/memory_reallocation.cpp b/test/memory_reallocation.cpp new file mode 100644 index 0000000..753c742 --- /dev/null +++ b/test/memory_reallocation.cpp
@@ -0,0 +1,280 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include "main.h" +#include <Eigen/SparseCore> + +#include <memory> + +#ifdef EIGEN_TEST_WRAP_MALLOC +static int allocations_before_failure = -1; +extern "C" void* real_malloc(std::size_t) asm("__real_malloc"); +extern "C" void* real_realloc(void*, std::size_t) asm("__real_realloc"); +extern "C" void* wrap_malloc(std::size_t) asm("__wrap_malloc"); +extern "C" void* wrap_realloc(void*, std::size_t) asm("__wrap_realloc"); + +static bool fail_allocation() { + if (allocations_before_failure < 0) return false; + return allocations_before_failure-- == 0; +} + +extern "C" void* wrap_malloc(std::size_t size) { return fail_allocation() ? nullptr : real_malloc(size); } +extern "C" void* wrap_realloc(void* ptr, std::size_t size) { + return fail_allocation() ? nullptr : real_realloc(ptr, size); +} +#endif + +struct ReallocationScalar { + struct ConstructionFailure {}; + + static int live_objects; + static int live_resources; + static int defaults_before_throw; + static int moves_before_throw; + + std::unique_ptr<int> value; + + ReallocationScalar() { +#ifdef EIGEN_EXCEPTIONS + if (defaults_before_throw == 0) throw ConstructionFailure{}; +#endif + if (defaults_before_throw > 0) --defaults_before_throw; + value.reset(new int(0)); + ++live_objects; + ++live_resources; + } + + ReallocationScalar(ReallocationScalar&& other) { +#ifdef EIGEN_EXCEPTIONS + if (moves_before_throw == 0) throw ConstructionFailure{}; +#endif + if (moves_before_throw > 0) --moves_before_throw; + value = std::move(other.value); + ++live_objects; + } + + ReallocationScalar& operator=(const ReallocationScalar& other) { + *value = *other.value; + return *this; + } + + ~ReallocationScalar() { + if (value) --live_resources; + --live_objects; + } + + static void reset_failures() { + defaults_before_throw = -1; + moves_before_throw = -1; + } +}; + +int ReallocationScalar::live_objects = 0; +int ReallocationScalar::live_resources = 0; +int ReallocationScalar::defaults_before_throw = -1; +int ReallocationScalar::moves_before_throw = -1; + +template <bool Align> +void reallocation_success() { + using Scalar = ReallocationScalar; + Scalar::reset_failures(); + Scalar* values = internal::conditional_aligned_new<Scalar, Align>(3); + for (int i = 0; i < 3; ++i) *values[i].value = i + 1; + for (int size : {5, 5, 2, 0}) { + const int old_size = Scalar::live_objects; + values = internal::conditional_aligned_realloc_new<Scalar, Align>(values, size, old_size); + VERIFY_IS_EQUAL(Scalar::live_objects, size); + VERIFY_IS_EQUAL(Scalar::live_resources, size); + for (int i = 0; i < size; ++i) VERIFY_IS_EQUAL(*values[i].value, i < 3 ? i + 1 : 0); + } + internal::conditional_aligned_delete<Scalar, Align>(values, 0); +} + +#ifdef EIGEN_EXCEPTIONS +template <bool Align> +void reallocation_failure(bool fail_move, int successful_constructions, int old_size) { + using Scalar = ReallocationScalar; + Scalar::reset_failures(); + Scalar* values = internal::conditional_aligned_new<Scalar, Align>(old_size); + for (int i = 0; i < old_size; ++i) *values[i].value = i + 1; + if (fail_move) + Scalar::moves_before_throw = successful_constructions; + else + Scalar::defaults_before_throw = successful_constructions; + + bool caught = false; + try { + values = internal::conditional_aligned_realloc_new<Scalar, Align>(values, old_size + 2, old_size); + } catch (const Scalar::ConstructionFailure&) { + caught = true; + } + Scalar::reset_failures(); + VERIFY(caught); + VERIFY_IS_EQUAL(Scalar::live_objects, old_size); + // Only a failing move can leave the original elements moved-from. + VERIFY_IS_EQUAL(Scalar::live_resources, fail_move ? old_size - successful_constructions : old_size); + if (!fail_move) { + for (int i = 0; i < old_size; ++i) VERIFY_IS_EQUAL(*values[i].value, i + 1); + } + internal::conditional_aligned_delete<Scalar, Align>(values, old_size); + VERIFY_IS_EQUAL(Scalar::live_objects, 0); + VERIFY_IS_EQUAL(Scalar::live_resources, 0); +} + +void sparse_reserve_failure() { + using Scalar = ReallocationScalar; + for (int successful_defaults : {0, 1}) { + Scalar::reset_failures(); + { + SparseVector<Scalar> vector(10); + vector.reserve(3); + VERIFY_IS_EQUAL(Scalar::live_objects, 3); + Scalar::defaults_before_throw = successful_defaults; + bool caught = false; + try { + vector.reserve(5); + } catch (const Scalar::ConstructionFailure&) { + caught = true; + } + Scalar::reset_failures(); + VERIFY(caught); + VERIFY_IS_EQUAL(vector.size(), 10); + VERIFY_IS_EQUAL(vector.nonZeros(), 0); + VERIFY_IS_EQUAL(Scalar::live_objects, 3); + VERIFY_IS_EQUAL(Scalar::live_resources, 3); + } + VERIFY_IS_EQUAL(Scalar::live_objects, 0); + VERIFY_IS_EQUAL(Scalar::live_resources, 0); + } +} + +#ifdef EIGEN_TEST_WRAP_MALLOC +void compressed_storage_trivial_allocation_failure() { + for (int operation : {0, 1, 2}) { + for (int successful_allocations : {0, 1}) { + internal::CompressedStorage<double, int> storage(3); + for (int i = 0; i < 3; ++i) { + storage.index(i) = 2 * i; + storage.value(i) = double(i + 1); + } + if (operation == 1) storage.reserve(2); + const Index capacity = storage.allocatedSize(); + const auto update = [&]() { + if (operation == 0) + storage.reserve(2); + else if (operation == 1) + storage.squeeze(); + else + storage.atWithInsertion(1, 7.0); + }; + bool caught = false; + allocations_before_failure = successful_allocations; + try { + update(); + } catch (const std::bad_alloc&) { + caught = true; + } + allocations_before_failure = -1; + VERIFY(caught); + VERIFY_IS_EQUAL(storage.size(), 3); + VERIFY_IS_EQUAL(storage.allocatedSize(), operation == 1 && successful_allocations == 1 ? 3 : capacity); + for (int i = 0; i < 3; ++i) { + VERIFY_IS_EQUAL(storage.index(i), 2 * i); + VERIFY_IS_EQUAL(storage.value(i), double(i + 1)); + } + update(); + VERIFY_IS_EQUAL(storage.size(), operation == 2 ? 4 : 3); + if (operation == 2) { + VERIFY_IS_EQUAL(storage.index(1), 1); + VERIFY_IS_EQUAL(storage.value(1), 7.0); + } + storage.reserve(4); + storage.squeeze(); + } + } +} + +void compressed_storage_allocation_failure() { + using Scalar = ReallocationScalar; + // Exercise reserve growth, squeeze shrinkage, and insertion growth. + for (int operation : {0, 1, 2}) { + for (int successful_allocations : {0, 1}) { + Scalar::reset_failures(); + { + internal::CompressedStorage<Scalar, int> storage(3); + for (int i = 0; i < 3; ++i) { + storage.index(i) = 2 * i; + *storage.value(i).value = i + 1; + } + if (operation == 1) storage.reserve(2); + const Index capacity = storage.allocatedSize(); + const Scalar* values = storage.valuePtr(); + const int* indices = storage.indexPtr(); + Scalar inserted; + *inserted.value = 7; + const auto update = [&]() { + if (operation == 0) + storage.reserve(2); + else if (operation == 1) + storage.squeeze(); + else + storage.atWithInsertion(1, inserted); + }; + bool caught = false; + allocations_before_failure = successful_allocations; + try { + update(); + } catch (const std::bad_alloc&) { + caught = true; + } + allocations_before_failure = -1; + VERIFY(caught); + VERIFY_IS_EQUAL(storage.size(), 3); + VERIFY_IS_EQUAL(storage.allocatedSize(), capacity); + VERIFY(storage.valuePtr() == values); + VERIFY(storage.indexPtr() == indices); + VERIFY_IS_EQUAL(Scalar::live_objects, capacity + 1); + VERIFY_IS_EQUAL(Scalar::live_resources, capacity + 1); + for (int i = 0; i < 3; ++i) { + VERIFY_IS_EQUAL(storage.index(i), 2 * i); + VERIFY_IS_EQUAL(*storage.value(i).value, i + 1); + } + update(); + VERIFY_IS_EQUAL(storage.size(), operation == 2 ? 4 : 3); + if (operation == 2) { + VERIFY_IS_EQUAL(storage.index(1), 1); + VERIFY_IS_EQUAL(*storage.value(1).value, 7); + } + storage.clear(); + storage.squeeze(); + VERIFY_IS_EQUAL(storage.allocatedSize(), 0); + VERIFY_IS_EQUAL(Scalar::live_objects, 1); + VERIFY_IS_EQUAL(Scalar::live_resources, 1); + } + VERIFY_IS_EQUAL(Scalar::live_objects, 0); + VERIFY_IS_EQUAL(Scalar::live_resources, 0); + } + } +} +#endif +#endif + +EIGEN_DECLARE_TEST(memory_reallocation) { + CALL_SUBTEST(reallocation_success<false>()); + CALL_SUBTEST(reallocation_success<true>()); +#ifdef EIGEN_EXCEPTIONS +#ifdef EIGEN_TEST_WRAP_MALLOC + CALL_SUBTEST(compressed_storage_allocation_failure()); + CALL_SUBTEST(compressed_storage_trivial_allocation_failure()); +#endif + CALL_SUBTEST(sparse_reserve_failure()); + for (int successful_constructions : {0, 1}) { + CALL_SUBTEST(reallocation_failure<false>(false, successful_constructions, 3)); + CALL_SUBTEST(reallocation_failure<true>(false, successful_constructions, 3)); + CALL_SUBTEST(reallocation_failure<false>(true, successful_constructions, 3)); + CALL_SUBTEST(reallocation_failure<true>(true, successful_constructions, 3)); + CALL_SUBTEST(reallocation_failure<false>(false, successful_constructions, 0)); + CALL_SUBTEST(reallocation_failure<true>(false, successful_constructions, 0)); + } +#endif +}
diff --git a/test/meta.cpp b/test/meta.cpp index 4656c6a..17e26ce 100644 --- a/test/meta.cpp +++ b/test/meta.cpp
@@ -9,6 +9,7 @@ // SPDX-License-Identifier: MPL-2.0 #include "main.h" +#include "NonCommutativeScalar.h" #include <array> #include <Eigen/src/Core/util/Meta.h> @@ -17,6 +18,16 @@ typedef int ReturnType; }; +struct CommutativeScalarOptIn {}; +struct UnannotatedScalar {}; +namespace Eigen { +template <> +struct ScalarBinaryOpTraits<CommutativeScalarOptIn, CommutativeScalarOptIn> { + using ReturnType = CommutativeScalarOptIn; + static constexpr bool IsCommutative = true; +}; +} // namespace Eigen + using Eigen::internal::apply_op_from_left; using Eigen::internal::apply_op_from_right; using Eigen::internal::arg_prod; @@ -253,6 +264,19 @@ } EIGEN_DECLARE_TEST(meta) { + using Quaternion = noncommutative_scalar::Quaternion; + STATIC_CHECK(NumTraits<Quaternion>::IsComplex); + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<double>>::value)); + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<std::complex<double>>>::value)); + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<double, float>>::value)); + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<double, std::complex<double>>>::value)); + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<std::complex<double>, double>>::value)); + STATIC_CHECK((!internal::functor_is_commutative<internal::scalar_product_op<Quaternion>>::value)); + STATIC_CHECK((!internal::functor_is_commutative<internal::scalar_product_op<Quaternion, double>>::value)); + STATIC_CHECK((!internal::functor_is_commutative<internal::scalar_product_op<double, Quaternion>>::value)); + STATIC_CHECK((internal::functor_is_commutative<internal::scalar_product_op<CommutativeScalarOptIn>>::value)); + STATIC_CHECK((!internal::functor_is_commutative<internal::scalar_product_op<UnannotatedScalar>>::value)); + VERIFY((std::is_same<float, float>::value)); VERIFY((!std::is_same<float, double>::value)); VERIFY((!std::is_same<float, float&>::value));
diff --git a/test/numext.cpp b/test/numext.cpp index 91cecd1..624bbcf 100644 --- a/test/numext.cpp +++ b/test/numext.cpp
@@ -9,6 +9,13 @@ // SPDX-License-Identifier: MPL-2.0 #include "main.h" +#include <Eigen/Cholesky> +#include <Eigen/Eigenvalues> +#include <Eigen/Geometry> +#include <Eigen/LU> +#include <Eigen/QR> +#include <Eigen/SVD> +#include <Eigen/Sparse> #include "twoprod_helpers.h" #include "fp_control.h" @@ -16,6 +23,436 @@ #include <cfenv> #endif +namespace log10_adl_test { +struct Scalar { + double value; +}; +inline Scalar log10(Scalar x) { return {std::log10(x.value)}; } +} // namespace log10_adl_test + +namespace numext_dispatch_test { +// Only numext supplies math for this scalar; generic packet fallbacks must not bypass it. +struct Scalar { + double value; +}; +} // namespace numext_dispatch_test + +namespace Eigen { +namespace numext { +template <> +inline numext_dispatch_test::Scalar sin(const numext_dispatch_test::Scalar& x) { + return {std::sin(x.value)}; +} +template <> +inline numext_dispatch_test::Scalar acosh(const numext_dispatch_test::Scalar& x) { + return {std::acosh(x.value)}; +} +template <> +inline numext_dispatch_test::Scalar atanh(const numext_dispatch_test::Scalar& x) { + return {std::atanh(x.value)}; +} +} // namespace numext +} // namespace Eigen + +void check_generic_packet_numext_dispatch() { + const numext_dispatch_test::Scalar x{0.25}, y{2.0}; + VERIFY_IS_EQUAL(internal::psin(x).value, std::sin(x.value)); + VERIFY_IS_EQUAL(internal::pacosh(y).value, std::acosh(y.value)); + VERIFY_IS_EQUAL(internal::patanh(x).value, std::atanh(x.value)); +} + +namespace adl_template_math_test { +struct Scalar { + double v; + Scalar() : v(0.0) {} + template <typename U, std::enable_if_t<std::is_arithmetic<U>::value, int> = 0> + Scalar(U x) : v(static_cast<double>(x)) {} + explicit operator double() const { return v; } + Scalar operator-() const { return Scalar(-v); } + friend Scalar operator+(const Scalar& a, const Scalar& b) { return Scalar(a.v + b.v); } + friend Scalar operator-(const Scalar& a, const Scalar& b) { return Scalar(a.v - b.v); } + friend Scalar operator*(const Scalar& a, const Scalar& b) { return Scalar(a.v * b.v); } + friend Scalar operator/(const Scalar& a, const Scalar& b) { return Scalar(a.v / b.v); } + Scalar& operator+=(const Scalar& b) { + v += b.v; + return *this; + } + Scalar& operator-=(const Scalar& b) { + v -= b.v; + return *this; + } + Scalar& operator*=(const Scalar& b) { + v *= b.v; + return *this; + } + Scalar& operator/=(const Scalar& b) { + v /= b.v; + return *this; + } + friend bool operator==(const Scalar& a, const Scalar& b) { return a.v == b.v; } + friend bool operator!=(const Scalar& a, const Scalar& b) { return a.v != b.v; } + friend bool operator<(const Scalar& a, const Scalar& b) { return a.v < b.v; } + friend bool operator<=(const Scalar& a, const Scalar& b) { return a.v <= b.v; } + friend bool operator>(const Scalar& a, const Scalar& b) { return a.v > b.v; } + friend bool operator>=(const Scalar& a, const Scalar& b) { return a.v >= b.v; } +}; + +// Unconstrained function templates in the scalar's namespace (like CGAL::abs / CGAL::sqrt). +// Unqualified calls after `using numext::...;` would be ambiguous with Eigen::numext templates. +template <typename T> +inline T abs(const T& x) { + return Scalar(std::abs(x.v)); +} +template <typename T> +inline T sqrt(const T& x) { + return Scalar(std::sqrt(x.v)); +} +template <typename T> +inline T sin(const T& x) { + return Scalar(std::sin(x.v)); +} +template <typename T> +inline T cos(const T& x) { + return Scalar(std::cos(x.v)); +} +template <typename T> +inline T acos(const T& x) { + return Scalar(std::acos(x.v)); +} +template <typename T> +inline T atan2(const T& y, const T& x) { + return Scalar(std::atan2(y.v, x.v)); +} +template <typename T> +inline T log(const T& x) { + return Scalar(std::log(x.v)); +} +template <typename T> +inline T ldexp(const T& x, int exp) { + return Scalar(std::ldexp(x.v, exp)); +} +// Not templates: the UCRT declares unconstrained isfinite, isnan and isinf templates. +inline bool(isfinite)(const Scalar& x) { return (std::isfinite)(x.v); } +inline bool(isnan)(const Scalar& x) { return (std::isnan)(x.v); } +inline bool(isinf)(const Scalar& x) { return (std::isinf)(x.v); } +} // namespace adl_template_math_test + +namespace adl_non_template_math_test { +struct Scalar { + double v = 0.0; + Scalar() = default; + Scalar(double x) : v(x) {} + Scalar operator-() const { return Scalar(-v); } + friend Scalar operator+(const Scalar& a, const Scalar& b) { return Scalar(a.v + b.v); } + friend Scalar operator-(const Scalar& a, const Scalar& b) { return Scalar(a.v - b.v); } + friend Scalar operator*(const Scalar& a, const Scalar& b) { return Scalar(a.v * b.v); } + friend Scalar operator/(const Scalar& a, const Scalar& b) { return Scalar(a.v / b.v); } + Scalar& operator+=(const Scalar& b) { + v += b.v; + return *this; + } + Scalar& operator-=(const Scalar& b) { + v -= b.v; + return *this; + } + Scalar& operator*=(const Scalar& b) { + v *= b.v; + return *this; + } + Scalar& operator/=(const Scalar& b) { + v /= b.v; + return *this; + } + friend bool operator==(const Scalar& a, const Scalar& b) { return a.v == b.v; } + friend bool operator!=(const Scalar& a, const Scalar& b) { return a.v != b.v; } + friend bool operator<(const Scalar& a, const Scalar& b) { return a.v < b.v; } + friend bool operator<=(const Scalar& a, const Scalar& b) { return a.v <= b.v; } + friend bool operator>(const Scalar& a, const Scalar& b) { return a.v > b.v; } + friend bool operator>=(const Scalar& a, const Scalar& b) { return a.v >= b.v; } +}; + +inline Scalar abs(const Scalar& x) { return Scalar(std::abs(x.v)); } +inline Scalar sqrt(const Scalar& x) { return Scalar(std::sqrt(x.v)); } +} // namespace adl_non_template_math_test + +namespace std { +template <> +class numeric_limits<adl_template_math_test::Scalar> : public numeric_limits<double> {}; +} // namespace std + +namespace Eigen { +template <> +struct NumTraits<adl_template_math_test::Scalar> : GenericNumTraits<adl_template_math_test::Scalar> { + static inline Real epsilon() { return Real(NumTraits<double>::epsilon()); } + static inline Real dummy_precision() { return Real(NumTraits<double>::dummy_precision()); } + static inline Real highest() { return Real(NumTraits<double>::highest()); } + static inline Real lowest() { return Real(NumTraits<double>::lowest()); } +}; + +// Intentionally omits std::numeric_limits and IsSigned so GenericNumTraits::IsSigned defaults to 0. +template <> +struct NumTraits<adl_non_template_math_test::Scalar> : GenericNumTraits<adl_non_template_math_test::Scalar> { + static inline Real epsilon() { return Real(NumTraits<double>::epsilon()); } + static inline Real dummy_precision() { return Real(NumTraits<double>::dummy_precision()); } + static inline Real highest() { return Real(NumTraits<double>::highest()); } + static inline Real lowest() { return Real(NumTraits<double>::lowest()); } +}; +} // namespace Eigen + +void check_adl_template_math_dispatch() { + using S = adl_template_math_test::Scalar; + using CS = std::complex<S>; + using Mat2 = Matrix<S, 2, 2>; + using Mat3 = Matrix<S, 3, 3>; + using Mat4 = Matrix<S, 4, 4>; + using MatX = Matrix<S, Dynamic, Dynamic>; + using CMat2 = Matrix<CS, 2, 2>; + using Vec2 = Matrix<S, 2, 1>; + using Vec3 = Matrix<S, 3, 1>; + using Vec4 = Matrix<S, 4, 1>; + + const double kTol = 1e-11; + auto max_abs = [](const auto& m) { return m.cwiseAbs().maxCoeff().v; }; + + Mat2 a; + a << S(4.0), S(1.0), S(1.0), S(3.0); + Vec2 b(S(1.0), S(2.0)); + const Mat2 a_upd = a + b * b.transpose(); + + LDLT<Mat2> ldlt(a); + ldlt.rankUpdate(b, S(1.0)); + Vec2 x_ldlt = ldlt.solve(b); + VERIFY_LE(max_abs(ldlt.reconstructedMatrix() - a_upd), kTol); + VERIFY_LE(max_abs(a_upd * x_ldlt - b), kTol); + + LLT<Mat2> llt(a); + llt.rankUpdate(b, S(1.0)); + VERIFY_LE(max_abs(llt.reconstructedMatrix() - a_upd), kTol); + VERIFY_LE(max_abs(a_upd * llt.solve(b) - b), kTol); + + BunchKaufman<Mat2> bk(a); + Vec2 x_bk = bk.solve(b); + VERIFY_LE(max_abs(a * x_bk - b), kTol); + + FullPivLU<Mat2> lu(a); + VERIFY_IS_EQUAL(lu.rank(), 2); + Vec2 x_lu = lu.solve(b); + Mat2 a_inv = a.inverse(); + VERIFY_LE(max_abs(a * x_lu - b), kTol); + VERIFY_LE(max_abs(a * a_inv - Mat2::Identity()), kTol); + + HouseholderQR<Mat2> hqr(a); + ColPivHouseholderQR<Mat2> cpqr(a); + FullPivHouseholderQR<Mat2> fpqr(a); + VERIFY_IS_APPROX(hqr.absDeterminant().v, 11.0); + VERIFY_IS_APPROX(cpqr.absDeterminant().v, 11.0); + VERIFY_IS_APPROX(fpqr.absDeterminant().v, 11.0); + VERIFY_IS_EQUAL(cpqr.rank(), 2); + VERIFY_IS_EQUAL(fpqr.rank(), 2); + VERIFY_LE(max_abs(a * hqr.solve(b) - b), kTol); + VERIFY_LE(max_abs(a * cpqr.solve(b) - b), kTol); + VERIFY_LE(max_abs(a * fpqr.solve(b) - b), kTol); + + JacobiSVD<Mat2, ComputeFullU | ComputeFullV> jsvd(a); + VERIFY_IS_EQUAL(jsvd.rank(), 2); + VERIFY_LE(max_abs(jsvd.matrixU() * jsvd.singularValues().asDiagonal() * jsvd.matrixV().transpose() - a), kTol); + VERIFY_LE(max_abs(a * jsvd.solve(b) - b), kTol); + + Mat4 ns4; + ns4 << S(0.0), S(2.0), S(-3.0), S(1.0), S(-4.0), S(1.0), S(2.0), S(-1.0), S(1.0), S(-7.0), S(0.5), S(2.0), S(3.0), + S(1.0), S(-2.0), S(-6.0); + Vec4 b4(S(1.0), S(-2.0), S(3.0), S(-1.0)); + const Mat4 spd4 = ns4.transpose() * ns4 + Mat4::Identity(); + + const MatX xns4 = ns4; + BDCSVD<MatX, ComputeFullU | ComputeFullV> bdcsvd(xns4); + VERIFY_IS_EQUAL(bdcsvd.rank(), 4); + VERIFY_LE(max_abs(bdcsvd.matrixU() * bdcsvd.singularValues().asDiagonal() * bdcsvd.matrixV().transpose() - xns4), + kTol); + + SelfAdjointEigenSolver<Mat2> eig2; + eig2.computeDirect(a); + VERIFY_IS_EQUAL(eig2.info(), Success); + VERIFY_LE(max_abs(eig2.eigenvectors() * eig2.eigenvalues().asDiagonal() * eig2.eigenvectors().transpose() - a), kTol); + + Mat3 spd3; + spd3 << S(5.0), S(1.0), S(0.5), S(1.0), S(4.0), S(-0.5), S(0.5), S(-0.5), S(3.0); + SelfAdjointEigenSolver<Mat3> eig3; + eig3.computeDirect(spd3); + VERIFY_IS_EQUAL(eig3.info(), Success); + VERIFY_LE(max_abs(eig3.eigenvectors() * eig3.eigenvalues().asDiagonal() * eig3.eigenvectors().transpose() - spd3), + kTol); + + SelfAdjointEigenSolver<MatX> eigx{MatX(spd4)}; + VERIFY_IS_EQUAL(eigx.info(), Success); + VERIFY_LE( + max_abs(eigx.eigenvectors() * eigx.eigenvalues().asDiagonal() * eigx.eigenvectors().transpose() - MatX(spd4)), + kTol); + + Tridiagonalization<Mat4> tridiag(spd4); + VERIFY_LE(max_abs(tridiag.matrixQ() * tridiag.matrixT() * tridiag.matrixQ().transpose() - spd4), kTol); + + RealSchur<Mat4> rschur(ns4); + VERIFY_IS_EQUAL(rschur.info(), Success); + VERIFY_LE(max_abs(rschur.matrixU() * rschur.matrixT() * rschur.matrixU().transpose() - ns4), kTol); + + RealQZ<Mat4> rqz(ns4, spd4); + VERIFY_IS_EQUAL(rqz.info(), Success); + VERIFY_LE(max_abs(rqz.matrixQ() * rqz.matrixS() * rqz.matrixZ() - ns4), kTol); + VERIFY_LE(max_abs(rqz.matrixQ() * rqz.matrixT() * rqz.matrixZ() - spd4), kTol); + + using PlainS = adl_non_template_math_test::Scalar; + Matrix<PlainS, 4, 4> pns4; + pns4 << PlainS(0.0), PlainS(2.0), PlainS(-3.0), PlainS(1.0), PlainS(-4.0), PlainS(1.0), PlainS(2.0), PlainS(-1.0), + PlainS(1.0), PlainS(-7.0), PlainS(0.5), PlainS(2.0), PlainS(3.0), PlainS(1.0), PlainS(-2.0), PlainS(-6.0); + Matrix<PlainS, 4, 1> pb4(PlainS(1.0), PlainS(-2.0), PlainS(3.0), PlainS(-1.0)); + const Matrix<PlainS, 4, 4> pspd4 = pns4.transpose() * pns4 + Matrix<PlainS, 4, 4>::Identity(); + + HouseholderQR<Matrix<PlainS, 4, 4>> phqr(pns4); + VERIFY_IS_APPROX(phqr.absDeterminant().v, 469.5); + FullPivLU<Matrix<PlainS, 4, 4>> plu(pns4); + VERIFY_IS_EQUAL(plu.rank(), 4); + VERIFY_LE(max_abs(pns4 * plu.solve(pb4) - pb4), kTol); + ColPivHouseholderQR<Matrix<PlainS, 4, 4>> pcpqr(pns4); + FullPivHouseholderQR<Matrix<PlainS, 4, 4>> pfpqr(pns4); + VERIFY_IS_EQUAL(pcpqr.rank(), 4); + VERIFY_IS_EQUAL(pfpqr.rank(), 4); + RealSchur<Matrix<PlainS, 4, 4>> prschur(pns4); + VERIFY_IS_EQUAL(prschur.info(), Success); + VERIFY_LE(max_abs(prschur.matrixU() * prschur.matrixT() * prschur.matrixU().transpose() - pns4), kTol); + SelfAdjointEigenSolver<Matrix<PlainS, 4, 4>> peig4(pspd4); + VERIFY_IS_EQUAL(peig4.info(), Success); + VERIFY_LE(max_abs(peig4.eigenvectors() * peig4.eigenvalues().asDiagonal() * peig4.eigenvectors().transpose() - pspd4), + kTol); + JacobiSVD<Matrix<PlainS, 4, 4>, ComputeFullU | ComputeFullV> pjsvd(pns4); + VERIFY_LE(max_abs(pjsvd.matrixU() * pjsvd.singularValues().asDiagonal() * pjsvd.matrixV().transpose() - pns4), kTol); + +#ifndef _MSC_VER // MSVC's std::complex<T> needs unary + and a conversion to double from T. + EigenSolver<Mat4> esolver(ns4); + VERIFY_IS_EQUAL(esolver.info(), Success); + VERIFY_LE( + max_abs((ns4.cast<CS>() * esolver.eigenvectors() - esolver.eigenvectors() * esolver.eigenvalues().asDiagonal()) + .cwiseAbs()), + kTol); + + GeneralizedEigenSolver<Mat4> gesolver(ns4, spd4); + VERIFY_IS_EQUAL(gesolver.info(), Success); + + ComplexSchur<Mat2> cschur_real(a); + VERIFY_IS_EQUAL(cschur_real.info(), Success); + VERIFY_LE( + max_abs( + (cschur_real.matrixU() * cschur_real.matrixT() * cschur_real.matrixU().adjoint() - a.cast<CS>()).cwiseAbs()), + kTol); + + CMat2 ca; + ca << CS(S(2.0), S(1.0)), CS(S(0.5), S(-0.5)), CS(S(0.5), S(0.5)), CS(S(3.0), S(-1.0)); + ComplexSchur<CMat2> cschur_cplx(ca); + VERIFY_IS_EQUAL(cschur_cplx.info(), Success); + VERIFY_LE(max_abs((cschur_cplx.matrixU() * cschur_cplx.matrixT() * cschur_cplx.matrixU().adjoint() - ca).cwiseAbs()), + kTol); + + using PlainCS = std::complex<PlainS>; + ComplexSchur<Matrix<PlainS, 4, 4>> plain_cschur_real(pns4); + VERIFY_IS_EQUAL(plain_cschur_real.info(), Success); + VERIFY_LE(max_abs((plain_cschur_real.matrixU() * plain_cschur_real.matrixT() * plain_cschur_real.matrixU().adjoint() - + pns4.cast<PlainCS>()) + .cwiseAbs()), + kTol); + + Matrix<PlainCS, 2, 2> pca; + pca << PlainCS(PlainS(2.0), PlainS(1.0)), PlainCS(PlainS(0.5), PlainS(-0.5)), PlainCS(PlainS(0.5), PlainS(0.5)), + PlainCS(PlainS(3.0), PlainS(-1.0)); + ComplexSchur<Matrix<PlainCS, 2, 2>> plain_cschur_cplx(pca); + VERIFY_IS_EQUAL(plain_cschur_cplx.info(), Success); + VERIFY_LE( + max_abs((plain_cschur_cplx.matrixU() * plain_cschur_cplx.matrixT() * plain_cschur_cplx.matrixU().adjoint() - pca) + .cwiseAbs()), + kTol); + + const PlainCS csqrt_neg = numext::sqrt(PlainCS(PlainS(-3.0), PlainS(-4.0))); + VERIFY_IS_APPROX(csqrt_neg.real().v, 1.0); + VERIFY_IS_APPROX(csqrt_neg.imag().v, -2.0); + + // Pin fallback behavior on negative zero: `< Scalar(0)` cannot see -0, so sqrt(-1 - 0i) yields +i. + const PlainCS csqrt_neg_zero = numext::sqrt(PlainCS(PlainS(-1.0), PlainS(-0.0))); + VERIFY_IS_APPROX(csqrt_neg_zero.real().v, 0.0); + VERIFY_IS_APPROX(csqrt_neg_zero.imag().v, 1.0); +#endif + + VERIFY_IS_EQUAL(numext::copysign(PlainS(2.0), PlainS(-1.0)).v, -2.0); + VERIFY_IS_EQUAL(numext::copysign(PlainS(-2.0), PlainS(1.0)).v, 2.0); + // Pin the `< Scalar(0)` fallback limitation on signed zero and NaN when copysign is unavailable. + VERIFY((std::signbit)(numext::copysign(PlainS(-0.0), PlainS(1.0)).v)); + VERIFY(!(std::signbit)(numext::copysign(PlainS(1.0), PlainS(-0.0)).v)); + VERIFY_IS_EQUAL(numext::copysign(PlainS(1.0), PlainS(-0.0)).v, 1.0); + VERIFY_IS_EQUAL(numext::copysign(PlainS(1.0), PlainS(-std::numeric_limits<double>::quiet_NaN())).v, 1.0); + + Quaternion<S> q(AngleAxis<S>(S(0.5), Vec3::UnitZ())); + AngleAxis<S> aa; + aa = q; + VERIFY_IS_APPROX(aa.angle().v, 0.5); + Rotation2D<S> rot2(S(0.25)); + Rotation2D<S> rot2_from_mat(rot2.toRotationMatrix()); + VERIFY_IS_APPROX(rot2_from_mat.angle().v, 0.25); + AlignedBox<S, 3> box(Vec3(S(0.0), S(0.0), S(0.0)), Vec3(S(1.0), S(1.0), S(1.0))); + VERIFY_IS_APPROX(box.exteriorDistance(Vec3(S(4.0), S(5.0), S(1.0))).v, 5.0); + ParametrizedLine<S, 3> line = ParametrizedLine<S, 3>::Through(Vec3::Zero(), Vec3(S(3.0), S(4.0), S(0.0))); + VERIFY_IS_APPROX(line.distance(Vec3(S(0.0), S(0.0), S(2.0))).v, 2.0); + VERIFY_IS_APPROX(b.stableNorm().v, std::sqrt(5.0)); + + SparseMatrix<S> sp_spd = spd4.sparseView(); + sp_spd.makeCompressed(); + SparseMatrix<S> sp_ns = ns4.sparseView(); + sp_ns.makeCompressed(); + SparseMatrix<S> sp_pruned = (sp_spd * sp_spd).pruned(S(1e-6)); + VERIFY(sp_pruned.nonZeros() > Index(0)); + + SimplicialLLT<SparseMatrix<S>> sllt(sp_spd); + VERIFY_IS_EQUAL(sllt.info(), Success); + VERIFY_LE(max_abs(spd4 * sllt.solve(b4) - b4), kTol); + + SimplicialLDLT<SparseMatrix<S>> sldlt(sp_spd); + VERIFY_IS_EQUAL(sldlt.info(), Success); + VERIFY_LE(max_abs(spd4 * sldlt.solve(b4) - b4), kTol); + + SparseLU<SparseMatrix<S>> splu(sp_ns); + VERIFY_IS_EQUAL(splu.info(), Success); + VERIFY_LE(max_abs(ns4 * splu.solve(b4) - b4), kTol); + + SparseQR<SparseMatrix<S>, COLAMDOrdering<int>> spqr(sp_ns); + VERIFY_IS_EQUAL(spqr.info(), Success); + VERIFY_LE(max_abs(ns4 * spqr.solve(b4) - b4), kTol); + + IncompleteCholesky<S> ichol(sp_spd); + VERIFY_IS_EQUAL(ichol.info(), Success); + ConjugateGradient<SparseMatrix<S>, Lower | Upper, IncompleteCholesky<S>> cg(sp_spd); + VERIFY_LE(max_abs(spd4 * cg.solve(b4) - b4), 1e-8); + + IncompleteLUT<S> ilut(sp_ns); + VERIFY_IS_EQUAL(ilut.info(), Success); + BiCGSTAB<SparseMatrix<S>, IncompleteLUT<S>> bicg(sp_ns); + VERIFY_LE(max_abs(ns4 * bicg.solve(b4) - b4), 1e-8); + + LeastSquaresConjugateGradient<SparseMatrix<S>> lscg(sp_ns); + VERIFY_LE(max_abs(ns4 * lscg.solve(b4) - b4), 1e-8); + + GMRES<SparseMatrix<S>> gmres(sp_ns); + VERIFY_LE(max_abs(ns4 * gmres.solve(b4) - b4), 1e-8); + + MINRES<SparseMatrix<S>> minres(sp_spd); + VERIFY_LE(max_abs(spd4 * minres.solve(b4) - b4), 1e-8); + + IDRS<SparseMatrix<S>> idrs(sp_ns); + VERIFY_LE(max_abs(ns4 * idrs.solve(b4) - b4), 1e-8); + + IDRSTABL<SparseMatrix<S>> idrstabl(sp_ns); + VERIFY_LE(max_abs(ns4 * idrstabl.solve(b4) - b4), 1e-8); + + LSMR<SparseMatrix<S>> lsmr(sp_ns); + VERIFY_LE(max_abs(ns4 * lsmr.solve(b4) - b4), 1e-8); +} + template <typename T, typename U> bool check_if_equal_or_nans(const T& actual, const U& expected) { return (numext::equal_strict(actual, expected) || ((numext::isnan)(actual) && (numext::isnan)(expected))); @@ -986,7 +1423,7 @@ std::numeric_limits<Scalar>::quiet_NaN()}; const int count = sizeof(values) / sizeof(values[0]); for (int mode = 0; mode < 4; ++mode) { -#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(SYCL_DEVICE_ONLY) && EIGEN_ARCH_i386_OR_x86_64 && \ +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) && EIGEN_ARCH_i386_OR_x86_64 && \ defined(_MM_SET_DENORMALS_ZERO_MODE) ScopedFlushToZero restore; const unsigned mask = _MM_FLUSH_ZERO_MASK | _MM_DENORMALS_ZERO_MASK; @@ -1070,7 +1507,7 @@ ScopedFlushToZero flush; if (!flush.isSupported()) return; check_binary_sign_no_flush_values<Scalar>(); -#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(SYCL_DEVICE_ONLY) && EIGEN_ARCH_i386_OR_x86_64 && \ +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) && EIGEN_ARCH_i386_OR_x86_64 && \ defined(_MM_SET_DENORMALS_ZERO_MODE) _MM_SET_DENORMALS_ZERO_MODE(_MM_DENORMALS_ZERO_ON); check_binary_sign_no_flush_values<Scalar>(); @@ -1120,7 +1557,26 @@ } } +template <typename Scalar> +void check_low_precision_math_rounding() { + for (unsigned int bits = 0; bits < 65536; ++bits) { + const Scalar x = numext::bit_cast<Scalar>(numext::uint16_t(bits)); + const Scalar actual[] = {numext::tanh(x), numext::sqrt(x), numext::log2(x)}; + const Scalar expected[] = {Scalar(std::tanh(float(x))), Scalar(std::sqrt(float(x))), Scalar(std::log2(float(x)))}; + for (int operation = 0; operation < 3; ++operation) { + if ((numext::isnan)(expected[operation])) { + VERIFY((numext::isnan)(actual[operation])); + } else { + VERIFY_IS_EQUAL(numext::bit_cast<numext::uint16_t>(actual[operation]), + numext::bit_cast<numext::uint16_t>(expected[operation])); + } + } + } +} + void check_scalar_tanh_saturation() { + check_low_precision_math_rounding<half>(); + check_low_precision_math_rounding<bfloat16>(); VERIFY_IS_EQUAL(internal::ptanh_float(20.0f), 1.0f); VERIFY_IS_EQUAL(internal::ptanh_float(-20.0f), -1.0f); VERIFY_IS_EQUAL(internal::ptanh_double(20.0), 1.0); @@ -1129,6 +1585,59 @@ VERIFY_IS_EQUAL(numext::tanh(-20.0f), -1.0f); } +namespace bit_cast_test_detail { +struct literal_pair16 { + numext::uint16_t low; + numext::uint16_t high; +}; + +struct non_literal_bits16 { + explicit non_literal_bits16(numext::uint16_t v = 0) : bits(v) {} + numext::uint16_t bits; +}; +} // namespace bit_cast_test_detail + +void check_bit_cast() { +#if (defined(__cpp_lib_bit_cast) && __cpp_lib_bit_cast >= 201806L) || EIGEN_HAS_BUILTIN(__builtin_bit_cast) + static_assert(numext::bit_cast<numext::uint32_t>(1.0f) == 0x3f800000u, "constexpr float -> uint32 bit_cast"); + static_assert(numext::bit_cast<numext::uint32_t>(numext::bit_cast<float>(0xbf800000u)) == 0xbf800000u, + "constexpr uint32 -> float bit_cast"); + constexpr bit_cast_test_detail::literal_pair16 kPair = {/*low=*/0x1234u, /*high=*/0x5678u}; + static_assert( + numext::bit_cast<bit_cast_test_detail::literal_pair16>(numext::bit_cast<numext::uint32_t>(kPair)).low == + 0x1234u && + numext::bit_cast<bit_cast_test_detail::literal_pair16>(numext::bit_cast<numext::uint32_t>(kPair)).high == + 0x5678u, + "constexpr struct <-> uint32 bit_cast"); +#endif + + const numext::uint16_t patterns16[] = {0x0000u, 0x8000u, 0x0001u, 0x3c00u, 0x3f80u, 0x7c00u, 0x7e2au, 0xfbffu}; + for (numext::uint16_t raw : patterns16) { + const half h = numext::bit_cast<half>(raw); + const bfloat16 bfloat = numext::bit_cast<bfloat16>(h); + const half h_roundtrip = numext::bit_cast<half>(bfloat); + VERIFY_IS_EQUAL(numext::bit_cast<numext::uint16_t>(bfloat), raw); + VERIFY_IS_EQUAL(numext::bit_cast<numext::uint16_t>(h_roundtrip), raw); + + const bit_cast_test_detail::non_literal_bits16 non_literal = + numext::bit_cast<bit_cast_test_detail::non_literal_bits16>(h); + VERIFY_IS_EQUAL(non_literal.bits, raw); + VERIFY_IS_EQUAL(numext::bit_cast<numext::uint16_t>(numext::bit_cast<half>(non_literal)), raw); + VERIFY_IS_EQUAL(numext::bit_cast<numext::uint16_t>(numext::bit_cast<bfloat16>(non_literal)), raw); + } + + const numext::uint32_t patterns32[] = {0x00000000u, 0x80000000u, 0x3f800000u, 0x7f800000u, 0x7fc01234u}; + for (numext::uint32_t raw : patterns32) { + VERIFY_IS_EQUAL(numext::bit_cast<numext::uint32_t>(numext::bit_cast<float>(raw)), raw); + } + + const numext::uint64_t patterns64[] = {0x0000000000000000ull, 0x8000000000000000ull, 0x3ff0000000000000ull, + 0x7ff0000000000000ull, 0x7ff8000012345678ull}; + for (numext::uint64_t raw : patterns64) { + VERIFY_IS_EQUAL(numext::bit_cast<numext::uint64_t>(numext::bit_cast<double>(raw)), raw); + } +} + EIGEN_DECLARE_TEST(numext) { for (int k = 0; k < g_repeat; ++k) { CALL_SUBTEST(check_copysign<half>()); @@ -1304,5 +1813,10 @@ CALL_SUBTEST(check_shift<uint64_t>()); CALL_SUBTEST(check_scalar_tanh_saturation()); + CALL_SUBTEST(check_bit_cast()); + CALL_SUBTEST(check_generic_packet_numext_dispatch()); + CALL_SUBTEST(check_adl_template_math_dispatch()); + VERIFY_IS_EQUAL(numext::log10(log10_adl_test::Scalar{10.0}).value, 1.0); + VERIFY_IS_APPROX(numext::log10(std::complex<double>(2.0, 3.0)), std::log10(std::complex<double>(2.0, 3.0))); } }
diff --git a/test/optimization_barrier.cpp b/test/optimization_barrier.cpp new file mode 100644 index 0000000..d88c8b9 --- /dev/null +++ b/test/optimization_barrier.cpp
@@ -0,0 +1,139 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include "main.h" +#include <Eigen/Core> +#ifdef __ARM_FEATURE_SVE +#include <arm_sve.h> +#endif + +enum class BarrierEnum : unsigned long long { Value = 0xfedcba9876543210ull }; +struct BarrierRecord { + unsigned long long value; + bool operator==(const BarrierRecord& other) const { return value == other.value; } +}; +union BarrierUnion { + unsigned long long value; + double real; + bool operator==(const BarrierUnion& other) const { return value == other.value; } +}; +// GCC rejects register and general asm constraints for this struct on every target. +struct BarrierTriple { + double x, y, z; + bool operator==(const BarrierTriple& other) const { return x == other.x && y == other.y && z == other.z; } +}; + +template <typename T> +void check_barrier_value(T value) { + const T expected = value; + EIGEN_OPTIMIZATION_BARRIER(value); + VERIFY(value == expected); + internal::pbarrier(value); + VERIFY(value == expected); +} + +template <typename Scalar> +EIGEN_DONT_INLINE Scalar rounded_by_barrier(Scalar x) { + // A constant bias: MSVC /fp:fast folds (x + bias) - bias only when it can see the constant. + constexpr Scalar bias = Scalar(1ull << (NumTraits<Scalar>::digits() - 1)); + Scalar rounded = x + bias; + EIGEN_OPTIMIZATION_BARRIER(rounded); + return rounded - bias; +} + +EIGEN_DECLARE_TEST(optimization_barrier) { + check_barrier_value(false); + check_barrier_value(static_cast<signed char>(-17)); + check_barrier_value(static_cast<unsigned short>(65521)); + check_barrier_value(-1234567); + check_barrier_value(0xfedcba9876543210ull); + check_barrier_value(BarrierEnum::Value); + check_barrier_value(nullptr); + check_barrier_value(1.25f); + check_barrier_value(1.25); + check_barrier_value(1.25L); + int pointee = 7; + check_barrier_value(&pointee); + volatile decltype(nullptr) null_value = nullptr; + EIGEN_OPTIMIZATION_BARRIER(null_value); + VERIFY(null_value == nullptr); + volatile double real_value = 1.25; + EIGEN_OPTIMIZATION_BARRIER(real_value); + VERIFY(real_value == 1.25); + + check_barrier_value(BarrierRecord{0xfedcba9876543210ull}); + check_barrier_value(BarrierUnion{0xfedcba9876543210ull}); + check_barrier_value(BarrierTriple{1.25, -2.5, 3.75}); + check_barrier_value(std::complex<double>(1.25, -2.5)); + int array[] = {1, -2, 3, -4}; + EIGEN_OPTIMIZATION_BARRIER(array); + for (int i = 0; i < 4; ++i) VERIFY_IS_EQUAL(array[i], (i % 2 ? -1 : 1) * (i + 1)); + double wide_array[8] = {1, -2, 3, -4, 5, -6, 7, -8}; + EIGEN_OPTIMIZATION_BARRIER(wide_array); + for (int i = 0; i < 8; ++i) VERIFY_IS_EQUAL(wide_array[i], (i % 2 ? -1 : 1) * (i + 1)); + internal::eigen_packet_wrapper<int, 7> wrapped(-1234567); + internal::pbarrier(wrapped); + VERIFY_IS_EQUAL(static_cast<int&>(wrapped), -1234567); +#ifdef __ARM_FEATURE_SVE + // Sizeless: sizeof(svfloat64_t) is ill-formed, and there is no operator== for check_barrier_value. + svfloat64_t sizeless = svdup_n_f64(1.25); + EIGEN_OPTIMIZATION_BARRIER(sizeless); + VERIFY(!svptest_any(svptrue_b64(), svcmpne_n_f64(svptrue_b64(), sizeless, 1.25))); + internal::pbarrier(sizeless); + VERIFY(!svptest_any(svptrue_b64(), svcmpne_n_f64(svptrue_b64(), sizeless, 1.25))); +#endif + +#if !defined(EIGEN_GPU_COMPILE_PHASE) && !defined(__SYCL_DEVICE_ONLY__) && \ + (EIGEN_COMP_GNUC || (defined(__clang__) && defined(_MSC_VER)) || (EIGEN_COMP_MSVC && defined(_M_FP_FAST))) + VERIFY_IS_EQUAL(rounded_by_barrier(1.25f), 1.0f); + VERIFY_IS_EQUAL(rounded_by_barrier(1.25), 1.0); +#endif + +#if !defined(EIGEN_GPU_COMPILE_PHASE) && (EIGEN_COMP_GNUC_STRICT || EIGEN_COMP_CLANG || EIGEN_COMP_MINGW) && \ + (EIGEN_ARCH_PPC || EIGEN_ARCH_ARM_OR_ARM64 || EIGEN_ARCH_i386_OR_x86_64) + using internal::optimization_barrier_in_memory; + STATIC_CHECK(optimization_barrier_in_memory<BarrierRecord>::value); + STATIC_CHECK(optimization_barrier_in_memory<BarrierUnion>::value); + STATIC_CHECK(optimization_barrier_in_memory<BarrierTriple>::value); + STATIC_CHECK(optimization_barrier_in_memory<std::complex<double>>::value); + STATIC_CHECK(optimization_barrier_in_memory<Eigen::half>::value); + STATIC_CHECK(optimization_barrier_in_memory<int[4]>::value); + STATIC_CHECK(optimization_barrier_in_memory<double[8]>::value); + STATIC_CHECK(!optimization_barrier_in_memory<double>::value); + STATIC_CHECK(!optimization_barrier_in_memory<long double>::value); + STATIC_CHECK(!optimization_barrier_in_memory<int*>::value); + STATIC_CHECK(!optimization_barrier_in_memory<decltype(nullptr)>::value); + STATIC_CHECK(!optimization_barrier_in_memory<BarrierEnum>::value); +#endif + +#if !defined(EIGEN_GPU_COMPILE_PHASE) && (EIGEN_COMP_GNUC_STRICT || EIGEN_COMP_CLANG || EIGEN_COMP_MINGW) && \ + (EIGEN_ARCH_ARM_OR_ARM64 || EIGEN_ARCH_i386_OR_x86_64) + using internal::optimization_barrier_in_register; +#if (EIGEN_ARCH_ARM_OR_ARM64 && defined(__ARM_FP) && (__ARM_FP & 4)) || (EIGEN_ARCH_i386_OR_x86_64 && defined(__SSE2__)) + STATIC_CHECK(optimization_barrier_in_register<float>::value); +#endif +#if (EIGEN_ARCH_ARM_OR_ARM64 && defined(__ARM_FP) && (__ARM_FP & 8)) || (EIGEN_ARCH_i386_OR_x86_64 && defined(__SSE2__)) + STATIC_CHECK(optimization_barrier_in_register<double>::value); +#endif + STATIC_CHECK(!optimization_barrier_in_register<long double>::value); + STATIC_CHECK(!optimization_barrier_in_register<int>::value); + STATIC_CHECK(!optimization_barrier_in_register<int*>::value); + STATIC_CHECK(!optimization_barrier_in_register<decltype(nullptr)>::value); + STATIC_CHECK(!optimization_barrier_in_register<BarrierEnum>::value); + STATIC_CHECK(!optimization_barrier_in_register<BarrierRecord>::value); + STATIC_CHECK(!optimization_barrier_in_register<BarrierUnion>::value); + STATIC_CHECK(!optimization_barrier_in_register<Eigen::half>::value); + STATIC_CHECK(!optimization_barrier_in_register<Eigen::bfloat16>::value); + STATIC_CHECK(!optimization_barrier_in_register<std::complex<double>>::value); + STATIC_CHECK(!optimization_barrier_in_register<int[4]>::value); +#ifdef __ARM_FEATURE_SVE + STATIC_CHECK(!optimization_barrier_in_register<svfloat64_t>::value); +#endif +#if (EIGEN_ARCH_i386_OR_x86_64 && defined(__SSE2__)) || defined(__ARM_NEON) + using BarrierVector = float __attribute__((vector_size(16))); + using WideBarrierVector = float __attribute__((vector_size(128))); + STATIC_CHECK(optimization_barrier_in_register<BarrierVector>::value); + STATIC_CHECK(!optimization_barrier_in_register<WideBarrierVector>::value); +#endif +#endif +}
diff --git a/test/packet_segment.cpp b/test/packet_segment.cpp index 3c16766..47a2d76 100644 --- a/test/packet_segment.cpp +++ b/test/packet_segment.cpp
@@ -150,11 +150,26 @@ } } } + static void verify_invalid_range(Index begin, Index count) { + VectorX<Scalar> data = VectorX<Scalar>::Zero(PacketSize); + const Packet a = internal::pset1<Packet>(Scalar(0)); + VERIFY_RAISES_ASSERT(internal::ploaduSegment<Packet>(data.data(), begin, count)); + VERIFY_RAISES_ASSERT((internal::pstoreuSegment<Scalar, Packet>(data.data(), a, begin, count))); + } + static void test_invalid_ranges() { + // Every specialization asserts 0 <= begin, 0 <= count, begin + count <= PacketSize. + verify_invalid_range(-1, 1); + verify_invalid_range(0, -1); + verify_invalid_range(0, PacketSize + 1); + verify_invalid_range(1, PacketSize); + verify_invalid_range(PacketSize + 1, 0); + } static void run() { test_unaligned(); test_aligned(); test_all_ranges(); test_zero_fill(); + test_invalid_ranges(); } }; @@ -197,6 +212,14 @@ } EIGEN_DECLARE_TEST(packet_segment) { +#if defined(EIGEN_VECTORIZE_AVX) && !defined(EIGEN_VECTORIZE_AVX512VL) + std::complex<float> data[2] = {}; + VERIFY_RAISES_ASSERT(internal::ploaduSegment<internal::Packet2cf>(data, 0, 3)); + VERIFY_RAISES_ASSERT(internal::ploaduSegment<internal::Packet2cf>(data, 0, -1)); + VERIFY_RAISES_ASSERT(internal::ploaduSegment<internal::Packet2cf>(data, -1, 0)); + VERIFY_RAISES_ASSERT(internal::ploaduSegment<internal::Packet2cf>(data, 3, 0)); + VERIFY_RAISES_ASSERT(internal::ploaduSegment<internal::Packet2cf>(data, (std::numeric_limits<Index>::max)(), 1)); +#endif for (int i = 0; i < g_repeat; i++) { test_packet_segment<bool>(); test_packet_segment<int8_t>();
diff --git a/test/packetmath.cpp b/test/packetmath.cpp index 33b8f83..7ee2610 100644 --- a/test/packetmath.cpp +++ b/test/packetmath.cpp
@@ -736,7 +736,7 @@ } }; -// Ensure optimization barrier compiles and doesn't modify contents. +// Ensure the macro and packet optimization barriers preserve every lane. // Only applies to raw types, so will not work for std::complex, Eigen::half // or Eigen::bfloat16. For those you would need to refer to an underlying // storage element. @@ -750,11 +750,18 @@ Packet, std::enable_if_t<!NumTraits<Packet>::IsComplex && !std::is_same<Packet, Eigen::half>::value && !std::is_same<Packet, Eigen::bfloat16>::value>> { static void run() { - typedef typename internal::unpacket_traits<Packet>::type Scalar; - Scalar s = internal::random<Scalar>(); - Packet barrier = internal::pset1<Packet>(s); + using Scalar = typename internal::unpacket_traits<Packet>::type; + constexpr int PacketSize = internal::unpacket_traits<Packet>::size; + Scalar input[PacketSize], output[PacketSize]; + for (int i = 0; i < PacketSize; ++i) input[i] = internal::random<Scalar>(); + Packet barrier = internal::ploadu<Packet>(input); EIGEN_OPTIMIZATION_BARRIER(barrier); - eigen_assert(s == internal::pfirst(barrier) && "EIGEN_OPTIMIZATION_BARRIER"); + internal::pstoreu(output, barrier); + for (int i = 0; i < PacketSize; ++i) VERIFY_IS_EQUAL(input[i], output[i]); + // Deduced, as in the kernels, so wrapped packets take the unwrapping overload. + internal::pbarrier(barrier); + internal::pstoreu(output, barrier); + for (int i = 0; i < PacketSize; ++i) VERIFY_IS_EQUAL(input[i], output[i]); } }; @@ -1074,6 +1081,57 @@ return numext::abs(actual - expected) <= RealScalar(8) * NumTraits<RealScalar>::epsilon() * numext::abs(expected); } +// phypot_complex, the modulus of plog, is within 0.5 + O(u) ulp of sqrt(a^2 + b^2): correctly rounded except within +// O(u) ulp of a rounding midpoint. The test allows 0.5 + 2^-8 ulp in the wider type W, whose own error is below 2^-8 +// ulp when it has 8 more significand bits. The larger component falls in either lane, so the test covers the order of +// the components and, without FMA, the exact residuals. +template <typename Scalar, typename Packet, typename W, std::enable_if_t<!internal::is_scalar<Packet>::value, int> = 0> +void complex_hypot_test() { + using RealScalar = typename Scalar::value_type; + constexpr int digits = std::numeric_limits<RealScalar>::digits; + if (std::numeric_limits<W>::digits < digits + 8) return; + const int PacketSize = internal::unpacket_traits<Packet>::size; + EIGEN_ALIGN_TO_BOUNDARY(unpacket_traits<Packet>::alignment) Scalar data1[PacketSize]; + EIGEN_ALIGN_TO_BOUNDARY(unpacket_traits<Packet>::alignment) Scalar data2[PacketSize]; + // The last case is a = b^2 with b the smallest odd integer whose square has ulp 1: sqrt(a^2 + a) = a + 1/2 - 1/(8a) + // + ..., which the result rounds up to a + 1. That is 0.5 + 2^-26 ulp off for float, within the bound but not the + // correctly rounded value. + RealScalar b = std::ceil(std::sqrt(std::ldexp(RealScalar(1), digits - 1))); + if (std::fmod(b, RealScalar(2)) == RealScalar(0)) b += RealScalar(1); + const Scalar cases[] = {Scalar(RealScalar(3), RealScalar(-4)), + Scalar(RealScalar(-4), RealScalar(3)), + Scalar(RealScalar(0), RealScalar(-2)), + Scalar(RealScalar(2), RealScalar(0)), + Scalar(RealScalar(5), RealScalar(5)), + Scalar(RealScalar(1), RealScalar(-1)), + Scalar(b * b, b)}; + constexpr int num_cases = 7; + for (int trial = 0; trial < 1024; ++trial) { + for (int i = 0; i < PacketSize; ++i) { + if (trial < num_cases) { + data1[i] = cases[trial]; + continue; + } + const RealScalar re = + std::ldexp(internal::random<RealScalar>(RealScalar(-4), RealScalar(4)), internal::random<int>(-20, 20)); + const RealScalar im = + std::ldexp(internal::random<RealScalar>(RealScalar(-4), RealScalar(4)), internal::random<int>(-20, 20)); + data1[i] = Scalar(re, im); + } + internal::pstore(data2, internal::phypot_complex(internal::pload<Packet>(data1))); + for (int i = 0; i < PacketSize; ++i) { + const W re = W(data1[i].real()), im = W(data1[i].imag()); + const W ref = std::sqrt(re * re + im * im); + const W ulp = std::ldexp(W(1), std::ilogb(ref) - digits + 1); + const W tol = (W(0.5) + std::ldexp(W(1), -8)) * ulp; + VERIFY(numext::abs(W(data2[i].real()) - ref) <= tol && "internal::phypot_complex"); + VERIFY_IS_EQUAL(data2[i].real(), data2[i].imag()); + } + } +} +template <typename Scalar, typename Packet, typename W, std::enable_if_t<internal::is_scalar<Packet>::value, int> = 0> +void complex_hypot_test() {} + // Notice that this definition works for complex types as well. // std::log2 only supports real types, not complex. template <typename Scalar> @@ -2864,6 +2922,8 @@ } } } + complex_hypot_test<Scalar, Packet, + typename std::conditional<std::is_same<RealScalar, float>::value, double, long double>::type>(); } exp_complex_test<Scalar, Packet>(data1, data2, ref, size); } @@ -3061,6 +3121,132 @@ } #endif +#if defined(EIGEN_VECTORIZE_AVX) || defined(EIGEN_VECTORIZE_AVX512) || defined(EIGEN_VECTORIZE_NEON) || \ + defined(EIGEN_VECTORIZE_SME) +// The bfloat16 arithmetic packets round a float result to nearest even. Check them against that rounding applied to +// the float packet op, over every encoding of one operand and a fixed set for the other that puts sums and products +// on exact midpoints: 1 and 3 against [256, 512), 1 + 2^-7 and 2^-7 against values near 1, plus the range ends and +// the special values. NaN lanes compare as NaN, since a kernel may keep the operand's payload. +template <typename Packet> +void packetmath_bfloat16_arithmetic_rounding() { + constexpr int packet_size = internal::unpacket_traits<Packet>::size; + using FloatPacket = + typename internal::find_packet_by_size<float, internal::plain_enum_min( + packet_size, internal::packet_traits<float>::size)>::type; + constexpr int float_size = internal::unpacket_traits<FloatPacket>::size; + static_assert(packet_size % float_size == 0, "the bfloat16 packet must span whole float packets"); + const numext::uint16_t others[] = {0x0000, 0x8000, 0x3f80, 0x4040, 0x3f81, 0x3c00, 0x4380, 0xbf81, + 0x7f7f, 0x0080, 0x0001, 0x7f80, 0xff80, 0x7fc0, 0x7f81, 0x3f7f}; + constexpr int count = sizeof(others) / sizeof(others[0]); + EIGEN_ALIGN_MAX bfloat16 a[packet_size], b[packet_size], c[packet_size], out[packet_size]; + EIGEN_ALIGN_MAX float fa[packet_size], fb[packet_size], fc[packet_size], ref[packet_size]; + // op is applied to the float operands one float packet at a time. + auto verify = [&](const Packet& result, auto op) { + internal::pstore(out, result); + for (int k = 0; k < packet_size; k += float_size) { + internal::pstore(ref + k, op(internal::pload<FloatPacket>(fa + k), internal::pload<FloatPacket>(fb + k), + internal::pload<FloatPacket>(fc + k))); + } + for (int i = 0; i < packet_size; ++i) { + numext::uint16_t got = numext::bit_cast<numext::uint16_t>(out[i]); + numext::uint16_t want = numext::bit_cast<numext::uint16_t>(static_cast<bfloat16>(ref[i])); + if ((want & 0x7fff) > 0x7f80) { + VERIFY((got & 0x7fff) > 0x7f80); + } else { + VERIFY_IS_EQUAL(got, want); + } + } + }; + for (int j = 0; j < count; ++j) { + for (int i = 0; i < packet_size; ++i) { + b[i] = numext::bit_cast<bfloat16>(others[j]); + c[i] = numext::bit_cast<bfloat16>(others[(j + 1) % count]); + fb[i] = static_cast<float>(b[i]); + fc[i] = static_cast<float>(c[i]); + } + Packet qb = internal::pload<Packet>(b), qc = internal::pload<Packet>(c); + for (unsigned first = 0; first < 65536; first += packet_size) { + for (int i = 0; i < packet_size; ++i) { + a[i] = numext::bit_cast<bfloat16>(static_cast<numext::uint16_t>(first + i)); + fa[i] = static_cast<float>(a[i]); + } + Packet qa = internal::pload<Packet>(a); + using P = const FloatPacket&; + verify(internal::padd(qa, qb), [](P x, P y, P) { return internal::padd(x, y); }); + verify(internal::psub(qa, qb), [](P x, P y, P) { return internal::psub(x, y); }); + verify(internal::pmul(qa, qb), [](P x, P y, P) { return internal::pmul(x, y); }); + verify(internal::pdiv(qa, qb), [](P x, P y, P) { return internal::pdiv(x, y); }); + verify(internal::pabsdiff(qa, qb), [](P x, P y, P) { return internal::pabsdiff(x, y); }); + verify(internal::pmadd(qa, qb, qc), [](P x, P y, P z) { return internal::pmadd(x, y, z); }); + verify(internal::pmsub(qa, qb, qc), [](P x, P y, P z) { return internal::pmsub(x, y, z); }); + verify(internal::pnmadd(qa, qb, qc), [](P x, P y, P z) { return internal::pnmadd(x, y, z); }); + verify(internal::pnmsub(qa, qb, qc), [](P x, P y, P z) { return internal::pnmsub(x, y, z); }); + } + } +} + +// The nearest bfloat16 to a double, ties to even. Rounding through float first could tip a bfloat16 tie, so the +// candidates are the truncation of float(value) and its neighbours, compared in double. +numext::uint16_t round_to_bfloat16_bits(double value) { + float single = static_cast<float>(value); + numext::uint16_t truncated = static_cast<numext::uint16_t>(numext::bit_cast<numext::uint32_t>(single) >> 16); + if (!(numext::isfinite)(single) || single == 0.0f) return truncated; + numext::uint16_t best = truncated; + double best_error = NumTraits<double>::infinity(); + for (int step = -1; step <= 1; ++step) { + numext::uint16_t candidate = static_cast<numext::uint16_t>(truncated + step); + double error = numext::abs(static_cast<double>(static_cast<float>(numext::bit_cast<bfloat16>(candidate))) - value); + if (error < best_error || (error == best_error && (candidate & 1) == 0)) { + best = candidate; + best_error = error; + } + } + return best; +} + +// psqrt and prsqrt over every bfloat16 encoding against references rounded once from double: the backends that +// enable this claim correctly rounded results. With flushed set, the FPU reads a subnormal input as a zero of its +// sign, and so does the reference; a kernel that passes such a lane through unchanged, as the AVX-512 fast-math +// psqrt does under DAZ, is accepted too. +template <typename Packet> +void packetmath_bfloat16_sqrt_exhaustive(bool flushed) { + constexpr int packet_size = internal::unpacket_traits<Packet>::size; + EIGEN_ALIGN_MAX bfloat16 input[packet_size], sqrt_output[packet_size], rsqrt_output[packet_size]; + for (unsigned first = 0; first < 65536; first += packet_size) { + for (int i = 0; i < packet_size; ++i) { + input[i] = numext::bit_cast<bfloat16>(static_cast<numext::uint16_t>(first + i)); + } + internal::pstoreu(sqrt_output, internal::psqrt(internal::ploadu<Packet>(input))); + internal::pstoreu(rsqrt_output, internal::prsqrt(internal::ploadu<Packet>(input))); + for (int i = 0; i < packet_size; ++i) { + numext::uint16_t input_bits = static_cast<numext::uint16_t>(first + i); + bool flushed_subnormal = flushed && (input_bits & 0x7f80) == 0 && (input_bits & 0x7f) != 0; + numext::uint16_t bits = flushed_subnormal ? static_cast<numext::uint16_t>(input_bits & 0x8000) : input_bits; + numext::uint16_t sqrt_bits = numext::bit_cast<numext::uint16_t>(sqrt_output[i]); + numext::uint16_t rsqrt_bits = numext::bit_cast<numext::uint16_t>(rsqrt_output[i]); + if ((bits & 0x7fff) > 0x7f80 || bits > 0x8000) { + // NaN, or negative and not -0. + VERIFY((sqrt_bits & 0x7fff) > 0x7f80); + VERIFY((rsqrt_bits & 0x7fff) > 0x7f80); + } else { + double x = static_cast<double>(static_cast<float>(numext::bit_cast<bfloat16>(bits))); + numext::uint16_t sqrt_expected = round_to_bfloat16_bits(std::sqrt(x)); + if (flushed_subnormal && sqrt_bits == input_bits) sqrt_expected = input_bits; + VERIFY_IS_EQUAL(sqrt_bits, sqrt_expected); + VERIFY_IS_EQUAL(rsqrt_bits, round_to_bfloat16_bits(1.0 / std::sqrt(x))); + } + } + } +} + +template <typename Packet> +void packetmath_bfloat16_sqrt_exhaustive_flushed() { + ScopedFlushToZero flush_to_zero(FlushToZeroMode::Both); + if (flush_to_zero.mode() != FlushToZeroMode::Both) return; + packetmath_bfloat16_sqrt_exhaustive<Packet>(true); +} +#endif + namespace Eigen { namespace test { @@ -3140,6 +3326,19 @@ CALL_SUBTEST_15(packetmath_bfloat16_sign_bits_flushed()); #endif +#if defined(EIGEN_VECTORIZE_AVX) || defined(EIGEN_VECTORIZE_AVX512) || defined(EIGEN_VECTORIZE_NEON) || \ + defined(EIGEN_VECTORIZE_SME) + using Bf16Packet = internal::packet_traits<bfloat16>::type; + using Bf16HalfPacket = internal::packet_traits<bfloat16>::half; + CALL_SUBTEST_15(packetmath_bfloat16_arithmetic_rounding<Bf16Packet>()); + CALL_SUBTEST_15(packetmath_bfloat16_arithmetic_rounding<Bf16HalfPacket>()); + // 32-bit ARM NEON flushes subnormals whatever FPSCR says. + CALL_SUBTEST_15(packetmath_bfloat16_sqrt_exhaustive<Bf16Packet>(EIGEN_ARCH_ARM != 0)); + CALL_SUBTEST_15(packetmath_bfloat16_sqrt_exhaustive<Bf16HalfPacket>(EIGEN_ARCH_ARM != 0)); + CALL_SUBTEST_15(packetmath_bfloat16_sqrt_exhaustive_flushed<Bf16Packet>()); + CALL_SUBTEST_15(packetmath_bfloat16_sqrt_exhaustive_flushed<Bf16HalfPacket>()); +#endif + #if defined(EIGEN_VECTORIZE_RVV10) CALL_SUBTEST_1((packetmath_redux_infinities<float, internal::Packet1Xf>())); CALL_SUBTEST_1((packetmath_redux_infinities<float, internal::Packet2Xf>()));
diff --git a/test/packetmath_fastmath.cpp b/test/packetmath_fastmath.cpp index bd3b055..c64f8e7 100644 --- a/test/packetmath_fastmath.cpp +++ b/test/packetmath_fastmath.cpp
@@ -387,6 +387,7 @@ CALL_SUBTEST((verify_mask_reduction<float, Eigen::internal::Packet2f>())); CALL_SUBTEST((verify_low_bit_mask_any_ftz<float, Eigen::internal::Packet2f>())); CALL_SUBTEST((verify_low_bit_mask_any_ftz<Eigen::bfloat16, Eigen::internal::Packet4bf>())); + CALL_SUBTEST((verify_low_bit_mask_any_ftz<Eigen::bfloat16, Eigen::internal::Packet8bf>())); #if EIGEN_ARCH_ARM64 CALL_SUBTEST((verify_low_bit_mask_any_ftz<double, Eigen::internal::Packet2d>())); #endif
diff --git a/test/pastix_support.cpp b/test/pastix_support.cpp index ff665f8..533b74b 100644 --- a/test/pastix_support.cpp +++ b/test/pastix_support.cpp
@@ -14,6 +14,104 @@ #include <Eigen/PaStiXSupport> #include <contrib/Eigen/SparseExtra> +template <typename Solver> +void test_pastix_reuse() { + using Scalar = typename Solver::Scalar; + using Real = typename NumTraits<Scalar>::Real; + using Sparse = typename Solver::MatrixType; + using Dense = Matrix<Scalar, Dynamic, Dynamic, RowMajor>; + Sparse matrix(5, 5); + const Scalar offDiagonal = Scalar(-1) + internal::random<Scalar>() * Scalar(0.25); + for (Index i = 0; i < matrix.rows(); ++i) { + matrix.insert(i, i) = Scalar(4); + if (i > 0) matrix.insert(i, i - 1) = offDiagonal; + if (i + 1 < matrix.rows()) matrix.insert(i, i + 1) = numext::conj(offDiagonal); + } + Dense expected(5, 2); + for (Index i = 0; i < expected.rows(); ++i) { + expected(i, 0) = Scalar(i + 1); + expected(i, 1) = Scalar(2 - i); + } + Solver solver; + solver.iparm(IPARM_THREAD_NBR) = 1; + solver.dparm(DPARM_EPSILON_REFINEMENT) = NumTraits<Real>::epsilon(); + solver.analyzePattern(matrix); + VERIFY_IS_EQUAL(solver.info(), Success); + for (int iteration = 0; iteration < 2; ++iteration) { + for (Index i = 0; i < matrix.rows(); ++i) matrix.coeffRef(i, i) += Scalar(1); + solver.factorize(matrix); + VERIFY_IS_EQUAL(solver.info(), Success); + const Dense rhs = matrix * expected; + const Dense result = solver.solve(rhs); + VERIFY_IS_EQUAL(solver.info(), Success); + const Real errorBound = Real(128) * NumTraits<Real>::epsilon() * expected.norm(); + VERIFY((result - expected).norm() <= errorBound); + } + solver.compute(matrix); + VERIFY_IS_EQUAL(solver.info(), Success); + const Dense rhs = matrix * expected; + const Dense result = solver.solve(rhs); + VERIFY_IS_EQUAL(solver.info(), Success); + VERIFY_IS_APPROX(result, expected); + + Sparse diagonal(2, 2); + diagonal.insert(0, 0) = Scalar(2); + diagonal.insert(1, 1) = Scalar(3); + solver.compute(diagonal); + VERIFY_IS_EQUAL(solver.info(), Success); + const Matrix<Scalar, 2, 1> diagonalRhs(Scalar(2), Scalar(3)); + const Matrix<Scalar, 2, 1> diagonalResult = solver.solve(diagonalRhs); + VERIFY_IS_EQUAL(solver.info(), Success); + VERIFY_IS_APPROX(diagonalResult, (Matrix<Scalar, 2, 1>::Ones())); + + Sparse empty(0, 0); + solver.compute(empty); + VERIFY_IS_EQUAL(solver.info(), Success); + const Dense emptyRhs(0, 2); + const Dense emptyResult = solver.solve(emptyRhs); + VERIFY_IS_EQUAL(solver.info(), Success); + VERIFY_IS_EQUAL(emptyResult.rows(), 0); + VERIFY_IS_EQUAL(emptyResult.cols(), 2); + solver.analyzePattern(empty); + solver.factorize(empty); + VERIFY_IS_EQUAL(solver.info(), Success); + solver.compute(matrix); + const Dense noRhs(matrix.rows(), 0); + const Dense noResult = solver.solve(noRhs); + VERIFY_IS_EQUAL(solver.info(), Success); + VERIFY_IS_EQUAL(noResult.cols(), 0); +} + +template <typename Scalar> +void test_pastix_special_cases() { + using Sparse = SparseMatrix<Scalar, RowMajor, long long>; + test_pastix_reuse<PastixLU<Sparse>>(); + test_pastix_reuse<PastixLLT<Sparse, Lower>>(); + test_pastix_reuse<PastixLDLT<Sparse, Upper>>(); + for (int emptyRow = 0; emptyRow < 2; ++emptyRow) { + Sparse singular(2, 2); + singular.insert(0, 0) = Scalar(1); + if (!emptyRow) { + singular.insert(0, 1) = Scalar(1); + singular.insert(1, 0) = Scalar(1); + singular.insert(1, 1) = Scalar(1); + } + PastixLU<Sparse> solver; + solver.iparm(IPARM_THREAD_NBR) = 1; + // A fixed tiny perturbation makes these inconsistent systems overflow during solve. + solver.dparm(DPARM_EPSILON_MAGN_CTRL) = -(std::numeric_limits<double>::min)(); + solver.compute(singular); + if (solver.info() == Success) { + const Matrix<Scalar, 2, 1> rhs(Scalar(1), Scalar(8)); + const Matrix<Scalar, 2, 1> result = solver.solve(rhs); + EIGEN_UNUSED_VARIABLE(result); + VERIFY_IS_EQUAL(solver.info(), NumericalIssue); + } else { + VERIFY_IS_EQUAL(solver.info(), NumericalIssue); + } + } +} + template <typename T> void test_pastix_T() { PastixLLT<SparseMatrix<T, ColMajor>, Eigen::Lower> pastix_llt_lower; @@ -22,6 +120,12 @@ PastixLDLT<SparseMatrix<T, ColMajor>, Eigen::Upper> pastix_ldlt_upper; PastixLU<SparseMatrix<T, ColMajor> > pastix_lu; + pastix_llt_lower.iparm(IPARM_THREAD_NBR) = 1; + pastix_ldlt_lower.iparm(IPARM_THREAD_NBR) = 1; + pastix_llt_upper.iparm(IPARM_THREAD_NBR) = 1; + pastix_ldlt_upper.iparm(IPARM_THREAD_NBR) = 1; + pastix_lu.iparm(IPARM_THREAD_NBR) = 1; + check_sparse_spd_solving(pastix_llt_lower); check_sparse_spd_solving(pastix_ldlt_lower); check_sparse_spd_solving(pastix_llt_upper); @@ -37,17 +141,13 @@ pastix_lu.dparm(); } -// There is no support for selfadjoint matrices with PaStiX. -// Complex symmetric matrices should pass though -template <typename T> -void test_pastix_T_LU() { - PastixLU<SparseMatrix<T, ColMajor> > pastix_lu; - check_sparse_square_solving(pastix_lu); -} - EIGEN_DECLARE_TEST(pastix_support) { CALL_SUBTEST_1(test_pastix_T<float>()); CALL_SUBTEST_2(test_pastix_T<double>()); - CALL_SUBTEST_3((test_pastix_T_LU<std::complex<float> >())); - CALL_SUBTEST_4(test_pastix_T_LU<std::complex<double> >()); + CALL_SUBTEST_3((test_pastix_T<std::complex<float>>())); + CALL_SUBTEST_4(test_pastix_T<std::complex<double>>()); + CALL_SUBTEST_1(test_pastix_special_cases<float>()); + CALL_SUBTEST_2(test_pastix_special_cases<double>()); + CALL_SUBTEST_3(test_pastix_special_cases<std::complex<float>>()); + CALL_SUBTEST_4(test_pastix_special_cases<std::complex<double>>()); }
diff --git a/test/product_assign.cpp b/test/product_assign.cpp new file mode 100644 index 0000000..74d2811 --- /dev/null +++ b/test/product_assign.cpp
@@ -0,0 +1,411 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// This Source Code Form is subject to the terms of the Mozilla +// Public License v. 2.0. If a copy of the MPL was not distributed +// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +// GEMM assignment, C = A * B, which kernels that can overwrite (gebp_kernel_overwrites) compute without zeroing C +// first. Every overwritten coefficient is alpha * acc + 0, so the result must be bitwise the one that zeroing C and +// accumulating gives; C starts out NaN, so a coefficient not written, or computed from C's old contents, fails the +// comparison. Small blocking budgets split the products over several depth, row and column blocks, of which only +// the first depth block may overwrite. + +#define EIGEN_DEBUG_SMALL_PRODUCT_BLOCKS +#include "main.h" +#include "product_test_helpers.h" + +#include <limits> + +template <typename Scalar> +Scalar nan_value() { + return Scalar(std::numeric_limits<typename NumTraits<Scalar>::Real>::quiet_NaN()); +} + +// Bitwise equality of every coefficient, so that +0 and -0 differ. +template <typename Real> +bool bits_equal(const Real& x, const Real& y) { + using Bits = typename numext::get_integer_by_size<sizeof(Real)>::unsigned_type; + return numext::bit_cast<Bits>(x) == numext::bit_cast<Bits>(y); +} +template <typename Real> +bool bits_equal(const std::complex<Real>& x, const std::complex<Real>& y) { + return bits_equal(x.real(), y.real()) && bits_equal(x.imag(), y.imag()); +} +template <typename A, typename B> +bool bitwise_equal(const DenseBase<A>& a, const DenseBase<B>& b) { + if (a.rows() != b.rows() || a.cols() != b.cols()) return false; + for (Index j = 0; j < a.cols(); ++j) + for (Index i = 0; i < a.rows(); ++i) + if (!bits_equal(a.derived().coeff(i, j), b.derived().coeff(i, j))) return false; + return true; +} + +// dst = scale * a * b through assignment, from a NaN-filled destination, against the zero-then-accumulate result and +// the coefficient-based product. The two must be bitwise equal where the product runs GEMM. The coefficient-based +// product and the GEMV fallbacks compile `=` and `+=` separately, and those may round differently. +template <typename Dst, typename Lhs, typename Rhs, typename Scalar> +void check_assign(Dst& dst, const Lhs& a, const Rhs& b, const Scalar& scale) { + using ScaledLhs = internal::remove_all_t<decltype(scale * a)>; + using Impl = internal::generic_product_impl<ScaledLhs, Rhs, DenseShape, DenseShape, GemmProduct>; + const bool gemm = dst.rows() > 1 && dst.cols() > 1 && !Impl::useRuntimeCoeffBasedProduct(dst, b); + // Accumulated into dst itself, so that it takes the same kernel (or external BLAS) as the assignment: a row-major or + // strided destination may take another, which rounds differently. + using Plain = typename Dst::PlainObject; + dst.setZero(); + dst.noalias() += scale * a * b; + const Plain expected = dst; + + dst.setConstant(nan_value<typename Dst::Scalar>()); + dst.noalias() = scale * a * b; + VERIFY(dst.allFinite()); + if (gemm) + VERIFY(bitwise_equal(dst, expected)); + else + VERIFY(verifyProduct(dst, expected, scale * a, b)); + + // Assignment without noalias goes through a temporary, whose storage order is the product's. + dst.setConstant(nan_value<typename Dst::Scalar>()); + dst = scale * a * b; + VERIFY(dst.allFinite()); + VERIFY(verifyProduct(dst, expected, scale * a, b)); + + if (a.cols() > 0) VERIFY(verifyProduct(expected, Plain((scale * a).lazyProduct(b)), scale * a, b)); +} + +// Sizes around the packet widths and the register and cache blocks of the kernels, at depths below, at and above +// the small blocking's depth block. +template <typename Scalar, int DstOrder> +void assign_shapes() { + using Mat = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + using DstMat = Matrix<Scalar, Dynamic, Dynamic, DstOrder>; + const Index sizes[] = {1, 3, 16, 17, 32, 33, 48, 65, 129}; + const Index depths[] = {1, 25, 64, 130, 300}; + for (Index m : sizes) + for (Index n : sizes) + for (Index k : depths) { + if (m * n * k > 1000000) continue; + const Mat a = Mat::Random(m, k), b = Mat::Random(k, n); + DstMat c(m, n); + check_assign(c, a, b, Scalar(1)); + if (m > 1 && n > 1) check_assign(c, a, b, Scalar(2)); + } + + // Transposed operands, and a column-major result read back through an outer stride with sentinel padding. + const Index m = 97, n = 70, k = 300; + const Mat at = Mat::Random(k, m), bt = Mat::Random(n, k); + DstMat c(m, n); + check_assign(c, at.transpose(), bt.transpose(), Scalar(1)); + + const Mat a = Mat::Random(m, k), b = Mat::Random(k, n); + Mat storage = Mat::Constant(m + 5, n, Scalar(7)); + Map<Mat, 0, OuterStride<>> mapped(storage.data(), m, n, OuterStride<>(m + 5)); + check_assign(mapped, a, b, Scalar(1)); + VERIFY((storage.bottomRows(5).array() == Scalar(7)).all()); +} + +// Outer stride 1 and inner stride n: a row-major layout seen through column-major strides, so the kernel's C has a +// unit column stride. +template <typename Scalar> +void assign_unit_col_stride() { + using Mat = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + for (Index m : {Index(17), Index(65), Index(129)}) + for (Index n : {Index(9), Index(40), Index(129)}) + for (Index k : {Index(25), Index(300)}) { + const Mat a = Mat::Random(m, k), b = Mat::Random(k, n); + Matrix<Scalar, Dynamic, 1> storage = Matrix<Scalar, Dynamic, 1>::Constant(m * n + 3, Scalar(7)); + Map<Mat, 0, Stride<Dynamic, Dynamic>> mapped(storage.data(), m, n, Stride<Dynamic, Dynamic>(1, n)); + check_assign(mapped, a, b, Scalar(1)); + for (Index i = m * n; i < storage.size(); ++i) VERIFY_IS_EQUAL(storage(i), Scalar(7)); + } +} + +// A depth past the blocking's depth block: only the first pass overwrites, the later ones accumulate. +template <typename Scalar> +void assign_depth_passes() { + using Mat = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + using Blocking = internal::gemm_blocking_space<ColMajor, Scalar, Scalar, Dynamic, Dynamic, Dynamic>; + for (Index m : {Index(33), Index(129)}) { + const Index n = 40; + // At least 48: the heuristic does not block a product with max(m, n, k) < 48, and wide packets shrink kc so far + // (16 at VLEN 1024) that 2 * kc can fall below that. The 7 leaves a partial final depth pass. + Index k = numext::maxi<Index>(2 * Blocking(m, n, 1 << 20, 1, true).kc(), 48) + 7; + VERIFY(Blocking(m, n, k, 1, true).kc() < k); + const Mat a = Mat::Random(m, k), b = Mat::Random(k, n); + Mat c(m, n); + check_assign(c, a, b, Scalar(1)); + } +} + +// An inner stride of two: the result coefficients are every other scalar, and the ones in between stay untouched. +template <typename Scalar> +void assign_inner_stride() { + using Mat = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + const Index m = 65, n = 40, k = 150; + const Mat a = Mat::Random(m, k), b = Mat::Random(k, n); + Matrix<Scalar, Dynamic, 1> storage = Matrix<Scalar, Dynamic, 1>::Constant(2 * m * n, Scalar(7)); + Map<Mat, 0, Stride<Dynamic, 2>> mapped(storage.data(), m, n, Stride<Dynamic, 2>(2 * m, 2)); + check_assign(mapped, a, b, Scalar(1)); + for (Index i = 1; i < storage.size(); i += 2) VERIFY_IS_EQUAL(storage(i), Scalar(7)); +} + +// Conjugated operands and scalar factors on either side, and a real operand on either side of a complex product. +template <typename Scalar> +void assign_conjugate() { + using Mat = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + using RealScalar = typename NumTraits<Scalar>::Real; + using RealMat = Matrix<RealScalar, Dynamic, Dynamic, ColMajor>; + const Index m = 70, n = 45, k = 140; + const Mat a = Mat::Random(m, k), b = Mat::Random(k, n), at = Mat::Random(k, m); + Mat c(m, n); + const Scalar s = internal::random<Scalar>(); + check_assign(c, a.conjugate(), b, s); + check_assign(c, at.adjoint(), b, s); + check_assign(c, a, (s * b).conjugate(), Scalar(1)); + const RealMat ra = RealMat::Random(m, k), rb = RealMat::Random(k, n); + check_assign(c, ra, b, RealScalar(1)); + check_assign(c, a, rb, RealScalar(1)); +} + +// Accumulation is unchanged: += and -= onto a nonzero result. +template <typename Scalar> +void accumulate_unchanged() { + using Mat = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + const Index m = 66, n = 33, k = 200; + const Mat a = Mat::Random(m, k), b = Mat::Random(k, n), c0 = Mat::Random(m, n); + Mat c = c0; + c.noalias() += a * b; + VERIFY(verifyProduct(Mat(c - c0), Mat(a.lazyProduct(b)), a, b)); + c = c0; + c.noalias() -= a * b; + VERIFY(verifyProduct(Mat(c0 - c), Mat(a.lazyProduct(b)), a, b)); +} + +// alpha * acc is -0 for alpha = -1 and an all-zero product, and overwriting must keep the sign that accumulating into +// a zeroed result gives: +0 for real scalars (0 + (-0)); for complex ones it depends on the packet pmadd (the AVX and +// SSE fmaddsub forms give a -0 real part). Odd sizes reach the scalar row and column tails as well as the packet +// stores. +template <typename Scalar> +void assign_signed_zero() { + using Mat = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + for (Index size : {64, 67}) { + const Mat a = Mat::Ones(size, size), b = Mat::Zero(size, size); + Mat expected = Mat::Zero(size, size); + expected.noalias() += Scalar(-1) * a * b; + Mat c = Mat::Constant(size, size, nan_value<Scalar>()); + c.noalias() = Scalar(-1) * a * b; + VERIFY(bitwise_equal(c, expected)); + for (Index j = 0; j < size; ++j) + for (Index i = 0; i < size; ++i) + VERIFY(c(i, j) == Scalar(0) && (NumTraits<Scalar>::IsComplex || !(std::signbit)(numext::real(c(i, j))))); + } +} + +// The driver's own contract: overwriting assigns, so a zero factor or an empty depth zeroes the result. The result is +// a padded column-major block whatever the default storage order, and the padding must stay untouched. Through +// gemm_run_overwrite, which takes run_overwrite where the driver has one (not the external-BLAS one). +template <typename Scalar> +void driver_overwrite() { + using Mat = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + using Gemm = + internal::general_matrix_matrix_product<Index, Scalar, ColMajor, false, Scalar, ColMajor, false, ColMajor, 1>; + using Blocking = internal::gemm_blocking_space<ColMajor, Scalar, Scalar, Dynamic, Dynamic, Dynamic>; + const Index m = 40, n = 30, k = 20, ldc = m + 5; + const Mat a = Mat::Random(m, k), b = Mat::Random(k, n); + Mat storage(ldc, n); + internal::GemmParallelInfo<Index>* no_info = nullptr; + for (Index depth : {Index(0), k}) { + for (Scalar alpha : {Scalar(0), Scalar(1)}) { + Blocking blocking(m, n, depth, 1, true); + storage.setConstant(Scalar(7)); + storage.topRows(m).setConstant(nan_value<Scalar>()); + internal::gemm_run_overwrite<Gemm>(m, n, depth, a.data(), m, b.data(), k, storage.data(), Index(1), ldc, alpha, + blocking, no_info); + const Mat expected = (depth == 0 || alpha == Scalar(0)) ? Mat(Mat::Zero(m, n)) : Mat(a.lazyProduct(b)); + VERIFY(storage.topRows(m).allFinite()); + VERIFY(verifyProduct(Mat(storage.topRows(m)), expected, a, b)); + VERIFY((storage.bottomRows(ldc - m).array() == Scalar(7)).all()); + } + } + // An empty result is left alone. + Blocking blocking(m, 0, k, 1, true); + internal::gemm_run_overwrite<Gemm>(m, Index(0), k, a.data(), m, b.data(), k, storage.data(), Index(1), ldc, Scalar(1), + blocking, no_info); +} + +// A kernel from before the overwrite contract: only the untagged call, which accumulates. +struct accumulate_only_kernel { + int* calls; + template <typename... Args> + void operator()(const Args&...) const { + ++*calls; + } +}; + +// A general_matrix_matrix_product specialization from before the contract: no run_overwrite and no CanOverwrite. +struct accumulate_only_gemm { + static int calls; + template <typename Scalar, typename Blocking> + static void run(Index rows, Index cols, Index, const Scalar*, Index, const Scalar*, Index, Scalar* res, Index, + Index resStride, Scalar, Blocking&, internal::GemmParallelInfo<Index>*) { + ++calls; + for (Index j = 0; j < cols; ++j) + for (Index i = 0; i < rows; ++i) res[i + j * resStride] += Scalar(1); + } +}; +int accumulate_only_gemm::calls = 0; + +// A scalar from before the overwrite contract whose product is an expression: it supports `dst += a * c` but not +// adding that expression to a scalar, so accumulating stores must ask for nothing more. +struct et_scalar; +struct et_product { + double v; +}; +struct et_scalar { + double v; + et_scalar& operator+=(const et_product& p) { + v += p.v; + return *this; + } +}; +et_product operator*(const et_scalar& a, const et_scalar& b) { return et_product{a.v * b.v}; } + +// Kernels with and without the profitability hook. +struct pays_never { + static bool overwrite_pays(Index, Index, Index) { return false; } +}; +struct no_pays_hook {}; + +#ifdef EIGEN_VECTORIZE_SME +// A kernel from before the contract that reads its LHS in place: only the untagged run_direct_lhs. +struct accumulate_only_direct_kernel { + int* calls; + template <typename... Args> + void run_direct_lhs(const Args&...) const { + ++*calls; + } +}; +#endif + +// Old interfaces keep working: the kernel only ever gets the untagged call, and a driver without run_overwrite is +// overwritten by zeroing the result and accumulating into it. +void old_interfaces() { + et_scalar dst{1}, alpha{2}, acc{3}; + internal::gebp_store_scalar<false>(true, dst, alpha, acc); + VERIFY_IS_EQUAL(dst.v, 7.0); + + VERIFY((internal::gebp_overwrite_pays<no_pays_hook>(Index(64), Index(64), Index(64)))); + VERIFY(!(internal::gebp_overwrite_pays<pays_never>(Index(64), Index(64), Index(64)))); + +#ifdef EIGEN_VECTORIZE_SME + int direct_calls = 0; + accumulate_only_direct_kernel direct{&direct_calls}; + internal::sme_direct_lhs_call(std::false_type(), direct, true, 1, 2, 3); + VERIFY_IS_EQUAL(direct_calls, 1); +#endif + + int calls = 0; + accumulate_only_kernel kernel{&calls}; + internal::gebp_run(std::false_type(), kernel, true, 1, 2, 3); + VERIFY_IS_EQUAL(calls, 1); + STATIC_CHECK(!(internal::gebp_kernel_overwrites<accumulate_only_kernel>::value)); + + STATIC_CHECK(!(internal::gemm_can_overwrite<accumulate_only_gemm>::value)); + const Index m = 6, n = 4, k = 3, ldc = 8; + Matrix<float, Dynamic, 1> res = Matrix<float, Dynamic, 1>::Constant(ldc * n, nan_value<float>()); + internal::gemm_blocking_space<ColMajor, float, float, Dynamic, Dynamic, Dynamic> blocking(m, n, k, 1, true); + const float* none = nullptr; + internal::GemmParallelInfo<Index>* no_info = nullptr; + accumulate_only_gemm::calls = 0; + internal::gemm_run_overwrite<accumulate_only_gemm>(m, n, k, none, m, none, k, res.data(), Index(1), ldc, 1.f, + blocking, no_info); + VERIFY_IS_EQUAL(accumulate_only_gemm::calls, 1); + for (Index j = 0; j < n; ++j) + for (Index i = 0; i < ldc; ++i) VERIFY(i < m ? res(i + j * ldc) == 1.f : (numext::isnan)(res(i + j * ldc))); +} + +// A complex-like custom scalar over a built-in real type. +struct custom_complex { + double re, im; +}; + +// int and half have no architecture kernel, so they take the generic one: built-in arithmetic opts in, half and +// custom scalars do not (assigning them is not known to match accumulating onto zero). Architecture kernels opt in +// where they implement it. +template <typename Lhs, typename Rhs = Lhs> +using kernel_for = + internal::gebp_kernel<Lhs, Rhs, Index, internal::blas_data_mapper<Lhs, Index, ColMajor>, + internal::gebp_traits<Lhs, Rhs>::mr, internal::gebp_traits<Lhs, Rhs>::nr, false, false>; +void capability_trait() { + STATIC_CHECK((internal::gebp_kernel_overwrites<kernel_for<int>>::value)); + STATIC_CHECK(!(internal::gebp_kernel_overwrites<kernel_for<half>>::value)); + STATIC_CHECK((internal::gebp_overwritable_scalar<std::complex<double>>::value)); + STATIC_CHECK(!(internal::gebp_overwritable_scalar<custom_complex>::value)); +#ifdef EIGEN_USE_BLAS + // With beta = 0, ?gemm may form alpha * t and return -0 where zero-then-accumulate gives +0 (reference BLAS does + // in its transposed branches), so the external-BLAS driver keeps zeroing first. + STATIC_CHECK( + !(internal::gemm_can_overwrite<internal::general_matrix_matrix_product<Index, float, ColMajor, false, float, + ColMajor, false, ColMajor, 1>>::value)); + STATIC_CHECK( + !(internal::gemm_can_overwrite<internal::general_matrix_matrix_product<Index, double, RowMajor, false, double, + ColMajor, false, ColMajor, 1>>::value)); +#endif +#ifdef EIGEN_VECTORIZE_SME + STATIC_CHECK((internal::gebp_kernel_overwrites<kernel_for<float>>::value)); + STATIC_CHECK((internal::gebp_kernel_overwrites<kernel_for<std::complex<float>>>::value)); + // Complex results overwrite up to 128 KB. + VERIFY((kernel_for<std::complex<float>>::overwrite_pays(128, 128, 64))); + VERIFY(!(kernel_for<std::complex<float>>::overwrite_pays(128, 129, 64))); +#ifdef EIGEN_VECTORIZE_SME_F64F64 + // 64-bit results overwrite up to 128 KB; a 32-bit Index must not overflow forming the byte count. + using DoubleKernel = internal::gebp_kernel<double, double, int, internal::blas_data_mapper<double, int, ColMajor>, + internal::gebp_traits<double, double>::mr, + internal::gebp_traits<double, double>::nr, false, false>; + VERIFY(DoubleKernel::overwrite_pays(128, 128, 64)); + VERIFY(!DoubleKernel::overwrite_pays(128, 129, 64)); + VERIFY(!DoubleKernel::overwrite_pays(16384, 16384, 64)); + VERIFY((kernel_for<std::complex<double>>::overwrite_pays(128, 64, 64))); + VERIFY(!(kernel_for<std::complex<double>>::overwrite_pays(128, 65, 64))); +#endif +#endif +} + +EIGEN_DECLARE_TEST(product_assign) { + CALL_SUBTEST_1(capability_trait()); + CALL_SUBTEST_1(old_interfaces()); + CALL_SUBTEST_1((assign_shapes<float, ColMajor>())); + CALL_SUBTEST_1((assign_shapes<float, RowMajor>())); + CALL_SUBTEST_1(assign_inner_stride<float>()); + CALL_SUBTEST_1(assign_unit_col_stride<float>()); + CALL_SUBTEST_1(assign_depth_passes<float>()); + CALL_SUBTEST_1(assign_signed_zero<float>()); + CALL_SUBTEST_1(accumulate_unchanged<float>()); + CALL_SUBTEST_1(driver_overwrite<float>()); + CALL_SUBTEST_2((assign_shapes<double, ColMajor>())); + CALL_SUBTEST_2((assign_shapes<double, RowMajor>())); + CALL_SUBTEST_2(assign_inner_stride<double>()); + CALL_SUBTEST_2(assign_unit_col_stride<double>()); + CALL_SUBTEST_2(assign_depth_passes<double>()); + CALL_SUBTEST_2(assign_signed_zero<double>()); + CALL_SUBTEST_2(driver_overwrite<double>()); + CALL_SUBTEST_3((assign_shapes<std::complex<float>, ColMajor>())); + CALL_SUBTEST_3((assign_shapes<std::complex<float>, RowMajor>())); + CALL_SUBTEST_3(assign_conjugate<std::complex<float>>()); + CALL_SUBTEST_3(assign_inner_stride<std::complex<float>>()); + CALL_SUBTEST_3(assign_unit_col_stride<std::complex<float>>()); + CALL_SUBTEST_3(assign_depth_passes<std::complex<float>>()); + CALL_SUBTEST_3(assign_signed_zero<std::complex<float>>()); + CALL_SUBTEST_3(accumulate_unchanged<std::complex<float>>()); + CALL_SUBTEST_3(driver_overwrite<std::complex<float>>()); + CALL_SUBTEST_4((assign_shapes<std::complex<double>, ColMajor>())); + CALL_SUBTEST_4((assign_shapes<std::complex<double>, RowMajor>())); + CALL_SUBTEST_4(assign_conjugate<std::complex<double>>()); + CALL_SUBTEST_4(assign_inner_stride<std::complex<double>>()); + CALL_SUBTEST_4(assign_unit_col_stride<std::complex<double>>()); + CALL_SUBTEST_4(assign_depth_passes<std::complex<double>>()); + CALL_SUBTEST_4(assign_signed_zero<std::complex<double>>()); + CALL_SUBTEST_4(driver_overwrite<std::complex<double>>()); +}
diff --git a/test/product_cast.cpp b/test/product_cast.cpp new file mode 100644 index 0000000..0a9facd --- /dev/null +++ b/test/product_cast.cpp
@@ -0,0 +1,308 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#define TEST_ENABLE_TEMPORARY_TRACKING +#define EIGEN_RUNTIME_NO_MALLOC +#define TEST_IGNORE_STACK_ALLOCATED_TEMPORARY + +#include "main.h" +#include "product_test_helpers.h" + +// GEMM operands of the form A.cast<T>() are packed straight from A, and half and bfloat16 GEMMs accumulate in float. + +// Temporaries per cast operand: none, or one with EIGEN_USE_BLAS or on SME, which materialize the cast. +#if !defined(EIGEN_USE_BLAS) && !defined(EIGEN_VECTORIZE_SME) +constexpr int kCastTemporaries = 0; +#else +constexpr int kCastTemporaries = 1; +#endif + +template <typename Src, int Order> +Matrix<Src, Dynamic, Dynamic, Order> random_small_integers(Index rows, Index cols, int bound) { + Matrix<Src, Dynamic, Dynamic, Order> m(rows, cols); + for (Index j = 0; j < cols; ++j) + for (Index i = 0; i < rows; ++i) m(i, j) = Src(internal::random<int>(-bound, bound)); + return m; +} + +// Integers in [-bound, bound], or for a floating-point Src, fractions there that a narrowing cast rounds. +template <typename Src, int Order> +Matrix<Src, Dynamic, Dynamic, Order> random_operand(Index rows, Index cols, int bound) { + if (NumTraits<Src>::IsInteger) return random_small_integers<Src, Order>(rows, cols, bound); + return Matrix<Src, Dynamic, Dynamic, Order>::Random(rows, cols) * Src(bound); +} + +// The lazily cast operands pack to the same blocks as the materialized casts, so the products agree bit for bit. +template <typename Src, typename Dst, int LhsOrder, int RhsOrder, int DstOrder> +void lazy_cast_product(Index rows, Index cols, Index depth) { + using DstMat = Matrix<Dst, Dynamic, Dynamic, DstOrder>; + using PlainDst = Matrix<Dst, Dynamic, Dynamic>; + Matrix<Src, Dynamic, Dynamic, LhsOrder> a = random_operand<Src, LhsOrder>(rows + 3, depth + 2, 8); + Matrix<Src, Dynamic, Dynamic, RhsOrder> b = random_operand<Src, RhsOrder>(depth + 1, cols + 4, 8); + // Strided operands: blocks of larger matrices. + auto A = a.block(1, 2, rows, depth); + auto B = b.block(1, 3, depth, cols); + Matrix<Dst, Dynamic, Dynamic, LhsOrder> Ad = A.template cast<Dst>(); + Matrix<Dst, Dynamic, Dynamic, RhsOrder> Bd = B.template cast<Dst>(); + + DstMat ref(rows, cols), dst(rows, cols); + ref.noalias() = Ad * Bd; + dst.setRandom(); + VERIFY_EVALUATION_COUNT(dst.noalias() = A.template cast<Dst>() * B.template cast<Dst>(), 2 * kCastTemporaries); + VERIFY_IS_EQUAL(dst, ref); + DstMat ref_update = ref; + ref_update.noalias() += Ad * Bd; + VERIFY_EVALUATION_COUNT(dst.noalias() += A.template cast<Dst>() * B.template cast<Dst>(), 2 * kCastTemporaries); + VERIFY_IS_EQUAL(dst, ref_update); + ref_update.noalias() -= Ad * Bd; + VERIFY_EVALUATION_COUNT(dst.noalias() -= A.template cast<Dst>() * B.template cast<Dst>(), 2 * kCastTemporaries); + VERIFY_IS_EQUAL(dst, ref_update); + + // One side already in the kernel's type. + ref.noalias() = Ad * Bd; + VERIFY_EVALUATION_COUNT(dst.noalias() = A.template cast<Dst>() * Bd, kCastTemporaries); + VERIFY_IS_EQUAL(dst, ref); + VERIFY_EVALUATION_COUNT(dst.noalias() = Ad * B.template cast<Dst>(), kCastTemporaries); + VERIFY_IS_EQUAL(dst, ref); + // A transposed operand below the cast. + DstMat dst_t(cols, rows), ref_t(cols, rows); + ref_t.noalias() = Bd.transpose() * Ad.transpose(); + VERIFY_EVALUATION_COUNT(dst_t.noalias() = B.transpose().template cast<Dst>() * A.transpose().template cast<Dst>(), + 2 * kCastTemporaries); + VERIFY_IS_EQUAL(dst_t, ref_t); + + // A scalar factor below the cast is not hoisted out of it, and the operand is materialized as before. + ref.noalias() = (Src(2) * A).template cast<Dst>().eval() * Bd; + dst.noalias() = (Src(2) * A).template cast<Dst>() * B.template cast<Dst>(); + VERIFY_IS_EQUAL(dst, ref); + // A scalar factor above the cast multiplies the result. + ref.noalias() = (Dst(3) * Ad) * Bd; + dst.noalias() = (Dst(3) * A.template cast<Dst>()) * B.template cast<Dst>(); + VERIFY_IS_EQUAL(dst, ref); + // Not a GEMM: the GEMV fallback for a single result column, which sums in another order than the GEMV kernel. + PlainDst col(rows, 1); + col.noalias() = A.template cast<Dst>() * B.col(0).template cast<Dst>(); + VERIFY_IS_APPROX(col, (Ad * Bd.col(0)).eval()); +} + +template <typename Src, typename Dst> +void lazy_cast_products() { + const Index rows = internal::random<Index>(20, 300); + const Index cols = internal::random<Index>(20, 300); + const Index depth = internal::random<Index>(20, 300); + CALL_SUBTEST((lazy_cast_product<Src, Dst, ColMajor, ColMajor, ColMajor>(rows, cols, depth))); + CALL_SUBTEST((lazy_cast_product<Src, Dst, RowMajor, ColMajor, ColMajor>(rows, cols, depth))); + CALL_SUBTEST((lazy_cast_product<Src, Dst, ColMajor, RowMajor, RowMajor>(rows, cols, depth))); + CALL_SUBTEST((lazy_cast_product<Src, Dst, RowMajor, RowMajor, RowMajor>(rows, cols, depth))); + // Sizes that are not a multiple of the packet or the register block. + CALL_SUBTEST((lazy_cast_product<Src, Dst, ColMajor, RowMajor, ColMajor>(17, 13, 9))); + CALL_SUBTEST((lazy_cast_product<Src, Dst, RowMajor, ColMajor, RowMajor>(31, 7, 23))); +} + +// A half or bfloat16 GEMM sums in float and rounds each coefficient once. +template <typename T, int LhsOrder, int RhsOrder, int DstOrder> +void float_accumulation(Index rows, Index cols, Index depth) { + using Mat = Matrix<T, Dynamic, Dynamic, DstOrder>; + // |sum| <= 256 * depth < 2^24 stays an integer that float holds exactly, so the reference does not depend on the + // order. Its spread of about 16^2 / 3 * sqrt(depth) exceeds half's and bfloat16's exact integers, so a sum in T + // rounds. + Matrix<T, Dynamic, Dynamic, LhsOrder> A = random_small_integers<T, LhsOrder>(rows, depth, 16); + Matrix<T, Dynamic, Dynamic, RhsOrder> B = random_small_integers<T, RhsOrder>(depth, cols, 16); + MatrixXf exact = A.template cast<float>().eval() * B.template cast<float>().eval(); + + Mat C(rows, cols); + C.setConstant(T(1000)); + C.noalias() = A * B; + VERIFY_IS_EQUAL(C, exact.cast<T>().eval()); + C = A * B; + VERIFY_IS_EQUAL(C, exact.cast<T>().eval()); + + Mat C0 = random_small_integers<T, DstOrder>(rows, cols, 64); + C = C0; + C.noalias() += A * B; + VERIFY_IS_EQUAL(C, (C0.template cast<float>() + exact).template cast<T>().eval()); + C = C0; + C.noalias() -= A * B; + VERIFY_IS_EQUAL(C, (C0.template cast<float>() - exact).template cast<T>().eval()); + + // Strided destination and operands. + Mat big = Mat::Constant(rows + 2, cols + 3, T(7)); + big.block(1, 2, rows, cols).noalias() = A.leftCols(depth) * B.topRows(depth); + VERIFY_IS_EQUAL(big.block(1, 2, rows, cols), exact.cast<T>().eval()); + VERIFY_IS_EQUAL(big.row(0), Mat::Constant(1, cols + 3, T(7))); + VERIFY_IS_EQUAL(big.col(0), Mat::Constant(rows + 2, 1, T(7))); + VERIFY_IS_EQUAL(big.rightCols(1), Mat::Constant(rows + 2, 1, T(7))); + VERIFY_IS_EQUAL(big.bottomRows(1), Mat::Constant(1, cols + 3, T(7))); +} + +template <typename T> +void float_accumulations() { + const Index rows = internal::random<Index>(20, 200); + const Index cols = internal::random<Index>(20, 200); + const Index depth = internal::random<Index>(20, 600); + CALL_SUBTEST((float_accumulation<T, ColMajor, ColMajor, ColMajor>(rows, cols, depth))); + CALL_SUBTEST((float_accumulation<T, RowMajor, ColMajor, RowMajor>(rows, cols, depth))); + CALL_SUBTEST((float_accumulation<T, RowMajor, RowMajor, ColMajor>(rows, cols, depth))); + CALL_SUBTEST((float_accumulation<T, ColMajor, RowMajor, RowMajor>(13, 9, 31))); +} + +template <typename T> +void float_accumulation_depth() { + // Summed in T, a run of ones stalls where T's spacing reaches 2: at 2048 for half and 256 for bfloat16. + const Index depth = std::is_same<T, half>::value ? 4096 : 1024; + using Mat = Matrix<T, Dynamic, Dynamic>; + Mat A = Mat::Ones(24, depth), B = Mat::Ones(depth, 21); + Mat C = A * B; + VERIFY_IS_EQUAL(C, Mat::Constant(24, 21, T(float(depth)))); + // Nonnegative operands, where no cancellation hides the growth of an error summed in T: here it stays one + // rounding to T at any depth. + Mat X = Mat::Random(24, depth).cwiseAbs(), Y = Mat::Random(depth, 21).cwiseAbs(); + C.noalias() = X * Y; + VERIFY(verifyProduct<float>(C.template cast<double>(), + (X.template cast<double>().eval() * Y.template cast<double>().eval()).eval(), X, Y, 1)); + + // The scalar factors combine in float: s * t = 2^-28 is below half's and above bfloat16's smallest subnormal, and + // depth * s * t = 2^-16 (half) or 2^-18 (bfloat16) is representable in both. + const T s = T(std::ldexp(1.0f, -14)); + C.noalias() = (s * A) * (B * s); + VERIFY_IS_EQUAL(C, Mat::Constant(24, 21, T(std::ldexp(float(depth), -28)))); + // Where s * s underflows in T, a product small enough for the coefficient-based path would evaluate 0 * inf there, + // so it takes the float GEMM as well. + for (Index n : {5, 24}) { + Mat P = Mat::Ones(n, n), Q = Mat::Ones(n, n), R(n, n); + P(0, 0) = NumTraits<T>::infinity(); + R.noalias() = (s * P) * (Q * s); + VERIFY(R.row(0).array().isInf().all()); + VERIFY_IS_EQUAL(R.bottomRows(n - 1), Mat::Constant(n - 1, n, T(std::ldexp(float(n), -28)))); + R.setOnes(); + R.noalias() += (s * P) * (Q * s); + VERIFY(R.row(0).array().isInf().all()); + VERIFY_IS_EQUAL(R.bottomRows(n - 1), Mat::Constant(n - 1, n, T(1.0f + std::ldexp(float(n), -28)))); + } +} + +// A cast to T is materialized in T, as before, rather than packed from the source and summed in float: the product +// sees T's overflow, its rounding of the source, and inf * 0. +template <typename T, typename Src> +void cast_to_float_accumulated() { + using SrcMat = Matrix<Src, Dynamic, Dynamic>; + using Mat = Matrix<T, Dynamic, Dynamic>; + const Index n = 32; + SrcMat A = random_small_integers<Src, ColMajor>(n, n, 8), B = random_small_integers<Src, ColMajor>(n, n, 8); + // Out of half's range, and above bfloat16's spacing of 2^10 at 2^17. + A(0, 0) = Src(std::is_same<T, half>::value ? 70000 : 205001); + // 2051 rounds to 2052 in half and 2048 in bfloat16. + A(1, 0) = Src(2051); + B.row(0).setConstant(Src(3)); + B(0, 1) = Src(0); + Mat ref = A.template cast<T>().eval() * B.template cast<T>().eval(); + Mat C = A.template cast<T>() * B.template cast<T>(); + // inf and NaN coefficients compare unequal, so compare their classification and the finite rest. + VERIFY((C.array().isNaN() == ref.array().isNaN()).all()); + VERIFY((C.array().isInf() == ref.array().isInf()).all()); + VERIFY_IS_EQUAL(C.array().isFinite().select(C, T(0)).eval(), ref.array().isFinite().select(ref, T(0)).eval()); + VERIFY_IS_EQUAL(C.array().isInf().any(), (std::is_same<T, half>::value)); +} + +// The float workspace goes where the packed blocks do: a fixed-size product above the stack limit compiles and runs, +// and a preallocated product below it does not allocate. +template <typename T> +void float_accumulation_workspace() { + using Fixed = Matrix<T, 192, 192>; + std::unique_ptr<Fixed> A(new Fixed), B(new Fixed), C(new Fixed); + *A = random_small_integers<T, ColMajor>(192, 192, 4); + *B = random_small_integers<T, ColMajor>(192, 192, 4); + C->noalias() = *A * *B; + using Mat = Matrix<T, Dynamic, Dynamic>; + MatrixXf Af = A->template cast<float>(), Bf = B->template cast<float>(); + VERIFY_IS_EQUAL(Mat(*C), (Af * Bf).template cast<T>().eval()); + + // With EIGEN_USE_BLAS or on SME the float product materializes both casts. +#if !defined(EIGEN_USE_BLAS) && !defined(EIGEN_VECTORIZE_SME) + Mat X = random_small_integers<T, ColMajor>(48, 40, 4), Y = random_small_integers<T, ColMajor>(40, 36, 4); + MatrixXf exact = X.template cast<float>() * Y.template cast<float>(); + Mat Z(48, 36); + internal::set_is_malloc_allowed(false); + Z.noalias() = X * Y; + internal::set_is_malloc_allowed(true); + VERIFY_IS_EQUAL(Z, exact.cast<T>().eval()); + MatrixXf twice = Z.template cast<float>() + exact; + internal::set_is_malloc_allowed(false); + Z.noalias() += X * Y; + internal::set_is_malloc_allowed(true); + VERIFY_IS_EQUAL(Z, twice.cast<T>().eval()); +#endif + + // A runtime vector result takes GEMV on T, which does not allocate for a row-major matrix by a column or a row by a + // column-major matrix. Entries in [-1, 1] at depth 40 keep every partial sum exact in T. + Matrix<T, Dynamic, Dynamic, RowMajor> R = random_small_integers<T, RowMajor>(48, 40, 1); + Mat v = random_small_integers<T, ColMajor>(40, 1, 1), w = random_small_integers<T, ColMajor>(1, 40, 1); + Mat V = random_small_integers<T, ColMajor>(40, 36, 1), y(48, 1), z(1, 36); + internal::set_is_malloc_allowed(false); + y.noalias() = R * v; + z.noalias() = w * V; + internal::set_is_malloc_allowed(true); + VERIFY_IS_EQUAL(y, (R.template cast<float>() * v.template cast<float>()).template cast<T>().eval()); + VERIFY_IS_EQUAL(z, (w.template cast<float>() * V.template cast<float>()).template cast<T>().eval()); + + // An empty product neither allocates nor rewrites dst, which keeps a NaN payload that a round trip through float + // would quieten. + Mat E0(200, 0), E1(0, 200), W = Mat::Ones(200, 200); + W(0, 0) = numext::bit_cast<T>(std::is_same<T, half>::value ? uint16_t(0x7d01) : uint16_t(0x7fa1)); + const Mat W0 = W; + internal::set_is_malloc_allowed(false); + W.noalias() += E0 * E1; + internal::set_is_malloc_allowed(true); + VERIFY_IS_EQUAL(numext::bit_cast<uint16_t>(W(0, 0)), numext::bit_cast<uint16_t>(W0(0, 0))); + VERIFY_IS_EQUAL(W.bottomRows(199), W0.bottomRows(199)); + W.noalias() = E0 * E1; + VERIFY_IS_EQUAL(W, Mat::Zero(200, 200)); +} + +EIGEN_DECLARE_TEST(product_cast) { + using MatrixXhalf = Matrix<half, Dynamic, Dynamic>; + using internal::gemm_accumulates_in_float; + using internal::gemm_operand_traits; +#if !defined(EIGEN_USE_BLAS) && !defined(EIGEN_VECTORIZE_SME) + STATIC_CHECK((std::is_same<gemm_operand_traits<decltype(MatrixXhalf().cast<float>())>::StorageScalar, half>::value)); + STATIC_CHECK(( + std::is_same<gemm_operand_traits<decltype(MatrixXf().transpose().cast<double>())>::StorageScalar, float>::value)); +#endif + STATIC_CHECK( + (std::is_same<gemm_operand_traits<decltype((MatrixXhalf() + MatrixXhalf()).cast<float>())>::StorageScalar, + float>::value)); + STATIC_CHECK((std::is_same<gemm_operand_traits<decltype((half(2) * MatrixXhalf()).cast<float>())>::StorageScalar, + float>::value)); + STATIC_CHECK((std::is_same<gemm_operand_traits<decltype(MatrixXf().cast<std::complex<float>>())>::StorageScalar, + std::complex<float>>::value)); + STATIC_CHECK((std::is_same<gemm_operand_traits<decltype(MatrixXf().cast<half>())>::StorageScalar, half>::value)); + STATIC_CHECK((gemm_accumulates_in_float<half, half>::value)); +#if !(defined(EIGEN_USE_BLAS) && defined(EIGEN_USE_OPENBLAS_BFLOAT16) && EIGEN_USE_OPENBLAS_BFLOAT16) + STATIC_CHECK((gemm_accumulates_in_float<bfloat16, bfloat16>::value)); +#endif + STATIC_CHECK((!gemm_accumulates_in_float<half, bfloat16>::value)); + STATIC_CHECK((!gemm_accumulates_in_float<float, float>::value)); + STATIC_CHECK((!gemm_accumulates_in_float<double, double>::value)); + + for (int i = 0; i < g_repeat; i++) { + CALL_SUBTEST_1((lazy_cast_products<half, float>())); + CALL_SUBTEST_2((lazy_cast_products<bfloat16, float>())); + CALL_SUBTEST_3((lazy_cast_products<float, double>())); + CALL_SUBTEST_3((lazy_cast_products<int, float>())); + CALL_SUBTEST_3((lazy_cast_products<double, float>())); + CALL_SUBTEST_4((lazy_cast_products<int, double>())); + CALL_SUBTEST_4((lazy_cast_products<half, double>())); + } + for (int i = 0; i < g_repeat; i++) { + CALL_SUBTEST_5(float_accumulations<half>()); + CALL_SUBTEST_6(float_accumulations<bfloat16>()); + } + CALL_SUBTEST_5(float_accumulation_depth<half>()); + CALL_SUBTEST_6(float_accumulation_depth<bfloat16>()); + CALL_SUBTEST_5((cast_to_float_accumulated<half, float>())); + CALL_SUBTEST_5((cast_to_float_accumulated<half, double>())); + CALL_SUBTEST_6((cast_to_float_accumulated<bfloat16, float>())); + CALL_SUBTEST_6((cast_to_float_accumulated<bfloat16, double>())); + CALL_SUBTEST_5(float_accumulation_workspace<half>()); + CALL_SUBTEST_6(float_accumulation_workspace<bfloat16>()); +}
diff --git a/test/product_extra.cpp b/test/product_extra.cpp index 00ddc52..bb98375 100644 --- a/test/product_extra.cpp +++ b/test/product_extra.cpp
@@ -9,6 +9,27 @@ // SPDX-License-Identifier: MPL-2.0 #include "main.h" +#include "NonCommutativeScalar.h" + +template <int Order> +void test_dense_quaternion_row_order() { + using Scalar = noncommutative_scalar::Quaternion; + const Scalar i(0, 1, 0, 0), j(0, 0, 1, 0); + Matrix<Scalar, Dynamic, Dynamic, Order> matrix = Matrix<Scalar, Dynamic, Dynamic, Order>::Constant(3, 4, j); + Matrix<Scalar, 1, Dynamic> row = Matrix<Scalar, 1, Dynamic>::Constant(3, i); + const Matrix<Scalar, 1, Dynamic> rowProduct = row * matrix; + const Matrix<Scalar, Dynamic, 1> column = row.transpose(); + const Matrix<Scalar, 1, Dynamic> transposedProduct = column.transpose() * matrix; + Matrix<Scalar, 1, Dynamic> scaledRow = Matrix<Scalar, 1, Dynamic>::Zero(matrix.cols()); + scaledRow.noalias() += j * (row * matrix); + for (Index col = 0; col < matrix.cols(); ++col) { + Scalar expected; + for (Index k = 0; k < row.size(); ++k) expected += row(k) * matrix(k, col); + VERIFY(rowProduct(col) == expected); + VERIFY(transposedProduct(col) == expected); + VERIFY(scaledRow(col) == j * expected); + } +} template <typename MatrixType> void product_extra(const MatrixType& m) { @@ -960,6 +981,8 @@ #endif EIGEN_DECLARE_TEST(product_extra) { + CALL_SUBTEST_17((test_dense_quaternion_row_order<ColMajor>())); + CALL_SUBTEST_17((test_dense_quaternion_row_order<RowMajor>())); CALL_SUBTEST_14(complex_outer_product_overflow<ColMajor>()); CALL_SUBTEST_14(complex_outer_product_overflow<RowMajor>()); CALL_SUBTEST_14(single_term_nullary_product<ColMajor>());
diff --git a/test/product_selfadjoint.cpp b/test/product_selfadjoint.cpp index a7fd13e..465fac3 100644 --- a/test/product_selfadjoint.cpp +++ b/test/product_selfadjoint.cpp
@@ -215,29 +215,44 @@ } } -// Same test for complex type (tests conjugation logic). -template <int> +// Same test for complex types, including buffers aligned only to alignof(RealScalar) < sizeof(Scalar). +template <typename Scalar> void product_selfadjoint_boundary_complex() { - typedef std::complex<float> Scalar; - typedef Matrix<Scalar, Dynamic, Dynamic> Mat; - typedef Matrix<Scalar, Dynamic, 1> Vec; + using RealScalar = typename NumTraits<Scalar>::Real; + using Mat = Matrix<Scalar, Dynamic, Dynamic>; + using Vec = Matrix<Scalar, Dynamic, 1>; + using RealVec = Matrix<RealScalar, Dynamic, 1>; - const int sizes[] = {1, 8, 47, 48, 49, 64}; - for (int si = 0; si < 6; ++si) { - int n = sizes[si]; + const int sizes[] = {1, 2, 3, 4, 8, 9, 16, 47, 48, 49, 64}; + for (int n : sizes) { Mat m1 = Mat::Random(n, n); m1 = (m1 + m1.adjoint()).eval(); // make Hermitian m1.diagonal() = m1.diagonal().real().template cast<Scalar>(); // real diagonal Vec v1 = Vec::Random(n); Mat rhs = Mat::Random(n, 3); + Vec ref = m1 * v1; - Mat m2 = m1.triangularView<Lower>(); - VERIFY_IS_APPROX(m2.selfadjointView<Lower>() * v1, m1 * v1); - VERIFY_IS_APPROX(m2.selfadjointView<Lower>() * rhs, m1 * rhs); + Mat m2 = m1.template triangularView<Lower>(); + VERIFY_IS_APPROX(m2.template selfadjointView<Lower>() * v1, ref); + VERIFY_IS_APPROX(m2.template selfadjointView<Lower>() * rhs, m1 * rhs); - m2 = m1.triangularView<Upper>(); - VERIFY_IS_APPROX(m2.selfadjointView<Upper>() * v1, m1 * v1); + m2 = m1.template triangularView<Upper>(); + VERIFY_IS_APPROX(m2.template selfadjointView<Upper>() * v1, ref); + + RealVec unaligned_res_storage(2 * n + 1); + RealVec unaligned_rhs_storage(2 * n + 1); + Map<Vec, Unaligned> unaligned_res(reinterpret_cast<Scalar*>(unaligned_res_storage.data() + 1), n); + Map<Vec, Unaligned> unaligned_rhs(reinterpret_cast<Scalar*>(unaligned_rhs_storage.data() + 1), n); + unaligned_rhs = v1; + + unaligned_res.setZero(); + unaligned_res.noalias() += m1.template selfadjointView<Lower>() * unaligned_rhs; + VERIFY_IS_APPROX(unaligned_res, ref); + + unaligned_res.setZero(); + unaligned_res.noalias() += m1.template selfadjointView<Upper>() * unaligned_rhs; + VERIFY_IS_APPROX(unaligned_res, ref); } } @@ -267,5 +282,6 @@ // Deterministic blocking boundary tests (outside g_repeat). CALL_SUBTEST_8(product_selfadjoint_boundary<0>()); - CALL_SUBTEST_9(product_selfadjoint_boundary_complex<0>()); + CALL_SUBTEST_9(product_selfadjoint_boundary_complex<std::complex<float>>()); + CALL_SUBTEST_9(product_selfadjoint_boundary_complex<std::complex<double>>()); }
diff --git a/test/product_small.cpp b/test/product_small.cpp index 3fe40a3..c152e60 100644 --- a/test/product_small.cpp +++ b/test/product_small.cpp
@@ -446,6 +446,27 @@ VERIFY(!verifyProduct(nan, zero, abs_a, abs_b, 4)); } +// verifyProductUpdate() bounds C0 + alpha*A*B by 5*sqrt(k+2)*eps*2*|| |alpha|*|A|*|B| + |C0| ||_F, here +// 20*eps*(2e-6*|alpha| + 1): an error of a few eps*|C0| passes although the bound on A*B alone rejects it. +template <typename Scalar> +void product_verify_update(const Scalar &large_alpha) { + using Mat = Matrix<Scalar, Dynamic, Dynamic>; + using Real = typename NumTraits<Scalar>::Real; + const Real eps = NumTraits<Scalar>::epsilon(); + const Mat A = Mat::Constant(1, 2, Scalar(1e-3f)), B = Mat::Constant(2, 1, Scalar(1e-3f)), C0 = Mat::Ones(1, 1); + for (const Scalar alpha : {Scalar(-2), large_alpha}) { + const Mat expected = C0 + alpha * A * B; + const Mat off4 = expected.array() + Scalar(4 * eps), off40 = expected.array() + Scalar(40 * eps); + VERIFY(verifyProductUpdate(off4, expected, alpha, A, B, C0)); + VERIFY(!verifyProduct(off4, expected, A, B)); + VERIFY(verifyProductUpdate(off40, expected, alpha, A, B, C0) == (numext::abs(alpha) > Real(2))); + VERIFY(!verifyProductUpdate(Mat::Constant(1, 1, Scalar(std::numeric_limits<Real>::infinity())), expected, alpha, A, + B, C0)); + VERIFY(!verifyProductUpdate(Mat::Constant(1, 1, Scalar(std::numeric_limits<Real>::quiet_NaN())), expected, alpha, A, + B, C0)); + } +} + EIGEN_DECLARE_TEST(product_small) { for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST_1(product(Matrix<float, 3, 2>())); @@ -492,6 +513,8 @@ CALL_SUBTEST_6(product_small_regressions<0>()); CALL_SUBTEST_6(product_transposed_triple_bound<0>()); CALL_SUBTEST_6(product_verify_rejects_nonfinite<0>()); + CALL_SUBTEST_6(product_verify_update(1e6f)); + CALL_SUBTEST_6(product_verify_update(std::complex<float>(0.0f, 1e6f))); // Deterministic sweep at transition boundaries (outside g_repeat). CALL_SUBTEST_54(product_transition_sizes<float>());
diff --git a/test/product_sme.cpp b/test/product_sme.cpp index 7a43822..ccd597c 100644 --- a/test/product_sme.cpp +++ b/test/product_sme.cpp
@@ -279,19 +279,19 @@ const SmeColMajorMat<Scalar> reference = deepA.lazyProduct(deepB); C = initial; C.noalias() += deepA * deepB; - VERIFY_IS_APPROX(SmeColMajorMat<Scalar>(C), SmeColMajorMat<Scalar>(initial + reference)); + VERIFY(verifyProductUpdate(C, initial + reference, Scalar(1), deepA, deepB, initial)); C = initial; C.noalias() += alpha * (deepA * deepB); - VERIFY_IS_APPROX(SmeColMajorMat<Scalar>(C), SmeColMajorMat<Scalar>(initial + alpha * reference)); + VERIFY(verifyProductUpdate(C, initial + alpha * reference, alpha, deepA, deepB, initial)); // A real alpha takes the complex store's fused path: -1 from -=, and a // non-unit real scale. const Scalar real_alpha(typename NumTraits<Scalar>::Real(1.375)); C = initial; C.noalias() -= deepA * deepB; - VERIFY_IS_APPROX(SmeColMajorMat<Scalar>(C), SmeColMajorMat<Scalar>(initial - reference)); + VERIFY(verifyProductUpdate(C, initial - reference, Scalar(-1), deepA, deepB, initial)); C = initial; C.noalias() += real_alpha * (deepA * deepB); - VERIFY_IS_APPROX(SmeColMajorMat<Scalar>(C), SmeColMajorMat<Scalar>(initial + real_alpha * reference)); + VERIFY(verifyProductUpdate(C, initial + real_alpha * reference, real_alpha, deepA, deepB, initial)); #ifdef EIGEN_VECTORIZE_SME_F64F64 // RowMajor GEMM normally transposes the computation. Call the packed kernel @@ -309,7 +309,7 @@ internal::sme_gebp_dispatch<Scalar, false, false>::run( C.data(), Index(ResMat::IsRowMajor ? n : 1), Index(ResMat::IsRowMajor ? 1 : n), packedA.data(), packedB.data(), Index(n), Index(65), Index(n), scale, Index(65), Index(65), Index(0), Index(0)); - VERIFY_IS_APPROX(SmeColMajorMat<Scalar>(C), SmeColMajorMat<Scalar>(initial + scale * reference)); + VERIFY(verifyProductUpdate(C, initial + scale * reference, scale, deepA, deepB, initial)); } } #endif @@ -510,16 +510,16 @@ MC C = MC::Random(m, n); const MC C0 = C; C.noalias() = A * B; - VERIFY_IS_APPROX(C, ref); + VERIFY(verifyProduct(C, ref, A, B)); C = C0; C.noalias() += A * B; - VERIFY_IS_APPROX(C, C0 + ref); + VERIFY(verifyProductUpdate(C, C0 + ref, Scalar(1), A, B, C0)); C = C0; C.noalias() -= Scalar(2) * A * B; - VERIFY_IS_APPROX(C, C0 - Scalar(2) * ref); + VERIFY(verifyProductUpdate(C, C0 - Scalar(2) * ref, Scalar(-2), A, B, C0)); const MA At = A.transpose(); C.noalias() = At.transpose() * B; - VERIFY_IS_APPROX(C, ref); + VERIFY(verifyProduct(C, ref, A, B)); } } @@ -776,7 +776,7 @@ const SmeColMajorMat<Scalar> Ac = ConjLhs ? A.conjugate().eval() : A; const SmeColMajorMat<Scalar> Bc = ConjRhs ? B.conjugate().eval() : B; const SmeColMajorMat<Scalar> ref = c_before + alpha * Ac.lazyProduct(Bc); - VERIFY_IS_APPROX(SmeColMajorMat<Scalar>(C), ref); + VERIFY(verifyProductUpdate(C, ref, alpha, Ac, Bc, c_before)); // Cells the strided map skips stay untouched. if (strided_c) { @@ -1092,6 +1092,38 @@ check_packed_rhs<Scalar, StorageOrder, PanelMode, Conjugate>(Mapper(V.data(), V.outerStride()), V, depth, cols); } +template <bool Conjugate, typename Scalar> +EIGEN_DONT_INLINE __arm_locally_streaming static void sme_copy_range(Scalar* dst, const Scalar* src, Index stride, + Index k0, Index k1, int width) { + internal::sve_copy_panel_range<Conjugate>(dst, src, stride, k0, k1, width); +} + +// Every partial-vector width, including copies ending with an unshifted tail and exact vector boundaries. +template <typename Scalar> +static void test_copy_panel_range() { + const int svl = internal::sme_packet_traits<typename NumTraits<Scalar>::Real>::size(); + for (int width = 1; width <= 2 * svl; ++width) { + for (Index k0 : {0, 1, svl - 1}) { + for (Index depth : {0, 1, 2, 3, svl - 1, svl, svl + 1, 2 * svl + 1}) { + const Index k1 = k0 + depth; + SmeRowMajorMat<Scalar> src = SmeRowMajorMat<Scalar>::Random(k1, width + 3); + SmeVector<Scalar> packed = SmeVector<Scalar>::Constant(k1 * width + 2, pack_sentinel<Scalar>()); + SmeVector<Scalar> ref = packed; + sme_copy_range<false>(packed.data() + 1, src.data(), src.outerStride(), k0, k1, width); + for (Index k = k0; k < k1; ++k) + for (Index c = 0; c < width; ++c) set_packed(ref.data() + 1, Index(width), k, c, Scalar(src(k, c))); + VERIFY_IS_EQUAL(packed, ref); + packed.setConstant(pack_sentinel<Scalar>()); + sme_copy_range<true>(packed.data() + 1, src.data(), src.outerStride(), k0, k1, width); + for (Index k = k0; k < k1; ++k) + for (Index c = 0; c < width; ++c) + set_packed(ref.data() + 1, Index(width), k, c, Scalar(numext::conj(src(k, c)))); + VERIFY_IS_EQUAL(packed, ref); + } + } + } +} + // Inner-strided blas mapper LHS: element(i, k) laid out with inner stride // `incr`. ColMajor takes the vectorised gather path; RowMajor takes the scalar // path (its packets would run along depth, not rows). @@ -1496,6 +1528,47 @@ } } +// GCC 14 must keep the full-grid store's ZA tuples intact across C loads. Exact integer results and sentinel +// padding also cover the single-tile stores used for ragged rows and columns. +template <typename Scalar> +EIGEN_DONT_INLINE __arm_locally_streaming __arm_new("za") void sme_test_store_grid(Scalar* out, Index ldc, + const Scalar* a, const Scalar* b, + int rows, int cols) { + using Traits = internal::sme_packet_traits<Scalar>; + const int svl = Traits::size(); + const auto pg = Traits::ptrue(); + svzero_za(); + internal::outer_product_2x2<Scalar>(internal::ploadu(pg, a), internal::ploadu(pg, a + svl), internal::ploadu(pg, b), + internal::ploadu(pg, b + svl)); + internal::sme_store_2x2_grid(out, Index(1), ldc, Scalar(2), Index(2), svl, rows - svl, Index(1), + internal::sme_min(cols, svl), cols > svl ? cols - svl : 0); +} + +template <typename Scalar> +static void test_tile_pair_stores() { + const int svl = internal::sme_packet_traits<Scalar>::size(); + const Index ldc = 2 * svl + 7; + std::vector<Scalar> a(2 * svl), b(2 * svl); + for (int i = 0; i < 2 * svl; ++i) { + a[i] = Scalar(i % 7 + 1); + b[i] = Scalar(i % 5 + 2); + } + const int widths[] = {1, 2, 3, 4, 5, svl - 1, svl, svl + 1, svl + 2, 2 * svl - 1, 2 * svl}; + std::vector<Scalar> out(ldc * (2 * svl + 3)), expected(out.size()); + for (int rows = svl + 1; rows <= 2 * svl; ++rows) { + for (int cols : widths) { + if (cols > 2 * svl) continue; + std::fill(out.begin(), out.end(), Scalar(-9999)); + expected = out; + for (int j = 0; j < cols; ++j) + for (int i = 0; i < rows; ++i) expected[2 + i + (1 + j) * ldc] += Scalar(2) * a[i] * b[j]; + internal::sme_fpsr_guard guard; + sme_test_store_grid(out.data(), ldc, a.data(), b.data(), rows, cols); + for (Index i = 0; i < Index(out.size()); ++i) VERIFY_IS_EQUAL(out[i], expected[i]); + } + } +} + EIGEN_DECLARE_TEST(product_sme) { CALL_SUBTEST_1(test_whilelt_operand_types<float>()); CALL_SUBTEST_1(test_packet_ops<float>()); @@ -1503,6 +1576,7 @@ CALL_SUBTEST_1(test_conjugated_products<float>()); CALL_SUBTEST_1(test_symm_pack<float>()); CALL_SUBTEST_1(test_pack_direct<float>()); + CALL_SUBTEST_1(test_copy_panel_range<float>()); CALL_SUBTEST_1(test_mapper_fallback<float>()); CALL_SUBTEST_1(test_neon_small_blocks<float>()); CALL_SUBTEST_1(test_small_products<float>()); @@ -1523,6 +1597,7 @@ CALL_SUBTEST_2(test_packet_ops<double>()); CALL_SUBTEST_2(test_symm_pack<double>()); CALL_SUBTEST_2(test_pack_direct<double>()); + CALL_SUBTEST_2(test_copy_panel_range<double>()); CALL_SUBTEST_2(test_mapper_fallback<double>()); CALL_SUBTEST_2(test_neon_small_blocks<double>()); CALL_SUBTEST_2(test_small_products<double>()); @@ -1538,6 +1613,7 @@ CALL_SUBTEST_3(test_conjugated_products<std::complex<float>>()); CALL_SUBTEST_3(test_symm_pack<std::complex<float>>()); CALL_SUBTEST_3(test_pack_direct<std::complex<float>>()); + CALL_SUBTEST_3(test_copy_panel_range<std::complex<float>>()); CALL_SUBTEST_3(test_mapper_fallback<std::complex<float>>()); CALL_SUBTEST_3(test_neon_small_blocks<std::complex<float>>()); CALL_SUBTEST_3(test_small_products<std::complex<float>>()); @@ -1554,6 +1630,7 @@ #ifdef EIGEN_VECTORIZE_SME_F64F64 CALL_SUBTEST_4(test_symm_pack<std::complex<double>>()); CALL_SUBTEST_4(test_pack_direct<std::complex<double>>()); + CALL_SUBTEST_4(test_copy_panel_range<std::complex<double>>()); CALL_SUBTEST_4(test_mapper_fallback<std::complex<double>>()); CALL_SUBTEST_4(test_neon_small_blocks<std::complex<double>>()); CALL_SUBTEST_4(test_small_products<std::complex<double>>()); @@ -1581,4 +1658,8 @@ CALL_SUBTEST(test_small_products<std::complex<double>>()); #endif #endif + CALL_SUBTEST_9(test_tile_pair_stores<float>()); +#ifdef EIGEN_VECTORIZE_SME_F64F64 + CALL_SUBTEST_9(test_tile_pair_stores<double>()); +#endif }
diff --git a/test/product_storage_order.cpp b/test/product_storage_order.cpp new file mode 100644 index 0000000..4a09e00 --- /dev/null +++ b/test/product_storage_order.cpp
@@ -0,0 +1,137 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// This Source Code Form is subject to the terms of the Mozilla +// Public License v. 2.0. If a copy of the MPL was not distributed +// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +// A product served by the coefficient-based evaluator must be evaluated in the storage order its expression reports: +// Reverse, Redux, PartialRedux, Block, Replicate and FindCoeff index its packets and coefficients by that order, or +// size their loops by it. Each consumer is checked against the materialized product. + +#include "main.h" + +// Integer entries stay small so that the products cannot overflow. +template <typename MatrixType> +MatrixType random_operand() { + using Scalar = typename MatrixType::Scalar; + return MatrixType::NullaryExpr([] { + return NumTraits<Scalar>::IsInteger ? internal::random<Scalar>(Scalar(-8), Scalar(8)) : internal::random<Scalar>(); + }); +} + +template <typename Prod> +void check_min_max(const Prod&, const Matrix<typename Prod::Scalar, Dynamic, Dynamic>&, std::true_type) {} + +template <typename Prod> +void check_min_max(const Prod& prod, const Matrix<typename Prod::Scalar, Dynamic, Dynamic>& ref, std::false_type) { + VERIFY_IS_APPROX(prod.minCoeff(), ref.minCoeff()); + Index i = -1, j = -1; + const typename Prod::Scalar m = prod.maxCoeff(&i, &j); + VERIFY(i >= 0 && i < ref.rows() && j >= 0 && j < ref.cols()); + VERIFY_IS_APPROX(m, ref(i, j)); + VERIFY_IS_APPROX(m, ref.maxCoeff()); +} + +template <typename Prod> +void check_consumers(const Prod& prod) { + using Scalar = typename Prod::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + using Plain = Matrix<Scalar, Dynamic, Dynamic>; + const Plain& lhs = prod.lhs(); + const Plain& rhs = prod.rhs(); + const Plain ref = lhs * rhs; + + VERIFY_IS_EQUAL(bool(Prod::IsRowMajor), bool(int(internal::evaluator<Prod>::Flags) & RowMajorBit)); + VERIFY_IS_APPROX(Plain(prod), ref); + VERIFY_IS_APPROX(Plain(prod.reverse()), Plain(ref.reverse())); + VERIFY_IS_APPROX(Plain(prod.colwise().reverse()), Plain(ref.colwise().reverse())); + VERIFY_IS_APPROX(Plain(prod.rowwise().reverse()), Plain(ref.rowwise().reverse())); + VERIFY_IS_APPROX(Plain(prod.colwise().sum()), Plain(ref.colwise().sum())); + VERIFY_IS_APPROX(Plain(prod.rowwise().sum()), Plain(ref.rowwise().sum())); + double magnitude = 0; + for (Index j = 0; j < rhs.cols(); ++j) + for (Index i = 0; i < lhs.rows(); ++i) + for (Index k = 0; k < lhs.cols(); ++k) + magnitude += static_cast<double>(numext::abs(lhs(i, k))) * static_cast<double>(numext::abs(rhs(k, j))); + // Two k-term products and mn-term reductions; the factor 4 also covers complex arithmetic. + const double sum_bound = + 4 * (double(lhs.cols()) + double(ref.size())) * double(NumTraits<RealScalar>::epsilon()) * magnitude; + VERIFY((numext::isfinite)(sum_bound)); + VERIFY(static_cast<double>(numext::abs(prod.sum() - ref.sum())) <= sum_bound); + VERIFY_IS_APPROX(prod.squaredNorm(), ref.squaredNorm()); + VERIFY_IS_APPROX(prod.cwiseAbs().sum(), ref.cwiseAbs().sum()); + check_min_max(prod, ref, internal::bool_constant<NumTraits<Scalar>::IsComplex>()); + const Matrix<Scalar, Dynamic, Dynamic, ColMajor> col_major = prod.block(0, 0, ref.rows(), ref.cols()); + const Matrix<Scalar, Dynamic, Dynamic, RowMajor> row_major = prod.block(0, 0, ref.rows(), ref.cols()); + VERIFY_IS_APPROX(Plain(col_major), ref); + VERIFY_IS_APPROX(Plain(row_major), ref); + VERIFY_IS_APPROX(Plain(prod.replicate(2, 3)), Plain(ref.replicate(2, 3))); +} + +template <typename Scalar, int Rows, int Cols, int Order> +using MatrixOf = typename internal::make_proper_matrix_type<Scalar, Rows, Cols, Order>::type; + +template <typename Scalar, int M, int K, int N> +void lazy_products() { + const MatrixOf<Scalar, M, K, RowMajor> lr = random_operand<MatrixOf<Scalar, M, K, RowMajor>>(); + const MatrixOf<Scalar, M, K, ColMajor> lc = random_operand<MatrixOf<Scalar, M, K, ColMajor>>(); + const MatrixOf<Scalar, K, N, RowMajor> rr = random_operand<MatrixOf<Scalar, K, N, RowMajor>>(); + const MatrixOf<Scalar, K, N, ColMajor> rc = random_operand<MatrixOf<Scalar, K, N, ColMajor>>(); + check_consumers(lr.lazyProduct(rr)); + check_consumers(lc.lazyProduct(rr)); + check_consumers(lr.lazyProduct(rc)); + check_consumers(lc.lazyProduct(rc)); + // A nested product has no preferred storage order. + const Matrix<Scalar, K, K> identity = Matrix<Scalar, K, K>::Identity(); + check_consumers((lc * identity).lazyProduct(rr)); + check_consumers(lr.lazyProduct(identity * rc)); + // Expression operands. + const MatrixOf<Scalar, K, M, ColMajor> lct = random_operand<MatrixOf<Scalar, K, M, ColMajor>>(); + const Matrix<Scalar, M + 2, K + 2, RowMajor> big = random_operand<Matrix<Scalar, M + 2, K + 2, RowMajor>>(); + check_consumers(lct.transpose().lazyProduct(rr)); + check_consumers((Scalar(2) * lr).lazyProduct(rr)); + check_consumers(lr.conjugate().lazyProduct(rr)); + check_consumers(big.template block<M, K>(1, 1).lazyProduct(rr)); + + const Matrix<Scalar, Dynamic, Dynamic, RowMajor> dlr = lr, drr = rr; + const Matrix<Scalar, Dynamic, Dynamic, ColMajor> dlc = lc, drc = rc; + check_consumers(dlr.lazyProduct(drr)); + check_consumers(dlc.lazyProduct(drr)); + check_consumers(dlr.lazyProduct(drc)); +} + +// Default products with a unit inner dimension are served by the coefficient-based evaluator without being +// materialized. +template <typename Scalar> +void outer_products() { + const Matrix<Scalar, 8, 8, RowMajor> a = random_operand<Matrix<Scalar, 8, 8, RowMajor>>(); + const Matrix<Scalar, 8, 8, ColMajor> b = random_operand<Matrix<Scalar, 8, 8, ColMajor>>(); + const Matrix<Scalar, 1, 7> v = random_operand<Matrix<Scalar, 1, 7>>(); + check_consumers(a.col(2) * v); + check_consumers(b.col(2) * v); + check_consumers(a.col(2) * a.row(3).template head<7>()); + // Depth 1 from the rhs alone: the lhs has a runtime number of columns. + const Matrix<Scalar, Dynamic, Dynamic, RowMajor> d = a.col(5); + check_consumers(d * v); +} + +EIGEN_DECLARE_TEST(product_storage_order) { + for (int i = 0; i < g_repeat; ++i) { + CALL_SUBTEST_1((lazy_products<float, 3, 3, 5>())); + CALL_SUBTEST_1((lazy_products<float, 8, 4, 5>())); + CALL_SUBTEST_1((lazy_products<float, 16, 2, 17>())); + CALL_SUBTEST_1(outer_products<float>()); + CALL_SUBTEST_2((lazy_products<double, 6, 2, 7>())); + CALL_SUBTEST_2((lazy_products<double, 4, 1, 9>())); + CALL_SUBTEST_2((lazy_products<double, 1, 3, 8>())); + CALL_SUBTEST_2(outer_products<double>()); + CALL_SUBTEST_3((lazy_products<std::complex<float>, 5, 3, 6>())); + CALL_SUBTEST_3(outer_products<std::complex<float>>()); + CALL_SUBTEST_4((lazy_products<std::complex<double>, 4, 2, 5>())); + CALL_SUBTEST_5((lazy_products<int, 8, 3, 8>())); + CALL_SUBTEST_5(outer_products<int>()); + } +}
diff --git a/test/product_symm.cpp b/test/product_symm.cpp index ed84763..cf752f8 100644 --- a/test/product_symm.cpp +++ b/test/product_symm.cpp
@@ -330,6 +330,74 @@ } } +template <typename Scalar, int RhsOrder> +void symm_blas_operand_strides() { + using Vector = Matrix<Scalar, Dynamic, 1>; + using Lhs = Matrix<Scalar, Dynamic, Dynamic, RowMajor>; + using Rhs = Matrix<Scalar, Dynamic, Dynamic, RhsOrder>; + using Dst = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + const Index rows = 4, cols = 3, lhs_stride = 8, rhs_stride = 5; + Vector lhs_storage = Vector::Constant(rows * lhs_stride, Scalar(-90)); + Vector rhs_storage = Vector::Constant(cols * rhs_stride, Scalar(-90)); + Map<Lhs, 0, OuterStride<>> lhs(lhs_storage.data(), rows, cols, OuterStride<>(lhs_stride)); + Map<Rhs, 0, OuterStride<>> rhs(rhs_storage.data(), cols, cols, OuterStride<>(rhs_stride)); + for (Index j = 0; j < cols; ++j) { + for (Index i = 0; i < rows; ++i) lhs(i, j) = Scalar(1 + i + rows * j); + for (Index i = j; i < cols; ++i) rhs(i, j) = Scalar(1 + i + cols * j); + } + Dst expected = Dst::Ones(rows, cols); + for (Index j = 0; j < cols; ++j) + for (Index i = 0; i < rows; ++i) + for (Index k = 0; k < cols; ++k) expected(i, j) += Scalar(2) * lhs(i, k) * (k >= j ? rhs(k, j) : rhs(j, k)); + Dst result = Dst::Ones(rows, cols); + result.noalias() += Scalar(2) * lhs * rhs.template selfadjointView<Lower>(); + VERIFY_IS_CWISE_EQUAL(result, expected); +#ifdef EIGEN_USE_BLAS + result.setOnes(); + internal::gemm_blocking_space<ColMajor, Scalar, Scalar, Dynamic, Dynamic, Dynamic> blocking(rows, cols, cols, 1, + false); + internal::product_selfadjoint_matrix<Scalar, Index, RowMajor, false, false, RhsOrder, true, false, ColMajor, 1>::run( + rows, cols, lhs.data(), lhs_stride, rhs.data(), rhs_stride, result.data(), Index(1), result.outerStride(), + Scalar(2), blocking); + VERIFY_IS_CWISE_EQUAL(result, expected); +#endif +} + +template <typename RealScalar, int LhsOrder, int RhsOrder> +void hemm_blas_empty_product() { + using Scalar = std::complex<RealScalar>; + using Lhs = Matrix<Scalar, Dynamic, Dynamic, LhsOrder>; + using Rhs = Matrix<Scalar, Dynamic, Dynamic, RhsOrder>; + using Dst = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + const Index shapes[][2] = {{0, 0}, {0, 3}, {3, 0}}; + for (const auto& shape : shapes) { + const Index rows = shape[0], cols = shape[1]; + const Lhs lhs = Lhs::Zero(rows, cols); + const Rhs rhs = Rhs::Identity(cols, cols); + const Dst result = lhs * rhs.template selfadjointView<Lower>(); + VERIFY_IS_EQUAL(result.rows(), rows); + VERIFY_IS_EQUAL(result.cols(), cols); + VERIFY_IS_EQUAL(result.size(), 0); +#ifdef EIGEN_USE_BLAS + std::vector<Index> strides{3}; + EIGEN_IF_CONSTEXPR (std::numeric_limits<Index>::digits > std::numeric_limits<BlasIndex>::digits) { + Index oversized = static_cast<Index>((std::numeric_limits<BlasIndex>::max)()); + strides.push_back(++oversized); + } + const Matrix<Scalar, 3, 3> input = Matrix<Scalar, 3, 3>::Constant(Scalar(7, -3)); + internal::gemm_blocking_space<ColMajor, Scalar, Scalar, Dynamic, Dynamic, Dynamic> blocking(rows, cols, cols, 1, + false); + for (Index stride : strides) { + Matrix<Scalar, 3, 3> storage = input; + internal::product_selfadjoint_matrix<Scalar, Index, LhsOrder, false, false, RhsOrder, true, false, ColMajor, + 1>::run(rows, cols, input.data(), stride, input.data(), stride, + storage.data(), Index(1), stride, Scalar(1), blocking); + VERIFY_IS_CWISE_EQUAL(storage, input); + } +#endif + } +} + EIGEN_DECLARE_TEST(product_symm) { for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST_1((symm<float, Dynamic, Dynamic>(internal::random<int>(1, EIGEN_TEST_MAX_SIZE), @@ -352,4 +420,8 @@ // Packed-buffer contract checks and RowMajor selfadjoint operands. CALL_SUBTEST_10(symm_packers_and_rowmajor_operands<0>()); + CALL_SUBTEST_11((symm_blas_operand_strides<float, ColMajor>())); + CALL_SUBTEST_11((symm_blas_operand_strides<double, RowMajor>())); + CALL_SUBTEST_12((hemm_blas_empty_product<float, ColMajor, RowMajor>())); + CALL_SUBTEST_12((hemm_blas_empty_product<double, RowMajor, ColMajor>())); }
diff --git a/test/product_test_helpers.h b/test/product_test_helpers.h index 746de37..41614eb 100644 --- a/test/product_test_helpers.h +++ b/test/product_test_helpers.h
@@ -33,48 +33,93 @@ // lambda: probability parameter; P(lambda) = 1 - 2*exp(-lambda^2/2). // lambda=5 gives P > 0.9999 per inner product. -// Returns lambda * sqrt(k) * epsilon * num_products * || |A|*|B| ||_F, with epsilon that of the -// operand scalar but the value kept in double: narrowed to half, a triple-product bound already -// overflows at k = 100 for all-one operands. +// Returns || |alpha|*|A|*|B| + |C0| ||_F, with |C0| given coefficient-wise by c0(i, j). // -// || |A|*|B| ||_F is accumulated in double with an explicit loop rather than formed as +// The norm is accumulated in double with an explicit loop rather than formed as // (A.cwiseAbs() * B.cwiseAbs()).norm(): a bound built with the product implementation would inherit // that implementation's defects, and one that overflowed to infinity would admit any result at all. -template <typename DerivedA, typename DerivedB> -double product_error_bound(const MatrixBase<DerivedA>& A, const MatrixBase<DerivedB>& B, int num_products = 1, - double lambda = 5) { - using Scalar = typename DerivedA::Scalar; - const Index k = A.cols(); +template <typename DerivedA, typename DerivedB, typename C0Coeff> +double abs_product_update_norm(double abs_alpha, const MatrixBase<DerivedA>& A, const MatrixBase<DerivedB>& B, + C0Coeff c0) { double squared_norm = 0.0; for (Index i = 0; i < A.rows(); ++i) for (Index j = 0; j < B.cols(); ++j) { double sum = 0.0; - for (Index l = 0; l < k; ++l) + for (Index l = 0; l < A.cols(); ++l) sum += static_cast<double>(numext::abs(A.coeff(i, l))) * static_cast<double>(numext::abs(B.coeff(l, j))); + sum = abs_alpha * sum + c0(i, j); squared_norm += sum * sum; } - return lambda * numext::sqrt(double(k)) * static_cast<double>(NumTraits<Scalar>::epsilon()) * double(num_products) * - numext::sqrt(squared_norm); + return numext::sqrt(squared_norm); } -// Verify that two computations of A*B agree within the Higham-Mary bound. -// Returns true if ||actual - expected||_F <= product_error_bound(A, B, ...). +// The type a product sums in: Accumulator, or the operand scalar when Accumulator is void. +template <typename Scalar, typename Accumulator> +using product_accumulator_t = std::conditional_t<std::is_void<Accumulator>::value, Scalar, Accumulator>; + +// Returns lambda * sqrt(k) * epsilon * num_products * || |A|*|B| ||_F, with epsilon that of the +// accumulator (by default the operand scalar) but the value kept in double: narrowed to half, a +// triple-product bound already overflows at k = 100 for all-one operands. +template <typename Accumulator = void, typename DerivedA, typename DerivedB> +double product_error_bound(const MatrixBase<DerivedA>& A, const MatrixBase<DerivedB>& B, int num_products = 1, + double lambda = 5) { + using Scalar = product_accumulator_t<typename DerivedA::Scalar, Accumulator>; + const double norm = abs_product_update_norm(1.0, A, B, [](Index, Index) { return 0.0; }); + return lambda * numext::sqrt(double(A.cols())) * static_cast<double>(NumTraits<Scalar>::epsilon()) * + double(num_products) * norm; +} + +// Returns, for C = C0 + alpha*A*B (for beta*C0 + alpha*A*B, pass beta*C0 as C0), +// lambda * sqrt(k+2) * epsilon * num_products * || |alpha|*|A|*|B| + |C0| ||_F. +// k+2: a k-term inner product, then the alpha scale and the sum with c0 (real arithmetic; complex +// scalars use the same depth by extension, as in product_error_bound). The |C0| term is needed: a +// bound on A*B alone fails once |alpha|*|A|*|B| << |C0|. +template <typename Accumulator = void, typename DerivedA, typename DerivedB, typename DerivedC0> +double product_update_error_bound(const typename DerivedA::Scalar& alpha, const MatrixBase<DerivedA>& A, + const MatrixBase<DerivedB>& B, const MatrixBase<DerivedC0>& C0, int num_products = 1, + double lambda = 5) { + using Scalar = product_accumulator_t<typename DerivedA::Scalar, Accumulator>; + eigen_assert(C0.rows() == A.rows() && C0.cols() == B.cols()); + const double norm = abs_product_update_norm(static_cast<double>(numext::abs(alpha)), A, B, [&C0](Index i, Index j) { + return static_cast<double>(numext::abs(C0.coeff(i, j))); + }); + return lambda * numext::sqrt(double(A.cols() + 2)) * static_cast<double>(NumTraits<Scalar>::epsilon()) * + double(num_products) * norm; +} + +// Returns ||M||_F accumulated in double, so that it overflows only when the norm itself does. +template <typename Derived> +double frobenius_norm_in_double(const MatrixBase<Derived>& M) { + double squared_norm = 0.0; + for (Index j = 0; j < M.cols(); ++j) + for (Index i = 0; i < M.rows(); ++i) { + const double d = static_cast<double>(numext::abs(M.coeff(i, j))); + squared_norm += d * d; + } + return numext::sqrt(squared_norm); +} + +// A product that sums in an Accumulator wider than its operand Scalar rounds each result coefficient +// once to Scalar, with error at most u|e| + denorm_min/2 (u = epsilon/2 of Scalar; the second term +// covers subnormal results): num_products * (u ||expected||_F + denorm_min/2 sqrt(mn)) more. Summed +// in Scalar, that rounding is one of the k+2 operations the bound already counts. +template <typename Scalar, typename Accumulator, typename Derived> +double result_rounding_bound(const MatrixBase<Derived>& expected, int num_products) { + if (std::is_same<product_accumulator_t<Scalar, Accumulator>, Scalar>::value) return 0.0; + return 0.5 * double(num_products) * + (static_cast<double>(NumTraits<Scalar>::epsilon()) * frobenius_norm_in_double(expected) + + static_cast<double>((std::numeric_limits<typename NumTraits<Scalar>::Real>::denorm_min)()) * + numext::sqrt(double(expected.size()))); +} + +// Returns true if ||actual - expected||_F <= bound. // // The error norm is accumulated in double like the bound, so that it overflows only when the // difference itself does. A non-finite error or bound fails: inf <= inf would otherwise accept // an overflowed result. -template <typename D1, typename D2, typename DA, typename DB> -inline bool verifyProduct(const MatrixBase<D1>& actual, const MatrixBase<D2>& expected, const MatrixBase<DA>& A, - const MatrixBase<DB>& B, int num_products = 2, double lambda = 5) { - const double bound = product_error_bound(A, B, num_products, lambda); - const typename D1::PlainObject diff = actual - expected; - double squared_error = 0.0; - for (Index j = 0; j < diff.cols(); ++j) - for (Index i = 0; i < diff.rows(); ++i) { - const double d = static_cast<double>(numext::abs(diff.coeff(i, j))); - squared_error += d * d; - } - const double error = numext::sqrt(squared_error); +template <typename D1, typename D2> +bool verify_within_product_bound(const MatrixBase<D1>& actual, const MatrixBase<D2>& expected, double bound) { + const double error = frobenius_norm_in_double((actual - expected).eval()); if (!((numext::isfinite)(bound) && error <= bound)) { std::cerr << "Product verification failed: error " << error << " exceeds bound " << bound << std::endl; return false; @@ -82,6 +127,32 @@ return true; } +// Verify that two computations of A*B agree within the Higham-Mary bound. +// Returns true if ||actual - expected||_F <= product_error_bound<Accumulator>(A, B, ...), plus the +// result_rounding_bound when Accumulator is wider than the operand scalar. Against an exact +// expected value, pass num_products = 1. +template <typename Accumulator = void, typename D1, typename D2, typename DA, typename DB> +inline bool verifyProduct(const MatrixBase<D1>& actual, const MatrixBase<D2>& expected, const MatrixBase<DA>& A, + const MatrixBase<DB>& B, int num_products = 2, double lambda = 5) { + return verify_within_product_bound( + actual, expected, + product_error_bound<Accumulator>(A, B, num_products, lambda) + + result_rounding_bound<typename DA::Scalar, Accumulator>(expected, num_products)); +} + +// Verify that two computations of C0 + alpha*A*B agree within the Higham-Mary bound. +// Returns true if ||actual - expected||_F <= product_update_error_bound<Accumulator>(alpha, A, B, C0, +// ...), plus the result_rounding_bound when Accumulator is wider than the operand scalar. +template <typename Accumulator = void, typename D1, typename D2, typename DA, typename DB, typename DC0> +bool verifyProductUpdate(const MatrixBase<D1>& actual, const MatrixBase<D2>& expected, const typename DA::Scalar& alpha, + const MatrixBase<DA>& A, const MatrixBase<DB>& B, const MatrixBase<DC0>& C0, + int num_products = 2, double lambda = 5) { + return verify_within_product_bound( + actual, expected, + product_update_error_bound<Accumulator>(alpha, A, B, C0, num_products, lambda) + + result_rounding_bound<typename DA::Scalar, Accumulator>(expected, num_products)); +} + } // namespace Eigen #endif // EIGEN_TEST_PRODUCT_TEST_HELPERS_H
diff --git a/test/product_threaded.cpp b/test/product_threaded.cpp index 6cb91d2..6ac2afe 100644 --- a/test/product_threaded.cpp +++ b/test/product_threaded.cpp
@@ -316,6 +316,49 @@ VERIFY_IS_APPROX(fc, fr); } +// Bitwise equality of every coefficient, so that +0 and -0 differ. +template <typename Real> +bool bits_equal(const Real& x, const Real& y) { + using Bits = typename numext::get_integer_by_size<sizeof(Real)>::unsigned_type; + return numext::bit_cast<Bits>(x) == numext::bit_cast<Bits>(y); +} +template <typename Real> +bool bits_equal(const std::complex<Real>& x, const std::complex<Real>& y) { + return bits_equal(x.real(), y.real()) && bits_equal(x.imag(), y.imag()); +} + +// Threaded assignment, c = a * b, which the kernels overwrite on the first depth pass instead of zeroing c, against +// zero-then-accumulate in the same pool: the same partition and the same summation order, so bitwise equal. c starts +// out NaN, and the depth spans several depth blocks, so the later passes accumulate. The complex<double> result is at +// most 128 KB, where the SME kernel overwrites rather than having the driver zero it. +template <typename Scalar> +void test_parallelize_gemm_assign() { + static ThreadPool pool(4); + Eigen::setGemmThreadPool(&pool); + using Mat = Matrix<Scalar, Dynamic, Dynamic>; + using Real = typename NumTraits<Scalar>::Real; + const Index m = NumTraits<Scalar>::IsComplex ? 64 : 517, n = NumTraits<Scalar>::IsComplex ? 128 : 283, k = 2049; + internal::gemm_blocking_space<ColMajor, Scalar, Scalar, Dynamic, Dynamic, Dynamic> blocking(m, n, k, 1, true); + VERIFY(blocking.kc() < k); + blocking.initParallel(m, n, k, 4); + VERIFY(blocking.kc() < k); + const Mat a = Mat::Random(m, k), b = Mat::Random(k, n); + Mat expected = Mat::Zero(m, n); + expected.noalias() += a * b; + Mat c = Mat::Constant(m, n, Scalar(std::numeric_limits<Real>::quiet_NaN())); + c.noalias() = a * b; + bool equal = true; + for (Index j = 0; j < n; ++j) + for (Index i = 0; i < m; ++i) equal = equal && bits_equal(c(i, j), expected(i, j)); + VERIFY(equal); + Mat serial(m, n); + { + ScopedSerialGemm guard; + serial.noalias() = a * b; + } + VERIFY_IS_APPROX(c, serial); +} + // Runs a test with the given SME unit count: a nonzero count splits products on the SME kernel into disjoint parts, // 0 keeps the shared parallel session, so both paths are covered on every host. template <typename Test> @@ -344,6 +387,8 @@ CALL_SUBTEST_8(with_sme_units(units, test_parallelize_gemm_parts_types<std::complex<float>>)); CALL_SUBTEST_8(with_sme_units(units, test_parallelize_gemm_parts_types<std::complex<double>>)); CALL_SUBTEST_9(with_sme_units(units, test_parallelize_gemm_parts_blocks<>)); + CALL_SUBTEST_10(with_sme_units(units, test_parallelize_gemm_assign<float>)); + CALL_SUBTEST_10(with_sme_units(units, test_parallelize_gemm_assign<std::complex<double>>)); } CALL_SUBTEST_3(test_balanced_gemm_range<>()); CALL_SUBTEST_5(test_parallelize_gemm_require_initialization<>());
diff --git a/test/product_trmm.cpp b/test/product_trmm.cpp index 0079356..c596b51 100644 --- a/test/product_trmm.cpp +++ b/test/product_trmm.cpp
@@ -9,6 +9,7 @@ // SPDX-License-Identifier: MPL-2.0 #include "main.h" +#include "product_test_helpers.h" template <typename T> int get_random_size() { @@ -174,6 +175,126 @@ trmm<Scalar, Mode, TriOrder, OtherOrder, ResOrder, Dynamic>(rows, cols, otherCols); } +// Maps whose types fix or bound some dimensions. When every maximum size is finite, TRMM packs into inline +// buffers sized by the operands, so only the partially fixed maps are large. +template <int Mode, typename LhsType, typename RhsType, typename DstType> +void trmm_maps(Index rows, Index depth, Index cols) { + using Scalar = typename DstType::Scalar; + using LhsX = Matrix<Scalar, Dynamic, Dynamic, LhsType::IsRowMajor ? RowMajor : ColMajor>; + using RhsX = Matrix<Scalar, Dynamic, Dynamic, RhsType::IsRowMajor ? RowMajor : ColMajor>; + using DstX = Matrix<Scalar, Dynamic, Dynamic, DstType::IsRowMajor ? RowMajor : ColMajor>; + const LhsX lhs = LhsX::Random(rows, depth); + const RhsX rhs = RhsX::Random(depth, cols); + DstX dst(rows, cols); + const Map<const LhsType> lhs_map(lhs.data(), rows, depth); + const Map<const RhsType> rhs_map(rhs.data(), depth, cols); + Map<DstType> dst_map(dst.data(), rows, cols); + const LhsX tri_lhs = lhs.template triangularView<Mode>(); + const RhsX tri_rhs = rhs.template triangularView<Mode>(); + + VERIFY_IS_APPROX(dst_map.noalias() = lhs_map.template triangularView<Mode>() * rhs_map, tri_lhs * rhs); + VERIFY_IS_APPROX(dst_map.noalias() = lhs_map * rhs_map.template triangularView<Mode>(), lhs * tri_rhs); +} + +template <typename Scalar, int Mode, int Order> +void trmm_panel_boundaries() { + using Mat = Matrix<Scalar, Dynamic, Dynamic, Order>; + using Real = typename NumTraits<Scalar>::Real; + { + // The coefficient-based path must evaluate nested products before accessing their coefficients. + Mat a = Mat::Random(3, 3), b = Mat::Random(3, 3), c = Mat::Random(3, 3), actual(3, 3); + const Mat ab = a * b, bc = b * c; + const Mat triangle = a.template triangularView<Mode>(); + const Mat productTriangle = ab.template triangularView<Mode>(); + actual.noalias() = a.template triangularView<Mode>() * (b * c); + VERIFY(verifyProduct(actual, triangle.lazyProduct(bc).eval(), triangle, bc)); + actual.noalias() = (b * c) * a.template triangularView<Mode>(); + VERIFY(verifyProduct(actual, bc.lazyProduct(triangle).eval(), bc, triangle)); + actual.noalias() = (a * b).template triangularView<Mode>() * c; + VERIFY(verifyProduct(actual, productTriangle.lazyProduct(c).eval(), productTriangle, c)); + actual.noalias() = c * (a * b).template triangularView<Mode>(); + VERIFY(verifyProduct(actual, c.lazyProduct(productTriangle).eval(), c, productTriangle)); + } + const Index panel = internal::product_triangular_matrix_matrix<Scalar, Index, Mode, true, ColMajor, false, ColMajor, + false, ColMajor, 1>::SmallPanelWidth; + // Small-product dispatch, partial diagonal panels, and multiple depth blocks under EIGEN_DEBUG_SMALL_PRODUCT_BLOCKS. + const Index sizes[] = {2, 3, 5, 6, 7, panel - 1, panel, panel + 1, 2 * panel + 3, 131}; + for (Index n : sizes) { + for (Index delta : {-1, 0, 1}) { + const Index rows = n + delta, depth = n - delta, columns = n == 131 ? 531 : n + 2; + Mat a = Mat::Random(rows, depth), b = Mat::Random(depth, columns); + const Mat triangle = a.template triangularView<Mode>(); + const Mat expected = triangle.lazyProduct(b); + // Poison coefficients that the triangular view does not own, including implicit diagonals. + for (Index j = 0; j < depth; ++j) + for (Index i = 0; i < rows; ++i) + if (((Mode & Lower) ? i < j : i > j) || ((Mode & (UnitDiag | ZeroDiag)) && i == j)) + a(i, j) = Scalar(NumTraits<Real>::quiet_NaN()); + Mat actual(rows, columns); + actual.noalias() = a.template triangularView<Mode>() * b; + VERIFY(verifyProduct(actual, expected, triangle, b)); + Mat saved = Mat::Random(rows, columns); + actual = saved; + actual.noalias() -= a.template triangularView<Mode>() * b; + VERIFY(verifyProductUpdate(actual, (saved - expected).eval(), Scalar(-1), triangle, b, saved)); + Mat scaledTriangle = Scalar(2) * triangle; + EIGEN_IF_CONSTEXPR ((Mode & UnitDiag) != 0) scaledTriangle.diagonal().setOnes(); + actual.noalias() = (Scalar(2) * a).template triangularView<Mode>() * b; + VERIFY(verifyProduct(actual, scaledTriangle.lazyProduct(b).eval(), scaledTriangle, b)); + // A strided destination also checks the untouched interleaved coefficients. + Mat storage = Mat::Random(2 * rows, 2 * columns), original = storage; + Map<Mat, 0, Stride<Dynamic, 2>> strided(storage.data(), rows, columns, + Stride<Dynamic, 2>(storage.outerStride(), 2)); + strided.noalias() = a.template triangularView<Mode>() * b; + VERIFY(verifyProduct(strided, expected, triangle, b)); + for (Index j = 0; j < storage.cols(); ++j) + for (Index i = 0; i < storage.rows(); ++i) { + const bool written = Order == ColMajor ? (i % 2 == 0 && j < columns) : (j % 2 == 0 && i < rows); + if (!written) VERIFY_IS_EQUAL(storage(i, j), original(i, j)); + } + } + } +} + +// The suite's reduced cache model caps kc at the left kernel's diagonal panel width on wide-register targets, so the +// public products above never slice a diagonal block into horizontal panels there; a hand-set blocking does. +template <typename Scalar, int Mode, int LhsOrder> +void trmm_wide_depth_blocks() { + struct Blocking : internal::level3_blocking<Scalar, Scalar> { + Blocking(Index mc, Index nc, Index kc) { + this->m_mc = mc; + this->m_nc = nc; + this->m_kc = kc; + } + }; + using Kernel = internal::product_triangular_matrix_matrix<Scalar, Index, Mode, true, LhsOrder, false, ColMajor, false, + ColMajor, 1>; + using Lhs = Matrix<Scalar, Dynamic, Dynamic, LhsOrder>; + using Mat = Matrix<Scalar, Dynamic, Dynamic>; + using Real = typename NumTraits<Scalar>::Real; + const Index panel = Kernel::HorizontalPanelWidth, nr = Kernel::Traits::nr; + // Depth blocks of one full horizontal panel plus a row, and of three plus a partial one; an RHS of one column, which + // only SME slices horizontally, and of one and two micro panels; an lhs block of every row, and of one panel, which + // packs a depth block one panel at a time. + for (Index kc : {panel + 1, 3 * panel + 5}) + for (Index cols : {Index(1), nr, 2 * nr + 1}) + for (bool narrowBlock : {false, true}) { + const Index n = 2 * kc + 3; + Lhs a = Lhs::Random(n, n); + const Mat b = Mat::Random(n, cols); + const Mat triangle = a.template triangularView<Mode>(); + for (Index j = 0; j < n; ++j) + for (Index i = 0; i < n; ++i) + if (((Mode & Lower) ? i < j : i > j) || ((Mode & (UnitDiag | ZeroDiag)) && i == j)) + a(i, j) = Scalar(NumTraits<Real>::quiet_NaN()); + Mat actual = Mat::Zero(n, cols); + Blocking blocking(narrowBlock ? panel + 1 : n, cols, kc); + Kernel::run(n, cols, n, a.data(), a.outerStride(), b.data(), b.outerStride(), actual.data(), 1, + actual.outerStride(), Scalar(1), blocking); + VERIFY(verifyProduct(actual, triangle.lazyProduct(b).eval(), triangle, b)); + } +} + #define CALL_ALL_ORDERS(NB, SCALAR, MODE) \ EIGEN_CAT(CALL_SUBTEST_, NB)((trmm<SCALAR, MODE, ColMajor, ColMajor, ColMajor>())); \ EIGEN_CAT(CALL_SUBTEST_, NB)((trmm<SCALAR, MODE, ColMajor, ColMajor, RowMajor>())); \ @@ -202,8 +323,47 @@ CALL_ALL(3, std::complex<float>); // EIGEN_SUFFIXES;13;113;23;123;33;133 CALL_ALL(4, std::complex<double>); // EIGEN_SUFFIXES;14;114;24;124;34;134 } + CALL_SUBTEST_11((trmm_panel_boundaries<float, Lower, ColMajor>())); + CALL_SUBTEST_12((trmm_panel_boundaries<double, Upper, RowMajor>())); + CALL_SUBTEST_13((trmm_panel_boundaries<std::complex<float>, Upper, ColMajor>())); + CALL_SUBTEST_14((trmm_panel_boundaries<std::complex<double>, Lower, RowMajor>())); + CALL_SUBTEST_21((trmm_panel_boundaries<float, UnitLower, ColMajor>())); + CALL_SUBTEST_22((trmm_panel_boundaries<double, UnitUpper, RowMajor>())); + CALL_SUBTEST_31((trmm_panel_boundaries<float, StrictlyLower, ColMajor>())); + CALL_SUBTEST_32((trmm_panel_boundaries<double, StrictlyUpper, RowMajor>())); + CALL_SUBTEST_11((trmm_wide_depth_blocks<float, Lower, ColMajor>())); + CALL_SUBTEST_12((trmm_wide_depth_blocks<double, Upper, ColMajor>())); + CALL_SUBTEST_13((trmm_wide_depth_blocks<std::complex<float>, Lower, RowMajor>())); + CALL_SUBTEST_14((trmm_wide_depth_blocks<std::complex<double>, Upper, ColMajor>())); + CALL_SUBTEST_21((trmm_wide_depth_blocks<float, UnitUpper, RowMajor>())); + CALL_SUBTEST_22((trmm_wide_depth_blocks<double, UnitLower, ColMajor>())); + CALL_SUBTEST_31((trmm_wide_depth_blocks<float, StrictlyLower, ColMajor>())); + CALL_SUBTEST_32((trmm_wide_depth_blocks<double, StrictlyUpper, RowMajor>())); // 1x1 and blocking-boundary sizes for the view-times-view products. CALL_SUBTEST_11((trmm<float, Lower, ColMajor, ColMajor, ColMajor>(1, 1, 1))); + CALL_SUBTEST_11((trmm<float, Lower, ColMajor, ColMajor, ColMajor>(6, 6, 6))); CALL_SUBTEST_14((trmm<std::complex<double>, Upper, RowMajor, ColMajor, RowMajor>(1, 1, 1))); CALL_SUBTEST_12((trmm<double, Upper, ColMajor, ColMajor, ColMajor>(EIGEN_TEST_MAX_SIZE, EIGEN_TEST_MAX_SIZE, 1))); + // Trapezoids spanning several depth blocks, with partial diagonal panels and packet tails. + CALL_SUBTEST_11((trmm<float, Lower, ColMajor, ColMajor, ColMajor>(513, 521, 517))); + CALL_SUBTEST_12((trmm<double, Upper, RowMajor, RowMajor, RowMajor>(521, 513, 517))); + CALL_SUBTEST_21((trmm<float, UnitLower, ColMajor, RowMajor, RowMajor>(521, 513, 517))); + CALL_SUBTEST_22((trmm<double, UnitUpper, RowMajor, ColMajor, ColMajor>(513, 521, 517))); + CALL_SUBTEST_31((trmm<float, StrictlyUpper, ColMajor, ColMajor, ColMajor>(521, 513, 517))); + CALL_SUBTEST_32((trmm<double, StrictlyLower, RowMajor, RowMajor, RowMajor>(513, 521, 517))); + CALL_SUBTEST_11((trmm_maps<Lower, Matrix<float, 24, 20>, Matrix<float, 20, 16>, Matrix<float, 24, 16>>(24, 20, 16))); + // Upper trapezoids filling the inline blockB, with a packed triangle (depth^2 scalars) that ends off a packet + // boundary, so the general panel after it is moved forward to realign. Large enough (rows + depth + cols >= + // EIGEN_GEMM_TO_COEFFBASED_THRESHOLD) to be packed rather than computed coefficient-wise. + CALL_SUBTEST_11((trmm_maps<Upper, Matrix<float, 8, 7>, Matrix<float, 7, 9>, Matrix<float, 8, 9>>(8, 7, 9))); + CALL_SUBTEST_12((trmm_maps<Upper, Matrix<double, 8, 7>, Matrix<double, 7, 9>, Matrix<double, 8, 9>>(8, 7, 9))); + CALL_SUBTEST_12( + (trmm_maps<UnitUpper, Matrix<double, 13, 17, RowMajor>, Matrix<double, 17, 11>, Matrix<double, 13, 11, RowMajor>>( + 13, 17, 11))); + CALL_SUBTEST_21( + (trmm_maps<Lower, Matrix<float, 517, Dynamic>, MatrixXf, Matrix<float, 517, Dynamic>>(517, 521, 513))); + CALL_SUBTEST_22( + (trmm_maps<StrictlyUpper, Matrix<double, Dynamic, Dynamic, RowMajor, 528, Dynamic>, + Matrix<double, Dynamic, Dynamic, RowMajor>, Matrix<double, Dynamic, Dynamic, RowMajor, 528, Dynamic>>( + 521, 513, 517))); }
diff --git a/test/product_trmv.cpp b/test/product_trmv.cpp index 067df4b..b7618b9 100644 --- a/test/product_trmv.cpp +++ b/test/product_trmv.cpp
@@ -117,6 +117,43 @@ trmv_negative_stride_destination_shapes<Scalar, RowMajor>(); } +// A real triangular matrix times a complex vector at a fixed size, through the column-major kernel (directly and as the +// transpose of a row-vector product) and the row-major one. The unevaluated s * v.adjoint() reaches the kernel as a +// conjugated vector and a scale factor, as in #3214. Without the workaround, GCC 15.2 at -O3 with AVX2 gets sizes 3 and +// 4 wrong here; which sizes it miscompiles depends on every product in the translation unit. +template <int Size, int Mode> +void trmv_real_times_complex_fixed_size() { + using CD = std::complex<double>; + using RealMat = Matrix<double, Size, Size>; + using ComplexMat = Matrix<CD, Size, Size>; + using ComplexVec = Matrix<CD, Size, 1>; + using ComplexRowVec = Matrix<CD, 1, Size>; + const RealMat m = RealMat::Random(); + const ComplexVec v = ComplexVec::Random(); + const CD s = internal::random<CD>(); + const ComplexMat dense = m.template triangularView<Mode>().toDenseMatrix().template cast<CD>(); + const ComplexMat dense_of_transpose = + m.transpose().template triangularView<Mode>().toDenseMatrix().template cast<CD>(); + + const ComplexVec sv = s * v; + VERIFY(verifyProduct(ComplexVec(m.template triangularView<Mode>() * sv), ComplexVec(dense * sv), dense, sv)); + VERIFY(verifyProduct(ComplexVec(m.transpose().template triangularView<Mode>() * sv), + ComplexVec(dense_of_transpose * sv), dense_of_transpose, sv)); + const ComplexRowVec svh = s * v.adjoint(); + VERIFY(verifyProduct(ComplexRowVec(svh * m.transpose().template triangularView<Mode>()), + ComplexRowVec(svh * dense_of_transpose), svh, dense_of_transpose)); + VERIFY(verifyProduct(ComplexRowVec(s * v.adjoint() * m.transpose().template triangularView<Mode>()), + ComplexRowVec(svh * dense_of_transpose), svh, dense_of_transpose)); +} + +template <int Size> +void trmv_real_times_complex_fixed_size_all_modes() { + trmv_real_times_complex_fixed_size<Size, Lower>(); + trmv_real_times_complex_fixed_size<Size, Upper>(); + trmv_real_times_complex_fixed_size<Size, UnitLower>(); + trmv_real_times_complex_fixed_size<Size, UnitUpper>(); +} + EIGEN_DECLARE_TEST(product_trmv) { int s = 0; for (int i = 0; i < g_repeat; i++) { @@ -132,6 +169,11 @@ s = internal::random<int>(1, EIGEN_TEST_MAX_SIZE); CALL_SUBTEST_6(trmv(Matrix<float, Dynamic, Dynamic, RowMajor>(s, s))); TEST_SET_BUT_UNUSED_VARIABLE(s); + + CALL_SUBTEST_8(trmv_real_times_complex_fixed_size_all_modes<2>()); + CALL_SUBTEST_8(trmv_real_times_complex_fixed_size_all_modes<3>()); + CALL_SUBTEST_8(trmv_real_times_complex_fixed_size_all_modes<4>()); + CALL_SUBTEST_8(trmv_real_times_complex_fixed_size_all_modes<5>()); } CALL_SUBTEST_7(trmv_negative_stride_destination_all<float>()); CALL_SUBTEST_7(trmv_negative_stride_destination_all<double>());
diff --git a/test/qr_rand_colpivoting.cpp b/test/qr_rand_colpivoting.cpp index 12223ee..c92806d 100644 --- a/test/qr_rand_colpivoting.cpp +++ b/test/qr_rand_colpivoting.cpp
@@ -664,11 +664,11 @@ CALL_SUBTEST_1(rcod<MatrixXf>()); CALL_SUBTEST_2(rcod<MatrixXd>()); CALL_SUBTEST_3(rcod<MatrixXcd>()); - CALL_SUBTEST_4((rcod_fixedsize<Matrix<float, 8, 10>, 4>())); + CALL_SUBTEST_10((rcod_fixedsize<Matrix<float, 8, 10>, 4>())); CALL_SUBTEST_5((rcod_fixedsize<Matrix<double, 12, 6>, 3>())); CALL_SUBTEST_2((rcod<Matrix<double, Dynamic, Dynamic, RowMajor>>())); - CALL_SUBTEST_4((rcod_fixedsize<Matrix<float, 8, 10, RowMajor>, 4>())); + CALL_SUBTEST_11((rcod_fixedsize<Matrix<float, 8, 10, RowMajor>, 4>())); } for (int i = 0; i < g_repeat; i++) {
diff --git a/test/qtvector.cpp b/test/qtvector.cpp deleted file mode 100644 index f545a4e..0000000 --- a/test/qtvector.cpp +++ /dev/null
@@ -1,141 +0,0 @@ -// This file is part of Eigen, a lightweight C++ template library -// for linear algebra. -// -// Copyright (C) 2008 Gael Guennebaud <gael.guennebaud@inria.fr> -// Copyright (C) 2008 Benoit Jacob <jacob.benoit.1@gmail.com> -// -// This Source Code Form is subject to the terms of the Mozilla -// Public License v. 2.0. If a copy of the MPL was not distributed -// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. -// SPDX-License-Identifier: MPL-2.0 - -#include "main.h" -#include <QtCore/QVector> -#include <Eigen/Geometry> -#include <Eigen/QtAlignedMalloc> - -template <typename MatrixType> -void check_qtvector_matrix(const MatrixType& m) { - Index rows = m.rows(); - Index cols = m.cols(); - MatrixType x = MatrixType::Random(rows, cols), y = MatrixType::Random(rows, cols); - QVector<MatrixType> v(10, MatrixType(rows, cols)), w(20, y); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], y); - } - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.fill(y, 22); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - VERIFY((size_t) & (v[22]) == (size_t) & (v[21]) + sizeof(MatrixType)); - - // do a lot of push_back such that the vector gets internally resized - // (with memory reallocation) - MatrixType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (int i = 23; i < v.size(); ++i) { - VERIFY(v[i] == w[(i - 23) % w.size()]); - } -} - -template <typename TransformType> -void check_qtvector_transform(const TransformType&) { - typedef typename TransformType::MatrixType MatrixType; - TransformType x(MatrixType::Random()), y(MatrixType::Random()); - QVector<TransformType> v(10), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.fill(y, 22); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - VERIFY((size_t) & (v[22]) == (size_t) & (v[21]) + sizeof(TransformType)); - - // do a lot of push_back such that the vector gets internally resized - // (with memory reallocation) - TransformType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; int(i) < v.size(); ++i) { - VERIFY(v[i].matrix() == w[(i - 23) % w.size()].matrix()); - } -} - -template <typename QuaternionType> -void check_qtvector_quaternion(const QuaternionType&) { - typedef typename QuaternionType::Coefficients Coefficients; - QuaternionType x(Coefficients::Random()), y(Coefficients::Random()); - QVector<QuaternionType> v(10), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.fill(y, 22); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - VERIFY((size_t) & (v[22]) == (size_t) & (v[21]) + sizeof(QuaternionType)); - - // do a lot of push_back such that the vector gets internally resized - // (with memory reallocation) - QuaternionType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; int(i) < v.size(); ++i) { - VERIFY(v[i].coeffs() == w[(i - 23) % w.size()].coeffs()); - } -} - -EIGEN_DECLARE_TEST(qtvector) { - // some non vectorizable fixed sizes - CALL_SUBTEST(check_qtvector_matrix(Vector2f())); - CALL_SUBTEST(check_qtvector_matrix(Matrix3f())); - CALL_SUBTEST(check_qtvector_matrix(Matrix3d())); - - // some vectorizable fixed sizes - CALL_SUBTEST(check_qtvector_matrix(Matrix2f())); - CALL_SUBTEST(check_qtvector_matrix(Vector4f())); - CALL_SUBTEST(check_qtvector_matrix(Matrix4f())); - CALL_SUBTEST(check_qtvector_matrix(Matrix4d())); - - // some dynamic sizes - CALL_SUBTEST(check_qtvector_matrix(MatrixXd(1, 1))); - CALL_SUBTEST(check_qtvector_matrix(VectorXd(20))); - CALL_SUBTEST(check_qtvector_matrix(RowVectorXf(20))); - CALL_SUBTEST(check_qtvector_matrix(MatrixXcf(10, 10))); - - // some Transform - CALL_SUBTEST(check_qtvector_transform(Affine2f())); - CALL_SUBTEST(check_qtvector_transform(Affine3f())); - CALL_SUBTEST(check_qtvector_transform(Affine3d())); - // CALL_SUBTEST(check_qtvector_transform(Transform4d())); - - // some Quaternion - CALL_SUBTEST(check_qtvector_quaternion(Quaternionf())); - CALL_SUBTEST(check_qtvector_quaternion(Quaterniond())); -}
diff --git a/test/random_matrix.cpp b/test/random_matrix.cpp index 978d370..f155545 100644 --- a/test/random_matrix.cpp +++ b/test/random_matrix.cpp
@@ -98,7 +98,48 @@ check_generateRandomMatrixSvs<MatrixType, RealScalar, RealVectorType>(rows, cols, diag_size, min_svs, max_svs); } +template <typename DenseType> +void check_random_range() { + using Scalar = typename DenseType::Scalar; + using RealScalar = typename NumTraits<Scalar>::Real; + DenseType matrix = DenseType::Constant(4, 5, Scalar(99)); + auto block = matrix.template block<2, 3>(1, 1); + setRandomInRange(block, RealScalar(2), RealScalar(4)); + for (Index i = 0; i < matrix.rows(); ++i) { + for (Index j = 0; j < matrix.cols(); ++j) { + if (i >= 1 && i < 3 && j >= 1 && j < 4) { + VERIFY_GE(numext::real(matrix(i, j)), RealScalar(2)); + VERIFY_LE(numext::real(matrix(i, j)), RealScalar(4)); + VERIFY_IS_EQUAL(numext::imag(matrix(i, j)), RealScalar(0)); + } else { + VERIFY_IS_EQUAL(matrix(i, j), Scalar(99)); + } + } + } + + const DenseType generated = RandomMatrix<DenseType>(4, 5, RealScalar(2), RealScalar(4)); + VERIFY_IS_EQUAL(generated.rows(), 4); + VERIFY_IS_EQUAL(generated.cols(), 5); + VERIFY((generated.real().array() >= RealScalar(2)).all()); + VERIFY((generated.real().array() <= RealScalar(4)).all()); + VERIFY(generated.imag().isZero()); + + setRandomDataInRange(matrix, Scalar(3), Scalar(3)); + VERIFY_IS_CWISE_EQUAL(matrix, DenseType::Constant(4, 5, Scalar(3))); + auto empty = matrix.block(0, 0, 0, 0); + setRandomInRange(empty, RealScalar(2), RealScalar(4)); + VERIFY_IS_CWISE_EQUAL(matrix, DenseType::Constant(4, 5, Scalar(3))); +} + EIGEN_DECLARE_TEST(random_matrix) { + CALL_SUBTEST_17((check_random_range<Array<float, 4, 5, ColMajor>>())); + CALL_SUBTEST_17((check_random_range<Array<double, Dynamic, Dynamic, RowMajor>>())); + CALL_SUBTEST_17((check_random_range<Array<int, 4, 5, RowMajor>>())); + CALL_SUBTEST_17((check_random_range<Array<unsigned char, Dynamic, Dynamic, ColMajor>>())); + CALL_SUBTEST_17((check_random_range<Array<std::complex<float>, 4, 5, ColMajor>>())); + CALL_SUBTEST_17((check_random_range<Array<std::complex<double>, Dynamic, Dynamic, RowMajor>>())); + CALL_SUBTEST_17((check_random_range<Matrix<float, 4, 5, RowMajor>>())); + CALL_SUBTEST_17((check_random_range<Matrix<std::complex<double>, Dynamic, Dynamic, ColMajor>>())); for (int i = 0; i < g_repeat; i++) { CALL_SUBTEST_1(check_random_matrix(Matrix<float, 1, 1>())); CALL_SUBTEST_2(check_random_matrix(Matrix<float, 4, 4>()));
diff --git a/test/random_matrix_helper.h b/test/random_matrix_helper.h index b96a371..2d3a0e6 100644 --- a/test/random_matrix_helper.h +++ b/test/random_matrix_helper.h
@@ -272,22 +272,30 @@ } // end namespace Eigen +// Fill in row-major coefficient order, including non-contiguous dense views. +// The range type determines the draws: real bounds produce real values even in a complex destination. +template <typename Derived, typename Scalar> +void setRandomInRange(Eigen::DenseBase<Derived>& matrix, Scalar min_value, Scalar max_value) { + for (Eigen::Index i = 0; i < matrix.rows(); ++i) { + for (Eigen::Index j = 0; j < matrix.cols(); ++j) { + matrix.derived().coeffRef(i, j) = Eigen::internal::random<Scalar>(min_value, max_value); + } + } +} + +// Requires contiguous storage, as in Tensor and TensorMap. template <typename DataContainer> void setRandomDataInRange(DataContainer& data_container, typename DataContainer::Scalar min_value, typename DataContainer::Scalar max_value) { - for (Eigen::Index i = 0; i < data_container.size(); ++i) { - data_container.data()[i] = Eigen::internal::random<typename DataContainer::Scalar>(min_value, max_value); - } + using Scalar = typename DataContainer::Scalar; + Eigen::Map<Eigen::Matrix<Scalar, Eigen::Dynamic, 1>> values(data_container.data(), data_container.size()); + setRandomInRange(values, min_value, max_value); } template <typename MatrixType, typename Scalar = typename MatrixType::Scalar> MatrixType RandomMatrix(Eigen::Index rows, Eigen::Index cols, Scalar min, Scalar max) { MatrixType M(rows, cols); - for (Eigen::Index i = 0; i < rows; ++i) { - for (Eigen::Index j = 0; j < cols; ++j) { - M(i, j) = Eigen::internal::random<Scalar>(min, max); - } - } + setRandomInRange(M, min, max); return M; }
diff --git a/test/redux.cpp b/test/redux.cpp index 7cbb914..1281e32 100644 --- a/test/redux.cpp +++ b/test/redux.cpp
@@ -25,9 +25,11 @@ MatrixType m1 = MatrixType::Random(rows, cols); - // The entries of m1 are uniformly distributed in [-1,1), so m1.prod() is very small. This may lead to test - // failures if we underflow into denormals. Thus, we scale so that entries are close to 1. - MatrixType m1_for_prod = MatrixType::Ones(rows, cols) + RealScalar(0.2) * m1; + // With N = rows*cols <= 10000, the spread s = min(0.2, 20/N) keeps every partial product above + // (1 - s)^N >= 0.8^100 ~ e^-22, even for all-negative inputs, and s = 0.2 up to N = 100 keeps a dropped + // factor visible to VERIFY_IS_APPROX. A fixed spread can underflow: for real inputs E[log(prod)] ~ -N. + const RealScalar product_scale = numext::mini(RealScalar(0.2), RealScalar(20) / RealScalar(rows * cols)); + MatrixType m1_for_prod = MatrixType::Ones(rows, cols) + product_scale * m1; Matrix<Scalar, MatrixType::RowsAtCompileTime, MatrixType::RowsAtCompileTime> m2(rows, rows); m2.setRandom(); @@ -58,8 +60,26 @@ VERIFY_IS_APPROX(m1.real().maxCoeff(), numext::real(maxc)); // test that partial reduction works if nested expressions is forced to evaluate early - VERIFY_IS_APPROX((m1.matrix() * m1.matrix().transpose()).cwiseProduct(m2.matrix()).rowwise().sum().sum(), - (m1.matrix() * m1.matrix().transpose()).eval().cwiseProduct(m2.matrix()).rowwise().sum().sum()); + const Scalar nested = (m1.matrix() * m1.matrix().transpose()).cwiseProduct(m2.matrix()).rowwise().sum().sum(); + const Scalar evaluated = + (m1.matrix() * m1.matrix().transpose()).eval().cwiseProduct(m2.matrix()).rowwise().sum().sum(); + double absolute_sum = 0; + for (Index i = 0; i < rows; ++i) { + for (Index j = 0; j < rows; ++j) { + double absolute_dot = 0; + for (Index k = 0; k < cols; ++k) + absolute_dot += static_cast<double>(numext::abs(m1(i, k))) * static_cast<double>(numext::abs(m1(j, k))); + absolute_sum += absolute_dot * static_cast<double>(numext::abs(m2(i, j))); + } + } + // Cancellation requires an absolute bound. Each evaluation has depth cols + 2*rows + 1; + // compare two rounding paths, allowing four real rounding stages per complex term. + const double bound = (NumTraits<Scalar>::IsComplex ? 8.0 : 2.0) * double(cols + 2 * rows + 1) * + static_cast<double>(NumTraits<RealScalar>::epsilon()) * absolute_sum; + VERIFY((numext::isfinite)(bound)); + const double error = static_cast<double>(numext::abs(nested - evaluated)); + if (!(error <= bound)) std::cerr << "nested reduction: error " << error << " > bound " << bound << std::endl; + VERIFY(error <= bound); // test slice vectorization assuming assign is ok Index r0 = internal::random<Index>(0, rows - 1); @@ -98,7 +118,8 @@ Index size = w.size(); VectorType v = VectorType::Random(size); - VectorType v_for_prod = VectorType::Ones(size) + Scalar(0.2) * v; // see comment above declaration of m1_for_prod + // Vectors have at most 100 entries, so a fixed spread around 1 keeps their products normal. + VectorType v_for_prod = VectorType::Ones(size) + Scalar(0.2) * v; if (Eigen::NumTraits<Scalar>::IsInteger) { Scalar kMaxVal = Scalar(8); v.array() = v.array() - kMaxVal * (v.array() / kMaxVal); @@ -256,6 +277,16 @@ Map<const Vec, Unaligned> mapped_for_prod(buf_for_prod.data() + offset, n); redux_vec_boundary_check(mapped, mapped_for_prod); } + { + using RealScalar = typename NumTraits<Scalar>::Real; + Matrix<RealScalar, Dynamic, 1> raw_buf(2 * n + 1); + Matrix<RealScalar, Dynamic, 1> raw_buf_for_prod(2 * n + 1); + Map<Vec, Unaligned> mapped(reinterpret_cast<Scalar*>(raw_buf.data() + 1), n); + Map<Vec, Unaligned> mapped_for_prod(reinterpret_cast<Scalar*>(raw_buf_for_prod.data() + 1), n); + mapped = v; + mapped_for_prod = v_for_prod; + redux_vec_boundary_check(mapped, mapped_for_prod); + } } } @@ -715,6 +746,8 @@ // Integer reductions are already tested by matrixRedux/vectorRedux with clamped values. CALL_SUBTEST_12(redux_vec_boundary<float>()); CALL_SUBTEST_12(redux_vec_boundary<double>()); + CALL_SUBTEST_12(redux_vec_boundary<std::complex<float>>()); + CALL_SUBTEST_12(redux_vec_boundary<std::complex<double>>()); CALL_SUBTEST_12(redux_block_boundary<float>()); CALL_SUBTEST_12(redux_block_boundary<double>()); CALL_SUBTEST_12(redux_block_boundary<std::complex<float>>());
diff --git a/test/redux_unrolling.cpp b/test/redux_unrolling.cpp new file mode 100644 index 0000000..31e977b --- /dev/null +++ b/test/redux_unrolling.cpp
@@ -0,0 +1,87 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// This Source Code Form is subject to the terms of the Mozilla +// Public License v. 2.0. If a copy of the MPL was not distributed +// with this file, You can obtain one at http://mozilla.org/MPL/2.0/. +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +// Reductions of a runtime size must not unroll, however large the unrolling budget grows. This limit reaches HugeCost, +// the cost of a runtime size, with or without vectorization. The default limit of 110 does so from 91 scalars per +// packet on (RVV at VLEN 1024 and LMUL 4), where the budget scales with the packet size. +#define EIGEN_UNROLLING_LIMIT 10000 +#include "main.h" +#include "random_for_arithmetic.h" + +template <typename Xpr> +typename Xpr::Scalar sum_by_coeff(const Xpr& x) { + typename Xpr::Scalar sum(0); + for (Index j = 0; j < x.cols(); ++j) + for (Index i = 0; i < x.rows(); ++i) sum += x.coeff(i, j); + return sum; +} + +template <typename Xpr> +constexpr int sum_unrolling() { + using Scalar = typename Xpr::Scalar; + return internal::redux_traits<internal::scalar_sum_op<Scalar, Scalar>, internal::redux_evaluator<Xpr>>::Unrolling; +} + +template <typename Panel> +constexpr int packetwise_sum_unrolling() { + using Scalar = typename Panel::Scalar; + return internal::packetwise_redux_traits<internal::scalar_sum_op<Scalar, Scalar>, + internal::redux_evaluator<Panel>>::Unrolling; +} + +template <typename Scalar> +void redux_runtime_size() { + using Vec = Matrix<Scalar, Dynamic, 1>; + using Mat = Matrix<Scalar, Dynamic, Dynamic, ColMajor>; + using ConstBlock = Block<const Mat>; + using Panel = Block<const Mat, 4, Dynamic>; + STATIC_CHECK(sum_unrolling<Vec>() == NoUnrolling); + STATIC_CHECK(sum_unrolling<Mat>() == NoUnrolling); + // Matters where a block takes DefaultTraversal: SliceVectorizedTraversal ignores Unrolling. + STATIC_CHECK(sum_unrolling<ConstBlock>() == NoUnrolling); + // LinearTraversal for complex, whose budget scales with the real packet size although it reads no packets. + STATIC_CHECK(sum_unrolling<typename Mat::CwiseAbs2ReturnType>() == NoUnrolling); + STATIC_CHECK(packetwise_sum_unrolling<Panel>() == NoUnrolling); + + // Shifted so that the sums do not cancel, and long enough to read packets of 128 scalars. + const Vec v = Vec::Constant(300, Scalar(8)) + random_for_arithmetic<Vec>(300); + const Mat m = Mat::Constant(300, 11, Scalar(8)) + random_for_arithmetic<Mat>(300, 11); + const ConstBlock block = m.block(1, 2, 15, 7); + VERIFY_IS_APPROX(v.sum(), sum_by_coeff(v)); + VERIFY_IS_APPROX(m.sum(), sum_by_coeff(m)); + VERIFY_IS_APPROX(block.sum(), sum_by_coeff(block)); + VERIFY_IS_APPROX(v.squaredNorm(), sum_by_coeff(v.cwiseAbs2())); + VERIFY_IS_APPROX(m.squaredNorm(), sum_by_coeff(m.cwiseAbs2())); + VERIFY_IS_APPROX(v.cwiseAbs2().sum(), sum_by_coeff(v.cwiseAbs2())); + VERIFY_IS_APPROX(m.cwiseAbs2().sum(), sum_by_coeff(m.cwiseAbs2())); + // rowwise() of a column-major operand and colwise() of a row-major one reduce panels of packets. + VERIFY_IS_APPROX(m.rowwise().sum().sum(), sum_by_coeff(m)); + VERIFY_IS_APPROX(m.transpose().colwise().sum().sum(), sum_by_coeff(m)); +} + +template <typename Scalar> +void redux_fixed_size() { + using Vec = Matrix<Scalar, 4, 1>; + using Panel = Block<const Matrix<Scalar, Dynamic, 4, ColMajor>, 4, 4>; + STATIC_CHECK(sum_unrolling<Vec>() == CompleteUnrolling); + STATIC_CHECK(packetwise_sum_unrolling<Panel>() == CompleteUnrolling); + const Vec v = Vec::Constant(Scalar(8)) + Vec::Random(); + VERIFY_IS_APPROX(v.sum(), sum_by_coeff(v)); +} + +EIGEN_DECLARE_TEST(redux_unrolling) { + CALL_SUBTEST(redux_runtime_size<float>()); + CALL_SUBTEST(redux_runtime_size<double>()); + CALL_SUBTEST(redux_runtime_size<long double>()); + CALL_SUBTEST(redux_runtime_size<std::complex<float>>()); + CALL_SUBTEST(redux_runtime_size<std::complex<double>>()); + CALL_SUBTEST(redux_runtime_size<int>()); + CALL_SUBTEST(redux_fixed_size<float>()); + CALL_SUBTEST(redux_fixed_size<std::complex<double>>()); +}
diff --git a/test/scalar_binary_op_overloads.cpp b/test/scalar_binary_op_overloads.cpp new file mode 100644 index 0000000..d5a50d9 --- /dev/null +++ b/test/scalar_binary_op_overloads.cpp
@@ -0,0 +1,580 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include "main.h" +#define EIGEN_TEST_ANNOYING_SCALAR_DONT_THROW +#include "AnnoyingScalar.h" +#include <Eigen/Geometry> +#include <Eigen/SparseCore> + +template <int Dimension> +struct QuantityScalar { + double value = 0; + QuantityScalar() = default; + QuantityScalar(double x) : value(x) {} +}; + +template <int A, int B> +QuantityScalar<A + B> operator*(QuantityScalar<A> lhs, QuantityScalar<B> rhs) { + return lhs.value * rhs.value; +} +template <int A> +QuantityScalar<A> operator+(QuantityScalar<A> lhs, QuantityScalar<A> rhs) { + return lhs.value + rhs.value; +} +template <int A> +QuantityScalar<A>& operator+=(QuantityScalar<A>& lhs, QuantityScalar<A> rhs) { + lhs.value += rhs.value; + return lhs; +} + +namespace Eigen { +template <int A> +struct NumTraits<QuantityScalar<A>> : NumTraits<double> { + using Real = QuantityScalar<A>; + using NonInteger = QuantityScalar<A>; + using Nested = QuantityScalar<A>; + using Literal = QuantityScalar<A>; + static constexpr bool RequireInitialization = true; +}; +template <int A, typename B> +struct ScalarBinaryOpTraits<QuantityScalar<A>, B, internal::scalar_product_op<QuantityScalar<A>, B>> { + using ReturnType = decltype(std::declval<QuantityScalar<A>>() * std::declval<B>()); +}; +template <typename A, int B> +struct ScalarBinaryOpTraits<A, QuantityScalar<B>, internal::scalar_product_op<A, QuantityScalar<B>>> { + using ReturnType = decltype(std::declval<A>() * std::declval<QuantityScalar<B>>()); +}; +template <int A, int B> +struct ScalarBinaryOpTraits<QuantityScalar<A>, QuantityScalar<B>, + internal::scalar_product_op<QuantityScalar<A>, QuantityScalar<B>>> { + using ReturnType = QuantityScalar<A + B>; +}; +} // namespace Eigen + +template <int Size, int Order, int RhsDimension> +void quantity_scalar_product() { + Matrix<QuantityScalar<1>, Size, Size, Order> lhs(2, 2); + Matrix<QuantityScalar<RhsDimension>, Size, Size, Order> rhs(2, 2); + for (Index j = 0; j < 2; ++j) + for (Index i = 0; i < 2; ++i) { + lhs(i, j) = double(i + 2 * j + 1); + rhs(i, j) = double(2 * i - j + 1); + } + const Matrix<QuantityScalar<1 + RhsDimension>, Size, Size, Order> result = lhs * rhs; + const Matrix<QuantityScalar<1>, Dynamic, Dynamic, Order> dynamic_lhs = lhs; + const Matrix<QuantityScalar<RhsDimension>, Dynamic, Dynamic, Order> dynamic_rhs = rhs; + const Matrix<QuantityScalar<1 + RhsDimension>, Dynamic, Dynamic, Order> lazy_result = + dynamic_lhs.lazyProduct(dynamic_rhs); + for (Index j = 0; j < 2; ++j) + for (Index i = 0; i < 2; ++i) { + double expected = 0; + for (Index k = 0; k < 2; ++k) expected += lhs(i, k).value * rhs(k, j).value; + VERIFY_IS_EQUAL(result(i, j).value, expected); + VERIFY_IS_EQUAL(lazy_result(i, j).value, expected); + } +} + +template <typename Kind> +struct EigenBaseScalar; +namespace Eigen { +namespace internal { +template <typename Kind> +struct traits<EigenBaseScalar<Kind>> : traits<Matrix2d> { + using XprKind = Kind; +}; +} // namespace internal +} // namespace Eigen + +template <typename Kind> +struct EigenBaseScalar : Eigen::EigenBase<EigenBaseScalar<Kind>> { + double value = 3; + Eigen::Index rows() const { return 2; } + Eigen::Index cols() const { return 2; } +}; + +template <typename Kind> +double operator*(double lhs, const EigenBaseScalar<Kind>& rhs) { + return lhs * rhs.value; +} +template <typename Kind> +double operator*(const EigenBaseScalar<Kind>& lhs, double rhs) { + return lhs.value * rhs; +} + +struct PrivateMatrixScalar : private Eigen::Matrix2d { + PrivateMatrixScalar() = default; + explicit PrivateMatrixScalar(double value) : value(value) {} + using Eigen::Matrix2d::derived; + double value = 0; +}; + +using BoundedRows = Eigen::Matrix<double, Eigen::Dynamic, 2, Eigen::ColMajor, 2, 2>; +using BoundedCols = Eigen::Matrix<double, 2, Eigen::Dynamic, Eigen::ColMajor, 2, 2>; +using BoundedArray = Eigen::Array<double, Eigen::Dynamic, Eigen::Dynamic, Eigen::ColMajor, 2, 2>; +using ScalarMatrix = Eigen::Matrix<double, 1, 1>; +using ScalarArray = Eigen::Array<double, 1, 1>; + +struct BaseReferenceScalar final { + BaseReferenceScalar(double value = 0) : value(value) {} + double value; +}; +using BaseReferenceMatrix = Eigen::Matrix<BaseReferenceScalar, 1, 1>; +using BaseReferenceArray = Eigen::Array<BaseReferenceScalar, 1, 1>; + +BaseReferenceScalar operator*(BaseReferenceScalar lhs, double rhs) { return lhs.value * rhs; } +BaseReferenceScalar operator*(double lhs, BaseReferenceScalar rhs) { return lhs * rhs.value; } +BaseReferenceScalar operator+(BaseReferenceScalar lhs, BaseReferenceScalar rhs) { return lhs.value + rhs.value; } +BaseReferenceScalar operator*(BaseReferenceScalar lhs, const ScalarMatrix& rhs) { return lhs.value + rhs(0); } +BaseReferenceScalar operator*(BaseReferenceScalar lhs, const ScalarArray& rhs) { return lhs.value + rhs(0); } + +struct DiagonalOperandScalar final { + DiagonalOperandScalar(double value = 0) : value(value) {} + double value; +}; +using DiagonalOperandMatrix = Eigen::Matrix<DiagonalOperandScalar, 1, 1>; +using ScalarDiagonal = Eigen::DiagonalMatrix<double, 1>; + +DiagonalOperandScalar operator*(const DiagonalOperandScalar& lhs, double rhs) { return lhs.value * rhs; } +DiagonalOperandScalar operator*(double lhs, const DiagonalOperandScalar& rhs) { return lhs * rhs.value; } +template <typename Derived> +DiagonalOperandScalar operator*(const DiagonalOperandScalar& lhs, const Eigen::DiagonalBase<Derived>& rhs) { + return lhs.value + rhs.diagonal()(0); +} +template <typename Derived> +DiagonalOperandScalar operator*(const Eigen::DiagonalBase<Derived>& lhs, const DiagonalOperandScalar& rhs) { + return lhs.diagonal()(0) + rhs.value; +} + +double operator*(double lhs, const PrivateMatrixScalar& rhs) { return lhs * rhs.value; } +double operator*(const PrivateMatrixScalar& lhs, double rhs) { return lhs.value * rhs; } +double operator/(const PrivateMatrixScalar& lhs, const Eigen::Matrix2d& rhs) { return lhs.value / rhs.sum(); } +Eigen::Matrix2d operator*(const PrivateMatrixScalar& lhs, const Eigen::Matrix2d& rhs) { return lhs.value * rhs; } +Eigen::Matrix2d operator*(const PrivateMatrixScalar& lhs, const BoundedRows& rhs) { return lhs.value * rhs; } +Eigen::Matrix2d operator*(const BoundedCols& lhs, const PrivateMatrixScalar& rhs) { return lhs * rhs.value; } +Eigen::Array22d operator*(const PrivateMatrixScalar& lhs, const BoundedArray& rhs) { return lhs.value * rhs; } +Eigen::Array22d operator*(const BoundedArray& lhs, const PrivateMatrixScalar& rhs) { return lhs * rhs.value; } +double operator*(const PrivateMatrixScalar& lhs, const ScalarMatrix& rhs) { return lhs.value + rhs(0); } +double operator*(const ScalarMatrix& lhs, const PrivateMatrixScalar& rhs) { return lhs(0) + rhs.value; } +double operator*(const PrivateMatrixScalar& lhs, const ScalarArray& rhs) { return lhs.value + rhs(0); } +double operator*(const ScalarArray& lhs, const PrivateMatrixScalar& rhs) { return lhs(0) + rhs.value; } +double operator+(const PrivateMatrixScalar& lhs, const ScalarArray& rhs) { return 2 * lhs.value + rhs(0); } +double operator+(const ScalarArray& lhs, const PrivateMatrixScalar& rhs) { return lhs(0) + 2 * rhs.value; } +double pow(const ScalarArray& base, const PrivateMatrixScalar& exponent) { return base(0) + exponent.value; } +double pow(const Eigen::Matrix2d& base, const PrivateMatrixScalar& exponent) { + return std::pow(base.sum(), exponent.value); +} + +AnnoyingScalar operator*(int scalar, const AnnoyingScalar& value) { return AnnoyingScalar(scalar) * value; } +AnnoyingScalar pow(const AnnoyingScalar& base, const AnnoyingScalar& exponent) { + return AnnoyingScalar(std::pow(*base.v, *exponent.v)); +} + +namespace Eigen { + +template <> +struct NumTraits<BaseReferenceScalar> : NumTraits<double> { + static constexpr bool RequireInitialization = true; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<BaseReferenceScalar, double, BinaryOp> { + using ReturnType = BaseReferenceScalar; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<double, BaseReferenceScalar, BinaryOp> { + using ReturnType = BaseReferenceScalar; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<BaseReferenceScalar, ScalarMatrix, BinaryOp> { + using ReturnType = BaseReferenceScalar; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<BaseReferenceScalar, ScalarArray, BinaryOp> { + using ReturnType = BaseReferenceScalar; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<BaseReferenceMatrix, double, BinaryOp> { + using ReturnType = BaseReferenceMatrix; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<MatrixBase<BaseReferenceMatrix>, double, BinaryOp> { + using ReturnType = BaseReferenceMatrix; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<BaseReferenceArray, double, BinaryOp> { + using ReturnType = BaseReferenceArray; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<ArrayBase<BaseReferenceArray>, double, BinaryOp> { + using ReturnType = BaseReferenceArray; +}; + +template <> +struct NumTraits<DiagonalOperandScalar> : NumTraits<double> {}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<DiagonalOperandScalar, double, BinaryOp> { + using ReturnType = DiagonalOperandScalar; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<double, DiagonalOperandScalar, BinaryOp> { + using ReturnType = DiagonalOperandScalar; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<DiagonalOperandScalar, ScalarDiagonal, BinaryOp> { + using ReturnType = DiagonalOperandScalar; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<ScalarDiagonal, DiagonalOperandScalar, BinaryOp> { + using ReturnType = DiagonalOperandScalar; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<DiagonalOperandMatrix, double, BinaryOp> { + using ReturnType = DiagonalOperandMatrix; +}; +template <typename BinaryOp> +struct ScalarBinaryOpTraits<double, DiagonalOperandMatrix, BinaryOp> { + using ReturnType = DiagonalOperandMatrix; +}; + +template <> +struct ScalarBinaryOpTraits<PrivateMatrixScalar, ScalarMatrix, + internal::scalar_product_op<PrivateMatrixScalar, ScalarMatrix>> { + using ReturnType = double; +}; +template <> +struct ScalarBinaryOpTraits<ScalarMatrix, PrivateMatrixScalar, + internal::scalar_product_op<ScalarMatrix, PrivateMatrixScalar>> { + using ReturnType = double; +}; +template <> +struct ScalarBinaryOpTraits<PrivateMatrixScalar, ScalarArray, + internal::scalar_product_op<PrivateMatrixScalar, ScalarArray>> { + using ReturnType = double; +}; +template <> +struct ScalarBinaryOpTraits<ScalarArray, PrivateMatrixScalar, + internal::scalar_product_op<ScalarArray, PrivateMatrixScalar>> { + using ReturnType = double; +}; +template <> +struct ScalarBinaryOpTraits<PrivateMatrixScalar, ScalarArray, + internal::scalar_sum_op<PrivateMatrixScalar, ScalarArray>> { + using ReturnType = double; +}; +template <> +struct ScalarBinaryOpTraits<ScalarArray, PrivateMatrixScalar, + internal::scalar_sum_op<ScalarArray, PrivateMatrixScalar>> { + using ReturnType = double; +}; +template <> +struct ScalarBinaryOpTraits<ScalarArray, PrivateMatrixScalar, + internal::scalar_pow_op<ScalarArray, PrivateMatrixScalar>> { + using ReturnType = double; +}; + +template <typename Kind> +struct ScalarBinaryOpTraits<double, EigenBaseScalar<Kind>, internal::scalar_product_op<double, EigenBaseScalar<Kind>>> { + using ReturnType = double; +}; +template <typename Kind> +struct ScalarBinaryOpTraits<EigenBaseScalar<Kind>, double, internal::scalar_product_op<EigenBaseScalar<Kind>, double>> { + using ReturnType = double; +}; + +template <> +struct ScalarBinaryOpTraits<PrivateMatrixScalar, BoundedRows, + internal::scalar_product_op<PrivateMatrixScalar, BoundedRows>> { + using ReturnType = Matrix2d; +}; + +template <> +struct ScalarBinaryOpTraits<BoundedCols, PrivateMatrixScalar, + internal::scalar_product_op<BoundedCols, PrivateMatrixScalar>> { + using ReturnType = Matrix2d; +}; + +template <> +struct ScalarBinaryOpTraits<PrivateMatrixScalar, BoundedArray, + internal::scalar_product_op<PrivateMatrixScalar, BoundedArray>> { + using ReturnType = Array22d; +}; + +template <> +struct ScalarBinaryOpTraits<BoundedArray, PrivateMatrixScalar, + internal::scalar_product_op<BoundedArray, PrivateMatrixScalar>> { + using ReturnType = Array22d; +}; + +template <typename BinaryOp> +struct ScalarBinaryOpTraits<double, PrivateMatrixScalar, BinaryOp> { + using ReturnType = double; +}; + +template <typename BinaryOp> +struct ScalarBinaryOpTraits<PrivateMatrixScalar, double, BinaryOp> { + using ReturnType = double; +}; + +template <> +struct ScalarBinaryOpTraits<PrivateMatrixScalar, Matrix2d, + internal::scalar_quotient_op<PrivateMatrixScalar, Matrix2d>> { + using ReturnType = double; +}; + +template <> +struct ScalarBinaryOpTraits<PrivateMatrixScalar, Matrix2d, internal::scalar_product_op<PrivateMatrixScalar, Matrix2d>> { + using ReturnType = Matrix2d; +}; + +template <> +struct ScalarBinaryOpTraits<Matrix2d, PrivateMatrixScalar, internal::scalar_pow_op<Matrix2d, PrivateMatrixScalar>> { + using ReturnType = double; +}; + +template <> +struct ScalarBinaryOpTraits<Quaterniond, Matrix3d, internal::scalar_product_op<Quaterniond, Matrix3d>> { + using ReturnType = Matrix3d; +}; + +// Broad traits must not make both generic scalar overloads compete with matrix/array operations. +template <typename T, typename BinaryOp> +struct ScalarBinaryOpTraits<AnnoyingScalar, T, BinaryOp> { + using ReturnType = AnnoyingScalar; +}; + +template <typename T, typename BinaryOp> +struct ScalarBinaryOpTraits<T, AnnoyingScalar, BinaryOp> { + using ReturnType = AnnoyingScalar; +}; + +template <typename BinaryOp> +struct ScalarBinaryOpTraits<AnnoyingScalar, AnnoyingScalar, BinaryOp> { + using ReturnType = AnnoyingScalar; +}; + +} // namespace Eigen + +template <typename MatrixType> +void scalar_binary_op_overloads() { + using Scalar = typename MatrixType::Scalar; + MatrixType a(3, 3), b(3, 3); + Matrix3f af, bf; + for (Index j = 0; j < 3; ++j) { + for (Index i = 0; i < 3; ++i) { + af(i, j) = float(i + 2 * j + 1); + bf(i, j) = float(2 * i - j + 1); + a(i, j) = Scalar(af(i, j)); + b(i, j) = Scalar(bf(i, j)); + } + } + + Matrix<Scalar, Dynamic, Dynamic, MatrixType::Options> result = a * b; + VERIFY_IS_APPROX(result.template cast<float>(), af * bf); + result = a.transpose() * b.template leftCols<2>(); + VERIFY_IS_APPROX(result.template cast<float>(), af.transpose() * bf.template leftCols<2>()); + result = a * b.col(0); + VERIFY_IS_APPROX(result.template cast<float>(), af * bf.col(0)); + const SparseMatrix<Scalar, MatrixType::Options> sparse_a = a.sparseView(); + result = sparse_a * b; + VERIFY_IS_APPROX(result.template cast<float>(), af * bf); + result = a * sparse_a; + VERIFY_IS_APPROX(result.template cast<float>(), af * af); + result = sparse_a * sparse_a; + VERIFY_IS_APPROX(result.template cast<float>(), af * af); + + result = a * Scalar(2); + VERIFY_IS_APPROX(result.template cast<float>(), af * 2); + result = Scalar(2) * a; + VERIFY_IS_APPROX(result.template cast<float>(), 2 * af); + result = a * 2; + VERIFY_IS_APPROX(result.template cast<float>(), af * 2); + result = 2 * a; + VERIFY_IS_APPROX(result.template cast<float>(), 2 * af); + + const auto aa = a.array(); + const auto ba = b.array(); + VERIFY_IS_APPROX((aa * ba).template cast<float>(), af.array() * bf.array()); + VERIFY_IS_APPROX((aa + ba).template cast<float>(), af.array() + bf.array()); + VERIFY_IS_APPROX((aa - ba).template cast<float>(), af.array() - bf.array()); + VERIFY_IS_APPROX((aa / (ba + Scalar(10))).template cast<float>(), af.array() / (bf.array() + 10)); + VERIFY_IS_APPROX((aa + Scalar(2)).template cast<float>(), af.array() + 2); + VERIFY_IS_APPROX((Scalar(2) + aa).template cast<float>(), 2 + af.array()); + VERIFY_IS_APPROX(Eigen::pow(aa, ba).template cast<float>(), af.array().pow(bf.array())); + VERIFY_IS_APPROX(Eigen::pow(af.array(), ba).template cast<float>(), af.array().pow(bf.array())); + VERIFY_IS_APPROX(Eigen::pow(aa, bf.array()).template cast<float>(), af.array().pow(bf.array())); + VERIFY_IS_APPROX(Eigen::pow(Scalar(2), ba).template cast<float>(), Eigen::pow(2.0f, bf.array())); + VERIFY_IS_APPROX(Eigen::pow(af.array(), 2).eval(), af.array().square()); + + struct DerivedMatrix : MatrixType { + explicit DerivedMatrix(const MatrixType& matrix) : MatrixType(matrix) {} + }; + DerivedMatrix derived_a(a), derived_b(b); + result = derived_a * derived_b; + VERIFY_IS_APPROX(result.template cast<float>(), af * bf); +} + +template <int Rows, int Cols> +void bounded_array_scalar_binary_op_overloads() { + Array<PrivateMatrixScalar, Rows, Cols> coefficients; + for (Index i = 0; i < coefficients.size(); ++i) coefficients(i) = PrivateMatrixScalar(i + 1); + const BoundedArray operand = Array22d::Random(); + const Array<Array22d, Rows, Cols> right = coefficients * operand; + const Array<Array22d, Rows, Cols> left = operand * coefficients; + for (Index i = 0; i < coefficients.size(); ++i) { + VERIFY_IS_APPROX(right(i), (i + 1) * operand); + VERIFY_IS_APPROX(left(i), (i + 1) * operand); + } +} + +EIGEN_DECLARE_TEST(scalar_binary_op_overloads) { + quantity_scalar_product<2, ColMajor, 2>(); + quantity_scalar_product<2, RowMajor, 2>(); + BaseReferenceMatrix base_matrix; + base_matrix(0) = BaseReferenceScalar(2); + const MatrixBase<BaseReferenceMatrix>& matrix_base = base_matrix; + const ScalarMatrix base_matrix_operand = ScalarMatrix::Constant(3); + const BaseReferenceMatrix base_matrix_result = matrix_base * base_matrix_operand; + VERIFY_IS_EQUAL(base_matrix_result(0).value, 5); + BaseReferenceArray base_array; + base_array(0) = BaseReferenceScalar(2); + const ArrayBase<BaseReferenceArray>& array_base = base_array; + const ScalarArray base_array_operand = ScalarArray::Constant(3); + const BaseReferenceArray base_array_result = array_base * base_array_operand; + VERIFY_IS_EQUAL(base_array_result(0).value, 5); + DiagonalOperandMatrix diagonal_coefficients; + diagonal_coefficients(0) = DiagonalOperandScalar(2); + ScalarDiagonal diagonal_operand; + diagonal_operand.diagonal()(0) = 3; + const DiagonalOperandMatrix diagonal_right = diagonal_coefficients * diagonal_operand; + const DiagonalOperandMatrix diagonal_left = diagonal_operand * diagonal_coefficients; + VERIFY_IS_EQUAL(diagonal_right(0).value, 5); + VERIFY_IS_EQUAL(diagonal_left(0).value, 5); + Matrix<PrivateMatrixScalar, 1, 1> registered_matrix; + registered_matrix(0) = PrivateMatrixScalar(2); + const ScalarMatrix matrix_operand = ScalarMatrix::Constant(3); + const ScalarMatrix matrix_right = registered_matrix * matrix_operand; + const ScalarMatrix matrix_left = matrix_operand * registered_matrix; + VERIFY_IS_EQUAL(matrix_right(0), 5); + VERIFY_IS_EQUAL(matrix_left(0), 5); + const Array<PrivateMatrixScalar, 1, 1> registered_array = registered_matrix.array(); + const ScalarArray array_operand = ScalarArray::Constant(3); + const ScalarArray array_right = registered_array * array_operand; + const ScalarArray array_left = array_operand * registered_array; + const ScalarArray sum_right = registered_array + array_operand; + const ScalarArray sum_left = array_operand + registered_array; + const ScalarArray power = Eigen::pow(array_operand, registered_array); + VERIFY_IS_EQUAL(array_right(0), 5); + VERIFY_IS_EQUAL(array_left(0), 5); + VERIFY_IS_EQUAL(sum_right(0), 7); + VERIFY_IS_EQUAL(sum_left(0), 7); + VERIFY_IS_EQUAL(power(0), 5); + const EigenBaseScalar<MatrixXpr> matrix_scale; + const Matrix2d matrix = Matrix2d::Ones(); + VERIFY_IS_EQUAL((matrix * matrix_scale).eval(), 3 * matrix); + VERIFY_IS_EQUAL((matrix_scale * matrix).eval(), 3 * matrix); + const EigenBaseScalar<ArrayXpr> array_scale; + const Array22d array = Array22d::Ones(); + VERIFY(((array * array_scale) == 3 * array).all()); + VERIFY(((array_scale * array) == 3 * array).all()); + + using SparseMatrixType = SparseMatrix<double>; + using SparseView = SparseSelfAdjointView<SparseMatrixType, Lower>; + using DenseView = SelfAdjointView<Matrix2d, Lower>; + static_assert(internal::has_generic_scalar_binary_ops<SparseMatrixType, SparseMatrixType, MatrixXpr>::value, + "sparse generic scalar operators"); + static_assert(internal::has_generic_scalar_binary_ops<Matrix2d, Matrix2d, MatrixXpr>::value, + "dense generic scalar operators"); + static_assert(internal::has_generic_scalar_binary_ops<Array22d, Array22d, ArrayXpr>::value, + "array generic scalar operators"); + static_assert(!internal::has_generic_scalar_binary_ops<SparseView, SparseView, MatrixXpr>::value, + "sparse self-adjoint scaling uses fixed scalar overloads"); + static_assert(!internal::has_generic_scalar_binary_ops<DenseView, DenseView, MatrixXpr>::value, + "dense self-adjoint scaling uses fixed scalar overloads"); + static_assert(!internal::has_generic_scalar_binary_ops<ScalarDiagonal, ScalarDiagonal, MatrixXpr>::value, + "diagonal scaling uses fixed scalar overloads"); + + struct PrivateMatrix : private Matrix3f {}; + struct UnrelatedDerived { + const UnrelatedDerived& derived() const; + }; + static_assert(internal::is_eigen_expression<const Matrix3f&>::value, "cv-qualified expressions"); + static_assert(internal::is_eigen_expression<MatrixBase<Matrix3f>>::value, "expression base references"); + static_assert(!internal::is_eigen_expression<AnnoyingScalar>::value, "custom scalars"); + static_assert(!internal::is_eigen_expression<PrivateMatrix>::value, "private implementation bases"); + static_assert(!internal::is_eigen_expression<PrivateMatrixScalar>::value, "public derived with private bases"); + static_assert(!internal::is_eigen_expression<UnrelatedDerived>::value, "unrelated derived methods"); + static_assert(internal::shape_max_dimensions<UnrelatedDerived>::Rows == Dynamic, "missing maximum metadata"); + static_assert(internal::dimension_relation<Dynamic, Dynamic, 2, 3>::Compatible, + "unequal maxima do not imply unequal dimensions"); + static_assert(internal::dimension_relation<2, Dynamic, 2, 2>::Compatible, "matching at the capacity limit"); + static_assert(!internal::dimension_relation<3, Dynamic, 3, 2>::Compatible, "fixed above dynamic maximum"); + static_assert(!internal::dimension_relation<Dynamic, 3, 2, 3>::Compatible, "dynamic maximum below fixed"); + static_assert(!internal::has_competing_expression_binary_op<Matrix<PrivateMatrixScalar, 1, 2>, BoundedRows, + internal::scalar_product_op, false>::value, + "registered scalar products remain valid even when dimensions also match"); + scalar_binary_op_overloads<Matrix<AnnoyingScalar, 3, 3, ColMajor>>(); + scalar_binary_op_overloads<Matrix<AnnoyingScalar, 3, 3, RowMajor>>(); + scalar_binary_op_overloads<Matrix<AnnoyingScalar, Dynamic, Dynamic, ColMajor>>(); + scalar_binary_op_overloads<Matrix<AnnoyingScalar, Dynamic, Dynamic, RowMajor>>(); + VERIFY_IS_EQUAL(AnnoyingScalar::instances, 0); + + Matrix<PrivateMatrixScalar, 1, 3> bounded_coefficients; + for (Index i = 0; i < 3; ++i) bounded_coefficients(i) = PrivateMatrixScalar(i + 1); + const BoundedRows bounded_rows = Matrix2d::Random(); + const BoundedCols bounded_cols = Matrix2d::Random(); + const Matrix<Matrix2d, 1, 3> bounded_right = bounded_coefficients * bounded_rows; + const Matrix<PrivateMatrixScalar, 3, 1> bounded_column = bounded_coefficients.transpose(); + const Matrix<Matrix2d, 3, 1> bounded_left = bounded_cols * bounded_column; + for (Index i = 0; i < 3; ++i) { + VERIFY_IS_APPROX(bounded_right(i), (i + 1) * bounded_rows); + VERIFY_IS_APPROX(bounded_left(i), (i + 1) * bounded_cols); + } + bounded_array_scalar_binary_op_overloads<3, 1>(); + bounded_array_scalar_binary_op_overloads<1, 3>(); + + const PrivateMatrixScalar private_scale(3); + const Matrix2d private_right = Matrix2d::Ones() * private_scale; + const Matrix2d private_left = private_scale * Matrix2d::Ones(); + VERIFY_IS_APPROX(private_right, 3 * Matrix2d::Ones()); + VERIFY_IS_APPROX(private_left, 3 * Matrix2d::Ones()); + + Array<Quaterniond, 2, 1> rotations; + rotations(0) = Quaterniond::Identity(); + rotations(1) = Quaterniond(AngleAxisd(0.5, Vector3d::UnitZ())); + const Matrix3d operand = Matrix3d::Random(); + const Array<Matrix3d, 2, 1> rotated = rotations * operand; + for (Index i = 0; i < rotations.size(); ++i) { + VERIFY_IS_APPROX(rotated(i), rotations(i).toRotationMatrix() * operand); + } + const Matrix<Quaterniond, 2, 1> matrix_rotations = rotations.matrix(); + const Matrix<Matrix3d, 2, 1> matrix_rotated = matrix_rotations * operand; + for (Index i = 0; i < rotations.size(); ++i) { + VERIFY_IS_APPROX(matrix_rotated(i), rotations(i).toRotationMatrix() * operand); + } + + Matrix<PrivateMatrixScalar, 2, 1> dividends; + dividends(0) = PrivateMatrixScalar(4); + dividends(1) = PrivateMatrixScalar(8); + const Matrix2d divisor = Matrix2d::Ones(); + const Vector2d quotients = dividends / divisor; + VERIFY_IS_EQUAL(quotients(0), 1); + VERIFY_IS_EQUAL(quotients(1), 2); + const Matrix<Matrix2d, 2, 1> multiplied = dividends * divisor; + VERIFY_IS_APPROX(multiplied(0), 4 * divisor); + VERIFY_IS_APPROX(multiplied(1), 8 * divisor); + const Array<PrivateMatrixScalar, 2, 1> exponents = dividends.array(); + const Array2d powers = Eigen::pow(divisor, exponents); + VERIFY_IS_EQUAL(powers(0), 256); + VERIFY_IS_EQUAL(powers(1), 65536); + + Array<Matrix2f, 2, 1> matrices; + matrices(0) = Matrix2f::Identity(); + matrices(1) = Matrix2f::Ones(); + const Matrix2f scale = 2 * Matrix2f::Identity(); + const Array<Matrix2f, 2, 1> scaled = matrices * scale; + VERIFY_IS_APPROX(scaled(0), scale); + VERIFY_IS_APPROX(scaled(1), 2 * Matrix2f::Ones()); + const Array<Matrix2f, 2, 1> left_scaled = scale * matrices; + VERIFY_IS_APPROX(left_scaled(0), scale); + VERIFY_IS_APPROX(left_scaled(1), 2 * Matrix2f::Ones()); +}
diff --git a/test/schur_real.cpp b/test/schur_real.cpp index 6d0390e..b38c837 100644 --- a/test/schur_real.cpp +++ b/test/schur_real.cpp
@@ -30,9 +30,10 @@ for (int row = 1; row < size; ++row) { if (!numext::is_exactly_zero(T(row, row - 1))) { VERIFY(row == size - 1 || numext::is_exactly_zero(T(row + 1, row))); - Scalar tr = T(row - 1, row - 1) + T(row, row); - Scalar det = T(row - 1, row - 1) * T(row, row) - T(row - 1, row) * T(row, row - 1); - VERIFY(4 * det > tr * tr); + // The test RealSchur::splitOffTwoRows keeps a block on: q = p^2 + bc < 0 with p = (a-d)/2. + // tr^2 - 4*det would cancel the common diagonal shift. + const Scalar half_difference = (T(row - 1, row - 1) - T(row, row)) / Scalar(2); + VERIFY(half_difference * half_difference + T(row - 1, row) * T(row, row - 1) < Scalar(0)); } } } @@ -299,6 +300,14 @@ } EIGEN_DECLARE_TEST(schur_real) { + // RealSchur keeps these blocks (|c| > eps*(|a|+|d|)), but tr^2 and 4*det round to the same value. + Matrix2f small_imaginary_float; + small_imaginary_float << 1, 0.1f, -3e-7f, 1; + CALL_SUBTEST_1(verifyIsQuasiTriangular(RealSchur<Matrix2f>(small_imaginary_float).matrixT())); + Matrix2d small_imaginary_double; + small_imaginary_double << 1, 0.1, -5e-16, 1; + CALL_SUBTEST_2(verifyIsQuasiTriangular(RealSchur<Matrix2d>(small_imaginary_double).matrixT())); + CALL_SUBTEST_1((schur<Matrix4f>())); CALL_SUBTEST_2((schur<MatrixXd>(internal::random<int>(1, EIGEN_TEST_MAX_SIZE / 4)))); CALL_SUBTEST_3((schur<Matrix<float, 1, 1> >()));
diff --git a/test/selfadjoint.cpp b/test/selfadjoint.cpp index 359a8a0..109a719 100644 --- a/test/selfadjoint.cpp +++ b/test/selfadjoint.cpp
@@ -175,6 +175,28 @@ EIGEN_UNUSED_VARIABLE(m); } +template <unsigned int UpLo, typename MatrixType> +EIGEN_DONT_INLINE void verify_selfadjoint_copy(const MatrixType& m, const MatrixType& dense) { + for (Index j = 0; j < m.cols(); ++j) + for (Index i = 0; i < m.rows(); ++i) + VERIFY_IS_EQUAL(dense(i, j), (UpLo == Upper ? i <= j : i >= j) ? m(i, j) : numext::conj(m(j, i))); +} + +// GCC PR target/124138 miscompiled this copy for 4x4 double with AVX-512 (see SelfAdjointView.h). It needs the copy +// materialized in a function of its own, and a reference built without the mirrored stores that GCC also miscompiles. +template <unsigned int UpLo, typename MatrixType> +EIGEN_DONT_INLINE void selfadjoint_copy(const MatrixType& m) { + const MatrixType dense = m.template selfadjointView<UpLo>(); + verify_selfadjoint_copy<UpLo>(m, dense); +} + +template <typename MatrixType> +void selfadjoint_copy_fixed() { + const MatrixType m = MatrixType::Random(); + selfadjoint_copy<Upper>(m); + selfadjoint_copy<Lower>(m); +} + EIGEN_DECLARE_TEST(selfadjoint) { for (int i = 0; i < g_repeat; i++) { int s = internal::random<int>(1, EIGEN_TEST_MAX_SIZE); @@ -190,6 +212,9 @@ } CALL_SUBTEST_1(bug_159()); + CALL_SUBTEST_2(selfadjoint_copy_fixed<Matrix4d>()); + CALL_SUBTEST_2((selfadjoint_copy_fixed<Matrix<double, 4, 4, RowMajor> >())); + CALL_SUBTEST_3(selfadjoint_copy_fixed<Matrix4cd>()); CALL_SUBTEST_1(selfadjoint_nested_expression<>()); CALL_SUBTEST_4(selfadjoint_l1norm_sizes<double>()); CALL_SUBTEST_4(selfadjoint_l1norm_sizes<std::complex<double> >());
diff --git a/test/selfadjoint_permutation.cpp b/test/selfadjoint_permutation.cpp new file mode 100644 index 0000000..bfaf885 --- /dev/null +++ b/test/selfadjoint_permutation.cpp
@@ -0,0 +1,248 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#define TEST_ENABLE_TEMPORARY_TRACKING +#define EIGEN_RUNTIME_NO_MALLOC +#include "main.h" +#include <Eigen/Core> +#include <algorithm> +#include <cstdint> + +template <unsigned int UpLo, typename MatrixType, typename Transformation> +void check_selfadjoint_permutation(MatrixType& matrix, const Transformation& transformation) { + using IndicesType = typename Transformation::IndicesType; + using Permutation = PermutationMatrix<IndicesType::SizeAtCompileTime, IndicesType::MaxSizeAtCompileTime, + typename IndicesType::Scalar>; + // NOLINTNEXTLINE(performance-unnecessary-copy-initialization): Transpositions inputs need this conversion. + const Permutation permutation(transformation); + using PlainObject = typename MatrixType::PlainObject; + const PlainObject original = matrix; + const Index size = matrix.rows(); + PlainObject full(size, size); + for (Index j = 0; j < size; ++j) { + for (Index i = 0; i < size; ++i) { + full(Index(permutation.indices()[i]), Index(permutation.indices()[j])) = + (UpLo == Lower ? i >= j : i <= j) ? original(i, j) : numext::conj(original(j, i)); + } + } + PlainObject expected = original; + expected.template triangularView<UpLo>() = full; + auto view = matrix.template selfadjointView<UpLo>(); + VERIFY(&view.applyPermutation(transformation) == &view); + VERIFY_IS_EQUAL(matrix, expected); + VERIFY_IS_EQUAL(view.toDenseMatrix(), full); + view.applyPermutation(transformation.inverse()); + VERIFY_IS_EQUAL(matrix, original); + view.applyPermutation(transformation.transpose()).applyPermutation(transformation); + VERIFY_IS_EQUAL(matrix, original); +} + +template <unsigned int UpLo, typename MatrixType> +void check_selfadjoint_transposition(MatrixType& matrix, Index i, Index j) { + using PlainObject = typename MatrixType::PlainObject; + const PlainObject original = matrix; + PlainObject full = original.template selfadjointView<UpLo>(); + full.row(i).swap(full.row(j)); + full.col(i).swap(full.col(j)); + PlainObject expected = original; + expected.template triangularView<UpLo>() = full; + auto view = matrix.template selfadjointView<UpLo>(); + internal::set_is_malloc_allowed(false); + VERIFY(&view.applyTransposition(i, j) == &view); + internal::set_is_malloc_allowed(true); + VERIFY_IS_EQUAL(matrix, expected); + view.applyTransposition(j, i); + VERIFY_IS_EQUAL(matrix, original); +} + +template <typename Scalar, int Order> +void selfadjoint_permutation_cycles() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + for (Index size = 0; size <= 5; ++size) { + MatrixType matrix = MatrixType::Random(size, size); + matrix.diagonal() = matrix.diagonal().real().template cast<Scalar>(); + PermutationMatrix<Dynamic> permutation(size); + permutation.setIdentity(); + do { + check_selfadjoint_permutation<Lower>(matrix, permutation); + check_selfadjoint_permutation<Upper>(matrix, permutation); + } while (size > 1 && std::next_permutation(permutation.indices().data(), permutation.indices().data() + size)); + for (Index i = 0; i < size; ++i) { + for (Index j = 0; j < size; ++j) { + check_selfadjoint_transposition<Lower>(matrix, i, j); + check_selfadjoint_transposition<Upper>(matrix, i, j); + } + } + } +} + +template <typename Scalar, int Order> +void selfadjoint_permutation_views() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + const Index size = internal::random<Index>(2, 30); + MatrixType storage = MatrixType::Random(2 * size + 3, 2 * size + 3); + const MatrixType original = storage; + auto block = storage.block(1, 2, size, size); + PermutationMatrix<Dynamic, Dynamic, std::int64_t> permutation(size); + randomPermutationVector(permutation.indices(), size); + const Map<PermutationMatrix<Dynamic, Dynamic, std::int64_t>> mapped_permutation(permutation.indices().data(), size); + check_selfadjoint_permutation<Lower>(block, mapped_permutation); + check_selfadjoint_permutation<Upper>(block, permutation.indices().asPermutation()); + VERIFY_IS_EQUAL(storage, original); + + Map<MatrixType, Unaligned, Stride<Dynamic, Dynamic>> strided(storage.data(), size, size, + Stride<Dynamic, Dynamic>(2 * storage.outerStride(), 2)); + check_selfadjoint_permutation<Lower>(strided, permutation); + check_selfadjoint_permutation<Upper>(strided, permutation); + for (Index i = 0; i < size; ++i) { + check_selfadjoint_transposition<Lower>(block, i, size - 1 - i); + check_selfadjoint_transposition<Upper>(strided, i, size - 1 - i); + } + VERIFY_IS_EQUAL(storage, original); +} + +template <typename Scalar, int Size> +void selfadjoint_permutation_fixed() { + Matrix<Scalar, Size, Size> matrix = Matrix<Scalar, Size, Size>::Random(); + PermutationMatrix<Size> permutation; + randomPermutationVector(permutation.indices(), Size); + internal::set_is_malloc_allowed(false); + check_selfadjoint_permutation<Lower>(matrix, permutation); + check_selfadjoint_permutation<Upper>(matrix, permutation); + Transpositions<Size> transpositions; + for (Index i = 0; i < Size; ++i) transpositions.indices()[i] = int(Size - 1 - i); + check_selfadjoint_permutation<Lower>(matrix, transpositions); + check_selfadjoint_permutation<Upper>(matrix, transpositions); + internal::set_is_malloc_allowed(true); +} + +void selfadjoint_permutation_preconditions() { + MatrixXd matrix = MatrixXd::Random(3, 3); + PermutationMatrix<Dynamic> permutation(2); + permutation.setIdentity(); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Lower>().applyPermutation(permutation)); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Upper>().applyPermutation(permutation.inverse())); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Lower>().applyTransposition(-1, 0)); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Upper>().applyTransposition(0, 3)); + Transpositions<Dynamic> transpositions(2); + transpositions.setIdentity(); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Lower>().applyPermutation(transpositions)); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Upper>().applyPermutation(transpositions.inverse())); + // Indices that do not form a permutation: a repeat would otherwise loop forever. + PermutationMatrix<Dynamic> invalid(3); + invalid.indices() << 1, 1, 2; + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Lower>().applyPermutation(invalid)); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Upper>().applyPermutation(invalid)); + invalid.indices() << 2, 0, 0; + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Lower>().applyPermutation(invalid)); + invalid.indices() << 3, 0, 1; + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Upper>().applyPermutation(invalid)); + matrix.resize(2, 3); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Lower>().applyPermutation(permutation)); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Upper>().applyPermutation(permutation)); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Lower>().applyTransposition(0, 1)); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Upper>().applyPermutation(transpositions)); + VERIFY_RAISES_ASSERT(matrix.selfadjointView<Lower>().applyPermutation(transpositions.transpose())); +} + +void selfadjoint_permutation_aliased_indices() { + Matrix3i matrix; + matrix << 1, 2, 0, 4, 5, 6, 7, 8, 9; + const PermutationMatrix<3> permutation(matrix.row(0).transpose()); + Matrix3i expected = matrix; + expected.selfadjointView<Upper>().applyPermutation(permutation); + matrix.selfadjointView<Upper>().applyPermutation(matrix.row(0).transpose().asPermutation()); + VERIFY_IS_EQUAL(matrix, expected); + matrix << 1, 2, 0, 4, 5, 6, 7, 8, 9; + expected = matrix; + expected.selfadjointView<Upper>().applyPermutation(permutation.inverse()); + matrix.selfadjointView<Upper>().applyPermutation(matrix.row(0).transpose().asPermutation().inverse()); + VERIFY_IS_EQUAL(matrix, expected); + + for (bool inverse : {false, true}) { + matrix << 1, 2, 0, 4, 5, 6, 7, 8, 9; + auto indices = matrix.row(0).transpose(); + TranspositionsWrapper<decltype(indices)> wrapped(indices); + const Transpositions<3> transpositions(wrapped); + expected = matrix; + if (inverse) { + expected.selfadjointView<Upper>().applyPermutation(transpositions.inverse()); + matrix.selfadjointView<Upper>().applyPermutation(wrapped.inverse()); + } else { + expected.selfadjointView<Upper>().applyPermutation(transpositions); + matrix.selfadjointView<Upper>().applyPermutation(wrapped); + } + VERIFY_IS_EQUAL(matrix, expected); + } +} + +template <typename Scalar, int Order> +void selfadjoint_transpositions() { + using MatrixType = Matrix<Scalar, Dynamic, Dynamic, Order>; + for (Index size = 0; size <= 4; ++size) { + MatrixType matrix = MatrixType::Random(size, size); + matrix.diagonal() = matrix.diagonal().real().template cast<Scalar>(); + Transpositions<Dynamic, Dynamic, std::int64_t> transpositions(size); + transpositions.indices().setZero(); + // All n^n sequences, including swaps with earlier indices and repeated targets. + bool finished = false; + do { + check_selfadjoint_permutation<Lower>(matrix, transpositions); + check_selfadjoint_permutation<Upper>(matrix, transpositions); + Index i = 0; + while (i < size && ++transpositions.indices()[i] == size) transpositions.indices()[i++] = 0; + finished = i == size; + } while (!finished); + transpositions.setIdentity(); + const Map<Transpositions<Dynamic, Dynamic, std::int64_t>> mapped(transpositions.indices().data(), size); + check_selfadjoint_permutation<Lower>(matrix, mapped); + check_selfadjoint_permutation<Upper>(matrix, mapped); + } +} + +void selfadjoint_permutation_temporary_count() { + MatrixXd matrix = MatrixXd::Random(8, 8); + PermutationMatrix<Dynamic> permutation(8); + randomPermutationVector(permutation.indices(), 8); + VERIFY_EVALUATION_COUNT(matrix.selfadjointView<Lower>().applyPermutation(permutation), 1); + VERIFY_EVALUATION_COUNT(matrix.selfadjointView<Upper>().applyPermutation(permutation.inverse()), 1); + VERIFY_EVALUATION_COUNT(matrix.selfadjointView<Lower>().applyPermutation(permutation.transpose()), 1); + Transpositions<Dynamic> transpositions(8); + transpositions.setIdentity(); + VERIFY_EVALUATION_COUNT(matrix.selfadjointView<Lower>().applyPermutation(transpositions), 1); + VERIFY_EVALUATION_COUNT(matrix.selfadjointView<Upper>().applyPermutation(transpositions.transpose()), 1); + VERIFY_EVALUATION_COUNT(matrix.selfadjointView<Lower>().applyTransposition(1, 7), 0); +} + +EIGEN_DECLARE_TEST(selfadjoint_permutation) { + CALL_SUBTEST_1((selfadjoint_permutation_cycles<float, ColMajor>())); + CALL_SUBTEST_1((selfadjoint_permutation_cycles<float, RowMajor>())); + CALL_SUBTEST_2((selfadjoint_permutation_cycles<double, ColMajor>())); + CALL_SUBTEST_2((selfadjoint_permutation_cycles<double, RowMajor>())); + CALL_SUBTEST_3((selfadjoint_permutation_cycles<std::complex<float>, ColMajor>())); + CALL_SUBTEST_3((selfadjoint_permutation_cycles<std::complex<float>, RowMajor>())); + CALL_SUBTEST_4((selfadjoint_permutation_cycles<std::complex<double>, ColMajor>())); + CALL_SUBTEST_4((selfadjoint_permutation_cycles<std::complex<double>, RowMajor>())); + CALL_SUBTEST_5((selfadjoint_permutation_cycles<int, ColMajor>())); + CALL_SUBTEST_5((selfadjoint_permutation_cycles<short, RowMajor>())); + CALL_SUBTEST_5((selfadjoint_permutation_cycles<half, ColMajor>())); + CALL_SUBTEST_5((selfadjoint_permutation_cycles<bfloat16, RowMajor>())); + for (int i = 0; i < g_repeat; ++i) { + CALL_SUBTEST_1((selfadjoint_permutation_views<float, ColMajor>())); + CALL_SUBTEST_2((selfadjoint_permutation_views<double, RowMajor>())); + CALL_SUBTEST_3((selfadjoint_permutation_views<std::complex<float>, RowMajor>())); + CALL_SUBTEST_4((selfadjoint_permutation_views<std::complex<double>, ColMajor>())); + CALL_SUBTEST_1((selfadjoint_permutation_fixed<float, 0>())); + CALL_SUBTEST_1((selfadjoint_permutation_fixed<float, 1>())); + CALL_SUBTEST_2((selfadjoint_permutation_fixed<double, 4>())); + CALL_SUBTEST_3((selfadjoint_permutation_fixed<std::complex<float>, 5>())); + } + CALL_SUBTEST_6((selfadjoint_transpositions<double, ColMajor>())); + CALL_SUBTEST_6((selfadjoint_transpositions<double, RowMajor>())); + CALL_SUBTEST_6((selfadjoint_transpositions<std::complex<double>, ColMajor>())); + CALL_SUBTEST_6((selfadjoint_transpositions<std::complex<double>, RowMajor>())); + CALL_SUBTEST_6((selfadjoint_transpositions<int, RowMajor>())); + CALL_SUBTEST_6(selfadjoint_permutation_temporary_count()); + CALL_SUBTEST_1(selfadjoint_permutation_preconditions()); + CALL_SUBTEST_5(selfadjoint_permutation_aliased_indices()); +}
diff --git a/test/shape_checks.cpp b/test/shape_checks.cpp index c1bad9b..7a7cd71 100644 --- a/test/shape_checks.cpp +++ b/test/shape_checks.cpp
@@ -277,6 +277,12 @@ STATIC_CHECK((!dimension_relation<3, 2>::Compatible)); STATIC_CHECK((dimension_relation<Dynamic, 3>::Compatible)); STATIC_CHECK((!dimension_relation<Dynamic, 3>::Known)); + STATIC_CHECK((!dimension_relation<3, Dynamic, 3, 2>::Compatible)); + STATIC_CHECK((!dimension_relation<Dynamic, 3, 2, 3>::Compatible)); + STATIC_CHECK((dimension_relation<Dynamic, 3, 4, 3>::Compatible)); + STATIC_CHECK((!dimension_relation<Dynamic, 3, 4, 3>::Proven)); + STATIC_CHECK((compile_time_shape<const Matrix<double, Dynamic, Dynamic, 0, 2, 3>&>::MaxRows == 2)); + STATIC_CHECK((compile_time_shape<const Matrix<double, Dynamic, Dynamic, 0, 2, 3>&>::MaxCols == 3)); STATIC_CHECK((compile_time_shape<MissingShape>::Rows == Dynamic)); STATIC_CHECK((compile_time_shape<PartialShape>::Rows == 3)); STATIC_CHECK((compile_time_shape<PartialShape>::Cols == Dynamic));
diff --git a/test/sparse_basic.cpp b/test/sparse_basic.cpp index 8462185..524562d 100644 --- a/test/sparse_basic.cpp +++ b/test/sparse_basic.cpp
@@ -21,6 +21,7 @@ #endif #include "sparse.h" +#include "sparse_test_helpers.h" template <typename SparseMatrixType> void sparse_basic(const SparseMatrixType& ref) { @@ -730,10 +731,8 @@ initSparse<Scalar>(density, refMat2, m2); initSparse<Scalar>(density, refMat3, m3); { - Map<SparseMatrixType> mapMat2(m2.rows(), m2.cols(), m2.nonZeros(), m2.outerIndexPtr(), m2.innerIndexPtr(), - m2.valuePtr(), m2.innerNonZeroPtr()); - Map<SparseMatrixType> mapMat3(m3.rows(), m3.cols(), m3.nonZeros(), m3.outerIndexPtr(), m3.innerIndexPtr(), - m3.valuePtr(), m3.innerNonZeroPtr()); + auto mapMat2 = make_map(m2); + auto mapMat3 = make_map(m3); VERIFY_IS_APPROX(mapMat2 + mapMat3, refMat2 + refMat3); VERIFY_IS_APPROX(mapMat2 + mapMat3, refMat2 + refMat3); } @@ -742,8 +741,7 @@ Index j = internal::random<Index>(0, cols - 1); m2.coeffRef(i, j) = 123; if (internal::random<bool>()) m2.makeCompressed(); - Map<SparseMatrixType> mapMat2(rows, cols, m2.nonZeros(), m2.outerIndexPtr(), m2.innerIndexPtr(), m2.valuePtr(), - m2.innerNonZeroPtr()); + auto mapMat2 = make_map(m2); VERIFY_IS_EQUAL(m2.coeff(i, j), Scalar(123)); VERIFY_IS_EQUAL(mapMat2.coeff(i, j), Scalar(123)); mapMat2.coeffRef(i, j) = -123; @@ -896,7 +894,7 @@ initSparse<Scalar>(density, refMat2, m2); m2.makeCompressed(); VectorXi res(rows); - for (Index i = 0; i < rows; ++i) res(i) = internal::random<int>(0, 3); + setRandomInRange(res, 0, 3); m2.reserve(res); m2 -= d.asDiagonal(); refMat2 -= d.asDiagonal();
diff --git a/test/sparse_block.cpp b/test/sparse_block.cpp index 567b1b5..1e0edcd 100644 --- a/test/sparse_block.cpp +++ b/test/sparse_block.cpp
@@ -9,46 +9,9 @@ // SPDX-License-Identifier: MPL-2.0 #include "sparse.h" +#include "sparse_test_helpers.h" #include "AnnoyingScalar.h" -template <typename T> -std::enable_if_t<(T::Flags & RowMajorBit) == RowMajorBit, typename T::RowXpr> innervec(T& A, Index i) { - return A.row(i); -} - -template <typename T> -std::enable_if_t<(T::Flags & RowMajorBit) == 0, typename T::ColXpr> innervec(T& A, Index i) { - return A.col(i); -} - -template <typename T> -typename T::Map make_map(T& A) { - return typename T::Map(A.rows(), A.cols(), A.nonZeros(), A.outerIndexPtr(), A.innerIndexPtr(), A.valuePtr(), - A.innerNonZeroPtr()); -} - -template <typename T> -auto innerpanel(T& A, Index start, Index size) - -> std::enable_if_t<(T::Flags & RowMajorBit) == RowMajorBit, decltype(A.middleRows(start, size))> { - return A.middleRows(start, size); -} - -template <typename T> -auto innerpanel(T& A, Index start, Index size) - -> std::enable_if_t<(T::Flags & RowMajorBit) == 0, decltype(A.middleCols(start, size))> { - return A.middleCols(start, size); -} - -template <int Options> -void init_inner_panel_diag(SparseMatrix<double, Options>& m) { - m.resize(4, 4); - m.insert(0, 0) = 10.0; - m.insert(1, 1) = 20.0; - m.insert(2, 2) = 30.0; - m.insert(3, 3) = 40.0; - m.makeCompressed(); -} - template <typename MatrixType> std::enable_if_t<(MatrixType::Flags & RowMajorBit) == 0> init_inner_panel_offset(MatrixType& m) { m.resize(4, 4); @@ -154,10 +117,10 @@ void check_inner_panel_compressed_api() { verify_plain_ref_map<SparseMatrix<double>>( - [](SparseMatrix<double>& matrix) { init_inner_panel_diag(matrix); }, + [](SparseMatrix<double>& matrix) { init_sparse_diagonal(matrix); }, [](auto& xpr, auto& matrix, int variant) { verify_inner_panel_coeffs(xpr, matrix, variant); }); verify_plain_ref_map<SparseMatrix<double, RowMajor>>( - [](SparseMatrix<double, RowMajor>& matrix) { init_inner_panel_diag(matrix); }, + [](SparseMatrix<double, RowMajor>& matrix) { init_sparse_diagonal(matrix); }, [](auto& xpr, auto& matrix, int variant) { verify_inner_panel_coeffs(xpr, matrix, variant); }); verify_plain_ref_map<SparseMatrix<double>>( [](SparseMatrix<double>& matrix) { init_inner_panel_offset(matrix); },
diff --git a/test/sparse_product.cpp b/test/sparse_product.cpp index a9c2a49..e59c9ca 100644 --- a/test/sparse_product.cpp +++ b/test/sparse_product.cpp
@@ -22,6 +22,7 @@ #define EIGEN_TEST_ANNOYING_SCALAR_DONT_THROW #include "AnnoyingScalar.h" +#include "NonCommutativeScalar.h" #define VERIFY_EVALUATION_COUNT(XPR, N) \ { \ @@ -970,6 +971,361 @@ VERIFY_IS_APPROX(MatrixXd(A.toDense()), ref); } +template <int Tag> +struct SparseCwiseScalar { + explicit SparseCwiseScalar(int v = 0) : value(v) {} + int value; +}; + +SparseCwiseScalar<2> operator*(const SparseCwiseScalar<0>& lhs, const SparseCwiseScalar<1>& rhs) { + return SparseCwiseScalar<2>(lhs.value * rhs.value); +} + +SparseCwiseScalar<2> operator+(const SparseCwiseScalar<0>& lhs, const SparseCwiseScalar<1>& rhs) { + return SparseCwiseScalar<2>(lhs.value + rhs.value); +} + +SparseCwiseScalar<3> operator*(const SparseCwiseScalar<1>& lhs, const SparseCwiseScalar<0>& rhs) { + return SparseCwiseScalar<3>(lhs.value * rhs.value); +} + +namespace Eigen { +template <> +struct ScalarBinaryOpTraits<SparseCwiseScalar<0>, SparseCwiseScalar<1>, + internal::scalar_product_op<SparseCwiseScalar<0>, SparseCwiseScalar<1>>> { + using ReturnType = SparseCwiseScalar<2>; +}; + +template <> +struct ScalarBinaryOpTraits<SparseCwiseScalar<0>, SparseCwiseScalar<1>, + internal::scalar_sum_op<SparseCwiseScalar<0>, SparseCwiseScalar<1>>> { + using ReturnType = SparseCwiseScalar<2>; +}; + +template <> +struct ScalarBinaryOpTraits<SparseCwiseScalar<1>, SparseCwiseScalar<0>, + internal::scalar_product_op<SparseCwiseScalar<1>, SparseCwiseScalar<0>>> { + using ReturnType = SparseCwiseScalar<3>; +}; +} // namespace Eigen + +template <int Order> +void test_sparse_cwise_product_operand_types() { + using Left = SparseCwiseScalar<0>; + using Right = SparseCwiseScalar<1>; + using Result = SparseCwiseScalar<2>; + SparseMatrix<Left, Order> lhs(3, 3); + lhs.reserve(VectorXi::Constant(3, 3)); + lhs.insert(0, 1) = Left(2); + lhs.insert(2, 0) = Left(-3); + Matrix<Right, Dynamic, Dynamic, Order> rhs(3, 3); + for (Index j = 0; j < 3; ++j) + for (Index i = 0; i < 3; ++i) rhs(i, j) = Right(int(i + 2 * j + 1)); + + for (int compressed = 0; compressed < 2; ++compressed) { + SparseMatrix<Result, Order> result = lhs.cwiseProduct(rhs); + VERIFY_IS_EQUAL(result.nonZeros(), 2); + for (Index j = 0; j < 3; ++j) + for (Index i = 0; i < 3; ++i) VERIFY_IS_EQUAL(result.coeff(i, j).value, lhs.coeff(i, j).value * rhs(i, j).value); + lhs.makeCompressed(); + } + + Matrix<Left, Dynamic, Dynamic, Order> denseLeft(3, 3); + SparseMatrix<Right, Order> sparseRight(3, 3); + sparseRight.reserve(VectorXi::Constant(3, 3)); + sparseRight.insert(0, 1) = Right(4); + sparseRight.insert(2, 0) = Right(5); + for (Index j = 0; j < 3; ++j) + for (Index i = 0; i < 3; ++i) denseLeft(i, j) = Left(int(i + 2 * j + 1)); + for (int compressed = 0; compressed < 2; ++compressed) { + const SparseMatrix<Result, Order> result = denseLeft.cwiseProduct(sparseRight); + VERIFY_IS_EQUAL(result.nonZeros(), 2); + for (Index j = 0; j < 3; ++j) + for (Index i = 0; i < 3; ++i) + VERIFY_IS_EQUAL(result.coeff(i, j).value, denseLeft(i, j).value * sparseRight.coeff(i, j).value); + sparseRight.makeCompressed(); + } +} + +template <int Order> +void test_sparse_cwise_sum_operand_types() { + using Left = SparseCwiseScalar<0>; + using Right = SparseCwiseScalar<1>; + using Result = SparseCwiseScalar<2>; + SparseMatrix<Left, Order> lhs(3, 3); + SparseMatrix<Right, Order> rhs(3, 3); + lhs.reserve(VectorXi::Constant(3, 3)); + rhs.reserve(VectorXi::Constant(3, 3)); + lhs.insert(0, 1) = Left(2); + lhs.insert(2, 0) = Left(-3); + rhs.insert(0, 1) = Right(4); + rhs.insert(1, 2) = Right(5); + Matrix<Left, Dynamic, Dynamic, Order> denseLhs(3, 3); + Matrix<Right, Dynamic, Dynamic, Order> denseRhs(3, 3); + for (Index j = 0; j < 3; ++j) + for (Index i = 0; i < 3; ++i) { + denseLhs(i, j) = Left(int(i + j + 1)); + denseRhs(i, j) = Right(int(i - j - 1)); + } + for (int compressed = 0; compressed < 2; ++compressed) { + const SparseMatrix<Result, Order> sparse = lhs + rhs; + const Matrix<Result, Dynamic, Dynamic, Order> denseSparse = denseLhs + rhs; + const Matrix<Result, Dynamic, Dynamic, Order> sparseDense = lhs + denseRhs; + for (Index j = 0; j < 3; ++j) + for (Index i = 0; i < 3; ++i) { + VERIFY_IS_EQUAL(sparse.coeff(i, j).value, lhs.coeff(i, j).value + rhs.coeff(i, j).value); + VERIFY_IS_EQUAL(denseSparse(i, j).value, denseLhs(i, j).value + rhs.coeff(i, j).value); + VERIFY_IS_EQUAL(sparseDense(i, j).value, lhs.coeff(i, j).value + denseRhs(i, j).value); + } + lhs.makeCompressed(); + rhs.makeCompressed(); + } + + Matrix<double, 2, 2, Order> real = Matrix<double, 2, 2, Order>::Ones(); + SparseMatrix<std::complex<double>, Order> complex(2, 2); + complex.insert(0, 1) = std::complex<double>(2, 3); + const Matrix<std::complex<double>, 2, 2, Order> result = real + complex; + for (Index j = 0; j < 2; ++j) + for (Index i = 0; i < 2; ++i) VERIFY_IS_EQUAL(result(i, j), std::complex<double>(real(i, j)) + complex.coeff(i, j)); +} + +struct SparseNoncommutingScalar { + explicit SparseNoncommutingScalar(int diagonal = 0) : values{{diagonal, 0, 0, diagonal}} {} + SparseNoncommutingScalar& operator=(int diagonal) { + values = {{diagonal, 0, 0, diagonal}}; + return *this; + } + std::array<int, 4> values; +}; + +double abs(const SparseNoncommutingScalar& scalar) { + double result = 0; + for (int value : scalar.values) result += numext::abs(value); + return result; +} + +namespace Eigen { +template <> +struct NumTraits<SparseNoncommutingScalar> : GenericNumTraits<SparseNoncommutingScalar> { + using Real = double; + static constexpr bool IsSigned = true; + static constexpr bool RequireInitialization = true; +}; +} // namespace Eigen + +SparseNoncommutingScalar operator*(const SparseNoncommutingScalar& lhs, const SparseNoncommutingScalar& rhs) { + SparseNoncommutingScalar result; + for (int i = 0; i < 2; ++i) + for (int j = 0; j < 2; ++j) + for (int k = 0; k < 2; ++k) result.values[2 * i + j] += lhs.values[2 * i + k] * rhs.values[2 * k + j]; + return result; +} + +template <int Order> +void test_sparse_diagonal_products() { + using DiagonalScalar = SparseCwiseScalar<0>; + using MatrixScalar = SparseCwiseScalar<1>; + using Scalar = SparseNoncommutingScalar; + DiagonalMatrix<DiagonalScalar, 3> diagonal; + DiagonalMatrix<Scalar, 3> blockDiagonal; + for (Index i = 0; i < 3; ++i) { + diagonal.diagonal()(i) = DiagonalScalar(int(i) + 2); + blockDiagonal.diagonal()(i).values = {{1, 0, int(i) + 2, 1}}; + } + SparseMatrix<MatrixScalar, Order> matrix(3, 3); + SparseMatrix<Scalar, Order> blocks(3, 3); + matrix.reserve(VectorXi::Constant(3, 3)); + blocks.reserve(VectorXi::Constant(3, 3)); + matrix.insert(0, 1) = MatrixScalar(5); + matrix.insert(2, 0) = MatrixScalar(-2); + blocks.insert(0, 1).values = {{1, 2, 0, 1}}; + blocks.insert(2, 0).values = {{2, 1, 0, 1}}; + for (int compressed = 0; compressed < 2; ++compressed) { + const SparseMatrix<SparseCwiseScalar<2>, Order> left = diagonal * matrix; + const SparseMatrix<SparseCwiseScalar<3>, Order> right = matrix * diagonal; + const SparseMatrix<Scalar, Order> blockLeft = blockDiagonal * blocks; + const SparseMatrix<Scalar, Order> blockRight = blocks * blockDiagonal; + VERIFY_IS_EQUAL(left.nonZeros(), matrix.nonZeros()); + VERIFY_IS_EQUAL(right.nonZeros(), matrix.nonZeros()); + for (Index j = 0; j < 3; ++j) + for (Index i = 0; i < 3; ++i) { + VERIFY_IS_EQUAL(left.coeff(i, j).value, diagonal.diagonal()(i).value * matrix.coeff(i, j).value); + VERIFY_IS_EQUAL(right.coeff(i, j).value, matrix.coeff(i, j).value * diagonal.diagonal()(j).value); + const Scalar expectedLeft = blockDiagonal.diagonal()(i) * blocks.coeff(i, j); + const Scalar expectedRight = blocks.coeff(i, j) * blockDiagonal.diagonal()(j); + VERIFY(blockLeft.coeff(i, j).values == expectedLeft.values); + VERIFY(blockRight.coeff(i, j).values == expectedRight.values); + } + matrix.makeCompressed(); + blocks.makeCompressed(); + } +} + +SparseNoncommutingScalar& operator+=(SparseNoncommutingScalar& lhs, const SparseNoncommutingScalar& rhs) { + for (int i = 0; i < 4; ++i) lhs.values[i] += rhs.values[i]; + return lhs; +} + +template <int LOrder, int ROrder, int DstOrder> +void test_sparse_product_operand_order() { + using Scalar = SparseNoncommutingScalar; + SparseMatrix<Scalar, LOrder> lhs(2, 3); + SparseMatrix<Scalar, ROrder> rhs(3, 2); + lhs.reserve(VectorXi::Constant(lhs.outerSize(), 3)); + rhs.reserve(VectorXi::Constant(rhs.outerSize(), 3)); + lhs.insert(0, 0).values = {{1, 2, 0, 1}}; + lhs.insert(0, 2).values = {{2, 1, 0, 1}}; + lhs.insert(1, 1).values = {{1, -1, 1, 1}}; + rhs.insert(0, 1).values = {{1, 0, 3, 1}}; + rhs.insert(2, 1).values = {{2, 0, 1, 1}}; + rhs.insert(1, 0).values = {{1, 1, 2, 1}}; + for (int compressed = 0; compressed < 2; ++compressed) { + const SparseMatrix<Scalar, DstOrder> sparse = lhs * rhs; + const Matrix<Scalar, Dynamic, Dynamic, DstOrder> dense = lhs * rhs; + VERIFY(sparse.innerIndicesAreSorted()); + VERIFY_IS_EQUAL(sparse.nonZeros(), 2); + for (Index j = 0; j < 2; ++j) + for (Index i = 0; i < 2; ++i) { + Scalar expected; + for (Index k = 0; k < 3; ++k) expected += lhs.coeff(i, k) * rhs.coeff(k, j); + VERIFY(sparse.coeff(i, j).values == expected.values); + VERIFY(dense(i, j).values == expected.values); + } + lhs.makeCompressed(); + rhs.makeCompressed(); + } +} + +template <int LOrder, int ROrder, int DstOrder> +void test_sparse_pruned_operand_order() { + using Scalar = SparseNoncommutingScalar; + SparseMatrix<Scalar, LOrder> lhs(2, 3); + SparseMatrix<Scalar, ROrder> rhs(3, 2); + lhs.reserve(VectorXi::Constant(lhs.outerSize(), 3)); + rhs.reserve(VectorXi::Constant(rhs.outerSize(), 3)); + lhs.insert(0, 0).values = {{1, 2, 0, 1}}; + lhs.insert(0, 2).values = {{2, 1, 0, 1}}; + rhs.insert(0, 1).values = {{1, 0, 3, 1}}; + rhs.insert(2, 1).values = {{2, 0, 1, 1}}; + for (int compressed = 0; compressed < 2; ++compressed) { + SparseMatrix<Scalar, DstOrder> result(2, 2); + internal::sparse_sparse_product_with_pruning_selector<decltype(lhs), decltype(rhs), decltype(result)>::run( + lhs, rhs, result, 0.0); + const SparseMatrix<Scalar, DstOrder> publicResult = (lhs * rhs).pruned(Scalar(0), 0.0); + VERIFY(result.innerIndicesAreSorted()); + VERIFY_IS_EQUAL(result.nonZeros(), 1); + for (Index j = 0; j < 2; ++j) + for (Index i = 0; i < 2; ++i) { + Scalar expected; + for (Index k = 0; k < 3; ++k) expected += lhs.coeff(i, k) * rhs.coeff(k, j); + VERIFY(result.coeff(i, j).values == expected.values); + VERIFY(publicResult.coeff(i, j).values == expected.values); + } + lhs.makeCompressed(); + rhs.makeCompressed(); + } +} + +template <typename = void> +void test_sparse_outer_operand_types() { + SparseVector<SparseCwiseScalar<0>> column(2); + column.insert(1) = SparseCwiseScalar<0>(3); + Matrix<SparseCwiseScalar<1>, 1, Dynamic> row(3); + for (Index i = 0; i < row.size(); ++i) row(i) = SparseCwiseScalar<1>(int(i + 2)); + using LeftEval = internal::sparse_dense_outer_product_evaluator<decltype(column), decltype(row), false>; + const LeftEval left(column, row); + for (Index j = 0; j < row.size(); ++j) { + typename LeftEval::InnerIterator it(left, j); + VERIFY(it); + VERIFY_IS_EQUAL(it.row(), 1); + VERIFY_IS_EQUAL(it.value().value, 3 * row(j).value); + ++it; + VERIFY(!it); + } + + SparseVector<SparseCwiseScalar<1>, RowMajor> sparseRow(3); + sparseRow.insert(2) = SparseCwiseScalar<1>(5); + Matrix<SparseCwiseScalar<0>, Dynamic, 1> denseColumn(2); + denseColumn(0) = SparseCwiseScalar<0>(2); + denseColumn(1) = SparseCwiseScalar<0>(4); + using RightEval = internal::sparse_dense_outer_product_evaluator<decltype(denseColumn), decltype(sparseRow), true>; + const RightEval right(denseColumn, sparseRow); + for (Index i = 0; i < denseColumn.size(); ++i) { + typename RightEval::InnerIterator it(right, i); + VERIFY(it); + VERIFY_IS_EQUAL(it.col(), 2); + VERIFY_IS_EQUAL(it.value().value, denseColumn(i).value * 5); + } +} + +template <int Order> +void test_sparse_quaternion_product_order() { + using Scalar = noncommutative_scalar::Quaternion; + const Scalar i(0, 1, 0, 0), j(0, 0, 1, 0), k(0, 0, 0, 1); + SparseMatrix<Scalar, Order> lhs(1, 1), rhs(1, 1), pruned(1, 1); + lhs.insert(0, 0) = i; + rhs.insert(0, 0) = j; + DiagonalMatrix<Scalar, 1> diagonal; + diagonal.diagonal()(0) = j; + Matrix<Scalar, 1, 1> dense; + SparseVector<Scalar, RowMajor> row(1); + row.insert(0) = j; + dense(0) = i; + using Outer = internal::sparse_dense_outer_product_evaluator<decltype(dense), decltype(row), true>; + const Outer outer(dense, row); + VERIFY(typename Outer::InnerIterator(outer, 0).value() == k); + for (int compressed = 0; compressed < 2; ++compressed) { + const SparseMatrix<Scalar, Order> left = diagonal * lhs, right = lhs * diagonal, product = lhs * rhs; + const Matrix<Scalar, 1, 1, Order> denseProduct = lhs * rhs; + VERIFY(left.coeff(0, 0) == -k); + VERIFY(right.coeff(0, 0) == k); + VERIFY(product.coeff(0, 0) == k); + VERIFY(denseProduct(0) == k); + internal::sparse_sparse_product_with_pruning_selector<decltype(lhs), decltype(rhs), decltype(pruned)>::run( + lhs, rhs, pruned, 0.0); + VERIFY(pruned.coeff(0, 0) == k); + lhs.makeCompressed(); + rhs.makeCompressed(); + } +} + +template <int Order> +void test_dense_sparse_quaternion_order() { + using Scalar = noncommutative_scalar::Quaternion; + const Scalar i(0, 1, 0, 0), j(0, 0, 1, 0); + Matrix<Scalar, Dynamic, Dynamic, Order> dense = Matrix<Scalar, Dynamic, Dynamic, Order>::Constant(3, 3, i); + Matrix<Scalar, 1, Dynamic> row = Matrix<Scalar, 1, Dynamic>::Constant(3, i); + SparseMatrix<Scalar, Order> sparse(3, 3); + sparse.reserve(VectorXi::Constant(sparse.outerSize(), 3)); + sparse.insert(0, 0) = j; + sparse.insert(1, 2) = j; + for (int compressed = 0; compressed < 2; ++compressed) { + const Matrix<Scalar, Dynamic, Dynamic> product = dense * sparse; + Matrix<Scalar, Dynamic, Dynamic> scaled = Matrix<Scalar, Dynamic, Dynamic>::Zero(3, 3); + scaled.noalias() += j * (dense * sparse); + Matrix<Scalar, Dynamic, Dynamic> sparseLeft = Matrix<Scalar, Dynamic, Dynamic>::Zero(3, 3); + sparseLeft.noalias() += i * (sparse * dense); + const Matrix<Scalar, 1, Dynamic> rowSparse = row * sparse; + const SparseMatrix<Scalar, Order> cwise = dense.cwiseProduct(sparse); + for (Index col = 0; col < sparse.cols(); ++col) { + Scalar expectedRow; + for (Index k = 0; k < row.size(); ++k) expectedRow += row(k) * sparse.coeff(k, col); + VERIFY(rowSparse(col) == expectedRow); + for (Index r = 0; r < dense.rows(); ++r) { + Scalar expected; + for (Index k = 0; k < dense.cols(); ++k) expected += dense(r, k) * sparse.coeff(k, col); + VERIFY(product(r, col) == expected); + VERIFY(scaled(r, col) == j * expected); + Scalar expectedLeft; + for (Index k = 0; k < dense.rows(); ++k) expectedLeft += sparse.coeff(r, k) * dense(k, col); + VERIFY(sparseLeft(r, col) == i * expectedLeft); + VERIFY(cwise.coeff(r, col) == dense(r, col) * sparse.coeff(r, col)); + } + } + sparse.makeCompressed(); + } +} + template <typename = void> void test_sparse_vector_dense_product() { SparseVector<double> sv(3); @@ -1001,6 +1357,31 @@ CALL_SUBTEST_5((test_mixing_types<float>())); CALL_SUBTEST_5((test_mixed_storage<>())); + CALL_SUBTEST_5((test_sparse_cwise_product_operand_types<ColMajor>())); + CALL_SUBTEST_7((test_sparse_cwise_sum_operand_types<RowMajor>())); + CALL_SUBTEST_8((test_sparse_diagonal_products<ColMajor>())); + CALL_SUBTEST_8((test_sparse_diagonal_products<RowMajor>())); + CALL_SUBTEST_9((test_sparse_product_operand_order<ColMajor, ColMajor, ColMajor>())); + CALL_SUBTEST_9((test_sparse_product_operand_order<ColMajor, ColMajor, RowMajor>())); + CALL_SUBTEST_9((test_sparse_product_operand_order<ColMajor, RowMajor, ColMajor>())); + CALL_SUBTEST_9((test_sparse_product_operand_order<ColMajor, RowMajor, RowMajor>())); + CALL_SUBTEST_9((test_sparse_product_operand_order<RowMajor, ColMajor, ColMajor>())); + CALL_SUBTEST_9((test_sparse_product_operand_order<RowMajor, ColMajor, RowMajor>())); + CALL_SUBTEST_9((test_sparse_product_operand_order<RowMajor, RowMajor, ColMajor>())); + CALL_SUBTEST_9((test_sparse_product_operand_order<RowMajor, RowMajor, RowMajor>())); + CALL_SUBTEST_10((test_sparse_outer_operand_types<>())); + CALL_SUBTEST_12((test_sparse_quaternion_product_order<ColMajor>())); + CALL_SUBTEST_12((test_sparse_quaternion_product_order<RowMajor>())); + CALL_SUBTEST_14((test_dense_sparse_quaternion_order<ColMajor>())); + CALL_SUBTEST_14((test_dense_sparse_quaternion_order<RowMajor>())); + CALL_SUBTEST_11((test_sparse_pruned_operand_order<ColMajor, ColMajor, ColMajor>())); + CALL_SUBTEST_11((test_sparse_pruned_operand_order<ColMajor, ColMajor, RowMajor>())); + CALL_SUBTEST_11((test_sparse_pruned_operand_order<ColMajor, RowMajor, ColMajor>())); + CALL_SUBTEST_11((test_sparse_pruned_operand_order<ColMajor, RowMajor, RowMajor>())); + CALL_SUBTEST_11((test_sparse_pruned_operand_order<RowMajor, ColMajor, ColMajor>())); + CALL_SUBTEST_11((test_sparse_pruned_operand_order<RowMajor, ColMajor, RowMajor>())); + CALL_SUBTEST_11((test_sparse_pruned_operand_order<RowMajor, RowMajor, ColMajor>())); + CALL_SUBTEST_11((test_sparse_pruned_operand_order<RowMajor, RowMajor, RowMajor>())); CALL_SUBTEST_6((test_pruned_product_custom_scalar<>())); CALL_SUBTEST_6((test_ambivector_discard_custom_scalar<>()));
diff --git a/test/sparse_ref.cpp b/test/sparse_ref.cpp index 503aae1..36961d2 100644 --- a/test/sparse_ref.cpp +++ b/test/sparse_ref.cpp
@@ -25,6 +25,7 @@ #include "main.h" #include <Eigen/SparseCore> +#include "sparse_test_helpers.h" #define VERIFY_EVALUATION_COUNT(XPR, N) \ { \ @@ -80,35 +81,6 @@ VERIFY_IS_EQUAL(a.toDense(), b.toDense()); } -template <typename T> -typename T::Map make_map(T &xpr) { - return typename T::Map(xpr.rows(), xpr.cols(), xpr.nonZeros(), xpr.outerIndexPtr(), xpr.innerIndexPtr(), - xpr.valuePtr(), xpr.innerNonZeroPtr()); -} - -template <typename T> -auto innerpanel(T &xpr, Index start, Index size) - -> std::enable_if_t<(T::Flags & RowMajorBit) == RowMajorBit, decltype(xpr.middleRows(start, size))> { - return xpr.middleRows(start, size); -} - -template <typename T> -auto innerpanel(T &xpr, Index start, Index size) - -> std::enable_if_t<(T::Flags & RowMajorBit) == 0, decltype(xpr.middleCols(start, size))> { - return xpr.middleCols(start, size); -} - -template <typename T> -auto innervector(T &xpr, Index index) - -> std::enable_if_t<(T::Flags & RowMajorBit) == RowMajorBit, decltype(xpr.row(index))> { - return xpr.row(index); -} - -template <typename T> -auto innervector(T &xpr, Index index) -> std::enable_if_t<(T::Flags & RowMajorBit) == 0, decltype(xpr.col(index))> { - return xpr.col(index); -} - template <typename XprType, typename ExpectedType> void verify_coeffs(const XprType &xpr, const ExpectedType &expected) { auto coeffs = xpr.coeffs(); @@ -116,16 +88,6 @@ for (Index i = 0; i < expected.size(); ++i) VERIFY_IS_EQUAL(coeffs[i], expected[i]); } -template <int Options> -void init_diag(SparseMatrix<float, Options> &m) { - m.resize(4, 4); - m.insert(0, 0) = 10.f; - m.insert(1, 1) = 20.f; - m.insert(2, 2) = 30.f; - m.insert(3, 3) = 40.f; - m.makeCompressed(); -} - template <typename MatrixType> std::enable_if_t<(MatrixType::Flags & RowMajorBit) == 0> init_noncompressed_inner_panel(MatrixType &m) { m.resize(4, 4); @@ -245,8 +207,8 @@ Array<float, 2, 1> expected; expected << 20.f, 30.f; - Ref<SparseVector<float>> inner(innervector(xpr, 2)); - Ref<const SparseVector<float>> const_inner(innervector(xpr, 2)); + Ref<SparseVector<float>> inner(innervec(xpr, 2)); + Ref<const SparseVector<float>> const_inner(innervec(xpr, 2)); const float updated0 = 211.f + variant; const float updated1 = 221.f + variant; @@ -270,10 +232,10 @@ void check_ref_slice_coeffs() { verify_ref_and_map<SparseMatrix<float>>( - [](SparseMatrix<float> &matrix) { init_diag(matrix); }, + [](SparseMatrix<float> &matrix) { init_sparse_diagonal(matrix); }, [](auto &xpr, auto &matrix, int variant) { verify_slice_coeffs(xpr, matrix, variant); }); verify_ref_and_map<SparseMatrix<float, RowMajor>>( - [](SparseMatrix<float, RowMajor> &matrix) { init_diag(matrix); }, + [](SparseMatrix<float, RowMajor> &matrix) { init_sparse_diagonal(matrix); }, [](auto &xpr, auto &matrix, int variant) { verify_slice_coeffs(xpr, matrix, variant); }); }
diff --git a/test/sparse_solver.h b/test/sparse_solver.h index 60dfdd1..43a0d20 100644 --- a/test/sparse_solver.h +++ b/test/sparse_solver.h
@@ -316,6 +316,7 @@ void check_sparse_determinant(Solver& solver, const typename Solver::MatrixType& A, const DenseMat& dA) { typedef typename Solver::MatrixType Mat; typedef typename Mat::Scalar Scalar; + typedef typename NumTraits<Scalar>::Real RealScalar; solver.compute(A); if (solver.info() != Success) { @@ -323,8 +324,17 @@ return; } - Scalar refDet = dA.determinant(); - VERIFY_IS_APPROX(refDet, solver.determinant()); + // d det(A) = det(A) tr(A^{-1} dA). With the LU backward error |dA| <= gamma_n |L||U| (Higham, ASNA, Thm. 9.3) and + // |L||U| ~ |A|, each determinant is off by about n eps cond |det(A)|, cond = sum_ij |A^{-1}_ji| |A_ij| >= n, which + // is unbounded for the matrices check_sparse_square_determinant draws. The factor 8 is empirical: partial pivoting + // stays below 0.5. The test_precision floor admits the growth of threshold pivoting, which |L||U| ~ |A| ignores; + // UMFPACK's symmetric strategy takes diagonal pivots down to 1e-3 of the column maximum. + const RealScalar cond = dA.inverse().transpose().cwiseAbs().cwiseProduct(dA.cwiseAbs()).sum(); + const RealScalar tolerance = + numext::maxi(test_precision<Scalar>(), RealScalar(8 * dA.rows()) * NumTraits<RealScalar>::epsilon() * cond); + VERIFY((numext::isfinite)(tolerance)); + const Scalar refDet = dA.determinant(); + VERIFY(numext::abs(solver.determinant() - refDet) <= tolerance * numext::abs(refDet)); } template <typename Solver, typename DenseMat> void check_sparse_abs_determinant(Solver& solver, const typename Solver::MatrixType& A, const DenseMat& dA) {
diff --git a/test/sparse_test_helpers.h b/test/sparse_test_helpers.h new file mode 100644 index 0000000..cc6ee3a --- /dev/null +++ b/test/sparse_test_helpers.h
@@ -0,0 +1,53 @@ +// SPDX-FileCopyrightText: 2008-2015 Gael Guennebaud <gael.guennebaud@inria.fr> +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_SPARSE_TEST_HELPERS_H +#define EIGEN_SPARSE_TEST_HELPERS_H + +#include <type_traits> + +#include <Eigen/SparseCore> + +namespace Eigen { + +template <typename T, std::enable_if_t<(T::Flags & RowMajorBit) != 0, int> = 0> +auto innervec(T& matrix, Index index) -> decltype(matrix.row(index)) { + return matrix.row(index); +} + +template <typename T, std::enable_if_t<(T::Flags & RowMajorBit) == 0, int> = 0> +auto innervec(T& matrix, Index index) -> decltype(matrix.col(index)) { + return matrix.col(index); +} + +template <typename T> +typename T::Map make_map(T& matrix) { + return typename T::Map(matrix.rows(), matrix.cols(), matrix.nonZeros(), matrix.outerIndexPtr(), + matrix.innerIndexPtr(), matrix.valuePtr(), matrix.innerNonZeroPtr()); +} + +template <typename T, std::enable_if_t<(T::Flags & RowMajorBit) != 0, int> = 0> +auto innerpanel(T& matrix, Index start, Index size) -> decltype(matrix.middleRows(start, size)) { + return matrix.middleRows(start, size); +} + +template <typename T, std::enable_if_t<(T::Flags & RowMajorBit) == 0, int> = 0> +auto innerpanel(T& matrix, Index start, Index size) -> decltype(matrix.middleCols(start, size)) { + return matrix.middleCols(start, size); +} + +template <typename MatrixType> +void init_sparse_diagonal(MatrixType& matrix) { + using Scalar = typename MatrixType::Scalar; + matrix.resize(4, 4); + matrix.insert(0, 0) = Scalar(10); + matrix.insert(1, 1) = Scalar(20); + matrix.insert(2, 2) = Scalar(30); + matrix.insert(3, 3) = Scalar(40); + matrix.makeCompressed(); +} + +} // namespace Eigen + +#endif // EIGEN_SPARSE_TEST_HELPERS_H
diff --git a/test/stddeque.cpp b/test/stddeque.cpp index 822d309..d8842bc 100644 --- a/test/stddeque.cpp +++ b/test/stddeque.cpp
@@ -12,6 +12,7 @@ #include "main.h" #include <Eigen/StdDeque> #include <Eigen/Geometry> +#include "stl_container_test_helpers.h" template <typename MatrixType> void check_stddeque_matrix(const MatrixType& m) { @@ -19,80 +20,23 @@ Index cols = m.cols(); MatrixType x = MatrixType::Random(rows, cols), y = MatrixType::Random(rows, cols); std::deque<MatrixType, Eigen::aligned_allocator<MatrixType> > v(10, MatrixType::Zero(rows, cols)), w(20, y); - v.front() = x; - w.front() = w.back(); - VERIFY_IS_APPROX(w.front(), w.back()); - v = w; - - typename std::deque<MatrixType, Eigen::aligned_allocator<MatrixType> >::iterator vi = v.begin(); - typename std::deque<MatrixType, Eigen::aligned_allocator<MatrixType> >::iterator wi = w.begin(); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(*vi, *wi); - ++vi; - ++wi; - } - - v.resize(21, MatrixType::Zero(rows, cols)); - v.back() = x; - VERIFY_IS_APPROX(v.back(), x); - v.resize(22, y); - VERIFY_IS_APPROX(v.back(), y); - v.push_back(x); - VERIFY_IS_APPROX(v.back(), x); + check_stl_container_front(v, w, x, y, MatrixType::Zero(rows, cols)); } template <typename TransformType> void check_stddeque_transform(const TransformType&) { - typedef typename TransformType::MatrixType MatrixType; + using MatrixType = typename TransformType::MatrixType; TransformType x(MatrixType::Random()), y(MatrixType::Random()), ti = TransformType::Identity(); std::deque<TransformType, Eigen::aligned_allocator<TransformType> > v(10, ti), w(20, y); - v.front() = x; - w.front() = w.back(); - VERIFY_IS_APPROX(w.front(), w.back()); - v = w; - - typename std::deque<TransformType, Eigen::aligned_allocator<TransformType> >::iterator vi = v.begin(); - typename std::deque<TransformType, Eigen::aligned_allocator<TransformType> >::iterator wi = w.begin(); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(*vi, *wi); - ++vi; - ++wi; - } - - v.resize(21, ti); - v.back() = x; - VERIFY_IS_APPROX(v.back(), x); - v.resize(22, y); - VERIFY_IS_APPROX(v.back(), y); - v.push_back(x); - VERIFY_IS_APPROX(v.back(), x); + check_stl_container_front(v, w, x, y, ti); } template <typename QuaternionType> void check_stddeque_quaternion(const QuaternionType&) { - typedef typename QuaternionType::Coefficients Coefficients; + using Coefficients = typename QuaternionType::Coefficients; QuaternionType x(Coefficients::Random()), y(Coefficients::Random()), qi = QuaternionType::Identity(); std::deque<QuaternionType, Eigen::aligned_allocator<QuaternionType> > v(10, qi), w(20, y); - v.front() = x; - w.front() = w.back(); - VERIFY_IS_APPROX(w.front(), w.back()); - v = w; - - typename std::deque<QuaternionType, Eigen::aligned_allocator<QuaternionType> >::iterator vi = v.begin(); - typename std::deque<QuaternionType, Eigen::aligned_allocator<QuaternionType> >::iterator wi = w.begin(); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(*vi, *wi); - ++vi; - ++wi; - } - - v.resize(21, qi); - v.back() = x; - VERIFY_IS_APPROX(v.back(), x); - v.resize(22, y); - VERIFY_IS_APPROX(v.back(), y); - v.push_back(x); - VERIFY_IS_APPROX(v.back(), x); + check_stl_container_front(v, w, x, y, qi); } EIGEN_DECLARE_TEST(stddeque) {
diff --git a/test/stddeque_overload.cpp b/test/stddeque_overload.cpp index 282f1bd..0e1c250 100644 --- a/test/stddeque_overload.cpp +++ b/test/stddeque_overload.cpp
@@ -13,6 +13,7 @@ #include <Eigen/StdDeque> #include <Eigen/Geometry> +#include "stl_container_test_helpers.h" EIGEN_DEFINE_STL_DEQUE_SPECIALIZATION(Vector4f) @@ -32,89 +33,23 @@ Index cols = m.cols(); MatrixType x = MatrixType::Random(rows, cols), y = MatrixType::Random(rows, cols); std::deque<MatrixType> v(10, MatrixType::Zero(rows, cols)), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.resize(22, y); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - - // do a lot of push_back such that the deque gets internally resized - // (with memory reallocation) - MatrixType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(v[i] == w[(i - 23) % w.size()]); - } + check_stl_container_indexed(v, w, x, y); } template <typename TransformType> void check_stddeque_transform(const TransformType&) { - typedef typename TransformType::MatrixType MatrixType; + using MatrixType = typename TransformType::MatrixType; TransformType x(MatrixType::Random()), y(MatrixType::Random()), ti = TransformType::Identity(); std::deque<TransformType> v(10, ti), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21, ti); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.resize(22, y); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - - // do a lot of push_back such that the deque gets internally resized - // (with memory reallocation) - TransformType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(v[i].matrix() == w[(i - 23) % w.size()].matrix()); - } + check_stl_container_indexed(v, w, x, y, ti); } template <typename QuaternionType> void check_stddeque_quaternion(const QuaternionType&) { - typedef typename QuaternionType::Coefficients Coefficients; + using Coefficients = typename QuaternionType::Coefficients; QuaternionType x(Coefficients::Random()), y(Coefficients::Random()), qi = QuaternionType::Identity(); std::deque<QuaternionType> v(10, qi), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21, qi); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.resize(22, y); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - - // do a lot of push_back such that the deque gets internally resized - // (with memory reallocation) - QuaternionType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(v[i].coeffs() == w[(i - 23) % w.size()].coeffs()); - } + check_stl_container_indexed(v, w, x, y, qi); } EIGEN_DECLARE_TEST(stddeque_overload) {
diff --git a/test/stdlist.cpp b/test/stdlist.cpp index cb65eff..b986a7f 100644 --- a/test/stdlist.cpp +++ b/test/stdlist.cpp
@@ -12,6 +12,7 @@ #include "main.h" #include <Eigen/StdList> #include <Eigen/Geometry> +#include "stl_container_test_helpers.h" template <typename MatrixType> void check_stdlist_matrix(const MatrixType& m) { @@ -19,80 +20,23 @@ Index cols = m.cols(); MatrixType x = MatrixType::Random(rows, cols), y = MatrixType::Random(rows, cols); std::list<MatrixType, Eigen::aligned_allocator<MatrixType> > v(10, MatrixType::Zero(rows, cols)), w(20, y); - v.front() = x; - w.front() = w.back(); - VERIFY_IS_APPROX(w.front(), w.back()); - v = w; - - typename std::list<MatrixType, Eigen::aligned_allocator<MatrixType> >::iterator vi = v.begin(); - typename std::list<MatrixType, Eigen::aligned_allocator<MatrixType> >::iterator wi = w.begin(); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(*vi, *wi); - ++vi; - ++wi; - } - - v.resize(21, MatrixType::Zero(rows, cols)); - v.back() = x; - VERIFY_IS_APPROX(v.back(), x); - v.resize(22, y); - VERIFY_IS_APPROX(v.back(), y); - v.push_back(x); - VERIFY_IS_APPROX(v.back(), x); + check_stl_container_front(v, w, x, y, MatrixType::Zero(rows, cols)); } template <typename TransformType> void check_stdlist_transform(const TransformType&) { - typedef typename TransformType::MatrixType MatrixType; + using MatrixType = typename TransformType::MatrixType; TransformType x(MatrixType::Random()), y(MatrixType::Random()), ti = TransformType::Identity(); std::list<TransformType, Eigen::aligned_allocator<TransformType> > v(10, ti), w(20, y); - v.front() = x; - w.front() = w.back(); - VERIFY_IS_APPROX(w.front(), w.back()); - v = w; - - typename std::list<TransformType, Eigen::aligned_allocator<TransformType> >::iterator vi = v.begin(); - typename std::list<TransformType, Eigen::aligned_allocator<TransformType> >::iterator wi = w.begin(); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(*vi, *wi); - ++vi; - ++wi; - } - - v.resize(21, ti); - v.back() = x; - VERIFY_IS_APPROX(v.back(), x); - v.resize(22, y); - VERIFY_IS_APPROX(v.back(), y); - v.push_back(x); - VERIFY_IS_APPROX(v.back(), x); + check_stl_container_front(v, w, x, y, ti); } template <typename QuaternionType> void check_stdlist_quaternion(const QuaternionType&) { - typedef typename QuaternionType::Coefficients Coefficients; + using Coefficients = typename QuaternionType::Coefficients; QuaternionType x(Coefficients::Random()), y(Coefficients::Random()), qi = QuaternionType::Identity(); std::list<QuaternionType, Eigen::aligned_allocator<QuaternionType> > v(10, qi), w(20, y); - v.front() = x; - w.front() = w.back(); - VERIFY_IS_APPROX(w.front(), w.back()); - v = w; - - typename std::list<QuaternionType, Eigen::aligned_allocator<QuaternionType> >::iterator vi = v.begin(); - typename std::list<QuaternionType, Eigen::aligned_allocator<QuaternionType> >::iterator wi = w.begin(); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(*vi, *wi); - ++vi; - ++wi; - } - - v.resize(21, qi); - v.back() = x; - VERIFY_IS_APPROX(v.back(), x); - v.resize(22, y); - VERIFY_IS_APPROX(v.back(), y); - v.push_back(x); - VERIFY_IS_APPROX(v.back(), x); + check_stl_container_front(v, w, x, y, qi); } EIGEN_DECLARE_TEST(stdlist) {
diff --git a/test/stdlist_overload.cpp b/test/stdlist_overload.cpp index 26558d8..51942ea 100644 --- a/test/stdlist_overload.cpp +++ b/test/stdlist_overload.cpp
@@ -13,6 +13,7 @@ #include <Eigen/StdList> #include <Eigen/Geometry> +#include "stl_container_test_helpers.h" EIGEN_DEFINE_STL_LIST_SPECIALIZATION(Vector4f) @@ -26,127 +27,29 @@ EIGEN_DEFINE_STL_LIST_SPECIALIZATION(Quaternionf) EIGEN_DEFINE_STL_LIST_SPECIALIZATION(Quaterniond) -template <class Container, class Position> -typename Container::iterator get(Container& c, Position position) { - typename Container::iterator it = c.begin(); - std::advance(it, position); - return it; -} - -template <class Container, class Position, class Value> -void set(Container& c, Position position, const Value& value) { - typename Container::iterator it = c.begin(); - std::advance(it, position); - *it = value; -} - template <typename MatrixType> void check_stdlist_matrix(const MatrixType& m) { Index rows = m.rows(); Index cols = m.cols(); MatrixType x = MatrixType::Random(rows, cols), y = MatrixType::Random(rows, cols); std::list<MatrixType> v(10, MatrixType::Zero(rows, cols)), w(20, y); - typename std::list<MatrixType>::iterator itv = get(v, 5); - typename std::list<MatrixType>::iterator itw = get(w, 6); - *itv = x; - *itw = *itv; - VERIFY_IS_APPROX(*itw, *itv); - v = w; - itv = v.begin(); - itw = w.begin(); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(*itw, *itv); - ++itv; - ++itw; - } - - v.resize(21); - set(v, 20, x); - VERIFY_IS_APPROX(*get(v, 20), x); - v.resize(22, y); - VERIFY_IS_APPROX(*get(v, 21), y); - v.push_back(x); - VERIFY_IS_APPROX(*get(v, 22), x); - - // do a lot of push_back such that the list gets internally resized - // (with memory reallocation) - MatrixType* ref = &(*get(w, 0)); - for (int i = 0; i < 30 || ((ref == &(*get(w, 0))) && i < 300); ++i) v.push_back(*get(w, i % w.size())); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY((*get(v, i)) == (*get(w, (i - 23) % w.size()))); - } + check_stl_container_indexed(v, w, x, y); } template <typename TransformType> void check_stdlist_transform(const TransformType&) { - typedef typename TransformType::MatrixType MatrixType; + using MatrixType = typename TransformType::MatrixType; TransformType x(MatrixType::Random()), y(MatrixType::Random()), ti = TransformType::Identity(); std::list<TransformType> v(10, ti), w(20, y); - typename std::list<TransformType>::iterator itv = get(v, 5); - typename std::list<TransformType>::iterator itw = get(w, 6); - *itv = x; - *itw = *itv; - VERIFY_IS_APPROX(*itw, *itv); - v = w; - itv = v.begin(); - itw = w.begin(); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(*itw, *itv); - ++itv; - ++itw; - } - - v.resize(21, ti); - set(v, 20, x); - VERIFY_IS_APPROX(*get(v, 20), x); - v.resize(22, y); - VERIFY_IS_APPROX(*get(v, 21), y); - v.push_back(x); - VERIFY_IS_APPROX(*get(v, 22), x); - - // do a lot of push_back such that the list gets internally resized - // (with memory reallocation) - TransformType* ref = &(*get(w, 0)); - for (int i = 0; i < 30 || ((ref == &(*get(w, 0))) && i < 300); ++i) v.push_back(*get(w, i % w.size())); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(get(v, i)->matrix() == get(w, (i - 23) % w.size())->matrix()); - } + check_stl_container_indexed(v, w, x, y, ti); } template <typename QuaternionType> void check_stdlist_quaternion(const QuaternionType&) { - typedef typename QuaternionType::Coefficients Coefficients; + using Coefficients = typename QuaternionType::Coefficients; QuaternionType x(Coefficients::Random()), y(Coefficients::Random()), qi = QuaternionType::Identity(); std::list<QuaternionType> v(10, qi), w(20, y); - typename std::list<QuaternionType>::iterator itv = get(v, 5); - typename std::list<QuaternionType>::iterator itw = get(w, 6); - *itv = x; - *itw = *itv; - VERIFY_IS_APPROX(*itw, *itv); - v = w; - itv = v.begin(); - itw = w.begin(); - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(*itw, *itv); - ++itv; - ++itw; - } - - v.resize(21, qi); - set(v, 20, x); - VERIFY_IS_APPROX(*get(v, 20), x); - v.resize(22, y); - VERIFY_IS_APPROX(*get(v, 21), y); - v.push_back(x); - VERIFY_IS_APPROX(*get(v, 22), x); - - // do a lot of push_back such that the list gets internally resized - // (with memory reallocation) - QuaternionType* ref = &(*get(w, 0)); - for (int i = 0; i < 30 || ((ref == &(*get(w, 0))) && i < 300); ++i) v.push_back(*get(w, i % w.size())); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(get(v, i)->coeffs() == get(w, (i - 23) % w.size())->coeffs()); - } + check_stl_container_indexed(v, w, x, y, qi); } EIGEN_DECLARE_TEST(stdlist_overload) {
diff --git a/test/stdvector.cpp b/test/stdvector.cpp index 0dd8f88..f5cb5b3 100644 --- a/test/stdvector.cpp +++ b/test/stdvector.cpp
@@ -11,6 +11,7 @@ #include "main.h" #include <Eigen/StdVector> #include <Eigen/Geometry> +#include "stl_container_test_helpers.h" template <typename MatrixType> void check_stdvector_matrix(const MatrixType& m) { @@ -18,92 +19,23 @@ Index cols = m.cols(); MatrixType x = MatrixType::Random(rows, cols), y = MatrixType::Random(rows, cols); std::vector<MatrixType, Eigen::aligned_allocator<MatrixType> > v(10, MatrixType::Zero(rows, cols)), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.resize(22, y); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - VERIFY((std::uintptr_t) & (v[22]) == (std::uintptr_t) & (v[21]) + sizeof(MatrixType)); - - // do a lot of push_back such that the vector gets internally resized - // (with memory reallocation) - MatrixType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(v[i] == w[(i - 23) % w.size()]); - } + check_stl_container_indexed(v, w, x, y); } template <typename TransformType> void check_stdvector_transform(const TransformType&) { - typedef typename TransformType::MatrixType MatrixType; + using MatrixType = typename TransformType::MatrixType; TransformType x(MatrixType::Random()), y(MatrixType::Random()); std::vector<TransformType, Eigen::aligned_allocator<TransformType> > v(10), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.resize(22, y); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - VERIFY((std::uintptr_t) & (v[22]) == (std::uintptr_t) & (v[21]) + sizeof(TransformType)); - - // do a lot of push_back such that the vector gets internally resized - // (with memory reallocation) - TransformType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(v[i].matrix() == w[(i - 23) % w.size()].matrix()); - } + check_stl_container_indexed(v, w, x, y); } template <typename QuaternionType> void check_stdvector_quaternion(const QuaternionType&) { - typedef typename QuaternionType::Coefficients Coefficients; + using Coefficients = typename QuaternionType::Coefficients; QuaternionType x(Coefficients::Random()), y(Coefficients::Random()), qi = QuaternionType::Identity(); std::vector<QuaternionType, Eigen::aligned_allocator<QuaternionType> > v(10, qi), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.resize(22, y); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - VERIFY((std::uintptr_t) & (v[22]) == (std::uintptr_t) & (v[21]) + sizeof(QuaternionType)); - - // do a lot of push_back such that the vector gets internally resized - // (with memory reallocation) - QuaternionType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(v[i].coeffs() == w[(i - 23) % w.size()].coeffs()); - } + check_stl_container_indexed(v, w, x, y); } // the code below triggered an invalid warning with gcc >= 7
diff --git a/test/stdvector_overload.cpp b/test/stdvector_overload.cpp index c084a52..dd5bf28 100644 --- a/test/stdvector_overload.cpp +++ b/test/stdvector_overload.cpp
@@ -13,6 +13,7 @@ #include <Eigen/StdVector> #include <Eigen/Geometry> +#include "stl_container_test_helpers.h" EIGEN_DEFINE_STL_VECTOR_SPECIALIZATION(Vector4f) @@ -32,92 +33,23 @@ Index cols = m.cols(); MatrixType x = MatrixType::Random(rows, cols), y = MatrixType::Random(rows, cols); std::vector<MatrixType> v(10, MatrixType::Zero(rows, cols)), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.resize(22, y); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - VERIFY((std::uintptr_t) & (v[22]) == (std::uintptr_t) & (v[21]) + sizeof(MatrixType)); - - // do a lot of push_back such that the vector gets internally resized - // (with memory reallocation) - MatrixType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(v[i] == w[(i - 23) % w.size()]); - } + check_stl_container_indexed(v, w, x, y); } template <typename TransformType> void check_stdvector_transform(const TransformType&) { - typedef typename TransformType::MatrixType MatrixType; + using MatrixType = typename TransformType::MatrixType; TransformType x(MatrixType::Random()), y(MatrixType::Random()); std::vector<TransformType> v(10), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.resize(22, y); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - VERIFY((std::uintptr_t) & (v[22]) == (std::uintptr_t) & (v[21]) + sizeof(TransformType)); - - // do a lot of push_back such that the vector gets internally resized - // (with memory reallocation) - TransformType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(v[i].matrix() == w[(i - 23) % w.size()].matrix()); - } + check_stl_container_indexed(v, w, x, y); } template <typename QuaternionType> void check_stdvector_quaternion(const QuaternionType&) { - typedef typename QuaternionType::Coefficients Coefficients; + using Coefficients = typename QuaternionType::Coefficients; QuaternionType x(Coefficients::Random()), y(Coefficients::Random()), qi = QuaternionType::Identity(); std::vector<QuaternionType> v(10, qi), w(20, y); - v[5] = x; - w[6] = v[5]; - VERIFY_IS_APPROX(w[6], v[5]); - v = w; - for (int i = 0; i < 20; i++) { - VERIFY_IS_APPROX(w[i], v[i]); - } - - v.resize(21); - v[20] = x; - VERIFY_IS_APPROX(v[20], x); - v.resize(22, y); - VERIFY_IS_APPROX(v[21], y); - v.push_back(x); - VERIFY_IS_APPROX(v[22], x); - VERIFY((std::uintptr_t) & (v[22]) == (std::uintptr_t) & (v[21]) + sizeof(QuaternionType)); - - // do a lot of push_back such that the vector gets internally resized - // (with memory reallocation) - QuaternionType* ref = &w[0]; - for (int i = 0; i < 30 || ((ref == &w[0]) && i < 300); ++i) v.push_back(w[i % w.size()]); - for (unsigned int i = 23; i < v.size(); ++i) { - VERIFY(v[i].coeffs() == w[(i - 23) % w.size()].coeffs()); - } + check_stl_container_indexed(v, w, x, y); } EIGEN_DECLARE_TEST(stdvector_overload) {
diff --git a/test/stl_container_test_helpers.h b/test/stl_container_test_helpers.h new file mode 100644 index 0000000..bad0db0 --- /dev/null +++ b/test/stl_container_test_helpers.h
@@ -0,0 +1,106 @@ +// SPDX-FileCopyrightText: 2008 Benoit Jacob <jacob.benoit.1@gmail.com> +// SPDX-FileCopyrightText: 2010 Hauke Heibel <hauke.heibel@gmail.com> +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#ifndef EIGEN_STL_CONTAINER_TEST_HELPERS_H +#define EIGEN_STL_CONTAINER_TEST_HELPERS_H + +#include <cstdint> +#include <iterator> +#include <vector> + +#include <Eigen/Geometry> + +template <typename Container> +typename Container::reference stl_container_at(Container& container, std::size_t index, + std::random_access_iterator_tag) { + return container[index]; +} + +template <typename Container> +typename Container::reference stl_container_at(Container& container, std::size_t index, + std::bidirectional_iterator_tag) { + auto iterator = container.begin(); + std::advance(iterator, index); + return *iterator; +} + +template <typename Container> +typename Container::reference stl_container_at(Container& container, std::size_t index) { + using Category = typename std::iterator_traits<typename Container::iterator>::iterator_category; + return stl_container_at(container, index, Category{}); +} + +template <typename Derived> +const Derived& stl_container_coeffs(const Eigen::MatrixBase<Derived>& value) { + return value.derived(); +} + +template <typename Scalar, int Dim, int Mode, int Options> +const typename Eigen::Transform<Scalar, Dim, Mode, Options>::MatrixType& stl_container_coeffs( + const Eigen::Transform<Scalar, Dim, Mode, Options>& value) { + return value.matrix(); +} + +template <typename Derived> +auto stl_container_coeffs(const Eigen::QuaternionBase<Derived>& value) -> decltype(value.coeffs()) { + return value.coeffs(); +} + +template <typename Container> +void verify_stl_contiguity(const Container&) {} + +template <typename Value, typename Allocator> +void verify_stl_contiguity(const std::vector<Value, Allocator>& values) { + VERIFY(reinterpret_cast<std::uintptr_t>(&values[22]) == + reinterpret_cast<std::uintptr_t>(&values[21]) + sizeof(Value)); +} + +template <typename Container, typename... Initializers> +void check_stl_container_front(Container& values, Container& other, const typename Container::value_type& x, + const typename Container::value_type& y, const Initializers&... initializers) { + values.front() = x; + other.front() = other.back(); + VERIFY_IS_APPROX(other.front(), other.back()); + values = other; + auto iterator = values.begin(); + auto other_iterator = other.begin(); + for (int i = 0; i < 20; ++i, ++iterator, ++other_iterator) VERIFY_IS_APPROX(*iterator, *other_iterator); + + values.resize(21, initializers...); + values.back() = x; + VERIFY_IS_APPROX(values.back(), x); + values.resize(22, y); + VERIFY_IS_APPROX(values.back(), y); + values.push_back(x); + VERIFY_IS_APPROX(values.back(), x); +} + +template <typename Container, typename... Initializers> +void check_stl_container_indexed(Container& values, Container& other, const typename Container::value_type& x, + const typename Container::value_type& y, const Initializers&... initializers) { + stl_container_at(values, 5) = x; + stl_container_at(other, 6) = stl_container_at(values, 5); + VERIFY_IS_APPROX(stl_container_at(other, 6), stl_container_at(values, 5)); + values = other; + for (int i = 0; i < 20; ++i) VERIFY_IS_APPROX(stl_container_at(other, i), stl_container_at(values, i)); + + values.resize(21, initializers...); + stl_container_at(values, 20) = x; + VERIFY_IS_APPROX(stl_container_at(values, 20), x); + values.resize(22, y); + VERIFY_IS_APPROX(stl_container_at(values, 21), y); + values.push_back(x); + VERIFY_IS_APPROX(stl_container_at(values, 22), x); + verify_stl_contiguity(values); + + // The old loop compared an address in the unchanged source, so it always appended 300 entries. + for (int i = 0; i < 300; ++i) values.push_back(stl_container_at(other, i % other.size())); + for (std::size_t i = 23; i < values.size(); ++i) { + VERIFY(stl_container_coeffs(stl_container_at(values, i)) == + stl_container_coeffs(stl_container_at(other, (i - 23) % other.size()))); + } +} + +#endif // EIGEN_STL_CONTAINER_TEST_HELPERS_H
diff --git a/test/stl_iterators.cpp b/test/stl_iterators.cpp index c560020..406fc79 100644 --- a/test/stl_iterators.cpp +++ b/test/stl_iterators.cpp
@@ -147,9 +147,9 @@ void test_strided_iterator_order(Index stride) { using VectorType = Matrix<int, Dynamic, 1>; using MappedVector = Matrix<int, Options == RowMajor ? 1 : Dynamic, Options == RowMajor ? Dynamic : 1, Options>; - VectorType storage = VectorType::Random(20); - // Keep both forward and reverse end pointers within the allocation, including with negative strides. - Map<MappedVector, Unaligned, InnerStride<Stride>> map(storage.data() + 9, 4, InnerStride<Stride>(stride)); + VectorType storage = VectorType::Random(3 * numext::abs(stride) + 1); + Map<MappedVector, Unaligned, InnerStride<Stride>> map(storage.data() + (stride < 0 ? storage.size() - 1 : 0), 4, + InnerStride<Stride>(stride)); const auto& constMap = map; check_begin_end_for_loop(map); check_iterator_order(map.begin(), map.begin(), map.size()); @@ -201,6 +201,65 @@ check_iterator_const_conversion(xpr.rbegin(), xpr.rend(), xpr.crbegin()); } +template <typename Xpr> +void check_iterator_endpoints(Xpr& xpr) { + check_begin_end_for_loop(xpr); + check_iterator_const_conversion(xpr.begin(), xpr.end(), xpr.cbegin()); + auto it = xpr.begin(); + it += xpr.size(); + VERIFY(it == xpr.end()); + it -= xpr.size(); + VERIFY(it == xpr.begin()); + for (Index i = 0; i < xpr.size(); ++i) { + VERIFY(it.operator->() == &xpr[i]); + VERIFY_IS_EQUAL(*it++, xpr[i]); + } + VERIFY(it == xpr.end()); + for (Index i = xpr.size(); i > 0; --i) { + VERIFY_IS_EQUAL(*--it, xpr[i - 1]); + VERIFY_IS_EQUAL(xpr.end()[-i], xpr[xpr.size() - i]); + } + VERIFY(it == xpr.begin()); + auto reverse = make_reverse_iterator(xpr.end()); + for (Index i = xpr.size(); i > 0; --i) VERIFY_IS_EQUAL(*reverse++, xpr[i - 1]); + VERIFY(reverse == make_reverse_iterator(xpr.begin())); +} + +void test_strided_iterator_endpoints() { + // A singleton needs no padding, regardless of the stride. Forming its end pointer can overflow. + int value = 42; + const Index largeStride = (std::numeric_limits<Index>::max)(); + for (Index stride : {Index(2), Index(-2), largeStride, -largeStride}) { + Map<VectorXi, Unaligned, InnerStride<Dynamic>> singleton(&value, 1, InnerStride<Dynamic>(stride)); + check_iterator_endpoints(singleton); + Map<VectorXi, Unaligned, InnerStride<Dynamic>> empty(nullptr, 0, InnerStride<Dynamic>(stride)); + check_iterator_endpoints(empty); + } + + int storage[] = {0, 1, 2, 3, 4}; + Map<VectorXi, Unaligned, InnerStride<2>> map(storage, 3); + check_iterator_endpoints(map); + for (int& coefficient : map) coefficient = 7; + for (Index i = 0; i < 5; ++i) VERIFY_IS_EQUAL(storage[i], i % 2 == 0 ? 7 : int(i)); + Map<VectorXi, Unaligned, InnerStride<-2>> reverseMap(storage + 4, 3); + check_iterator_endpoints(reverseMap); + + Matrix<int, 3, 4, ColMajor> columns = Matrix<int, 3, 4, ColMajor>::Random(); + Matrix<int, 3, 4, RowMajor> rows = columns; + for (Index i = 0; i < columns.rows(); ++i) { + auto row = columns.row(i); + check_iterator_endpoints(row); + auto it = columns.row(i).begin(); + VERIFY_IS_EQUAL(*it, columns(i, 0)); + } + for (Index j = 0; j < rows.cols(); ++j) { + auto column = rows.col(j); + check_iterator_endpoints(column); + auto it = rows.col(j).begin(); + VERIFY_IS_EQUAL(*it, rows(0, j)); + } +} + template <typename Scalar, int Rows, int Cols> void test_stl_iterators(int rows = Rows, int cols = Cols) { typedef Matrix<Scalar, Rows, 1> VectorType; @@ -713,6 +772,13 @@ static_assert(std::ranges::contiguous_range<DynVectorType>); static_assert(std::ranges::contiguous_range<const DynVectorType>); + using StridedVector = Map<DynVectorType, Unaligned, InnerStride<2>>; + using RuntimeStridedVector = Map<DynVectorType, Unaligned, InnerStride<Dynamic>>; + static_assert(std::random_access_iterator<typename StridedVector::iterator>); + static_assert(std::random_access_iterator<typename RuntimeStridedVector::iterator>); + static_assert(!std::contiguous_iterator<typename StridedVector::iterator>); + static_assert(!std::contiguous_iterator<typename RuntimeStridedVector::iterator>); + // Sized range. static_assert(std::ranges::sized_range<VectorType>); static_assert(std::ranges::sized_range<DynVectorType>); @@ -725,6 +791,10 @@ // ranges::data returns the same pointer as data(). VERIFY(std::ranges::data(v) == v.data()); VERIFY(static_cast<Index>(std::ranges::size(v)) == v.size()); + VERIFY(std::to_address(v.end()) == v.data() + v.size()); + DynVectorType empty; + VERIFY(std::to_address(empty.begin()) == empty.data()); + VERIFY(std::to_address(empty.end()) == empty.data()); // Column expression of a column-major matrix is contiguous. ColMatrixType A = ColMatrixType::Random(rows, cols); @@ -736,6 +806,7 @@ #endif EIGEN_DECLARE_TEST(stl_iterators) { + CALL_SUBTEST_1(test_strided_iterator_endpoints()); for (Index stride : {-2, -1, 1, 2}) { CALL_SUBTEST_1((test_strided_iterator_order<Dynamic, ColMajor>(stride))); CALL_SUBTEST_1((test_strided_iterator_order<Dynamic, RowMajor>(stride)));
diff --git a/test/svd_common.h b/test/svd_common.h index f1e6a6f..e737a01 100644 --- a/test/svd_common.h +++ b/test/svd_common.h
@@ -253,8 +253,10 @@ MatrixType2 m2(rank, cols); m2.setRandom(); if (SVD_FOR_MIN_NORM(MatrixType2)(m2).setThreshold(test_precision<Scalar>()).rank() != rank) { - // Ensure full row rank by making the leading square block diagonally dominant. - for (Index i = 0; i < rank; ++i) m2(i, i) += Scalar(1); + // With m2 = [B R] and s = 2 |m2|_F, sigma_min(m2 + s [I 0]) >= sigma_min(B + s I) >= s - |B|_2 >= |m2|_F (Weyl) + // and |m2 + s [I 0]|_2 <= 3 |m2|_F, so cond(m2) <= 3. + const Scalar shift = Scalar(2 * m2.norm()); + m2.diagonal().array() += shift; } RhsType2 rhs2 = RhsType2::Random(rank); @@ -277,7 +279,11 @@ typedef Matrix<Scalar, RowsAtCompileTime3, 1> RhsType3; Index rows3 = ColsAtCompileTime == Dynamic ? internal::random<Index>(rank + 1, 2 * cols) : Index(ColsAtCompileTime + 1); - Matrix<Scalar, RowsAtCompileTime3, Dynamic> C = Matrix<Scalar, RowsAtCompileTime3, Dynamic>::Random(rows3, rank); + // Orthonormal columns give m3 the nonzero singular values of m2, so sigma_0 / sigma_{rank-1} = cond(m2) stays far + // below 1/threshold() = 1/(diagSize * eps), and solve() keeps every direction of m2. + using MatrixTypeC = Matrix<Scalar, RowsAtCompileTime3, Dynamic>; + const MatrixTypeC C = + HouseholderQR<MatrixTypeC>(MatrixTypeC::Random(rows3, rank)).householderQ() * MatrixTypeC::Identity(rows3, rank); MatrixType3 m3 = C * m2; RhsType3 rhs3 = C * rhs2; SVD_STATIC_OPTIONS(MatrixType3, Options) svd3(m3);
diff --git a/test/svd_lapacke.cpp b/test/svd_lapacke.cpp new file mode 100644 index 0000000..0e168df --- /dev/null +++ b/test/svd_lapacke.cpp
@@ -0,0 +1,156 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#define EIGEN_USE_LAPACKE +#include "main.h" +#include <Eigen/SVD> + +namespace { +// The plain copy of the matrix being decomposed, of the scalar type under test. +const void* expected_a = nullptr; +int svd_calls = 0; +bool args_match = false; + +// Columns of U, or rows of V^T, that a LAPACK job references; 0 when it references none. +lapack_int job_extent(char job, lapack_int full, lapack_int thin) { return job == 'A' ? full : job == 'S' ? thin : 0; } + +// Minimum leading dimension of a rows x cols array, or 1 for an array the job does not reference. +lapack_int min_ld(bool col_major, lapack_int rows, lapack_int cols) { + return rows == 0 || cols == 0 ? 1 : col_major ? rows : cols; +} + +// Whether the arguments describe the matrix being decomposed: a holds it in the given layout, and u and vt have +// leading dimensions the LAPACKE contract allows. +template <typename Scalar> +bool matches_expected(int layout, char jobu, char jobvt, lapack_int m, lapack_int n, const Scalar* a, lapack_int lda, + lapack_int ldu, lapack_int ldvt) { + const auto& expected = *static_cast<const Matrix<Scalar, Dynamic, Dynamic>*>(expected_a); + const bool col_major = layout == LAPACK_COL_MAJOR; + const lapack_int k = numext::mini(m, n); + // The binding decomposes a fresh compact copy. Requiring its exact leading dimension also keeps the reads below + // inside the copy. + if (m != expected.rows() || n != expected.cols() || lda != numext::maxi<lapack_int>(1, col_major ? m : n)) { + return false; + } + if (ldu < min_ld(col_major, m, job_extent(jobu, m, k)) || ldvt < min_ld(col_major, job_extent(jobvt, n, k), n)) { + return false; + } + for (lapack_int i = 0; i < m; ++i) { + for (lapack_int j = 0; j < n; ++j) { + if (a[col_major ? i + j * lda : i * lda + j] != expected(i, j)) return false; + } + } + return true; +} + +// Probes return a failure code, so Eigen reads no output but the singular values, which it checks for finiteness. +template <typename Scalar, typename RealScalar> +lapack_int svd_probe(int layout, char jobu, char jobvt, lapack_int m, lapack_int n, const Scalar* a, lapack_int lda, + RealScalar* s, lapack_int ldu, lapack_int ldvt) { + ++svd_calls; + args_match = expected_a != nullptr && matches_expected(layout, jobu, jobvt, m, n, a, lda, ldu, ldvt); + std::fill_n(s, numext::mini(m, n), RealScalar(0)); + return 1; +} +} // namespace + +// The bindings take the address of every scalar type's routine, so an unoptimized build needs all of them. +// TYPE and RTYPE form declarations, not expressions. +// NOLINTBEGIN(bugprone-macro-parentheses) +#define EIGEN_SVD_PROBES(PREFIX, TYPE, RTYPE) \ + extern "C" lapack_int LAPACKE_##PREFIX##gesvd(int layout, char jobu, char jobvt, lapack_int m, lapack_int n, \ + TYPE* a, lapack_int lda, RTYPE* s, TYPE*, lapack_int ldu, TYPE*, \ + lapack_int ldvt, RTYPE*) { \ + return svd_probe(layout, jobu, jobvt, m, n, a, lda, s, ldu, ldvt); \ + } \ + extern "C" lapack_int LAPACKE_##PREFIX##gesdd(int layout, char jobz, lapack_int m, lapack_int n, TYPE* a, \ + lapack_int lda, RTYPE* s, TYPE*, lapack_int ldu, TYPE*, \ + lapack_int ldvt) { \ + return svd_probe(layout, jobz, jobz, m, n, a, lda, s, ldu, ldvt); \ + } +// NOLINTEND(bugprone-macro-parentheses) +EIGEN_SVD_PROBES(s, float, float) +EIGEN_SVD_PROBES(d, double, double) +EIGEN_SVD_PROBES(c, lapack_complex_float, float) +EIGEN_SVD_PROBES(z, lapack_complex_double, double) +#undef EIGEN_SVD_PROBES + +template <typename SVD, typename Input> +void check_lapacke_input(const Input& input) { + const Matrix<typename SVD::Scalar, Dynamic, Dynamic> expected = input; + expected_a = &expected; + args_match = false; + const int calls_before = svd_calls; + SVD svd(input); + expected_a = nullptr; + VERIFY_IS_EQUAL(svd_calls, calls_before + 1); + VERIFY(args_match); + VERIFY_IS_EQUAL(svd.info(), NoConvergence); +} + +// LAPACK rejects the zero leading dimensions of an empty input, and given valid ones it leaves U and VT unwritten. The +// bindings must not call it, and must return the native path's identity unitaries. +template <typename SVD> +void check_lapacke_empty(Index rows, Index cols) { + using MatrixType = typename SVD::MatrixType; + const int calls_before = svd_calls; + SVD svd(MatrixType(rows, cols)); + VERIFY_IS_EQUAL(svd_calls, calls_before); + VERIFY_IS_EQUAL(svd.info(), Success); + VERIFY_IS_EQUAL(svd.singularValues().size(), 0); + if (svd.computeU()) VERIFY_IS_EQUAL(svd.matrixU(), MatrixType::Identity(rows, svd.matrixU().cols())); + if (svd.computeV()) VERIFY_IS_EQUAL(svd.matrixV(), MatrixType::Identity(cols, svd.matrixV().cols())); +} + +// The bindings hand LAPACK a copy in MatrixType's layout (#3217): an input whose own layout or outer stride differs +// must not leak into the arguments that describe the copy. +template <typename SVD> +void svd_lapacke_input_layout() { + using Scalar = typename SVD::Scalar; + using ColMatrix = Matrix<Scalar, Dynamic, Dynamic>; + using RowMatrix = Matrix<Scalar, Dynamic, Dynamic, RowMajor>; + const Index m = internal::random<Index>(4, 12), n = internal::random<Index>(1, m - 1); + const ColMatrix tall = ColMatrix::Random(m, n), wide = ColMatrix::Random(n, m), square = ColMatrix::Random(m, m); + const ColMatrix big = ColMatrix::Random(m + 3, m + 2); + const RowMatrix row_major = tall, big_row_major = big; + check_lapacke_input<SVD>(tall); + check_lapacke_input<SVD>(row_major); + check_lapacke_input<SVD>(tall.transpose()); + check_lapacke_input<SVD>(wide.transpose()); + check_lapacke_input<SVD>(square.transpose()); + check_lapacke_input<SVD>(big.block(1, 2, m, n)); + check_lapacke_input<SVD>(big.block(1, 2, n, m).transpose()); + check_lapacke_input<SVD>(big_row_major.block(1, 2, m, n)); + check_lapacke_input<SVD>(Scalar(2) * tall); + check_lapacke_empty<SVD>(0, 0); + check_lapacke_empty<SVD>(0, n); + check_lapacke_empty<SVD>(m, 0); +} + +template <typename Scalar, int Options> +void svd_lapacke_options() { + using ColMatrix = Matrix<Scalar, Dynamic, Dynamic>; + using RowMatrix = Matrix<Scalar, Dynamic, Dynamic, RowMajor>; + svd_lapacke_input_layout<JacobiSVD<ColMatrix, Options>>(); + svd_lapacke_input_layout<JacobiSVD<RowMatrix, Options>>(); + svd_lapacke_input_layout<BDCSVD<ColMatrix, Options>>(); + svd_lapacke_input_layout<BDCSVD<RowMatrix, Options>>(); +} + +// ThinU | FullV is the case where the BDCSVD binding computes U in a local buffer. +template <typename Scalar> +void svd_lapacke() { + svd_lapacke_options<Scalar, 0>(); + svd_lapacke_options<Scalar, ComputeThinU | ComputeThinV>(); + svd_lapacke_options<Scalar, ComputeFullU | ComputeFullV>(); + svd_lapacke_options<Scalar, ComputeThinU | ComputeFullV>(); +} + +EIGEN_DECLARE_TEST(svd_lapacke) { + for (int i = 0; i < g_repeat; i++) { + CALL_SUBTEST_1(svd_lapacke<double>()); + CALL_SUBTEST_2(svd_lapacke<float>()); + CALL_SUBTEST_3(svd_lapacke<std::complex<double>>()); + CALL_SUBTEST_4(svd_lapacke<std::complex<float>>()); + } +}
diff --git a/test/sycl_basic.cpp b/test/sycl_basic.cpp index ad9ba04..577bba3 100644 --- a/test/sycl_basic.cpp +++ b/test/sycl_basic.cpp
@@ -351,7 +351,7 @@ void test_predux_any_float() { auto operation = [](const float* in, int* out) { -#ifdef SYCL_DEVICE_ONLY +#ifdef EIGEN_SYCL_DEVICE_ONLY const cl::sycl::cl_int4 int_none(0); const cl::sycl::cl_int4 int_some(0, -1, 0, 0); const cl::sycl::cl_int4 int_all(-1); @@ -370,7 +370,7 @@ void test_predux_any_half() { auto operation = [](const float* in, int* out) { -#ifdef SYCL_DEVICE_ONLY +#ifdef EIGEN_SYCL_DEVICE_ONLY const cl::sycl::cl_short8 short_none(0); const cl::sycl::cl_short8 short_some(0, -1, 0, 0, 0, 0, 0, 0); const cl::sycl::cl_short8 short_all(-1); @@ -389,7 +389,7 @@ void test_predux_any_double() { auto operation = [](const float* in, int* out) { -#ifdef SYCL_DEVICE_ONLY +#ifdef EIGEN_SYCL_DEVICE_ONLY const cl::sycl::cl_long2 long_none(0); const cl::sycl::cl_long2 long_some(0, -1); const cl::sycl::cl_long2 long_all(-1);
diff --git a/test/sycl_include_order.cpp b/test/sycl_include_order.cpp new file mode 100644 index 0000000..34ef38f --- /dev/null +++ b/test/sycl_include_order.cpp
@@ -0,0 +1,105 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +// EIGEN_SUFFIXES;1;2 + +// Core use inside a SYCL kernel does not require EIGEN_USE_SYCL. +#if defined(EIGEN_TEST_PART_2) +#include <Eigen/Core> +#endif +#include <CL/sycl.hpp> +#include <Eigen/Core> +#include <contrib/Eigen/SpecialFunctions> + +template <typename Scalar> +bool sycl_first_vector_addition() { + using Vector = Eigen::Matrix<Scalar, 4, 1>; + Vector a, b, result, expected; + a << 1, 2, 3, 4; + b << 5, 6, 7, 8; + expected << 6, 8, 10, 12; + cl::sycl::queue queue; + { + cl::sycl::buffer<Scalar, 1> input_a(a.data(), cl::sycl::range<1>(4)); + cl::sycl::buffer<Scalar, 1> input_b(b.data(), cl::sycl::range<1>(4)); + cl::sycl::buffer<Scalar, 1> output(result.data(), cl::sycl::range<1>(4)); + queue.submit([&](cl::sycl::handler& handler) { + auto x = input_a.template get_access<cl::sycl::access::mode::read>(handler); + auto y = input_b.template get_access<cl::sycl::access::mode::read>(handler); + auto z = output.template get_access<cl::sycl::access::mode::discard_write>(handler); + handler.single_task([=]() { + Eigen::Map<const Vector> a_map(&x[0]), b_map(&y[0]); + Eigen::Map<Vector> result_map(&z[0]); + result_map = a_map + b_map; + }); + }); + queue.wait_and_throw(); + } + return result == expected; +} + +template <typename Scalar> +bool sycl_scalar_math() { + using Vector = Eigen::Matrix<Scalar, 4, 1>; + Vector input, result_sin, result_lgamma; + input << 1, 2, 3, 4; + cl::sycl::queue queue; + { + cl::sycl::buffer<Scalar, 1> input_buffer(input.data(), cl::sycl::range<1>(4)); + cl::sycl::buffer<Scalar, 1> output_sin(result_sin.data(), cl::sycl::range<1>(4)); + cl::sycl::buffer<Scalar, 1> output_lgamma(result_lgamma.data(), cl::sycl::range<1>(4)); + queue.submit([&](cl::sycl::handler& handler) { + auto x = input_buffer.template get_access<cl::sycl::access::mode::read>(handler); + auto y = output_sin.template get_access<cl::sycl::access::mode::discard_write>(handler); + auto z = output_lgamma.template get_access<cl::sycl::access::mode::discard_write>(handler); + handler.single_task([=]() { + Eigen::Map<const Vector> input_map(&x[0]); + Eigen::Map<Vector> sin_map(&y[0]); + sin_map = input_map.array().sin().matrix(); + for (Eigen::Index i = 0; i < 4; ++i) z[i] = Eigen::numext::lgamma(input_map[i]); + }); + }); + queue.wait_and_throw(); + } + const Scalar tolerance = Scalar(32) * Eigen::NumTraits<Scalar>::epsilon(); + for (Eigen::Index i = 0; i < 4; ++i) { + const Scalar expected_sin = std::sin(input[i]); + const Scalar expected_lgamma = std::lgamma(input[i]); + if (!(std::abs(result_sin[i] - expected_sin) <= tolerance * (Scalar(1) + std::abs(expected_sin)))) return false; + if (!(std::abs(result_lgamma[i] - expected_lgamma) <= tolerance * (Scalar(1) + std::abs(expected_lgamma)))) + return false; + } + return true; +} + +template <typename Scalar> +bool sycl_low_precision_fma() { + float input[] = {1.5f, 2.f, 0.5f}, result = 0; + cl::sycl::queue queue; + { + cl::sycl::buffer<float, 1> inputs(input, cl::sycl::range<1>(3)); + cl::sycl::buffer<float, 1> output(&result, cl::sycl::range<1>(1)); + queue.submit([&](cl::sycl::handler& handler) { + auto x = inputs.template get_access<cl::sycl::access::mode::read>(handler); + auto y = output.template get_access<cl::sycl::access::mode::discard_write>(handler); + handler.single_task([=]() { y[0] = float(Eigen::numext::fma(Scalar(x[0]), Scalar(x[1]), Scalar(x[2]))); }); + }); + queue.wait_and_throw(); + } + const float expected = + float(Scalar(std::fma(float(Scalar(input[0])), float(Scalar(input[1])), float(Scalar(input[2]))))); + return result == expected; +} + +// The shared test driver explicitly enables CUDA when __CUDACC__ is defined; +// this test must leave backend selection to Eigen, as a user translation unit does. +int main() { + if (!sycl_first_vector_addition<float>() || !sycl_scalar_math<float>() || !sycl_low_precision_fma<Eigen::half>() || + !sycl_low_precision_fma<Eigen::bfloat16>()) + return 1; + cl::sycl::queue queue; + if (queue.get_device().has(cl::sycl::aspect::fp64) && + (!sycl_first_vector_addition<double>() || !sycl_scalar_math<double>())) + return 1; + return 0; +}
diff --git a/test/sycl_macro_collision.cpp b/test/sycl_macro_collision.cpp new file mode 100644 index 0000000..ffe1fc2 --- /dev/null +++ b/test/sycl_macro_collision.cpp
@@ -0,0 +1,69 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +// EIGEN_SUFFIXES;1;2 + +#if defined(EIGEN_TEST_PART_2) +// Older DPC++ NVPTX passes define __CUDA_ARCH__ without CUDA language mode. +#define __SYCL_DEVICE_ONLY__ // NOLINT(bugprone-reserved-identifier): simulate the compiler's device pass. +#define __CUDA_ARCH__ 500 // NOLINT(bugprone-reserved-identifier): simulate older DPC++ NVPTX targets. +#ifndef EIGEN_NO_ASSERTION_CHECKING +#define EIGEN_NO_ASSERTION_CHECKING +#endif +#endif + +// AdaptiveCpp defines this flag even without Eigen's explicit SYCL integration. +#define SYCL_DEVICE_ONLY +#include "main.h" +#include "fp_control.h" +#if !defined(EIGEN_TEST_PART_2) +#include <contrib/Eigen/Tensor> +#endif + +#ifdef EIGEN_SYCL_DEVICE_ONLY +#error Eigen SYCL integration requires EIGEN_USE_SYCL +#endif +#ifndef SYCL_DEVICE_ONLY +#error Eigen must preserve externally defined macros +#endif + +#if defined(EIGEN_TEST_PART_2) +#if defined(EIGEN_CUDA_ARCH) || defined(EIGEN_GPU_COMPILE_PHASE) +#error SYCL device compilation alone must not select CUDA intrinsic paths +#endif +// Keep stringification nonempty for the MSVC traditional preprocessor. +static_assert(sizeof(EIGEN_MAKESTRING((EIGEN_DEVICE_FUNC))) == 3, + "Implicit SYCL use must not enable Eigen's flattening policy"); +#endif + +EIGEN_DECLARE_TEST(sycl_macro_collision) { +#if !defined(EIGEN_TEST_PART_2) && EIGEN_ARCH_x86_64 + ScopedFlushToZero fp_mode(FlushToZeroMode::None); + VERIFY(fp_mode.isSupported()); +#endif +#if defined(EIGEN_TEST_PART_2) +#if EIGEN_COMP_GNUC + VERIFY_IS_EQUAL(std::string(EIGEN_MAKESTRING(EIGEN_ALWAYS_INLINE)), + std::string("__attribute__((always_inline)) inline")); + VERIFY_IS_EQUAL(std::string(EIGEN_MAKESTRING(EIGEN_LAMBDA_ALWAYS_INLINE)), + std::string("__attribute__((always_inline))")); +#endif + const half half_a(33.f / 32), half_b(65.f / 64), half_c((std::numeric_limits<half>::denorm_min)()); + VERIFY_IS_EQUAL(float(numext::fma(half_a, half_b, half_c)), + float(half(numext::fma(float(half_a), float(half_b), float(half_c))))); +#endif + Vector4d a, b; + a << 1, 2, 3, 4; + b << 5, 6, 7, 8; + Vector4d expected; + expected << 6, 8, 10, 12; + VERIFY_IS_EQUAL((a + b).eval(), expected); + VERIFY_IS_EQUAL(numext::mini(2, 3), 2); + VERIFY_IS_EQUAL(numext::maxi(2.0, 3.0), 3.0); +#if !defined(EIGEN_TEST_PART_2) + Tensor<float, 1> tensor(3), result(3); + tensor.setConstant(2.0f); + result = tensor + tensor; + for (Index i = 0; i < result.size(); ++i) VERIFY_IS_EQUAL(result(i), 4.0f); +#endif +}
diff --git a/test/sycl_math.cpp b/test/sycl_math.cpp new file mode 100644 index 0000000..5854ab8 --- /dev/null +++ b/test/sycl_math.cpp
@@ -0,0 +1,122 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#define EIGEN_USE_SYCL +#define EIGEN_TEST_NO_LONGDOUBLE +#include "main.h" +#include <Eigen/Geometry> + +template <typename Scalar> +void sycl_scalar_math(sycl::queue& queue) { + constexpr int kOutputs = 38; + std::vector<Scalar> result(4 * kOutputs); + { + sycl::buffer<Scalar, 1> buffer(result.data(), sycl::range<1>(result.size())); + queue.submit([&](sycl::handler& handler) { + auto output = buffer.template get_access<sycl::access::mode::write>(handler); + handler.parallel_for(sycl::range<1>(4), [=](sycl::id<1> id) { + const Scalar x = Scalar(id[0] + 1) / Scalar(4); + const int offset = int(id[0]) * kOutputs; + output[offset + 0] = internal::psin(x); + output[offset + 1] = internal::pcos(x); + output[offset + 2] = internal::ptan(x); + output[offset + 3] = internal::pasin(x); + output[offset + 4] = internal::pacos(x); + output[offset + 5] = internal::patan(x); + output[offset + 6] = internal::psinh(x); + output[offset + 7] = internal::pcosh(x); + output[offset + 8] = internal::ptanh(x); + output[offset + 9] = internal::pasinh(x); + output[offset + 10] = internal::pacosh(x + Scalar(1)); + output[offset + 11] = internal::patanh(x / Scalar(2)); + output[offset + 12] = numext::log(x); + output[offset + 13] = numext::log2(x); + output[offset + 14] = numext::log10(x); + output[offset + 15] = numext::exp(x); + output[offset + 16] = Array<Scalar, 1, 1>::Constant(x).pow(Scalar(1.5))(0); + output[offset + 17] = numext::cbrt(x); + output[offset + 18] = numext::sqrt(x); + output[offset + 19] = numext::hypot(x, Scalar(2)); + output[offset + 20] = numext::atan2(x, Scalar(2)); + output[offset + 21] = numext::copysign(x, Scalar(-1)); + output[offset + 22] = numext::rint(x + Scalar(0.25)); + const Matrix<Scalar, 3, 1> vector(Scalar(1), x, x / Scalar(2)); + output[offset + 23] = vector.template lpNorm<3>(); + output[offset + 24] = vector.norm(); + const Quaternion<Scalar> rotation(AngleAxis<Scalar>(x, Matrix<Scalar, 3, 1>::UnitZ())); + output[offset + 25] = rotation.angularDistance(Quaternion<Scalar>::Identity()); + output[offset + 26] = + rotation.slerp(Scalar(0.5), Quaternion<Scalar>::Identity()).angularDistance(Quaternion<Scalar>::Identity()); + output[offset + 27] = Rotation2D<Scalar>(x).toRotationMatrix()(0, 1); + JacobiRotation<Scalar> jacobi; + jacobi.makeJacobi(Scalar(2), x, Scalar(3)); + output[offset + 28] = numext::abs2(jacobi.c()) + numext::abs2(jacobi.s()); + output[offset + 29] = Rotation2D<Scalar>(Rotation2D<Scalar>(x).toRotationMatrix()).angle(); + output[offset + 30] = numext::copysign(Scalar(0), Scalar(-1)); + output[offset + 31] = numext::atan2(Scalar(-0.0), Scalar(-1)); + output[offset + 32] = Scalar(float(numext::tanh(half(float(x))))); + output[offset + 33] = Scalar(float(numext::tanh(bfloat16(float(x))))); + output[offset + 34] = Scalar(float(numext::sqrt(half(float(x))))); + output[offset + 35] = Scalar(float(numext::sqrt(bfloat16(float(x))))); + output[offset + 36] = Scalar(float(numext::log2(half(float(x))))); + output[offset + 37] = Scalar(float(numext::log2(bfloat16(float(x))))); + }); + }); + queue.wait_and_throw(); + } + for (int i = 0; i < 4; ++i) { + const Scalar x = Scalar(i + 1) / Scalar(4); + const Scalar expected[] = {std::sin(x), + std::cos(x), + std::tan(x), + std::asin(x), + std::acos(x), + std::atan(x), + std::sinh(x), + std::cosh(x), + std::tanh(x), + std::asinh(x), + std::acosh(x + Scalar(1)), + std::atanh(x / Scalar(2)), + std::log(x), + std::log2(x), + std::log10(x), + std::exp(x), + std::pow(x, Scalar(1.5)), + std::cbrt(x), + std::sqrt(x), + std::hypot(x, Scalar(2)), + std::atan2(x, Scalar(2)), + -x, + std::rint(x + Scalar(0.25)), + std::pow(Scalar(1) + x * x * x * Scalar(1.125), Scalar(1) / Scalar(3)), + std::sqrt(Scalar(1) + x * x * Scalar(1.25)), + x, + x / Scalar(2), + -std::sin(x), + Scalar(1), + x}; + // Each composed rotation/norm uses O(10) operations and a few elementary functions. + for (int j = 0; j < 30; ++j) { + const Scalar tolerance = + Scalar(128) * NumTraits<Scalar>::epsilon() * (std::max)(Scalar(1), std::abs(expected[j])); + VERIFY(std::abs(result[i * kOutputs + j] - expected[j]) <= tolerance); + } + VERIFY_IS_EQUAL(result[i * kOutputs + 30], Scalar(0)); + VERIFY((numext::signbit)(result[i * kOutputs + 30])); + VERIFY(result[i * kOutputs + 31] < Scalar(0)); + VERIFY(std::abs(result[i * kOutputs + 31] + Scalar(EIGEN_PI)) <= Scalar(4) * NumTraits<Scalar>::epsilon()); + VERIFY_IS_EQUAL(result[i * kOutputs + 32], Scalar(float(half(std::tanh(float(x)))))); + VERIFY_IS_EQUAL(result[i * kOutputs + 33], Scalar(float(bfloat16(std::tanh(float(x)))))); + VERIFY_IS_EQUAL(result[i * kOutputs + 34], Scalar(float(half(std::sqrt(float(x)))))); + VERIFY_IS_EQUAL(result[i * kOutputs + 35], Scalar(float(bfloat16(std::sqrt(float(x)))))); + VERIFY_IS_EQUAL(result[i * kOutputs + 36], Scalar(float(half(std::log2(float(x)))))); + VERIFY_IS_EQUAL(result[i * kOutputs + 37], Scalar(float(bfloat16(std::log2(float(x)))))); + } +} + +EIGEN_DECLARE_TEST(sycl_math) { + sycl::queue queue; + CALL_SUBTEST(sycl_scalar_math<float>(queue)); + if (queue.get_device().has(sycl::aspect::fp64)) CALL_SUBTEST(sycl_scalar_math<double>(queue)); +}
diff --git a/test/trig_reduction_fastmath.cpp b/test/trig_reduction_fastmath.cpp new file mode 100644 index 0000000..036b416 --- /dev/null +++ b/test/trig_reduction_fastmath.cpp
@@ -0,0 +1,89 @@ +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 + +#include "packetmath_test_shared.h" + +void check_trig_reduction() { + // MPFR 4.2.1, 256-bit precision, rounded to binary64. Include both sides of the medium/large reduction cutoffs. + const double cases[][4] = { + {1.5707963267948963, 1, 2.8327694488239898e-16, 3530114321217157.5}, + {1.5707963267948966, 1, 6.123233995736766e-17, 16331239353195370}, + {1.5707963267948968, 1, -1.6081226496766366e-16, -6218431163823738}, + {3.1415926535897927, 5.6655388976479796e-16, -1, -5.6655388976479796e-16}, + {3.1415926535897931, 1.2246467991473532e-16, -1, -1.2246467991473532e-16}, + {3.1415926535897936, -3.2162452993532732e-16, -1, 3.2162452993532732e-16}, + {14.137166941154067, 1, 2.3274478990165594e-15, 429655160239049.94}, + {14.137166941154069, 1, 5.5109105961630896e-16, 1814582150355041}, + {14.137166941154071, 1, -1.2252657797839415e-15, -816149456305174.88}, + {0.29999999999999999, 0.29552020666133955, 0.95533648912560598, 0.30933624960962325}, + {4.7123889803846897, -1, -1.8369701987210297e-16, 5443746451065123}, + {14.999999999999998, 0.65028784015711827, -0.75968791285882009, -0.85599340090852183}, + {15, 0.65028784015711683, -0.75968791285882131, -0.85599340090851872}, + {15.000000000000002, 0.65028784015711549, -0.75968791285882242, -0.85599340090851572}, + {20, 0.91294525072762767, 0.40808206181339196, 2.2371609442247422}, + {100, -0.50636564110975879, 0.86231887228768389, -0.58721391515692911}, + {123.456, -0.80393736857282394, -0.5947139710921574, 1.3518050821917635}, + {1000, 0.82687954053200252, 0.56237907629070294, 1.4703241557027185}, + {12345.678900000001, -0.70344192126325633, 0.71075274421521484, -0.98971397154430851}, + {123456.789, -0.99866408234322457, 0.051672532718701383, -19.326787943215852}, + {1000000, -0.34999350217129294, 0.93675212753314474, -0.37362445398759903}, + {10000000, 0.4205477931907825, -0.90727038618173961, -0.46353082785018906}, + {100000000, 0.93163902710972601, -0.36338508935569053, -2.5637789067283774}, + {1000000000, 0.54584344944869956, 0.83788718136390239, 0.65145220214514132}, + {10000000000, -0.48750602508751067, 0.87311962267685606, -0.55834963781124181}, + {100000000000, 0.92869366049659197, 0.37084779216471114, 2.5042448144982212}, + {1000000000000, -0.61123870237688949, 0.79144630185289022, -0.77230596813187613}, + {6907362400000, 0.5754525274479354, 0.81783518428457402, 0.70362896889955862}, + {10000000000000, -0.28888529481752512, 0.95736371690083999, -0.30175082856983471}, + {99999999999999.984, -0.19410480015836853, -0.9809807982603328, 0.1978680933435121}, + {100000000000000, -0.20940830749645231, -0.97782828796853249, 0.21415652428250395}, + {100000000000000.02, -0.22466069079959752, -0.97443705492425092, 0.23055433869667633}, + {1000000000000000, 0.85827279317023586, -0.51319373778697031, -1.672414782127583}, + // Medium-path inputs within 6e-19 to 9e-7 of k*pi/2, for k from 29 to 6e13. + {45.553093477052002, 1, -6.1898063658835771e-19, -1.6155594228467482e+18}, + {364.42474781641602, 4.9518450927068616e-18, 1, 4.9518450927068616e-18}, + {857879.41830474522, -1, -1.2354480663331819e-14, 80942293508783.953}, + {956274858.90980339, -1, 2.2360885370100136e-11, -44720948363.571968}, + {93293944418765.062, 9.1063643066365511e-07, 0.99999999999958533, 9.1063643066403267e-07}, + }; + const Index count = sizeof(cases) / sizeof(cases[0]); + ArrayXd x(2 * count), sin_ref(2 * count), cos_ref(2 * count), tan_ref(2 * count); + for (Index i = 0; i < count; ++i) { + x[2 * i] = cases[i][0]; + x[2 * i + 1] = -cases[i][0]; + sin_ref[2 * i] = cases[i][1]; + sin_ref[2 * i + 1] = -cases[i][1]; + cos_ref[2 * i] = cos_ref[2 * i + 1] = cases[i][2]; + tan_ref[2 * i] = cases[i][3]; + tan_ref[2 * i + 1] = -cases[i][3]; + } + for (Index offset = 0; offset < 2; ++offset) { + for (Index size = 1; size <= x.size() - offset; ++size) { + const ArrayXd sine = x.segment(offset, size).sin(); + const ArrayXd cosine = x.segment(offset, size).cos(); + const ArrayXd tangent = x.segment(offset, size).tan(); + VERIFY(test::areWithinUlps(sin_ref.data() + offset, sine.data(), static_cast<int>(size), 4)); + VERIFY(test::areWithinUlps(cos_ref.data() + offset, cosine.data(), static_cast<int>(size), 4)); + VERIFY(test::areWithinUlps(tan_ref.data() + offset, tangent.data(), static_cast<int>(size), 4)); + } + } +} + +template <typename Packet> +void check_error_free_transformations() { + const double low = std::ldexp(1.0, -54); + Packet hi, lo; + internal::twosum(internal::pset1<Packet>(low), internal::pset1<Packet>(1.0), hi, lo); + VERIFY_IS_EQUAL(internal::pfirst(hi), 1.0); + VERIFY_IS_EQUAL(internal::pfirst(lo), low); + const Packet x = internal::pset1<Packet>(1.0 + std::ldexp(1.0, -30)); + internal::twoprod<Packet, true>(x, x, hi, lo); + VERIFY_IS_EQUAL(internal::pfirst(hi), 1.0 + std::ldexp(1.0, -29)); + VERIFY_IS_EQUAL(internal::pfirst(lo), std::ldexp(1.0, -60)); +} + +EIGEN_DECLARE_TEST(trig_reduction_fastmath) { + CALL_SUBTEST(check_trig_reduction()); + CALL_SUBTEST(check_error_free_transformations<double>()); + CALL_SUBTEST(check_error_free_transformations<internal::packet_traits<double>::type>()); +}
diff --git a/test/unalignedcount.cpp b/test/unalignedcount.cpp index 1bf94d4..fabeab3 100644 --- a/test/unalignedcount.cpp +++ b/test/unalignedcount.cpp
@@ -32,23 +32,34 @@ } #include "main.h" +#include <Eigen/Core> -template <typename Scalar> +template <typename Scalar, int Options = ColMajor> void assignment_packet_boundaries() { - using VectorType = Matrix<Scalar, Dynamic, 1>; + using VectorType = Matrix<Scalar, Options == RowMajor ? 1 : Dynamic, Options == RowMajor ? Dynamic : 1, Options>; + using MapType = Map<VectorType>; + using Traits = internal::copy_using_evaluator_traits<internal::evaluator<MapType>, internal::evaluator<MapType>, + internal::assign_op<Scalar, Scalar>>; + STATIC_CHECK((!internal::packet_traits<Scalar>::Vectorizable || Traits::Traversal == LinearVectorizedTraversal)); + STATIC_CHECK(Traits::Unrolling == NoUnrolling); constexpr int packetSize = internal::packet_traits<Scalar>::size; for (Index offset = 0; offset < packetSize; ++offset) { - for (Index size = 0; size <= 2 * packetSize + 1; ++size) { - VectorType source(size), storage(size + 2 * packetSize); - for (Index i = 0; i < size; ++i) source(i) = Scalar(i + 1); + for (Index size = 0; size <= 9 * packetSize + 1; ++size) { + VectorType sourceStorage(size + 2 * packetSize), storage(size + 2 * packetSize); + MapType source(sourceStorage.data() + (packetSize - offset) % packetSize, size); + for (Index i = 0; i < size; ++i) source(i) = Scalar(i % 7 + 1); storage.setConstant(Scalar(-1)); - Map<VectorType> destination(storage.data() + offset, size); + MapType destination(storage.data() + offset, size); destination = source; - for (Index i = 0; i < size; ++i) VERIFY_IS_EQUAL(destination(i), Scalar(i + 1)); + for (Index i = 0; i < size; ++i) VERIFY_IS_EQUAL(destination(i), Scalar(i % 7 + 1)); destination += source; - for (Index i = 0; i < size; ++i) VERIFY_IS_EQUAL(destination(i), Scalar(2 * (i + 1))); + for (Index i = 0; i < size; ++i) VERIFY_IS_EQUAL(destination(i), Scalar(2 * (i % 7 + 1))); destination -= source; - for (Index i = 0; i < size; ++i) VERIFY_IS_EQUAL(destination(i), Scalar(i + 1)); + for (Index i = 0; i < size; ++i) VERIFY_IS_EQUAL(destination(i), Scalar(i % 7 + 1)); + destination.array() = source.array().square(); + for (Index i = 0; i < size; ++i) VERIFY_IS_EQUAL(destination(i), Scalar((i % 7 + 1) * (i % 7 + 1))); + destination.array() += destination.array(); + for (Index i = 0; i < size; ++i) VERIFY_IS_EQUAL(destination(i), Scalar(2 * (i % 7 + 1) * (i % 7 + 1))); for (Index i = 0; i < offset; ++i) VERIFY_IS_EQUAL(storage(i), Scalar(-1)); for (Index i = offset + size; i < storage.size(); ++i) VERIFY_IS_EQUAL(storage(i), Scalar(-1)); } @@ -58,6 +69,10 @@ EIGEN_DECLARE_TEST(unalignedcount) { CALL_SUBTEST(assignment_packet_boundaries<float>()); CALL_SUBTEST(assignment_packet_boundaries<double>()); + CALL_SUBTEST((assignment_packet_boundaries<float, RowMajor>())); + CALL_SUBTEST(assignment_packet_boundaries<int>()); + CALL_SUBTEST(assignment_packet_boundaries<std::complex<float>>()); + CALL_SUBTEST(assignment_packet_boundaries<std::complex<double>>()); #if defined(EIGEN_VECTORIZE_AVX512) VectorXf a(48), b(48); a.fill(0);
diff --git a/test/vectorization_logic.cpp b/test/vectorization_logic.cpp index 7f50af5..77f3352 100644 --- a/test/vectorization_logic.cpp +++ b/test/vectorization_logic.cpp
@@ -35,12 +35,16 @@ using internal::demangle_traversal; using internal::demangle_unrolling; +// The traits of the kernel that `Dst = Src` runs, with the source evaluator the assignment uses. +template <typename Dst, typename Src> +using assign_traits = internal::copy_using_evaluator_traits< + internal::evaluator<Dst>, typename internal::dense_assignment_source_evaluator<internal::evaluator<Dst>, Src>::type, + internal::assign_op<typename Dst::Scalar, typename Src::Scalar>>; + template <typename Dst, typename Src> bool test_assign(const Dst&, const Src&, int traversal, int unrolling) { EIGEN_STATIC_ASSERT_SAME_MATRIX_SIZE(Dst, Src); - typedef internal::copy_using_evaluator_traits<internal::evaluator<Dst>, internal::evaluator<Src>, - internal::assign_op<typename Dst::Scalar, typename Src::Scalar> > - traits; + using traits = assign_traits<Dst, Src>; // If traversal or unrolling are negative, ignore. bool res = traversal > -1 ? traits::Traversal == traversal : true; if (unrolling > -1) { @@ -67,9 +71,7 @@ template <typename Dst, typename Src> bool test_assign(int traversal, int unrolling) { EIGEN_STATIC_ASSERT_SAME_MATRIX_SIZE(Dst, Src); - typedef internal::copy_using_evaluator_traits<internal::evaluator<Dst>, internal::evaluator<Src>, - internal::assign_op<typename Dst::Scalar, typename Src::Scalar> > - traits; + using traits = assign_traits<Dst, Src>; bool res = traits::Traversal == traversal && traits::Unrolling == unrolling; if (!res) { std::cerr << "Src: " << demangle_flags(Src::Flags) << std::endl; @@ -222,6 +224,17 @@ int(Matrix11::SizeAtCompileTime) * CostP11 <= BudgetP11 ? CompleteUnrolling : int(Matrix11::InnerSizeAtCompileTime) * CostP11 <= BudgetP11 ? InnerUnrolling : NoUnrolling)); + + // An assignment evaluates a lazy product along the destination's storage order, so it vectorizes whenever the + // operand along that order is contiguous, whatever order the product expression reports. + using MatrixC = Matrix<Scalar, PacketSize, PacketSize, ColMajor>; + using MatrixR = Matrix<Scalar, PacketSize, PacketSize, RowMajor>; + VERIFY(test_assign(MatrixC(), MatrixC().lazyProduct(MatrixC()), InnerVectorizedTraversal, -1)); + VERIFY(test_assign(MatrixC(), MatrixC().lazyProduct(MatrixR()), InnerVectorizedTraversal, -1)); + VERIFY(test_assign(MatrixR(), MatrixC().lazyProduct(MatrixR()), InnerVectorizedTraversal, -1)); + VERIFY(test_assign(MatrixR(), MatrixR().lazyProduct(MatrixR()), InnerVectorizedTraversal, -1)); + // The strided column cannot vectorize the product, the contiguous row can. + VERIFY(test_assign(MatrixR(), MatrixR().col(0).lazyProduct(MatrixR().row(0)), InnerVectorizedTraversal, -1)); } VERIFY(test_redux(Vector1(), LinearVectorizedTraversal, CompleteUnrolling)); @@ -371,8 +384,10 @@ #if !EIGEN_GCC_AND_ARCH_DOESNT_WANT_STACK_ALIGNMENT VERIFY(test_assign(Vector1(), Vector1(), InnerVectorizedTraversal, CompleteUnrolling)); VERIFY(test_assign(Vector1(), Vector1() + Vector1(), InnerVectorizedTraversal, CompleteUnrolling)); + // A full-packet segment retains alignment when inferred as an inner panel. VERIFY(test_assign(Vector1(), Vector1().template segment<MinVSize>(0).derived(), - EIGEN_UNALIGNED_VECTORIZE ? InnerVectorizedTraversal : LinearVectorizedTraversal, + EIGEN_UNALIGNED_VECTORIZE || MinVSize == PacketTraits::size ? InnerVectorizedTraversal + : LinearVectorizedTraversal, CompleteUnrolling)); VERIFY(test_assign(Vector1(), Scalar(RealScalar(2.1)) * Vector1() - Vector1(), InnerVectorizedTraversal, CompleteUnrolling)); @@ -380,7 +395,9 @@ Vector1(), (Scalar(RealScalar(2.1)) * Vector1().template segment<MinVSize>(0) - Vector1().template segment<MinVSize>(0)) .derived(), - EIGEN_UNALIGNED_VECTORIZE ? InnerVectorizedTraversal : LinearVectorizedTraversal, CompleteUnrolling)); + EIGEN_UNALIGNED_VECTORIZE || MinVSize == PacketTraits::size ? InnerVectorizedTraversal + : LinearVectorizedTraversal, + CompleteUnrolling)); VERIFY(test_assign(Vector1(), Vector1().cwiseProduct(Vector1()), InnerVectorizedTraversal, CompleteUnrolling)); VERIFY(test_assign(Vector1(), Vector1().template cast<Scalar>(), InnerVectorizedTraversal, CompleteUnrolling));
diff --git a/test/zerosized.cpp b/test/zerosized.cpp index 7facccd..7b36873 100644 --- a/test/zerosized.cpp +++ b/test/zerosized.cpp
@@ -53,6 +53,8 @@ Matrix<Scalar, 2, 1> fixed; fixed << Scalar(2), Scalar(3); + struct UnusedIdentity {}; + VERIFY_IS_EQUAL(double(internal::redux_with_identity(fixed, sum, UnusedIdentity{})), 5.0); dynamic = fixed; VERIFY_IS_EQUAL(double(internal::redux_with_identity(fixed, sum, identity)), 5.0); VERIFY_IS_EQUAL(double(internal::redux_with_identity(
diff --git a/unsupported/Eigen/CMakeLists.txt b/unsupported/Eigen/CMakeLists.txt index c6620e5..0f41e5b 100644 --- a/unsupported/Eigen/CMakeLists.txt +++ b/unsupported/Eigen/CMakeLists.txt
@@ -23,6 +23,7 @@ NumericalDiff OpenGLSupport Polynomials + Riccati SparseExtra SpecialFunctions Splines
diff --git a/unsupported/Eigen/Riccati b/unsupported/Eigen/Riccati new file mode 100644 index 0000000..21733f7 --- /dev/null +++ b/unsupported/Eigen/Riccati
@@ -0,0 +1,17 @@ +// This file is part of Eigen, a lightweight C++ template library +// for linear algebra. +// +// SPDX-FileCopyrightText: The Eigen Authors +// SPDX-License-Identifier: MPL-2.0 +// +// Forwarding header: the top-level unsupported/ directory was renamed +// to contrib/ in Eigen 5.x. New code should use +// #include <contrib/Eigen/Riccati> +// directly. This shim preserves the legacy include path. + +#ifndef EIGEN_UNSUPPORTED_RICCATI_FORWARD_H +#define EIGEN_UNSUPPORTED_RICCATI_FORWARD_H + +#include "../../contrib/Eigen/Riccati" // IWYU pragma: export + +#endif // EIGEN_UNSUPPORTED_RICCATI_FORWARD_H