Achieving Performance Parity Between the oneTBB and OpenMP oneMKL Backends

Choosing between OpenMP and oneTBB when linking oneMKL is an architectural choice between coordinated work-sharing and dynamic work-stealing. OpenMP routinely wins on large, regular dense kernels like DGEMM due to coordinated cache and memory reuse, but it breaks application composability when nested in larger parallel frameworks. oneTBB provides composable worker pools across complex software stacks, but tends to lose out on high performance cooperative threading that requires barriers. oneTBB threaded oneMKL often wins on heterogeneous and smaller task granularity. For applications like sparse direct factorization this presents a dilemma because they routinely have a perfect blend of heterogneneous tasks (e.g. the leaves of the elimination tree) as well as large regular tasks (the root of the elimination tree). I devise a benchmark here to simulate this and see how we can try to bridge this with a single threading runtime.

Unless stated otherwise, all benchmarks were run on a Google Cloud h3-standard-88 VM featuring 88 physical Intel Xeon Platinum 8481C cores across four 22-core NUMA nodes, private 2 MiB L2 caches, and two 105 MiB shared last-level caches. SMT was disabled in the guest environment. Executables were built against oneMKL 2026.1 using either mkl_intel_thread with Intel libiomp5, or mkl_tbb_thread with oneTBB 2023.1. Matrices were touched in parallel prior to timed regions, and reported values reflect medians across warm runs.

Heterogeneous Tasks

Here, work-sharing and work-stealing refer to the execution models used by the two oneMKL backends:

• OpenMP-oneMKL: Executes each kernel through a coordinated, bulk-synchronous team. OpenMP may schedule calls dynamically, but each call retains its own predetermined inner team.
• oneTBB-oneMKL: Exposes application tasks and nested kernel tasks to the same worker pool, allowing capacity to move between calls.

I built a benchmark imitating the task structure of a supernodal factorization to find where this trade-off shifts in practice. It consists of eight levels. Each task performs a dense Cholesky factorization (DPOTRF) followed by a level-3 Schur complement update (DGEMM). The task count halves at each level while matrix dimensions scale upward, with matrix orders spread deterministically by ±30% within each level to introduce realistic load imbalance:

OpenMP executed outer tasks using schedule(dynamic, 1) with thread-local oneMKL allocations:

#pragma omp parallel for schedule(dynamic, 1)
for (int i = 0; i < tasks.size(); ++i) {
    mkl_set_num_threads_local(inner_threads);
    factor_and_update(tasks[i]);
}

The corresponding oneTBB outer loop can be written with grain-one range partitioning:

tbb::parallel_for(
    tbb::blocked_range<int>(0, task_count, 1),
    [&](const auto& range) {
        for (int i = range.begin(); i != range.end(); ++i)
            factor_and_update(tasks[i]);
    },
    tbb::simple_partitioner{});

OpenMP’s dynamic schedule and TBB’s grain-one range both balance the outer calls. OpenMP cannot redistribute an inner oneMKL team among those calls. TBB exposes both levels to one scheduler, so idle capacity can move across the entire nested workload.

Across Levels 0–3, oneTBB wins by up to 18.7% because dynamic work-stealing smooths out iteration imbalance that fixed OpenMP inner-team reservations cannot absorb. Once the hierarchy narrows to Levels 4–7, inner-kernel compute efficiency begins to dominate outer imbalance.

At Level 7 (the single dense root), there is no outer imbalance left to absorb. OpenMP’s coordinated kernel outperforms oneTBB by 52.2%. That single root node accounts for nearly the entire 10% deficit across the full tree traversal.

Large xGEMM: Panel Packing

Isolating large regular level-3 BLAS as a primary slowdown target I set up a benchmark with n = 15,840 to reproduce the issue and investigate.

I first suspected scheduler overhead or task over-decomposition. I profiled on-CPU stack samples with perf record -e cpu-clock:u -F 99 -g:

Despite my suspicion of the work-stealing scheduler hurting us, sampling found little visible scheduler time, while task instrumentation found exactly 88 TBB tasks. The difference was kernel selection: OpenMP executed mkl_blas_avx512_dgemm_kernel_0 alongside packing helpers (dgemm_dcopy_*), while oneTBB spent 99.86% of its execution time in mkl_blas_avx512_dgemm_kernel_nocopy_nn_b0/b1. Therefore oneTBB-oneMKL prioritized no-copy xGEMM over panel packing.

This choice fits the oneTBB runtime well. Bounded shared operand packing is fundamentally a work-sharing pattern: workers cooperatively fill a shared panel buffer, synchronize at a barrier, compute against that buffer, and synchronize again before overwriting it:

parallel_region(workers, [&](int worker) {
    for (KPanel k : k_panels) {
        pack_assigned_part(worker, k, shared_panel);
        barrier();

        compute_assigned_tiles(worker, shared_panel);
        barrier();
    }
});

Global barriers constrain a work-stealing scheduler because waiting participants cannot service unrelated work. They buy coordinated reuse at the cost of composability. The no-copy kernel preserves task independence, but gives up the packed microkernel’s performance on this large matrix. I attempted to solve this first using oneMKL’s provided panel packing helpers to do this manually

Prototype 1: Public oneMKL Packing APIs

My first attempt imposed work-sharing inside TBB using oneMKL’s public packing interface. I divided matrix B into column panels, packed those panels across parallel TBB tasks, and dispatched long-lived row workers against the packed representations:

struct PackedPanel {
    int column, width;
    aligned_buffer bytes; // cblas_dgemm_pack_get_size(...)
};

// Epoch 1: pack B panels in parallel
tbb::parallel_for(0, panel_count, [&](int p) {
    cblas_dgemm_pack(CblasRowMajor, CblasBMatrix, CblasNoTrans,
                     rows_per_owner, panels[p].width, K, 1.0,
                     &B[panels[p].column], ldb, panels[p].bytes.data());
});

// Epoch 2: row owners compute against each packed panel
tbb::parallel_for(0, machine_threads, [&](int owner) {
    auto [r0, r1] = equal_row_slab(owner, machine_threads, M);
    for (const PackedPanel& panel : panels)
        cblas_dgemm_compute(/* A[r0:r1, :] * packed(panel) -> C slab */);
});

Panel widths between 720 and 1,440 columns recovered OpenMP-level wall time (1.494 s). However oneMKL’s public packing interface (cblas_dgemm_pack) is largely designed for pre-packing a larger matrix, which means storing a lot of extra data which the OpenMP-oneMKL backend is not doing.

Prototype 2: OpenBLAS Microkernels + NUMA Arenas

To save on memory I repurposed OpenBLAS’s internal packing routines and microkernels. I distributed 88 long-lived row-owner tasks across four NUMA-constrained tbb::task_arena instances, 22 per node.

For each K panel, the row owners packed their private A slabs, cooperatively packed a single shared B panel, synchronized through a quickly written CyclicBarrier, and executed the microkernel:

tbb::parallel_for(row_owners, [&](Owner owner) {
    for (KPanel k : k_panels) {
        pack_private_A(owner.rows, k, owner.packed_A);
        pack_assigned_B_columns(owner, k, shared_packed_B);
        barrier.arrive_and_wait();

        openblas_microkernel(owner.rows, all_columns, k.width,
                             owner.packed_A, shared_packed_B,
                             C[owner.rows]);
        barrier.arrive_and_wait();
    }
});

The smaller buffers reduced explicit packed storage from 2.154 GiB to 247.5 MiB. With NUMA task arenas, execution time reached 1.453 s, within 0.7% of OpenMP oneMKL. Thus while this was the most invasive fix to the problem, it did mostly recover the lost performance.

Incorporating into a Library

OpenMP-oneMKL composes dynamic outer scheduling with internally work-shared kernels, whereas oneTBB-oneMKL exposes both layers to one scheduler. Coordinated work-sharing simplifies ownership and reuse. A unified scheduler makes it easier to redistribute resources across nested workloads.

A library using oneTBB could support two operational tiers:

1. Composable Mode: Default stealable tasks across a global worker pool for heterogeneous trees and multi-component pipelines.
2. Exclusive Mode Synchronized panel-packed work-sharing executed across long-lived TBB workers when a large dense kernel is granted dedicated ownership of the machine.

If a user selects (1) the underlying library simply exposes tasking to the global scheduler. Within oneTBB it is possible for the user to use their own NUMA-fenced arenas, so this need not necessarily be done by the library and the user could get the best of both worlds when NUMA effects as well as load balancing are the dominant performance wins. If a user selects (2) they are communicating to the library that resources are exclusively owned by this process and we may make use of much cheaper static scheduling, cooperative barriers, and long-lived tasks to promote better cache locality. In this case the user gives up composability but gets back in return a faster library.