1. 22 Feb, 2024 1 commit
  2. 18 Feb, 2024 2 commits
  3. 13 Feb, 2024 1 commit
  4. 07 Feb, 2024 1 commit
    • Aymen Qader's avatar
      Improve performance of simde_mm512_add_epi32 (#1126) · 8392c69a
      Aymen Qader authored
      Improve and simplify implementation of simde_mm512_add_epi32 as follows:
      
      (1): Remove the explicit SVE implementation. For SVE vector lengths of
      VL={128, 256}, this explicit vector length agnostic (VLA) SVE loop
      performs significantly worse than the Neon equivalent, which can be
      executed using fewer instructions. This sequence of SVE intrinsics is
      also malformed according to clang, so it fails to compile altogether.
      
      (2): Preferentially use GCC's vector extension if available, instead of
      repeated calls to simde_mm256_add_epi32. There are a couple of reasons
      for this:
      
      (a) The added indirection results in worse code generation. See the code
          generation attached to commit message for an example with GCC 13.
      
      (b) GCC's vector extension is an easier optimization target for
          compilers, allowing them to appropriately output performant code
          generation depending on their own internal cost & tuning models.
          See the snippets attached to commit message for an example of
          improved code-gen in a vector length specific (VLS) context.
      
      This brings the implementation of simde_mm512_add_epi32 back in line
      with other similar AVX512 intrinsics, such as simde_mm512_sub_epi32 and
      simde_mm512_mul_ps.
      
      Fixes #980.
      
      An example of code-gen difference is shown below. Source is a function
      containing a single call to simde_mm512_add_epi32.
      
      Compiler: GCC 13.2.0
      Compile flags: -O3 -march=armv8-a
      
      Before this patch:
         0:    ld1    {v28.16b-v31.16b}, [x0]
         4:    sub    sp, sp, 0x90
         8:    ld1    {v24.16b-v27.16b}, [x1]
         c:    add    x2, sp, 0x3f
        10:    and    x2, x2, 0xffffffffffffffc0
        14:    add    x0, x2, 0x40
        18:    add    x1, x2, 0x20
        1c:    add    v24.4s, v24.4s, v28.4s
        20:    add    v25.4s, v25.4s, v29.4s
        24:    add    v26.4s, v30.4s, v26.4s
        28:    add    v27.4s, v31.4s, v27.4s
        2c:    stp    q24, q25, [x2, 64]
        30:    ld1    {v28.16b, v29.16b}, [x0]
        34:    stp    q26, q27, [x2, 64]
        38:    ld1    {v30.16b, v31.16b}, [x0]
        3c:    st1    {v28.16b, v29.16b}, [x2]
        40:    st1    {v30.16b, v31.16b}, [x1]
        44:    ld1    {v28.16b-v31.16b}, [x2]
        48:    st1    {v28.16b-v31.16b}, [x8]
        4c:    add    sp, sp, 0x90
        50:    ret
      
      With this patch:
         0:    ld1    {v28.16b-v31.16b}, [x0]
         4:    ld1    {v24.16b-v27.16b}, [x1]
         8:    add    v24.4s, v28.4s, v24.4s
         c:    add    v25.4s, v29.4s, v25.4s
        10:    add    v26.4s, v30.4s, v26.4s
        14:    add    v27.4s, v31.4s, v27.4s
        18:    st1    {v24.16b-v27.16b}, [x8]
        1c:    ret
      
      Another example of code-gen difference is shown below, targeting an SVE
      enabled microarchitecture with a 512-bit vector length in a vector
      length specific (VLS) context.
      
      Compiler: GCC 13.2.0
      Compile flags: -O3 -march=armv8-a+sve -msve-vector-bits=512
      
      Before this patch:
         0:    sub     sp, sp, 0xf0
         4:    mov     x2, 0x10 // 16
         8:    ptrue   p6.b, vl64
         c:    add     x4, sp, 0x3f
        10:    mov     x3, x2
        14:    ld1d    {z31.d}, p6/z, [x0]
        18:    and     x4, x4, 0xffffffffffffffc0
        1c:    ld1d    {z30.d}, p6/z, [x1]
        20:    ptrue   p7.s, vl16
        24:    add     x6, x4, 0x40
        28:    add     x5, x4, 0x80
        2c:    st1d    {z30.d}, p6, [x4]
        30:    st1d    {z31.d}, p6, [x4, 1, mul vl]
        34:    nop
        38:    nop
        3c:    nop
        40:    add     x0, x4, x2, lsl 2
        44:    add     x1, x6, x2, lsl 2
        48:    ld1w    {z30.s}, p7/z, [x0, -1, mul vl]
        4c:    ld1w    {z31.s}, p7/z, [x1, -1, mul vl]
        50:    add     x0, x5, x2, lsl 2
        54:    add     z30.s, z31.s, z30.s
        58:    st1w    {z30.s}, p7, [x0, -1, mul vl]
        5c:    whilelo p7.s, x2, x3
        60:    add     x2, x2, 0x10
        64:    b.ne    40 // b.any
        68:    ptrue   p7.b, vl64
        6c:    ld1d    {z31.d}, p7/z, [x4, 2, mul vl]
        70:    st1d    {z31.d}, p7, [x8]
        74:    add     sp, sp, 0xf0
        78:    ret
      
      With this patch:
         0:    ptrue   p0.b, vl64
         4:    ld1d    {z0.d}, p0/z, [x0]
         8:    ld1d    {z1.d}, p0/z, [x1]
         c:    add     z0.s, z0.s, z1.s
        10:    st1d    {z0.d}, p0, [x8]
        14:    ret
      8392c69a
  5. 12 Jan, 2024 2 commits
  6. 15 Dec, 2023 1 commit
  7. 22 Nov, 2023 5 commits
  8. 20 Nov, 2023 2 commits
    • Michael R. Crusoe's avatar
      prepare to release 0.8.0 · e96949e3
      Michael R. Crusoe authored
      e96949e3
    • Yi-Yen Chung's avatar
      NEON: implement all bf16-related intrinsics (#1110) · f73d72e4
      Yi-Yen Chung authored
      * [Feat] Add BF16 when the machine is supported.
      
      Finished: vld1_bf16_x4 and vld1q_bf16_x2
      
      * [NEON] Add a C implementation of the bf16 type
      
      * [NEON] Add all ld_*_bf16 intrinsics.
      
      * [NEON] Add all st*_bf16 intrinsics.
      
      * [Test] Add vbfdot_f32 test case
      
      * [NEON] Complete converting function from float32 to bfloat16.
      
      - Also add bf-related functions in three series
      - cvt, dot, dot_lane
      
      * [Feat] Add option '+bf16' in cross-file
      
      * [NEON] Completed initial implementation of bf-16 related intrinsics.
      
      * [Fix] Remove redundant commment
      
      * [Fix] Correct native aliases
      
      * [Fix] The test generation code has been completed.
      f73d72e4
  9. 17 Nov, 2023 1 commit
  10. 16 Nov, 2023 1 commit
    • Yi-Yen Chung's avatar
      NEON: implement all intrinsics supported by architecture A64-remaining part (#1093) · d6271b3f
      Yi-Yen Chung authored
      * [NEON] Add 5 intrinsics (vdiv{h/q}_f{16/64}).
      * [NEON] Add 11 dup_lane series intrinsics.
      - 2 dup{q}_laneq_f16
      - 9 dup{b,h}_lane{q}_{s/u}{8,16}, duph_laneq_f16
      * [NEON] Add 8 intrinsics (veor3q{s/u}{8/16/32/64}).
      * [NEON] Add fmlal, fmlsl, maxnmv, minnmv, pmaxnm, pminnm.
      * [NEON] Add 12 fmlal series intrinsics.
      * [NEON] Add 12 fmlsl series intrinsics.
      * [NEON] Add 11 vmax series intrinsics.
      - 1 vmaxh_f16
      - 3 vmaxnm{/h/q}_f16
      - 5 vmaxnmv{q}_f{16/32/64}
      - 2 vmaxv{q}_f16
      * [NEON] Add 11 vmin series intrinsics.
      - 1 vminh_f16
      - 3 vminnm{/h/q}_f16
      - 5 vminnmv{q}_f{16/32/64}
      - 2 vminv{q}_f16
      * [NEON] Add 8 vpmax series intrinsics.
      - 1 vpmaxq_f16
      - 7 vpmaxnm{/s/q/qd}_f{16/32/64}
      * [NEON] Add 9 vpmin series intrinsics.
      - 2 vpmin{q}_f16
      - 7 vpminnm{/s/q/qd}_f{16/32/64}
      * [NEON] Add 8 intrinsic function families.
      mmlaq, mull_high_lane, mull_high_n, mulx,
      mulx_lane, mulx_n, qrdmlah, qmovun_high.
      * [NEON] Add 3 vmmlaq series intrinsics.
      * [NEON] Add 41 vmul-related intrinsics.
      - 8 mull_high_lane series intrinsics
      - 4 mull_high_n series intrinsics
      - 9 vmulx series intrinsics
      - 2 vmulx{q}_n_f16 series intrinsics
      - 18 vmulx_lane series intrinsics
      * [NEON] Add 1 vpaddq_f16 intrinsic.
      * [NEON] Add 3 vqmovun_high_s{16/32/64} intrinsic.
      * [NEON] Add 6 vqrdmlah series intrinsics.
      * [NEON] Add 11 series intrinsics.
      qrdmlah_lane, qrdmlsh, qrdmlsh_lane, qshrun_high_n,
      rnd32x, rnd32z, rnd64x, rnd64z, rnda, rndx, shll_high_n.
      * [NEON] Add 30 vqrdmlah, vqrdmlsh related intrinsics.
      - 12 vqrdmlah{h/s/q}_lane{q}_s{16/32}
      - 6 vqrdmlsh{h/s/q}_s{16/32}
      - 12 vqrdmlsh{h/s/q}_lane{q}_s{16/32}
      * [NEON] Add 2 vqrdmulhh_lane{q}_s16 intrinsics.
      * [NEON] Add 5 vqsh related intrinsics.
      - 1 vqshluh_n_s16
      - 3 vqshrun_high_n_s{16/32/64}
      - 1 vqshrun_n_s16
      * [NEON] Add 16 vrnd32x, vrnd32z, vrnd64x, vrnd64z related intrinsics.
      - 4 vrnd32x{q}_f{32/64}
      - 4 vrnd32z{q}_f{32/64}
      - 4 vrnd64x{q}_f{32/64}
      - 4 vrnd64x{q}_f{32/64}
      * [NEON] Add vrnd{/a/i/m/p/x} related intrinsics.
      - 3 vrnd{/h/q}_f16
      - 3 vrndi{/h/q}_f16
      - 3 vrndm{/h/q}_f16
      - 3 vrndp{/h/q}_f16
      - 7 vrnda{q}_f{16/32/64}, vrndah_f16
      - 7 vrndx{q}_f{16/32/64}, vrndxh_f16
      * [NEON] Add 6 vshll_high_n series intrinsics.
      * [NEON] Add 7 intrinsic series.
      cadd_rot270, cadd_rot90, shrn_high_n, subhn_high,
      sudot_lane, usdot, usdot_lane
      * [NEON] Add 2 vcmla{q}_f16 intrinsics
      * [NEON] Add 6 vshrn_high_n series intrinsics
      * [NEON] Add 6 vsubhn_high series intrinsics
      * [NEON] Add 10 vsudot_lane, vusdot, and vusdot_lane series intrinsics.
      - 4 sudot{q}_lane{q}_s32
      - 2 vusdot{q}_s32
      - 4 vusdot{q}_lane{q}_s32
      * [NEON] Add 10 vadd{q}_rot{90/270}_f{16/32/64} intrinsics.
      * [NEON] Add 5 series intrinsics.
      cmla_lane, cmla_rot180_lane, cmla_rot270_lane, cmla_rot90_lane, recpx.
      * [NEON] Add 38 vcmla related intrinsics.
      - 8 cvmla{q}_lane{q}_f{16/32}
      - 2 cvmla{q}_rot90_f16
      - 8 cvmla{q}_rot90_lane{q}_f{16/32}
      - 2 cvmla{q}_rot180_f16
      - 8 cvmla{q}_rot180_lane{q}_f{16/32}
      - 2 cvmla{q}_rot270_f16
      - 8 cvmla{q}_rot270_lane{q}_f{16/32}
      * [NEON] Add vrecpeh_f16 and vrecpsh_f16 intrinsics.
      * [NEON] Add 3 vrecpx{h,s,d}_f{16,32,64} intrinsics.
      * [NEON] Add 8 series intrinsics.
      __crc32, ras, sha1, sha256, sha512, sm3, sm4
      * [NEON] Add 8 __crc series intrinsics.
      * [NEON] Add vrax1q_u64 intrinsic
      * [NEON] Add sha1, sha256, and sha512 series intrinsics
      * [NEON] Add sm3 and sm4 series intrinsics
      * [NEON] Include <arm_acle.h> for __crc32 intrinsics
      * [NEON] Use uint to simulate the poly type and implement it
      * [NEON] Add poly type related intrinsics
      * [NEON] Add ldr and str related intrinsics.
      Co-authored-by: default avatarEric Yi-Yen Chung <eric681@andestech.com>
      Co-authored-by: default avatarMichael R. Crusoe <michael.crusoe@gmail.com>
      d6271b3f
  11. 13 Nov, 2023 1 commit
  12. 11 Nov, 2023 1 commit
  13. 09 Nov, 2023 1 commit
  14. 07 Nov, 2023 2 commits
  15. 03 Nov, 2023 1 commit
  16. 31 Oct, 2023 2 commits
  17. 27 Oct, 2023 3 commits
  18. 25 Oct, 2023 2 commits
  19. 24 Oct, 2023 1 commit
    • Yi-Yen Chung's avatar
      NEON: part 1 of implement all intrinsics supported by architecture A64 (#1090) · 70f70262
      Yi-Yen Chung authored
      Add 368 initial implementations and corresponding test cases in 88 families which are listed below:
      - `abd`, `abdl_high`, `add`, `addhn_high`, `bsl`, `ceq`, `ceqz`, `cgez`, `cgtz`, `cle`,
      - `cltz`, `cmla`, `cmla_rot180`, `cmla_rot270`, `cmla_rot90`, `cnt`, `copy_lane`, `cvt`, `cvt_n`, `cvtm`,
      -  `cvtp`, `dot`, `dot_lane`, `dup_n`, `eor`, `fms_n`, `ld1`, `ld3`, `ld4`, `maxnm`,
      -  `maxv`, `minnm`, `minv`, `mull`, `mull_high`, `mvn`, `pmin`, `qrdmulh_lane`, `qrshl`, `qrshrn_high_n`,
      -  `qrshrun_high_n`, `qshl_n`, `qshlu_n`, `qshrn_high_n`, `qshrn_n`, `qshrun_n`, `qtbl`, `qtbx`, `raddhn`, `raddhn_high`,
      -  `rbit`, `reinterpret`, `rev16`, `rev32`, `rev64`, `rnd`, `rndi`, `rndm`, `rndp`, `rshrn_high_n`,
      -  `rsubhn`, `rsubhn_high`, `shr_n`, `shrn_n`, `sli_n`, `sri_n`, `st1`, `st1_lane`, `st1_x2`, `st1_x3`,
      -  `st1_x4`, `st1q_x2`, `st1q_x3`, `st1q_x4`, `st2_lane`, `st3`, `st3_lane`, `st4`, `st4_lane`, `tbl`,
      -  `tbx`, `trn`, `trn1`, `trn2`, `tst`, `uzp`, `uzp1`, `uzp2`
      70f70262
  20. 20 Oct, 2023 2 commits
  21. 19 Oct, 2023 3 commits
  22. 18 Oct, 2023 2 commits
    • Yi-Yen Chung's avatar
    • Yi-Yen Chung's avatar
      NEON: more fp16 using intrinsics supported by architecture v7 (skip version) (#1081) · 5634cec0
      Yi-Yen Chung authored
      * [NEON] Add vabal_{s/u}{8/16/32}
      
      * [NEON] Add vabal_high_{s/u}{8/16/32}
      
      * [NEON] Add all vcale* intrinsics (9)
      
      * [NEON] Add all vcalt intrinsics (9)
      
      * [NEON] Add vcreate_f16
      
      * [NEON] Add vreinterpret_u64_f16
      
      * [NEON] Add vcvth_f16_s16 and vcvth_f16_u16
      
      * [NEON] Add vduph_lane_f16, vdup_lane_f16, and vdupq_lane_f16
      
      * [NEON] Add vext_f16
      
      * [NEON] Add 16 vcvt{q}_n_* intrinsics
      
      * [Fix] Correct function input parameters
      
      * [NEON] Add 6 vcvtn_{s/u}{16/32/64}_f{*} intrinsics
      
      * [Fix] Correct vdup_lane_f16 and vdupq_lane_f16.
      
      * [Fix] Correct function input parameters.
      
      * [NEON] Add 24 vcvt{q}_n_* intrinsics
      
      * [NEON] Add all vcvtn* intrinsics
      
      * [NEON] Add vfmah_f16 and vfma_f16
      
      * [NEON] Add vfma_n_f16 and vfmaq_n_f16
      
      * [NEON] Add vmulh_f16
      
      * [NEON] Add fma_lane related intrinsics.
      
      * [NEON] Add 5 vmul* related intrinsics
      vmulh_lane_f16, vmulh_laneq_f16, vmul_lane_f16,
      vmul_laneq_f16, vmulq_laneq_f16.
      
      * [NEON] Add neg related intrinsics.
      
      * [NEON] Add all fms, fms_n, and fms_lane intrinsics
      
      * [NEON] Add types float16x{4/8}x{2/3/4}
      
      * [NEON] Add 9 vld1 related intrinsics
      
      * [Fix] Modified wrong rounding implementation.
      Modified wrong implementation "Ties to Away" to "rounding to nearest
      with ties to Away"
      add.h: Remove redundant code.
      
      * [Fix] Fix wrong intrinsic alias names.
      
      * [Refactor] Remove redundant functions.
      
      * [NEON] Add 45 ld2 related intrinsics
      one ld2_f16, twenty-two ld2_lane series, and twenty-two ld2_dup series.
      
      * [NEON] Add ld3_dup, ld3_lane, and ld4_dup
      
      * [NEON] Add vld3_f16 and vld4_f16.
      
      * [NEON] Add vld{3/4}_{dup/lane} series intrinsics
      
      * [NEON] Add mla_{high}_lane series intrinsics
      
      * [NEON] Add qdmlal_{high}_{lane} series intrinsics.
      
      * [NEON] Add qdmlal_lane and qdmlal_n series intrinsics
      
      * [NEON] Add mls_lane and mlsl_high_lane series intrinsics
      
      * [NEON] Add 22 qdmlsl series intrinsics
      
      * [NEON] Add 10 qdmull_* series intrinsics
      
      * [NEON] Add 3 qdmulh series intrinsics
      
      * [Fix] Fix wrong function name.
      
      * [Fix] Correct the wrong alias function name.
      
      * [NEON] Add qdmullh_lane{q}_s{16/32} related intrinsics
      
      * [NEON] Add qdmull_n and qdmull_high_lane series intrinsics
      
      * [Fix] Add conditions for fp16 intrinsics
      
      * [Hack] Skip functions that trigger compiler bugs.
      5634cec0
  23. 17 Oct, 2023 1 commit
    • Chi-Wei Chu's avatar
      arm neon: Complex operations from Armv8.3-a (#1077) · cfd91723
      Chi-Wei Chu authored
      * [Neon] Add vcadd_rot270_f{16/32} and vcaddq_rot270_f{16/32/64}
      * [Neon] Add vcadd_rot90_f{16/32} and vcaddq_rot90_f{16/32/64}
      * [Neon] Add vcmla_lane_f{16/32} and vcmla_laneq_f{16/32} and vcmlaq_lane_f{16/32} and vcmlaq_laneq_f{16/32}
      * [Neon] Add vcmla_rot90_lane_f{16/32} and vcmla_rot90_laneq_f{16/32} and vcmlaq_rot90_lane_f{16/32} and vcmlaq_rot90_laneq_f{16/32}
      * [Neon] Add vcmla_rot180_lane_f{16/32} and vcmla_rot180_laneq_f{16/32} and vcmlaq_rot180_lane_f{16/32} and vcmlaq_rot180_laneq_f{16/32}
      * [Neon] Add vcadd_rot270_f{16/32} and vcaddq_rot270_f{16/32/64}
      cfd91723
  24. 16 Oct, 2023 1 commit