Merge remote-tracking branch 'origin/tx_symbol_processing' into integration_2026_w15 (!3489)

Thread-pool support for TX symbol processing.

This MR adds thread-pool support for TX symbol processing. It allows modulation
/mapping/layer-precoding to run in parallel and offers a speedup of around 3
compared to single-thread execution. This is particularly important for large
bandwidths and 4 or more TX antenna ports where the precoding operation is quite
computationally-intensive.

Please see the description of !3489 for the performance comparison results and
the nr_dlsim timing measurements.

doc/NR_SA_Tutorial_COTS_UE.md

@@ -193,6 +193,7 @@ sudo ethtool -G enp1s0f0 tx 4096 rx 4096
 - Enable Performance Mode `sudo cpupower idle-set -D 0`
 - If you get real-time problems on heavy UL traffic, reduce the maximum UL MCS using an additional command-line switch: `--MACRLCs.[0].ul_max_mcs 14`.
 - You can also reduce the number of LDPC decoder iterations, which will make the LDPC decoder take less time: `--L1s.[0].max_ldpc_iterations 4`.
+- Read more on system tuning in the [dedicated document on performance tuning](./tuning_and_security.md#performance-tuning).
 
 ### 6.3 Uplink issues related with noise on the DC carriers
 

doc/tuning_and_security.md

@@ -69,6 +69,109 @@ sudo sysctl -n -e -q -w net.core.wmem_default=134217728
 sudo sysctl -n -e -q -w net.core.wmem_max=134217728
 ```
 
+### System tuning
+In order to get an optimal real-time behavior, a few tunings can be performed on the host system:
+- The use of isolated cores for the softmodem prevents competitions on the usage of core between the softmodem and other processes.
+  Core isolation is enabled through the kernel command line.  
+  **Warning: modifying the kernel command line can harm the OS behavior. Proceed with caution.**  
+  Refer to the [OAI 7.2 Fronthaul Interface Tutorial](./ORAN_FHI7.2_Tutorial.md) for examples.
+
+### Softmodem tuning
+The way the NR softmodem uses the computing ressource can be configured.
+It can have a significant effect on the performance and real-time behavior:
+- The L1 TX and L1 RX threads are the two main threads
+  executing the L1 RX and L1 TX pipelines.  
+  These threads are ideally assigned to two dedicated cores.
+  To be dedicated, the cores should be isolated in the
+  kernel parameters and not be assigned elsewhere.  
+  They can be assigned to specified cores with options
+  `--L1s.[0].L1_tx_thread_core` and `--L1s.[0].L1_rx_thread_core`
+  followed by a core id.
+- The thread pool is a group of processor cores over which
+  some baseband processing worker cores execute.  
+  It is configured by providing a list of core ids
+  after option `--thread-pool`.  
+  `-1` can also be passed instead of a core id
+  in order to use a floating core.  
+  By default, the thread pool is 8 floating cores.
+- PDSCH generation (i.e., layer mapping and precoding) is by default executed
+  in the L1 TX thread but can be multithreaded using the thread pool.  
+  This is enabled by option `--L1s.[0].L1_num_tx_sym_per_thread` followed
+  by the number of symbols that should be processed in each thread.  
+  This option can also be set in the gNB configuration file in field
+  `L1_num_tx_sym_per_thread` in the `L1s` section.
+
+### Workarounds
+If the real-time performance remains bad after tuning the system and softmodem,
+some workarounds allow to lower the computing demand
+at the cost of lower network performance:
+- If you get real-time problems on heavy UL traffic,
+  reduce the maximum UL MCS using an additional
+  command-line switch: `--MACRLCs.[0].ul_max_mcs 14`.  
+  This comes at the cost of a lower spectral efficiency
+  (i.e., less data for the same radio resource).
+- You can also reduce the number of LDPC decoder iterations,
+  which will make the LDPC decoder take less time:
+  `--L1s.[0].max_ldpc_iterations 4`.  
+  The default number of LDPC iterations is 8.
+  Lowering the number of iteration comes at the cost
+  of more unsuccessful transmissions.  
+  OAI offers multiple implementation of LDPC coding,
+  including offloading to an accelerator,
+  the number of LDPC iteration should be chosen accordingly.
+
+### Known hardware behaviors
+Here is a **non-exhaustive** list of known behaviors related to hardware architecture:
+- On some AMD EPYC series processors with Zen architecture
+  (at least every Zen4, Zen4c, Zen5 and Zen5c based processors
+  experience this behavior),  
+  the processor is made of multiple dies holding one or multiple
+  core complexes which are groups of cores with an L3 cache.  
+  This means that cores from different core complexes
+  do not share the same L3 cache and communication between
+  these cores implies inter L3 cache communication  
+  within a die or, even worse, between dies,
+  which has a cost in term of latency.  
+  Depending on the system configuration,
+  the NUMA topology may reflect this physical topology,
+  which can induce even further latency for
+  inter core complex communication.  
+  The softmodem is sensitive to this latency and its performance
+  can be harmed if it uses cores across the border of dies or core complexes,  
+  especially when multithreading of PDSCH generation is enabled
+  (argument of `--L1s.[0].L1_num_tx_sym_per_thread` is superior to 0).  
+  **Solution**: We recommend to use only one core complex
+  or one die for allocating cores to nr-softmodem process.
+- NUMA architecture: Make sure you don't assign cores to nr-softmodem from
+  different numa nodes. Cores from different numa cores can induce latency.
+  It is preferred to use the cores from numa node
+  which is used by the Fronthaul NIC.
+
+**Example**: How to assign cores to `nr-softmodem` process for gNB/DU connected to an o-ran 7.2 O-RU on EPYC 9575F (64 Zen5)
+assuming that we have configured `L1s.[0].L1_num_tx_sym_per_thread = 1`
+
+There are 8 cores per core complex and one core complex per die. You can visulize this topology using 
+
+```bash
+# Ubuntu 25.04
+for X in $(seq 0 63); do echo -n "cpu$X -> die "; cat /sys/devices/system/cpu/cpu$X/topology/die_id; done
+```
+   
+A 100MHz 4x4 FR1 gNB/DU with FHI 7.2 can be executed with full capabilities (4DL and 2UL layers)
+on a single core complex (cpus 0-7 in this example) by folowing the instructions of the
+[OAI 7.2 Fronthaul Interface Tutorial](./ORAN_FHI7.2_Tutorial.md)
+with the following core assignment:
+
+- `L1s.[0].L1_tx_thread_core = 0`
+- `L1s.[0].L1_rx_thread_core = 1`
+- `RUs.[0].ru_thread_core = 2`
+- `fhi_72.system_core = 3`
+- `fhi_72.io_core = 4`
+- `fhi_72.worker_cores.[0] = 5` (minimum 1 core)
+- `thread-pool: 2,3,6,7`
+  (thread pool can overlap with `RUs.[0].ru_thread_core` and `fhi_72.system_core`)
+
+
 ## Capabilities
 
 Historically, all softmodems are executed as `root`, typically using `sudo`.

executables/nr-gnb.c

@@ -208,8 +208,7 @@ static size_t dump_L1_meas_stats(PHY_VARS_gNB *gNB, RU_t *ru, char *output, size
   output += print_meas_log(&gNB->dlsch_encoding_stats, "DLSCH encoding", NULL, NULL, output, end - output);
   output += print_meas_log(&gNB->dlsch_scrambling_stats, "DLSCH scrambling", NULL, NULL, output, end-output);
   output += print_meas_log(&gNB->dlsch_modulation_stats, "DLSCH modulation", NULL, NULL, output, end - output);
-  output += print_meas_log(&gNB->dlsch_resource_mapping_stats, "DLSCH resource mapping", NULL, NULL, output,end-output);
-  output += print_meas_log(&gNB->dlsch_precoding_stats, "DLSCH precoding", NULL, NULL, output,end-output);
+  output += print_meas_log(&gNB->dlsch_pdsch_generation_stats, "PDSCH generation", NULL, NULL, output, end - output);
   output += print_meas_log(&gNB->phy_proc_rx, "L1 Rx processing", NULL, NULL, output, end - output);
   output += print_meas_log(&gNB->ts_deinterleave, "UL segment deinterleaving", NULL, NULL, output, end - output);
   output += print_meas_log(&gNB->ts_rate_unmatch, "UL segment rate recovery", NULL, NULL, output, end - output);
@@ -280,8 +279,7 @@ void *nrL1_stats_thread(void *param) {
   reset_meas(&gNB->rx_pusch_stats);
   reset_meas(&gNB->dlsch_scrambling_stats);
   reset_meas(&gNB->dlsch_modulation_stats);
-  reset_meas(&gNB->dlsch_resource_mapping_stats);
-  reset_meas(&gNB->dlsch_precoding_stats);
+  reset_meas(&gNB->dlsch_pdsch_generation_stats);
   while (!oai_exit) {
     sleep(1);
     if (ftruncate(fileno(fd), 0) != 0 || fseek(fd, 0, SEEK_SET) != 0) {
@@ -309,8 +307,6 @@ void init_gNB_Tpool(int inst)
   PHY_VARS_gNB *gNB;
   gNB = RC.gNB[inst];
   gNB_L1_proc_t *proc = &gNB->proc;
-  // PUSCH symbols per thread need to be calculated by how many threads we have
-  gNB->num_pusch_symbols_per_thread = 1;
   // ULSCH decoding threadpool
   initTpool(get_softmodem_params()->threadPoolConfig, &gNB->threadPool, cpumeas(CPUMEAS_GETSTATE));
 

openair1/PHY/defs_gNB.h

@@ -485,6 +485,7 @@ typedef struct PHY_VARS_gNB_s {
   notifiedFIFO_t L1_rx_out;
   tpool_t threadPool;
   int num_pusch_symbols_per_thread;
+  int num_pdsch_symbols_per_thread;
   int dmrs_num_antennas_per_thread;
   pthread_t L1_rx_thread;
   int L1_rx_thread_core;

openair1/SIMULATION/NR_PHY/dlsim.c

@@ -382,6 +382,8 @@ int main(int argc, char **argv)
   uint8_t  dlsch_threads = 0;
   int chest_type[2] = {0};
   uint8_t  max_ldpc_iterations = 5;
+  // number of PDSCH symbols per thread = 0 means do not use thread pool
+  int num_pdsch_symbols_per_thread = 0;
   if ((uniqCfg = load_configmodule(argc, argv, CONFIG_ENABLECMDLINEONLY)) == 0) {
     exit_fun("[NR_DLSIM] Error, configuration module init failed\n");
   }
@@ -583,6 +585,10 @@ int main(int argc, char **argv)
       gNBthreads[sizeof(gNBthreads)-1]=0;
       break;
 
+    case 'Y':
+      num_pdsch_symbols_per_thread = atoi(optarg);
+      break;
+
     case 'Z' :
       filename_csv = strdup(optarg);
       AssertFatal(filename_csv != NULL, "strdup() error: errno %d\n", errno);
@@ -657,6 +663,7 @@ int main(int argc, char **argv)
       printf("-T Enable PTRS, arguments list L_PTRS{0,1,2} K_PTRS{2,4}, e.g. -T 2 0 2 \n");
       printf("-U Change DMRS Config, arguments list DMRS TYPE{0=A,1=B} DMRS AddPos{0:2} DMRS ConfType{1:2}, e.g. -U 3 0 2 1 \n");
       printf("-X gNB thread pool configuration, n => no threads\n");
+      printf("-Y Number of symbols processed per PDSCH generation thread\n");
       printf("-Z Output filename (.csv format) for stats\n");
       exit (-1);
       break;
@@ -715,6 +722,7 @@ int main(int argc, char **argv)
 
   AssertFatal((gNB->if_inst = NR_IF_Module_init(0)) != NULL, "Cannot register interface");
   gNB->if_inst->NR_PHY_config_req = nr_phy_config_request;
+  gNB->num_pdsch_symbols_per_thread = num_pdsch_symbols_per_thread;
 
   NR_ServingCellConfigCommon_t *scc = calloc(1,sizeof(*scc));;
   prepare_scc(scc);

openair2/GNB_APP/L1_nr_paramdef.h

@@ -28,6 +28,10 @@
 #define L1_MAX_LDPC_ITERATIONS               "max_ldpc_iterations"
 #define L1_RX_THREAD_CORE                    "L1_rx_thread_core"
 #define L1_TX_THREAD_CORE                    "L1_tx_thread_core"
+#define L1_NUM_RX_SYM_PER_THREAD             "L1_num_rx_sym_per_thread"
+#define HLP_L1_NUM_RX_SYM_PER_THREAD         "number of symbols processed per PUSCH generation thread"
+#define L1_NUM_TX_SYM_PER_THREAD             "L1_num_tx_sym_per_thread"
+#define HLP_L1_NUM_TX_SYM_PER_THREAD         "number of symbols processed per PDSCH generation thread"
 #define HLP_TP_SIZ "thread_pool_size paramter removed, please use --thread-pool"
 #define L1_TX_AMP_BACKOFF_dB                 "tx_amp_backoff_dB"
 #define HLP_L1TX_BO "Backoff from full-scale output at the L1 entity(frequency domain), ex. 12 would corresponding to 14-bit input level (6 dB/bit). Default 36 dBFS for OAI RU entity"
@@ -58,6 +62,8 @@
   {L1_MAX_LDPC_ITERATIONS,               NULL,       0,         .uptr=NULL,           .defintval=8,               TYPE_UINT,     0},         \
   {L1_RX_THREAD_CORE,                    NULL,       0,         .iptr=NULL,           .defintval=-1,              TYPE_INT,      0},         \
   {L1_TX_THREAD_CORE,                    NULL,       0,         .iptr=NULL,           .defintval=-1,              TYPE_INT,      0},         \
+  {L1_NUM_RX_SYM_PER_THREAD,             HLP_L1_NUM_RX_SYM_PER_THREAD, 0, .iptr=NULL, .defintval=1,               TYPE_INT,      0},         \
+  {L1_NUM_TX_SYM_PER_THREAD,             HLP_L1_NUM_TX_SYM_PER_THREAD, 0, .iptr=NULL, .defintval=0,               TYPE_INT,      0},         \
   {L1_TX_AMP_BACKOFF_dB,                 HLP_L1TX_BO,0,         .uptr=NULL,           .defintval=36,              TYPE_UINT,     0},         \
   {L1_PHASE_COMP,                        HLP_L1_PHASE_COMP,PARAMFLAG_BOOL, .uptr=NULL,.defintval=1,               TYPE_UINT,     0},         \
   {L1_NUM_ANTENNAS_PER_THREAD,           HLP_NUM_ARX,0,         .uptr=NULL,           .defintval=1,               TYPE_UINT,     0},         \

openair2/GNB_APP/gnb_config.c

@@ -841,6 +841,9 @@ void RCconfig_NR_L1(void)
       gNB->L1_rx_thread_core = *gpd(params, np, L1_RX_THREAD_CORE)->iptr;
       gNB->L1_tx_thread_core = *gpd(params, np, L1_TX_THREAD_CORE)->iptr;
       LOG_I(NR_PHY, "thread cores for L1_RX %d L1_TX %d\n", gNB->L1_rx_thread_core, gNB->L1_tx_thread_core);
+      // PUSCH symbols per thread need to be calculated by how many threads we have
+      gNB->num_pusch_symbols_per_thread = *gpd(params, np, L1_NUM_RX_SYM_PER_THREAD)->iptr;
+      gNB->num_pdsch_symbols_per_thread = *gpd(params, np, L1_NUM_TX_SYM_PER_THREAD)->iptr;
       gNB->TX_AMP = min(32767.0 / pow(10.0, .05 * (double)(*gpd(params, np, L1_TX_AMP_BACKOFF_dB)->uptr)), INT16_MAX);
       LOG_I(NR_PHY, "TX_AMP = %d (-%d dBFS)\n", gNB->TX_AMP, *gpd(params, np, L1_TX_AMP_BACKOFF_dB)->uptr);
       AssertFatal(gNB->TX_AMP > 300, "TX_AMP is too small, must be larger than 300 (is %d)\n", gNB->TX_AMP);