Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
O
OpenXG-RAN
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Operations
Operations
Metrics
Environments
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
OpenXG
OpenXG-RAN
Commits
f4b394a3
Commit
f4b394a3
authored
2 years ago
by
Robert Schmidt
Browse files
Options
Browse Files
Download
Plain Diff
Merge remote-tracking branch 'origin/ci-clean-dashboard-drop-old-images' into integration_2022_wk06
parents
adb02aa2
a36ed24b
Branches unavailable
2025.w14
2025.w13
2025.w12
2025.w11
2025.w10
2025.w09
2025.w08
2025.w07
2025.w06
2025.w05
2025.w04
2025.w03
2025.w02
2024.w51
2024.w50
2024.w49
2024.w48
2024.w47
2024.w46
2024.w45
2024.w44
2024.w43
2024.w42
2024.w41
2024.w40
2024.w39
2024.w38
2024.w36
2024.w35
2024.w34
2024.w33
2024.w32
2024.w31
2024.w30
2024.w29
2024.w28
2024.w27
2024.w26
2024.w25
2024.w24
2024.w23
2024.w22
2024.w21
2024.w18
2024.w17
2024.w16
2024.w15
2024.w14
2024.w13
2024.w12
2024.w11
2024.w10
2024.w09
2024.w08
2024.w06
2024.w05
2024.w04
2024.w03
2024.w02
2024.w01
2023.w51
2023.w50
2023.w49
2023.w48
2023.w47
2023.w45
2023.w43
2023.w42
2023.w41
2023.w40
2023.w39
2023.w38
2023.w37
2023.w36
2023.w34
2023.w33
2023.w32
2023.w31
2023.w30
2023.w29
2023.w28
2023.w27
2023.w26
2023.w25
2023.w24
2023.w23
2023.w22
2023.w21
2023.w20
2023.w19
2023.w18
2023.w18b
2023.w16
2023.w15
2023.w14
2023.w13
2023.w12
2023.w11
2023.w11b
2023.w10
2023.w10b
2023.w09
2023.w08
2023.w08b
2023.w07
2023.w06
v2.2.0
v2.1.0
v2.0.0
ARC_1.3
No related merge requests found
Changes
11
Show whitespace changes
Inline
Side-by-side
Showing
11 changed files
with
17 additions
and
1417 deletions
+17
-1417
ci-scripts/Jenkinsfile-git-dashboard
ci-scripts/Jenkinsfile-git-dashboard
+0
-48
ci-scripts/Jenkinsfile-tmp-multi-enb-nsa
ci-scripts/Jenkinsfile-tmp-multi-enb-nsa
+0
-13
ci-scripts/cls_cluster.py
ci-scripts/cls_cluster.py
+8
-0
ci-scripts/ran_dashboard/Hdashboard.py
ci-scripts/ran_dashboard/Hdashboard.py
+9
-511
ci-scripts/ran_dashboard/mr_styles.css
ci-scripts/ran_dashboard/mr_styles.css
+0
-136
ci-scripts/ran_dashboard/ran_dashboard.py
ci-scripts/ran_dashboard/ran_dashboard.py
+0
-405
ci-scripts/ran_dashboard/ran_dashboard_cfg.yaml
ci-scripts/ran_dashboard/ran_dashboard_cfg.yaml
+0
-26
ci-scripts/ran_dashboard/readme.txt
ci-scripts/ran_dashboard/readme.txt
+0
-3
ci-scripts/ran_dashboard/sqlconnect.py
ci-scripts/ran_dashboard/sqlconnect.py
+0
-107
ci-scripts/ran_dashboard/test_styles.css
ci-scripts/ran_dashboard/test_styles.css
+0
-103
ci-scripts/template.xsl
ci-scripts/template.xsl
+0
-65
No files found.
ci-scripts/Jenkinsfile-git-dashboard
deleted
100644 → 0
View file @
adb02aa2
#
!
/bin/
groovy
/*
* Licensed to the OpenAirInterface (OAI) Software Alliance under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The OpenAirInterface Software Alliance licenses this file to You under
* the OAI Public License, Version 1.1 (the "License"); you may not use this file
* except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.openairinterface.org/?page_id=698
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*-------------------------------------------------------------------------------
* For more information about the OpenAirInterface (OAI) Software Alliance:
* contact@openairinterface.org
*/
// Template Jenkins Declarative Pipeline script to run Test w/ RF HW
// Location of the python executor node shall be in the same subnet as the others servers
def
pythonExecutor
=
params
.
pythonExecutor
pipeline
{
agent
{
label
pythonExecutor
}
stages
{
stage
(
"gDashboard"
)
{
steps
{
script
{
dir
(
"ci-scripts/ran_dashboard"
)
{
//retrieve MR data from gitlab / mySQL db, build HTML pages and load them to AWS S3 bucket (configured as static web page hosting)
//deprecated method : sh returnStdout: true, script: 'python3 ran_dashboard.py'
sh
returnStdout:
true
,
script:
'python3 Hdashboard.py'
}
}
}
}
}
}
This diff is collapsed.
Click to expand it.
ci-scripts/Jenkinsfile-tmp-multi-enb-nsa
View file @
f4b394a3
...
...
@@ -376,19 +376,6 @@ pipeline {
}
}
}
stage
(
"Result Update"
){
when
{
expression
{
params
.
DataBaseHost
!=
"none"
}
}
agent
{
label
DataBaseHost
}
steps
{
script
{
dir
(
'ci-scripts/ran_dashboard'
)
{
sh
"python3 Hdashboard.py testevent ${params.eNB_MR} "
}
}
}
}
}
}
}
...
...
This diff is collapsed.
Click to expand it.
ci-scripts/cls_cluster.py
View file @
f4b394a3
...
...
@@ -230,6 +230,14 @@ class Cluster:
HTML
.
CreateHtmlTestRow
(
'N/A'
,
'KO'
,
CONST
.
OC_PROJECT_FAIL
)
return
False
# delete old images by Sagar Arora <sagar.arora@openairinterface.org>:
# 1. retrieve all images and their timestamp
# 2. awk retrieves those whose timestamp is older than 4 weeks
# 3. issue delete command on corresponding istags (the images are dangling and will be cleaned by the registry)
delete_cmd
=
"oc get istag -o go-template --template '{{range .items}}{{.metadata.name}} {{.metadata.creationTimestamp}}{{
\"\\
n
\"
}}{{end}}' | awk '$2 <=
\"
'$(date -d '-4weeks' -Ins --utc | sed 's/+0000/Z/')'
\"
{ print $1 }' | xargs --no-run-if-empty oc delete istag"
response
=
self
.
cmd
.
run
(
delete_cmd
)
logging
.
debug
(
f"deleted images:
\n
{
response
.
stdout
}
"
)
self
.
_recreate_entitlements
()
status
=
True
# flag to abandon compiling if any image fails
...
...
This diff is collapsed.
Click to expand it.
ci-scripts/ran_dashboard/Hdashboard.py
View file @
f4b394a3
...
...
@@ -32,488 +32,34 @@
#Author Remi
import
boto3
import
shlex
import
subprocess
import
json
#json structures
import
datetime
#now() and date formating
from
datetime
import
datetime
import
re
import
gitlab
import
yaml
import
os
import
time
import
sys
from
sqlconnect
import
SQLConnect
#-----------------------------------------------------------
# Class Declaration
#-----------------------------------------------------------
class
Dashboard
:
def
__init__
(
self
):
#init with data sources : git, yaml config file, test results databases
#init with data sources : git, test results databases
print
(
"Collecting Data"
)
cmd
=
"""curl --silent "https://gitlab.eurecom.fr/api/v4/projects/oai%2Fopenairinterface5g/merge_requests?state=opened&per_page=100" """
self
.
git
=
self
.
__getGitData
(
cmd
)
#git data from Gitlab
self
.
tests
=
self
.
__loadCfg
(
'ran_dashboard_cfg.yaml'
)
#tests table setup from yaml
self
.
db
=
self
.
__loadFromDB
()
#test results from database
self
.
mr_list
=
[]
#mr list in string format
for
x
in
range
(
len
(
self
.
git
)):
self
.
mr_list
.
append
(
str
(
self
.
git
[
x
][
'iid'
]))
def
__loadCfg
(
self
,
yaml_file
):
with
open
(
yaml_file
,
'r'
)
as
f
:
tests
=
yaml
.
load
(
f
)
return
tests
def
__getGitData
(
self
,
cmd
):
#cmd="""curl --silent "https://gitlab.eurecom.fr/api/v4/projects/oai%2Fopenairinterface5g/merge_requests?state=opened&per_page=100" """
process
=
subprocess
.
Popen
(
shlex
.
split
(
cmd
),
stdout
=
subprocess
.
PIPE
)
output
=
process
.
stdout
.
readline
()
tmp
=
output
.
decode
(
"utf-8"
)
d
=
json
.
loads
(
tmp
)
return
d
def
__loadFromDB
(
self
):
mr_list
=
[]
for
x
in
range
(
len
(
self
.
git
)):
mr_list
.
append
(
str
(
self
.
git
[
x
][
'iid'
]))
mydb
=
SQLConnect
()
for
MR
in
mr_list
:
mydb
.
get
(
MR
)
mydb
.
close_connection
()
return
mydb
.
data
def
singleMR_initHTML
(
self
,
date
):
self
.
f_html
.
write
(
'<!DOCTYPE html>
\n
'
)
self
.
f_html
.
write
(
'<head>
\n
'
)
self
.
f_html
.
write
(
'<link rel="stylesheet" href="../test_styles.css">
\n
'
)
self
.
f_html
.
write
(
'<title>Test Dashboard</title>
\n
'
)
self
.
f_html
.
write
(
'</head>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<table>
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td class="Main">OAI RAN TEST Status Dashboard</td>
\n
'
)
self
.
f_html
.
write
(
'</td>
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<tr></tr>
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td class="Date">Update : '
+
date
+
'</td>
\n
'
)
self
.
f_html
.
write
(
'</td>
\n
'
)
self
.
f_html
.
write
(
'</tr>
\n
'
)
self
.
f_html
.
write
(
'</table>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
def
Test_initHTML
(
self
,
date
):
self
.
f_html
.
write
(
'<!DOCTYPE html>
\n
'
)
self
.
f_html
.
write
(
'<head>
\n
'
)
self
.
f_html
.
write
(
'<link rel="stylesheet" href="test_styles.css">
\n
'
)
self
.
f_html
.
write
(
'<title>Test Dashboard</title>
\n
'
)
self
.
f_html
.
write
(
'</head>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<table>
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td class="Main">OAI RAN TEST Status Dashboard</td>
\n
'
)
self
.
f_html
.
write
(
'</td>
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td class="DashLink"> <a href="https://oairandashboard.s3.eu-west-1.amazonaws.com/index.html">Merge Requests Dashboard</a></td>
\n
'
)
self
.
f_html
.
write
(
'</td>
\n
'
)
self
.
f_html
.
write
(
'<tr></tr>
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td class="Date">Update : '
+
date
+
'</td>
\n
'
)
self
.
f_html
.
write
(
'</td>
\n
'
)
self
.
f_html
.
write
(
'</tr>
\n
'
)
self
.
f_html
.
write
(
'</table>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
def
Test_terminateHTML
(
self
):
self
.
f_html
.
write
(
'</body>
\n
'
)
self
.
f_html
.
write
(
'</html>
\n
'
)
self
.
f_html
.
close
()
def
MR_initHTML
(
self
,
date
):
self
.
f_html
.
write
(
'<!DOCTYPE html>
\n
'
)
self
.
f_html
.
write
(
'<head>
\n
'
)
self
.
f_html
.
write
(
'<link rel="stylesheet" href="mr_styles.css">
\n
'
)
self
.
f_html
.
write
(
'<title>MR Dashboard</title>
\n
'
)
self
.
f_html
.
write
(
'</head>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<table>
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td class="Main">OAI RAN MR Status Dashboard</td>
\n
'
)
self
.
f_html
.
write
(
'</td>
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td class="DashLink"> <a href="https://oaitestdashboard.s3.eu-west-1.amazonaws.com/index.html">Tests Dashboard</a></td>
\n
'
)
self
.
f_html
.
write
(
'</td>
\n
'
)
self
.
f_html
.
write
(
'<tr></tr>
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td class="Date">Update : '
+
date
+
'</td>
\n
'
)
self
.
f_html
.
write
(
'</td>
\n
'
)
self
.
f_html
.
write
(
'</tr>
\n
'
)
self
.
f_html
.
write
(
'</table>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<br>
\n
'
)
self
.
f_html
.
write
(
'<table class="MR_Table">
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<th class="MR">MR</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="CREATED_AT">Created_At</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="AUTHOR">Author</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="TITLE">Title</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="ASSIGNEE">Assignee</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="REVIEWER">Reviewer</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="CAN_START">CAN START</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="IN_PROGRESS">IN PROGRESS</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="COMPLETED">COMPLETED</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="REVIEW_FORM">Review Form</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="OK_MERGE">OK Merge</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="MERGE_CONFLICTS">Merge Conflicts</th>
\n
'
)
self
.
f_html
.
write
(
'</tr>
\n
'
)
def
MR_terminateHTML
(
self
):
self
.
f_html
.
write
(
'</table>
\n
'
)
self
.
f_html
.
write
(
'</body>
\n
'
)
self
.
f_html
.
write
(
'</html>
\n
'
)
self
.
f_html
.
close
()
def
MR_rowHTML
(
self
,
row
):
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td><a href=
\"
'
+
row
[
0
]
+
'
\"
>'
+
row
[
1
]
+
'</a></td>
\n
'
)
self
.
f_html
.
write
(
'<td>'
+
row
[
2
]
+
'</td>
\n
'
)
self
.
f_html
.
write
(
'<td>'
+
row
[
3
]
+
'</td>
\n
'
)
self
.
f_html
.
write
(
'<td class="title_cell">'
+
row
[
4
]
+
'</td>
\n
'
)
self
.
f_html
.
write
(
'<td>'
+
row
[
5
]
+
'</td>
\n
'
)
self
.
f_html
.
write
(
'<td>'
+
row
[
6
]
+
'</td>
\n
'
)
if
row
[
7
]
==
'X'
:
self
.
f_html
.
write
(
'<td style="background-color: orange;">'
+
row
[
7
]
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
if
row
[
8
]
==
'X'
:
self
.
f_html
.
write
(
'<td style="background-color: yellow;">'
+
row
[
8
]
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
if
row
[
9
]
==
'X'
:
self
.
f_html
.
write
(
'<td style="background-color: rgb(144, 221, 231);">'
+
row
[
9
]
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
if
row
[
10
]
==
'X'
:
self
.
f_html
.
write
(
'<td style="background-color: rgb(58, 236, 58);">'
+
row
[
10
]
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
if
row
[
11
]
==
'X'
:
self
.
f_html
.
write
(
'<td style="background-color: rgb(58, 236, 58);">'
+
row
[
11
]
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
if
row
[
12
]
==
'YES'
:
self
.
f_html
.
write
(
'<td style="background-color: red;">'
+
row
[
12
]
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
self
.
f_html
.
write
(
'</tr>
\n
'
)
def
Build
(
self
,
type
,
mr
,
htmlfilename
):
if
type
==
'MR'
:
self
.
Build_MR_Table
(
htmlfilename
)
elif
type
==
'Tests'
:
self
.
Build_Test_Table
(
htmlfilename
)
elif
type
==
'singleMR'
:
self
.
Build_singleMR_Table
(
mr
,
htmlfilename
)
else
:
print
(
"Undefined Dashboard Type, options : MR or Tests"
)
def
Build_Test_Table
(
self
,
htmlfilename
):
print
(
"Building Tests Dashboard..."
)
self
.
f_html
=
open
(
htmlfilename
,
'w'
)
###update date/time, format dd/mm/YY H:M:S
now
=
datetime
.
now
()
dt_string
=
now
.
strftime
(
"%d/%m/%Y %H:%M"
)
#HTML table header
self
.
Test_initHTML
(
dt_string
)
#1 table per MR if test results exist
for
x
in
range
(
len
(
self
.
git
)):
mr
=
str
(
self
.
git
[
x
][
'iid'
])
if
'PASS'
not
in
self
.
db
[
mr
]:
self
.
f_html
.
write
(
'<h3><a href="https://gitlab.eurecom.fr/oai/openairinterface5g/-/merge_requests/'
+
mr
+
'">'
+
mr
+
'</a>'
+
' '
+
self
.
git
[
x
][
'title'
]
+
'</h3>
\n
'
)
self
.
f_html
.
write
(
'<table class="Test_Table">
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<th class="Test_Name">Test Name</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Test_Descr">Bench</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Test_Descr">Test</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Pass"># Pass</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Fail"># Fail</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Last_Pass">Last Pass</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Last_Fail">Last Fail</th>
\n
'
)
self
.
f_html
.
write
(
'</tr>
\n
'
)
#parsing the tests
for
t
in
self
.
tests
:
row
=
[]
short_name
=
t
hyperlink
=
self
.
tests
[
t
][
'link'
]
job
=
self
.
tests
[
t
][
'job'
]
if
job
in
self
.
db
[
mr
]:
if
'PASS'
in
self
.
db
[
mr
][
job
]:
row
.
append
(
self
.
db
[
mr
][
job
][
'PASS'
])
else
:
row
.
append
(
''
)
if
'FAIL'
in
self
.
db
[
mr
][
job
]:
row
.
append
(
self
.
db
[
mr
][
job
][
'FAIL'
])
else
:
row
.
append
(
''
)
#2 columns for last_pass and last_fail links
if
'last_pass'
in
self
.
db
[
mr
][
job
]:
lastpasshyperlink
=
self
.
db
[
mr
][
job
][
'last_pass'
][
1
]
lastpasstext
=
self
.
db
[
mr
][
job
][
'last_pass'
][
0
]
else
:
lastpasshyperlink
=
''
lastpasstext
=
''
if
'last_fail'
in
self
.
db
[
mr
][
job
]:
lastfailhyperlink
=
self
.
db
[
mr
][
job
][
'last_fail'
][
1
]
lastfailtext
=
self
.
db
[
mr
][
job
][
'last_fail'
][
0
]
else
:
lastfailhyperlink
=
''
lastfailtext
=
''
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td><a href='
+
hyperlink
+
'>'
+
short_name
+
'</a></td>
\n
'
)
self
.
f_html
.
write
(
'<td>'
+
self
.
tests
[
t
][
'bench'
]
+
'</td>
\n
'
)
self
.
f_html
.
write
(
'<td>'
+
self
.
tests
[
t
][
'test'
]
+
'</td>
\n
'
)
if
row
[
0
]
!=
''
:
self
.
f_html
.
write
(
'<td style="background-color: rgb(58, 236, 58);">'
+
str
(
row
[
0
])
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
if
row
[
1
]
!=
''
:
self
.
f_html
.
write
(
'<td style="background-color: red;">'
+
str
(
row
[
1
])
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
self
.
f_html
.
write
(
'<td><a href='
+
lastpasshyperlink
+
'>'
+
lastpasstext
+
'</a></td>
\n
'
)
self
.
f_html
.
write
(
'<td><a href='
+
lastfailhyperlink
+
'>'
+
lastfailtext
+
'</a></td>
\n
'
)
self
.
f_html
.
write
(
'</tr>
\n
'
)
self
.
f_html
.
write
(
'</table>
\n
'
)
#terminate HTML table and close file
self
.
Test_terminateHTML
()
def
Build_singleMR_Table
(
self
,
singlemr
,
htmlfilename
):
print
(
"Building single MR Tests Results..."
)
self
.
f_html
=
open
(
htmlfilename
,
'w'
)
###update date/time, format dd/mm/YY H:M:S
now
=
datetime
.
now
()
dt_string
=
now
.
strftime
(
"%d/%m/%Y %H:%M"
)
#HTML table header
self
.
singleMR_initHTML
(
dt_string
)
#1 table per MR if test results exist => 1 table for matching mr
for
x
in
range
(
len
(
self
.
git
)):
mr
=
str
(
self
.
git
[
x
][
'iid'
])
if
mr
==
singlemr
:
#if 'PASS' not in self.db[mr]:
self
.
f_html
.
write
(
'<h3><a href="https://gitlab.eurecom.fr/oai/openairinterface5g/-/merge_requests/'
+
mr
+
'">'
+
mr
+
'</a>'
+
' '
+
self
.
git
[
x
][
'title'
]
+
'</h3>
\n
'
)
self
.
f_html
.
write
(
'<table class="Test_Table">
\n
'
)
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<th class="Test_Name">Test Name</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Test_Descr">Bench</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Test_Descr">Test</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Pass"># Pass</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Fail"># Fail</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Last_Pass">Last Pass</th>
\n
'
)
self
.
f_html
.
write
(
'<th class="Last_Fail">Last Fail</th>
\n
'
)
self
.
f_html
.
write
(
'</tr>
\n
'
)
#parsing the tests
for
t
in
self
.
tests
:
row
=
[]
short_name
=
t
hyperlink
=
self
.
tests
[
t
][
'link'
]
job
=
self
.
tests
[
t
][
'job'
]
if
job
in
self
.
db
[
mr
]:
if
'PASS'
in
self
.
db
[
mr
][
job
]:
row
.
append
(
self
.
db
[
mr
][
job
][
'PASS'
])
else
:
row
.
append
(
''
)
if
'FAIL'
in
self
.
db
[
mr
][
job
]:
row
.
append
(
self
.
db
[
mr
][
job
][
'FAIL'
])
else
:
row
.
append
(
''
)
#2 columns for last_pass and last_fail links
if
'last_pass'
in
self
.
db
[
mr
][
job
]:
lastpasshyperlink
=
self
.
db
[
mr
][
job
][
'last_pass'
][
1
]
lastpasstext
=
self
.
db
[
mr
][
job
][
'last_pass'
][
0
]
else
:
lastpasshyperlink
=
''
lastpasstext
=
''
if
'last_fail'
in
self
.
db
[
mr
][
job
]:
lastfailhyperlink
=
self
.
db
[
mr
][
job
][
'last_fail'
][
1
]
lastfailtext
=
self
.
db
[
mr
][
job
][
'last_fail'
][
0
]
else
:
lastfailhyperlink
=
''
lastfailtext
=
''
self
.
f_html
.
write
(
'<tr>
\n
'
)
self
.
f_html
.
write
(
'<td><a href='
+
hyperlink
+
'>'
+
short_name
+
'</a></td>
\n
'
)
self
.
f_html
.
write
(
'<td>'
+
self
.
tests
[
t
][
'bench'
]
+
'</td>
\n
'
)
self
.
f_html
.
write
(
'<td>'
+
self
.
tests
[
t
][
'test'
]
+
'</td>
\n
'
)
if
row
[
0
]
!=
''
:
self
.
f_html
.
write
(
'<td style="background-color: rgb(58, 236, 58);">'
+
str
(
row
[
0
])
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
if
row
[
1
]
!=
''
:
self
.
f_html
.
write
(
'<td style="background-color: red;">'
+
str
(
row
[
1
])
+
'</td>
\n
'
)
else
:
self
.
f_html
.
write
(
'<td></td>
\n
'
)
self
.
f_html
.
write
(
'<td><a href='
+
lastpasshyperlink
+
'>'
+
lastpasstext
+
'</a></td>
\n
'
)
self
.
f_html
.
write
(
'<td><a href='
+
lastfailhyperlink
+
'>'
+
lastfailtext
+
'</a></td>
\n
'
)
self
.
f_html
.
write
(
'</tr>
\n
'
)
self
.
f_html
.
write
(
'</table>
\n
'
)
#terminate HTML table and close file
self
.
Test_terminateHTML
()
def
Build_MR_Table
(
self
,
htmlfilename
):
print
(
"Building Merge Requests Dashboard..."
)
self
.
f_html
=
open
(
htmlfilename
,
'w'
)
###update date/time, format dd/mm/YY H:M:S
now
=
datetime
.
now
()
dt_string
=
now
.
strftime
(
"%d/%m/%Y %H:%M"
)
#HTML table header
self
.
MR_initHTML
(
dt_string
)
###MR data lines
for
x
in
range
(
len
(
self
.
git
)):
hyperlink
=
'https://gitlab.eurecom.fr/oai/openairinterface5g/-/merge_requests/'
+
str
(
self
.
git
[
x
][
'iid'
])
text
=
str
(
self
.
git
[
x
][
'iid'
])
date_time_str
=
self
.
git
[
x
][
'created_at'
]
date_time_obj
=
datetime
.
strptime
(
date_time_str
,
'%Y-%m-%dT%H:%M:%S.%fZ'
)
milestone1
=
milestone2
=
milestone3
=
milestone4
=
""
if
self
.
git
[
x
][
'milestone'
]
!=
None
:
if
self
.
git
[
x
][
'milestone'
][
'title'
]
==
"REVIEW_CAN_START"
:
milestone1
=
"X"
elif
self
.
git
[
x
][
'milestone'
][
'title'
]
==
"REVIEW_IN_PROGRESS"
:
milestone2
=
"X"
elif
self
.
git
[
x
][
'milestone'
][
'title'
]
==
"REVIEW_COMPLETED_AND_APPROVED"
:
milestone3
=
"X"
elif
self
.
git
[
x
][
'milestone'
][
'title'
]
==
"OK_TO_BE_MERGED"
:
milestone4
=
"X"
else
:
pass
else
:
pass
#check if empty or not
if
self
.
git
[
x
][
'assignee'
]
!=
None
:
assignee
=
str
(
self
.
git
[
x
][
'assignee'
][
'name'
])
else
:
assignee
=
""
#check if empty or not
if
len
(
self
.
git
[
x
][
'reviewers'
])
!=
0
:
reviewer
=
str
(
self
.
git
[
x
][
'reviewers'
][
0
][
'name'
])
else
:
reviewer
=
""
if
self
.
git
[
x
][
'has_conflicts'
]
==
True
:
conflicts
=
"YES"
else
:
conflicts
=
""
#add a column flagging that the review form is present
#we use gitlab API to parse the MR notes
gl
=
gitlab
.
Gitlab
.
from_config
(
'OAI'
)
project_id
=
223
project
=
gl
.
projects
.
get
(
project_id
)
#get the opened MR in the project
mrs
=
project
.
mergerequests
.
list
(
state
=
'opened'
,
per_page
=
100
)
review_form
=
''
for
m
in
range
(
0
,
len
(
mrs
)):
if
mrs
[
m
].
iid
==
self
.
git
[
x
][
'iid'
]:
#check the iid is the one we are on
mr_notes
=
mrs
[
m
].
notes
.
list
(
all
=
True
)
n
=
0
found
=
False
while
found
==
False
and
n
<
len
(
mr_notes
):
res
=
re
.
search
(
'Code Review by'
,
mr_notes
[
n
].
body
)
#this is the marker we are looking for in all notes
if
res
!=
None
:
review_form
=
"X"
found
=
True
n
+=
1
#build final row to be inserted
row
=
[
hyperlink
,
text
,
str
(
date_time_obj
.
date
()),
str
(
self
.
git
[
x
][
'author'
][
'name'
]),
str
(
self
.
git
[
x
][
'title'
]),
\
assignee
,
reviewer
,
\
milestone1
,
milestone2
,
milestone3
,
review_form
,
milestone4
,
conflicts
]
self
.
MR_rowHTML
(
row
)
#terminate HTML table and close file
self
.
MR_terminateHTML
()
def
CopyToS3
(
self
,
htmlfilename
,
bucket
,
key
):
print
(
"Uploading to S3 bucket"
)
#Creating Session With Boto3.
s3
=
boto3
.
client
(
's3'
)
#Creating S3 Resource From the Session.
result
=
s3
.
upload_file
(
htmlfilename
,
bucket
,
key
,
ExtraArgs
=
{
'ACL'
:
'public-read'
,
'ContentType'
:
'text/html'
})
#unused
def
CopyCSS
(
self
,
path
):
s3
=
boto3
.
resource
(
's3'
)
copy_source
=
{
'Bucket'
:
'oaitestdashboard'
,
'Key'
:
'test_styles.css'
}
s3
.
meta
.
client
.
copy
(
copy_source
,
'oaitestdashboard'
,
path
+
'/'
+
'test_styles.css'
)
def
PostGitNote
(
self
,
mr
,
commit
,
args
):
#current date and time to be posted with test results
#now = datetime.now()
#dt_string = now.strftime("%d/%m/%Y %H:%M")
if
len
(
args
)
%
4
!=
0
:
print
(
"Wrong Number of Arguments"
)
return
...
...
@@ -528,7 +74,7 @@ class Dashboard:
editable_mr
=
project
.
mergerequests
.
get
(
int
(
mr
))
mr_notes
=
editable_mr
.
notes
.
list
(
all
=
True
)
body
=
'[Consolidated Test Results]
(https://oaitestdashboard.s3.eu-west-1.amazonaws.com/MR'
+
mr
+
'/index.html)
\\\n
'
body
=
'[Consolidated Test Results]
\\\n
'
body
+=
'Tested CommitID: '
+
commit
for
i
in
range
(
0
,
n_tests
):
...
...
@@ -544,54 +90,16 @@ class Dashboard:
})
editable_mr
.
save
()
def
AWSCleanup
(
self
,
mode
):
#first build MR list from aws S3 bucket
if
mode
!=
'report'
and
mode
!=
'delete'
:
print
(
"incorrect mode for awsclean"
)
return
aws_mr_list
=
[]
s3
=
boto3
.
resource
(
's3'
)
my_bucket
=
s3
.
Bucket
(
'oaitestdashboard'
)
for
my_bucket_object
in
my_bucket
.
objects
.
all
():
#MR objects are like MR1407/index.html
res
=
re
.
search
(
r'^MR([0-9]+)'
,
my_bucket_object
.
key
)
if
res
!=
None
:
aws_mr_list
.
append
(
res
.
group
(
1
))
#store MR number as a string
#open MR list from GIt already exists as an attribute of this class self.mr_list
#parse aws MR list and delete those MR that are no longer open
for
aws_mr
in
aws_mr_list
:
if
aws_mr
not
in
self
.
mr_list
:
if
mode
==
"report"
:
print
(
aws_mr
+
' can be deleted from AWS S3'
)
else
:
awspath
=
"MR"
+
aws_mr
+
"/"
print
(
'deleting '
+
aws_mr
)
my_bucket
.
objects
.
filter
(
Prefix
=
awspath
).
delete
()
def
main
():
#call from slave Jenkinsfile : sh "python3 Hdashboard.py testevent ${params.eNB_MR} "
#call from master Jenkinsfile : sh "python3 Hdashboard.py gitpost ${GitPostArgs}"
if
len
(
sys
.
argv
)
>
1
:
if
len
(
sys
.
argv
)
>
1
:
#individual MR test results + test dashboard, event based (end of slave jenkins pipeline)
if
sys
.
argv
[
1
]
==
"testevent"
:
mr
=
sys
.
argv
[
2
]
htmlDash
=
Dashboard
()
if
mr
in
htmlDash
.
mr_list
:
#single MR test results
htmlDash
.
Build
(
'singleMR'
,
mr
,
'/tmp/MR'
+
mr
+
'_index.html'
)
htmlDash
.
CopyToS3
(
'/tmp/MR'
+
mr
+
'_index.html'
,
'oaitestdashboard'
,
'MR'
+
mr
+
'/index.html'
)
#all MR test results
htmlDash
.
Build
(
'Tests'
,
'0000'
,
'/tmp/Tests_index.html'
)
htmlDash
.
CopyToS3
(
'/tmp/Tests_index.html'
,
'oaitestdashboard'
,
'index.html'
)
if
sys
.
argv
[
1
]
!=
"gitpost"
:
print
(
"error: only gitpost subcommand is supported"
)
exit
(
1
)
#git post with MR test results, event based (end of master jenkins pipeline)
elif
sys
.
argv
[
1
]
==
"gitpost"
:
mr
=
sys
.
argv
[
2
]
commit
=
sys
.
argv
[
3
]
...
...
@@ -603,25 +111,15 @@ def main():
htmlDash
.
PostGitNote
(
mr
,
commit
,
args
)
else
:
print
(
"Not a Merge Request => this build is for testing/debug purpose, no report to git"
)
elif
sys
.
argv
[
1
]
==
"awsclean"
:
mode
=
sys
.
argv
[
2
]
#report or delete
htmlDash
=
Dashboard
()
htmlDash
.
AWSCleanup
(
mode
)
else
:
print
(
"Wrong argument at position 1"
)
exit
(
1
)
#test and MR status dashboards, cron based
else
:
htmlDash
=
Dashboard
()
#all MR status dashboard
htmlDash
.
Build
(
'MR'
,
'0000'
,
'/tmp/MR_index.html'
)
htmlDash
.
CopyToS3
(
'/tmp/MR_index.html'
,
'oairandashboard'
,
'index.html'
)
#all MR test results
htmlDash
.
Build
(
'Tests'
,
'0000'
,
'/tmp/Tests_index.html'
)
htmlDash
.
CopyToS3
(
'/tmp/Tests_index.html'
,
'oaitestdashboard'
,
'index.html'
)
print
(
"error: only gitpost subcommand is supported"
)
exit
(
1
)
if
__name__
==
"__main__"
:
# execute only if run as a script
...
...
This diff is collapsed.
Click to expand it.
ci-scripts/ran_dashboard/mr_styles.css
deleted
100755 → 0
View file @
adb02aa2
body
{
font-family
:
'lato'
,
sans-serif
;
}
.Main
{
text-align
:
left
;
font-size
:
30px
;
font-weight
:
bold
;
}
.DashLink
{
text-align
:
left
;
font-size
:
16px
;
}
.DashLink
:hover
{
font-weight
:
bold
;
}
.Date
{
text-align
:
left
;
font-size
:
16px
;
font-weight
:
bold
;
}
a
{
text-decoration
:
none
;
}
a
:visited
{
text-decoration
:
none
;
color
:
blue
}
a
:hover
{
text-decoration
:
none
;
font-weight
:
bold
;
color
:
blue
}
.MR_Table
{
border-collapse
:
collapse
;
}
.MR_Table
tr
{
font-size
:
12px
;
text-align
:
center
;
}
.MR_Table
tr
:hover
{
background-color
:
lightgray
;
}
.MR_Table
td
{
border
:
1px
solid
black
;
padding
:
5px
;
}
.MR_Table
td
:hover
{
font-weight
:
bold
;
}
.MR_Table
th
{
font-size
:
14px
;
text-align
:
center
;
padding
:
5px
;
}
.title_cell
{
text-align
:
left
;
}
.MR
{
table-layout
:
fixed
;
width
:
50px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.CREATED_AT
{
table-layout
:
fixed
;
width
:
100px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.AUTHOR
{
table-layout
:
fixed
;
width
:
150px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.TITLE
{
table-layout
:
fixed
;
width
:
500px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.ASSIGNEE
{
table-layout
:
fixed
;
width
:
150px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.REVIEWER
{
table-layout
:
fixed
;
width
:
150px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.CAN_START
{
background-color
:
orange
;
table-layout
:
fixed
;
width
:
150px
;
}
.IN_PROGRESS
{
background-color
:
yellow
;
table-layout
:
fixed
;
width
:
150px
;
}
.COMPLETED
{
background-color
:
rgb
(
144
,
221
,
231
);
table-layout
:
fixed
;
width
:
150px
;
}
.REVIEW_FORM
{
table-layout
:
fixed
;
width
:
100px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.OK_MERGE
{
background-color
:
rgb
(
58
,
236
,
58
);
table-layout
:
fixed
;
width
:
150px
;
}
.MERGE_CONFLICTS
{
table-layout
:
fixed
;
width
:
100px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
This diff is collapsed.
Click to expand it.
ci-scripts/ran_dashboard/ran_dashboard.py
deleted
100644 → 0
View file @
adb02aa2
#/*
# * Licensed to the OpenAirInterface (OAI) Software Alliance under one or more
# * contributor license agreements. See the NOTICE file distributed with
# * this work for additional information regarding copyright ownership.
# * The OpenAirInterface Software Alliance licenses this file to You under
# * the OAI Public License, Version 1.1 (the "License"); you may not use this file
# * except in compliance with the License.
# * You may obtain a copy of the License at
# *
# * http://www.openairinterface.org/?page_id=698
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# *-------------------------------------------------------------------------------
# * For more information about the OpenAirInterface (OAI) Software Alliance:
# * contact@openairinterface.org
# */
#---------------------------------------------------------------------
# Merge Requests Dashboard for RAN on googleSheet
#
# Required Python Version
# Python 3.x
#
#---------------------------------------------------------------------
#-----------------------------------------------------------
# Import
#-----------------------------------------------------------
#author Remi
#import google spreadsheet API
import
gspread
from
oauth2client.service_account
import
ServiceAccountCredentials
import
subprocess
import
shlex
#lexical analysis
import
json
#json structures
import
datetime
#now() and date formating
from
datetime
import
datetime
import
re
import
gitlab
import
yaml
import
os
import
pickle
import
time
from
sqlconnect
import
SQLConnect
#-----------------------------------------------------------
# Class Declaration
#-----------------------------------------------------------
class
gDashboard
:
def
__init__
(
self
,
creds_file
,
spreadsheet
,
worksheet
):
#"creds.json", 'OAI RAN Dashboard', 'MR Status'
self
.
scope
=
[
"https://spreadsheets.google.com/feeds"
,
'https://www.googleapis.com/auth/spreadsheets'
,
"https://www.googleapis.com/auth/drive.file"
,
"https://www.googleapis.com/auth/drive"
]
self
.
creds
=
ServiceAccountCredentials
.
from_json_keyfile_name
(
creds_file
,
self
.
scope
)
self
.
client
=
gspread
.
authorize
(
self
.
creds
)
#spreadsheet
self
.
ss
=
self
.
client
.
open
(
spreadsheet
)
#worksheet
self
.
sheet
=
self
.
ss
.
worksheet
(
worksheet
)
self
.
ss
.
del_worksheet
(
self
.
sheet
)
#start by deleting the old sheet
self
.
sheet
=
self
.
ss
.
add_worksheet
(
title
=
worksheet
,
rows
=
"100"
,
cols
=
"30"
)
#create a new one
#init with data sources : git, yaml config file, test results databases
cmd
=
"""curl --silent "https://gitlab.eurecom.fr/api/v4/projects/oai%2Fopenairinterface5g/merge_requests?state=opened&per_page=100" """
self
.
git
=
self
.
__getGitData
(
cmd
)
#git data from Gitlab
self
.
tests
=
self
.
__loadCfg
(
'ran_dashboard_cfg.yaml'
)
#tests table setup from yaml
self
.
db
=
self
.
__loadFromDB
()
#test results from database
def
__loadCfg
(
self
,
yaml_file
):
with
open
(
yaml_file
,
'r'
)
as
f
:
tests
=
yaml
.
load
(
f
)
return
tests
def
__getGitData
(
self
,
cmd
):
#cmd="""curl --silent "https://gitlab.eurecom.fr/api/v4/projects/oai%2Fopenairinterface5g/merge_requests?state=opened&per_page=100" """
process
=
subprocess
.
Popen
(
shlex
.
split
(
cmd
),
stdout
=
subprocess
.
PIPE
)
output
=
process
.
stdout
.
readline
()
tmp
=
output
.
decode
(
"utf-8"
)
d
=
json
.
loads
(
tmp
)
return
d
def
__loadFromDB
(
self
):
mr_list
=
[]
for
x
in
range
(
len
(
self
.
git
)):
mr_list
.
append
(
str
(
self
.
git
[
x
][
'iid'
]))
mydb
=
SQLConnect
()
for
MR
in
mr_list
:
mydb
.
get
(
MR
)
mydb
.
close_connection
()
return
mydb
.
data
def
gBuild
(
self
,
destinationSheetName
):
###line 1 : update date/time, format dd/mm/YY H:M:S
now
=
datetime
.
now
()
dt_string
=
"Update : "
+
now
.
strftime
(
"%d/%m/%Y %H:%M"
)
row
=
[
dt_string
]
self
.
sheet
.
insert_row
(
row
,
index
=
1
,
value_input_option
=
'RAW'
)
###line 2 is for the test short names (links to jenkins pipeline), updated at the end
###line 3 is for the column names
i
=
3
row
=
[
"MR"
,
"Created_at"
,
"Author"
,
"Title"
,
"Assignee"
,
"Reviewer"
,
"CAN START"
,
"IN PROGRESS"
,
"COMPLETED"
,
"Review Form"
,
"OK MERGE"
,
"Merge conflicts"
,
""
]
#tests
for
t
in
range
(
0
,
len
(
self
.
tests
)):
row
.
append
(
"# PASS"
)
row
.
append
(
"# FAIL"
)
row
.
append
(
"Last Pass"
)
row
.
append
(
"Last Fail"
)
self
.
sheet
.
insert_row
(
row
,
index
=
i
,
value_input_option
=
'RAW'
)
###line 4 onward, MR data lines
for
x
in
range
(
len
(
self
.
git
)):
i
=
i
+
1
date_time_str
=
self
.
git
[
x
][
'created_at'
]
date_time_obj
=
datetime
.
strptime
(
date_time_str
,
'%Y-%m-%dT%H:%M:%S.%fZ'
)
milestone1
=
milestone2
=
milestone3
=
milestone4
=
""
if
self
.
git
[
x
][
'milestone'
]
!=
None
:
if
self
.
git
[
x
][
'milestone'
][
'title'
]
==
"REVIEW_CAN_START"
:
milestone1
=
"X"
elif
self
.
git
[
x
][
'milestone'
][
'title'
]
==
"REVIEW_IN_PROGRESS"
:
milestone2
=
"X"
elif
self
.
git
[
x
][
'milestone'
][
'title'
]
==
"REVIEW_COMPLETED_AND_APPROVED"
:
milestone3
=
"X"
elif
self
.
git
[
x
][
'milestone'
][
'title'
]
==
"OK_TO_BE_MERGED"
:
milestone4
=
"X"
else
:
pass
else
:
pass
#check if empty or not
if
self
.
git
[
x
][
'assignee'
]
!=
None
:
assignee
=
str
(
self
.
git
[
x
][
'assignee'
][
'name'
])
else
:
assignee
=
""
#check if empty or not
if
len
(
self
.
git
[
x
][
'reviewers'
])
!=
0
:
reviewer
=
str
(
self
.
git
[
x
][
'reviewers'
][
0
][
'name'
])
else
:
reviewer
=
""
if
self
.
git
[
x
][
'has_conflicts'
]
==
True
:
conflicts
=
"YES"
else
:
conflicts
=
""
#add a column flagging that the review form is present
#we use gitlab API to parse the MR notes
gl
=
gitlab
.
Gitlab
.
from_config
(
'OAI'
)
project_id
=
223
project
=
gl
.
projects
.
get
(
project_id
)
#get the opened MR in the project
mrs
=
project
.
mergerequests
.
list
(
state
=
'opened'
,
per_page
=
100
)
review_form
=
''
for
m
in
range
(
0
,
len
(
mrs
)):
if
mrs
[
m
].
iid
==
self
.
git
[
x
][
'iid'
]:
#check the iid is the one we are on
mr_notes
=
mrs
[
m
].
notes
.
list
(
all
=
True
)
n
=
0
found
=
False
while
found
==
False
and
n
<
len
(
mr_notes
):
res
=
re
.
search
(
'Code Review by'
,
mr_notes
[
n
].
body
)
#this is the marker we are looking for in all notes
if
res
!=
None
:
review_form
=
"X"
found
=
True
n
+=
1
#build final row to be inserted, the first column is left empty for now, will be filled afterward with hyperlinks to gitlab MR
row
=
[
""
,
str
(
date_time_obj
.
date
()),
str
(
self
.
git
[
x
][
'author'
][
'name'
]),
str
(
self
.
git
[
x
][
'title'
]),
\
assignee
,
reviewer
,
\
milestone1
,
milestone2
,
milestone3
,
review_form
,
milestone4
,
conflicts
,
""
]
#and append the test results coming from self.db
mr
=
str
(
self
.
git
[
x
][
'iid'
])
for
t
in
self
.
tests
:
if
mr
in
self
.
db
:
job
=
self
.
tests
[
t
][
'job'
]
if
job
in
self
.
db
[
mr
]:
if
'PASS'
in
self
.
db
[
mr
][
job
]:
row
.
append
(
self
.
db
[
mr
][
job
][
'PASS'
])
else
:
row
.
append
(
''
)
if
'FAIL'
in
self
.
db
[
mr
][
job
]:
row
.
append
(
self
.
db
[
mr
][
job
][
'FAIL'
])
else
:
row
.
append
(
''
)
#leave 2 columns for last_pass and last_fail links
row
.
append
(
''
)
row
.
append
(
''
)
else
:
#4 columns are empty
row
.
append
(
''
)
row
.
append
(
''
)
row
.
append
(
''
)
row
.
append
(
''
)
#insert the final row to worksheet
self
.
sheet
.
insert_row
(
row
,
index
=
i
,
value_input_option
=
'RAW'
)
time
.
sleep
(
10
)
#add MR hyperlinks in a list of requests to be sent as one update batch; this to save API calls (quotas)
i
=
3
requests
=
[]
for
x
in
range
(
len
(
self
.
git
)):
rowIndex
=
i
colIndex
=
0
hyperlink
=
'
\"
'
+
"https://gitlab.eurecom.fr/oai/openairinterface5g/-/merge_requests/"
+
str
(
self
.
git
[
x
][
'iid'
])
+
'
\"
'
text
=
'
\"
'
+
str
(
self
.
git
[
x
][
'iid'
])
+
'"'
requests
.
append
(
self
.
addHyperlink
(
hyperlink
,
text
,
destinationSheetName
,
rowIndex
,
colIndex
))
mr
=
str
(
self
.
git
[
x
][
'iid'
])
colIndex
=
15
for
t
in
self
.
tests
:
job
=
self
.
tests
[
t
][
'job'
]
if
job
in
self
.
db
[
mr
]:
if
'last_pass'
in
self
.
db
[
mr
][
job
]:
hyperlink
=
'
\"
'
+
self
.
db
[
mr
][
job
][
'last_pass'
][
1
]
+
'
\"
'
text
=
'
\"
'
+
self
.
db
[
mr
][
job
][
'last_pass'
][
0
]
+
'"'
requests
.
append
(
self
.
addHyperlink
(
hyperlink
,
text
,
destinationSheetName
,
rowIndex
,
colIndex
))
if
'last_fail'
in
self
.
db
[
mr
][
job
]:
hyperlink
=
'
\"
'
+
self
.
db
[
mr
][
job
][
'last_fail'
][
1
]
+
'
\"
'
text
=
'
\"
'
+
self
.
db
[
mr
][
job
][
'last_fail'
][
0
]
+
'"'
requests
.
append
(
self
.
addHyperlink
(
hyperlink
,
text
,
destinationSheetName
,
rowIndex
,
colIndex
+
1
))
colIndex
+=
4
#move to next test
i
=
i
+
1
#increment row index for next MR
body
=
{
"requests"
:
requests
}
self
.
ss
.
batch_update
(
body
)
###line 2 is for the test names
#add MR hyperlinks in a list of requests to be sent as one update batch; this to save API calls (quotas)
requests
=
[]
rowIndex
=
1
colIndex
=
13
for
t
in
self
.
tests
:
hyperlink
=
'
\"
'
+
self
.
tests
[
t
][
'link'
]
+
'
\"
'
short_name
=
'
\"
'
+
t
+
'
\"
'
requests
.
append
(
self
.
addHyperlink
(
hyperlink
,
short_name
,
destinationSheetName
,
rowIndex
,
colIndex
))
colIndex
+=
4
body
=
{
"requests"
:
requests
}
self
.
ss
.
batch_update
(
body
)
def
addHyperlink
(
self
,
hyperlink
,
text
,
destinationSheetName
,
rowIndex
,
colIndex
):
sheetId
=
self
.
ss
.
worksheet
(
destinationSheetName
).
_properties
[
'sheetId'
]
request
=
\
{
"updateCells"
:
{
"rows"
:
[
{
"values"
:
[
{
"userEnteredValue"
:
{
"formulaValue"
:
"=HYPERLINK({},{})"
.
format
(
hyperlink
,
text
)
}
}
]
}
],
"fields"
:
"userEnteredValue"
,
"start"
:
{
"sheetId"
:
sheetId
,
"rowIndex"
:
rowIndex
,
"columnIndex"
:
colIndex
}
}
}
return
request
def
gFormat
(
self
,
sourceSheetName
,
destinationSheetName
):
# "Formating" , "MR Status"
#the requests are appended in a list of requests to be sent as one update batch; this to save API calls (quotas)
#copy formating template
sourceSheetId
=
self
.
ss
.
worksheet
(
sourceSheetName
).
_properties
[
'sheetId'
]
destinationSheetId
=
self
.
ss
.
worksheet
(
destinationSheetName
).
_properties
[
'sheetId'
]
requests
=
[]
requests
.
append
(
{
"copyPaste"
:
{
"source"
:
{
"sheetId"
:
sourceSheetId
,
"startRowIndex"
:
1
,
"endRowIndex"
:
40
,
"startColumnIndex"
:
0
,
"endColumnIndex"
:
30
},
"destination"
:
{
"sheetId"
:
destinationSheetId
,
"startRowIndex"
:
1
,
"endRowIndex"
:
40
,
"startColumnIndex"
:
0
,
"endColumnIndex"
:
30
},
"pasteType"
:
"PASTE_FORMAT"
}
}
)
#resize columns fit to data, except col 0
sheetId
=
self
.
ss
.
worksheet
(
destinationSheetName
).
_properties
[
'sheetId'
]
requests
.
append
(
{
'autoResizeDimensions'
:
{
'dimensions'
:
{
'sheetId'
:
sheetId
,
'dimension'
:
'COLUMNS'
,
'startIndex'
:
1
,
'endIndex'
:
12
}
}
}
)
#resize col 0
sheetId
=
self
.
ss
.
worksheet
(
destinationSheetName
).
_properties
[
'sheetId'
]
requests
.
append
(
{
"updateDimensionProperties"
:
{
"range"
:
{
"sheetId"
:
sheetId
,
"dimension"
:
"COLUMNS"
,
"startIndex"
:
0
,
"endIndex"
:
1
},
"properties"
:
{
"pixelSize"
:
100
},
"fields"
:
"pixelSize"
}
}
)
#resize milestones to be cleaner
sheetId
=
self
.
ss
.
worksheet
(
destinationSheetName
).
_properties
[
'sheetId'
]
requests
.
append
(
{
"updateDimensionProperties"
:
{
"range"
:
{
"sheetId"
:
sheetId
,
"dimension"
:
"COLUMNS"
,
"startIndex"
:
6
,
"endIndex"
:
11
},
"properties"
:
{
"pixelSize"
:
120
},
"fields"
:
"pixelSize"
}
}
)
#group MR related columns
# sheetId = self.ss.worksheet(destinationSheetName)._properties['sheetId']
# requests.append(
# {
# "addDimensionGroup": {
# "range": {
# "dimension": "COLUMNS",
# "sheetId": sheetId,
# "startIndex": 3,
# "endIndex": 12
# },
# }
# }
# )
#
body
=
{
"requests"
:
requests
}
self
.
ss
.
batch_update
(
body
)
def
main
():
my_gDashboard
=
gDashboard
(
"/opt/dashboard/g_creds.json"
,
'OAI RAN Dashboard'
,
'MR Status'
)
my_gDashboard
.
gBuild
(
"MR Status"
)
my_gDashboard
.
gFormat
(
"Formating"
,
"MR Status"
)
if
__name__
==
"__main__"
:
# execute only if run as a script
main
()
This diff is collapsed.
Click to expand it.
ci-scripts/ran_dashboard/ran_dashboard_cfg.yaml
deleted
100644 → 0
View file @
adb02aa2
LTE-2x2
:
#short name used in the dashboard
job
:
'
RAN-LTE-2x2-Module-OAIEPC'
#job name from Jenkins, used in the database
link
:
'
https://jenkins-oai.eurecom.fr/view/RAN/job/RAN-LTE-2x2-Module-OAIEPC'
bench
:
'
Obelix-N310-OAIEPC-Quectel(nrmodule2)'
test
:
'
TM1
+
TM2,
TDD,
40MHz,
MCS9,
26Mb
DL,
7Mb
UL'
NSA-B200
:
job
:
'
RAN-NSA-B200-Module-LTEBOX'
link
:
'
https://jenkins-oai.eurecom.fr/view/RAN/job/RAN-NSA-B200-Module-LTEBOX'
bench
:
'
Nepes-B200-Obelix-B200-LTEBOX-Quectel(idefix)'
test
:
'
20MHz,
60Mb
DL,
3Mb
UL'
NSA-2x2
:
job
:
'
RAN-NSA-2x2-Module-OAIEPC'
link
:
'
https://jenkins-oai.eurecom.fr/view/RAN/job/RAN-NSA-2x2-Module-OAIEPC'
bench
:
'
Asterix-N310-Obelix-N310-OAIEPC-Quectel(nrmodule2)'
test
:
'
TDD,
40MHz,
60Mb
DL,
3Mb
UL'
SA-N310
:
job
:
'
RAN-SA-Module-CN5G'
link
:
'
https://jenkins-oai.eurecom.fr/view/RAN/job/RAN-SA-Module-CN5G'
bench
:
'
Asterix-N310-OAICN5G-Quectel(nrmodule2)'
test
:
'
TDD,
40MHz,
60Mb
DL,
3Mb
UL'
SA-OAIUE-N310-X300
:
job
:
'
RAN-SA-OAIUE-N310-X300-CN5G'
link
:
'
https://jenkins-oai.eurecom.fr/view/RAN/job/RAN-SA-OAIUE-N310-X300-CN5G/'
bench
:
'
Asterix-N310-OAICN5G-OAIUE-N310'
test
:
'
TDD,
40MHz,
Ping,
(to
be
implemented
:
iperf)'
This diff is collapsed.
Click to expand it.
ci-scripts/ran_dashboard/readme.txt
deleted
100644 → 0
View file @
adb02aa2
The code ran_dashboard.py was initially developped to bring MR status and test resuts to a google sheet, using google sheets API.
This method is now deprecated, and replaced by Hdashboard.py that builds the results as HTML page and loads it to AWS S3.
ran_dashboard_cfg.yaml is still used by Hdashboard.py as tests config file.
This diff is collapsed.
Click to expand it.
ci-scripts/ran_dashboard/sqlconnect.py
deleted
100644 → 0
View file @
adb02aa2
#/*
# * Licensed to the OpenAirInterface (OAI) Software Alliance under one or more
# * contributor license agreements. See the NOTICE file distributed with
# * this work for additional information regarding copyright ownership.
# * The OpenAirInterface Software Alliance licenses this file to You under
# * the OAI Public License, Version 1.1 (the "License"); you may not use this file
# * except in compliance with the License.
# * You may obtain a copy of the License at
# *
# * http://www.openairinterface.org/?page_id=698
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# *-------------------------------------------------------------------------------
# * For more information about the OpenAirInterface (OAI) Software Alliance:
# * contact@openairinterface.org
# */
#---------------------------------------------------------------------
# Merge Requests Dashboard for RAN on googleSheet
#
# Required Python Version
# Python 3.x
#
#---------------------------------------------------------------------
#author Remi
import
pymysql
import
sys
from
datetime
import
datetime
import
pickle
#This is the script/package used by the dashboard to retrieve the MR test results from the database
class
SQLConnect
:
def
__init__
(
self
):
self
.
connection
=
pymysql
.
connect
(
host
=
'172.22.0.2'
,
user
=
'root'
,
password
=
'ucZBc2XRYdvEm59F'
,
db
=
'oaicicd_tests'
,
port
=
3306
)
self
.
data
=
{}
#retrieve data from mysql database and organize it in a dictionary (per MR passed as argument)
def
get
(
self
,
MR
):
self
.
data
[
MR
]
=
{}
cur
=
self
.
connection
.
cursor
()
#get counters per test
sql
=
"select TEST,STATUS, count(*) AS COUNT from test_results where MR=(%s) group by TEST, STATUS;"
cur
.
execute
(
sql
,
MR
)
response
=
cur
.
fetchall
()
if
len
(
response
)
==
0
:
#no test results yet
self
.
data
[
MR
][
'PASS'
]
=
''
self
.
data
[
MR
][
'FAIL'
]
=
''
else
:
for
i
in
range
(
0
,
len
(
response
)):
test
=
response
[
i
][
0
]
status
=
response
[
i
][
1
]
count
=
response
[
i
][
2
]
if
test
in
self
.
data
[
MR
]:
self
.
data
[
MR
][
test
][
status
]
=
count
else
:
self
.
data
[
MR
][
test
]
=
{}
self
.
data
[
MR
][
test
][
status
]
=
count
#get last failing build and link
sql
=
"select TEST,BUILD, BUILD_LINK from test_results where MR=(%s) and STATUS='FAIL' order by DATE DESC;"
cur
.
execute
(
sql
,
MR
)
response
=
cur
.
fetchall
()
if
len
(
response
)
!=
0
:
for
i
in
range
(
0
,
len
(
response
)):
test
=
response
[
i
][
0
]
build
=
response
[
i
][
1
]
link
=
response
[
i
][
2
]
if
'last_fail'
not
in
self
.
data
[
MR
][
test
]:
self
.
data
[
MR
][
test
][
'last_fail'
]
=
[]
self
.
data
[
MR
][
test
][
'last_fail'
].
append
(
build
)
self
.
data
[
MR
][
test
][
'last_fail'
].
append
(
link
)
#get last passing build and link
sql
=
"select TEST,BUILD, BUILD_LINK from test_results where MR=(%s) and STATUS='PASS' order by DATE DESC;"
cur
.
execute
(
sql
,
MR
)
response
=
cur
.
fetchall
()
if
len
(
response
)
!=
0
:
for
i
in
range
(
0
,
len
(
response
)):
test
=
response
[
i
][
0
]
build
=
response
[
i
][
1
]
link
=
response
[
i
][
2
]
if
'last_pass'
not
in
self
.
data
[
MR
][
test
]:
self
.
data
[
MR
][
test
][
'last_pass'
]
=
[]
self
.
data
[
MR
][
test
][
'last_pass'
].
append
(
build
)
self
.
data
[
MR
][
test
][
'last_pass'
].
append
(
link
)
#close database connection
def
close_connection
(
self
):
self
.
connection
.
close
()
This diff is collapsed.
Click to expand it.
ci-scripts/ran_dashboard/test_styles.css
deleted
100755 → 0
View file @
adb02aa2
body
{
font-family
:
'lato'
,
sans-serif
;
}
.Main
{
text-align
:
left
;
font-size
:
30px
;
font-weight
:
bold
;
}
.DashLink
{
text-align
:
left
;
font-size
:
16px
;
}
.DashLink
:hover
{
font-weight
:
bold
;
}
.Date
{
text-align
:
left
;
font-size
:
16px
;
font-weight
:
bold
;
}
h3
{
font-size
:
16px
;
}
a
{
text-decoration
:
none
;
}
a
:visited
{
text-decoration
:
none
;
color
:
blue
}
a
:hover
{
text-decoration
:
none
;
font-weight
:
bold
;
color
:
blue
}
.Test_Table
{
border-collapse
:
collapse
;
}
.Test_Table
tr
{
font-size
:
12px
;
text-align
:
center
;
}
.Test_Table
tr
:hover
{
background-color
:
lightgray
;
}
.Test_Table
td
{
border
:
1px
solid
black
;
padding
:
5px
;
}
.Test_Table
td
:hover
{
font-weight
:
bold
;
}
.Test_Table
th
{
font-size
:
14px
;
text-align
:
center
;
padding
:
5px
;
}
.Test_Name
{
table-layout
:
fixed
;
width
:
100px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.Test_Descr
{
table-layout
:
fixed
;
width
:
400px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.Pass
{
table-layout
:
fixed
;
width
:
100px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.Fail
{
table-layout
:
fixed
;
width
:
100px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.Last_Pass
{
table-layout
:
fixed
;
width
:
100px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
.Last_Fail
{
table-layout
:
fixed
;
width
:
100px
;
background-color
:
rgb
(
143
,
154
,
216
);
}
This diff is collapsed.
Click to expand it.
ci-scripts/template.xsl
deleted
100644 → 0
View file @
adb02aa2
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet
version=
"2.0"
xmlns:xsl=
"http://www.w3.org/1999/XSL/Transform"
>
<xsl:output
method=
"html"
version=
"4"
encoding=
"UTF-8"
indent=
"no"
omit-xml-declaration=
"yes"
/>
<!-- main body -->
<xsl:template
match=
"/"
>
<html>
<body>
<h3>
TEMPLATE Results Summary
</h3>
<table
border=
"1"
>
<tr
bgcolor=
"lightcyan"
>
<!--Header only so select first row to get headers-->
<th>
Hostname
</th>
<th>
Nb Tests
</th>
<th>
Failures
</th>
<th>
Timestamp
</th>
</tr>
<!--Get all the other rows-->
<xsl:for-each
select=
"testsuites/testsuite"
>
<tr>
<td>
<xsl:value-of
select=
"@hostname"
/>
</td>
<td>
<xsl:value-of
select=
"@tests"
/>
</td>
<td>
<xsl:value-of
select=
"@failures"
/>
</td>
<td>
<xsl:value-of
select=
"@timestamp"
/>
</td>
</tr>
</xsl:for-each>
</table>
<h4>
Details
</h4>
<table
border=
"1"
>
<tr
bgcolor=
"lightcyan"
>
<!--Header only so select first row to get headers-->
<th>
Test Name
</th>
<th>
Result
</th>
<th>
Time
</th>
<th>
Description
</th>
</tr>
<!--Get all the other rows-->
<xsl:for-each
select=
"testsuites/testsuite/testcase"
>
<tr>
<td>
<xsl:value-of
select=
"@name"
/>
</td>
<td>
<xsl:value-of
select=
"@RESULT"
/>
</td>
<td>
<xsl:value-of
select=
"@time"
/>
</td>
<td>
<xsl:value-of
select=
"@description"
/>
</td>
</tr>
</xsl:for-each>
</table>
</body>
</html>
</xsl:template>
</xsl:stylesheet>
This diff is collapsed.
Click to expand it.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment