Create and Install a Relocating Analysis Data Server on Linux with Docker (CodeSonar 7.2 and earlier)

This page describes how to set up a relocating analysis data server to manage your CodeSonar analysis files after the analysis has completed. The analysis data server will store these files and use them to service hub requests for information such as source file listings and procedure-granularity metrics.

These instructions are designed for use with CodeSonar 7.2 and earlier. If you are using CodeSonar 7.3 or later, we recommend that you use a standard analysis data server in combination with codesonar analyze -remote-archive instead.

If you are using CodeSonar SaaS, or if your hub already has suitable associated remote analysis launch daemons, you do not need to set up a separate analysis data server.

We also provide instructions to create and install a relocating analysis data server on other systems:

Prerequisites

A CodeSonar hub.
A suitable machine for running the analysis data server.

Overview

There are three stages:

A. Prepare the hub
B. Prepare the analysis data server host machine
C. Start a remote analysis launch daemon and a GitLab Runner instance

A. Prepare the hub

Log in to your hub as Administrator, or as another user with administrative privileges.
Create a new launchd group to contain the launch daemons that you will create for your analysis data server.

For details, see the manual: Settings > Other Links > Analysis Cloud > Create New Launchd Group.

The steps below will assume that the launchd group path name is '/analysis-data-server'.
Create a hub user account that you can use to run the launch daemons and associated pipeline jobs. Make sure the user has sufficient permissions: with factory settings, it is sufficient to assign the User and Enabled roles.

The remainder of these instructions will assume that the username for this account is gitlab.

B. Prepare the analysis data server host machine

If you have not already done so, identify a suitable host machine for the analysis data server container.
- This can be a physical machine or a virtual machine.
- It must have Docker installed.
- It will not need to listen on any ports, and does not need a stable host name.
- It will make outgoing network connections to your CodeSonar hub.
- It will need plenty of disk space where you can store analysis data files.
Log in to the host machine as root.
If Docker Engine is not already installed, use your package manager to install it.

See: https://docs.docker.com/engine/install/#server for more information
Create a new analysis data service user account. This account will run the CodeSonar launch daemon process (don't run the launch daemon as root).

These instructions will assume that the CI user name is csonar_ci.
Ensure you have a user that can build and run Docker containers (don't run the Docker containers as root).

If necessary, create another new user that has permission to execute Docker commands.

These instructions will assume that the Docker user name is docker_ci.
Create directories for saving analysis data and configuration.

You will need a directory for saving CodeSonar analysis data, and one for the GitLab Runner to use for saving its configuration files.

The following instructions create these directories under a bind-mounted local directory in your analysis server container. You could use a Docker volume instead, but we will not describe this option here.

The directories should be owned by the analysis data user account you previously created (csonar_ci).

Analysis data can take plenty of space depending on the size of your code. You will need to arrange to clean up old data periodically using a cron job or similar.
```
CI_USER=csonar_ci
CI_USER_GROUP=csonar_ci
HOST_ROOT=/srv/codesonar
HOST_ANALYSIS_DATA=$HOST_ROOT/analysis_data
HOST_RUNNER_CONFIGDIR=$HOST_ROOT/analysis.server.config

mkdir -p "$HOST_ROOT" "$HOST_ANALYSIS_DATA" "$HOST_RUNNER_CONFIGDIR"
chown $CI_USER:$CI_USER_GROUP "$HOST_ROOT" "$HOST_ANALYSIS_DATA" "$HOST_RUNNER_CONFIGDIR"
```
Log in to the host machine as the Docker user (docker_ci).
Install the CodeSonar-GitLab Integration tools.

We will use these tools to create the Docker container. They are not used directly by service processes.
1. Download the CodeSonar-GitLab integration tools package (e.g. codesonar-gitlab-integration-1.6p0.tar.gz).
2. Extract the package.
```
cd /opt
tar -xzf /path/to/codesonar-gitlab-integration-1.6p0.tar.gz
```
3. Make a version-neutral symlink so that integration tools are easier for pipeline scripts to find.
```
ln -s codesonar-gitlab-integration-1.6p0  codesonar-gitlab-integration
```
The example commands above will install CodeSonar-GitLab integration tools to /opt/codesonar-gitlab-integration.

The remainder of these instructions will refer to this installation directory as <extract-dir>

Note that the extracted material includes directory <extract-dir>/examples/analysis-server.relocate.docker/: This contains example scripts that we will use in a later step.
[HTTPS hubs only] Download a copy of your hub's hub server certificate. We will use this copy to ensure that the CodeSonar command line tools in your Docker image will trust your hub.
- If your CodeSonar hub uses plain HTTP (and not secure HTTPS), skip this step.
- Otherwise:
  1. Download the hub server certificate from the hub Configure HTTPS page.
    
    See the Troubleshooting document for additional information about how to download this file.
  2. Save the certificate to your local machine in Base-64 ASCII text format (often called "PEM" format). A typical name for saving the certificate file is "cacert.pem".

C. Create Docker images

Create a base Docker image containing CodeSonar.

Change directory to <extract-dir>/distro-image, where <extract-dir> is the directory where you installed the CodeSonar-GitLab integration tools package.
Check to make sure that this directory contains a copy of the CodeSonar installer archive. The name will be something like codesonar-7.3p0.20230330-x86_64-pc-linux.tar.gz (version number and datestamp will vary).
[HTTPS hubs only] Copy your downloaded hub server certificate to this directory.
```
cp <path-to>/cacert.pem .
```

Execute the following. Make any necessary changes to variable settings before executing.

Variable	Setting
`CODESONAR_PACKAGE`	The name of the CodeSonar installer archive file located in `<extract-dir>/distro-image`.
`CODESONAR_HUB_CACERT`	For HTTPS hubs, the path to the hub server certificate that you downloaded. For HTTP hubs, leave the setting unchanged.

CODESONAR_PACKAGE=codesonar-7.3p0.20230330-x86_64-pc-linux.tar.gz
CODESONAR_IMAGE=codesonar_launchd
CODESONAR_IMAGE_VERSION=latest
CODESONAR_HUB_CACERT=cacert.pem
BASE_IMAGE=ubuntu:18.04

docker build --tag $CODESONAR_IMAGE:$CODESONAR_IMAGE_VERSION \
       --build-arg BASE_IMAGE=$BASE_IMAGE \
       --build-arg CODESONAR_PACKAGE=$CODESONAR_PACKAGE \
       --build-arg CODESONAR_HUB_CACERT=$CODESONAR_HUB_CACERT \
       --build-arg TELEMETRY=1 \
       --build-arg HUB=0 \
       --build-arg JAVA_ANALYSIS=0 \
       --build-arg PYTHON_ANALYSIS=0 \
       --build-arg ECLIPSE=0 \
       .

Create a Docker image containing GitLab Runner software.

Change directory to <extract-dir>/examples/analysis-server.relocate.docker, where <extract-dir> is the directory where you installed the CodeSonar-GitLab integration tools package.
Check the directory contents. The following files should be present.
- Dockerfile: an image built from this Dockerfile will execute the ci-runner.sh script.
- ci-runner.sh

Execute the following. Make any necessary changes to variable settings before executing.

Variable	Setting
`CI_USER`	The username of the analysis data service user account that you created in part B.

CI_USER=csonar_ci
CI_USER_HOME=/home/$CI_USER
CI_USER_UID=$(id -u $CI_USER)
CI_USER_GID=$(id -g $CI_USER)
CI_USER_GROUP=$(id -ng $CI_USER)
CODESONAR_IMAGE=codesonar_launchd
CODESONAR_IMAGE_VERSION=latest
CI_RUNNER_IMAGE=codesonar_analysis_server_runner
CI_RUNNER_IMAGE_VERSION=latest

docker image build \
  --build-arg BASE_IMAGE=$CODESONAR_IMAGE:$CODESONAR_IMAGE_VERSION \
  --build-arg CI_USER=$CI_USER \
  --build-arg CI_USER_HOME=$CI_USER_HOME \
  --build-arg CI_USER_UID=$CI_USER_UID \
  --build-arg CI_USER_GID=$CI_USER_GID \
  --build-arg CI_USER_GROUP=$CI_USER_GROUP \
  -t $CI_RUNNER_IMAGE:$CI_RUNNER_IMAGE_VERSION \
  .

Get a GitLab Runner registration token from the "CI/CD" Runner settings page on your GitLab instance. Note that you can register a runner at the project, group, or global level.

Set up hub credentials.

If your hub has HTTPS enabled, we recommend using certificate-based authentication as described here. If your hub is HTTP-only, or does not permit certificate-based authentication, you will need to modify this step to set up password-based authentication instead.

Execute the following to start a temporary Docker container that generates a hub certificate and saves it to a Docker volume. This volume will be used when starting the launch daemon container later.

Set up variables. Make any necessary changes to variable settings before executing.

Variable	Setting
`CODESONAR_HUB`	The location of your CodeSonar hub (protocol://host:port).
`CODESONAR_HUBUSER`	The username of the analysis data service user account that you created in part A.

CI_RUNNER_IMAGE=codesonar_analysis_server_runner
CI_RUNNER_IMAGE_VERSION=latest
CI_RUNNER_USER_VOL=$CI_RUNNER_IMAGE.config
CODESONAR_HUB=https://codesonar.example.com:7340
CODESONAR_HUBUSER=cshub_ci
CI_USER_HOME=/home/csonar_ci
CS_USER_HOME=$CI_USER_HOME/cs_home
CSONAR=/opt/codesonar

Create the volume, if it does not exist.

docker volume inspect $CI_RUNNER_USER_VOL  2> /dev/null || \
  docker volume create $CI_RUNNER_USER_VOL

Start the temporary container.

docker container run \
  -v $CI_RUNNER_USER_VOL:$CS_USER_HOME \
  --env CS_USER_HOME=$CS_USER_HOME \
  --rm \
  -it \
  $CI_RUNNER_IMAGE:$CI_RUNNER_IMAGE_VERSION \
  $CSONAR/codesonar/bin/codesonar generate-hub-cert -foruser "$CODESONAR_HUBUSER" "$CODESONAR_HUB"

You will be prompted to enter a hub username and password to authorize certificate creation. This does not have to be the same as the CODESONAR_HUBUSER hub user account.

Start the analysis data server container.

Execute the following. Make any necessary changes to variable settings before executing.

Variable	Setting
`HOST_ROOT`, `HOST_ANALYSIS_DATA`, and `HOST_RUNNER_CONFIGDIR`	These variables should have the same values that you used when you created the directories in part B. They are all directory paths on your Docker host machine (outside the countainer) and will be "bind mounted" into the container.
`CODESONAR_ANALYSIS_DATA`	The home directory for your launch daemon: should be a subdirectory of `CODESONAR_DATA`. Data for each analysis managed by the launch daemon will be stored under a subdirectory of this directory. The launch daemon will be responsible for creating this directory. This is a path inside the container.
`CODESONAR_HUB`	The location of your CodeSonar hub (protocol://host:port).
`CODESONAR_HUBUSER`	The username of the hub user account that you created in part A.
`CODESONAR_LAUNCHD_GROUP`	The launchd group you created in part A.
`CODESONAR_LAUNCHD_KEY`	A unique identifier to distinguish this launch daemon from any others you may wish to start on the analysis data server.
`CI_SERVER_URL`	GitLab server URL.
`CI_RUNNER_REG_TOKEN`	Provided by your GitLab site under the "CI/CD" Runner settings.
`CI_RUNNER_IMAGE`	The name of your analysis server Docker image.
`CI_RUNNER_IMAGE_VERSION`	A versioning tag for your analysis server Docker image.
`CI_RUNNER_USER_VOL`	The Docker volume where your credentials are saved.
`CS_USER_HOME`	This variable is recognized by CodeSonar and influences hub authentication commands. Do not use the same directory as `CI_USER_HOME`.

HOST_ROOT=/srv/codesonar
HOST_ANALYSIS_DATA=$HOST_ROOT/analysis_data
HOST_RUNNER_CONFIGDIR=$HOST_ROOT/analysis.server.config
CODESONAR_ANALYSIS_DATA=/srv/codesonar/analysis_data
CODESONAR_HUB=https://codesonar.example.com:7340
CODESONAR_HUBUSER=cshub_ci
CODESONAR_LAUNCHD_GROUP=/analysis-data-server
CODESONAR_LAUNCHD_KEY=analysis-data-ld
CI_SERVER_URL=https://gitlab.example.com
CI_RUNNER_REG_TOKEN=abcd1234
CI_RUNNER_TAG=codesonar_analysis_server
CI_RUNNER_HOSTNAME=csonar-analysis
CI_RUNNER_DESCRIPTION=codesonar_analysis_server
CI_RUNNER_CONFIGDIR=$CI_USER_HOME/gitlab-runner/config
CI_RUNNER_IMAGE=codesonar_analysis_server_runner
CI_RUNNER_IMAGE_VERSION=latest
CI_RUNNER_CONTAINER=$CI_RUNNER_IMAGE
CI_RUNNER_USER_VOL=$CI_RUNNER_IMAGE.config
CI_USER_HOME=/home/csonar_ci
CS_USER_HOME=$CI_USER_HOME/cs_home

mkdir -p $HOST_ROOT $HOST_ANALYSIS_DATA $HOST_RUNNER_CONFIGDIR

docker container run \
    --env "CODESONAR_HUB=$CODESONAR_HUB" \
    --env "CODESONAR_HUBUSER=$CODESONAR_HUBUSER" \
    --env "CODESONAR_LAUNCHD_GROUP=$CODESONAR_LAUNCHD_GROUP" \
    --env "CODESONAR_LAUNCHD_KEY=$CODESONAR_LAUNCHD_KEY" \
    --env "CI_SERVER_URL=$CI_SERVER_URL" \
    --env "CI_RUNNER_TAG=$CI_RUNNER_TAG" \
    --env "CI_RUNNER_REG_TOKEN=$CI_RUNNER_REG_TOKEN" \
    --env "CI_RUNNER_DESCRIPTION=$CI_RUNNER_DESCRIPTION" \
    --env CS_USER_HOME=$CS_USER_HOME \
    -v $HOST_ANALYSIS_DATA:$CODESONAR_ANALYSIS_DATA \
    -v $HOST_RUNNER_CONFIGDIR:$CI_RUNNER_CONFIGDIR \
    -v $CI_RUNNER_USER_VOL:$CS_USER_HOME \
    --hostname $CI_RUNNER_HOSTNAME \
    --name $CI_RUNNER_CONTAINER \
    --init \
    --detach \
    $CI_RUNNER_IMAGE:$CI_RUNNER_IMAGE_VERSION

The ci-runner.sh script will attempt to register the runner only if necessary. The runner configuration is stored on a bind-mount directory so that if you start a new container you can re-use your existing configuration rather than registering a new runner.

Next Steps

If you are setting up a GitLab pipeline to perform CodeSonar analysis, go on to step C to create and install a pipeline build runner.

Notes

Modifications for password authentication

If you need to authenticate your launch daemon with a password instead of with a certificate, make the following changes.

When you start the temporary container, replace the line that begins

$CSONAR/codesonar/bin/codesonar generate-hub-cert [...]

with the following command:

$CSONAR/codesonar/bin/codesonar generate_hubpwfile.py "$CS_USER_HOME/hubpwfile"

When you start the temporary container provide the hub user account password when prompted. The password will be saved in file $CS_USER_HOME/hubpwfile in the Docker volume. Make sure only your Docker user (docker_ci) has access to this file.