Places that gather data

The LANforge server and client save data and reports in different places:

LANforge Server:

This is the manager node. (If you have just one system it is both a manager and a resource.)

Client:

The LANforge software that can generate report output could be:

• Java GUI: when you run Chamber View (CV) reports, those reports and their underlying data are saved to the laptop or the server desktop that runs the GUI.
• Perl scripts: output from the perl scripts are saved on the system running the script.
• Python scripts: output from the perl scripts are saved in two places:
  • Local report data: In the GUI, the Report menu->Report Manager->GUI Settings tab sets the place where the local CSV data is saved. It also sets the default location of the Chamber View reports.
  • Local report data is saved in various places:
    • /home/lanforge/html-reports: the HTML and PDF reports
    • /home/lanforge/report-data: this is the folder holding CSV data and Wireshark capture data (PCAPs).
    • /home/lanforge/lf_reports: this folder is an older default location for saving CSV data for the GUI tabs (Port, Layer 3, Layer 4-7, etc).
  • Script report data: when running a script from the py-scripts directory, the output is typically saved in the directory you start the script in. If you run these scripts from your laptop, they might be saving in your C:\Users\bob\Desktop or your C:\Users\bob\git\lanforge-scripts\py-scripts directory.
    • Please DO NOT save report output or changed scripts anywhere in your git/lanforge-scripts folder.

The challenge

• Your CV reports can produce 100s of megabytes of data (CSV, HTML and PDF files)
• Your GUI produces CSV data as a multiple of Endpoints X Report-Interval X Test Instance, so your report-data directory will accrue at least 100s of MB of report data.
• Your Chamber View (CV) reports save their own copy of CSV, HTML and PDF reports to the html-reports directory where the GUI runs.
• Your Python scripts can add to that by saving CSV and graph output. This output lands wherever the script is run from on your laptop or on your LANforge desktop.

Scripts that remove data

• lf_kinstall.pl --install_large_file_cron: creates a cronjob that removes files. This calls check_large_files.bash -a and will remove some files.
• check_large_files.bash: this script removes a variety of files:
  • old LANforge downloads
  • trash can folders
  • logs from /var/log, /var/tmp, journalctl, and nginx logs
  • wifi output logs
  • saved LANforge databases
  • core files and crash dumps

Your automation

Your automation scripts can produce data in at least two of these areas (typically the report-data, html-reports and sometimes the lf_reports dirs.) Your Linux OS comes with a few areas that can help automate the removal of those files.

Systemd timers: you can craft a system service-unit file and a system timer-task that calls it.

Systemd timers and services are somewhat simpler to understand because there is less strange syntax. You need to create both a service and timer file to for the job.

Cron

This legacy service has been with us for some time; it enables system-scope and user-scope scripts to run on a regular basis. Some of of the notable places you can find crontab scripts:

• /var/spool/cron/<user> - a user's crontab
• /etc/crontab - one file that the root user can edit
• /etc/cron.d - a directory of files that a root user can add scripts to
• /etc/cron.(hourly|daily|weekly|monthly) - where root can place other scripts
• User jobs are created with crontab -e or crontab -u lanforge -e. Crontab syntax is easily searched for. Please ask support@candelatech.com for help.
• on-service-start: the /home/lanforge/lfcustom_ctrl.bash file provides areas you can execute shell commands that will start immediately before the LANforge manager service starts. There is a template script provided to edit. Please do not edit it without consulting Candelatech support (support@candelatech.com).

File selection techniques

The report data you want to purge is always time-sensitive: you probably want to avoid deleting collected data that is actively being collected by the test your are executing. It is prudent to consider how old the data you want to delete is.

Candela Technologies encourages you to save your report data to a safe location. LANforge systems are often operated for prolonged periods in high-temerature environments that shorten the life of SSDs.

How old is the data?

• Maybe

  Do you want to compress it?

  The check_large_files.bash script has an option to compress the reports and data present. Your team might benefit from compressing old output before deleting it in case you need it.

Consider your existing automation:

• Long-running tests can run longer than the period between cleanup tasks
• Days of the week: choosing a day of the week is understandible, but keep an eye out for those long-running tests that span the next cleanup
• Do you need PCAPs after reports are generated? PCAPs take significant space
• Do you need both HTML reports and PDFs? You might benefit from deleting one before the other.
• How old is the data? Maybe you want to delete everything older than one or two days?
• Do you want to compress it? The _check_largefiles script has an option that can compress your reports. It might make sense to only delete report data that has already been compressed.
• Consider you existing automation: your CICD pipeline might be able to back up just the reports you want and also trigger a cleanup script.

System techniques

Below are some examples of how to script the removal of data:

Systemd timers

Systemd timers operate systemd service units.

1. Create a bash script that removes the data you want to delete, /home/lanforgee/remove-data.sh

   #!/bin/bash
   find /home/lanforge/lf_reports -type f -mtime +1 -exec rm -f {} \;

   Make the file executable with chmod +x remove-data.sh

2. Create a systemd service: /etc/systemd/system/remove-data.service

   
   [Unit]
   Description=Remmove report data older than one day

[Service] Type=oneshot ExecStart=/home/lanforge/remove-data.sh

3. Create a timer: `/etc/systemd/system/remove-data.timer`

[Unit] Description=Remove LANforge report data older than one day every week Unit=remove-data.service

[Timer] OnCalendar=weekly Persistent=true

OR on a specific weekday:

OnCalendar=Fri --* 18:00:00

[Install] WantedBy=timers.target

4. Inspect the settings: 
```shell
systemd-analyze calendar weekly

Cron

The cron system is slowly being abandoned by Linux distributions. If you are familiar with the service, there's no harm in using it.

1. Script: /home/lanforge/remove-data.sh

   #!/bin/bash
   find /home/lanforge/lf_reports -type f -mtime +1 -exec rm -f {} \;

2. Find the weekly cron directory: /etc/cron.weekly/
3. Symlink the script into the directory: sudo ln -s /home/lanforge/remove-data.sh /etc/cron.weekly/