Set up a Jenkins pipeline storage policy

During the daily iteration of functions, project personnel use pipelines to automate business processes, which generates a large number of pipeline execution records. These records are stored in the etcd component of the platform. If not processed in a timely manner, it will reduce the storage performance of etcd and affect the overall use of the platform.

To address this issue, the platform provides a comprehensive cleaning and archiving strategy and mechanism, which supports automatic cleaning of execution records from etcd or archiving execution records to other databases, meeting storage needs while balancing storage pressure.

Cautions

When using the cleanup and archiving functions of the platform, depending on whether you have deployed Elasticsearch when installing the platform, please note the following:

• Elasticsearch is deployed on the platform: When the platform detects that you have deployed Elasticsearch, it will enable the pipeline execution record archiving function by default and support manual disabling.

• Elasticsearch is not deployed on the platform: When the platform detects that you have not deployed Elasticsearch, it will disable the pipeline execution record archiving function by default, and the button will be grayed out, and cannot be manually enabled.

  Caution: The platform supports manual modification of archiving and cleaning settings. The following is the default pipeline archiving and cleaning execution strategy set by the platform:

  • When the total number of all pipeline execution records on the platform is less than 20000, the pipeline execution records will be stored in ETCD.

  • When the total number of all pipeline execution records on the platform exceeds 20000, the part exceeding 20000 will be deleted from ETCD. If the platform detects that you have deployed Elasticsearch, the part exceeding 20000 will be synchronized and archived to Elasticsearch.

Procedure

1. In the left navigation bar, click Pipelines > Jenkins Storage Settings.

2. Click Update and configure the relevant parameters according to the following instructions.

   • In the cleanup setting section, configure the pipeline cleanup policy.

     • Keep at least x execution records for each pipeline: the input value range is 5 - 20, and only positive integers are supported.

     • Maximum storage of x pipeline execution records for the platform: the total number of execution records for all pipelines in the platform, the input value range is 1 - 20000, and only positive integers are supported.

       Tip:

       • If the preset value range is More than or Lower Than, the system will automatically adjust to the normal range closest to the value.

       • If the archive function is not enabled, pipeline execution records exceeding the maximum storage limit will be automatically cleaned up in reverse order of pipeline execution time.

       • If the archive function is enabled and Elasticsearch is deployed on the platform, pipeline execution records exceeding the maximum storage limit will be automatically archived to Elasticsearch in reverse order of pipeline execution time.

   • In the archive setting section, if you have deployed Elasticsearch in the platform, you can configure the pipeline archiving policy here.

     • Enable: If enabled, select the retention time for pipeline execution records in the drop-down box. The supported time periods for selection are: one year, half year, three months. Archived execution records that are not within the retention time range will be cleaned up, and execution record information that exceeds three months of archiving time will no longer retain detailed logs.