Pipeline CI/CD security

A CI/CD pipeline is a set of automated steps used in software development  to take code from creation to production deployment. It is made up of three key phases: Continuous Integration (CI), Continuous Delivery (CD), and Continuous Deployment (CD).

• Continuous Integration (CI): focuses on the regular integration of developers' code into a shared repository. Every change to the code is automatically integrated and verified by running unit and integration tests. This allows errors to be detected early, preventing problems in the code from accumulating throughout development. CI automates code verification, ensuring that each new piece of code that is integrated into the existing codebase works correctly with previous changes.
• Continuous Delivery (CD): at this stage, code that has successfully passed through CI is packaged and prepared to be delivered to a production or pre-production environment. Continuous Delivery automates the delivery of this production-ready code, but typically requires manual approval before final deployment takes place. This ensures that the software is in a ready-to-deploy state at any time but maintains manual control over the exact timing of its deployment.
• Continuous Deployment (CD): this phase takes automation a step further, eliminating the need for manual intervention in deployment. In a pipeline with Continuous Deployment, the software that goes through all the tests and validations is automatically deployed in the production environment. This enables a fully automated development lifecycle, from coding to final delivery, reducing turnaround times and allowing for rapid iteration and response to changes in requirements.

Un pipeline CI/CD seguro ofrece numerosas ventajas en el desarrollo de software, ya que automatiza la integración de prácticas de seguridad en cada etapa del ciclo de vida del desarrollo, como el análisis de código y las pruebas automatizadas, lo que permite detectar y mitigar vulnerabilidades de manera proactiva. 

These practices, aligned with the DevSecOps philosophy, ensure that security remains a priority from development to deployment, fostering collaboration between development, operations, and security teams to ensure software integrity and security  throughout its entire lifecycle.

- CI/CD pipeline diagram. -

The following points review some of the most interesting security measures that we can introduce in the CI/CD stages.

Continuous integration

This stage focuses on producing source code free of bugs, vulnerabilities, and quality issues, thus maintaining stable and efficient development.

Secure source code repository management

Source code storage and version control are critical in software development. You need a place, or repository, where you can store your code and maintain multiple versions, as your code is constantly updated with new features and improvements. To choose a code repository management solution, it is crucial to maintain the integrity of changes from the first release, ensuring that all modifications are logged and can be tracked at any time. Version control solutions are essential for managing frequent changes and allowing multiple developers to work on the same code simultaneously. Common Git-based tools (such as Gitlab or Github) and SVN (such as TortoiseSVN) make it easier to manage source code; Git, for example, is a distributed system where each contributor has their copy of the code, while SVN is a centralized system. 

A commonly insecure configuration is to allow any user on an internal network to register on an instance of a versioning system and access public repositories, which expands the attack surface. In large organizations, this can result in the exposure of sensitive information due to the mistaken belief that the internal network is a sufficient security barrier. Automated tools, such as scripts that use the Release Manager API, can exploit this vulnerability to enumerate and download entire public repositories. And you should also keep in mind that if your development includes insecure code or sensitive information at any point, it may still be visible or accessible to users of the system, even if it has been fixed in later versions. This risk is amplified in the case of open repositories, where any user with access can review the full history of changes and find vulnerabilities or sensitive data, such as credentials, which could compromise the security of the project. 

To ensure source code security in a development environment generally, it is critical to implement several security practices:

• Branch protection prevents changes from being made directly to critical branches, such as the main branch, without going through code reviews and automated testing, ensuring that only reviewed and approved code is merged. 
• Environment variables allow sensitive data, such as passwords or API keys, to be stored securely, outside of the source code, preventing it from being exposed in the repository. Additionally, if you use a secret management solution, it is important to review its encryption mechanisms.
• Granular access controls define specific permissions for each user or group, ensuring that only authorized individuals can access or modify sensitive parts of the code.
• Tools such as GittyLeaks, gitleaks o trufflehog can help audit the level of sensitive information from previous versions stored in the repository.

Secure dependency management

Dependency management is a crucial aspect of software development, as many times the code we write is only a small part of the total, relying heavily on libraries and software development kits  (SDKs). These dependencies allow developers to reuse existing code, speed up development, and reduce the need to reinvent the wheel, improving overall efficiency. However, the extensive use of dependencies also introduces significant security risks that need to be carefully managed. Dependencies can be external, or internal, developed and maintained by an organization for its own use. 

External dependencies such as publicly available libraries in managers such as PyPi for Python, NuGet for .NET, or Gems for Ruby present security risks because we do not have full control over their maintenance and evolution; these libraries could contain vulnerabilities or be compromised through supply chain attacks. For example, if a package manager or CDN that distributes these libraries is compromised, an attacker could insert malicious code into a widely used library, potentially affecting all applications that use it. As an example, we can cite the case of the security incident with the 'event-stream' library in 2018. 

To mitigate these risks, it is crucial:

• Update and apply patches to dependencies on a regular basis, performing emergency patches in the event of serious vulnerabilities. 
• Additionally, copying and hosting these externally developed dependencies internally can reduce the attack surface, limiting exposure to external sources, but it is essential to regularly synchronize them with the latest versions of public repositories to ensure the latest security patches are applied. 
• Using  Sub-resource Integrity (SRI) helps protect JavaScript libraries from tampering during loading. Supplementing SRI with  regular security audits ensures that the libraries used do not contain vulnerabilities from the source.

On the other hand, internal dependencies also carry risks. Although they are developed and maintained in-house, they can become obsolete if they stop receiving updates or if the team that developed them is no longer available. In addition, a vulnerability in an internal library could affect all applications that use it within the organization, generating widespread risk. To protect these dependencies, it is essential:

• Actively maintain them, ensuring that they do not become unsupported legacy code and that any vulnerabilities in them do not affect multiple applications. 
• In addition, the hosting infrastructure of these dependencies must be secured, protect packages using scoped control to restrict the use of dependencies to specific applications or services within the organization by ensuring that only applications that actually need a certain dependency have access to it, and use client-side verification features such as version locking to detect and prevent the execution of malicious code. It should be noted that this version lock prevents the installation of unapproved dependencies, but it should be supplemented with regular security scans to ensure that locked versions do not have known vulnerabilities.

SAST Security Testing

The integration of SAST (Static Application Security Testing) in CI allows security problems to be detected early, during software development, and provides immediate feedback to developers. However, while they are effective at saving time compared to manual reviews, these tools can generate false positives, and while many offers advanced settings to reduce their frequency, it is important to adjust scanning rules and perform manual validations, ensuring that real vulnerabilities are dealt with efficiently. Systems like Gitlab and GitHub integrate SAST natively. Some guidelines for improving SAST integration into CI include:

• Integrating tools like Semgrep into developers' IDEs allows security analysis to be performed during the coding phase, improving code quality before the integration phases. For real-time alerts, other tools such as SonarLint can be used, which offer problem detection while the developer is writing the code. In addition, these tools allow you to define severity thresholds that determine when the continuous integration process should be stopped.
• In addition to SAST, data flow analysis and data contamination detection can be performed, using complementary tools, providing an additional layer of security before software deployment.
• Vulnerability management tools, such as Jira, GitLab Issues, or GitHub Issues, can also be integrated to track and manage vulnerabilities detected by SAST in detail and manage their remediation systematically within the   development pipeline. These tools allow you to automatically create tickets or tasks when security issues are detected, assigning them to the appropriate developers for review and remediation. This ensures that each vulnerability is monitored from detection to resolution.

Continuous delivery

This phase ensures that versions of the software that have passed initial testing are delivered quickly and securely, allowing for additional revisions, and ensuring that the product is ready to be released under approval.

DAST Security Testing

Unlike  manual application pentesting, DAST (Dynamic Application Security Testing) in a  CI/CD pipeline usually involves the automatic execution of security tests at different stages of the development and deployment process, which complemented by manual tests allows the detection of vulnerabilities in dynamic applications.

DAST integration may seem straightforward, but it requires key decisions such as properly handling false positives, determining at what stages the scans will run, what events will trigger them, and the intensity of each scan to avoid slowing down the development team. It is essential that these decisions are made in collaboration with the development team to ensure that security requirements are met without significantly disrupting established processes:

• For example, you can use an automated tool such as OWASP ZAP, executed using Zap2docker, which allows you to run security scans in an integrated way in Docker containers during the pipeline construction process. 
• If we use orchestrators, such as Jenkins, it is also possible to configure a new stage in the orchestrator to run these scans. 
• Finally, implementing monitoring and review mechanisms for the results of the scans allows a rapid response to detected vulnerabilities, thus optimizing development security without compromising the speed of software delivery.

Continuous deployment

Continuous deployment takes automation to the next level, allowing every approved change to code to be automatically deployed to the desired environment. This stage eliminates manual intervention in the deployment process, ensuring fast and efficient delivery of the software to the end user, minimizing the risks of human error and downtime.

Risks in environment configurations

It is common to have multiple environments for development, testing, or production in a  CD pipeline for different stages, but misconfiguring these environments also introduces risk factors that are enhanced by automation. For example, sharing runners across environments can allow a vulnerability in a less secure environment to compromise a more critical one, such as PROD, facilitating cross-attacks and privilege escalations that could compromise pipeline integrity. In addition, if proper access controls are not implemented, there is a risk of unauthorized tampering with runners, which could lead to the execution of malicious code or unauthorized access to sensitive environments. Lack of resource isolation between environments and inconsistent security configurations increase exposure to potential internal and external attacks, putting the security of the entire development pipeline  at risk.

Some guidelines for improving security in environments include: 

• Isolate environments: ensure that DEV  and PRO pipelines are isolated, using  separate build runners or environment-specific runner tags  . This prevents any compromise in the DEV environment from directly affecting the PRO environment. Each environment should have dedicated resources to prevent shared vulnerabilities.
• Restrict access to CD jobs: it is important to limit access to machines running  build runners (such as GitLab Runners) to authorized personnel only. Strict access controls should be implemented and GitLab's 'Protected CD Environments' feature should be used to define who can deploy in specific environments. This feature allows you to set permissions on who can modify CD configurations, including the .gitlab-ci.yml file, ensuring that only authorized users can make changes.
• Monitoring and alerting: it is recommended to configure alert systems for the  CD pipeline and  build runners. These systems should include alerts for any suspicious activity, such as unauthorized access attempts or failed builds that may indicate compromise. It is critical to regularly review and audit access permissions to CD environments and configurations to ensure that only necessary personnel have access. Access to any users or roles that no longer require it should also be revoked, maintaining tight control over who can affect the build and deployment processes.
   

Conclusion

With the introduction of automated tools and processes, the attack surface is greatly expanded. Every component of the  CI/CD pipeline, such as runners, repositories, and image logs, becomes a potential attack vector if security controls are not properly managed, so an attacker no longer needs to focus solely on the end application.

If an  automated pipeline is not well protected, it can be exploited to inject malicious code, modify configurations, or extract sensitive data before the software reaches the production environment. This possibility increases the risk of compromising the end application, as any vulnerability in the pipeline can facilitate attacks on production if robust security controls are not in place. Proper segmentation of environments and continuous pipeline validation  can significantly reduce this risk.

Therefore, it is critical to implement secure automation within the pipeline to ensure that it does not contribute to the risk of application compromise. This involves building security controls into every stage of the pipeline, from code integration and testing to deployment and continuous delivery. It is necessary to establish strict access policies, such as multi-factor authentication (MFA), secure runners and execution environments, properly manage secrets and sensitive variables through automatic rotation, and continuously monitor activities in the pipeline to detect possible anomalies or unauthorized access attempts. Additionally, training the development team on security best practices and maintaining constant vigilance through regular audits and reviews are essential steps to ensure that automation, rather than being a vulnerability, becomes a strength that drives both software development efficiency and security.