--- title: Improve Build Times on Kubernetes-Based Jenkins with Stateful Agents description: An effective continuous integration system is a crucial component for fast-paced software development. At Hiya, we built our CI system using Jenkins deployed on Kubernetes via a fork of the official Helm chart. image: https://blog.hiya.com/hubfs/Coding.png --- [![Hiya logo](https://blog.hiya.com/hubfs/hiya-new-purple.svg)](https://www.hiya.com/) Solutions Solutions [For Mobile Operators Protect your network](https://www.hiya.com/solutions/operators) [For Businesses Brand your calls](https://www.hiya.com/solutions/businesses) [For People Get the mobile app](https://www.hiya.com/products/apps/hiya-spam-blocker) ###### Webinar Branded Calling 101 Weekly Webinar Make your company's calls more recognizable. Learn how Hiya can drive value for your business. [Sign up today Arrow Right](https://www.hiya.com/lp/webinar-series-intro-to-hiya-branded-call) Products Connect ###### [Branded Call Display your branded caller ID](https://www.hiya.com/products/connect/branded-call) ###### [Number Registration Display your branded caller ID](https://www.hiya.com/products/connect/number-registration) ###### [View Plans Flexible pricing for teams of all sizes](https://www.hiya.com/products/connect/pricing) Protect ###### [Spam Analytics Stop spam & fraud on your mobile network](https://www.hiya.com/products/protect/spam-analytics) ###### [AI Voice Detection Real-time AI voice detection](https://www.hiya.com/products/protect/ai-voice-detection) Mobile apps ###### [Hiya Spam Blocker Fraud & AI voice protection](https://www.hiya.com/products/apps/hiya-spam-blocker) ###### [Hiya AI Phone Productivity for busy people](https://www.hiya.com/products/apps/hiya-ai-phone) ![](https://blog.hiya.com/hubfs/sotc-report.avif) ###### Report State of the Call 2026 86% of unidentified calls go unanswered. Read the benchmark report for what is happening in voice today, and what you can do to drive business. [Read the report Arrow Right](https://www.hiya.com/state-of-the-call) Why Hiya? Overview [Why Hiya Your voice innovation partner](https://www.hiya.com/why-hiya) [How it works Get started quickly & easily](https://www.hiya.com/how-it-works) [Customer Stories Real companies, real results](https://www.hiya.com/case-studies) [Voice Intelligence Platform Industry’s leading voice platform](https://www.hiya.com/why-hiya/voice-intelligence-platform) [Trust Center Compliance, security, & privacy](https://www.hiya.com/why-hiya/trust-center) Company [About Leadership and history](https://www.hiya.com/company/about) [Careers We're hiring!](https://www.hiya.com/company/careers) [Contact us Get in touch](https://www.hiya.com/contact-us) ![](https://blog.hiya.com/hubfs/bclc-hero-img.webp) ###### Customer Story BCLC increases business KPIs with Hiya With Hiya Branded Call BCLC was able to increase contact rates, campaign efficiency, and revenue. [Read their story Arrow Right](https://www.hiya.com/customer-stories/bclc) Resources Resources [Resource Center](https://www.hiya.com/resources) [Partner Program](https://partners.hiya.com/) [Get Support](https://hiya.com/support) [Developer Docs](https://developer.hiya.com/) [Hiya Blog](https://blog.hiya.com/) [Newsroom](https://www.hiya.com/newsroom) [Events](https://www.hiya.com/events) ###### eBook 10 Tips for customer-friendly phone calls Prevent caller reputation issues and complaints with customer-friendly calling practices. [Read eBook Arrow Right](https://www.hiya.com/resources/ebooks/10-tips-for-customer-friendly-phone-calls) [Get Started Arrow Right](https://www.hiya.com/#audience) [Log in](https://connect.hiya.com/login) [Get started](https://www.hiya.com/#audience) - [Home](https://hiya.com) - [Blog](https://blog.hiya.com) - [Improve Build Times on Kubernetes-Based Jenkins with ...](https://blog.hiya.com/kubernetes-base-jenkins-stateful-agents/) # Improve Build Times on Kubernetes-Based Jenkins with Stateful Agents [Jake Utley](https://blog.hiya.com/author/jutley) Sep. 20, 2020 [![](https://blog.hiya.com/hubfs/icon-fb.svg)](https://www.facebook.com/sharer/sharer.php?u=https://blog.hiya.com/kubernetes-base-jenkins-stateful-agents/) [![](https://blog.hiya.com/hubfs/icon-x.svg)](https://twitter.com/intent/tweet?url=https://blog.hiya.com/kubernetes-base-jenkins-stateful-agents/&text=Improve+Build+Times+on+Kubernetes-Based+Jenkins+with+Stateful+Agents) [![](https://blog.hiya.com/hubfs/icon-linkedin.svg)](https://www.linkedin.com/shareArticle?mini=true&url=https://blog.hiya.com/kubernetes-base-jenkins-stateful-agents/) ## At a glance ![](https://blog.hiya.com/hubfs/Coding.png) ### Our Use Case An effective continuous integration system is a crucial component for fast-paced software development. At Hiya, we built our CI system using Jenkins deployed on Kubernetes via a fork of the official [Helm chart](https://github.com/kubernetes/charts/tree/master/stable/jenkins). This decision was largely inspired by Lachlan Evenson’s tutorial video[ Zero to Kubernetes CI/CD in 5 minutes with Jenkins and Helm](https://www.youtube.com/watch?v=eMOzF_xAm7w). Traditionally, running Jenkins on Kubernetes involves dynamically creating Jenkins agents for each job. There are many benefits with this approach, but in our experience it has one critical downside: **Short-lived Jenkins agents do not retain the dependencies they pull in**. Each of our jobs spent 20 minutes pulling dependencies. All 20 minute delays added up to an enormous loss in productivity. We were unable to find any community solutions to this problem, so **we created an approach to deploy stateful Jenkins agents on Kubernetes**. Running stateful agents allows us to persist our cache of dependencies between jobs/pods, dramatically speeding up build times. This article will explain our approach and provide a demo that can be run in the reader’s Kubernetes cluster. Please note that all resources mentioned in this article can be found in this git repository: [https://github.com/hiyainc/jenkins-stateful-agents-demo](https://github.com/hiyainc/jenkins-stateful-agents-demo). ### Alternatives Considered Before we get into our stateful agent solution, we will cover some other approaches we tried and the downfalls we encountered with each. We build the majority of our services with Scala, so some of our approaches are JVM specific. The “stateful agent” solution, however, is technology-agnostic. First, we tried caching our dependency artifacts on a persistent [Artifactory OSS](https://www.jfrog.com/open-source/) instance running within our cluster. This made no significant impact to our build times, indicating that the bottleneck was in dependency resolution, not the network. Our second approach built agent docker images with all our dependencies statically cached. This helped initially, but degraded as projects updated their dependencies. We could regularly build new images with updated dependencies, but this is tedious and costs developer time. The third iteration mounted a persistent volume onto each dynamically created Jenkins agent. This works, but since most persistent volume types can only be mounted onto one pod at a time, we cannot run multiple agents at the same time. We considered modifying our third approach with a ReadWriteMany persistent volume, allowing multiple agents to run in parallel while sharing a cache. In some cases this may work, but the general approach introduces potential for race conditions. We would rather not worry about this complexity. After all these attempts, we decided to create long-lived stateful agents that each have their own persistent storage. While we could not find an existing example anywhere, we felt that this was the simplest and most reliable solution and worth figuring out.   ![ssh-agent-flow-3](https://blog.hiya.com/hs-fs/hubfs/ssh-agent-flow-3.png?width=600&name=ssh-agent-flow-3.png) We will need to create: - A Jenkins master, fully configured for our needs - A Jenkins agent with a static hostname and persistent volume, which automatically configures Jenkins master to use the agent - A test job that demonstrates the stateful behavior of our agent. Our Jenkins master will not be stateful. In a production environment, a Jenkins master *should* be stateful, but in this article we have decided to not worry about this for simplicity’s sake. There are plenty of resources online to help you set up a persistent volume for your Jenkins master if you need help. **Note:** Jenkins “agents” used to be called “slaves”, and some plugins and docker images still use the old name. We will use “agent” unless referring to one of these plugins or images. ## Communication between master and agent There are a number of ways a Jenkins master and agent can connect to each other. We will initiate this connection from the Jenkins master over SSH, using SSH credentials to secure the connection. This approach requires that we install the [ssh-slaves](https://wiki.jenkins.io/display/JENKINS/SSH+Slaves+plugin) plugin to our master, mount ssh credentials into the master and agent, and base our agent off of an [ssh-slave](https://hub.docker.com/r/jenkinsci/ssh-slave/) image. Additionally, the Jenkins master needs to connect to each agent via unique static hostnames, which we get by deploying our agents using a [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/). ## Initialization of agent(s) Jenkins master will not connect to an ssh-based agent unless it is configured to do so. We can bootstrap this configuration by giving each agent pod an [Init Container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) in charge of configuring Jenkins master. By the time the agent starts up, Jenkins master will already be trying to connect. The Init Container will need to complete the following: - Download the Jenkins CLI from master - Use the CLI to check if master is already configured to use this agent - If needed, use the CLI to configure master to use this agent This will require some configuration within our agent pod: - [fsGroup](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) set to the “jenkins” user group. This is needed because persistent volumes mount as the root user by default, making them unusable for the “jenkins” user. - Environment variables within the Init Container: - JENKINS\_URL: http address of Jenkins master - JENKINS\_LABEL: label for our agent - Environment variables within the main container: - JENKINS\_SLAVE\_SSH\_PUBKEY: public key to authenticate connection attempts ## Initialization of master Our Jenkins master will need to start up with some configuration pre-applied. Specifically, we will need to: - Install the [ssh-slaves](https://wiki.jenkins.io/display/JENKINS/SSH+Slaves+plugin) plugin - Enable SSHD - Set “admin” user’s public key (so the Init Container can authenticate as “admin”) - Add an SSH key credential (to connect the agent) While we will not go into detail about how this is all configured, all the necessary configuration is in the [values.yaml](https://github.com/hiyainc/jenkins-stateful-agents-demo/blob/master/values.yaml) file in this demo’s resources on Github. ### How to **Note**: All steps assume that we are deploying into the default kubectl context. You may need to set your default context, or modify the commands to manually provide a context. ## Jenkins master 1. Clone our [repository](https://github.com/hiyainc/jenkins-stateful-agents-demo) containing this demo’s Kubernetes resources, then apply our Kubernetes secret containing our ssh credentials. ``` ➜ git clone https://github.com/hiyainc/jenkins-stateful-agents-demo \ && cd jenkins-stateful-agents-demo \ && kubectl --namespace jenkins-demo apply -f jenkins-ssh-secret.yaml ``` 2. Use Helm with the provided [values.yaml](https://github.com/hiyainc/jenkins-stateful-agents-demo/blob/master/values.yaml) file to configure and create a Jenkins master. All settings in values.yaml are explained in the previous section. ``` ➜ helm install --name jenkins-demo --namespace jenkins-demo stable/jenkins -f values.yaml ``` 3. Verify that Jenkins master is properly configured. Specifically: - The ‘admin’ user should be configured with our public ssh key. - There should be a single ssh credential, referencing files on disk. - The sshd port should be enabled and set to 22. - The ‘[ssh-slaves](https://wiki.jenkins.io/display/JENKINS/SSH+Slaves+plugin)’ plugin should be installed. To do this, log into Jenkins master via the web console, and check following paths: - /user/admin/configure - /credentials/store/system/domain/\_/credential/jenkins-agent-ssh-key/update - /configureSecurity/ - /pluginManager/installed **Note**: The following commands use the shell tool ‘jq’. To get the url to use in your browser: ``` ➜ echo http://$(kubectl get nodes -o json | jq -r '.items[0].status.addresses[0].address'):$(kubectl --namespace jenkins-demo get svc jenkins-demo-jenkins -o json | jq -r '.spec.ports[0].nodePort')http://10.0.10.15:31639 ``` To get the password for the admin user: ``` ➜ kubectl --namespace jenkins-demo get secret jenkins-demo-jenkins -o json | jq -r '.data["jenkins-admin-password"]' | base64 -dyn5nUy0jLz ``` 4. The service provided by the Helm chart does not include the sshd port. To resolve this, we must add an additional service. More on this under “best practices”. ``` ➜ kubectl --namespace jenkins-demo apply -f jenkins-master-svc.yaml ``` ## Jenkins agent 1. Deploy the [ConfigMap](https://github.com/hiyainc/jenkins-stateful-agents-demo/blob/master/jenkins-agent-bootstrap-configmap.yaml) with our initialization script. This will be mounted into our Init Container to configure master to use this agent. ``` ➜ kubectl --namespace jenkins-demo apply -f jenkins-agent-bootstrap-configmap.yaml ``` 2. Deploy the [StatefulSet](https://github.com/hiyainc/jenkins-stateful-agents-demo/blob/master/jenkins-agent-statefulset.yaml). This will create our agent(s), each of which will have a unique persistent volume and static hostname. **If you do not have a default storage-class in your cluster**, first edit this statefulset to use a storage-class your cluster supports. ``` ➜ kubectl --namespace jenkins-demo apply -f jenkins-agent-statefulset.yaml ``` 3. Within a few minutes, “jenkins-agent-0” should be visible as a “build executor” from the master’s web console! ## Testing From our Jenkins master’s web console, create a new Freestyle job, and configure it as follows: - “Restrict where the project can run” should be set to the label for our stateful agents (“stateful”, in our demo). - Give a single “Execute shell” build step, with the following script: ``` FILE=/mnt/pv/buildsecho "Adding build id to $FILE"echo $BUILD_ID >> $FILEecho "Current state of $FILE:"cat $FILE ``` Save this job and run it a few times. Each run will concatenate the build number to a file, then print out the file. After 3 runs, you should see output that looks like this: ``` Started by user adminBuilding remotely on jenkins-agent-0 (stateful) in workspace /home/jenkins/workspace/test-job[test-job] $ /bin/sh -xe /tmp/jenkins6384725398355425391.sh+ FILE=/mnt/pv/builds+ echo Adding build id to /mnt/pv/buildsAdding build id to /mnt/pv/builds+ echo 3+ echo Current state of /mnt/pv/builds:Current state of /mnt/pv/builds:+ cat /mnt/pv/builds123Finished: SUCCESS ``` ### Final Thoughts/Best Practices With this approach, our job durations decreased from ~20 minutes to ~3 minutes! This is a huge win for us. While the end result is not as simple as using stateless, short-lived agents, it has not been difficult to maintain. Occasionally a job *does* put our workspace into a buggy state, but since our persistent volumes only cache our dependencies, we can simply bounce the agent pod. Our dependency cache is preserved, and the pod starts with a fresh workspace. If you would like to follow our example and add stateful agents to your Jenkins server, here are a couple points to bear in mind: **Be mindful of what parts of your agent actually need to be stateful**, and limit your persistent volume to contain exactly those parts. In our case, our persistent volumes only contain our Ivy dependencies. Other parts of the agent, such as its workspace, are not treated as persistent. Introducing state to any application adds complexity, so minimize your state as much as possible. **Use a persistent storage backend that is reliable**. Our Kubernetes clusters run on AWS, and we have dealt with a lot of pain around EBS-backed persistent volumes, particularly around the automated mounting/unmounting of the volumes onto the underlying EC2 instances. This led us to try installing the [efs-provisioner](https://github.com/kubernetes-incubator/external-storage/tree/master/aws/efs), which has proven much more reliable. Our pains with EBS were fixed when we upgraded to Kubernetes 1.7, but the lesson stands that the persistent storage backend that you choose matters, so choose wisely! **Setting an external address for your Jenkins master will break this flow** by changing the advertised SSHD address. If you want an external domain for Jenkins master, you need to set the “org.jenkinsci.main.modules.sshd.SSHD.hostName” system property to the preferred host. **The Jenkins Helm chart’s service does not include the sshd port**. In our demo, we exposed this port via an additional service, and hardcoded our agents to access the Jenkins master via this service’s shortname. In a production system, this manual configuration is not recommended. Instead, add this port into a service dynamically. In our actual Jenkins, we do this by forking the Helm chart and adding the extra port into the Jenkins master service. [![banner-hiya-reg](https://blog.hiya.com/hubfs/banner-hiya-reg.jpg)](https://www.hiya.com/products/connect/number-registration) ### Latest Articles ⚡️ - Hiya awarded 16 G2 Summer 2026 badges for branded caller ID Lena Prickett Jul. 9, 2026 - How Canada is leading the fight to stop bank scams Stephanie Boulanger Jun. 26, 2026 - How to check phone numbers for spam labels Michelle Wallace Jun. 9, 2026 ## Related articles ![](https://blog.hiya.com/hubfs/rogers-toronto-event-blog-hero.webp) ## [How Canada is leading the fight to stop bank scams](https://blog.hiya.com/how-canada-is-leading-the-fight-to-stop-bank-scams?hsLang=en) Stephanie Boulanger Jun. 26, 2026 ![](https://blog.hiya.com/hubfs/Blog_How-to-check-phone-numbers-for-spam-labels_B.webp) ## [How to check phone numbers for spam labels](https://blog.hiya.com/how-to-check-phone-numbers-for-spam-labels?hsLang=en) Michelle Wallace Jun. 9, 2026 ![](https://blog.hiya.com/hubfs/Hiya%20x%20DNA-blog@2x.webp) ## [Hiya and DNA Finland launch network-level call protection](https://blog.hiya.com/hiya-and-dna-finland-launch-network-level-call-protection?hsLang=en) Stephanie Boulanger Jun. 2, 2026 [![](https://blog.hiya.com/hubfs/hiya-new-purple.svg)](https://www.hiya.com/) [![](https://blog.hiya.com/hubfs/linkedin-1.svg)](https://www.linkedin.com/company/hiyainc) [![](https://blog.hiya.com/hubfs/facebook-1.svg)](https://www.facebook.com/hiyainc/) ###### Products - Connect - [Number Registration](https://www.hiya.com/products/connect/number-registration) - [Branded Call](https://www.hiya.com/products/connect/branded-call) - [View Plans](https://www.hiya.com/products/connect/pricing) - Protect - [Spam Analytics](https://www.hiya.com/products/protect/spam-analytics) - [AI Voice Detection](https://www.hiya.com/products/protect/ai-voice-detection) - Apps - [Hiya Spam Blocker](https://www.hiya.com/products/apps/hiya-spam-blocker) - [Hiya AI Phone](https://www.hiya.com/products/apps/hiya-ai-phone) ###### Solutions - Company size - [Enterprise](https://www.hiya.com/solutions/enterprise) - [Call Centers](https://www.hiya.com/solutions/call-center) - [Small and Medium](https://www.hiya.com/solutions/businesses) - Service providers - [Operators](https://www.hiya.com/solutions/operators) - [OEMs and technology](https://www.hiya.com/solutions/technology-partners) ###### Considering Hiya? - [Why Hiya](https://www.hiya.com/why-hiya) - [How it Works](https://www.hiya.com/how-it-works) - [Customer Stories](https://www.hiya.com/case-studies) - [Voice Intelligence Platform](https://www.hiya.com/why-hiya/voice-intelligence-platform) - [Trust Center](https://www.hiya.com/why-hiya/trust-center) - [Modern Slavery](https://www.hiya.com/company/modern-slavery) - [About Hiya](https://www.hiya.com/company/about) - [Careers: We're hiring!](https://www.hiya.com/company/careers) - [Contact us](https://www.hiya.com/contact-us) ###### Resources - [Resource Center](https://www.hiya.com/resources) - [Partner Program](https://partners.hiya.com/) - [Get Support](https://hiya.com/support) - [Developer Docs](https://developer.hiya.com/) - [Hiya Blog](https://hiya.com/blog) - [Events](https://www.hiya.com/events) - [Press Kit](https://www.hiya.com/newsroom#press-kit) ![ANAB](https://blog.hiya.com/hubfs/logo-anab.svg) ![ISO 27001](https://blog.hiya.com/hubfs/logo-iso27017.png) ![AICPA](https://blog.hiya.com/hubfs/logo-aicpa.svg) ![IAF](https://blog.hiya.com/hubfs/logo-iaf.svg) [Privacy Policy](https://www.hiya.com/legal/privacy) [Terms of Service](https://www.hiya.com/legal/terms-of-service) [App Data Privacy Policy](https://www.hiya.com/legal/app-data-protection-and-privacy-policy) ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Jake Utley", "url" : "https://blog.hiya.com/author/jutley" }, "dateModified" : "2022-08-30T22:41:39.445Z", "datePublished" : "2020-09-20T23:21:22.000Z", "headline" : "Improve Build Times on Kubernetes-Based Jenkins with Stateful Agents", "image" : [ "https://blog.hiya.com/hubfs/Coding.png" ], "mainEntityOfPage" : { "@id" : "https://blog.hiya.com/kubernetes-base-jenkins-stateful-agents/", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://blog.hiya.com/hubfs/hiya-logo-3.svg" }, "name" : "Hiya Inc." } } ```