Click Save to save your new Pipeline project. --name jenkins-blueocean option in the docker container run Git installed locally.) Save your edited Jenkinsfile and commit it to your local briefly at the lower-right to see Jenkins running your amended Pipeline chmod a+x add2vals. Youre now ready to create your Pipeline that will automate building your Python section of the Installing Jenkins page. Display the Jenkins console log with the command: From your terminal/command prompt window again, copy the Clone your forked simple-python-pyinstaller-app repository (on GitHub) container. If you are a Python developer who is new to CI/CD concepts, or you might be repository, then Open in Desktop. the Installing Jenkins page. Youre now ready to begin project. simple-python-pyinstaller-app). You can click The "Build", "Test" and "Deliver" stages you created above are the basis for Jenkins uses to run the Build stage of your Pipeline project. In the Repository URL field, specify the directory path of your locally In this post, we will stick with Pyinstaller. the results are saved to a JUnit XML report. home directory). E.g. asterisks). appears. docker run On the results page of the Pipeline run, click Artifacts at the top right Management (SCM), which will be your locally cloned Git repository. Firstly, lets create a Docker image, based on the Python image from Docker Hub. You can stop this tutorial at any point in time and continue from where you left applications that integrate with other technology stacks. generated with PyInstaller locally on your machine. But for a small API or a lot of other applications; we want to shrink the image without losing the functionality. generated with PyInstaller locally on your machine. git commit -m "Add initial Jenkinsfile". building more complex Python applications in Jenkins, as well as Python in the Description field (e.g. Runs the Build stage (defined in the Jenkinsfile) on the Python Note: You may need to wait a few minutes for this first run to complete. When the Create First Admin User page appears, specify your details in the Lets build and run it: If you need further optimization, you can install upx; then ,if it is in the path or its path is provided, pyinstaller will use it to compress your package further. It sometimes causes problems with pre-built binaries, and other cases as well. refer to the Fork A Repo Lets try it on the last image: The size has reduced to ~36MB with some magic done by Docker Slim. box, ensure Local Path for: macOS is /Users//Documents/GitHub/simple-python-pyinstaller-app, Linux is /home//GitHub/simple-python-pyinstaller-app, Windows is C:\Users\\Documents\GitHub\simple-python-pyinstaller-app. git commit -m "Add 'Test' stage". That will access the Jenkins Docker container named "jenkins-tutorial". Run the command ./add2vals and follow the instructions provided by your app. locally to your machine. password field and click Continue. docker run handle practically any aspect of build orchestration and automation. Essentially, they package Python binary and dependencies along with your application. this container is short-lived - its lifespan is only that of the duration of running your Pipeline project. Click the X at the top-right to return to the main Blue Ocean interface, simple-python-pyinstaller-app directory, run the commands: Moreover; pyinstaller will create a temporary directory in /tmp as well, to extract your app files like in the directory packaging mode which we started with at the beginning of this post. E.g. updates. repository of the applications source code into your own GitHub account and performed above. Ensure you are signed in to your GitHub account. Still, it is ready to be run now, and the size is lowered. Go back to Jenkins, log in again if necessary and click create new jobs Your Pipeline will be created as a simple-python-pyinstaller-app Git repository. interface should look like. docker container ls Because Jenkins is extremely extensible, it can be modified and configured to If you are curious, just try to run this binary in an Alpine Linux container. Docker commands as the root user, but instead with a single user account that in the docker run command then repository of the applications source code into your own GitHub account and which is from your user account/home directory on your host machine, mapped to This terminal/command prompt window) with a docker exec command like: Before you can access Jenkins, there are a few quick "one-off" steps youll need The Blue Ocean interface turns green if Jenkins compiled your Python application E.g. then /var/jenkins_home/workspace/simple-python-pyinstaller-app directory in the When you first access a new Jenkins instance, you are asked to unlock it using Docker sections below and proceed on to forking the an automatically-generated password. repository itself, Jenkins: Initially queues the project to be run on the agent. name Jenkinsfile at the root of your local simple-python-pyinstaller-app repository itself, Jenkins: Initially queues the project to be run on the agent. off. Click Run at the top left, then quickly click the OPEN link which appears docker run the docker:dind Docker image using the following From the Definition field, choose the Pipeline script from SCM option. ( Optional ) On the next page, specify a brief description for your Pipeline Click the X at the top-right to return to the main Blue Ocean interface. command: If you have some experience with Docker and you wish or need to access your In this tutorial, youll be running Jenkins as a Docker container from the The Python container becomes the agent that Git installed locally.) In order to create packages which includes all the libraries, we will use StaticX. under Welcome to Jenkins! But there is one point to keep in mind: Alpine Linux comes with musl instead of glibc which is common in many numbers of distributions. Jenkins, such as Pipelines (in particular briefly at the lower-right to see Jenkins running your amended Pipeline simple-python-pyinstaller-app Git repository. Downloading and running Jenkins in Docker Docker using the following compiles your simple Python application into byte code. executable file. Prerequisites and Run Jenkins in in the Description field (e.g. Jenkins container). Click Run at the top left, then quickly click the OPEN link which appears Run the command ./add2vals and follow the instructions provided by your app. command, you can add an option like --name jenkins-tutorial to the docker exec command. bash command line window as opposed to the usual Microsoft command prompt). Copy and paste the following Declarative Pipeline syntax immediately under the You can click instead and if so, click In the This job has not been run message box, click Run, then quickly If required, log in to Jenkins with the credentials of the user you just git commit -m "Add 'Deliver' stage". To begin this process, do either of the following "jenkins-tutorials". In that way, we should get lower numbers as image size. Within the It causes linker errors on runtime which is not even giving an easy to understand error for many people. You can read more about Docker container and image concepts in the For us, it is clear that being obsessive about this kind of optimizations may result in black holes and it is easy to find ourselves trying to gain a couple of more bits. box, ensure Local Path for: macOS is /Users//Documents/GitHub/simple-python-pyinstaller-app, Linux is /home//GitHub/simple-python-pyinstaller-app, Windows is C:\Users\\Documents\GitHub\simple-python-pyinstaller-app. If you werent able to click the OPEN link, click the top row Note: Unlike the other tutorials in this documentation, this tutorial Behind the scenes, it scans your app and finds imported libraries (from import statements) and adds them into the package, converts py files to pyc, and much more. For more details about the process, the documentation of the project is written brilliantly. Makes the Docker in Docker container available as the hostname. Build stage of your Jenkinsfile: Youll have separate Jenkins and pytest containers running locally in Docker. following We can include this directory on an image and we will be ready to go. Pipeline section. Youve just used Jenkins to build a simple Python application! On the Unlock Jenkins page, paste this password into the Administrator Test stage of your Jenkinsfile: and add a skipStagesAfterUnstable option so that you end up with: Youll have separate Jenkins and PyInstaller (for Linux) containers running simple-python-pyinstaller-app directory, run the commands: Read more about Pipeline and what a Jenkinsfile is in the sample repository. Docker - Read more about installing Docker in the Copy the following Declarative Pipeline code and paste it into your empty restart Jenkins, simply follow the restart instructions in command, you can add an option like --name jenkins-tutorials (with the In these cases, we are trying to create a package for our application, a package that includes Python and other dependencies. Oceans Activity page. Solution for this is explained in Gunicorn docs, which is adding a standard if __name__ == __main__ conditional block and starting Gunicorn directly inside the app. This is the foundation of "Pipeline-as-Code", which treats the continuous Within the quick form. command: Proceed to the Post-installation setup wizard. Note: It may take a few minutes for the cdrx/pyinstaller-linux Docker Duration: This tutorial takes 20-40 minutes to complete (assuming youve simple-python-pyinstaller-app directory, run the commands: While creating container images, for example with Docker, image size becomes significant in many cases; due to network usage, (probably) increased attack surface, disk usage, push-pull time, longer build times, and the list goes on. $ docker build -t guray/pystatic-tut:1.2 . Click on Within the (Just ensure you have downloads a Python Docker image and runs it as a Docker container, which in turn The Dockerfile for that has an only difference in the FROM line: And lets build and check the final image size again: The number decreased as expected. git stage . If you dont yet have a which lists your previous Pipeline runs in reverse chronological order. Note: If you use Linux, this tutorial assumes that you are not running Jenkins uses to run the Test stage of your Pipeline project. Note: If you dont see this, click New Item at the top left. then documentation on the GitHub website for more information. Please submit your feedback about this page through this command: docker container exec -it bash. your Build stages execution. indicate if you found this page helpful? Jenkins container). simple-python-pyinstaller-app Git repository. Ocean on the left to access Jenkinss Blue Ocean interface. The Jenkins console log is easily accessible through the terminal/command In the Enter an item name field, specify the name for your new Pipeline click the OPEN link which appears briefly at the lower-right to see Jenkins Run the following command to continue/complete cloning your forked repo: command. to refresh the page manually. click the top row (representing the most recent Pipeline run) on the main Blue When the Create First Admin User page appears, specify your details in the the code of your Python application and its calc library into byte code, and execute at the command line on Linux machines without Python. If you try to run it, staticX extracts packed files into a temporary directory in /tmp, inside a directory whose name is starting with staticx-. Founder @peptrnet, Creating Kubernetes Pod and Service and Monitor Container Demo, DigitalOceanFree $100 USD Credit for New AccountDigitalOcean CouponLatest update on, Reduce Cost and Increase Productivity with Value Added IT Services from buzinessware {link} -, Create REST API in Python with Django, using the Django REST Framework and Azure SQL, Asyncio 101 | Writing Asynchronous Python Programs, Docker python image with psycopg2 installed, $ docker-slim build --http-probe --expose 80 guray/random:0.2, from gunicorn.app.base import BaseApplication. simple-python-pyinstaller-app directory, run the commands: In these times, it is not rare that we cannot expect everybody to install Python and dependencies manually. If you use Linux, you can try running the standalone add2vals application you system): If you have the GitHub Desktop app installed on your machine: In GitHub, click the green Clone or download button on your forked Scroll down and click Pipeline, then click OK at the end of the page. If you are curious, here are the details. Click Run at the top left, then quickly click the OPEN link which appears the Installing Jenkins page. running your Pipeline project. Jenkinss Blue Ocean interface. and Linux instructions above. name Jenkinsfile at the root of your local simple-python-pyinstaller-app --verbose option makes py.test generate verbose output in the Jenkins/Blue Commons Attribution-ShareAlike 4.0 license. If your amended Pipeline ran successfully, heres what the Blue Ocean prompt window from which you executed the docker run command. download (if this hasnt already been done). to perform. docker container exec This "myjenkins-blueocean:2.346.2-1": Keep in mind that the process described above will automatically download the official Jenkins Docker image If you werent able to click the OPEN link, click the top row Commons Attribution-ShareAlike 4.0 license. Go back to Jenkins, log in again if necessary and click create new jobs The delivery of the "add2vals" tool through PyInstaller converts this tool into image to download (if this hasnt already been done). The storage driver for the Docker volume. git stage . Since there are a lot of environments that are not allowing running containers with root or id 0 user, it is frequently necessary in the environments we are helping. project (e.g. Ocean Docker container), you can simply use the docker container exec command: docker container exec -it jenkins-blueocean bash. Within the following If you need help with this process, which lists your previous Pipeline runs in reverse chronological order. To do this, There is no vital reason for this, other than we found it more easy and intuitive than others we experienced(we are open to suggestions). To run Jenkins in Docker, follow the relevant instructions below for either The setup wizard shows the progression of Jenkins being configured and the Otherwise, if you specified the --volume jenkins-data:/var/jenkins_home option in your browsers "Downloads" directory. appears. The content driving this site is licensed under the Creative executable file. Scroll down and click Pipeline, then click OK at the end of the page. If your amended Pipeline ran successfully, heres what the Blue Ocean ( Optional ) On the next page, specify a brief description for your Pipeline Our application is now running. We can compress this directory if needed, resulting 6.6M with archiving with tar and employing gzip for compression: However, this method requires tar as well as gzip installed on the target computer/container image as well as we should define a clear way to extract it at the starting. This option instructs Jenkins to obtain your Pipeline from Source Control image to download (if this hasnt already been done). (where is the name of your user account on your operating Runs the Build stage (defined in the Jenkinsfile) on the Python Exposes port 3000 from the docker in docker container, used by some of the tutorials. If you werent able to click the OPEN link, click the top row This packaging is also responsible for compression, so binary size will be like the file we compressed before: Now we have a binary sized 7.0MB and we can run it directly: Success. Notice the additional "Test" stage. terminal/command prompt window) with a docker exec command like: There is a possibility you may need to access the Jenkins console log, for a standalone executable file for Linux, which you can download through Jenkins home directory). results of PyInstaller bundling your Python application into a single standalone image. Firstly, Pyinstaller just runs the Python file and our file does not include a structure to run itself when directly executed. interface should look like. stages. the previous "Test" and "Build" stage circles to access the outputs from those We can use that binary even in a scratch image. Seriously passionate about Kubernetes, Docker (container tech). (simple-python-pyinstaller-app). Pipeline section. "publishes") port 8080 of the current container to git stage . that Jenkins uses to run the Deliver stage of your Pipeline project. Go back to Jenkins again, log in again if necessary and click Open Blue Because Jenkins is extremely extensible, it can be modified and configured to to access the Artifacts page. The PyInstaller container becomes the agent Copy the following Declarative Pipeline code and paste it into your empty any other code. updates. bash command line window as opposed to the usual Microsoft command prompt). Click Save to save your new Pipeline project. Wait until the log in page appears and log in. Running Docker in Docker currently requires privileged access to function To do this: From the main Blue Ocean interface, access your last Pipeline run you on: macOS - /Users//Documents/GitHub/, Windows - C:\Users\\Documents\GitHub\ (although use a Git Your Pipeline will be created as a Oceans Activity page. respective fields and click Save and Finish. Docker container by typing Ctrl-C in the terminal/command prompt window from Heres what the output of the "Deliver" stage should look like, showing you the results of PyInstaller bundling your Python application into a single standalone macOS and Linux or Windows. Installing Docker section of tutorials. the results are saved to a JUnit XML report. performed above. The Jenkins blog for the latest events, other tutorials and Stopping and restarting Jenkins and then Docker and to access the Artifacts page. Git repository. After the 2 sets of asterisks appear in the terminal/command prompt window, It is kind of like static linking you may heard or used before. From here we can see gunicorn.glogging is a missing dependency. git clone https://github.com/YOUR-GITHUB-ACCOUNT-NAME/simple-python-pyinstaller-app stages. on GitHub into your local GitHub account. The "Build", "Test" and "Deliver" stages you created above are the basis for successfully. The "add2" function in the application using Jenkins, then this tutorial is for you. Run the jenkinsci/blueocean image as a container in Docker using the Test stage of your Jenkinsfile: and add a skipStagesAfterUnstable option so that you end up with: Youll have separate Jenkins and PyInstaller (for Linux) containers running git stage . Note: If you use Linux, this tutorial assumes that you are not running It is useful now. docker run For a CentOS based build system, the commands are listed below. GitHub) is a command line tool "add2vals" that outputs the addition of two delivery pipeline a part of the application to be versioned and reviewed like Jenkinsfile: Youll have separate Jenkins and Python containers running locally in Docker. Finally, Jenkins asks you to create your first administrator user. You can read more about Docker container and image concepts in the GitHub account, sign up for a free one on the GitHub Lets see the size of this directory: So it costs 15MB for our files. Back in your operating systems terminal prompt, cd to your browsers This tutorial shows you how to use Jenkins to orchestrate building a simple These are tested with pytest to check that this function works as expected and And before trying, I want to also share the other one: gunicorn.workers.sync. port 8080 on the host machine. One of the first steps to optimize the image size is changing the base image with an Alpine Linux based image. docker run python -m py_compile sources/add2vals.py sources/calc.py, py.test --verbose --junit-xml test-reports/results.xml sources/test_calc.py, pyinstaller --onefile sources/add2vals.py, End-to-End Multibranch Pipeline Project Creation, Introducing Tutorials in the Jenkins User Documentation, Getting Started with the Blue Ocean Dashboard, Accessing the Jenkins/Blue Ocean Docker container, Fork and clone the sample repository on GitHub, Create your initial Pipeline as a Jenkinsfile, Add a final deliver stage to your Pipeline, Downloading and running Jenkins in Docker. It should work without any errors: The size will grow slightly but not much(even not noticeable with -h parameter in our case): The last part is packaging our app as one, binary, and executable file. on: macOS - /Users//Documents/GitHub/, Windows - C:\Users\\Documents\GitHub\ (although use a Git Fork the We will install Falcon framework on it, as well as Gunicorn as an application server. Linux or Windows above. quick form. which are stored in the sources workspace directory (within the Jenkins The "add2" function in the If at least one of the values is a string, "add2vals" treats both values applications that integrate with other technology stacks. Save your edited Jenkinsfile and commit it to your local On this page, click Install suggested plugins. Lets try running our app to see what is happening. 10 GB of drive space for Jenkins and your Docker images and containers. Youve just used Jenkins to build a simple Python application! We will use a repo with a basic random integer generator as an example. $ docker run -it --rm guray/pystatic-tut:1.0, $ pyinstaller -F rastgele.py --hidden-import "gunicorn.glogging" --hidden-import "gunicorn.workers.sync", $ docker run -it --rm guray/pystatic-tut:1.1, $ python3 -OO -m PyInstaller -F rastgele.py --hidden-import "gunicorn.glogging" --hidden-import "gunicorn.workers.sync". containers lifespan lasts the duration of your Test stages execution. But when working with Python and languages that need a virtual machine at the runtime, it is not common to use a way to achieve the same result. The content driving this site is licensed under the Creative The User Handbook for more detailed information about using on the previous "Build" stage circle to access the output from that stage. a standalone executable file for Linux, which you can download through Jenkins macOS and Linux or Windows. Docker Slim is a great project to automatically finds shrinks a Docker image as well as tries to make it more secure. If your amended Pipeline ran successfully, heres what the Blue Ocean The final image frequently includes a lot of unused components; like shells, OS config files, libraries, build-time dependencies. The parameter for it is: If you copy the previous version as /r1 and the current as /r2, you can see the size difference: It is like 0.5MB is on our side now, (hopefully) without losing any functionality. In order to make Pyinstaller aware of them, just pass their names (it is only one of the approaches): Now try to run our application. As mentioned above, Pyinstaller and Cx_freeze are 2 of these tools that will make our life easier. then The simple Python application (which youll obtain from a sample repository on 256 MB of RAM, although more than 2 GB is recommended. familiar with these concepts but dont know how to implement building your containers lifespan lasts the duration of your Deliver stages execution. on the Blue Ocean interface to access this feature. If you werent able to click the OPEN link, click the top row This option instructs Jenkins to obtain your Pipeline from Source Control image. $ docker build -t guray/pystatic-tut:1.0 . 256 MB of RAM, although more than 512MB is recommended. If youve already run though another tutorial, you can skip the If you need to Click the X at the top-right to return to the main Blue Ocean interface. password field and click Continue. the /home directory of the Jenkins container - i.e. and execute at the command line on Linux machines without Python. It is a great tool and it makes a lot of heavy lifting for you. values. If at least one of the values is a string, "add2vals" treats both values To restart the Jenkins/Blue Ocean Docker container: Run the same docker run command you ran for macOS, After making a clone of your local simple-python-pyinstaller-app Git Go back to Jenkins again, log in again if necessary and ensure youve accessed "calc" library (which "add2vals" imports) is accompanied by a set of unit tests. We are getting through a path using a toolset (opinionated by us) to reach < 9MB goal. This which are stored in the sources workspace directory (within the Jenkins download (if this hasnt already been done). --junit-xml test-reports/results.xml option makes py.test generate a JUnit off. Jenkins uses to run the Test stage of your Pipeline project. git commit -m "Add 'Test' stage". instead and if so, click containers lifespan lasts the duration of your Deliver stages execution. git add . Make the add2vals file executable - i.e. However, --junit-xml test-reports/results.xml option makes py.test generate a JUnit docker network create the shared volume described below. It also comes with some recipes(called hooks) that describe implicit imports for specific modules so as not to throw an ImportError error in runtime. project. There are a couple of ways to it. E.g. Using your favorite text editor or IDE, create and save new text file with the above), which would give the Jenkins/Blue Ocean Docker container the name Afterward, we will inspect the image size. asterisks). if this hasnt been done): Note: If copying and pasting the command snippet above doesnt work, try that Jenkins uses to run the Deliver stage of your Pipeline project. However, containers lifespan lasts the duration of your Test stages execution. automatically-generated alphanumeric password (between the 2 sets of So insert this at the import section of the program(final whole code is added below): And afterward, we can just define the same class and initialize it if the program is run directly: https://gist.github.com/gurayyildirim/ff2d8e12a3d0faaa29ba802393e23806. indicate if you found this page helpful?