![]() I could then make a few simple plots to look at memory. Thank goodness that Singularity has seamless connection to the host (binds, environment), because I asked watchme to record all metrics every 1 second, and I piped the json In the above, I monitored the command to run the container, singularity run watchme-mnist_latest.sif,įrom inside of the container. $ singularity exec watchme-mnist_latest.sif watchme monitor -seconds 1 python /run.py plots.png > mnist.json In parallel on my cluster, but I have no idea how much memory to ask for! Let’s step it up a notch, and try measuring the training of a model. You can also see higher user CPU times for a Then when we pull and extract layers, albeit the change being small, (represented by the terminal logging) corresponds with what we’ve measured:ĭid you see that? The first two seconds when we were “Starting Build” likely correspond with Sure, the units of change are very small, but we can watch a pull to see how behavior (slightly) more CPU intensive, like extraction of layers and then building the SIF binary. I would guess that Singularity starts running, and at around 2 seconds starts to do something But what I do find interesting is the kink The place, and hard to make any sort of conclusion. Yeah, I was a bit lazy to just show all the iterations on the same plot. And this (might be?) what we see - here is a pull ![]() I would expect that asking for less memory for a jobĬorresponds with getting less user CPU time. A higher value for this metric means that ![]() The processor spends pulling our container. What is cpu time in user space, as opposed to system / kernel space? It’s the amount of time First, let’s look at cpu time in user space during the pull of alpine. Most of these pulls are between 4 and 10 seconds, so there isn’t a ton of data recorded, but I’ll quickly show an example I installed watchme, and opted to pipe results directly into files named according to the parameters. How I’m handing off the process for watchme to run and watch. Notice how the “singularity pull” command is wrapped with “watchme monitor” - this is Is my quick submission loop (the sbatch command submits the job in the file pull-job.sh): ![]() This meant that I launched a job, and manipulated only the amount of memory. Is there varying performance based on the amount of memory available? The goal was to create plots, taking a measurement each second, and I wanted to collect resource usage during a Singularity pull of severalĬontainers including ubuntu, busybox, centos, alpine, and nginx. Why Should I Care? and then talk about why in the world you should care at all.įeel free to jump around if one is more interesting to you.Measure Memory Usage for a containerized sklearn model.Monitor Container Pulls on the Sherlock cluster using Singularity.If you are interested, here is an asciinema video of that in action.īut let’s skip over the dummy examples and jump into something a little more fun - using ![]()
0 Comments
Leave a Reply. |