site/_site/feed/til.xml

<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.3.2">Jekyll</generator><link href="http://localhost:4000/feed/til.xml" rel="self" type="application/atom+xml" /><link href="http://localhost:4000/" rel="alternate" type="text/html" /><updated>2023-05-03T21:25:52+01:00</updated><id>http://localhost:4000/feed/til.xml</id><title type="html"> | Til</title><author><name>Andrew Conlin</name></author><entry><title type="html">Running a Python script periodically in a Docker container using cron</title><link href="http://localhost:4000/til/running-python-in-cron-in-docker/" rel="alternate" type="text/html" title="Running a Python script periodically in a Docker container using cron" /><published>2023-04-19T00:00:00+01:00</published><updated>2023-04-19T00:00:00+01:00</updated><id>http://localhost:4000/til/running-python-in-cron-in-docker</id><content type="html" xml:base="http://localhost:4000/til/running-python-in-cron-in-docker/"><![CDATA[<p>Recently, my partner gave a great idea for utilising my old Kindle: generate a “newspaper” each morning from a bunch of RSS feeds, and email it to the Kindle using “Send-to-Kindle” feature (a blog post about this project is in the works).</p>

<p>I loved this idea, and thought it would be no problem to get a Python script up and running periodically on my Raspberry Pi home server using <code class="language-plaintext highlighter-rouge">cron</code>. However, I ran into various issues along the way (some of which were not so easy to resolve), so I’m collating all the configuration changes I made in the hopes that it will be useful to someone one day. You can find the full repo for this project <a href="https://github.com/andrwcnln/watchman">here</a>, and I have also included my Dockerfile, docker-compose.yml and crontab at the end of this TIL.</p>

<h2 id="1-double-check-the-user">1. Double check the user</h2>

<p>A lot of problems with <code class="language-plaintext highlighter-rouge">cron</code> come down to user privileges. Each user has their own <code class="language-plaintext highlighter-rouge">crontab</code>, and then there is the system-wide <em>root</em> <code class="language-plaintext highlighter-rouge">crontab</code>. The first issue I ran into with creating a <code class="language-plaintext highlighter-rouge">cron</code> job inside a container was that Docker created the crontab as a non-root user. This issue presented itself to me when I tried to run the following command, to  list the current cronjobs in the Docker container:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker-compose exec container-name crontab -l
</code></pre></div></div>
<p>This returned the following output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>no crontab for root
</code></pre></div></div>
<p>Now, it is not necessarily a problem to have non-root <code class="language-plaintext highlighter-rouge">cron</code> jobs, but just make absolutely certain that you are creating the jobs with the user you expect. For me, I wanted to run as <code class="language-plaintext highlighter-rouge">root</code>, so I added to following line to my docker-compose.yml:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>user: root
</code></pre></div></div>
<p>Now, the <code class="language-plaintext highlighter-rouge">root</code> user will be used when building your Docker image and the created <code class="language-plaintext highlighter-rouge">crontab</code> will be where you expect.</p>

<h2 id="2-missing-dependencies">2. Missing dependencies</h2>
<p>When <code class="language-plaintext highlighter-rouge">cron</code> calls your Python script, you may run into issues with <code class="language-plaintext highlighter-rouge">ModuleNotFoundError</code> or <code class="language-plaintext highlighter-rouge">ImportError</code>, where Python cannot find your installed packages. This is because <code class="language-plaintext highlighter-rouge">cron</code> does not have access to your system environment variables, including the Python path. You can resolve most of these errors with imports by adding the <code class="language-plaintext highlighter-rouge">PYTHONPATH</code> environment variable to your <code class="language-plaintext highlighter-rouge">crontab</code>. This should be the path to your <code class="language-plaintext highlighter-rouge">site-packages</code> folder, something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PYTHONPATH=/usr/bin/local/python3
</code></pre></div></div>
<p>You may also need to add a shebang (<code class="language-plaintext highlighter-rouge">#!</code>) to your Python script to direct <code class="language-plaintext highlighter-rouge">cron</code> to the correct version. You can find the Python location with one of the following commands:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>which python
which py
which python3
</code></pre></div></div>
<p><em>NOTE</em>: These commands must be performed in your Docker container when it is up and running. In <code class="language-plaintext highlighter-rouge">docker-compose</code> syntax this would be the following (with the name of your container instead of <code class="language-plaintext highlighter-rouge">container-name</code>):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker-compose exec container-name which python3
</code></pre></div></div>
<p>You can then add this to the top of your Python script, as follows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/usr/bin/local/python3
</code></pre></div></div>
<h2 id="3-still-missing-dependencies">3. Still missing dependencies</h2>
<p>Some modules will still run into errors even when the PYTHONPATH variable has been set. In particular, I ran into problems with <code class="language-plaintext highlighter-rouge">reportlab</code> and <code class="language-plaintext highlighter-rouge">Pillow/PIL</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ImportError: cannot import name '_imaging' from 'PIL'
</code></pre></div></div>
<p>This was solved by adding the system PATH to the <code class="language-plaintext highlighter-rouge">crontab</code> as well. The system path is included in the default <code class="language-plaintext highlighter-rouge">crontab</code> that is created when you first run <code class="language-plaintext highlighter-rouge">crontab -e</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
</code></pre></div></div>
<p>Therefore, it is a good idea to include it if you are making a new <code class="language-plaintext highlighter-rouge">crontab</code> to make sure <code class="language-plaintext highlighter-rouge">cron</code> can find everything it needs to.</p>

<h2 id="4-check-relative-paths-in-python">4. Check relative paths in Python</h2>
<p>By default, <code class="language-plaintext highlighter-rouge">cron</code> runs from the default root path. Therefore, both your call to Python in your <code class="language-plaintext highlighter-rouge">crontab</code> and the filepaths within Python should either be relative to <code class="language-plaintext highlighter-rouge">root</code> (i.e <code class="language-plaintext highlighter-rouge">/main.py</code> rather than <code class="language-plaintext highlighter-rouge">main.py</code>) or just use full paths instead.</p>

<h2 id="5-failed-to-build-wheel-and-related-errors">5. “Failed to build wheel” and related errors</h2>
<p>This error is related to Python inside a Docker container rather than <code class="language-plaintext highlighter-rouge">cron</code>. However, someone might still find it useful. When you install your <code class="language-plaintext highlighter-rouge">requirements.txt</code>, you may encounter errors such as</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>legacy-install-failure
error: command '/usr/bin/gcc' failed with exit code 1
fatal error: Python.h: No such file or directory
</code></pre></div></div>
<p>I was able to resolve these by adding <code class="language-plaintext highlighter-rouge">python3-dev</code>, <code class="language-plaintext highlighter-rouge">wheel</code> and <code class="language-plaintext highlighter-rouge">Cmake</code> to my <code class="language-plaintext highlighter-rouge">requirements.txt</code>. These are sometimes required when packages include other binaries or need to compile other code when installed.</p>

<h2 id="6-other-useful-tips">6. Other useful tips</h2>
<ul>
  <li><a href="https://crontab.guru">crontab.guru</a> is a great resource for checking <code class="language-plaintext highlighter-rouge">cron</code> syntax</li>
  <li>Installing vim/nano in your Docker container to make the debugging stage easier. This is especially useful for changing your crontab to run much more frequently, or adding debugging messages etc., when the container is up.</li>
</ul>

<p>I hope this helped you resolve some errors! I’ve included my Dockerfile, docker-compose.yml and crontab below if you want to set up a similar project or adjust your own files. The full repo is also available <a href="https://github.com/andrwcnln/watchman">here</a>.</p>

<p>Dockerfile:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM python:3

COPY . .
RUN python3.11 -m pip install --no-cache-dir -r requirements.txt

RUN touch /var/log/cron.log

RUN apt-get update \  
&amp;&amp; apt-get install cron -y

RUN chmod +x main.py

RUN crontab crontab 

CMD cron -f
</code></pre></div></div>
<p>docker-compose.yml:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>version: "2.4"

services:
  watchman:
    platform: "linux/arm64/v8"
    image: watchman:latest
    container_name: watchman
    restart: always
    user: root
    build:
      context: build
      dockerfile: Dockerfile
</code></pre></div></div>
<p>crontab:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH=/usr/bin/local/python3
15 7 * * * python3 /main.py &gt;&gt; /var/log/cron.log 2&gt;&amp;1

</code></pre></div></div>]]></content><author><name>Andrew Conlin</name></author><category term="til" /><category term="cron" /><category term="python" /><category term="docker" /><category term="docker-compose" /><category term="kindle" /><summary type="html"><![CDATA[Recently, my partner gave a great idea for utilising my old Kindle: generate a “newspaper” each morning from a bunch of RSS feeds, and email it to the Kindle using “Send-to-Kindle” feature (a blog post about this project is in the works). I loved this idea, and thought it would be no problem to get a Python script up and running periodically on my Raspberry Pi home server using cron. However, I ran into various issues along the way (some of which were not so easy to resolve), so I’m collating all the configuration changes I made in the hopes that it will be useful to someone one day. You can find the full repo for this project here, and I have also included my Dockerfile, docker-compose.yml and crontab at the end of this TIL. 1. Double check the user A lot of problems with cron come down to user privileges. Each user has their own crontab, and then there is the system-wide root crontab. The first issue I ran into with creating a cron job inside a container was that Docker created the crontab as a non-root user. This issue presented itself to me when I tried to run the following command, to list the current cronjobs in the Docker container: docker-compose exec container-name crontab -l This returned the following output: no crontab for root Now, it is not necessarily a problem to have non-root cron jobs, but just make absolutely certain that you are creating the jobs with the user you expect. For me, I wanted to run as root, so I added to following line to my docker-compose.yml: user: root Now, the root user will be used when building your Docker image and the created crontab will be where you expect. 2. Missing dependencies When cron calls your Python script, you may run into issues with ModuleNotFoundError or ImportError, where Python cannot find your installed packages. This is because cron does not have access to your system environment variables, including the Python path. You can resolve most of these errors with imports by adding the PYTHONPATH environment variable to your crontab. This should be the path to your site-packages folder, something like this: PYTHONPATH=/usr/bin/local/python3 You may also need to add a shebang (#!) to your Python script to direct cron to the correct version. You can find the Python location with one of the following commands: which python which py which python3 NOTE: These commands must be performed in your Docker container when it is up and running. In docker-compose syntax this would be the following (with the name of your container instead of container-name): docker-compose exec container-name which python3 You can then add this to the top of your Python script, as follows: #!/usr/bin/local/python3 3. Still missing dependencies Some modules will still run into errors even when the PYTHONPATH variable has been set. In particular, I ran into problems with reportlab and Pillow/PIL: ImportError: cannot import name '_imaging' from 'PIL' This was solved by adding the system PATH to the crontab as well. The system path is included in the default crontab that is created when you first run crontab -e: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Therefore, it is a good idea to include it if you are making a new crontab to make sure cron can find everything it needs to. 4. Check relative paths in Python By default, cron runs from the default root path. Therefore, both your call to Python in your crontab and the filepaths within Python should either be relative to root (i.e /main.py rather than main.py) or just use full paths instead. 5. “Failed to build wheel” and related errors This error is related to Python inside a Docker container rather than cron. However, someone might still find it useful. When you install your requirements.txt, you may encounter errors such as legacy-install-failure error: command '/usr/bin/gcc' failed with exit code 1 fatal error: Python.h: No such file or directory I was able to resolve these by adding python3-dev, wheel and Cmake to my requirements.txt. These are sometimes required when packages include other binaries or need to compile other code when installed. 6. Other useful tips crontab.guru is a great resource for checking cron syntax Installing vim/nano in your Docker container to make the debugging stage easier. This is especially useful for changing your crontab to run much more frequently, or adding debugging messages etc., when the container is up. I hope this helped you resolve some errors! I’ve included my Dockerfile, docker-compose.yml and crontab below if you want to set up a similar project or adjust your own files. The full repo is also available here. Dockerfile: FROM python:3 COPY . . RUN python3.11 -m pip install --no-cache-dir -r requirements.txt RUN touch /var/log/cron.log RUN apt-get update \ &amp;&amp; apt-get install cron -y RUN chmod +x main.py RUN crontab crontab CMD cron -f docker-compose.yml: version: "2.4" services: watchman: platform: "linux/arm64/v8" image: watchman:latest container_name: watchman restart: always user: root build: context: build dockerfile: Dockerfile crontab: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PYTHONPATH=/usr/bin/local/python3 15 7 * * * python3 /main.py &gt;&gt; /var/log/cron.log 2&gt;&amp;1]]></summary></entry><entry><title type="html">Fixing php errors in a Nextcloud docker-compose configuration</title><link href="http://localhost:4000/til/nextcloud-php-errors/" rel="alternate" type="text/html" title="Fixing php errors in a Nextcloud docker-compose configuration" /><published>2023-03-01T00:00:00+00:00</published><updated>2023-03-01T00:00:00+00:00</updated><id>http://localhost:4000/til/nextcloud-php-errors</id><content type="html" xml:base="http://localhost:4000/til/nextcloud-php-errors/"><![CDATA[<p>I was trying to rescan the files in my Nextcloud server (running on Raspberry Pi 4 with DietPi), and kept running into some weird <code class="language-plaintext highlighter-rouge">php</code> errors.</p>

<h3 id="first-error---could-not-find-driver">First error - “could not find driver”</h3>

<p>I was trying using the following syntax to call <code class="language-plaintext highlighter-rouge">occ</code> and scan the files:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -u www-data php /path/to/nextcloud/occ files:scan --all
</code></pre></div></div>

<p>but I kept running into a PHP error. Specifically this error:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: could not find driver in /path/to/nextcloud/lib/private/DB/Connection.php:139
</code></pre></div></div>

<p>followed by a long, verbose stack trace.</p>

<p>It took me a decent amount of time to diagnose the exact issue, but eventually I found <a href="https://docs.nextcloud.com/server/20/admin_manual/installation/source_installation.html#prerequisites-for-manual-installation">this list</a> of required PHP modules in the Nextcloud admin manual.</p>

<p>Running <code class="language-plaintext highlighter-rouge">php -m</code> will print out the list of currently installed PHP modules. I noticed I was missing quite a few of the required modules, but the one that was causing my issue was the missing <code class="language-plaintext highlighter-rouge">pdo_mysql</code> module.
This can be installed by running:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get install php7.4-mysql
</code></pre></div></div>
<p><strong>Note: This command will change based on your OS, PHP version and database type</strong></p>

<p>This resolved the error! However (as is always the case), this only meant I got a shiny new error instead:</p>

<h3 id="second-error---name-or-service-not-known">Second error - “Name or service not known”</h3>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Name or service not known in /path/to/nextcloud/lib/private/DB/Connection.php:139
</code></pre></div></div>

<p>From first glance, this looks like something wrong in the DNS name resolution. This sent me a long way down the wrong path, changing a whole bunch of things in my docker-compose.yml file.
Eventually however, after a long and perilous journey over the high seas of Nextcloud forums and StackOverflow, I found <a href="https://techoverflow.net/2020/07/17/how-to-run-nextcloud-php-occ-in-a-docker-compose-configuration/">this example</a> of running <code class="language-plaintext highlighter-rouge">php occ</code> in a docker-compose configuration.
This led me to running this command:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker-compose exec -u www-data nextcloud-app php occ files:scan --all
</code></pre></div></div>
<p><strong>Note: replace nextcloud-app with the name of your Nextcloud container. Also, this command must be run from the directory of your Nextcloud docker-compose.yml</strong></p>

<p>….aaaaaand, <em>voila!</em> The command runs, the files are scanned and everything is up to date.</p>

<p><img src="/assets/images/occ.png" alt="terminal output showing the files that have been successfully been scanned" /></p>]]></content><author><name>Andrew Conlin</name></author><category term="til" /><category term="php" /><category term="nextcloud" /><category term="docker" /><category term="docker-compose" /><summary type="html"><![CDATA[I was trying to rescan the files in my Nextcloud server (running on Raspberry Pi 4 with DietPi), and kept running into some weird php errors.]]></summary></entry></feed>