From 83590d3916b6fbe8ee729b3a94b3a86d9a05e920 Mon Sep 17 00:00:00 2001 From: andrew Date: Sun, 6 Aug 2023 18:21:39 +0100 Subject: [PATCH] [06/08/23] sorted blog and til pages --- _config.yml | 14 +- ...2023-03-10-putting-thoughts-into-action.md | 9 +- ...-04-19-running-python-in-cron-in-docker.md | 9 +- .../putting-thoughts-into-action/index.html | 156 ++++++++++++++++++ _site/blog/category/til/index.html | 0 _site/blog/index.html | 21 +-- _site/blog/tag/cron/index.html | 0 _site/blog/tag/docker-compose/index.html | 0 _site/blog/tag/docker/index.html | 0 _site/blog/tag/kindle/index.html | 0 _site/blog/tag/nextcloud/index.html | 0 _site/blog/tag/php/index.html | 0 _site/blog/tag/python/index.html | 0 _site/feed.xml | 24 ++- _site/feed/til.xml | 139 ---------------- _site/sitemap.xml | 28 +--- _site/til/index.html | 9 +- blog.html | 8 +- til.html | 10 +- 19 files changed, 203 insertions(+), 224 deletions(-) rename {_drafts => _posts}/blog/2023-03-10-putting-thoughts-into-action.md (97%) create mode 100644 _site/blog/2023/03/10/putting-thoughts-into-action/index.html delete mode 100644 _site/blog/category/til/index.html delete mode 100644 _site/blog/tag/cron/index.html delete mode 100644 _site/blog/tag/docker-compose/index.html delete mode 100644 _site/blog/tag/docker/index.html delete mode 100644 _site/blog/tag/kindle/index.html delete mode 100644 _site/blog/tag/nextcloud/index.html delete mode 100644 _site/blog/tag/php/index.html delete mode 100644 _site/blog/tag/python/index.html delete mode 100644 _site/feed/til.xml diff --git a/_config.yml b/_config.yml index 51483fd..c847b06 100644 --- a/_config.yml +++ b/_config.yml @@ -29,17 +29,5 @@ defaults: category: til layout: post -jekyll-archives: - enabled: - - categories - - tags - layouts: - category: category - tag: tag - permalinks: - category: "/blog/category/:name/" - tag: "/blog/tag/:name/" - feed: - categories: - - til \ No newline at end of file + category: til \ No newline at end of file diff --git a/_drafts/blog/2023-03-10-putting-thoughts-into-action.md b/_posts/blog/2023-03-10-putting-thoughts-into-action.md similarity index 97% rename from _drafts/blog/2023-03-10-putting-thoughts-into-action.md rename to _posts/blog/2023-03-10-putting-thoughts-into-action.md index 2eb58cd..885530c 100644 --- a/_drafts/blog/2023-03-10-putting-thoughts-into-action.md +++ b/_posts/blog/2023-03-10-putting-thoughts-into-action.md @@ -1,13 +1,6 @@ --- title: A measure of how easily thoughts can be translated to action -categories: -- thoughts -tags: -- html -- css -- bootstrap -- jekyll -- webdev +category: blog --- Recently, I've been using multiple different languages and frameworks (both at work and personally) and I've been thinking about why I prefer some over others. Ultimately, I think it comes down to this: diff --git a/_posts/til/2023-04-19-running-python-in-cron-in-docker.md b/_posts/til/2023-04-19-running-python-in-cron-in-docker.md index 5e0c2a7..0e8ef95 100644 --- a/_posts/til/2023-04-19-running-python-in-cron-in-docker.md +++ b/_posts/til/2023-04-19-running-python-in-cron-in-docker.md @@ -1,13 +1,6 @@ --- title: Running a Python script periodically in a Docker container using cron -categories: -- til -tags: -- cron -- python -- docker -- docker-compose -- kindle +category: til --- Recently, my partner gave a great idea for utilising my old Kindle: generate a "newspaper" each morning from a bunch of RSS feeds, and email it to the Kindle using "Send-to-Kindle" feature (a blog post about this project is in the works). diff --git a/_site/blog/2023/03/10/putting-thoughts-into-action/index.html b/_site/blog/2023/03/10/putting-thoughts-into-action/index.html new file mode 100644 index 0000000..439c049 --- /dev/null +++ b/_site/blog/2023/03/10/putting-thoughts-into-action/index.html @@ -0,0 +1,156 @@ + + + + + + + + + andrew conlin + + + + + + + +
+

+ ac +

+
+
+ +
+
+

+ A measure of how easily thoughts can be translated to action +

+

+ +

+

+  10 Mar 2023 +    |    +  2 minutes +

+
+ +

Recently, I’ve been using multiple different languages and frameworks (both at work and personally) and I’ve been thinking about why I prefer some over others. Ultimately, I think it comes down to this:

+ +

How easily can I translate what I am thinking about into a working setup?

+ +

That is, how much effort needs to go into converting my idea into a solution? (If there is not already a word for this, then there should be. Don’t worry, I am willing to take the hit and have it named after me.)

+ +

For example, I’ve been building this website over the past month or so. Generally, for static websites, I think that HTML + CSS are very good at allowing me to get what I see in my head running on localhost. At this point CSS has so many different options that you just need to find the right search terms in order to get what you want.

+ +

However, they are both lacking somewhat in intuitiveness. Especially when starting out, it can get very confusing to know the right tags and the right syntax and the right placement (and so on) to realise your vision.

+ +

This is where services like Squarespace et al. come into play. They give you that intuitive UI, dragging and dropping of features, resizing and changing colours. They make it super duper easy to do the actual building, at the cost of the range of bricks you can use.

+ +

So, we have two measures:

+
    +
  1. Can I do everything I want to?
    2. +
  2. How easy is it?
    3. +
+ +

These are usually a direct tradeoff, with ease typically preferred over options. This tends to frustrate me, as I want full control of all those little configurations.

+ +

The solution to this, at least for me, are frameworks such as Bootstrap and Jekyll. They put a layer between the user and the options; essentially translating and simplying all the little configurations into something much easier to work with. Columns in Bootstrap make page layouts simple; markdown-to-html parsing in Jekyll makes text formatting a breeze. But, most importantly of all, these frameworks don’t sacrifice on the possibilities. They make it easier to apply complex setups, but if you want to go in and tweak absolutely everything to your liking, go right ahead. By providing the user with simpler ways of doing the same things as before, they manage to bridge the gap between Can I do everything I want to? and How easy is it?. And I think that is beautiful.

+ + +
+
+
+
+ + + + \ No newline at end of file diff --git a/_site/blog/category/til/index.html b/_site/blog/category/til/index.html deleted file mode 100644 index e69de29..0000000 diff --git a/_site/blog/index.html b/_site/blog/index.html index 045ec68..dec8e4c 100644 --- a/_site/blog/index.html +++ b/_site/blog/index.html @@ -121,29 +121,22 @@ + + +
  • -

    Running a Python script periodically in a Docker container using cron

    +

    A measure of how easily thoughts can be translated to action

    -  19 Apr 2023   |    -   til  +  10 Mar 2023
    - Recently, my partner gave a great idea for utilising my old Kindle: generate a “newspaper” each morning from a bunch of RSS feeds, and email it to the Kindle using... + Recently, I’ve been using multiple different languages and frameworks (both at work and personally) and I’ve been thinking about why I prefer some over others. Ultimately, I think it comes...

    • -
  • -

    Fixing php errors in a Nextcloud docker-compose configuration

    -
    -  01 Mar 2023   |    -   til  -
    - I was trying to rescan the files in my Nextcloud server (running on Raspberry Pi 4 with DietPi), and kept running into some weird php errors. First error - “could... -
    -
    -
    • +
    diff --git a/_site/blog/tag/cron/index.html b/_site/blog/tag/cron/index.html deleted file mode 100644 index e69de29..0000000 diff --git a/_site/blog/tag/docker-compose/index.html b/_site/blog/tag/docker-compose/index.html deleted file mode 100644 index e69de29..0000000 diff --git a/_site/blog/tag/docker/index.html b/_site/blog/tag/docker/index.html deleted file mode 100644 index e69de29..0000000 diff --git a/_site/blog/tag/kindle/index.html b/_site/blog/tag/kindle/index.html deleted file mode 100644 index e69de29..0000000 diff --git a/_site/blog/tag/nextcloud/index.html b/_site/blog/tag/nextcloud/index.html deleted file mode 100644 index e69de29..0000000 diff --git a/_site/blog/tag/php/index.html b/_site/blog/tag/php/index.html deleted file mode 100644 index e69de29..0000000 diff --git a/_site/blog/tag/python/index.html b/_site/blog/tag/python/index.html deleted file mode 100644 index e69de29..0000000 diff --git a/_site/feed.xml b/_site/feed.xml index f2957c2..7f9f37e 100644 --- a/_site/feed.xml +++ b/_site/feed.xml @@ -1,4 +1,4 @@ -Jekyll2023-08-06T18:05:21+01:00http://localhost:4000/feed.xmlAndrew ConlinRunning a Python script periodically in a Docker container using cron2023-04-19T00:00:00+01:002023-04-19T00:00:00+01:00http://localhost:4000/til/running-python-in-cron-in-dockerRecently, my partner gave a great idea for utilising my old Kindle: generate a “newspaper” each morning from a bunch of RSS feeds, and email it to the Kindle using “Send-to-Kindle” feature (a blog post about this project is in the works).

    +Jekyll2023-08-06T18:20:12+01:00http://localhost:4000/feed.xmlAndrew ConlinRunning a Python script periodically in a Docker container using cron2023-04-19T00:00:00+01:002023-04-19T00:00:00+01:00http://localhost:4000/til/running-python-in-cron-in-dockerRecently, my partner gave a great idea for utilising my old Kindle: generate a “newspaper” each morning from a bunch of RSS feeds, and email it to the Kindle using “Send-to-Kindle” feature (a blog post about this project is in the works).

    I loved this idea, and thought it would be no problem to get a Python script up and running periodically on my Raspberry Pi home server using cron. However, I ran into various issues along the way (some of which were not so easy to resolve), so I’m collating all the configuration changes I made in the hopes that it will be useful to someone one day. You can find the full repo for this project here, and I have also included my Dockerfile, docker-compose.yml and crontab at the end of this TIL.

    @@ -94,7 +94,27 @@ services: PYTHONPATH=/usr/bin/local/python3 15 7 * * * python3 /main.py >> /var/log/cron.log 2>&1 -
    ]]>Andrew ConlinFixing php errors in a Nextcloud docker-compose configuration2023-03-01T00:00:00+00:002023-03-01T00:00:00+00:00http://localhost:4000/til/nextcloud-php-errorsI was trying to rescan the files in my Nextcloud server (running on Raspberry Pi 4 with DietPi), and kept running into some weird php errors.

    +]]>    Andrew Conlin    A measure of how easily thoughts can be translated to action2023-03-10T00:00:00+00:002023-03-10T00:00:00+00:00http://localhost:4000/blog/2023/03/10/putting-thoughts-into-actionRecently, I’ve been using multiple different languages and frameworks (both at work and personally) and I’ve been thinking about why I prefer some over others. Ultimately, I think it comes down to this:

    + +

    How easily can I translate what I am thinking about into a working setup?

    + +

    That is, how much effort needs to go into converting my idea into a solution? (If there is not already a word for this, then there should be. Don’t worry, I am willing to take the hit and have it named after me.)

    + +

    For example, I’ve been building this website over the past month or so. Generally, for static websites, I think that HTML + CSS are very good at allowing me to get what I see in my head running on localhost. At this point CSS has so many different options that you just need to find the right search terms in order to get what you want.

    + +

    However, they are both lacking somewhat in intuitiveness. Especially when starting out, it can get very confusing to know the right tags and the right syntax and the right placement (and so on) to realise your vision.

    + +

    This is where services like Squarespace et al. come into play. They give you that intuitive UI, dragging and dropping of features, resizing and changing colours. They make it super duper easy to do the actual building, at the cost of the range of bricks you can use.

    + +

    So, we have two measures:

    +
      +
    1. Can I do everything I want to?
      2. +
    2. How easy is it?
      3. +
    + +

    These are usually a direct tradeoff, with ease typically preferred over options. This tends to frustrate me, as I want full control of all those little configurations.

    + +

    The solution to this, at least for me, are frameworks such as Bootstrap and Jekyll. They put a layer between the user and the options; essentially translating and simplying all the little configurations into something much easier to work with. Columns in Bootstrap make page layouts simple; markdown-to-html parsing in Jekyll makes text formatting a breeze. But, most importantly of all, these frameworks don’t sacrifice on the possibilities. They make it easier to apply complex setups, but if you want to go in and tweak absolutely everything to your liking, go right ahead. By providing the user with simpler ways of doing the same things as before, they manage to bridge the gap between Can I do everything I want to? and How easy is it?. And I think that is beautiful.

    ]]>    Andrew Conlin    Fixing php errors in a Nextcloud docker-compose configuration2023-03-01T00:00:00+00:002023-03-01T00:00:00+00:00http://localhost:4000/til/nextcloud-php-errorsI was trying to rescan the files in my Nextcloud server (running on Raspberry Pi 4 with DietPi), and kept running into some weird php errors.

    First error - “could not find driver”

    diff --git a/_site/feed/til.xml b/_site/feed/til.xml deleted file mode 100644 index a441047..0000000 --- a/_site/feed/til.xml +++ /dev/null @@ -1,139 +0,0 @@ -Jekyll2023-08-06T18:05:21+01:00http://localhost:4000/feed/til.xml | TilAndrew ConlinRunning a Python script periodically in a Docker container using cron2023-04-19T00:00:00+01:002023-04-19T00:00:00+01:00http://localhost:4000/til/running-python-in-cron-in-dockerRecently, my partner gave a great idea for utilising my old Kindle: generate a “newspaper” each morning from a bunch of RSS feeds, and email it to the Kindle using “Send-to-Kindle” feature (a blog post about this project is in the works).

    - -

    I loved this idea, and thought it would be no problem to get a Python script up and running periodically on my Raspberry Pi home server using cron. However, I ran into various issues along the way (some of which were not so easy to resolve), so I’m collating all the configuration changes I made in the hopes that it will be useful to someone one day. You can find the full repo for this project here, and I have also included my Dockerfile, docker-compose.yml and crontab at the end of this TIL.

    - -

    1. Double check the user

    - -

    A lot of problems with cron come down to user privileges. Each user has their own crontab, and then there is the system-wide root crontab. The first issue I ran into with creating a cron job inside a container was that Docker created the crontab as a non-root user. This issue presented itself to me when I tried to run the following command, to list the current cronjobs in the Docker container:

    -
    docker-compose exec container-name crontab -l
-
    -

    This returned the following output:

    -
    no crontab for root
-
    -

    Now, it is not necessarily a problem to have non-root cron jobs, but just make absolutely certain that you are creating the jobs with the user you expect. For me, I wanted to run as root, so I added to following line to my docker-compose.yml:

    -
    user: root
-
    -

    Now, the root user will be used when building your Docker image and the created crontab will be where you expect.

    - -

    2. Missing dependencies

    -

    When cron calls your Python script, you may run into issues with ModuleNotFoundError or ImportError, where Python cannot find your installed packages. This is because cron does not have access to your system environment variables, including the Python path. You can resolve most of these errors with imports by adding the PYTHONPATH environment variable to your crontab. This should be the path to your site-packages folder, something like this:

    -
    PYTHONPATH=/usr/bin/local/python3
-
    -

    You may also need to add a shebang (#!) to your Python script to direct cron to the correct version. You can find the Python location with one of the following commands:

    -
    which python
-which py
-which python3
-
    -

    NOTE: These commands must be performed in your Docker container when it is up and running. In docker-compose syntax this would be the following (with the name of your container instead of container-name):

    -
    docker-compose exec container-name which python3
-
    -

    You can then add this to the top of your Python script, as follows:

    -
    #!/usr/bin/local/python3
-
    -

    3. Still missing dependencies

    -

    Some modules will still run into errors even when the PYTHONPATH variable has been set. In particular, I ran into problems with reportlab and Pillow/PIL:

    -
    ImportError: cannot import name '_imaging' from 'PIL'
-
    -

    This was solved by adding the system PATH to the crontab as well. The system path is included in the default crontab that is created when you first run crontab -e:

    -
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-
    -

    Therefore, it is a good idea to include it if you are making a new crontab to make sure cron can find everything it needs to.

    - -

    4. Check relative paths in Python

    -

    By default, cron runs from the default root path. Therefore, both your call to Python in your crontab and the filepaths within Python should either be relative to root (i.e /main.py rather than main.py) or just use full paths instead.

    - - -

    This error is related to Python inside a Docker container rather than cron. However, someone might still find it useful. When you install your requirements.txt, you may encounter errors such as

    -
    legacy-install-failure
-error: command '/usr/bin/gcc' failed with exit code 1
-fatal error: Python.h: No such file or directory
-
    -

    I was able to resolve these by adding python3-dev, wheel and Cmake to my requirements.txt. These are sometimes required when packages include other binaries or need to compile other code when installed.

    - -

    6. Other useful tips

    -
      -
    • crontab.guru is a great resource for checking cron syntax
      • -
    • Installing vim/nano in your Docker container to make the debugging stage easier. This is especially useful for changing your crontab to run much more frequently, or adding debugging messages etc., when the container is up.
      • -
    - -

    I hope this helped you resolve some errors! I’ve included my Dockerfile, docker-compose.yml and crontab below if you want to set up a similar project or adjust your own files. The full repo is also available here.

    - -

    Dockerfile:

    -
    FROM python:3
-
-COPY . .
-RUN python3.11 -m pip install --no-cache-dir -r requirements.txt
-
-RUN touch /var/log/cron.log
-
-RUN apt-get update \  
-&& apt-get install cron -y
-
-RUN chmod +x main.py
-
-RUN crontab crontab 
-
-CMD cron -f
-
    -

    docker-compose.yml:

    -
    version: "2.4"
-
-services:
-  watchman:
-    platform: "linux/arm64/v8"
-    image: watchman:latest
-    container_name: watchman
-    restart: always
-    user: root
-    build:
-      context: build
-      dockerfile: Dockerfile
-
    -

    crontab:

    -
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-PYTHONPATH=/usr/bin/local/python3
-15 7 * * * python3 /main.py >> /var/log/cron.log 2>&1
-
-
    ]]>    Andrew Conlin    Fixing php errors in a Nextcloud docker-compose configuration2023-03-01T00:00:00+00:002023-03-01T00:00:00+00:00http://localhost:4000/til/nextcloud-php-errorsI was trying to rescan the files in my Nextcloud server (running on Raspberry Pi 4 with DietPi), and kept running into some weird php errors.

    - -

    First error - “could not find driver”

    - -

    I was trying using the following syntax to call occ and scan the files:

    - -
    sudo -u www-data php /path/to/nextcloud/occ files:scan --all
-
    - -

    but I kept running into a PHP error. Specifically this error:

    - -
    Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: could not find driver in /path/to/nextcloud/lib/private/DB/Connection.php:139
-
    - -

    followed by a long, verbose stack trace.

    - -

    It took me a decent amount of time to diagnose the exact issue, but eventually I found this list of required PHP modules in the Nextcloud admin manual.

    - -

    Running php -m will print out the list of currently installed PHP modules. I noticed I was missing quite a few of the required modules, but the one that was causing my issue was the missing pdo_mysql module. -This can be installed by running:

    - -
    sudo apt-get install php7.4-mysql
-
    -

    Note: This command will change based on your OS, PHP version and database type

    - -

    This resolved the error! However (as is always the case), this only meant I got a shiny new error instead:

    - -

    Second error - “Name or service not known”

    - -
    Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Name or service not known in /path/to/nextcloud/lib/private/DB/Connection.php:139
-
    - -

    From first glance, this looks like something wrong in the DNS name resolution. This sent me a long way down the wrong path, changing a whole bunch of things in my docker-compose.yml file. -Eventually however, after a long and perilous journey over the high seas of Nextcloud forums and StackOverflow, I found this example of running php occ in a docker-compose configuration. -This led me to running this command:

    - -
    docker-compose exec -u www-data nextcloud-app php occ files:scan --all
-
    -

    Note: replace nextcloud-app with the name of your Nextcloud container. Also, this command must be run from the directory of your Nextcloud docker-compose.yml

    - -

    ….aaaaaand, voila! The command runs, the files are scanned and everything is up to date.

    - -

    terminal output showing the files that have been successfully been scanned

    ]]>    Andrew Conlin     \ No newline at end of file diff --git a/_site/sitemap.xml b/_site/sitemap.xml index f39839d..2f76ffc 100644 --- a/_site/sitemap.xml +++ b/_site/sitemap.xml @@ -5,6 +5,10 @@ 2023-03-01T00:00:00+00:00 +http://localhost:4000/blog/2023/03/10/putting-thoughts-into-action/ +2023-03-10T00:00:00+00:00 + + http://localhost:4000/til/running-python-in-cron-in-docker/ 2023-04-19T00:00:00+01:00 @@ -17,28 +21,4 @@ http://localhost:4000/til/ - -http://localhost:4000/blog/tag/php/ - - -http://localhost:4000/blog/tag/nextcloud/ - - -http://localhost:4000/blog/tag/docker/ - - -http://localhost:4000/blog/tag/docker-compose/ - - -http://localhost:4000/blog/tag/cron/ - - -http://localhost:4000/blog/tag/python/ - - -http://localhost:4000/blog/tag/kindle/ - - -http://localhost:4000/blog/category/til/ - diff --git a/_site/til/index.html b/_site/til/index.html index 2b9d6a0..976e2d8 100644 --- a/_site/til/index.html +++ b/_site/til/index.html @@ -121,12 +121,10 @@ -

  • Running a Python script periodically in a Docker container using cron

    -  19 Apr 2023   |    -   til  +  19 Apr 2023
    Recently, my partner gave a great idea for utilising my old Kindle: generate a “newspaper” each morning from a bunch of RSS feeds, and email it to the Kindle using...
    @@ -137,11 +135,12 @@ + +

  • Fixing php errors in a Nextcloud docker-compose configuration

    -  01 Mar 2023   |    -   til  +  01 Mar 2023
    I was trying to rescan the files in my Nextcloud server (running on Raspberry Pi 4 with DietPi), and kept running into some weird php errors. First error - “could...
    diff --git a/blog.html b/blog.html index 2c9e508..c5eaafa 100644 --- a/blog.html +++ b/blog.html @@ -9,17 +9,17 @@ permalink: /blog/
      {% for post in site.posts %} - {% assign categories = post.categories %} - {% assign category = site.data.categories[categories.first] %} + {% assign category = post.category %} + {% if category == 'blog' %}

    • {{ post.title }}

      -  {{ post.date | date_to_string }}   |    -   {{ categories.first }}  +  {{ post.date | date_to_string }}
      {{ post.content | strip_html | truncatewords: 30 }}

      • + {% endif %} {% endfor %}
    \ No newline at end of file diff --git a/til.html b/til.html index bd29107..aeec96d 100644 --- a/til.html +++ b/til.html @@ -3,22 +3,18 @@ layout: default title: latest tils subtitle: my collection of tils (today-i-learned). i use this mainly to collect solutions to weird errors i have faced, and detail the little projects that i work on. this has its own atom feed (above) as they are typically very different from my other posts. all tils are blog posts but not all blog posts are tils. permalink: /til/ -categories: -- til ---
      {% for post in site.posts %} - {% assign categories = post.categories %} - {% assign category = site.data.categories[categories.first] %} - {% if categories.first == 'til' %} + {% assign category = post.category %} + {% if category == 'til' %}

    • {{ post.title }}

      -  {{ post.date | date_to_string }}   |    -   {{ categories.first }}  +  {{ post.date | date_to_string }}
      {{ post.content | strip_html | truncatewords: 30 }}