- andrew conlin
- software engineer @ MathWorks
-
- -
-
- 
blog
-
-
-
-
- 
til
-
-
-
-
- 
projects
-
-
-
-
- 
photos↗
-
-
- - -
-
- Musings on the power of cinema -
-- -
-
- 
- -
I’ve realised that going to the cinema makes me sad; a quiet, contemplative melancholy. Ennui. No matter the film, something about the cinema experience is meditative for me. I always look at things differently in the aftermath; like my perception is temporarily knocked off balance. No, more like a fog is lifted for a while. I especially enjoy that time immediately after, the cycle home. I notice so many little details, scenes of beauty and of strange nostalgia. Like I’m finally seeing the world how it is, or rather, I begin to see rather than look, listen rather than hear. I feel as though these moments, however brief, are when I am truly in touch with myself. My soul, my inner essence. This is why I love the cinema. These moments of private melancholy, of temporary clarity. It is a feeling I cannot replicate anywhere else in my life.
- - --
-
- andrew conlin
- software engineer @ MathWorks
-
- -
-
- blog
-
-
-
-
- til
-
-
-
-
- projects
-
-
-
-
- photos↗
-
-
- - -
-
- Yuri Felsen on love -
-- -
-
-
- -
“Like everyone, I have my own, maybe obsessive futile, maybe in some way authentic, vision: all of a sudden, I will imagine the entire homogenous world as it is revealed to us-the streets, the cities, the rooms, those intelligent beasts of a sad and predatory nature, who have learn to stand on their hind legs, who have built all this but are fated to disappear, who, despite this, still try to cling to something solid and lasting, still try to ward off the inevitability of death, who dreamt up fairy tales and, now that these stories have been disproved, are disconsolate - and for me the only means of defending myself from our terrible fate is love, my love - Lyolya. Without love we fall into a stupor or despair, it covers our naked animal essence; with the fear of death, with deliberate attempts to grab hold of some kind of eternity, one that is at once a mystery to us and yet devised by us, even the remains of love, even its very echo in music, imbues us with a semblance of fearlessness, dignity and the spiritual range to disregard death. Only by loving, by knowing about love, hoping for love, are we inspired and meaningfully engaged in life, able to banish the sovereignty of petty day-to-day cares, to stop waiting for the end to come”
- -- Yuri Felsen (trans. Bryan Karetnyk)
- -From ‘Deceit’, first published in 1930 and translated to English in 2022.
- - --
-
- andrew conlin
- software engineer @ MathWorks
-
- -
-
- blog
-
-
-
-
- til
-
-
-
-
- projects
-
-
-
-
- photos↗
-
-
- - -
-
- A measure of how easily thoughts can be translated to action -
-- -
-
-
- -
Recently, I’ve been using multiple different languages and frameworks (both at work and personally) and I’ve been thinking about why I prefer some over others. Ultimately, I think it comes down to this:
- -How easily can I translate what I am thinking about into a working setup?
- -That is, how much effort needs to go into converting my idea into a solution? (If there is not already a word for this, then there should be. Don’t worry, I am willing to take the hit and have it named after me.)
- -For example, I’ve been building this website over the past month or so. Generally, for static websites, I think that HTML + CSS are very good at allowing me to get what I see in my head running on localhost. At this point CSS has so many different options that you just need to find the right search terms in order to get what you want.
However, they are both lacking somewhat in intuitiveness. Especially when starting out, it can get very confusing to know the right tags and the right syntax and the right placement (and so on) to realise your vision.
- -This is where services like Squarespace et al. come into play. They give you that intuitive UI, dragging and dropping of features, resizing and changing colours. They make it super duper easy to do the actual building, at the cost of the range of bricks you can use.
- -So, we have two measures:
--
-
- Can I do everything I want to? -
- How easy is it? -
These are usually a direct tradeoff, with ease typically preferred over options. This tends to frustrate me, as I want full control of all those little configurations.
- -The solution to this, at least for me, are frameworks such as Bootstrap and Jekyll. They put a layer between the user and the options; essentially translating and simplying all the little configurations into something much easier to work with. Columns in Bootstrap make page layouts simple; markdown-to-html parsing in Jekyll makes text formatting a breeze. But, most importantly of all, these frameworks don’t sacrifice on the possibilities. They make it easier to apply complex setups, but if you want to go in and tweak absolutely everything to your liking, go right ahead. By providing the user with simpler ways of doing the same things as before, they manage to bridge the gap between Can I do everything I want to? and How easy is it?. And I think that is beautiful.
- - --
-
- andrew conlin
- software engineer @ MathWorks
-
- -
-
- blog
-
-
-
-
- til
-
-
-
-
- projects
-
-
-
-
- photos↗
-
-
- - -
-
- Links aren't performances -
-- -
-
-
- -
“For starters, it finds that publishing a link can be a “performance” of a copyrighted work, which meant that when Linkletter linked to the world-viewable Youtube files that Proctorio had posted, he infringed on copyright.
- -This is a perverse, even surreal take on copyright. The court rejects Linkletter’s argument that even Youtube’s terms of service warned Proctorio that publishing world-viewable material on its site constituted permission for people to link to and watch that material.”
- -Via: Pluralistic
- - --
-
In this post, I will set out the existing problems with the game, then how I aim to resolve them, and conclude with a summary of the amazing of the brand-spanking new edition.
- -For the record, I am writing this post because the game is bad, not because I always lose to my girlfriend and have decided to change the rules rather than accept defeat1.
- -1. The problems
--
-
- Lack of options -
- Scissors picked disproportionately -
- General lack of whimsy -
2. Fixing the problems
-How many extra options should we add?
-Graph theory
-Before we dive in, let me give you some Graph Theory 101. First some terms:
--
-
- vertex: a point in space -
- edge: connects any two points -
- vertices are joined with edges to create graphs -
- to begin with, we will have undirected graphs, where edges have no direction associated with them -
Therefore, to represent the current setup for RPS, we can have the following
- - - - - -Each possible hand is a vertex, and each edge is a game. In graph theory, a complete graph is one where every pair of vertices is connected by a unique edge. Intuitively you can see that rock, paper scissors is complete, otherwise there would be combinations that couldn’t be played against each other.
- -Another concept that I now need to introduce is that of degree. The degree of a vertex is the total number of edges it is connected to. This is the second criterion that we need to fulfill: Every vertex must have the same degree. This is known as a regular graph. The exisiting RPS graph is also regular, with degree 2 (2-regular). We need to satisfy this with our new graph.
- -It is now time to introduce direction to our graph, used to represent winners and losers. Here is the RPS graph with directions representing who will win each game, also known as a digraph.
- - - -Rock beats scissors etc. etc.
- -More specifically, we want an oriented graph. There shouldn’t be any bidirectionality between vertices, which would indicate a draw.
- -A complete, oriented graph is called a tournament.
- -Buidling on out definition of regularity from the non-directed graphs, we have directional regularity (indegree = outdegree). A tournament where each of the vertices is regular is called a “regular tournament”. This specific type of digraph satisfies all our criteria, and any new game that we create must be regular tournament.
- --
-
- complete (every hand can play every hand) -
- regular (every hand can participate in the same number of possible games) -
- oriented (every game results in a winner) -
- indegree = outdegree for all vertices (each hand wins as many as it loses) -
Number of regular tournaments for numbers of nodes (even no. of nodes excluded because they have an odd degree, and therefore no regular tournaments):
- -| Nodes | -Degree | -Regular Tournaments | -
|---|---|---|
| 1 | -0 | -1 | -
| 3 | -2 | -2 | -
| 5 | -4 | -24 | -
| 7 | -6 | -2640 | -
| . | -. | -. | -
| . | -. | -. | -
as much as I would love to look at all of the possible 2640 combinations for a 7 player tournament, 5 possible hands looks like our sweetspot here, both in terms of the possible winner/loser combinations and in terms of the brain capacity of people playing the game.
- - - -This took me so long to figure out2
- -]]>Background
-Rock, Paper, Scissors needs an update. -In this post I will set out the threory behind
-How many extra options should we add?
-Graph theory
-Before we dive in, let me give you some Graph Theory 101. FIrst some terms:
--
-
- vertex: a point in space -
- edge: connects any two points -
- vertices are joined with edges to create graphs -
- to begin with, we will have undirected graphs, where edges have no direction associated with them -
Therefore, to represent the current setup for RPS, we can have the following
- “rock”
- o
- /
- /
- /
- paper o——-o scissors
Each possible hand is a vertex, and each edge is a game. In graph theory, a complete graph is one where every pair of vertices is connected by a unique edge. Intuitively you can see that rock, paper scissors is complete, otherwise there would be combinations that couldn’t be played against each other.
- -Another concept that I now need to introduce is that of degree. The degree of a vertex is the total number of edges it is connected to. This is the second criterion that we need to fulfill: Every vertex must have the same degree. This is known as a regular graph. The exisiting RPS graph is also regular, with degree 2 (2-regular). We need to satisfy this with our new graph.
- -It is now time to introduce direction to our graph, used to represent winners and losers. Here is the RPS graph with directions representing who will win each game, also known as a digraph.
- - "rock"
- o
- ^ \
- / \
- / v paper o <------o scissors
-Rock beats scissors etc. etc.
- -More specifically, we want an oriented graph. There shouldn’t be any bidirectionality between vertices, which would indicate a draw.
- -A complete, oriented graph is called a tournament.
- -Buidling on out definition of regularity from the non-directed graphs, we have directional regularity (indegree = outdegree). A tournament where each of the vertices is regular is called a “regular tournament”. This specific type of digraph satisfies all our criteria, and any new game that we create must be regular tournament.
- --
-
- complete (every hand can play every hand) -
- regular (every hand can participate in the same number of possible games) -
- oriented (every game results in a winner) -
- indegree = outdegree for all vertices (each hand wins as many as it loses) -
Number of regular tournaments for numbers of nodes (even no. of nodes excluded because they have an odd degree, and therefore no regular tournaments): -| Nodes | Degree | Regular Tournaments | -|—————- | ————— | ————— | -| 1 | 0 | 1 | -| 3 | 2 | 2 | -| 5 | 4 | 24 | -| 7 | 6 | 2640 | -and so on…
- -as much as I would love to look at all of the possible 2640 combinations for a 7 player tournament, 5 possible hands looks like our sweetspot here, both in terms of the possble winner/loser combinations and in terms of the brain capacity of people playing the game.
]]>This is a perverse, even surreal take on copyright. The court rejects Linkletter’s argument that even Youtube’s terms of service warned Proctorio that publishing world-viewable material on its site constituted permission for people to link to and watch that material.”
- -Via: Pluralistic
]]>I loved this idea, and thought it would be no problem to get a Python script up and running periodically on my Raspberry Pi home server using cron. However, I ran into various issues along the way (some of which were not so easy to resolve), so I’m collating all the configuration changes I made in the hopes that it will be useful to someone one day. You can find the full repo for this project here, and I have also included my Dockerfile, docker-compose.yml and crontab at the end of this TIL.
1. Double check the user
- -A lot of problems with cron come down to user privileges. Each user has their own crontab, and then there is the system-wide root crontab. The first issue I ran into with creating a cron job inside a container was that Docker created the crontab as a non-root user. This issue presented itself to me when I tried to run the following command, to list the current cronjobs in the Docker container:
docker-compose exec container-name crontab -l
-This returned the following output:
-no crontab for root
-Now, it is not necessarily a problem to have non-root cron jobs, but just make absolutely certain that you are creating the jobs with the user you expect. For me, I wanted to run as root, so I added to following line to my docker-compose.yml:
user: root
-Now, the root user will be used when building your Docker image and the created crontab will be where you expect.
2. Missing dependencies
-When cron calls your Python script, you may run into issues with ModuleNotFoundError or ImportError, where Python cannot find your installed packages. This is because cron does not have access to your system environment variables, including the Python path. You can resolve most of these errors with imports by adding the PYTHONPATH environment variable to your crontab. This should be the path to your site-packages folder, something like this:
PYTHONPATH=/usr/bin/local/python3
-You may also need to add a shebang (#!) to your Python script to direct cron to the correct version. You can find the Python location with one of the following commands:
which python
-which py
-which python3
-NOTE: These commands must be performed in your Docker container when it is up and running. In docker-compose syntax this would be the following (with the name of your container instead of container-name):
docker-compose exec container-name which python3
-You can then add this to the top of your Python script, as follows:
-#!/usr/bin/local/python3
-3. Still missing dependencies
-Some modules will still run into errors even when the PYTHONPATH variable has been set. In particular, I ran into problems with reportlab and Pillow/PIL:
ImportError: cannot import name '_imaging' from 'PIL'
-This was solved by adding the system PATH to the crontab as well. The system path is included in the default crontab that is created when you first run crontab -e:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-Therefore, it is a good idea to include it if you are making a new crontab to make sure cron can find everything it needs to.
4. Check relative paths in Python
-By default, cron runs from the default root path. Therefore, both your call to Python in your crontab and the filepaths within Python should either be relative to root (i.e /main.py rather than main.py) or just use full paths instead.
5. “Failed to build wheel” and related errors
-This error is related to Python inside a Docker container rather than cron. However, someone might still find it useful. When you install your requirements.txt, you may encounter errors such as
legacy-install-failure
-error: command '/usr/bin/gcc' failed with exit code 1
-fatal error: Python.h: No such file or directory
-I was able to resolve these by adding python3-dev, wheel and Cmake to my requirements.txt. These are sometimes required when packages include other binaries or need to compile other code when installed.
6. Other useful tips
--
-
- crontab.guru is a great resource for checking
cronsyntax
- - Installing vim/nano in your Docker container to make the debugging stage easier. This is especially useful for changing your crontab to run much more frequently, or adding debugging messages etc., when the container is up. -
I hope this helped you resolve some errors! I’ve included my Dockerfile, docker-compose.yml and crontab below if you want to set up a similar project or adjust your own files. The full repo is also available here.
- -Dockerfile:
-FROM python:3
-
-COPY . .
-RUN python3.11 -m pip install --no-cache-dir -r requirements.txt
-
-RUN touch /var/log/cron.log
-
-RUN apt-get update \
-&& apt-get install cron -y
-
-RUN chmod +x main.py
-
-RUN crontab crontab
-
-CMD cron -f
-docker-compose.yml:
-version: "2.4"
-
-services:
- watchman:
- platform: "linux/arm64/v8"
- image: watchman:latest
- container_name: watchman
- restart: always
- user: root
- build:
- context: build
- dockerfile: Dockerfile
-crontab:
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-PYTHONPATH=/usr/bin/local/python3
-15 7 * * * python3 /main.py >> /var/log/cron.log 2>&1
-
-How easily can I translate what I am thinking about into a working setup?
- -That is, how much effort needs to go into converting my idea into a solution? (If there is not already a word for this, then there should be. Don’t worry, I am willing to take the hit and have it named after me.)
- -For example, I’ve been building this website over the past month or so. Generally, for static websites, I think that HTML + CSS are very good at allowing me to get what I see in my head running on localhost. At this point CSS has so many different options that you just need to find the right search terms in order to get what you want.
However, they are both lacking somewhat in intuitiveness. Especially when starting out, it can get very confusing to know the right tags and the right syntax and the right placement (and so on) to realise your vision.
- -This is where services like Squarespace et al. come into play. They give you that intuitive UI, dragging and dropping of features, resizing and changing colours. They make it super duper easy to do the actual building, at the cost of the range of bricks you can use.
- -So, we have two measures:
--
-
- Can I do everything I want to? -
- How easy is it? -
These are usually a direct tradeoff, with ease typically preferred over options. This tends to frustrate me, as I want full control of all those little configurations.
- -The solution to this, at least for me, are frameworks such as Bootstrap and Jekyll. They put a layer between the user and the options; essentially translating and simplying all the little configurations into something much easier to work with. Columns in Bootstrap make page layouts simple; markdown-to-html parsing in Jekyll makes text formatting a breeze. But, most importantly of all, these frameworks don’t sacrifice on the possibilities. They make it easier to apply complex setups, but if you want to go in and tweak absolutely everything to your liking, go right ahead. By providing the user with simpler ways of doing the same things as before, they manage to bridge the gap between Can I do everything I want to? and How easy is it?. And I think that is beautiful.
]]>- Yuri Felsen (trans. Bryan Karetnyk)
- -From ‘Deceit’, first published in 1930 and translated to English in 2022.
]]>php errors.
-
-First error - “could not find driver”
- -I was trying using the following syntax to call occ and scan the files:
sudo -u www-data php /path/to/nextcloud/occ files:scan --all
-but I kept running into a PHP error. Specifically this error:
- -Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: could not find driver in /path/to/nextcloud/lib/private/DB/Connection.php:139
-followed by a long, verbose stack trace.
- -It took me a decent amount of time to diagnose the exact issue, but eventually I found this list of required PHP modules in the Nextcloud admin manual.
- -Running php -m will print out the list of currently installed PHP modules. I noticed I was missing quite a few of the required modules, but the one that was causing my issue was the missing pdo_mysql module.
-This can be installed by running:
sudo apt-get install php7.4-mysql
-Note: This command will change based on your OS, PHP version and database type
- -This resolved the error! However (as is always the case), this only meant I got a shiny new error instead:
- -Second error - “Name or service not known”
- -Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Name or service not known in /path/to/nextcloud/lib/private/DB/Connection.php:139
-From first glance, this looks like something wrong in the DNS name resolution. This sent me a long way down the wrong path, changing a whole bunch of things in my docker-compose.yml file.
-Eventually however, after a long and perilous journey over the high seas of Nextcloud forums and StackOverflow, I found this example of running php occ in a docker-compose configuration.
-This led me to running this command:
docker-compose exec -u www-data nextcloud-app php occ files:scan --all
-Note: replace nextcloud-app with the name of your Nextcloud container. Also, this command must be run from the directory of your Nextcloud docker-compose.yml
- -….aaaaaand, voila! The command runs, the files are scanned and everything is up to date.
- -
In this post, I will set out the existing problems with the game, then how I aim to resolve them, and conclude with a summary of the amazing of the brand-spanking new edition.
- -For the record, I am writing this post because the game is bad, not because I always lose to my girlfriend and have decided to change the rules rather than accept defeat1.
- -1. The problems
--
-
- Lack of options -
- Scissors picked disproportionately -
- General lack of whimsy -
2. Fixing the problems
-How many extra options should we add?
-Graph theory
-Before we dive in, let me give you some Graph Theory 101. First some terms:
--
-
- vertex: a point in space -
- edge: connects any two points -
- vertices are joined with edges to create graphs -
- to begin with, we will have undirected graphs, where edges have no direction associated with them -
Therefore, to represent the current setup for RPS, we can have the following
- - - - - -Each possible hand is a vertex, and each edge is a game. In graph theory, a complete graph is one where every pair of vertices is connected by a unique edge. Intuitively you can see that rock, paper scissors is complete, otherwise there would be combinations that couldn’t be played against each other.
- -Another concept that I now need to introduce is that of degree. The degree of a vertex is the total number of edges it is connected to. This is the second criterion that we need to fulfill: Every vertex must have the same degree. This is known as a regular graph. The exisiting RPS graph is also regular, with degree 2 (2-regular). We need to satisfy this with our new graph.
- -It is now time to introduce direction to our graph, used to represent winners and losers. Here is the RPS graph with directions representing who will win each game, also known as a digraph.
- - - -Rock beats scissors etc. etc.
- -More specifically, we want an oriented graph. There shouldn’t be any bidirectionality between vertices, which would indicate a draw.
- -A complete, oriented graph is called a tournament.
- -Buidling on out definition of regularity from the non-directed graphs, we have directional regularity (indegree = outdegree). A tournament where each of the vertices is regular is called a “regular tournament”. This specific type of digraph satisfies all our criteria, and any new game that we create must be regular tournament.
- --
-
- complete (every hand can play every hand) -
- regular (every hand can participate in the same number of possible games) -
- oriented (every game results in a winner) -
- indegree = outdegree for all vertices (each hand wins as many as it loses) -
Number of regular tournaments for numbers of nodes (even no. of nodes excluded because they have an odd degree, and therefore no regular tournaments):
- -| Nodes | -Degree | -Regular Tournaments | -
|---|---|---|
| 1 | -0 | -1 | -
| 3 | -2 | -2 | -
| 5 | -4 | -24 | -
| 7 | -6 | -2640 | -
| . | -. | -. | -
| . | -. | -. | -
as much as I would love to look at all of the possible 2640 combinations for a 7 player tournament, 5 possible hands looks like our sweetspot here, both in terms of the possible winner/loser combinations and in terms of the brain capacity of people playing the game.
- - - -This took me so long to figure out2
- -]]>This is a perverse, even surreal take on copyright. The court rejects Linkletter’s argument that even Youtube’s terms of service warned Proctorio that publishing world-viewable material on its site constituted permission for people to link to and watch that material.”
- -Via: Pluralistic
]]>How easily can I translate what I am thinking about into a working setup?
- -That is, how much effort needs to go into converting my idea into a solution? (If there is not already a word for this, then there should be. Don’t worry, I am willing to take the hit and have it named after me.)
- -For example, I’ve been building this website over the past month or so. Generally, for static websites, I think that HTML + CSS are very good at allowing me to get what I see in my head running on localhost. At this point CSS has so many different options that you just need to find the right search terms in order to get what you want.
However, they are both lacking somewhat in intuitiveness. Especially when starting out, it can get very confusing to know the right tags and the right syntax and the right placement (and so on) to realise your vision.
- -This is where services like Squarespace et al. come into play. They give you that intuitive UI, dragging and dropping of features, resizing and changing colours. They make it super duper easy to do the actual building, at the cost of the range of bricks you can use.
- -So, we have two measures:
--
-
- Can I do everything I want to? -
- How easy is it? -
These are usually a direct tradeoff, with ease typically preferred over options. This tends to frustrate me, as I want full control of all those little configurations.
- -The solution to this, at least for me, are frameworks such as Bootstrap and Jekyll. They put a layer between the user and the options; essentially translating and simplying all the little configurations into something much easier to work with. Columns in Bootstrap make page layouts simple; markdown-to-html parsing in Jekyll makes text formatting a breeze. But, most importantly of all, these frameworks don’t sacrifice on the possibilities. They make it easier to apply complex setups, but if you want to go in and tweak absolutely everything to your liking, go right ahead. By providing the user with simpler ways of doing the same things as before, they manage to bridge the gap between Can I do everything I want to? and How easy is it?. And I think that is beautiful.
]]>- Yuri Felsen (trans. Bryan Karetnyk)
- -From ‘Deceit’, first published in 1930 and translated to English in 2022.
]]>I loved this idea, and thought it would be no problem to get a Python script up and running periodically on my Raspberry Pi home server using cron. However, I ran into various issues along the way (some of which were not so easy to resolve), so I’m collating all the configuration changes I made in the hopes that it will be useful to someone one day. You can find the full repo for this project here, and I have also included my Dockerfile, docker-compose.yml and crontab at the end of this TIL.
1. Double check the user
- -A lot of problems with cron come down to user privileges. Each user has their own crontab, and then there is the system-wide root crontab. The first issue I ran into with creating a cron job inside a container was that Docker created the crontab as a non-root user. This issue presented itself to me when I tried to run the following command, to list the current cronjobs in the Docker container:
docker-compose exec container-name crontab -l
-This returned the following output:
-no crontab for root
-Now, it is not necessarily a problem to have non-root cron jobs, but just make absolutely certain that you are creating the jobs with the user you expect. For me, I wanted to run as root, so I added to following line to my docker-compose.yml:
user: root
-Now, the root user will be used when building your Docker image and the created crontab will be where you expect.
2. Missing dependencies
-When cron calls your Python script, you may run into issues with ModuleNotFoundError or ImportError, where Python cannot find your installed packages. This is because cron does not have access to your system environment variables, including the Python path. You can resolve most of these errors with imports by adding the PYTHONPATH environment variable to your crontab. This should be the path to your site-packages folder, something like this:
PYTHONPATH=/usr/bin/local/python3
-You may also need to add a shebang (#!) to your Python script to direct cron to the correct version. You can find the Python location with one of the following commands:
which python
-which py
-which python3
-NOTE: These commands must be performed in your Docker container when it is up and running. In docker-compose syntax this would be the following (with the name of your container instead of container-name):
docker-compose exec container-name which python3
-You can then add this to the top of your Python script, as follows:
-#!/usr/bin/local/python3
-3. Still missing dependencies
-Some modules will still run into errors even when the PYTHONPATH variable has been set. In particular, I ran into problems with reportlab and Pillow/PIL:
ImportError: cannot import name '_imaging' from 'PIL'
-This was solved by adding the system PATH to the crontab as well. The system path is included in the default crontab that is created when you first run crontab -e:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-Therefore, it is a good idea to include it if you are making a new crontab to make sure cron can find everything it needs to.
4. Check relative paths in Python
-By default, cron runs from the default root path. Therefore, both your call to Python in your crontab and the filepaths within Python should either be relative to root (i.e /main.py rather than main.py) or just use full paths instead.
5. “Failed to build wheel” and related errors
-This error is related to Python inside a Docker container rather than cron. However, someone might still find it useful. When you install your requirements.txt, you may encounter errors such as
legacy-install-failure
-error: command '/usr/bin/gcc' failed with exit code 1
-fatal error: Python.h: No such file or directory
-I was able to resolve these by adding python3-dev, wheel and Cmake to my requirements.txt. These are sometimes required when packages include other binaries or need to compile other code when installed.
6. Other useful tips
--
-
- crontab.guru is a great resource for checking
cronsyntax
- - Installing vim/nano in your Docker container to make the debugging stage easier. This is especially useful for changing your crontab to run much more frequently, or adding debugging messages etc., when the container is up. -
I hope this helped you resolve some errors! I’ve included my Dockerfile, docker-compose.yml and crontab below if you want to set up a similar project or adjust your own files. The full repo is also available here.
- -Dockerfile:
-FROM python:3
-
-COPY . .
-RUN python3.11 -m pip install --no-cache-dir -r requirements.txt
-
-RUN touch /var/log/cron.log
-
-RUN apt-get update \
-&& apt-get install cron -y
-
-RUN chmod +x main.py
-
-RUN crontab crontab
-
-CMD cron -f
-docker-compose.yml:
-version: "2.4"
-
-services:
- watchman:
- platform: "linux/arm64/v8"
- image: watchman:latest
- container_name: watchman
- restart: always
- user: root
- build:
- context: build
- dockerfile: Dockerfile
-crontab:
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-PYTHONPATH=/usr/bin/local/python3
-15 7 * * * python3 /main.py >> /var/log/cron.log 2>&1
-
-php errors.
-
-First error - “could not find driver”
- -I was trying using the following syntax to call occ and scan the files:
sudo -u www-data php /path/to/nextcloud/occ files:scan --all
-but I kept running into a PHP error. Specifically this error:
- -Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: could not find driver in /path/to/nextcloud/lib/private/DB/Connection.php:139
-followed by a long, verbose stack trace.
- -It took me a decent amount of time to diagnose the exact issue, but eventually I found this list of required PHP modules in the Nextcloud admin manual.
- -Running php -m will print out the list of currently installed PHP modules. I noticed I was missing quite a few of the required modules, but the one that was causing my issue was the missing pdo_mysql module.
-This can be installed by running:
sudo apt-get install php7.4-mysql
-Note: This command will change based on your OS, PHP version and database type
- -This resolved the error! However (as is always the case), this only meant I got a shiny new error instead:
- -Second error - “Name or service not known”
- -Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Name or service not known in /path/to/nextcloud/lib/private/DB/Connection.php:139
-From first glance, this looks like something wrong in the DNS name resolution. This sent me a long way down the wrong path, changing a whole bunch of things in my docker-compose.yml file.
-Eventually however, after a long and perilous journey over the high seas of Nextcloud forums and StackOverflow, I found this example of running php occ in a docker-compose configuration.
-This led me to running this command:
docker-compose exec -u www-data nextcloud-app php occ files:scan --all
-Note: replace nextcloud-app with the name of your Nextcloud container. Also, this command must be run from the directory of your Nextcloud docker-compose.yml
- -….aaaaaand, voila! The command runs, the files are scanned and everything is up to date.
- -
- andrew conlin
- software engineer @ MathWorks
-
- -
-
- blog
-
-
-
-
- til
-
-
-
-
- projects
-
-
-
-
- photos↗
-
-
- - -
-
- latest posts -
-- all the posts, in reverse chronological. there is an atom feed above. -
--
-
-
-
-
-
-
-
Enhancing Rock Paper Scissors
--- At least where I live, Rock Paper Scissors (henceforth referred to as RPS) is still the dominant way of quickly making a decision between friends. However, it is in desperate... -
-
-
-
-
-
-
-
-
-
- -
-
Links aren't performances
--- “For starters, it finds that publishing a link can be a “performance” of a copyrighted work, which meant that when Linkletter linked to the world-viewable Youtube files that Proctorio had... -
-
-
-
-
-
-
-
-
-
- -
-
A measure of how easily thoughts can be translated to action
--- Recently, I’ve been using multiple different languages and frameworks (both at work and personally) and I’ve been thinking about why I prefer some over others. Ultimately, I think it comes... -
-
-
-
-
-
-
- -
-
Yuri Felsen on love
--- “Like everyone, I have my own, maybe obsessive futile, maybe in some way authentic, vision: all of a sudden, I will imagine the entire homogenous world as it is revealed... -
-
-
-
-
-
-
-
-
-
- -
-
Musings on the power of cinema
--- I’ve realised that going to the cinema makes me sad; a quiet, contemplative melancholy. Ennui. No matter the film, something about the cinema experience is meditative for me. I always... -
-
-
-
-
-
-
-
- andrew conlin
- software engineer @ MathWorks
-
- -
-
- blog
-
-
-
-
- til
-
-
-
-
- projects
-
-
-
-
- photos↗
-
-
- - -
-
- latest tils -
-- my collection of tils (today-i-learned). i use this mainly to collect solutions to weird errors i have faced, and detail the little projects that i work on. this has its own atom feed (above) as they are typically very different from my other posts. all tils are blog posts but not all blog posts are tils. -
--
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Running a Python script periodically in a Docker container using cron
--- Recently, my partner gave a great idea for utilising my old Kindle: generate a “newspaper” each morning from a bunch of RSS feeds, and email it to the Kindle using... -
-
-
-
-
-
-
-
-
-
-
-
-
- -
-
Fixing php errors in a Nextcloud docker-compose configuration
--- I was trying to rescan the files in my Nextcloud server (running on Raspberry Pi 4 with DietPi), and kept running into some weird php errors. First error - “could... -
-
-
-
-
-
-
-
-
-
-
- andrew conlin
- software engineer @ MathWorks
-
- -
-
- blog
-
-
-
-
- til
-
-
-
-
- projects
-
-
-
-
- photos↗
-
-
- - -
-
- Fixing php errors in a Nextcloud docker-compose configuration -
-- -
-
-
- -
I was trying to rescan the files in my Nextcloud server (running on Raspberry Pi 4 with DietPi), and kept running into some weird php errors.
First error - “could not find driver”
- -I was trying using the following syntax to call occ and scan the files:
sudo -u www-data php /path/to/nextcloud/occ files:scan --all
-but I kept running into a PHP error. Specifically this error:
- -Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: could not find driver in /path/to/nextcloud/lib/private/DB/Connection.php:139
-followed by a long, verbose stack trace.
- -It took me a decent amount of time to diagnose the exact issue, but eventually I found this list of required PHP modules in the Nextcloud admin manual.
- -Running php -m will print out the list of currently installed PHP modules. I noticed I was missing quite a few of the required modules, but the one that was causing my issue was the missing pdo_mysql module.
-This can be installed by running:
sudo apt-get install php7.4-mysql
-Note: This command will change based on your OS, PHP version and database type
- -This resolved the error! However (as is always the case), this only meant I got a shiny new error instead:
- -Second error - “Name or service not known”
- -Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Name or service not known in /path/to/nextcloud/lib/private/DB/Connection.php:139
-From first glance, this looks like something wrong in the DNS name resolution. This sent me a long way down the wrong path, changing a whole bunch of things in my docker-compose.yml file.
-Eventually however, after a long and perilous journey over the high seas of Nextcloud forums and StackOverflow, I found this example of running php occ in a docker-compose configuration.
-This led me to running this command:
docker-compose exec -u www-data nextcloud-app php occ files:scan --all
-Note: replace nextcloud-app with the name of your Nextcloud container. Also, this command must be run from the directory of your Nextcloud docker-compose.yml
- -….aaaaaand, voila! The command runs, the files are scanned and everything is up to date.
- -
-
-
- andrew conlin
- software engineer @ MathWorks
-
- -
-
- blog
-
-
-
-
- til
-
-
-
-
- projects
-
-
-
-
- photos↗
-
-
- - -
-
- Running a Python script periodically in a Docker container using cron -
-- -
-
-
- -
Recently, my partner gave a great idea for utilising my old Kindle: generate a “newspaper” each morning from a bunch of RSS feeds, and email it to the Kindle using “Send-to-Kindle” feature (a blog post about this project is in the works).
- -I loved this idea, and thought it would be no problem to get a Python script up and running periodically on my Raspberry Pi home server using cron. However, I ran into various issues along the way (some of which were not so easy to resolve), so I’m collating all the configuration changes I made in the hopes that it will be useful to someone one day. You can find the full repo for this project here, and I have also included my Dockerfile, docker-compose.yml and crontab at the end of this TIL.
1. Double check the user
- -A lot of problems with cron come down to user privileges. Each user has their own crontab, and then there is the system-wide root crontab. The first issue I ran into with creating a cron job inside a container was that Docker created the crontab as a non-root user. This issue presented itself to me when I tried to run the following command, to list the current cronjobs in the Docker container:
docker-compose exec container-name crontab -l
-This returned the following output:
-no crontab for root
-Now, it is not necessarily a problem to have non-root cron jobs, but just make absolutely certain that you are creating the jobs with the user you expect. For me, I wanted to run as root, so I added to following line to my docker-compose.yml:
user: root
-Now, the root user will be used when building your Docker image and the created crontab will be where you expect.
2. Missing dependencies
-When cron calls your Python script, you may run into issues with ModuleNotFoundError or ImportError, where Python cannot find your installed packages. This is because cron does not have access to your system environment variables, including the Python path. You can resolve most of these errors with imports by adding the PYTHONPATH environment variable to your crontab. This should be the path to your site-packages folder, something like this:
PYTHONPATH=/usr/bin/local/python3
-You may also need to add a shebang (#!) to your Python script to direct cron to the correct version. You can find the Python location with one of the following commands:
which python
-which py
-which python3
-NOTE: These commands must be performed in your Docker container when it is up and running. In docker-compose syntax this would be the following (with the name of your container instead of container-name):
docker-compose exec container-name which python3
-You can then add this to the top of your Python script, as follows:
-#!/usr/bin/local/python3
-3. Still missing dependencies
-Some modules will still run into errors even when the PYTHONPATH variable has been set. In particular, I ran into problems with reportlab and Pillow/PIL:
ImportError: cannot import name '_imaging' from 'PIL'
-This was solved by adding the system PATH to the crontab as well. The system path is included in the default crontab that is created when you first run crontab -e:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-Therefore, it is a good idea to include it if you are making a new crontab to make sure cron can find everything it needs to.
4. Check relative paths in Python
-By default, cron runs from the default root path. Therefore, both your call to Python in your crontab and the filepaths within Python should either be relative to root (i.e /main.py rather than main.py) or just use full paths instead.
5. “Failed to build wheel” and related errors
-This error is related to Python inside a Docker container rather than cron. However, someone might still find it useful. When you install your requirements.txt, you may encounter errors such as
legacy-install-failure
-error: command '/usr/bin/gcc' failed with exit code 1
-fatal error: Python.h: No such file or directory
-I was able to resolve these by adding python3-dev, wheel and Cmake to my requirements.txt. These are sometimes required when packages include other binaries or need to compile other code when installed.
6. Other useful tips
--
-
- crontab.guru is a great resource for checking
cronsyntax
- - Installing vim/nano in your Docker container to make the debugging stage easier. This is especially useful for changing your crontab to run much more frequently, or adding debugging messages etc., when the container is up. -
I hope this helped you resolve some errors! I’ve included my Dockerfile, docker-compose.yml and crontab below if you want to set up a similar project or adjust your own files. The full repo is also available here.
- -Dockerfile:
-FROM python:3
-
-COPY . .
-RUN python3.11 -m pip install --no-cache-dir -r requirements.txt
-
-RUN touch /var/log/cron.log
-
-RUN apt-get update \
-&& apt-get install cron -y
-
-RUN chmod +x main.py
-
-RUN crontab crontab
-
-CMD cron -f
-docker-compose.yml:
-version: "2.4"
-
-services:
- watchman:
- platform: "linux/arm64/v8"
- image: watchman:latest
- container_name: watchman
- restart: always
- user: root
- build:
- context: build
- dockerfile: Dockerfile
-crontab:
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-PYTHONPATH=/usr/bin/local/python3
-15 7 * * * python3 /main.py >> /var/log/cron.log 2>&1
-
--
-