How to waste time and overcomplicate things

26/02/22

A quick word

This post ended up being fairly popular on Hacker News and sparked conversations about project management and development in the IT industry. You can find the post here.

The problem

My homelab is connected to the internet with a 20mb/s down and 5mb/s up connection, just enough for a remote 720p stream at decent quality without completely saturating the link. The problem is that the Infuse app I use will not request my Jellyfin server to transcode the video. So if I play a 4K video file on Infuse, my 5mb/s upload speed can’t keep up.

The solution

I found this application called Tdarr. A system for transcoding large libraries of media files. Great!

This is how I made it way more complicated than it should have been and wasted my time.

Step 1 - Make assumptions

Create a solution without understanding how it all works

While waiting for Tdarr to download and install I began thinking about how to manage the transcoded files. I assumed that the transcoded media files would be placed into the same directory as the source file.

So, I wrote a bash script to traverse my media directories looking for these new 720p files and then move them into a directory which would contain only 720p files.

Note

My media directory is structured like media/shows/[Series]/[Season]/[Episode] The 720p videos are structured media/720pShows/[Series]/[Season]/[Episode]

Writing this script included a method of detecting the current path of the 720p media file within my media directory and then replicating that in the 720p media directory. So directories would be created if they didn’t exist and then the 720p file was moved.

So I began setting up Tdarr and then realised that you can select the output of the converted file. This made all of my time and effort planning and then writing the bash script completely pointless.

Reflecting upon this, it seems like it is pretty obvious Tdarr would let you choose the output directory. I can’t tell you why I just assumed it wouldn’t.

Step 2 - Continue to make assumptions

Continue creating a solution without understanding how it all works

Ok, my media files are beginning to be converted and then saved into a temp directory I created to hold them. From there I’ll write a python script which reads the names of the files, parses them to get their series name and the season number. With the series name and the series number the script will create directories in the 720p directory so it would be structured like 720pShows/[Series]/[Season]/[Episode] and then the files will be moved.

I assumed Tdarr would place the converted video file into the temp directory as I had set it to. Makes sense right? I copied some video files into the temp directory to test it - and it worked! I set up a systemd timer to run my script every minute so that as newly converted files are placed into the temp directory they are moved into their new home.

I came back a short while later and checked the status of my systemd timer and all was well. I checked the 720p directory and all was not well there. It now contained directories named ‘Futurama’, ‘Season 2’, ‘Season 3’. Season folders which belong inside of series folders were not where they are supposed to be.

I assumed there must be some kind of error in my script’s parsing of the file title it uses to determine the directory structure. That wasn’t the case. If I copied the video file which was in the wrong folder into the temp directory and then ran my script manually the video file ended up in the correct place!

It was then that I realised that after converting a video file Tdarr will place that video file into the new directory with the same structure it came from. For example, a video which came from shows/[Series]/[Season]/[video file] will end up in temp/[Series]/[Season]/[video file].

My python script was pointless! If it creates the directories then all I need to do is move whatever is in the temp directory into the 720p directory.

Step 3 - Just fix the mistakes, don’t learn from them

After something has caused you multiple issues, don’t stop and reconsider your approach

Now that I know Tdarr already takes care of the directory structure for me all I need to do is move the temp directory contents into the 720p directory. I’ll set up an rsync move command on a systemd timer to do it every minute.

But wait… if Tdarr lets you choose the output directory and then even puts the output into the correct directory structure, why do I even need this temp directory and the rsync move command? Why not just point the output of Tdarr to the 720p directory?

It wasn’t until this point that I realised just how badly I had biffed this whole process.

Conclusion

I had a relatively simple task. Convert a batch of 4K and 1080p videos into 720p. I managed to overcomplicate and waste my time because I made assumptions and then failed to recognise mistakes and then repeated it.

In the future I plan on avoiding these mistakes by asking myself two questions:

  • Why am I doing this?

and then more importantly,

  • What is the evidence that this actually needs to be done?

I had a good reason to write those scripts because I thought I had a problem. If I had tried to find evidence that I actually had that problem and I needed those scripts (by just doing a test run) I quickly would have realised it was all unnecessary.

On a positive note, I was able to practice my bash and python skills so it wasn’t all for nothing.