I use Plex for managing and watching media that I own. Recently I got the boxed set of Game Of Thrones Blu-ray and wanted to put them in Plex so I could watch them on devices of my choice. Ripping them is easy enough – just use makemkv. But there were some problems in getting them into the form of naming that Plex wants for it to identify and present the episodes correctly:
- Plex wants TV series to be organized in folders by season. So Season 1 is in folder Season 01 and so on. This part is easy enough – when ripping we can specify the folder since we know what season we are ripping and we have to specify a folder anyway.
- Plex wants each episode file to be named
<series> S<season_number>E<episode_number>– this part is harder as makemkv just creates files with names<series>_season <season_number> Disc <disc_number>_t<number>.mkv - To complicate matters further, in this case, each disc had multiple versions of each episode and a bunch of extras. So apart from identifying which file is which episode number, I also had to de-duplicate the different versions of each episode and identify the extras (which is relatively easy by looking at file size).
After doing this for one disc manually, I decided it was too tedious (cue the xkcd comic on ROI of automating things) and looked for a better solution. Turns out there really isn’t a great solution out there. So, I ended up building my own – thanks to AI assisted coding, this was just a few hours of work and a lot of fun!
The general idea I had was the following:
- Given the series (and possibly the season information), I want to identify the TV episode number for a given video file.
- You could identify it through various means (image identification by taking frames and doing some kind of image search, video identification with similar approaches). The problem was, these were all quite resource intensive and also very error prone IMO. To be fair, I didn’t even bother trying them out – I briefly considered them and discarded them as too costly and too error prone based on instinct.
- Then I had this idea that text identification is a much cheaper and more robust way of doing this – but how do you get text? The idea I had here was utilizing subtitles.
- This then became the high-level plan:
- Given a video file, extract a sample of subtitles from it in the form of text.
- To identify the episode, just ask an LLM to identify the season and episode number given the series name and the sample of subtitles.
- Duplicates can be identified by duplication of the subtitles (even though the versions are different they are cuts of the same episode and mostly the same dialogs).
- For building this, Python was the most natural choice for me. I knew I would need some form of OCR and I knew Python has pytesseract which makes it trivial to do OCR.
Coding this with AI was interesting. I just use the free tiers of co-pilot, Gemini coding assistant in Visual Studio Code for my personal projects. Attempting this with those assistants was easy enough – co-pilot in particular with Claude Haiku pretty much got me the initial working version within an hour or so of prompting. However, I was not very happy with the initial version because it was quite slow. The initial version’s approach (since I am not very familiar with video file formats or using ffmpeg to process them, all my knowledge relied heavily on what the LLM told me):
- Extract subtitle event timestamps from the MKV file
- Extract frames from the MKV file corresponding to these event timestamps
- Do OCR on these extracted frames to get the subtitles
- Call the LLM with the OCR’d subtitles
The problem was both step 1 and step 2 were pretty slow – in particular, step 1 was processing the entire file. Another problem was that the extracted subtitle quality was pretty poor – roughly 50% of the subtitles were gibberish. I made some attempts at optimizing it but none were very effective and I wasted about 2 hours here.
After running into a brick wall on the optimization front (and really, I should have stepped back and asked better questions of the AI here), I switched to Gemini Pro on my Perplexity account to ask it how to improve OCR. Somewhere in that very helpful response, Gemini Pro just dropped this gem that you can ask ffmpeg to extract the subtitle stream directly as a data stream of images in SUP format. That was a light bulb moment for me – I asked it to immediately explain this better to me and this led me to the final optimized version of the pipeline:
- Extract a portion of the subtitle stream (PGS) in SUP format.
- Do OCR on the images extracted from this SUP format to get the subtitles.
- Call the LLM with the OCR’d subtitles
In the process, I also realized some basics of improving OCR quality that I really never thought of before but made perfect sense once the AI explained it to me:
- Converting the image from white text on black background to black text on white background since tesseract is optimized for OCR from paper.
- Scaling up the cropped portion with text so OCR can be more successful by having a magnified text image to analyze.
The final result is available as a python package and here’s the source for the utility. Here is the example usage of the utility:
tvidentify /path/to/TVShows/Game\ Of\ Thrones/Season\ 02/ --max-frames 10 --offset 3 --series-name "Game Of Thrones" --scan-duration 5 --output-dir ~/gots2 --model gemini-3-pro-preview --rename --skip-already-named