I recently setup a MythTV setup at my house and use a PS3 to stream over WiFi to my TV and sound system. Because stations broadcast digital signals now, what is broadcasted and received are 100% identical (provided no signal loss). The codec that my TV card captures in is straight MPEG-2 at a very high bitrate. “How high is it?” Well, I’m glad you asked! Depending on the station and program, a 1-hour show (with commercials) can run to just over 8 GB. Run some numbers, and this is combined bitrate is over 18.6 Mbps, or over 2.3 MB/s. 2.3 Megabytes a second. For reference, the absolute max that a video DVD can provide is 10.08 Mbps for both audio and video. (http://en.wikipedia.org/wiki/DVD-Video#Data_rate) It looks fantastic. The problem I run into though is wireless bandwidth. The PS3 only has 802.11g wireless (54 Mbps theoretical max). Take into count that that maximum is combine sending and receiving, signal degradation depending on countless factors, and other usage on the wireless network, and you are left being good to push 1 MB/s. Video compression to the rescue!
Robert Houghton wrote a perl script (that he modified from Nigel Pearson’s “Copy and transcode” script) and posted it on the MythTV wiki. In order to use it, FFmpeg has to be compiled and the script uses the x264 codec, a static video bitrate, the (expiremental encoding) AAC codec, and wouldn’t actually run for me due to the preset “hq” not existing. Since November, I had been playing with FFmpeg and have modified the script to more my taste. You still have to compile it, but FFmpeg’s Wiki outlines how to do this. The configure flags are correct, as they enable non-free and fdk_aac. A problem that I ran into (that fellow Ubuntu and Debain users can follow) is that by compiling as outlined on the FFmpeg wiki, only the compiling user would have access to FFmpeg and the codecs and filters. This can be fixed by following the guidance posted by using the checkinstall package.
Now the question as to why you would use my modified script over the source one.
- x264’s CRF feature in favor of a set bitrate. By setting a video bitrate, you determine the file-size and not the quality. Using the Constant Rate Factor, you set the quality and not the file-size. This means that for two shows of the exact same run-time can have very different bitrates. Animated shows compress much better than live-action, but they would be the approximate quality. The default CRF is 23, I have my scripts set to 22. With CRF, lower is better quality, and every 6 values result in approximately 2x the bitrate. (CRF 22 is 1/2 the size of CRF 16). Tailor to liking and viewing source.
- Mapping commands removed. I was having some problems with the script not running the command because of problems with mapping the audio and video streams. Commenting them out and removing them has only affected a single recording in the months of use. Most likely safe to leave out.
- AAC Codec. The AAC codec built-in to FFmpeg is experimental for encoding audio and the source script uses the FAAC codec. However, both of these codecs are not as good as other codecs. FFmpeg is actually banned in scene releases, as it either uses the FAAC codec or the built-in AAC codec and is inferior to their recommended Nero AAC encoder. HOWEVER, FFmpeg can use another AAC codec made by Fraunhofer. If that name is familiar, it should be. They are one of the companies that help make the MP3 standard, as well as AAC. Tests show that the Fraunhofer codec actually performs better than the Nero codec. This is one of the major reasons why FFmpeg must be compiled from source – in order to use a non-free, however highly superior audio codec. The bitrate remains unchanged. The FFmpeg AAC Encoding Guide has this to say about which AAC codec to use: “libfdk_aac > libfaac > Native FFmpeg AAC encoder (aac) > libvo_aacenc”
- A second script that sizes the video down. I have on my MythTV setup two scripts – a 720P resolution (1280 x 720) version for content I intend to watch in the living room and a 704×396 version. I use this for content that I intend to watch on my phone (Samsung Galaxy S III) and results in files about 1/2 the size of the 720p script. The audio bitrate is the same and the CRF is the same, only the video resolution is changed.
- Thread information removed. If your backend has more than 2 cores, use them all. The ffmpeg command is initiated using nice, which will give priority to other tasks anyways, so there is no reason to limit it.
- Preset changed from “HQ” to “veryfast”. As mentioned above, I was having issues even FINDING a hq preset. Also because of FFmpeg fork libav being the default packages in Debian and Ubuntu apt packages, the settings are different and sometimes incompatible. Which is another reason why you must compile FFmpeg proper and not apt-get install it! From my own tests and from other tests, the “veryfast” preset provides roughly the same filesize at a fraction of the encoding time. The quality is slightly lower than the “slow” preset, but it is, in my opinion, worth the speed boost. I have seen the conflicting results in this test; do note that this particular test was performed in 2010 before the presets were further refined and more presets were added. The 2013 Multimedia Mash results reflect that of my own.
Caveats and other notes:
- As mentioned above, the mapping doesn’t change anything for me – but it might with different broadcast areas.
- The audio streams tend to result in stereo only, but that is not an issue with me so I have chosen to not investigate it.
- Compile FFmpeg from source as outlined, and do a system install using checkinstall. Test the script running as your mythtv user (easy way to do so is su to root, then su to mythtv).
- “Masterpiece” shows from PBS encode, but there is an error in the name generation portion of the script (existing from the source script), so it will result in a file “Masterpiece_Classic_”. This is a valid MP4 video and is transcoded, but there isn’t a file extension on it and it isn’t added to the MythTV recordings database. I will be looking into this and how to fix it soon. UPDATE: Found the problem. The scripts at the bottom of the page have been fixed and updated.
- All successfully completed conversions have the same resolution flags, commercial flags, and other flags (including the auto-expire flag) from the source recording and are located in the “Recording” storage group / menu alongside the original MPEG-2 version.
- My backend is a Core2Duo with 1 GB of RAM (I have another 1 GB stick sitting around – just need to get around to installing it) and FFmpeg uses ~170-190 indicated in top (as it is using two cores) and typically will run at about 0.5x – 0.75x realtime (a 1 hour transcode will finish in 1 hour, 30 minutes typically at the 720p resolution).
- You must run the mythtv setup to allow user jobs, the commands for the respective user jobs (usage outlined in the scripts themselves), and chmod the script files to at least 755. I have chosen to place the scrips in my /usr/bin directory so all users have access to execute the scripts.
- Due to WordPress, I cannot upload the files as a .pl. I renamed them to ….pl.doc. After downloading, change the extension back to ….pl.
And now, here are the scripts that I’ve been using. If you find any changes, please let me know and I will be happy to look into it and change my scripts. Comment if this post has helped at all!
Gomez Development 
So I’ve just started running a MythTV backend myself – it’s crazy how hard it is to find up-to-date information for the simplest things, like x264 transcoding! Your script was a lifesaver. My only question is, this leaves me with two files, one original and one transcoded, both still in the database. I can always manually remove the original and manually fix the database with find_orphans.py, but I was wondering if you knew of a script I could use to remove the original after the transcode is complete? I’m unfortunately not well-enough versed in SQL to dive in myself. Regardless, thanks for the above script, it’s awesome!
Shannon, glad to hear that you got some use out of the script! I’ve upped the quality on my full-size transcode to “-crf 18,” and looks much better so keep that in mind if the video quality isn’t as good as you’d like. I preferred to keep the original recording handy just in case I were wanting to transcode it another way or archive it, I’d still have the original captured recording.
I haven’t tried it at all, but you might want to try adding the following lines after the existing script inserts the new, transcoded recording into the database and before it disconnects (line 349 or so):
$query = $db->prepare("DELETE FROM recorded $whereChanAndStarttime;");$query->execute || Die "Unable to query recorded table";
$query->finish;
$command = "rm $dir/$file";
system $command;
and experiment to see if it is indeed deleted from the database that way.
And of course, I have no idea if that’ll work or not, so do please backup your database in case it pulls a Bobby Tables.
Ahh! Don’t put it right at the end! Try to put it right after the FFmpeg command executes and before the transcoded version is added to the database. If you put it at the end as I mentioned before, the original and transcoded version would not be in the database and the mp4 will remain on the drive.
Wow, that was a fast response! (And a fast correction, as well! 😀 ) I can see (conceptually) why it would do that; one you put the new record in, they both have the same channel and start time. >.< I haven't had a chance to get on my system and play around yet, but I'll try putting that snippet between the transcode and the database entry and let you know what I see.
So far the original settings for the full-size script seem good on one of our mid-size TVs, but I haven't watched anything on the BIG TV yet, so I may make that change yet, depending on how it looks.
Okay, I think I got it hammered out; I put your codesnip right at line 327, after you pull the recording details from the old file, but before you insert them for the new file. If you run the delete before this, there are no old-file-details to get anymore; after, and it deletes both database entries because they both have the same channel/time information. I’ve tested it on several shows so far and it seems to work great, as long as the transcode completes without erroring out. 🙂
Oh, one other thing I did is I changed
$command = "rm $dir/$file";to$command = "rm $dir/$file*";– this takes care of any thumbnails that MythTV may have created as well as the original file, so they’re not cluttering the directory.Glad you figured out where to put the code snippet! It occurred to me that it had to go right before the transcoded file is added to the database, but hadn’t had a chance to reply back. Good call also on the thumbnails – forgot all about those!
Would you guys mind sharing the patched/enhanced script? Thanks!
Stefan,
I inserted the code snippet in above and made the change to include thumbnails as Shannon had recommended. The modified script is here. Again, I had to upload it as a .doc file to keep wordpress happy, so do
mv mpg2_to_x264_delete_original-pl.doc mpg2_to_x264_delete_original.pland be sure tochmod 755 mpg2_to_x264_delete_original.pl. Do note that this will delete the original MPEG (or whatever it is recorded as) and if you aren’t happy with the resulting quality, you will be up a creek. If you are happy with a CRF of 22, you’re set. For watching on my phone, CRF 22 is great. For watching on my living room TV, I use a separate script with CRF 18.Thanks a lot, no problem with your “doc-style” 😉
Unfortunately I have to debug some local problem with the usage of tvdb-stuff.
The mythtv-package on gentoo does currently not install the required python bindings, and your script calls /usr/share/mythtv/metadata/Television/ttvdb.py .. which doesn’t work for me currently. But I am on my way 😉
Thanks, Stefan
For now I commented the stuff around ttvdb.py to get to testing. I contacted the maintainer for mythtv in gentoo and will see if we get those bindings corrected.
So now it basically works and transcodes, I used the original non-removing version for getting comfortable … and see how I like CRF=18. Does adjusting CRF also influence overall conversion times? (My backend has a Xeon E3-1220 CPU, so it shouldn’t matter that much)
As I now get 2 files with same name listed (and 2 sizes): I suggest a parameter like “$newfile_prefix” which would be applied to the transcoded file. So that would make it easier to spot the transcoded versions in mythfrontend or similar. Just an idea, I will try if I can patch that by myself 😉
Thanks so far, Stefan
From my limited testing, a higher CRF (lower quality) tends to compress faster than one with a lower CRF. I am not sure if there is a consistent percentage to this though.
A prefix would be very nice to have! I want to say I looked into it, but my MySQL and scripting kung-fu are not very good, and gave up. If you do get it working, I’d be happy to add it to my personal scripts and update the scripts on my site!
You’re very welcome! As mentioned in the post, there didn’t seem to be a very good (and modern) MP4 conversion script out there so I made my own. I’m just glad that I’m able to help others who are having the same problem I was!
Hi again! Still poking around in this. I was able to get it to stop downcoding the audio from 6-channel to stereo by commenting out the following lines:
if ( $audiostreamsurround eq "" )
{
$audiochannels = 2;
$audiostream = $audiostreamstereo;
}
As best I can tell, the code that determines whether the audio is 6-channel or stereo (lines 193 and 198 in my script) are somehow always overridden at the lines above so that the output is almost always just stereo. Someone better with python than I would need to try and suss out exactly why
$audiostreamsurroundis blank even when the stream is definitely 5.1.The only drawback to commenting those lines out is, I have no idea what happens if you actually try to transcode a source that’s originally in steeo. As everything I record is in HD/6 channel, and all the devices I play back on support converting to stereo on their own, I don’t see it as ever being a problem for me.
Shannon, I did a quick ffmpeg test – telling FFMpeg to do 6 channels from stereo source will “convert” it to 5.1. Opening up this file in Audacity to see how FFMpeg handles the upconversion. The left and right audio tracks remain as left and right, and the remaining channels (rear left, rear right, center, sub) are completely empty. So, playing this on a surround sound system would result in only the front left and right channels playing. FYI.