FFROGG - FFmpeg Recursive Ogg Encoding Script

What happens when you have thousands of songs that you want encoded, but either you

a) Have a command line system where no gui app can encode them for you?

b) Want more efficient use of your processor therefore do not use a gui?

c) are not good at computing and just want to get'r done?

This is one itch I just had to scratch. Now I'm aware of great apps like handbreak and I know amarok 1.4.x used to have a script to encode songs in a playlist. However, not only am I a big fan off ffmpeg, I needed to test out my bash-fu.

The scenario is this:

Joe user has tons of music spread out all over his hard drive. Lets say he has 10 000, none of which are contained in any one folder as more OCD folks would do. (Or an organized heirarchy of folders even)

Joe decides that one day, he wants to move to a patent-free codec such as .OGG format, but not only does he not have the time to FIND all his mp3's, he's clueless about how to encode them and knows of no gui apps (except for above mentioned in this article).

Joe user visits this article and decides that he wants all his files encoded into one folder on an external hard drive (which could be his mp3 player).

All Joe has to do is ensure that ffmpeg and libvorbis/libtheora are installed correctly on his *nix system and he can go about watching TV or playing with his dog/kids/wife while the machine does all the work.

Here's How:

1) Download the script @ [ http://www.mediafire.com/?lc023alg2bsfx9x ] to your home folder

2) Make sure it's executable, use chmod +x ffrogg.sh to do so.

3) Use the command sh ffrogg.sh --input=/home/$USER/ --output=/media/externaldisk/encoded --export --recursive (as long as your --input is linked to the top folder where your music is located, it should be fine)

4) Navigate to your output folder, in Joe User's case, it's /media/externaldisk/encoded/. There should be a script called "ffrogg-encodethis-[numbers].sh".

5) At his/your convenience, run the script. Example: sh ffrogg-encodethis-1279967631.sh

Technical Aspect:

Here's the really cool technical details of the script.

THIS SCRIPT IS CURRENTLY IN ALPHA, IT DOES NOT WORK 100%!

This script at its greatest potential will dynamically produce a script using the ffmpeg and theora-vorbis backend that can be run which will encode ALL RECOGNIZED AUDIO FILES that were found when it was invoked. It can recursively find all audio files starting at the specified input directory. Since this is alpha software, below is a list of what DOES work in the current version (0.01.05 ALPHA):

sh ffrogg.sh [commands]

• --input=[folder or file]
• --output=[folder only]
• --logging (this is for debugging purposes only, can specify a folder with --logging=[folder] if directory is read-only)
• --recursive (this flag is required to go into directories to find audio files. Otherwise it just looks at contents of specified directories)
• --export (Create the script and exit. Omitting this flag will cause the script to encode before exiting, although it gives a period to exit before starting)
• --force (No use for it yet, as the recovery handling bit is not quite implemented. Will remove temp files if they get in your way)
• --help (Gives a list of options, some of which will not work yet)
• --version (displays version of program)

This script creates a few temporary files which is inefficient, but this is by no means a professional work. It's also licensed under the GPL version 2 (although I may have omitted that in the script itself).

If the --output and --logging= and --save-state= flags are given without parameters, the current folder is automatically used "." .

Running the script without parameters will cause it to exit.

I may not get around to improving it much, and I do have a graphical user interface planned which would require basically a re-write of the script into a QT application, so it may not get past Alpha. Feel free to take it on yourselves as long as you give credit in the comments at the very least, to PeonDevelopments 2010.

Thanks, and happy listening.

Since I don't usually look at comments, send them to my gmail.

peon.developments --at-- gmail --dot-- com.

=-=-=-=-=
Powered by Blogilo

Scripting in Bash: Arrays, Files and Spaces

I recently encountered some strange behaviour in my bash shell while writing a script.

My script would take the contents of a folder and store them into an array. Sounds simple enough, as it should be. But it's not.

If you type verbatim:

" declare -a ARRAY=`ls -1 -x $directory` "

You will find that every space in every filename will produce its own entry in the array.

A file called "An Introduction To Bash.txt" would be seen in the array as

An

Introduction

To

Bash.txt

which is an abomination for what I need these the filenames for. It does make sense why this behaviour appears, but if we want the entire filename per entry, we need to take the filename before it's put into the $ARRAY variable and do something with it to ensure spaces either do not exist or do not denote the start of a new array token.

The former being the easiest and resourced method, I converted all spaces to underscores (temporarily) to prove my point.

The command is now " declare -a ARRAY=`ls -1 -x $directory | sed 's/ /_/g'` " and it will output filename entries properly, albeit with underscores, no spaces.

To ensure this effect is permanent and not just for show, one would need to rename (using the 'mv' command) all the files in that directory so no spaces are in the filenames.

Bash isn't one of my stronger languages, so I may have missed a blaringly obvious detail.

Also of note: I'm still working on the freeworld project, albeit slowly after gaining employment. My 1st revision document is approximately 70-80 pages, but that's not including proper use case tables and UML/flowchart diagrams. Least of all the prototypes for necessary components.

I hope to make another entry soon enough, in the coming couple weeks.

Peon out.

=-=-=-=-=
Powered by Blogilo

Minimalism on a Tablet

I've been experimenting with minimalist environments lately. While I'm a KDE guy all the way, I do have a tablet that even with KDE 4.4 on Slackware, it feels a little sluggish and gets ~ 2 hours of battery. I'm a bit of a perfectionist so onward I search for the perfect tablet PC configuration.

I'm no expert, but I've been using Blackbox as a window manager on a Gentoo system and have been quite pleased with the results. My average cpu while idle is 0-1% and currently my ram is sitting at 7% usage, or 76 MB out of 1 GB. (Courtesy of Conky, a great system monitor app)

Computer stats:

IBM Thinkpad X41

1.5 GHz LV Centrino (Pentium M)

1 GB DDR2 Ram (probably 533MHz or 667 MHz)

Intel GMA 910 integrated graphics chipset

ICH6 Intel Motherboard

1024x768 px screen resolution

Wacom Penabled

Functionality varies, and a minimal system isn't for everyone. I just compiled bmpanel (which looks great with transparency) for my bottom panel. Right click is the menu and after installation of a program, it must be regenerated. Some things have to be configured by hand and to use awesome programs from KDE and GNOME, both sets of libraries must be installed which amounts to ~70-90 MB for KDE and about ~33 MB for GNOME.

One program I would recommend for tablets is the GTK-based character input application, Cellwriter. I've only played with it currently, but it looks good. Once I add some script functionality to get the screen to rotate, a system tray, and a customized monitor layout, all will be good. Will update with screen shots later.

=-=-=-=-=
Powered by Blogilo

The FreeWorld Project *

It's time for the pre-introduction (unveiling) of a project years in the making. With an intermittant design phase of 7+ years and insight as a developer and programmer, PeonDevelopments presents a strong probability of a new multiplatform Java-based FOSS application dedicated to providng the greatest 2D Role Playing Game Engine to all Gamers, Casual Players, Developers, Marketers, Businesses and Educational Institutions interested.

Our mission statement is To provide developers and non-experienced users with a modular and FLOSS role playing game engine that performs reasonably well on multiple platforms and aids in the production of dynamic and advanced battle systems for the purpose of relating a story, plot or series of meaningful actions.

As the project is not mature and currently in vapourware status, I would like to remind readers that documentation is being developed to lay groundwork for the structure of the Engine. Expect a release of such documentation in 3 months of the posting of this article at the latest.

I will be posting a road map at a later date. I wish to produce a working proof-of-concept prototype with Installer before allowing others to work on developing this project. With that in mind, I have big expectations of this project and hope it will fill a niche that currently is not being filled to the extent it could be.

Projects such as Sphere do exist but are not maintained or as powerful as I want. More updates at a later stage.

Current documentaiton progress: At 40 pages. Expect 5x that.

* Not the official release name of the project, used as a placeholder

=-=-=-=-=
Powered by Blogilo

Technical-tainment

So as I'm sitting here at the table, drinking my light-roast coffee and feeling pretty good about life, it's time to let my readers know (all 1 of them, me) about some cool techy stuff that my mind is on.

http://kernelnewbies.org/Linux_2_6_32#head-935c2274be57c1585f23d91d185888ed0b6e0567

The above link is to the changelog for kernel 2.6.32 (which I currently use) and its improvements. Notably, some great improvements have been done in virtualization as with KSM (Kernel Shared Memory), a daemon process goes through page files determining which files are the same and merges them to reduce memory usage. That in itself is great but it doesn't put anything in perspective. Until you read the following:

"The result is a dramatic decrease in memory usage in virtualization environments. In a virtualization server, Red Hat found that thanks to KSM, KVM can run as many as 52 Windows XP VMs with 1 GB of RAM each on a server with just 16 GB of RAM."

If an XP machine was to be run under minimum requirements, using 128 MB of Ram, you'd get a barely functional OS and 52 of them would use a total of 6.656 GB of Ram. I've used XP in a VirtualBox VM using 512 MB of Ram and got some decent performance out of it. (Ran SAP and Borland Together in it; not at the same time) So assuming that each user is limited to a 512 MB Windows XP Pro OS, you'd be looking at a total of 26.624 GB of Ram required if these were dedicated machines. By sharing memory and having 52 XP machines running on a bare-bones Hardware Virtualization using Red Hat, each OS would require 307 MB of Ram if it were dedicated, but as it's sharing memory, the end result would mean more Ram to use. Assuming having shared memory resulted in each OS feeling as though it had 512 MB of Ram, that's a savings of ~10 GB of Ram, or an efficiency of 166.4% (2/3rds greater) over dedicated Ram resources.

And that's just one kernel improvement. I wonder how Windows kernel is doing in comparison.

The 2.6.32 kernel also has scheduler improvements in its CFQ low-latency mode.

Some people have had issues with KDE's Akonadi, getting a "Not Registered At DBUS" message. From my limited experience in dealing with this error, it generally only happens when trying to use an external SQL database. I believe that this is an upstream issue, as Akonadi is constantly being worked on during this time and big things are expected of it. (This has been tested on a KDE 4.3.5 and 4.4.2 system) The internal database doesn't seem to have this issue; it works at least, though sometimes produces a warning message. It may take a few more releases before this bug is dealt with,

In one Fedora forum, it seems that using an external sql database works under the root user. I wouldn't recommend this route.

Others say that it's an AppArmour issue, that it is blocking akonadi. It seems odd, as I still got this issue using Gentoo which doesn't have that firewall installed. The developer team which worked on it has been layed off as well.

http://news.cnet.com/8301-13580_3-9796140-39.html?part=rss&subj=news&tag=2547-1_3-0-5

There are some issues with Gentoo's KDE Unstable 4.4.2 release, though that's to be expected.

Sometimes closing Kontact does not completely close it and attempting to run it again will have no effect. The process must be killed before it can be sucessfully launched again.

Kile (2.1 beta 2 & 3) seems to have an issue where closing the program results in an unexpected error minutes later or if the program is left open, unattended for too long. It doesn't seem to impact my work (autosave it on) and is more of an annoyance. Still, it's a great program for working in LaTeX.

Umbrello is a nice UML modeller program, but I get segfaults semi-regularly. I'm not sure why that is and I still use it. This doesn't seem restricted to the unstable KDE release, but I thought I'd point it out anyway.

Some final thoughts on KDE 4.4.2: It's faster, looks better, and has more functionality. I'm getting used to tabbed file browsing and I like where the KDE team is headed.

A special thanks to Aaron J. Seigo, a KDE guru and expert. This guy is real inspiration and I hope to do some work on KDE in the near future.

=-=-=-=-=
Powered by Blogilo

Long Weekend

In writing the title of this post, the song by Scooter comes to mind.

Well it's the weekend, for me at least. Currently jobless and bored, it'll be a great geeky weekend indeed. In fact, my joblessness has become a catalyst for getting more involved in my personal work.

School is out and I went out yesterday to congradulate my fellow graduates from college (though I, for reasons none too serious, did not graduate this at the end of this year) and wish them the best. Once school was out, my body and mind did a crash nose-dive into my bed. For about a week or two I was sleeping approximately 12 hours a day, catching up on all those late nights/all nighters spent on our Capstone Project. (2-semeter long enterprise-level software project with proper documentation and installation)

Back to the topic at hand. Once I recovered from my exhaustion, I started back into the world of programming and computing. Re-built my resume from scratch using a personal custom design (in OpenOffice.org Draw program none-the-less) and managed to hook in one possible job prospect of which I'm only now getting the full force of what I've done.

As an aside, draw programs that I really recommend for GNU/Linux users are in fact, OpenOffice.org Draw and Krita for wonderful paint/publishing works and KolourPaint for the equivalent of Window's paint.exe program. Not much experience with The Gimp, but once they implement tabs and one window (I think there's a plugin for that, photogimp or something) then I'll go for a more professional bit.

So I met one of the management team at the Calgary Open Source Software Festival (2010) as he was a speaker there. Had a little chat afterwards and got a business card. Weeks later when I got around to applying for jobs, I mentioned this past experience and added a cover letter. Well, about a week later the company and I touched bas. They are very busy, so I am quite lucky to have gotten the attention I did; And while they don't have many positions open, it's quite possible that they will have room to train me in multiple areas and the *best* part is, they may create a System Administration position later on and have me in charge of that!

Now, the mere thought of the potential here is exciting for me. As I mentioned in the interview and will mention here now,

"In 5 years I see myself as a Software Developer. In 25 years, I see myself as a Senior System Administrator."

Basically saying that while I'm quite a geek and willing to do any odd jobs including software development of which I truly enjoy, I have a career goal and in one near-coincidental meeting (assuming I actually believe in those), my entire life as a geek may have been set up perfectly. It's much too early to say for sure and anything could get in the way of this. Nothing is conclusive and I have alternate paths if this blows up. But to be taken seriously, me, a young 22-year old whippersnapper as a possible dba / system admin, this is almost unprecedented! Basically this entire series of events rubbed my ego to the hilt. =)

Best part of all: It's mainly GNU/Linux work. Right up my alley.

So what am I going to do this weekend? Catch up on System Administration tools and information. Read a book. Play around with a project of mine. Catch up on tech news feeds. Learn some more C++ and start on Python. Build a Plasmoid. Something, anything productive. And I'm going to enjoy it.

We'll see where this leads. God willing, everything will work out in the end. It always does though, even if it's not how I want it to. But even in my darkest hours, I become stronger and more knowledgable so I don't see any downside to being rejected as a Sys Admin either.

I hope everyone enjoys their weekend.

- NP

=-=-=-=-=
Powered by Blogilo

Life to CURDATE();

School's almost done and the whether is cold and blizzardy. Or, at least it was a few nights ago; That's true Canadian weather for you.

I'm currently finished writing all my assignments and have a quiz and presentation left before I am free at long last. All this free time has also meant I've been tinkering around with my computer more often than usual. Take, for instance, yesterday. While no one was around to see it, I basically spent the whole day compiling on my laptop. A lot has happened since my last post and allow me to bring you up to speed.

+ Sold my Alienware machine that I previously had Gentoo on. Sad, I miss the machine, but some hardware or driver hiccups meant that it was constantly freezing. I'm wary that the problems stemmed either from a bad hard drive or the Vista drivers it came with. Perhaps it is also indicitive of some poor Kernel drivers too, I dunno. Doubtful though. One strange issue I had is that after installing Kubuntu 8.10, it would freeze if and only if the power manager was not on performance. Well such is life. Sold it, payed for another month of insurance, and purchased a beautiful Thinkpad T60 for a fraction of the Alienware's cost.

+ T60 Specs: 2.0 GHz Core Duo, 2GB DDR2 Ram, 512 MB x1400 ATI Radeon video (128 MB dedicated), 100GB 5400 rpm hard drive, 1400x1050 resolution, keyboard light, security fingerprint reader, and a decent speaker system. Total Price? $375. It was a deal to be sure, the last one in stock. Looked brand new except for the outer lid which had some scratches on it. This runs really cool, standard at 40 degrees idle to around 50-70 when working, compiling, and of course, running flash videos. It doesn't have amd64 support unfortunately, one thing I miss. But the keyboard on this thing is where it really shines. IBM and keyboards are like bread and butter. Perhaps I'll pick up an IBM Model M in my lifetime. I remember using them way back and enjoying every minute of it.

+ Installed Gentoo on the T60. Overall, things went smoothly on the third install attempt. I have issues with perfection and things weren't up to snuff the first two times. But I'm becoming more comfortable with the Gentoo environment and more prone to try fixing problems as opposed to the 'reinstall when something goes wrong' methodology which I have yet to shake off. I installed KDE 4.3.5 and that ran wonderfully.

+ Compiled a kernel for my T60. I need to enable plug-and-play usb mouse and keyboard support as those don't work yet, as well as the radeon fb compiled into the kernel. Currently it's as a module which doesn't work. However, responsiveness since the upgrade has increased wonderfully; A pre-emptive high low-latency kernel also helps.

+ Installed KDE 4.4 series unstable on Gentoo. Had some issues, will mention them in my next post. Still loved every minute of using this faster, more streamlined system.
+ Preparing my resume for my life as a cubicle slave. Just kidding though. While I have no qualms about being a code monkey/data entry clerk/ tech documentor, it's not something my ambition will sit well with over the years. I am in fact hoping to eventually work with the government on education, but we'll see where that leads me.

+ Excited to work on my own projects and plans, make some money, pay off debts (first priority), and collect more computer toys. While I love having the fastest and greatest/latest electronic toys in my paws, I also have learned to appreciate the stability and endurance of legacy machines. Remember when towers were made of steel? IBM Keyboard Model M's are still known for their quality. Nowadays, we get cheap keyboard crap built into plastic laptops such as the Toshiba A9, with one of the most gimped keyboards I've ever used. I'm still not sure which is worse: a small backspace key, or a small shift key. Both irritate me like a papercut to the eyeball.

And so, now with a moderately set up Gentoo machine with KDE I feel I am ready to start programming full force, starting with basic plasmoids and simple GUI's in QT4. I still hope to work in Java and am very proud of a Binary Search Tree I built last year for my college. Why are people still using ArrayLists and the like?


Check out my next post for KDE-specific notes and a very frustrating error I think I've gained an understanding on.