Onlyblog... sample blog

Problem statement:

Grab HD clips from Sony HDR-HC7 HDV camcorder and convert them to HD videos on the web (Vimeo, YouTube, etc.) fairly easily on Linux - specifically, Ubuntu

I've been trying to address this problem for a while now. It has been about a year since I tried last. I remember using Kino and giving Cinelerra a shot. For variety of reasons, I did not like either and I gave up on the effort.

Recently, when Ubuntu 10.04 shipped with PiTiVi as the default video editor, I decided to give it another go. Since I had used OpenShot briefly in Ubuntu 9.10, I thought it would be educational to try both the video editors to accomplish the task. So I created a short sample video in both these editors - embedded below.

I am a video pro by no means. I am just a novice user trying hard to shoot least shaky HD movies and sharing them with family and friends - while not embarrassing myself! So please take my input and opinions with a grain of salt. But at the end of it all, I think I prefer OpenShot - at least for my camcorder, my level of skill and expertise using video editors.

Now that the result is out of the way, here's the rest of the story...

Workflow

Workflow I would like to set up is simple:

1. Grab clips from the camcorder
2. Convert clips to a format the video editor likes (I would rather not, but necessary with some editors)
3. Import them into the video editor
4. Edit/mix the clips and compose the video
5. Export it to desired target

Grabbing clips from the camcorder

This is something I learned from my previous experience with Kino. I use dvgrab to grab clips from the camcorder. Here is the source of the script I use:

Sony HDR-HC7 is an HDV camcorder. As such, the above script grabs the clips in form of a sequence of MPEG-2 transport stream (.m2t) files. I use the -timestamp option so that the names of the files include the time when the video was recorded. I also store the output of this script for later use. It includes useful information about the clips - like the timecode, length of the clip in frames, size of the clip etc. Here's a sample output:

Using OpenShot

Importing video

One thing I really loved about OpenShot is the fact that it can consume .m2t files! Kino did not, Cinelerra did not, PiTiVi does not. So this is great. It completely eliminates one step in my workflow.

Import function in OpenShot (the big green '+' sign in the toolbar) allows you to add video (my .m2t files), audio and other media files.

Editing video

For a simple video, like the one I created for this experiment, all you need to do is drag the clips into the timeline - the lower part of the interface - and position them to define the sequence of scenes in the movie.

Here are some editing/mixing features of OpenShot I used in the making of the sample video:

• The razor tool can be used to snip clips at any given position
• Markers can be added on the timeline to mark significant positions in the timeline
• Video in a clip can be turned ON or OFF
• Audio in a clip can be turned ON or OFF
• Effects like Fade in/Fade out can be applied to individual tracks
• There are many transitions and effects, which I did not use in the sample video

I did not play with titles, credits or subtitles, so I cannot say much about those features. All in all, you can do some basic editing and mixing really well.

Exporting video

Another thing you got to love is that OpenShot comes with canned export profiles. Profiles have export parameters set to work with DVD, Web (Vimeo, Youtube and the likes) etc. The fact that export to Vimeo-HD and Youtube-HD worked so well, make me feel very optimistic about DVD export - which is going to be my next endeavor.

Here's a screenshot of the export dialog.

And finally, here are the videos out of OpenShot, on both Vimeo (exported with Vimeo-HD profile) and Youtube (exported with Youtube-HD profile).

OpenShot Vimeo HD video:

OpenShot Youtube HD video:

Using PiTiVi

Importing video

Like I mentioned, PiTiVi does not consume .m2t files. Apparently, it has the capability, but there's probably a bug currently, that keeps it from recognizing the files correctly. So there's hope!

But for now, you must convert the .m2t files into a format that PiTiVi can read. I used ffmpeg to do so. Here's the command (and its output) I used for .m2t to .mpg conversion:

Converted .mpg files can now be imported and worked with in PiTiVi.

Editing video

Much like OpenShot, PiTiVi allows you to add video, audio and other media files to a project. You then drag the clips in the time line and position them to define the sequence of scenes in the movie.

Here are some editing/mixing features of PiTiVi I used in the making of the sample video:

• The scissor tool can be used to snip clips at any given position
• I could only add one marker (the cursor), but I am sure there is a way to add more. I just did not try to find it
• Instead of Fade In/Out effects or completely turning ON or OFF a clip, PiTiVi allows you to set track levels (brightness for video tracks, volume for audio tracks). The levels can vary through the track depending on the levels in the beginning and the end of track. I think this provides an added level of flexibility. But also, at the same time, makes simple Fade In/Out slightly harder to implement. I like it though
• Audio from a video track can be separated into an independent track. This, also, I think is a very handy feature

Again, I did not play with titles, credits or subtitles, so I cannot say much about those features. I also did not easily find any transitions or effects, that could be applied to tracks.

I had trouble synchronizing audio and video in PiTiVi (synchronizing the 'gong' with dropping of CD sleeve). I was finally able to synchronize the two with a lot of trial-and-error. The one that rendered kind of synchronized (one below) does not actually look synchronized in the preview. The preview in PiTiVi did not seem very accurate.

Exporting video

PiTiVi does not come with canned export profiles. Although I found a way to export to Vimeo-HD on the Vimeo help sites. It is left to the users to do the research. As you might notice though, the audio/video out of PiTiVi is a bit choppy at times. I probably did not get all the export parameters right. But then again, PiTiVi does not make it any easier.

Here's the screenshot of the export dialog of PiTiVi:

And here is the resulting Video:

Conclusion

I tend to compare video editors in three different categories - Import, Editing and Export. A good video editor should support a wide variety of input formats. It should have a good balance of powerful and easy-to-use editing features. It should export video in formats that can be played flawlessly on popular targets like the Web (Youtube, Vimeo, etc.) and media players (DVD/VCD players, desktop media players, etc.).

At the time of this writing, I think OpenShot trumps in all the three categories.

• In my case, PiTiVi could not import .m2t files. This added a step in my workflow. Hopefully this is temporary and will be fixed soon
• PiTiVi has some neat editing features, but lack of transitions and effects shifts the balance in OpenShot's favor. Also everything you can do in PiTiVi can be achieved in OpenShot to some extent.
• OpenShot has export profile for popular targets. This is a must for anyone who doesn't want to get lost in the many gory details of conversion parameters and their optimal values - certainly for me

Looks like the Linux video editor scene is starting to look up. I hope it only gets better from here.

If something I mentioned in this post is inaccurate/wrong, please point it out in the comments. Hope this helps!

It's been around a week since Ubuntu 9.10 - Karmic Koala was released. Both my Linux systems are now up-to-date.

Linux systems

One of the two machines is a Sony Vaio Laptop (VGN-T140P) and other is a desktop built on a Asus P5K-e Wifi/AP motherboard with an Nvidia 8400GS video card.

Install strategy

In both cases I chose to do a clean install. I always keep /home on a separate partition, so clean installs are not much of a hassle. I usually move all the dot-files (and directories) from /home/me out of the way before a clean install. After the install, I bring the ones I need manually. This, I believe, prevents any configuration issues arising from new versions of applications adding/removing/changing any configuration items. I do have to configure some applications - like compiz - all over again, but it is not a big deal most of the time.

I also make a copy of /etc directory before a clean install, just in case I need to refer to the hacks that worked around bugs in the previous release.

Trends

If I look at the way thinks are working on the two of my systems since Ubuntu 8.10 - Hardy Heron, I see pretty consistent trends. My laptop was at its best in Hardy, things are not quite that good now. The desktop, however, is at its best in Karmic! I wonder if it has to do with the video hardware (Intel in laptop, Nvidia in desktop) or wireless usage (I use it in laptop all the time, never on the desktop).

Suspend/resume has gotten worse on the laptop, but almost perfect on the desktop.

Laptop is plagued with wireless issues, but I know I am not alone there (Launchpad bug reports as of today).

Sound, at least on my systems, has greatly improved. It worked out of the box on both of them.

Laptop experience

The general feel of the OS has definitely improved. With the new boot experience, artwork, new themes, Karmic looks pretty neat. It is snappier than ever, even on the laptop with mediocre specifications. All that good stuff is marred by the issues that have come up in this release - from back in the alpha days.

There has always been a problem with the laptop hardware, that when it resumes from suspend (to RAM), the backlight is missing. In the past, I have been able to work around it by setting quirks in suspend/resume scripts. Those hacks don't work anymore. I haven't found any work-around for Karmic yet. Launchpad bug #417599.

An even worse issue that has come up is with the wireless networking. In my case, it just doesn't connect. Even when it does, it doesn't stay connected for long. It is extremely annoying. Launchpad but #429035. So its mostly wired network for now. I know, laptop wired to the router? That sucks :( I hope not for very long though.

Desktop experience

On the desktop, Karmic is just sweet. Out of the box, everything just worked. Sound, video, suspend/resume, everything!

The only thing worth mentioning is that when I first rebooted the system after installing Karmic, Windows entry was missing from the Grub2 menu. All I did to fix this was:

  % sudo update-grub

I wonder why it missed the Windows entry during installation. I am sure it runs update-grub as an installation step, doesn't it? In any case, if you see Windows entry missing from your Grub menu after installing Karmic on a dual-boot machine, just run the above command.

One more thing I haven't checked is if wake-on-lan works. If it doesn't I'll need to do what I usually do after clean Ubuntu installs - Howto: Wake on LAN with ASUS P5WDG2 WS Pro in Debian/Ubuntu. Hope it works this time too!

Well, there. Now that the computer are all working, time to get some work done!

Incidentally, I set up a github account for all my little hobby projects. Check it out. All the projects are purely experimental and educational. Nothing is going to blow your mind.

I like to read blogs/news items on Open Source/Linux related topics. Far too often, mostly in the comments, I sense the following sentiment:

I use [insert Linux distribution / some Open Source Software] and I love it! It fails in [insert some scenario/hardware/etc]. But hey, I can't complain, it's free!

I don't know if this is a result of someone misinterpreting the term free or just not being responsible enough.

From what I understand, Free Software is software licensed such that anyone can freely copy, extend and redistribute it. It just so happens that with such a license, the software is easily available free of charge - so you don't need to pay anyone any amount to install and use it.

What keeps free software alive is the community behind it. The community of developers (God bless them!) who make the software. Also the community of responsible users, who take it upon themselves to contribute in one way or the other. Many choose to make monetary donations to keep the development of free software funded, others choose to make contributions in form of testing. Using the software, reporting bugs when something doesn't appear quite right, helping the project evolve and stay healthy.

Being free doesn't mean it is any less in quality. In fact, price has nothing to do with quality. Look at Microsoft Windows and on the other end of the spectrum, look at Mozilla Firefox, Ubuntu or VLC to name a few good quality free software projects.

So when someone says - so what it's broken, it's free. You get what you pay for - it really bugs me. If you find something broken in any free software it should be reported to the developers. Most free software projects make it fairly easy to report bugs. Consider it your contribution to the software you use.

Users of free software who talk about the software they use on various platforms - blogs, comments, news websites, etc. - are also the marketing force for the free software. When we users are not careful of what we say about the free software we use, love and support, it is the project that is getting negative publicity.

Although free software is free of cost, it is - IMHO - not free of moral responsibility.

Kind of a rant, but had to say it!

After days of going back and forth between registrars, this domain name - mahesha.com - is finally transferred. What a relief!

Many apologies for the downtime over the past few days.

The process of transferring domain between registrars, as simple as it sounds, was anything but. At least in my case. In about a month's time, took three attempts, coordination with losing/gaining registrars and more than a week's downtime to get it right.

After all the confusion, I think, mahesha.com and all its sub-domains of are now up and running.

I must say, so far, I am loving my new host - DreamHost (Here's a promo code if you want to sign up with them: MMAGOTUDREAM). I hope it stays that way!

Cheers!

I was watching the Apple Get a Mac ads today and this is what I got:

This is on Ubuntu 9.04 (Jaunty Jackalope), where Firefox uses GStreamer plug-in to display video. Movie Player (Applications -> Sound & Video -> Movie Player) also uses GStreamer as its back-end.

So to fix the hue issue, I opened Movie Player preferences (Edit -> Preferences). In the Display tab, under section Color Balance noticed the Hue. In my case it was set to the minimum value. I clicked on the Reset to Defaults button. It reset the Hue slider to a center position.

Closed Movie Player preferences and reloaded the movie. It looked so much better.

Now I wonder what changed the Hue setting in the first place...

The requirement is simple:

It is desirable that when we use our browser on different computers, we want to feel at home on all of the computers. If we bookmark a website on one computer, other computers should also get it seamlessly. If we end up using a loaner computer, the bookmarks should be readily available online.

When I say feel at home, I have something very Mozilla Firefox specific in mind. I use Firefox's smart URL bar, or what is called the AwesomeBar, extensively. When I start typing in the URL bar, I like to get suggestions based on my bookmarks and history. When I get the suggestions, I want them to be derived from the latest and the greatest of my bookmarks - bookmarks added on my current computer, or some other computer. Getting history from the other computer would be great too, but just bookmarks should be just fine.

So what are the options? From what I read, there are quite a few...

Weave

Weave from Mozilla Labs, to me, is the most promising candidate to solve the problem at hand. But it is far from ready for general consumption. Every new version (that comes out pretty frequently) seems to break something or the other - It is fixed very promptly too - but there is disruption. It is indeed in lab-mode.

What I like about Weave is that it synchronizes almost your entire Firefox profile between multiple Firefox installations - Bookmarks, History, Tabs, Saved Passwords, may be even more in the future. The data is stored (by default) on Mozilla's servers in an encrypted format, that only your can read. Privacy freaks concerned can setup their own servers, so their data stays with them. But then they must keep the server accessible from the internet.

All this is achieved using the UI elements that already exist in Firefox - namely, the star in the URL bar. It is very intuitive, and in most cases very seamless.

It is really neat.

The down side to weave, is that there is no web interface to the data on the server. So if I am on a loaner computer, or in an internet cafe, my bookmarks are not accessible to me. I cannot use my existing bookmarks. I cannot add a bookmark if I come across something useful.

Delicious

Delicious is one of the most widely used social bookmarking service. Delicious add-on for Firefox provides an additional bookmarking system where the bookmarks are stored on the Delicious website. The add-on provides handy buttons to manage the online bookmarks.

Although the Delicious website says that the add-on enhances Firefox's bookmarking system, in my opinion, it is more like an additional bookmarking system. The add-on adds UI elements to Firefox toolbar to interact with Delicious bookmarks. A tag button that functions exactly like the star in Firefox's URL bar. A new window to add tags to the new bookmark, where you input almost the same information that you do in Firefox's native interface. A bookmarks menu item, that is very similar to the native Firefox bookmarks menu. There's just too much redundancy.

Since Delicious is mainly an online service, it has a very mature and efficient web interface. And because all your bookmarks are online, you can access them from any computer with an internet connection - whether it is your own or a loaner or one in an internet cafe.

Google Bookmarks

Google Bookmarks is a part of Google Web History. Like Delicious, the bookmarks are saved online and they can be accessed from any online computer.

Google Bookmarks integrate well with the Google Toolbar for Firefox. Also like Delicious, the interface provided by the Google toolbar for bookmarking is exactly like Firefox's own. The toolbar adds its own star, which is used to bookmark a website, or edit (add tags to) an existing bookmark.

Foxmarks A.K.A. Xmarks

Foxmarks which is now known as Xmarks is also a strong contender in this space. It has the goods of both online bookmarking services and native Firefox bookmark synchronizing. Xmarks provides an add-on for Firefox (in addition to Internet Explorer and Safari, which is an added bonus), that just synchronizes your bookmarks with an online storage on xmarks.com. They have a neat web interface to manage the online bookmarks. Any changes made online, or on any computer are automatically synchronized.

The missing link

Although Delicious and Google Bookmarks provide a usable interface with Firefox, there is one deficiency in both of them that is very significant to me (an AwesomeBar user). Bookmarks on Delicious or Google Bookmarks don't become a part of Firefox Bookmarks, and hence are not accounted in the search algorithm used by the AwesomeBar. If I add a bookmark to either of the two services, it will not be suggested when I type something in the URL bar.

Conclusion

I love the fact that online bookmarks (Delicious/Google Bookmarks) are available from anywhere on the internet. However, no matter how much I try to get used to one of the two services, I miss that fact that online bookmark are not included in the AwesomeBar suggestions.

A web interface to manage Weave's bookmarks online will be a welcome feature. Or If add-ons provided by online bookmark services somehow integrate their bookmarks with Firefox's bookmarks, that would be cool too.

Until then, it will have to be an inconvenient combination of the two.

I have not used Xmarks long enough to form a solid opinion about it. I'll just have to play around with it for a while.

Edit: Added Foxmarks/Xmarks as a viable alternative, after this post was first written.

Edit (05/12/2009): Well, looks like the Delicious add-on for Firefox just came a step closer to the way I like it. With its version 2.1.041 released yesterday (?) delicious bookmarks are now integrated with the AwesomeBar. In addition, if you start typing in the address bar with ??, AwesomeBar will show only Delicious bookmarks. If you start typing with >>, AwesomeBar will show Delicious tags. This is great! Now if it could just reuse Firefox's native UI elements...

Its been a while since I booted into Windows XP on my desktop. Today I did. Since Adobe Flash Player 10 is out, I decided to upgrade the existing flash installation - In fact, my bank's website won't work without the upgrade. I downloaded the installer and started it, only to get an error:

You do not have sufficient disk space to complete this installation. Please free 5000 KB and try again.

My disk is nowhere close to being full. A little Googling pointed at a known bug in the installer - Apparently, the installer has C:\ hard coded in its installation path. So if your boot/system drive is not C:\, the installer dies with a very misleading message, quoted above.

I tried to follow the workaround suggested for Windows XP, but it did not quite work, since Changing the drive letter of the system volume or the boot volume is not a built-in feature of the Disk Management snap-in.

Further Googling pointed at a very simple solution. All you need to do is map C: to an existing folder (G:\ in my case). So I opened the Command Prompt, and executed the following command:

subst C: G:\

There. I started the installer again and had no problem this time!

Way to go Adobe..

PS: Don’t ask me how I ended up having G: as the system drive instead of the standard C:.

Looks like the Google Street-View guy is spreading some cheer on Google's 10^th birthday!

Happy 10^th, Google!

Lately, I have developed a liking for the Alpine Messaging System. It is a text based email client that runs on a variety of platforms. I use it with Gmail IMAP at home and MS Exchange at work. In both cases, it works beautifully.

Pre-compiled binaries for a good number of platforms are available on the apline website. But if you are not in a position to use the pre-compiled binaries - say because you do not have root privileges, or binary is not available for your platform, or may be you just want to use the bleeding edge development code - you can build the alpine application from source. The following steps are specific to building the latest SVN snapshot on Ubuntu Hardy Heron. It may/may not work on other OSes.

Prepare your system

Get all the (known) prerequisites. From my experience, these are required:

• libncurses5-dev to get rid of the following error:

configure: error: Terminfo/termcap not found

• libpam0g-dev and libssl-dev to get rid of some SSL related errors.

To get the above use the following in terminal:

  % sudo apt-get install subversion libncurses5-dev \
                         libpam0g-dev libssl-dev

Get the source code

Get the latest source snapshot. Following few steps are one time. After the initial snapshot is acquired with svn checkout, svn update can be used to update the source snapshot:

  % mkdir ..../alpine
  % cd ..../alpine
  % svn checkout \
        https://svn.cac.washington.edu/public/alpine/snapshots/

Accept the certificate (permanently) if required.

Configure, compile and install

Now use the familiar configure, make, make install routine to build and install Alpine.

  % cd .../alpine/snapshots
  % svn update
  % ./configure --prefix=/opt/apps/alpine
                --with-local-password-cache-method
  % make
  % sudo make install

I was overly optimistic in using --with-local-password-cache-method. I was hoping that Alpine would store my account passwords in Gnome Keyring! It did not work quite like that. I need to enter my email passwords every time I start Alpine.

Other way to automate password entry is to use the --with-passfile configure option. But it is known to be unsafe.

Happy messaging!

EDIT: Trying to recompile alpine on Intrepid, I discovered that libssl-dev is also required.

If you use Weave, you've probably noticed that Mozilla servers are currently overloaded. Accounts are not working. My account authenticates (If I access

https://services.mozilla.com/user/<user_name>/

it asks for and accepts my credentials), but I get a "Not Found" 404 error. The accounts that do work, are very slow.

Like always, I went forumming, trying to get to the bottom of this trouble. I came across some helpful posts at the Weave forums. Posts from covidium and netoak are especially useful. This post is just a compilation of steps I followed to get my Weave working again.

Get a WebDAV host I guess there are only a handful of these, that are free. I tried to find one in the US, but couldn't. I just used the one that covidium suggested. I opened a free account at myDisk.se.

Create directory structure Weave expects the files to be in a specific directory structure. If you have myserver.com as your server, it will look for

myserver.com/user/user_name

To cater to this, I created a directory called user, then inside it, I created one called user_name.

In retrospect, I should have instead made

weave/user/user_name

directory structure. That would keep everything Weave together.

Set Weave's Server Settings Open Weave's Server Settings (Tools -> Weave -> Preferences...) and then go to the Advanced tab. Replace the default Server Location with

https://mydisk.se/user_name/

mydisk makes my files available at https://mydisk.se/user_name/, other hosts may use some other path.

If I had used

weave/user/user_name

directory structure, I would set the Server Location to

https://mydisk.se/user_name/weave/

Try Weave again In Weave preferences' Advanced tab, click on the Advanced... button. Click on Reset Login and Reset Server Lock buttons. I did not think it would be required, but I had to quit Firefox and restart it.

Now, sign in to Weave again (Tools -> Weave -> Sign In). Follow the setup wizard as if you are opening a new account. Use username and password from your WebDAV server. Pick a strong passphrase.

At this point, you should be all set.

Do leave feedback to point out errors in the above process, so other readers can benefit too.

My current web host does not provide WebDAV. From what I read, DreamHost does. I wonder if Weave users who have DreamHost accounts already host their own Weave servers.