Solved: weird problem with special characters in filenames

After moving my iTunes library from my Mac to a directory on an Ubuntu Linux server mounted via CIFS, iTunes could no longer find a good number of songs. All of the problematic songs had special characters such as accents in their filenames.

After trying a few different things over the course of several weeks, I finally noticed that the problematic names had the plain characters followed by so-called combining accents.

For example, instead of instead of a simple Ñ they had an N followed by a combining tilde. CIFS clients, including OS X Finder and smbclient could not handle these combined versions, and the files were inaccessible.

Fortunately, the convmv utility can fix this problem in bulk for a whole directory tree:

convmv -r -f utf8 -t utf8 --nfc .

It appears that the key was to convert the filenames into NFC Unicode normalization form (--nfc argument), typically used on Linux. It does not seem to work with NFD, typically used on Mac OS.

Update on Monday, August 17, 2009, 20:36

This problem just came back to haunt me when trying to synchronize the recently modified version of the iTunes library on OS X (NFD encoding) with the one on the Linux server (NFC encoding) using the Unison file synchronizer via ssh (not via CIFS, for better reliability). It turns out that the released and beta versions of Unison do not yet solve this problem. After I found this discussion, I downloaded and built the most recent revision from the repository and am running it with the unicode=true option, which makes it handle this situation correctly. The non-GUI version built out of the box without problems on OS X and Ubuntu with the most recent OCaml. That's one more loose end tied up...

Hauppauge Win-TV-HVR-950Q Works With Ubuntu Hardy on Mac Mini G4

I have had a very robust MythTV setup since spring 2005 using a Hauppauge WinTV-PVR-150 Media Center edition. Given the impending conversion to ATSC digital TV in February 2009, I ordered a Hauppauge WinTV-HVR-950Q external USB digital TV capture device.

Meanwhile, to reduce noise, heat, and power consumption, I am also switching from an i386-based workstation/server to an almost silent Mac Mini PowerPC G4 1.25 GHz running Ubuntu Hardy server with the lightweight Xubuntu desktop.

Today, I received the 950Q and was able to install it on the Mini without any trouble following Scott Bronson's detailed instructions to the letter. Thanks, Scott, for providing these wonderful, accurate instructions!

The Mini is too slow to play live video without mplayer's -framedrop option, but it is mostly a server anyway. The next step is to configure MythTV to record from the 950Q...no problem.

Meanwhile, I have upgraded all my systems to Jaunty, where the device is supported out of the box once you copy the firmware to /lib/firmware. Please see my recent comment for more details.

Simplified Gizmo5 Backdoor Dialing Asterisk Gateway Interface Script

Using Gizmo5 backdoor dialing, one can make free VoIP calls to a growing double-digit percentage of domestic US phones. This article shows how to integrate this mechanism into FreePBX.

In an earlier article, I described a DYI VoIP setup that uses plain Asterisk for routing calls among various VoIP providers, analog phone adapters (ATAs), and the building door phone. Since I wrote that article, I have moved this Asterisk installation to a Linksys NSLU2 ("slug") and want to keep it as minimal as possible.

Starting with an updated version of the Monetra callerID lookup script, which performs some very basic screen scraping to look up callerID information from several web sites, I wrote a similar script that checks whether a given phone number can be called for free using Gizmo5 backdoor dialing and caches the result. Gizmo5 has a very simple web service for this purpose that powers its lookup form.

The script can be used in extensions.conf as follows, assuming your Gizmo5 account is set up correctly in sip.conf.

[macro-gizmo5backdoor]
exten => s,1,AGI(gizmo5backdoor_shell.agi|${MACRO_EXTEN})
exten => s,n,GotoIf($[ "${foundroute}" = "yes" ]?backdoor:pstn)
exten => s,n(backdoor),Goto(backdoor,010${MACRO_EXTEN},1)
exten => s,n(pstn),Dial(SIP/other-provider/${MACRO_EXTEN})

[domestic]
exten => _1NXXNXXXXXX,1,Macro(gizmo5backdoor,${EXTEN})

[backdoor]
exten => _0101NXXNXXXXXX,1,Dial(SIP/gizmo5/${EXTEN})

Both scripts are available from the Asterizmo project site.

Exporting Contacts from Plaxo to Nokia 6100

Instead of using up my equipment credit from T-Mobile for signing a new contract, I am using a Nokia 6100 my friend from Germany had given to me. This is a nice and small mid-range phone.

The challenge was to prepopulate the new phone with a selection and projection of my contacts stored on Plaxo. I was not even looking for bidirectional synchronization because I don't mind making incremental changes on both Plaxo and the phone, as long as I have the ability to back up the phone data reliably.

Based on information available on the Nokia support site, I ordered an original Nokia CA-42 true USB cable (among other accessories) for US$22 instead of a cheap DKU-5 clone that merely emulates a serial connection over USB.

The situation:

• This phone is not supported by Apple iSync.
• It is supported by the Nokia PC Suite. Because I don't run Windows natively on any machine (not a safe practice in my opinion), I installed the Nokia PC Suite on a VirtualBox VM running Windows XP. On a Mac host, I couldn't connect because the Mac grabs the USB device so Windows thinks its already in use. On a Linux host (I use Ubuntu Hardy 8.04), I could connect the phone just fine, but the phone book component of the suite kept crashing miserably and repeatedly and I decided to abandon this route.
• It is also supported by a shareware program for the Mac called MegaCellX. Although this is a nice program that supports a number of phone models and integrates with Address Book, on certain Nokia phones it does not support multi-number contacts, so you get multiple entries on the phone for people with more than one number.
  

I eventually got the job done this morning around 3 as follows:

1. Plaxo itself does not support selective export, but Address Book does (synchronized via Plaxo for Mac, and Plaxo's user-defined categories show up very nicely as groups in Address Book. I exported a selection of about 150 contacts in the (text-based) vCard 2.1 format.
2. Using a combination of grep and Emacs, I manually fixed the following kinks that cause problems with the next step:
• illegal characters in phone numbers (anything other than digits, *, +, or p) (everything up to this point does get imported, but you need to switch to SIM contacts and back to memory contacts to refresh the view)
  
• use of FN: instead of the expected N:
• strings in non-quoted-printable UTF-8 (I cannot believe this is still broken in Address Book) show up as empty fields but don't disrupt the import
  
• unnecessary fields (this is the projection part)
3. Using Wammu on my Linux box, I transfered these contacts to my phone! I had to cycle a few times between the preceding step and this one until all contacts got imported cleanly. I still had to do some editing in Wammu to add special characters in names etc.
   
4. For some multi-number contacts, I changed the default number on the phone.

Finally Switching from Verizon Wireless to T-Mobile

I finally threw off the shackles of Verizon Wireless last Wednesday. I chose T-Mobile instead for the following reasons:

• I sometimes travel internationally and am much better off with a pure GSM provider.
• Verizon is very slow to establish agreements with countries that do have CDMA networks.
  
• There are countries where dual-mode phones from Verizon will not work at all. This is generally the case for GSM countries with American frequencies (850/1900 MHz) such as Bolivia and Paraguay.
  
• T-Mobile doesn't lock down its devices the way Verizon does. I can install my own apps much more easily.

It also turns out that myFaves is a real winner. If you have a provider with a non-800 access number, you get close to unlimited outbound calling.

See also:

• HowardForums T-Mobile forum
  
• HowardForums T-Mobile wiki
• Free incoming calls on T-Mobile
  

Dude, You're Getting VoIP: Moving Past POTS

AKA a case against gardening

My colleague George and I had been toying for a long time with the idea of getting rid of POTS ("plain old telephone service", more properly called PSTN for public switched telephone network) at home and were considering different alternatives. An obvious one is to get lots of minutes or unlimited calling on your cell phone—now on option on the major carriers. But that might not be the most effective approach because the network gets overloaded, the call quality is inconsistent, and international rates are sky-high (unless you are using a suitable calling card). Your brain might also get fried if you are not using a headset. But after several months in a row of local phone bills in the high double digits, we knew we had to do something soon.

Because we were already paying for a broadband connection (cable modem, in both cases, though ADSL will work fine), we have a big communication pipe coming into our homes. That pipe should be able to handle occasional voice traffic in addition to other uses of the internet, shouldn't it? This is where VoIP (voice over internet protocol) comes into the picture (see voip-info.org, an excellent resource on all aspects of this topic). Most people are already familiar with Skype, primarily a software client for free voice and video calls from computer to computer using a proprietary protocol but requiring only minimal, if any, configuration. There is also Gizmo5, which now supports video on its Windows and Mac versions and is compatible with the SIP (session initiation protocol) standard. These and other providers also have (non-free, usually prepaid) ways to make or receive calls to and from the PSTN, e.g., SkypeOut and SkypeIn. There are various open-source or free proprietary SIP soft phones available for different operating systems, including Twinkle for Linux and XMeeting for Mac OS.

But both of us were looking for something more: a seamless drop-in replacement for the conventional home phone service that particularly our spouses enjoys so much. Specifically, we wanted the existing phones to ring when a call came in while no one was using the computer, and we wanted to be able to make outgoing calls from the existing phones. An ATA (analog telephone adapter) bridges this gap: these devices typically have an LAN connection as well as a phone jack where you can plug in a phone and get a dial tone. And now we've reached a junction where we could choose between the easy and the hard way to continue.

The easy way

George choose the easy way: He signed up with PhonePower.com, a consumer-oriented VoIP provider (reviews are available at, e.g., voipreview.org), received a preconfigured ATA, connected power, Ethernet, and a cordless phone, and was ready to go. Once his spouse approved, he instructed the VoIP provider to port his existing number and canceled his land line. Many providers have flat nationwide packages for US$20 per month or less, and you can use your ATA in other locations around the world as long as you have an internet connection. Some providers even have regular E911 service, where the emergency call taker automatically sees your address and handles the call accordingly. In case of a power failure, however, your internet connection would be down and you might not be able to make that call. In that case, your best bet would be your cell phone, given the slim likelihood of both networks failing independently at the same time.

The kewl way

Needless to say, I choose the hard but kewl way and in return wanted free service (no recurring monthly charge) and the opportunity to reuse as much existing hardware as possible. As an aside, when I mentioned to my friend Hans-Jürgen a few years ago that I was planning to play with a software-based phone system at home, he said I had too much time on my hands. But now I realized that we simply have different hobbies. I enjoy these activities much more than, say, gardening.

Receiving calls

The first step was to set up DID (direct inward dialing) so others could reach me from the PSTN. Most providers have a single-digit monthly charge for DID, but Matt Butcher, a technologically savvy philosophy graduate student, told me about a service called GrandCentral. Whether or not you are using VoIP, GrandCentral is useful as a single-number service (SNS): a caller dials a single number, zero or more phones—home, work, cell, according to your configuration settings—ring at once, and you can pick up the call on any one of them. You can also send calls to voice mail, listen in while someone is leaving you a message, switch to another phone during the call, etc. You can even customize by caller which of your phones ring. The catch is that GrandCentral used to have an invite-only private beta program that has since been discontinued, but they haven't yet announced when they will open to the general public. Since GrandCentral was recently acquired by Google, I am hopeful that it won't go away, but for now, you will probably have to pay for DID if you want a local number; in return, you can probably port your existing number. So what makes GrandCentral work as DID for VoIP? The ability to forward calls not only to PSTN numbers but also to Gizmo5 numbers! I was already able to make and receive SIP calls through my existing Gizmo5 account, but now others can call my GrandCentral number from an ordinary phone, and it rings on my Gizmo5 soft phone. So that part is solved!

Making calls

Like SkypeOut, Gizmo5 also supports outbound calling into the PSTN as a prepaid service with no recurring charges and reasonable international rates. As an optimization, I am using Future Nine, a reliable VoIP provider with very competitive rates for outbound calling. Another advantage of Future Nine is that you can set the caller ID for outgoing calls to your GrandCentral number for free. You can even have multiple SIP accounts as part of a single VoIP account with separate caller IDs for outgoing calls; this capability is useful to me because my wife and I have different GrandCentral numbers. In addition, your Future Nine account works like a calling card. As a further optimization, I set up free toll-free and free VoIP backdoor calling through Gizmo5. Since you can't call back a Gizmo SIP number from outside, I also decided to pay Gizmo5 US$4 per year for letting me set the caller ID on that account to my GrandCentral number as well.

Connecting existing analog phones

The next big step was to bring the analog phones back on board, since SIP phones are rather expensive. A while ago, I had bought a Digium S101I IAXy ATA but never used it. If I had DID from Future Nine or another provider that supports the IAX protocol (Inter-Asterisk eXchange), I could provision the IAXy to connect directly with my Future Nine account. That would be just like George's easy approach described above. (Linksys, formerly known as Sipura, Grandstream, and several others make high-quality SIP-compatible ATAs.) But my situation is more complicated because it involves different accounts for incoming and outgoing calls and Gizmo5 speaks SIP but not IAX. Rescue comes in the form of a software telephony engine such as Asterisk (asterisk.org, excellent tips and tricks at nerdvittles.com), an open-source system from Digium. Asterisk (and similar systems such as Yate or FreeSWITCH) are full-featured software equivalents of PBX systems handling voice mail, music on hold, etc. But since GrandCentral provides all the features already, we need only a minimal Asterisk installation that handles registration and call routing among the different pieces:

• SIP registration with one or more Gizmo5 accounts for incoming calls from GrandCentral and free outgoing calls to other Gizmo5 users, toll-free numbers, and VoIP backdoor numbers.

  [general]
  context=incoming
  dtmfmode=rfc2833 ; required to answer incoming GrandCentral calls
  srvlookup=yes
  register => 1747NXXNXXX:password@proxy01.sipphone.com/1747NXXNXXX
  
  [gizmo]
  type=peer
  disallow=all
  allow=ulaw
  allow=ilbc
  host=proxy01.sipphone.com
  fromdomain=proxy01.sipphone.com
  insecure=very
  fromuser=1747NXXNXXX
  authuser=1747NXXNXXX
  username=1747NXXNXXX
  secret=password
  canreinvite=no
  context=incoming
  

• SIP credentials for using Future Nine for other outgoing calls.

  [future-nine]
  context=default
  type=peer
  host=sip.future-nine.com
  username=username
  secret=password
  qualify=no
  insecure=invite
  canreinvite=no
  

• IAX configuration for the IAXy ATA.

  [myiaxy]
  context=default
  type=friend
  accountcode=myiaxy
  host=dynamic
  secret=password
  disallow=all
  allow=ulaw
  allow=adpcm
  callerid="My Name" x1111
  trunk=no
  

• SIP configuration for any softphones for in-house calling (optional).

  [1117]
  context=default
  type=friend
  username=1117
  callerid="My Name" x1117
  host=dynamic
  canreinvite=no
  

• Dial plan rules for making the IAXy (and registered softphones) ring on incoming calls.

  [incoming]
  exten => 1747NXXNXXX,n,Dial(SIP/1117&IAX2/192.168.1.50@myiaxy/s)
  

• Dial plan rules for mapping dialed numbers to the right provider for outgoing calls.

  [gizmo]
  exten => _1747XXXXXXX,1,Dial(SIP/${EXTEN}@gizmo)
  exten => _0101NXXNXXXXXX,1,Dial(SIP/${EXTEN}@gizmo)
  
  [local]
  exten => _NXXXXXX,1,Dial(SIP/1773${EXTEN}@$gizmo)
  
  [tollfree]
  exten => _1800XXXXXXX,1,Dial(SIP/${EXTEN}@gizmo)
  ...
  
  [domestic]
  exten => _1NXXNXXXXXX,1,Dial(SIP/${EXTEN}@$futurenine)
  
  [international]
  exten => _011X.,1,Dial(SIP/${EXTEN:3}@futurenine)
  
  [intercom]
  exten => s,1,Set(CALLERID(all)=Lobby Phone <773nxxnxxx>)
  exten => s,n,Dial(SIP/1117&IAX2/192.168.1.50@myiaxy/s)
  
  [default]
  include => internal
  include => gizmo
  include => local
  include => tollfree
  include => domestic
  include => international
  

The image below shows how all these pieces fit together. The arrows point in the direction of the call, i.e., from the calling to the called party; the hollow tip of the arrow from the PSTN into my other phones indicates that bypassing GrandCentral is the exception rather than the rule.

Instead of plugging a particular phone into the IAXy, I simply left the existing phones plugged into the original jacks and plugged the IAXy into another free jack. In this way, the IAXy provides a phone signal to the entire apartment (it can handle the load of a few modern phones). I had already disconnected the incoming PSTN line from the apartment wiring so that it would not interfere with the ATA during the brief transitional period.

Reconnecting the door phone

I generally like to keep working hardware as long as there is a use for it, so I was wondering what to do about the Digium X100P clone FXO card I had put into my home server a few years back. This is basically a controllerless (soft) modem card that allows the computer to make and receive phone calls. I remembered an innocuous comment a crafty neighbor of mine, Louis Hubbard, had made during an earlier discussion of the switch to VoIP. He said that the lobby phone for the doorperson to announce visitors (I live in a highrise) should still work even without PSTN service as long as I simply hook up a phone to the PSTN line coming in from outside. And he was right! So I rewired the apartment a bit and managed to connect the outside line to the X100P and configured Asterisk to receive incoming calls from the card, complete with caller ID and all. That was the dot on the i, so to speak, because other residents who switched only from an ILEC (incumbent local exchange carrier, such as the local phone company) to a CLEC (competitive local exchange carrier, such as the cable company) lost that capability, to the annoyance of building staff. You can probably use a similar setup for other types of door phones or get a (more expensive) SIP door phone.

So how hard was it?

It was straightforward to get most pieces to work on Ubuntu Edgy or Hardy (yes, I made that upgrade in the process, too, but had to resolve some other lingering complications before I could go on) and configure them according to the instructions. The Zaptel driver for the X100P card included with Ubuntu, however, is buggy, and I had to perform a manual install of Digium's latest version of that driver (svn checkout http://svn.digium.com/svn/zaptel/branches/1.4 zaptel) to get everything to work including proper initialization at boot time.

In conclusion, yes, it was a bit hard, but it sure was great fun. Both George and I ended up saving big bucks amounting to 75% or more compared to our previous plans. We use our regular phones more than cell phones, especially for long calls. Call quality is better—even with VoIP—, not to mention fewer dropped packets and delays of one form or another. I would gladly do it again! Definitely beats gardening by the widest of margins.

Acknowledgments

Thanks to Ron Senykoff for his valuable advice on this project.

Converting MPEG-4 (h.264) Videos for Viewing with mvpmc

mvpmc is a media player running on the Hauppauge MediaMVP. Besides serving as a MythTV front end, it allows me to access multimedia content stored on a file server and show it on a conventional TV set.

Increasingly, TV content is archived in MPEG-4 format (example). Using ffmpeg, it can be converted to a DVD-like format suitable for viewing with mvmpc as follows:

ffmpeg -i xyz.mp4 -target ntsc-dvd -s 320x240 -acodec mp2 -ab 160k xyz.vob

Unfortunately, this conversion causes the file size to increase six-fold. Any suggestions to minimize the file size without introducing playback problems are welcome.

Gift Ideas for Friends in Europe

If you live in the US or Canada and have friends or family abroad who lead an active lifestyle, you may consider taking some of these items as gifts (conveniently available at the nearby Target):

• Gatorade powder
• CLIF bars (for example, in Germany, you can get only the much less tasty Power Bars for an even less tasty price)
• sweatproof sunblock (they only have the waterproof kind)
  

Propane Exchange Pricing in the Chicago Area

Last night, I went to Home Depot in Evanston for a quick propane exchange. Out of curiosity, I called a few other places this morning to get their prices. Here is a summary (all prices in US$, not including about 10% sales tax):

• Clark Devon Hardware:
• CVS:
  
• Home Depot:
• Lowe's:
  
• Menards: 17.82
• 7-11:
  

I think it would be cool if Chicago Gas Prices included propane exchange...