Saturday, November 13, 2010

MyApp: Spb Wallet to KeePass convertor

Spb Wallet is one of the best wallet applications that I've used across various mobile platforms. However, it isn't available on all mobile platforms and it isn't free either. Once I moved on to Android, I hit a road-block because Spb Wallet isn't available for Android (yet). I'm sure you would understand how painful it is to not have a wallet app, once you are used to.

KeePass is the alternate. KeePass is an open-source software for password management. To be fair, KeePass isn't as great as Spb Wallet, but does its job. Being open-source, it is available on almost all platforms including desktops.

The pain here is the conversion. I have tonnes of data on Spb Wallet that manually entering them on KeePass is a no-go. Unfortunately, and mostly intentionally, Spb Wallet doesn't export to any well known format for import into KeePass. There weren't any handy tools to convert either. Thankfully Spb Wallet had a text export and KeePass had many import options. But it isn't directly compatible, because Spb Wallet doesn't have any proper structure to the exported TXT file and the grammar is quite ambiguous. KeePass has a well defined XML structure for import (found it by doing a sample XML export from KeePass). I wrote this python script to convert the Spb Wallet TXT export file into the XML format that KeePass can understand. In reality, Spb Wallet has more "specific" fields than KeePass, so there isn't always a direct mapping. Any non-mappable field (Account Number for example) will be appended in the notes section of KeePass so no information is lost.

This script is a simple parser that understands and converts the Spb Wallet TXT export file. It maintains the internal state of parsing and learns dynamically about what the current token is -- some times even looking ahead into the file to resolve ambiguities.

This script is probably not so robust on errors. But this did the job for my Wallet export which is reasonably big.

If you find any bug on this script that you want me to fix, report here. I *MAY* fix it for you.

Here is the source:
#
# Code to Convert Spb Wallet TXT export file to KeePass XML import file.
#
# Original Author: Gerald Naveen A (http://geraldnaveen.blogspot.com)
#
# License:
# You are free to modify/distribute/use for commercial/non-commercial/
# personal applications. Any modification to this code needn't be published. 
# However, any publication of this code or the derivative of this code, should 
# include the original author and this license text.
#
import sys

def mywrite(f, str):
f.write("{0}\n".format(str));
def main():
print "\nSpb Wallet to KeePass Convertor v 1.0 by Gerald Naveen\n";
print "Report bugs at http://geraldnaveen.blogspot.com/2010/11/myapp-spb-wallet-to-keepass-convertor.html\n";
if len(sys.argv) < 3:
 print "Usage: spb_wallet_to_keepass.py <spb_txt_export_file> <keepass_xml_import_file>";
 print "\nWhere,\nspb_txt_export_file: path to the TXT file exported from Spb Wallet.";
 print "keepass_txt_import_file: path to the output XML file, that shall be imported into KeePass.";
 return;
try:
 ifile = open (sys.argv[1], 'r');
except:
 print "Could not open input file", sys.argv[1];
 return;

try:
 ofile = open (sys.argv[2], 'w');
except:
 print "Could not open output file", sys.argv[2];
 return;

FOLDER_NAME_TOKEN=1;
ENTRY_NAME_TOKEN=FOLDER_NAME_TOKEN+1;
BEFORE_NOTES_TOKEN=ENTRY_NAME_TOKEN+1;
NOTES_TOKEN=BEFORE_NOTES_TOKEN+1;
INVALID_VALUE='invalid';

next_token=ENTRY_NAME_TOKEN;
folder_name = INVALID_VALUE;
entry_name = INVALID_VALUE;
user_name = INVALID_VALUE;
password = INVALID_VALUE;
url = INVALID_VALUE;
notes = INVALID_VALUE;
valid_entry = False;

mywrite(ofile, '<?xml version="1.0" encoding="utf-8" standalone="yes"?>');
mywrite(ofile, '<pwlist>');
try:
 for line in ifile:
  line = line.strip('\r\n');
  if len(line) == 0:
   # empty line
   if valid_entry == False:
    # entry name found after folder name
    folder_name = entry_name;
    entry_name = INVALID_VALUE;
   else:
    # found the last line of the entry..dump
    mywrite(ofile, '<pwentry>');
    if folder_name != INVALID_VALUE:
     mywrite(ofile, '<group>{0}</group>'.format(folder_name));
    mywrite(ofile, '<title>{0}</title>'.format(entry_name));    
    if user_name != INVALID_VALUE:
     mywrite(ofile, '<username>{0}</username>'.format(user_name));
    if password != INVALID_VALUE:
     mywrite(ofile, '<password>{0}</password>'.format(password));
    if url != INVALID_VALUE:
     mywrite(ofile, '<url>{0}</url>'.format(url));
    if notes != INVALID_VALUE:
     notes=notes.replace('\n', '&#xD;&#xA;');
     mywrite(ofile, '<notes>{0}</notes>'.format(notes));
    mywrite(ofile, '</pwentry>');
    user_name = INVALID_VALUE;
    password = INVALID_VALUE;
    url = INVALID_VALUE;
    notes = INVALID_VALUE;
   valid_entry = False;
   next_token=ENTRY_NAME_TOKEN;
  else:
   if next_token == ENTRY_NAME_TOKEN:
    entry_name = line;
    next_token = BEFORE_NOTES_TOKEN;
   else:
    valid_entry = True;
    if next_token == BEFORE_NOTES_TOKEN:
     if line.startswith('User Name:'):
      user_name = line[len('User Name:'):].strip(' ');
     elif line.startswith('Password:'):
      password = line[len('Password:'):].strip(' ');
     elif line.startswith('Web Site:'):
      url = line[len('Web Site:'):].strip(' ');
     elif line.startswith('Notes:'):
      if notes == INVALID_VALUE:
       notes = line[len('Notes:'):].strip(' ');
      else:
       notes += '\n' + line[len('Notes:'):].strip(' ');
      next_token = NOTES_TOKEN;
     else:
      # any unknown params should go as notes.
      if notes == INVALID_VALUE:
       notes = line;
      else:
       notes += '\n' + line;
    elif next_token == NOTES_TOKEN:
     # any thing from the notes section.
     notes += '\n' + line;
except:
  print "Unknown error occured while processing the input file.";
mywrite(ofile, '</pwlist>');    
ifile.close();
ofile.close();
print "Success. Now import {0} in KeePass as KeePass XML".format(sys.argv[2]);
if __name__ == "__main__":
   main()
Download spb_wallet_to_keepass.py

Update [26-Nov-2010]:

If the script ran successfully, but the output XML file didn't work on import, it could most likely be a CRLF issue. Try this in such cases:

Lets assume your file is test.txt.

1. Open test.txt in Notepad
2. "Save as" another file, say test2.txt (Note: select Utf8 as Encoding).
3. Use test2.txt as input

38 comments:

  1. Gerald
    Thanks for this nice piece of code!
    I'm not a python programmer so I used a portable version of python to run your program. I have probably a version problem. Some versions give a syntax error at the "print" commands (I correct adding brackets), others at "+=" (I can correct expanding the expression). Nevertheless at most I arrive to:

    File "test.py", line 113
    except:
    ^
    SyntaxError: invalid syntax

    Could you please tell the version of python?
    Thank you!

    ReplyDelete
  2. I developed this using ActivePython 2.6.

    Also do remember to download the source file using the link above and not copy the source text from my post. Indentations matter a lot in python.

    ReplyDelete
  3. It worked perfectly! I had to change indentation, I don't know why, but after a few trials...

    BIG THANK YOU!!!!!

    ReplyDelete
  4. I have also problems with indentation in the downloaded file. What should I do to solve it?
    Thank in advance for your help.

    ReplyDelete
  5. This is strange. I had uploaded a working file. which platform are you trying on?

    ReplyDelete
  6. I use WinXP SP3 and PythonWin (latest version). The indentation of the last except is faulty (easy to correct). But it doesn't work for me. Could you give an example of the way you enter the arguments of input and output files?
    I am not an expert in Python. Therefore it would be useful to have an example of how you execute your program. Maybe there is something special in the way the arguments are entered.
    Thank your for your help. If it could work, it would save me a lot of time...

    ReplyDelete
  7. I guess that extra tab got inserted accidentally. I've fixed that indentation issue in 'except' now.

    When you say it doesn't work, what does that mean? Did you get an XML? or an error? or the script itself doesn't run? If you invoke the script wrongly it should print out the usage:

    Usage: spb_wallet_to_keepass.py <spb_txt_export_file> <keepass_xml_import_file>

    Try ActiveState Python 2.6 -- that's the version I've.

    ReplyDelete
  8. After a few tests more (with short path names without spaces in it for input and output files), I can obtain now an .xml output.
    But this output file is only 1 KB and contains only:


    It seems that the python script doesn't actually read the content of my input file .txt
    Maybe it is necessary to "prepare" this file for your script. I removed the long texte file explanation at the beginning of my spb export file and now it starts with a folder name on the 1st line. The 2nd is empty, the 3rd contains a "title name", the 4th a "website", the 5th a "user_name", the 6th a "password" ... after there is an empty line and the next "title name" in the same folder ... after several more lines a new folder name, etc.
    Depending on the folders and cards, data fields are more complex. But even in the simple case above, it doesn't work for me.
    Maybe there is still indent problems in other parts (?)

    ReplyDelete
  9. This output file contains only:
    xml version="1.0" encoding="utf-8" standalone="yes"
    pwlist

    ReplyDelete
  10. I forgot also to say that I have installed also ActiveState Python 2.6.6.

    ReplyDelete
  11. The format you have mentioned seems to be correct.

    I can guess another possible issue -- the CRLF issue. Try this.

    Lets assume your file is test.txt.

    1. Open test.txt in Notepad
    2. "Save as" another file, say test2.txt (Note: select Utf8 as Encoding).
    3. Use test2.txt as input

    This might just work. Revert here if this works.

    ReplyDelete
  12. Thanks a lot for your explanation. It was a problem of "CRLF". My .txt file was in Unicode and after saving it in "UTF8" encoding, I could get a much bigger XML file (115 KB in my case).
    But I had an error in Keepass version 2.13 (for Windows) when trying to import this file with Keepass 1.x XML format.
    I discovered that it was linked to complex URL addresses with for example at the end "SignOnPL?msg=&goto=MembersMainMenu"
    Keepass 2.13 doesn't recognize the sign '=' in this case.
    I don't know if there is a possible improvement of your script to avoid such errors. Do you think that this kind of errors is avoidable or should I cut all my URL in Spb Wallet before exporting my wallet to a .txt file?
    Thank you in advance for your answer and for your work with this very nice script.

    ReplyDelete
  13. A small update on my experience of conversion.
    If I use Keepass 1.18 plus VariousImport plugin 1.2, there is no more errors (even with long complex URL) in importing the output file .xml of your script.
    Afterwards I can export it with Keepass 1.18 to another .xml file, and then I can import this new .xml without any problem in Keepass 2.13. It is the mystery of conversions...

    ReplyDelete
  14. I am now extremly happy with this script. It has saved me days of work and I can now use Keepass on different platform (Windows XP, Debian, WM6.1 and maybe Android in a near future).
    Spb Wallet was good, but it cannot function under Linux (even with Wine) or Android. It is nevertheless a pity that they don't intend to develop a multiplatform product.
    Thanks a lot!

    ReplyDelete
  15. Hi,

    Python is not my thing but even with a simple file I get

    Traceback (most recent call last):
    File "E:\KeePass\spb_wallet_to_keepass.py", line 120, in
    main()
    File "E:\KeePass\spb_wallet_to_keepass.py", line 52, in main
    mywrite(ofile, '');
    File "E:\KeePass\spb_wallet_to_keepass.py", line 16, in mywrite
    f.write("{0}\n".format(str));
    AttributeError: 'str' object has no attribute 'format'
    Script terminated.

    ReplyDelete
  16. Such errors don't exist in the code. You most likely have an older version of python. Use ActiveState Python 2.6.

    Also, read through other comments, lots of common issues have been talked about, already.

    ReplyDelete
  17. sorry but clearly i don't have a clue ......

    IDLE 2.6.6
    >>> spb_wallet_to_keepass.py E:\KeePass\file.txt E:\KeePass\outfile.txt

    File "", line 1
    spb_wallet_to_keepass.py E:\KeePass\file.txt E:\KeePass\outfile.txt
    ^
    IndentationError: unexpected indent

    ReplyDelete
  18. From your file paths, I can sense that you are using Windows.

    Don't use IDLE. Just open Windows command prompt and type the command "spb_wallet_to_keepass.py E:\KeePass\file.txt E:\KeePass\outfile.txt"

    That should do, if you had installed python correctly.

    You can also try "path-to-python.exe spb_wallet_to_keepass.py E:\KeePass\file.txt E:\KeePass\outfile.txt"

    ReplyDelete
  19. Thanks the second option works fine

    ReplyDelete
  20. Thank you so much for the tool! Finally my Spb Wallet data has been set free!

    ReplyDelete
  21. Thank you so much for this script! I just transfered all my data from Spb Wallet in a WinXP VM to KeePassX on Linux so that I can migrate to an Android phone. The categories came out funky in many cases, but they were easily remedied with drag-and-drop in KeePassX. Nothing was lost. You've done the community a great service, Gerald!

    ReplyDelete
  22. Thanks for your script
    But after convert during import I got this message

    Parsing error: File is no valid KeePassX XML file.

    Thanks

    ReplyDelete
    Replies
    1. Gerald - thank you so much for keeping this script online. I've been keen to get away from SPB Wallet for ages as it's a dying product and iphone support is pathetic.

      Ran your script and presto - I'm now in Keepass land :)

      Note I had to use Keepass ver1 and then use various import plugin on the xml file your script generated from the .txt file. Then I could import the ver1 file into Keepass ver2.

      Delete
    2. Glad to know that it helped. And thanks for letting us know the version specifics, it shall definitely help others. It is quite possible when I first wrote this script (back in 2010), it was just version 1.

      Delete
  23. Any way to import v2 with custom string fields?

    ReplyDelete
  24. Thank you alot for this very helpful script! You saved my day!!

    ReplyDelete
  25. I have to note that the method described above (use keypass 1/x and 'various import plugin') didn't help in my case. I have various foreign characters (Greek letters) in my original password file and they are all lost (become not readable) with this method. I used the following:
    Run your helpful script
    Run keypass 2.x and import the file with the XML version 1.x option.
    Keypass found various errors (offending characters) in the XML file to import. Keypass reports the line number and position of the offending character. All these characters were '=' in some URL codes.
    I used a simple text editor (notepad +) with line numbering to remove the spaces that existed in those positions that Keypass complained.
    After some tries, it worked OK, with the foreign (Greek) characters preserved.

    Thank you again, Gerald for this very helpful script!

    ReplyDelete
  26. Welcome :) Glad, it helped.

    Thanks for the additional info about unicode chars -- would be useful for others.

    ReplyDelete
  27. Hi Gerald

    all goes well until I try to import the file and get the error message can't open xml file?

    ReplyDelete
  28. I save file as UTF8 and converter starts but I see error:

    C:\>"c:\Python27\python.exe" c:\Documents\Wallet\exp\spbx.txt c:\Documents\Walle
    t\exp\out.txt
    File "c:\Documents\Wallet\exp\spbx.txt", line 5
    Alior Bank - Kantor Walutowy
    ^
    SyntaxError: invalid syntax

    My input file looks like:

    spb

    Bank

    Alior Bank - Kantor Walutowy X0
    Web Site: https://kantor.aliorbank.pl/login
    User Name: my@myemail.pl
    Password: erase

    Could you help me?

    ReplyDelete
    Replies
    1. You aren't invoking my script at all. You are trying to execute your spbx.txt file.

      Use:

      python_exe myscript.py input_file output_file

      Delete
  29. What a great script. It safes me hours of work. Thank you Gerald for making it public.

    I got a problem with missing XML escaping because I use a lot of special characters within my passwords. So I added XML escaping to your script.

    from xml.sax.saxutils import escape

    < ...>

    mywrite(ofile, '{0}'.format(escape(password)));

    Maybe you think of adding it to your script, too.

    Cheers

    Reiner

    ReplyDelete
  30. Excellent! Thank you very much, indeed!

    Martin

    ReplyDelete
  31. This comment has been removed by the author.

    ReplyDelete
  32. Hi, thanks for your tool. I had spb wallet entries containing chars "<", ">", and "&". These entries made the generated xml invalid.

    regards

    Tom

    ReplyDelete
  33. I like your suggestions they are really helpful. Thank you so much for sharing this post.

    Funky Phone Cases

    ReplyDelete
  34. hi i have never used python before and need an idiots guide as my test2.txt is in C:\keepass\ so have been gtrying the following without luck

    C:\keepass\spb_wallet_to_keepass.py C:\keepass\test2.txt C:\keepass\export.txt which isn't working!

    I have also tried


    C:\keepass\python.exe spb_wallet_to_keepass.py C:\keepass\test2.txt C:\keepass\export.txt

    would very much appreciate a by the nose guide

    Thanks

    ReplyDelete