disable kernel debugging before using ubuntu’s kernel config

Posted on July 30, 2008
Filed Under system administration | Leave a Comment

if you ever need to rebuild an ubuntu kernel beware when using their supplied config file from /boot/config-$(uname -r)

i went through this exercise recently and wasted quite a bit of time trying to figure out why my new kernel package including modules was over 700mb uncompressed when the original package is around 60mb uncompressed.

once i found out that debugging symbols were to blame this was the process i used to rebuild the kernel:


cd /usr/src

apt-get install build-essential kernel-package linux-kernel-devel

apt-get build-dep linux-image-$(uname -r)

apt-get source linux-image-$(uname -r)

cd linux-* # might fail if there's more than one match

make-kpkg clean

cp /boot/config-$(uname -r) .config

make-kpkg configure

make menuconfig

# disable kernel debugging under "kernel hacking"

# not sure how my copy keeps getting this sticky bit set but it causes a failure
chmod -s debian

make-kpkg -us -uc binary

if everything works you should end up with some .deb files in the parent directory.

script to delete all iphone sms messages

Posted on July 28, 2008
Filed Under iphone | 3 Comments

my coworker, dan smith, recently expressed frustration with apple’s lack of bulk message deletion in the iphone sms app.  sometimes our monitoring scripts go a little crazy and send hundreds of emails to us when things go wrong and its extremely tedious to delete these extra useless messages one at a time.

after jailbreaking my iphone and installing several of the available packages i was poking around the filesystem and looking at the source code to the example python application.  the gui stuff is beyond me for now but the sqlite access caught my eye - from /Applications/HelloPython.app/HelloPython.py (with the iphone-python package installed through cydia):


...
from sqlite3 import dbapi2 as sqlite
...
db = sqlite.connect(self.userHomeDirectory() + "/Library/AddressBook/AddressBook.sqlitedb")
...

next i did a search of /var/mobile for any other sqlite databases:


cd /var/mobile
find . -name '*.sqlitedb'

this returns:


./Library/AddressBook/AddressBook.sqlitedb
./Library/AddressBook/AddressBookImages.sqlitedb
./Library/Caches/MapTiles/MapTiles.sqlitedb
./Library/Calendar/Calendar.sqlitedb

hmm…no sms database. poking around /var/mobile/Library i found the SMS directory. in there is a file called “sms.db”. at first i thought it was a berkley database or something but i wasn’t able to open it using python’s anydbm module so i tried the “file” command:


iPhone:/var/mobile/Library/SMS root# file sms.db
sms.db: SQLite 3.x database

sweet - its an sqlite database after all. searching for other files ending with a .db in /var/mobile reveals several other sqlite databases including call_history.db, notes.db, voicemail.db, and databases for several of the installed applications.

using sqlite’s command line utility i inspected the schema:


iPhone:/var/mobile/Library/SMS root# sqlite3 sms.db
SQLite version 3.5.4
Enter ".help" for instructions
sqlite> .tables
_SqliteDatabaseProperties message
group_member msg_group
sqlite> .schema message
CREATE TABLE message (ROWID INTEGER PRIMARY KEY AUTOINCREMENT, address TEXT, date INTEGER, text TEXT, flags INTEGER, replace INTEGER, svc_center TEXT, group_id INTEGER, association_id INTEGER, height INTEGER, UIFlags INTEGER, version INTEGER);
CREATE INDEX message_flags_index ON message(flags);
CREATE INDEX message_group_index ON message(group_id, ROWID);
CREATE TRIGGER delete_message AFTER DELETE ON message WHEN NOT read(old.flags) BEGIN UPDATE msg_group SET unread_count = (SELECT unread_count FROM msg_group WHERE ROWID = old.group_id) - 1 WHERE ROWID = old.group_id; END;
CREATE TRIGGER delete_newest_message AFTER DELETE ON message WHEN old.ROWID = (SELECT newest_message FROM msg_group WHERE ROWID = old.group_id) BEGIN UPDATE msg_group SET newest_message = (SELECT ROWID FROM message WHERE group_id = old.group_id AND ROWID = (SELECT max(ROWID) FROM message WHERE group_id = old.group_id)) WHERE ROWID = old.group_id; END;
CREATE TRIGGER insert_newest_message AFTER INSERT ON message WHEN new.ROWID >= IFNULL((SELECT MAX(ROWID) FROM message WHERE message.group_id = new.group_id), 0) BEGIN UPDATE msg_group SET newest_message = new.ROWID WHERE ROWID = new.group_id; END;
CREATE TRIGGER insert_unread_message AFTER INSERT ON message WHEN NOT read(new.flags) BEGIN UPDATE msg_group SET unread_count = (SELECT unread_count FROM msg_group WHERE ROWID = new.group_id) + 1 WHERE ROWID = new.group_id; END;
CREATE TRIGGER mark_message_read AFTER UPDATE ON message WHEN NOT read(old.flags) AND read(new.flags) BEGIN UPDATE msg_group SET unread_count = (SELECT unread_count FROM msg_group WHERE ROWID = new.group_id) - 1 WHERE ROWID = new.group_id; END;
CREATE TRIGGER mark_message_unread AFTER UPDATE ON message WHEN read(old.flags) AND NOT read(new.flags) BEGIN UPDATE msg_group SET unread_count = (SELECT unread_count FROM msg_group WHERE ROWID = new.group_id) + 1 WHERE ROWID = new.group_id; END;

the hairy thing about this is that there are triggers on the table for deletes. a “select * from message;” shows all sms messages so i first tried the easy route:


sqlite> delete from message;
SQL error: no such function: read

looking more closely at the triggers i saw where it was calling the read() function in the triggers. googling for “sqlite user defined functions” provided some explanation on what they are and how to define them. one of the results showed an example in python.

i did some experiments and found that the flags column appears to be a bitmap and that the second bit indicates whether a message has been read or not. so the triggers update an unread message count in another table on deletion, the flags column is updated, or when a row is inserted.

so here’s my script to delete all sms messages:


#! /usr/bin/env python

from sqlite3 import dbapi2 as sqlite

def message_read(flags):
"""reimplementation of an sqlite user defined function called by a trigger
on the messages table.

the trigger checks the message flags to see if a message has been read to
see if the counter of unread messages in another needs to be updated when
a message is deleted.
"""

# second bit is the "was read" flag
return (int(flags) & 0x02) >> 1

db = sqlite.connect('/var/mobile/Library/SMS/sms.db')

# register the user-defined function used by delete trigger
db.create_function('read', 1, message_read)

c = db.cursor()

c.execute('delete from message;')

db.commit()

# vim:set ts=4 sw=4 ai et tw=80:

you can download the source here.

i’m not much of a gui programmer and have no experience at all with apple’s cocoa interface or objective-c so for now this is just a command-line script. i hope to spend some time learning how to use pyobjc and working with the cocoa libraries to make an interface for it.

its easy enough to run the script through the mobile terminal app after copying the script to the iphone, chmod 755 delete_sms.py, then run it:


./delete_sms.py

no output from the script is a good thing. if something goes wrong it should spew several lines of errors describing what happened.

my favorite iphone apps

Posted on July 26, 2008
Filed Under iphone | 2 Comments

after holding out and suffering with my treo 650 through the launch of the first iphone i couldn’t help myself when the iphone 3g came out and had to get one.

it’s awesome.

sure, it crashes at least once a day, apps randomly stop working, it doesn’t record video or send/receive mms, the on-screen keyboard takes some getting used to, and apple has draconian software policies.

however, imho, the pros outweigh the cons.

by far the #1 app for the iphone is the safari web browser. combined with faster 3g networking and taking advantage of wifi where it’s available, it makes my treo seem like a commodore 64 by comparison - not that the commodore 64 wasn’t a great machine in it’s time.

my second favorite app is airme. with lots of out-of-state family who don’t get to see my kids very often i can take pictures of moments with my phone which typically would be lost before digging the nikon out. that in itself isn’t impressive but what is impressive is how it immediately uploads to flickr so my family can actually see the pictures instead of them staying on my phone until i get around to uploading them.

next up is tuner. i’m a news junkie. it drives me nuts whenever i can’t listen to npr, preferably via kut, when i want to. tuner lets you tune into any free mp3 stream on the net and kut has a great stream 24/7.

when i’m coding and want to listen to music - typically techno - i turn to the variety of streams available through pandora’s app. the quality is good and there’s something for everyone.

one of the things that made the iphone 3g nearly irresistible for me was the likelihood that it would be jailbroken like its predecessor. sure enough it took about two weeks before a jailbreaking app for macs was available. i wasted no time liberating mine from apple’s lock over what i could install. what makes this particularly appealing for this linux geek is that under the hood the iphone runs mac osx - apple’s variant of freebsd - which isn’t linux but still has a command prompt, a compiler and a nice selection of my favorite tools.

the jailbreaking utility provides a convenient package manager for installing non-sanctioned software. the nes emulator available through this is awesome and already has consumed too much of my time. the pacman game - macman - is also cool. i was also quick to install the terminal app - my preferred working environment. it gives you a command prompt for the bsd system from which it’s possible to ssh to a server, run scripts, etc

after installing python and the sample python app i was pleased to discover while reading the sample code that apple uses sqlite for a few of it’s databases including contacts, calendar, and sms messages. it took me about 20 minutes to write a script to purge all text messages - should be handy the next time a server decides to spam me with 200 text messages before someone can silence it.

i’m eagerly awaiting a video recording app - and will gladly pay for one. a bonus goes to the one that can auto upload to flickr or youtube. there’s an app already for the first iphone running the 1.x software so hopefully it won’t be long.

mms would be nice but with airme it isn’t that big of as deal without it.

i’m travelling next month and will post how ad-hoc tethering works.

testing threads with py.test

Posted on May 12, 2008
Filed Under python | Leave a Comment

i’m a big fan of automated unit tests. my preferred testing utility is py.test. for most things it works pretty well and its super easy to use.

threads are tricky to deal with but especially while trying to test them if you have a lot going on at the same time. by default py.test will not catch exceptions which occur in threads which, unless you’re careful to check for them, could lead to inaccurate test results.

i wrote the following class to help with thread testing:

class ThreadMonitor(object):
   """Helper class for catching exceptions generated in threads.

   Usage:

      mon = ThreadMonitor() 

      th = threading.Thread(target=mon.wrap(myFunction))
      th.start()

      th.join()

      mon.check() # raises any exception generated in the thread

   Any raised exception will include a traceback from the original
   thread, not the function calling mon.check()

   Works for multiple threads
   """
   def __init__(self):
      self.queue = Queue.Queue()

   def wrap(self, function):
      def threadMonitorWrapper(*args, **kw):
         try:
            ret = function(*args, **kw)
         except Exception, e:
            self.queue.put(sys.exc_info())
            raise

         return ret

      return threadMonitorWrapper

   def check(self):
      try:
         item = self.queue.get(block=False)
      except Queue.Empty:
         return

      klass, value, tb = item

      raise klass, value, tb # note the last parameter - traceback

this doesn’t fill all of one’s py.test thread testing needs, but it helps for certain cases.

the final line with the “raise” keyword raises the exception caught in the thread but the traceback argument causes py.test to show the original context of the exception. so with the “–pdb” option to py.test you’ll automatically drop into the context of the first exception on a failure in a monitored thread.

awesome screen/vim hack

Posted on April 22, 2008
Filed Under programming | Leave a Comment

add this to ~/.vimrc:

if $TERM=='screen'
   exe "set title titlestring=vim:%f"
   exe "set title t_ts=\<ESC>k t_fs=\<ESC>\\"
endif

and vim will automatically set the title of screen windows to:

vim:<file opened>

thanks go to:

http://gunkies.org/blog/?p=19

back home from pycon

Posted on March 19, 2008
Filed Under python | Leave a Comment

made it back home yesterday with only a 45 minute delay after nearly getting stuck in san antonio because of bad weather in austin.

i hung out in the bazaar/olpc sprint room monday. i’m pretty new to bazaar and wasn’t really able to contribute anything but i spent the time familiarizing myself with the code and trying out the bzr-svn plugin to evaluate whether we’d be able to use it at work. i listened to mark hammond and ian clatworthy discuss bazaar internals and mark’s strategy for developing tortoisebzr.

tummy.com was very generous and ordered pizza for all of the sprinters (35 pizzas for 300 people).

in the airport and on the plane yesterday i spent more time digging around the bazaar code. its quite readable and very well organized into layers where modules can be swapped out at various layers to change the behavior. the bzr-loom plugin adds quilt-like functionality in bazaar. looking around i saw that while its functional it doesn’t have the ability to change a loomified bazaar branch into a regular bazaar branch. it seems straightforward to add this so i started working on it on the plane. i got a crude version working which just discards any changes in threads but ran into some locking issues and wasn’t able to work them out before the battery died on my laptop. i’ll write more about that once i have more time to look at it.

pycon 2008 - day 3

Posted on March 16, 2008
Filed Under python | 2 Comments

the first keynote address was really good, calling for an easier way of getting a standardized python distribution on windows to make it easier to distribute single python scripts. the second keynote address, snake charming the dragon, was also good but dragged on a little too long with perhaps too much detail. mark hammond discussed pyxpcom and the lack of community support as well as the direction of the mozilla project. the olpc talk was excellent. despite ivan losing his original presentation and having to rewrite it during the first two on someone else’s computer, it was an excellent overview of where the olpc project is and some of its early successes in the field.

next i went to the consuming html talk by ian bicking. he started with presenting a case why you might want to consume html instead of either xml or xhtml and continued by discussing various python libraries for parsing and working with html. i didn’t realize that the lxml parser was a suitable alternative to BeautifulSoup and will certainly give it a closer look. there was a mini-argument about the merits of xhtml following ian’s assertion that xhtml will never be widely accepted, including annoying interjections during the talk. otherwise it was a pretty good talk but the level of detail was a bit lower than i had expected.

the wingware presentation was interesting. i’ve never been much of an IDE user - i’ve tried using slickedit and others but i keep falling back to trusty vim. i don’t expect to start using either wing or komodo - the nature of my company’s product makes it difficult to develop solely on one’s laptop and i haven’t been comfortable with remotely editing, given that i’ve typically had gateway hops between my workstation and the code i’m editing. its impressive that the wing ide and the supporting business is managed by two guys.

titus brown’s olpc testing techniques talk was very informative and unexpectedly critical of the testing techniques, or lack thereof, used by the olpc project, asserting that without testing the project will inevitably lead to a quality death spiral. he’s doing some cool stuff with code coverage, using python’s sys.settrace() function to record the execution path of olpc’s sugar interface, sending it to a web server where it can be immediately analyzed with provisions for clearing previous history and omitting uninteresting execution paths.

i sat in on the wing ide bof for a bit while eating lunch but it was mostly wrapping up with discussions about possible and future features by the time i got there.

the lightning talks today were much better. you can see the links i saved on my del.icio.us page. i was going to write more about each of them but i didn’t take any notes and don’t have a great memory.

following the lightning talks today was brett cannon’s humorous and well presented intro to sprinting, after which the sprint coaches introduced themselves and answered various questions about sprinting.

the superhuman duo from wingware described earlier were generous enough to provide free licenses of their premium product to all sprinters. i got my activation key without much delay and will download & try it out in the next few days.

after considering the rather large list of of sprints i settled on the bazaar sprint because it has the possibility of being most directly applicable to my day job and seems to have a low barrier to entry. also version control is something i’ve spent quite a bit of time thinking about in the course of trying to implement release management for my employer.

there was a good turnout for the bazaar sprinting tutorial. the core bazaar developers from canonical really have their stuff together and provided a very thorough introduction of both how to use bazaar and how to contribute to bazaar, either by creating a plugin or changing the core project itself. again, the core bazaar team and its surrounding community really deserve some recognition for the outstanding quality of the source code and development process - everything from a complete test suite to a patch submission tracking page where each patch must be reviewed by two core developers before it is allowed to be merged into the central bazaar repository.

tomorrow is my last day here in chicago and i plan to spend it sprinting with the bazaar bunch.

pycon 2008 - day 2

Posted on March 15, 2008
Filed Under python | 5 Comments

some of the keynotes from this morning were great. others, less than exciting - informative at best. i used the time to check out allmydata.org’s tahoe distributed filesystem, connecting to an ad-hoc p2p network started by other pycon participants. cool stuff. i haven’t quite figured out how to replicate data from other nodes, though.

the first talk i went to today was supervisor as a platform, about http://supervisord.org. overall it seems like a cool project for managing processes on multiple machines. however i wish that it didn’t require processes to run in the foreground - this might be nice for new programs but for well established programs which daemonize themselves i think it would have been better to use the standard /etc/init.d/ script interface.

the pypy talk was pretty good. i’ve been following their blog for a while so i didn’t see much new but it was well presented and received a good response from the crowd.

the django talk was really good. the example showing how to make a view reusable by adding parameters for the form class, the template, and the redirect url makes a lot of sense. its good to see there’s a way for doing reverse url lookups for dynamically determining where an application is rooted in the site’s url tree - that was a point of frustration for me before. the emphasis on writing multiple smaller, narrowly defined applications made me realize that this would have been a much better way of working for the django apps i’ve written.

lunch was better today.

the multitouch presentation was excellent. i’ve met with the guys from enthought at the austin python user’s group meeting they host once a month and saw a preview of this presentation some months ago. they’ve come a long way since then. too bad it would be a monumental and/or expensive effort to get one of these things.

i haven’t looked at trac too much but the plugin architecture seems solid and well thought out. the presentation was excellent and gave a great overview of what is possible when developing trac plugins and how to go about doing so. the company i work for could probably benefit from switching to trac - seems like someone there looked at it before but i’m not sure where that ended.

the stackless + twisted talk is probably most directly relevant to my day job. a coworker and i implemented a test driver using stackless to take advantage of lightweight threads and were surprised to find no readily available solution for dealing with blocking socket operations. the stacklesssocket module comes close but had some problems conforming to the socket module’s behavior and wastes quite a bit of processor (looking at the page for stacklesssocket now i see that richard tew has integrated some fixes for these issues). after several rounds of finding and killing bugs we got something usable and have been quite happy with the results. i’m not sure if using twisted is the best option for our environment because it doesn’t completely hide the fact that twisted is used. i’ll have to make our changes to stacklesssocket available somewhere sometime.

the lightning talks were different today. a few plugs for various projects, tripping over podiums, possessed computers, etc. the resolver spreadsheet product is cooler than i had imagined it would be - generating ironpython code behind the scenes to reproduce the displayed spreadsheet, allowing python formulas, etc. overall the talks from yesterday were better than today’s but i’ll admit that i zoned out after a few of the less interesting talks and bailed out early for food so i could make it to the bazaar introduction i mostly missed yesterday.

speaking of which, the bazaar introduction was great. before yesterday i had only briefly looked at bazaar and regarded it as one might a strange looking bug, much the same way i initially regarded python upon seeing the whitespace treatment. but, like python, upon further inspection it appears to be more useful than i initially thought. not requiring a centralized repository allows a more flexible workflow between coworkers. merge tracking is far superior than subversion’s and would appear to eliminate spurious conflicts when merging multiple times from the same source. additionally, much work has gone into not only allowing multiple use case scenarios with varying degrees of molding bazaar to work more like subversion on the one extreme or git on the other, but to provide some level of interaction between other version control systems. the subversion plugin to bazaar appears to be the most mature and allegedly allows a nearly seemless flow of changes to and from subversion to a bazaar branch. i’m certainly going to test this with our subversion-hosted source code to see if its feasible.

overall, today was good. i think i learned more about more projects yesterday but perhaps i learned more about a few individual projects today, specifically tahoe and bazaar.

pycon 2008 - day 1

Posted on March 14, 2008
Filed Under python | 2 Comments

day 1 of pycon 2008 is wrapping up.  i’d say it turned out pretty well.  aside from wireless issues and some minor registration problems everything seemed to run smoothly and on schedule.

all of the talks i attended were interesting:

the lightning talks were probably the most entertaining, especially those given for the daemonize module and tinypy.

projects that i learned about today which i’d like to check out are:

i’ve heard of a few of these before but learned more about them today which piqued my interest.  i think i’ll check out bazaar first.

more tomorrow.

adding pjsip to my openmoko overlay - part 2

Posted on February 28, 2008
Filed Under openmoko | 2 Comments

since my last post i’ve made some progress with my bitbake recipe to build the pjsip project for openmoko.

the first problem i ran into with the install was bitbake complaining that files from a .svn subdirectory from the source tree couldn’t be installed because of a permission problem. those files shouldn’t be installed anyway but were as a side effect of pjsip’s install Makefile rule, which does a recursive copy of the include directories within each subdirectory to /usr/local/include. this fixes it:

# removes all of the .svn files to avoid problems
# during the install
do_remove_svn_files () {
   # "|| true" is to avoid problems with recursing into directories
   # that are removed
   find ${S} -type d -name '.svn' -exec rm -rf {} \; > /dev/null \
      2>&1 || true
}
addtask remove_svn_files after do_unpack before do_patch

pjsip’s makefile is hardcoded to install the libraries and header files into the /usr/local prefix. the default bitbake recipes don’t look for installed files in these directories and complained that the files were installed but not listed in any package. so, copying what i saw in the ffmpeg recipe, i tried this:

FILES_pjsip-dev = “/usr/local/lib/lib*.a \
/usr/local/include/*\
/usr/local/lib/pkgconfig/libpj.pc”

this appears to have no effect. after some trial and error and finally discovering the “-e” option to bitbake, i saw from this command - bitbake -e pjsip | less - that the FILES variable is already assigned by one of the bitbake classes. i thought my assignment would override it but i guess not. instead, something like this is necessary:

FILES_pjsip-dev += "/usr/local/lib/lib*.a \
/usr/local/include/*\
/usr/local/lib/pkgconfig/libpj.pc"

or the more conventional:

FILES_${PN}-dev += "/usr/local/lib/lib*.a \
/usr/local/include/*\
/usr/local/lib/pkgconfig/libpj.pc"

this works perfectly and produces an ipkg file in build/tmp/deploy/glibc/ipk/armv4t.

pjsip’s install Makefile rule doesn’t install any of the command line utilities or the python bindings. i’ll take a stab at those next.

keep looking »