Armitage struck in each half for Irish, whose other nine points came from three Tom Homer penalties.
England fly-half Charlie Hodgson was the stand-out performer for Saracens, kicking all five of his penalties in addition to two drop goals and converting Sarries' only try, scored by flanker Will Fraser.
Irish were even denied a losing bonus point because of Hodgson's last minute drop goal.
Flanker Jamie Gibson was sin-binned in the 52nd minute.
Alex Corbisiero became the youngest London Irish player, at 23 years old, to captain his side in the professional era.