Current aspects in machine authentication of security documents

In part 1 of this publication Ulrich Schneider and Uwe Seidel discussed the current situation regarding the application of machine authentication systems to security documents, with respect to international guidelines of the International Civil Aviation Organization^1,2, as well as the key components involved in the process of machine authentication: the security document itself and the reader hardware and software. In part 2, the authors present the result of the analysis of the authentication performance of a number of currently available inspection systems.

Although the current article, being the follow-up of part 1 published in issue 41 of the Journal³ obviously needs only little motivation, the authors would like to briefly discuss another real case of a manipulated e-Passport. 

Figure 1 shows an image of an illegally ‘issued’ stolen blank European passport with an overlay of its X-ray image. It is obvious that there are two inlays inside: the authentic one with the antenna in a rectangular shape (RFID chip dysfunctional) and an additional inlay with the irregular shape (RFID chip ‘alive’). The data page was personalised in very good correspondence with regularly issued authentic documents of this series.

Leaving the electronic component aside, the optical inspection of the data page of that document, using three state-of-the-art full-page readers gave three different responses: 

• System A: no decision – no clear identification of the series, verification response indifferent.
• System B: wrong decision – series correctly identified, all specified checks were passed leading to a false acceptance.
• System C: correct decision – series correctly identified, the check on the infrared properties of the ink used for personalisation discovered a different absorption of the illegally versus the genuine ink used.

In this article the results of the systematic analysis of the authentication performance of currently available inspection systems are presented using a reference set of passports of the EU Member States (EU-MS) and the Schengen Associated Countries (SAC).

For a closer look, a subset of 20 was carefully chosen out of 36 models of valid series (EU + SAC). Figure 2 shows a typical example of the project’s data set.

R&D project IDEAL
A rigorous analytical approach to the current situation as sketched in part 1 of this article directly leads to the following core questions:
• Is it possible to generalise successful strategies for machine authentication? Is there potential for standardisation?
• How do commercial document inspection systems perform?
• Is it possible to specify self defined inspection processes to analyse and optimise the performance of commercial systems?

Initially, commercially available machine authentication (MA) systems consisting of hardware and software of the following six manufacturers were selected: 3M, AssureTec, Avalon Biometrics, Bundesdruckerei, Crossmatch and Regula. For a common logging scheme of authentication results a prototypical application interfacing with an established middleware standard was developed (secunet biomiddle). Because of the heterogeneous log formats, the syntax and semantics of the systems, this step was necessary to facilitate the evaluation based on a comparable syntax⁴. 

Generalized check routines
As commercial systems use proprietary designations to reference the check routines they perform – sometimes even different from document to document – for a generalised technical debate there is an obvious need to agree upon a common terminology in order to facilitate comparability. To solve this task and as a prerequisite for further research, a complete catalogue of check routines for machine authentication was identified and generically defined. Subsequently, the existing proprietary routines were mapped to these generic ones.

A ‘generic’ check routine is characterised by:
I) the light source under which it operates; and
II) the security feature (or area) it is operating on; and
III) the (physical) property of this feature the check routine exploits.

Furthermore, it was deemed necessary to differentiate between basic check routines as defined above and extended check routines. Basic check routines check for existence or absence of a property or feature in one of the images (VIS, IR and UV). Extended check routines logically link two or more basic check routines and are applied across more than one image. 

Figure 3 illustrates the definition of the generalised nomenclature for the generic check routines based on features and their properties revealed by different light sources.

To systematically denote the relevant parameters, the designations of the generalised check routines are constructed using the following scheme: ‘lightsource_property_feature’, leading to a table of two letter abbreviations (see table 1). The colours refer to three of the categories contained in the minimum security standards of EU Council Regulation (EC) No 2252/2004 for passports and travel documents⁵.

Table 1.

While the majority of check routines and their application to a passport model strongly depends on the security design and therefore on the availability of specific security features, the following generic check routines shall be applied to every document, regardless of whether the model can be uniquely identified or not, since they are common to all standard compliant security documents:

• IR_AB_MR (IR absorbance of the MRZ lines) and
• UV_BR_PH + UV_BR_VZ + UV_BR_MR (UV brightness in separate regions) or
• UV_BR_FU (UV brightness of the full page).

Using this scheme, an in-depth analysis of 20 selected passport models of EU-MS+SAC was performed to examine the potential inherent in these passports for machine authentication.

Figure 4 shows a wide variety in the numbers of applicable check routines. For this exercise, quality and reliability issues were left aside. Moreover, while in the left part of figure 4 every applicable check routine is counted only once per passport, the complete picture has to take into account the redundant applicability on more than one ink (for example, routine UV_LU_IS for red and blue luminescent ink), as shown in the right part of figure 4.

The passport model FRA_2006 shows the highest potential for MA (25 check routines applicable), whereas DNK_2006 shows the lowest (13 check routines applicable). The number of extended check routines varies from three (FRA_2006, POL_2008) to zero (PRT_2006). 

New passport models do not necessarily lead to an enhanced design with respect to the (limited) machine point of view: while LTU_2008 shows an improvement with respect to machine authentication, GBR_2010 and BEL_2008 show fewer possibilities than their predecessors.

Performance of commercial systems
Inspection system potential
Figure 5 shows a comparison between theoretically checkable features according to figure 4 and actually deployed routines for the three examined systems. Basis is the preceding theoretical analysis for all 20 MRTDs.

As shown in figure 5, it is clear that the systems vary considerably with respect to their usage of the MA potential offered by the documents: typically they do not use more than 50% of the overall potential. For this figure, the numbers count for routines only, without the investigation of quality and reliability of the individually defined check routines. Therefore, this graph can only serve as a hint for the checking depths of the commercial systems. The authors’ analysis also showed that commercial systems mostly lack the ability to use the logically linked result of two or more base routines in different wavelength regions (extended check routines).

Identification rates
The correct identification of the passport model is the basis for a subsequent successful machine authentication. If a system fails to identify the correct model, not only is it unable to apply model specific routines, but it will, in the worst case, lead to false-positive decisions. 

The performed evaluation was aimed to answer the following questions:
– Which rate of live image data sets (documents put on the reader) will be assigned to the correct reference data sets? (identification rate, IDR).
– For which rate of live image data sets do the systems correctly assign the country, but not the actual passport model? (human assisted identification rate, TIR).
– Which rate of live image data sets documents put on the reader will be assigned to the wrong reference data sets? (false identification rate, FIR).
– Which rate of live image data sets documents put on the reader will be assigned to no reference data sets? (non-identification rate, NIR).

The results of the evaluation are shown in figure 6.

 

Genuine documents
All systems are able to identify genuine documents quite reliably and show an IDR > 90%. System C performs best.

Reproductions
For reproductions, systems A and B still show an IDR = 80%, whereas system C shows an IDR = 30%. This is due to C’s identification algorithm, which takes the colour distribution into account. As the reproductions show colours slightly deviating from the original, they tend to be difficult to identify, resulting in system C’s NIR of 70%.

Counterfeits
For counterfeits, system C performs best (IDR = 70%): only this system uses the white light image to identify the document if IR reading of the MRZ fails, as this happens to be the case for some counterfeits. Therefore, this approach of system C leads to the best identification performance for counterfeits. 
Only system B allows for a manual choice to identify the correct model if the automatic process fails to do so or is in doubt. This proves a useful feature for manually assisted document inspection, but is obviously not suitable for unattended (ABC) scenarios.

Verification rates
After a document model is  successfully identified, the systems shall correctly verify genuine documents and counterfeits. The evaluation provided answers to the following questions:
– Which rate of genuine documents will be rejected by the commercial systems? (false reject rate, FRR)
– Which rate of false documents (reproductions and counterfeit documents) will be accepted by the commercial systems? (false acceptance rate, FAR)

Figure 7 shows the error rates FRR for genuine documents and FAR (FAR_R) for reproductions and ‘true’ counterfeits (FAR_A).

For the false reject rate (FRR), the best system is B with an FRR = 0% (i.e. no genuine document was falsely rejected), followed by C with an FRR = 8%. System A shows an unacceptable 45%⁶.
For the false acceptance rate (FAR), all systems show a comparable performance with a FAR between 10% and 20%. System A is the best, followed by C and then B.

Generally and not surprisingly, ‘true’ counterfeits are more often accepted than the reproductions, since they aim to mimic the genuine documents in more aspects than were tried to model with the relatively simple reproductions. A limiting factor to this study is the small number of individual documents, especially counterfeits and therefore the lack of a broad statistical basis.

FRONTEX Document Challenge II – sneak preview of additional technical tests
From 16 September to 2 October 2013, seven commercial systems took part in the FRONTEX Document Challenge II exercise in Lisbon, Portugal, which is part of the R&D Project IDCheck 2013. Six of the seven systems were performing optical machine authentication. 
In addition to the main Document Challenge Field test phase, which is currently being evaluated and will be published in the first quarter of 2014, the Forensic Science Institute of German Bundeskriminalamt (BKA) provided 82 test cases for optical machine authentication similar to the project IDEAL: 24 genuine ‘specimen’ passports plus 24 reproductions of their personal data page and an additional 24 ‘specimen’ ID-cards (including one reproduction) plus 10 ‘real’ passport forgeries. Similar to the IDEAL project reported on earlier, the systems FRR and FAR were evaluated with the genuine and false documents, this time including the reproductions.

The exercise showed the following results (see figure 8): 

• 6 out of 7 systems provided machine authentication results (‘D’ serves only as an intelligent multispectral camera plus chip reader).
• Systems B and E have the lowest FAR ~5%.
•  Systems A, B and G have the lowest FRR ~10% or lower⁸.
• The sum of FAR and FRR is lowest for system B, followed by system E.
• System G has an enormous and unacceptable FAR close to 70%, having accepted 22 out of 24 reproductions as genuine.

The systems’ response was also tested to the presentation of ID1-size ID cards. The following, partly very surprising results were recorded:
– 6 out of 7 systems require starting the evaluation with the cards’ back (MRZ side).
– As an interesting exception, system F requires the front first.
– Systems A and E denote the front confusingly as ‘the second page’ or ‘page 1’.
– System B actually requests to place the front on the reader.
– Systems C and G do not read (identify) the front of ID cards at all, but stop after reading the back of the card.

Generally, the systems show a very heterogeneous performance. Moreover, the overall inspection results of two-sided authentications unfortunately tend to be mixed up in an intransparent way.

In summary: while the performance with respect to FRR/FAR shown in the BKA test of the FRONTEX exercise closely resembles the findings of the IDEAL project, the additional test on ID cards revealed major issues in the handling and authentication logic of the commercial systems. This is especially important for Europe since ID cards are a valid means of crossing inner-European borders or even entering the Schengen area from abroad.

Summary and conclusion
The documents analysed in the project were found to show a wide variety in their potential for machine authentication. This supports the claim that MRTDs nowadays are still not designed with (optical) machine authentication in mind. However, commercial inspection systems do not even use this (sometimes limited) potential to its full capacity. This result proves the lack of communication between security document designers on the one hand and the IT architects of commercial MA systems on the other. 

The performance of commercial inspection systems including their authentication processes is very heterogeneous. This holds for their ability to identify document models as well as for their verification performance. There is no clear ‘winner’ among the systems evaluated.
All systems represent a ‘black box’ for the user without any influence to control the key processes identification and verification. Only one system offers the possibility to select an operational sensitivity (high – as used in this study -, normal and low) resulting in a corresponding FRR/FAR ratio. It is also not possible for the operating agency to adapt the systems according to different scenarios: staffed, but machine assisted inspection vs. ABC gates might result in different configurations of the MA systems. Having full control over the systems’ parameters, one might even consider a ‘situation dependent’ modification of the systems’ checking depth.

Check routines and their designations vary considerably between vendors. As one of the major results of the project, 34 generic check routines were defined. It is the authors’ current opinion that these describe possible machine authentication processes and will be put forward for future standardisation. 
As model identification is one of the key steps at the beginning of the whole process of machine authentication, the unique identification of a document model by MA systems might be greatly supported by a standardised, machine readable ‘imprint’, for example in the form of an additional 2D barcode or as part of the MRZ. As this was already recommended by Interpol as far back as in 1992⁹, it is discussed as a future work item of ICAO’s New Technologies Working Group. 

References and footnotes
1 International Civil Aviation Organization (ICAO), Doc 9303: Machine Readable Travel Documents, Part 1: Machine Readable Passports, current version: sixth Edition 2006; http://www.icao.int/Security/mrtd/Pages/Document9303.aspx.
2 ICAO Technical Report Machine Assisted Document Security Verification, September 2011. www.icao.int/Security/mrtd/Downloads/Machine%20Authentication/Machine_Authentication_TR_v10%20(2).pdf.
3 Schneider, U. and Seidel, U. Current Aspects in Machine Authentication of Security Documents – Part I: Do we need optical document security? Keesing Journal of Documents & Identity, Issue 41, 2013.
4 For reasons of reproducibility the authors did neither enclose new passport models nor product software updates published during the project’s runtime except for debugging of severe errors negatively influencing functionality.
5 Council Regulation (EC) No 2252/2004 of 13 December 2004, OJ L 385 of 29 December 2004, p. 1-6, entered into force as of 18 January 2005.
6 This result is most likely a consequence of the authors’ evaluation procedure not considering the systems integral weight function.
7 Gariup, M. and Soederlind, G. Document Fraud Detection at the Border: Preliminary Observations on Human and Machine Performance. EISIC Proceedings 2013, IEEE Computer Society.
8 Please note that the ‘Specimen’ imprint shown on most of the samples might have influenced (increased) the FRR, but does not explain the high error rates completely.
9 Interpol resolution AGN/61/RES/7 Travel Documents, 1992.