Copyright Detection of Copied Paper Documents
Developed a system to automatically manage copyrighted printed materials by
monitoring photocopiers to detect the copying of pages from copyrighted
publications. The goal is to reimburse publishers for copying done in schools
in countries without a "fair use" doctrine. This approach produces precision
and recall above 98% with millions of page images. The customer really liked
demonstrations of the system but a pilot deployment was delayed a couple of
times by stalled negotiations due to reorganizations until the current delay
caused by COVID-19.
Technical challenges. Started with an adaptation of our Embedded
Media Marker approach that turned out to be unsuitable for more than a few
thousand images. Used the same SIFT-like image feature descriptors and turned
them into visual words for bag-of-words matching. Visual words produced very
good results but required a compression technique for the inverted index to
keep it in memory for millions of pages, each containing thousands of
keypoints. Hierarchical k-means clustering of feature vectors to
determine visual words also required techniques to reduce memory usage by never
copying feature vectors, only swapping them (2.5 billion 40-byte vectors for 1
million images). Geometric verification by RANSAC affine-transform fitting
proved to be insufficient because of matching patches on pages. Developed a
grid-based, uniform-distribution approach for the geometric verification of
whole pages.
Technologies. C++, Java (for server), MongoDB, DynamoDB, AWS Lambda,
SIFT-like features, hierarchical k-means clustering, visual words.
- A. Girgensohn, L. Wilcox, Q. Liu. Automatic Rights Management for Photocopiers. Document Engineering, article 21, doi:10.1145/3209280.3209531, 2018.
- X. Yang, Q. Liu, C. Liao, A. Girgensohn. Large-Scale EMM Identification With Geometry-Constrained Visual Word Correspondence Voting. ACM International Conference on Multimedia Retrieval (ICMR), article 35, doi:10.1145/1991996.1992031, 2011.
Document Linking with Embedded Media Markers
Designed and developed a system that links paper to digital media. Faint
marks on the paper indicate the presence of a link. The link is established
via an image search from keypoint features in a patch of the paper in a
database of links to digital content. This work formed the basis of a Fuji
Xerox product.
Technical challenges. Developed a high-performance Java web server
that used the Java Native Interface (JNI) to detect keypoints in a submitted
image and to perform a search in an approximate nearest neighbor tree (ANN).
Created an Android app that used video capture to frequently check whether the
captured image had a match in the database. Designed and developed an
authoring tool that indicated to the user whether a region on a page was
unsuitable for linking due to insufficient or even blank content.
Technologies. Android, Java (for app and server), JavaScript, MySQL, JNI, C++,
SIFT-like features.
- Q. Liu, A. Girgensohn, L. Wilcox, F. Shipman, T. Dunnigan. MET: Media Embedded Target for Connecting Paper to Digital Media. International Journal of Multimedia Information Retrieval, 3(3), 177-192, doi:10.1007/s13735-014-0059-7, 2014.
- X. Yang, Q. Liu, C. Liao, A. Girgensohn. Large-Scale EMM Identification With Geometry-Constrained Visual Word Correspondence Voting. ACM International Conference on Multimedia Retrieval (ICMR), article 35, doi:10.1145/1991996.1992031, 2011.
- A. Girgensohn, F. Shipman, L. Wilcox, Q. Liu, C. Liao, Y. Oneda. A Tool for Authoring Unambiguous Links From Printed Content to Digital Media. ACM Multimedia, 1561-1564, doi:10.1145/2072298.2072065, 2011.
Document Browsing and Searching
Created a system that ingested a large collection of office documents and
made them browseable and searchable via a web interface. Documents were
converted to PDF to extract page thumbnails and text. Text phrases were
detected and used to summarize documents.
Technical challenges. Created a pipeline that opened documents with
OpenOffice to save them as PDF. Used Ghostscript to extract page images that
were scaled down to different sizes for a zoomable UI. Summarized directory
trees by showing popup collages of thumbnails of documents best matching a
search or having interesting properties. Developed a zoomable UI that adapted
document boxes and thumbnail sizes. Offered a document clip UI that showed
parts of document pages matching a search.
Technologies. Java, JavaScript, PDF, OpenOffice, Ghostscript.
- F. Chen, A. Girgensohn, M. Cooper, Y. Lu, G. Filby. Genre Identification for Office Document Search and Browsing. International Journal on Document Analysis and Recognition (IJDAR), 15(3), 167-182, doi:10.1007/s10032-011-0163-7, 2012.
- A. Girgensohn, F. Shipman, F. Chen, L. Wilcox. DocuBrowse: Faceted Searching, Browsing, and Recommendations in an Enterprise Context. IUI, 189-198, doi:10.1145/1719970.1719997, 2010.