An End-to-End Trainable Neural Network for Image-Based Sequence Recognition

What this paper solves

Optical character recognition and scene text reading require mapping a sequence of visual features to a sequence of labels of unknown length. Prior systems segmented the input image into individual characters before recognition, requiring precise segmentation as a preprocessing step. This paper introduces the Convolutional Recurrent Neural Network (CRNN): a fully convolutional feature extractor feeding into a bidirectional LSTM decoder trained with Connectionist Temporal Classification (CTC) loss, enabling end-to-end training without any segmentation step.

The architectural contribution is the combination of spatial feature extraction with temporal sequence modeling in a single differentiable pipeline. The same architecture generalizes to any problem where the input has spatial structure and the output is a variable-length sequence.

Key mathematical framework

Feature sequence extraction

Given input image $I\in {\mathbb{R}}^{H\times W\times C}$, a convolutional feature extractor produces a feature map $F\in {\mathbb{R}}^{H^{\prime }\times W^{\prime }\times D}$. Vertical slices of this map (columns) form the feature sequence:

Each $x_t$ captures the visual features at horizontal position$t$ across the full height of the feature map. The sequence length $W^{\prime }$ depends on the input width through the convolutional downsampling factor.

Bidirectional LSTM

The feature sequence is processed by a two-layer bidirectional LSTM. At each timestep $t$, the forward and backward hidden states are:

The concatenated state $h_t=[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$is projected to a per-class score over the label vocabulary at each position:

where $|\mathcal{V}|+1$ includes a blank token for CTC.

Connectionist Temporal Classification loss

The label sequence $\mathbf{l}=(l_1,\dots ,l_S)$ is shorter than the output sequence $T=W^{\prime }$. CTC defines a set of paths${\mathcal{B}}^{-1}(\mathbf{l})$ (output sequences that collapse to $\mathbf{l}$ by merging repeated labels and removing blanks) and trains to maximize:

This sum over paths is computed efficiently with a forward-backward dynamic programming algorithm, making the loss differentiable with respect to the per-timestep output scores. The result is that neither the alignment between input positions and output labels, nor the segmentation boundaries, need to be specified during training.

Empirical results

• Scene text: State-of-the-art accuracy on IIIT5K, SVT, and ICDAR benchmarks at the time of publication, without scene-specific fine-tuning
• End-to-end training: No segmentation preprocessing required. A single backward pass through the CTC loss trains the full convolutional plus recurrent pipeline
• Speed: Real-time on a GPU, with the convolutional backbone shared between position encoding and sequence features

What this means for robotic automation

The CRNN architecture establishes the template for any system that must map a spatially structured input to a variable-length output sequence without prior knowledge of the alignment between input regions and output elements. CTC loss is the correct objective when the alignment is unknown and should be inferred from data.

For sequential anomaly observation models, the key transfer is the bidirectional LSTM component. A system that processes a temporal sequence of observations to produce a state label (confidence class, heading decision, localization region) faces the same alignment ambiguity: which observations correspond to which state transitions. The bidirectional LSTM's ability to capture context from both past and future frames in a sequence makes it the correct architecture for this class of problem.