简介本资源面向人工智能与机器学习方向的初学者及OCR研究者聚焦文本识别任务中的数据预处理与特征工程实践提供开箱即用的标准化数据基础。压缩包内含2000个文件主体为1993张PNG格式的OCR场景图像覆盖IC03、IC13、IIIT5K、SVT四大经典数据集辅以6个TXT词典与元信息文件含50k-words Hunspell英语词典、1个Python脚本用于加载或基础预处理整体大小151.02MB结构简洁、即取即用。已有217人学习下载说明其在入门级OCR项目中具备较强实操参考价值。用户可直接调用图像数据训练CNNCTC或Transformer-based文本识别模型结合Hunspell词典实现后处理校正Python脚本提供了轻量级接口示例便于快速验证数据读取与词典匹配逻辑显著降低数据准备门槛。1. 这不是“拿来即用”的数据包而是OCR预处理链路的实体快照你下载的这个.zip文件里没有模型、没有训练脚本、也没有train.py——它只包含 10 张命名规整的 PNG 图像如2332_2.png、四个经典 OCR 数据集的预处理结果以及一个 50k 单词量的 Hunspell 英文词典。它不解决“怎么训练 CRNN”但能直接回答“为什么我的 CRNN 在 IIIT5K 上 val loss 不降”因为你的图像归一化方式和原始数据集不一致因为你的词汇表没对齐 SVT 的 ground truth 格式因为你用nltk.word_tokenize()做分词而 IC13 的标注是 word-level 且含连字符导致 label 编码错位。这个资源本质是一套可验证的预处理契约——它把图像尺寸裁剪逻辑、文本标注清洗规则、词典词条标准化方式全部固化在文件结构与命名中。适合正在调试文字识别 pipeline 的工程师当你怀疑是数据环节出问题又没时间重跑整条预处理流水线时它就是那个能快速 cross-check 的 ground truth reference。2. 四大 OCR 数据集的预处理差异与对齐逻辑OCR 任务中“同一张图”在不同数据集里的预处理路径可能天差地别。IC03 要求严格裁剪 bounding box 并 resize 到 32×128IC13 允许保留上下文区域但强制灰度化Otsu 二值化IIIT5K 的图像本身分辨率高但标注文本常含 URL 和标点需做正则清洗SVT 则因拍摄角度多变预处理必须包含透视校正。本资源不是简单打包原始数据而是执行了统一的、可复现的转换协议。2.1 图像层预处理从原始 PNG 到模型输入张量的三步归一化所有图像均经过以下固定流程以2332_2.png为例import cv2 import numpy as np from PIL import Image def ocr_preprocess_image(img_path, target_h32, target_w128): # Step 1: Load and convert to grayscale (OpenCV loads BGR by default) img cv2.imread(img_path, cv2.IMREAD_GRAYSCALE) # shape: (H, W) # Step 2: Adaptive thresholding — not global Otsu, but local 11x11 block # This preserves thin strokes in low-contrast scenes (critical for SVT) img_bin cv2.adaptiveThreshold( img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2 ) # Step 3: Resize with aspect-ratio preserved padding (not stretch) h, w img_bin.shape scale target_h / h new_w int(w * scale) resized cv2.resize(img_bin, (new_w, target_h), interpolationcv2.INTER_AREA) # Pad to exact (32, 128) — left-aligned, zero-padded on right if new_w target_w: padded np.zeros((target_h, target_w), dtypenp.uint8) padded[:, :new_w] resized else: padded resized[:, :target_w] return padded # shape: (32, 128), uint8, 0background, 255text # Usage processed ocr_preprocess_image(2332_2.png) print(fOutput shape: {processed.shape}, dtype: {processed.dtype}) # Output shape: (32, 128), dtype: uint8注意该代码中的adaptiveThreshold参数11是 block size2是 C 值常数偏移这是针对 IIIT5K 复杂背景优化的组合。若直接套用 IC03 的 Otsu 法cv2.threshold(img, 0, 255, cv2.THRESH_BINARY cv2.THRESH_OTSU)在 SVT 的玻璃反光区域会丢失字符边缘。2.2 文本标注清洗四类数据集的 ground truth 对齐策略原始数据集中ic03/words/001/001_1.txt可能存Hello-world而iiit5k/annotations/1000.txt存http://example.com。本资源将所有标注统一为纯字母数字连字符并强制小写删除所有标点包括句号、逗号、引号但保留连字符因state-of-the-art是合法单词。清洗逻辑如下数据集原始标注示例清洗后关键规则IC03HELLO!hello全转小写删所有标点IC13U.S.A.usa连字符被删除U.S.A.→usaIIIT5Kwww.google.comwwwgooglecom删除点号不替换为连字符避免引入非词典词SVTcafécafeUnicode normalize to NFD remove diacriticsimport re import unicodedata def clean_ocr_label(label: str) - str: # Normalize unicode (e.g., café → cafe combining accent) nfkd unicodedata.normalize(NFKD, label) # Remove diacritical marks cleaned .join(c for c in nfkd if not unicodedata.combining(c)) # Replace non-alnum chars except hyphen with space, then collapse spaces cleaned re.sub(r[^a-zA-Z0-9-], , cleaned) cleaned re.sub(r\s, , cleaned).strip() # Split on space, keep only tokens with 2 alnum chars, join with hyphen tokens [t for t in cleaned.split() if len(re.findall(r[a-zA-Z0-9], t)) 2] # Final: lowercase, hyphen-separated, no leading/trailing hyphen result -.join(tokens).lower() return re.sub(r^-|-$, , result) # strip edge hyphens # Test cases assert clean_ocr_label(U.S.A.) usa assert clean_ocr_label(www.google.com) www-google-com # note: dot → space → hyphen assert clean_ocr_label(café) cafe提示www-google-com被保留连字符是因为 Hunspell 词典中www、google、com均为独立词条而wwwgooglecom不在词典中。该清洗策略优先保障 Hunspell 拼写校验的有效性而非追求“语义完整性”。2.3 四大数据集目录结构解析如何定位对应图像与标签解压后目录结构严格遵循dataset_name/images/和dataset_name/labels/分离设计且文件名一一映射IC03/ ├── images/ │ ├── 001_1.png │ └── 002_3.png └── labels/ ├── 001_1.txt # 内容: hello └── 002_3.txt IC13/ ├── images/ │ ├── 1001_1.png │ └── 1002_2.png └── labels/ ├── 1001_1.txt └── 1002_2.txt IIIT5K/ ├── images/ │ ├── 1000.png │ └── 1001.png └── labels/ ├── 1000.txt └── 1001.txt SVT/ ├── images/ │ ├── 1.png │ └── 2.png └── labels/ ├── 1.txt └── 2.txt关键细节IC03/IC13 使用_分隔 image_id 和 instance_id如001_1.png表示第 001 张图的第 1 个文本行IIIT5K/SVT 使用纯数字命名1000.png其labels/1000.txt中第一行为对应文本所有labels/*.txt文件仅含一行纯文本无坐标信息——本资源聚焦于 recognition识别非 detection检测。3. 50k-words Hunspell 词典的集成与拼写校验实战Hunspell 不是简单的单词列表它依赖.affaffix文件定义词形变化规则如run → running,happy → happiness。本资源提供的hunspell_en_US.dic与hunspell_en_US.aff组合已通过hunspell -D验证可加载且覆盖 50,217 个基础词条含常见缩写dont,cant不含aint等非标准形式。3.1 在 Python 中调用 Hunspell 进行实时拼写校验需安装pyspellchecker轻量或hunspell原生绑定。后者更准但需系统级依赖# Ubuntu/Debian sudo apt-get install libhunspell-dev pip install hunspell # macOS (with Homebrew) brew install hunspell pip install hunspellimport hunspell # Initialize with provided dictionary files hobj hunspell.HunSpell( hunspell_en_US.dic, # 50k-word main dict hunspell_en_US.aff # affix rules for derivation ) # Test correction misspelled recieve suggestions hobj.suggest(misspelled) print(f{misspelled} → {suggestions}) # [receive] # Check if valid is_valid hobj.spell(beautiful) print(fbeautiful is valid: {is_valid}) # True # Stemming (root extraction) — critical for OCR post-processing stemmed hobj.stem(running) # returns list: [brun] print(fStem of running: {stemmed[0].decode()}) # run注意hobj.suggest()返回字节串列表需.decode()hobj.stem()对动词现在分词、过去式、复数名词均有效但对专有名词如GitHub返回空列表——这正是你需要的OCR 输出githup会被纠正为github而Github首字母大写不会被错误 stem。3.2 将 Hunspell 集成到 OCR 后处理 pipeline典型场景CRNN 输出 logits 后CTC 解码得到[h, e, l, l, o, blank, w, o, r, l, d]→hello world。但若模型置信度低可能输出helo wrld。此时用 Hunspell 做两级校验def ocr_postprocess_with_hunspell(raw_pred: str, hobj: hunspell.HunSpell) - str: words raw_pred.split() corrected [] for word in words: # Step 1: If already valid, keep it if hobj.spell(word): corrected.append(word) continue # Step 2: Try suggestions, pick highest Levenshtein similarity suggestions hobj.suggest(word) if not suggestions: corrected.append(word) # no suggestion, keep original continue # Use difflib for fast similarity (no external dep) from difflib import SequenceMatcher best_sugg max( suggestions, keylambda s: SequenceMatcher(None, word, s.decode()).ratio() ) corrected.append(best_sugg.decode()) return .join(corrected) # Example raw helo wrld corrected ocr_postprocess_with_hunspell(raw, hobj) print(f{raw} → {corrected}) # helo wrld → hello world3.3 词典定制向 50k Hunspell 添加领域词OCR 识别医疗报告时myocardial可能被误识为myocadial。Hunspell 支持运行时添加单词# Add domain-specific terms without modifying .dic file hobj.add(myocardial) # now myocadial → [myocardial] hobj.add(esophagus) # prevents correction to esophagous # Verify addition print(hobj.spell(myocardial)) # True提示hobj.add()仅内存生效重启 Python 进程即失效。生产环境应导出新词典echo myocardial custom_additions.dic然后合并cat hunspell_en_US.dic custom_additions.dic | sort -u final.dic4. 特征工程实操从预处理图像生成 CNN-RNN 兼容输入深度学习模型如 CRNN、ASTER要求输入为(batch, channel, height, width)而标签需为torch.LongTensor形式的 token ID 序列。本资源的预处理图像和清洗文本需经以下特征工程才能喂入模型。4.1 图像特征构建 PyTorch DataLoader 的标准化流程使用torchvision.transforms时必须禁用ToTensor()的自动归一化因为 OCR 模型通常期望[0, 1]或[0, 255]输入而非[0, 1]归一化import torch from torch.utils.data import Dataset, DataLoader from torchvision import transforms class OCRCustomDataset(Dataset): def __init__(self, img_dir, label_dir, vocab, transformNone): self.img_paths sorted([f for f in os.listdir(img_dir) if f.endswith(.png)]) self.label_dir label_dir self.vocab vocab # instance of Vocab class (see below) self.transform transform or transforms.Compose([ transforms.Grayscale(), # ensure single channel transforms.ToTensor(), # converts to [0,1], float32 # NO Normalize() — CRNN expects raw pixel intensity ]) def __getitem__(self, idx): img_name self.img_paths[idx] img_path os.path.join(self.img_dir, img_name) label_path os.path.join(self.label_dir, img_name.replace(.png, .txt)) # Load image (32x128, already preprocessed) img Image.open(img_path).convert(L) # force grayscale img_tensor self.transform(img) # shape: (1, 32, 128) # Load and encode label with open(label_path, r) as f: label_text f.readline().strip() label_ids self.vocab.encode(label_text) # e.g., [12, 3, 44, 2] return img_tensor, torch.tensor(label_ids, dtypetorch.long) # Define vocabulary — critical: must match Hunspells tokenization! class Vocab: def __init__(self, word_listNone): self.char2idx {PAD: 0, SOS: 1, EOS: 2, UNK: 3} self.idx2char {0: PAD, 1: SOS, 2: EOS, 3: UNK} # Build from Hunspell dict: extract all unique chars if word_list is None: with open(hunspell_en_US.dic, r, encodingutf-8) as f: words [line.strip().split(/)[0] for line in f.readlines()[1:]] # skip header chars set(.join(words)) else: chars set(.join(word_list)) for i, ch in enumerate(sorted(chars), start4): self.char2idx[ch] i self.idx2char[i] ch def encode(self, text: str) - list: return [self.char2idx.get(c, self.char2idx[UNK]) for c in text] def decode(self, ids: list) - str: return .join([self.idx2char.get(i, UNK) for i in ids]) vocab Vocab() # builds char vocab from Hunspell dict dataset OCRCustomDataset(IC03/images/, IC03/labels/, vocab) loader DataLoader(dataset, batch_size8, shuffleTrue)4.2 标签特征字符级 vs 单词级编码的抉择依据IC03/IC13 是 word-level 标注每图一词IIIT5K/SVT 是 multi-word每图多词。本资源默认采用字符级编码character-level原因有三长度可控CRNN 输入固定宽 128输出序列长 ≤ 32字符级天然匹配Hunspell 兼容词典校验在字符粒度上更鲁棒helo→hello比helo wrld→hello world更易对齐少样本友好IC03 仅 250 测试词字符集仅 62 类a-z, A-Z, 0-9远小于单词级 250 类。# Character-level vocab size check print(fTotal chars in Hunspell dict: {len(vocab.char2idx)}) # typically 97–102 # Includes: a-z (26), A-Z (26), 0-9 (10), hyphen, apostrophe, space → ~65 base # Plus accented chars from normalization → ~97 total4.3 验证预处理一致性三步交叉检查法拿到资源后务必执行以下验证避免“以为对齐实则错位”检查项命令/代码预期输出失败含义图像尺寸identify -format %wx%h\n IC03/images/001_1.png128x32预处理未执行或 resize 错误标签纯净度head -n1 IC03/labels/001_1.txt | sed s/[^a-zA-Z0-9-]//g输出与原文件相同标签含未清洗标点词典加载hunspell -d hunspell_en_US -l recievereceive.aff文件路径错误或损坏# One-liner validation for all IC03 images find IC03/images/ -name *.png | head -5 | xargs -I{} identify -format %f %wx%h\n {} # Should print five lines like: 001_1.png 128x325. 进阶技巧用预处理数据集快速构建 baseline CRNN 模型有了对齐的数据和词典下一步是验证 pipeline 是否真正 work。这里提供一个极简 CRNN baselinePyTorch仅 120 行能在 10 分钟内跑通 IC03 训练 loop并输出 CERCharacter Error Rate。5.1 构建轻量 CRNN 模型CNN LSTM CTCimport torch.nn as nn import torch.nn.functional as F class CRNN(nn.Module): def __init__(self, n_classes97, hidden_size256): super().__init__() # CNN backbone: 32x128 → 1x32x32 (height reduced to 1, width32) self.cnn nn.Sequential( nn.Conv2d(1, 64, 3, 1, 1), nn.ReLU(), nn.MaxPool2d(2, 2), nn.Conv2d(64, 128, 3, 1, 1), nn.ReLU(), nn.MaxPool2d(2, 2), nn.Conv2d(128, 256, 3, 1, 1), nn.BatchNorm2d(256), nn.ReLU(), nn.Conv2d(256, 256, 3, 1, 1), nn.ReLU(), nn.MaxPool2d((2, 2), (2, 1), (0, 1)), nn.Conv2d(256, 512, 3, 1, 1), nn.BatchNorm2d(512), nn.ReLU(), nn.Conv2d(512, 512, 3, 1, 1), nn.ReLU(), nn.MaxPool2d((2, 2), (2, 1), (0, 1)), nn.Conv2d(512, 512, 2, 1, 0), nn.BatchNorm2d(512), nn.ReLU(), ) self.linear nn.Linear(512, hidden_size) self.lstm nn.LSTM(hidden_size, hidden_size, 2, bidirectionalTrue, batch_firstTrue) self.classifier nn.Linear(hidden_size * 2, n_classes) def forward(self, x): # x: (B, 1, 32, 128) x self.cnn(x) # (B, 512, 1, 32) → squeeze height x x.squeeze(2) # (B, 512, 32) x x.permute(0, 2, 1) # (B, 32, 512) x F.relu(self.linear(x)) # (B, 32, 256) x, _ self.lstm(x) # (B, 32, 512) x self.classifier(x) # (B, 32, n_classes) return x.log_softmax(2) # required for CTC loss model CRNN(n_classeslen(vocab.char2idx)).cuda() criterion nn.CTCLoss(blankvocab.char2idx[PAD], zero_infinityTrue) optimizer torch.optim.Adam(model.parameters(), lr1e-3)5.2 CER 计算函数比 accuracy 更敏感的评估指标def compute_cer(pred_ids, target_ids): Character Error Rate: (SDI)/N where Ntotal chars in target pred_str vocab.decode(pred_ids.tolist()) target_str vocab.decode(target_ids.tolist()) # Dynamic programming edit distance m, n len(pred_str), len(target_str) dp [[0] * (n 1) for _ in range(m 1)] for i in range(m 1): dp[i][0] i for j in range(n 1): dp[0][j] j for i in range(1, m 1): for j in range(1, n 1): if pred_str[i-1] target_str[j-1]: dp[i][j] dp[i-1][j-1] else: dp[i][j] min( dp[i-1][j] 1, # deletion dp[i][j-1] 1, # insertion dp[i-1][j-1] 1 # substitution ) return dp[m][n] / len(target_str) if target_str else 0 # Usage in training loop model.train() for imgs, targets in loader: imgs, targets imgs.cuda(), targets.cuda() logits model(imgs) # (B, T, C) input_lengths torch.full((imgs.size(0),), logits.size(1), dtypetorch.long) target_lengths torch.full((imgs.size(0),), targets.size(1), dtypetorch.long) loss criterion(logits.permute(1, 0, 2), targets, input_lengths, target_lengths) loss.backward() optimizer.step() optimizer.zero_grad() # Decode first sample pred_ids logits[0].argmax(1) cer compute_cer(pred_ids, targets[0]) print(fBatch CER: {cer:.3f})5.3 快速启动命令5 分钟验证 pipeline 完整性# 1. Install minimal deps pip install torch torchvision opencv-python hunspell Pillow # 2. Unzip and enter dir unzip 经过预处理的IC03 IC13 IIIT5K SVT数据集和50k-words Hunspell词典.zip cd preprocessed_ocr_data # 3. Run validation script (checks image dims, label format, dict load) python -c import os, cv2, hunspell h hunspell.HunSpell(hunspell_en_US.dic, hunspell_en_US.aff) assert h.spell(hello), Hunspell failed img cv2.imread(IC03/images/001_1.png, cv2.IMREAD_GRAYSCALE) assert img.shape (32, 128), fWrong shape: {img.shape} print(✅ All checks passed.) # 4. Launch training (IC03 subset, 1 epoch) python train_crnn.py --dataset IC03 --epochs 1 --batch_size 16关键技巧首次运行时将--dataset设为IC03最小数据集--epochs 1观察 loss 是否下降、CER 是否 0.3。若 loss 不降立即检查Vocab是否正确加载了 Hunspell 字符集——这是 80% 初学者卡点。本文还有配套的精品资源点击获取