Why Table Scraping Is Always Wrong the First Time
Extracting a table looks like a two-line job: grab table.rows, map each row's cells to text. It works on the table you wrote by hand and fails on approximately every table in production. The reason is merged cells:
<table>
<tr><th rowspan="2">Region</th><th colspan="2">2025</th><th colspan="2">2026</th></tr>
<tr><th>Q3</th><th>Q4</th><th>Q3</th><th>Q4</th></tr>
<tr><td>EMEA</td><td>12</td><td>15</td><td>18</td><td>21</td></tr>
</table>
Row 2 has five <th> elements in the markup but occupies five logical columns only because "2025" spans two of them. Row 3 has five cells and five values, but naively indexing them against row 2's position numbers puts "EMEA" under Q4/2025 — every column shifted. There is no per-row fix for this, because the semantics are two-dimensional: a cell's position depends on spans from previous rows.
The robust solution is a single up-front transformation: normalize the table into a dense occupancy grid — a matrix where every logical (row, column) position exists, merged cells appear once with their covered positions marked — and do all extraction, header detection, and export on the matrix. Rendering quirks (CSS, whitespace, layout) stop mattering the moment the grid is built. This article builds that grid, then solves the four problems that remain on real pages: multi-row headers, text extraction, nested tables, and CSV export.
The Occupancy Grid
Two invariants make the algorithm work, and every bug in home-made versions is a violation of one of them:
- Process cells left to right within each row, maintaining a current column cursor. Before placing a cell, advance the cursor past any positions already occupied by
rowspancells from earlier rows. - After placing a cell, mark its full rectangle as occupied: its own row and column plus every position to the right (
colspan) and below (rowspan). Future rows consult these marks to skip positions.
A Map<column, origin> per row is enough to carry occupancy forward — sparse, cheap, and it doesn't need to know the table's total width (which, with rowspans, you cannot know until you finish).
function normalizeTable(table) {
const rows = [...table.rows];
const grid = rows.map(() => []); // text at origin, null where covered
const origin = rows.map(() => []); // [r0, c0] that each position maps to
const carried = rows.map(() => new Map()); // col -> origin, occupied from above
for (let r = 0; r < rows.length; r++) {
let c = 0; // column cursor
for (const cell of rows[r].cells) {
while (carried[r].has(c)) c++; // invariant 1: skip rowspan holes
// IDL properties, not getAttribute: the DOM already clamps these
// per the HTML spec — colSpan to 1..1000, rowSpan to 0..65534 — so
// colspan="junk" and colspan="-3" arrive here as colSpan === 1.
const colspan = cell.colSpan;
let rowspan = cell.rowSpan;
// rowspan="0" is legal and means "to the end of my row group"
if (rowspan === 0) {
const section = cell.closest('thead, tbody, tfoot') || table;
const sectionRows = [...section.rows];
rowspan = sectionRows.length - sectionRows.indexOf(rows[r]);
}
rowspan = Math.min(rowspan, rows.length - r); // never spill past the table
const text = readCellText(cell);
for (let dr = 0; dr < rowspan; dr++) {
for (let dc = 0; dc < colspan; dc++) {
const rr = r + dr, cc = c + dc;
grid[rr][cc] = (dr === 0 && dc === 0) ? text : null;
origin[rr][cc] = [r, c]; // invariant 2: record every position
if (dr > 0) carried[rr].set(cc, [r, c]); // block this column below
}
}
c += colspan;
}
}
return { grid, origin, rowCount: rows.length };
}
Details that matter, each learned from a real page:
table.rowsis scoped correctly. It lists rows of this table across all row groups (thead/tbody/tfoot) in document order — and crucially, it does not include rows of a nested table. The one DOM detail that pays for itself here.- Use
cell.colSpan/cell.rowSpan, notgetAttribute. These IDL properties reflect the attributes through the spec's clamping rules, so malformed markup — and it is everywhere in hand-written HTML — can't produce a negative or absurd span. You inherit the browser's parsing work for free. rowspan="0"is not broken markup. It's valid HTML meaning "span to the end of the row group". Generators emit it for section-spanning labels. Handle it explicitly, or you'll ship a table where one cell shadows everything below it.- Clamp the rowspan to the remaining rows. A
rowspan="99"on the last row of a five-row table is a real thing (WYSIWYG editors produce it); without the clamp,carried[r + dr]writes into future-row maps thatgriddoesn't have, and the while-loop can walk off the end of the world. - The unguarded while loop is a hazard: if a malformed table ever carries occupancy into every column up to some huge index, the cursor walk is unbounded. Belt and braces: cap
cat the maximum occupied column + the widest colspan seen so far, which is a two-line change. It should never trigger; it will the day someone opens a table exported from a broken CMS.
Blank, Repeat, or Both: Resolving Merged Values
The grid stores each merged cell's text exactly once, at its origin, with null at covered positions. That's the right internal representation because it loses nothing — both common output conventions are derivable:
function toMatrix(ctx, fill = 'blank') {
return ctx.grid.map((row, r) => row.map((v, c) => {
if (v !== null) return v;
if (fill === 'blank') return ''; // spreadsheet 'merged cells' look
const [r0, c0] = ctx.origin[r][c];
return ctx.grid[r0][c0] ?? ''; // repeat the origin value
}));
}
Which mode to use is not arbitrary. Blank matches how Excel models merges and reads best for display. Repeat is what you want for data analysis — every row carrying "EMEA" in the Region column survives a GROUP BY, a pivot, or a filter, none of which know what a merge is. My default for CSV export is repeat, with blank available as an option, and I've never regretted making it a parameter instead of a decision.
Worth stating because it surprises people: origin is also the key to un-merging in the other direction. Given a target position, you can always answer "which cell visually covers this spot" — needed later for header flattening, and for hit-testing when the export feature offers "extract this column" on click.
Extracting Cell Text Without the Traps
textContent on a cell concatenates everything with no separators — <td><div>Product</div><div>Price</div></td> becomes ProductPrice. innerText gets this right by consulting layout — but it forces style and layout recalculation per cell, which on a thousand-row table is a measurable stall inside a content script. The middle path: walk the nodes and insert a space at every element that renders as a block, plus map <br> and images' alt text explicitly:
const BLOCK = new Set(['DIV','P','LI','UL','OL','SECTION','ARTICLE',
'H1','H2','H3','H4','H5','H6','PRE','BLOCKQUOTE']);
function readCellText(cell) {
const parts = [];
const walk = (node) => {
for (const child of node.childNodes) {
if (child.nodeType === Node.TEXT_NODE) {
parts.push(child.data);
} else if (child.nodeType === Node.ELEMENT_NODE) {
if (child.tagName === 'TABLE') continue; // nested table: handled elsewhere
if (child.tagName === 'BR') { parts.push(' '); continue; }
if (child.tagName === 'IMG') { parts.push(child.alt || ''); continue; }
const space = BLOCK.has(child.tagName);
if (space) parts.push(' ');
walk(child);
if (space) parts.push(' ');
}
}
};
walk(cell);
return parts.join('').replace(/\s+/g, ' ').trim();
}
Three decisions baked in. Nested tables are skipped, not flattened — their text belongs to the inner table's own grid, and silently concatenating it into a parent cell produces cells containing three unrelated rows' worth of text; handle the inner table as its own extraction (below). \s+ collapse also folds non-breaking spaces — JavaScript's \s includes \u00A0, which is exactly the character tables are full of ( is the universal way to keep an "empty" cell from collapsing, and it should export as empty, not as an invisible character). The block-tag set is a heuristic, honestly labeled as one. If your extraction fidelity needs to be perfect — legal tables, invoices — swap BLOCK.has(child.tagName) for getComputedStyle(child).display checks and pay the style-recalc cost. For the 95% case, the name list runs in microseconds and produces identical output.
Nested tables, in practice: walk cell.querySelectorAll('table'), normalize each with the same function, and decide representation by use case — a placeholder like [3×4 table] in the parent cell (my default, since it keeps the parent grid rectangular and the data reachable), or flatten the inner table into the parent with a separator row. What you should not do is pretend the nesting doesn't exist: the moment one layout table wraps the real content is the moment a flattened extraction goes from "data" to "haystack".
Multi-Row Headers, Flattened
Real report tables have grouped headers: "2025" spanning Q3 and Q4. Exported flat, the user loses the grouping; exported as two header rows, half of every downstream tool misreads it. The standard answer is a path per column: the column is "2025 / Q3".
Header detection first. Reliable signals in priority order: rows inside <thead>; rows where every cell is a <th>; a scope attribute on cells. In the wild, header rows are a prefix of the rows — so the pragmatic rule is: count leading rows that are entirely header cells, using thead as a tiebreaker:
function headerRowCount(table) {
const rows = [...table.rows];
let n = 0;
for (const row of rows) {
const inThead = row.closest('thead') !== null;
const allTh = [...row.cells].every(c => c.tagName === 'TH');
if (inThead || allTh) n++;
else break; // headers are a prefix, in practice
}
return n;
}
With the normalized grid in hand, flattening is a short walk — and it's where the origin map pays off, because a header cell with rowspan="2" under "2025" resolves to a single path element instead of repeating on every row:
function columnHeaderPath(ctx, col, headerRows) {
const parts = [];
for (let r = 0; r < headerRows; r++) {
const [r0, c0] = ctx.origin[r][col];
const value = ctx.grid[r0][c0];
if (value && value !== parts[parts.length - 1]) parts.push(value);
}
return parts.join(' / '); // '2025 / Q3'
}
Two refinements from use. When the group header equals the sub-header ("Q3 / Q3" from a merged first row), the dedupe above drops the repeat. And when the header path is empty for some column — no header row covers it, which happens with off-by-one markup — fall back to the column letter (A, B, …) rather than emitting an empty string that turns into a nameless CSV column.
CSV Export That Opens Correctly
Writing CSV looks trivial and has exactly three failure modes, all with the same symptom: someone opens the file and says "it's wrong".
function toCSV(matrix) {
const esc = (v) => {
let s = String(v ?? '');
// CSV injection: =, +, @, TAB, CR as the first character execute as
// formulas in Excel / Sheets. '-' only if not a plain negative number.
if (/^[=+@\t\r]/.test(s) || /^-(?!\d+(\.\d+)?$)/.test(s)) s = "'" + s;
return /[",\r\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s;
};
const bom = '\uFEFF'; // without this, Excel mangles non-ASCII as mojibake
return bom + matrix.map(row => row.map(esc).join(',')).join('\r\n');
}
Quoting (RFC 4180): fields containing comma, quote, or newline get wrapped in quotes, and embedded quotes double. The classic bug is quoting only for commas, then discovering a cell with a line break splits the row on import. Formula injection: a cell beginning with = is a formula — in a file exported from a hostile page, that's an attack that fires on the downloader's machine (=HYPERLINK(...), DDE, and worse). Prefix with a single quote — Excel's text marker, invisible in the cell — and skip the prefix for ordinary negative numbers so numeric columns stay numeric. The BOM: UTF-8 without a byte-order mark is correctly parsed by everything except the program most users will open it in. One \uFEFF at the start costs 3 bytes and removes an entire support category.
File download has one extension-specific wrinkle worth knowing before you build it: in an MV3 service worker, URL.createObjectURL does not exist. Options: a data: URL (fine for small exports, memory-heavy for a 50 MB CSV), or — better — create the blob and object URL in an offscreen document and pass the URL to chrome.downloads.download from the service worker, which is the same offscreen pattern that keeps long jobs alive and is the reason heavy export logic belongs there in the first place.
Production Checklist
- Normalize once into a grid + origin map; never index
row.cellsdirectly for anything positional. - Advance the column cursor past carried occupancy before placing each cell; record origin for every merged position.
- Use
cell.colSpan/cell.rowSpan(IDL) instead of raw attributes; handlerowSpan === 0as section-end semantics; clamp to remaining rows. - Cap the cursor-skip loop defensively against malformed occupancy.
- Make blank-vs-repeat an explicit fill mode; default to repeat for analysis exports.
- Extract text with a node walker: block-boundary spaces,
<br>, alt text, nested-table skip, whitespace and nbsp collapse. - Treat nested tables as sub-problems, with a placeholder or recursive extraction — never silent concatenation.
- Detect header rows as an all-th/thead prefix; flatten grouped headers into "Group / Column" paths through the origin map.
- Emit RFC 4180 CSV: quote comma/quote/newline, double embedded quotes, \r\n line endings, UTF-8 BOM.
- Guard formula injection on export; exempt plain negative numbers.
- Create blob URLs in an offscreen document; hand downloads to
chrome.downloads.downloadfrom the service worker. - Test against the ugly corpus: rowspan=0, colspan=0, nested tables, missing tbody, 1000-row tables, and a table inside the first table.
References
- HTML Standard — Tables
- HTMLTableCellElement.colSpan — MDN
- HTMLTableCellElement.rowSpan — MDN
- RFC 4180 — CSV Format
- CSV Injection — OWASP