What XMLTV is, and what a parser has to survive in one

XMLTV is the format almost every television guide outside a broadcaster's own stack is delivered in. It predates the streaming industry entirely, it is plain XML, and its specification is a DTD rather than a schema with a validator anyone runs.

That last part is the whole story. XMLTV is not difficult because the format is complex. It is difficult because anyone can generate one, nobody checks it, and the files that reach a player have been through a chain of scrapers and converters with no test suite between them.

The format, in one screen#

There are 2 elements that matter.

<tv>
  <channel id="bbc1.uk">
    <display-name>BBC One</display-name>
    <display-name lang="en">BBC One HD</display-name>
    <icon src="https://example.com/bbc1.png" />
  </channel>

  <programme channel="bbc1.uk" start="20260831180000 +0100" stop="20260831190000 +0100">
    <title lang="en">The Six O'Clock News</title>
    <desc lang="en">National and international news.</desc>
    <category lang="en">News</category>
  </programme>
</tv>

A channel declares an identifier and one or more human-readable names. A programme points at a channel identifier, carries a start time and usually a stop time, and holds a title plus whatever optional metadata the generator felt like including.

That is the format. Everything else is optional, and in practice everything else is inconsistently present.

The identifier is the whole join#

channel id is a join key, and it is the only join key. That is the single most important thing about XMLTV.

Your playlist says a channel is tvg-id="bbc1.uk". Your guide file has a <channel id="bbc1.uk">. Those two strings have to match, exactly, or the channel has no schedule.

They are not derived from anything. There is no registry. There is no normalisation the format mandates. bbc1.uk, BBCOne.uk, BBC1.uk and bbc-one.uk are 4 different channels as far as any parser is concerned, and it is entirely normal for a provider's playlist to use one convention while the guide file they recommend uses another.

This is why "my guide is empty" is usually a matching problem rather than a fetch problem or a parse problem. Both files can be perfectly formed and completely useless together.

The display-name elements do not help with the join. They are for showing to people, they are frequently several per channel in several languages, and using them for matching is a heuristic rather than a rule.

Timestamps, and the reason so many guides look an hour out#

An XMLTV timestamp is a wall-clock string followed by an optional offset:

20260831180000 +0100    six in the evening, one hour ahead of UTC
20260831180000          six in the evening UTC, per the spec's default

The offset is optional. That one design choice generates more support questions than everything else in the format combined.

We got this wrong in an earlier draft of this article, which described a bare timestamp as unplaceable. It is not. The DTD is explicit that a timestamp with no timezone is to be read as UTC, so there is a defined default and a conforming parser follows it.

The problem is that a generator omitting the offset is usually not publishing UTC. It is publishing the local wall clock of wherever the guide was assembled, and relying on the reader to be in the same place. That is not what the specification says, and it is what a great many files do. A parser that follows the DTD is then correct and an hour or two wrong at the same time.

Our parser takes an assume-timezone from the caller for exactly this case, which turns the discrepancy into a setting rather than a silent decision. Both device ingest paths leave it null, so a bare timestamp is read as UTC and the machine's own timezone never enters the calculation.

The failure this produces is distinctive: an entire day shifted by a whole number of hours, with the programme durations all correct. If your schedule is right but consistently one or two hours off, why guide times are wrong is the article with the details.

A guide file is untrusted input#

Invisible to a user. Everything to a parser.

A guide file is XML from an arbitrary URL, and XML has a long history of parser attacks. So the reader is configured defensively before it reads a byte: document type definitions are ignored rather than processed, the external resolver is set to null so no entity can cause a network fetch, and expansion from entities is capped at zero characters.

Those three settings close the classic entity-expansion and external-entity problems. They are not hardening we might relax for a friendlier parser; a guide URL is exactly the shape of input those attacks target.

There is a matching rule on the server side, and it is stated to users rather than hidden: your devices download guide files directly, and our servers never fetch them for you. A server that fetched arbitrary user-supplied URLs would be a request-forgery engine, so it does not.

Eleven limits, and why each one exists#

Hardening the reader is not enough, because the dangerous input in practice is not malicious. It is a guide with 40 title elements.

depth                32       nesting
text                 64 KB    any single text node
description          4 KB     aggregate across repeats
title                1 KB     aggregate across repeats
credits              15       per programme
categories           6        per programme
alternate names      8        per channel
channel batch        512      rows written at a time
programme batch      2,000    rows written at a time
programme duration   6 hours  ceiling
retention            8 days forward, 1 day back

The title cap is the one with the most instructive reason. A programme may legally carry one title per language. A guide that emits 40 of them is not malformed, it is thorough, and without a cap all 40 get concatenated into a single stored title. The description cap is the same rule for the same reason.

The batch sizes are throughput rather than safety: a national guide is hundreds of thousands of programmes, and writing them one at a time is the difference between a guide that loads and a guide that appears to hang.

The retention window is the one that surprises people. Programmes outside it are dropped during the parse rather than stored and pruned afterwards. Forward defaults to 8 days and back to 1, clamped to 14 and 7. A provider publishing 14 days of schedule into a client configured for 8 is not losing data through a bug; it is being told that a player does not need next fortnight's schedule badly enough to store it.

That has a direct consequence for catch-up: catch-up entries are built from programme rows, so guide retention is a ceiling on how far back you can reach. What catch-up and timeshift are covers the interaction.

Eight ways a fetch fails#

Failures are classified into 8 kinds rather than reported as one error: no URL, not XML, network, size limit, decompression limit, security limit, cancelled, and unknown. Decompression limit and security limit are separate from network failure on purpose, because they mean genuinely different things: one is a compression bomb, one is a hostile document, and one is a bad connection.

The three faults that a fixture pack found#

Real guide files are worse than the format allows. We have numbers on that rather than an impression.

A fixture pack of 51 real-world guide files uncovered 3 distinct faults. Each one was capable of destroying an entire provider's guide over a single byte.

An unescaped ampersand in a programme title. XML requires &, plenty of generators write a bare &, and a strict parser stops at that point. Everything after it is gone.

A file declaring UTF-8 while carrying Latin-1 bytes. The header says one thing and the body is another, and the first accented character in a French film title is where it ends.

A stray control character in a description. Not legal XML at any encoding, and again fatal at the point it appears.

All 3 are now repaired rather than fatal, and the repairs are counted, so a file that needed patching says so instead of quietly appearing to work. The reason to publish this rather than to fix it silently is that it calibrates expectations: if you assume guide files are well formed, the failures make no sense.

Before the XML: working out what the file even is#

A guide URL ending .xml.gz is a hint, not a fact. Providers serve gzip from paths ending .xml, plain XML from paths ending .gz, and both from paths ending in nothing at all, and a content type header is no more reliable than the path.

So the container is decided by reading the first few bytes and comparing them against known magic numbers:

1F 8B                gzip
50 4B 03 04          zip
FD 37 7A 58 5A 00    xz
anything else        treat as uncompressed

Four outcomes, and the fourth is the important one: an unrecognised header is not an error, it is a file assumed to be plain XML. That is the right default, because the common case for an unrecognised header is a plain XMLTV document, and the wrong case fails cleanly at the parser a moment later rather than at the decompressor with something more confusing.

The one shape this cannot help with is a URL that answers with an HTML error page. It has no magic number, so it is treated as uncompressed, handed to the XML reader, and rejected as not XML. The failure is correct and it is one step removed from the cause, which is worth knowing when you are diagnosing a guide URL that used to work.

Two ceilings on decompression, and why one of them is a ratio#

A compressed guide is an untrusted archive, and an archive that expands to fill a disk is an old attack with a name. So decompression is bounded twice rather than once.

The absolute ceiling is 2 GiB of output. That is deliberately generous: national guides for several countries genuinely run to hundreds of megabytes uncompressed, and a limit that clipped a legitimate guide would be worse than the problem it prevents.

The second bound is the interesting one. Output is also capped at 400 times the compressed bytes actually received. A ratio guard catches the case the absolute ceiling misses: a 40 KB file expanding toward gigabytes is stopped at 16 MB, long before it approaches 2 GiB, because nothing about its size profile resembles a real guide.

XML compresses extremely well, so 400 to 1 is not a tight limit for honest data. It is the point past which a file has stopped looking like a schedule and started looking like a payload.

What identifier coverage actually looks like#

The identifier join is only useful if playlists carry identifiers, and coverage varies more than you would hope.

Measured over our own playlist corpus, 2,203 entries in total:

playlist_01.m3u    308 entries    308 with tvg-id (100%)    283 group-title     0 tvg-chno
playlist_02.m3u8 1,895 entries  1,392 with tvg-id (73.5%)  1,895 group-title  1,355 tvg-chno

Both are curated, directory-style playlists rather than exports from a provider panel, and that caveat matters more here than usual: curated lists are maintained by people who care about identifiers, and a panel export is generated by software that may not.

Two details in that table matter. The second file has 20 entries carrying an identifier attribute that is present and empty. That is a different state from an absent attribute, and a parser has to treat it as absent rather than as an identifier that happens to be the empty string, or all 20 of those channels join to each other.

And group coverage is 100 percent in the second file while identifier coverage is 73.5 percent. Grouping and guide matching are separate concerns carried by separate attributes, which is why a playlist can arrive perfectly organised into categories and still show no schedule at all.

Why display names cannot rescue a failed join#

The obvious next idea, when identifiers do not match, is to match on the human-readable name instead. That is a much harder problem than it looks.

A display name is free text, there may be several per channel in several languages, and providers decorate them. The same channel across two sources can carry a country prefix, a quality suffix, a separator that is a pipe in one list and a colon in another, and any amount of unicode ornamentation. None of that is malformed; it is how the field is used.

So name matching is a normalisation problem with a long tail, and a wrong match is worse than no match: a channel showing another channel's schedule is a bug a viewer will trust for several minutes before noticing. The identifier exists precisely so that this guesswork is unnecessary, and the right fix when it fails is to make the identifiers agree rather than to guess harder.

Which is the practical argument for preferring your provider's own XMLTV over a third-party one. Not because it is better data, but because their guide and their playlist come out of one system, so the identifiers match by construction.

Reading the numbers a parse hands back#

A parse reports more than success. It returns 7 fields, and they answer different questions:

Channels           how many channel declarations were kept
Programmes         how many programme rows were kept
Dropped            how many were discarded, mostly by the retention window
BytesCompressed    what arrived over the network
BytesUncompressed  what it expanded to
ParseMs            how long it took
CompletedAtUtc     when

That combination answers questions a boolean cannot. A guide with 200,000 programmes and 40,000 dropped is doing something different from one with 200,000 and 0. Several diagnoses fall straight out of combinations like that.

Compressed and uncompressed sizes that are nearly equal mean the file was not compressed, which for a URL ending .gz means you are being served something other than what the name promises.

A large programme count with a large dropped count is a provider publishing further ahead than your retention window keeps. That is normal, and it is the thing to change if catch-up is shallower than you expected.

A parse that finished in 40 milliseconds on a file that should be 60 MB did not read a guide. It read an error page, and the failure kind will say so.

A channel count in the thousands with a programme count of zero is the shape of a guide carrying channel declarations and no schedule, which some providers publish between generation runs.

What to take from this#

If your guide is empty, check the identifiers before anything else. That is the join, and it is where most of it goes wrong.

If your guide is present but shifted, check whether the file carries offsets.

If your guide is partial, check whether it was truncated in transit, which is a specific failure with a specific signature: a guide that stops halfway.

And when you can choose between a provider's own XMLTV and a third-party one, prefer the provider's, for the reason above. The guide is empty walks the diagnosis end to end.