Running Telly: Filtering

Filtering in 1.0 and 1.1 involves creating a regular expression [or "regex"] that will include [or exclude in 1.0] entries from the input M3U to get it below 420 channels.

If your requirements are too complex to be expressed in a telly filter, there are other ways to "filter" the channel list from your provider. Some are discussed on the channel number limit page.

An M3U is just a test file listing channels. Each channel entry looks like this:

#EXTINF:-1 tvg-id="channel.id" tvg-name="channel.name" tvg-logo="channel.logo.url" group-title="group",channel.name
http://stream.url/999.ts

An actual example:

#EXTINF:-1 tvg-id="sky1.uk" tvg-name="UK SKY 1" tvg-logo="http://picon.helixhosting.ninja/25.png" group-title="UK ENTERTAINMENT",UK SKY 1
http://irislinks.net:83/live/REDACTED/REDACTED/21074.ts

Of course, all these fields will vary by provider, and even for a particular provider may change as channels come and go in the provider's lineup.

Teaching regular expressions [regex] is out of scope for this article, but in short a regular expression is a way of specifying a recipe for extracting information from some text.

For example, if you had a list of all your CDs, you might want to pull out just those CDs where the artist contains "Who" OR "Garbage" OR "Blues" (but only if Blues is NOT followed by "Traveler"). That sort of recipe is expressible via a regex.

The most common regex used in telly is something like: "USA" OR "HBO" OR "ESPN". The vertical bar means "OR", so this is expressed as "USA|HBO|ESPN".

The telly config file filter section consists of three fields:

  Filter = "Some|String|Here"
  FilterKey = "group-title"
  FilterRaw = false

Telly will apply the regex in Filter to the field in FilterKey, allowing just those channels into your final channel list.

If FilterRaw is true, the regex is applied to the entire line, so if you want to write filters that span tags you can do that. For example, say you want to include tvg-id="channel.id" tvg-name="English.channel.name" but not tvg-id="channel.id" tvg-name="French.channel.name". This provision is not often used.

Take the example line above:

#EXTINF:-1 tvg-id="sky1.uk" tvg-name="UK SKY 1" tvg-logo="http://picon.helixhosting.ninja/25.png" group-title="UK ENTERTAINMENT",UK SKY 1
http://irislinks.net:83/live/REDACTED/REDACTED/21074.ts

There are four fields that could go in the FilterKey:

tvg-id
tvg-name
tvg-logo
group-title

As a ridiculous example, this:

  Filter = "sky1.uk"
  FilterKey = "group-title"
  FilterRaw = false

finds 0 channels, because there are no channels with "sky1.uk" in the group-title tag, while this:

  Filter = "sky1.uk"
  FilterKey = "tvg-id"
  FilterRaw = false

matches the channel above, because it has "sky1.uk" in the tvg-id line, and this:

  Filter = "sky1.uk"
  FilterKey = "group-title"
  FilterRaw = true

Finds the channels that have "sky1.uk" anywhere in the M3U info line [because FilterRaw is turned on].

By default it's going to apply the filter to group-title, and that's typically what people want to filter on. They want all the movie channels, or the "UK ENTERTAINMENT" group, or that sort of thing. Using the FilterRaw option makes filtering considerably more complex, because telly's filtering doesn't allow "NOT". Using the RAW option can lead to many unintended matches that will be hard to account for.

The simplest way to get just a list of specific channels is to filter on tvg-name:

  Filter = "TLC|HBO|Some News Channel|Discovery Investigation"
  FilterKey = "tvg-name"
  FilterRaw = false

This can still lead to some extra channels; for example imagine you had these two channels and you wanted the non-FHD one:

The Movie Channel
The Movie Channel FHD

If you add "The Movie Channel" to your filter, you'll get both. They probably both have the same tvg-id, so you can't disambiguate them using that. Depending on your specific channel names, you may run into situations like this.

Telly uses the Go regex library, which is somewhat limited compared to some you may be used to [for example, I don't think my CD example above can be expressed in Go Regex]; however, for the purpose it typically works well. Just think in terms of coming up with a list of things you want to include rather than trying to leverage things you want to leave behind.

As mentioned above, that vertical bar means "OR", so the example above reads as "find channels where the group-title field contains 'Some' OR 'String' OR 'Here'".

Examples

I'm using the test scripts mentioned at the end of this article. Note that these test scripts, as of this writing, only support filtering on `group-title``.

First let's get the unfiltered list of groups from my provider:

➜  telly-test-harness ./groups.sh .
Using filter: .
      57
24/7 A-G TV SHOWS
24/7 ADULT CARTOONS
24/7 H-N TV SHOWS
24/7 KIDS
24/7 MOVIES
24/7 O-U TV SHOWS
24/7 V-Z TV SHOWS
ABC LOCAL NETWORKS
ADULTS ONLY
AFL, NRL & RUGBY PASS
AUSTRALIA/NZ
CANADA
CANADA TEST
CBS LOCAL NETWORKS
EPL / SPFL / iFOLLOW
ESPN NETWORKS
EX-YU
FOX LOCAL NETWORKS
FOX REGIONAL SPORTS
GERMAN
HQ VUE TEST
INTERNATIONAL SPORTS
IRISH
LATIN
MLB PACKAGE
MUSIC CHOICE CHANNELS
NASCAR CUP SERIES
NBA PACKAGE
NBC LOCAL NETWORKS
NCAA FOOTBALL
NFL PACKAGE
NHL PACKAGE
PAY PER VIEW
PREMIUM HD USA NETWORKS
PREMIUM SD USA NETWORKS
PREMIUM USA SPORTS NETWORKS
RADIO
RELIGIOUS NETWORKS
SWEDEN
SWITZERLAND / DENMARK
TEST-XYZ UK Sports
UK +1
UK DOCUMENTARIES
UK ENTERTAINMENT
UK KIDS NETWORKS
UK MOVIE NETWORKS
UK NEWS NETWORKS
UK SPORTS NETWORKS
UK VIP HD/FHD
UK VIP SD
USA & CANADA SPORTS
USA ENTERTAINMENT
USA KIDS NETWORKS
USA MOVIE NETWORKS
USA NEWS NETWORKS
VIP USA ENTERTAINMENT
VIP USA SPORTS NETWORKS

Say I want just UK channels:

  Filter = "UK"
  FilterKey = "group-title"
  FilterRaw = false

➜  telly-test-harness ./groups.sh UK
Using filter: UK
      10
TEST-XYZ UK Sports
UK +1
UK DOCUMENTARIES
UK ENTERTAINMENT
UK KIDS NETWORKS
UK MOVIE NETWORKS
UK NEWS NETWORKS
UK SPORTS NETWORKS
UK VIP HD/FHD
UK VIP SD

All of those groups made it through, because all their names contain "UK".

Maybe I want only those "NETWORKS" groups, so let's try "UK, then a space, then some characters, then a space, then NETWORK":

  Filter = "UK .* NETWORKS"
  FilterKey = "group-title"
  FilterRaw = false

➜  telly-test-harness ./groups.sh "UK .* NETWORKS"
Using filter: UK .* NETWORKS
       4
UK KIDS NETWORKS
UK MOVIE NETWORKS
UK NEWS NETWORKS
UK SPORTS NETWORKS

All those groups contain "UK ", then some characters, then " NETWORKS".

Taking off the UK brings a bunch more groups in:

  Filter = ".* NETWORKS"
  FilterKey = "group-title"
  FilterRaw = false

➜  telly-test-harness ./groups.sh ".* NETWORKS"
Using filter: .* NETWORKS
      17
ABC LOCAL NETWORKS
CBS LOCAL NETWORKS
ESPN NETWORKS
FOX LOCAL NETWORKS
NBC LOCAL NETWORKS
PREMIUM HD USA NETWORKS
PREMIUM SD USA NETWORKS
PREMIUM USA SPORTS NETWORKS
RELIGIOUS NETWORKS
UK KIDS NETWORKS
UK MOVIE NETWORKS
UK NEWS NETWORKS
UK SPORTS NETWORKS
USA KIDS NETWORKS
USA MOVIE NETWORKS
USA NEWS NETWORKS
VIP USA SPORTS NETWORKS

You can see that all those group-titles contain "some characters, then a space, then NETWORKS"

Say I want to include these groups:

ABC LOCAL NETWORKS
CBS LOCAL NETWORKS
ESPN NETWORKS
FOX LOCAL NETWORKS
NBC LOCAL NETWORKS
PREMIUM HD USA NETWORKS
UK DOCUMENTARIES
UK ENTERTAINMENT
UK MOVIE NETWORKS
USA ENTERTAINMENT
USA MOVIE NETWORKS
USA NEWS NETWORKS
VIP USA ENTERTAINMENT

One filter would be:

  Filter = "LOCAL NETWORKS|ESPN|PREMIUM HD |UK D| ENTERTAINMENT| MOVIE NETWORKS|USA NEWS NETWORKS|VIP USA ENTERTAINMENT"
  FilterKey = "group-title"
  FilterRaw = false

➜  telly-test-harness ./groups.sh "LOCAL NETWORKS|ESPN|PREMIUM HD |UK D| ENTERTAINMENT| MOVIE NETWORKS|USA NEWS NETWORKS|VIP USA ENTERTAINMENT"
Using filter: LOCAL NETWORKS|ESPN|PREMIUM HD |UK D| ENTERTAINMENT| MOVIE NETWORKS|USA NEWS NETWORKS|VIP USA ENTERTAINMENT
      13
ABC LOCAL NETWORKS
CBS LOCAL NETWORKS
ESPN NETWORKS
FOX LOCAL NETWORKS
NBC LOCAL NETWORKS
PREMIUM HD USA NETWORKS
UK DOCUMENTARIES
UK ENTERTAINMENT
UK MOVIE NETWORKS
USA ENTERTAINMENT
USA MOVIE NETWORKS
USA NEWS NETWORKS
VIP USA ENTERTAINMENT

What are those filter components doing?

LOCAL NETWORKS: we want all the local network groups, so this works

ESPN: There is only one group with "ESPN" in the name

PREMIUM HD : there are other groups with "PREMIUM", so this is qualified a bit more

UK D: only one group contains "UK, then space, then D"

[SPACE]ENTERTAINMENT: we want both the groups that contain "A space, then ENTERTAINMENT"

[SPACE]MOVIE NETWORKS: we want both the groups that contain "A space, then MOVIE NETWORKS"

USA NEWS NETWORKS: full group name

VIP USA ENTERTAINMENT: full group name

The expressions are all linked by the OR operator, since we want groups where any of those things match.

Now you can pick up anything

Hopefully this provides some guidance on how the filtering works.

ONE BIG CAVEAT: All these examples are using sed and egrep to apply the test regex; this may differ from Go's regex-processing, but it's easily accessible for quick and dirty testing like this. Your results may vary somewhat between sed, egrep and telly.

You can find those test scripts for preflighting filters here.