Feed Creator 1.2: merge and filter feeds

The new version of our Feed Creator application is now available. Feed Creator started life as a tool letting you create feeds for sites which don’t offer their own.

With this update, it now also lets you combine existing feeds, filter feeds (e.g. only keep items with a certain keyword), and convert existing feeds to JSON. You can jump in and try it out now, or read on to find out more.

New endpoint to merge, filter and convert feeds

Since Yahoo announced it will be discontinuing Yahoo Pipes (no new Pipes can be created from 30 August 2015 and the service will be shut down on 30 September 2015), we’ve had many requests for a tool to allow feed merging and filtering. We decided to extend Feed Creator with a new endpoint (mergefeeds.php) to allow this. It’s no replacement for everything Yahoo Pipes offers, but if you used Pipes mainly for feed merging, simple filtering, or converting feeds to JSON, you might find this new tool covers those needs. (And because it’s free software, you don’t need to worry about us pulling the plug.)

Merging feeds

Let’s say we want a single, combined feed of articles by John Pilger, Chris Hedges and Glenn Greenwald. Here’s what we do:

  1. Visit the new merge feeds page

  2. Enter the feed URLs, one per line:

    https://firstlook.org/theintercept/staff/glenn-greenwald/feed/?rss
http://feeds.feedburner.com/Truthdig/ChrisHedges
http://feeds.feedburner.com/johnpilger
  3. Click ‘Create Feed’

This will produce a new feed URL embedding all the source feeds above in the url querystring parameter. Click the link below to see the result.

Note: We’re omitting item descriptions in our output using the description=0 parameter to keep things more compact for this example.

Filtering feeds

Now let’s say we’re not really interested in everything these authors publish, but only interested in what they have to say about Greece. We use one of the filtering parameters (text_contains) to keep only items which mention Greece:

Converting to JSON

Finally, let’s tell Feed Creator to produce JSON instead of RSS. We use the format parameter here:

Things to note

  • Although the endpoint is called mergefeeds.php you can apply filtering and feed conversion on single feeds too. No need to supply more than one feed.

  • By default, all the parameters you supply get embedded in the URL we generate. This presents a problem if you subscribe to the feed we generate in your news reading application and then want to change the parameters (e.g. to add a new feed, or change the filtering rules). If you want to make such changes without affecting the feed URL (to save yourself the hassle of updating the feed in your news reading application), you can add the parameters to the config file that comes with Feed Creator. You can create a set of parameters in this way and give the set a name. You then pass the set’s name using the saved parameter to tell Feed Creator where to find the additional parameters it should use. Doing things this way allows you to make changes to the parameters by editing the config file without changing the feed URL.

    Here’s what the entry will look like in the config file for the parameters above:

    $options->saved = array(
  'greece-articles' => array(
    'text_contains' => 'Greece',
    'url' => array(
      'https://firstlook.org/theintercept/staff/glenn-greenwald/feed/?rss',
      'http://feeds.feedburner.com/Truthdig/ChrisHedges',
      'http://feeds.feedburner.com/johnpilger'
    )
  )
);

    We can then pass saved=greece-articles in the querystring to tell Feed Creator to apply these parameters:

    http://createfeed.fivefilters.org/mergefeeds.php?saved=greece-articles

    (This feature, because it requires editing the config file, is only available for users who buy the self-hosted version and run it themselves.)

Hosting

The free hosted service we offer at the moment is only intended for light, personal use and for demo purposes. We might restrict it if there’s too much strain put on the server. If there’s enough demand, we might also offer a premium hosted service in the future.

For now, if you intend to do anything serious with the application, we suggest you host it yourself. If you don’t already have PHP hosting, please see our hosting page for suggestions. (That entry was created for our Full-Text RSS application, but applies equally to Feed Creator.)

Full changelog

  • New endpoint: mergefeeds.php – combine feeds, filter feeds, convert feeds to JSON (combine as you like)
  • New parameter: item_date – CSS selector to pick out item dates (extract.php endpoint)
  • New parameter: parser – You can now specify HTML5 parsing (extract.php endpoint)
  • New parameter: order – Pass order=reverse to have extracted items returned in reverse document order (extract.php endpoint)
  • Added proxy support – new config option: proxy_servers
  • Humble HTTP Agent library updated
  • HTML5-PHP library added for HTML5 parsing support
  • Other minor fixes

Available to buy

Feed Creator 1.2 is now available to buy. If you’re an existing customer, please wait for an email from us with an upgrade link.

We’re open to suggestions. If there’s something you think might be a nice addition to this application, let us know in the comments.