HeadlinesPlugin < TWiki

TWiki Web>HeadlinesPlugin (2018-07-14, TWikiContributor) (raw view)
---+!! Headlines Plugin
<!--
   Contributions to this plugin are appreciated. Please update the plugin page at
   http://twiki.org/cgi-bin/view/Plugins/HeadlinesPlugin or provide feedback at
   http://twiki.org/cgi-bin/view/Plugins/HeadlinesPluginDev.
   If you are a TWiki contributor please update the plugin in the SVN repository.
-->
<sticky>
<div class="twikiTocFloat">
%TOC{ title="Page contents" depth="2" }%
</div>
</sticky>
%SHORTDESCRIPTION%

---++ Description

This plugin displays RSS and ATOM feeds from news sites. Use it to build news portals that show headline news.

__Notes:__

   * [[Wikipedia:RSS][RSS]] (Rich Site Summary) is an XML standard used to publish frequently updated works such as blog entries, news headlines and audio.
   * [[Wikipedia:Atom_%28standard%29][Atom]] is an XML syndication format used for web feeds.
   * Syndic8.com at http://www.syndic8.com/ lists many RSS and ATOM feeds.

#SyntaxRules
---++ Syntax Rules

=%<nop>HEADLINES{"..."}%=

| *Parameter* | *Explanation* | *Default* |
| ="..."= | Source of RSS or ATOM feed; this can be an url (starting with http) or a <nop>web.topic location for internal feeds | None; is required |
| =href="..."= | (Alternative to above) | N/A |
| =refresh="60"= | Refresh rate in minutes for caching feed; ="0"= for no caching | Global =REFRESH= setting |
| =limit="12"= | Maximum number of items shown | Global =LIMIT= setting |
| =header="..."= | Header. May include these variables: %BR% \
             - =$channeltitle=, =$title=: title of channel (channel.title) %BR% \
             - =$channellink=, =$link=: link of channel (channel.link) %BR% \
             - =$channeldescription=, =$description=: description (channel.description) %BR% \
             - =$channeldate=, =$date=: publication date of the channel (channel.pubDate) %BR% \
             - =$rights=: copyrights of the channel (channel.copyright) %BR% \
             - =$imagetitle=: title text for site (image.title) %BR% \
             - =$imagelink=: link for site (image.link) %BR% \
             - =$imageurl=: URL of image (image.url) %BR% \
             - =$imagedescription=: description of image (image.description) \
             | Global =HEADER= setting |
| =format="..."= | Format of one item. May include these variables: %BR% \
             - =$title=: news item title (item.title) %BR% \
             - =$link=: news item link (item.link)  %BR% \
             - =$description=: news item description (item.description) %BR% \
             - =$date=: the publication date (item.pubDate, item.date) %BR% \
             - =$category=: the article category (item.category) \
             | Global =FORMAT= setting |
| =newline="$br"= | Convert newlines in feed; ="$br"= becomes =&lt;br /&gt;= tag | ="$n"= (newline) |
| =filter="..."= | Filter out content from feed; for example to delete an encoded break tag specify ="&amp;lt;br&amp;gt;"= | =""= |
| =touch="..."= | Touch (edit/save) topics if the feed has updates. Specify a comma-space delimited list of !TopicNames or !Web.TopicNames, such as ="%<nop>TOPIC%, !NewsLetter"=. Useful to send out newsletter using MailerContrib, showing new feeds since last newsletter. To update feeds, visit topics with feeds in regular intervals (using cron with wget or the like). | N/A |

The =header= and =format= parameters might also use variables rendering the =dc=, =image= and =content= namespace information. Note, that only bits of interest have been implemented so far and those namespaces might not be implemented fully yet.

---+++ Rendering the =dc= namespace

The following variables are extracting the =dc= namespace info, that could be used in =header= and =format=. Note that some of the variables are already used above. This is done by purpose to use different feeds with the same formatting parameters. If there's a conflict the non-dc tags have higher precedence, i.e. a =&lt;title&gt; content &lt;/title&gt;= is preferred over =&lt;dc:title&gt; content &lt;/dc:title&gt;=.
   * =$title=: channel/article title (dc:title)
   * =$creator=: channel creator (dc:creator)
   * =$subject=: subject text; this will also add an image according to the subject hash
               list, see [[#Subject_Images][above]] (dc:subject)
   * =$description=: ... (dc:description)
   * =$publisher=: the channel/article publisher (dc:publisher)
   * =$contributor=: ... (dc:contributor)
   * =$date=: ... (dc:date)
   * =$type=: ... (dc:type)
   * =$format=: ... (dc:format)
   * =$identifier=: ... (dc:identifier)
   * =$source=: ... (dc:source)
   * =$language=: ... (dc:language)
   * =$relation=: ... (dc:relation)
   * =$coverage=: ... (dc: coverage)
   * =$rights=: ... (dc: rights)

---+++ Rendering the =image= namespace

An =image:item= is converted into an &lt;img&gt; tag using the following mappings:
   * =src=: image url (rdf:about attribute of the image.item tag)
   * =alt=: image title (title)
   * =width=: image width (image:width)
   * =height=: image height image:height)

---+++ Rendering the =content= namespace

The variable =$content= is referring to the =&lt;content:encoding&gt; content &lt;/content:encoding&gt;=.

---++ Examples

---+++ Slashdot News

Write
<verbatim>
%HEADLINES{ "http://slashdot.org/slashdot.rdf" 
  header="*[[$link][$title]]:* $description" 
  format="$t* [[$link][$title]]"
  limit="4"
}%
</verbatim>  
to get the latest Slashdot news as a bullet list format:

%HEADLINES{ "http://slashdot.org/slashdot.rdf" header="---+!! [[$link][$title]]$n $description" format="$t* [[$link][$title]]" limit="4" }%

---++ Plugin Settings

Plugin settings are stored as preferences settings. Do __not__ change the settings here, they are here only for illustration purposes showing the default values. Define the settings in [[%LOCALSITEPREFS%]]. For example, to customize the =HEADLINESPLUGIN_USERAGENTNAME= setting, add a =* Set HEADLINESPLUGIN_USERAGENTNAME = ...= bullet in [[%LOCALSITEPREFS%]].

   * Refresh rate in minutes for cached feeds. Set to =0= to disable caching:
      * Set HEADLINESPLUGIN_REFRESH = 60

   * Maximum number of items shown:
      * Set HEADLINESPLUGIN_LIMIT = 100

   * Use LWP::UserAgent if set to =1=, or fallback to TWiki's internal =getUrl()= method if set to =0=:
      * Set HEADLINESPLUGIN_USELWPUSERAGENT = 1

   * Timeout fetching a feed using the LWP::UserAgent:
      * Set HEADLINESPLUGIN_USERAGENTTIMEOUT = 20

<noautolink>
   * Name of user agent:
      * Set HEADLINESPLUGIN_USERAGENTNAME = TWikiHeadlinesPlugin/2011-07-08
</noautolink>

   * Default header: (variables are explained in the syntax rules)
<verbatim>
      * Set HEADLINESPLUGIN_HEADER = <div class="headlinesChannel"><div class="headlinesLogo"><img src="$imageurl" alt="$imagetitle" border="0" />%BR%</div><div class="headlinesTitle">$n---+!! <a href="$link">$title</a></div><div class="headlinesDate">$date</div><div class="headlinesDescription">$description</div><div class="headlinesRight">$rights</div></div>
</verbatim>

   * Default format of one item: (variables are explained in the syntax rules)
<verbatim>
      * Set HEADLINESPLUGIN_FORMAT = <div class="headlinesArticle"><div class="headlinesTitle"><a href="$link">$title</a></div>$n<span class="headlinesDate">$date</span> <span class="headlinesCreator"> $creator</span> <span class="headlinesSubject"> $subject </span>$n<div class="headlinesText"> $description</div></div>
</verbatim>

   * Allow HTML in header and format parameters of the HEADLINES variable. If set to =0=, HTML is *not* allowed those in parameters. This is to guard against Cross-Site Scripting (XSS) attacks. The HEADLINESPLUGIN_HEADER and HEADLINESPLUGIN_FORMAT plugin settings allow HTML regardless of this setting.
      * Set HEADLINESPLUGIN_ALLOWHTML = 0

   * Values taken from configure: (only supported if CPAN:LWP is installed)
      * =$TWiki::cfg{PROXY}{HOST}= - proxy host, such as ="proxy.example.com";=
      * =$TWiki::cfg{PROXY}{PORT}= - proxy port, such as ="8080";=
      * =$TWiki::cfg{PROXY}{SkipProxyForDomains}= - domains excluded from proxy, such as ="intra.example.com, bugs.example.com";=

---++ Style Sheets

The default HEADER and FORMAT settings use the following styles. See the [[%ATTACHURL%/style.css][style.css]] file defining the default CSS properties  (indentation illustrates enclosure).

   * =headlinesRss=: output of the HeadlinesPlugin (div)
      * =headlinesChannel=: channel header (div)
         * =headlinesLogo=: channel logo (div)
         * =headlinesTitle=: channel title (div)
         * =headlinesDate=: channel date (div)
         * =headlinesDescription=: channel description (div)
         * =headlinesRight=: channel copyright (div)
      * =headlinesArticle=: one news item (div)
         * =headlinesTitle=: article title (div)
         * =headlinesDate=: article date (span)
         * =headlinesCreator=: author of article (span)
         * =headlinesSubject=: subect category of the article (span)
         * =headlinesText=: article text (div)

---++ Plugin Installation Instructions

This is a pre-installed plugin. TWiki administrators can upgrade the plugin as needed on the TWiki server.
%TWISTY{
 mode="div"
 showlink="Show details %ICONURL{toggleopen}% "
 hidelink="Hide details %ICONURL{toggleclose}% "
}%

__Note:__ You do not need to install anything on the browser to use this plugin. The following instructions are for the administrator who installs the plugin on the TWiki server.

   * For an __automated installation__, run the [[%SCRIPTURL{configure}%][configure]] script and follow "Find More Extensions" in the in the __Extensions__ section.

   * Or, follow these __manual installation__ steps:
      * Download the ZIP file from the Plugins home (see below).
      * Unzip ==%TOPIC%.zip== in your twiki installation directory. Content:
        | *File:* | *Description:* |
        | ==data/TWiki/%TOPIC%.txt== | Plugin topic |
        | ==pub/TWiki/%TOPIC%/style.css== | Default CSS |
        | ==lib/TWiki/Plugins/%TOPIC%.pm== | Plugin Perl module |
        | ==lib/TWiki/%TOPIC%/Core.pm== | Plugin core |
      * Set the ownership of the extracted directories and files to the webserver user.
      * Make sure the dependencies listed in the table below are resolved.
        <table border="1"><tr><th>Name</th><th>Version</th><th>Description</th></tr><tr><td align="left">Digest::MD5</td><td align="left">&gt;=2.33</td><td align="left">Required. Download from CPAN:Digest::MD5</td></tr><tr><td align="left">LWP::UserAgent</td><td align="left">&gt;=5.803</td><td align="left">Optional. Download from CPAN:LWP::UserAgent</td></tr></table>

   * Plugin __configuration and testing__:
      * Run the [[%SCRIPTURL{configure}%][configure]] script, enable the plugin in the __Plugins__ section
      * Configure the plugin: See plugin settings above.
      * Test if the installation was successful: See example above.

%ENDTWISTY%

---++ Plugin Info

   * One line description, shown in the TextFormattingRules topic:
      * Set SHORTDESCRIPTION = Show headline news in TWiki pages based on RSS and ATOM news feeds from external sites

%TABLE{ tablewidth="100%" columnwidths="170," }%
|  Plugin Author: | TWiki:Main.PeterThoeny, TWiki:Main.MichaelDaum |
|  Copyright: | &copy; 2002-2018 Peter Thoeny, [[http://twiki.org/][TWiki.org]];%BR% &copy; 2002-2018 TWiki:TWiki.TWikiContributor; %BR% &copy; 2005-2007 Michael Daum wikiring.de |
|  License: | GPL ([[http://www.gnu.org/copyleft/gpl.html][GNU General Public License]]) |
|  Plugin Version: | 2018-07-13 |
%TWISTY{
 mode="div"
 showlink="Show Change History %ICONURL{toggleopen}%"
 hidelink="Hide Change History %ICONURL{toggleclose}% "
}%
%TABLE{ tablewidth="100%" columnwidths="170," }%
|  2018-07-13: | TWikibug:Item7846: Sanitize parameters; add HEADLINESPLUGIN_ALLOWHTML setting; sanitize malicious scripts in feed; remove non-functional Wired feed example |
|  2018-07-05: | TWikibug:Item7841: Copyright update to 2018 |
|  2016-01-08: | TWikibug:Item7708: Copyright update to 2016 |
|  2015-11-06: | TWikibug:Item7697: New filter and newline parameters; fix Atom feed parser to support Google Alerts |
|  2015-07-28: | TWikibug:Item7678: HeadlinesPlugin not working well on Fast CGI and mod_perl |
|  2015-01-10: | TWikibug:Item7604: Switch from GPL v2 to v3 |
|  2013-11-18: | TWikibug:Item7385: Encode "[" and "]" to avoid problems with TWiki =[<nop>[...][...]]= links; code-refactor to use registerTagHandler and OO style. |
|  2013-02-16: | TWikibug:Item7091: Use TWISTY in installation instructions section and change history |
|  2013-01-29: | TWikibug:Item7131: HeadlinesPlugin doesn't match pattern in multiple lines. |
|  2012-12-10: | TWikibug:Item7070: HeadlinesPlugin doesn't support multiple disks. |
|  2012-11-13: | TWikibug:Item7016: Restore changes for earlier versions with LWP |
|  2012-11-13: | TWikibug:Item7020: Categorize TWiki Variable |
|  2012-10-31: | TWikibug:Item7016: Utilize &lt;guid&gt; value when it is available |
|  2012-09-05: | TWikibug:Item6837: Doc fixes |
|  2011-07-17: | TWikibug:Item6764: Add VarHEADLINES variable documentation; doc improvements; setting NO_PREFS_IN_TOPIC |
|  2011-07-08: | TWikibug:Item6725: Change global package variables from "use vars" to "our" |
|  2010-05-16: | TWikibug:Item6433: More doc improvements |
|  2010-04-25: | TWikibug:Item6433: Doc fix: Changing TWIKIWEB to SYSTEMWEB |
|  2010-02-27: | TWikibug:Item6313: Fixed bug in ATOM feed with =&lt;link ...&gt;&lt;/link&gt;= instead of =&lt;link ... /&gt;= -- Peter Thoeny |
|  2009-09-30: | fixed bug in lastBuildDate of feeds affecting touch parameter functionality -- Peter Thoeny |
|  2009-08-29: | added touch parameter -- Peter Thoeny |
|  12 Feb 2009: | {PROXY}{HOST} supports domain with and without protocol -- Peter Thoeny |
|  06 Feb 2009: | added {PROXY}{SkipProxyForDomains} configure setting, added USERAGENTNAME plugin setting -- Peter Thoeny |
|  11 Dec 2008: | added {PROXY}{HOST} and {PROXY}{PORT} configure settings -- Peter Thoeny |
|  13 Sep 2007: | fixed parsing of content:encoded |
|  23 Jul 2006: | improved atom parser; if a posting has no title default to 'Untitled' |
|  26 Apr 2006: | added lazy compilation |
|  10 Feb 2006: | packaged using the TWiki:Plugins/BuildContrib; minor fixes |
|  03 Feb 2006: | off-by-one: limit="n" returned n+1 articles; make FORMAT and HEADER format strings more robust |
|  23 Jan 2006: | released v2.00 |
|  05 Dec 2005: | internal feed urls must be absolute |
|  02 Dec 2005: | added <nop>web.topic shorthand for internal feeds |
|  29 Nov 2005: | fixed CDATA handling |
|  21 Nov 2005: | added ATOM support; extended RSS support; added dublin core support; added content support; optionally using LWP to fetch feeds to follow redirections; corrected CPAN dependencies ; recoding special chars from html integer to entity encoding to increase browser compatibility; added css support; use =getWorkArea()= if available |
|  11 May 2005: | TWiki:Main.WillNorris: added <nop>DevelopBranch compatability |
|  31 Oct 2004: | Fixed taint issue by TWiki:Main.AdrianWeiler; small performance improvement |
|  29 Oct 2004: | Fixed issue of external caching if mod_perl or <nop>SpeedyCGI is used |
|  02 Aug 2002: | Implemented caching of feeds, thanks to TWiki:Main/RobDuarte |
|  11 Jun 2002: | Initial version (V1.000) |
%ENDTWISTY%
%TABLE{ tablewidth="100%" columnwidths="170," }%
|  Perl Version: | 5.8 |
|  [[TWiki:Plugins/Benchmark][Plugin Benchmark]]: | %SYSTEMWEB%.GoodStyle 100%, %SYSTEMWEB%.FormattedSearch 99.5%, %TOPIC% 94% |
|  Plugin Home: | http://TWiki.org/cgi-bin/view/Plugins/%TOPIC% |
|  Feedback: | http://TWiki.org/cgi-bin/view/Plugins/%TOPIC%Dev |
|  Appraisal: | http://TWiki.org/cgi-bin/view/Plugins/%TOPIC%Appraisal |

__Related Topics:__ %SYSTEMWEB%.VarHEADLINES, %SYSTEMWEB%.TWikiPlugins, %SYSTEMWEB%.AdminDocumentationCategory, %SYSTEMWEB%.TWikiPreferences
Topic revision: r11 - 2018-07-14 - TWikiContributor
TWiki
Webs
LAPPD
Sandbox
TWiki
Copyright © 1999-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
Note: Please contribute updates to this topic on TWiki.org at TWiki:TWiki.HeadlinesPlugin.