Mirrors, P2P, and easier file distribution

["anthony l. bryan" <albryan@comcast.net>]

Hi everyone,

I've been working on something tentatively called Metalink that stores the
many ways to get a certain file (ftp, http, P2P) in a sort of universal
download container for easy one click file distribution.

This is mainly useful for large multi-located files and for multi-threaded
(segmented/accelerated) downloads with download managers. I think it could
be good for free software/open source community files that are stored on
lots of mirrors. Things like Linux distribution ISOs & OpenOffice.org.

A few download managers plan to implement this. I'm no XML expert, so I'd
appreciate any comments.

thank you...
ant

-----------------------------------------------

http://home.comcast.net/~albryan/metalink/

What is a Metalink?

Metalinks bundle the various ways (FTP/HTTP/P2P) to acquire a file into one
format. This can increase download performance, reliability, usability, &
efficiency. A Metalink is a Multi Method Metalinker (MMM or M3talinker)
document. MMM is a dialect of XML and MMM files must conform to the XML 1.0
specification. MMM might also look like WWW mirrored.

Metalinks (MMM .metalink files) use XML to catalog the various ways (P2P,
FTP, HTTP) to get a file/URL. I see this as especially useful for
multi-threaded (segmented) downloads of large files that are mirrored (&
integrated with P2P downloads) & also to list alternate addresses for web
pages (for instance during a slashdotting).

Benefits over traditional methods

    * Standard unified format that collects links for automatic accelerated
(segmented) downloads from multiple sources.
    * Automatic load balancing distributes traffic so individual servers are
under less strain.
    * No Single Point of Failure (SPOF) like FTP or HTTP URLs.
    * Links already collected for segmented (accelerated) downloading from
multiple sources and they are used automatically.
    * No long confusing list of possibly outdated Mirrors.
    * Simplifies download options (language, Operating System, location,
etc).
    * Combines Peer-to-peer (shared bandwidth) with FTP and HTTP.
    * Can finish BitTorrent download even if no full seeds are shared.
    * No separate MD5/SHA-1 file & manual process for verification.

Automatic use: click on a link for a 3 gigabyte Linux distribution ISO & all
the mirror locations are in your download manager & downloading from
BitTorrent peers & grabbing a portion of the file from each mirror, maxing
out your downstream bandwidth. Alternate links can be used as a fallback to
finish downloading a .torrent that is no longer seeded 100%. Once you have a
100% seed, you will keep sharing and help the other people's .torrent
downloads finish.

And take a look at the OpenOffice.org download page. It has three choices
for the user. Language, 41 options to choose from, operating system, 8
options, Mirror location, 71 options. And three types of P2P (BitTorrent,
Magnet, & ED2K). Language, operating system, and location can be set in the
client. 

There is a section for MD5/SHA-1 hashes and PGP signatures for verification.
MD5/SHA-1 hashes are unique to each file & could be used to search for
alternate locations (instead of just size & filename). So if all links went
bad (or slow), the client could use a Web/P2P search engine to find
mirrors/alternate locations files for the SHA-1 sum it is looking for.

An example, this time for OpenOffice.org 2.0 with a BitTorrent .torrent file
embedded, magnet, & ed2k links. The final file would include all OS/language
combinations, and for single click downloads, the download client will
automatically grab the correct version according to options you have
specified.

<?xml version="1.0" encoding="UTF-8"?>
<metalink version="1.9" xmlns="http://home.comcast.net/~albryan/metalink/";
  origin="http://www.openoffice.org/mdc/ooo2.metalink";
  type="static" pubdate="2005-11-08-22:07:22"
refreshdate="2005-11-09-03:24:18">

<files>
  <file name="OpenOffice.org_2.0.0_Win32Intel_install.exe">
    <identity>OpenOffice.org</identity>
    <version>2.0.0</version>
    <description>OpenOffice.org 2.0.0 - free office suite</description>
    <tags>OpenOffice.org, office suite, OpenDocument, open source</tags>
    <language>en-US</language>
    <os>Windows32</os>    
    <size>78597807</size>
    <verification>
      <md5>ec048f0e09431196959c6ed7bed22cae</md5>
    </verification>    
    <publisher>
      <name>OpenOffice.org</name>
      <url>http://www.openoffice.org/</url>
    </publisher>
    <license>
      <name>LGPL</name>
      <url>http://www.gnu.org/copyleft/lesser.html</url>
    </license>    
    <copyright>Copyright 2000-2005 Sun Microsystems Inc.</copyright>
    <resources>
      <magnet>
        <url>
        magnet:?xt=urn:bitprint:EVYF3SWQSV4CCSK46VIT7QHM2OEDEHYL.V
 
TYBIIBLAEUMSVPQCDUAK7DXJJH2MEK7DZHTVXY&xt=urn:ed2k:4b4fb5116c0a4b261fb7e
        d0d721178da&dn=OpenOffice.org_2.0.0_Win32Intel_install.exe
	</url>
	<preference>90</preference>
      </magnet>
      <ed2k>
        <url>
        ed2k://|file|OpenOffice.org_2.0.0_Win32Intel_install.exe|
 
78597807|4B4FB5116C0A4B261FB7ED0D721178DA|h=UD55DQJEU7EYEWL5QIYLJINNQAIQ
        H5P7|/
        </url>
	<preference>90</preference>        
      </ed2k>
      <bittorrent>
	<torrent encoding="base64">
	
ZDg6YW5ub3VuY2U0NTpodHRwOi8vYm9yZnQuc3R1ZGVudC51dHdlbnRlLm5sOjY5NjkvYW5u
	
b3VuY2UxMzpjcmVhdGlvbiBkYXRlaTExMjY3Njg5MzdlNDppbmZvZDY6bGVuZ3RoaTY3NDIz
	
MTQ4ZTQ6bmFtZTMyOk9Pb18xLjEuNV9XaW4zMkludGVsX2luc3RhbGwuemlwMTI6cGllY2Ug
	
bGVuZ3RoaTI2MjE0NGU2OnBpZWNlczUxNjA6fKscPdm2FUu3r/I9s0dCoSXwzKdgMqd/0Rjv
	
SQCYp0QAkXNPYq2SVapoq/A/H5OZxFJvg5O2Ej0KidtQoiuTik3nTVDa+Xh9x0KD1HSnwCsQ
	
I/23xsnITV9xIN2vRUJiRD3HE4sY7Syw2QgaNmpIUJIFiq7gXqxnZrqlxR6nkC6p1hqiXUxv
	...TRIMMED TO SAVE SPACE...
	
3nBjypUCJOsy/NzaYnnXJx8LL7Z1PRUCYYlP4P0pZ9DQQbmbAY9VeXX1W/1rI9k66DikYI3f
	
rQVEqy8e2RmpN1uGlDEBb4G+KoGeVfHs9Lyvn/iHBmVJ7GCnAK1HPFXjocpe3rLSWWeNwVdo
	l0IOZWU=
        </torrent>
	<preference>100</preference>
      </bittorrent>
      <http>
	
<url>http://mirrors.isc.org/pub/openoffice/stable/2.0.0/OOo_2.0.0_Win32Intel
_install.exe</url>
	<location>US</location>
	<preference>80</preference>
      </http>
      <ftp>
	
<url>ftp://ftp.ussg.iu.edu/pub/openoffice/stable/2.0.0/OOo_2.0.0_Win32Intel_
install.exe</url>
	<location>US</location>
	<preference>20</preference>
      </ftp>
      <http>
	
<url>http://mirrors.ibiblio.org/pub/mirrors/openoffice/stable/2.0.0/OOo_2.0.
0_Win32Intel_install.exe</url>
	<location>US</location>
	<preference>20</preference>
      </http>
      <ftp>
	
<url>ftp://openofficeorg.secsup.org/pub/software/openoffice/stable/2.0.0/OOo
_2.0.0_Win32Intel_install.exe</url>
	<location>US</location>
	<preference>40</preference>
      </ftp>
    </resources>
  </file>
</files> 

</metalink>