apt/doc/files.sgml

<!doctype debiandoc system>
<!-- -*- mode: sgml; mode: fold -*- -->
<book>
<title>APT Files</title>

<author>Jason Gunthorpe <email>jgg@debian.org</email></author>
<version>$Id: files.sgml,v 1.2 1998/07/12 02:11:09 jgg Exp $</version>

<abstract>
This document describes the complete implementation and format of the 
installed APT directory structure. It also serves as guide to how APT 
views the Debian archive.
</abstract>

<copyright>
Copyright &copy; Jason Gunthorpe, 1998.
<p>
"APT" and this document are free software; you can redistribute them and/or
modify them under the terms of the GNU General Public License as published
by the Free Software Foundation; either version 2 of the License, or (at your
option) any later version.

<p>
For more details, on Debian GNU/Linux systems, see the file
/usr/doc/copyright/GPL for the full license.
</copyright>

<toc sect>

<chapt>Introduction
<!-- General		                                               {{{ -->
<!-- ===================================================================== -->
<sect>General

<p>
This document serves two purposes. The first is to document the installed
directory structure and the format and purpose of each file. The second
purpose is to document how APT views the Debian archive and deals with 
multiple package files.

<p>
The var directory structure is as follows:
<example>
  /var/state/apt/
		  lists/
		       partial/
		  xstatus
  /var/cache/apt/
                  pkgcache.bin
		  srcpkgcache.bin
		  archives/
		          partial/
  /etc/apt/
           sources.list
	   cdromdevs.list
  /usr/lib/apt/
                methods/
		       cdrom
		       ftp
		       http
</example>

<p>
As is specified in the FHS 2.0 /var/state/apt is used for application 
data that is not expected to be user modified. /var/cache/apt is used
for regeneratable data and is where the package cache and downloaded .debs
go.
</sect>
                                                                  <!-- }}} -->

<chapt>Files
<!-- Distribution Source List					       {{{ -->
<!-- ===================================================================== -->
<sect>Distribution Source list (sources.list)

<p>
The distribution source list is used to locate archives of the debian
distribution. It is designed to support any number of active sources and to
support a mix of source media. The file lists one source per line, with the 
fastest source listed first. The format of each line is:

<p>
<var>type ui args</var>

<p>
The first item, <var>type</var>, indicates the format for the remainder 
of the line. It is designed to indicate the structure of the distribution
the line is talking about. Currently the only defined value is <em>deb</em>
which indicates a standard debian archive with a dists dir.

<sect1>The deb Type
   <p>
   The <em>deb</em> type is to be a typical two level debian distributions,
   dist/<var>distribution</var>/<var>component</var>. Typically distribution
   is one of stable, unstable or frozen while component is one of main, 
   contrib, non-free or non-us. The format for the deb line is as follows:

   <p>
   deb <var>uri</var> <var>distribution</var> <var>compontent</var> 
   [<var>component</var> ...]

   <p>
   <var>uri</var> for the <em>deb</em> type must specify the base of the 
   debian distribution. APT will automatically generate the proper longer 
   URIs to get the information it needs. <var>distribution</var> can specify
   an exact path, in this case the components must be omitted and
   <var>distribution</var> must end in a slash.
   
   <p>
   Since only one distribution can be specified per deb line it may be
   necessary to list a number of deb lines for the same URI. APT will
   sort the URI list after it has generated a complete set to allow 
   connection reuse. It is important to order things in the sourcelist
   from most prefered to least prefered (fastest to slowest).
</sect1>

<sect1>URI specification
<p> 
URIs in the source list support a large number of access schemes.

<taglist>
<tag>cdrom<item>
   The cdrom scheme is special in that If Modifed Since queries are never
   performed and that APT knows how to match a cdrom to the name it
   was given when first inserted. It does this by examining the date
   and size of the package file. APT also knows all of the possible 
   prefix paths for the cdrom drives and that the user should be prompted
   to insert a CD if it cannot be found. The path is relative to an 
   arbitary mount point (of APT's choosing) and must not start with a 
   slash. The first pathname component is the given name and is purely
   descriptive and of the users choice. However, if a file in the root of 
   the cdrom is called 'cdname' its contents will be used instead of 
   prompting. The name serves as a tag for the cdrom and should be unique.
   APT will track the CDROM's based on their tag and package file
   properties.
   <example>
   cdrom:Debian 1.3/debian
   </example>

<tag>http<item>
   This scheme specifies a HTTP server for the debian archive. HTTP is prefered
   over FTP because If Modified Since queries against the Package file are
   possible. Newer HTTP protcols may even support reget which would make
   http the protocol of choice.
   <example>
   http://www.debian.org/archive
   </example>

<tag>ftp<item>
   This scheme specifies a FTP connection to the server. FTP is limited because
   there is no support for IMS and is hard to proxy over firewalls.
   <example>
   ftp://ftp.debian.org/debian
   </example>

<tag>file<item>
   The file scheme allows an arbitary directory in the file system to be 
   considered as a debian archive. This is usefull for NFS mounts and 
   local mirrors/archives.
   <example>
   file:/var/debian
   </example>
   
<tag>mirror<item>
   The mirror scheme is special in that it does not specify the location of a
   debian archive but specifies the location of a list of mirrors to use
   to access the archive. Some technique will be used to determine the
   best choice for a mirror. The mirror file is specified in the Mirror File
   section. If/when URIs take off they should obsolete this field.
   <example>
   mirror:http://www.debian.org/archivemirrors
   </example>
   
<tag>smb<item>
   A possible future expansion may be to have direct support for smb (Samba 
   servers).
   <example>
   smb://ftp.kernel.org/pub/mirrors/debian
   </example>
</taglist>
</sect1>

<sect1>Hashing the URI
<p>
All permanent information aquired from any of the sources is stored in the
lists directory. Thus, there must be a way to relate the filename in the
lists directory to a line in the sourcelist. To simplify things this is
done by quoting the URI and treating _'s as quoteable characters and
converting / to _. The URI spec says this is done by converting a 
sensitive character into %xx where xx is the hexadecimal representation 
from the ascii character set. Examples:

<example>
http://www.debian.org/archive/dists/stable/binary-i386/Packages 
/var/state/apt/lists/www.debian.org_archive_dists_stable_binary-i386_Packages

cdrom:Debian 1.3/debian/Packages
/var/state/apt/info/Debian%201.3_debian_Packages
</example>

<p> 
The other alternative that was considered was to use a deep directory 
structure but this poses two problems, it makes it very difficult to prune
directories back when sources are no longer used and complicates the handling
of the partial directory. This gives a very simple way to deal with all
of the situations that can arise. The equals sign was choosen on the 
suggestion of Manoj because it is very infrequently used in filenames.
Also note that the same rules described in the <em>Archive Directory</>
section regarding the partial sub dir apply here as well.
</sect1>

</sect>
                                                                  <!-- }}} -->
<!-- Extra Status						       {{{ -->
<!-- ===================================================================== -->
<sect>Extra Status File (xstatus)

<p>
The extra status file serves the same purpose as the normal dpkg status file 
(/var/lib/dpkg/status) except that it stores information unique to diety.
This includes the autoflag, target distribution and version and any other
uniqe features that come up over time. It duplicates nothing from the normal
dpkg status file.  Please see other APT documentation for a discussion
of the exact internal behavior of these fields. The Package field is
placed directly before the new fields to indicate which package they
apply to. The new fields are as follows:

<taglist>
<tag>X-Auto<item>
   The Auto flag can be Yes or No and controls whether the package is in
   auto mode. 

<tag>X-TargetDist<item>
   The TargetDist item indicates which distribution versions are offered for
   installation from. It should be stable, unstable or frozen.
   
<tag>X-TargetVersion<item>
   The target version item is set if the user selects a specific version, it
   overrides the TargetDist selection if both are present.
</taglist>
</sect>
                                                                  <!-- }}} -->
<!-- Binary Package Cache					       {{{ -->
<!-- ===================================================================== -->
<sect>Binary Package Cache (pkgcache.bin)

<p>
Please see cache.sgml for a complete description of what this file is. The 
cache file is updated whenever the contents of the lists directory changes.
If the cache is erased, corrupted or of a non-matching version it will
be automatically rebuilt by all of the tools that need it. 
<em>srcpkgcache.bin</> contains a cache of all of the package files in the 
source list. This allows regeneration of the cache when the status files 
change to use a prebuilt version for greater speed.
</sect>
                                                                  <!-- }}} -->
<!-- Downloads Directory					       {{{ -->
<!-- ===================================================================== -->
<sect>Downloads Directory (archives)

<p>
The archives directory is where all downloaded .deb archives go. When the
file transfer is initiated the deb is placed in partial. Once the file
is fully downloaded and its MD5 hash and size are verifitied it is moved
from partial into archives/. Any files found in archives/ can be assumed 
to be verified.

<p>
No dirctory structure is transfered from the receiving site and all .deb
file names conform to debian conventions. No short (msdos) filename should
be placed in archives. If the need arises .debs should be unpacked, scanned
and renamed to their correct internal names. This is mostly to prevent
file name conflicts but other programs may depend on this if convenient.
Downloaded .debs must be found in one of the package lists with an exact
name + version match..
</sect>
                                                                  <!-- }}} -->
<!-- The Methods Directory					       {{{ -->
<!-- ===================================================================== -->
<sect> The Methods Directory (/usr/lib/apt/methods)

<p>
Like dselect, APT will support plugable acquisition methods to complement
its internaly supported methods. The files in
this directory are execultables named after the URI type. APT will
sort the required URIs and spawn these programs giving a full sorted, quoted 
list of URIs.

<p>
The interface is simple, the program will be given a list
of URIs on the command line. The URIs will be a pairs of strings, the first 
being the actual URI and the second being the filename to write the data to. 
The current directory will be set properly by APT and it is 
expected the method will put files relative to the current directory. 
The output of these programs is strictly speficied. The programs must accept
nothing from stdin (stdin will be an invalid fd) and they must output 
status information to stdout according to the format below.
Stderr will be redirected to the logging facility.

<p>
Each line sent to stdout must be a line that has a single letter and a
space. Strings after the first letter do not need quoting, they are taken
as is till the end of the line. The tag letters, listed in expected order, 
is as follows:

<taglist>

<tag>F - Change URI<item>
This specifies a change in URI. All information after this will be applied
to the new URI. When the URI is changed it is assumed that the old URI has
completed unless an error is set. The format is <var>F URI</>

<tag>S - Object Size<item>
This specifies the expected size of the object. APT will use this to 
compute percent done figures. If it is not sent then a kilobyte meter
will be used instead of a percent display. The foramat is <var>S INTEGER</>

<tag>E - Error Information<item>
Exactly one line of error information can be set for each URI. The
information will be summarized for the user. If an E tag is send before
any F tags then the error is assumed to be a fatal method error and all URI
fetches for that method are aborted with that error string. The format 
is <var>E String</>

<tag>I - Informative progress information<item>
The I tag allows the method to specify the status of the connection. 
Typically the GUI will show the last recieved I line. The format is
<var>I String</> As a general rule an I tag should be ommitted before a
lengthy operation only. Things that always take a short period are not
suited for I tags. I tags should change wnenever the methods state changes.
Some standard forms, in order of occurance, are <var>Connecting to SITE</>,
<var>Connecting to SITE (1.1.1.1)</>, <var>Waiting for file</>, 
<var>Authenticating</>, <var>Downloading</>, <var>Resuming (size)</>, 
<var>Computing MD5</> <var>I</> lines should never print out information that 
APT is already aware of, such as file names.

<tag>R - Set final path<item>
The R tag allows the method to tell APT that the file is present in the
local file system. APT might copy it into a the download directory. The format
is <var>R String</>

<tag>M - MD5Sum of the file<item>
The method is expected to compute the md5 hash on the fly as the download
progresses. The final md5 of the file is to be output when the file is 
completed. If the md5 is not output it will not be checked! Some methods
such as the file method will not check md5's because they are most
commonly used on mirrors or local CD-ROM's, a paranoid option may be
provided in future to force checking. The format is <var>M MD5-String</>

<tag>L - Log output<item>
This tag indicates a string that should be dumped to some log file. The
string is for debugging and is not ment to be seen by the user. The format
is <var>L String</> Log things should only be used in a completed method
if they have special relavence to what is happening.
</taglist>

<p>
APT monitors the progress of the transfer by watching the file size. This
means the method must not create any temp files and must use a fairly small
buffer. The method is also responsible for If-Modified-Since (IMS) queries
for the object. It should check ../outputname to get the time stamp but not
size. The size may be different because the file was uncompressed after
it was transfed. A method must <em>never</> change the file in .., it may
only change the output file in the current directory.

<p>
The APT 'http' program is the reference implementation of this specification, 
it implements all of the features a method is expected to do.
</sect>
                                                                  <!-- }}} -->
<!-- The Mirror List						       {{{ -->
<!-- ===================================================================== -->
<sect> The Mirror List

<p>
The mirror list is stored on the primary debian web server (www.debian.org)
and contains a machine readable list of all known debian mirrors. The mirror
URI type will cause this list to be downloaded and considered. It has the
same form as the source list. When the source list specifies mirror
as the target the mirror list is scanned to find the nescessary parts for 
the requested distributions and components. This means the user could 
have a line like:

<var>deb mirror:http://www.debian.org/mirrorlist stable main non-us</var>

which would likely cause APT to choose two separate sites to download from,
one for main and another for non-us.

<p>
Some form of network measurement will have to be used to gauge performance
of each of the mirrors. This will be discussed later, initial versions
will use the first found URI.
</sect>
                                                                  <!-- }}} -->
<!-- The Release File						       {{{ -->
<!-- ===================================================================== -->
<sect> The Release File

<p>
This file plays and important role in how APT presents the archive to the 
user. Its main purpose is to present a descriptive name for the source
of each version of each package. It also is used to detect when new versions
of debian are released. It augments the package file it is associated with 
by providing meta information about the entire archive which the Packages
file describes.

<p>
The full name of the distribution for presentation to the user is formed
as 'label version archive', with a possible extended name being 
'label version archive component'.

<p>
The file is formed as the package file (RFC-822) with the following tags
defined:

<taglist>
<tag>Archive<item>
This is the common name we give our archives, such as <em>stable</> or
<em>unstable</>.

<tag>Component<item>
Referes to the sub-component of the archive, <em>main</>, <em>contrib</>
etc.

<tag>Version<item>
This is a version string with the same properties as in the Packages file.
It represents the release level of the archive.

<tag>Origin<item>
This specifies who is providing this archive. In the case of Debian the
string will read 'Debian'. Other providers may use their own string

<tag>Label<item>
This carries the encompassing name of the distribution. For Debian proper
this field reads 'Debian'. For derived distributions it should contain their 
proper name.

<tag>Architecture<item>
When the archive has packages for a single architecture then the Architecture
is listed here. If a mixed set of systems are represented then this should
contain the keyword <em>mixed</em>.

<tag>NotAutomatic<item>
A Yes/No flag indicating that the archive is extremely unstable and its
version's should never be automatically selected. This is to be used by 
experimental.

<tag>Description<item>
Description is used to describe the release. For instance experimental would
contain a warning that the packages have problems.
</taglist>

<p>
The location of the Release file in the archive is very important, it must 
be located in the same location as the packages file so that it can be 
located in all situations. The following is an example for the current stable
release, 1.3.1r6 

<example>
Archive: stable
Compontent: main
Version: 1.3.1r6
Origin: Debian
Label: Debian
Architecture: i386
</example>

This is an example of experimental,
<example>
Archive: experimental
Version: 0
Origin: Debian
Label: Debian
Architecture: mixed
NotAutomatic: Yes
</example>

And unstable,
<example>
Archive: unstable
Compontent: main
Version: 2.1
Origin: Debian
Label: Debian
Architecture: i386
</example>

</sect>
                                                                  <!-- }}} -->

</book>