Home > Distfiles, Dowloader, Fetcher, Gentoo, GSOC, Linux, Packages, Portage > WR#1 – “Getting Nervous”

WR#1 – “Getting Nervous”

The purpose of the project is to optimize software installation process,
by means of making distfile fetcher more intelligent and increasing
effectiveness of network connection utilization. The idea of the project
is not to rewrite the whole Portage system, but rather the part that
actually contiguous with the bottle neck of the network connection – the
distfile fetcher.

For more information on the project please see website:

git repository for idfetch project:

git repository for changes to Portage:

You can share your ideas on idfetch by joining IRC channel
#gentoo-idfetch at freenode or just sending me an email.

The progress report:

1) I've started from joining the mainstream and becoming pretty nervous
of the thoughts whether i can manage this (for some people seemingly
easy) project. After importing chocolate and coffee modules i tried to
switch to more productive things πŸ˜‰

2) First thing to do was to export some data from the current portage
system:Β  basename, mirror URLs, size, checksums. I ended up with some
changes to fetch.py file that provided me with the following results:

# list of pkgs to be installed
Tidfetch_pkg_list : list of Tidfetch_pkg;

Tidfetch_pkg : dict
['pkg_name'] : string;
['distfile_list'] : list of Tidfetch_distfile;

Tidfetch_distfile : dict
['name'] : string;
['url_list'] : string;
['size'] : int;

3) I started to use pickle module to exchange data between fetch.py and
twrapper (threaded wrapper for simultaneous downloads)

4) Following advice from Robin H. Johnson (my mentor) replaced pickle
module by json [1]. So now pkg_list.list (export file) looks this way:

"distfile_list": [
"RMD160": "10a19a10d0388bc084a7c1d3da845068d7169054",
"SHA1": "2a7198e8178b2e7dba87cb5794da515200b568f5",
"name": "ghostscript-fonts-std-8.11.tar.gz",
"size": 3752871,
"url_list": [
...........more mirrors - skipped ......
"pkg_name": "gnu-gs-fonts-std-8.11"
"distfile_list": [
"RMD160": "ae50d9eaccb3cc6aa48669eb5ea44a2857e80952",
"SHA1": "d6c3ed6f0c0deab9ee4f6d63f7b2c7ce3cbae280",
"name": "util-macros-1.6.1.tar.bz2",
"size": 62130,
"url_list": [
...........more mirrors - skipped ......
"pkg_name": "util-macros-1.6.1"

5) Development of simple-threaded-twrapper started. Twrapper reads the
data from pkg_list.list file and starts downloading simultaneously
distfiles from the list (according to MAX_ACTIVE_DOWNLOADS in

5.1) Before downloading file it checks if file already exists, it's
complete, check sums are ok. If so, skips this distfile. Otherwise
downloads it and checks its check sums.

6) Interface development: it's possible to choose between curses
(USE_CURSES_FLAG=1) and simple log-like output (USE_CURSES_FLAG=0).
Probably i'll do tput-interface implementation, because log-like output
is hard to follow and curses don't like buggy code 😦

To see examples of output please follow these links:

7) Robin H. Johnson suggested that Portage changes might be better in a
separate repo, that tracks the main Portage repo. Therefore, repository
for portage-idfetch project was created:

[1] http://docs.python.org/library/json.html

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: