Home > Distfiles, Dowloader, Fetcher, Gentoo, GSOC, Linux, Packages, Portage > WR#2 – “Replacing Stuff”

WR#2 – “Replacing Stuff”

Though, twrapper.py is a good thing to play with: it takes list
of files from pkg.list and starts downloading them
simultaneously. Nevertheless it uses wget tool for this and
communication between twrapper.py has some serious limitations
especially when it comes to controling simultaneous downloads
(speed, segments, etc). So next step for idfetch project was to
make its own tool for segmented file fetching.


1. [Replacing stuff]
Started from replacing chocolate module by dates [1] (i still
use coffee though), python by cpp, wget by libcurl [2].

2. [JSON struggles]. Going cpp grows my love to python.

2.1 Using python

2.1.1 Saving json data with python

To save data to json-formated pkg.list fetch.py
uses following lines:

import json

idfetch_pkg_list_file.write(json.dumps(idfetch_pkg_list, sort_keys=1, indent=4))

2.1.2 Loading json data with python

To load data from json-formated pkg.list
twrapper.py uses these 2 lines:

import json

idfetch_pkg_list=json.loads(idfetch_pkg_list_file.read())

2.2 Using cpp


2.2.1 Loading json data with cpp
Reading data in cpp resulted in:
a) #emerge json-c
b) g++ -o task task.c -ljson (where
task.c has 146 lines of code = 5037
bytes);
c++) minus one day spend on these 146
lines of code.

3. [Discovering libcurl].
It's the first time I use libcurl for my project (Thanks to
Robin H. Johnson). Libcurl happened to be pretty interesting lib
with lots of opportunities to discover and implement into
idfetch. Therefore I only hope not to bump into some grueling
bug or limitation.

4. [Downloading with libcurl].

Experiments at stage 3 resulted in segget.cpp. Segget downloads
files, splitting them into segments. It's possible to set number of
parallel connections for segget. After fetching segments, segget
combines them together to provide original file (no checksum check
implemented yet).

[1] http://en.wikipedia.org/wiki/Phoenix_dactylifera
[2] http://curl.haxx.se/libcurl/

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: