~grubng-dev/grubng/clients-python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
Grub Next Generation Python Client

2008 Giorgos Logiotatidis <seadog@sealabs.net>

Version 0.4

FEATURES

    * Python >= 2.4 Compatibility
    * Multiple crawlers support
    * Configurable


DESCRIPTION

This is a python based crawler for the grub project. 

The program by default searches the ~/.grub and the current directory for the file 'account' to find username the password information. See CONFIG FILE Section. Alternativelly you can use the --username and --password flags on run time. 

It spawns 5 threads but you can add more with the -t flag. A tested, working number is 20 which produces arround 3000 arcs per day on a moderate DSL line.


CONFIG FILE
The format of the config file should be file the password file. E.g.

username:passwd

EXAMPLES

python grub.py -t 5 -u username -p password

Runs python client with 5 crawlers. Use -h flag for more options


Changelog 0.4
* Fix fetchWorkUnit return values. Closes Bug http://grub.org/?q=en/node/264
* Support for new workunits. More HTTP headers per request.
* Replacing socket.gaierror with socket.error to catch all socket module exceptions. Closes bug: http://grub.org/?q=en/node/283
* Added Uploading thread. Closing bug http://grub.org/?q=en/node/302
* Added upload speed meter by Tbohdan. Closes bug http://grub.org/?q=en/node/319
* Fix --help
* and code cleanup, small code fixes
 


KNOWN BUGS
 * None for this release