[gentoo-soc] Project IDFetch - Weekly report #4 ("Growing muscles on skeleton")

[Kostyantyn Ovechko]

-========================================================================-
-  Project IDFetch - Weekly report #4 ("Growing muscles on skeleton")    - 
-========================================================================-

Fresh milk and cottage cheese gradually enforce the skeleton, at the
same time growing muscles on it can take a lot of efforts. Nevertheless
pulling-ups, mirror benchmarking [1], and other things were added to my
program.

NOTE: 
                Previous weeks i kept writing reports, but while sending
                them i used my default email account, which
                unfortunately happened to be not the one subscribed to
                [gentoo-soc]. Hence, i don't think anybody got them.
                Yesterday i resent these reports (better later then
                never):
                   - Project IDFetch - Weekly report #2 ("Replacing
                stuff")
                   - Project IDFetch - Weekly report #3 ("Strawberry
                issues")
                
                Since i've already tried writing reports without sending
                them, why not to do the opposite - to send a report
                without actually writing it ;o)
                So please take a look at the list of the options
                implemented in segget during the 4th week of IDFetch
                project.
                
IDFetch: adding muscles to segget.conf skeleton:
================================================

[folders]
# DISTFILES_DIR
# Define a dir to store distfiles
# Default:
# distfiles_dir=./distfiles
distfiles_dir=./distfiles

# SEGMENTS_DIR
# Define a dir to store distfiles' segments
# Default:
# segments_dir=./tmp
segments_dir=./tmp

[distfiles]
# MAX_CONNECTION_NUM_PER_DISTFILE
# Each distfile can have up to max_connection_num_per_distfile
# simultaneous connections.
# default:
# max_connection_num_per_distfile=3
max_connection_num_per_distfile=3

[segments]
# MAX_SEGMENT_SIZE
# Define maximum segment size in bytes.
# Default:
# max_segment_size=500000
max_segment_size=500000

# SYNOPSIS: resume_on=0 | 1
# - If resume_on set to 1:
# Before starting downloading a segment segget checks whether this
# segment is already downloaded, checks segments size and if size
# mathces considers this segment to be downloaded and skips downloading
# process.
# - If resume_on set to 0:
# Segget always starts new fetch for a segment regardless of the 
# fact whether it is downloaded or not.
# Default:
# resume_on=1
resume_on=1

# MAX_TRIES
# If segment download was unsuccessful, new attempts are made. When 
# attempts number reaches max_tries, segment gets FAILED status and 
# error logged to error_log.
# Default:
# max_tries=10
max_tries=10

[connections]
# MAX_CONNECTIONS
# Define maximum number of connections
max_connections=10

# CONNECTION_TIMEOUT
# Set the number of seconds to wait while trying to connect. Use 0 to 
# wait indefinitely. Pass a long. It should contain the maximum time in 
# seconds that you allow the connection to the server to take. This 
# only limits the connection phase, once it has connected, this option 
# is of no more use. Set to zero to disable connection timeout (it will
# then only timeout on the system's internal timeouts). See also the 
# TIMEOUT option.
# Default:
# connection_timeout=15
connection_timeout=15

# FTP_RESPONSE_TIMEOUT
# Set a timeout period (in seconds) on the amount of time that the 
# server is allowed to take in order to generate a response message for 
# a command before the session is considered hung. While awaiting for a 
# response, this value overrides TIMEOUT. It is recommended that if 
# used in conjunction with TIMEOUT, you set FTP_RESPONSE_TIMEOUT to a 
# value smaller than TIMEOUT.
# Default:
# ftp_response_timeout=180
ftp_response_timeout=180

# TIMEOUT
# maximum amount of time to download segment in seconds
# Set the maximum number of seconds for a connection to execute.
# Pass a long as parameter containing the maximum time in seconds that 
# you allow the transfer operation to take. Normally, name lookups can 
# take a considerable 
# time and limiting operations to less than a few minutes risk aborting 
# perfectly normal operations.
# Default:
# timeout=500
timeout=500

# LOW_CONNECTION_SPEED_LIMIT
# Define the low speed limit for connection. Pass a long as parameter. 
# It contains the transfer speed in bytes per second that the transfer 
# should be below during LOW_CONNECTION_SPEED_TIME seconds to consider 
# it too slow and abort.
# Default:
# low_connection_speed_limit=1000
low_connection_speed_limit=1000

# LOW_CONNECTION_SPEED_TIME
# Pass a long as parameter. It contains the time in seconds that the 
# transfer should be below the LOW_CONNECTION_SPEED_LIMIT to consider 
# it too slow and abort.
# Default:
# low_connection_speed_time=10
low_connection_speed_time=10

# MAX_CONNECTION_SPEED
# If a download exceeds this speed (counted in bytes per second) on 
# cumulative average during the transfer, the transfer will pause to 
# keep the average rate less than or equal to the parameter value. 
# Defaults to unlimited speed.
# Default:
# max_connection_speed=0
max_connection_speed=0

# BIND INTERFACE / IP
# Pass a string as parameter. This sets the interface name to use as 
# outgoing network interface. The name can be an interface name, an IP
# address, or a host name. No binding is set by default.
# Default:
# bind_interface=none
bind_interface=none

[mirrors]
# MAX_CONNECTIONS_NUM_PER_MIRROR
# Define how many simultaneous downloads from one mirror segget is 
# allowed to have. While choosing a mirror segget will skip mirrors 
# with max_connections_num_per_mirror active downloads.
# Default:
# max_connections_num_per_mirror=1
max_connections_num_per_mirror=1

# SYNOPSIS: collect_benchmark_stats_on=0 | 1
# - If set to 1, stats on mirrors performance will be collected.
# default:
# collect_benchmark_stats_on=1 
# ***Note: at the moment collect_benchmark_stats_on can NOT be set to 0
collect_benchmark_stats_on=1

# SYNOPSIS: use_benchmark_stats=0 | 1
# If use_benchmark_stats=1 statistics on mirrors is used to rate them 
# and therefore improve performance.
# Each time connection from a particular mirror closes mirror->dld_time, 
# and mirror->dld_size get increased (in case of unsuccessful connection
# only time gets increased), so avg speed for a mirror can be
# calculated:
# 
#           mirror->avg_speed=mirror->dld_size/mirror->dld_time.       (1)
#
# When new segment is going to be started segget goes through the list of
# the mirrors distfile/segment has, and asks each mirror for self_rating:
#
#          "ulong self_rating=mirror->mirror_on_the_wall();". 
#
# This way segget chooses a mirror with the best self_rating. To calculate 
# self_rating mirrors use the following formula:
#
# 		self_rating=dld_time/dld_size*honesty.                 (2)
#
# So mirrors actually say how bad they are.
# Even mirrors can have critical times, so to give mirrors another chance 
# honesty was added to the formula (2). honesty can get values in 
# interval (0,1]. Each time connection from a mirror opens or closes mirror
# sets its honesty=1. If mirror was asked for self_evaluation with 
# mirror->mirror_on_the_wall(), but wasn't chosen its honesty decreases
# somewhat (see [mirror].benchmark_oblivion option), so next time it will 
# lie little bit more about how bad it's.
# Default:
# use_benchmark_results=1
# ***Note: at the moment use_benchmark_results can NOT be set to 0
use_benchmark_stats=1

# BENCHMARK_OBLIVION
# benchmark_oblivion option allows to adjust how fast segget "forgets" 
# benchmarking statistics on mirrors performance. 
# Each time mirror->mirror_on_the_wall() called, mirror decreases its
# honesty (to have more chances next time) using the following formula:
#
#       honesty=honesty*100/(100+settings.benchmark_oblivion)         (3)
#
# Therefore, setting benchmark_oblivion=100 will make mirror look twice
# less bad next time mirror->mirror_on_the_wall() called.
# Default:
# benchmark_oblivion=5
benchmark_oblivion=5

[user-data]
# USER_AGENT
# Set the User-Agent: header in the http request sent to the remote 
# server. 
# This can be used to fool servers or scripts.
# Default:
# user_agent=segget
user_agent=segget

[proxy]
# PROXY_IP_OR_NAME
# Specify a proxy to use (address and port).
# Set HTTP proxy to use. The parameter should be a string holding the 
# proxy host name or dotted IP address. To specify port number in this 
# string, append :[port] to the end of the host name. The proxy string 
# may be prefixed with [protocol]:// since any such prefix will be 
# ignored. The proxy's port number may optionally be specified with the 
# separate option. If not specified, by default port 1080 will be used 
# for proxies. When you tell segget to use an HTTP proxy, segget will 
# transparently convert operations to HTTP even if you specify an FTP 
# URL etc. Segget respects the environment variables http_proxy, 
# ftp_proxy, all_proxy etc, if any of those are set. The PROXY option 
# does however override any possibly set environment variables.
# Default:
# proxy_ip_or_name=none
proxy_ip_or_name=none

# PROXY_PORT
# Set the proxy port to connect to unless it is specified in the PROXY 
# option.
# Default:
# proxy_port=3128
proxy_port=3128

# PROXY_USER
# Set user name to use for the transfer while connecting to Proxy.
# The PROXY_USER option should be used in same way as the 
# PROXY_PASSWORD is used. In order to specify the password to be used 
# in conjunction with the user name use the PROXY_PASSWORD option.
# Default:
# proxy_user=none
proxy_user=none

# PROXY_PASSWORD
# Set password to use for the transfer while connecting to Proxy.
# The PROXY_PASSWORD option should be used in conjunction with the 
# PROXY_USER option.
# Default:
# proxy_password=none
proxy_password=none

# SYNOPSIS: proxy_off=0 | 1
# Setting the proxy_off=1 will explicitly disable the use of a proxy, 
# even if there is an environment variable set for it.
# Default:
# proxy_off=1
proxy_off=1

[logs]
# LOGS_DIR
# Define a dir to store log files.
# Default:
# logs_dir=./logs
logs_dir=./logs

# GENERAL_LOG_FILE
# Define a file name to store general log.
# Default:
# general_log_file=segget.log
general_log_file=segget.log

# ERROR_LOG_FILE
# Define a file name to store error log.
# Default:
# error_log_file=segget.log
error_log_file=error.log

# DEBUG_LOG_FILE
# Define a file name to store debug log.
# Default:
# debug_log_file=segget.log
debug_log_file=debug.log

[1]http://www.twolia.com/blogs/daily-beauty-break/files/2010/04/mirror-on-the-wall.jpg

Best regards,
Kostyantyn Ovechko AKA simka