[gentoo-soc] Project IDFetch - Weekly report #6 ("Network/Connection Management")

[Kostyantyn Ovechko]

-=====================================================================-
-Project IDFetch - Weekly report #6 ("Network/Connection Management") - 
-=====================================================================-

Amway, Oriflame, Mary Kay ... looks like the whole world is covered with
these networks. I don't mind network marketing [1], but in most cases
network brokers so eager to cover all of your needs, that probably you
don't even have a chance to buy from the others or even from a local
place [2]. Trying to impose limits on your connections with network
brokers doesn't really work - they already know how to get you via your
cell phone, LAN and in some cases even the address you are bound to[3].

Although, sometimes i'd like to provide something myself [4], but
mostly it's getting extremely annoyinggggggggg. Hopefully development of
Network/Connection Management System [5] will mitigate at least some of 
these problems. 
We'll see...

From IDFetch project timeline [5]:
Fetchers-Connection-Manager will allow fetcher to manage multiple
connections, in order to increase throughput, and prevent downloads via
some network connections ( e.i. to prevent connection via cell-phone,
when you pay for the traffic).

Progress on IDFetch project
===========================
Development of Network/Connection Management System resulted in adding
the following options to the segget's configuration files:

----------------- Added to segget.conf file -----------------

[networks]
# NETWORK0_PRIORITY
# Define priority as a value in range from lowest 0 to highest 10.
# Segget tries to use networks with higher priority levels first, and in
# case of failure will switch to networks with lower priority levels.
# Segget will NOT use network if its priority level set to 0. Therefore
# at least one network must have priority level higher than 0.
# Networks with local mirrors usually would have higher priority than
# that of networks with remote mirrors.
# Segget can have up to 10 networks (from network0 to network9).
# Settings for each network should be defined in network#.conf file,
# where instead of # should be a network number.
# For network0 it's network0.conf
# Default:
# network0_priority=10
# network1_priority=0
# network2_priority=0
# network3_priority=0
# network4_priority=0
# network5_priority=0
# network6_priority=0
# network7_priority=0
# network8_priority=0
# network9_priority=0
network0_priority=10

[provide_mirror_to_others]
# PROVIDE_MIRROR_DIR
# Define a dir for making symlinks to downloaded distfiles. This dir can
# be used to provide local mirror for other hosts (with help of Apache,
# vsftp, etc).
# If set to none, segget will not make symlinks.
# Default:
# provide_mirror_dir=none
provide_mirror_dir=./provide_mirror_dir

# SYNOPSIS: PROVIDE_MIRROR_FILES_RESTRICT_LIST_ON= 0 | 1
# If PROVIDE_MIRROR_DIR=none this option will be ignored.
# - If set to 1, segget will compare distfile name with the list of
# forbiden patterns from the restricted.conf file. If distfile name
# contains any of the patterns, no symlink will be provided to this 
# distfile.
# Default:
# provide_mirror_files_restrict_list_on=0
provide_mirror_files_restrict_list_on=1

----------------- Added to network#.conf files -----------------

[network_mirrors] 
# SYNOPSIS: NETWORK_USES_OWN_MIRROR_LIST_ONLY_ON=0 | 1
# - If set to 1, segget will replace mirror list provided by portage
# system with the list from network0_mirrors.conf file
# - If set to 0, segget will use ONLY mirror list provided by portage
# system, and will NOT use the list from network0_mirrors.conf file
# In some cases it's necessary to make segget prefer local mirrors over
# the remote ones. For this purpose define settings for 2 networks: 
# settings for network0 (to provide access to local mirrors), 
# settings for network1 (to provide access to remote ones).
# 1) Set the following options in segget.conf file:
# [networks]
# network0_priority=10
# network1_priority=9
# As you can see network0 (with local mirrors) has higher priority than 
# network1 (with mirrors provided by portage).
# 2) Set NETWORK_USES_OWN_MIRROR_LIST_ONLY_ON=1 in network0.conf file.
# 3) Create network0_mirrors.conf file with the list of your local 
# mirrors.
# For example, network0_mirrors.conf may look like this:
# http://192.168.210.12/
# ftp://192.168.210.205/
# http://192.168.210.56/
# 4) Set NETWORK_USES_OWN_MIRROR_LIST_ONLY_ON=0 in network1.conf file, 
# so segget will use remote mirrors working via this network.
# NOTE: Actually network0 and network1 can be the same LAN with only 
# one ip address set on the host. The only difference is that in case of
# network1 segget will have to use a gateway to access remote mirrors.
# Default:
# use_own_mirror_list_only_on=0
use_own_mirror_list_only_on=0
 
# SYNOPSIS: ONLY_LOCAL_WHEN_POSSIBLE=0 | 1
# If NETWORK_USES_OWN_MIRROR_LIST_ONLY_ON=0 this option will be ignored.
# - If set to 1, segget will not use remote mirrors with equal or lower
# priority until all mirrors in network#_mirrors.conf file have failed.
# - If set to 0, segget will use remote mirrors with equal priority or
# mirrors with lower priority when this network has NO free connections 
# (see option NETWORK_MAX_CONNECTIONS in [network_connections] section 
# of this file).
# NOTE: Following example for NETWORK_USES_OWN_MIRROR_LIST_ONLY_ON
option,
# if in network0.conf has option ONLY_LOCAL_WHEN_POSSIBLE=1, segget
# will NOT start to use network1 for a particular distfile until all
# mirrors specified in network0_mirrors.conf file will have failed 
# to provide this distfile.
# On the other hand if ONLY_LOCAL_WHEN_POSSIBLE=0 segget will start
# to use network1 as soon as NETWORK_MAX_CONNECTIONS limit, set
# in network0.conf file has been reached.
# Default:
# only_local_when_possible=1
only_local_when_possible=1 

------------- Moved from segget.conf to network#.conf files ------------

[network_bind]
# BIND INTERFACE / IP
# Pass a string as parameter. This sets the interface name to use as
# outgoing network interface. The name can be an interface name, an IP 
# address, or a host name. No binding is set by default.
# Default:
# bind_interface=none
bind_interface=none

# BIND LOCALPORT
# Pass a long. This sets the local port number of the socket used for
# connection.
# This can be used in combination with BIND_INTERFACE and you are
# recommended to use BIND_LOCALPORTRANGE as well when this is set.
# Valid port numbers are 1 - 65535.

# BIND_LOCALPORTRANGE
# Pass a long. This is the number of attempts segget should make to find
# a working local port number. It starts with the given BIND_LOCALPORT 
# and adds one to the number for each retry. Setting this to 1 or below 
# will make segget do only one try for the exact port number. Port 
# numbers by nature are scarce resources that will be busy at times so 
# setting this value to something too low might cause unnecessary 
# connection setup failures.

[network_connections]
# NETWORK_MAX_CONNECTIONS
# Define maximum number of connections
# Minimum value: 1
# Maximum value: 20
# Default:
# max_connections=10
max_connections=2

# CONNECTION_TIMEOUT
# Set the number of seconds to wait while trying to connect. Use 0 to
# wait indefinitely. Pass a long. It should contain the maximum time in
# seconds that you allow the connection to the server to take. This
# only limits the connection phase, once it has connected, this option 
# is of no more use. Set to zero to disable connection timeout (it will 
# then only timeout on the system's internal timeouts). See also the 
# TIMEOUT option.
# Minimum value: 1
# Maximum value: 1000
# Default:
# connection_timeout=15
connection_timeout=15

# FTP_RESPONSE_TIMEOUT
# Set a timeout period (in seconds) on the amount of time that the
# server is allowed to take in order to generate a response message for 
# a command before the session is considered hung. While awaiting for a 
# response, this value overrides TIMEOUT. It is recommended that if 
# used in conjunction with TIMEOUT, you set FTP_RESPONSE_TIMEOUT to a 
# value smaller than TIMEOUT.
# Minimum value: 1
# Maximum value: -1 (for no limit)
# Default:
# ftp_response_timeout=180
ftp_response_timeout=180

# TIMEOUT
# maximum amount of time to download segment in seconds
# Set the maximum number of seconds for a connection to execute.
# Pass a long as parameter containing the maximum time in seconds that
# you allow the transfer operation to take. Normally, name lookups can 
# take a considerable time and limiting operations to less than a few 
# minutes risk aborting perfectly normal operations.
# Minimum value: 100
# Maximum value: -1 (for no limit)
# Default:
# timeout=500
timeout=500

# LOW_CONNECTION_SPEED_LIMIT
# Define the low speed limit for connection. Pass a long as parameter.
# It contains the transfer speed in bytes per second that the transfer
# should be below during LOW_CONNECTION_SPEED_TIME seconds to consider 
# it too slow and abort.
# Minimum value: 1
# Maximum value: -1 (-1 for no limit)
# Default:
# low_connection_speed_limit=1000
low_connection_speed_limit=1000

# LOW_CONNECTION_SPEED_TIME
# Pass a long as parameter. It contains the time in seconds that the
# transfer should be below the LOW_CONNECTION_SPEED_LIMIT to consider 
# it too slow and abort.
# Minimum value: 1
# Maximum value: 600
# Default:
# low_connection_speed_time=10
low_connection_speed_time=10

# MAX_CONNECTION_SPEED
# If a download exceeds this speed (counted in bytes per second) on
# cumulative average during the transfer, the transfer will pause to 
# keep the average rate less than or equal to the parameter value. 
# Defaults to unlimited speed.
# Minimum value: 1
# Maximum value: -1 (-1 for no limit)
# Default:
# max_connection_speed=0
max_connection_speed=3000

[network_user_data]
# USER_AGENT
# Set the User-Agent: header in the http request sent to the remote
# server. 
# This can be used to fool servers or scripts.
# Default:
# user_agent=segget
user_agent=segget

[network_proxy]
# PROXY_IP_OR_NAME
# Specify a proxy to use (address and port).
# Set HTTP proxy to use. The parameter should be a string holding the
# proxy host name or dotted IP address. To specify port number in this 
# string, append :[port] to the end of the host name. The proxy string 
# may be prefixed with [protocol]:// since any such prefix will be 
# ignored. The proxy's port number may optionally be specified with the
# separate option. If not specified, by default port 1080 will be used 
# for proxies.
# When you tell segget to use an HTTP proxy, segget will transparently
# convert operations to HTTP even if you specify an FTP URL etc.
# Segget respects the environment variables http_proxy, ftp_proxy,
# all_proxy etc, if any of those are set. The PROXY option does however 
# override any possibly set environment variables.
# Default:
# proxy_ip_or_name=none
proxy_ip_or_name=none

# PROXY_PORT
# Set the proxy port to connect to unless it is specified in the PROXY
# option.
# Minimum value: 1
# Maximum value: 65535
# Default:
# proxy_port=3128
proxy_port=3128

# PROXY_USER
# Set user name to use for the transfer while connecting to Proxy.
# The PROXY_USER option should be used in same way as the PROXY_PASSWORD
# is used.
# In order to specify the password to be used in conjunction with the
# user name use the PROXY_PASSWORD option.
# Default:
# proxy_user=none
proxy_user=none

# PROXY_PASSWORD
# Set password to use for the transfer while connecting to Proxy.
# The PROXY_PASSWORD option should be used in conjunction with 
# the PROXY_USER option.
# Default:
# proxy_password=none
proxy_password=none

# SYNOPSIS: proxy_off=0 | 1
# Setting the proxy_off=1 will explicitly disable the use of a proxy,
# even if there is an environment variable set for it.
# Default:
# proxy_off=1
proxy_off=1

[1]http://en.wikipedia.org/wiki/Multi-level_marketing
[2]http://idfetch.isgreat.org/_content2/segget_doc.html#mirrors
[3]http://idfetch.isgreat.org/_content2/segget_doc.html#network_bind
[4]http://idfetch.isgreat.org/_content2/segget_doc.html#provide_mirror_to_others
[5]http://idfetch.isgreat.org/index.php/timeline