Re: [gentoo-user] kde-apps/kde-l10n-16.04.3:5/5::gentoo conflicting with kde-apps/kdepim-l10n-15.12.3:5/5::gentoo

[james]

On 08/09/2016 01:41 PM, Michael Mol wrote:
> On Tuesday, August 09, 2016 01:23:57 PM james wrote:
>> On 08/09/2016 09:17 AM, Michael Mol wrote:
>>> On Tuesday, August 09, 2016 09:13:31 AM james wrote:
>>>> On 08/09/2016 07:42 AM, Michael Mol wrote:
>>>>> On Monday, August 08, 2016 10:45:09 PM Alan McKinnon wrote:
>>>>>> On 08/08/2016 19:20, Michael Mol wrote:
>>>>>>> On Monday, August 08, 2016 06:52:15 PM Alan McKinnon wrote:
>>>>>>>> On 08/08/2016 17:02, Michael Mol wrote:
>
>>> I use Zabbix extensively at work, and have the Zabbix agent on my
>>> workstation reporting back various supported metrics. There's a great
>>> deal you can use (and--my favorite--abuse) Zabbix for, especially once
>>> you understand how it thinks.
>>
>> Congradualtions! Of the net-analyzer crowd, you've manage to find one I
>> have not spent time with........
>
> Oh, man, are you in for a treat. I recently had a conversation with a guy I
> happened to sit next to while traveling about how, were I in his position, I'd
> improve his cash crop and hydroponics operations (he periodically tests soil
> and sunlight properties) continually using a combination of cheap, custom
> probes and SBCs, feeding the data into Zabbix for monitoring and trend
> analysis / prediction. Zabbix will do time-series graphing and analysis of
> arbitrary input data; it may have been designed for watching interface
> counters, but there's no reason it need be limited to that...

Not sure of your tendencies, but yea, I tend to be more hardware and EE 
oriented, than CS. Yep, I spent too many years with time-sequenced data 
(turds) to not be totally excited about what we can now do with 
clusters, analog (16 bit+) IO  and enough processors and memory to keep
a simulation going and in RT(color).  You sure know how to instigate an 
itch.....

Besides, as I transcend  retirement, I'm looking for greener  pastures
and methodologies to enhance da(tm) dream state  ......
(thx)


>>>> Any specific kernel tweaks?
>>>
>>> Most of my tweaks for KDE revolved around tuning mysqld itself. But for
>>> sysctls improving workstation responsiveness as it relates to memory
>>> interactions with I/O, these are my go-tos:
>>>
>>>
>>>
>>> vm.dirty_background_bytes = 1048576
>>> vm.dirty_bytes = 10485760
>>> vm.swappiness = 0
>>
>> Mine are::
>> cat dirty_bytes
>> 0
>> cat dirty_background_bytes
>> 0
>
> So, that means you have vm.dirty_bytes_ratio and vm.dirty_background_ratio
> set, instead. I forget what those default to, but I think
> dirty_bacgkround_ratio defaults to something like 10, which means *10%* of
> your memory may get used for buffering disk I/O before it starts writing data
> to disk. dirty_bytes_ratio will necessarily be higher, which means that if
> you're performing seriously write-intensive activities on a system with 32GiB
> of RAM, you may find yourself with a system that will halt until it finishes
> flushing 3+GiB of data to disk.
>
>> cat swappiness
>> 60
>
> Yeah, you want that set to lower than that.
>
>>
>>> vm.dirty_background_bytes ensures that any data (i.e. from mmap or
>>> fwrite, not from swapping) waiting to be written to disk *starts*
>>> getting written to disk once you've got at least the configured amount
>>> (1MB) of data waiting. (If you've got a disk controller with
>>> battery-backed or flash-backed write cache, you might consider
>>> increasing this to some significant fraction of your write cache. I.e.
>>> if you've got a 1GB FBWC with 768MB of that dedicated to write cache,
>>> you might set this to 512MB or so. Depending on your workload. I/O
>>> tuning is for those of us who enjoy the dark arts.)
>>>
>>>
>>> vm.dirty_bytes says that once you've got the configured amount (10MB) of
>>> data waiting to be disk, then no more asynchronous I/O is permitted
>>> until you have no more data waiting; all outstanding writes must be
>>> finished first. (My rule of thumb is to have this between 2-10 times the
>>> value of vm.dirty_background_bytes. Though I'm really trying to avoid it
>>> being high enough that it could take more than 50ms to transfer to disk;
>>> that way, any stalls that do happen are almost imperceptible.)
>>>
>>>
>>>
>>> You want vm.dirty_background_bytes to be high enough that your hardware
>>> doesn't spend its time powered on if it doesn't have to be, and so that
>>> your hardware can transfer data in large, efficient, streamable chunks.
>>>
>>>
>>>
>>> You want vm.dirty_bytes enough higher than your first number so that
>>> your hardware has enough time to spin up and transfer data before you
>>> put the hammer down and say, "all right, nobody else gets to queue
>>> writes until all the waiting data has reached disk."
>>>
>>> You want vm.dirty_bytes *low* enough that when you *do* have to put that
>>> hammer down, it doesn't interfere with your perceptions of a responsive
>>> system. (And in a server context, you want it low enough that things
>>> can't time out--or be pushed into timing out--waiting for it. Call your
>>> user attention a matter of timing out expecting things to respond to
>>> you, and the same principle applies...)
>>>
>>> Now, vm.swappiness? That's a weighting factor for how quickly the kernel
>>> should try moving memory to swap to be able to speedily respond to new
>>> allocations. Me, I prefer the kernel to not preemptively move
>>> lesser-used data to swap, because that's going to be a few hundred
>>> megabytes worth of data all associated with one application, and it'll
>>> be a real drag when I switch back to the application I haven't used for
>>> half an hour. So I set vm.swappiness to 0, to tell the kernel to only
>>> move data to swap if it has no other alternative while trying to satisfy
>>> a new memory allocation request.
>>
>> OK, OK, OK. I need to read a bit about these. Any references or docs or
>> is the result of parsing out what is the least painful for a
>> workstation? I do not run any heavy databases on my workstation; they
>> are only there to hack on them. I test db centric stuff on domain
>> servers, sometimes with limited resources. I run lxde and I'm moving to
>> lxqt for workstations and humanoid (terminal) IO.
>
> https://www.kernel.org/doc/Documentation/sysctl/vm.txt
> https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/

Excellent docs, thx.

>> Do you set these differently for servers?
>
> On my servers, I keep these values similar, because I'd rather have a little
> bit lower throughput than risk a catastrophic cascade failure stemming from an
> I/O stall.
>
>>
>> Nodes in a cluster?
>
> Same story.
>
> The exception is my storage cluster, which has dirty_bytes much higher, as
> it's very solidly battery backed, so I can use its oodles of memory as a write
> cache, giving its kernel time to reorder writes and flush data to disk
> efficiently, and letting clients very rapidly return from write requests.

Are these TSdB (time series data) by chance?

OK, so have your systematically experimented with these parameter 
settings, collected and correlated the data, domain (needs) specific ?

As unikernels collide with my work on building up  minimized and 
optimized linux clusters, my pathway forward is to use several small 
clusters, where the codes/frameworks can be changed, even the 
tweaked-tuned kernels and DFS and note the performance differences for 
very specific domain solutions. My examples are quite similar to that 
aforementioned  flight sim above, but the ordinary and uncommon 
workloads of regular admin (dev/ops) work is only a different domain.

Ideas on automating the exploration of these settings 
(scripts/traces/keystores) are keenly of interest to me, just so you know.



>> I use OpenRC, just so you know. I also have a motherboard with IOMMU
>> that is currently has questionable settings in the kernel config file. I
>> cannot find consensus if/how IOMMU that affects IO with the Sata HD
>> devices versus mm mapped peripherals.... in the context of 4.x kernel
>> options. I'm trying very hard here to avoid a deep dive on these issues,
>> so trendy strategies are most welcome, as workstation and cluster node
>> optimizations are all I'm really working on atm.
>
> Honestly, I'd suggest you deep dive. An image once, with clarity, will last
> you a lot longer than ongoing fuzzy and trendy images from people whose
> hardware and workflow is likely to be different from yours.
>
> The settings I provided should be absolutely fine for most use cases. Only
> exception would be mobile devices with spinning rust, but those are getting
> rarer and rarer...

I did a quick test with games-arcade/xgalaga. It's an old, quirky game 
with sporadic lag variations. On a workstation with 32G ram and (8) 4GHz 
64bit cores, very lightly loaded, there is no reason for in game lag. 
Your previous settings made it much better and quicker the vast majority 
of the time; but not optimal (always responsive). Experiences tell me if 
I can tweak a system so that that game stays responsive whilst the 
application(s) mix is concurrently running then the  quick 
test+parameter settings is reasonably well behaved. So thats becomes a 
baseline for further automated tests and fine tuning for a system under 
study.


Perhaps Zabbix +TSdB can get me further down the pathway.  Time 
sequenced and analyzed data is over kill for this (xgalaga) test, but 
those coalesced test-vectors  will be most useful for me as I seek a 
gentoo centric pathway for low latency clusters (on bare metal).

TIA,


James