Gentoo Archives: gentoo-user

From: Dale <rdalek1967@×××××.com>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] PCIe x1 or PCIe x4 SATA controller card
Date: Mon, 13 Mar 2023 14:35:01
Message-Id: b779f1ba-273c-2a30-9774-bd0c44c0e08c@gmail.com
In Reply to: Re: [gentoo-user] PCIe x1 or PCIe x4 SATA controller card by Rich Freeman
1 Rich Freeman wrote:
2 > On Mon, Mar 13, 2023 at 8:24 AM Dale <rdalek1967@×××××.com> wrote:
3 >> According to my google searches, PCIe x4 is faster
4 >> than PCIe x1. It's why some cards are PCIe x8 or x16. I think video
5 >> cards are usually x16. My question is, given the PCIe x4 card costs
6 >> more, is it that much faster than a PCIe x1?
7 > It could be slower than PCIe x1, because you didn't specify the version.
8 >
9 > PCIe uses lanes. Each lane provides a certain amount of bandwidth
10 > depending on the version in use.
11 >
12 > For example, a v1 4x card has 1 GB/s of bandwidth. A v4 1x card has
13 > 2GB/s of bandwidth.
14 >
15 > Note that slot size is only loosely coupled with the number of lanes.
16 > Lots of motherboards have a second 16x slot that only provides 4-8
17 > lanes to save on the cost of a PCIe swich. You can also use adapters
18 > to connect a 16x card to a 1x slot, or you might find a motherboard
19 > that has an open-ended slot so that you can just fit a 16x card onto
20 > the 1x slot. It will of course only use a single lane that way.
21 >
22 > So what you need to do is consider the following:
23 >
24 > 1. How much bandwidth do you actually need? If you're using spinning
25 > disks you aren't going to sustain more than 200MB/s to a single drive,
26 > and the odds of having 10 drives using all that bandwidth are pretty
27 > low. If you're using SSDs then you're more likely to max them out
28 > since the seek cost is much lower.
29 > 2. What PCIe version does your motherboard support? Sticking a v4
30 > card on an old motherboard that only supports v2 is going to result in
31 > it running at v2 speeds, so don't pay a premium for something you
32 > won't use. Likewise, if they cut down on the number of lanes assuming
33 > they'll have more bandwidth you might have less than you expected to
34 > have.
35 >
36 > Then look up the number of lanes and the PCIe version and see what you
37 > can expect:
38 > https://en.wikipedia.org/wiki/PCI_Express#History_and_revisions
39 >
40 > I think odds are you aren't going to want to pay a premium if you're
41 > just using spinning disks. If you actually wanted solid state storage
42 > then I'd also be avoiding SATA and trying to use NVMe, though doing
43 > that at scale requires a lot of IO, and that will cost you quite a
44 > bit. There is a reason your motherboard has mostly 1x slots - PCIe
45 > lanes are expensive to support. On most consumer motherboards they're
46 > only handled by the CPU, and consumer CPUs are very limited in how
47 > many they offer. Higher end motherboards may have a switch and offer
48 > more lanes, but they'll still bottleneck if they're all maxed out
49 > getting into the CPU. If you buy a server CPU for several thousand
50 > dollars one of the main features they offer is a LOT more PCIe lanes,
51 > so you can load up on NVMes and have them running at v4-5. (Typical
52 > NVMe uses a 4x M.2 slot, and of course you can have 16x cards offering
53 > multiples of those.)
54 >
55 > The whole setup is pretty analogous to networking. If you have a
56 > computer with 4 network ports you can bond them together and run them
57 > to a switch that supports this with 4 cables, and get 4x the
58 > bandwidth. However, you can also get a single connection to run at
59 > higher speeds (1Gb, 2.5Gb, 10Gb, etc), and you can do both. PCIe
60 > lanes are just like bonded network cables - they are just pairs of
61 > signal wires that use differential signaling, just like twisted pairs
62 > in an ethernet cable. Longer slots just add more of them. Everything
63 > is packet switched, so if there are more lanes it just spreads the
64 > packets across them. Higher versions mean higher speeds in each lane.
65 >
66
67
68 This is why I asked.  I didn't even think about the different PCIe
69 versions available.  My mobo, had to go dig out the manual, says it is
70 PCIe 2.0.  This is a Gigabyte 970A-UD3P motherboard.  Yes, I'm thinking
71 about building a new rig.  Turn this into a NAS maybe.  Anyway, I'm
72 assuming 2.0 isn't the slowest or fastest but as you point out, it'll be
73 a bottleneck.  Everything has a bottleneck somewhere. 
74
75 The difference in price isn't that large.  What I did was, I saw the
76 PCIe x1 and bought it.  It supported Linux.  Later on I noticed the PCIe
77 x4 and then wondered if I should upgrade to that.  Given the limits of
78 my mobo and the fact I won't be maxing out the drives anyway, I don't
79 see the need to upgrade.  Your info pretty much makes that clear.  I
80 might, just might, see a small difference when using pvmove.  Maybe. 
81 Given that it generally maxes out the drive as it is, even if it does go
82 faster, it won't be much.  Add in that when I start a pvmove, I go nap
83 and do other things anyway, I won't notice it.  Last pvmove took a
84 little over 19 hours.  Even 20 or 30 minutes isn't much difference in
85 the grand scheme of things. 
86
87 Another question.  My rig is getting a bit aged.  I have a AMD FX-8350 8
88 core CPU running at 4GHz.  I also have 32GBs of memory.  I've read that
89 Intel currently has the best bang for buck on CPUs nowadays.  I'm open
90 to the idea of switching.  As far as speed goes, if I built a new rig
91 that is using a reasonably cost CPU and memory, would I see any real
92 improvements?  When I say 'reasonably cost', I usually find the
93 fastest/newest then drop down a bit to get out of that 'brand new' price
94 point.  Generally, the difference in price is quite large but the
95 difference in speed isn't that much.  Also, I got hard drives, I don't
96 spend much on video cards either since I don't game, except solitaire. 
97 So, let's say a mobo, CPU and memory.  What price range would I need to
98 look for?  Just a rough idea.  I figure the CPU will be a few hundred. 
99 Memory may be half that.  Mobo will likely be close to $200 or so.  I'm
100 thinking $500 to $700 or so.  Then comes case, video card and all that. 
101 The CPU, memory and mobo is the ones I try to buy all at once from the
102 same vendor, like Newegg or Tigerdirect.  Thoughts?  Am I close?
103
104 Thanks for the info.  At least I know I'm good on drive speed.  For my
105 use anyway. 
106
107 Dale
108
109 :-)  :-) 

Replies

Subject Author
Re: [gentoo-user] PCIe x1 or PCIe x4 SATA controller card Mark Knecht <markknecht@×××××.com>