Page 1 of 3 123 LastLast
Results 1 to 30 of 75

Thread: Internet: 0.99 MB/s (xDSL) and 5.85 MB/s (Cable/Fiber)

  1. #1
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts

    Internet: 0.99 MB/s (xDSL) and 5.85 MB/s (Cable/Fiber)

    When designing web-related compression systems, it can be useful to have a good understanding of internet broadband speeds. EU just published a study about this: https://ec.europa.eu/digital-agenda/...nd-services-eu

    Page 13, Key findings:

    """xDSL services averaged 8.27Mbps in Europe and 7.67Mbps in theUS. Cable services averaged 66.57Mbps in Europe and 25.48Mbps in the US. Thesame pattern was found for FTTx services too, with Europe averaging 53.09Mbpsand US achieving 41.35Mbps."""

    Averaged for US/Europe in MB/s (not Mbps):
    xDSL: 0.99 MB/s
    Cable and FTTx: 5.85 MB/s

  2. Thanks (5):

    Cyan (24th October 2015),inikep (28th January 2016),load (25th October 2015),nemequ (24th October 2015),Turtle (29th January 2016)

  3. #2
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Trying to make more concrete the (somewhat absurd) question 'how fast is the internet'.

    http://www.webpagetest.org/result/160128_85_95H/ shows that encode.su/threads/2313-Brotli/page3 loads 3.3 seconds, repeated views 2.5 seconds.

    Click image for larger version. 

Name:	encode-ru-speed-zpng.png 
Views:	209 
Size:	33.5 KB 
ID:	4044

    Speed Index does the same beautifully for a large group of webpages (for alexa top 100'000). https://sites.google.com/a/webpagete...cs/speed-index

    There, the slowest 5 % of web pages take more than 10 seconds on 5 Mbps Cable and more than 12 seconds 1.5 Mbps DSL.
    Last edited by Jyrki Alakuijala; 28th January 2016 at 10:02.

  4. #3
    Member
    Join Date
    Jul 2013
    Location
    United States
    Posts
    194
    Thanks
    44
    Thanked 140 Times in 69 Posts
    Sorry, I should have posted this a long time ago…

    While doing research for the (still in development) Squash Corpus, I came across the HTTP Archive, which contains a lot of very interesting data about the composition of the web. It's basically metadata from the archive.org project.

    There are a lot of charts, but IMHO they put the best one first. It's probably no surprised that images are the lion's share of the data, but I was definitely surprised that the average page contains more CSS data than HTML, and even more surprised that fonts are almost as much as HTML and CSS combined! And, of course, JS dwarfs all three (CSS, HTML, fonts) combined. That said, CSS, JS, and fonts (as well as images) are generally more cache-friendly than HTML, and are shared across multiple pages…

    Before focusing too heavily on JS, though, it's probably wise to keep the upcoming WebAssembly specification in mind. They are still working on the binary serialization, but it has the potential to change the composition of JS pretty drastically in a few years.

    There are some histograms later on in the page which show how common different size chunks of data are. For example, over half of the HTML requests are for ≤ 40 KiB, and the vast majority are under 200 KiB.

    And, if there is something you don't see in the graphs, the data is available for download or for use with Google BigQuery.

  5. Thanks (2):

    Cyan (29th January 2016),Turtle (29th January 2016)

  6. #4
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    'how fast is the internet'
    I'm starting to become more and more annoyed about Google forcing Internet "standards".

    First Google forced websites to enable compression and other tricks to make websites faster otherwise they lose their rankings. This was good for visitors who love speed but bad for the website server CPU use.

    Then Google forced websites to mobile designs otherwise they lose their rankings. Nice for mobile users but not for desktop/laptop users, font sizes to big for easy reading, text and menu structures do not fit and fall from the screen and are missing, information or options disappeared to make more space, indirect endless scrolling without next option to go to page xx, try to find the news messages of five days ago at a big news website, overall it's slower loading and navigating or even hanging sometimes.

    Last year Google started to force websites to HTTPS otherwise they lose their ranking. Nice for important websites and users who like WiFi or shared wired networks but not for people like me who like to type URLs in address bar and only retype last part and not nice for websites who are small (95%) and are hosted where installing certificates is not easy or costly. Most websites let not post info in their website at all or info what need to be secure, loading is much slower special when checking the certificate. Also webmasters need to rename all links in their website to HTTPS and external links are not working anymore without redirect what need technical knowledge and root access and then hope they keep their ranking.

    Now Google push slowly Brotli to the mass in web browsers, can we expect in the future again a move like when your website do not support the better Brotli compression you lose ranking?

    A funny thing, I remember that Matrixview showed 10 years ago to big companies their ABO compression for websites with speed/size data sheets for HTML what was much better then GZIP and it looked like nobody was interested.

  7. #5
    Member
    Join Date
    Jul 2013
    Location
    United States
    Posts
    194
    Thanks
    44
    Thanked 140 Times in 69 Posts
    Quote Originally Posted by Sportman View Post
    First Google forced websites to enable compression and other tricks to make websites faster otherwise they lose their rankings. This was good for visitors who love speed but bad for the website server CPU use.
    I don't see the problem. Google search results are about pointing people to the best resource to answer their query. If multiple sites are capable of answering the query then why shouldn't Google rank results by other criteria, in an effort to point me to the page with the best experience?

    Quote Originally Posted by Sportman View Post
    Then Google forced websites to mobile designs otherwise they lose their rankings. Nice for mobile users but not for desktop/laptop users, font sizes to big for easy reading, text and menu structures do not fit and fall from the screen and are missing, information or options disappeared to make more space, indirect endless scrolling without next option to go to page xx, try to find the news messages of five days ago at a big news website, overall it's slower loading and navigating or even hanging sometimes.
    I don't have a problem with this, either. It's possible to have a mobile-friendly design that doesn't cause the desktop version to suck. Sites which aren't mobile-friendly are difficult to deal with on mobile devices. All else being equal, I'd rather see the mobile-friendly site.

    Quote Originally Posted by Sportman View Post
    Last year Google started to force websites to HTTPS otherwise they lose their ranking. Nice for important websites and users who like WiFi or shared wired networks but not for people like me who like to type URLs in address bar and only retype last part and not nice for websites who are small (95%) and are hosted where installing certificates is not easy or costly. Most websites let not post info in their website at all or info what need to be secure, loading is much slower special when checking the certificate. Also webmasters need to rename all links in their website to HTTPS and external links are not working anymore without redirect what need technical knowledge and root access and then hope they keep their ranking.
    This time I completely disagree. TLS isn't just about providing privacy for your connection to an AP, it's about everything between the client and the server. That means not letting everyone between the client and server snoop on your traffic (ISPs, backbone providers, etc.).

    Perhaps even more important than confidentiality is integrity. TLS means no Verizon/AT&T supercookies, no ISPs injecting ads, no MITM JS DDoS attacks, etc.

    With Let's Encrypt you can easily get a free certificate (StartSSL has also been offering free certs for a while, though Let's Encrypt is much more streamlined), and with SNI shared hosting is no longer a barrier.

    Frankly, I'd like to see Google (and other search engines) "punish" sites for not supporting more things, like IPv6. Also, I'd love for a search engine to take into account how many ads and tracking services are present. I think it's pretty clear that Google will not be punishing sites for showing ads any time soon, but I can dream…

    Quote Originally Posted by Sportman View Post
    Now Google push slowly Brotli to the mass in web browsers, can we expect in the future again a move like when your website do not support the better Brotli compression you lose ranking?
    The point about Google pushing brotli out to everyone doesn't really interest me, but… I don't like how brotli is getting into browsers. IIUC it's required for WOFF2, so browsers need a brotli decoder anyways, so they figure they may as well support it as a content-encoding.

    I think we absolutely need to be thinking about alternatives to gzip, but we are getting ahead of ourselves with brotli. It's certainly one of the more interesting options, but I would like to see the IETF pick up the issue and pick 1 or 2 codecs; one highly asymmetric with a good compression ratio (like brotli or lzham), and geared towards more symmetrical and smaller use cases (somewhere in the lz4 or zstd range). The important point here is to take some time and open the discussion up a bit. Maybe someone with a proprietary codec (Oodle, nanozip, LZTurbo, etc.) would be willing to open it up a bit if they thought they could get it standardized and integrated into every major browser… Maybe someone will come up with something new. Maybe it would inspire some big companies to commit some devs to working on a couple codecs (look at what is happening with Daala, Thor, VP10, etc.).

    Quote Originally Posted by Sportman View Post
    A funny thing, I remember that Matrixview showed 10 years ago to big companies their ABO compression for websites with speed/size data sheets for HTML what was much better then GZIP and it looked like nobody was interested.
    AFAIK ABO was(is?) patented and proprietary. Brotli is open-source and… wow, okay, is there any info regarding patents? I'm pretty sure Google would be providing a patent grant (similar to VP9) if they thought they had a relevant patent, and would be disclosing it if they thought someone else did, but it would be nice if they made this explicit somewhere… Also, I think the timing is much better now than 10 years ago… there is a much bigger gap between deflate and modern codecs now than there was then.
    Last edited by nemequ; 29th January 2016 at 06:23. Reason: fix let's encrypt link

  8. #6
    Member
    Join Date
    Feb 2015
    Location
    United Kingdom
    Posts
    155
    Thanks
    20
    Thanked 66 Times in 37 Posts
    *in response to sportman
    I don't see how that's a bad thing, changing how webpages are processed and viewed for different devices does have its perks. As an end-user all I look forward to is fast, consistent internet connectivity, which Google has started offering. In comparison to the sly marketing techniques other ISP's use like labeling their speed in Mb rather than the expected MB, (in my opinion) I think Google is doing a fine job shaping the internet.

  9. #7
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by Sportman View Post
    This was good for visitors who love speed but bad for the website server CPU use.

    I'm not a webmaster, but my understanding of the internet business is that many websites have to pay for bandwidth. I have no real experience on the cost calculation, but I'm giving it a try. Some searching and back-of-the-envelope: The bandwidth cost can be about $100/Mbps per year (See https://blog.cloudflare.com/the-rela...und-the-world/). A server spitting out uncompressed data at a continuous rate of 100 MB/s, i.e., 1000 Mbps/per second generates a cost of $100000. Now, the server uses one of the cores to do 75 % compression, and spits out 25 MB/s, generating a cost of $25000. The cost of the one core for compression is about $500. This is an annual saving of $74500 for the website.


    Everyone is a winner in compression; user waits less, user pays less for data transfer, website pays less for data transfer. The biggest winner is the website.


    I am also annoyed that we can compress only HTTPS with brotli. What is your proposal for it?

  10. #8
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by nemequ View Post
    I think we absolutely need to be thinking about alternatives to gzip, but we are getting ahead of ourselves with brotli. It's certainly one of the more interesting options, but I would like to see the IETF pick up the issue and pick 1 or 2 codecs; one highly asymmetric with a good compression ratio (like brotli or lzham), and geared towards more symmetrical and smaller use cases (somewhere in the lz4 or zstd range).
    While brotli's design goal was not to be a symmetric format/codec, the new quality settings 0 and 1 are more symmetric. The new quality setting 0 is about 2x as fast in compression as the quality setting 1 was in the initial release. In a simple test over the 10 GB corpus brotli quality setting 0 is surprisingly symmetric (29 seconds for compression, 19 for decompression).

    See http://encode.su/threads/2313-Brotli...ll=1#post46303

    I'm excited to see more complete performance testing of the new quality 0 and quality 1 settings in the squash benchmark.

  11. Thanks:

    nemequ (30th January 2016)

  12. #9
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,505
    Thanks
    741
    Thanked 665 Times in 359 Posts
    Jyrki: sites may pay for traffic or bandwidth. the cheapest solution i've seen is just 3euro/200mbit, i.e. 1gbit will cost 180 euro per year. the largest prices are ~$100/TB for Amazon EC2, MS Azure and many CDN services. Since 1 Gbps == 4 PB/year, this means $400K/year. But these prices aren't for pure traffic but for services measured in the gigabytes processed, so they of course include compression, encryption and any other processing required

    PS: here you can buy 1 core for 12 euro/year. and here 1 core is 3 euro/year
    Last edited by Bulat Ziganshin; 30th January 2016 at 00:35.

  13. #10
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    Since 1 Gbps == 4 PB/year, this means $400K/year.
    I left out the 6x multiplier for Asia and Latin American, and the 20x multiplier for Australia. If you take those into my cost estimate, and get some part of the traffic from Asia, you are closer to $400k with that, too. Anyways, the cpu cost to compress data is extremely low in comparison to the bandwidth costs. In a simple economic estimate the external bandwidth is 100x to 1000x more expensive than the cpu use to consume that bandwidth.

  14. #11
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by nemequ View Post
    Brotli is open-source and… wow, okay, is there any info regarding patents? I'm pretty sure Google would be providing a patent grant (similar to VP9) if they thought they had a relevant patent, and would be disclosing it if they thought someone else did, but it would be nice if they made this explicit somewhere…
    https://datatracker.ietf.org/ipr/2396/ is "Google Inc.'s Statement about IPR related to draft-alakuijala-brotli-01"

  15. Thanks:

    nemequ (30th January 2016)

  16. #12
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    I'm Internet user since Trumpet Winsock was needed to login and since then I spend daily all my not sleeping, eating and sporting time online. I saw all the changes, first it wend to slow, then it accelerated and now it go to fast.

    The problem lay in backwards compatibility and at what point you want to mass promote new tech and to punish who did not upgrade.

    If you are a hosting provider and host 600-6000 websites at one server you try to host it as long as possible at the same old hardware. Server hardware accept consumer hard disks can easy run 5 years 24 hours without problems, a small change in CPU use for each website can have impact at all that hosted websites.

    Bandwidth was a big and expensive problem in the past, but not anymore for the average website or company. Ten years ago it started to become so cheap that you need to be a very large website or host large files or crawl the web at high rates or send a lot of data between internal networks via public networks before it started to become expensive.

    Mobile still was a problem till last year when prices for bandwidth use started to drop and this year even more. I don't know much about mobile, but maybe this high bandwidth prices where related with the maximum slots what can be used at the same time at a base station and how faster the data transfer how more mobile phones users can connect in the same time period.

    If you try to connect to an IPMI or storage card, power switch or a virtual machine management LAN interface you shall notice already quick you can't connect with modern browsers because browser only support new standards, or with one you can connect and a other not. You need to have installed Internet Explorer, Firefox, Chrome and an old version of Opera and try them one by one to get access. I do not talk about very old hardware but 2-5 years old. There are system administrators who think that their hardware web server is broken for long time and do not know it's their modern browser what's the problem.

    I visit once a while people to help them with computer troubles and they keep hardware even longer running, PC's and laptops can easy be 5-12 years old. Try to upgrade a Windows XP, Vista or Windows 7 OS with an IE 6,7,8 browser still installed to modern web browser. Most websites do not load anymore or partially, even not web browser download pages from big companies like Microsoft. This people can't upgrade even if they want and do not know websites do not load because they use software what do not support new standards, this people are not technical.

    Most website owners I know earn some money with ads, almost all use Google Adsense, to earn 500 dollar monthly there are round 3000 visitors a day needed where by most websites 20-60% of their visitors come from Google Search. When Google change their ranking to punish a website not upgrading to the latest standards that can cost serious money for those website owner, where most have no clue why it happened and no idea how to fix it, they are not technical at all, often good writers.

    A better compressor was nice in the ISDN line and the 500Kbit ADSL time, but now most have many Mbit speed and in my country 500Mbit up and down is rolled out to consumers for ADSL/VDSL prices and consumer cable had already till 200Mbit speed for the same prices. Ad blockers are installed by mass now and it looks like Google raised their click price in the same pace to not loss income. Companies can go bankrupt, individuals can die, I think important new standards need an independent and long term steady place where a decision is not based on corporate gain, but an improvement without to much hassle for everybody and can avoid a patent transfer and abuse by bankruptcy.

    I think as long not 40-50% use a new technology, old technology need still to be 100% supported. For websites and web browser I think 10-15 years backwards compatibility is a reasonable time to support old hardware and old software in their life span. There are enormous costs made only to do “forced” upgrade after upgrade for some improvements in speed, functionality or security.

    It's not good for a big company to let decisions be based by academic people only, ask plumbers what problems they face with their websites to be visible in a search engine and ask 55+ old people what problems they face when browsing the web.

    There are also other ways instead of punishing, Google could also not change the ranking and show a turtle symbol by slow websites and a hare symbol for fast websites and nothing for websites with average speed. The same is possible by showing a symbol for a mobile only and a symbol for a desktop only and nothing by a responsive design website. Users can then set a filter what they want to see as default or see all but can influence the order. Google could also make each day a copy of their whole web index at a fixed point in a day and keep that for ever, so people could search in an index of every date back in time they wish by selecting the date of the index they want to search in.
    Last edited by Sportman; 31st January 2016 at 05:08.

  17. #13
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by Sportman View Post
    Ten years ago it started to become so cheap that you need to be a very large website or host large files or crawl the web at high rates or send a lot of data between internal networks via public networks before it started to become expensive.
    Wouldn't both the bandwidth cost and the cpu cost scale roughly linearly together with the website size?

    Concrete example of a small website, with one query per second, and response size 100 kB: There is an average of 100 kB/s bandwidth need, i.e., you need 1 Mbps of internet bandwidth. According to Cloudflare's published bandwidth costs 1 mbps costs about $100 per year in the cheapest continents, and 6-20x more in the expensive continents. To compress 100 kB/s, you need 0.25 % of a core (to run a slow compression algorithm with 40 MB/s). 0.25 % of a core costs less than $100-2000 per year, so the small site is making savings, too.

  18. #14
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by nemequ View Post
    Before focusing too heavily on JS, though, it's probably wise to keep the upcoming WebAssembly specification in mind. They are still working on the binary serialization, but it has the potential to change the composition of JS pretty drastically in a few years.
    From the docs, WebAssembly's binary serialization has three layers:
    1. the binary format,
    2. polyfill (https://github.com/WebAssembly/desig...er/Polyfill.md),
    3. general purpose compression.


    The binary format is an encoding of 'abstract syntax tree nodes'. I think they don't explicitly state that the goal of WebAssembly is to be smaller than minified and compressed JavaScript. I'm not defending JavaScript, just that it is not yet easy to reason about the impact of WebAssembly.

  19. #15
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Wouldn't both the bandwidth cost and the cpu cost scale roughly linearly together with the website size?
    Yes, but mostly bandwidth increase do not come with a problem. For small websites shared hosting ($3-6) monthly data transfer allowed is often between 25-100GB, for medium shared hosting ($6-9) between 100-500GB and for large shared hosting ($9-14) from 500GB and above, or data transfer is flat and limited with a 10Gbit or 100Gbit connection. Dollar ranges are total hosting costs.

    Most websites are small and have less then 1000 visitors a day and under 25GB data transfer a month, the medium websites have 1k-10k visitors a day and the large websites above 10k visitors a day. On average each visitor use between 25-45KB so for 1000 visitors x 30 days x 45KB = 1350MB this is far under the limit they have.

    Quote Originally Posted by Jyrki Alakuijala View Post
    According to Cloudflare's published bandwidth costs 1 mbps costs about $100 per year in the cheapest continents.
    10 years ago 1Mbit prices where 3 dollar a Mbit wholesale so 36 dollar a year, now they are almost 10 times lower at 0.32 dollar a Mbit so 3.84 dollar a year. See Specials https://www.he.net/

    Shared hosting is often slow (high CPU load) because servers are loaded till the max with clients, cloud hosting is a solution but most website owners have no idea how to configure an OS, web server, script language and database etc. Hosting companies disable websites who are using to much CPU and enable them when client fixed it, often by disabling some Wordpress plugins.

    By moving an old server to a virtual server at new hardware ($$) high CPU use can be solved without much down time, this way buying a new OS license is not needed, but old operating systems and other software get a longer life span.

    There are many hosting servers still running under very old operating systems where installing a certificate can be a risky business and I do not know if it's possible to change compression in old web server software. There are also websites who can't enable compression because they use framework software to build their website what do not support it.

    Building a new website can be very slow for big companies or governments. If they do it themselves it can easy take 1-3 years, by a third party it can take much longer. First they need to find the right third party what can take up 1 year, then there is a transfer period from old third party to new third party from 1-2 years, then this new third party design a new platform part by part what can take 3-10 years till everything is replaced.
    Last edited by Sportman; 31st January 2016 at 17:29.

  20. #16
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by Sportman View Post
    Most websites are small and have less then 1000 visitors a day and under 25GB data transfer a month, the medium websites have 1k-10k visitors a day and the large websites above 10k visitors a day. On average each visitor use between 25-45KB so for 1000 visitors x 30 days x 45KB = 1350MB this is far under the limit they have.
    1350 MB/month is about 520 bytes per second. If a compression algorithm can handle 50 MB/s, half a kilobyte per second gives a requirement of 0.01 % of a core to handle the compression load. The website cannot afford the 0.01 % of a core? One core can run 10000 such websites?! If it is too expensive, they can compress 150-300 MB/s, and handle 30-60000 websites using one core for compression. (perhaps there is a mistake in my calculation)
    Last edited by Jyrki Alakuijala; 31st January 2016 at 19:45.

  21. #17
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by Sportman View Post
    On average each visitor use between 25-45KB so for 1000 visitors x 30 days x 45KB = 1350MB this is far under the limit they have.
    This is wrong, I used HTML only data.

    When I take a worst case total unoptimized rich media website with 5-15 images average each page and round 600 words text each page I get:

    280KB HTML (dynamic and static)
    120KB css and js scripts
    1150KB png, jpg, gif images
    --------
    1550KB each visitor

    1000 visitors x 30 days x 1550KB = 46.5GB this is above the small website traffic limit of 25GB, till 537 visitors it's allowed in this case.

    There are 350mln active domains, at least 99% of it has less then 300 visitors a day.

    With compression enabled the HTML, css and js scripts shall be smaller.

    I did a test with one rich media page request existing out 36 files and used gzip (normal mode) for every single file to see what compression do:

    50,868 bytes HTML - 10,865 bytes gzip
    167,187 bytes css and js scripts - 38,212 bytes gzip
    746,928 bytes png, jpg, gif, ico images - 725,391 bytes gzip
    ----------------
    926,771 bytes - 774,468 bytes gzip
    Last edited by Sportman; 1st February 2016 at 04:48.

  22. #18
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    The website cannot afford the 0.01 % of a core?
    I took today my oldest server:

    Intel Pentium III 550E 550 MHz
    2 x 256MB ECC SDRAM PC100 100MHz
    2 x 20GB harddisk RAID 1

    Microsoft Windows Server 2003 SP1
    IIS 6.0

    I connected it via 100Mbit ethernet to my fastest OC workstation and copied 7628 HTML pages to the web server.
    At the workstation I started a web crawler with one thread to get all 7628 pages behind each other.
    First test with compression disabled in IIS, second test with compression enabled in IIS, both test I cleaned temp files and reboot and wait some minutes before starting the test.

    Results:

    Test 1:
    264.515 sec. compression disabled
    CPU load round 10%
    Traffic round 7Mbit

    Test 2:
    308.548 sec. compression enabled
    CPU load round 70%
    Traffic round 6Mbit

    After test 2 the "IIS Temporary Compressed Files" folder contained 3,488 GZIP HTML files total 27,103,748 bytes

    Test 3 (repeat test 2 without clean and reboot):
    61.289 sec. compression enabled
    CPU load round 75%
    Traffic round 24Mbit

    After test 3 the "IIS Temporary Compressed Files" folder contained 4,762 GZIP HTML files total 36,839,673 bytes

    I think in test 2 IIS do not send compressed files yet only create them, in test 3 network sent bytes was 45% less then test 1 and test 2.

    Test 4 (repeat test 3 without clean and reboot):
    56.263 sec. compression enabled
    Did not made CPU/traffic graphs

    After test 4 the "IIS Temporary Compressed Files" folder contained 5,923 GZIP HTML files total 46,005,401 bytes

    Test 5 (repeat test 1 without clean and reboot):
    134.394 sec. compression disabled
    CPU load round 19%
    Traffic round 14Mbit

    Added 4 x CPU load, 4 x network traffic load.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	cpu-no-comp.jpg 
Views:	79 
Size:	73.9 KB 
ID:	4053   Click image for larger version. 

Name:	cpu-comp.jpg 
Views:	81 
Size:	81.5 KB 
ID:	4054   Click image for larger version. 

Name:	traffic-no-comp.jpg 
Views:	115 
Size:	67.1 KB 
ID:	4055   Click image for larger version. 

Name:	traffic-comp.jpg 
Views:	78 
Size:	72.3 KB 
ID:	4056   Click image for larger version. 

Name:	cpu-comp2.jpg 
Views:	78 
Size:	74.3 KB 
ID:	4057  

    Click image for larger version. 

Name:	traffic-comp2.jpg 
Views:	75 
Size:	61.6 KB 
ID:	4058   Click image for larger version. 

Name:	cpu-no-comp2.jpg 
Views:	71 
Size:	73.1 KB 
ID:	4059   Click image for larger version. 

Name:	traffic-no-comp2.jpg 
Views:	72 
Size:	65.6 KB 
ID:	4060  
    Last edited by Sportman; 2nd February 2016 at 17:22.

  23. #19
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by Sportman View Post
    I connected it via 100Mbit ethernet to my fastest OC workstation and copied 7628 HTML pages to the web server.
    At the workstation I started a web crawler with one thread to get all 7628 pages behind each other.
    First test with compression disabled in IIS, second test with compression enabled in IIS, both test I cleaned temp files and reboot and wait some minutes before starting the test.

    Results:

    Test 1:
    264.515 sec. compression disabled
    CPU load round 10%
    Traffic round 7Mbit

    Test 2:
    308.548 sec. compression enabled
    CPU load round 70%
    Traffic round 6Mbit
    Trying to understand your test setup: 7 Mbit for 7628 documents, it means 917 bytes per document. Is this correct?

    Your test 2 spends 308 seconds compressing 7 Mbit of documents, i.e., compression speed tops at 5 kbytes/sec/core:

    7 Mbit / 8 (bits/byte) / 308.548 / ((70 % core - 10 % core)/100 %) = ~4.7 kilobytes per second/core, correct?

  24. #20
    Member
    Join Date
    Jul 2013
    Location
    United States
    Posts
    194
    Thanks
    44
    Thanked 140 Times in 69 Posts
    FWIW, while this is interesting, I don't really think it is relevant to your point about Google penalizing sites for not using compression. Even discounting the fact that, it's not generally cost effective to keep using hardware that old (I just read this morning that the current useful window for data center hardware is 5 years).

    Responsiveness of the web site contributes to user experience, and for the *vast* majority of users it is better to receive compressed content because they can spare CPU cycles more than bandwidth. It is Google's job to recommend a site to the user which is relevant to their query, not to promote your web site, and I don't see a reason they shouldn't be taking user experience into account when making a recommendation.

    That said, from a technical standpoint, I like the discussion about performance on older CPUs

  25. #21
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by nemequ View Post
    Responsiveness of the web site contributes to user experience, and for the *vast* majority of users it is better to receive compressed content because they can spare CPU cycles more than bandwidth. It is Google's job to recommend a site to the user which is relevant to their query, not to promote your web site, and I don't see a reason they shouldn't be taking user experience into account when making a recommendation.
    My rule-of-thumb compression cost distribution:
    1 unit: cost of cpu, including power, ram, disk, housing, etc.
    100 units: cost of the data transfer (for xDSL/Cable/Fiber)
    10000 units: cost of human waiting for the data

    Almost the only thing that matters is the human waiting for the data. There are three main scenarios that can contribute to waiting: full compress-transfer-decode cycle, static data transfer and client-side decode, and a client-side decode from a client-side cached version. If we only maximize for compression density and almost ignore decoding speed, the last scenario can start to significantly contribute to waiting. Some resources are almost solely taken from the cache.

  26. #22
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by nemequ View Post
    It is Google's job to recommend a site to the user which is relevant to their query, not to promote your web site, and I don't see a reason they shouldn't be taking user experience into account when making a recommendation.
    I found this site about Google's ranking performance: https://sites.google.com/site/webmas...xing---ranking

  27. #23
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Trying to understand your test setup: 7 Mbit for 7628 documents, it means 917 bytes per document. Is this correct?
    No, that 7Mbit an 6Mbit I estimated from the graphs but looks like not far off:

    7Mbit x 1,024 = 7,168Kbit x 1,024 = 7,340,032 bits / 8 = 917,504 Bytes a second x 264.515 sec. = 242,693,571 bytes / 7,628 = 31,816 bytes per page.
    6Mbit x 1,024 = 6,144Kbit x 1,024 = 6,291,456 bits / 8 = 786,432 Bytes a second x 308.548 sec. = 242,652,021 bytes / 7,628 = 31,811 bytes per page.

    You are missing some info here I did not provided before, the crawler do something extra as crawling, it pack the crawled data in the end what took some time included in my timings from Acutimer. I just tested it and that part took 0.92 sec and the crawler must have other overhead too, so the size of the pages is smaller.

    I know for sure it's less then 233,089,000 bytes / 7,628 = 30,557 bytes per page average, the exact true size shall not be much smaller.

  28. #24
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Your test 2 spends 308 seconds compressing 7 Mbit of documents, i.e., compression speed tops at 5 kbytes/sec/core:

    7 Mbit / 8 (bits/byte) / 308.548 / ((70 % core - 10 % core)/100 %) = ~4.7 kilobytes per second/core, correct?
    No, I think this:

    During test 2 only 3,488 pages where compressed assume avg. 30,557 bytes = 3,488 x 30,557 = 106,582,816 bytes / 308.548 sec. = 345,434 bytes a sec. / (70% - 10%) = 60% / 100 = 575,723 bytes a sec a core by 100% compression CPU utilization.

    Not total accurate because it took longer with compression so crawling alone took probably less then 10%. The traffic graph show the real time server was busy serving pages and the CPU graph show the real time server was busy serving pages and compressing, I think horizontal the graph is 300 seconds also I/O looks like a bottle neck and can possible give CPU slow down.
    Last edited by Sportman; 2nd February 2016 at 05:07.

  29. #25
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by nemequ View Post
    Even discounting the fact that, it's not generally cost effective to keep using hardware that old
    This old 550Mhz 180nm CPU from begin 2000 has an TDP from only 14.5 Watt.

    I agree CPU's after this model with higher clock frequencies became power and airco power eaters, it can be cheaper to replace that hardware.

    Quote Originally Posted by nemequ View Post
    I like the discussion about performance on older CPUs
    I started this discussion because everything what a big company push have big consequences, same as the penalties. VGA video standard started in 1987 and only recent Intel removed VGA support out of their Skylake CPU's, that's almost 30 years VGA support. Google is more from penalizing, blocking and/or stopping support, what makes others doing the same, what speed up the process of not supporting old standards anymore.

    I think this is not a good idea because we face a future with electric cars and Internet of things, full of home automation where everything shall be connected to the Internet and using standards to get things from the Internet for device decision making.

    In the past you bought a car or thermostat and could use it 15-30 years, I think the most people never bought a new thermostat. Now they are connected and get a weather report, what if the standards change in some years and the connection fails. You can upgrade software, but I do not see manufacturers supporting their old hardware that long. After some years they probably already stopped selling that model you bought and advice you to buy a new model.
    Last edited by Sportman; 2nd February 2016 at 05:50.

  30. #26
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by Sportman View Post
    This old 550Mhz 180nm CPU from begin 2000 has an TDP from only 14.5 Watt.
    You can use an ARM board instead of this old machine, save energy and have more cpu power. An arm board costs less than a fraction of the energy cost for your current setup.

    Quote Originally Posted by Sportman View Post
    I started this discussion because everything what a big company push have big consequences, same as the penalties.
    I'm not experienced in Windows nor maintaining a website. Your topic would be a great discussion to have in the forum I linked in my previous post. You would hear what other webmasters think and how they see the ethics of the situation.

  31. #27
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    You can use an ARM board instead of this old machine, save energy and have more cpu power. An arm board costs less than a fraction of the energy cost for your current setup.
    I think I have one ARM Windows RT tablet somewhere.

    For compare I found a 32-bit benchmark http://www.primatelabs.com/geekbench2/ what was capable running at my oldest and newest server for compare tests and 15 years progress, I added speedup between brackets. Harddisk speed I tested with http://www.hdtune.com/ the free 12 February 2008 version:

    Intel Pentium III 550MHz - 1 core 1 thread
    TDP 14 Watt

    Geekbench 2 Score 310

    Integer 363
    Floating point 350
    Memory 207
    Stream 200

    Text compress single core 351 1.12 MB/sec
    Text decompress single core 399 1.64MB/sec

    Image compress single core 367 3.04 Mp/s
    Image decompress single core 350 5.88 Mp/s

    20GB harddisk:

    Avg. 18.8 MB/sec 14.0 ms, 22.3 MB/sec max

    ----------------------------------------------------------------

    2 x Intel Xeon v3 - 6 core 12 threads 3.4GHz 3.7GHz turbo (6.7x)
    TDP 2 x 135 Watt = 270 Watt (19.2x)

    Geekbench 2 Score 26458 (75.6x)

    Integer 32301 (89.0x)
    Floating point 38120 (108.9x)
    Memory 6277 (30.3x)
    Stream 5562 (27.81x)

    Text compress single core 3734 (10.6x) 11.9 MB/sec
    Text decompress single core 4349 (10.9x) 17.9 MB/sec

    Image compress single core 3339 (9.1x) 27.6 Mp/s
    Image decompress single core 3620 (10.3x) 60.8 Mp/s

    Text compress multi core 55321 (157.6x) 181 MB/sec
    Text decompress multi core 65990 (165.4x) 262 MB/sec

    Image compress multi core 51624 (140.7x) 434 Mp/s
    Image decompress multi core 39161 (111.9x) 639 Mp/s

    8TB (400x) harddisk:

    Avg. 195.5 MB/sec (10.4x) 8.7 ms, 217.2MB/s max

    Details tests GeekBench 2:
    http://support.primatelabs.com/kb/ge...h-2-benchmarks

    I you have a Brotli 32-bit compile what can run at this PIII and modern XEON CPU I can compare it against GZIP at both servers.
    Last edited by Sportman; 4th February 2016 at 17:13.

  32. #28
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    I have added test 5 what's a repeat of test 1 including CPU and traffic graphs. Disk and probably software cache give double transfer speed after next run with compression disabled, compression enabled give almost three times (2.8x) transfer speed over compression disabled in this test setup.

  33. #29
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by Sportman View Post
    I you have a Brotli 32-bit compile what can run at this PIII and modern XEON CPU I can compare it against GZIP at both servers.
    We only offer the compile-it-yourself solution from github, but it is two or three cmd-lines to get and compile.

    Geekbench 2 runs with bzip2. Gzip is around 10-20x faster. Why not try gzip directly?

  34. #30
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    788
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Why not try gzip directly?
    GZIP was already running, found a Brotli 32-bit (Bulat) compile from September 2015:

    Source same 7,628 HTML files one by one, some failed because space in file name.

    Old server PIII 550MHz dual 20GB disk RAID 1:
    bro 0: 306.974 sec
    gzip -1: 231.770 sec

    New server Xeon v3 3700MHz single 8TB disk:
    bro 0: 47.058 sec (6.5x)
    gzip -1: 33.706 sec (6.9x)

    Both values from second run (after disk/memory caching).
    Last edited by Sportman; 2nd February 2016 at 20:14.

Page 1 of 3 123 LastLast

Similar Threads

  1. Private Internet Access PIA VPN 403 error
    By cbloom in forum The Off-Topic Lounge
    Replies: 2
    Last Post: 3rd February 2015, 06:46

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •