Results 1 to 8 of 8

Thread: large pages

  1. #1
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,135
    Thanks
    320
    Thanked 1,397 Times in 802 Posts

    large pages

    On win10, Large Pages only work when
    1) Program executable is x64
    2) Runs under admin
    3) Policy allows large pages (“Computer Configuration” , “Windows Settings”, “Security Settings”, “Local Policies” , “User Rights Assignment”, “Lock Pages in memory”)
    4) Unfragmented 2MB pages physically exist in memory manager (ie test is done soon after reboot)

    Code:
    z:\021>2mpages.exe
    
    OpenProcessToken: <The operation completed successfully.  >
    LookupPrivilegeValue: <The operation completed successfully.  >
    AdjustTokenPrivileges: <The operation completed successfully.  >
    LPM.size=200000: 
    VirtualAlloc flags=20001000: <The operation completed successfully.  >
    p=00C00000 Flags=20001000
    
    Z:\021>timetest 7z a -mx1 -md27 -slp -mmt=1 1 "D:\000\enwik8"
    
    7-Zip 19.02 alpha (x64) : Copyright (c) 1999-2019 Igor Pavlov : 2019-09-05
    Archive size: 31739530 bytes (31 MiB)
    Tested program has wasted 12.563s
    
    Z:\021>timetest 7z a -mx1 -md27 -slp- -mmt=1 2 "D:\000\enwik8"
    
    7-Zip 19.02 alpha (x64) : Copyright (c) 1999-2019 Igor Pavlov : 2019-09-05
    Archive size: 31739530 bytes (31 MiB)
    Tested program has wasted 15.547s
    
    (15.547/12.563-1)*100 = 23.75%
    Attached Files Attached Files

  2. Thanks:

    Bulat Ziganshin (9th November 2020)

  3. #2
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    so what exactly does your utility do, sets OS to use 2mb pages?

  4. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,135
    Thanks
    320
    Thanked 1,397 Times in 802 Posts
    Yes, its tricky enough on windows - there's source in archive.

  5. #4
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    I`m not strong in coding - does app sets it until next reboot system wide?

  6. #5
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,135
    Thanks
    320
    Thanked 1,397 Times in 802 Posts
    > does app sets it until next reboot system wide

    That's impossible - well, Igor Pavlov had a project that attempted to do that - http://sevenmax.sourceforge.net/

    Normally 4k and 2M (and 1GB) virtual memory pages are managed separately, and you need a specific allocation method in each case.

  7. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,135
    Thanks
    320
    Thanked 1,397 Times in 802 Posts
    Here's a linux example.
    On linux though, its also necessary to force the system to actually allocate the pages (as root):
    Code:
    echo 16 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    ./2Mlin
    echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    #include <stdio.h>
    #include <string.h>

    #include <linux/mman.h>
    #include <sys/mman.h>
    #include <errno.h>

    int main( void ) {

    void* p = 0;
    int N = 16<<20;

    p = 0; errno=0;
    p = mmap( 0, N, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB|MAP_HUGE_2MB, 0, 0);
    printf( "!p = %p errno=%i <%s>!\n", p, errno, strerror(errno) );

    p = 0; errno=0;
    p = mmap( 0, N, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0);
    printf( "!p = %p errno=%i <%s>!\n", p, errno, strerror(errno) );

    }
    Attached Files Attached Files

  8. Thanks:

    Bulat Ziganshin (9th November 2020)

  9. #7
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    Sorry, still see no answer to what this utility does- 1. just shows info 2. sets OS to use 2mb pages 3. sets OS to use 2mb pages until reboot?

  10. #8
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,135
    Thanks
    320
    Thanked 1,397 Times in 802 Posts
    Current PC virtual memory architecture supports different page sizes - 4K,2M,1G; https://en.wikipedia.org/wiki/Page_(...age_table_size
    Large pages were added in hardware long after the memory management in popular operating systems was already established,
    so large pages are managed separately, and are not allocated automatically by default, eg. just because requested memory block is large enough.

    > 1. just shows info

    2mpages.cpp attempts to acquire permissions and allocate a memory block using large pages.
    It does print error messages for called winapi functions, so it can be said that it "shows info".

    2Mlin.cpp is just an example of memory allocation using large pages.

    > 2. sets OS to use 2mb pages

    That's currently impossible without using non-default OS memory manager.

    Linux example shows how to reserve large pages via device FS, so they'd stay reserved until next change.
    But even then linux won't automatically use them.

    > 3. sets OS to use 2mb pages until reboot?

    No, these pages have to be allocated in a special way, so I posted examples on how to do it.

  11. Thanks (2):

    Bulat Ziganshin (9th November 2020),necros (6th November 2020)

Similar Threads

  1. Large Text Compression Benchmark
    By CompressMaster in forum Data Compression
    Replies: 26
    Last Post: 17th February 2021, 00:25
  2. Compressing two (or more) large but similar files
    By nanoflooder in forum Data Compression
    Replies: 1
    Last Post: 2nd March 2017, 20:41
  3. Adler32 on Large Blocks
    By encode in forum Data Compression
    Replies: 1
    Last Post: 2nd January 2015, 22:38
  4. LZSS with a large dictionary
    By encode in forum Data Compression
    Replies: 31
    Last Post: 31st July 2008, 22:15
  5. Large text benchmark
    By Matt Mahoney in forum Forum Archive
    Replies: 39
    Last Post: 13th January 2008, 01:57

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •