Page 1 of 2 12 LastLast
Results 1 to 30 of 39

Thread: Command Line Process Profiling Tool

  1. #1
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts

    Command Line Process Profiling Tool

    Here is a simple process profiling tool I made. It shows process times (including user/kernel breakdown), peak memory usage (pool/working set/pagefile), page faults, process/thread ids and exit codes, and IO operation counts including total data transferred. The "byte" version shows memory and IO counts in bytes instead of kilobytes.
    Attached Files Attached Files
    Last edited by david_werecat; 6th December 2013 at 16:27.

  2. Thanks (2):

    Bulat Ziganshin (6th December 2013),Sebastian W (6th December 2013)

  3. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    thanks, but i ask you to change license to the Public Domain. and optionally push it to github, or i can do it

    i want to further modify it providing output templates; also i want to build archiver benchmarking tool that will measure these parameters too

  4. #3
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    So I would just need to remove the GPL header to make the code public domain?

  5. #4
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    i think it's better to write something like "Copyright (C) 2013 David Catt; placed into public domain"

    so i would push it into github?

  6. #5
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    Yes, I don't have a github account.

  7. #6
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,610
    Thanks
    30
    Thanked 65 Times in 47 Posts
    "Copyright (C) 2013 David Catt; placed into public domain"
    This is self contradictory. Public domain = no copyright.
    "Written in 2013 by David Catt; placed into public domain"
    would be more like it.

  8. Thanks:

    Bulat Ziganshin (6th December 2013)

  9. #7
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    Thanks David!
    I have a bug to report and a couple of suggestions.

    1.) There is something wrong with ProcProfile64-Byte.exe. I tried to measure the 7z -m0=LZMA:a1:d30:mf=bt4 and got the following results.
    Code:
    Process ID       : 103988
    Thread ID        : 104004
    Process Exit Code: 0
    Thread Exit Code : 0
    
    User Time        :            389.468s
    Kernel Time      :              2.843s
    Process Time     :            392.311s
    Clock Time       :            271.046s
    
    Working Set      :         1739468800 bytes     its absolutely incorrect cause lzma allocates more than 10GB with d30 bt4
    Paged Pool       :              84480 bytes
    Nonpaged Pool    :               6208 bytes
    Pagefile         :        -1579122688 bytes     negative value
    Page Fault Count : 2516921
    
    IO Read          :         1022457950 bytes (in             248 reads )
    IO Write         :          131356710 bytes (in            2010 writes)
    IO Other         :             212278 bytes (in             547 others)
    ProcProfile64.exe shows the correct results.
    Code:
    Process ID       : 104396
    Thread ID        : 104468
    Process Exit Code: 0
    Thread Exit Code : 0
    
    User Time        :         394.750s
    Kernel Time      :           2.609s
    Process Time     :         397.359s
    Clock Time       :         273.812s
    
    Working Set      :        10087308 KB
    Paged Pool       :              82 KB
    Nonpaged Pool    :               6 KB
    Pagefile         :        11040800 KB
    Page Fault Count : 2516921
    
    IO Read          :          998494 KB (in             248 reads )
    IO Write         :          128278 KB (in            2010 writes)
    IO Other         :             207 KB (in             547 others)
    2.) I think it would be better to rename Pagefile: to something else. Actually it shows the same value as Peak Private Bytes in Process Explorer and Task Manager. IMHO the Pagefile: can be confusing for some people.

    3.) It would be just great if ProcProfile will be available in x86 too.

    4.) For precesses with large pages turned on, ProcProfile shows incorrect Working Set value. Well, its overall problem with all timing tools I tested so far, so maybe its impossible at all due the nature of Windows internals. Please correct me if I wrong here.

    5.) I would like to suggest the same alignment for all outputed stats. Something like this:
    Code:
    
    Process ID       : 104396
    Thread ID        : 104468
    Process Exit Code: 0
    Thread Exit Code : 0
    
    User Time        : 394.750s
    Kernel Time      : 2.609s
    Process Time     : 397.359s
    Clock Time       : 273.812s
    
    Working Set      : 10087308 KB
    Paged Pool       : 82 KB
    Nonpaged Pool    : 6 KB
    Pagefile         : 11040800 KB
    Page Fault Count : 2516921
    
    IO Read          : 998494 KB (in 248 reads)
    IO Write         : 128278 KB (in 2010 writes)
    IO Other         : 207 KB (in 547 others)
    In my opinion its a little bit more comfortable for reading and for parsing the log file in case of complex test runs.

    6.) Options to control the amount of displayed (and collected) data.
    We have 16 values: Process ID is the first, Process Exit Code is the third, and so on... So, if no options are given for ProcProfile then all values are measured and displayed. If for example -d8,14 option used, then only Clock Time and IO Read are measured. Such exclusion has not only the functional purpose but probably can increase the precision due to less number of counters invoked.
    Of course in this case its good to have the -- option to stop processing options. So suggested syntax is: ProcProfile [options] <commandline>

    7.) IMHO is not so good to provide a standalone Byte version. It would be nice to have a possibility to control the unit type. Something like: -b will show bytes, -m will show megabytes, kilobytes by default.

    Well, thats all

  10. Thanks (2):

    Bulat Ziganshin (7th December 2013),david_werecat (8th December 2013)

  11. #8
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    created https://github.com/Bulat-Ziganshin/proc-profile


    1. works for me
    Code:
    C:\>ProcProfile64.exe 7z a m:\a z:\1g -mx -md1g -mmt2
    ....
    Working Set      :         9871956 KB
    Paged Pool       :             113 KB
    Nonpaged Pool    :               8 KB
    Pagefile         :        11041820 KB
    
    
    C:\>ProcProfile64-Byte.exe 7z a m:\a z:\1g -mx -md1g -mmt2
    
    7-Zip [64] 9.32 alpha  Copyright (c) 1999-2013 Igor Pavlov  2013-12-01
    ....
    Working Set      :        10108936192 bytes
    Paged Pool       :             116664 bytes
    Nonpaged Pool    :               7976 bytes
    Pagefile         :        11306840064 bytes

    3. some counters are limited to 32-bit in x86 API, so the same command prints:
    Code:
    Working Set      :         4294967295 bytes
    Paged Pool       :             116664 bytes
    Nonpaged Pool    :               8096 bytes
    Pagefile         :         4294967295 bytes


    i've attached version with two minor bugfixes and x86 executables
    Attached Files Attached Files
    Last edited by Bulat Ziganshin; 7th December 2013 at 18:00.

  12. Thanks:

    Skymmer (7th December 2013)

  13. #9
    Member FatBit's Avatar
    Join Date
    Jan 2012
    Location
    Prague, CZ
    Posts
    191
    Thanks
    0
    Thanked 36 Times in 27 Posts
    Dear Mr. Ziganshin,

    something is wrong. I am not able to run programs ProcProfile 101 and Tornado 05a on Win XP SP3 Czech version. I obtained messages "It is not valid win32 application." Is it possible to fix?
    Attachment 2603
    Best regards,

    FatBit

  14. #10
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    I just made a github account to continue development on ProcProfile. I have code that accepts command line arguments for different display units, but I can't commit it because my username needs commit privileges to the repository. My github username is davidcatt.

  15. #11
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    David, i've added you to the list of collaboratoes, but it may be better to manage this repo in your account. at least if you want to further update it. so it will be your version and may be i will have my own differen version

    FatBit, i was unable to compile the code using MSVC2010, so i used MSVC2012. may be it may be fixed in exe headers, may be some APIs are unsupported there - i don't know. Tornado was compiled just for Nania. i prefer now to publish only the fazip, is it not enough?
    Last edited by Bulat Ziganshin; 7th December 2013 at 22:35.

  16. #12
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    I've updated ProcProfile to version 1.1 which includes command line options for output units, process exit detection type, and left aligned output. I've includes MSVC2013 builds for both 32 and 64 bit.
    Attached Files Attached Files

  17. #13
    Member FatBit's Avatar
    Join Date
    Jan 2012
    Location
    Prague, CZ
    Posts
    191
    Thanks
    0
    Thanked 36 Times in 27 Posts
    To Mr. Ziganshin: It is more than enough. When it was publicly downloadable… It was only info from user to author.
    To Mr. Werecat: ProcProfile32 does not work under Win XP SP3 Czech 32 bit - "It is not valid win32 application."

    Best regards,

    FatBit

  18. #14
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    There is a serious problem I must say...

    Bulat's version doesn't run on XP due to 0006 value of MajorOperatingSystemVersion and MajorSubsystemVersion in optional header. But after patching its appeared that:
    Code:
    The procedure entry point K32GetProcessMemoryInfo could not be located in the dynamic link library KERNEL32.dll
    David's 1.1 version doesn't runs on XP too. Its also has the 0006 value but after patching we got the message that MSVCR120.dll is missing. Thats right - we have a dynamic compile here.
    Redistributables can be downloaded from http://www.microsoft.com/en-us/downl....aspx?id=40784
    But even with these redistrs we will finally got the same message:
    The procedure entry point K32GetProcessMemoryInfo could not be located in the dynamic link library KERNEL32.dll

    Its really dissapointing because ProcProfile has a rich functionallity and have a potential to become the best measurement tool in its class. But it will not be possible without XP support and static compiles I think.
    Last edited by Skymmer; 8th December 2013 at 00:08.

  19. #15
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    Static compiles are easy. I'll work on a tool to change the executable headers to works on Windows XP. As for the issue with K32GetProcessMemoryInfo, I just need to add a define in the source file to fix it.

    edit:
    I've fixed the issues. Here are some compiles with the new fixes:
    Attached Files Attached Files
    Last edited by david_werecat; 8th December 2013 at 01:34.

  20. Thanks:

    Bulat Ziganshin (8th December 2013)

  21. #16
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    I've updated ProcProfile to version 1.2.1 which includes command line options for enabling individual statistic output types, including flags to quickly enable groups.
    Attached Files Attached Files
    Last edited by david_werecat; 8th December 2013 at 07:06.

  22. #17
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    what about win32/win64 problem? i think it may be fixed by providing double-mode executable, like DOS EXE stub in early win32 times. or it's impossible?

  23. #18
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Not sure if the issue was fixed - but it seems that K32GetProcessMemoryInfo is on XP called GetProcessMemoryInfo:
    http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  24. #19
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    it is detailed in Remarks on this page:

    Programs that must run on earlier versions of Windows as well as Windows 7 and later versions should always call this function as GetProcessMemoryInfo. To ensure correct resolution of symbols, add Psapi.lib to the TARGETLIBS macro and compile the program with -DPSAPI_VERSION=1.

  25. #20
    Member FatBit's Avatar
    Join Date
    Jan 2012
    Location
    Prague, CZ
    Posts
    191
    Thanks
    0
    Thanked 36 Times in 27 Posts
    Dear members of encode forum,

    it looks like both ProcProfile32.exe from package without file version number (initial) and from version 121 work Win XP SP3 Czech 32 bit. Thank you.

    Best regards,

    FatBit

  26. #21
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    a few more feature requests:

    1. options for setting priority and affinity as in the "start" command: /low, /high...

    2. special template for compressors benchmarking:

    1,000,000,000 -> 252,474,982: 25.25% Cpu 6 mb/s (147.889 sec), real 11 mb/s (87.219 sec) = 170%

    here the first two numbers are bytes in and out, respectively. then origsize=max(insize,outsize), compsize=min(insize,outsize), ratio=compsize/origsize, speeds are measured relative to origsize (not compsize!)

    3. print large numbers with "," delimiter. i do it using the following procedure:
    Code:
    static char* show3 (uint64_t n, char *buf, const char *prepend=""){
        char *p = buf + 27+strlen(prepend);
        int i = 4;
    
        *p = '\0';
        do {
            if (!--i) *--p = ',', i = 3;
            *--p = '0' + (n % 10);
        } while (n /= 10);
    
        memcpy (p-strlen(prepend), prepend, strlen(prepend));
        return p-strlen(prepend);
    }
    
    
    Usage:
      char temp1[100], temp2[100];
      fprintf(stderr, "Compressed %s -> %s bytes = %.2lf%%\n", show3(origsize,temp1), show3(compsize,temp2), (double(compsize)/origsize)*100);
    Last edited by Bulat Ziganshin; 9th December 2013 at 15:06.

  27. Thanks:

    david_werecat (9th December 2013)

  28. #22
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    I've updated ProcProfile to version 1.3. It now supports the compression benchmarking template and has the ability to specify the priority of the process.
    Attached Files Attached Files

  29. Thanks (2):

    Bulat Ziganshin (9th December 2013),Edison007 (9th December 2013)

  30. #23
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    thank you very much!!! it's almost ideal for me!

  31. #24
    Member
    Join Date
    Nov 2011
    Location
    Estonia
    Posts
    9
    Thanks
    13
    Thanked 7 Times in 4 Posts
    thank.s
    1. Bad, that when use option "-tb", not work option "-r...".
    2. What about, always show speed in kB/s?

  32. #25
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    I've updated ProcProfile to version 1.3.1. The -b, -k and -m options now set the speed unit for the compression benchmark template and the -a option now allows the processor affinity mask of the process to be specified.
    Attached Files Attached Files

  33. Thanks (2):

    Bulat Ziganshin (9th December 2013),Edison007 (9th December 2013)

  34. #26
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    I've updated ProcProfile to version 1.3.2. It is now possible to select multiple templates for output.
    Attached Files Attached Files

  35. Thanks:

    Bulat Ziganshin (9th December 2013)

  36. #27
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    I've update ProcProfile to version 1.3.3. There is now a template for xml output.
    Attached Files Attached Files

  37. Thanks:

    Skymmer (10th December 2013)

  38. #28
    Member
    Join Date
    Aug 2011
    Location
    Canada
    Posts
    113
    Thanks
    9
    Thanked 22 Times in 15 Posts
    Here are two tools that I made for modifying executables to run on XP or making an executable large address aware. These tools are related to ProcProfile in that I use the MakeXP program on the executables posted in this thread.
    Attached Files Attached Files

  39. Thanks (2):

    adends (3rd March 2019),Bulat Ziganshin (11th December 2013)

  40. #29
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    thanks for LX utility. although the second problem solved by MSVC utility: EDITBIN /LARGEADDRESSAWARE[:NO] [files], or just by providing /LARGEADDRESSAWARE (or -large-address-aware for gcc) option to the linker

  41. #30
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    a few more crazy ideas:
    1. update stats while program runs. i.e. display the current live stats. it looks especially useful with "compression benchmarking" template, but may be also good with multiline output, i.e. entire screen will be refreshed

    2. use some AI and win apis to detect input and output files and display only their stats. also it will allow to display "% done" indicator, in particular to the windows taskbar

    2a. as simpler variant, provide option to specify input size/input file and display "%done" comparing bytes read to this file size; as well as cmdline template to substitute this file name:
    "procfile -fz:\4g fazip lzma {-f} nul" executes "fazip lzma z:\4g nul" and displays progress indicator comparing bytes read so far with the z:\4g size

    3. add to "compression benchmarking" template memories like 7z 9.32 does. i.e. it may be virtual memory used or real memory usage, or both

Page 1 of 2 12 LastLast

Similar Threads

  1. Replies: 39
    Last Post: 10th April 2014, 22:26
  2. Replies: 2
    Last Post: 23rd February 2013, 06:01
  3. Replies: 4
    Last Post: 2nd December 2012, 02:55
  4. command-line calculator for Windows?
    By Alexander Rhatushnyak in forum The Off-Topic Lounge
    Replies: 5
    Last Post: 29th January 2012, 23:40
  5. DARK - a new BWT-based command-line archiver
    By encode in forum Forum Archive
    Replies: 138
    Last Post: 23rd September 2006, 21:42

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •