Command Line Process Profiling Tool

[david_werecat]

Here is a simple process profiling tool I made. It shows process times (including user/kernel breakdown), peak memory usage (pool/working set/pagefile), page faults, process/thread ids and exit codes, and IO operation counts including total data transferred. The "byte" version shows memory and IO counts in bytes instead of kilobytes.

[Bulat Ziganshin]

thanks, but i ask you to change license to the Public Domain. and optionally push it to github, or i can do it

i want to further modify it providing output templates; also i want to build archiver benchmarking tool that will measure these parameters too

[david_werecat]

So I would just need to remove the GPL header to make the code public domain?

[Bulat Ziganshin]

i think it's better to write something like "Copyright (C) 2013 David Catt; placed into public domain"

so i would push it into github?

[david_werecat]

Yes, I don't have a github account.

[m^2]

"Copyright (C) 2013 David Catt; placed into public domain"
This is self contradictory. Public domain = no copyright.
"Written in 2013 by David Catt; placed into public domain"
would be more like it.

[Skymmer]

Thanks David!
I have a bug to report and a couple of suggestions.

1.) There is something wrong with ProcProfile64-Byte.exe. I tried to measure the 7z -m0=LZMA:a1:d30:mf=bt4 and got the following results.

Process ID       : 103988
Thread ID        : 104004
Process Exit Code: 0
Thread Exit Code : 0

User Time        :            389.468s
Kernel Time      :              2.843s
Process Time     :            392.311s
Clock Time       :            271.046s

Working Set      :         1739468800 bytes     its absolutely incorrect cause lzma allocates more than 10GB with d30 bt4
Paged Pool       :              84480 bytes
Nonpaged Pool    :               6208 bytes
Pagefile         :        -1579122688 bytes     negative value
Page Fault Count : 2516921

IO Read          :         1022457950 bytes (in             248 reads )
IO Write         :          131356710 bytes (in            2010 writes)
IO Other         :             212278 bytes (in             547 others)

ProcProfile64.exe shows the correct results.

Process ID       : 104396
Thread ID        : 104468
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         394.750s
Kernel Time      :           2.609s
Process Time     :         397.359s
Clock Time       :         273.812s

Working Set      :        10087308 KB
Paged Pool       :              82 KB
Nonpaged Pool    :               6 KB
Pagefile         :        11040800 KB
Page Fault Count : 2516921

IO Read          :          998494 KB (in             248 reads )
IO Write         :          128278 KB (in            2010 writes)
IO Other         :             207 KB (in             547 others)

2.) I think it would be better to rename Pagefile: to something else. Actually it shows the same value as Peak Private Bytes in Process Explorer and Task Manager. IMHO the Pagefile: can be confusing for some people.

3.) It would be just great if ProcProfile will be available in x86 too.

4.) For precesses with large pages turned on, ProcProfile shows incorrect Working Set value. Well, its overall problem with all timing tools I tested so far, so maybe its impossible at all due the nature of Windows internals. Please correct me if I wrong here.

5.) I would like to suggest the same alignment for all outputed stats. Something like this:

Process ID       : 104396
Thread ID        : 104468
Process Exit Code: 0
Thread Exit Code : 0

User Time        : 394.750s
Kernel Time      : 2.609s
Process Time     : 397.359s
Clock Time       : 273.812s

Working Set      : 10087308 KB
Paged Pool       : 82 KB
Nonpaged Pool    : 6 KB
Pagefile         : 11040800 KB
Page Fault Count : 2516921

IO Read          : 998494 KB (in 248 reads)
IO Write         : 128278 KB (in 2010 writes)
IO Other         : 207 KB (in 547 others)

In my opinion its a little bit more comfortable for reading and for parsing the log file in case of complex test runs.

6.) Options to control the amount of displayed (and collected) data.
We have 16 values: Process ID is the first, Process Exit Code is the third, and so on... So, if no options are given for ProcProfile then all values are measured and displayed. If for example -d8,14 option used, then only Clock Time and IO Read are measured. Such exclusion has not only the functional purpose but probably can increase the precision due to less number of counters invoked.
Of course in this case its good to have the -- option to stop processing options. So suggested syntax is: ProcProfile [options] <commandline>

7.) IMHO is not so good to provide a standalone Byte version. It would be nice to have a possibility to control the unit type. Something like: -b will show bytes, -m will show megabytes, kilobytes by default.

Well, thats all

[Bulat Ziganshin]

created https://github.com/Bulat-Ziganshin/proc-profile


1. works for me

C:\>ProcProfile64.exe 7z a m:\a z:\1g -mx -md1g -mmt2
....
Working Set      :         9871956 KB
Paged Pool       :             113 KB
Nonpaged Pool    :               8 KB
Pagefile         :        11041820 KB


C:\>ProcProfile64-Byte.exe 7z a m:\a z:\1g -mx -md1g -mmt2

7-Zip [64] 9.32 alpha  Copyright (c) 1999-2013 Igor Pavlov  2013-12-01
....
Working Set      :        10108936192 bytes
Paged Pool       :             116664 bytes
Nonpaged Pool    :               7976 bytes
Pagefile         :        11306840064 bytes

3. some counters are limited to 32-bit in x86 API, so the same command prints:

Working Set      :         4294967295 bytes
Paged Pool       :             116664 bytes
Nonpaged Pool    :               8096 bytes
Pagefile         :         4294967295 bytes

i've attached version with two minor bugfixes and x86 executables

[FatBit]

Dear Mr. Ziganshin,

something is wrong. I am not able to run programs ProcProfile 101 and Tornado 05a on Win XP SP3 Czech version. I obtained messages "It is not valid win32 application." Is it possible to fix?
Attachment 2603
Best regards,

FatBit

[david_werecat]

I just made a github account to continue development on ProcProfile. I have code that accepts command line arguments for different display units, but I can't commit it because my username needs commit privileges to the repository. My github username is davidcatt.

[Bulat Ziganshin]

David, i've added you to the list of collaboratoes, but it may be better to manage this repo in your account. at least if you want to further update it. so it will be your version and may be i will have my own differen version

FatBit, i was unable to compile the code using MSVC2010, so i used MSVC2012. may be it may be fixed in exe headers, may be some APIs are unsupported there - i don't know. Tornado was compiled just for Nania. i prefer now to publish only the fazip, is it not enough?

[david_werecat]

I've updated ProcProfile to version 1.1 which includes command line options for output units, process exit detection type, and left aligned output. I've includes MSVC2013 builds for both 32 and 64 bit.

[FatBit]

To Mr. Ziganshin: It is more than enough. When it was publicly downloadable… It was only info from user to author.
To Mr. Werecat: ProcProfile32 does not work under Win XP SP3 Czech 32 bit - "It is not valid win32 application."

Best regards,

FatBit

[Skymmer]

There is a serious problem I must say...

Bulat's version doesn't run on XP due to 0006 value of MajorOperatingSystemVersion and MajorSubsystemVersion in optional header. But after patching its appeared that:

The procedure entry point K32GetProcessMemoryInfo could not be located in the dynamic link library KERNEL32.dll

David's 1.1 version doesn't runs on XP too. Its also has the 0006 value but after patching we got the message that MSVCR120.dll is missing. Thats right - we have a dynamic compile here.
Redistributables can be downloaded from http://www.microsoft.com/en-us/downl....aspx?id=40784
But even with these redistrs we will finally got the same message:
The procedure entry point K32GetProcessMemoryInfo could not be located in the dynamic link library KERNEL32.dll

Its really dissapointing because ProcProfile has a rich functionallity and have a potential to become the best measurement tool in its class. But it will not be possible without XP support and static compiles I think.

[david_werecat]

Static compiles are easy. I'll work on a tool to change the executable headers to works on Windows XP. As for the issue with K32GetProcessMemoryInfo, I just need to add a define in the source file to fix it.

edit:
I've fixed the issues. Here are some compiles with the new fixes:

[david_werecat]

I've updated ProcProfile to version 1.2.1 which includes command line options for enabling individual statistic output types, including flags to quickly enable groups.

[Bulat Ziganshin]

what about win32/win64 problem? i think it may be fixed by providing double-mode executable, like DOS EXE stub in early win32 times. or it's impossible?

[Black_Fox]

Not sure if the issue was fixed - but it seems that K32GetProcessMemoryInfo is on XP called GetProcessMemoryInfo:
http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx

[Bulat Ziganshin]

it is detailed in Remarks on this page:

Programs that must run on earlier versions of Windows as well as Windows 7 and later versions should always call this function as GetProcessMemoryInfo. To ensure correct resolution of symbols, add Psapi.lib to the TARGETLIBS macro and compile the program with -DPSAPI_VERSION=1.

[FatBit]

Dear members of encode forum,

it looks like both ProcProfile32.exe from package without file version number (initial) and from version 121 work Win XP SP3 Czech 32 bit. Thank you.

Best regards,

FatBit

[Bulat Ziganshin]

a few more feature requests:

1. options for setting priority and affinity as in the "start" command: /low, /high...

2. special template for compressors benchmarking:

1,000,000,000 -> 252,474,982: 25.25% Cpu 6 mb/s (147.889 sec), real 11 mb/s (87.219 sec) = 170%

here the first two numbers are bytes in and out, respectively. then origsize=max(insize,outsize), compsize=min(insize,outsize), ratio=compsize/origsize, speeds are measured relative to origsize (not compsize!)

3. print large numbers with "," delimiter. i do it using the following procedure:

static char* show3 (uint64_t n, char *buf, const char *prepend=""){
    char *p = buf + 27+strlen(prepend);
    int i = 4;

    *p = '\0';
    do {
        if (!--i) *--p = ',', i = 3;
        *--p = '0' + (n % 10);
    } while (n /= 10);

    memcpy (p-strlen(prepend), prepend, strlen(prepend));
    return p-strlen(prepend);
}


Usage:
  char temp1[100], temp2[100];
  fprintf(stderr, "Compressed %s -> %s bytes = %.2lf%%\n", show3(origsize,temp1), show3(compsize,temp2), (double(compsize)/origsize)*100);

[david_werecat]

I've updated ProcProfile to version 1.3. It now supports the compression benchmarking template and has the ability to specify the priority of the process.

[Bulat Ziganshin]

thank you very much!!! it's almost ideal for me!

[Edison007]

thank.s
1. Bad, that when use option "-tb", not work option "-r...".
2. What about, always show speed in kB/s?

[david_werecat]

I've updated ProcProfile to version 1.3.1. The -b, -k and -m options now set the speed unit for the compression benchmark template and the -a option now allows the processor affinity mask of the process to be specified.

[david_werecat]

I've updated ProcProfile to version 1.3.2. It is now possible to select multiple templates for output.

[david_werecat]

I've update ProcProfile to version 1.3.3. There is now a template for xml output.

[david_werecat]

Here are two tools that I made for modifying executables to run on XP or making an executable large address aware. These tools are related to ProcProfile in that I use the MakeXP program on the executables posted in this thread.

[Bulat Ziganshin]

thanks for LX utility. although the second problem solved by MSVC utility: EDITBIN /LARGEADDRESSAWARE[:NO] [files], or just by providing /LARGEADDRESSAWARE (or -large-address-aware for gcc) option to the linker

[Bulat Ziganshin]

a few more crazy ideas:
1. update stats while program runs. i.e. display the current live stats. it looks especially useful with "compression benchmarking" template, but may be also good with multiline output, i.e. entire screen will be refreshed

2. use some AI and win apis to detect input and output files and display only their stats. also it will allow to display "% done" indicator, in particular to the windows taskbar

2a. as simpler variant, provide option to specify input size/input file and display "%done" comparing bytes read to this file size; as well as cmdline template to substitute this file name:
"procfile -fz:\4g fazip lzma {-f} nul" executes "fazip lzma z:\4g nul" and displays progress indicator comparing bytes read so far with the z:\4g size

3. add to "compression benchmarking" template memories like 7z 9.32 does. i.e. it may be virtual memory used or real memory usage, or both