StartMT (an idea)

[SvenBent]

Im wondering if anyoee would be intersted in making this idea come true

StartMT <Options> [Filelist] [Command]
Start commands for each files in multiple threads.

<Options> Optional
-t#     Set numbers of threads to use to # (default: Autodetects)
-n0     Dont use CPU0 (Enables -al)
-NoHT   Dont count Virtual cores on HT enabled processors (enables -al)
-al     Locks each thread to a cores (default: threads are not affinity locked
-s      Treverse Subdiretories

[Filelist] Specifies the the files to execute command one

{Command} Specifies what command to run on files
          use %file% in command line for file selection.
          use %thread% in command line to insert thread number

Example:
StartMT -n0 *.png Pngout %file% /kp
   runs pngout with parameter /kp on all *.png files. in a sepraet thread    for all cpu except core0

With this tool it would be easy to make simple file Command compression multit hreaded as StartMT.exe will autmatic execute the command in parralel and feed an individual filename from the filelist to each command
using above example command in the kodak suit it would execute these command in sequence

kodim01.png

pngout kodim01.png /kp
pngout kodim02.png /kp
pngout kodim03.png /kp
pngout kodim04.png /kp
pngout kodim05.png /kp
pngout kodim06.png /kp
pngout kodim07.png /kp
pngout kodim08.png /kp
pngout kodim09.png /kp
pngout kodim10.png /kp
pngout kodim11.png /kp
pngout kodim12.png /kp
pngout kodim13.png /kp

etc etc
but only enough at a time to fill out the current amount of cpu execution units exempt for core0 (set affinity for the individual thread to cpu 1 2 3 on a 4 core system)

The disabled HT option would set affinity on threads to each first cpu in a pairs e.g. 0 2 4 6 on a Quadcore cpu with ht (8 execution units, virtual cores0-7)

-al is the options to use affinity on each thread without it it woudl ust star the threads without affininty. Which is why is set by -n0 and -NoHT options
-s options will run the command and file in each sub directory


Offcouse thes options would be icing on the cake and the core functionality of easy multithreading would be nice.

anyway just an idea. i f i had the ability/skill i would have made it myself

[nburns]

This exists on *nix systems. It's called xargs. Doing a web search on xargs led to some other, similar programs that I didn't know about.

[Sebastian W]

For Windows I use PPX2.exe. The source code is available.
http://www.pirosa.co.uk/demo/wxargs/wxargs.html

You can use it like this:

dir *.png /B /O-S | ppx2.exe -P 8 -L 1 pngcompressor.bat "{}"

It does not handle filename-wildcards by itself. You can pipe any list of files or anything else, like in my example above. It is not aware of threads and cores and hyperthreding, like your suggestion is.

[SvenBent]

its the multithreading that I'm the most concerned about. Atm i have to rely and very ugly tricks in batch files like this

copy %1 %1.thread1.png
echo tag > %1.1.tag
echo for /l %%%%r in (1,1,%2) do ( >%1.pngmix1.bat
echo   pngout  %1.thread1.png %1.tmp1.png /r /force /f6 /kp >>%1.pngmix1.bat
echo   huffmix %1.thread1.png %1.tmp1.png %1.thread1.png  >>%1.pngmix1.bat
echo   del %1.tmp1.png  >>%1.pngmix1.bat
echo )  >>%1.pngmix1.bat
echo del %1.1.tag >>%1.pngmix1.bat
echo exit >>%1.pngmix1.bat

copy %1 %1.thread2.png
echo tag > %1.2.tag
echo for /l %%%%r in (1,1,%2) do ( >%1.pngmix2.bat
echo   pngout  %1.thread2.png %1.tmp2.png /r /force /f6 /kp >>%1.pngmix2.bat
echo   huffmix %1.thread2.png %1.tmp2.png %1.thread2.png  >>%1.pngmix2.bat
echo   del %1.tmp2.png  >>%1.pngmix2.bat
echo )  >>%1.pngmix2.bat
echo del %1.2.tag >>%1.pngmix2.bat
echo exit >>%1.pngmix2.bat

copy %1 %1.thread3.png
echo tag > %1.3.tag
echo for /l %%%%r in (1,1,%2) do ( >%1.pngmix3.bat
echo   pngout  %1.thread3.png %1.tmp3.png /r /force /f6 /kp >>%1.pngmix3.bat
echo   huffmix %1.thread3.png %1.tmp3.png %1.thread3.png  >>%1.pngmix3.bat
echo   del %1.tmp3.png  >>%1.pngmix3.bat
echo )  >>%1.pngmix3.bat
echo del %1.3.tag >>%1.pngmix3.bat
echo exit >>%1.pngmix3.bat

copy %1 %1.thread4.png
echo tag > %1.4.tag
echo for /l %%%%r in (1,1,%2) do ( >%1.pngmix4.bat
echo   pngout  %1.thread4.png %1.tmp4.png /r /force /f6 /kp >>%1.pngmix4.bat
echo   huffmix %1.thread4.png %1.tmp4.png %1.thread4.png  >>%1.pngmix4.bat
echo   del %1.tmp4.png  >>%1.pngmix4.bat
echo )  >>%1.pngmix4.bat
echo del %1.4.tag >>%1.pngmix4.bat
echo exit >>%1.pngmix4.bat

Start %1.pngmix1.bat %1 %2
Start %1.pngmix2.bat %1 %2
Start %1.pngmix3.bat %1 %2
Start %1.pngmix4.bat %1 %2

:keepwait
Ping 127.0.0.1
if exist %1.?.tag goto keepwait

huffmix %1 %1.thread1.png %1
huffmix %1 %1.thread2.png %1
huffmix %1 %1.thread3.png %1
huffmix %1 %1.thread4.png %1

del %1.thread?.png
del %1.pngmix?.bat

I have to spawn sub batch files and make small tag files to keep score if a thread is done or not.
ugly ping test as delays the loops to check if threads are done
etc etc etc

This is off cause different kind of multithreading as my batch multithreads on a single/same file.

[T. Hautamäki]

but only enough at a time to fill out the current amount of cpu execution units exempt for core0 (set affinity for the individual thread to cpu 1 2 3 on a 4 core system)

The disabled HT option would set affinity on threads to each first cpu in a pairs e.g. 0 2 4 6 on a Quadcore cpu with ht (8 execution units, virtual cores0-7)

-al is the options to use affinity on each thread without it it woudl ust star the threads without affininty. Which is why is set by -n0 and -NoHT options
-s options will run the command and file in each sub directory

I'm not sure if there's any significant advantage in manual thread handling. AFAIK, the task of assigning threads to physical/logical cores is pretty well handled by modern kernels. After all, you could have a PC with multiple HT cores in 2002.

Offcouse thes options would be icing on the cake and the core functionality of easy multithreading would be nice.

anyway just an idea. i f i had the ability/skill i would have made it myself

If a computational problem is embarassingly parallel, there's usually an easy way of making it run on all cores.

Unix has always been the more natural platform for doing parallel processing.

For example, if you have bash script with a serial loop:

for ((i=1 ; i<=$operations ; i++))
do
  do_something "$arg1" "$arg2"
  if [[ $? -gt 0 ]] ; then exit 1 ; fi
done

Then by using 'xargs' (or 'parallel'..) you can convert it into a parallel loop like this:

threads=$(grep -c ^processor /proc/cpuinfo)
procargs=""
for ((i=1 ; i<=$operations ; i++))
do
  procargs="$procargs \"$arg1\" \"$arg2\""
  let buffer=$i%$threads
  if [[ $buffer -eq 0 ]] || [[ $i -eq $operations ]] ; then
    echo $procargs | xargs --max-procs=$threads --max-args=2 do_something
    if [[ $? -gt 0 ]] ; then exit 1 ; fi
    procargs=""
  fi
done

Or you could write a suitable Makefile and use 'make -j'. In Unix/Linux there are many possibilities.

On the other hand, Windows has been developed starting from a single-user, single-core environment. Only recently has there been features introduced in Windows Powershell for running things in parallel. But then there's MinGW/MSYS..