Page 2 of 2 FirstFirst 12
Results 31 to 59 of 59

Thread: CHK v1.03 is here!

  1. #31
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    I haven't tried the buf[ptr]^=ch optimization yet. I didn't use it in my first version because I wanted the code to be platform independent. But in the SSSE3 version I guess there's no reason not to.

    I would say that SHA-1 is weak, not broken. So far nobody has found a collision yet. The best known attack would require 2^61 hashes. That's a little different than CRC where finding a collision or pre-image is trivial. MD4 and MD5 are not collision resistant but AFAIK they are still pre-image resistant.
    http://en.wikipedia.org/wiki/SHA-1#SHA-1

    Also, we don't really know if SHA-3 and SHA-512 are more secure than SHA-256. I would say that SHA-256 is more secure than SHA-3 just because it is older and people have had more time to attack it. There is no practical reason for using hashes of more than about 200 bits to protect against brute force attacks.

    Edit: tested the optimization. In Keccak<r>:: put(int ch) I replaced the line:

    state[ptr/8]^=UINT64(ch)<<(ptr%8*;

    with

    ((unsigned char*)state)[ptr]^=(unsigned char)ch;

    and time to hash enwik9 improves from 27.0s to 23.3s (2 GHz T3200, Win32, g++ 4.7.0 -O3 -mssse3) averaged over 2 tests.
    Last edited by Matt Mahoney; 26th November 2012 at 20:08.

  2. #32
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    Work of duplicate file finder - trash bin indicates duplicate files...
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	chk110.png 
Views:	250 
Size:	62.4 KB 
ID:	2102  

  3. #33
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    Please tell me what you think
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	flat16.png 
Views:	256 
Size:	53.3 KB 
ID:	2103   Click image for larger version. 

Name:	flat16v2.png 
Views:	260 
Size:	52.6 KB 
ID:	2104   Click image for larger version. 

Name:	flat20.png 
Views:	253 
Size:	51.2 KB 
ID:	2105   Click image for larger version. 

Name:	flat20v2.png 
Views:	231 
Size:	50.1 KB 
ID:	2106  

  4. #34
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Smaller icons and consistency is neat (the tick and cross is always "encircled", be it update.exe or nu.exe) -> 1st screenshot is the best, but the iconset as such is ugly.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  5. #35
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    It's Windows 8 icons in action... Okay, so I'll keep these icons. Just thinking about what icon is the best for duplicate files. Sheep icon is the great idea, but there is not Sheep in my icon set. Might user delete all files marked with trash bin?? (Current icon for duplicate files - all duplicate files are marked with trash bin, including original files)

  6. #36
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Yes, Windows8 icons, it's just that it's either simple/Metro or it's pretty, depends what you choose What are all current states that there can be? I can see at least 5 from the screenshots.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  7. #37
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    5 states exactly!

  8. #38
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    I implemented SHA-256 from scratch (using description in Wikipedia) and ran some benchmarks comparing with SHA3-512 above and SHA-1 from libzpaq and fsum. For SHA1 and SHA256 I compiled with g++ -O3 -msse2 and with cl /O2 /arch:SSE2 in 32 bit Windows. I timed them on a 2.0 GHz T3200 using real times which seemed to give more consistent results than process times (1 thread) and took the fastest of several tests on enwik9.

    5.6s fsum -sha1
    11.1s sha1 (libzpaq, cl)
    11.4s sha1 (libzpaq, g++)
    13.4s fsum -sha256
    16.0s sha256 (below, cl)
    16.45s sha256 (below, g++)
    25.1s sha3-512 (above, g++ with ssse3 intrinsics)

    So at least my code is not too much worse than fsum this time although I still don't know how they made sha1 so fast. I used the same interface for sha256 as I did for sha1 in libzpaq.

    Code:
    // sha256.cpp - Compute SHA-256 checksums of filename arguments.
    // Written by Matt Mahoney. Public domain.
    
    #define _CRT_DISABLE_PERFCRIT_LOCKS
    #include <stdio.h>
    #include <stdint.h>
    
    // For computing SHA-256 checksums
    // http://en.wikipedia.org/wiki/SHA-2
    class SHA256 {
    public:
      void put(int c) {  // hash 1 byte
        unsigned& r=w[len0>>5&15];
        r=(r<<8)|(c&255);
        if (!(len0+=8)) ++len1;
        if ((len0&511)==0) process();
      }
      double size() const {return len0/8+len1*536870912.0;} // size in bytes
      uint64_t usize() const {return len0/8+(uint64_t(len1)<<29);} //size in bytes
      const char* result();  // get hash and reset
      SHA256() {init();}
    private:
      void init();           // reset, but don't clear hbuf
      unsigned len0, len1;   // length in bits (low, high)
      unsigned s[8];         // hash state
      unsigned w[16];        // input buffer
      char hbuf[32];         // result
      void process();        // hash 1 block
    };
    
    void SHA256::init() {
      len0=len1=0;
      s[0]=0x6a09e667;
      s[1]=0xbb67ae85;
      s[2]=0x3c6ef372;
      s[3]=0xa54ff53a;
      s[4]=0x510e527f;
      s[5]=0x9b05688c;
      s[6]=0x1f83d9ab;
      s[7]=0x5be0cd19;
    }
    
    void SHA256::process() {
    
    #define m(i) \
        t1=w[(i-15)&15], t2=w[(i-2)&15], \
        w[i&15]+=w[(i-7)&15]+((t1>>7|t1<<25)^(t1>>18|t1<<14)^(t1>>3)) \
          +((t2>>17|t2<<15)^(t2>>19|t2<<13)^(t2>>10)) \
    
    #define r(a,b,c,d,e,f,g,h,i) \
        if (i>15) m(i); \
        t2=((a>>2|a<<30)^(a>>13|a<<19)^(a>>22|a<<10))+((a&b)^(a&c)^(b&c)); \
        t1=((e>>6|e<<26)^(e>>11|e<<21)^(e>>25|e<<7))+((e&f)^(~e&g)) \
          +h+k[i]+w[i&15]; \
        d+=t1; \
        h=t1+t2
    
    #define r8(i) \
        r(a,b,c,d,e,f,g,h,i);   \
        r(h,a,b,c,d,e,f,g,i+1); \
        r(g,h,a,b,c,d,e,f,i+2); \
        r(f,g,h,a,b,c,d,e,i+3); \
        r(e,f,g,h,a,b,c,d,i+4); \
        r(d,e,f,g,h,a,b,c,i+5); \
        r(c,d,e,f,g,h,a,b,i+6); \
        r(b,c,d,e,f,g,h,a,i+7);
    
      static const unsigned k[64]={
        0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5,
        0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
        0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3,
        0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
        0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc,
        0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
        0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7,
        0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
        0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13,
        0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
        0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3,
        0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
        0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5,
        0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
        0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208,
        0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2};
    
      unsigned a=s[0];
      unsigned b=s[1];
      unsigned c=s[2];
      unsigned d=s[3];
      unsigned e=s[4];
      unsigned f=s[5];
      unsigned g=s[6];
      unsigned h=s[7];
      unsigned t1, t2;
    
      r8(0);
      r8(8);
      r8(16);
      r8(24);
      r8(32);
      r8(40);
      r8(48);
      r8(56);
    
      s[0]+=a;
      s[1]+=b;
      s[2]+=c;
      s[3]+=d;
      s[4]+=e;
      s[5]+=f;
      s[6]+=g;
      s[7]+=h;
    
      #undef m
      #undef r8
      #undef r
    };
    
    // Return old result and start a new hash
    const char* SHA256::result() {
    
      // pad and append length
      const unsigned s1=len1, s0=len0;
      put(0x80);
      while ((len0&511)!=448) put(0);
      put(s1>>24);
      put(s1>>16);
      put(s1>>8);
      put(s1);
      put(s0>>24);
      put(s0>>16);
      put(s0>>8);
      put(s0);
    
      // copy s to hbuf
      for (int i=0; i<8; ++i) {
        hbuf[4*i]=s[i]>>24;
        hbuf[4*i+1]=s[i]>>16;
        hbuf[4*i+2]=s[i]>>8;
        hbuf[4*i+3]=s[i];
      }
    
      // return hash prior to clearing state
      init();
      return hbuf;
    }
    
    int main(int argc, char** argv) {
      SHA256 sha256;
      for (int i=1; i<argc; ++i) {
        FILE* in=fopen(argv[i], "rb");
        if (!in)
          perror(argv[i]);
        else {
          for (int c; (c=getc(in))!=EOF; sha256.put(c));
          fclose(in);
          double sz=sha256.size();
          const char* p=sha256.result();
          for (int j=0; j<32; ++j) printf("%02x", p[j]&255);
          printf(" %1.0f %s\n", sz, argv[i]);
        }
      }
      return 0;
    }

  9. #39
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Quote Originally Posted by encode View Post
    5 states exactly!
    Haha, sure But I don't know what does which status mean.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  10. #40
    Member ivan2k2's Avatar
    Join Date
    Nov 2012
    Location
    Russia
    Posts
    35
    Thanks
    13
    Thanked 6 Times in 3 Posts
    Quote Originally Posted by Matt Mahoney View Post
    So at least my code is not too much worse than fsum this time although I still don't know how they made sha1 so fast. I used the same interface for sha256 as I did for sha1 in libzpaq.
    Hi, Matt. Try to replace (a&b)^(a&c)^(b&c) in your code to (a&b)^(c&(a^b)) and this could make your sha256 faster.

  11. #41
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    571
    Thanks
    219
    Thanked 204 Times in 96 Posts
    I profiled sha256.cpp using the following commands on a 560 MiB test file and a 2.4 GHz M520:

    Code:
    g++ -O3 -msse2 -pg sha256.cpp -osha256_prof.exe
    sha256_prof.exe testfile
    gprof sha256_prof.exe gmon.out | more
    This gave the following output:

    Code:
      %   cumulative   self              self     total
     time   seconds   seconds    calls  ns/call  ns/call  name
     63.26      5.20     5.20  9175041   566.75   566.75  SHA256::process()
     36.74      8.22     3.02                             main
    So even though SHA256:: process takes most of the time as it should, it looks like main() could be worth looking at.
    With the I/O benchmarks from here in mind, I modified main() to use fread instead of getc:

    Code:
    #define BUF_SIZE 1024
    
    int main(int argc, char** argv) {
      SHA256 sha256;
    
      int bytes_read;
      char* buf = new char[BUF_SIZE];
    
      for (int i=1; i<argc; ++i) {
        FILE* in=fopen(argv[i], "rb");
        if (!in)
          perror(argv[i]);
        else {
          //old code: for (int c; (c=getc(in))!=EOF; sha256.put(c));
          for (;;) {
            bytes_read = fread(buf, 1, BUF_SIZE, in);
            if (bytes_read == 0) break;
            for (int j = 0; j < BUF_SIZE; j++) {
              sha256.put(buf[j]);
            }
          }
          fclose(in);
          double sz=sha256.size();
          const char* p=sha256.result();
          for (int j=0; j<32; ++j) printf("%02x", p[j]&255);
          printf(" %1.0f %s\n", sz, argv[i]);
        }
      }
    
      delete[] buf;
    
      return 0;
    }
    This resulted in an improvement of around 20% in speed and a better profiling result:

    Code:
      %   cumulative   self              self     total
     time   seconds   seconds    calls  ns/call  ns/call  name
     78.29      4.94     4.94  9175041   538.42   538.42  SHA256::process()
     21.71      6.31     1.37                             main
    Quote Originally Posted by ivan2k2 View Post
    Hi, Matt. Try to replace (a&b)^(a&c)^(b&c) in your code to (a&b)^(c&(a^b)) and this could make your sha256 faster.
    This improves the result by another 2% (6.17 s, ~490 ns/call) - there might be additional boolean operation optimizations like this, comparing with existing SHA-256 code would reveal them most likely.

    The modified version doesn't bridge the gap to fsum (5.63 s) completely, but it's getting much closer.
    Last edited by schnaader; 29th November 2012 at 17:43.
    http://schnaader.info
    Damn kids. They're all alike.

  12. #42
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Thanks for the ideas. I also found http://download.intel.com/embedded/p...per/327457.pdf although I prefer a portable implementation without assembler.

    Edit: optimized version below. Time for enwik9 is improved to 15.3s for g++ and 15.1s for VC++. Optimizations are as follows:

    - Used fread() instead of getc().
    - Replaced (a&b)^(a&c)^(b&c) with (a&b)^(c&(a^b)
    - Optimized S0(a) and S1(e) as suggested in the Intel doc. I also tried this on the scheduler for w[i-2] and w[i-15] but there was no improvement.
    - Rearranged the round function to eliminate one variable. Big gain here.

    Other changes are for readability and had no effect on speed. That includes removing the if-statement from the round function, which was optimized out anyway. Neither did removing the variables from the scheduler or defining the ror() macro.

    Code:
    // sha256.cpp v2 - Compute SHA-256 checksums of filename arguments.
    // Written by Matt Mahoney. Public domain.
    
    #define _CRT_DISABLE_PERFCRIT_LOCKS
    #include <stdio.h>
    #include <stdint.h>
    
    // For computing SHA-256 checksums
    // http://en.wikipedia.org/wiki/SHA-2
    class SHA256 {
    public:
      void put(int c) {  // hash 1 byte
        unsigned& r=w[len0>>5&15];
        r=(r<<8)|(c&255);
        if (!(len0+=8)) ++len1;
        if ((len0&511)==0) process();
      }
      double size() const {return len0/8+len1*536870912.0;} // size in bytes
      uint64_t usize() const {return len0/8+(uint64_t(len1)<<29);} //size in bytes
      const char* result();  // get hash and reset
      SHA256() {init();}
    private:
      void init();           // reset, but don't clear hbuf
      unsigned len0, len1;   // length in bits (low, high)
      unsigned s[8];         // hash state
      unsigned w[16];        // input buffer
      char hbuf[32];         // result
      void process();        // hash 1 block
    };
    
    void SHA256::init() {
      len0=len1=0;
      s[0]=0x6a09e667;
      s[1]=0xbb67ae85;
      s[2]=0x3c6ef372;
      s[3]=0xa54ff53a;
      s[4]=0x510e527f;
      s[5]=0x9b05688c;
      s[6]=0x1f83d9ab;
      s[7]=0x5be0cd19;
    }
    
    void SHA256::process() {
    
      #define ror(a,b) ((a)>>(b)|(a<<(32-(b))))
    
      #define m(i) \
         w[(i)&15]+=w[(i-7)&15] \
           +(ror(w[(i-15)&15],7)^ror(w[(i-15)&15],18)^(w[(i-15)&15]>>3)) \
           +(ror(w[(i-2)&15],17)^ror(w[(i-2)&15],19)^(w[(i-2)&15]>>10))
    
      #define r(a,b,c,d,e,f,g,h,i) \
        t1=ror(e,14)^e; \
        t1=ror(t1,5)^e; \
        h+=ror(t1,6)+((e&f)^(~e&g))+k[i]+w[(i)&15]; \
        d+=h; \
        t1=ror(a,9)^a; \
        t1=ror(t1,11)^a; \
        h+=ror(t1,2)+((a&b)^(c&(a^b)));
    
      #define mr(a,b,c,d,e,f,g,h,i) m(i); r(a,b,c,d,e,f,g,h,i);
    
      #define r8(i) \
        r(a,b,c,d,e,f,g,h,i);   \
        r(h,a,b,c,d,e,f,g,i+1); \
        r(g,h,a,b,c,d,e,f,i+2); \
        r(f,g,h,a,b,c,d,e,i+3); \
        r(e,f,g,h,a,b,c,d,i+4); \
        r(d,e,f,g,h,a,b,c,i+5); \
        r(c,d,e,f,g,h,a,b,i+6); \
        r(b,c,d,e,f,g,h,a,i+7);
    
      #define mr8(i) \
        mr(a,b,c,d,e,f,g,h,i);   \
        mr(h,a,b,c,d,e,f,g,i+1); \
        mr(g,h,a,b,c,d,e,f,i+2); \
        mr(f,g,h,a,b,c,d,e,i+3); \
        mr(e,f,g,h,a,b,c,d,i+4); \
        mr(d,e,f,g,h,a,b,c,i+5); \
        mr(c,d,e,f,g,h,a,b,i+6); \
        mr(b,c,d,e,f,g,h,a,i+7);
    
      static const unsigned k[64]={
        0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5,
        0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
        0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3,
        0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
        0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc,
        0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
        0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7,
        0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
        0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13,
        0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
        0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3,
        0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
        0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5,
        0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
        0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208,
        0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2};
    
      unsigned a=s[0];
      unsigned b=s[1];
      unsigned c=s[2];
      unsigned d=s[3];
      unsigned e=s[4];
      unsigned f=s[5];
      unsigned g=s[6];
      unsigned h=s[7];
      unsigned t1;
    
      r8(0);
      r8(8);
      mr8(16);
      mr8(24);
      mr8(32);
      mr8(40);
      mr8(48);
      mr8(56);
    
      s[0]+=a;
      s[1]+=b;
      s[2]+=c;
      s[3]+=d;
      s[4]+=e;
      s[5]+=f;
      s[6]+=g;
      s[7]+=h;
    
      #undef mr8
      #undef r8
      #undef mr
      #undef r
      #undef m
      #undef ror
    };
    
    // Return old result and start a new hash
    const char* SHA256::result() {
    
      // pad and append length
      const unsigned s1=len1, s0=len0;
      put(0x80);
      while ((len0&511)!=448) put(0);
      put(s1>>24);
      put(s1>>16);
      put(s1>>8);
      put(s1);
      put(s0>>24);
      put(s0>>16);
      put(s0>>8);
      put(s0);
    
      // copy s to hbuf
      for (int i=0; i<8; ++i) {
        hbuf[4*i]=s[i]>>24;
        hbuf[4*i+1]=s[i]>>16;
        hbuf[4*i+2]=s[i]>>8;
        hbuf[4*i+3]=s[i];
      }
    
      // return hash prior to clearing state
      init();
      return hbuf;
    }
    
    int main(int argc, char** argv) {
      SHA256 sha256;
      for (int i=1; i<argc; ++i) {
        FILE* in=fopen(argv[i], "rb");
        if (!in)
          perror(argv[i]);
        else {
          const int BUFSIZE=4096;
          int n;
          unsigned char buf[BUFSIZE];
          while ((n=fread(buf, 1, BUFSIZE, in))>0) {
            for (int i=0; i<n; ++i)
              sha256.put(buf[i]);
          }
          fclose(in);
          double sz=sha256.size();
          const char* p=sha256.result();
          for (int j=0; j<32; ++j) printf("%02x", p[j]&255);
          printf(" %1.0f %s\n", sz, argv[i]);
        }
      }
      return 0;
    }
    Last edited by Matt Mahoney; 30th November 2012 at 04:41.

  13. #43
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    Quote Originally Posted by Black_Fox View Post
    Haha, sure But I don't know what does which status mean.
    My bad, if you're asking. It must be clear without any hints - 100% intuitive...

    Delete command from Pop-Up menu now have same behavior as in Windows Explorer - i.e. files will be moved to the Recycle Bin, if different option is not set in Windows.

    Same hashes will be marked with a green background, instead of a special icon.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	chk110.png 
Views:	234 
Size:	56.2 KB 
ID:	2108  

  14. #44
    Member FatBit's Avatar
    Join Date
    Jan 2012
    Location
    Prague, CZ
    Posts
    189
    Thanks
    0
    Thanked 36 Times in 27 Posts
    Dear Mr. Encode,

    I recommend as option more aggressive strategy when CHK will search same files inside various archives.

    Sincerely yours,

    FatBit

  15. #45
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Some timings for chk v1.03 on a 2.0 GHz T3200, 3 GB, 2 cores, Win32 (Vista) on enwik9:

    crc32 19s (30-40% CPU)
    crc64 19s (35-40% CPU)
    md4 18s (35-40% CPU)
    md5 18s (~45% CPU)
    sha1 17s (~50% CPU)
    sha256 24s (~50% CPU)
    sha512 58s (~50% CPU)

    I timed with a watch and estimated CPU from watching Windows Task Manager (100% = both cores). The time is from when I click on a new hash in the options menu until the new hash appears with only enwik9 in the window.

  16. #46
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    Good news is that new CHK v1.10 is much faster than v1.03 BETA. I optimized ALL hashes! Especially the huge difference in SHA1 computation.
    From your timings I see how slow HDD affects final time. With SSD we will see really different results. Some numbers from the top of my head (ENWIK9):
    CHK v1.03 BETA
    CRC32 -> 3 sec
    CRC64 -> 4.3 sec
    MD4 -> 3.1 sec
    MD5 -> 3.9 sec
    SHA1 -> 5.4 sec
    SHA256 -> 10 sec
    SHA512 -> 21 sec

    And, for comparison, CHK v1.10 (tested with the same hardware as v1.03 BETA)
    CRC16 -> 2.6 sec
    CRC32 -> 2.6 sec
    CRC64 -> 4.1 sec
    MD4 -> 2.9 sec
    MD5 -> 3.4 sec
    SHA1 -> 4.3 sec [!]
    SHA256 -> 8.1 sec
    SHA512 -> 17 sec
    SHA3 -> 30 sec

    In addition, new version will show processing time in CHKs status bar! No more watches!


  17. #47
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Timing will be nice (or a command line version).
    Disk I/O was indeed slowing down the times, even though I should have had enough memory to keep enwik9 in cache. I managed to fix the problem after closing some programs. In this test the disk light stayed off and task manager showed 50% CPU for all tests.

    crc32 7 sec
    crc64 11 sec
    md4 9 sec
    md5 14 sec
    sha1 16 sec
    sha256 24 sec (no change)
    sha512 58 sec (no change)

    Looking forward to your new faster version.

  18. #48
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    Command line version will be much faster (because of Intel/Visual C++ compile). All I'm doing now is fighting with such inefficient Borland C++. BTW, very soon, Embarcadero will release 64-bit C++ Builder. And instantly, CHK will have a 64-bit release.
    In my opinion, GUI is much more user friendly. Actually, we already have a bunch of nice command line hash tools...

  19. #49
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    bcc32 -O -6 sha256.cpp
    sha256 enwik9 -> 30.0 sec (twice as slow as g++ or VC++)

    That's Borland C++ 5.5.1. Really old version (2000). Doesn't have <stdint.h>

    Also, dmc -6 -> 37.6 sec. (Mars 8.42n, 2004. -o flag causes an internal compiler error, so not optimized).

  20. #50
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    Borland/Embarcadero C++ compiler untouched for decades, you'll be surprised!

    Anyway, check out more signature design. Please tell me what you think, or keep silence for a long time - a new version will be released within a few days, and no future releases expected for a few months (at least). I'm just performing some extra testing - for the first non beta and because previous CHK (1.03) contains too many bugs (and no one reported it to me - I need much more user count for my software!). And I just want to get back to BIM/GAR and BCM, finally.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	1.png 
Views:	244 
Size:	58.9 KB 
ID:	2109  

  21. #51
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    you can generate dll with icl and use it from GUI binary. it's how i made freearc ~10% faster (arc.exe can be compiled only with ghc 3.x)

  22. #52
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    No DLLs in CHKs philosophy. Anyway, I may statically link compiled with ICL code as well. As a note, Delphi/C++ Builder really dislikes alien code. Even with custom .MANIFEST all things may go wrong.
    Such things may be used in future anyway...

  23. #53
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,475
    Thanks
    26
    Thanked 121 Times in 95 Posts
    What about dllmerge from Eugene: http://encode.su/threads/1584-dllmerge ?

  24. #54
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    No need in DLLs. Why I should compile my code to DLLs and then bind it to CHK, if I may compile to OBJ/LIB and statically link it?

  25. #55
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,475
    Thanks
    26
    Thanked 121 Times in 95 Posts
    OK, I overlooked that.

  26. #56
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    you can't link obj files from different c++ compilers due to different startup/instrinsincs code. altough for the small computation-only functions it may work
    Last edited by Bulat Ziganshin; 1st December 2012 at 15:35.

  27. #57
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,475
    Thanks
    26
    Thanked 121 Times in 95 Posts
    So dllmerge would solve the problem then? It should be possible to find and use a calling convention understandable by Borland C++ and Intel compiler simultaneously.

  28. #58
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    pascal/cdecl calling conventions are compatible among compilers and dll/dllmerge will work

  29. #59
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,423
    Thanks
    223
    Thanked 1,052 Times in 565 Posts
    Actually both stdcall/cdecl are supported well enough everywhere (most because winapi is stdcall).
    And there're good OMF/COFF converters, for example http://agner.org/optimize/objconv.zip .
    I even managed to link an IntelC-compiled .obj into a delphi app (using {$L name}), and delphi requires a special kind of OMF.

    I suspect that Ilia didn't already do it mostly because IDEs (both VS and BC) don't really support build scripts
    So I'd suggest to look into #pragma-comment-lib thing - http://msdn.microsoft.com/en-us/libr...(v=vs.80).aspx
    which might allow to link a module without any external build scripts.

Page 2 of 2 FirstFirst 12

Similar Threads

  1. CHK wishlist
    By encode in forum The Off-Topic Lounge
    Replies: 64
    Last Post: 17th March 2012, 14:51
  2. CHK 1.02 - file analysis tool
    By encode in forum Data Compression
    Replies: 6
    Last Post: 24th July 2011, 16:46
  3. CHK 1.01 is here! (New GUI MD5/SHA1 file checker)
    By encode in forum Data Compression
    Replies: 24
    Last Post: 20th July 2011, 09:45

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •