Creating our IRC channel

[Bulat Ziganshin]

today i've got the following message:

Hello Bulat,

I don't want to bother you by writing this message. I know
you as a very active and committed member of encode.su. As I
tried today to register there, I found out, registration is
disabled there, thats a pity. Well, my intention is the
following: I got interested in archivers/compressors maybe
since the beginning of this year and followed the forum of
encode every day and i want to get in contact with the admin
of the forum, encode because i'm an netadmin of a German IRC
Network and want to ask him, if he is interested in another
communication way (well, i call it "live" :-; ) to give the
members of the forum the possibility to talk at realtime
together. My intention is NOT to say, forum is dead, use IRC
but mayby both ways are a common way for interacting
together. Such active members as you and others can talk in
realtime and have not to wait for a response post.
You can naturally post encode this message too. It would be
my pleasure, if you or encode don't hesitate to contact me
in different ways:
1. here, as an answer
2. email: azrael@german-irc.net
3. IRC server: matrix.german-irc.net port 6667 channel
#german-irc (i am mostly there, but won't answer anytime,
cuz of work)

It would be very honest of you, if you answer or just give
this message to encode and it would be very great, if i get
an answer.

With warm regards from Germany,

Azrael

and then, when i asked why he doesn't establish channel myself, Azrael answered:

thanks a lot for your very fast answer. You/Ilya can do what ever you want,
it would be very great if you publish the mess. in the forum. You can
establish a channel on your own, if you like; just enter the IRC with the
data i wrote you, register your name with
1. /ns register password email
2. /join #desired channel
3. register channel with: /cs register channel password description

thats it

warm regards

Azrael

Netadmin of the German-IRC Network
http://www.german-irc.net
http://forum.german-irc.net/
IRC: matrix.german-irc.net #german-irc

so that. if there are people interested in IRC it may be a good day to establish a channel

[Surfer]

Good idea.
Why not #freearc channel ?

[Shelwien]

Actually I started #compression on irchighway.net a while ago,
but it wasn't any successful.
Well, lets try again: irc://irc.irchighway.net/compression
And as to german-irc.net - its in german and seems too strict,
banned me while I was reading its stats, probably because I
didn't join any channel.

Btw, http://www.mibbit.com/chat/ can be used to join via browser.

[LovePimple]

I prefer to use the forum.

[Piotr Tarsa]

I agree with LovePimple. I think that shoutbox on this forum would be sufficient. People then will be able to announce some updates for programs, benchmarks, ask for quick answers etc without much discussion.

[Black_Fox]

I have nothing against IRC, but I'm afraid it would suffer from inactivity, there's not that much to say

[Shelwien]

Well, two people did appear until now - Surfer and chornobyl...

[Shelwien]

Seen two more people.
Also, added a bot with DCC 1991-2004 archives on it.
Would maybe later upload also some compression-related books there.

[Shelwien]

#compression is slightly more lively now.
(server is irc.irchighway.net)
It seem like we have 3 persons who even talk sometimes
irchighway banned mibbit for some reason though,
so http://webchat.irchighway.net/ can be used instead.

[Black_Fox]

Freenode banned mibbit too, reportedly it's been abused too much recently.

[Shelwien]

* Simon|B has joined #compression
* Alibert has joined #compression
<Simon|B> hi guys
<Shelwien> hi ;)
<toffer> wow so many people!
<Shelwien> a monster: http://shelwien.googlepages.com/img_0061_.jpg
<Simon|B> haha nice
<Simon|B> don't want to meet him in a dark forest ;)
<toffer> is there any good cross platform irc client (which works on win and linux) ?
<Simon|B> opera :-P
<toffer> i usually use programs which share their configs accross different platforms like firefox, psi, ...
<Simon|B> firefox has a plugin too I am sure...
<Shelwien> http://www.irssi.org/download ?
<Simon|B> http://www.mozilla.org/projects/rt-messaging/chatzilla/
<toffer> that mozilla stuff requires java
<toffer> unfortunately -.-
<Simon|B> yes also read this 
* Alibert has quit IRC (Ping timeout: 363 seconds)
<Simon|B> I use opera so I don't have the problem with freaking firefox plugin search.
<Simon|B> Do you have java script disabled?
<toffer> i don't like that stuff. it's bloated
<toffer> i'd prefer some standalone client
<toffer> and it doesn't require java script but java, thus the vm needs to run

<toffer> and did anything else happen meanwhile?
<Shelwien> 1. i optimized the 09h2 mixer up to 207961 on book1
<toffer> 09h2 is what?
<Shelwien> 2. i added that exponent/mantissa stuff to distance coding, and also a second runlength-coded probability interval
<Shelwien> http://shelwien.googlepages.com/p2.txt
<Shelwien> and 09h2 is the thing from mix_test v9 / BWTmix v1
<chornobyl> beware of googlepages shutdown
<Shelwien> who cares
<chornobyl> lovepimple warned you
<Shelwien> i have two paid hostings ;)
<toffer> at least it now seems to be equally fast
<toffer> now it's time to improve compression i guess ^^
<Shelwien> new one is slightly faster
<toffer> oh yeah 3sec
<toffer> i just noticed
<Shelwien> and you can notice that compression is slightly better in first version (with one interval)
<Shelwien> probably it would be better to somehow improve the distance modelling
<Shelwien> like use log2i(x^2) for exponent maybe?
<Shelwien> and I kinda want to make it all distance-coded now...
<toffer> really?
<toffer> the normal coding, too?
<Shelwien> well, i know that it won't be really good for speed ;)
<Shelwien> though that not certain too
<toffer> unsure
<toffer> what about 0.5 ?
<toffer> and i thought you don't model distances
<toffer> you just used exponent coding
<Shelwien> i mean, the probability range can be split into subranges
<Shelwien> which encoded with separate distance models
<Shelwien> I currently have two such subranges
<Shelwien> and subrange new 0.5 can be just stored uncompressed
<Shelwien> I already checked that its ok for the range (16384-1000;16384+1000), with SCALE=32768
<toffer> two subranges?
<Shelwien> 0..48 and 48..128 or something
<toffer> so subdividing the mentioned initial interval piece?
<Shelwien> you can notice there're versions with 444M rc calls and 352M
<Shelwien> lumping the bits from specific probability subrange together
<Shelwien> and runlength coding them
<toffer> mh i already calculated (at least as an estimation) that direct rle won't work
<toffer> you still need to add a model
<Shelwien> ?
<toffer> just using rle on bit runs 
<Shelwien> i have it working already as you can see by compressed code size
<Shelwien> well, of course i mean to compress rle distances
<toffer> ah
<toffer> i thought you meant direct compression of runs 
<Shelwien> well, currently it uses binary coding for distance exponent
<Shelwien> and puts mantissa bits uncompressed
<Shelwien> and somehow it still even improved compression on enwik8 (in first case)
<toffer> weird
<toffer> but still it's like halving the rc calls
<toffer> wonder why it's just 3secs
<toffer> is there still that branch?
<toffer> guess i'd have to look at the code 
<Shelwien> yeah.. its even less actually
<Shelwien> the compression speed is probably improved due to buffered input
<Shelwien> also it doesn't have that branch in the model
<toffer> did you cut off get/putc?
<toffer> already
<toffer> guess yes
<Shelwien> not really
<toffer> why not try i first?
<Shelwien> well, i wanted to fix the distance model first, and that i did yesterday 
<Shelwien> so next i'd fix i/o, right
<Shelwien> i can post the source if you want
<toffer> you could upload it somewhere
<toffer> i'd have a look
<toffer> i hope you log all of that stuff?
<toffer> and may post something interesting on the forum
<Shelwien> http://shelwien.googlepages.com/09h2-p3.rar
<Shelwien> and log which stuff? this channel? I do of course...
* toffer has quit IRC (Quit: Page closed)

[Shelwien]

Here goes another advertisement
#compression on irc.irchighway.net
http://webchat.irchighway.net/ can be used for access via browser.

* osman has joined #compression
<osman> hi everyone!
<osman> Hi "the respective bot" and Shelwien :)
* PAQer has joined #compression
* PAQer has quit IRC (Quit: Page closed)
* toffer has joined #compression
* Lasse has joined #compression
<Lasse> heyo
<Lasse> quicklz author here :)
<Lasse> gonna bbl
* Lasse has quit IRC (Client closed connection)
* Lasse has joined #compression
<osman> see you all
* osman has quit IRC (Quit: Page closed)
<Shelwien> damn, and I was sleeping...
<Shelwien> Somebody should have asked what osman is doing now ;)
* toffer has quit IRC (Quit: Page closed)
<Shelwien> ...
<Shelwien> as to qpress btw
<Shelwien> do you have any specific experience as to optimizing file i/o on windows?
<Shelwien> Lasse?
<Shelwien> qpress seems to be a rather rare example of an open source compressor with async i/o
<Shelwien> but its very "general"
<Shelwien> and I think it might be possible to further improve the speed under windows
<Shelwien> couldn't find any informative posts on that though
<Lasse> yeah, have a deal of trial-and-error experience
<Lasse> how would you optimize it?
<Shelwien> well, windows has some special i/o modes
<Shelwien> overlapped i/o for example
<Shelwien> and actually its a very complicated task
<Shelwien> as there're different optimal solutions for different cases
<Shelwien> one example is that when you call fwrite()
<Shelwien> it executes some standard library wrapper
<Shelwien> then goes to kernel32.dll and executes one more wrapper
<Shelwien> then to ntdll.dll and one more wrapper there, with int 2E/syscall/sysenter in it
<Shelwien> then stack arguments get copied to ring0 stack
<Shelwien> and some more weird stuff happens, until
execution reaches the filesystem driver and then hdd
driver
<Shelwien> and that's only one direction - i/o via a custom driver might be really noticeable faster
<Shelwien> (of course not on hdd level though, as completely ignoring windows caches is bad too)
<Shelwien> ...
<Lasse> FILE_FLAG_OVERLAPPED? tried that one, it has
problems depending on security settings, such as on NT and
Vista Home, where Windows starts a lazy writer that mucks
up everything
<Lasse> i'm not gonna touch FILE_FLAG_OVERLAPPED again :)
<Shelwien> well, i'd try
<Shelwien> for an example, there's this thing called XBMC
<Lasse> "XBMC Media Center"?
<Shelwien> XBox Media Center
<Shelwien> yeah
<Shelwien> its all skin-driven
<Shelwien> and has a pretty large skin archive to load
<Lasse> but what about it? :)
<Shelwien> (with images compressed with LZO btw)
<Shelwien> and it uses overlapped i/o ;)
<Lasse> ah
<Lasse> ok
<Lasse> but xbox probably don't have these security settings (forgot their names)
<Shelwien> XBMC is ported to windows/linux/macos
<Shelwien> well, and another thing is memory-mapped files
<Shelwien> and then the thing i wanted to talk about next ;)
<Shelwien> specifically, some delay modelling for i/o
<Lasse> the problem is that FILE_FLAG_OVERLAPPED makes it
possible to close a file beyond EOF so that you can read
deleted data on disk, so system admins often prevent this
on Windows Server. And all the Home and Basic editions
have it prevented by default because it doesn't let the
user specify them
<Shelwien> %)
<Lasse> and when this is deactivated, then Windows starts
a lazy writer that writes 0 in parallel with your own disk
I/O and that fucks everything up
<Shelwien> well, yeah, perfection here certainly requires
writing handlers for too much different cases ;)
<Lasse> mem mapped files aren't really meant to be used for sequential I/O
<Shelwien> i thought about it
<Shelwien> still it has benefit of avoiding data copying
<Lasse> I havn't timed mem I/O though
<Lasse> data copying?
<Shelwien> at least in theory the file data is read directly where they are mapped
<Shelwien> (same like with overlapped i/o)
<Lasse> ah, you mean bypassing cache?
<Lasse> yeah
<Shelwien> i think there's not only cache
<Shelwien> or how to say this...
<Shelwien> i mean, the data for a common readfile() are
read into some system buffer (or copied from cache)
<Lasse> but I'm already supporting FILE_FLAG_NO_BUFFERING which also prevents caching
<Shelwien> and then copied into your memory
<Lasse> yup
<Shelwien> and its even worse with writefile()
<Shelwien> as at least XP, it seems
<Shelwien> tries to really write the data and blocks it until all's done
<Lasse> WriteFile doesn't block if you issue just 8-64 kbyte at a time, or thereabout :)
<Lasse> yeah, they do
<Shelwien> anyway, what i wanted to say
<Shelwien> is that overlapped i/o and mapped files avoid some memcpys
<Lasse> FILE_FLAG_NO_BUFFERING does too which I'm using already :)
<Shelwien> it does anyway, i measured it, and it becomes slower when you write over a cluster size
<Shelwien> yeah, i noticed that ;)
<Shelwien> still, i think mapped i/o is worth trying too
<Lasse> maybe :)
<Shelwien> though for reading files like that
<Shelwien> you'd probably have to add some prefetching
<Shelwien> basically pages accesses to read the data before your (de)compressor would need it
<Lasse> yup
<Lasse> but optimizing disk I/O on Windows is a hell
<Shelwien> ...anyway, I'd like to have it all already
tested by somebody so that I could just read a nice report
and choose a method i'd like ;)
<Lasse> if the user has 1 physical disk, prefetch is hard because it interferes with writing
<Shelwien> yeah, that's why i mentioned i/o modelling ;)
<Lasse> i'd like that too ;D

* mathiasr has joined #compression
<mathiasr> Hello
<Shelwien> hi ;)
<Lasse> heyo mathiasr 
<mathiasr> I've been reading encode.su forums for a while
<mathiasr> but I cannot register to post
<Shelwien> you can mail to "encode"
<Shelwien> apparently he has some problems with spammers and bots
<mathiasr> What is his complete email?
<Shelwien> http://encode.su/forum/sendmessage.php ?
<mathiasr> Does not work since I'm not registered !
<Shelwien> well, some encode@encode.su then i guess ;)
<Shelwien> ICQ# 224716672 also, but he's rarely online recently
<mathiasr> I tried that too does not work
<mathiasr> Could you send him a message asking him to take contact whith me mathiasr@free.fr
<Shelwien> ok
<mathiasr> Many thanks
* mathiasr has quit IRC (Quit: )

<Shelwien> i mean, i/o operation delays and read/write overlapping
<Shelwien> are series of data too, and can be predicted with a CM ;)
<Lasse> problem is, you have caches in disk, in controller
and in Windows. And, user can compress from a physical
disk to the same physical disk, or from one physical disk
to another. So many different cases to take care of. Also,
Windows kernel is indeterministic. WriteFile with 32 KB
blocks and you get 100 MB/s. Use 64 KB blocks and you get
50 MB/s. Use 4 MB blocks and get 100 MB/s again
<Shelwien> yeah, thats why i'm talking about i/o behavior modelling ;)
<Lasse> for example, see what I bumped into:
http://social.technet.microsoft.com/Forums/en-US/winserverfiles/thread/09dd046e-8127-4550-8e26-5fba7a5a0743
<Lasse> so I've pretty much given up optimizing for disk
I/O these days. Think disk I/O should be optimized by
Microsoft instead
<Lasse> unfortunatly that's not easy on Windows because it behaves indeterministic
<Lasse> I found that performance flaw on the forum, btw :)
<Shelwien> ah, thanks for that link, as its cases like that which I wanted to know about ;)
<Shelwien> as I might have ideas on how it should be done in theory
<Shelwien> but then if it doesn't work like that for many other people, it'd be kinda wasted time ;)
<Shelwien> one example is how many compressor writers
<Shelwien> like to read all the input into memory, then compress, and then write all the output
<Lasse> haha yeah
<Shelwien> thinking that its a fastest possible design, even if it kinda consumes too much memory ;)
<Lasse> THOR has pretty good I/O in most use cases (hardware cases I mentioned earlier)
<Lasse> impressive since he's apparently using Pascal
<Lasse> which probably has some naive I/O wrapper
<Shelwien> well, stuff like that is most weird
<Lasse> yeah
<Shelwien> like when replacing fread/fwrite with winapi
calls caused a slight slowdown in one of my experiments
<Lasse> fread/fwrite performs very fast using 4 MB blocks,
whereas ReadFile/WriteFile performs extremely slow (at
perhaps 60% of optimal speed) with the same 4 MB blocks
<Lasse> exactly
<Lasse> THOR is using 32 KB blocks
<Lasse> afair
<Shelwien> btw, did you try different compilers?
<Shelwien> like gcc vs intelc?
<Lasse> not for testing file I/O (tested for in-memory performance, though)
<Shelwien> and?
<Lasse> icc and vs was ~30% faster than gcc
<Shelwien> err... which gcc do you use then?
<Lasse> I used the very latest of all three, about 4-5 months ago
<Shelwien> gcc 4.3+ is usually comparable to intel
<Shelwien> which is significantly faster than vs
<Shelwien> also i've seen quite a few cases, exactly with LZ-like stuff
<Shelwien> where gcc-compiled program was faster
<Lasse> in the few tests where I've compared icc and vs,
they have been the *exact* same speed. So exact that I was
thinking if MS bought Intel's backend or something
<Shelwien> but that implies using PGO and recent gcc
<Shelwien> ah, they might have
<Shelwien> or, more like its probably somewhat of a joint development
<Shelwien> as intelc was initially a plugin for VS and all
<Lasse> yea

[osmanturan]

<Shelwien> Somebody should have asked what osman is doing now

I'm at least still alive But, busy with 3 jobs. Recently, I've graduated from my university with B.S. degree. And looking for master/doctorate without master program applications. Also some universities' research assistance position application made me busy too. Still waiting the other universities' applications. Ah, forgot... there are also some exams which I should pass for master/doctorate without master program and research assistance

For compression, I've changed BIT's command line parser to a more generic class. And also I've created another LZ codec: TAZI. In Turkish, Tazi (actually Tazı) means greyhound dog. And if somebody talks about a really fast thing/person, he says "he/it runs/goes like a tazı". So, that's why it has this name But, for now, it's far from actual tazi speed (tazi can reach 72 km/h in 1.5 seconds!!!). I would like to see a 72 MiB file processed in 1.5 seconds (=48 MiB/sec).

For signal processing, I've uploaded my GPU accelerated edge-detection test program on my web site. Also, experienced with GPU accelerated wavelet decomposition (just made a program which applies DWT on "only" rows). You can also download the test application from my site.

[Shelwien]

another advertisement
#compression on irc.irchighway.net
http://webchat.irchighway.net/ can be used for access via browser.

Random discussion:
http://shelwien.googlepages.com/cmp1.txt
(sorry, in russian this time

[Shelwien]

#compression on irc.irchighway.net
http://webchat.irchighway.net/ can be used for access via browser.

Random talk:

<Shelwien> i just remebered about the thing called "branch prediction attack" a few days ago
<Shelwien> BPA is originally a cryptography/security idea
<Shelwien> to use cpu's branch prediction cache as a so-called "side channel"
<Shelwien> so, i started to think how to implement some "telepathic data transmission" using that
<Shelwien> the original BPA idea
<Shelwien> is that cryptoalgorithms, like RSA
<Shelwien> can be cracked using info from branch prediction
<Shelwien> like, imagine running a "sniffer" utility
<Shelwien> which loops a precoded sample with a lot of branches
<Shelwien> and measures how many clocks it takes
<Shelwien> basically, flushing the BP cache and getting some info on cached branches
<Shelwien> and then, if there were branches dependent on key bits
<Shelwien> (like in RSA)
<toffer> i wondered how you want to read the branch cache - so like that? 
<toffer> there's no direct way i think?
<Shelwien> there is actually, but its not user-level
<Shelwien> not accessible without a driver basically
<Shelwien> and layout of these registers is cpu-specific and not really documented
<Shelwien> ...
<Shelwien> anyway, i just thought that a bitwise compressor is exactly such kind of algorithm
<Shelwien> with bit-dependent branches i mean
<Shelwien> and started thinking about how to implement such a thing
<Shelwien> like, run a data (de)compression loop in one process
<Shelwien> and branch sniffing loop (via a branch pool and RDTSC) in another
<Shelwien> well, the interesting point in this
<Shelwien> is analysis of sniffer output
<Shelwien> like, we can initially process the known random data sample
<Shelwien> and then we get a TSC trace from sniffer
<Shelwien> and we want to produce a model
<Shelwien> which would allow to get as much info as possible from the trace data
<Shelwien> (given known sample data)
<Shelwien> i think, basically
<Shelwien> it can be reformulated as 
<Shelwien> trace compressing using data sample as a context
<Shelwien> ...
<Shelwien> so initially i just want a demo like this
<Shelwien> 1. we start the "source" with known data sample processing in a loop
<Shelwien> 2. we start a sniffer and collect some trace data
<Shelwien> 3. then a trace data model is optimized
<Shelwien> for current circumstances 
<Shelwien> (results would be different on different cpus and OSes etc)
<Shelwien> 4. then we take an "unknown" data sample and try to "guess" it
<Shelwien> then, if there'd be any really impressive results, 
<Shelwien> i guess i could try to really apply that for some password cracking ;)
<Shelwien> ,,,
<Shelwien> so that's where i got my troubles about context clustering
<Shelwien> just think about finding correlations with data bits in a huge TSC trace
<Shelwien> where samples correlated with data bits would be separated 
<Shelwien> by variable-length strings of unrelated samples
<Shelwien> and there'd be "noise" in some cases too

[Shelwien]

<Shelwien> ?
* Krugz waves
<Krugz> hello
<Shelwien> hi
<Krugz>
Just randomly dropped in, saw it on the channel
list and it seemed interesting
<Shelwien>
well, people forgot how to lurk so its never
crowded here
<Krugz> that's ok
<Shelwien> but we have a log
<Krugz>
hmm I'll think about it, but I'm not much for
reading logs
<Shelwien>
well, if you have any suggestions about making
them more readable, its welcome. also its relatively
readable, if anything, we have real discussions here
<Krugz> heh you sure? I'm not exactly anyone particularly
qualified to do so
<Shelwien>
i don't see how it requires any qualification
<Krugz>
idk, I assumed there would be a lot of technical jargon
<Shelwien>
err... afair there was a 3 hour bodybuilding
chat here - that's a problem for me too
<Krugz> lol
<Shelwien> its not that topic-specific really
<Shelwien> just people from a forum chatting about random stuff
* Krugz nods
<Krugz> ya I'm reading the first log
<Krugz>
I get the idea, though I'm sure if you guys went
into the details I'd be lost
<Shelwien> that's not that frequent unfortunately
<Krugz>
well I'm assuming the people who discuss things in
here are already to the point where talking through all
the details would be pointless for everyone
it'd like watching professors explain for and
while loops to each other :P
<Shelwien>
dunno
user's point is very important too
<Krugz>
true enough
well I don't have any real use for data
compression, I'm just interested because it's ...
interesting lol
<Shelwien>
so most of the people on the forum, and half here, are
just people interested in testing experimental tools
and in fact, i don't have any use for data
compression too - that's kinda normal with modern hdds etc
<Krugz>
hah that's kinda counterintuitive, I would assume
the person making the most of the technology would be
someone inspired by a problem they themselves faced
<Shelwien>
but the applications of algorithms developed for data
compression are everywhere basically
<Krugz>
well ya, anywhere there's data..
:P
<Shelwien>
there're even compression-related algorithms in any
modern x86 CPUs, like in caches
or, especially, branch prediction
<Krugz>
hmm that I didn't really think about
<Shelwien>
and any optimization or prediction or storage
or error-correction or cryptography or communications or
recognition... etc... are related to that too
as it all deals with data, like you said
<Krugz>
ya cryptography would have been my first choice as
an alternate application
<Shelwien>
well, steganography is more related, but there're even
image scaling algorithms based on context models, and
plain data compression is just a good and objective
measure of model quality
<Krugz>
lol you certainly know a lot about data
compression and it's applications :P
<Shelwien>
basically, if your algorithm is good for
compressing data, then it would be also good for
predicting that data, and manipulating it
<Krugz> ya that makes sense
<Shelwien> well, its both my hobby and job
<Krugz>
did you get the job because it was your hobby or
the other way around?
<Shelwien>
well, i guess first
once I decided to help with compression.ru maintenance
and that eventually brought me some jobs
<Krugz> ah gotcha
<Shelwien>
btw, do you know about this program? http://soundslimmer.com/
<Krugz>
another guy in here, Chernobl, was also Russian
(or seemingly to me), is this like a mini-russian group?
<Shelwien> asmodean here is japanese
<Shelwien> and one more guy
<Krugz>
oic
well I guess I tip the balance towards Russian :P
but don't try to speak Russian with me, it won't get
either of us anywhere lol
<Shelwien>
well, the main language here is supposed to be english
<Krugz>
ya I figured :P, fine with me
<Shelwien>
forum is english-only in fact
but half of the people there are russians
<Krugz>
anyhow, no I haven't seen this sound slimmer
program before, I've never actively tried to compress
files unless you're talking about things like .zip/.rar
<Shelwien>
http://forum.websound.ru/index.php?showtopic=24601
<Krugz>
ah it's really too bad I can't read Russian, I've
been trying to relearn but I'm too busy and it's not as
easy as I'd hoped
<Shelwien>
well, then this: http://web.archive.org/web/200611170...bout_us_i.html
somehow they don't have it there anymore
<Krugz>
hmm weird, oh well though
but it's cool that you have your name on it :P
hmm there's a lot of things I'd like to ask, trying to
decide where I'd like to start
what're you currently working on, if anything?
<Shelwien>
a remote backup application i guess, that's one job
and a proprietary audio codec which followed after that
soundslimmer
<Krugz>
hmm are you having any fun with either of them?
<Shelwien>
why, its interesting enough
i'm only responsible for entropy coding (compression)
part for that audio codec, though
also, remote diffing is an interesting problem too
<Krugz>
I've heard that somewhere before..
I might know someone who does something related
<Shelwien>
say, you uploaded a big video file to a server via ftp
then checked its crc, and it doesn't match
and downloading it back to build a patch and fix it on
the server is bad
and uploading it again is even worse
<Krugz>
ya lol you keep losing or distorting it
<Shelwien>
yeah, so there's a workaround
you can build a hash table from the remote file, and
only download it (its small)
and then generate a patch using only the hash table
it would be more redundant than a "precise" patch
but still by a few orders smaller than the whole file
and then, my algorithm even works with stuff like
remuxed video files, kinda like these xdelta patches,
but without having both versions on one side
and i'm thinking about eventually applying it for p2p
as unlike torrent hashing system, my algorithm would
allow to locate usable fragments anywhere
<Krugz>
hmm I sort of understand
I didn't know you could patch a Hash though..
I don't really even think I understand how that's possible lol
sounds impressive, but I've only minorly worked with hashing
<Shelwien>
err... its simple... remote diff can be done with a torrent too
suppose you'd generate a torrent for the local file
and then "download" it over the broken remote file
<Krugz> ya
<Shelwien>
but, torrent isn't really the right solution
because even if you only delete a first byte from remote file
it won't find any matches at all
<Krugz> why's that?
<Shelwien> dunno. its implemented like that
<Krugz> ah ok
<Shelwien>
it only computes hashes for fixed data blocks
so if a block moves somewhere, it won't be found anymore
<Krugz>
Ok, I think I understand
<Shelwien>
and i actually experienced such problems with
ftp/http, especially when proxy is used
<Krugz>
so, the result is a corrupt file, right?
<Shelwien>
yeah, these protocols don't have any checks for corruption
<Krugz>
ok I see
<Shelwien>
but the real problem is when data offsets in
the file don't match
<Krugz>
heh because the program is just running wild?
<Shelwien>
more like because of internet instability
for example, a tcp packet only has 16-bit crc
so when you're trying to transfer like 10GB of data
some error is very likely to appear
<Krugz>
hmm ya that makes sense
<Shelwien>
and its quite possible that ftp client would
send the packet with 1000 bytes of data
and ftp server would receive a (broken) packet with 100
bytes of data, but accidentally the same crc
<Krugz>
well when you say quite possible, isn't it a very low chance?
<Shelwien>
say, we have a 1% of a tcp packet breaking in
transmission (its much higher for wifi or just loaded
networks btw)
<Krugz> ok
<Shelwien>
and 1/65536 of such broken packets would have
accidentally matching crcs
because its 16 bits
<Krugz> ok
<Shelwien>
and a packet size is ~1400 bytes
so, once per 1400*100*65536 bytes
you'd almost certainly have a crc collision and data error
and thats only 9G
<Krugz>
hmm
then isn't it weird that it doesn't happen more
often? Or is it just not noticeable? or maybe I don't
download very large files often?
<Shelwien>
it does actually
why do you think there's par2 and rar recovery?
<Krugz> par2?
<Shelwien>
its a usenet tool similar to rar recovery, but standalone
<Krugz> ah ya, I googled it
<Shelwien>
its just that torrents are mostly used for big files now
<Krugz> ya
<Shelwien>
and torrent downloaders do check their data and retry
when necessary, unlike ftp/http
<Krugz>
so you're working on/made an algorithm that gets around this problem?
<Shelwien>
well, kinda
you see, its not always possible to use torrent on the
remote server
like, hostings sometimes provide shell accounts
but they don't allow to run torrents
<Krugz> ya
<Shelwien>
and main benefit of my algorithm
is that it allows to detect file similarities
(matching fragments etc)
without having full content for any file
<Krugz>
ah I see
sounds complicated lol
<Shelwien>
basically, its like if you could start
downloading torrents of both v1 and v2 videos
and the downloader would automatically detect similar
fragments in the files, and won't download it twice
<Krugz>
oh that's interesting
<Shelwien>
unfortunately, the current torrent hashing
algorithm doesn't allow that
<Krugz> because the hash is too small?
<Shelwien> no, because block offsets are fixed
<Krugz> ah
<Shelwien>
it won't change for torrents even if you could
set it to block size = 1 byte, but different hashing
schemes allow that
<Krugz>
ah I see
<Shelwien>
and that's not all
http://schnaader.info/precomp.html
there're also tools like that
so eventually it would be possible
to find a content for the rar archive you're downloading
in an alternative release in a zip archive
<Krugz>
hmm
that's a cool way to think about it
<Shelwien>
well, it requires a completely different p2p engine though
<Krugz>
ya, but the basics are there already
sounds like a great idea
<Shelwien>
sure
utorrent is really annoying as it downloads the
same file with different names multiple times
<Krugz>
I've used a few different torrent downloaders, I
prefer bittorrent so far. nothing technical about it
though, just feels the best to me :P
<Shelwien>
well, utorrent is mainstream now
after bittorrent developers bought it
<Krugz>
oh they bought it? I really don't keep up with these things..
<Shelwien> i also recently used a client called transmission
<Krugz> hmm I don't think I've ever heard of that one
<Shelwien> its a C++ open source client with most features
<Krugz> ah ok, that's cool
<Shelwien> and it has a headless mode etc
<Krugz> headless?
<Shelwien> you can run it on a remote server without GUI etc
<Krugz> ah ok
<Shelwien> and control via webinterface or something
<Krugz> gotcha
<Shelwien>
there're also some 3rd-party remote control guis
and it has very low memory usage
my server here (ps16893) has a 150M memory limit
but it doesn't look like any problem at all
<Krugz>
so you were thinking of using that open source
client as a base for your idea?
<Shelwien> not really
<Krugz> lol nice server
<Shelwien> just downloading warez
<Shelwien> $5/month
<Krugz> 5$?
<Krugz> for what?
<Shelwien> well, actually $15 but i have a discount
<Shelwien> a virtual server
<Krugz> ah ok
<Shelwien>
unlim traffic/storage
and full root access inside (there's debian)
so its not really different from a full dedicated server
<Krugz>
cool, so you just have your warez available to you
wherever you go
<Shelwien>
yeah, kinda
and as to "base for my idea"
i'm more interested in japanese networks for that
like winny/share/PD
<Krugz> why's that?
<Shelwien>
they have some unique concepts
and torrent might die soon
<Krugz> you think so?
<Shelwien>
piratebay promoting magnet links and DHT etc
and various filters and shapers with torrent support
its just the same story that happened in japan long ago
<Krugz> ?
<Shelwien>
people going in jail for p2p etc
so their p2p uses different protocols from what
europeans accustomized to
<Krugz>
you think piratebay will be jailed?
<Shelwien>
dunno, it seems like they have money to avoid that
<Krugz>
different protocols? Are they really that different?
<Shelwien>
yeah
they're designed with an idea in mind
that you can't prove that a person shares illegal content
even if you downloaded that content from that person
one of their features is something like cache
the p2p client downloads random stuff and stores there
in encrypted form - a few GBs, and then shares it
and you don't know what's there, unless you'd
accidentally try to download a file, which is already cached
<Krugz>
I don't see how that protects them though
<Krugz>
it's a little confusing, but I think I'm just tired
<Shelwien>
as i said, you can't put a person to jail after
demonstrating that you can download an illegal file from
that person's computer because he doesn't control that,
unlike torrents
<Krugz>
I kinda understand
<Shelwien>
well, also encryption and user identification is unique too
its much harder to filter these like they do with
torrents. also file search too - you can never acquire a
list of files shared by specific user
err... i mean, by a user with specific ip
<Krugz>
ah well, not to cut you short but I'm going to
have to go to sleep soon
<Krugz> it's 4am here :P
<Shelwien> well, its 11 here
<Krugz> ah ok
<Shelwien> +2 or something
<Shelwien> ukraine
<Krugz> EST, US east coast
<Krugz>
anyways thanks a lot for the chat, that was fun,
I learned a lot in a short period of time lol
<Shelwien>
well, it'd be useful if you could lurk here
<Krugz>
I'm currently on a holiday break (thanksgiving),
so I'll be around for a while, and I'll try to stay after too
<Shelwien>
well, i can't promise much news, unless somebody talks
to me, but having more people here would be useful
<Krugz>
sure I'd be willing to help out, if I can lol
just curious, do you know anything about the Edsac?
<Shelwien> http://en.wikipedia.org/wiki/EDSAC ?
<Krugz> yup
<Shelwien> are you building a model?
<Krugz>
no nothing that complicated
just writing code for the simulator
I have a hardware architecture class, and an early
assignment was to program for this Edsac simulator
<Shelwien> nice too
<Krugz>
our professor is very lax on the due dates, so I can still do it
<Shelwien> i'd write a compiler though
<Krugz> lolol I'm not good enough for that kind of thing
<Shelwien>
its not really harder than interpreter
but works much faster
like JIT etc
<Krugz>
I honestly think it would be out of my league
I'm way behind on what I should know in computer
science, my own laziness
<Shelwien>
http://www-jpc.physics.ox.ac.uk/doom.html
like that
<Krugz>
ah ya, that would take me a while to make lol
the applet seems to be down, but I know what you
were trying to show me
<Shelwien>
nah, i checked and it worked