Results 1 to 7 of 7

Thread: Data Visualization tools

  1. #1
    Member
    Join Date
    Jun 2020
    Location
    Meadowood
    Posts
    6
    Thanks
    2
    Thanked 2 Times in 2 Posts

    Data Visualization tools

    Hello all. I recently took on a project where I was compressing large amounts of scientific data. Adjacent to that project, I was wondering, what is the best way to visualize huge data files(like 2 GB of floats) that have low dimensionality(<5 variables)? Ideally I am picturing something kinda like looking at an audio waveform, where each variable can be visualized individually and the user can scrub along the waveform along with zoom in to see things more clearly. However, when I try to look up data visualization tools, I keep getting stumped by results for high dimensionality visualization. I don't need many dimensions, just the ability to zoom and scroll in one dimension with many points!(one of the variables is a time variable, so that would be the independent variable for these visualized waveforms).

    Thank you for any recommendations, I will gladly research any leads.

    Alex

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,946
    Thanks
    294
    Thanked 1,286 Times in 728 Posts
    There's color + 2D screen + time so you can visualize up to 4D directly as an animation (5D with 3D->2D projection).
    Or you can simply combine individual plots of each column (in different colors).
    Compression-wise, there're also methods like this: http://www.fantascienza.net/leonardo...tatistics.html

    There're many posts about this on the net, eg:
    https://towardsdatascience.com/the-a...a-6c7202990c57
    https://en.wikipedia.org/wiki/Data_v..._visualization
    But in the end you have to choose the best method based on your data.

  3. #3
    Member JamesWasil's Avatar
    Join Date
    Dec 2017
    Location
    Arizona
    Posts
    95
    Thanks
    89
    Thanked 17 Times in 16 Posts
    There was a program called File View or FV.exe from Mahoney on his site. It's rather old but it might be useful still for data visualization, although the above is more certain.

  4. #4
    Member
    Join Date
    Jun 2020
    Location
    Meadowood
    Posts
    6
    Thanks
    2
    Thanked 2 Times in 2 Posts
    Thank you for your responses. I don't think I was clear with my issue though.

    I am ideally hoping to visualize my data with a scatterplot, and it is a good idea by Shelwein to use different colors to represent the different sensor readings against time. My main issue is that I have too many samples to visualize all at once effectively(on the order of 100,000,000 points). Also, I would like to be able to zoom and scrub to different points on the visualized chart. Effectively, I think this would require downsampling the signal for visualizing large time scales, and reconstructing it for zooming in to specific points. Are there any tools that are able to do this smoothly?

    Thanks for any more help!

  5. Thanks:

    JamesWasil (31st July 2020)

  6. #5
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,946
    Thanks
    294
    Thanked 1,286 Times in 728 Posts

  7. Thanks:

    JamesWasil (31st July 2020)

  8. #6
    Member
    Join Date
    Jun 2020
    Location
    Meadowood
    Posts
    6
    Thanks
    2
    Thanked 2 Times in 2 Posts
    Exactly! Thank you so much. For some reason google searches kept pointing me to tools for many dimensional data, and not for many points. I will work on implementing this.

  9. Thanks:

    JamesWasil (31st July 2020)

  10. #7
    Member JamesWasil's Avatar
    Join Date
    Dec 2017
    Location
    Arizona
    Posts
    95
    Thanks
    89
    Thanked 17 Times in 16 Posts
    Google search results are no longer good after 2013. They return bad results and the opposite of what you search for, or things that are not relevant. Part of this is due to commercialization and ads, but other reasons are censorship and other agendas that make things no longer findable there and search results from Google are now often useless.

    Usually asking here on the forum, someone will know. Stackexchange sometimes, and if you have to use a search engine, duckduckgo, Bing, dogpile, and Yahoo (for older but still semi-relevant results) will be better than Google for anything.

Similar Threads

  1. Tools to display lz compressed data content
    By smjohn1 in forum Data Compression
    Replies: 2
    Last Post: 4th May 2020, 23:40
  2. Replies: 9
    Last Post: 18th May 2014, 06:50
  3. Non open source Data compression Tools
    By ehsansad in forum Data Compression
    Replies: 9
    Last Post: 22nd September 2011, 00:41
  4. Interesting tools
    By lunaris in forum Data Compression
    Replies: 2
    Last Post: 25th August 2009, 23:50

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •