-1

I need to write a very large amount of Numbers to a File in a very memory efficient way with java.

The Numbers are in range from 0 to 2,132,487 so that a Short data type(16bit) would not be enough but a Integer(32bit) would be a waste of space.

The most efficient way would be to use 22Bits for each Number (2²²= 4,194,394)

How could i implement that?

Johannes Fuchs
  • 1
  • 1
  • Please list what you have tried already. – Brien Foss Jan 19 '18 at 17:39
  • Chances are using the simple thing (32 bit) and compressing the resulting stream is "good enough" for what you need. – Joachim Sauer Jan 19 '18 at 17:41
  • *"very large amount"* and *"very memory efficient"* are *very* broad. 10 numbers, 10000000 numbers or 10000000000000 numbers? – luk2302 Jan 19 '18 at 17:43
  • ~90,000,000 Numbers. I dont wandt to use any bits that dont encode any information but i dont need compression as this would probably get too complicated – Johannes Fuchs Jan 19 '18 at 17:48
  • If you want to do this kind of low level stuff, you'll probably have an easier time using C or C++. – Gab Jan 19 '18 at 17:52
  • gzip compression for example is very basic and widely supported, you should at least give it a try. You are basically trying to do the compression by hand, which will cost you quite some time and brain power and is not even guaranteed to be fast. – luk2302 Jan 19 '18 at 17:53

1 Answers1

1

It would be very difficult to allocate 22 bits, but you could allocate 24 bits (3 bytes).

A simple way would be to convert each number to a byte array. Using ByteBuffer from https://stackoverflow.com/a/2183279/708790.

int i = 3763; // For example

byte[] intBytes = ByteBuffer.allocate(4).putInt(i).array();
byte[] lessBytes = Arrays.copyOfRange(intBytes,1, 4);

for (byte b : lessBytes) {
    System.out.format("0x%x ", b);
}

Which gives 0x0 0xe 0xb3.

This assumes that the most significant byte is not used - as mentioned in the question.

When reading the file back, make sure to pad the bytes back to 4 before converting back to an int.

starf
  • 1,053
  • 11
  • 15