Support big arrays

[dtolnay]

Servo does this to support big arrays in one of their proc macros:

https://github.com/servo/heapsize/blob/44e86d6d48a09c9cbc30a122bc8725b188d017b2/derive/lib.rs#L36-L41

Let's do the same but only if the size of the array exceeds our biggest builtin impl.

Thanks @nox.

[dtolnay]

One relatively easy workaround for serialization is coercing to a slice:

struct S {
    #[serde(serialize_with = "<[_]>::serialize")]
    arr: [u8; 256],
}

Deserialization is still annoying I think.

[sdleffler]

Hey folks, this feature is important to me because I'd like to be able to serialize a 512-bit hash (so, 64 bytes) and because the serde impls necessarily only go up to [u8; 32] I cannot serialize a [u8; 64].

As workarounds I'm considering using [[u8; 32]; 2], GenericArray or just lazily using a Box<[u8]>. I'm piqued by the idea of the workaround shown above - @dtolnay did you ever find a deserialization workaround?

Would it be okay to add impls up to 64? Or is there some reason that hasn't been done?

[clarfonthey]

In the meantime, perhaps we should add impls for the sizes that arrayvec provides?

impl<T> Array for [T; 40]
impl<T> Array for [T; 48]
impl<T> Array for [T; 50]
impl<T> Array for [T; 56]
impl<T> Array for [T; 64]
impl<T> Array for [T; 72]
impl<T> Array for [T; 96]
impl<T> Array for [T; 100]
impl<T> Array for [T; 128]
impl<T> Array for [T; 160]
impl<T> Array for [T; 192]
impl<T> Array for [T; 200]
impl<T> Array for [T; 224]
impl<T> Array for [T; 256]
impl<T> Array for [T; 384]
impl<T> Array for [T; 512]
impl<T> Array for [T; 768]
impl<T> Array for [T; 1024]
impl<T> Array for [T; 2048]
impl<T> Array for [T; 4096]
impl<T> Array for [T; 8192]
impl<T> Array for [T; 16384]
impl<T> Array for [T; 32768]
impl<T> Array for [T; 65536]

[dtolnay]

@clarcharr I would prefer to stick with what the standard library does, which is 0 to 32 (inclusive).

[dtolnay]

Here is a workaround for deserializing.

#[macro_use]
extern crate serde_derive;

extern crate serde;
extern crate serde_json;

use std::fmt;
use std::marker::PhantomData;
use serde::ser::{Serialize, Serializer, SerializeTuple};
use serde::de::{Deserialize, Deserializer, Visitor, SeqAccess, Error};

trait BigArray<'de>: Sized {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
        where S: Serializer;
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
        where D: Deserializer<'de>;
}

macro_rules! big_array {
    ($($len:expr,)+) => {
        $(
            impl<'de, T> BigArray<'de> for [T; $len]
                where T: Default + Copy + Serialize + Deserialize<'de>
            {
                fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
                    where S: Serializer
                {
                    let mut seq = serializer.serialize_tuple(self.len())?;
                    for elem in &self[..] {
                        seq.serialize_element(elem)?;
                    }
                    seq.end()
                }

                fn deserialize<D>(deserializer: D) -> Result<[T; $len], D::Error>
                    where D: Deserializer<'de>
                {
                    struct ArrayVisitor<T> {
                        element: PhantomData<T>,
                    }

                    impl<'de, T> Visitor<'de> for ArrayVisitor<T>
                        where T: Default + Copy + Deserialize<'de>
                    {
                        type Value = [T; $len];

                        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
                            formatter.write_str(concat!("an array of length ", $len))
                        }

                        fn visit_seq<A>(self, mut seq: A) -> Result<[T; $len], A::Error>
                            where A: SeqAccess<'de>
                        {
                            let mut arr = [T::default(); $len];
                            for i in 0..$len {
                                arr[i] = seq.next_element()?
                                    .ok_or_else(|| Error::invalid_length(i, &self))?;
                            }
                            Ok(arr)
                        }
                    }

                    let visitor = ArrayVisitor { element: PhantomData };
                    deserializer.deserialize_tuple($len, visitor)
                }
            }
        )+
    }
}

big_array! {
    40, 48, 50, 56, 64, 72, 96, 100, 128, 160, 192, 200, 224, 256, 384, 512,
    768, 1024, 2048, 4096, 8192, 16384, 32768, 65536,
}

#[derive(Serialize, Deserialize)]
struct S {
    #[serde(with = "BigArray")]
    arr: [u8; 64],
}

fn main() {
    let s = S { arr: [1; 64] };
    let j = serde_json::to_string(&s).unwrap();
    println!("{}", j);
    serde_json::from_str::<S>(&j).unwrap();
}

[Binero]

As long as you're not working with primes:

#[derive(Serialize, Deserialize, Debug)]
struct MyStruct {
    data: [[u8; 32]; 16],
}

impl MyStruct {
    fn data(&self) -> &[u8; 512] {
        use std::mem::transmute;
        unsafe { transmute(&self.data) }
    }
}

This is a pretty neat workaround for when never expect a human to use the serialised version (e.g. bincode), as it creates a nested array. Added bonus: it also works for Debug, PartialEq, etc.

[Boscop]

FWIW, I use this:

use serde::{Serialize, Serializer};

pub fn serialize_array<S, T>(array: &[T], serializer: S) -> Result<S::Ok, S::Error>
where S: Serializer, T: Serialize {
	array.serialize(serializer)
}

#[macro_export]
macro_rules! serde_array { ($m:ident, $n:expr) => {
	pub mod $m {
		use std::{ptr, mem};
		use serde::{Deserialize, Deserializer, de};
		pub use $crate::serialize_array as serialize;
		use super::*;

		pub fn deserialize<'de, D, T>(deserializer: D) -> Result<[T; $n], D::Error>
		where D: Deserializer<'de>, T: Deserialize<'de> + 'de {
			let slice: Vec<T> = Deserialize::deserialize(deserializer)?;
			if slice.len() != $n {
				return Err(de::Error::custom("input slice has wrong length"));
			}
			unsafe {
				let mut result: [T; $n] = mem::uninitialized();
				for (src, dst) in slice.into_iter().zip(&mut result[..]) {
					ptr::write(dst, src);
				}
				Ok(result)
			}
		}
	}
}}

serde_array!(a64, 64);
serde_array!(a120, 120);
serde_array!(a128, 128);
serde_array!(a384, 384);

And then

struct Foo {
	#[serde(with = "a128")]
	bar: [f32; 128],
}