At Thu, 25 May 2006 13:42:11 +0400, Bulat Ziganshin wrote:
Hello Jeremy,
Monday, May 22, 2006, 12:20:54 AM, you wrote:
For my own needs, I cabalized and debianized the Streams library. It generates binary debs for ghc6 and hugs -- but I think the hugs version is broken. In any case, it is a start, you can download the packaging at:
can i include your work in the library itself?
Absolutely.
is it better to include 'debian' directory to my archive or left this to the debian packagers?
If someone volunteers to maintain the package -- then it is probably better to not keep a copy of the debian directory in your archive -- because it will often be out of date and confuse people -- and debian users will be able to get the debianized source easily by typing, 'apt-get source haskell-streams'.
On the other hand -- if there is no one officially maintaing it -- it would be useful to provide the debian directory (with a disclaimer) so that debian users can easily build and install the .deb, since subverting the debian package system tends to lead to long-term complications.
can you say how you are use my library? it's both interesting for me and can be helpful in deciding how it should be further developed
I am using it to serialize/deserialize haskell data structures so I can store them in a Berkeley database.
To get them into BDB I need to convert the haskell data structure into a C structure that looks like this:
struct __db_dbt { void *data; /* Key/data */ u_int32_t size; /* key/data length */ };
Currently I am doing it like this -- but this will clearly fail if the serialized data structure is longer than 512 bytes...
withDBT :: (Binary a) => a -> (Ptr DBT -> IO b) -> IO b withDBT thedata f = allocaBytes #{size DBT} $ \dbtPtr -> allocaBytes 512 $ \dataPtr -> do h <- openMemBuf dataPtr 512 withByteAlignedLE h $ flip put_ thedata wrote <- vTell h vClose h #{poke DBT, data} dbtPtr (castPtr dataPtr) #{poke DBT, size} dbtPtr ((fromIntegral wrote) :: Int) f dbtPtr
I don't really need the file-system interface for this project -- what would be nice is something like 'withCStringLen' and 'peekCString' for the encode/decode functions:
type PtrLen a = (Ptr a, Int) encodePtrLen :: (Binary a) => a -> (PtrLen a -> IO b) -> IO b decodePtr :: (Binary a) => Ptr a -> IO a
I could simulate this by using 'encode' to convert the data structure to a String and then use 'withCStringLen' to get the pointer and length -- but having the intermediate String seems like it could be a big performance hit.
Two alternative ideas are:
(1) accurately pre-calculate the size of the serialized structure and allocate the correct amount of memory from the start
(2) start with a 'guess' and realloc the memory if the initial guess is too small.
Both of those alternatives have their own problems -- so I think only testing will tell what works best...
I have not looked at the library exhaustively, so if there is already a good way to do this, let me know.
Thanks! j.
Hello Jeremy,
Sunday, May 28, 2006, 1:29:02 AM, you wrote:
can i include your work in the library itself?
Absolutely.
thanks
is it better to include 'debian' directory to my archive or left this to the debian packagers?
If someone volunteers to maintain the package -- then it is probably
afaiu, you say about maintaining their debian ports? it seems that i should include these files now and omit them when someone will start to maintain the package?
would be useful to provide the debian directory (with a disclaimer) so
what disclaimer?
To get them into BDB I need to convert the haskell data structure into a C structure that looks like this:
struct __db_dbt { void *data; /* Key/data */ u_int32_t size; /* key/data length */ };
Currently I am doing it like this -- but this will clearly fail if the serialized data structure is longer than 512 bytes...
withDBT :: (Binary a) => a -> (Ptr DBT -> IO b) -> IO b withDBT thedata f = allocaBytes #{size DBT} $ \dbtPtr -> allocaBytes 512 $ \dataPtr -> do h <- openMemBuf dataPtr 512 withByteAlignedLE h $ flip put_ thedata wrote <- vTell h vClose h #{poke DBT, data} dbtPtr (castPtr dataPtr) #{poke DBT, size} dbtPtr ((fromIntegral wrote) :: Int) f dbtPtr
i will prefer to split it into two parts. and, DBD-interfacing part can be also implemented using binary i/o:
withDBT00 :: Ptr a -> Int -> (Ptr DBT -> IO b) -> IO b withDBT00 f buf size = do h <- createMemBuf 20 >>= openByteAlignedLE put_ h buf putWord32 h size vRewind h (dbt,_) <- vReceiveBuf h result <- f dbt vClose h return result
withDBT f thedata = encodeMemBufLE (withDBT00 f) thedata
I don't really need the file-system interface for this project -- what would be nice is something like 'withCStringLen' and 'peekCString' for the encode/decode functions:
type PtrLen a = (Ptr a, Int) encodePtrLen :: (Binary a) => a -> (PtrLen a -> IO b) -> IO b decodePtr :: (Binary a) => Ptr a -> IO a
encodeMemBufLE f thedata = do h <- createMemBuf 512 >>= openByteAlignedLE put_ h thedata vRewind h (buf,size) <- vReceiveBuf h result <- f buf size vClose h return result
decodeMemBufLE buf size = do h <- openMemBuf buf size >>= openByteAlignedLE result <- get h vClose h return result
but it will work only with Streams 0.1. you have spotted the problem that there is no official way to get access to the whole buffer contents if buffer was created with createMemBuf.
I could simulate this by using 'encode' to convert the data structure to a String and then use 'withCStringLen' to get the pointer and length -- but having the intermediate String seems like it could be a big performance hit.
Strings are slow by itself and moreover 'encode' has O(n^2) complexity
Two alternative ideas are:
(1) accurately pre-calculate the size of the serialized structure and allocate the correct amount of memory from the start
it's good idea to have 'binarySize :: Binary a => a->Int' function, although using it will halve the speed, so for you it's not the best solution
(2) start with a 'guess' and realloc the memory if the initial guess is too small.
createMemBuf does exactly this :) it's for why the whole Streams part exist. actually i have started with trivial
instance ByteStream Handle where vPutByte h n = do hPutChar h (chr (fromEnum n)) vGetByte h = do c <- hGetChar h return $! (toEnum (ord c))
and only after Binary part was enough matured, i goes to adding all those fancy Stream features
I have not looked at the library exhaustively, so if there is already a good way to do this, let me know.
the way exist but it's not guaranteed by library interface and uses my knowledge of library internals. i will add interface which guarantees access to full buffer's contents. after that, 'encodeMemBufLE' can be written using official library capabilities. i will also add encodeMemBuf*/decodeMemBuf* to the lib, although i'm not sure that these functions are universal enough
Hello Bulat,
Sunday, May 28, 2006, 2:44:37 PM, you wrote:
type PtrLen a = (Ptr a, Int) encodePtrLen :: (Binary a) => a -> (PtrLen a -> IO b) -> IO b decodePtr :: (Binary a) => Ptr a -> IO a
Finally i've implemented the following (you then would use 'withForeignPtr' to work with contents of ForeignPtr):
-- ----------------------------------------------------------------------------- -- Encode/decode contents of memory buffer
encodePtr :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtrLE :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtrBitAligned :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtrBitAlignedLE :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtr = encodePtr' openByteAligned encodePtrLE = encodePtr' openByteAlignedLE encodePtrBitAligned = encodePtr' openBitAligned encodePtrBitAlignedLE = encodePtr' openBitAlignedLE
decodePtr :: (Binary a, Integral size) => Ptr x -> size -> IO a decodePtrLE :: (Binary a, Integral size) => Ptr x -> size -> IO a decodePtrBitAligned :: (Binary a, Integral size) => Ptr x -> size -> IO a decodePtrBitAlignedLE :: (Binary a, Integral size) => Ptr x -> size -> IO a
decodePtr = decodePtr' openByteAligned decodePtrLE = decodePtr' openByteAlignedLE decodePtrBitAligned = decodePtr' openBitAligned decodePtrBitAlignedLE = decodePtr' openBitAlignedLE
-- Universal function what encodes data with any alignment encodePtr' open thedata = do h <- createMemBuf 512 >>= open put_ h thedata vFlush h vRewind h (buf,size) <- vReceiveBuf h READING -- FIXME: MemBuf-implementation specific fptr <- newForeignPtr finalizerFree (castPtr buf) -- FIXME: also MemBuf-implementation specific return (fptr,size)
-- Universal function what decodes data written with any alignment decodePtr' open ptr size = do h <- openMemBuf ptr size >>= open result <- get h vClose h return result
debian-haskell@lists.urchin.earth.li