правильно так:
<StructLayout(LayoutKind.Sequential, Pack:=1, CharSet:=CharSet.Ansi)> Public Structure inbuf
<MarshalAs(UnmanagedType.U4)> Public length As UInt32
<MarshalAs(UnmanagedType.U4)> Public svcid As UInt32
<MarshalAs(UnmanagedType.U2)> Public infosvcid As UInt16
End Structure
<StructLayout(LayoutKind.Sequential, Pack:=1)> Public Structure outbuf
<MarshalAs(UnmanagedType.U4)> Public length As UInt32
<MarshalAs(UnmanagedType.ByValArray, SizeConst:=256)> Public data() As Byte
End Structure
маршалить структуры, в обертке указывать их byref
Public Declare Function serialNum Lib "op20pt32.dll" Alias "PassThruIoctl" (ByVal ChannelID As UInt32, ByVal IoctlID As UInt32, ByRef pInput As inbuf, ByRef pOutput As outbuf) As Integer
и вызов так:
serialNum(DevID, TX_IOCTL_APP_SERVICE, inbuf1, outbuf1)
managed
<->unmanaged
.